Investigating SerializedProperty, custom controls, OnSceneGUI, Handles and multi-editing

I've been developing tools for the Unity editor for quite some time now, but I keep stumbling over basics. Today, I went over a few concepts and thought I'd share my results with the community. Some of this may help beginners understand different systems and for advanced users, I would be happy to ask some questions myself.

My test setup consists of a MonoBehaviour component added to a GameObject in the scene. The GameObject is then turned into a prefab and duplicated. We can now test the following cases:

• Edit a prefab instance in the scene via the inspector
• Edit a prefab in prefab mode
• Multi-Edit prefab instances in the scene
• Edit via SceneView Handles

And here is my test code:

Code (CSharp):

1. using UnityEditor;
2. using UnityEngine;
3.  
4. public class PrefabEditTest : MonoBehaviour
5. {
6.     public Vector3 someValue;
7.     public bool useCustomSceneViewGUICallback = true;
8. }
9.  
10. [CustomEditor(typeof(PrefabEditTest))]
11. [CanEditMultipleObjects]
12. public class PrefabEditTestEditor : Editor
13. {
14.     public override void OnInspectorGUI()
15.     {
16.         DrawToggle();
17.         DrawSerializedProperty();
18.         DrawCustomControl();
19.     }
20.  
21.     private void DrawToggle()
22.     {
23.         // This is only here to easily switch between using Editor.SceneViewGUI and
24.         // a custom delegate added to SceneView.duringSceneGui.
25.         serializedObject.Update();
26.         var toggleProp = serializedObject.FindProperty("useCustomSceneViewGUICallback");
27.         EditorGUILayout.PropertyField(toggleProp);
28.         serializedObject.ApplyModifiedProperties();
29.     }
30.  
31.     private void DrawSerializedProperty()
32.     {
33.         // Regular inspector fields via SerializedProperty.
34.         // Everything works and is handled automatically:
35.         // - Undo
36.         // - Prefab instance overrides
37.         // - Editing in prefab mode
38.         // - Setting scene dirty and saving changes.
39.         // - Multi-object-editing
40.  
41.         // Update the object representation from serialization. This ensures that the MonoBehaviour
42.         // class instance of PrefabEditTest will have the most recent values that are actually stored on disk.
43.         serializedObject.Update();
44.         SerializedProperty prop = serializedObject.FindProperty("someValue");
45.  
46.         // This automatically records undo und handles prefab overrides (bold text and blue markings).
47.         // It basically does everything Unity supports in one call.
48.         EditorGUILayout.PropertyField(prop);
49.  
50.         // Make sure we write changed values to disk.
51.         serializedObject.ApplyModifiedProperties();
52.     }
53.  
54.     private void DrawCustomControl()
55.     {
56.         // Try to handle the same functionality with individual steps (because sometimes we need control).
57.  
58.         // This time we directly edit the reference to the target object (our MonoBehaviour class).
59.         PrefabEditTest script = (PrefabEditTest)target;
60.  
61.         // However, we still need the SerializedProperty to retrieve information such as prefab overrides.
62.         SerializedProperty prop = serializedObject.FindProperty("someValue");
63.  
64.         float height = EditorGUIUtility.wideMode ? EditorGUIUtility.singleLineHeight : EditorGUIUtility.singleLineHeight * 2f;
65.         Rect rect = EditorGUILayout.GetControlRect(true, height);
66.  
67.         // Handles bold label when this property is a prefab override and shows mixed values indicator when multi-editing.
68.         // However, small ISSUE_A: When a Vector3 component is overriden but only the x value is different,
69.         // it makes all three components bold, although only x should be shown as a bold override.
70.         var label = EditorGUI.BeginProperty(rect, new GUIContent(prop.displayName), prop);
71.  
72.         // Only do something if the values from our inspector control have actually changed.
73.         EditorGUI.BeginChangeCheck();
74.  
75.         // Do not write the value back to the script immediately, because we first need to record an Undo step.
76.         var value = EditorGUI.Vector3Field(rect, label, script.someValue);
77.         if (EditorGUI.EndChangeCheck())
78.         {
79.             // Handle multi-object editing. Apply changes to all instances, not only the first one.
80.             for (int i = 0; i < targets.Length; i++)
81.             {
82.                 var target = targets[i];
83.                 script = (PrefabEditTest)target;
84.  
85.                 // Record undo of previous values on object (works nicely).
86.                 Undo.RecordObject(target, "Change Value");
87.  
88.                 // Now change the values on the MonoBehaviour class.
89.                 script.someValue = value;
90.  
91.                 #region ISSUE_B
92.                 // Documentation says: "If this method is not called, changes made to the instance are lost."
93.                 // However, in my tests, this does not change the behaviour at all, changes don't seem to be lost if I don't use it.
94.                 // Also, should I only call this on prefab instances in the scene or can I safely call it on all GameObjects or Prefabs?
95.                 // For Example, should I check for '!EditorUtility.IsPersistent(target)' first?
96.                 PrefabUtility.RecordPrefabInstancePropertyModifications(target);
97.                 #endregion
98.             }
99.  
100.             #region ISSUE_C
101.             // At this point, most things seem to work, except:
102.             // When multi-editing targets with multiple different values, their values all become the same under the hood (which is correct),
103.             // however, when moving the slider or changing values in the field, the inspector still shows "mixed values" until the
104.             // target is deselected and selected again. Basically, prop.hasMultipleDifferentValues is still true and only reset
105.             // once the editor is recreated. Updating or applying the serializedObject does not help.
106.             // To fix this, it seems, I need to assign the values back to the SerializedProperty and apply the SerializedObject.
107.             prop.vector3Value = script.someValue;
108.             serializedObject.ApplyModifiedProperties();
109.             #endregion
110.         }
111.  
112.         EditorGUI.EndProperty();
113.     }
114.  
115.     public void OnSceneGUI()
116.     {
117.         // Using serializedObject in this method results in an error:
118.         // "The serializedObject should not be used inside OnSceneGUI or OnPreviewGUI. Use the target property directly instead."
119.         // It seems, multi-object-editing cannot be easily implemented. At least, if we want to draw
120.         // a single handle control and change the values of multiple instances.
121.         // Instead, this method will draw multiple handles, one for each instance separately.
122.  
123.         PrefabEditTest script = (PrefabEditTest)target;
124.  
125.         // Disable this example if we are using the custom callback.
126.         if (script.useCustomSceneViewGUICallback)
127.             return;
128.  
129.         EditorGUI.BeginChangeCheck();
130.         Vector3 value = Handles.Slider(script.someValue, Vector3.right);
131.         if (EditorGUI.EndChangeCheck())
132.         {
133.             // Next error: when trying to use the targets array to support multi-editing:
134.             // "The targets array should not be used inside OnSceneGUI or OnPreviewGUI. Use the single target property instead."
135.  
136.             // Record undo of previous values on object (works nicely).
137.             Undo.RecordObject(target, "Change Value");
138.  
139.             // Now change the values on the MonoBehaviour class.
140.             script.someValue = value;
141.  
142.             #region ISSUE_B
143.             // Documentation says: "If this method is not called, changes made to the instance are lost."
144.             // However, in my tests, this does not change the behaviour at all, changes don't seem to be lost if I don't use it.
145.             // Also, should I only call this on prefab instances in the scene or can I safely call it on all GameObjects or Prefabs?
146.             // For Example, should I check for '!EditorUtility.IsPersistent(target)' first?
147.             PrefabUtility.RecordPrefabInstancePropertyModifications(target);
148.             #endregion
149.         }
150.  
151.         #region ISSUE_D
152.         // Using Handles in OnSceneGUI this way, results in the basic functionality working, but multi-object-editing is not fully supported.
153.         #endregion
154.     }
155.  
156.     private void OnEnable()
157.     {
158.         // Here we add a callback to the delegate, which hooks into the
159.         // same loop as OnSceneGUI, but allows us to use SerializedProperty.
160.         SceneView.duringSceneGui += MySceneGUI;
161.     }
162.  
163.     private void OnDisable()
164.     {
165.         SceneView.duringSceneGui -= MySceneGUI;
166.     }
167.  
168.     private void MySceneGUI(SceneView obj)
169.     {
170.         PrefabEditTest script = (PrefabEditTest)target;
171.  
172.         // Disable this example if we are not using the custom callback.
173.         if (!script.useCustomSceneViewGUICallback)
174.             return;
175.  
176.         // With a custom callback, we can support multi-editing simply by using serializedObject.
177.         serializedObject.Update();
178.  
179.         // This also makes the rest of the code very easy.
180.         SerializedProperty prop = serializedObject.FindProperty("someValue");
181.         prop.vector3Value = Handles.Slider(prop.vector3Value, Vector3.right);
182.  
183.         serializedObject.ApplyModifiedProperties();
184.     }
185. }
186.  

So here are my own questions:

ISSUE_A

When I use a custom control to draw "Some Value" I use BeginProperty and EndProperty to make sure that the label is drawn bold and the blue line is added to indicate a prefab override. However, it makes all three vector components x, y and z bold as well, whereas the SerializedProperty implementation only highlights x. How do I implement this correctly? Or is this a bug in BeginProperty?

ISSUE_B
The documentation advises to call PrefabUtility.RecordPrefabInstancePropertyModifications(target) after changes have been made to a prefab instance. In my test, however, there was no observable difference. Values were still saved. Do I really need to call this? Are there any cases which I haven't tested which are affected by this? And do I need to first test, if the target is actually a prefab instance, e.g. use !EditorUtility.IsPersistent(target) ?

Overall I feel like there are a few things I would like to see in the documentation. For example, why can't I use serializedObject in Editor.OnSceneGUI, but it works fine in SceneView.duringSceneGUI? And what are the recommended approaches for Handles and multi-editing?

 

I am having this issue with OnSceneGUI and multi-object editing too. However, I am just trying to draw the different instances.

 

Wow, this is so well documented and still no reply. I'm interested in the answer too. This would be the most important information from Unity to understand generic patterns to follow when a tool is implemented for Unity.