Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

Investigating SerializedProperty, custom controls, OnSceneGUI, Handles and multi-editing

Discussion in 'Immediate Mode GUI (IMGUI)' started by Xarbrough, Dec 6, 2019.

  1. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    1,173
    I've been developing tools for the Unity editor for quite some time now, but I keep stumbling over basics. Today, I went over a few concepts and thought I'd share my results with the community. Some of this may help beginners understand different systems and for advanced users, I would be happy to ask some questions myself.

    My test setup consists of a MonoBehaviour component added to a GameObject in the scene. The GameObject is then turned into a prefab and duplicated. We can now test the following cases:
    • Edit a prefab instance in the scene via the inspector
    • Edit a prefab in prefab mode
    • Multi-Edit prefab instances in the scene
    • Edit via SceneView Handles
    And here is my test code:

    Code (CSharp):
    1. using UnityEditor;
    2. using UnityEngine;
    3.  
    4. public class PrefabEditTest : MonoBehaviour
    5. {
    6.     public Vector3 someValue;
    7.     public bool useCustomSceneViewGUICallback = true;
    8. }
    9.  
    10. [CustomEditor(typeof(PrefabEditTest))]
    11. [CanEditMultipleObjects]
    12. public class PrefabEditTestEditor : Editor
    13. {
    14.     public override void OnInspectorGUI()
    15.     {
    16.         DrawToggle();
    17.         DrawSerializedProperty();
    18.         DrawCustomControl();
    19.     }
    20.  
    21.     private void DrawToggle()
    22.     {
    23.         // This is only here to easily switch between using Editor.SceneViewGUI and
    24.         // a custom delegate added to SceneView.duringSceneGui.
    25.         serializedObject.Update();
    26.         var toggleProp = serializedObject.FindProperty("useCustomSceneViewGUICallback");
    27.         EditorGUILayout.PropertyField(toggleProp);
    28.         serializedObject.ApplyModifiedProperties();
    29.     }
    30.  
    31.     private void DrawSerializedProperty()
    32.     {
    33.         // Regular inspector fields via SerializedProperty.
    34.         // Everything works and is handled automatically:
    35.         // - Undo
    36.         // - Prefab instance overrides
    37.         // - Editing in prefab mode
    38.         // - Setting scene dirty and saving changes.
    39.         // - Multi-object-editing
    40.  
    41.         // Update the object representation from serialization. This ensures that the MonoBehaviour
    42.         // class instance of PrefabEditTest will have the most recent values that are actually stored on disk.
    43.         serializedObject.Update();
    44.         SerializedProperty prop = serializedObject.FindProperty("someValue");
    45.  
    46.         // This automatically records undo und handles prefab overrides (bold text and blue markings).
    47.         // It basically does everything Unity supports in one call.
    48.         EditorGUILayout.PropertyField(prop);
    49.  
    50.         // Make sure we write changed values to disk.
    51.         serializedObject.ApplyModifiedProperties();
    52.     }
    53.  
    54.     private void DrawCustomControl()
    55.     {
    56.         // Try to handle the same functionality with individual steps (because sometimes we need control).
    57.  
    58.         // This time we directly edit the reference to the target object (our MonoBehaviour class).
    59.         PrefabEditTest script = (PrefabEditTest)target;
    60.  
    61.         // However, we still need the SerializedProperty to retrieve information such as prefab overrides.
    62.         SerializedProperty prop = serializedObject.FindProperty("someValue");
    63.  
    64.         float height = EditorGUIUtility.wideMode ? EditorGUIUtility.singleLineHeight : EditorGUIUtility.singleLineHeight * 2f;
    65.         Rect rect = EditorGUILayout.GetControlRect(true, height);
    66.  
    67.         // Handles bold label when this property is a prefab override and shows mixed values indicator when multi-editing.
    68.         // However, small ISSUE_A: When a Vector3 component is overriden but only the x value is different,
    69.         // it makes all three components bold, although only x should be shown as a bold override.
    70.         var label = EditorGUI.BeginProperty(rect, new GUIContent(prop.displayName), prop);
    71.  
    72.         // Only do something if the values from our inspector control have actually changed.
    73.         EditorGUI.BeginChangeCheck();
    74.  
    75.         // Do not write the value back to the script immediately, because we first need to record an Undo step.
    76.         var value = EditorGUI.Vector3Field(rect, label, script.someValue);
    77.         if (EditorGUI.EndChangeCheck())
    78.         {
    79.             // Handle multi-object editing. Apply changes to all instances, not only the first one.
    80.             for (int i = 0; i < targets.Length; i++)
    81.             {
    82.                 var target = targets[i];
    83.                 script = (PrefabEditTest)target;
    84.  
    85.                 // Record undo of previous values on object (works nicely).
    86.                 Undo.RecordObject(target, "Change Value");
    87.  
    88.                 // Now change the values on the MonoBehaviour class.
    89.                 script.someValue = value;
    90.  
    91.                 #region ISSUE_B
    92.                 // Documentation says: "If this method is not called, changes made to the instance are lost."
    93.                 // However, in my tests, this does not change the behaviour at all, changes don't seem to be lost if I don't use it.
    94.                 // Also, should I only call this on prefab instances in the scene or can I safely call it on all GameObjects or Prefabs?
    95.                 // For Example, should I check for '!EditorUtility.IsPersistent(target)' first?
    96.                 PrefabUtility.RecordPrefabInstancePropertyModifications(target);
    97.                 #endregion
    98.             }
    99.  
    100.             #region ISSUE_C
    101.             // At this point, most things seem to work, except:
    102.             // When multi-editing targets with multiple different values, their values all become the same under the hood (which is correct),
    103.             // however, when moving the slider or changing values in the field, the inspector still shows "mixed values" until the
    104.             // target is deselected and selected again. Basically, prop.hasMultipleDifferentValues is still true and only reset
    105.             // once the editor is recreated. Updating or applying the serializedObject does not help.
    106.             // To fix this, it seems, I need to assign the values back to the SerializedProperty and apply the SerializedObject.
    107.             prop.vector3Value = script.someValue;
    108.             serializedObject.ApplyModifiedProperties();
    109.             #endregion
    110.         }
    111.  
    112.         EditorGUI.EndProperty();
    113.     }
    114.  
    115.     public void OnSceneGUI()
    116.     {
    117.         // Using serializedObject in this method results in an error:
    118.         // "The serializedObject should not be used inside OnSceneGUI or OnPreviewGUI. Use the target property directly instead."
    119.         // It seems, multi-object-editing cannot be easily implemented. At least, if we want to draw
    120.         // a single handle control and change the values of multiple instances.
    121.         // Instead, this method will draw multiple handles, one for each instance separately.
    122.  
    123.         PrefabEditTest script = (PrefabEditTest)target;
    124.  
    125.         // Disable this example if we are using the custom callback.
    126.         if (script.useCustomSceneViewGUICallback)
    127.             return;
    128.  
    129.         EditorGUI.BeginChangeCheck();
    130.         Vector3 value = Handles.Slider(script.someValue, Vector3.right);
    131.         if (EditorGUI.EndChangeCheck())
    132.         {
    133.             // Next error: when trying to use the targets array to support multi-editing:
    134.             // "The targets array should not be used inside OnSceneGUI or OnPreviewGUI. Use the single target property instead."
    135.  
    136.             // Record undo of previous values on object (works nicely).
    137.             Undo.RecordObject(target, "Change Value");
    138.  
    139.             // Now change the values on the MonoBehaviour class.
    140.             script.someValue = value;
    141.  
    142.             #region ISSUE_B
    143.             // Documentation says: "If this method is not called, changes made to the instance are lost."
    144.             // However, in my tests, this does not change the behaviour at all, changes don't seem to be lost if I don't use it.
    145.             // Also, should I only call this on prefab instances in the scene or can I safely call it on all GameObjects or Prefabs?
    146.             // For Example, should I check for '!EditorUtility.IsPersistent(target)' first?
    147.             PrefabUtility.RecordPrefabInstancePropertyModifications(target);
    148.             #endregion
    149.         }
    150.  
    151.         #region ISSUE_D
    152.         // Using Handles in OnSceneGUI this way, results in the basic functionality working, but multi-object-editing is not fully supported.
    153.         #endregion
    154.     }
    155.  
    156.     private void OnEnable()
    157.     {
    158.         // Here we add a callback to the delegate, which hooks into the
    159.         // same loop as OnSceneGUI, but allows us to use SerializedProperty.
    160.         SceneView.duringSceneGui += MySceneGUI;
    161.     }
    162.  
    163.     private void OnDisable()
    164.     {
    165.         SceneView.duringSceneGui -= MySceneGUI;
    166.     }
    167.  
    168.     private void MySceneGUI(SceneView obj)
    169.     {
    170.         PrefabEditTest script = (PrefabEditTest)target;
    171.  
    172.         // Disable this example if we are not using the custom callback.
    173.         if (!script.useCustomSceneViewGUICallback)
    174.             return;
    175.  
    176.         // With a custom callback, we can support multi-editing simply by using serializedObject.
    177.         serializedObject.Update();
    178.  
    179.         // This also makes the rest of the code very easy.
    180.         SerializedProperty prop = serializedObject.FindProperty("someValue");
    181.         prop.vector3Value = Handles.Slider(prop.vector3Value, Vector3.right);
    182.  
    183.         serializedObject.ApplyModifiedProperties();
    184.     }
    185. }
    186.  
    So here are my own questions:

    ISSUE_A
    upload_2019-12-6_22-6-56.png
    When I use a custom control to draw "Some Value" I use BeginProperty and EndProperty to make sure that the label is drawn bold and the blue line is added to indicate a prefab override. However, it makes all three vector components x, y and z bold as well, whereas the SerializedProperty implementation only highlights x. How do I implement this correctly? Or is this a bug in BeginProperty?

    ISSUE_B
    The documentation advises to call PrefabUtility.RecordPrefabInstancePropertyModifications(target) after changes have been made to a prefab instance. In my test, however, there was no observable difference. Values were still saved. Do I really need to call this? Are there any cases which I haven't tested which are affected by this? And do I need to first test, if the target is actually a prefab instance, e.g. use !EditorUtility.IsPersistent(target) ?

    Overall I feel like there are a few things I would like to see in the documentation. For example, why can't I use serializedObject in Editor.OnSceneGUI, but it works fine in SceneView.duringSceneGUI? And what are the recommended approaches for Handles and multi-editing?
     
    Last edited: Jan 21, 2023
    Yany likes this.
  2. unity_OZTrCwWrfgwrKA

    unity_OZTrCwWrfgwrKA

    Joined:
    Feb 11, 2022
    Posts:
    1
    I am having this issue with OnSceneGUI and multi-object editing too. However, I am just trying to draw the different instances.
     
  3. Yany

    Yany

    Joined:
    May 24, 2013
    Posts:
    61
    Wow, this is so well documented and still no reply. I'm interested in the answer too. This would be the most important information from Unity to understand generic patterns to follow when a tool is implemented for Unity.