Search Unity

  1. We want you to join us at GDC this year! Take a peek at all of the events we will be hosting during the week of GDC.
    Dismiss Notice
  2. Tell us about your experience here and you’ll get early access to the 2018 Game Studios report + more goodies.
    Dismiss Notice
  3. Unity 2017.3 has arrived! Read about it here.
    Dismiss Notice
  4. Want to see the most recent patch releases? Take a peek at the patch release page.
    Dismiss Notice
  5. We've closed the job boards. If you're looking for work, or looking to hire check out Unity Connect. You can see more information here.
    Dismiss Notice

Filter which objects I can select in the scene object via editor tool - how to do custom selection

Discussion in 'Extensions & OnGUI' started by Xarbrough, Feb 9, 2018.

  1. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    250
    I would like to implement a tool (probably a simple editor window), which adds selection modes to the Unity scene. In my case, I would like to only make specific object selectable while in my special mode, because I'm working in a large scene with many objects and clicking on an object by accident may scroll the hierarchy down to a place far away from where I was working.

    I know that I can set the current selection via Selection.activeGameObject and that I can block clicks in the scene view by using the current event with Event.Use().

    Now, when a click happens (I can check in the current event), I would need to see if the mouse has selected one of my special objects and here comes the tricky part. I can't use raycasting because the objects are only visualized via editor handles (more specifically lines and boxes). So I would want to achieve the default selection behavior of these selectable controls, such as Handles.Button(), but I don't want to actually implement those controls in my additional selection mode tool. Instead, I want to keep all of the individual handles and their custom editor code (Gizmos etc) in their respective MonoBehaviors and custom editors and then somehow filter the selection to exclude all other objects via an additional editor window.

    Any ideas how this could be done?
     
  2. shawn

    shawn

    Unity Technologies

    Joined:
    Aug 4, 2007
    Posts:
    546
    Unfortunately, there's no nice public API to do this that I can think of. There is the following delegate in UnityEditor.HandleUtility that you could hook into it. It's internal, so would require some reflection.

    Code (CSharp):
    1.     internal delegate GameObject PickClosestGameObjectFunc(Camera cam, int layers, Vector2 position, GameObject[] ignore, GameObject[] filter, out int materialIndex);
    2.     internal static PickClosestGameObjectFunc pickClosestGameObjectDelegate;
    You could add a handler for it, add whichever objects you want to the ignore/filter arrays. Then call the following internal method. Which is what would normally would be called if pickClosestGameObjectDelegate == null.

    Code (CSharp):
    1. internal static GameObject Internal_PickClosestGO (Camera cam, int layers, Vector2 position, GameObject[] ignore, GameObject[] filter, out int materialIndex)
    Using this internal API is at your own risk. Expect that we will break this in the future, with no backwards compatibility support.
     
    Xarbrough likes this.
  3. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    250
    Thank you for the suggestion! I've got the basic callback working. At least, when I put GameObjects into the ignore and filter arrays, the picking behavior in the scene changes.

    Code (CSharp):
    1. using System.Reflection;
    2.  
    3. public class SceneSelectionFilter : EditorWindow
    4. {
    5.     [MenuItem("Window/Scene Selection Filter")]
    6.     public static void ShowWindow()
    7.     {
    8.         EditorWindow.GetWindow<SceneSelectionFilter>("Selection Filter").Show();
    9.     }
    10.  
    11.     private MethodInfo Internal_PickClosestGO;
    12.  
    13.     private void OnEnable()
    14.     {
    15.         Assembly editorAssembly = typeof(Editor).Assembly;
    16.         System.Type handleUtilityType = editorAssembly.GetType("UnityEditor.HandleUtility");
    17.  
    18.         FieldInfo pickClosestDelegateInfo = handleUtilityType.GetField("pickClosestGameObjectDelegate", BindingFlags.Static | BindingFlags.NonPublic);
    19.         Delegate pickHandler = Delegate.CreateDelegate(pickClosestDelegateInfo.FieldType, this, "OnPick");
    20.         pickClosestDelegateInfo.SetValue(null, pickHandler);
    21.  
    22.         Internal_PickClosestGO = handleUtilityType.GetMethod("Internal_PickClosestGO", BindingFlags.Static | BindingFlags.NonPublic);
    23.     }
    24.  
    25.     private GameObject OnPick(Camera cam, int layers, Vector2 position, GameObject[] ignore, GameObject[] filter, out int materialIndex)
    26.     {
    27.         materialIndex = -1;
    28.  
    29.         ignore = new GameObject[] { /* ? */ };
    30.         filter = new GameObject[] { /* ? */ };
    31.  
    32.         return (GameObject)Internal_PickClosestGO.Invoke(null, new object[] { cam, layers, position, ignore, filter, materialIndex });
    33.     }
    34. }
    However, I don't understand how those arrays need to be filled to achieve my desired behavior:

    • GameObjects should not be pickable by default (ground tiles, models, etc)
    • A set of specific objects, which I can define, should be pickable
    I've tried a few configurations but couldn't manage to completely exclude objects from being selected. It does look like the right approach to handle the problem, hopefully, I just missed something.
     
  4. shawn

    shawn

    Unity Technologies

    Joined:
    Aug 4, 2007
    Posts:
    546
    Just skimming the code here it looks like the behavior is the following:

    If the filter array is null, it'll pick all objects minus the ones in the ignore.
    If the filter array is not null (even if it's empty), it'll pick objects in the filter array minus the ones in the ignore.

    So it looks like you want to add specific set of objects, you'll want to pass null or empty array to ignore, and pass your specific set to filter.
     
    Xarbrough likes this.
  5. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    250
    @shawn: Thank you for explaining. The filtering by Internal_PickClosestGO returns the desired result: either one of the pickable objects or null. :)

    However, I noticed, that null is not a valid value for the internal method PickGameObjectDelegated, which will call the internal method again without the filtering applied to get a non-null pickable object. I managed to force the selection to stay at the current correct object and not select anything else with the following code. The behavior is how I want it, but Unity logs an error.

    I don't expect any sort of technical support for these internal features, so I don't want to be asking anymore. I'm still posting my current code, with the small error-logging problem for reference, if anyone else has a similar issue.

    Code (CSharp):
    1. private GameObject OnPick(Camera cam, int layers, Vector2 position, GameObject[] ignore, GameObject[] filter, out int materialIndex)
    2. {
    3.     materialIndex = -1;
    4.  
    5.     filter = GetPickableObjects();
    6.  
    7.     Assembly editorAssembly = typeof(Editor).Assembly;
    8.     Type handleUtilityType = editorAssembly.GetType("UnityEditor.HandleUtility");
    9.     MethodInfo Internal_PickClosestGO = handleUtilityType.GetMethod("Internal_PickClosestGO", BindingFlags.Static | BindingFlags.NonPublic);
    10.     GameObject go = (GameObject)Internal_PickClosestGO.Invoke(null, new object[] { cam, layers, position, ignore, filter, materialIndex });
    11.  
    12.     // The returned object is either one of our desired ones, or null.
    13.     // If we return null, the internal Unity method 'PickGameObjectDelegated' will ignore our custom filter
    14.     // and find a different pick.
    15.     if (go == null)
    16.     {
    17.         // Now, if we continue here, everything works as I want it to,
    18.         // however, Unity will throw the following error, when we have a pickable object ("MyCube") already selected
    19.         // and attempt to click on something, which shouldn't be selected:
    20.         // "GetAllOverlapping failed, could not ignore game object 'MyCube' when picking".
    21.         // I'm not explicitly ignoring this object, I actually want it to remain selection. I tried to:
    22.         //    add the current selection ("MyCube") to ignore
    23.         //    add the current selection to filter and everything else to ignore
    24.         // The error still gets logged, but the behavior is what I want, the old correct selection remains.
    25.  
    26.         // Instead of letting Unity pick something else, I want to stay at the current selection.
    27.         return Selection.activeGameObject;
    28.     }
    29.  
    30.     // This is one the pickable desired objects, all good if we reach this.
    31.     return go;
    32. }
    I've tried to find the issue in the source, but it looks like the error is logged, but then the null is still handled correctly, so I'm not sure how to change the ignore and filter arrays in this special case to keep the selection.

    For now, I can also use the following workaround:

    Code (CSharp):
    1. public class SceneSelectionFilter : EditorWindow
    2. {
    3.     [MenuItem("Window/Scene Selection Filter")]
    4.     public static void ShowWindow()
    5.     {
    6.         EditorWindow.GetWindow<SceneSelectionFilter>("Selection Filter").Show();
    7.     }
    8.  
    9.     private void OnEnable()
    10.     {
    11.         SceneView.onSceneGUIDelegate += OnSceneGUI;
    12.     }
    13.  
    14.     private void OnDisable()
    15.     {
    16.         SceneView.onSceneGUIDelegate -= OnSceneGUI;
    17.     }
    18.  
    19.     private void OnSceneGUI(SceneView sceneView)
    20.     {
    21.         // Block all clicks from propagating to the scene view.
    22.  
    23.         Event e = Event.current;
    24.         if (e.type == EventType.MouseUp)
    25.         {
    26.             // This returns a picked object as it normally does, but does not select it yet.
    27.             GameObject picked = HandleUtility.PickGameObject(e.mousePosition, false);
    28.  
    29.             if (ShouldBeSelected(picked))
    30.             {
    31.                 // We select it, if valid for our needs.
    32.                 Selection.activeObject = picked;
    33.                 e.Use();
    34.             }
    35.         }
    36.     }
    37.  
    38.     private bool ShouldBeSelected(GameObject go)
    39.     {
    40.         // TODO: go contained in my set of selectable objects.
    41.         return go == GameObject.Find("MyCube");
    42.     }
    43. }
    This is more like the original idea I had: Block all clicks from the scene view, then pick an object close to the mouse, but only select it if within our defined set. Made possible by the supported PickGameObject method. Not sure if there are any other side-effects, but if we only want to select an object, this works fine.

    One drawback, however, is that since we are blocking the scene view completely, our relevant components do not receive a click event themselves. My original use-case was: The selection filter tool blocks selection of unwanted objects, but lets the desired click through to my objects, which have a custom editor and gizmo /handle code with buttons, etc. This would work flawlessly in the first approach with the internal delegate if I could get rid of the logged error, but in the second approach, I need to forward scene events captured by the selection filter to those selected objects, which is a little ugly (because now this code needs to change if any of the other tools requires a different click or layout message).