Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice
  2. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  3. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Instance segmentation for separate objects

Discussion in 'Computer Vision' started by dusansvilarkovic, Jun 8, 2021.

  1. dusansvilarkovic

    dusansvilarkovic

    Joined:
    Jun 8, 2021
    Posts:
    9
    Hi,
    So far what I saw is that Instance segmentation is able to give 2d camera view of the objects included, but I do not see is there any way to give instance segmentation per object?

    In first image Instance_8_normal.png, you see there is instance segmentation for every object,
    and in next 3 images Instance_8_{i}.png (where i = 1,2,3) you see what I would like to have from your perception package. How can I achieve this?

    You might suggest me to just darken every pixel I don't want and have single objects by writing my own implementation, but the problem is, this doesn't work when there is occlusion in between the objects, as in Instance_8_overlap.png
     

    Attached Files:

  2. MohsenK_Unity

    MohsenK_Unity

    Unity Technologies

    Joined:
    Jun 8, 2021
    Posts:
    10
    Hi,

    I need to understand this issue better. In the case of Instance_8_overlap.png, what is the desired outcome? Do you need the red object to be fully displayed including the occluded part?
     
  3. dusansvilarkovic

    dusansvilarkovic

    Joined:
    Jun 8, 2021
    Posts:
    9

    Yes, exactly that, I want to have a mask of it in that case. My idea was to call instance segmentation each time for each object and disable visibility from the rest of the objects included, but don't have idea how to do it.
     
  4. MohsenK_Unity

    MohsenK_Unity

    Unity Technologies

    Joined:
    Jun 8, 2021
    Posts:
    10
    As you pointed out, one way to do this would be to deactivate all other labeled objects and have only one active at the time of each capture. If you are using the Perception package's Scenarios to control the flow of your simulation, you can achieve this goal by running multiple frames inside of each Iteration, and in each frame having only one labeled object active. For documentation on how to use Scenarios, checkout the Perception Tutorial. Especially, I recommend following Phase 2 of the tutorial, which focuses on writing custom Randomizers. That will help follow the example below.

    The project in the tutorial is setup so that the simulation is randomized on each Iteration of the Scenario, and each Iteration only runs for one frame. For your purpose, you can set the number of frames per Iteration to the number of objects that will be present in each Iteration. So 3, if you will always have 3 objects, as seen below:

    upload_2021-6-8_15-20-11.png


    You can also write a custom Scenario class that can be manually moved forward to the next Iteration at runtime, so that you do not have to specify an exact number of frames per Iteration beforehand. Let us know if that's something you need and we can provide some sample code.

    Then, you will need to create a custom Randomizer that is tasked with switching objects on and off during each Iteration. Here is a sample Randomizer:

    Code (CSharp):
    1.  
    2. [Serializable]
    3. [AddRandomizerMenu("SynthDet/Active Object Switcher")]
    4. public class ActiveObjectSwitcher : Randomizer
    5. {
    6.     int m_FrameInIteration;
    7.     List<ObjectSwitcherTag> m_SwitcherTagsInIteration;
    8.     protected override void OnIterationStart()
    9.     {
    10.         m_FrameInIteration = 0;
    11.         m_SwitcherTagsInIteration = tagManager.Query<ObjectSwitcherTag>().ToList();
    12.     }
    13.  
    14.     protected override void OnUpdate()
    15.     {
    16.         foreach (var tag in m_SwitcherTagsInIteration)
    17.         {
    18.             tag.gameObject.SetActive(false);
    19.         }
    20.        
    21.         if (m_FrameInIteration < m_SwitcherTagsInIteration.Count)
    22.         {
    23.             m_SwitcherTagsInIteration[m_FrameInIteration].gameObject.SetActive(true);
    24.         }
    25.  
    26.         m_FrameInIteration++;
    27.     }
    28. }
    29.  
    At the start of each Iteration, this Randomizer looks for all active objects in the Scene that have the
    ObjectSwitcherTag
    component attached. Then, on each update (frame), one of these objects is activated and the rest are deactivated.

    For this to work, you will also need to create the
    ObjectSwitcherTag
    class, which looks like this:

    Code (CSharp):
    1.  
    2. [AddComponentMenu("SynthDet/RandomizerTags/ObjectSwitcherTag")]
    3. public class ObjectSwitcherTag : RandomizerTag { }
    4.  
    Now, to use these in your project, the prefabs that you instantiate in the scene will need to have the
    ObjectSwitcherTag
    component attached, and the list of Randomizers in your Scenario will need to include
    ActiveObjectSwitcher
    .

    A sample output looks like this (three objects in the scene simultaneously): rgb.jpg
    instance.jpg
     
  5. dusansvilarkovic

    dusansvilarkovic

    Joined:
    Jun 8, 2021
    Posts:
    9
    I have a script which creates randomly objects around the surface and destroys them afterwards (SimulationManager.cs) and is attached to unused GameObject, and I am wondering is there any way I can manipulate these scenarios through it?

    My idea is, use the for loop, and for each created object, disable visibility of others and trigger instance segmentation?
     

    Attached Files:

    Last edited: Jun 9, 2021
  6. dusansvilarkovic

    dusansvilarkovic

    Joined:
    Jun 8, 2021
    Posts:
    9
    Nevermind, please provide me with some sample code if possible
     
  7. MohsenK_Unity

    MohsenK_Unity

    Unity Technologies

    Joined:
    Jun 8, 2021
    Posts:
    10
    Actually, I had a look at your SimulationManager class and was able to achieve the same effect. I added some code that iterates through spawned objects, one per frame, and keeps only one active. When they have all been captured, it destroys them and creates new ones. Here is the change:
    Code (CSharp):
    1.  
    2. int frameCounter;
    3. void Update()
    4. {
    5.     if (frameCounter % NumOfObj == 0)
    6.     {
    7.         DestroyAll();
    8.         RandomizeStage();
    9.         NewPositions();
    10.         SpawnObjects(NumOfObj);
    11.         camera.transform.localRotation = Quaternion.Euler(30, Random.Range(-30f,30f), 0);    
    12.     }
    13.    
    14.     SwitchActiveObject();
    15.     frameCounter++;
    16. }
    17.  
    18. void SwitchActiveObject()
    19. {
    20.     foreach (var obj in gameList)
    21.     {
    22.         obj.SetActive(false);
    23.     }
    24.     gameList[frameCounter % NumOfObj].SetActive(true);
    25. }
    26.  
     
  8. Huynh-Anh

    Huynh-Anh

    Joined:
    Mar 28, 2022
    Posts:
    1
    how would one save the images in different folder ? so one folder per object with their segmented region
     
  9. cameronsun

    cameronsun

    Unity Technologies

    Joined:
    Jan 10, 2020
    Posts:
    21
    I just responded to your post on github (I think you're the same person?) with an answer. Adding the link here in case anyone down the road has the same questions.