Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice

How to do 3D bounding box labeling for an object with many children?

Discussion in 'Computer Vision' started by Roboserg, May 25, 2022.

  1. Roboserg

    Roboserg

    Joined:
    Jun 3, 2018
    Posts:
    83
    Convinced my company to give Unity Perception a try :) It works great for labeling individual objects.
    However, the current problem I am solving is the following:
    • I have a model of an object (let's say a spaceship) that consists of 300+ parts. So I have an empty container parent GameObject (representing the whole spaceship) that holds those 300+ 3D models as separate GameObjects as children.
    • Now I'd like to have a single 3D bounding box covering all those 300+ parts. So a single 3D bbox for the whole spaceship.
    • What would be the fastest and easier way of accomplishing this?
    What I tried so far:
    • Googled for hours to calculate bounds from the Renderer component of all children. Drawing a Gizmo and using
      Code (csharp):
      1. Gizmos.matrix = transform.localToWorldMatrix
      seems to work.
    • But now I need to create a mesh and orient it the same way as the Gizmos Wire Cube, which I have no clue how to do.
    Since this problem statement could be quite common, e.g. having a single 3D bounding box for all the children of a parent object, I am wondering if there is a simpler solution? It boils down to the problem statement of an axis aligned bounding box for all children and then having a transparent material for Unity Perception to only label it, without rendering.
     
  2. cameronsun

    cameronsun

    Unity Technologies

    Joined:
    Jan 10, 2020
    Posts:
    21
    Hey there Roboserg. This should "just work" if you add a labelling component only to the topmost parent (and not any of the children). How is your scene currently set up? If it's set up like I've described here and it's still not working, I'll file a bug.

    Also, if you're using Perception at work, our team would love to talk and figure out how we can help meet any needs you have. Feel free to email shounak.mitra@unity3d.com if you'd like to set up some time to do that.

    - Cameron
     
  3. Roboserg

    Roboserg

    Joined:
    Jun 3, 2018
    Posts:
    83
    Hi Cameron,
    thanks for the quick reply. I've your suggestion, the coordinates of the labelled object with 3D bbox do not add up. I don't know if it's a bug, but certainly not an expected behavior. Let me expand on that:

    I have a car at position (0,0,0), notice the pivot being on the ground. (The white bounding box is just a wired cube Gizmo, my own calculation, more on that later).
    upload_2022-5-27_2-28-58.png

    I have a camera at position (0, 0.74, -5). Y is set to 0.74, this is the height of the geometric center of the car (the center of the bounding box). So that after labeling a 3d bounding box with Unity Perception I would expect a relative position of the camera to the car being (0, 0, -5). I have calculated it with the "test.cs" script attached to the camera.
    upload_2022-5-27_2-25-17.png

    The "Vehicle" GameObject is an empty container, holding the car model, has the "Labelling" component attached to it, as you've suggested.

    Result:
    upload_2022-5-27_2-39-54.png

    The camera has the correct translational parameters after labeling:
    {
    "x": 0
    "y": 0
    "z": -5
    }

    However, the car, instead of having a relative traslational vector of (0,0,5) for some reason has an error of 0.16424 on the Z axis. Also notice X and Y are not really 0 either, but should be:
    upload_2022-5-27_2-43-5.png

    And if the car is rotated 90 degrees, Unity Perception labels the position of the car as following:
    {
    "x": 0.0164241809,
    "y": 2.38418579e-7,
    "z": 4.99999857
    }

    Notice an error in X, same 0.0164241809 offset.
     

    Attached Files:

    Last edited: May 27, 2022
  4. Roboserg

    Roboserg

    Joined:
    Jun 3, 2018
    Posts:
    83
    Because of the error shown above, I calculate the bounding box of the whole group of objects on my own and create a cube with the size of the bounding box:
    upload_2022-5-27_2-49-58.png

    I render the center of the bounding box with the white sphere. My calculation seems to be correct, because the Unity's move tool, set to "Center" (instead of pivot) is perfectly aligned with the white sphere:
    upload_2022-5-27_2-50-45.png

    I then set the material of the cube, the bounding box mesh, to transparent and label it with Unity Perception:
    upload_2022-5-27_3-0-14.png

    Result:
    Correct position of the car relative to camera at (0,0,5):
    upload_2022-5-27_3-1-12.png

    The car rotated 90 degrees also has correct labels:
    {
    "x": 0,
    "y": 0,
    "z": 5
    }

    TLDR: There is definitely something going on. Any help or advice is very appreciated!
     
  5. cameronsun

    cameronsun

    Unity Technologies

    Joined:
    Jan 10, 2020
    Posts:
    21
    Hey there, sorry for the slow reply.

    This does certainly seem like it could be a bug, but it's hard to say without seeing the source asset / how your own bounding box is being calculated. If you're comfortable with it, would you please be able to send a minimal unity project with this scene and the bounding box script to computer-vision@unity3d.com? From there, we can log a ticket for this and have an engineer look at exactly what you're seeing.

    One other thought - are any of the child components of the car using something other than a MeshFilter? Maybe a SkinnedMeshRenderer? We currently do not support SkinnedMeshRenderers with the 3D bounding box labeler. As a result, if certain parts of the car are meant to be articulated with a skinned mesh + armature, they will be ignored by the bounding box calculation and therefore may throw off the results.

    Best,
    Cameron