Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice
  2. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  3. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Keypoint annotations for non-human objects?

Discussion in 'Computer Vision' started by VivekSagar, Jul 1, 2021.

  1. VivekSagar

    VivekSagar

    Joined:
    Nov 9, 2016
    Posts:
    8
    I've been integrating the Perception SDK into our simulation workflow and am running into an issue - I have objects in the scene that I need annotations for, but instead of segmentation maps or bounding boxes, I'd like to annotate the position of specific points (specified in the prefab). In the image here for eg, I'd like point annotations at each of the red, blue, green and yellow spheres.
    They KeyPointLabeler does exactly this, but as far as I can tell, is designed only for human skeleton annotations (the KeypointTemplate for eg. has only skeletal joints that it can bind to). Is the plan to extend KeypointLabeler to non-human objects in the future? Is there another way this should be done?
     

    Attached Files:

  2. StevenBorkman

    StevenBorkman

    Joined:
    Jun 17, 2020
    Posts:
    16
    Hey Vivek,

    The KeypointLabeler can be used to do exactly this right now. The keypoint labeler works with a keypoint template to assign the points of interest that you are trying to label. By default, and the typical use case, the keypoints in the template are associated with an animation rig of a humanoid model. But this does not have to be the case.

    The perception package, when cloned from github, has sample scenes included with it. The 'PoseSample' does exactly what you are looking for. After opening the 'PoseSample' scene, click on the Main Camera in the Hierarchy view. In the Inspector window for the main camera, you will see that the PerceptionCamera game object contains multiple KeypointLabelers (only one of them can be active at a time). The first KeypointLabeler, the one with the CocoKeypointTemplate is for the human model. The second KeypointLabeler, the one with the BoxKeypointTemplate, is the one used to track the corners of the box. By looking through this scene I think you will probably find all of the info you need, but here are the general basics to accomplishing this:
    1. Create a new KeypointTemplate by right clicking in Assets and choosing 'Create->Perception->Keypoint Template'
    2. Create entries in your new keypoint template for all of the keypoints that you want to track
    3. Create game objects (no geometry is needed to be associated to them) for the points of interest that you want to track and put them in the scene where they should be. In our example these would be the all of the gameobjects under the KeypointCube for the corners of the box.
    4. Add a JointLabel to each of these new gameobjects
    5. Under the TemplateInformation for the JointLabel associate each label to a label found in your new KeypointTemplate
    6. On your PerceptionCamera add your new KeypointTemplate to KeypointLabeler 'Active Template' field

    I hope this helps, and do not hesitate to reach out if you have any issues following this.

    Steve
     
  3. NPSimulation

    NPSimulation

    Joined:
    Oct 6, 2020
    Posts:
    7
    Hi Steven, I'm trying to implement your solution which seems perfect for my project, however there seems to be a shader incompatibility problem.
    I'm using this commit of the perception SDK and whether in my project or in the 'PoseSample' scene, running the scene results in this error message:

    Code (CSharp):
    1. ArgumentNullException: Value cannot be null.
    2. Parameter name: shader
    3. UnityEngine.Material..ctor (UnityEngine.Shader shader) (at <127e81e1cb3441cc97d26b1910daae77>:0)
    4. UnityEngine.Perception.GroundTruth.KeypointLabeler.Setup () (at C:/Users/r_ang/Documents/GitHub/com.unity.perception/com.unity.perception/Runtime/GroundTruth/Labelers/KeypointLabeler.cs:148)
    5. UnityEngine.Perception.GroundTruth.CameraLabeler.Init (UnityEngine.Perception.GroundTruth.PerceptionCamera camera) (at C:/Users/r_ang/Documents/GitHub/com.unity.perception/com.unity.perception/Runtime/GroundTruth/Labelers/CameraLabeler.cs:122)
    6. UnityEngine.Perception.GroundTruth.PerceptionCamera.LateUpdate () (at C:/Users/r_ang/Documents/GitHub/com.unity.perception/com.unity.perception/Runtime/GroundTruth/PerceptionCamera.cs:228)
    It seems that the shader KeypointDepthCheck.shader is only compatible with HDRP despite being in the URP repository.
    Screenshot1.png
    Am I missing something?
    Screenshot2.png

    In advance, thank you :)
     
  4. JonHUnity

    JonHUnity

    Joined:
    Sep 16, 2016
    Posts:
    21
    NPSimulation, What version of Unity are you using? The 0.9.0 version of the Perception package is only compatible with 2020.3.17f1 or later. Previous versions will encounter issues like this.
     
    NPSimulation likes this.
  5. NPSimulation

    NPSimulation

    Joined:
    Oct 6, 2020
    Posts:
    7
    Thank you very much for the explanation.

    I was on an older version and updating to a newer one fixed the problem!
     
  6. shashank_prakash

    shashank_prakash

    Joined:
    Apr 18, 2023
    Posts:
    1
    Hi Steve,
    I followed all the above steps but when I try recording the data none of the keypoints show up, could you please help me with that. Screenshot from 2023-04-24 12-19-14.png Screenshot from 2023-04-24 12-19-02.png
     
  7. asmonix

    asmonix

    Joined:
    Dec 9, 2017
    Posts:
    1
    Is there any way of checking the type of object occluding the Keypoint? I'm interested in detecting two scenarios:
    1. the keypoint is occluded by other mug object (Mug_225)
    2. the keypoint is occluded by something else (Mug_224)
    upload_2023-6-7_14-24-41.png
     
  8. KlaraHusonuk

    KlaraHusonuk

    Joined:
    Sep 26, 2023
    Posts:
    1
    You can achieve point annotations for non-human objects by customizing the KeypointLabeler. Although it's designed for human skeleton annotations, you can adapt it for your specific needs. Modify the KeypointTemplate to include points for your non-human objects. This way, you can annotate the position of specific points on the red, blue, green, and yellow spheres or any other objects in your scene. There might be plans to extend KeypointLabeler's functionality in the future and you can find this, but customizing it should work for now.