Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice
  2. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  3. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question Save object information as metrics

Discussion in 'Computer Vision' started by Argosama, Jul 5, 2022.

  1. Argosama

    Argosama

    Joined:
    Feb 25, 2017
    Posts:
    3
    Hello,
    First of all, I'd want to get the opportunity to thank the team for the amazing job on the Perception package!

    I've been using it for a couple of months now, and I have a question on how to save information on the position of the instanced foreground objects in the 3D space.
    Specifically, I have randomly positioned and rotated flat objects with a front and a back face, and I'd want to save the "up" vector information for each instance.

    I've been trying saving the information as a metric using the same approach adopted in the SynthDet project, that is, using a "fake" randomizer to report data, but this way I've not been able to get the "instance id" of the objects (the one that also appears in RenderedObjectInfo).
    I've also been trying investigating a different approach, trying to get the job done from a Labeler as shown in the documentation, but this looks even worse since I still have no access to the instance id and I'm also missing the query method of the randomizer to get the game object instances. And even if I could get both (as lists, I suppose), I wonder how I could pair them.

    Am I missing something? Any suggestion on how I could accomplish my task?
    I've already spent quite a few hours trying to figure things out, and all methods I'm approaching seem overly complicated compared to what I'm trying to achieve.

    Many thanks to whoever tries to come to my help! :)

    I'm using latest Perception version 0.11.2-preview.2
     
  2. StevenBorkman

    StevenBorkman

    Joined:
    Jun 17, 2020
    Posts:
    16
    Hi Argosama,

    First, let me thank you for being a perception user and thanks for reaching out.

    It sounds like you are moving in the right direction but just need a couple of pointers to get you there. I would suggest that you take a look at the 3D bounding box labeler and either use it for inspiration or modify it to fit your needs. It has a lot going on it that I think you will be able to ignore.

    The key thing that it does is iterate through all of the labeled objects in your seen. The labeled objects are the objects that have been registered with perception to be captured by the perception camera and labelers (thus providing an instance ID).

    I have wrote up a labeler for you that will hopefully get your started (or maybe all of the way there):

    Code (CSharp):
    1.  
    2. using System;
    3. using System.Collections.Generic;
    4. using UnityEngine;
    5. using UnityEngine.Perception.GroundTruth;
    6. using UnityEngine.Perception.GroundTruth.DataModel;
    7. using UnityEngine.Rendering;
    8.  
    9. namespace DefaultNamespace
    10. {
    11.     public class ObjectNormal : IMessageProducer
    12.     {
    13.         public int labelId { get; set; }
    14.         public string labelName { get; set; }
    15.         public uint instanceId { get; set; }
    16.         public Vector3 normal { get; set; }
    17.         public void ToMessage(IMessageBuilder builder)
    18.         {
    19.             builder.AddInt("instanceId", (int)instanceId);
    20.             builder.AddInt("labelId", labelId);
    21.             builder.AddString("labelName", labelName);
    22.             builder.AddFloatArray("normal", MessageBuilderUtils.ToFloatVector(normal));
    23.         }
    24.     }
    25.  
    26.     public class ObjectNormalAnnotation : Annotation
    27.     {
    28.         public ObjectNormalAnnotation(AnnotationDefinition definition, string sensorId, List<ObjectNormal> normals)
    29.             : base(definition, sensorId)
    30.         {
    31.             this.normals = normals;
    32.         }
    33.  
    34.         public List<ObjectNormal> normals { get; set; }
    35.  
    36.         public override void ToMessage(IMessageBuilder builder)
    37.         {
    38.             base.ToMessage(builder);
    39.             foreach (var e in normals)
    40.             {
    41.                 var nested = builder.AddNestedMessageToVector("values");
    42.                 e.ToMessage(nested);
    43.             }
    44.         }
    45.  
    46.         public override bool IsValid() => true;
    47.     }
    48.  
    49.     public class ObjectNormalDefinition : AnnotationDefinition
    50.     {
    51.         public ObjectNormalDefinition(string id, IdLabelConfig.LabelEntrySpec[] spec)
    52.             : base(id)
    53.         {
    54.             this.spec = spec;
    55.         }
    56.  
    57.         internal const string labelDescription = "Produces normals for labeled objects";
    58.         public IdLabelConfig.LabelEntrySpec[] spec { get; }
    59.  
    60.         public override string modelType => "type.your_company.com/ObjectNormalAnnotation";
    61.         public override string description => labelDescription;
    62.  
    63.         public override void ToMessage(IMessageBuilder builder)
    64.         {
    65.             base.ToMessage(builder);
    66.             foreach (var e in spec)
    67.             {
    68.                 var nested = builder.AddNestedMessageToVector("spec");
    69.                 e.ToMessage(nested);
    70.             }
    71.         }
    72.     }
    73.  
    74.     public class NormalLabeler : CameraLabeler
    75.     {
    76.         public override string description => ObjectNormalDefinition.labelDescription;
    77.         public string annotationId = "Normal Labeler";
    78.         public override string labelerId => annotationId;
    79.         protected override bool supportsVisualization => false;
    80.  
    81.         public IdLabelConfig idLabelConfig;
    82.         ObjectNormalDefinition m_Definition;
    83.  
    84.         public NormalLabeler(){}
    85.  
    86.         public NormalLabeler(IdLabelConfig labelConfig)
    87.         {
    88.             idLabelConfig = labelConfig;
    89.         }
    90.  
    91.         protected override void Setup()
    92.         {
    93.             if (idLabelConfig == null)
    94.                 throw new InvalidOperationException("NormalLabeler's idLabelConfig field must be assigned");
    95.  
    96.             m_Definition = new ObjectNormalDefinition(annotationId, idLabelConfig.GetAnnotationSpecification());
    97.             DatasetCapture.RegisterAnnotationDefinition(m_Definition);
    98.             visualizationEnabled = supportsVisualization;
    99.         }
    100.  
    101.         protected override void OnBeginRendering(ScriptableRenderContext scriptableRenderContext)
    102.         {
    103.             var toReport = new List<ObjectNormal>();
    104.             foreach (var label in LabelManager.singleton.registeredLabels)
    105.             {
    106.                 if (idLabelConfig.TryGetLabelEntryFromInstanceId(label.instanceId, out var labelEntry))
    107.                 {
    108.                     var entityGameObject = label.gameObject;
    109.                     var up = entityGameObject.transform.up;
    110.                     toReport.Add(new ObjectNormal
    111.                     {
    112.                         labelId = labelEntry.id,
    113.                         labelName = labelEntry.label,
    114.                         instanceId = label.instanceId,
    115.                         normal = up
    116.                     });
    117.                 }
    118.             }
    119.  
    120.             perceptionCamera.SensorHandle.ReportAnnotation(m_Definition, new ObjectNormalAnnotation(m_Definition, perceptionCamera.ID, toReport));
    121.         }
    122.     }
    123. }
    124.  
    There's a lot of boiler plate in there that hopefully, some day, we will simplify. You will also need to adapt the PerceptionEndpoint to handle this new data class and write it out to disk.

    But I hope that you find this useful.
     
    DragonCoder likes this.