Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question Additional observation from RaySensor

Discussion in 'ML-Agents' started by artbotva, Jun 7, 2023.

  1. artbotva

    artbotva

    Joined:
    Nov 8, 2014
    Posts:
    23
    I use RayPerceptionSensor2d. I have had some success in learning. But I have a few questions.
    1) I'm not sure, but when ray encounters a detectable object, does it go no further? Judging by Gizmos it doesn't go any further. So I have to use 3 components of the ray sensor so that the agent sees the wall that is right behind the food?

    2) Even though I looked at
    https://github.com/Unity-Technologies/ml-agents/blob/08a66c4dc33340f4294643654bbe1dae74ca8ce7/com.unity.ml-agents/Runtime/Sensors/RayPerceptionSensor.cs#L164

    but didn't really understand what kind of observations the Ray Sensor transmits to the python? I need to pass the following observations into python:
    • Distance to object.
    • The speed of the object (rigidbody2d)
    • Weight of the object (rigidbody2d)
    How can I do it? I would be very grateful for code examples
     
  2. artbotva

    artbotva

    Joined:
    Nov 8, 2014
    Posts:
    23
    Code (CSharp):
    1. using UnityEngine;
    2. using UnityEngine.Serialization;
    3.  
    4. namespace Unity.MLAgents.Sensors
    5. {
    6.     public class CustomRayPerceptionOutput : RayPerceptionOutput
    7.     {
    8.         public CustomRayPerceptionOutput()
    9.         {
    10.             CustomObservationSizePerRay = 3;
    11.         }
    12.  
    13.         public override void GetCustomObservationData(RayOutput rayOutput, float[] buffer)
    14.         {
    15.             var rb = rayOutput.HitGameObject.GetComponent<Rigidbody>();
    16.             if (rb != null)
    17.             {
    18.                 buffer[0] = rb.velocity.x;
    19.                 buffer[1] = rb.velocity.z;
    20.                 buffer[2] = rb.mass;
    21.             }
    22.         }
    23.     }
    24.  
    25.     /// <summary>
    26.     /// A component for Custom 3D Ray Perception.
    27.     /// </summary>
    28.     [AddComponentMenu("ML Agents/Custom Ray Perception Sensor 3D", (int)MenuGroup.Sensors)]
    29.     public class RayPerceptionSensorComponentCustom3D : RayPerceptionSensorComponent3D
    30.     {
    31.         public override ISensor[] CreateSensors()
    32.         {
    33.             var rayPerceptionInput = GetRayPerceptionInput();
    34.             var rayPerceptionOutput = new CustomRayPerceptionOutput();
    35.  
    36.             m_RaySensor = new RayPerceptionSensor(SensorName, rayPerceptionInput, rayPerceptionOutput);
    37.  
    38.             if (ObservationStacks != 1)
    39.             {
    40.                 var stackingSensor = new StackingSensor(m_RaySensor, ObservationStacks);
    41.                 return new ISensor[] { stackingSensor };
    42.             }
    43.  
    44.             return new ISensor[] { m_RaySensor };
    45.         }
    46.     }
    47. }

    Found this on the internet, if it worked in the project it would be very cool (but it doesn't work, no methods/fields like this). Who knows, maybe this is a new version? I don't really want to make a custom raycast when there is already a ready-made solution
     
  3. smallg2023

    smallg2023

    Joined:
    Sep 2, 2018
    Posts:
    130
  4. artbotva

    artbotva

    Joined:
    Nov 8, 2014
    Posts:
    23
  5. artbotva

    artbotva

    Joined:
    Nov 8, 2014
    Posts:
    23
  6. smallg2023

    smallg2023

    Joined:
    Sep 2, 2018
    Posts:
    130
    you need a reference to your perception ray (can pass this like a normal reference or get it with GetComponent etc)
    i.e.

    Code (CSharp):
    1. [SerializeField]
    2. RayPerceptionSensorComponent2D rayPerceptionSensor;
    then in your CollectObservations function you can call something like

    Code (CSharp):
    1. // Get the ray perception output from the sensor
    2.       RayPerceptionOutput rayPerceptionOutput = rayPerceptionSensor.RaySensor.RayPerceptionOutput;
    3.       var rayOutputs = rayPerceptionOutput.RayOutputs;
    4.  
    5.       for (int i = 0; i < rayOutputs.Length; i++)
    6.       {
    7.         Debug.Log($"ray {i} hit something ? {rayOutputs[i].HasHit}");
    8.         if (rayOutputs[i].HasHit)
    9.         {
    10.           Debug.Log($"hit gameobject = {rayOutputs[i].HitGameObject.name}");
    11.         }
    12.       }

    so to get the info you asked for you can do something like this

    Code (CSharp):
    1. // Get the ray perception output from the sensor
    2.       RayPerceptionOutput rayPerceptionOutput = rayPerceptionSensor.RaySensor.RayPerceptionOutput;
    3.       RayPerceptionOutput.RayOutput[] rayOutputs = rayPerceptionOutput.RayOutputs;
    4.  
    5.       for (int i = 0; i < rayOutputs.Length; i++)
    6.       {
    7.         Debug.Log($"ray {i} hit something ? {rayOutputs[i].HasHit}");
    8.         if (rayOutputs[i].HasHit)
    9.         {
    10.           Debug.Log($"hit gameobject = {rayOutputs[i].HitGameObject.name}");
    11.           Debug.Log($"hit distance = {rayOutputs[i].HitFraction}");
    12.           Rigidbody2D hitRB = rayOutputs[i].HitGameObject.GetComponentInParent<Rigidbody2D>();
    13.           if (hitRB)
    14.           {
    15.             Debug.Log($"hit velo = {hitRB.velocity}");
    16.             Debug.Log($"hit weight = {hitRB.mass}");
    17.           }
    18.         }
    19.       }
    (should just be a case of swapping the relevant commands to 3D if required)
     
    Last edited: Jul 3, 2023