Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

How to create these perception rays?

Discussion in 'ML-Agents' started by JPhilipp, Jan 27, 2020.

  1. JPhilipp

    JPhilipp

    Joined:
    Oct 7, 2014
    Posts:
    56
    Using Ray Perception Sensor Component 3D, how would I get the following rays set up: ForwardLeft, Forward, ForwardRight + ForwardDown, Down + Back? As in this image:



    And also, just to confirm, I don't need to add the ray count to the Behavior Parameters -> Vector Observation -> Stack Size anymore, but it's now somehow managed automatically in the background, right?

    And a side question, how could I have the debug gizmo ray visualization show outside of training, while still in non-play mode in editor? I can't see any lines when setting up this component, making it hard to understand what's happening.

    Thanks!
     
    JPhilipp, Jan 27, 2020
    #1
    createtheimaginable likes this.

  2. celion_unity

    celion_unity

    Unity Technologies

    Joined:
    Jun 12, 2019
    Posts:
    289
    Hi,
    You should be able to do this setup if you're using ML-Agents 0.13.0 or higher. In that version, we added a "Use Child Sensors" option to the Agent, so it will look for SensorComponents (e.g. Ray Perception sensors) on GameObjects nested below the Agent in the hierarchy.

    So you should need:
    1. a RayPerceptionSensorComponent3D attached to the Agent's GameObject with RaysPerDirection=1 and MaxRayDegrees=45 for the forward, forward-left, and forward right rays
    2. a GameObject attached to the Agent's GameObject with a transform to point it backwards, and a RayPerceptionSensorComponent3D attached to this with RaysPerDirection=0 (will still shoot 1 ray) for the backwards ray
    3. for the forward-down ray, either do similar to #2, or attach to the Agent's GameObject with a negative End Vertical Offset

    Correct.

    Good question, I don't think we can currently do this, but I think there is some options for gizmos to render even outside of play mode. I'll look into this for a future version.
     
    celion_unity, Jan 27, 2020
    #2
    Haneferd, Sab_Rango and JPhilipp like this.

  3. JPhilipp

    JPhilipp

    Joined:
    Oct 7, 2014
    Posts:
    56
    Thank you for the fast reply! Great support!
     
    JPhilipp, Jan 29, 2020
    #3
    celion_unity likes this.

  4. JPhilipp

    JPhilipp

    Joined:
    Oct 7, 2014
    Posts:
    56
    Implementing this now! I wonder if for learning and usability sake, it would make sense to rename RaysPerDirection to say RaysPerSideDirection or something. Considering that forward also is a direction, and right now, RaysPerDirection=1 gives me 3 rays in your example, and RaysPerDirection=0 gives me 1 ray. (Maybe there's also something that could be done to have it easier understandable that RayPerceptionComponent is basically forward-facing by default, it wasn't immediately clear to me. In Unity code, such would usually be very clear via e.g. "transform.forward * foobar".)

    Either way, loving ML-Agents, and I don't have a deep understanding of the framework so please ignore this if it doesn't make sense.
     
    JPhilipp, Jan 30, 2020
    #4

  5. celion_unity

    celion_unity

    Unity Technologies

    Joined:
    Jun 12, 2019
    Posts:
    289
    celion_unity, Jan 31, 2020
    #5
    JPhilipp likes this.

  6. celion_unity

    celion_unity

    Unity Technologies

    Joined:
    Jun 12, 2019
    Posts:
    289
    celion_unity, Feb 25, 2020
    #6
    Haneferd and JPhilipp like this.

  7. JPhilipp

    JPhilipp

    Joined:
    Oct 7, 2014
    Posts:
    56
    Great! Thanks!
     
    JPhilipp, Feb 26, 2020
    #7

  8. jinhueck

    jinhueck

    Joined:
    May 25, 2017
    Posts:
    4
    @celion_unity
    Hello. It's a different issue from the one above, but I have a question, so I'm writing a comment. I'm currently working on a 2D project, and when I used ray permception2D in that project, it's hard to detect properly because I keep hitting my own collider. Can you tell me what I should do when I don't want to detect a collider that I don't want?

    For your information, I added the following inspector value.
    Sensor Name->RayPerceptionSensor
    Detectable Tags->player(that's not my agent's tag name now, but I want to use this tag for my agent)
    Ray Per Direction -> 5
    Max Ray Degrees -> 70
    Sphere Cast Radius -> 0.1
    Ray Layer Mask -> EveryThing
    Observation Stacks -> 1
    Use World Position -> true
     
    Last edited: May 12, 2020
    jinhueck, May 12, 2020
    #8

  9. celion_unity

    celion_unity

    Unity Technologies

    Joined:
    Jun 12, 2019
    Posts:
    289
    Hi @jinhueck,
    I think the 2D Physics raycasts behave a bit differently that the 3D ones by default. The simplest fix should be to go to Project Settings > Physics2D and uncheck "Queries Start In Colliders", but beware that this will affect all queries, not just the ML-Agents ones.

    If that's too big of a change, I think you can either
    1) Use the transformed GameObject approach described above, so that the rays for the SensorComponent start outside of the Agent's collider
    2) Add a specific layer to the Agent's collider, and use the Ray Layer Mask to ignore that layer

    Hope one of those works for you.
     
    celion_unity, May 12, 2020
    #9

  10. Joe-Soap

    Joe-Soap

    Joined:
    Oct 10, 2020
    Posts:
    2
    Quick question regarding the Ray Perception Sensor Component. I get the setup of the ray perception sensor component in the editor. Now how do I "React" to the raycast hit infos that arise from those rays hitting things. What is the Event name? OnCollisionEnter ? OnTriggerEnter? OnRayCastHit ? Nowhere can I find this vital bit of information.Please put me out of my misery
     
    Joe-Soap, Oct 11, 2020
    #10

  11. andrewcoh_unity

    andrewcoh_unity

    Unity Technologies

    Joined:
    Sep 5, 2019
    Posts:
    162
    A way to access the collision info associated with raycasts is not directly accessible through our API. Typically, we let the agent learn how to 'react' to the data captured by the raycasts. Can you describe what you're trying to do and maybe we can offer an alternative?

    If you have game logic that depends on getting this info, you can try something similar to how we implement the laser mechanic in our FoodCollector agent.
     
    andrewcoh_unity, Oct 13, 2020
    #11

  12. prismspecsx

    prismspecsx

    Joined:
    Nov 27, 2018
    Posts:
    13
    This is a very helpful thread, thank you.

    So, I understand that child ray perception sensors are "automatic" in the sense that no code is required for agents to use them and learn/act accordingly. You don't add them as observations, and you don't account for them in the vector observation space size, correct?
     
    prismspecsx, Feb 11, 2021
    #12

  13. celion_unity

    celion_unity

    Unity Technologies

    Joined:
    Jun 12, 2019
    Posts:
    289
    Yes, this is correct. I tried to make this point in the section in the documentation about sensors, but if you have suggestions on how to make it clearer, I can try to add them.
     
    celion_unity, Feb 12, 2021
    #13

  14. unity_6479390318sh

    unity_6479390318sh

    Joined:
    Mar 30, 2021
    Posts:
    1
    I see "RayPerceptionSensorComponent - Allows information from set of ray-casts to be used as observation." in that documentation.Is it possible to turn RaycastHit into an observation without using Ray Perception Sensor?

    I made a custom "eye" for my agent with lots of raycasts, but I can't get it to work with ML agents. Another idea I have is to connect a Vector3(a hit point) and an int(use a number to define a type of obejct, walls can be 0 and buttons is 1) so the agent can learn to avoid walls and find buttons. Is this possible?
     
    unity_6479390318sh, Jul 3, 2021
    #14

  15. akhrifmuhaed07

    akhrifmuhaed07

    Joined:
    Mar 16, 2022
    Posts:
    1
    @celio_unity how we can get the information like distances from Ray sensor component 3D ?
     
    akhrifmuhaed07, May 28, 2022
    #15