Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Unite 2023 Registrations are now LIVE! We are thrilled to announce that Unite 2023 Registration is now live and open for all!
    Dismiss Notice
New Forum User Notice Unite 2023 Registrations are now LIVE!

Is it possible to take more than one action per timestep?

Discussion in 'ML-Agents' started by Sira-, Jul 29, 2021.

  1. Sira-

    Sira-

    Joined:
    Jul 8, 2020
    Posts:
    4
    As in, request decision, take action, get new observation, then use that observation to make a new decision, all in the same timestep?

    What I’m trying to do:

    I have an RL model that isn’t bound to the navMesh, which has been trained to move to a location on the mesh that meets some criteria. Then, an A-star agent that is bound to the mesh actually goes to that location. I’m wondering if I can speed up the first part of the process. I’d like to do it all at once, or at least, take as many actions per timestep as possible without slowing down the scene.

    Something similar is definitely possible with timescale shenanigans, but my thing is meant to be a plug-in to a larger system, so I’d like to avoid that.
     
    Sira-, Jul 29, 2021
    #1

  2. ruoping_unity

    ruoping_unity

    Unity Technologies

    Joined:
    Jul 10, 2020
    Posts:
    134
    Hey correct me if my understanding is wrong since I don't think I fully understand your question.
    So you can take multiple actions in each environment step, but the actions are generated all at once so they can't be sequential actions. This is due to the RL formulations where the process is defined as series of (observation, action, reward) pairs.
    On that note, none of those environment stepping are tied to wall-clock time, or timestep. So it might helpful to clarify more on what you mean by "take as many actions per timestep as possible without slowing down the scene".
     
    ruoping_unity, Jul 29, 2021
    #2

  3. Sira-

    Sira-

    Joined:
    Jul 8, 2020
    Posts:
    4
    What I want to do is take multiple RL-steps per Unity frame update (Academy step? Environment step?). So, take an action, move the agent to a new spot, get new observations (1), run those observations through the model to get the next action (I have no idea how to do this), then repeat.

    Impossible things that, I think, would be equivalent if they were possible:
    • I want to set the decision period in the decision requester to something like 0.1.
    • I want to set the timescale really high for just the agent, but have everything else in the scene run as normal.
    I know what I'm trying to do is theoretically possible, because if I set the timescale to 10, the scene runs 10 times faster. (2) If ray observations can only be collected once per environment step, though, it might not be practically possible.

    1. I don't think CollectObservations() does this, because I think it only gets vector observations, where I have both vector and ray observations.
    2. Setting the timescale to 20 also results in a 10x speedup, so I take it that's the practical limit for how fast my computer can compute things. So, at timescale 1, trying to get the agent to do 20 updates in a single frame could slow down the scene, which is what I meant by "take as many actions per timestep as possible without slowing down the scene". What I want to do is run just the agent as fast as my computer allows, while the rest of the scene runs at a timescale of 1.
     
    Sira-, Jul 31, 2021
    #3