Search Unity

  1. We are migrating the Unity Forums to Unity Discussions by the end of July. Read our announcement for more information and let us know if you have any questions.
    Dismiss Notice
  2. Dismiss Notice

Setting external rewards to multiple agents

Discussion in 'ML-Agents' started by Sephraih, Apr 19, 2020.

  1. Sephraih

    Sephraih

    Joined:
    Jun 15, 2019
    Posts:
    11
    Hello there,

    So far I have been using " GetComponent<PlayerAgent>().SetReward(x); " to apply environment rewards to my agents, which have been controlled by PlayerAgent - I am now trying to extend the variety of agents - to, for example, a MageAgent and a WarriorAgent.

    My issue now is that I would like some way to determine what kind of Agent script is attached to the current transform to apply the rewards, instead of adapting every script, such as "character stats" or "health controller" to "warrior stats", "mage stats" etc. I believe there was a way to do this with monobehaviour scripts, but not with agents.

    I stumbled upon this issue in other contexts as well, but haven't yet found an ideal way to solve it. What would you suggest?
    Thanks a lot!
     
  2. Sephraih

    Sephraih

    Joined:
    Jun 15, 2019
    Posts:
    11
    i considered a few approaches
    - an inheritance structure (wouldnt be sure how to manage it well, though)
    - a helper object, that is static for all characters, receiving rewards -> the agents then collect those rewards during their observations (which would be one step late, which might affect the learning process)
     
  3. Sephraih

    Sephraih

    Joined:
    Jun 15, 2019
    Posts:
    11
    This is an option too I guess, a horrible one.:

    if (dmger.GetComponent<PlayerAgent>() != null)
    {
    dmger.GetComponent<PlayerAgent>().SetReward(damage * rewardmodifier); //reward to attacker (dmger)
    }
    if (dmger.GetComponent<MageAgent>() != null)
    {
    dmger.GetComponent<MageAgent>().SetReward(damage * rewardmodifier); //reward to attacker (dmger)
    }
    if (dmger.GetComponent<WarriorAgent>() != null)
    {
    dmger.GetComponent<WarriorAgent>().SetReward(damage * rewardmodifier); //reward to attacker (dmger)
    }
     
  4. mbaske

    mbaske

    Joined:
    Dec 31, 2017
    Posts:
    473
    If it's just a matter of scaling rewards for different agent types, then you could simply serialize the rewardmodifier field and set its values per agent instance in the inspector.

    If your reward logic is more complex than that, then you can create a BasicAgent class inheriting from MLAgents.Agent and put all shared behaviour in there (Agent extends Monobehaviour btw). Agent subclasses extending BasicAgent would implement their specific reward logic. Alternatively, you could encapsulate the reward logic in a seperate object, https://en.wikipedia.org/wiki/Strategy_pattern

    Either way, you shouldn't call GetComponent at every agent step for performance reasons. Cache a reference to the agent during initilization and use that instead.
     
  5. Sephraih

    Sephraih

    Joined:
    Jun 15, 2019
    Posts:
    11
    Sorry, the rewards were a bit misleading, thanks a bunch though, I believe your answer put me on the right track.

    It was all about the issue of getting access to the agent methods regardless of varying scripts.

    Currently trying to implement this n launch a training.
    I'll have to refactor afterwards, but to keep it as simple as possible i just made one class with BasicAgent : Agent and let the other (3 so far) inherit from that as suggested - now simply calling .GetComponent<BasicAgent>.AddReward()

    I had earlier tried to call GetComponent<Agent> to achieve this, which wouldn't work - I'm not exactly sure why, though.

    -> I guess the way i have it implemented now assumes that there is only one agent class attached to the transform?
    I am wondering this because i once had in mind to implement a set of abilities with inheritance, so that, for example, abilities could be assigned to a character or agent, which then would call "UseAbility" or so on the parent class and trigger whatever ability is assigned to the transform - what would happen if there were multiple abilities assigned to the transform, though?

    Having a look at the caching, i feel like I'm using GetComponent<> way too often in my project - I didn't quite figure this one out yet :p

    Thanks again!
     
  6. mbaske

    mbaske

    Joined:
    Dec 31, 2017
    Posts:
    473
    That sounds like you would want to use composition rather than inheritance. Like defining an IAbility interface and then assign concrete Ability instances to your agents. Either through code during runtime, or perhaps with scriptable objects, something like this https://dev.to/eriksk/implementing-...attern-using-scriptable-objects-in-unity-292i
     
  7. Sephraih

    Sephraih

    Joined:
    Jun 15, 2019
    Posts:
    11
    Thanks, I have stumbled upon the scriptable objects once, but believe i got stuck at some point and decided to revert to independent classes, which worked for that purpose - yet, it is certainly something i want to redo in the next project :)
    Thanks as well for the link :)

    Live long and prosper!
    Greetings,
    Sephraih