Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Question RequestDecision do nothing

Discussion in 'ML-Agents' started by Mikahide, Mar 12, 2024.

  1. Mikahide

    Mikahide

    Joined:
    Aug 4, 2022
    Posts:
    4
    Hello! I'm trying to create a game where the agent act as the enemy and choose the attack based on their attack effectiveness. So, here's my code:

    Code (CSharp):
    1. using System;
    2. using System.Collections;
    3. using System.Collections.Generic;
    4. using Unity.MLAgents;
    5. using Unity.MLAgents.Actuators;
    6. using Unity.MLAgents.Sensors;
    7. using Unity.MLAgents.Sensors.Reflection;
    8. using UnityEngine;
    9.  
    10. public class EnemyAIAgent : Agent
    11. {
    12.     public float attackPower = 10f;
    13.     public float weaknessMultiplier = 2f;
    14.     public float resistanceMultiplier = 0.5f;
    15.     public float blockMultiplier = 0.2f;
    16.  
    17.     private int NUM_ELEMENT;
    18.     private float reward = 0f;
    19.  
    20.     public AttackScript attackScript;
    21.  
    22.     void Start()
    23.     {
    24.         attackScript = GetComponent<AttackScript>();
    25.     }
    26.  
    27.     public override void OnEpisodeBegin()
    28.     {
    29.         RequestDecision();
    30.         // RequestAction();
    31.     }
    32.  
    33.     public override void CollectObservations(VectorSensor sensor)
    34.     {
    35.         // Observe the attack element
    36.         NUM_ELEMENT = (int)AttackScript.magicElement.LastElement;
    37.         sensor.AddObservation(NUM_ELEMENT);
    38.        
    39.     }
    40.  
    41.     public void AgentAttack(ActionSegment<int> act)
    42.     {
    43.         FighterStats fighterStats = GetComponent<FighterStats>();
    44.         FighterAction fighterAction = GetComponent<FighterAction>();
    45.         var dirToGo = Vector3.zero;
    46.         var rotateDir = Vector3.zero;
    47.         var physicalAttack = act[0];
    48.         var fireAttack = act[1];
    49.  
    50.         if(physicalAttack == 1)
    51.         {
    52.             attackScript = GameObject.Find("EMeleePrefab").GetComponent<AttackScript>();
    53.             fighterAction.SelectAttack("melee");
    54.             Debug.Log("Melee attack");
    55.         } else if (fireAttack == 1)
    56.         {
    57.             attackScript = GameObject.Find("EFireMagicPrefab").GetComponent<AttackScript>();
    58.             fighterAction.SelectAttack("Fire");
    59.             Debug.Log("Fire attack");
    60.         }
    61.  
    62.         if (physicalAttack == 1 || fireAttack == 1)
    63.         {
    64.             if (attackScript.IsBlockingAttack)
    65.             {
    66.                 // reward = -attackPower * blockMultiplier;
    67.                 reward = -1f;
    68.             } else if (attackScript.IsResistingAttack)
    69.             {
    70.                 // reward = -attackPower * resistanceMultiplier;
    71.                 reward = -0.5f;
    72.             } else if (attackScript.IsWeakToAttack)
    73.             {
    74.                 // reward = attackPower * weaknessMultiplier;
    75.                 reward = 1f;
    76.  
    77.             } else
    78.             {
    79.                 // reward = attackPower;
    80.                 reward = 0.5f;
    81.             }
    82.         }
    83.     }
    84.  
    85.  
    86.     public override void OnActionReceived(ActionBuffers actions)
    87.     {
    88.  
    89.         AgentAttack(actions.DiscreteActions);
    90.  
    91.         // Apply the reward
    92.         AddReward(reward);
    93.         Debug.Log("Reward: " + reward);
    94.  
    95.         // NextTurn();
    96.         EndEpisode();
    97.     }
    98. }
    So, I notice my CollectObservation method seem not work because when I call RequestDecision on my GameController, the agent does nothing. Any suggestion about what I should do?
     
  2. smallg2023

    smallg2023

    Joined:
    Sep 2, 2018
    Posts:
    154
    so what makes you say collectobservation isn't doing anything? is it being executed?
    do you have 2 discrete branches of size 2?
     
    Mikahide likes this.
  3. Mikahide

    Mikahide

    Joined:
    Aug 4, 2022
    Posts:
    4
    Well CollectObservation method is executed after I try, i try debug my NUM_ELEMENT. I have 2 discrete branches of size 1. Gosh, it really worked out when the size is 2, but I have two follow up question:
    1. Why should the size be 2 not 1? Because my if method just use 1 instead of 2 (e.g if physicalAtttack == 1, fireAttack == 1)
    2. My mana is being quickly depleted to 0. Is this a bug or because the ML Agents still run too quickly? Even I use time-scale = 1 when i run ML Agents. This is also my curiosity because when I use Debug.Log at NUM_ELEMENT, the log came up endlessly.
     
  4. smallg2023

    smallg2023

    Joined:
    Sep 2, 2018
    Posts:
    154
    arrays are 0 indexed so branch of 1 is going to always output 0, so if your 'if' statements are checking for 1 they'll never be true until you make them branch size of 2 (now the outputs are 0 or 1)

    at timescale 1 ML will run in fixed update so 50 frames in 1second, you are constantly ending and starting the episode in each step so it will run the same logic 50 times in 1 second, if you want the agent to run less you need to call requestdecision less often
     
    Mikahide likes this.
  5. Mikahide

    Mikahide

    Joined:
    Aug 4, 2022
    Posts:
    4
    Thank you very much for your explanation, I want to ask one last question (hopefully so), I tried to use
    Academy.Instance.AutomaticSteppingEnabled = false;
    , to limit the step, but i think it still the problem because I called the EnvironmentStep at FixedUpdate, can I reduce the 50 frame or somehting to limit it so it called once?

    Here is my full code:

    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4. using UnityEngine.UI;
    5. using System.Transactions;
    6. using UnityEngine.SocialPlatforms;
    7. using Unity.MLAgents;
    8. using Unity.MLAgents.Actuators;
    9. using Unity.MLAgents.Sensors;
    10.  
    11. public class GameController : MonoBehaviour
    12. {
    13.     private List<FighterStats> fighterStats;
    14.  
    15.     private GameObject battleMenu;
    16.  
    17.     public Text battleText;
    18.  
    19.     private void Awake()
    20.     {
    21.         battleMenu = GameObject.Find("ActionMenu");
    22.     }
    23.     void Start()
    24.     {
    25.         Academy.Instance.AutomaticSteppingEnabled = false;
    26.         fighterStats = new List<FighterStats>();
    27.         GameObject hero = GameObject.FindGameObjectWithTag("Hero");
    28.         FighterStats currentFighterStats = hero.GetComponent<FighterStats>();
    29.         currentFighterStats.CalculateNextTurn(0);
    30.         fighterStats.Add(currentFighterStats);
    31.  
    32.         GameObject enemy = GameObject.FindGameObjectWithTag("Enemy");
    33.         FighterStats currentEnemyStats = enemy.GetComponent<FighterStats>();
    34.         currentEnemyStats.CalculateNextTurn(0);
    35.         fighterStats.Add(currentEnemyStats);
    36.  
    37.         fighterStats.Sort();
    38.  
    39.         NextTurn();
    40.     }
    41.  
    42.     private void FixedUpdate()
    43.     {
    44.         Academy.Instance.AutomaticSteppingEnabled = false;
    45.         Academy.Instance.EnvironmentStep();
    46.     }
    47.  
    48.     public void NextTurn()
    49.     {
    50.         battleText.gameObject.SetActive(false);
    51.         FighterStats currentFighterStats = fighterStats[0];
    52.         fighterStats.Remove(currentFighterStats);
    53.         if (!currentFighterStats.GetDead())
    54.         {
    55.             GameObject currentUnit = currentFighterStats.gameObject;
    56.             currentFighterStats.CalculateNextTurn(currentFighterStats.nextActTurn);
    57.             Debug.Log("NextActTurn: " + currentFighterStats.nextActTurn);
    58.             fighterStats.Add(currentFighterStats);
    59.             fighterStats.Sort();
    60.             if (currentUnit.CompareTag("Hero"))
    61.             {
    62.                 battleMenu.SetActive(true);
    63.                 Debug.Log("Hero's turn");
    64.             }
    65.             else
    66.             {
    67.                 battleMenu.SetActive(false);
    68.                 Debug.Log("Enemy's turn");
    69.  
    70.                 // Scripted AI
    71.                 // string attackType = Random.Range(0, 2) == 1 ? "melee" : "range";
    72.                 // currentUnit.GetComponent<FighterAction>().SelectAttack(attackType);
    73.  
    74.                 // ML-Agents AI
    75.                 // float[] actions = new float[2];
    76.                 // float[] actions = new float[] { 0, 1, 0, 0, 0, 0, 0 };
    77.                 // ActionBuffers actionBuffers = ActionBuffers.FromDiscreteActions(actions);
    78.                 // currentUnit.GetComponent<EnemyAIAgent>().OnActionReceived(actionBuffers);
    79.                 currentUnit.GetComponent<EnemyAIAgent>().RequestDecision();
    80.            
    81.             }
    82.         }
    83.         else
    84.         {
    85.             NextTurn();
    86.         }
    87.     }
    88. }
    89.  
     
  6. smallg2023

    smallg2023

    Joined:
    Sep 2, 2018
    Posts:
    154
    simplest method is perhaps removing the manual calling of request decision / environmentstep and using the decision requester component as that lets you add a custom delay (if you need more control then you can code your own)

    p.s. you have an infinite loop of calling NextTurn when the agent is dead
     
    Mikahide likes this.
  7. Mikahide

    Mikahide

    Joined:
    Aug 4, 2022
    Posts:
    4
    Thank you very much for your answer, I'm experimenting between using RequestDecision method or DecisionRequester component.

    And I appreaciate your notice about my infinite loop, I'm currently fixing it.