ML agents: 3D ball agent script explanation.

Could someone explain this script used in ML agents 3d ball example?

I know how rewarding and punishing works, but I'm having trouble figuring out what's going on in AgentStep(). Also what variables should be put in CollectState()?

This post can be very useful for other beginners.

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4.  
5. public class Ball3DAgent : Agent
6. {
7.     [Header("Specific to Ball3D")]
8.     public GameObject ball;
9.  
10.     public override List<float> CollectState()
11.     {
12.         List<float> state = new List<float>();
13.         state.Add(gameObject.transform.rotation.z);
14.         state.Add(gameObject.transform.rotation.x);
15.         state.Add((ball.transform.position.x - gameObject.transform.position.x));
16.         state.Add((ball.transform.position.y - gameObject.transform.position.y));
17.         state.Add((ball.transform.position.z - gameObject.transform.position.z));
18.         state.Add(ball.transform.GetComponent<Rigidbody>().velocity.x);
19.         state.Add(ball.transform.GetComponent<Rigidbody>().velocity.y);
20.         state.Add(ball.transform.GetComponent<Rigidbody>().velocity.z);
21.         return state;
22.     }
23.  
24.     // to be implemented by the developer
25.     public override void AgentStep(float[] act)
26.     {
27.         if (brain.brainParameters.actionSpaceType == StateType.continuous)
28.         {
29.             float action_z = act[0];
30.             if (action_z > 2f)
31.             {
32.                 action_z = 2f;
33.             }
34.             if (action_z < -2f)
35.             {
36.                 action_z = -2f;
37.             }6
38.             if ((gameObject.transform.rotation.z < 0.25f && action_z > 0f) ||
39.                 (gameObject.transform.rotation.z > -0.25f && action_z < 0f))
40.             {
41.                 gameObject.transform.Rotate(new Vector3(0, 0, 1), action_z);
42.  
43.             }
44.             float action_x = act[1];
45.             if (action_x > 2f)
46.             {
47.                 action_x = 2f;
48.             }
49.             if (action_x < -2f)
50.             {
51.                 action_x = -2f;
52.             }
53.             if ((gameObject.transform.rotation.x < 0.25f && action_x > 0f) ||
54.                 (gameObject.transform.rotation.x > -0.25f && action_x < 0f))
55.             {
56.                 gameObject.transform.Rotate(new Vector3(1, 0, 0), action_x);
57.             }
58.                
59.  
60.             if (done == false)
61.             {
62.                 reward = 0.1f;
63.             }
64.         }
65.         else
66.         {
67.             int action = (int)act[0];
68.             if (action == 0 || action == 1)
69.             {
70.                 action = (action * 2) - 1;
71.                 float changeValue = action * 2f;
72.                 if ((gameObject.transform.rotation.z < 0.25f && changeValue > 0f) ||
73.                     (gameObject.transform.rotation.z > -0.25f && changeValue < 0f))
74.                 {
75.                     gameObject.transform.Rotate(new Vector3(0, 0, 1), changeValue);
76.                 }
77.             }
78.             if (action == 2 || action == 3)
79.             {
80.                 action = ((action - 2) * 2) - 1;
81.                 float changeValue = action * 2f;
82.                 if ((gameObject.transform.rotation.x < 0.25f && changeValue > 0f) ||
83.                     (gameObject.transform.rotation.x > -0.25f && changeValue < 0f))
84.                 {
85.                     gameObject.transform.Rotate(new Vector3(1, 0, 0), changeValue);
86.                 }
87.             }
88.             if (done == false)
89.             {
90.                 reward = 0.1f;
91.             }
92.         }
93.         if ((ball.transform.position.y - gameObject.transform.position.y) < -2f ||
94.             Mathf.Abs(ball.transform.position.x - gameObject.transform.position.x) > 3f ||
95.             Mathf.Abs(ball.transform.position.z - gameObject.transform.position.z) > 3f)
96.         {
97.             done = true;
98.             reward = -1f;
99.         }
100.  
101.     }
102.  
103.     // to be implemented by the developer
104.     public override void AgentReset()
105.     {
106.         gameObject.transform.rotation = new Quaternion(0f, 0f, 0f, 0f);
107.         gameObject.transform.Rotate(new Vector3(1, 0, 0), Random.Range(-10f, 10f));
108.         gameObject.transform.Rotate(new Vector3(0, 0, 1), Random.Range(-10f, 10f));
109.         ball.GetComponent<Rigidbody>().velocity = new Vector3(0f, 0f, 0f);
110.         ball.transform.position = new Vector3(Random.Range(-1.5f, 1.5f), 4f, Random.Range(-1.5f, 1.5f)) + gameObject.transform.position;
111.     }
112. }

 

I think that you are looking at some really old code. AgentStep isn't a method that is overrideable in the 0.3+ versions, that I know of.

n/m I see that your post is from a long time ago....

 

still need help on this, where do I find a good tutorial for ML agents.

 

Surajsirohi,
Here is lots of into on ml-agents (V0.4 for now, that's what I'm using) https://github.com/Unity-Technologies/ml-agents/tree/master/docs

And within that doc is a good example (the first one that I did): https://github.com/Unity-Technologies/ml-agents/blob/master/docs/Learning-Environment-Create-New.md

 

than

thanks, I will check it out.

 

I tried the Roller Agent, why does the training stops at 50000 steps? How can I increase the number of steps? After training I'm getting a mean reward of 2, it's too low.

 

You can change any part of the training in the "trainer_config.yaml" file (located in the Assets\ml_agents_master\python folder)
Look at some of the "brain names" in that file. Maybe even rename your "brain" GameObject so that it uses a different brain trainer in that file. It will use default or will use the brain name listed in the trainer yaml file if it matches your gameobject brain name.
You could copy one of the brain sections, paste it at the bottom of the file, and rename it to the name of your GameObject that has the brain component on it.

 

thanks again mate.

 

@surajsirohi1008 Note my edit above

"You could copy one of the brain sections, paste it at the bottom of the file, and rename it to the name of your GameObject that has the brain component on it. "

 

Noted, thanks.

 

Also, note that you can modify the trainer file between runs. If you run it and it look like you need another 100k steps or something, just edit the trainer yaml file to up the max_steps by that amount and run again. It will continue from the number of steps it did before up to the new max steps. Works well when viewing the summaries in the browser as well.

 

Muchas gracias