is there full-walkthrough or tutorial for custom python algorithm?

hello, i want to apply my custom python RL algorithm to Unity Environment that is also made by me.

i want to learn how to make my own python algorithm for ml agents and how to apply it to custom unity environment.

i tried to find any good tutorial for that, but seems like there aren't any good tutorial for up-to-date version of ml-agents.

it will be really thankful if i get any advice or tutorial links like that

 

really appreciate for advice.

well actually what i want to know that is, how should i change my own python algorithm when there are multiple training prefabs in Unity.

i saw Unity's docs which describe that i'm able to duplicate training object prefab, which is called parallel training, but it describes only when you use built-in algorithm on ml-agents.

could you give me any tutorials for this matter or advice?

 

well, first of all, i really want to say thank you all of those kinda tutorials that i need.

i don't know why i couldn't find this one, actually i tried to find any kind of this tuts pretty hard.

and your colab codes are really helpful.

here's some questions.

1. how should i change code if Unity Environment has '5 decision period' like in your
ML-Agents Q-Learning with GridWorld ?

2. to be related above question, should i widen(increase) decision period if my unity environment need to calculate many physical things(e.g collision ...)?

 

+ i tried to modify my code, and could run it with unity but somehow

after few random episodes,

below error is occurred.

https://imgur.com/a/QqPt8si

and here's my code also if can help.

https://github.com/schrodinger8834/custom_ddpg_mlagents/blob/main/test_ddpg.py

can i find something other algorithm like DDPG which i'm trying now?

because i got stuck here for few weeks..

can i get some advice for my code and above error?

 

so should i input trajectory which has DecisionSteps > 0 into replay memory and ignore when got 0 DecisionSteps and some TerminalSteps?

reason why i think like this is if code like :

Code (CSharp):

1. # Main
2. if __name__ == '__main__':
3.     # set env
4.     env = UnityEnvironment(file_name=None, seed=1, side_channels=[])
5.  
6.     # channel.set_configuration_parameters()
7.  
8.     env.reset()
9.  
10.     print('env.behavior_specs : {}\n'.format(list(env.behavior_specs)))
11.  
12.     behavior_name = list(env.behavior_specs)[0]
13.     decision_steps, terminal_steps = env.get_steps(behavior_name)
14.     spec = env.behavior_specs[behavior_name]
15.  
16.     tracked_agent = -1
17.  
18.     # DDPGAgent
19.     agent = DDPGAgent()
20.     rewards = deque(maxlen=print_interval)
21.     success_cnt = 0
22.     step = 0
23.  
24.     myobs = deque()
25.     myaction = deque()
26.     myreward = deque()
27.     myobs2 = deque()
28.     mydone = deque()
29.  
30.     # save into replay memory
31.     for episode in range(run_episode + test_episode):
32.         if episode == run_episode:
33.             train_mode = False
34.  
35.         if tracked_agent == -1 and len(decision_steps) >= 1:
36.             tracked_agent = decision_steps.agent_id[0]
37.  
38.         env.reset()
39.         decision_steps, terminal_steps = env.get_steps(behavior_name)
40.  
41.         state = decision_steps.obs[0]
42.  
43.         episode_rewards = 0
44.         done = False
45.  
46.         while not done:
47.             step += 1
48.  
49.             action = agent.get_action(state)
50.  
51.             env.set_actions(behavior_name, action)
52.             env.step()
53.  
54.             decision_steps, terminal_steps = env.get_steps(behavior_name)
55.  
56.             myobs.append(state)
57.             myaction.append(action)
58.  
59.             for agent_id_terminated in terminal_steps:
60.  
61.                 myreward.append(terminal_steps[agent_id_terminated].reward)
62.  
63.                 myobs2.append(terminal_steps[agent_id_terminated].obs[0])
64.                 mydone.append(not terminal_steps[agent_id_terminated].interrupted)
65.  
66.                 if tracked_agent == agent_id_terminated:
67.                     episode_rewards += terminal_steps[tracked_agent].reward                
68.                     done = True
69.  
70.             for agent_id_decisions in decision_steps:
71.  
72.                 myreward.append(decision_steps[agent_id_decisions].reward)
73.  
74.                 myobs2.append(decision_steps[agent_id_decisions].obs[0])
75.                 mydone.append(False)
76.  
77.                 if tracked_agent == agent_id_decisions:
78.                     episode_rewards += decision_steps[tracked_agent].reward
79.  
80.             # state <- next_state to generate new trajectory
81.             state = (np.asarray(decision_steps.obs[0])).reshape((-1, 9))
82.  
83.             if train_mode:
84.                
85.                 myobs_ = (np.asarray(myobs)).reshape((-1, 9))
86.                 myaction_ = (np.asarray(myaction)).reshape((-1, 3))
87.                 myreward_ = (np.asarray(myreward)).reshape((-1, 1))
88.                 myobs2_ = (np.asarray(myobs2)).reshape((-1, 9))
89.                 mydone_ = (np.asarray(mydone)).reshape((-1, 1))
90.  
91.                 agent.append_sample(myobs_, myaction_, myreward_, myobs2_, mydone_)
92.  
93.                 myobs.clear()
94.                 myaction.clear()
95.                 myreward.clear()
96.                 myobs2.clear()
97.                 mydone.clear()

then decision_steps.obs[0] must be empty list in

state = (np.asarray(decision_steps.obs[0])).reshape((-1, 9)) here.

so to be short,

1. should i ignore generating specific trajectory when there are NOT decision_steps.obs[0] at some timestep?

2. if not, how should i handle this problem?

 

+edit) please check this image also.

https://imgur.com/a/V6EYhGb

 

thank you and just finished implementing,

is there any possibility of occurring "KeyError"
in "generate_trajectories" function? because that error occurs after few episodes randomly.

specially, at "
obs=dict_last_obs_from_agent[agent_id_terminated].copy()" in
for agent_id_terminated in terminal_steps:
loop.

 

to be more specific, after few random episodes,

Code (CSharp):

1. def generate_trajectories(self, env, buffer_size):
2.        
3.         # Create an empty Buffer
4.         buffer: Buffer = []
5.  
6.         env.reset()
7.  
8.         behavior_name = list(env.behavior_specs)[0]
9.         spec = env.behavior_specs[behavior_name]      
10.  
11.         # Create a Mapping from AgentId to Trajectories.
12.         # This will help us create trajectories for each Agents.
13.         dict_trajectories_from_agent: Dict[int, Trajectory] = {}
14.         dict_last_obs_from_agent: Dict[int, np.ndarray] = {}
15.         dict_last_action_from_agent: Dict[int, np.ndarray] = {}
16.         # Only for reporting
17.         dict_cumulative_reward_from_agent: Dict[int, float] = {}
18.  
19.         cumulative_rewards: List[float] = []
20.  
21.         # while not enough data in the buffer
22.         while len(buffer) < buffer_size:
23.            
24.             # Get the Decision Steps and Terminal Steps of the Agents
25.             decision_steps, terminal_steps = env.get_steps(behavior_name)
26.      
27.             # For all Agents with a Terminal Step:
28.             for agent_id_terminated in terminal_steps:
29.                 # Create its last experience (is last because the Agent terminated)
30.                 last_experience = Experience(
31.                     obs=dict_last_obs_from_agent[agent_id_terminated].copy(),
32.                     # obs=dict_last_obs_from_agent[agent_id_terminated].copy(),
33.                     reward=terminal_steps[agent_id_terminated].reward,
34.                     done=not terminal_steps[agent_id_terminated].interrupted,
35.                     action=dict_last_action_from_agent[agent_id_terminated].copy(),
36.                     next_obs=terminal_steps[agent_id_terminated].obs[0]
37.                 )

at the beginning of generate_trajectories and first loop,

decision_steps has [0, 1, 2, 3, 4, 5, 6, 7, 8] and terminal_steps has [3] while dict_last_obs_from_agent is initialized with empty dictionary.

so KeyError is occurred at obs=dict_last_obs_from_agent[agent_id_terminated].copy() because dict_last_obs_from_agent is empty.

seems like there wouldn't be any problem if there is only decision_steps and empty terminal_steps at very first loop but i got KeyError because there is also terminal_steps at very first loop.

how can i solve this problem?

 

how can i deal with "ignoring" in the code?

as i know so far, when agent_id is in decision_steps, it needs "action" to proceed which means doing nothing couldn't step() agent.

could you explain more specifically in terms of writing code? i have no idea how to "do nothing" in code.

(and, this happens in my own environment not in GridWorld)

edit) i just added simply :

if agent_id_terminated not in dict_last_obs_from_agent:
break

this one just after

decision_steps, terminal_steps = env.get_steps(behavior_name)

in while len(buffer) < buffer_size loop.

is there better option?