Question How to resume training after 2 billions steps?

Hi.
I'm trying to train my 2d ragdoll character to take the right position on 14 points - 1 for each bone.
I use PPO and can see my entropy slowly goes down from 1.5 to 0.26 until now.

Here i've met some problem: when training reached 2147400000+ step - it stopped with NaN relative exceptions. So when i'm trying to resume the training - it's just starts from: -2147483625 step (from negative step) and immediately stops.

So i need help about how to resume training after 2 billions steps? Is that the limit?

 

Maybe start a new training run using --initialize-from
Haven't tried this myself yet, but from what I've read in the docs, it sounds like that could be a workaround for your issue.

 

Already tried. And even --initialize-from from other training which was started with --initialize-from from origin 2 bills train - the same exception result. It feels like some int var is stored inside origin brain.

 

Here is first exceptions and secondary after resume

Spoiler: First exception when steps count became too large

2020-10-07 21:44:14 INFO [stats.py:111] PatientPose: Step: 2147200000. Time Elapsed: 68309.679 s Mean Reward: 439.154. Std of Reward: 77.079. Training.
2020-10-07 21:45:40 INFO [stats.py:111] PatientPose: Step: 2147300000. Time Elapsed: 68395.498 s Mean Reward: 432.955. Std of Reward: 89.715. Training.
2020-10-07 21:46:55 INFO [stats.py:111] PatientPose: Step: 2147400000. Time Elapsed: 68470.389 s Mean Reward: 448.907. Std of Reward: 69.031. Training.
2020-10-07 21:47:50 INFO [environment.py:418] Environment shut down with return code 0.
Process Process-6:
2020-10-07 21:47:50 INFO [environment.py:418] Environment shut down with return code 0.
Process Process-1:
2020-10-07 21:47:50 INFO [environment.py:418] Environment shut down with return code 0.
Process Process-4:
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
env.step()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/m File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
env.step()
ultiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
self._update_state(rl_output)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
self._update_state(rl_output)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
env.step()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
agent_info_list, self._env_specs[brain_name]
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
agent_info_list, self._env_specs[brain_name]
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
self._update_state(rl_output)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
obs_index, obs_shape, decision_agent_info_list
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
obs_index, obs_shape, decision_agent_info_list
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
agent_info_list, self._env_specs[brain_name]
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
_raise_on_nan_and_inf(np_obs, "observations")
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
_raise_on_nan_and_inf(np_obs, "observations")
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
raise RuntimeError(f"The {source} provided had NaN values.")
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)

raise RuntimeError(f"The {source} provided had NaN values.")
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
obs_index, obs_shape, decision_agent_info_list
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
_raise_on_nan_and_inf(np_obs, "observations")
RuntimeError: The observations provided had NaN values.
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
raise RuntimeError(f"The {source} provided had NaN values.")
RuntimeError: The observations provided had NaN values.
RuntimeError: The observations provided had NaN values.
2020-10-07 21:47:50 INFO [trainer_controller.py:234] Learning was interrupted. Please wait while the graph is generated.
2020-10-07 21:47:50 INFO [environment.py:418] Environment shut down with return code 0.
Process Process-5:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
env.step()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
self._update_state(rl_output)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
agent_info_list, self._env_specs[brain_name]
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
obs_index, obs_shape, decision_agent_info_list
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
_raise_on_nan_and_inf(np_obs, "observations")
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
raise RuntimeError(f"The {source} provided had NaN values.")
RuntimeError: The observations provided had NaN values.
2020-10-07 21:47:50 INFO [environment.py:418] Environment shut down with return code 0.
Process Process-2:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
env.step()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
self._update_state(rl_output)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
agent_info_list, self._env_specs[brain_name]
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
obs_index, obs_shape, decision_agent_info_list
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
_raise_on_nan_and_inf(np_obs, "observations")
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
raise RuntimeError(f"The {source} provided had NaN values.")
RuntimeError: The observations provided had NaN values.
2020-10-07 21:47:50 INFO [trainer_controller.py:108] Saved Model
2020-10-07 21:47:50 INFO [model_serialization.py:203] List of nodes to export for brain atientPose?team=0
2020-10-07 21:47:50 INFO [model_serialization.py:205] is_continuous_control
2020-10-07 21:47:50 INFO [model_serialization.py:205] trainer_major_version
2020-10-07 21:47:50 INFO [model_serialization.py:205] trainer_minor_version
2020-10-07 21:47:50 INFO [model_serialization.py:205] trainer_patch_version
2020-10-07 21:47:50 INFO [model_serialization.py:205] version_number
2020-10-07 21:47:50 INFO [model_serialization.py:205] memory_size
2020-10-07 21:47:50 INFO [model_serialization.py:205] action_output_shape
2020-10-07 21:47:50 INFO [model_serialization.py:205] action
2020-10-07 21:47:50 INFO [model_serialization.py:205] action_probs
Converting results/393_Patient_Pose_Train/PatientPose/frozen_graph_def.pb to results/393_Patient_Pose_Train/PatientPose.nn
IGNORED: Cast unknown layer
IGNORED: Shape unknown layer
IGNORED: StopGradient unknown layer
GLOBALS: 'is_continuous_control', 'trainer_major_version', 'trainer_minor_version', 'trainer_patch_version', 'version_number', 'memory_size', 'action_output_shape'
IN: 'vector_observation': [-1, 1, 1, 153] => 'sub_2'
OUT: 'action', 'action_probs'
DONE: wrote results/393_Patient_Pose_Train/PatientPose.nn file.
2020-10-07 21:47:51 INFO [model_serialization.py:83] Exported results/393_Patient_Pose_Train/PatientPose.nn file
2020-10-07 21:47:52 INFO [environment.py:418] Environment shut down with return code 0.
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 89, in recv
response: EnvironmentResponse = self.conn.recv()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
buf = self._recv(4)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/bin/mlagents-learn", line 8, in <module>
sys.exit(main())
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/learn.py", line 322, in main
run_cli(parse_command_line())
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/learn.py", line 318, in run_cli
run_training(run_seed, options)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/learn.py", line 163, in run_training
tc.start_learning(env_manager)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/trainer_controller.py", line 243, in start_learning
raise ex
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/trainer_controller.py", line 216, in start_learning
external_brain_behavior_ids = set(env_manager.external_brains.keys())
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 301, in external_brains
return self.env_workers[0].recv().payload
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 95, in recv
raise UnityCommunicationException("UnityEnvironment worker: recv failed.")
mlagents_envs.exception.UnityCommunicationException: UnityEnvironment worker: recv failed.

Spoiler: Exceptions when triying to resume this training - negative steps appearance

2020-10-07 21:57:04 INFO [tf_policy.py:165] Loading model for brain PatientPose?team=0 from results/393_Patient_Pose_Train/PatientPose.
2020-10-07 21:57:04 INFO [tf_policy.py:196] Resuming training from step -2147483625.
2020-10-07 21:57:06 INFO [environment.py:418] Environment shut down with return code 0.
Process Process-6:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
env.step()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
self._update_state(rl_output)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
agent_info_list, self._env_specs[brain_name]
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
obs_index, obs_shape, decision_agent_info_list
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
_raise_on_nan_and_inf(np_obs, "observations")
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
raise RuntimeError(f"The {source} provided had NaN values.")
RuntimeError: The observations provided had NaN values.
2020-10-07 21:57:06 INFO [environment.py:418] Environment shut down with return code 0.
Process Process-2:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
env.step()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
self._update_state(rl_output)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
agent_info_list, self._env_specs[brain_name]
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
2020-10-07 21:57:06 INFO [environment.py:418] Environment shut down with return code 0.
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
obs_index, obs_shape, decision_agent_info_list
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
_raise_on_nan_and_inf(np_obs, "observations")
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
raise RuntimeError(f"The {source} provided had NaN values.")
RuntimeError: The observations provided had NaN values.
Process Process-4:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
env.step()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
self._update_state(rl_output)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
agent_info_list, self._env_specs[brain_name]
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
obs_index, obs_shape, decision_agent_info_list
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
_raise_on_nan_and_inf(np_obs, "observations")
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
raise RuntimeError(f"The {source} provided had NaN values.")
RuntimeError: The observations provided had NaN values.
2020-10-07 21:57:06 INFO [environment.py:418] Environment shut down with return code 0.
Process Process-3:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
env.step()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
self._update_state(rl_output)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
agent_info_list, self._env_specs[brain_name]
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
obs_index, obs_shape, decision_agent_info_list
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
_raise_on_nan_and_inf(np_obs, "observations")
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
raise RuntimeError(f"The {source} provided had NaN values.")
RuntimeError: The observations provided had NaN values.
2020-10-07 21:57:06 INFO [environment.py:418] Environment shut down with return code 0.
Process Process-5:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
env.step()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
self._update_state(rl_output)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
agent_info_list, self._env_specs[brain_name]
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
obs_index, obs_shape, decision_agent_info_list
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
_raise_on_nan_and_inf(np_obs, "observations")
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
raise RuntimeError(f"The {source} provided had NaN values.")
RuntimeError: The observations provided had NaN values.
2020-10-07 21:57:06 INFO [environment.py:418] Environment shut down with return code 0.
Process Process-1:
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
self.run()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
self._target(*self._args, **self._kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
env.step()
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
self._update_state(rl_output)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
agent_info_list, self._env_specs[brain_name]
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
obs_index, obs_shape, decision_agent_info_list
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
return func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
_raise_on_nan_and_inf(np_obs, "observations")
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
raise RuntimeError(f"The {source} provided had NaN values.")
RuntimeError: The observations provided had NaN values.
^C2020-10-07 21:58:17 INFO [trainer_controller.py:234] Learning was interrupted. Please wait while the graph is generated.
2020-10-07 21:58:17 INFO [trainer_controller.py:108] Saved Model
2020-10-07 21:58:17 INFO [model_serialization.py:203] List of nodes to export for brain atientPose?team=0
2020-10-07 21:58:17 INFO [model_serialization.py:205] is_continuous_control
2020-10-07 21:58:17 INFO [model_serialization.py:205] trainer_major_version
2020-10-07 21:58:17 INFO [model_serialization.py:205] trainer_minor_version
2020-10-07 21:58:17 INFO [model_serialization.py:205] trainer_patch_version
2020-10-07 21:58:17 INFO [model_serialization.py:205] version_number
2020-10-07 21:58:17 INFO [model_serialization.py:205] memory_size
2020-10-07 21:58:17 INFO [model_serialization.py:205] action_output_shape
2020-10-07 21:58:17 INFO [model_serialization.py:205] action
2020-10-07 21:58:17 INFO [model_serialization.py:205] action_probs
Converting results/393_Patient_Pose_Train/PatientPose/frozen_graph_def.pb to results/393_Patient_Pose_Train/PatientPose.nn
IGNORED: Cast unknown layer
IGNORED: Shape unknown layer
IGNORED: StopGradient unknown layer
GLOBALS: 'is_continuous_control', 'trainer_major_version', 'trainer_minor_version', 'trainer_patch_version', 'version_number', 'memory_size', 'action_output_shape'
IN: 'vector_observation': [-1, 1, 1, 153] => 'sub_2'
OUT: 'action', 'action_probs'
DONE: wrote results/393_Patient_Pose_Train/PatientPose.nn file.