Search Unity

  1. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Question How to resume training after 2 billions steps?

Discussion in 'ML-Agents' started by gr1ns1de, Oct 8, 2020.

  1. gr1ns1de

    gr1ns1de

    Joined:
    Feb 14, 2015
    Posts:
    4
    Hi.
    I'm trying to train my 2d ragdoll character to take the right position on 14 points - 1 for each bone.
    I use PPO and can see my entropy slowly goes down from 1.5 to 0.26 until now.

    Here i've met some problem: when training reached 2147400000+ step - it stopped with NaN relative exceptions. So when i'm trying to resume the training - it's just starts from: -2147483625 step (from negative step) and immediately stops.

    So i need help about how to resume training after 2 billions steps? Is that the limit?
     
  2. mbaske

    mbaske

    Joined:
    Dec 31, 2017
    Posts:
    473
    Maybe start a new training run using --initialize-from
    Haven't tried this myself yet, but from what I've read in the docs, it sounds like that could be a workaround for your issue.
     
  3. gr1ns1de

    gr1ns1de

    Joined:
    Feb 14, 2015
    Posts:
    4
    Already tried. And even --initialize-from from other training which was started with --initialize-from from origin 2 bills train - the same exception result. It feels like some int var is stored inside origin brain.
     
  4. andrewcoh_unity

    andrewcoh_unity

    Unity Technologies

    Joined:
    Sep 5, 2019
    Posts:
    162
    Can you share the full stack trace/log containing the exceptions?
     
  5. gr1ns1de

    gr1ns1de

    Joined:
    Feb 14, 2015
    Posts:
    4
    Here is first exceptions and secondary after resume

    2020-10-07 21:44:14 INFO [stats.py:111] PatientPose: Step: 2147200000. Time Elapsed: 68309.679 s Mean Reward: 439.154. Std of Reward: 77.079. Training.
    2020-10-07 21:45:40 INFO [stats.py:111] PatientPose: Step: 2147300000. Time Elapsed: 68395.498 s Mean Reward: 432.955. Std of Reward: 89.715. Training.
    2020-10-07 21:46:55 INFO [stats.py:111] PatientPose: Step: 2147400000. Time Elapsed: 68470.389 s Mean Reward: 448.907. Std of Reward: 69.031. Training.
    2020-10-07 21:47:50 INFO [environment.py:418] Environment shut down with return code 0.
    Process Process-6:
    2020-10-07 21:47:50 INFO [environment.py:418] Environment shut down with return code 0.
    Process Process-1:
    2020-10-07 21:47:50 INFO [environment.py:418] Environment shut down with return code 0.
    Process Process-4:
    Traceback (most recent call last):
    Traceback (most recent call last):
    Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
    env.step()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/m File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
    env.step()
    ultiprocessing/process.py", line 297, in _bootstrap
    self.run()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
    self._update_state(rl_output)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
    self._update_state(rl_output)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
    env.step()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
    agent_info_list, self._env_specs[brain_name]
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
    agent_info_list, self._env_specs[brain_name]
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
    self._update_state(rl_output)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
    obs_index, obs_shape, decision_agent_info_list
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
    obs_index, obs_shape, decision_agent_info_list
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
    agent_info_list, self._env_specs[brain_name]
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
    _raise_on_nan_and_inf(np_obs, "observations")
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
    _raise_on_nan_and_inf(np_obs, "observations")
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
    raise RuntimeError(f"The {source} provided had NaN values.")
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)

    raise RuntimeError(f"The {source} provided had NaN values.")
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
    obs_index, obs_shape, decision_agent_info_list
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
    _raise_on_nan_and_inf(np_obs, "observations")
    RuntimeError: The observations provided had NaN values.
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
    raise RuntimeError(f"The {source} provided had NaN values.")
    RuntimeError: The observations provided had NaN values.
    RuntimeError: The observations provided had NaN values.
    2020-10-07 21:47:50 INFO [trainer_controller.py:234] Learning was interrupted. Please wait while the graph is generated.
    2020-10-07 21:47:50 INFO [environment.py:418] Environment shut down with return code 0.
    Process Process-5:
    Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
    env.step()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
    self._update_state(rl_output)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
    agent_info_list, self._env_specs[brain_name]
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
    obs_index, obs_shape, decision_agent_info_list
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
    _raise_on_nan_and_inf(np_obs, "observations")
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
    raise RuntimeError(f"The {source} provided had NaN values.")
    RuntimeError: The observations provided had NaN values.
    2020-10-07 21:47:50 INFO [environment.py:418] Environment shut down with return code 0.
    Process Process-2:
    Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
    env.step()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
    self._update_state(rl_output)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
    agent_info_list, self._env_specs[brain_name]
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
    obs_index, obs_shape, decision_agent_info_list
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
    _raise_on_nan_and_inf(np_obs, "observations")
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
    raise RuntimeError(f"The {source} provided had NaN values.")
    RuntimeError: The observations provided had NaN values.
    2020-10-07 21:47:50 INFO [trainer_controller.py:108] Saved Model
    2020-10-07 21:47:50 INFO [model_serialization.py:203] List of nodes to export for brain :patientPose?team=0
    2020-10-07 21:47:50 INFO [model_serialization.py:205] is_continuous_control
    2020-10-07 21:47:50 INFO [model_serialization.py:205] trainer_major_version
    2020-10-07 21:47:50 INFO [model_serialization.py:205] trainer_minor_version
    2020-10-07 21:47:50 INFO [model_serialization.py:205] trainer_patch_version
    2020-10-07 21:47:50 INFO [model_serialization.py:205] version_number
    2020-10-07 21:47:50 INFO [model_serialization.py:205] memory_size
    2020-10-07 21:47:50 INFO [model_serialization.py:205] action_output_shape
    2020-10-07 21:47:50 INFO [model_serialization.py:205] action
    2020-10-07 21:47:50 INFO [model_serialization.py:205] action_probs
    Converting results/393_Patient_Pose_Train/PatientPose/frozen_graph_def.pb to results/393_Patient_Pose_Train/PatientPose.nn
    IGNORED: Cast unknown layer
    IGNORED: Shape unknown layer
    IGNORED: StopGradient unknown layer
    GLOBALS: 'is_continuous_control', 'trainer_major_version', 'trainer_minor_version', 'trainer_patch_version', 'version_number', 'memory_size', 'action_output_shape'
    IN: 'vector_observation': [-1, 1, 1, 153] => 'sub_2'
    OUT: 'action', 'action_probs'
    DONE: wrote results/393_Patient_Pose_Train/PatientPose.nn file.
    2020-10-07 21:47:51 INFO [model_serialization.py:83] Exported results/393_Patient_Pose_Train/PatientPose.nn file
    2020-10-07 21:47:52 INFO [environment.py:418] Environment shut down with return code 0.
    Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 89, in recv
    response: EnvironmentResponse = self.conn.recv()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 250, in recv
    buf = self._recv_bytes()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 407, in _recv_bytes
    buf = self._recv(4)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/connection.py", line 383, in _recv
    raise EOFError
    EOFError

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.7/bin/mlagents-learn", line 8, in <module>
    sys.exit(main())
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/learn.py", line 322, in main
    run_cli(parse_command_line())
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/learn.py", line 318, in run_cli
    run_training(run_seed, options)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/learn.py", line 163, in run_training
    tc.start_learning(env_manager)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/trainer_controller.py", line 243, in start_learning
    raise ex
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/trainer_controller.py", line 216, in start_learning
    external_brain_behavior_ids = set(env_manager.external_brains.keys())
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 301, in external_brains
    return self.env_workers[0].recv().payload
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 95, in recv
    raise UnityCommunicationException("UnityEnvironment worker: recv failed.")
    mlagents_envs.exception.UnityCommunicationException: UnityEnvironment worker: recv failed.


    2020-10-07 21:57:04 INFO [tf_policy.py:165] Loading model for brain PatientPose?team=0 from results/393_Patient_Pose_Train/PatientPose.
    2020-10-07 21:57:04 INFO [tf_policy.py:196] Resuming training from step -2147483625.
    2020-10-07 21:57:06 INFO [environment.py:418] Environment shut down with return code 0.
    Process Process-6:
    Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
    env.step()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
    self._update_state(rl_output)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
    agent_info_list, self._env_specs[brain_name]
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
    obs_index, obs_shape, decision_agent_info_list
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
    _raise_on_nan_and_inf(np_obs, "observations")
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
    raise RuntimeError(f"The {source} provided had NaN values.")
    RuntimeError: The observations provided had NaN values.
    2020-10-07 21:57:06 INFO [environment.py:418] Environment shut down with return code 0.
    Process Process-2:
    Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
    env.step()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
    self._update_state(rl_output)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
    agent_info_list, self._env_specs[brain_name]
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    2020-10-07 21:57:06 INFO [environment.py:418] Environment shut down with return code 0.
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
    obs_index, obs_shape, decision_agent_info_list
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
    _raise_on_nan_and_inf(np_obs, "observations")
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
    raise RuntimeError(f"The {source} provided had NaN values.")
    RuntimeError: The observations provided had NaN values.
    Process Process-4:
    Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
    env.step()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
    self._update_state(rl_output)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
    agent_info_list, self._env_specs[brain_name]
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
    obs_index, obs_shape, decision_agent_info_list
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
    _raise_on_nan_and_inf(np_obs, "observations")
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
    raise RuntimeError(f"The {source} provided had NaN values.")
    RuntimeError: The observations provided had NaN values.
    2020-10-07 21:57:06 INFO [environment.py:418] Environment shut down with return code 0.
    Process Process-3:
    Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
    env.step()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
    self._update_state(rl_output)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
    agent_info_list, self._env_specs[brain_name]
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
    obs_index, obs_shape, decision_agent_info_list
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
    _raise_on_nan_and_inf(np_obs, "observations")
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
    raise RuntimeError(f"The {source} provided had NaN values.")
    RuntimeError: The observations provided had NaN values.
    2020-10-07 21:57:06 INFO [environment.py:418] Environment shut down with return code 0.
    Process Process-5:
    Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
    env.step()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
    self._update_state(rl_output)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
    agent_info_list, self._env_specs[brain_name]
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
    obs_index, obs_shape, decision_agent_info_list
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
    _raise_on_nan_and_inf(np_obs, "observations")
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
    raise RuntimeError(f"The {source} provided had NaN values.")
    RuntimeError: The observations provided had NaN values.
    2020-10-07 21:57:06 INFO [environment.py:418] Environment shut down with return code 0.
    Process Process-1:
    Traceback (most recent call last):
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents/trainers/subprocess_env_manager.py", line 157, in worker
    env.step()
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 319, in step
    self._update_state(rl_output)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/environment.py", line 275, in _update_state
    agent_info_list, self._env_specs[brain_name]
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 188, in steps_from_proto
    obs_index, obs_shape, decision_agent_info_list
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/timers.py", line 305, in wrapped
    return func(*args, **kwargs)
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 152, in _process_vector_observation
    _raise_on_nan_and_inf(np_obs, "observations")
    File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/mlagents_envs/rpc_utils.py", line 130, in _raise_on_nan_and_inf
    raise RuntimeError(f"The {source} provided had NaN values.")
    RuntimeError: The observations provided had NaN values.
    ^C2020-10-07 21:58:17 INFO [trainer_controller.py:234] Learning was interrupted. Please wait while the graph is generated.
    2020-10-07 21:58:17 INFO [trainer_controller.py:108] Saved Model
    2020-10-07 21:58:17 INFO [model_serialization.py:203] List of nodes to export for brain :patientPose?team=0
    2020-10-07 21:58:17 INFO [model_serialization.py:205] is_continuous_control
    2020-10-07 21:58:17 INFO [model_serialization.py:205] trainer_major_version
    2020-10-07 21:58:17 INFO [model_serialization.py:205] trainer_minor_version
    2020-10-07 21:58:17 INFO [model_serialization.py:205] trainer_patch_version
    2020-10-07 21:58:17 INFO [model_serialization.py:205] version_number
    2020-10-07 21:58:17 INFO [model_serialization.py:205] memory_size
    2020-10-07 21:58:17 INFO [model_serialization.py:205] action_output_shape
    2020-10-07 21:58:17 INFO [model_serialization.py:205] action
    2020-10-07 21:58:17 INFO [model_serialization.py:205] action_probs
    Converting results/393_Patient_Pose_Train/PatientPose/frozen_graph_def.pb to results/393_Patient_Pose_Train/PatientPose.nn
    IGNORED: Cast unknown layer
    IGNORED: Shape unknown layer
    IGNORED: StopGradient unknown layer
    GLOBALS: 'is_continuous_control', 'trainer_major_version', 'trainer_minor_version', 'trainer_patch_version', 'version_number', 'memory_size', 'action_output_shape'
    IN: 'vector_observation': [-1, 1, 1, 153] => 'sub_2'
    OUT: 'action', 'action_probs'
    DONE: wrote results/393_Patient_Pose_Train/PatientPose.nn file.
     
    Last edited: Oct 14, 2020