Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Bug Agents stucks after some policie updates

Discussion in 'ML-Agents' started by alvaro_unity903, Nov 10, 2020.

  1. alvaro_unity903

    alvaro_unity903

    Joined:
    Jan 20, 2020
    Posts:
    21
    Environment:
    • Custom, with 1 to many agents, depending on the config, in the same scenario. It breaks in all of them.

    Version information:
    • Release 4
    • ml-agents: 0.18.0,
    • ml-agents-envs: 0.18.0,
    • Communicator API: 1.0.0,
    • TensorFlow: 2.2.0

    Training configuration:
    • Default and custom (also in our custom algorithms)

    Observation and action spaces:
    • Observations: continuous (from 11 to many, depend if we use custom algorithms or default)
    • Action vector: discrete (17)

    Rewards:
    • The same that worked before, in previous versions

    Bug:
    It works as used to, with some policy updates, and then the agent keep like in a loop of an animation, moving but with no changes in Transform. This bug use to start with one of the agents stucked and then the rest of them get infected.

    Checks:
    • The action that seems to be the last before the clog has been worked earlier. It is not the same action always.
    • The reset seems to work also, because it is restarted hundreds of times before de bug. The stagnation doesn't look like it is related with reset because is stuck when there is no action to restart.
    • Max steps are so big to be the cause.
    • The brain is in default mode. I also tested with heuristic only.
    • With --no-graphics it doesn't work either.
    • With one scenario it doesn't work either.
    • With one agent it doesn't work either.
    • The action vector it still very random because the little updates.
    • I checked if it happened with collisions, but in a empty space with agent running it happened also
    • I ran the training in another computers and it occur again, and the environment is very cheap in requierements.
    • I can't identify any related cause, it doesn't happen when update, nor reset, nor collisions...

    Clues:
    • When it is stucked I close the .exe and then it return this (I edited env and brain names):
    Code (Boo):
    1. (mla) C:\Users\alvar\Downloads\my behavior log idjubot>mlagents-learn "C:\Users\alvar\RiderProjects\Representador-Dev\Assets\ML-Agents\config\ppo\my behavior.yaml" --env=my behavior --force
    2. 2020-11-10 15:16:45.239332: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
    3. 2020-11-10 15:16:45.239507: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    4. WARNING:tensorflow:From C:\Users\alvar\miniconda3\lib\site-packages\tensorflow\python\compat\v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
    5. Instructions for updating:
    6. non-resource variables are not supported in the long term
    7.  
    8.  
    9.                         ▄▄▄▓▓▓▓
    10.                    ╓▓▓▓▓▓▓█▓▓▓▓▓
    11.               ,▄▄▄m▀▀▀'  ,▓▓▓▀▓▓▄                           ▓▓▓  ▓▓▌
    12.            ▄▓▓▓▀'      ▄▓▓▀  ▓▓▓      ▄▄     ▄▄ ,▄▄ ▄▄▄▄   ,▄▄ ▄▓▓▌▄ ▄▄▄    ,▄▄
    13.           ▄▓▓▓▀        ▄▓▓▀   ▐▓▓▌     ▓▓▌   ▐▓▓ ▐▓▓▓▀▀▀▓▓▌ ▓▓▓ ▀▓▓▌▀ ^▓▓▌  ╒▓▓▌
    14.         ▄▓▓▓▓▓▄▄▄▄▄▄▄▄▓▓▓      ▓▀      ▓▓▌   ▐▓▓ ▐▓▓    ▓▓▓ ▓▓▓  ▓▓▌   ▐▓▓▄ ▓▓▌
    15.         ▀▓▓▓▓▀▀▀▀▀▀▀▀▀▀▓▓▄     ▓▓      ▓▓▌   ▐▓▓ ▐▓▓    ▓▓▓ ▓▓▓  ▓▓▌    ▐▓▓▐▓▓
    16.           ^█▓▓▓        ▀▓▓▄   ▐▓▓▌     ▓▓▓▓▄▓▓▓▓ ▐▓▓    ▓▓▓ ▓▓▓  ▓▓▓▄    ▓▓▓▓`
    17.             '▀▓▓▓▄      ^▓▓▓  ▓▓▓       └▀▀▀▀ ▀▀ ^▀▀    `▀▀ `▀▀   '▀▀    ▐▓▓▌
    18.                ▀▀▀▀▓▄▄▄   ▓▓▓▓▓▓,                                      ▓▓▓▓▀
    19.                    `▀█▓▓▓▓▓▓▓▓▓▌
    20.                         ¬`▀▀▀█▓
    21.  
    22.  
    23. Version information:
    24.   ml-agents: 0.18.0,
    25.   ml-agents-envs: 0.18.0,
    26.   Communicator API: 1.0.0,
    27.   TensorFlow: 2.2.0
    28. 2020-11-10 15:16:50.911286: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
    29. 2020-11-10 15:16:50.911548: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    30. WARNING:tensorflow:From C:\Users\alvar\miniconda3\lib\site-packages\tensorflow\python\compat\v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
    31. Instructions for updating:
    32. non-resource variables are not supported in the long term
    33. 2020-11-10 15:17:08 INFO [environment.py:108] Connected to Unity environment with package version 1.2.0-preview and communication version 1.0.0
    34. 2020-11-10 15:17:09 INFO [environment.py:265] Connected new brain:
    35. my behavior?team=0
    36. 2020-11-10 15:17:09.330051: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
    37. 2020-11-10 15:17:09.347787: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x220fb868240 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
    38. 2020-11-10 15:17:09.347996: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
    39. 2020-11-10 15:17:09.354594: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
    40. 2020-11-10 15:17:09.355032: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
    41. 2020-11-10 15:17:09.368375: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-VPPQ3NN
    42. 2020-11-10 15:17:09.370196: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-VPPQ3NN
    43. 2020-11-10 15:17:09 WARNING [stats.py:198] events.out.tfevents.1605017682.DESKTOP-VPPQ3NN was left over from a previous run. Deleting.
    44. 2020-11-10 15:17:09 INFO [stats.py:131] Hyperparameters for behavior name my behavior:
    45.         trainer_type:   ppo
    46.         hyperparameters:
    47.           batch_size:   128
    48.           buffer_size:  2048
    49.           learning_rate:        0.0003
    50.           beta: 0.005
    51.           epsilon:      0.2
    52.           lambd:        0.95
    53.           num_epoch:    3
    54.           learning_rate_schedule:       linear
    55.         network_settings:
    56.           normalize:    False
    57.           hidden_units: 256
    58.           num_layers:   2
    59.           vis_encode_type:      simple
    60.           memory:       None
    61.         reward_signals:
    62.           extrinsic:
    63.             gamma:      0.99
    64.             strength:   1.0
    65.         init_path:      None
    66.         keep_checkpoints:       5
    67.         checkpoint_interval:    500000
    68.         max_steps:      20000000
    69.         time_horizon:   128
    70.         summary_freq:   20000
    71.         threaded:       True
    72.         self_play:      None
    73.         behavioral_cloning:     None
    74. 2020-11-10 15:15:11 INFO [stats.py:112] my behavior: Step: 20000. Time Elapsed: 223.148 s Mean Reward: -0.012. Std of Reward: 0.000. Training.
    75. 2020-11-10 15:17:12 INFO [subprocess_env_manager.py:184] UnityEnvironment worker 0: environment stopping.
    76. 2020-11-10 15:17:12 INFO [trainer_controller.py:196] Learning was interrupted. Please wait while the graph is generated.
    77. 2020-11-10 15:17:12 INFO [trainer_controller.py:76] Saved Model
    78. 2020-11-10 15:17:13 INFO [model_serialization.py:203] List of nodes to export for brain :my behavior
    79. 2020-11-10 15:17:13 INFO [model_serialization.py:205]   is_continuous_control
    80. 2020-11-10 15:17:13 INFO [model_serialization.py:205]   trainer_major_version
    81. 2020-11-10 15:17:13 INFO [model_serialization.py:205]   trainer_minor_version
    82. 2020-11-10 15:17:13 INFO [model_serialization.py:205]   trainer_patch_version
    83. 2020-11-10 15:17:13 INFO [model_serialization.py:205]   version_number
    84. 2020-11-10 15:17:13 INFO [model_serialization.py:205]   memory_size
    85. 2020-11-10 15:17:13 INFO [model_serialization.py:205]   action_output_shape
    86. 2020-11-10 15:17:13 INFO [model_serialization.py:205]   action
    87. Converting results\ppo\my behavior/frozen_graph_def.pb to results\ppo\my behavior.nn
    88. 2020-11-10 15:17:13 INFO [environment.py:418] Environment shut down with return code 0 (CTRL_C_EVENT).
    89. GLOBALS: 'is_continuous_control', 'trainer_major_version', 'trainer_minor_version', 'trainer_patch_version', 'version_number', 'memory_size', 'action_output_shape'
    90. IN: 'vector_observation': [-1, 1, 1, 11] => 'policy/main_graph_0/hidden_0/BiasAdd'
    91. IN: 'action_masks': [-1, 1, 1, 17] => 'policy_1/strided_slice'
    92. OUT: 'policy_1/concat_2/concat', 'action'
    93. DONE: wrote results\ppo\my behavior.nn file.
    94. 2020-11-10 15:17:13 INFO [model_serialization.py:83] Exported results\ppo\my behavior.nn file
    95.  
    96.  
     
    Last edited: Nov 10, 2020
    betike likes this.
  2. ruoping_unity

    ruoping_unity

    Unity Technologies

    Joined:
    Jul 10, 2020
    Posts:
    134
    Hi,

    Just to clarify:
    1. I'm not quite sure what you mean by "stuck" here. Can you provide more details on what you observed when agent is stuck (agent not moving/the training crashed/the editor not responding/etc.)? What's the agent behavior? Is your python thread/Unity editor still keep running? How long does this stuck persist or it just seems to stuck at some point forever?
    2. You said that it worked before, do you mean that the exact same code works with an older version but breaks with the version you listed above?
    3. It's hard for us to debug on custom environment which we have no access to. Can you reproduce the issue with any of the ML-Agents example environment?

    From the log provided above I can't really tell what went wrong since it's not showing any error and it terminated normally when you closed the execution file. Also the times in your log looks a bit weird - did you modified anything else other than env and brain names?
     
  3. alvaro_unity903

    alvaro_unity903

    Joined:
    Jan 20, 2020
    Posts:
    21
    1. Agent doesn't move (no translation or rotation) with a failed animation: The agent runs the running animation but this is not fluid, we are not sure if this is an animation in a loop or in forward-backwards frames... and the error also produce that the agent doesn't translate or rotate.

    When this happens, the Academy doesn't reset the episode again. The python thread still running but the suck persist forever in the environment. We don't run this with the editor, we don't know why but it doesn't work, but this is a previous and independent problem, we have been training with the standalone.

    2. Yes, but we have migrated from 0.4 to Release 4 and from Unity 2018 to 2019.

    3. I've just done some trials with the next results:

    I played the 3DBall and these message and error happened:

    Message:
    Code (Boo):
    1. Couldn't connect to trainer on port 5004 using API version 1.0.0. Will perform inference instead.
    2. 0x00007FF7406A723C (Unity) StackWalker::GetCurrentCallstack
    3. 0x00007FF7406AB1F1 (Unity) StackWalker::ShowCallstack
    4. 0x00007FF73ED7AB05 (Unity) GetStacktrace
    5. 0x00007FF74132A20E (Unity) DebugStringToFile
    6. 0x00007FF7407065F5 (Unity) DebugLogHandler_CUSTOM_Internal_Log
    7. 0x00000199948ECA6B (Mono JIT Code) (wrapper managed-to-native) UnityEngine.DebugLogHandler:Internal_Log (UnityEngine.LogType,UnityEngine.LogOption,string,UnityEngine.Object)
    8. 0x00000199948EC8EB (Mono JIT Code) UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
    9. 0x00000199948EBD6E (Mono JIT Code) UnityEngine.Logger:Log (UnityEngine.LogType,object)
    10. 0x00000199948EB9BD (Mono JIT Code) UnityEngine.Debug:Log (object)
    11. 0x000001999484BC93 (Mono JIT Code) [Academy.cs:394] Unity.MLAgents.Academy:InitializeEnvironment ()
    12. 0x000001999484B02B (Mono JIT Code) [Academy.cs:218] Unity.MLAgents.Academy:LazyInitialize ()
    13. 0x000001999484AC2B (Mono JIT Code) [Academy.cs:206] Unity.MLAgents.Academy:.ctor ()
    14. 0x000001999484AA03 (Mono JIT Code) [Academy.cs:78] Unity.MLAgents.Academy/<>c:<.cctor>b__80_0 ()
    15. 0x000001999484A35B (Mono JIT Code) System.Lazy`1<T_REF>:CreateValue ()
    16. 0x0000019994849863 (Mono JIT Code) System.Lazy`1<T_REF>:LazyInitValue ()
    17. 0x0000019994848FB3 (Mono JIT Code) System.Lazy`1<T_REF>:get_Value ()
    18. 0x0000019994848CD3 (Mono JIT Code) [Academy.cs:93] Unity.MLAgents.Academy:get_Instance ()
    19. 0x0000019994847443 (Mono JIT Code) [DecisionRequester.cs:49] Unity.MLAgents.DecisionRequester:Awake ()
    20. 0x000001999424CC28 (Mono JIT Code) (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr)
    21. 0x00007FFDCEF9D6B0 (mono-2.0-bdwgc) [mini-runtime.c:2809] mono_jit_runtime_invoke
    22. 0x00007FFDCEF22912 (mono-2.0-bdwgc) [object.c:2921] do_runtime_invoke
    23. 0x00007FFDCEF2B96F (mono-2.0-bdwgc) [object.c:2968] mono_runtime_invoke
    24. 0x00007FF7406198EE (Unity) scripting_method_invoke
    25. 0x00007FF74061361D (Unity) ScriptingInvocation::Invoke
    26. 0x00007FF7406138EE (Unity) ScriptingInvocation::InvokeChecked
    27. 0x00007FF74067B873 (Unity) SerializableManagedRef::CallMethod
    28. 0x00007FF7405DCEFD (Unity) MonoBehaviour::CallAwake
    29. 0x00007FF7405DB884 (Unity) MonoBehaviour::AddToManager
    30. 0x00007FF7405DC28C (Unity) MonoBehaviour::AwakeFromLoad
    31. 0x00007FF73ED03F07 (Unity) AwakeFromLoadQueue::InvokePersistentManagerAwake
    32. 0x00007FF73ED04A20 (Unity) AwakeFromLoadQueue::PersistentManagerAwakeFromLoad
    33. 0x00007FF7400D7933 (Unity) LoadSceneOperation::CompleteAwakeSequence
    34. 0x00007FF7400D7C03 (Unity) LoadSceneOperation::CompletePreloadManagerLoadSceneEditor
    35. 0x00007FF7400D88B9 (Unity) LoadSceneOperation::IntegrateMainThread
    36. 0x00007FF7400DC209 (Unity) PreloadManager::UpdatePreloadingSingleStep
    37. 0x00007FF7400DCA5F (Unity) PreloadManager::WaitForAllAsyncOperationsToComplete
    38. 0x00007FF73E9B67E8 (Unity) EditorSceneManager::RestoreSceneBackups
    39. 0x00007FF73E417CA4 (Unity) PlayerLoopController::EnterPlayMode
    40. 0x00007FF73E42A415 (Unity) PlayerLoopController::SetIsPlaying
    41. 0x00007FF73E42D0F2 (Unity) Application::TickTimer
    42. 0x00007FF73EDA0560 (Unity) MainMessageLoop
    43. 0x00007FF73EDAA738 (Unity) WinMain
    44. 0x00007FF741DC7DC2 (Unity) __scrt_common_main_seh
    45. 0x00007FFED6F47034 (KERNEL32) BaseThreadInitThunk
    46. 0x00007FFED8BBCEC1 (ntdll) RtlUserThreadStart
    and this:

    Code (Boo):
    1. Request error (error):
    2. 0x00007FF7406A723C (Unity) StackWalker::GetCurrentCallstack
    3. 0x00007FF7406AB1F1 (Unity) StackWalker::ShowCallstack
    4. 0x00007FF73ED7AB05 (Unity) GetStacktrace
    5. 0x00007FF74132A20E (Unity) DebugStringToFile
    6. 0x00007FF7407065F5 (Unity) DebugLogHandler_CUSTOM_Internal_Log
    7. 0x00000199948ECA6B (Mono JIT Code) (wrapper managed-to-native) UnityEngine.DebugLogHandler:Internal_Log (UnityEngine.LogType,UnityEngine.LogOption,string,UnityEngine.Object)
    8. 0x00000199948EC8EB (Mono JIT Code) UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
    9. 0x00000199948EBD6E (Mono JIT Code) UnityEngine.Logger:Log (UnityEngine.LogType,object)
    10. 0x0000019948F46EFA (Mono JIT Code) UnityEngine.Debug:LogError (object)
    11. 0x0000019948F3E20B (Mono JIT Code) UnityEditor.AssetStoreClient:ParseContent (UnityEditor.IAsyncHTTPClient)
    12. 0x0000019948F3D753 (Mono JIT Code) UnityEditor.AssetStoreClient/<>c__DisplayClass45_0:<WrapJsonCallback>b__0 (UnityEditor.IAsyncHTTPClient)
    13. 0x0000019948F3D3A3 (Mono JIT Code) UnityEditor.AsyncHTTPClient:Done (UnityEditor.AsyncHTTPClient/State,int)
    14. 0x000001999474DEDF (Mono JIT Code) (wrapper runtime-invoke) <Module>:runtime_invoke_void__this___int_int (object,intptr,intptr,intptr)
    15. 0x00007FFDCEF9D6B0 (mono-2.0-bdwgc) [mini-runtime.c:2809] mono_jit_runtime_invoke
    16. 0x00007FFDCEF22912 (mono-2.0-bdwgc) [object.c:2921] do_runtime_invoke
    17. 0x00007FFDCEF2B96F (mono-2.0-bdwgc) [object.c:2968] mono_runtime_invoke
    18. 0x00007FF7406198EE (Unity) scripting_method_invoke
    19. 0x00007FF74061361D (Unity) ScriptingInvocation::Invoke
    20. 0x00007FF73F3D0DD8 (Unity) AsyncHTTPClientRequest::Done
    21. 0x00007FF7412CF1A6 (Unity) CurlRequest::Check
    22. 0x00007FF7412CF769 (Unity) CurlRequestCheck
    23. 0x00007FF73E42C40B (Unity) Application::TickTimer
    24. 0x00007FF73EDA0560 (Unity) MainMessageLoop
    25. 0x00007FF73EDAA738 (Unity) WinMain
    26. 0x00007FF741DC7DC2 (Unity) __scrt_common_main_seh
    27. 0x00007FFED6F47034 (KERNEL32) BaseThreadInitThunk
    28. 0x00007FFED8BBCEC1 (ntdll) RtlUserThreadStart

    With VisualPyramids I got this one message:

    Code (Boo):
    1. Couldn't connect to trainer on port 5004 using API version 1.0.0. Will perform inference instead.
    2. 0x00007FF7406A723C (Unity) StackWalker::GetCurrentCallstack
    3. 0x00007FF7406AB1F1 (Unity) StackWalker::ShowCallstack
    4. 0x00007FF73ED7AB05 (Unity) GetStacktrace
    5. 0x00007FF74132A20E (Unity) DebugStringToFile
    6. 0x00007FF7407065F5 (Unity) DebugLogHandler_CUSTOM_Internal_Log
    7. 0x000001998C8CE02B (Mono JIT Code) (wrapper managed-to-native) UnityEngine.DebugLogHandler:Internal_Log (UnityEngine.LogType,UnityEngine.LogOption,string,UnityEngine.Object)
    8. 0x000001998C8CDEAB (Mono JIT Code) UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
    9. 0x000001998C8CD32E (Mono JIT Code) UnityEngine.Logger:Log (UnityEngine.LogType,object)
    10. 0x000001998C8CCF7D (Mono JIT Code) UnityEngine.Debug:Log (object)
    11. 0x000001998C81D1A3 (Mono JIT Code) [Academy.cs:394] Unity.MLAgents.Academy:InitializeEnvironment ()
    12. 0x000001998C81C53B (Mono JIT Code) [Academy.cs:218] Unity.MLAgents.Academy:LazyInitialize ()
    13. 0x000001998C81C13B (Mono JIT Code) [Academy.cs:206] Unity.MLAgents.Academy:.ctor ()
    14. 0x000001998C81BF13 (Mono JIT Code) [Academy.cs:78] Unity.MLAgents.Academy/<>c:<.cctor>b__80_0 ()
    15. 0x000001998C81B86B (Mono JIT Code) System.Lazy`1<T_REF>:CreateValue ()
    16. 0x000001998C81AD73 (Mono JIT Code) System.Lazy`1<T_REF>:LazyInitValue ()
    17. 0x000001998C81A4C3 (Mono JIT Code) System.Lazy`1<T_REF>:get_Value ()
    18. 0x000001998C81A1E3 (Mono JIT Code) [Academy.cs:93] Unity.MLAgents.Academy:get_Instance ()
    19. 0x000001998C818953 (Mono JIT Code) [DecisionRequester.cs:49] Unity.MLAgents.DecisionRequester:Awake ()
    20. 0x00000199903AD638 (Mono JIT Code) (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr)
    21. 0x00007FFDCEF9D6B0 (mono-2.0-bdwgc) [mini-runtime.c:2809] mono_jit_runtime_invoke
    22. 0x00007FFDCEF22912 (mono-2.0-bdwgc) [object.c:2921] do_runtime_invoke
    23. 0x00007FFDCEF2B96F (mono-2.0-bdwgc) [object.c:2968] mono_runtime_invoke
    24. 0x00007FF7406198EE (Unity) scripting_method_invoke
    25. 0x00007FF74061361D (Unity) ScriptingInvocation::Invoke
    26. 0x00007FF7406138EE (Unity) ScriptingInvocation::InvokeChecked
    27. 0x00007FF74067B873 (Unity) SerializableManagedRef::CallMethod
    28. 0x00007FF7405DCEFD (Unity) MonoBehaviour::CallAwake
    29. 0x00007FF7405DB884 (Unity) MonoBehaviour::AddToManager
    30. 0x00007FF7405DC28C (Unity) MonoBehaviour::AwakeFromLoad
    31. 0x00007FF73ED03F07 (Unity) AwakeFromLoadQueue::InvokePersistentManagerAwake
    32. 0x00007FF73ED04A20 (Unity) AwakeFromLoadQueue::PersistentManagerAwakeFromLoad
    33. 0x00007FF7400D7933 (Unity) LoadSceneOperation::CompleteAwakeSequence
    34. 0x00007FF7400D7C03 (Unity) LoadSceneOperation::CompletePreloadManagerLoadSceneEditor
    35. 0x00007FF7400D88B9 (Unity) LoadSceneOperation::IntegrateMainThread
    36. 0x00007FF7400DC209 (Unity) PreloadManager::UpdatePreloadingSingleStep
    37. 0x00007FF7400DCA5F (Unity) PreloadManager::WaitForAllAsyncOperationsToComplete
    38. 0x00007FF73E9B67E8 (Unity) EditorSceneManager::RestoreSceneBackups
    39. 0x00007FF73E417CA4 (Unity) PlayerLoopController::EnterPlayMode
    40. 0x00007FF73E42A415 (Unity) PlayerLoopController::SetIsPlaying
    41. 0x00007FF73E42D0F2 (Unity) Application::TickTimer
    42. 0x00007FF73EDA0560 (Unity) MainMessageLoop
    43. 0x00007FF73EDAA738 (Unity) WinMain
    44. 0x00007FF741DC7DC2 (Unity) __scrt_common_main_seh
    45. 0x00007FFED6F47034 (KERNEL32) BaseThreadInitThunk
    46. 0x00007FFED8BBCEC1 (ntdll) RtlUserThreadStart
    With Pyramids I got this message and error:

    Message:

    Code (Boo):
    1. Couldn't connect to trainer on port 5004 using API version 1.0.0. Will perform inference instead.
    2. 0x00007FF7406A723C (Unity) StackWalker::GetCurrentCallstack
    3. 0x00007FF7406AB1F1 (Unity) StackWalker::ShowCallstack
    4. 0x00007FF73ED7AB05 (Unity) GetStacktrace
    5. 0x00007FF74132A20E (Unity) DebugStringToFile
    6. 0x00007FF7407065F5 (Unity) DebugLogHandler_CUSTOM_Internal_Log
    7. 0x000001998C88E02B (Mono JIT Code) (wrapper managed-to-native) UnityEngine.DebugLogHandler:Internal_Log (UnityEngine.LogType,UnityEngine.LogOption,string,UnityEngine.Object)
    8. 0x000001998C88DEAB (Mono JIT Code) UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
    9. 0x000001998C88D32E (Mono JIT Code) UnityEngine.Logger:Log (UnityEngine.LogType,object)
    10. 0x000001998C88CF7D (Mono JIT Code) UnityEngine.Debug:Log (object)
    11. 0x000001998C7ED933 (Mono JIT Code) [Academy.cs:394] Unity.MLAgents.Academy:InitializeEnvironment ()
    12. 0x000001998C7ECCCB (Mono JIT Code) [Academy.cs:218] Unity.MLAgents.Academy:LazyInitialize ()
    13. 0x000001998C7EC8CB (Mono JIT Code) [Academy.cs:206] Unity.MLAgents.Academy:.ctor ()
    14. 0x000001998C7EC6A3 (Mono JIT Code) [Academy.cs:78] Unity.MLAgents.Academy/<>c:<.cctor>b__80_0 ()
    15. 0x000001998C7EBFFB (Mono JIT Code) System.Lazy`1<T_REF>:CreateValue ()
    16. 0x000001998C7EB503 (Mono JIT Code) System.Lazy`1<T_REF>:LazyInitValue ()
    17. 0x000001998C7EAC53 (Mono JIT Code) System.Lazy`1<T_REF>:get_Value ()
    18. 0x000001998C7EA973 (Mono JIT Code) [Academy.cs:93] Unity.MLAgents.Academy:get_Instance ()
    19. 0x000001998C7E90E3 (Mono JIT Code) [DecisionRequester.cs:49] Unity.MLAgents.DecisionRequester:Awake ()
    20. 0x000001998CB98458 (Mono JIT Code) (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr)
    21. 0x00007FFDCEF9D6B0 (mono-2.0-bdwgc) [mini-runtime.c:2809] mono_jit_runtime_invoke
    22. 0x00007FFDCEF22912 (mono-2.0-bdwgc) [object.c:2921] do_runtime_invoke
    23. 0x00007FFDCEF2B96F (mono-2.0-bdwgc) [object.c:2968] mono_runtime_invoke
    24. 0x00007FF7406198EE (Unity) scripting_method_invoke
    25. 0x00007FF74061361D (Unity) ScriptingInvocation::Invoke
    26. 0x00007FF7406138EE (Unity) ScriptingInvocation::InvokeChecked
    27. 0x00007FF74067B873 (Unity) SerializableManagedRef::CallMethod
    28. 0x00007FF7405DCEFD (Unity) MonoBehaviour::CallAwake
    29. 0x00007FF7405DB884 (Unity) MonoBehaviour::AddToManager
    30. 0x00007FF7405DC28C (Unity) MonoBehaviour::AwakeFromLoad
    31. 0x00007FF73ED03F07 (Unity) AwakeFromLoadQueue::InvokePersistentManagerAwake
    32. 0x00007FF73ED04A20 (Unity) AwakeFromLoadQueue::PersistentManagerAwakeFromLoad
    33. 0x00007FF7400D7933 (Unity) LoadSceneOperation::CompleteAwakeSequence
    34. 0x00007FF7400D7C03 (Unity) LoadSceneOperation::CompletePreloadManagerLoadSceneEditor
    35. 0x00007FF7400D88B9 (Unity) LoadSceneOperation::IntegrateMainThread
    36. 0x00007FF7400DC209 (Unity) PreloadManager::UpdatePreloadingSingleStep
    37. 0x00007FF7400DCA5F (Unity) PreloadManager::WaitForAllAsyncOperationsToComplete
    38. 0x00007FF73E9B67E8 (Unity) EditorSceneManager::RestoreSceneBackups
    39. 0x00007FF73E417CA4 (Unity) PlayerLoopController::EnterPlayMode
    40. 0x00007FF73E42A415 (Unity) PlayerLoopController::SetIsPlaying
    41. 0x00007FF73E42D0F2 (Unity) Application::TickTimer
    42. 0x00007FF73EDA0560 (Unity) MainMessageLoop
    43. 0x00007FF73EDAA738 (Unity) WinMain
    44. 0x00007FF741DC7DC2 (Unity) __scrt_common_main_seh
    45. 0x00007FFED6F47034 (KERNEL32) BaseThreadInitThunk
    46. 0x00007FFED8BBCEC1 (ntdll) RtlUserThreadStart
    47.  
    Error:

    Code (Boo):
    1. ag: switchOn is not defined.
    2. 0x00007FF7406A723C (Unity) StackWalker::GetCurrentCallstack
    3. 0x00007FF7406AB1F1 (Unity) StackWalker::ShowCallstack
    4. 0x00007FF73ED7AB05 (Unity) GetStacktrace
    5. 0x00007FF74132A20E (Unity) DebugStringToFile
    6. 0x00007FF74007DAC1 (Unity) ExtractTagErroringObject
    7. 0x00007FF740723890 (Unity) GameObject_CUSTOM_CompareTag
    8. 0x000001998C932A49 (Mono JIT Code) (wrapper managed-to-native) UnityEngine.GameObject:CompareTag (UnityEngine.GameObject,string)
    9. 0x000001998C9480B3 (Mono JIT Code) [RayPerceptionSensor.cs:481] Unity.MLAgents.Sensors.RayPerceptionSensor:PerceiveSingleRay (Unity.MLAgents.Sensors.RayPerceptionInput,int,Unity.MLAgents.Sensors.DebugDisplayInfo/RayInfo&)
    10. 0x000001998C945CB3 (Mono JIT Code) [RayPerceptionSensor.cs:322] Unity.MLAgents.Sensors.RayPerceptionSensor:Write (Unity.MLAgents.Sensors.ObservationWriter)
    11. 0x000001998C9449FA (Mono JIT Code) [GeneratorImpl.cs:119] Unity.MLAgents.Inference.VectorObservationGenerator:Generate (Unity.MLAgents.Inference.TensorProxy,int,System.Collections.Generic.IEnumerable`1<Unity.MLAgents.Inference.AgentInfoSensorsPair>)
    12. 0x000001998C940EB2 (Mono JIT Code) [TensorGenerator.cs:141] Unity.MLAgents.Inference.TensorGenerator:GenerateTensors (System.Collections.Generic.IEnumerable`1<Unity.MLAgents.Inference.TensorProxy>,int,System.Collections.Generic.IEnumerable`1<Unity.MLAgents.Inference.AgentInfoSensorsPair>)
    13. 0x000001998C93F773 (Mono JIT Code) [ModelRunner.cs:165] Unity.MLAgents.Inference.ModelRunner:DecideBatch ()
    14. 0x000001998C93F203 (Mono JIT Code) [BarracudaPolicy.cs:60] Unity.MLAgents.Policies.BarracudaPolicy:DecideAction ()
    15. 0x000001998C93EF5E (Mono JIT Code) [Agent.cs:1196] Unity.MLAgents.Agent:DecideAction ()
    16. 0x000001996906B357 (Mono JIT Code) (wrapper delegate-invoke) <Module>:invoke_void ()
    17. 0x000001998C915E3E (Mono JIT Code) [Academy.cs:507] Unity.MLAgents.Academy:EnvironmentStep ()
    18. 0x000001998C9158A3 (Mono JIT Code) [Academy.cs:34] Unity.MLAgents.AcademyFixedUpdateStepper:FixedUpdate ()
    19. 0x000001998CB98458 (Mono JIT Code) (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr)
    20. 0x00007FFDCEF9D6B0 (mono-2.0-bdwgc) [mini-runtime.c:2809] mono_jit_runtime_invoke
    21. 0x00007FFDCEF22912 (mono-2.0-bdwgc) [object.c:2921] do_runtime_invoke
    22. 0x00007FFDCEF2B96F (mono-2.0-bdwgc) [object.c:2968] mono_runtime_invoke
    23. 0x00007FF7406198EE (Unity) scripting_method_invoke
    24. 0x00007FF74061361D (Unity) ScriptingInvocation::Invoke
    25. 0x00007FF7405DD0D5 (Unity) MonoBehaviour::CallMethodIfAvailable
    26. 0x00007FF7405DD1E6 (Unity) MonoBehaviour::CallUpdateMethod
    27. 0x00007FF73FC73678 (Unity) BaseBehaviourManager::CommonUpdate<FixedBehaviourManager>
    28. 0x00007FF73FC7C844 (Unity) FixedBehaviourManager::Update
    29. 0x00007FF7400B6C6C (Unity) `InitPlayerLoopCallbacks'::`2'::FixedUpdateScriptRunBehaviourFixedUpdateRegistrator::Forward
    30. 0x00007FF7400A04F8 (Unity) ExecutePlayerLoop
    31. 0x00007FF7400A05CD (Unity) ExecutePlayerLoop
    32. 0x00007FF7400A5ABC (Unity) PlayerLoop
    33. 0x00007FF73E42EEBC (Unity) PlayerLoopController::UpdateScene
    34. 0x00007FF73E42CE58 (Unity) Application::TickTimer
    35. 0x00007FF73EDA0560 (Unity) MainMessageLoop
    36. 0x00007FF73EDAA738 (Unity) WinMain
    37. 0x00007FF741DC7DC2 (Unity) __scrt_common_main_seh
    38. 0x00007FFED6F47034 (KERNEL32) BaseThreadInitThunk
    39. 0x00007FFED8BBCEC1 (ntdll) RtlUserThreadStart

    And with soccer two I got the inference mode again:
    Code (Boo):
    1. Couldn't connect to trainer on port 5004 using API version 1.0.0. Will perform inference instead.
    2. 0x00007FF7406A723C (Unity) StackWalker::GetCurrentCallstack
    3. 0x00007FF7406AB1F1 (Unity) StackWalker::ShowCallstack
    4. 0x00007FF73ED7AB05 (Unity) GetStacktrace
    5. 0x00007FF74132A20E (Unity) DebugStringToFile
    6. 0x00007FF7407065F5 (Unity) DebugLogHandler_CUSTOM_Internal_Log
    7. 0x0000019990DDBD7B (Mono JIT Code) (wrapper managed-to-native) UnityEngine.DebugLogHandler:Internal_Log (UnityEngine.LogType,UnityEngine.LogOption,string,UnityEngine.Object)
    8. 0x0000019990DDBBFB (Mono JIT Code) UnityEngine.DebugLogHandler:LogFormat (UnityEngine.LogType,UnityEngine.Object,string,object[])
    9. 0x0000019990DDB07E (Mono JIT Code) UnityEngine.Logger:Log (UnityEngine.LogType,object)
    10. 0x0000019990DDACCD (Mono JIT Code) UnityEngine.Debug:Log (object)
    11. 0x0000019990D3B0E3 (Mono JIT Code) [Academy.cs:394] Unity.MLAgents.Academy:InitializeEnvironment ()
    12. 0x0000019990D3A47B (Mono JIT Code) [Academy.cs:218] Unity.MLAgents.Academy:LazyInitialize ()
    13. 0x0000019990D3A07B (Mono JIT Code) [Academy.cs:206] Unity.MLAgents.Academy:.ctor ()
    14. 0x0000019990D39E53 (Mono JIT Code) [Academy.cs:78] Unity.MLAgents.Academy/<>c:<.cctor>b__80_0 ()
    15. 0x0000019990D397AB (Mono JIT Code) System.Lazy`1<T_REF>:CreateValue ()
    16. 0x0000019990D38CB3 (Mono JIT Code) System.Lazy`1<T_REF>:LazyInitValue ()
    17. 0x0000019990D38403 (Mono JIT Code) System.Lazy`1<T_REF>:get_Value ()
    18. 0x0000019990D38123 (Mono JIT Code) [Academy.cs:93] Unity.MLAgents.Academy:get_Instance ()
    19. 0x0000019990D36893 (Mono JIT Code) [DecisionRequester.cs:49] Unity.MLAgents.DecisionRequester:Awake ()
    20. 0x00000199906684C8 (Mono JIT Code) (wrapper runtime-invoke) object:runtime_invoke_void__this__ (object,intptr,intptr,intptr)
    21. 0x00007FFDCEF9D6B0 (mono-2.0-bdwgc) [mini-runtime.c:2809] mono_jit_runtime_invoke
    22. 0x00007FFDCEF22912 (mono-2.0-bdwgc) [object.c:2921] do_runtime_invoke
    23. 0x00007FFDCEF2B96F (mono-2.0-bdwgc) [object.c:2968] mono_runtime_invoke
    24. 0x00007FF7406198EE (Unity) scripting_method_invoke
    25. 0x00007FF74061361D (Unity) ScriptingInvocation::Invoke
    26. 0x00007FF7406138EE (Unity) ScriptingInvocation::InvokeChecked
    27. 0x00007FF74067B873 (Unity) SerializableManagedRef::CallMethod
    28. 0x00007FF7405DCEFD (Unity) MonoBehaviour::CallAwake
    29. 0x00007FF7405DB884 (Unity) MonoBehaviour::AddToManager
    30. 0x00007FF7405DC28C (Unity) MonoBehaviour::AwakeFromLoad
    31. 0x00007FF73ED03F07 (Unity) AwakeFromLoadQueue::InvokePersistentManagerAwake
    32. 0x00007FF73ED04A20 (Unity) AwakeFromLoadQueue::PersistentManagerAwakeFromLoad
    33. 0x00007FF7400D7933 (Unity) LoadSceneOperation::CompleteAwakeSequence
    34. 0x00007FF7400D7C03 (Unity) LoadSceneOperation::CompletePreloadManagerLoadSceneEditor
    35. 0x00007FF7400D88B9 (Unity) LoadSceneOperation::IntegrateMainThread
    36. 0x00007FF7400DC209 (Unity) PreloadManager::UpdatePreloadingSingleStep
    37. 0x00007FF7400DCA5F (Unity) PreloadManager::WaitForAllAsyncOperationsToComplete
    38. 0x00007FF73E9B67E8 (Unity) EditorSceneManager::RestoreSceneBackups
    39. 0x00007FF73E417CA4 (Unity) PlayerLoopController::EnterPlayMode
    40. 0x00007FF73E42A415 (Unity) PlayerLoopController::SetIsPlaying
    41. 0x00007FF73E42D0F2 (Unity) Application::TickTimer
    42. 0x00007FF73EDA0560 (Unity) MainMessageLoop
    43. 0x00007FF73EDAA738 (Unity) WinMain
    44. 0x00007FF741DC7DC2 (Unity) __scrt_common_main_seh
    45. 0x00007FFED6F47034 (KERNEL32) BaseThreadInitThunk
    46. 0x00007FFED8BBCEC1 (ntdll) RtlUserThreadStart
    47.  


    4. About times, yes, I inserted the training line with a previous log and adjusted the difference with the next time to emulate and I've just found this is not compatible with the previous time. Sorry I paste here the log of a new trial:

    Code (Boo):
    1. (mla) C:\Users\alvar\Downloads\my behavior log idjubot>mlagents-learn "C:\Users\alvar\RiderProjects\Representador-Dev\Assets\ML-Agents\config\ppo\my behavior.yaml" --env=my behavior --force
    2. 2020-11-12 10:08:00.860955: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
    3. 2020-11-12 10:08:00.866241: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    4. WARNING:tensorflow:From C:\Users\alvar\miniconda3\lib\site-packages\tensorflow\python\compat\v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
    5. Instructions for updating:
    6. non-resource variables are not supported in the long term
    7.  
    8.  
    9.                         ▄▄▄▓▓▓▓
    10.                    ╓▓▓▓▓▓▓█▓▓▓▓▓
    11.               ,▄▄▄m▀▀▀'  ,▓▓▓▀▓▓▄                           ▓▓▓  ▓▓▌
    12.            ▄▓▓▓▀'      ▄▓▓▀  ▓▓▓      ▄▄     ▄▄ ,▄▄ ▄▄▄▄   ,▄▄ ▄▓▓▌▄ ▄▄▄    ,▄▄
    13.           ▄▓▓▓▀        ▄▓▓▀   ▐▓▓▌     ▓▓▌   ▐▓▓ ▐▓▓▓▀▀▀▓▓▌ ▓▓▓ ▀▓▓▌▀ ^▓▓▌  ╒▓▓▌
    14.         ▄▓▓▓▓▓▄▄▄▄▄▄▄▄▓▓▓      ▓▀      ▓▓▌   ▐▓▓ ▐▓▓    ▓▓▓ ▓▓▓  ▓▓▌   ▐▓▓▄ ▓▓▌
    15.         ▀▓▓▓▓▀▀▀▀▀▀▀▀▀▀▓▓▄     ▓▓      ▓▓▌   ▐▓▓ ▐▓▓    ▓▓▓ ▓▓▓  ▓▓▌    ▐▓▓▐▓▓
    16.           ^█▓▓▓        ▀▓▓▄   ▐▓▓▌     ▓▓▓▓▄▓▓▓▓ ▐▓▓    ▓▓▓ ▓▓▓  ▓▓▓▄    ▓▓▓▓`
    17.             '▀▓▓▓▄      ^▓▓▓  ▓▓▓       └▀▀▀▀ ▀▀ ^▀▀    `▀▀ `▀▀   '▀▀    ▐▓▓▌
    18.                ▀▀▀▀▓▄▄▄   ▓▓▓▓▓▓,                                      ▓▓▓▓▀
    19.                    `▀█▓▓▓▓▓▓▓▓▓▌
    20.                         ¬`▀▀▀█▓
    21.  
    22.  
    23. Version information:
    24.   ml-agents: 0.18.0,
    25.   ml-agents-envs: 0.18.0,
    26.   Communicator API: 1.0.0,
    27.   TensorFlow: 2.2.0
    28. 2020-11-12 10:08:13.875716: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
    29. 2020-11-12 10:08:13.875933: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
    30. WARNING:tensorflow:From C:\Users\alvar\miniconda3\lib\site-packages\tensorflow\python\compat\v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
    31. Instructions for updating:
    32. non-resource variables are not supported in the long term
    33. 2020-11-12 10:08:34 INFO [environment.py:108] Connected to Unity environment with package version 1.2.0-preview and communication version 1.0.0
    34. 2020-11-12 10:08:35 INFO [environment.py:265] Connected new brain:
    35. my behavior?team=0
    36. 2020-11-12 10:08:35.510556: I tensorflow/core/platform/cpu_feature_guard.cc:143] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
    37. 2020-11-12 10:08:35.546781: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2592da12c80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
    38. 2020-11-12 10:08:35.547167: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
    39. 2020-11-12 10:08:35.554268: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'nvcuda.dll'; dlerror: nvcuda.dll not found
    40. 2020-11-12 10:08:35.554514: E tensorflow/stream_executor/cuda/cuda_driver.cc:313] failed call to cuInit: UNKNOWN ERROR (303)
    41. 2020-11-12 10:08:35.570569: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information for host: DESKTOP-VPPQ3NN
    42. 2020-11-12 10:08:35.572785: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: DESKTOP-VPPQ3NN
    43. 2020-11-12 10:08:35 WARNING [stats.py:198] events.out.tfevents.1605171416.DESKTOP-VPPQ3NN was left over from a previous run. Deleting.
    44. 2020-11-12 10:08:35 INFO [stats.py:131] Hyperparameters for behavior name my behavior:
    45.         trainer_type:   ppo
    46.         hyperparameters:
    47.           batch_size:   128
    48.           buffer_size:  2048
    49.           learning_rate:        0.0003
    50.           beta: 0.005
    51.           epsilon:      0.2
    52.           lambd:        0.95
    53.           num_epoch:    3
    54.           learning_rate_schedule:       linear
    55.         network_settings:
    56.           normalize:    False
    57.           hidden_units: 256
    58.           num_layers:   2
    59.           vis_encode_type:      simple
    60.           memory:       None
    61.         reward_signals:
    62.           extrinsic:
    63.             gamma:      0.99
    64.             strength:   1.0
    65.         init_path:      None
    66.         keep_checkpoints:       5
    67.         checkpoint_interval:    500000
    68.         max_steps:      20000000
    69.         time_horizon:   128
    70.         summary_freq:   20000
    71.         threaded:       True
    72.         self_play:      None
    73.         behavioral_cloning:     None
    74. 2020-11-12 10:14:28 INFO [stats.py:112] my behavior: Step: 20000. Time Elapsed: 376.686 s Mean Reward: 0.046. Std of Reward: 0.253. Training.
    75. 2020-11-12 10:20:03 INFO [stats.py:112] my behavior: Step: 40000. Time Elapsed: 711.619 s Mean Reward: -0.009. Std of Reward: 0.001. Training.
    76. 2020-11-12 10:24:52 INFO [stats.py:112] my behavior: Step: 60000. Time Elapsed: 1000.646 s Mean Reward: -0.009. Std of Reward: 0.003. Training.
    77. 2020-11-12 10:27:50 INFO [stats.py:112] my behavior: Step: 80000. Time Elapsed: 1178.742 s Mean Reward: -0.006. Std of Reward: 0.001. Training.
    78. 2020-11-12 10:30:54 INFO [stats.py:112] my behavior: Step: 100000. Time Elapsed: 1362.565 s Mean Reward: -0.006. Std of Reward: 0.001. Training.
    79. 2020-11-12 10:31:47 INFO [subprocess_env_manager.py:184] UnityEnvironment worker 0: environment stopping.
    80. 2020-11-12 10:31:47 INFO [trainer_controller.py:196] Learning was interrupted. Please wait while the graph is generated.
    81. 2020-11-12 10:31:47 INFO [trainer_controller.py:76] Saved Model
    82. 2020-11-12 10:31:47 INFO [model_serialization.py:203] List of nodes to export for brain :my behavior
    83. 2020-11-12 10:31:47 INFO [model_serialization.py:205]   is_continuous_control
    84. 2020-11-12 10:31:47 INFO [model_serialization.py:205]   trainer_major_version
    85. 2020-11-12 10:31:47 INFO [model_serialization.py:205]   trainer_minor_version
    86. 2020-11-12 10:31:47 INFO [model_serialization.py:205]   trainer_patch_version
    87. 2020-11-12 10:31:47 INFO [model_serialization.py:205]   version_number
    88. 2020-11-12 10:31:47 INFO [model_serialization.py:205]   memory_size
    89. 2020-11-12 10:31:47 INFO [model_serialization.py:205]   action_output_shape
    90. 2020-11-12 10:31:47 INFO [model_serialization.py:205]   action
    91. Converting results\ppo\my behavior/frozen_graph_def.pb to results\ppo\my behavior.nn
    92. 2020-11-12 10:31:48 INFO [environment.py:418] Environment shut down with return code 0 (CTRL_C_EVENT).
    93. GLOBALS: 'is_continuous_control', 'trainer_major_version', 'trainer_minor_version', 'trainer_patch_version', 'version_number', 'memory_size', 'action_output_shape'
    94. IN: 'vector_observation': [-1, 1, 1, 11] => 'policy/main_graph_0/hidden_0/BiasAdd'
    95. IN: 'action_masks': [-1, 1, 1, 17] => 'policy_1/strided_slice'
    96. OUT: 'policy_1/concat_2/concat', 'action'
    97. DONE: wrote results\ppo\my behavior.nn file.
    98. 2020-11-12 10:31:48 INFO [model_serialization.py:83] Exported results\ppo\my behavior.nn file
     
  4. ruoping_unity

    ruoping_unity

    Unity Technologies

    Joined:
    Jul 10, 2020
    Posts:
    134
    If you were even not able to run the example environments, it seems to me that there's something wrong in your setup (installation, version, etc). You mentioned that you upgraded from 0.4 to release_4 so it might be the case that something breaks when you upgraded.
    I just tried the release 4 with Unity 2019 and I wasn't able to reproduce the errors. I would suggest double checking if your Unity and ML-Agents are both set up correctly first and make sure you're using a compatible version of python and C# of ML-Agent.
     
  5. alvaro_unity903

    alvaro_unity903

    Joined:
    Jan 20, 2020
    Posts:
    21
    Hi @unity_ruoping ,

    I have been researching, debugging a lot, and just now, reading some documentation, I realize that I could train in the editor running from Python with no env specified in the commands. Doing that it works as it should do with the example environments. If I run directly from the editor it runs in inference mode (the logs I pasted in my posts), but if I run it from python it trains, as expected.

    Conclusion, after the migration the setup seems to be ok, at least in that part.

    The rest of the versions (Python, C#, Unity and ML-Agents) should be compatible also.