Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Question Setting tensorboard to frequently update while training

Discussion in 'ML-Agents' started by GeorgGrech, Feb 15, 2023.

  1. GeorgGrech

    GeorgGrech

    Joined:
    Nov 26, 2021
    Posts:
    6
    Hello there,

    I am working on a project with Unity ML as part of a dissertation, I've never used it before. The project consists of an agent that goes around gathering resource and filling its inventory, then goes back to its base and deposits the resources for points, rewards with being given based on points and penalties on distances, along with some other minor rules.
    upload_2023-2-15_15-13-10.png

    Episodes last quite a long time, as I've set episodes to end after an amount of points have been gathered, after which the level resets. The game also automatically exits playmode after an amount of episodes so I can stop training and extract the .onnx file.

    The issue I've been having is with TensorBoard. I need it to update frequently so I can observe how well the agent is training so that if need be I can tweak some values for better training. However, TensorBoard seems to update very irregularly, especially the cumulative reward which is what I require.

    I tried adjusting the 'summary-freq' in the config to little avail. Is there any way to have TensorBoard update more frequently during training?

    Thanks a ton
     

    Attached Files:

  2. hughperkins

    hughperkins

    Joined:
    Dec 3, 2022
    Posts:
    195
    AFAIK summary-freq is indeed what determines how often results are written to the logs.

    Note that tensorboard itself sometimes needs reloading and stuff, to see the results changes. Did you try refreshing the tensorboard page?