Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Unite 2023 Registrations are now LIVE! We are thrilled to announce that Unite 2023 Registration is now live and open for all!
    Dismiss Notice
New Forum User Notice Unite 2023 Registrations are now LIVE!

Objects not showing up in camera observation

Discussion in 'ML-Agents' started by camigord_unity, Feb 12, 2021.

  1. camigord_unity

    camigord_unity

    Joined:
    Feb 12, 2021
    Posts:
    5
    Hi,

    I am experiencing some weird problem when using visual observations and instantiating prefabs when launching the game. This is going to be hard to explain, but please, bear with me...

    I know you can hide objects from your camera by configuring layers and so on, but that is not the problem right here. In fact, everything works fine when running on the editor, the problem shows up after building the game.

    I do have a "game area" in my editor containing one agent and a given map. I have made a prefab out of that object and I now dynamically instantiate many of such areas when launching the game. As as I said, this works fine when executed from the editor.

    When I build the game and try to train some agents, however, the images coming from the "instantiated" agents are not complete. By that I mean that some textures do not show up in the observation (similar to what would happen if the camera was ignoring some layer, which again, is not the case).

    My work around this issue was to stop the dynamic instantiation and just place as many game areas as I want before building the game. This then works fine when training.

    Even though this works out, I am quite curious about the reason why this issue was happening. What I presented above is rather a border case where the whole game area is instantiated at runtime, but what if I was instantiating other objects as training goes on?

    I hope I managed to properly explain the issue. By the way, I am actually using some assets/prefabs from Unity's 3D Starter Kit and some of those are the ones not showing up in the image (shader: MossMask).

    Thanks,
     
    camigord_unity, Feb 12, 2021
    #1

  2. ruoping_unity

    ruoping_unity

    Unity Technologies

    Joined:
    Jul 10, 2020
    Posts:
    134
    Hi,

    If I'm understanding correctly, some textures were missing in your prefab area if you dynamically instantiate the areas and it only happens in a build not in editor.

    If that's the case I have a few things to clarify:
    1. Can you verify if those missing textures actually being instantiated correctly during your dynamic instantiation? If they were not perceived by the camera/observations they were probably not created properly.
    2. How did you tell that the texture was missing? By the images received in trainer?
     
    ruoping_unity, Feb 12, 2021
    #2

  3. camigord_unity

    camigord_unity

    Joined:
    Feb 12, 2021
    Posts:
    5
    Hi, thanks for taking the time to answer.
    1. I do not understand what you mean there. I have verified that everything looks fine when I run the Editor. Everything is properly instantiated and the camera observations from all the agents look fine.
    2. Exactly. I plotted the images received from the environment and all but one are always wrong.

    I took the time to build a very simple project showcasing the issue here. I also wrote a simple jupyter notebook launching the environment and plotting the generated images. As you can see, only the figure to the right shows the platform in front of the agent, the textures do not show up for the other agents (which I assume are the ones been instantiated in code).

    Hope that helps clarify the issue. Any suggestions would be highly appreciated.
     
    camigord_unity, Feb 13, 2021
    #3

  4. ruoping_unity

    ruoping_unity

    Unity Technologies

    Joined:
    Jul 10, 2020
    Posts:
    134
    Thanks for making the sample project, it's very helpful for me to investigate this.

    While I'm looking into it, can you provide the Unity version and ML-Agents version you're using?
     
    ruoping_unity, Feb 14, 2021
    #4

  5. camigord_unity

    camigord_unity

    Joined:
    Feb 12, 2021
    Posts:
    5
    Sorry, forgot to mention:
    Unity version 2019.4.11f1, and ML-Agents package 1.0.5. I tried using the most recent version of ML-Agents but there was no difference.
     
    camigord_unity, Feb 14, 2021
    #5

  6. ruoping_unity

    ruoping_unity

    Unity Technologies

    Joined:
    Jul 10, 2020
    Posts:
    134
    I checked out your project and I noticed that the glitches actually changes with frames. I think the cause is the frame rate for graphics rendering and time scale you're using.
    The texture was not missing and was perceived normally in the fifth image in your notebook
    My guess is the texture you used were rendered slower so you're seeing those glitches.
    I would recommend you to try modify your EngineConfig parameters and see if that helps.
     
    ruoping_unity, Feb 15, 2021
    #6

  7. camigord_unity

    camigord_unity

    Joined:
    Feb 12, 2021
    Posts:
    5
    Hi,
    keep in mind that the images in the notebook are not different frames. Those are the first observations from the 5 agents in the game. One of those agents, as described above, is created and defined in the editor while the remaining 4 are instantiated the moment the game launches.

    Moreover, I am not playing with the timescale right now, so I do not expect that to be a problem.

    Finally, if I were to create all the 5 game areas (and agents) directly on the editor (instead of instantiating them in code) this problem would not occur. That's why I do not think it is because the textures are rendered slower.

    ** If I step the environment (even though nothing happens in this particular sample project) the images I receive from the agents do not change. New frames are still not rendering those textures.
     
    Last edited: Feb 15, 2021
    camigord_unity, Feb 15, 2021
    #7

  8. ruoping_unity

    ruoping_unity

    Unity Technologies

    Joined:
    Jul 10, 2020
    Posts:
    134
    If you try stepping the environment multiple times you can see the glitches actually changes with frames (I found it most obvious by looking at the observation of the fifth agent). Also if you call set_configuration_parameters() with different parameters, the pattern of the glitches is slightly different. The areas be instantiated one by one might also contribute to why you're seeing different output of 5 agents, versus instantiating them all at once seems fine.

    Also instantiating them all at once before hand doesn't completely solve the issue. I tried that and plotted the output with the same python notebook. The first step looks perfect, but again, if you try stepping the environment multiple times you can see the glitches appearing and disappearing some times. That's why I'd suggest it has something to do with frame rate and you might want to do something with relevant framerate settings in set_configuration_parameters().

    The images of observations plotted out on your notebook matches what I see in the game build display (glitches are in the game, not introduced by ML-Agent collecting the observations), so I'd say this is not an ML-Agent bug. If you still have issues around getting the texture render properly for each frame, I'd recommend you to ask for help from unity communities of that package or texture rendering.
     
    ruoping_unity, Feb 15, 2021
    #8

  9. camigord_unity

    camigord_unity

    Joined:
    Feb 12, 2021
    Posts:
    5
    You are right, I do notice that now.
    I played around with all the parameters provided by the EngineConfig but nothings seems to make it work. The only difference with that asset is the material and the associated shader (things I do not know much about). I will try to ask around and check if I can understand what's going on.

    Thanks for the help,
     
    camigord_unity, Feb 16, 2021
    #9