Search Unity

Mipmap steaming and memory usage

Discussion in 'General Graphics' started by pokruchin, Mar 24, 2021.

  1. pokruchin

    pokruchin

    Joined:
    Apr 24, 2019
    Posts:
    36
    Hello guys! Since there is almost no information about streaming mipmaps and the documenation is poor about this topic I would like to ask: can I use the mipmap streaming for reducing ram usage on mobile? Because textures eat a lot of ram and it would be great to reduce this amount.
     
  2. lyndon_unity

    lyndon_unity

    Unity Technologies

    Joined:
    Nov 2, 2017
    Posts:
    43
    In summary - yes you can use texture mip streaming to reduce GPU memory usage.

    The following page attempts to introduce the texture mip streaming
    https://docs.unity3d.com/Manual/TextureStreaming.html

    "The Mip Map Streaming system gives you control over which mipmap levels Unity loads into memory. This system reduces the total amount of memory Unity needs for Textures, because it only loads the mipmaps Unity needs to render the current Camera position in a Scene, instead of loading all of them by default. It trades a small amount of CPU resources to save a potentially large amount of GPU memory."

    For textures in a 3D scene with mip maps you can enable texture mip streaming. It will use the object bounds along with camera position, screen size and the mesh uv density to calculate the resolution the texture needs to be on screen. It will use this information to determine how many mips are needed for that texture (considering all occurrences of the texture in the view). It will then load (or unload) texture mips to constrain the GPU memory usage to the texture budget you provide (within the limitations such as max mip reduction level permitted in the chosen settings).

    The manual page and individual API settings give more detail, such as the memory budget include the streamed and non-streamed textures.
     
    pokruchin likes this.
  3. pokruchin

    pokruchin

    Joined:
    Apr 24, 2019
    Posts:
    36
    Thank you for answer! It's quite good if so. Going to try it asap!
     
  4. bfoddy

    bfoddy

    Joined:
    Mar 27, 2012
    Posts:
    50
    @lyndon_unity does this feature actually work in the 2020 LTS build? We find that if it's enabled, in both build and in editor, many mipmaps just stop loading at the lowest level, no matter how much VRAM we have free and how high the streaming budget is.

    For example, in my scene I am using about 3Gb of VRAM with streaming off. I have about 10,000 renderers visible (most are single-quad impostors).

    If I turn streaming on, and set a memory budget of 16384Mb, and set the max mip reduction to 7 (just for visibility's sake) about half the renderers load in mip7 and stop there. Memory usage is about 2.8Gb.

    Setting a mip bias on the Camera's streaming manager component doesn't change it. Changing the mip priorities doesn't change it. It seems to just be choking after processing a certain number of renderers (rather than splitting the work across multiple frames). Happens in build and in editor.

    (here you see one impostor that has loaded properly, and a bunch that are stalled at LOD7)

    Screen Shot 2021-10-04 at 3.21.42 PM.png

    I reported a bug via Hub, but in my experience the QA folks just ignore bugs that don't have full projects attached, so I figured I'd also ask here. Is this a feature we are supposed to know not to use, like dynamic resolution?
     
  5. lyndon_unity

    lyndon_unity

    Unity Technologies

    Joined:
    Nov 2, 2017
    Posts:
    43
    Hi, The mip streaming feature is supported in 2020 LTS. However there are some limitations on what sub systems supported which are listed in the above documentation. E.g if you are calling a draw mesh function directly without a renderer object the texture mip streaming system cannot calculate the desired mips and therefore will only load up the the max reduction. This is because a renderer object is required to provide the bounding box for the mip calculations. Also if you are using a custom shader with complex uv scaling, then the calculated mip could be incorrect as the system isn't aware of scaling within the shader.

    It you provide a reproduction case in the bug report then the feedback could be more accurate for your specific situation.

    Using the debugging API functions listed on this page can help identify exactly what mips the system has calculated, is attempting to load and the memory usage vs the budget.
    https://docs.unity3d.com/Manual/TextureStreaming-API.html

    At the texture level:
    At the system level:
    The following document released on the blog has a debug stats script to help with debugging memory usage and a script to help debugging an individual object in front of the camera. Both may help you identify the issue in your case
    https://docs.google.com/document/d/1P3OUoQ_y6Iu9vKcI5B3Vs2kWhQYSXe02h6YrkDcEpGM/edit
     
  6. bfoddy

    bfoddy

    Joined:
    Mar 27, 2012
    Posts:
    50
    Ok that's useful info - I mean, the impostor shader does do some UV scaling of course since it's loading from a flipbook. For this reason I tried setting a mip bias of -5 both on the shader and on the textures AND on the Streaming Manager component, but it didn't change anything.

    Besides, even allowing for the fact the wrong mip is being targeted by the shader, it still doesn't seem to be working right. In this case, it's loading mip 7 and stopping there. If I set it to a minimum reduction of 2 mip levels it loads mip 2 and stops there. It doesn't matter if I get closer to the object, it never loads mip1 or mip0. Even allowing for the fact that some of these textures are 16x16 flipbooks, that should only result in (at most) a reduction of 4 mip levels, not 7. The important thing is it never ever loads in a higher mip, no matter what. It's not seeking the wrong mip level as a target, it's just not streaming.

    As additional evidence, the debug view shows these props in red, which says 'insufficient resources to load the mips'. But it can't be the memory allowance (which is vastly higher than the total texture memory used), and it can't be the IO or MR usage because then it would eventually stream in after you waited for a while. Nothing in the debug view is green.

    Plus, some almost identical props are happily loading their 0th or 1st mips right next to other props that are at mip7. It really seems like it's not just 'it calculated the wrong target mip level', this is 'it's not progressively streaming in lower mips'.

    As for a repro case, it's not really feasible, right? My git repo is 140gb. Your uploader does not work on projects over 200mb, it just stalls out and stops uploading. I have uploaded literally dozens of repro projects over the years, and I have found reliably from 2012 to 2021, that if you try any method other than the in-editor bug report uploader, QA will leave the bug open and not look at it. So if the repro is too big for the uploader then it just won't ever be addressed.

    What would be helpful is if you could check if your system works on one of your larger-scale internal projects. Right now I don't know if it is confirmed working in any large project at all, since I asked friends and twitter followers who is using it, and the answer is 'nobody', and there are basically no threads on the forums or websites talking about using it, it really seems like I might be the only person trying to get it to work.

    Screen Shot 2021-10-05 at 2.49.12 PM.png
     
    Last edited: Oct 5, 2021
  7. lyndon_unity

    lyndon_unity

    Unity Technologies

    Joined:
    Nov 2, 2017
    Posts:
    43
    It seems something about the imposter setup is likely to be a key factor. Our main mip streaming tests are focused on standard meshes.

    It would be helpful to know more about how you setup the imposters in your project, so we can attempt to reproduce the issue. One possible issue is that the 'mesh metrics' may not have been correctly calculated by the system that created the imposters. If these values are only set to 1 it can be an indicator that they haven't been calculated.

    One approach to sharing the setup is to create a new empty unity project and drop in just the key assets that are problematic. Then create a minimal test scene with those assets present. Sending this much smaller prpject in a bug report would allow us to inspect the setup in detail and make sure we target your specific issue.

    If you list the bug id here then I would be able to review the report quicker.
     
  8. passeridaepc

    passeridaepc

    Joined:
    Jun 16, 2019
    Posts:
    267
    What exactly is meant by:
    If I use shadergraph where I set my own tiling and offset for the UVs - is it complex or not? What if I rotate UVs?
    When does scaling become complex and when is it okay?
    Thanks.
     
  9. lyndon_unity

    lyndon_unity

    Unity Technologies

    Joined:
    Nov 2, 2017
    Posts:
    43
    Offset or rotation would not be a problem as they don't change the texel density on screen.

    Tiling does introduce scale that changes the texel density on screen so could be problematic.
    Scale applied via the material using Material.SetTextureScale or mainTextureScale will be accounted for in the texture streaming system and the correct mip will be loaded.
    If you are calculating or modifying the scale amount within the shader then the mip streaming system will be unaware of that modification, so will be unable to account for in the calculation of the required mip to load.

    (If the scaling is a reduction then it will be be visually ok as the system will request a mip detail lever higher than needed and the GPU rendering will simply use the lower mip. So visually ok, but not saving as much memory as it potentially could. If the scaling is zooming in, it will be problematic. To be visually correct a more detailed mip would be needed and the streaming system would not be aware of that).

    If you need to bias the mip level due to your own scaling you can request a mip level directly for a texture with requestedMipmapLevel. You could read the calculated mip level via calculatedMipmapLevel, then apply the scaling to this level and call requestedMipmapLevel with this modified value.

    Another option is to explore using Streaming Virtual Texturing if you can support its requirements. This performs screenspace uv calculations and therefore handles the custom shader code directly.
     
  10. passeridaepc

    passeridaepc

    Joined:
    Jun 16, 2019
    Posts:
    267
    Thanks!
    I'm already using VT, but it doesn't support lightmaps, shadowmasks and everything that is assigned through a material property block. So tons of textures still have to be streamed somehow. Therefore mipmap streaming for them. Seem like it won't be a problem for lightmaps, because they don't use scaling at all. For other textures I use the scaling that is zooming out (tiles the material), so should be also okay.
    As for "mainTextureScale" is it a reference I should set in shadergrpah property that controls my scaling, so it would be picked up by the mipmap streaming system?
     
  11. BartPieters

    BartPieters

    Unity Technologies

    Joined:
    Apr 12, 2019
    Posts:
    17
    Hi,
    Virtual Texturing does not support Material Property Blocks. You cannot set Textures that are part of a Virtual Texture stack dynamically, including textures on a Material Property Block (see documentation). Mip map streaming also currently does not support Material Property Blocks. This is something however we would like to fix in the future.
     
  12. passeridaepc

    passeridaepc

    Joined:
    Jun 16, 2019
    Posts:
    267
    Yep, that's what I'm saying.

    That I didn't know. But does mipmap streaming support lightmaps and shadowmasks?
     
  13. lyndon_unity

    lyndon_unity

    Unity Technologies

    Joined:
    Nov 2, 2017
    Posts:
    43
    Yes it does. I remember testing with lightmaps myself.
     
  14. BartPieters

    BartPieters

    Unity Technologies

    Joined:
    Apr 12, 2019
    Posts:
    17
    I forgot to answer your question on 'mainTextureScale'. You should be able to follow this approach: Link. By naming your property
    _MainTex_ST
    you get access to the scale and translate property of a texture. That way, in shader graph, you can query the
    mainTextureScale
    . So your shader can use the scaling set by
    mainTextureScale
    and the mip map streaming should use the same property.
     
unityunity