Search Unity

Terrain.UpdateMaterials very expensive

Discussion in 'General Graphics' started by FallenTreeGames, Feb 1, 2019.

  1. FallenTreeGames

    FallenTreeGames

    Joined:
    Jun 22, 2016
    Posts:
    25
    Hey,

    We're looking at performance issues in our game and I've discovered that Terrain.UpdateMaterials (which is performed as part of the culling step) is taking around 2ms a frame. Which is particularly annoying since we never change those materials.

    Anyone else seen anything similar or know how to fix this?

    We're using unity 2018.3.2f1, though the problem has been there throughout previous versions. We use a custom terrain material.

    Thanks!
     
  2. dizzy2003

    dizzy2003

    Joined:
    Nov 25, 2013
    Posts:
    91
    twobob likes this.
  3. chris_schubert

    chris_schubert

    Joined:
    Jan 8, 2019
    Posts:
    28
    Any luck resolving this?
     
  4. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,888
    If anyone can share a simple project that shows the issue we can take a look :)
     
  5. Fewes

    Fewes

    Joined:
    Jul 1, 2014
    Posts:
    253
    I don't have a sample project, but I am seeing a similar loss in performance (about 1ms per camera, with no GI enabled) on Terrain.UpdateMaterials. Could you perhaps shed some light on what this function does/when it runs?
     
  6. BluudNobbut

    BluudNobbut

    Joined:
    Aug 31, 2019
    Posts:
    3
    Did you find a solution to this or still struggling?
     
  7. Rademanc123

    Rademanc123

    Joined:
    Aug 1, 2018
    Posts:
    2
    We have the same problem.
     
  8. Philip_Zhang

    Philip_Zhang

    Joined:
    Feb 15, 2017
    Posts:
    24
    Hi, I noticed the same problem here. I'm using Unity 2018.4.0f1.

    I was testing the performance of my own terrain shader. What my own shader does is I modified the standard shader and enabled terrain tessellation and TerrainLayer mask texture in the standard render pipeline.

    I was hoping that I will have performance loss due to the added features but ironically I don't have the Loading.ReadObject in Terrain.UpdateMaterials so I actually have 16+ms performance gain. I really don't understand why.

    I tried to enable and disable GI but the results are the same.

    Here is the snapshot of my own terrain shader.
    My Own Shader.png

    Here is the snapshot of the standard terrain shader, with 16+ms Loading.ReadObject
    Standard Shader.png
     
  9. Philip_Zhang

    Philip_Zhang

    Joined:
    Feb 15, 2017
    Posts:
    24
    Hi, another find here.

    I was trying to figure out what modification I made cause Loading.ReadObject disappears.

    And I find at the bottom of Standard-FirstPass.shader or MyTerrain-FirstPass.shader

    Dependency "AddPassShader"    = "Hidden/TerrainEngine/Splatmap/Standard-AddPass"
    Dependency "BaseMapShader" = "Hidden/TerrainEngine/Splatmap/Standard-Base"
    Dependency "BaseMapGenShader" = "Hidden/TerrainEngine/Splatmap/Standard-BaseGen"


    If I have any Dependency using these standard shaders, I have that Loading.ReadObject, but if I change them all to mine like

    Dependency "AddPassShader"    = "Hidden/TerrainEngine/Splatmap/MyTerrain-AddPass"
    Dependency "BaseMapShader" = "Hidden/TerrainEngine/Splatmap/MyTerrain-Base"
    Dependency "BaseMapGenShader" = "Hidden/TerrainEngine/Splatmap/MyTerrain-BaseGen"


    Then Loading.ReadObject disappears.

    I don't quite understand how Dependency works because I can't find any documentation about it. But this seems like a clue for you to track down.
     
  10. zhujiangbo

    zhujiangbo

    Joined:
    Nov 25, 2013
    Posts:
    6
    could please take a look at the report of @Philip_Zhang
     
  11. zhujiangbo

    zhujiangbo

    Joined:
    Nov 25, 2013
    Posts:
    6
    the problem only existed in editor.
     
  12. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,888
    Sure - what’s the case number?
     
  13. Philip_Zhang

    Philip_Zhang

    Joined:
    Feb 15, 2017
    Posts:
    24
    Hi @richardkettlewell, I have submitted a bug report (case number 1204901) regarding this issue.

    We noticed that Loading.ReadObject doesn't exist in the built program so it's editor only.

    I don't see Loading.ReadObject when using a newly created terrain or after import heightmap but It appears after I painted TerrainLayers on it and we have Splatmaps.

    I failed to reproduce that Loading.ReadObject disappears when I use my own terrain shader as I mentioned previously.
     
    richardkettlewell likes this.
  14. cpaunFunLabs

    cpaunFunLabs

    Joined:
    Jun 13, 2017
    Posts:
    13
    Bump,
    We are using Unity 2018.4.18f1 and Terrain.UpdateMaterials takes 4 ms (64 calls due to our map being tiled) while the GenerateBaseMap called inside Terrain.UpdateMaterials takes only 0.01 ms!
    It looks like some bad C# code is happening inside Terrain.UpdateMaterials and not the generation itself is the problem.

    We are also using GPU computed basemaps by overriding the BaseMap and BaseMapGen dependencies in shader.
     
  15. Turniper

    Turniper

    Joined:
    Nov 2, 2013
    Posts:
    10
    I appear to be experiencing the same issue, both with the standard terrain shader, and microsplat. Each terrain chunk takes 0.3 ms per frame, adding a good 8-10ms every frame to Camera.Render, with the vast majority of the time within being consumed by Loading.ReadObject. I'm using the legacy pipeline, and have tried with and without runtime GI and am seeing it in both. Edit: Seeing this in 2019.4.0f1. I'm also using MapMagic, but don't see why that would cause this.
     
  16. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,888
    Unfortunately, case 1204901 was closed by QA because the reporter didn't come back to them to answer a follow-up question. Someone will need to report a bug and see it through to the point where QA reproduce it and send it to the dev team to fix, in order for progress to be made on this.
     
  17. Turniper

    Turniper

    Joined:
    Nov 2, 2013
    Posts:
    10
    Alright, I'll see about seeing how small a test scene I can reproduce it with tonight and then open a new one.
     
    richardkettlewell likes this.
  18. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    944
    This crazy CPU cost has forced me to basically re-implement the terrain using compute shaders. Almost 3ms on Switch with 9 terrains. That's all you need to repro it: create a scene with 10+ terrain components, run on Switch (or XB1, or PS4, it's not that much better there either) and watch it eat away your main thread in the profiler.

    It seems to be whatever code is responsibly for setting the material parameters needed for terrain rendering is doing int very, very badly. The only way to make that bar do down is by assigning a null material to the terrain component.
     
    Turniper likes this.
  19. Turniper

    Turniper

    Joined:
    Nov 2, 2013
    Posts:
    10
    It happens on PC too, which is where I'm experiencing it. I will get around to that report and repro scene eventually, had to spend yesterday on my actual paying job.
     
    richardkettlewell likes this.
  20. Turniper

    Turniper

    Joined:
    Nov 2, 2013
    Posts:
    10
    Alright, there's a report submitted for it.
     
    richardkettlewell likes this.
  21. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,888
    Let us know the case number once you have it, and I’ll keep an eye on it.
     
  22. Turniper

    Turniper

    Joined:
    Nov 2, 2013
    Posts:
    10
    It's case 1256373.
     
    richardkettlewell likes this.
  23. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    944
    It would be amazing if this actually gets fixed. The high CPU cost of the standard terrain has been a known problem since forever and it is why many resort to solutions like converting terrain into meshes when they get bitten by it late in development.
     
  24. Lesnikus5

    Lesnikus5

    Joined:
    May 20, 2016
    Posts:
    131
    I want to have a lot of terrains (256 pieces), Terrain.UpdateMaterials eats up 4-5ms, is it possible to somehow reduce the load? Maybe some third party shader from the asset store can handle this?

    An attempt to dynamically enable and disable distant landscapes, depending on the distance to the player (for example, activating only 36 of the 256 terrains near the player enabled), leads to the fact that when a piece is activated, Terrain.UpdateMaterials gives a spike for 5 ms since it calculates all enabled terrains.

    1111.jpg
     
    Last edited: Aug 13, 2020
  25. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    944
    The only way to get rid of the Terrain.UpdateMaterials cost is by setting drawHeightmap flag to false *and* set the terrain materialTemplate to null. Which means, not drawing the terrain at all.

    Pretty much everyone uses the Terrain-to-mesh asset to generate plain mesh renderers to draw instead of the terrain. The downside is that it puts more strain on the GPU, since it doesn't have the progressive LOD an actual terrain should have, increasing the total number of vertices and micro polygons on screen.
     
  26. Lesnikus5

    Lesnikus5

    Joined:
    May 20, 2016
    Posts:
    131
    Strange, this method does not work for me. Terrains are no longer visible, but still Terrain.UpdateMaterials eats performance unchanged. Perhaps I am doing something wrong? Or did they change something in the Unity versions? I am using 2019.4.6.f1 (LTE).

    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4.  
    5. namespace Test
    6. {
    7.     public class Optimizer : MonoBehaviour
    8.     {
    9.  
    10.         public GameObject[] terrainsGO;
    11.         Terrain[] terrains;
    12.  
    13.         private void Start()
    14.         {
    15.             terrainsGO = GameObject.FindGameObjectsWithTag("Terrain");
    16.             terrains = new Terrain[terrainsGO.Length];
    17.  
    18.             for (int i = 0; i < terrainsGO.Length; i++)
    19.             {
    20.                 terrains[i] = terrainsGO[i].GetComponent<Terrain>();
    21.                 terrains[i].drawHeightmap = false;
    22.                 terrains[i].materialTemplate = null;
    23.             }
    24.         }
    25.     }
    26. }
     
  27. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    944
    Strange. Is the time reported in Terrain.UpdateMaterials any different if you assign a valid terrain material?
     
  28. Lesnikus5

    Lesnikus5

    Joined:
    May 20, 2016
    Posts:
    131
    I didn't bother to deal with this. I found another solution. Each terrain needs to set a different grouping ID and disable Auto Connect. Then each terrain is processed separately and activation of hidden terrain pieces no longer causes spikes.

    I don’t know if this will cause problems with shadows on the seams in the future or something else, so far everything works well

    3333.jpg
     
    Turniper and Neto_Kokku like this.
  29. Turniper

    Turniper

    Joined:
    Nov 2, 2013
    Posts:
    10
    That's good to know. I'll have to try that out with my setup. As an update, the bug report is still pending, a QA guy has looked at it but is still trying to reproduce it.
     
  30. Turniper

    Turniper

    Joined:
    Nov 2, 2013
    Posts:
    10
    FYI, I was just notified that this issue should be fixed in Unity 2021.1.0a1 and above.
     
  31. Raul_T

    Raul_T

    Joined:
    Jan 10, 2015
    Posts:
    343
    Will the fix get backported to stable versions of Unity? @richardkettlewell

    We are currently running into this on 2019.4 in a production project that is about to release in the upcoming weeks and it currently bugs out our QA Builds with abysimal performance even on high end computers (i7 + rtx2080 etc) - 36 terrain tiles takes about 5ms to "update materials" each frame - which is half of our total render time.

    Upgrading the project to an ALPHA version of Unity (2021.1) to get this somewhat serious issue fixed before release seems like a no go for us, so will this get backported or should we start looking into other hacks to solve this?
     
  32. richardkettlewell

    richardkettlewell

    Unity Technologies

    Joined:
    Sep 9, 2015
    Posts:
    1,888
    i looked at the code change, and it seems pretty complex, so my guess is no it won’t be.
     
  33. i2eilly

    i2eilly

    Joined:
    Feb 27, 2018
    Posts:
    45
    Did upgrading your project improve your CPU thread time? It is weird that I find myself on 2019.4 LTS and these issues aren't going to be fixed :(
     
  34. bdubbert

    bdubbert

    Joined:
    May 18, 2018
    Posts:
    5
    Is there any update here? Was anyone able to find a temporary workaround for 2019.4?
     
  35. Raul_T

    Raul_T

    Joined:
    Jan 10, 2015
    Posts:
    343
    Sorry for the late reply, didn't saw the notification for this back then.

    We haven't upgraded yet, still on 2019 LTS. We plan upgrading to 2020 LTS later this year, probably Q2. We are definitely not upgrading to 2021 TECH though since our game is already released and we don't want to introduce new engine related issues, and from what I recall the fix is not in 2020 either.
     
    i2eilly likes this.
  36. RubberBandGames

    RubberBandGames

    Joined:
    Jul 20, 2020
    Posts:
    122
    Yeah we are also having this issue in 2020.1.16f1, going to try this group id changing on each terrain to see if that makes a difference...
     
  37. RubberBandGames

    RubberBandGames

    Joined:
    Jul 20, 2020
    Posts:
    122
    Just to follow up on this, changing the group id and turning off auto connect, didn't solve the issue
     
  38. Lesnikus5

    Lesnikus5

    Joined:
    May 20, 2016
    Posts:
    131
    Just by the way. Changing the group id allows to disable/enable distant terrains without spikes, thus keeping the number of simultaneously worked terrains low. The number of Terrain.UpdateMaterials calls is reduced. But the material at the seams is poorly processed and the joints of the terranes become visible, so I decided not to.
     
  39. bdubbert

    bdubbert

    Joined:
    May 18, 2018
    Posts:
    5
    I'm in the process of making a copy of my project and upgrading to 2021.1.0b7, but it makes me super nervous to do so since I'm on a pretty limited schedule to get this stuff working. It seems like my only other options are either a) having my terrain guy try to heavily reduce our tile count while still maintaining the same texture quality or b) swapping to a mesh terrain. I've looked at some assets on the asset store to do the terrain to mesh swap but its hugely important that our textures (which are real world satellite images) stay the same quality. Does anyone know if assets like Terrain to Mesh are able to correctly preserve textures during the conversion?
     
  40. zeroyao

    zeroyao

    Unity Technologies

    Joined:
    Mar 28, 2013
    Posts:
    169
    Hey,

    As @richardkettlewell found out the fix is not trivial and the team has to rewrite many code to make it work. Therefore we think backporting to 2019 LTS would consider to be too risky.

    Some more information:

    1. The original slowness is mainly due to repeatedly collecting splat material properties (e.g. "_Splat0", "_Splat1") from Terrain Layer objects and copying them into the internal Material object that is created for rendering the terrain, on every frame, on the main CPU thread.

    2. It is an overhead per active Terrain object in the scene. It appears to me that there is no other way on the user side could avoid that.

    3. The fix takes another approach, that only does the property copying when Terrain Layer objects change (or any internal resources that are used by the terrain rendering material change). This approach involves hashing of several states, and we already found a few related regressions (Worry not! They have already been spotted and fixed.) If you find more related issues in 2020.2+ you are more than welcome to make reports and we'd greatly appreciate them.

    Again, sorry we couldn't do further backports prior to 2020.2 and we recommend you upgrade your project if the issue is the bottleneck of your project's performance.
     
    Lesnikus5 likes this.
  41. gecko

    gecko

    Joined:
    Aug 10, 2006
    Posts:
    2,183
    So to be clear: This is backported to 2020.2 (not only in 2021.1 as stated initially above) -- and that should soonish be in LTS. Right?
     
  42. isaac-ashdown

    isaac-ashdown

    Joined:
    Jan 30, 2019
    Posts:
    69
    @zeroyao could you confirm if the fix is in 2020.3.0f1? It appears this is the issue tracker:
    https://issuetracker.unity3d.com/is...-usage-when-using-terrain-dot-updatematerials
    And there it says "Planned for 2020.3.X", but it seems you are saying it was already in 2020.2? We have compared performance between 2019.4 and 2020.3 on Switch and it does appear to have improved, so we are wondering if we should expect more improvements soon in a future 2020 version. Thanks!
     
  43. wlad_s

    wlad_s

    Joined:
    Feb 18, 2011
    Posts:
    140
    Now the issue tracker says it's "Fixed in 2021.1.X".

    But @zeroyao mentioned it being in 2020.2, the issue tracker doesn't mention it being found in 2020.3, so is it fixed in 2020 LTS or not?
     
    Dev-Raz and knxrb like this.
  44. Lesnikus5

    Lesnikus5

    Joined:
    May 20, 2016
    Posts:
    131
    I am using 2020.3.3f1 (LTS) and no have problem. I have 256 terrains on stage at the same time and it takes about 1-2 ms per frame to process them. I think that I can reduce the cost to a fraction of a millisecond if the terrains do not have layers at all, and the visualization is completely entrusted to Microsplat shader with custom splatmaps. Unfortunately, it automatically adds layers to the terrain, even when using custom splatmaps, but I can turn it off as soon as I get my hands on it. jbooth said it was not important to function: https://forum.unity.com/threads/fre...or-unity-terrains.487458/page-79#post-6890489
     
    wlad_s likes this.
  45. Tomcat9

    Tomcat9

    Joined:
    Dec 10, 2020
    Posts:
    4
    Another deal-breaking bug on Unity's side forcing us to upgrade our projects to another editor version with other bugs. At this point Unity seems to have more bugs than Bugs Life and more glitches in it's core systems than Metal Arms: Glitch in the system.
     
    Last edited: Jun 20, 2021
    Dev-Raz likes this.
unityunity