GPU Bound Performance Issues After Lengthy Play Session

Getting an untenable frame rate in my game after playing for an hour and a half or so and finishing many stages.Thought it was a memory leak, but after profiling a PC build to death, it looks like the GPU is becoming very slow at finishing Camera.Render/Drawing/BatchRenderer.RenderSingle/Render.Mesh

There's a huge difference between playing a near-identical stage in the same scene when performance is "being bad" and when it isn't.

[Edit: Removed a bit about Total System Memory Usage as I realized it was fine]

Here are some composite images for the game's GPU profile and memory profile playing the same level:

Good performance:


Bad Performance:


I have also profiled the GPU's Render.Mesh performance throughout the entire playthrough (from the start to when things get bad) and it doesn't seem to gradually get worse over time as I finish more and more levels. It just suddenly starts taking forever for the GPU to render at a certain point.

I am using 2019.3.0f4.

Very stumped and don't know what to begin testing to change. There are many recommendations for correcting GPU bound games, but if my game is performant most of the time, what could be wrong with it after a long test? Any theories on what could be wrong and what I can try changing would be appreciated!

 

Your issue does sound strange. A few things that spring to mind.

1. Are you profiling your hardware gpu and cpu? I don’t mean using Unity profiler I mean some actual hardware profilers to check if usage or temps go up? You mention it happens suddenly and after a prolonged period of play some I’m wondering if thermal throttling might be a cause?

2.is it possible to play levels in a random order or repeatedly play the same level? That might help pin down the cause to say a specific level, your levelling loading code, not releasing some resources etc.

3. Is hard to know what else to look at. As far as I can see from you profiles there aren’t any obvious differences ( it’s a bit hard to see though as Kim currently looking on mobile and the forum keeps closing the image pop up when I try to zoom in). In which case the next stage will be to add more custom profiling stuff.

for example if it were me I’d add a way for the game to play itself and log performance. That way I can let it play for an hour or more without actually having to physically play it. Not sure if you can programmatically generate a profiler log, but that would be nice.

4. Perhaps take some RenderDoc snaps and see if the additional information it provides will give you a clue.

edit
Was able to zoom in on your profiler and spotted something odd. In the bad performance one you seem to have some red spikes in gpu graph, one at the start of the profile and one near the end. Those spikes are categorised as ‘other’ so not very helpful, but the spikes themselves seem uncharacteristic. Might be worth digging into that a bit.

also the strangeness of the problem might suggest perhaps some error occurred on the gpu. Something not bad enough to crash the gpu driver, but maybe something that causes progressive performance lost as the gpu tries to recover or is simply now in a broken state. This doesn’t give a specific solution, though as it only happens after prolong play it might point to an issue with a specific level.

 

Thanks, Noisecrime. That is all very helpful and stuff I will try out. As for the red "other" spikes, they are from doing a detailed memory sample, so nothing to worry about there.