Bug Crash caused by High vram usage on latest unity versions

Hello,

We have been working on a game (Easy Red 2) for several years and we always managed to make it run with a very good performance, the game is in fact out also on Nintendo switch.

However recently we upgraded unity from 2021.3.3 to 2021.3.25 due to the fact that Nintendo was going to stop accepting submissions with 2021.3.3 soon.

Since the upgrade we noticed some weird issues with VRAM on the build: Apparently VRAM usage now keeps growing constantly until at some point, on some hardware the game will crash.
Interestingly the game normally does crash normally only on RTX cards (also it crashes quite frequently on those cards) and only when the graphic API utilized is Direct X11 or 12, while on other platforms (even with wat way lower specs, like Nintendo switch or pc with integrated gpu) the game doesn't crash, but it still present weird high VRAM usage. When using Vulkan API the game normally never crashes on any platform, but again VRAM usage is way too high, basically it is always near the maximum VRAM available on the GPU, it can be 1GB or 16GB of vram, it doesn't matter, the game will use all of it.

We want to emphasize that the game is normally very well optimized and the "memory leak" doesn't look like it's coming from the game itself: The game uses usually less than 2GB of RAM at max graphic settings, and we profile frequently the amount of memory used by textures and every graphical aspect and we shouldn't ever use more than 1.5-2Gb of vram at max graphic settings.
Also we don't have any issue like creating Textures and never destroying them at runtime, in fact we rarely do that anyway.

We tried to profile the game using the profiler at runtime but we couldn't see weird memory usage in the game, only using MSI afterburner and game log files after a crash we notice that the VRAM usage was actually constantly growing until reaching max usage and again, eventually crash when using RTX family cards.
We also want to emphasize that the issue started to present very frequently with unity update, but was not there at all on 2021.3.3

What could we do to investigate the VRAM usage even more to find out what can be causing this? At this point we have a doubt that it is an issue related with some errors made by us (neither stuff coming from the very few asset store packages we have), but rather on the engine side expecially after the update.

Btw, we use URP, we use almost only asset bundles to store our assets, and we use almost only custom shaders, but they are very few and lightweight ones.

Thank you for the help

 

Just a hint for debugging this if you suspect this is due to upgrading Unity. You should take the original 2021.3.3 project and upgrade it to 2021.3.(25/2) = 13 (rounded). Basically divide and conquer, then test whether the issue appears in this version. If it does, try a lower patch version. If it doesn't, try a higher patch level. That way you should be able to hone in on a version that runs your game without these issues and also report a bug about it. You probably don't necessarily have to upgrade to the latest patch level so going back a couple patch versions for now should be a good compromise while waiting for a fix.

You may also want to consult the changelog(s) of all of these versions since 2021.3.3 for "known issues" - or once you found the version where the bug appears, check the changelog for any related fixes/features. Sometimes these provide some clues on how to work around issues.

And of course you could try upgrading to 2022.3 even though that's pretty new, it can't hurt to give it a shot.

 

Thank you, we'll surely try this way at some point because definetly this could give a good idea about what could be the issue, it's going to take a while for us to test this tho because we will probably have to do this process something like 10 times to find the update that started giving the issue, and every time we do this process once we will have to install the unity version we need, upgrade the project with it, compile it, uploading it to a provate steam branch (the game is a lot hardcoded to use steam drm for multiple things and can't start without it) and this can potentially require an entire day for us just for checking one single unity version, so I hoped to have a faster way to check what's going on.

Maybe is there a way to sniff what's inside the vram apart from loaded assets to find if there is something that is unnecessary allocated would be a solution but as far as we know, the profiler is the only way and it doesn't look like he can spot a lot of data about VRAM, but it just shows what texture / models etc are loaded in memory and according to it everything it's fine :/

Another detail regarding the issue tho, it looks like it happens also on 2022.something (i don't remember the exact version) so I wonder if it might be related to some new URP version

 

That is bad. No, that's downright terrible. A build should always be playable from a dev's machine at all times. Typically controlled through a scripting symbol.

This is particularly bad because it adds a lot of complexity (like you said) in order to have a testable build, which means the build will almost certainly be under-tested. A day for each Unity version upgrade is just way too much IMO, which adds to the risk of making an update and when you, as in this case, you cannot properly analyze issues within a reasonable timeframe.

Okay I may be stating the obvious and completely off-topic. But perhaps it's not so hardcoded after all. It may just take a programmer an hour or two to #if out the calls to Steam and return bogus data and refactor to interfaces rather than concrete classes so you can use mocks.

Not sure how far into releasing this game you are and how long you expect to support it afterwards, but if it's more than I'd say two months (assuming a team of less than 10) then you will certainly reap benefits by making the build runnable without Steam.

 

The game has been released 2 years ago and we are still supporting it for at least another year. Unfortunately it's quite a complex game, and it uses steam DRM for a lot of features such as verification of multiplayer users, anti piracy, controller input, workshop mods, and a bunch of other things. It would be possible to make a very limited version of the game for the sake of this test, surely not in like 2 hours, we really never had this necessity before because we normally can easily test our game from within unity editor first, then since normally a change impact only a few asset bundles or files the build of it + the upload can be resolved in like 30 minutes, which is not too much at all. The game is definitely much more tested than developed, it's solid and has 95% positive score on steam counting thousands of reviews, it's not really a skill issue of the team here about developing and testing the game, neither I would say the game is undertested.


While normally we can test the game on unity editor immediatly or by private steam branch in like 30 minutes of compiling + upload, doing a test of the game in various steam branches is a completely different thing, and the steam upload part of the new branch when everything is compiled from zero is long but even without it it still will take us a massime amount of time as we can estimate the time to be something like:
-Importing the game into a new unity engine version: 2-3 hours probably, the game project is something like 30-40GB
-Compiling the game and all asset bundles: Normally we recompile only what's changed, but compiling everything from zero usually takes 2-3 hours as the game and all of it's asset bundles will take approx 12GB of build
-If we need to add the steam upload part of those 12GB on our network it will add something like 1-2 extra hours, but at that point we would anyway be at the end of the working day, meaning that basically we can test the game on a completely new unity version once per day approx, and considering between 2021.3.3 and 2021.3.25 there are 22 versions it's likely that we will have to test like 5-10 different versions, it's going to take a massive amount of time in any case.
It's unlucky that there is not just a way to just detect what is causing what appears to be a VRAM leak.

I don't want to sound rude or negative toward this kind of suggested testing, it's just hard for us to decise to start focusing on something like this because the team is tiny, and this will completely end up stopping the development process for at least a week wich is something we can't really afford for an issue that atm can be workarounded by letting users use Vulkan API instead of Direct X

 

Thanks for the insights! I didn't mean this in a rude way either. I know how things develop over time and then you end up with something that's hardcoded everywhere.

Still, I'm inclined to argue that something like Steam DRM - or any 3rd party framework that you rely on in any kind of software - should be abstracted in such a manner that you can plug-in a different mock version of that framework that it allows the game to be run (minus the framework features). It kind of sounds to me like you have references to Steam framework namespaces in more than one assembly, possibly everywhere where the Steam DRM is used - networking, input, antipiracy, mods and so on.

I'm sure you'll be making another game and for this I want to express that it is a necessity to wrap away any kind of framework to a single assembly that all calls route through using your own interface(s). A lot of patterns and principles come to mind, Abstract Factory and Liskov Substitution Principle first and foremost. About 20 years ago we had a very, very nasty DRM and anticheat thing that hooked into everything code and asset related. None of the developers had anything to do with it except for that very senior dev integrating that library. Essentially, it was just another "MASTER" build config that controlled whether it was active or not, and when it was active you could ONLY play it after installing the game, entering a valid license key, and with a special CD inserted of which we had very few. We could not have shipped if we had been forced to test this way.

Being able to run builds can also matter a lot if you can run and test a build vs edit mode. Not just for performance reasons, and for technical reasons like things that may only occur in a build. But for gameplay reasons as well. Well over 20 years ago we've worked on the Dave Mirra Freestyle BMX conversion for the GameBoy Color and we made only one, but grave, mistake: we didn't test enough on the device! We've had an emulator that ran on the PC so we controlled everything with a keyboard that had pinpoint accuracy. But the actual d-pad of the GameBoy was more inaccurate than we knew, so the game got critiqued first and foremost for its unforgiving input. We noticed to some degree that it was slightly less comfortable to play on the Gameboy, but given our experience with the game we adapted quickly and did not see the problem for what it was.

 

Update:

After followign this recent thread that was reporting an issue similar to mine (the biggest difference was that the issue was happening on the editor instead of on the build in their case) I've been able to resolve the issue;
It seems in fact that the problem was a Unity bug and it was fixed in the latest Unity version a few days ago (2021.3.28).
So far neither me nor the players that were reporting bugs could reach more crashes, so I would say it's finally fixed!