Search Unity

Allocation Failure, Regardless of Lightmap Size or Available GPU Memory

Discussion in 'Global Illumination' started by Kleptine, Aug 20, 2020.

  1. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    282
    I've been seeing this issue since 2019.3. We recently upgraded to 2020.1, hoping it would fix it, but no luck.

    I have a Geforce RTX 2060, and am trying to bake a small scene with a single 2048x2048 map. Essentially, the first time I start my computer, I am able to bake it just fine, but after that it always fails with:
    upload_2020-8-20_16-2-24.png

    The obvious answer is "your GPU has run out of memory", however this happens even if all other programs are closed. Further, I've checked in Process Explorer, and no other application besides Unity is using GPU memory, besides some small allocations by firefox:
    upload_2020-8-20_16-5-10.png

    So.. what gives? Is there anything we can do?
     
  2. thefranke

    thefranke

    Unity Technologies

    Joined:
    Jun 20, 2015
    Posts:
    153
    Hey Kleptine,

    this is weird. Does this happen on a specific scene, or does this fail with for instance a scene with just a baked light and a baked cube in it?

    Cheers
     
  3. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    282
    Hi,

    It happens in all scenes. I just created a simple scene with a cube, and invoked the GPU lightmapper three times:
    1. Built fine.
    2. Fell back to CPU Lightmapper with same error as above.
    3. Just hangs. Nothing happens. No output in the console, just a spinning busy signal and the bouncing 'global illumination' progress bar.

    I actually hadn't seen scenario #3 before, but now I'm getting it for all invocations after.

    I've attached more system information, and my lightmapping settings.
     

    Attached Files:

  4. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    282
    Edit: Yow, I just tried to exit Unity after this test and now Unity has hung!
    upload_2020-8-25_10-58-32.png

    It's odd because I was able to cancel the hung lightmap baking. I guess something in the background wasn't released properly.
     
  5. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    282
    Hi, any updates on this? I still can't run a GPU lightmapper bake on my device. My partner, who has an AMD GPU, can bake the exact same scene just fine, but mine still fails with this error.
     
  6. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    282
    Bump. Tried this on the latest Unity 2020.1 build. Still can't build lighting.
     
  7. Pema-Malling

    Pema-Malling

    Unity Technologies

    Joined:
    Jul 3, 2020
    Posts:
    324
  8. Bordeaux_Fox

    Bordeaux_Fox

    Joined:
    Nov 14, 2018
    Posts:
    589
    I think the old versions of the GPU lightmapper have the problem that a certain Nvidia cache folder is not cleared. Deleting this temp folder worked for me back than.

    I also have a RTX 2060. You can only use 2GB of the complete memory. If your resolution is too high, it will fail.
     
  9. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    282
    Which temp folder did you delete? Deleting the Unity Temp folder has no effect.

    The resolution is really low. Only a single 2048 textures.
     
  10. Bordeaux_Fox

    Bordeaux_Fox

    Joined:
    Nov 14, 2018
    Posts:
    589
    %APPDATA%\Roaming\NVIDIA
     
    Pema-Malling likes this.
  11. VincentAbert

    VincentAbert

    Joined:
    May 2, 2020
    Posts:
    123
    Hi, is there any news on this ?

    I get a similar problem on my Nvidia RTX 2060 Super. First bake of the day goes great, and then the lightmapper just doesn't want anymore. Just like Kleptine, it falls back to the cpu, it just hangs...

    I also get a ton of these errors :

    Code (CSharp):
    1. [PathTracer] InitializeLightProbeData job with hash: 73d3dda406a533a7ead114d9a02d4edd failed with exit code 1.
    But it doesn't make any sense, even if the bake is 10 times bigger the first time it will work fine, and then there is just no way to do anything...
    Drivers are updated.

    Thank you !
     
  12. Pema-Malling

    Pema-Malling

    Unity Technologies

    Joined:
    Jul 3, 2020
    Posts:
    324
    Hey there. This thread is a bit old and I don't think a bug was ever reported. I suggest filing a report if this is blocking you.

    Also, which version of Unity are you on? I know there were some fixes preventing erroneous fallbacks recently.
     
  13. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    282
    I fixed this by purchasing a 1080Ti, sorry. :\

    For what it's worth I think the issue I was seeing was that it was attempting to allocate too large of a memory buffer, not running out of VRAM. On the 2060RTX I think the max allocation is around 2gb. The latest version of Unity have tiling that have improved this a lot.
     
  14. VincentAbert

    VincentAbert

    Joined:
    May 2, 2020
    Posts:
    123
    Haha I wish I could afford that fix ;)

    I'm interested in the tiling feature you're talking about though, what are you refering to exactly ?

    I guess in the meantime I'll try and report a bug, but I'm having a hard time understanding where this comes from, since some scenes don't create the issue...

    @Pema-Malling I'm on 2020.3 LTS's latest version.

    EDIT : Oh while I'm at it, for the light probes error I'm wondering if it could be because I have a lot of light probe groups, each comprised of very few light probes (because they are integrated in different prefabs), rather than a few big ones, can this be an issue ? It still wouldn't explain why it always works the first time though... But I'm just trying to get over this as it is really slowing down my iteration capacity.

    I should maybe start a new thread rather than use this one ?
     
    Last edited: Aug 18, 2021
  15. Pema-Malling

    Pema-Malling

    Unity Technologies

    Joined:
    Jul 3, 2020
    Posts:
    324
    About tiling: If you don't have enough memory for the bake, lightmaps will be split into tiles which are baked separately.

    Using this thread is fine, but if you really want this to be investigated, you should submit a bug report.
     
    VincentAbert likes this.
  16. VincentAbert

    VincentAbert

    Joined:
    May 2, 2020
    Posts:
    123
    Okay thank you... I'm trying to understand what the issue is, and to make at least possible to reproduce it so that I can file a bug, but I'm having a very hard time doing it, especially since I'm no expert in these matters... It seems I get a whole set of errors, different each time. Right now I just got :
    Not enough GPU memory to fit lightmap in memory, even in tiled baking mode. Available GPU memory: 44.3 MB

    I don't understand how my RTX 2060 can have so little available memory, Unity being the only open program ?
     
  17. Pema-Malling

    Pema-Malling

    Unity Technologies

    Joined:
    Jul 3, 2020
    Posts:
    324
    This could be due to a variety of things. For instance, we had an issue not long ago where the GPU lightmapper was preferring integrated graphics or the CPU itself as compute devices, causing issues similar to this. Might also be a driver-related issue. It's pretty hard to say without looking at your logs, which should be included if you submit a bug report using the in-editor tool.

    If you managed to get en error like the one you posted, vital troubleshooting info may be present in your editor logs, which you can also take a look at yourself if you'd like. https://docs.unity3d.com/Manual/LogFiles.html
     
    VincentAbert likes this.
  18. Kleptine

    Kleptine

    Joined:
    Dec 23, 2013
    Posts:
    282
    Two things:
    1. It's usually not your GPU running out of memory, it's just there's a maximum allocation size, which returns the same out of memory error. The allocation size is usually ~20-30% of the full GPU VRAM, I think.
    2. In our case, we had the AA settings set *way* too high in the lightmap parameter asset. This mean the allocation being made for baking was much too big and blowing past the max allocation size.
     
  19. PublicVoidStudios

    PublicVoidStudios

    Joined:
    Apr 14, 2017
    Posts:
    2
    I had simillar problem, but solved it by simply cleaning GI Cache in Preferences (not sure how it works, but this helped me)
     
  20. plmx

    plmx

    Joined:
    Sep 10, 2015
    Posts:
    308
    Since this is the first thread to pop up on Google with the error message "not enough gpu memory to fit lightmap in memory, even in tiled baking mode", let me report that I am still seeing this error.

    Unfortunately, this happens to me pretty randomly. Sometimes it works flawlessly for dozens of builds, then it absolutely refuses to work with the same settings. This is on 2021.3.13f1 (LTS) with an NVIDIA GeForce RTX 3060 Ti.

    Since it simply works™ sometimes and doesn't at other times with identical settings, the suggestions made in this thread and elsewhere (upgrading CPU drivers, changing sample size, changing bounces, AA settings, etc.) don't seem to have an impact (and yes, I tried). I also tried rebooting, deleting GI cache, deleting the scene lightmapping files, etc. etc.

    Just now I clicked on Bake to get this error with "Available GPU memory: 0.5GB". Frustrated, I canceled, switched back to GPU, and clicked Bake again, to be presented with the same error message with "Available GPU memory: 2.05GB". I then made a full CPU build, and then - out of curiosity - changed back to GPU and clicked Bake again. Voila*, it works.

    The reason I am posting this here is that I would like to help with more detailed error logs, but the error log is, sadly, not very informative; I am not seeing anything except the error message shown above. May I humbly suggest that you add more debugging information to these logs such that we have a chance to report back to you what went wrong?

    And I would also like to report a bug, but I can't submit a 10GB project, and since, as I said, this happens randomly, I could spend weeks trying to reproduce this in a small example project. It's an unsatisfying situation.

    Philip

    * which is ... French ... for ... "and then I found out"
     
    Last edited: Nov 18, 2022
  21. Pema-Malling

    Pema-Malling

    Unity Technologies

    Joined:
    Jul 3, 2020
    Posts:
    324
    To whom it may concern: We've recently fixed a bunch of bugs relating to tiled baking. I suspect these might be partially responsible for the behavior you guys are seeing. Most of those fixes are currently being backported to 21.3 and 22.2.

    Also, in 23.1 and going forward, we've taken a more manual approach to tiling. You can now select directly in the lighting window which tile size to use, giving you control over the tradeoff between baking speed and memory usage. If you still face these issues on 23.1 and future versions, fiddling with that setting might solve them.
     
    Artemko_one and plmx like this.