Official Progressive GPU Lightmapper preview

@KEngelstoft i finally found the reason for getting black parts of the meshes all over the scene in all the projects i tested !

It is the "Pushoff" value. I never thought that this might cause any issue because it is set by default to 0.0001 which seems correct to me as far as i am aware of other renderers, but it does produce issues with the CPU and GPU lightmappers !

Note that the objects in the images below are placed around a couple of KMs away from the origin !

Result with default tolerance of 0.0001 - incorrect ( this value is set to all the presets and that's the reason i did not get good results with any preset loaded ) Could this be a floating point precision issues with Radeon Rays API ? But it is also valid for the CPU Lightmapper !

GPU


CPU




Result with tolerance set to 0.001 - correct - I created a custom preset and did change the value !

GPU



CPU

 

Hi again guys,

I've tried again to bake with GPU PLM in latest 2019.2.1f1: same error using my RTX 2080 or my GTX 1070 with any lowest setting possible. I've tried forcing to use another GPU (Intel Graphics) to render the Editor scene; it doesn't help.

I still the same error:

Code (CSharp):

1. (Filename: C:\buildslave\unity\build\Editor/Src/GI/Progressive/OpenCL/OpenCLCheck.cpp Line: 134)
2. OpenCL Error: 'GetBuffer(kRRBuf_lightRayIndexToPathRayIndexBuffer).EnqueueClearBuffer(openCLState)' returned -4 (CL_MEM_OBJECT_ALLOCATION_FAILURE)!
3. OpenCL Error. Falling back to CPU lightmapper. Error callback from context: CL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_NDRANGE_KERNEL on GeForce RTX 2080 (Device 0).

It works perfecly fine on 2019.1.14f1.

Any ideas to help with this? I've got latest nVidia drivers; should I roll back the one recommended in the very first post? Seems like a pretty old driver so haven't tried that yet.

Thanks,
Charles

 

Hi, i am using GTX 1060 with this latest 2019.2.1f1 and it is working great.
Have you tried the suggestion of cleaning the compute cache folder ?
I assume you are on windows.
Just go in that location and delete this folder - "ComputeCache": I actually deleted everything in the NVIDIA folder !
C:\Users\"Your User Name"\AppData\Roaming\NVIDIA\ComputeCache

 

Hi, thank you for replying!

I did indeed delete this folder; and I also deleted all NVIDIA folders in all 3 Roaming / Local / LocalLow and reinstalled entirely latest drivers and didn't help either

I just discovered that in a very small level it works - but strangely it doesn't work in a real level, where it works well as soon as I downgrade back 2019.1.14f1

I will try this again, and maybe delete parts of the level gradually to see if a specific object could create this error (or level size, but would be weird that it's only "too big for my RTX2080" even with 32x32 lightmap size and resolution set to 1 from 2019.2.1f1, whereas I can bake 2k resolution 28 on that same level on 2019.1.14f1... confused

 

It does not make sense to me. It should be an issue. Unfortunately i can not think of a fix.
Have you tried other OpenCL based applications like Blender renderer benchmarks and the Lux renderer, just to be sure that they are working ?
Also there are some things that you can make to allow driver ( or OS - not sure ) to allow using more memory for OpenCL like this "GPU_FORCE_64BIT_PTR=1" go and search the web about "Environment Variables". I know there are some entries that can be added in order to improve memory allocation - well mostly for AMD cards, but you can go and search the web for these things !
Also do you have the latest windows updates ?

Another thing you can try ( i would try it for sure, but i doubt Unity would recommend this ), is to go to 2019.1 Unity install folder and copy the OpenCL.dll file and paste it to the 2019.2 folder. There might be some things changed if eventually the 19.2 version is using a dll compiled with a newer Radeon Rays SDK version ( just guessing ) ! Just be sure, back up the OpenCL.dll from the 19.2 folder just in case so you do not have to re install it if it does not work !

 

Thanks a lot for these suggestions and your lengthy reply I'm going to try this and keep you posted on what happens

 

Hello,
Thanks for your reply.

I tried with Filter set to None too, and all settings to a minimum, it doesn't help unfortunately. I've tested the DLL replacement suggested above, but there isn't any OpenCL.dll it seems so I tried with various other DLLs, didn't help either.

EDIT: It happens on multiple machines with various settings, it maybe is project-related to my project only as with other scenes it seems to work fine. I did clear all caches / delete lighting data, delete all NVIDIA folders, reinstall latest drivers, etc.

Thanks,
Charles

 

Does the Project Architecture matter - i mean, if the Windows Project is set to 32 bit try to set it to 64 bit !

 

Unfortunately seemingly tiny changes to a level can cause the GPU lightmapper to blow up and fail. We just had an issue (with 2019.2.0f1) where suddenly a level started failing to bake, even when the settings were all turned way down. After losing hours and hours of dev time, we finally narrowed it down to too many light probe groups (someone happened to try disabling all light probe groups in the Light Probes tab of light explorer, and suddenly the bakes worked fine).

 

Thanks, I'm going to try this and tell you how it goes

 

For me it works on small scenes but anything big it falls back to cpu baker. This is using a gtx1080ti.

 

@icefallgames @Aze_ We are also getting the same issue with the GPU lightmapper not able to bake when there are any Light Probes on the map. This seems to be a regression in the latest Unity. Is there a bug tracked for this?

I tried to bake the same map with CPU lightmapper, and it doesn't work either and seems to run out of memory after a while.

 

@KEngelstoft Thanks! That completely fixed the issue and I baked without errors. Do you happen to know what the lack of baking on LOD Groups in GPU Lightmapper actually means? Does it mean any static object with a LOD Group component gets no indirect lighting baked?

For any onlookers who want to save time, copy/pasting of Light Probes between groups is not trivial. To do it you have to select all the light probes, hit CTRL-C, go to the new light probe group and select one of the light probes in that group before pressing CTRL-V. We had hundreds or probes so this helped save a lot of time.

 

I'm also assuming lightmapping on Terrains isn't working yet? I get good results with CPU but bad with GPU.

 

[PathTracer] InitializeLightmapData job with hash: b9f2fdc6080521d91d4e0858a1dc35e5 failed with exit code 2. How fix this?

This is a nightmare. I not can create lightmaps more one month for one city map, why I have this error? On CPU I have more 20 hours for baking, on GPU I have this error.

Please get the script, which ignore this error or something, I can pay for that, because I really need create lightmaps.

 

@KEngelstoft Apologies, it looks like it might be a general lighting issue and not just related to the GPU. Thought I'd show you a screenshot anyway. Do you know why the shadows from trees would be blocky like that on the terrain?

There is also several instances in my game of where mixed lights do not seem to render any direct light either. I'm Forward rendering and the pixel light count is at 4.

 

Oookaay. Thank you. I spent one month for understanding this error... Why not write in console: Out of memory? This is standart message in other engines and programs.
This problem is randomly, I will start baking 3-4 times and will have this error on start or on finish. 16GB ram, about 12 free and 4GB vram - how much need for normal baking?

Okay, I have more problems:
1) why scene is dark after baking?
2) why some objects have a white or black spots, how fast fix this problem? I have not this problem in Unreal, so I think that problem not in meshes. Also, sometimes this spots dissapear, how it works?

Sorry. but I really tired. After one hard month I have this poroblems again and again...Unfortunately I not can upgrade my PC at this time.

 

Just got a Radeon VII in a sonnet e-gpu for my MacBook 2016.
It's working fine in Mojave and unity 2019.3.0b2.

On some scenes I get LightmapRasterize.cpp(574) in RasterTriangle - Triangle failed to raster! v1
What does this mean?

 

Thanks I will check for the model

 

Hello there, recently got around to baking parts of my project. Ran into a problem. No matter how low I set the lighting settings, the light mapper fails to bake in GPU mode. I have a 980Ti, so the memory should not be an issue. The errors I get are the following,

"OpenCL Error. Falling back to CPU lightmapper. Error callback from context: CL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_NDRANGE_KERNEL on GeForce GTX 980 Ti (Device 0)"

"OpenCL Error. Falling back to CPU lightmapper. Error callback from context: CL_MEM_OBJECT_ALLOCATION_FAILURE"

"OpenCL Error. Falling back to CPU lightmapper. Error callback from context: CL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_WRITE_BUFFER on GeForce GTX 980 Ti (Device 0)"

Any suggestions or help would be appreciated.

My Unity Version is 2019.2.5f1

 

As I replied on the other thread, reducing lightmap resolution helped for a while for me. But when my scene started to grow again, the same error appeared. Now I deleted all light probes and I'm able to bake with gpu again. Deleting light probes might not be a proper fix but have you tried that?

 

It looks like it only has 6GB?

- try reducing the number of light probe groups you have (not light probes)
- reduce your lightmap size. I have an 8GB card, and usually can't bake any higher than a 512 lightmap size for our levels.
- make sure your graphics drivers are up to date - if it fails no matter what the settings, even for small scenes, this might be the issue

 

6GB is over kill for the scene I'm trying to bake trust me lol. There's almost nothing here. I figured out that having multiple small terrains causes it to sometimes fail. So I just created one giant terrain. Problem solved.

There are no light probe groups in the scene at the moment.

Another strange problem is that if I scale any mesh up too high, the GPU fails the bake. Very odd. I'm trying to upscale some background structures, and bringing them past 1.5X the size causes the GPU bake to fail.

Drivers are up to date. Light map size is on 512. Just got home today and tried it on my personal PC (1080Ti) and the same issues persisted.


Edit: Anybody able to shed some light on why up sizing meshes past a certain size can cause the GPU light bake to fail?

 

Nvidia gpu have a limit of 25% ram usage when using OpenCL . So you can only use around 1.5GB ram on 980ti for baking .

 

This saved me, I was about to throw my system out the window.. thanks so much!

Successfully baked all my scenes at 2048 resolution in around 6 minutes... amazing stuff!!

// WildStyle

 

So, for me GPU lightmapper always crashes with even single 1024x map. 1050Ti 8GB RAM.

Will this be resolved in the future? Will getting another 8GB of RAM help? Or will it just allow me to bake 1024 and nothing higher?

 

GTX 1050 Ti could have at max 4 GB of memory.
How much is the memory of your GPU - 2 GB or 4 GB
When you say 8 GB, are you referring to system memory, or GPU memory. Because the system memory is only used to prepare and cache data that will be stored to the GPU memory for computation. 8 GB of system memory could be enough depending on scene. A small scene also could fit in 2GB or 4GB of GPU memory bu for lightmapping it is recommended to have 6GB or more !

In order to get proper answer you may need to share some errors that unity throws in the console !

 

There are no errors, it always crashes with maps bigger than 512x. I didn't check logs.

Will it ever be able to get around this memory limitation? Other GPU lightmappers do.

 

I successfully have been baking small scenes on GTX 750Ti with 2 GB of memory before even at 2K resolution !
Your hardware looks capable and when Unity crashes then the problem might be GPU drivers or something else !
In case you are out of memory Unity will throw an error and not crash. It always has been notifying me when the PC is running out of memory !

What version of Unity you are using. Could you try on some newer version and see if this is the case, because i've never experienced crashes on GTX 750 Ti and GTX 1060 whatever version i am using !

P.S. - there is also one folder that caches GPU kernels. You may want to try clean up that folder as the system may use some older kernel. Get back to the thread and searched for cached kernels folder !

 

It's been crashing in all versions in the past 6 months I think. Also it seem to have gotten worse and crashes with smaller scenes lately, not sure.

Is that what you mean regarding kernel:

So it's probably not that if its supposed to be fixed.

 

So if you are on Windows PC, go to this folder ->
C:\Users\...your user name...\AppData\Roaming\NVIDIA\ComputeCache
and delete all ( or back it up if you want ) and restart and try again !

 

Thanks. Didn't work unfortunately. I restarted Unity after deleting the folder, idk if you meant the PC instead.

 

The gpu plm didn't work until I removed all probes from the scene.. including the reflection probe.. strange.
Unity 2019.2.8f
Default HDRP scene.
750ti with 2GB vram.

 

Hey everyone,

I have been struggling with this problem since the inception of GPU lightmapping in Unity. Today we solved it. Although this might not be a practical solution, I can tell you that it's the only one we found and I have been making crazy good lightmaps since.

I replaced my 1080 Ti with an AMD Radeon RX 580. This is the only component I have changed. We tried multiple AMD and Nvidia GPUs and this is the ONLY ONE that works.

The CPU is a 7th generation Intel i7 7700K. DDR4-2200 RAM.

All of my problems are gone.

 

Will there ever be a proper fix to the terrain baking issue?

GPU lightmapper will ran out of memory always when baking even some arbitrary medium size terrain. E.g. 1.5x1.5km.
I get it that 1050TI might not cut it anymore, but cmon. Shouldn't it be optimized to bake large objects slower but consume less VRAM?

Terrain only makes GPU lightmapper completely unusable for me. Please consider fixing this at some closer point in time.

 

Windows 10 blocked unity 19.2 for using the GPU while trying to bake a small scene, notification that i see for first time and then the editor stopped hanged !

 

Related to the post above, i reinstalled Windows recently and i saw the option to install "Studio Drivers". I am currently having them installed ?

However is any of both drivers packs recommended or both should work because Unity hangs even if lightmapping option is disabled and only bake the environment lighting ?

P.S. the GPU is working in UE4 and even works with the realtime RayTracer. May be you would like to test these drivers because are recommended for users that are not playing games but are using the GPU for working in general !

Thanks !



Unity stuck right away after start baking !

 

Well, lower memory consumption will not matter. Unless actual algorithm will change.

Right now I've got estimated 12GB VRAM scene in size, so there's no way Progressive GPU will handle that in a single chunk, with lower consumtion or not.

Either Progressive should have chunked baking, or its a complete bust for me.