Resolved Progressive GPU not working for my project on 2020.3.7

UPDATE: I updated the drivers for the GPU and it works now, if you're having the same problem try updating your drivers. If you have the newest drivers available, try using Studio Drivers for NVIDIA GPUs.

I've been trying to bake lighting on my project but the Progressive GPU method gives me the errors below.
My CPU (3500x) is considerably weaker than my GPU, whenever it falls back to Progressive CPU it says it can take days to bake.

I've manage to bake lighting on the new default HDRP scene so I tried copying the lighting settings from that scene onto mine, yet it still gives me the running out of memory error among others.
I'm trying to bake around 12 lights in Mixed mode.

I've lowered the lightmap size to 512 and have lowered the texels to 20 per unit, I've basically lowered everything to ensure fast but low quality bake but it just gives me new errors:

OpenCL Error. Falling back to CPU lightmapper. Error callback from context: Unknown error executing clFlush on GeForce RTX 3070 (Device 0).

OpenCL Error. Falling back to CPU lightmapper. Error callback from context: CL_UNKNOWN_ERROR

OpenCL Error. Falling back to CPU lightmapper. Error callback from context: CL_OUT_OF_RESOURCES error executing CL_COMMAND_WRITE_BUFFER on GeForce RTX 3070 (Device 0).​

This error happens when my lightmap settings are 1024 for Lightmap size and 40 texels per unity. It requests 2.06GB out of the 2.00GB allocated.

OpenCL Error. Falling back to CPU lightmapper. Error callback from context: CL_MEM_OBJECT_ALLOCATION_FAILURE


OpenCL Error. Falling back to CPU lightmapper. Error callback from context: CL_MEM_OBJECT_ALLOCATION_FAILURE error executing CL_COMMAND_NDRANGE_KERNEL on GeForce RTX 3070 (Device 0).​

I can't really wait days every time I want bake something, especially since I don't how it'll turn out on my scene, I would have to keep iterating and it would take me weeks from my time...

What can I do? I'm not sure why this project doesn't let me bake light with GPU yet a new project lets me bake the default HDRP scene.