Build times of same scene random between 5min and 5hrs! Why?

In the same scene I running into build issues. This same scene sometimes builds in 5 minutes, but sometimes it takes forever (hours and hours) compiling the shader variants. And each time I clean the previous build.
I've disabled any GPU instancing on materials, but still there's a ton of compiling going on.
The thing also is I see Unity compiling xxxx/shadername - Forward:....
My scene is setup for Deferred rendering and not Forward, so why compile these?

Anyway, these extreme build time are simply unworkable. Does anyone have any smart solution to drastically reduce the build times?

Cheers,
Marco
20291.4.1, HDRP 7.4.1