Spikes caused by compiling shader variants that should have been warmed up in a ShaderVariollection

I have an issue where I get lots of spikes as a result of shader compiling. I do have shadervariantcollections for all the shaders in a specific scene, and it is warmed up at loading time for that scene. When I enable the Graphics setting to show shader compilation, I can clearly see that the cause of the spike is a variant compiling, BUT that shader variant have actually already been added to the variantcollection. It looks like it does not warm everything up maybe? Is there a known issue around this or a workaround?

One suggestion was that shaders in assetbundles cannot also be in shadervariantcollections because their source is not added to the build. How can I ensure that the shaders do not get added to assetbundles? I add scenes to the assetbundles, then I add all the shaders to their own assetbundle. Is there a way to say "dont include these shaders"? Or to force include the source for them?

 

I also have similar issue