Search Unity

  1. We are migrating the Unity Forums to Unity Discussions by the end of July. Read our announcement for more information and let us know if you have any questions.
    Dismiss Notice
  2. Dismiss Notice

Feedback Ship pre-compiled Unity shader variants with Editor?

Discussion in 'Unity 6 Beta' started by Peter77, Mar 16, 2024.

  1. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,642
    I'm curious if it would be possible to include pre-compiled Unity shader variants (URP, HDRP, etc) in the Editor.

    This would mean that the shaders are only compiled on your end, helping to reduce user project build times and CO2 emissions.

    I'm asking because I've been staring at a progress bar for over an hour while URP shaders compile.
     
    Last edited: Mar 17, 2024
    XiangAloha and mahdi_jeddi like this.
  2. icauroboros

    icauroboros

    Joined:
    Apr 30, 2021
    Posts:
    177
    Shaders can be cached in theory if you targeting a specific gpu and driver. In console world it is a good solution. But on pc it's not viable due to huge number of combinations of driver and gpu.

    quoting from https://www.yosoygames.com.ar/wp/2020/08/a-little-clarification-on-modern-shader-compile-times/

    "
    Because the PSO is tied to the GPU and driver version.

    GPUs are not like CPUs. PCs for example have the x86 instruction set which both Intel and AMD must conform to and that’s why both CPUs can run the same exe without any modification.

    However GPUs are vastly different between vendors (e.g. Intel, NVIDIA, AMD) and even between models (e.g. GeForce 680 works very different from a GeForce 1080) thus they can’t run a PSO that was compiled for a different GPU.

    What you’re mentioning (compile PSOs once, ship it with the game) is what’s done in consoles (PS4. XBox, Switch) because there’s only 1 or 2 GPUs to support (the PS4 & PS4Pro, the XBox One & XBox One X, only one for the Switch)

    > because this sounds like a huge waste of energy

    Yes it is : (

    > everyone recompiling the same thing for 20minutes, this is not eco-friendly at all

    They’re not recompiling *the same thing* if they have different GPUs. But everyone with the same GPU are indeed recompiling the same thing.

    I agree it’s not eco-friendly at all

    Right now the only way to ship a PSO that works on every machine would be to buy every GPU in existence and compile them all once. This is very expensive.

    If vendors (Intel, NV, AMD) would have a tool to allow gamedevs to compile for each GPU without actually owning the device, that would be a game changer
    "
     
    Frais254 likes this.
  3. AcidArrow

    AcidArrow

    Joined:
    May 20, 2010
    Posts:
    11,988
    Do you mean on the device or in the editor? I believe the editor supported precompiled shaders back in the Unity 5 days. Then it was dropped because of... reasons?

    I think shader compilation times went up in 2022+ in general and it's a problem.
     
  4. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,642
    Thank you for the answers.

    I believe I wasn't clear. When I wrote "pre-compile", I meant whatever Unity's shader compiler generates. This way, we can skip the "Compile shader variants" step for built-in, URP, HDRP, Unity Post Processing, etc. shaders.

    upload_2024-3-16_20-9-3.png

    I believe that the compiled shader is then compiled again at runtime by the GPU driver. I don't mean the GPU driver compiled shader, but rather the Unity compiled variants.

    Basically every shader that comes from Unity could be pre-compiled.
     
    Last edited: Mar 16, 2024
    mahdi_jeddi likes this.
  5. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,062
    @Peter77 it is being considered. We're not sure this would be a better time investment than, say, reducing the number of variants one needs to compile in the first place.

    That's correct.
    These are not PSOs yet, that's the shader code that gets sent to the driver at runtime as part of PSO.

    WRT staring at the progress bar for hours, it heavily depends on the settings in the pipeline assets. AFAIK someone sat down and cleaned them up for the new URP template and it went down from hours to something around 20 minutes (building for Android).
     
    ImpossibleRobert likes this.
  6. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,642
    Reducing the number of variants across all shaders would be ideal, I guess.

    However, considering that variant stripping needs to be turned off when you enable GPU Resident Drawer, I'm not sure how far reducing variants can go.

    upload_2024-6-2_8-34-46.png

    My project is using Unity 6.0.4f1, URP 17.0.3 Forward+, and GPU Resident Drawer enabled. The URP Standard Lit shader has almost 20000 variants. It takes hours to compile, but then it proceeds with various other URP Standard shaders that take a similar amount of time.

    Which brings me to another issue I encounter where you might have a tip. Since it takes so long to make a build, I often spread the build process across several days. That means I start the build, let it work for a few hours, cancel the build, put the PC in hibernate, wake up the PC a day later, and click Build again.

    Since Unity is or was able to build incrementally, shaders that were compiled earlier didn't recompile when starting the build again. This behavior seems to be different with Unity 6.

    When I start a build in Unity 6, wait for all URP Standard Lit shader variants to be compiled, stop the build, put the PC in hibernate, wake up the PC, start a build again, Unity compiles all those 20000 URP Standard Lit shader variants again.

    I'm now stuck at not being able to build the project anymore because it takes too long. I went over the "reduce shader variants" blog/forum posts that were recently published, and it seems there is nothing I can do anymore to make that process faster.

    My question is: Why is Unity recompiling shader variants when literally nothing changed between builds? One idea that comes to mind is, if there is a shader build cache somewhere that perhaps exceeds and throws out all those compiled variants again.

    I also see that the editor is showing cyan materials for many of the same objects over and over again when I restart Unity.

    EDIT: So... several hours later the build completed. I then immediately triggered a new build, selected the same output directory, and Unity started to recompile all shader variants again. Something obviously is not be working properly in Unity 6. It seems compiled shader variations are not cached. Is anybody else having the same issue?
     
    Last edited: Jun 2, 2024
    mahdi_jeddi likes this.
  7. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,062
    I don't think it's all stripping gets turned off, though. It's only the variants for BRG - DOTS_INSTANCING_ON.

    This is the first time I hear about something like that. Can you check the Editor log where it says about variant compilation - does it say there that it recompiled the variants too?

    The shader build cache is purged based on reimports of the given shader. IIRC we keep something like 10 most recent versions.

    Hm. May I ask you to file a bug report and post the issue number here? I'd like to take a look at what's happening.
     
  8. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    6,642
    I have submitted bug-report
    IN-77989
    with a project to reproduce, that shows that shaders seem to recompile in the editor every time the editor is restarted. I suspect that it's either the same or at least a related problem that I experience with building a player.
     
  9. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    3,062
    Thank you!
    Although, if you see the cyan shader there, it's not indicative of the same problem. This simply means we didn't pull the required variant out of the cache just yet.