Search Unity

  1. Get the latest news, tutorials and offers directly to your inbox with our newsletters. Sign up now.
    Dismiss Notice

New shader preprocessor

Discussion in '2020.1 Beta' started by aleksandrk, Dec 8, 2019.

  1. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,130
    @aleksandrk I just wanted to thank you and anyone else who worked on this feature, it has really sped up my iteration times when making builds of my app. It really means a lot that you listened to the feedback also and fixed the small issues I had!
     
    aleksandrk likes this.
  2. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,900
    Thank you! :)
     
  3. zezba9000

    zezba9000

    Joined:
    Sep 28, 2010
    Posts:
    800
    Can the new "shader preprocessor" handle scripting define symbols?

    I'm developing a custom SRP that needs to support multiple differed normal-map texture formats that in turn pack or don't pack normals depending on your platform target etc.

    To remove complexity I really would like to use define custom shader preprocessors but can't get it to work and thought this is what the new feature was?

    upload_2020-7-20_2-47-40.png
     

    Attached Files:

    Arycama likes this.
  4. zezba9000

    zezba9000

    Joined:
    Sep 28, 2010
    Posts:
    800
    If this is not possible I can just use a .hlsl include file that manually must be updated which uses #define as a work-around but its not convenient.
     
  5. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,900
    @zezba9000 Scripting define symbols are, well, for scripting, so they only exist for C# code, not shaders.
    Currently there's no way to pass custom defines from C# to the shader compiler.
     
  6. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    116
    zezba9000 likes this.
  7. zezba9000

    zezba9000

    Joined:
    Sep 28, 2010
    Posts:
    800
    Thanks that answers my question. I always thought variants were for internal use only and I couldn't customize it. Ugg how did I not know this lol
     
  8. VoxellStudios

    VoxellStudios

    Joined:
    Oct 8, 2019
    Posts:
    23
    @aleksandrk Hi there, not sure if I missed any details but I can't seem to use #pragma once in compute shaders in 2021.1.0f1...
     
  9. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,900
  10. VoxellStudios

    VoxellStudios

    Joined:
    Oct 8, 2019
    Posts:
    23
    Hi @aleksandrk thanks for the quick response! It seems that it did not compile normally and says that it is ignored?
     

    Attached Files:

  11. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,900
    Yes, that's this exact issue. A fix will be available in 2021.1.2f1.
     
    VoxellStudios likes this.
  12. VoxellStudios

    VoxellStudios

    Joined:
    Oct 8, 2019
    Posts:
    23
    where can i get the version or is it not out yet? will the alpha version have this fixed? (2021.2.0a11)
     
  13. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,900
    It should be available soonish - it's the next version.
    Yes, alpha has this fixed as well, in 2021.2.0a12 (probably not out yet either).
     
    VoxellStudios likes this.
  14. Arycama

    Arycama

    Joined:
    May 25, 2014
    Posts:
    161
    #include_with_pragmas is very useful. I'm planning to switch many of my shaders to include some common multi compiles such as shadows, volumetric lights, etc, in a single file and include them amongst several shaders, so I don't have to copy/paste 10 or so multi_compile lines between different shaders.

    Theres a few other macro features which would be very useful if they can be included, eg:

    #if UNITY_EDITOR

    Combined with #include_with_pragmas, this would be extremely useful for having lots of editor-only variants for debugging, but then swapping them out with hardcoded #defines in builds to keep variants down.

    As an example, I may wish to have several effects inside many of my shaders, which I want to be able to enable/disable in editor for testing/debug purposes. However this causes a crazy amount of shader variants for complex projects, and is completely unneccessary at runtime. So you could replace all of the multi compiles with hardcoded defines in a build, for example:

    Inside each shader, #include_with_pragmas for the below file:

    Code (CSharp):
    1. #if UNITY_EDITOR
    2.     #pragma multi_compile _ VOLUMETRIC_LIGHTING_ON
    3.     #pragma multi_compile _ ATMOSPHERIC_SCATTERING_ON
    4. #else
    5.     #define VOLUMETRIC_LIGHTING_ON
    6.     #define ATMOSPHERIC_SCATTERING_ON
    7. #endif
    It would also be very useful to be able to pass defines into the shader compiler pipeline. Eg Unity has built in macros such as UNITY_NO_DXT5nm or UNITY_UV_STARTS_AT_TOP. If we could also define things on the C# side (Or maybe in player settings) it would really help.

    Lastly, is there any chance that Unity will support runtime shader compilation? I believe some AAA games re-compile their shaders after the user changes graphics settings, rather than compiling a ton of variants and including them into the build. This would open up more optimisation opportunities, especially combined with the above feature of modifying #define values through C#.

    For example you could pass a #define value from C# at runtime as the result of a graphics setting (Say, max shadow distance) and then shaders could be re-compiled with that value hardcoded/inlined into the shader, which allows more optimised shader code to be produced, as the compiler can optimise a lot more of the code ahead of time.
     
  15. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    116
    Although not as elegant and simple, you can strip unnecessary variants at build time using IPreprocessComputeShaders.OnProcessComputeShader

    But I agree and was also looking for something simpler, like an UNITY_EDITOR define.

    Another thing that I noticed, is that the editor can compile all the different variants of a (compute) shader in the background, but when it detects that a variant is required, it will compile that ahead of the others, so it can be used before all the others are completed. This is awesome for writing and debugging shaders. What would make this even more useful, is if I could make the editor not compile the unused variants at all in the editor. Currently, our master shader takes half a minute to compile all variants, even though the used variants would take less than a second.
     
  16. Arycama

    Arycama

    Joined:
    May 25, 2014
    Posts:
    161
    It's not really the same though, because I can't replace multi-compiles with defines.

    The point was, I want to be able to have lots of variants in editor, for things to switch on and off, but in the build of the game, there are things which I'm never going to need to turn off, so I want them to be always on, without unneccessarily increasing variant count.
     
  17. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,900
    Yep, noted :)

    Yes, it would be. But then it becomes a dependency nightmare :)
    How do we know we need to recompile a shader if we, for example, allow to pass random defines from C#?
    Player settings could work.
    Although, right now there's nothing preventing you from having your own include file with all those defines. You could even modify it using a script :)

    There is a chance for that, but there's a couple of things to consider.
    First, this just won't work on all platforms. There are some that do not support generating/recompiling shaders at runtime.
    Then, if we were to have a compiler at runtime, this would increase the final binary size. Significantly. With many developers striving to have as little applications as possible, it's a no-go.

    Noted, we'll discuss this :)
     
    Arycama and DrummerB like this.
  18. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    391
    We use
    shader_feature
    for editor only variations and make sure to never reference those in materials.
    You might want to go reverse then like:

    Code (CSharp):
    1. #pragma shader_feature _ DISABLE_VOLUMETRIC_LIGHTING
     
    Arycama likes this.
  19. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    116
    @aleksandrk Do other platforms always use IEEE 754 floating point math? The documentation seems to suggest that they do.
    Does this mean that D3D10_SHADER_IEEE_STRICTNESS is enabled?

    In a quick experiment, the last claim (same results as on CPU) does not seem to hold. In a compute shader running for a 500x500x500 3D texture, I set all voxels inside an ellipsoid to white and outside to black. Using the same formula, types and input parameters on the CPU produces different results. About 5 voxels on the ellipsoid surface are classified differently (out of 125M voxels total). I suspect that the float division is to blame, because using a sphere formula without division produces exactly the same results on the GPU and CPU.

    This is an issue, because it leads to data desynchronization on GPU vs CPU or between multiple instances running on different GPUs. I almost have a repro done for a bug report.
     
  20. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,900
    There's... room for interpretation here :)
    It doesn't say they do.
    No, it's not enabled.
     
  21. Arycama

    Arycama

    Joined:
    May 25, 2014
    Posts:
    161
    Being able to specify Shader defines in player settings would be really useful, and per-platform/build target as well, along with an API to set this from code. (Even though it would have to be editor only)

    I understand this won't work on all platforms, but it doesn't need to. For consoles, we are working specific hardware and know exactly what we need to optimise for. (You generally can't change graphics settings on console either) However for PC, it would be much more useful, and we know players love to tweak their graphics settings there.

    As for build-size, how significant are we talking? PC/Console games are dozens of gigabytes on average. For graphically complex games, the size of compiled shaders can be quite large. Especially when they need to handle every combination of graphics settings, and quality levels, etc. If it is a possiblity, it seems like the logical option would be to add this as a player setting, (Enable runtime shader compilation y/n) and let the devleopers decide for themselves.

    As this is a fairly advanced use case, the average developer isn't likely to need it, but there are plenty of advanced graphics programmers using Unity, and I can think of many situations where this kind of thing is useful not only for including less shader variants in the build, and reducing build times significantly, but also improving runtime shader performance, as hardcoding numerical values into a shader is always going to be faster than reading them from memory, and this also allows the compiler to make more optimisations. (A simple example is the filter-radius for shadowmap PCF, in a PC game this may be in the graphics settings under "Shadow Quality". But if we have to have three different shader variants for three different filter radii, then we're talking up to 8x the amount of shader variants for each shadow-receiving shader)

    Several Unity versions ago, it was possible to create Shaders at runtime by passing in a string containing shader source code, so it was definitely possible at some point, but was removed without a clear reason. If it was purely removed due to build-sizes, then just give developers the choice. With PC and Console games approaching 100GB in file size, is a shader compiler really going to be the thing that makes it "too big"?
     
    Last edited: May 1, 2021
  22. Arycama

    Arycama

    Joined:
    May 25, 2014
    Posts:
    161
    This is an interesting approach, and I can understand why it would work, but this is obviously still a hack, and there's the chance that an option will unintentionally be enabled resulting in that variant being included in the build.

    It would be much better to have a proper solution by Unity without the potential for unintended side effects.

    But thanks for the tip either way, I'll try it out!
     
  23. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,900
    Yes.

    This is about mobile, of course, not PC :)

    No, as far as I know, the reason was different.

    I fully understand the usefulness of being able to optimise shaders for the particular setup and settings. But I won't promise anything at this point :)
     
    Arycama likes this.
  24. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    116
    If you're considering UNITY_EDITOR at some point, maybe it would also be worth adding a DEVELOPMENT_BUILD and DEBUG flag too, if possible. This would allow us to include debug version of our shaders in dev builds, but not in release. This would be especially useful if something like this worked:
    Code (CSharp):
    1. #if DEVELOPMENT_BUILD
    2. #pragma enable_d3d11_debug_symbols
    3. #endif
     
  25. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,900
    Definitely.
    But this will likely mean that we'll need to reimport at least some shaders if this settings changes.
     
unityunity