Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Join us on Dec 8, 2022, between 7 am & 7 pm EST, in the DOTS Dev Blitz Day 2022 - Q&A forum, Discord, and Unity3D Subreddit to learn more about DOTS directly from the Unity Developers.
    Dismiss Notice
  3. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice

New shader preprocessor

Discussion in '2020.1 Beta' started by aleksandrk, Dec 8, 2019.

  1. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    1,382
    @aleksandrk I just wanted to thank you and anyone else who worked on this feature, it has really sped up my iteration times when making builds of my app. It really means a lot that you listened to the feedback also and fixed the small issues I had!
     
    aleksandrk likes this.
  2. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    Thank you! :)
     
  3. zezba9000

    zezba9000

    Joined:
    Sep 28, 2010
    Posts:
    906
    Can the new "shader preprocessor" handle scripting define symbols?

    I'm developing a custom SRP that needs to support multiple differed normal-map texture formats that in turn pack or don't pack normals depending on your platform target etc.

    To remove complexity I really would like to use define custom shader preprocessors but can't get it to work and thought this is what the new feature was?

    upload_2020-7-20_2-47-40.png
     

    Attached Files:

    Arycama likes this.
  4. zezba9000

    zezba9000

    Joined:
    Sep 28, 2010
    Posts:
    906
    If this is not possible I can just use a .hlsl include file that manually must be updated which uses #define as a work-around but its not convenient.
     
  5. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    @zezba9000 Scripting define symbols are, well, for scripting, so they only exist for C# code, not shaders.
    Currently there's no way to pass custom defines from C# to the shader compiler.
     
  6. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    134
    zezba9000 likes this.
  7. zezba9000

    zezba9000

    Joined:
    Sep 28, 2010
    Posts:
    906
    Thanks that answers my question. I always thought variants were for internal use only and I couldn't customize it. Ugg how did I not know this lol
     
  8. voxellstudios

    voxellstudios

    Joined:
    Oct 8, 2019
    Posts:
    41
    @aleksandrk Hi there, not sure if I missed any details but I can't seem to use #pragma once in compute shaders in 2021.1.0f1...
     
  9. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
  10. voxellstudios

    voxellstudios

    Joined:
    Oct 8, 2019
    Posts:
    41
    Hi @aleksandrk thanks for the quick response! It seems that it did not compile normally and says that it is ignored?
     

    Attached Files:

  11. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    Yes, that's this exact issue. A fix will be available in 2021.1.2f1.
     
    voxellstudios likes this.
  12. voxellstudios

    voxellstudios

    Joined:
    Oct 8, 2019
    Posts:
    41
    where can i get the version or is it not out yet? will the alpha version have this fixed? (2021.2.0a11)
     
  13. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    It should be available soonish - it's the next version.
    Yes, alpha has this fixed as well, in 2021.2.0a12 (probably not out yet either).
     
    voxellstudios likes this.
  14. Arycama

    Arycama

    Joined:
    May 25, 2014
    Posts:
    179
    #include_with_pragmas is very useful. I'm planning to switch many of my shaders to include some common multi compiles such as shadows, volumetric lights, etc, in a single file and include them amongst several shaders, so I don't have to copy/paste 10 or so multi_compile lines between different shaders.

    Theres a few other macro features which would be very useful if they can be included, eg:

    #if UNITY_EDITOR

    Combined with #include_with_pragmas, this would be extremely useful for having lots of editor-only variants for debugging, but then swapping them out with hardcoded #defines in builds to keep variants down.

    As an example, I may wish to have several effects inside many of my shaders, which I want to be able to enable/disable in editor for testing/debug purposes. However this causes a crazy amount of shader variants for complex projects, and is completely unneccessary at runtime. So you could replace all of the multi compiles with hardcoded defines in a build, for example:

    Inside each shader, #include_with_pragmas for the below file:

    Code (CSharp):
    1. #if UNITY_EDITOR
    2.     #pragma multi_compile _ VOLUMETRIC_LIGHTING_ON
    3.     #pragma multi_compile _ ATMOSPHERIC_SCATTERING_ON
    4. #else
    5.     #define VOLUMETRIC_LIGHTING_ON
    6.     #define ATMOSPHERIC_SCATTERING_ON
    7. #endif
    It would also be very useful to be able to pass defines into the shader compiler pipeline. Eg Unity has built in macros such as UNITY_NO_DXT5nm or UNITY_UV_STARTS_AT_TOP. If we could also define things on the C# side (Or maybe in player settings) it would really help.

    Lastly, is there any chance that Unity will support runtime shader compilation? I believe some AAA games re-compile their shaders after the user changes graphics settings, rather than compiling a ton of variants and including them into the build. This would open up more optimisation opportunities, especially combined with the above feature of modifying #define values through C#.

    For example you could pass a #define value from C# at runtime as the result of a graphics setting (Say, max shadow distance) and then shaders could be re-compiled with that value hardcoded/inlined into the shader, which allows more optimised shader code to be produced, as the compiler can optimise a lot more of the code ahead of time.
     
  15. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    134
    Although not as elegant and simple, you can strip unnecessary variants at build time using IPreprocessComputeShaders.OnProcessComputeShader

    But I agree and was also looking for something simpler, like an UNITY_EDITOR define.

    Another thing that I noticed, is that the editor can compile all the different variants of a (compute) shader in the background, but when it detects that a variant is required, it will compile that ahead of the others, so it can be used before all the others are completed. This is awesome for writing and debugging shaders. What would make this even more useful, is if I could make the editor not compile the unused variants at all in the editor. Currently, our master shader takes half a minute to compile all variants, even though the used variants would take less than a second.
     
  16. Arycama

    Arycama

    Joined:
    May 25, 2014
    Posts:
    179
    It's not really the same though, because I can't replace multi-compiles with defines.

    The point was, I want to be able to have lots of variants in editor, for things to switch on and off, but in the build of the game, there are things which I'm never going to need to turn off, so I want them to be always on, without unneccessarily increasing variant count.
     
  17. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    Yep, noted :)

    Yes, it would be. But then it becomes a dependency nightmare :)
    How do we know we need to recompile a shader if we, for example, allow to pass random defines from C#?
    Player settings could work.
    Although, right now there's nothing preventing you from having your own include file with all those defines. You could even modify it using a script :)

    There is a chance for that, but there's a couple of things to consider.
    First, this just won't work on all platforms. There are some that do not support generating/recompiling shaders at runtime.
    Then, if we were to have a compiler at runtime, this would increase the final binary size. Significantly. With many developers striving to have as little applications as possible, it's a no-go.

    Noted, we'll discuss this :)
     
    Arycama and DrummerB like this.
  18. Hyp-X

    Hyp-X

    Joined:
    Jun 24, 2015
    Posts:
    421
    We use
    shader_feature
    for editor only variations and make sure to never reference those in materials.
    You might want to go reverse then like:

    Code (CSharp):
    1. #pragma shader_feature _ DISABLE_VOLUMETRIC_LIGHTING
     
    Arycama likes this.
  19. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    134
    @aleksandrk Do other platforms always use IEEE 754 floating point math? The documentation seems to suggest that they do.
    Does this mean that D3D10_SHADER_IEEE_STRICTNESS is enabled?

    In a quick experiment, the last claim (same results as on CPU) does not seem to hold. In a compute shader running for a 500x500x500 3D texture, I set all voxels inside an ellipsoid to white and outside to black. Using the same formula, types and input parameters on the CPU produces different results. About 5 voxels on the ellipsoid surface are classified differently (out of 125M voxels total). I suspect that the float division is to blame, because using a sphere formula without division produces exactly the same results on the GPU and CPU.

    This is an issue, because it leads to data desynchronization on GPU vs CPU or between multiple instances running on different GPUs. I almost have a repro done for a bug report.
     
  20. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    There's... room for interpretation here :)
    It doesn't say they do.
    No, it's not enabled.
     
  21. Arycama

    Arycama

    Joined:
    May 25, 2014
    Posts:
    179
    Being able to specify Shader defines in player settings would be really useful, and per-platform/build target as well, along with an API to set this from code. (Even though it would have to be editor only)

    I understand this won't work on all platforms, but it doesn't need to. For consoles, we are working specific hardware and know exactly what we need to optimise for. (You generally can't change graphics settings on console either) However for PC, it would be much more useful, and we know players love to tweak their graphics settings there.

    As for build-size, how significant are we talking? PC/Console games are dozens of gigabytes on average. For graphically complex games, the size of compiled shaders can be quite large. Especially when they need to handle every combination of graphics settings, and quality levels, etc. If it is a possiblity, it seems like the logical option would be to add this as a player setting, (Enable runtime shader compilation y/n) and let the devleopers decide for themselves.

    As this is a fairly advanced use case, the average developer isn't likely to need it, but there are plenty of advanced graphics programmers using Unity, and I can think of many situations where this kind of thing is useful not only for including less shader variants in the build, and reducing build times significantly, but also improving runtime shader performance, as hardcoding numerical values into a shader is always going to be faster than reading them from memory, and this also allows the compiler to make more optimisations. (A simple example is the filter-radius for shadowmap PCF, in a PC game this may be in the graphics settings under "Shadow Quality". But if we have to have three different shader variants for three different filter radii, then we're talking up to 8x the amount of shader variants for each shadow-receiving shader)

    Several Unity versions ago, it was possible to create Shaders at runtime by passing in a string containing shader source code, so it was definitely possible at some point, but was removed without a clear reason. If it was purely removed due to build-sizes, then just give developers the choice. With PC and Console games approaching 100GB in file size, is a shader compiler really going to be the thing that makes it "too big"?
     
    Last edited: May 1, 2021
  22. Arycama

    Arycama

    Joined:
    May 25, 2014
    Posts:
    179
    This is an interesting approach, and I can understand why it would work, but this is obviously still a hack, and there's the chance that an option will unintentionally be enabled resulting in that variant being included in the build.

    It would be much better to have a proper solution by Unity without the potential for unintended side effects.

    But thanks for the tip either way, I'll try it out!
     
  23. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    Yes.

    This is about mobile, of course, not PC :)

    No, as far as I know, the reason was different.

    I fully understand the usefulness of being able to optimise shaders for the particular setup and settings. But I won't promise anything at this point :)
     
    Arycama likes this.
  24. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    134
    If you're considering UNITY_EDITOR at some point, maybe it would also be worth adding a DEVELOPMENT_BUILD and DEBUG flag too, if possible. This would allow us to include debug version of our shaders in dev builds, but not in release. This would be especially useful if something like this worked:
    Code (CSharp):
    1. #if DEVELOPMENT_BUILD
    2. #pragma enable_d3d11_debug_symbols
    3. #endif
     
  25. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    Definitely.
    But this will likely mean that we'll need to reimport at least some shaders if this settings changes.
     
  26. Alekxss

    Alekxss

    Joined:
    Mar 25, 2013
    Posts:
    168
    Hello! Can you help here a little? :) I have some problems in 2021.3.3 with own shaders which have nested include files, when im include something which include something

    Shader error in 'Custom/Skybox': unrecognized identifier 'LIGHT_UNIFORMS' at Assets/Shaders/skybox.cginc(27) (on d3d11)



    It works great in Unity 2018, and it will work if add all code in first level included shader, but its not handy for me, as in main project i have many shaders which handy to share stuff.

    import this package to 2021.3.3
    https://drive.google.com/file/d/1s1lrfssRGIBd9pbAT4JPuHfrbHH79S6r/view?usp=sharing

    and you will see errors on import, or select material



    Thanks in advance
     
  27. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    Hi @Alekxss
    Looks like it doesn't like the fact that there's no space between the macro name and the '\'.
    So,
    Code (CSharp):
    1. #define FOO\
    2. float
    doesn't work, but
    Code (CSharp):
    1. #define FOO \
    2. float
    works.
    I'll make a note to myself to fix it.
     
    Alekxss likes this.
  28. DrSharky

    DrSharky

    Joined:
    Dec 7, 2016
    Posts:
    17
    Is this a feature that only works for compute shaders?
     
  29. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    No, it works for all shader types.
     
  30. voxellstudios

    voxellstudios

    Joined:
    Oct 8, 2019
    Posts:
    41
    Hi there, I still encounter the same error in 2021.3.10f1, it says it's fixed in 2021.2.x
    Is this #pragma once applicable to compute shader and hlsl files? For context, I am using it in a compute shader file:

    Code (CSharp):
    1. #pragma once
    2.  
    3. #include "Packages/23Parallel/TwoThree.Parallel.Compute/HLSL/Mathx.hlsl"
    4. #include "Packages/23Physics/TwoThree.Physics.Algorithm/Computes/ParticleSolver/ParticleSolver.hlsl"
    5.  
    6. ... StructuredBuffers here
    7.  
    8. #pragma kernel InitColorPalette
    9. [numthreads(64, 1, 1)]
    10. void InitColorPalette(uint DTid : SV_DISPATCHTHREADID)
    11. {
    12.     ... code here
    13. }
    14.  
    in Mathx.hlsl there is no #includes, but in ParticleSolver.hlsl, I #include Mathx.hlsl

    *edit, same warning, it's not an error
     
  31. voxellstudios

    voxellstudios

    Joined:
    Oct 8, 2019
    Posts:
    41
  32. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    @voxellstudios this directive is for include files, not for the main shader source.
     
  33. voxellstudios

    voxellstudios

    Joined:
    Oct 8, 2019
    Posts:
    41
    I see, then where should I put #pragma once in this case then?

    Edit: looks like it works if I put #pragma once at both the hlsl files
    Thanks @aleksandrk !!
     
    Last edited: Oct 25, 2022
    aleksandrk likes this.
  34. Zergling103

    Zergling103

    Joined:
    Aug 16, 2011
    Posts:
    369
    I found a way to replace UnityCG.cginc with custom code so that it affects ALL shaders that use the code. For example, if you want a custom fog formulation that affects all built-in shaders and third party shaders that use UnityCG.cginc, you can do this:

    - Copy the original UnityCG.cginc your project uses (it is in the editor files)
    - Paste it beside the Assets folder.
    - Make your changes.
    - Close unity.
    - Delete Library/ShaderCache
    - Re-open.

    After all shaders will be affected by your custom UnityCG.cginc.

    However, is there away to avoid closing and reopening, deleting stuff, etc.? Ideally Unity should just "know" to recompile when it detects you pasted or edited your custom replacement file. But a single button click that clears the cache and throws out any compiled shaders in memory, forcing an on-demand recompile of shaders currently being rendered would be extremely handy.
     
  35. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    It does, but only for the files under the "Assets" folder.
     
  36. Zergling103

    Zergling103

    Joined:
    Aug 16, 2011
    Posts:
    369
    Suppose I did this trick, but the UnityGC.gcinc replacement file beside the Assets folder simply redirected to Assets/TheRealUnityGC.gcinc, in other words UnityGC.gcinc only contained #include "Assets/TheRealUnityGC.gcinc"

    Would that allow changes made to TheRealUnityGC.gcinc to propigate to all shaders that use UnityGC.gcinc including builtin ones without needing to restart Unity as one currently needs to?
     
  37. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    No. As soon as any part of the include file dependency chain is outside of the Assets folder, this won't work.
     
  38. Zergling103

    Zergling103

    Joined:
    Aug 16, 2011
    Posts:
    369
    That sucks.Any chance on getting this fixed? Possibilities:
    • Shaders look for include files in these directories in this order:
      1. Path relative to your .shader file.
      2. Absolute path, starting from your project directory (the directiory containing the "Assets" folder).
      3. Your editor's install path.
      Thus, one could add another folder in this list that is included in the Assets folder, or in another folder (like ProjectSettings) that is considered for dependency chain searching. Examples:
      • Project/Assets
      • Project/Assets/ShaderIncludes
      • Project/ShaderIncludes
    • The file dependency chain is allowed to be followed outside of the Assets folder when an asset IN the Assets folder references it. (These files should ideally be also shown in the project view.)
    • An extension of Edit > Graphics > Built-in Shader Settings, which allows one to specify what #include files to provide substitutes for.
    • Moving all built-in shaders into a packages folder that is included with projects using the built-in render pipeline. This will allow one to move the package contents into the /Project/Packages folder where they can be edited. (To me this makes the most sense. Why are they in the Editor folder anyway?)
     
  39. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    Include files are assets, so Asset database would need to support this.
    You can send us a feature request, we'll figure out, how to provide this functionality properly.
     
  40. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    2,601
    Actually... Can you try updating this file outside of Assets and changing an include file that is under Assets folder? Changing like adding a space at the end. Maybe this will work :)
     
  41. Zergling103

    Zergling103

    Joined:
    Aug 16, 2011
    Posts:
    369
    Alright, I'll set up a demo project to see if I can get it working. Would be nice! I'll let you know.
     
  42. Zergling103

    Zergling103

    Joined:
    Aug 16, 2011
    Posts:
    369
    While I'm not sure I understood exactly what you wanted, but there was no luck with what I tried:
    • Two files named UnityCG.cginc, one IN Assets, one BESIDE Assets. The latter one contains only #include "Assets/UnityCG.cginc". Edited both with spaces, then reimported the former.
    • Two files like before, but both contained the entire UnityCG.cginc source. Edited both with spaces, reimported.
    I changed it so the fog lerp function uses 1 instead of "FogFac" to make fog appear 100% thick everywhere. No visible change, no evidence that it detected the change like pauses or loading bars.
     
    Last edited: Nov 26, 2022
  43. Zergling103

    Zergling103

    Joined:
    Aug 16, 2011
    Posts:
    369
    Lel, I managed to make Unity draw EVERYTHING pink by putting an error in the Project/UnityCG.cginc file, deleting cache, and restarting, lel.

    I'm trying another way. Just FindObjectsOfType<Shader>() and reimport them all in an editor thingy
     
  44. Zergling103

    Zergling103

    Joined:
    Aug 16, 2011
    Posts:
    369
    Hold on, I got it to recompile Legacy Shaders/Diffuse by deleting Assets/UnityCG.cginc...
     
  45. Zergling103

    Zergling103

    Joined:
    Aug 16, 2011
    Posts:
    369
    Hmm, I can get all the shaders showing using Resources.FindObjectsOfTypeAll, but then there's no command to tell unity to reimport them.
    Always gotta be some arbitrary wall set up to keep one from doing something simple it seems.