New shader preprocessor

@aleksandrk I just wanted to thank you and anyone else who worked on this feature, it has really sped up my iteration times when making builds of my app. It really means a lot that you listened to the feedback also and fixed the small issues I had!

 

Can the new "shader preprocessor" handle scripting define symbols?

I'm developing a custom SRP that needs to support multiple differed normal-map texture formats that in turn pack or don't pack normals depending on your platform target etc.

To remove complexity I really would like to use define custom shader preprocessors but can't get it to work and thought this is what the new feature was?

 

If this is not possible I can just use a .hlsl include file that manually must be updated which uses #define as a work-around but its not convenient.

 

@zezba9000 There are some predefined platform macros for shader code. Otherwise, wouldn't shader variants be a solution?

 

Thanks that answers my question. I always thought variants were for internal use only and I couldn't customize it. Ugg how did I not know this lol

 

@aleksandrk Hi there, not sure if I missed any details but I can't seem to use #pragma once in compute shaders in 2021.1.0f1...

 

Hi @aleksandrk thanks for the quick response! It seems that it did not compile normally and says that it is ignored?

 

where can i get the version or is it not out yet? will the alpha version have this fixed? (2021.2.0a11)

 

#include_with_pragmas is very useful. I'm planning to switch many of my shaders to include some common multi compiles such as shadows, volumetric lights, etc, in a single file and include them amongst several shaders, so I don't have to copy/paste 10 or so multi_compile lines between different shaders.

Theres a few other macro features which would be very useful if they can be included, eg:

#if UNITY_EDITOR

Combined with #include_with_pragmas, this would be extremely useful for having lots of editor-only variants for debugging, but then swapping them out with hardcoded #defines in builds to keep variants down.

As an example, I may wish to have several effects inside many of my shaders, which I want to be able to enable/disable in editor for testing/debug purposes. However this causes a crazy amount of shader variants for complex projects, and is completely unneccessary at runtime. So you could replace all of the multi compiles with hardcoded defines in a build, for example:

Inside each shader, #include_with_pragmas for the below file:

Code (CSharp):

1. #if UNITY_EDITOR
2.     #pragma multi_compile _ VOLUMETRIC_LIGHTING_ON
3.     #pragma multi_compile _ ATMOSPHERIC_SCATTERING_ON
4. #else
5.     #define VOLUMETRIC_LIGHTING_ON
6.     #define ATMOSPHERIC_SCATTERING_ON
7. #endif

It would also be very useful to be able to pass defines into the shader compiler pipeline. Eg Unity has built in macros such as UNITY_NO_DXT5nm or UNITY_UV_STARTS_AT_TOP. If we could also define things on the C# side (Or maybe in player settings) it would really help.

Lastly, is there any chance that Unity will support runtime shader compilation? I believe some AAA games re-compile their shaders after the user changes graphics settings, rather than compiling a ton of variants and including them into the build. This would open up more optimisation opportunities, especially combined with the above feature of modifying #define values through C#.

For example you could pass a #define value from C# at runtime as the result of a graphics setting (Say, max shadow distance) and then shaders could be re-compiled with that value hardcoded/inlined into the shader, which allows more optimised shader code to be produced, as the compiler can optimise a lot more of the code ahead of time.

 

Although not as elegant and simple, you can strip unnecessary variants at build time using IPreprocessComputeShaders.OnProcessComputeShader

But I agree and was also looking for something simpler, like an UNITY_EDITOR define.

Another thing that I noticed, is that the editor can compile all the different variants of a (compute) shader in the background, but when it detects that a variant is required, it will compile that ahead of the others, so it can be used before all the others are completed. This is awesome for writing and debugging shaders. What would make this even more useful, is if I could make the editor not compile the unused variants at all in the editor. Currently, our master shader takes half a minute to compile all variants, even though the used variants would take less than a second.

 

It's not really the same though, because I can't replace multi-compiles with defines.

The point was, I want to be able to have lots of variants in editor, for things to switch on and off, but in the build of the game, there are things which I'm never going to need to turn off, so I want them to be always on, without unneccessarily increasing variant count.

 

We use

shader_feature

for editor only variations and make sure to never reference those in materials.
You might want to go reverse then like:

Code (CSharp):

1. #pragma shader_feature _ DISABLE_VOLUMETRIC_LIGHTING

 

@aleksandrk Do other platforms always use IEEE 754 floating point math? The documentation seems to suggest that they do.

Does this mean that D3D10_SHADER_IEEE_STRICTNESS is enabled?

In a quick experiment, the last claim (same results as on CPU) does not seem to hold. In a compute shader running for a 500x500x500 3D texture, I set all voxels inside an ellipsoid to white and outside to black. Using the same formula, types and input parameters on the CPU produces different results. About 5 voxels on the ellipsoid surface are classified differently (out of 125M voxels total). I suspect that the float division is to blame, because using a sphere formula without division produces exactly the same results on the GPU and CPU.

This is an issue, because it leads to data desynchronization on GPU vs CPU or between multiple instances running on different GPUs. I almost have a repro done for a bug report.

 

Being able to specify Shader defines in player settings would be really useful, and per-platform/build target as well, along with an API to set this from code. (Even though it would have to be editor only)

I understand this won't work on all platforms, but it doesn't need to. For consoles, we are working specific hardware and know exactly what we need to optimise for. (You generally can't change graphics settings on console either) However for PC, it would be much more useful, and we know players love to tweak their graphics settings there.

As for build-size, how significant are we talking? PC/Console games are dozens of gigabytes on average. For graphically complex games, the size of compiled shaders can be quite large. Especially when they need to handle every combination of graphics settings, and quality levels, etc. If it is a possiblity, it seems like the logical option would be to add this as a player setting, (Enable runtime shader compilation y/n) and let the devleopers decide for themselves.

As this is a fairly advanced use case, the average developer isn't likely to need it, but there are plenty of advanced graphics programmers using Unity, and I can think of many situations where this kind of thing is useful not only for including less shader variants in the build, and reducing build times significantly, but also improving runtime shader performance, as hardcoding numerical values into a shader is always going to be faster than reading them from memory, and this also allows the compiler to make more optimisations. (A simple example is the filter-radius for shadowmap PCF, in a PC game this may be in the graphics settings under "Shadow Quality". But if we have to have three different shader variants for three different filter radii, then we're talking up to 8x the amount of shader variants for each shadow-receiving shader)

Several Unity versions ago, it was possible to create Shaders at runtime by passing in a string containing shader source code, so it was definitely possible at some point, but was removed without a clear reason. If it was purely removed due to build-sizes, then just give developers the choice. With PC and Console games approaching 100GB in file size, is a shader compiler really going to be the thing that makes it "too big"?

 

This is an interesting approach, and I can understand why it would work, but this is obviously still a hack, and there's the chance that an option will unintentionally be enabled resulting in that variant being included in the build.

It would be much better to have a proper solution by Unity without the potential for unintended side effects.

But thanks for the tip either way, I'll try it out!

 

If you're considering UNITY_EDITOR at some point, maybe it would also be worth adding a DEVELOPMENT_BUILD and DEBUG flag too, if possible. This would allow us to include debug version of our shaders in dev builds, but not in release. This would be especially useful if something like this worked:

Code (CSharp):

1. #if DEVELOPMENT_BUILD
2. #pragma enable_d3d11_debug_symbols
3. #endif

 

Hello! Can you help here a little? I have some problems in 2021.3.3 with own shaders which have nested include files, when im include something which include something

Shader error in 'Custom/Skybox': unrecognized identifier 'LIGHT_UNIFORMS' at Assets/Shaders/skybox.cginc(27) (on d3d11)



It works great in Unity 2018, and it will work if add all code in first level included shader, but its not handy for me, as in main project i have many shaders which handy to share stuff.

import this package to 2021.3.3
https://drive.google.com/file/d/1s1lrfssRGIBd9pbAT4JPuHfrbHH79S6r/view?usp=sharing

and you will see errors on import, or select material



Thanks in advance

 

Is this a feature that only works for compute shaders?

 

Hi there, I still encounter the same error in 2021.3.10f1, it says it's fixed in 2021.2.x
Is this #pragma once applicable to compute shader and hlsl files? For context, I am using it in a compute shader file:

Code (CSharp):

1. #pragma once
2.  
3. #include "Packages/23Parallel/TwoThree.Parallel.Compute/HLSL/Mathx.hlsl"
4. #include "Packages/23Physics/TwoThree.Physics.Algorithm/Computes/ParticleSolver/ParticleSolver.hlsl"
5.  
6. ... StructuredBuffers here
7.  
8. #pragma kernel InitColorPalette
9. [numthreads(64, 1, 1)]
10. void InitColorPalette(uint DTid : SV_DISPATCHTHREADID)
11. {
12.     ... code here
13. }
14.  

in Mathx.hlsl there is no #includes, but in ParticleSolver.hlsl, I #include Mathx.hlsl

*edit, same warning, it's not an error

 

I see, then where should I put #pragma once in this case then?

Edit: looks like it works if I put #pragma once at both the hlsl files
Thanks @aleksandrk !!

 

I found a way to replace UnityCG.cginc with custom code so that it affects ALL shaders that use the code. For example, if you want a custom fog formulation that affects all built-in shaders and third party shaders that use UnityCG.cginc, you can do this:

- Copy the original UnityCG.cginc your project uses (it is in the editor files)
- Paste it beside the Assets folder.
- Make your changes.
- Close unity.
- Delete Library/ShaderCache
- Re-open.

After all shaders will be affected by your custom UnityCG.cginc.

However, is there away to avoid closing and reopening, deleting stuff, etc.? Ideally Unity should just "know" to recompile when it detects you pasted or edited your custom replacement file. But a single button click that clears the cache and throws out any compiled shaders in memory, forcing an on-demand recompile of shaders currently being rendered would be extremely handy.

 

Suppose I did this trick, but the UnityGC.gcinc replacement file beside the Assets folder simply redirected to Assets/TheRealUnityGC.gcinc, in other words UnityGC.gcinc only contained #include "Assets/TheRealUnityGC.gcinc"

Would that allow changes made to TheRealUnityGC.gcinc to propigate to all shaders that use UnityGC.gcinc including builtin ones without needing to restart Unity as one currently needs to?

 

That sucks.Any chance on getting this fixed? Possibilities:

• Shaders look for include files in these directories in this order:
  1. Path relative to your .shader file.
  2. Absolute path, starting from your project directory (the directiory containing the "Assets" folder).
  3. Your editor's install path.
  Thus, one could add another folder in this list that is included in the Assets folder, or in another folder (like ProjectSettings) that is considered for dependency chain searching. Examples:
  • Project/Assets
  • Project/Assets/ShaderIncludes
  • Project/ShaderIncludes
• The file dependency chain is allowed to be followed outside of the Assets folder when an asset IN the Assets folder references it. (These files should ideally be also shown in the project view.)
• An extension of Edit > Graphics > Built-in Shader Settings, which allows one to specify what #include files to provide substitutes for.
• Moving all built-in shaders into a packages folder that is included with projects using the built-in render pipeline. This will allow one to move the package contents into the /Project/Packages folder where they can be edited. (To me this makes the most sense. Why are they in the Editor folder anyway?)

 

Alright, I'll set up a demo project to see if I can get it working. Would be nice! I'll let you know.

 

While I'm not sure I understood exactly what you wanted, but there was no luck with what I tried:

• Two files named UnityCG.cginc, one IN Assets, one BESIDE Assets. The latter one contains only #include "Assets/UnityCG.cginc". Edited both with spaces, then reimported the former.
• Two files like before, but both contained the entire UnityCG.cginc source. Edited both with spaces, reimported.

I changed it so the fog lerp function uses 1 instead of "FogFac" to make fog appear 100% thick everywhere. No visible change, no evidence that it detected the change like pauses or loading bars.

 

Lel, I managed to make Unity draw EVERYTHING pink by putting an error in the Project/UnityCG.cginc file, deleting cache, and restarting, lel.

I'm trying another way. Just FindObjectsOfType<Shader>() and reimport them all in an editor thingy

 

Hold on, I got it to recompile Legacy Shaders/Diffuse by deleting Assets/UnityCG.cginc...

 

Hmm, I can get all the shaders showing using Resources.FindObjectsOfTypeAll, but then there's no command to tell unity to reimport them.
Always gotta be some arbitrary wall set up to keep one from doing something simple it seems.