Search Unity

  1. Curious about what's going to be in 2020.1? Have a look at the 2020.1 beta blog post.
    Dismiss Notice
  2. Want to see 2020.1b in action? Sign up for our Beta 2020.1 Overview Webinar on April 20th for a live presentation from our evangelists and a Q&A session with guests from R&D.
    Dismiss Notice
  3. Interested in giving us feedback? Join our online research interviews on a broad range of topics and share your insights with us.
    Dismiss Notice
  4. New Unity Live Help updates. Check them out here!

    Dismiss Notice

New shader preprocessor

Discussion in '2020.1 Beta' started by aleksandrk, Dec 8, 2019.

  1. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    Hello Unity alpha users!

    2020.1.0a15 is out and it brings a new shader preprocessor (experimental). You can enable it by passing a command line option “-force-new-shader-pp” to the Editor executable.
    EDITED on 09.01.2020: As of 2020.1.0a18 you can enable it per project via Project Settings -> Editor Settings -> Caching Preprocessor checkbox.

    It’s going to be faster (performance improvements coming in 2020.1.0a16, shader compilation time is reduced by 5-25%), comes with new features and follows the C standard more thoroughly.

    New features

    Show preprocessed source
    Shader inspector now has a checkbox “Preprocess only” which toggles between compiling shader variants and showing preprocessed source. The checkbox is only active for the new preprocessor.

    Conditionals can now affect #pragma directives
    #pragma directive parsing is now done using the new preprocessor, which means preprocessor conditionals can be used to influence, which #pragma directives are selected.

    Restrictions:
    #pragma directives are not searched for in include files (this would increase shader compilation time significantly).

    Conditionals will affect #pragma directives if the expression depends only on:
    • Any custom #define
    • The following platform keywords: SHADER_API_MOBILE, SHADER_API_DESKTOP, UNITY_NO_RGBM, UNITY_USE_NATIVE_HDR, UNITY_FRAMEBUFFER_FETCH_AVAILABLE, UNITY_NO_CUBEMAP_ARRAY
    • Unity version macro
    The list may be expanded in the future.

    Accurate include dependency tracking
    The same rules that are used for conditionals affecting #pragma directives are now applied to include dependency tracking. This reduces the chance of shader reimport when some include file changes.

    Support for `#pragma once` directive
    Instead of writing the regular include guards
    Code (CSharp):
    1. #ifndef MY_ASWESOME_INCLUDE_FILE_INCLUDED
    2. #define MY_ASWESOME_INCLUDE_FILE_INCLUDED
    3. <actual include code here>
    4. #endif //MY_ASWESOME_INCLUDE_FILE_INCLUDED
    It’s now possible to just write
    Code (CSharp):
    1. #pragma once
    2. <actual include code here>
    Support for #pragma warning directive
    #pragma warning behaves the same as #pragma error, but issues a warning.

    New #include_with_pragmas directive (from 2020.1.0a18)
    This is a new directive introduced for the new preprocessor. It is exactly the same as the standard #include directive, but it is not ignored during shader import time, which makes it possible to put shader snippet parameters (#pragma directives) in an include file.
    This directive has to be used explicitly (e.g. if you use #include_with_pragmas inside a file included using a regular #include directive, it won't be checked).

    Differences in behaviour with the previous preprocessor
    • Octal number recognition is deferred to the compiler unless the number is part of a preprocessor conditional expression.
    • No number truncation to 64 bits happens in the preprocessor unless the number is part of a preprocessor conditional expression.
    • String with escape characters are now handled correctly.
    • #line directive can use non-10-base integers as line number
    • #line directive arguments will be macro-expanded if they don’t match <Integer> or <Integer + String> pattern
    • __FILE__ and __LINE__ macros can be used.
    • “\\\n” is not treated as a space anymore.
    • “\\\n” can now be used anywhere in the code, for example “#\\\nd\\\ne\\\nf\\\ni\\\nn\\\ne\\\n A\\\n 1\n” is equivalent to “#define A 1\n”
    • When a macro is redefined, the new preprocessor issues a warning if the definition is different and uses the last definition.
    • defined(x) can now come as a result of macro expansion
    • Macros with 0 parameters are now supported. Expamle: #define A() B
    • Self-referring macros are now handled correctly. Example: #define A 1 + A
    • Empty macro arguments are now supported. Example: #define A(x, y)... A(,)
    • Commas inside brackets (“[]”) and braces (“{}”) will now be treated as a delimiter between macro arguments.
    • Generic preprocessing numbers are now recognized as such.
    • Self-expanding macros are now handled correctly.
    • In general, macro expansion now follows the C standard.
    • Ill-formed #undef directives are no longer ignored when the first argument matches <Identifier>.
    • Full stringizing support.
    • Concatenation works within macros without arguments. Example: #define A x ## y
    • Invalid concatenation now issues a warning.
    • Macro parameters are not macro-expanded when used for concatenation or stringizing.
    • Character constants are now treated as integers.
    • No support for % operator in conditionals.
    • Shift operators are now supported in conditionals.
    • #include directive arguments are now macro-expanded if they don’t match the regular format.
    • When the new preprocessor encounters a macro name without parameters no substitution happens (previously resulted in an error).
    Restrictions (EDITED on 20.12.2019)

    The following platforms are supported starting from Unity 2020.1.0a17:
    • Xbox One
    • PS4
    The following shader types are supported starting from Unity 2020.1.0a17:
    • Compute shaders
    The following shader types are not yet supported:
    • Ray tracing shaders
    • Surface shaders
    These will gradually become available in the upcoming alpha releases.

    Thanks for reading this far :)
    Stay tuned!
     
    Last edited: Jan 9, 2020
  2. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    Greetings Unity alpha users!

    Unity 2020.1.0a17 is out, and brings Xbox Ona and PS4 support as well as Compute shader support.
    We also introduced UNITY_OLD_PREPROCESSOR macro, which is defined only for the old preprocessor.

    More to follow! :)
     
    GliderGuy likes this.
  3. Peter77

    Peter77

    QA Jesus

    Joined:
    Jun 12, 2013
    Posts:
    4,642
    Maybe also update the initial post "Restriction" section?
     
    aleksandrk likes this.
  4. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    @Peter77 updated, thanks for the suggestion!
     
    Peter77 likes this.
  5. hippocoder

    hippocoder

    Digital Ape Moderator

    Joined:
    Apr 11, 2010
    Posts:
    26,456
    Thanks for this - really exciting. Will it eventually cover performance improvements or is it mostly going to be a quality of life authoring thing?

    No complaints though, quality of life is a very nice thing in shader land :)
     
    ROBYER1 likes this.
  6. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    As it's only the preprocessor changing, the shaders should be the same, so runtime performance shouldn't be affected :)
    The new preprocessor should not be slower. I've seen anything between 1% and 25% faster import and compilation time when using the new one, but I haven't seen it perform slower than the old system. If you find a case where it is slower, please report a bug.
     
    hippocoder likes this.
  7. a436t4ataf

    a436t4ataf

    Joined:
    May 19, 2013
    Posts:
    607
    It's awesome that #pragma's will finally be conditionally compilable (although the only use case I have left, that I haven't ended up with good workarounds for, is: including multiple vert/frags within a single file).

    When you say:

    ... how bad is it, and why can't we make that decision ourselves?
     
  8. a436t4ataf

    a436t4ataf

    Joined:
    May 19, 2013
    Posts:
    607
    Also ... any chance for conditional-compile of passes? :) (currently I'm scouring the forums for best practices on ways to clean-up code that has divergent use of multiple passes, and finding only ugly hacks :(. On the plus side, it led me to this thread, and it's good to see what's coming in 2020.1, even if I won't be able to use it for another year or two once everyone else has upgraded)
     
  9. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    58
    Great news!
    Could you elaborate on this one? Was this not possible before? For instance this works in 2019.2:
    Code (CSharp):
    1. #define DEBUG defined(DEBUG_MODE) && !defined(RENDERING_CAMERA)
    2. // and later:
    3. #if DEBUG
    4. // code
    5. #endif
    Do you have any suggestions on how to toggle different debug modes of a compute shader from script?

    In a normal shader I'm able to define keywords and then toggle that keyword on a Material by script. Something similar would be super useful for compute shaders as well. Currently I have to enable debug modes by uncommenting #defines in the shader. I can use bool properties, but that's bad for performance and isn't compiled out of the shader when not needed.
     
  10. DrummerB

    DrummerB

    Joined:
    Dec 19, 2013
    Posts:
    58
    And just after posting this I've seen that multi compile for compute shaders is coming in 2020.1 as well! Compute shaders will be even more powerful! Thanks a lot for working on this!
     
  11. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    something like
    Code (CSharp):
    1. #if SHADER_API_DESKTOP
    2. #pragma vertex DesktopVert
    3. #else
    4. #pragma vertex MobileVert
    5. #endif
    should work now :)

    too bad to allow this directly (several times longer shader compilation)

    we're rolling the feature out step by step, just wait a bit ;)

    the change would be too drastic, so unlikely

    some more advanced macros did not work properly with this, can't say from the top of my head, what exactly didn't work.
     
    DrummerB and a436t4ataf like this.
  12. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    Happy New Year everyone!

    2020.1.0a18 is out and it bring two more updates:
    • New preprocessor can now be enabled/disabled per project (Project Settings -> Editor Settings -> Caching Preprocessor checkbox)
    • #include_with_pragmas directive (@a436t4ataf this is the thing you were asking for ;))
    I updated the original post with details.
    Stay tuned!
     
    Lars-Steenhoff and a436t4ataf like this.
  13. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    749
    Will it be on by default in a new 2020 version?
     
  14. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    @ROBYER1 the plan is to make it default during the 2020 cycle, most likely during the 2020.2 alpha.
     
  15. Malmer

    Malmer

    Joined:
    Nov 10, 2013
    Posts:
    18
    It would be super useful if the new preprocessor also worked outside of the HLSL blocks. Primarily in the Properties block, but also while setting up render states. Using an include and not having to constantly copy-paste all those shared properties would be a huge improvement.
     
    Peter77 likes this.
  16. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    Well, one doesn't need a preprocessor for that. Just some additional syntax to allow such behaviour :)
     
  17. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    749
    @aleksandrk This preprocessor (on 2020.1.0a24) seems to fall over looking for Xbox Shaders and Blit shaders? This happened when I activated the preprocessor and reopened my project. It has 2 shader graphs in. I can try to reproduce on a smaller repro but I was using Universal RP at this commit https://github.com/Unity-Technologi...mmit/1799d2ceeba0a7ea257eb6cfa6335651e489a3c8

    Shader error in 'Hidden/Universal Render Pipeline/Blit': Couldn't open include file 'Packages/com.unity.render-pipelines.xboxone/ShaderLibrary/API/XBoxOne.hlsl'. at Common.hlsl(159)


    Shader error in 'Universal Render Pipeline/Lit': undeclared identifier 'unity_Builtins0' at SpaceTransforms.hlsl(12) (on d3d11)


    Shader error in 'Universal Render Pipeline/Nature/SpeedTree8': undeclared identifier 'unity_Builtins1' at SpaceTransforms.hlsl(12) (on d3d11)


    Shader error in 'Graphs/MeshRenderer/MeshRenderer-ColorRim': undeclared identifier 'unity_Builtins0' at SpaceTransforms.hlsl(12) (on d3d11)


    It all goes away if I disable that new preprocessor and hit play. Also causes shadergraph to be unusable while erroring out before I disable the preprocessor.
     
    Last edited: Feb 28, 2020
  18. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    @ROBYER1 a fix is coming for the "Can't open include file" issue, I'll post here whan it's available in 2020.1 :)
    If you can make a small repro for the others, would be great!
     
  19. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    @ROBYER1 a fix for "Couldn't open include file" will be available in 2020.1.0b2.
     
  20. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    749
    Thankyou for this, I will get round to submitting a bug report for the other shader issues next week. Have had a lot of shader unrelated XR bugs to report during our development!

    Any word on when 2020.1.0b2 will hit?
     
  21. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    Can't promise anything, unfortunately. This depends on many factors, and a specific version can get delayed if e.g. a critical bug is found.
     
  22. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    749
    Reported at (Case 1226235) [URP][ShaderPreProcessor]Using new shader preprocessor on 2020.1.0a25, shader errors for Shader graphs and Speedtree shaders

    As it's a very simple test scene using the Unity Sample Assets which breaks with the preprocessor, I would urge you to test this with different shader graphs in URP/HDRP. a repository like this has great examples of what breaks with the new preprocessor in it. https://github.com/UnityTechnologies/ShaderGraph_ExampleLibrary

    Seeing as it's focused on shader compilation, is there an internal Unity Shader Graph examples project/scene with all possible variations or at least all shader graph nodes used within?

    I would also like to have such a scene for our own tests, we are forced to stay up to date with the latest URP/Shader graph versions within Unity Alpha 2020 until the current issues with Oculus Quest/Go and XR are fixed for us.
     
  23. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    Great, thanks a lot!

    Neat, didn't know about it :)
    I will definitely test with those!

    I'm not sure, but it seems there's a test project in SRP repo: https://github.com/Unity-Technologies/ScriptableRenderPipeline/tree/master/TestProjects
     
    ROBYER1 likes this.
  24. laurentlavigne

    laurentlavigne

    Joined:
    Aug 16, 2012
    Posts:
    2,502
    could this be linked to upgrade project that crash unity?
     
  25. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    @laurentlavigne it's disabled by default, so it shouldn't be the case.
    What's in the log before the crash?
     
  26. laurentlavigne

    laurentlavigne

    Joined:
    Aug 16, 2012
    Posts:
    2,502
    the import window is modal so no access to the console but one of the QA person found that repeatedly happening then crash "A polygon of stw_spine_weight_2_jnt_proxy_geo is self-intersecting and has been discarded."
    Note: import worked before beta 1 and before the upgrade to the latest HDRP
     
  27. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    well, as I said, the feature is in opt-in mode, so you have to manually enable it
    you don't need any window to access the log, here's how to find it: https://docs.unity3d.com/Manual/LogFiles.html
     
  28. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    749
    These errors have not returned in 2020.1.0b3!
     
    aleksandrk likes this.
  29. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    749
    I now get the following string of errors in a very simple test scene that has a tree, a shader graph of water and some UI as well as the Unity Sample Scene assets. Built using OpenGLES, IL2CPP, Universal RP, Shader Graph and running in XR on Oculus Quest.

    Will add the bug case back here soon: (Case 1231169) [XR][URP] New Shader Preprocessor creates incompatible shaders for the Oculus Quest in URP

    Code (CSharp):
    1. 2020/03/27 12:06:12.255 6311 6327 Debug Unity WARNING: Shader
    2. 2020/03/27 12:06:12.255 6311 6327 Debug Unity Unsupported: 'Hidden/Nature/Terrain/Utilities' - All passes removed
    3. 2020/03/27 12:06:12.255 6311 6327 Debug Unity WARNING: Shader
    4. 2020/03/27 12:06:12.256 6311 6327 Debug Unity Did you use #pragma only_renderers and omit this platform?
    5. 2020/03/27 12:06:12.256 6311 6327 Debug Unity ERROR: Shader
    6. 2020/03/27 12:06:12.256 6311 6327 Debug Unity Shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
    7. 2020/03/27 12:06:12.256 6311 6327 Debug Unity WARNING: Shader
    8. 2020/03/27 12:06:12.256 6311 6327 Debug Unity Unsupported: 'Hidden/Nature/Terrain/Utilities' - Setting to default shader.
    9. 2020/03/27 12:06:12.362 6311 6327 Debug Unity WARNING: Shader
    10. 2020/03/27 12:06:12.362 6311 6327 Debug Unity Unsupported: 'Hidden/Universal Render Pipeline/Terrain/Lit (Basemap Gen)' - Pass '' has no vertex shader
    11. 2020/03/27 12:06:12.362 6311 6327 Debug Unity ERROR: Shader
    12. 2020/03/27 12:06:12.362 6311 6327 Debug Unity Shader is not supported on this GPU (none of subshaders/fallbacks are suitable)
    13. 2020/03/27 12:06:12.362 6311 6327 Debug Unity WARNING: Shader
    14. 2020/03/27 12:06:12.362 6311 6327 Debug Unity Unsupported: 'Hidden/Universal Render Pipeline/Terrain/Lit (Basemap Gen)' - Setting to default shader.
    15. 2020/03/27 12:06:12.490 6311 6327 Debug Unity WARNING: Shader
    16. 2020/03/27 12:06:12.490 6311 6327 Debug Unity Unsupported: 'Universal Render Pipeline/Lit' - Pass 'ForwardLit' has no vertex shader
    17. 2020/03/27 12:06:12.490 6311 6327 Debug Unity WARNING: Shader
    18. 2020/03/27 12:06:12.490 6311 6327 Debug Unity Unsupported: 'Universal Render Pipeline/Lit' - Pass 'ShadowCaster' has no vertex shader
    19. 2020/03/27 12:06:12.490 6311 6327 Debug Unity WARNING: Shader
    20. 2020/03/27 12:06:12.490 6311 6327 Debug Unity Unsupported: 'Universal Render Pipeline/Lit' - Pass 'DepthOnly' has no vertex shader
    21. 2020/03/27 12:06:12.490 6311 6327 Debug Unity WARNING: Shader
    22. 2020/03/27 12:06:12.490 6311 6327 Debug Unity Unsupported: 'Universal Render Pipeline/Lit' - Pass 'Universal2D' has no vertex shader
    23. 2020/03/27 12:06:12.490 6311 6327 Debug Unity WARNING: Shader
    24. 2020/03/27 12:06:12.490 6311 6327 Debug Unity Unsupported: 'Universal Render Pipeline/Lit' - All passes removed
    25. 2020/03/27 12:06:12.490 6311 6327 Debug Unity WARNING: Shader
    26. 2020/03/27 12:06:12.491 6311 6327 Debug Unity Did you use #pragma only_renderers and omit this platform?
    27.  
     
    Last edited: Mar 27, 2020
  30. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    Do these errors disappear when you turn off the Caching shader preprocessor?
     
  31. ROBYER1

    ROBYER1

    Joined:
    Oct 9, 2015
    Posts:
    749
    Yes, but then currently on 2020.1.0b3 using OpenGLES on Oculus Quest, nothing renders but the UI and we get constant errors I have reported in (Case 1230906) [XR][URP]OpenGLES 3.0 runs OPENGL NATIVE PLUG-IN ERRORS in build on Oculus Quest

    (same scene very simple only a canvas UI, a speedtree, shader graph of water and sample scene assets)

    New report for the preprocessor errors: (Case 1231169) [XR][URP] New Shader Preprocessor creates incompatible shaders for the Oculus Quest in URP
     
    aleksandrk likes this.
  32. aleksandrk

    aleksandrk

    Unity Technologies

    Joined:
    Jul 3, 2017
    Posts:
    1,248
    Thanks a lot! I'll take a look next week :)
     
unityunity