Search Unity

Unity Unity is adding a new "DXC" HLSL compiler backend option

Discussion in 'Shaders' started by Aras, Apr 2, 2021.

  1. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,748
    Hey! So Unity has been using Microsoft's "FXC" HLSL compiler (plus our own HLSLcc when targeting Vulkan/Metal/OpenGL) for years. However FXC has a bunch of issues, like no support for some modern GPU features, extremely slow compile times for some shaders, and so on.

    Over the past year we've been chipping away at integrating Microsoft's "new" HLSL compiler, called "DXC". Right now in 2021.2 alpha builds it kinda mostly works, with a bunch of caveats and gotchas. On the plus side, it does support modern GPU features (like wave/simd-group functions), and overall is about 4x faster at compiling shaders than the old compiler was.

    If you have a bunch of shader code written or are just interested in playing around with it, we'd love your feedback! The new compiler is completely opt-in per-shader.

    Here's a long google doc: DXC shader compiler integration, 2021.2 - with instructions how to use it via a #pragma in a shader, what works (DX12, Vulkan, Metal), what does not work (DX11, GL), what are the various HLSL syntax bits that were supported by the old compiler but not by the new one, and so on.

    At this point due to various gotchas (see the doc above) we would not recommend using DXC in production, but testing your shader codebases with it, or playing around with it would be much appreciated.
     
    Last edited: Apr 2, 2021
    Camarent, PutridEx, Mauri and 3 others like this.
  2. JiangBaiShi

    JiangBaiShi

    Joined:
    Aug 3, 2019
    Posts:
    25
    I tried to use #pragma require WaveMultiPrefix to enable shder model 6.5 features.
    I'm using Unity2020.2.0b14, URP 11;
    The gfx backend is DX12, on windows platform.

    But these following error occurred:
    • Shader Compiler Socket Exception: Terminating shader compiler process
    • Shader Compiler: Compile StampVertexGenerator.compute - StampVertexGenerator_SpawnStampVertices: Internal error communicating with the shader compiler process. Please report a bug including this shader and the editor log.
     
  3. Neto_Kokku

    Neto_Kokku

    Joined:
    Feb 15, 2018
    Posts:
    914
    The document says FXC doesn't support ray tracing shaders, but then how did Unity compiled them up to now?
     
  4. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,748
    Unity already used DXC just for the raytracing shaders.
     
  5. TLukas

    TLukas

    Unity Technologies

    Joined:
    Jan 29, 2019
    Posts:
    2
    We could not reproduce this on 2020.2.0b14.3668 by just adding #pragma require WaveMultiPrefix to any shader. Could you file a bug report with the exact shader as the error message mentions, preferably from your exact your version of Unity.

    Note that our DXC backend has been updated multiple times since 2020.2.0b14 so the crash is likely fixed in latest 2021.2

    However,
    in 2021.2 we do now observe that #pragma require WaveMultiPrefix produces "error: validator version 1,4 does not support target profile." which looks like a bug with DXC or with the Windows SDK. We will look into this more.
     
    PutridEx likes this.
  6. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,065
    This is both exciting and terrifying. Having barycentrics available is something I've often hacked around with baking colors into the mesh, being able to get them natively will be really huge. And a 12x compile speedup on a ton of shaders is amazing.

    Currently I'll have to adapt things for a few issues present in the doc. Better Shaders uses VFace instead of SV_IsFrontFace, but that should be an easy switch. The one which is harder is being able to write to a global or cbuffer variable - I use that pattern a lot with Stackables, as coupled with a define it makes it easy for different stackables to leave optional scratchpad data for other stackables to use. However, I can come up with some kind of blackboard concept that builds a struct that gets passed between stackables to get around this I think. To be clear, this means you cannot write to a variable in the global namespace or in the PerMaterial CBuffer from within the shader, correct? Also, another doc mentions a -gec flag - is there a downside to using this flag to maintain backwards compatibility? And could it be enabled somehow from the shader?

    The move to a #pragma require system over shader targets also sounds great - but can the requirements not be inferred from parsing the code? IE: If I use barycentrics, then the requirement is known. (Though I suppose this would require understanding the state of various conditional compiles, which could be involved).

    tex2D is extremely common in shaders and apparently not allowed anymore (and I assume all the old variants as well). That seems like a huge one, especially for people who are quickly porting surface shaders to Better Shaders, since switching every texture sample around is a lot more work (it's almost copy/paste now). Perhaps adding warnings about it to the current compiler would be a good thing to get people to start moving now?

    Given the current state in that doc, it does seem like it'll be a few years before the old compiler can be pulled. Still, I'd like to get BetterShaders in compliance sooner rather than later, as the community using it is still pretty small.
     
  7. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,748
    FWIW barycentrics do require GPU (and the graphics API) to support it, so for example they will never work on DX11 :(

    Yeah, this means you can't write into any cbuffer (including the implicit globals cbuffer, i.e. loose global variables).

    "-gec" flag has downsides in that while it enables this "allow writing into cbuffers" behavior, it disables some of more recent DXC features, so it's kinda neither here nor there.

    The problem is that you need to pass the "which shader target model?" to the DXC (and FXC) compiler before starting compilation, so it's either "compile twice, once to max supported feature set to find the features used, then into the minimal featureset that still has all the features", or to do a text-based search through the shader for what is used or not (and text based searches are extremely brittle).

    That's actually not a problem! We emulate it on DXC just like we emulate lack of it in some console shader compilers. If you look at HLSLSupport.cginc in recent Unity versions, there's like
    Code (CSharp):
    1. // DXC no longer supports DX9-style HLSL syntax of sampler2D, tex2D and friends.
    2. // Emulate those using our own small structs & functions that have a combined sampler & texture.
    3. #if defined(UNITY_COMPILER_DXC) && !defined(DXC_SAMPLER_COMPATIBILITY)
    4. #define DXC_SAMPLER_COMPATIBILITY 1
    5. struct sampler1D            { Texture1D t; SamplerState s; };
    6. struct sampler2D            { Texture2D t; SamplerState s; };
    7. struct sampler3D            { Texture3D t; SamplerState s; };
    8. struct samplerCUBE          { TextureCube t; SamplerState s; };
    9.  
    10. float4 tex1D(sampler1D x, float v)              { return x.t.Sample(x.s, v); }
    11. float4 tex2D(sampler2D x, float2 v)             { return x.t.Sample(x.s, v); }
    12. // ...
    And then the rest of the code can use it and things work out fine.

    Yes quite likely some time until we can pull it. It will have to happen at some point though, which is exactly why we thought it would be good to give a heads up on "hey DXC is coming!".
     
  8. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,065
    Damn, so stuffing colors for the next 8 years or so. One day we'll have nice things..

    Great. I switched over from VFace to SV_isFrontFace this morning- does that have any issue on older APIs? I chose VFace because it was the older semantic and potentially supported on more platforms, but wasn't sure if SV_isFrontFace was.

    Which reminds me, it would be very nice if these abstraction layers were included for switch/xbox/ps4 - I suspect they are not for legal reasons, but they are all over the net, even in Unity's own public github hosted projects. So I've managed to hack together my own version for Better Shaders from multiple sources, but I can't know if they are really correct or updated because I don't have hardware access and the sources are from random versions of Unity. (I'm pushing users to write stuff using the newer semantics, so make the same API available on Built In, because it's a much better abstraction layer than the sorta "This is what we needed to wrap" version in Built In)

    But perhaps a solution is just to include them into the engine, so I can reference them there regardless of which render pipeline is used or which consoles are installed. Since SRPs are now tied to Unity versions, having a way to path to them in the engine would make sense- and then when the abstractions are updated with new consoles or whatever, everything keeps working and is updated automatically. Right now I'm stuffing them into every pass of a shader for the built in pipeline because it's the only way to keep the file portable (since I don't want to require any include files be installed). So that would save about 8k lines of cruft if I could just #include them from somewhere.
     
  9. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    11,035
    If you look at the compiled shaders, Unity's been replacing
    VFACE
    with
    SV_IsFrontFacing
    and adding a ternary to get a -1.0 or +1.0 value for while because GLSL has never supported
    VFACE
    . GLSL used
    gl_FrontFacing
    from the start, which is equivalent to
    SV_IsFrontFacing
    . hlsl2glslfork changed that back in 2014. It's really only Direct3D 9.0 that used
    VFACE
    .
     
    jbooth likes this.
  10. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,748
    Yup, various bits of various NDA platforms can't be included everywhere easily.
     
  11. jbooth

    jbooth

    Joined:
    Jan 6, 2014
    Posts:
    5,065
    Right, and that's understandable - but if I could include a file that existed in the engine (not in the packages) which included all the various platforms that are installed on a machine, then it wouldn't matter. Right now, because all of that stuff is shipped in packages, I have to put together my own version of this stuff for the built in pipeline because it doesn't exist as an include in that pipeline (well, it has it's own version- but the macro's are not the same or complete, so you'd end up having to write different code for SRPs vs. standard again). Since SRPs are going to be shipping as part of Unity now, would I be able to include those files from srp core even if the user doesn't have an SRP installed/active?
     
  12. JiangBaiShi

    JiangBaiShi

    Joined:
    Aug 3, 2019
    Posts:
    25
    Hi, Sorry for the late reply, I tried to upgrade Unity from 2020.2.0b14 to 2020.3.4f1c1, and upgraded my graphics dirver & windows OS to the newest version. However this problem still presents in my project:
    I tested both on Quadro M2000 and GTX 1080, both produces "error: validator version 1,4 does not support target profile" when wavemultiprefix is used in shader code. Can I solve this issue by alternating my windows SDK version? Or I can only wait for you guys to fix it:(
     
    Last edited: Apr 18, 2021
  13. Arycama

    Arycama

    Joined:
    May 25, 2014
    Posts:
    174
    This is pretty neat, I'm looking forward to trying it out.

    Any estimates on how long until it's ready for production? I'm working on a project which is still a while away from release, and is targeting console and PC. So I'd have plenty of time to catch any bugs that pop up.

    Will this make it possible to support proper sampler states in shaders, with the full range of features such as anisotropy, border modes, additional filter modes (Such as min/max, which are handy for Hi-Z generation) etc? There's not really a reason to couple textures and sampler states anymore, right? (Performance-wise, it's also better to re-use sampler states within a shader)

    eg:
    Code (CSharp):
    1. SamplerState _LinearSampler
    2. {
    3.     Filter = MIN_MAG_MIP_LINEAR;
    4.     AddressU = Border;
    5.     AddressV = Border;
    6.     BorderColor = float4(1, 0, 0, 1);
    7. };
    Some other features such as being able to sample a read-only depth buffer, stream-out, adjacency topology, and dual source blending are also noticeably absent and would be handy for advanced graphics features, and I'm sure HDRP can utilize some of them too.
     
  14. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,748
    No, that's completely unrelated to the shader compiler backend. So this will not change by itself just because the compiler is different.

    All of these are also not related to the shader compiler being FXC or DXC. If someone from the graphics teams will decide to expose them, then great. But none of them are related to the shader compiler.
     
    Arycama likes this.
  15. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    778
    I finally got around to try this as well, and while I get this to work:

    fixed4 frag (v2f i, float3 PerspectiveBaryWeights : SV_Barycentrics) : SV_Target


    moving the SV_Barycentrics to the v2f struct

    Code (CSharp):
    1. struct v2f
    2. {
    3.   float4 vertex : SV_POSITION;
    4.   float2 uv : TEXCOORD0;
    5.   float3 PerspectiveBaryWeights : SV_Barycentrics;
    6. };
    throws a shader error
    Shader error in 'Unlit/NewUnlitShader': Semantic SV_Barycentrics is invalid for shader model: vs Use /Zi for source location. (on d3d11)


    Is this expected? I'm on DX12 here right now.
     
  16. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,748
    fherbst likes this.
  17. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    778
    Aras and aleksandrk like this.
  18. Arycama

    Arycama

    Joined:
    May 25, 2014
    Posts:
    174
    Although I don't think this is shader compiler related, @fherbst you should be able to wrap the corresponding line in a macro, eg:

    Code (CSharp):
    1. struct v2f
    2. {
    3.   float4 vertex : SV_POSITION;
    4.   float2 uv : TEXCOORD0;
    5. #ifdef SHADER_STAGE_FRAGMENT
    6.   float3 PerspectiveBaryWeights : SV_Barycentrics;
    7. #endif
    8. };
    Similar macros exist for the other shader stages (Vertex, Domain, Hull, Compute) as detailed in the "Shader Stage Being Compiled" section of: https://docs.unity3d.com/Manual/SL-BuiltinMacros.html

    The DX11 documentation has some more info on which semantics are supported as inputs/outputs for each shader stage: https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-semantics
     
    Aras likes this.
  19. AquaGeneral

    AquaGeneral

    Joined:
    Oct 30, 2008
    Posts:
    140
    @Aras Are there any plans to export PBD for DXC compiled compute shaders shaders? In both NVIDIA Nsight and Microsoft's PIX it's not seemingly possible to step-by-step debug compute shaders from a Unity project with shaders compiled with DXC. Thanks for exposing the new functionality by the way! It's tremendously useful.
     
  20. TLukas

    TLukas

    Unity Technologies

    Joined:
    Jan 29, 2019
    Posts:
    2
    Thank you for bringing this to our attention. I'm also guessing it doesn't work with non-compute shaders as well? Looking at the code it seems like we might need to add extra compiler flags to embed PDB into the debug data but this is true for both compute and graphics shaders. We'll investigate this shortly.
     
    Last edited: May 3, 2021
    fherbst and AquaGeneral like this.
  21. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    778
    Good idea! Another change I had to do to get that to work was to ifdef the entire fragment shader as well – a bit clunky, but seems to work.

    The DX11 documentation semantics page is missing the SV_Barycentrics semantic entirely and hasn't been updated in a while...
     
    Last edited: May 4, 2021
  22. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    11,035
    It's missing a lot of semantics. I kind of don't think it should be up to Unity to keep the semantics page up to date with every single option. Even the official Microsoft HLSL documentation isn't able to do that.
     
  23. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,748
    What is "the semantics page"? Is that about our docs, or Microsoft docs, or...?

    (if it's about our docs, then today nothing related to DXC is documented yet -- this thread & the google doc is kinda a "sneak peek" before we make it more public. Still have to solve DX11 issue somehow or otherwise DXC is not terribly useful in the wild)
     
  24. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    11,035
    Yeah, I assumed @fherbst was referring to Unity's own Semantics documentation. Which definitely has not been updated in a long time. The MSDN semantics link is to a page that technically doesn't even exist anymore, though thankfully Microsoft's documentation is generally good about redirecting dead links. However they could have been referring to Microsoft's HLSL Semantics documentation, which also doesn't include SV_Barycentrics ... it's missing a lot of ShaderModel 6.0+ info. They haven't even finished documenting features of DirectX 11.

    The only place I've found any info about SV_Barycentrics is Microsoft's wiki for DXC itself.
    https://github.com/microsoft/DirectXShaderCompiler/wiki/SV_Barycentrics
    Which does have a lot of good documentation on Shader Model 6.0 features.
    https://github.com/microsoft/DirectXShaderCompiler/wiki
     
  25. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    778
    Sorry for the confusion, I was referring to the DX11 Docs Semantics page that Arycama had linked to which seems to be somewhat outdated as bgolus said better than I - updated my post above to make that more clear :)

    @Aras I totally understand this is all Sneak Peek Preview Alpha Experimental Unsupported Land, and super happy you decided to test this "in the wild"!

    @bgolus yes, the DXC Wiki was what I so far read to piece together the few shaders I tried out, I wasn't able to find understandable sample shaders for 6.0+.
     
  26. hellkbd

    hellkbd

    Joined:
    Mar 16, 2021
    Posts:
    3
    Nice to see Unity make progress on shader programming language, is there any plan to support template function/class?
     
  27. Aras

    Aras

    Unity Technologies

    Joined:
    Nov 7, 2005
    Posts:
    4,748
    What do you mean exactly? (FWIW we are not inventing any new shading languages here -- just updating to the new Microsoft's HLSL compiler. Whatever Microsoft decides to add to HLSL the language, is up to them)
     
  28. AquaGeneral

    AquaGeneral

    Joined:
    Oct 30, 2008
    Posts:
    140
    To add to this, there is early experimental support for templates with the DirectX Compiler itself: https://github.com/microsoft/DirectXShaderCompiler/issues/2145
     
unityunity