Official Unity is adding a new "DXC" HLSL compiler backend option

I tried to use #pragma require WaveMultiPrefix to enable shder model 6.5 features.
I'm using Unity2020.2.0b14, URP 11;
The gfx backend is DX12, on windows platform.

But these following error occurred:

• Shader Compiler Socket Exception: Terminating shader compiler process
• Shader Compiler: Compile StampVertexGenerator.compute - StampVertexGenerator_SpawnStampVertices: Internal error communicating with the shader compiler process. Please report a bug including this shader and the editor log.

 

The document says FXC doesn't support ray tracing shaders, but then how did Unity compiled them up to now?

 

This is both exciting and terrifying. Having barycentrics available is something I've often hacked around with baking colors into the mesh, being able to get them natively will be really huge. And a 12x compile speedup on a ton of shaders is amazing.

Currently I'll have to adapt things for a few issues present in the doc. Better Shaders uses VFace instead of SV_IsFrontFace, but that should be an easy switch. The one which is harder is being able to write to a global or cbuffer variable - I use that pattern a lot with Stackables, as coupled with a define it makes it easy for different stackables to leave optional scratchpad data for other stackables to use. However, I can come up with some kind of blackboard concept that builds a struct that gets passed between stackables to get around this I think. To be clear, this means you cannot write to a variable in the global namespace or in the PerMaterial CBuffer from within the shader, correct? Also, another doc mentions a -gec flag - is there a downside to using this flag to maintain backwards compatibility? And could it be enabled somehow from the shader?

The move to a #pragma require system over shader targets also sounds great - but can the requirements not be inferred from parsing the code? IE: If I use barycentrics, then the requirement is known. (Though I suppose this would require understanding the state of various conditional compiles, which could be involved).

tex2D is extremely common in shaders and apparently not allowed anymore (and I assume all the old variants as well). That seems like a huge one, especially for people who are quickly porting surface shaders to Better Shaders, since switching every texture sample around is a lot more work (it's almost copy/paste now). Perhaps adding warnings about it to the current compiler would be a good thing to get people to start moving now?

Given the current state in that doc, it does seem like it'll be a few years before the old compiler can be pulled. Still, I'd like to get BetterShaders in compliance sooner rather than later, as the community using it is still pretty small.

 

Damn, so stuffing colors for the next 8 years or so. One day we'll have nice things..

Great. I switched over from VFace to SV_isFrontFace this morning- does that have any issue on older APIs? I chose VFace because it was the older semantic and potentially supported on more platforms, but wasn't sure if SV_isFrontFace was.

Which reminds me, it would be very nice if these abstraction layers were included for switch/xbox/ps4 - I suspect they are not for legal reasons, but they are all over the net, even in Unity's own public github hosted projects. So I've managed to hack together my own version for Better Shaders from multiple sources, but I can't know if they are really correct or updated because I don't have hardware access and the sources are from random versions of Unity. (I'm pushing users to write stuff using the newer semantics, so make the same API available on Built In, because it's a much better abstraction layer than the sorta "This is what we needed to wrap" version in Built In)

But perhaps a solution is just to include them into the engine, so I can reference them there regardless of which render pipeline is used or which consoles are installed. Since SRPs are now tied to Unity versions, having a way to path to them in the engine would make sense- and then when the abstractions are updated with new consoles or whatever, everything keeps working and is updated automatically. Right now I'm stuffing them into every pass of a shader for the built in pipeline because it's the only way to keep the file portable (since I don't want to require any include files be installed). So that would save about 8k lines of cruft if I could just #include them from somewhere.

 

If you look at the compiled shaders, Unity's been replacing

VFACE

with

SV_IsFrontFacing

and adding a ternary to get a -1.0 or +1.0 value for while because GLSL has never supported

VFACE

. GLSL used

gl_FrontFacing

from the start, which is equivalent to

SV_IsFrontFacing

. hlsl2glslfork changed that back in 2014. It's really only Direct3D 9.0 that used

VFACE

.

 

Right, and that's understandable - but if I could include a file that existed in the engine (not in the packages) which included all the various platforms that are installed on a machine, then it wouldn't matter. Right now, because all of that stuff is shipped in packages, I have to put together my own version of this stuff for the built in pipeline because it doesn't exist as an include in that pipeline (well, it has it's own version- but the macro's are not the same or complete, so you'd end up having to write different code for SRPs vs. standard again). Since SRPs are going to be shipping as part of Unity now, would I be able to include those files from srp core even if the user doesn't have an SRP installed/active?

 

Hi, Sorry for the late reply, I tried to upgrade Unity from 2020.2.0b14 to 2020.3.4f1c1, and upgraded my graphics dirver & windows OS to the newest version. However this problem still presents in my project:
I tested both on Quadro M2000 and GTX 1080, both produces "error: validator version 1,4 does not support target profile" when wavemultiprefix is used in shader code. Can I solve this issue by alternating my windows SDK version? Or I can only wait for you guys to fix it

 

This is pretty neat, I'm looking forward to trying it out.

Any estimates on how long until it's ready for production? I'm working on a project which is still a while away from release, and is targeting console and PC. So I'd have plenty of time to catch any bugs that pop up.

Will this make it possible to support proper sampler states in shaders, with the full range of features such as anisotropy, border modes, additional filter modes (Such as min/max, which are handy for Hi-Z generation) etc? There's not really a reason to couple textures and sampler states anymore, right? (Performance-wise, it's also better to re-use sampler states within a shader)

eg:

Code (CSharp):

1. SamplerState _LinearSampler
2. {
3.     Filter = MIN_MAG_MIP_LINEAR;
4.     AddressU = Border;
5.     AddressV = Border;
6.     BorderColor = float4(1, 0, 0, 1);
7. };

Some other features such as being able to sample a read-only depth buffer, stream-out, adjacency topology, and dual source blending are also noticeably absent and would be handy for advanced graphics features, and I'm sure HDRP can utilize some of them too.

 

I finally got around to try this as well, and while I get this to work:

fixed4 frag (v2f i, float3 PerspectiveBaryWeights : SV_Barycentrics) : SV_Target

moving the SV_Barycentrics to the v2f struct

Code (CSharp):

1. struct v2f
2. {
3.   float4 vertex : SV_POSITION;
4.   float2 uv : TEXCOORD0;
5.   float3 PerspectiveBaryWeights : SV_Barycentrics;
6. };

throws a shader error

Shader error in 'Unlit/NewUnlitShader': Semantic SV_Barycentrics is invalid for shader model: vs Use /Zi for source location. (on d3d11)

Is this expected? I'm on DX12 here right now.

 

Ah, thanks. Guess my understanding was wrong then, I thought semantics could be moved between frag parameters and a parameter struct "by design".

Posted an issue here: https://github.com/microsoft/DirectXShaderCompiler/issues/3729
Also, reported a Unity crash, not sure if that's "URP is crashing" or DXC-specific though: (Case 1332972)



And here's DXC in Shader Graph See https://twitter.com/hybridherbst/status/1388147666106847237

 

Although I don't think this is shader compiler related, @fherbst you should be able to wrap the corresponding line in a macro, eg:

Code (CSharp):

1. struct v2f
2. {
3.   float4 vertex : SV_POSITION;
4.   float2 uv : TEXCOORD0;
5. #ifdef SHADER_STAGE_FRAGMENT
6.   float3 PerspectiveBaryWeights : SV_Barycentrics;
7. #endif
8. };

Similar macros exist for the other shader stages (Vertex, Domain, Hull, Compute) as detailed in the "Shader Stage Being Compiled" section of: https://docs.unity3d.com/Manual/SL-BuiltinMacros.html

The DX11 documentation has some more info on which semantics are supported as inputs/outputs for each shader stage: https://docs.microsoft.com/en-us/windows/win32/direct3dhlsl/dx-graphics-hlsl-semantics

 

@Aras Are there any plans to export PBD for DXC compiled compute shaders shaders? In both NVIDIA Nsight and Microsoft's PIX it's not seemingly possible to step-by-step debug compute shaders from a Unity project with shaders compiled with DXC. Thanks for exposing the new functionality by the way! It's tremendously useful.

 

Good idea! Another change I had to do to get that to work was to ifdef the entire fragment shader as well – a bit clunky, but seems to work.

The DX11 documentation semantics page is missing the SV_Barycentrics semantic entirely and hasn't been updated in a while...

 

It's missing a lot of semantics. I kind of don't think it should be up to Unity to keep the semantics page up to date with every single option. Even the official Microsoft HLSL documentation isn't able to do that.

 

Yeah, I assumed @fherbst was referring to Unity's own Semantics documentation. Which definitely has not been updated in a long time. The MSDN semantics link is to a page that technically doesn't even exist anymore, though thankfully Microsoft's documentation is generally good about redirecting dead links. However they could have been referring to Microsoft's HLSL Semantics documentation, which also doesn't include SV_Barycentrics ... it's missing a lot of ShaderModel 6.0+ info. They haven't even finished documenting features of DirectX 11.

The only place I've found any info about SV_Barycentrics is Microsoft's wiki for DXC itself.
https://github.com/microsoft/DirectXShaderCompiler/wiki/SV_Barycentrics
Which does have a lot of good documentation on Shader Model 6.0 features.
https://github.com/microsoft/DirectXShaderCompiler/wiki

 

Sorry for the confusion, I was referring to the DX11 Docs Semantics page that Arycama had linked to which seems to be somewhat outdated as bgolus said better than I - updated my post above to make that more clear

@Aras I totally understand this is all Sneak Peek Preview Alpha Experimental Unsupported Land, and super happy you decided to test this "in the wild"!

@bgolus yes, the DXC Wiki was what I so far read to piece together the few shaders I tried out, I wasn't able to find understandable sample shaders for 6.0+.

 

Nice to see Unity make progress on shader programming language, is there any plan to support template function/class?

 

To add to this, there is early experimental support for templates with the DirectX Compiler itself: https://github.com/microsoft/DirectXShaderCompiler/issues/2145

 

Hi! I'm using #pragma require WaveMultiPrefix to force DXC to enable shader model 6.5 features. My shader uses

Code (CSharp):

1. uint shadingRate                 : SV_ShadingRate;

in order to do primitive shading rate (the required extension is already activated). This seems to compile for D3D but not for Vulkan. Getting the trace "shader is not supported on this GPU (none of subshaders/fallbacks are suitable)". Any tips on how to make this compile for Vulkan? many thanks!

 

I use WaveMultiPrefix because there is no other way from Unity (at least that I know) to force DCX to compile using HLSL 6.5. The question then will be, is there any other way of doing that?
It'd be nice to have a #pragma ShadingRate so SV_ShadingRate semantic could be supported.

 

Hi TLukas, thanks for the reply.
We are using Vulkan and we have easilly activated the FSR extension by using the Rendering Plugin API (intercepting the required calls). FSR can operate at 3 different levels/methods: pipeline, where you set the rate in the graphics pipeline or in a command buffer; attachment, where you can use an attachment image to control the shading rate and primitive, where you can control the shading rate in the vertex shader (or in the last active pre-rasterization stage).

The pipeline and attachment methods will require a proper implementation provided by Unity, unless you want to do some nasty tricks to intercept the right calls. But, as far as I know, the primitive method could be used just by activating the extension, activate also the primitive method in the physical device (both are easilly doable) and then using the SV_ShadingRate semantic in HLSL, which translates to the appropiate PrimitiveShadingRateKHR SPIR-V decorator according to this document: https://github.com/microsoft/Direct...PIR-V.rst#implicit-location-number-assignment

 

HLSL 2021:

https://devblogs.microsoft.com/directx/announcing-hlsl-2021/

 

HLSL 2021 come with some pretty awesome features, and it can help to simplify the shader code dramatically. Wish Unity will integrate that in near future.

 

I just noticed that in 2021.2.12f1, UNITY_COMPILER_DXC is not defined for ray tracing shaders, I think it's certainly a bug caused some shader code will be shared between graphics and DXR pipeline, and ray tracing shader certainly compiled with DXC.

 

I just tried using

Code (CSharp):

1. #pragma require Native16Bit

in our shaders (PC, Vulkan, unity 2021.2.19f1 with URP) - and it crashes the editor with no crash popup

 

Not sure if this is the right place to post this, but there appears to be a bug with the DXC compiler and the multiview stereo mode used by the Oculus Quest 2 when using vulkan as the graphics api. Specifically, a shader compiled with DXC doesn't get the correct stereo eye index (0 for both eyes) while the same shader compiled with FXC does. Looking at the URP shader include files, in multiview stereo mode the eye index is supposed to be given to the shader in a special cbuffer as gl_ViewID (see com.unity.render-pipelines.universal/ShaderLibrary/UnityInput.hlsl line 176). The DXC doc mentions that the compiler is doing some remapping of resources for Vulkan shaders to match unity's expectations. My uneducated guess is that this cbuffer isn't getting mapped to the correct resource.

 

Yeah, I've already filed a bug report (IN-6122). Attached is the repro project as well, which was made with unity 2021.3.4f1. If you feel like checking it out, you'll need to actually build and run the project on a quest/quest 2 since multiview stereo is a hardware specific rendering mode that only works on those headsets.

 

Ok, so I think I've nailed down exactly where the issue lies. FXC seems to treat the constant buffer OVR_multiview in a special way, not actually using the constant buffer and instead replacing it with a uniform bound to the builtin Vulkan variable ViewIndex (shows up as "Decorate ## BuiltIn ViewIndex" in compiled code). DXC doesn't do this. It keeps the dummy constant buffer and does not add a uniform mapped to ViewIndex. I'm guessing the reason this was done with FXC was there is no equivalent in DirectX to this builtin (until SM6.1), so to keep FXC happy a dummy constant of the same name is declared in the shader and somewhere during the hlsl to glsl translation process swapped out with the real builtin uniform.

Shader model 6.1 added SV_ViewID which is equivalent to ViewIndex (https://github.com/microsoft/DirectXShaderCompiler/wiki/SV_ViewID). DXC maps SV_ViewID to ViewIndex (https://github.com/microsoft/Direct...PIR-V.rst#implicit-location-number-assignment). However, this requires that the shader target model 6.1 to actually use SV_ViewID, even when compiling to SPIR-V. Since it was decided we shouldn't have access to specifying the minimum shader model, the only option is to try to require a feature introduced in the same or later shader model. For example, using #pragma require barycentrics enables SM6.1. Enabling this also makes SV_ViewID a valid semantic. Compiling the shader like this shows that the SV_ViewID gets correctly mapped to the ViewIndex builtin. Unfortunately, its still completely unusable, since enabling barycentrics causes the "shader is not supported on this GPU" in both the editor and on the quest hardware. So close! The only other require that boosts the shader model to higher than 6.0 is the native 16bit storage extension, which is somehow not supported on the quest, and in editor causes the shader to appear completely corrupted. EDIT: The corruption happens because NVidia apparently doesn't support 16 bit types in interpolators.

I think we need a special pragma require that tells DXC to target 6.1 or greater without enabling any other vulkan extensions. In Vulkan ViewIndex is part of the default 1.1 spec so there is no specific extension to enable, we just need the compiler to target 6.1 so SV_ViewID is a recognized semantic.

 

EDIT: I was wrong about the ViewIndex builtin being part of the base spec, on account of the vulkan docs being confusing. There is a device and corresponding shader extension for multiview and enabling ViewIndex (https://www.khronos.org/registry/vulkan/specs/1.3-extensions/man/html/VK_KHR_multiview.html).

Multiview stereo is currently used by the Oculus XR plugin for quest/quest 2. If you can, please add a #pragma require for multiview stereo which sets DXC to target 6.1 and adds the SPV_KHR_MULTIVIEW extension so that SV_ViewID can be used to get the eye index.

 

Hey there!
Is there any way we can add or alter dxc compiler flags?
And can you also tell us if and which of the flags are activated during compilation by default?

 

Hi,
On Metal M1, the following kernel fails with no error with dxc (does not get appended, no kernel code executed at all as far as user can tell):
(appendbuffer stride 68)

Code (CSharp):

1. struct TestStruct
2. {
3.     float4x4 a;
4.     float b;
5. };
6. AppendStructuredBuffer<TestStruct> _testBuffer;
7. #pragma kernel Test
8. [numthreads(1, 1, 1)]
9. void Test (uint3 id : SV_DispatchThreadID)
10. {
11.     TestStruct test;
12.     test.a = 0.0;
13.     test.b = 0.0;
14.     _testBuffer.Append(test);
15. }

but this executes and works as expected (appendbuffer stride 64):

Code (CSharp):

1. struct TestStruct
2. {
3.     float4x4 a;
4. };
5. AppendStructuredBuffer<TestStruct> _testBuffer;
6. #pragma kernel Test
7. [numthreads(1, 1, 1)]
8. void Test (uint3 id : SV_DispatchThreadID)
9. {
10.     TestStruct test;
11.     test.a = 0.0;
12.     _testBuffer.Append(test);
13. }

float4x4 + float combo seems to mess something up. Without #pragma use_dxc both execute properly.

Shader compiler outputs:

Aligned struct works again, is this intended behaviour (To silently fail)?

Code (CSharp):

1. struct TestStruct
2. {
3.    float4x4 a;
4.    float b;
5.    float c;
6.    float d;
7.    float e;
8. };

 

ooh this sounds promising, i hope MS have finally fixed their out of control loop unrolling bugs that just tank processing for 10-15 minutes and can totally just be skipped with compilation tags without any perf changes lol.

 

I've tried using pragma use_dxc for my custom deferred SRP pipeline. This SRP pipeline also used RenderPass API. It worked well under Android and Vulkan API. But under Metal ios, something went wrong. I can not figure out. Can you guys help me with that? Thx a lot

 

Can we have the option to make dxc compile shader with HLSL 2021? HLSL 2021 has so many features to make shader programming more convenient and easy to maintain.