DXR in virtual reality, possible or not?

Hi, I have a DXR project which has very good FPS and I would like to try it in Oculus Quest using Link. But when I try to start Oculus, I am forced to switch from DirectX 12 to 11 when raytracing doesn't work.

"D3D11 is currently the only graphics API compatible with the Oculus XR Plugin on desktop platforms. Please change the preferred Graphics API setting in Player Settings.
UnityEditor.EditorApplication:Internal_PlayModeStateChanged (UnityEditor.PlayModeStateChange)"

Does DXR work for you in virtual reality - and on what hardware?

 

Trying to use OpenVR and Steam instead of Oculus drivers. Steam should be compatible with Oculus Quest Link.

SteamVR from Steam installed, in Quest I see Steam enviromnent.

Next step is Unity project for Steam....

OpenVR in not standart in Unity XR plug-in manager:
https://github.com/ValveSoftware/unity-xr-plugin

https://github.com/ValveSoftware/steamvr_unity_plugin/releases/tag/2.6.0b4

Cannot run preview mode to headset, Unity build crashes instantly.

 

I think the problem is that raytracing needs DirectX 12 but VR drivers are still DirectX 11.

 

Just hit the same thing... Raytracing is a DX12 feature (also in Unreal), so I think we're out of luck for now. I really just want to try it, I don't think it's viable, but... you know... I'm willing to suffer through low FPS, blink rapidly and let the brain interpolate..

 

I have a scene tuned with raytracing with about 90-130 fps which should be enough. I don't need to have DXR lighting, I only use raytracing for reflection. I can burn shadows / lights the old way but for more complex geometry I need raytraced reflections (self-reflection).

 

Same here, This https://docs.unity3d.com/Packages/c....high-definition@9.0/changelog/CHANGELOG.html said that "Ray tracing support for VR single-pass" but I have no ideas how to work with it, basically DXR needs DX12 and VR is only supported in DX11, correct me if I'm wrong.

 

I just tried the following combinations:

VR DX12 + Oculus XR plugin (1.5.0)
VR DX12 + OpenVR XR plugin (1.0.1)

Both work in DX11 but break in DX12.



I also tried the dev/UpdatedHeaders branch of the OpenVR XR plugin, but it does not work in DX12 either.

I have a Unity 2020.1 alpha project here that works flawless when using DX12 & Ray Tracing in VR. (using the old com.unity.xr.oculus.standalone package, which is no longer supported)



As soon as I try upgrading project to 2020.1 beta or later, I get the following message.



After pressing OK, upgrading the project and using the new XR Plug-in Management, we lose DX12.

So they have deprecated the old working package, while the new one is not yet ready. There's not a lot you can do right now except manually copying the old package to your /packages/ directory and use the last working editor before it got deprecated (2020.1.0a15), which is almost a year old by now.

Edit
I tried to hack in the legacy package in a modern editor (2020.1, 2020.2, 2021.1) and forcing the XR settings to read only to stop it from updating but no luck. I'm afraid we're stuck in a broken migration process here.

 

Would be nice to get at least some heads up from Unity if DX12 VR support is coming back in the future..

 

I have also forwarded this to ValveSoftware/unity-xr-plugin (issue #68), maybe this is something Valve can handle from their side.

Edit
Response over at Valve:

 

Does anyone know who to contact at Unity to get more information about this? It would be nice to know if this is solution to this on the horizon.

 

It does indeed start, but the display/graphics break completely. (and all other VR related assets/packages I tested seem incompatible with OpenXR at the moment)

So we either:

- Port modern ray tracing shader code to 2020.1.0a15 and use an optimized acceleration structure manager to compensate for the performance loss that comes with this downgrade, but it will never be on par with 2020.2/2021.1

- Find some kind of hack to run legacy XR on 2020.2/2021.1, but you obviously lose future support

- Wait for Oculus XR / OpenVR plugins to support DX12, this is the easiest since this is what current projects are built on, however we don't have confirmation DX12 support will land here in the future

- Start over on the OpenXR platform, report all graphics/input related bugs to the team & asset creators, I think this is the best option because OpenXR indeed launches in DX12, but we will have to invest a lot of time to get everything to fit, and it is a bit of a gamble, hoping everyone will adapt to OpenXR in the future.

Edit
I just got OpenXR to work in DX12, using Multi Pass mode. It could be that URP/HDRP actually work in Single Pass Instanced mode, because I am currently still testing on legacy pipeline.

 

I too would vote for the last option, Openxr and Raytracing are both future looking technologies that should really be able to work together in DX12.

 

A couple of days ago I tested HDRP through OpenXR in DX12, with DXR and DLSS in Single Pass Instanced mode. I ran this setup on a Oculus Quest 1 (72hz, 1440 x 1600 pixels per eye) running on a PC with a RTX 3080.

With the full ray tracing stack enabled (RTGI/RTR/RTAO/RTSS) and DLSS in Maximum Performance mode it was constantly locked on the Oculus Quest's 72hz frame limiter, so it is looking promising so far.

There is an issue with RTGI/RTSS that appear to render black after you turn a certain amount of degrees in VR. I have submitted a bug report (Case 1336608). I got an email this morning saying that they successfully reproduced the issue.

If you'd like to optimize this for lower end hardware, you could tweak the effects MIP bias, use a better performing HDRP asset and use half resolution effects and/or combine the minimum smoothness with reflection probes. Optimizing the ray tracing acceleration structure will help too.

For anyone looking to get started with DXR and DLSS in VR, I have created another boilerplate project. I will update this repository once a fix for case 1336608 lands.

https://github.com/m0nsky/hdrp_dxr_dlss_vr

 

Very cool of you! Thanks.

 

Thank you! I installed it and it works, but my pc (TUF RTX3080, ryzen 9 5900x and 16GB of RAM) can only reach 22fps. I'm using oculus rift cv1 with Open XR plugin. My Open XR Runtime is set "system default" (Steam VR in my case). Soorry for my bad english.

 

Hi @JanMarm

- Any chance you could try this through the Oculus runtime in OpenXR? Oculus is the runtime I used to test. I wonder if this is related to the selected runtime.



- Are you running this in 2021.2.0a17 with DLSS enabled in Maximum Performance mode?



Your PC specs are better than mine so it should run fine, especially because the CV1 uses a lower resolution (1080x1200 pixels per eye) than the Quest 1 I used to test.

 

A fix for case 1336608 has landed on the Graphics repository about 2 hours ago. It has not yet been merged into master but I've pushed it to the hdrp_dxr_dlss_vr repository and can confirm the issue has been fixed. So if you want to start developing a VR game with DLSS & ray tracing you should be good to go.

Edit
Found a remaining issue related to ray traced shadows in XR and added a comment to the PR. I tested the fix for RT shadows locally and can confirm this fixes the issue.

Edit 2
Here's a bit of an extreme standalone build, which has all of the ray tracing effects, including ray traced shadows.
It still manages to run steady at the 72hz frame limiter on a RTX 3080 / Quest 1.

I've only tested this using the Oculus runtime.

https://hyperproject.net/download/hdrp_dxr_dlss_vr_standalone.zip

Edit 3
I've pushed the changes (openxr 1.2.0, optimized hdrp asset & support for rt shadows in xr) to the hdrp_dxr_dlss_vr repository.

Edit 4
Video. It could use some denoiser radius tweaks here and there, but it feels like a huge milestone that we're able to do this now. Let's hope the constant deltatime fix lands on DX12 XR soon.

 

I get like 36 fps with Oculus on 2080 / Quest 1.
With SteamVR I get like 3fps and lots of crashes, so seems not to be working yet properly.
There seem to be some rendering issues in Global Illumination, and stability is not that good, but overall looks promising! At least for the next generation of GPUs

 

Cool! Are these results from the standalone build I posted?

 

No, I used the project. But did not change anything besides the runtime target (trying both Oculus and Steam)

 

Does DLSS upscaling work now? Any input on this is very much appreciated. Thanks for sharing the repo!

 

It should be working indeed. I've noticed an issue where DLSS sometimes will not update correctly after changing the settings. Toggling the MIP bias in the DRS settings off and on will cause it to "reset" and apply the correct settings.

 

Do you have to choose a upscaling resolution (for example 50%), or is this controlled in the DLSS quality settings?

 

You can do both.





There's a lot more info in the original PR here and some additional info in the official docs here. I don't think DLSS has been included in the latest HDRP release yet, so if you're using this in another project you'll either have to copy the packages from my repository or do a fresh github clone.

 

Is it required to use DX12? I have had some performance issues while testing it out..

 

I've managed to reproduce these crashes in a minimal project and submitted a bugreport (Case 1342985), I will let you know once the issue has been resolved.

 

m0nsky, thanks for the project. I'm trying to switch from OpenVR/SteamVR to OpenXR specifcally for DX12/Raytracing/DLSS....

Regardless of what options and combinations I try from your Biolerplate GitHub project and I can not get my Quest 2 to get an image. SteamVR loads upon running, but nothing appears. (Both in editor and after compiling your project).

Your precompiled Standalone.zip however works instantly.

Is there anything I need to change from your GitHub project to get working with Quest 2 under SteamVR?

As I've been scratching my head for many hours as to why I cannot get this to work..

Thanks for any help on the matter.



Edit: Found it for others who are having trouble.

Disabling Depth Submission mode from '16Bit' to None immediately allowed me to see the scene in the Quest 2, and have the scene work normally.

Cheers.

 

Thanks for testing and mentioning the workaround @Freakish. I have updated the repository to use the latest editor (2021.2.0b1), updated the HDRP packages to todays master and disabled depth submission mode for now.

I found out ray binning does not yet work in XR. Ray binning sorts the rays in groups close to eachother before dispatch which could trigger a lot of cache (geometry/texture sampling) hits (and result in a performance increase), especially in XR. Auzaiffe thinks it got lost in the backlog and is going to take a look at it. (Case 1346374)

I have also found a temporary workaround for the crashes that occur when exiting play mode (Case 1342985). Pausing the editor before exiting play mode will prevent it from crashing.

I will try and reproduce the SteamVR + depth submission mode issue and submit a bugreport.

Edit
Reproduced and reported. (Case 1346510)

Edit 2
Just got confirmation that QA managed to reproduce the issue.

 

Thanks, I wasn't sure what that setting is meant to do, and the docs are pretty light on any real info. As I don't see any similar option offered in SteamVR, or OpenVR for any kind of depth options, I had wondered if it was more a WMR or similar specific option.

Anyway much appreciate your effort in regards to the progressing of DX12 VR and the bug solutions... Will keep track of the incoming fixes and your efforts..

 

I've done some changes to the repository.

Updates
- Updated Editor (2021.2.0b2)
- Updated HDRP (4f8b69d7f47480572ceb3db0e49fd09f397720c9)
- Updated OpenXR (1.2.3)

Changes
- Locally merged the PR for ray binning in XR
- Disabled motion blur, vignette and chromatic aberration
- Disabled motion rejection (RTGI + RTAO) to avoid flickering

I did some testing on my Oculus Quest 1 this morning to check out if there are any issues related to ray binning and everything seems alright. Let me know if you run into any issues so I can try to reproduce and report them.

 

Because DX12 and Ray Tracing have recently gone out of experimental, I decided to upgrade the project to 2023.1.0a26. A couple of things on the engine/HDRP side have broken since the last update, however, MockHMD still works. I have just finished sending out the required bug reports, special thanks to "VR HOT Dev" in Discord for helping me with the investigation of these issues.

IN-29319 - [HDRP VR] Ray traced reflections in standalone do not match reflections in editor
IN-29324 - [HDRP VR] Black screen flickering when DLSS is enabled in standalone build

 

I have been e-mailing with QA back and forth all week, but they can not reproduce the black screen issue. I have tried multiple NVIDIA drivers and DLSS versions, but this seems to have no impact on my side.

If anyone with a RTX card and a VR headset, could download this build, and let me know what you see, it would be much appreciated. (please mention which GPU and VR headset you are using)

 

Black screen, couple white flashes at start,
sometimes can see reflective shiny ball and plane on the right eye for 1 frame (if move head around).

tested with oculus air link, rtx3090, log doesnt have anything interesting i think. (attached)

 

Thanks for confirming this so quick, this is exactly what I see too. I hope more people will be able to test this so I can forward this to QA.

 

same here, on RTX 3080

 

So far issue IN-29319 has been confirmed by QA as OXRB-124. Their side of the report seems a little messy, but let's hope it arrives correctly on the HDRP team side.

And about IN-29324, it seems to only happen on RTX 30 series hardware. So far we've got (thanks to you guys!):

Confirmed black screen
5/5 (100%) RTX 30 series users

Confirmed no issues
- 2/2 (100%) RTX 20 series users
- 1/1 (100%) RTX 40 series users

This issue is already taking almost 2 months for QA to confirm. I've tried to make this as easy as possible for them (to hopefully speed this up), by sending them a tiny build which does not even require a connected VR headset. This way they can simply run it on a system with a RTX 30 series GPU and observe the black screen, however, another 2 weeks have gone by.

 

Thanks @chap-unity , I will keep that in mind.

I think it would be great to see improvements on the QA side here. Since there have been a lot of improvements internally to streamline the outgoing flow from the SRP team to the users, it would be great to also streamline the ingoing flow from the users to the SRP team. This way the entire flow (user -> internal -> user) will be able to benefit from recent improvements and not result in a possibly worse user experience in iteration times than before (due to internal speed improvements getting cancelled out by QA choking).

Edit
The issue has just been confirmed.

 

Now this looks real interesting, DLSS with VR in general is just such a given, really odd its not natively supported. What would be really cool would be to get the path tracer to render in vr with dlss

 

2060 Super, Quest 2 via Air Link
Looks like this

 

Thanks @nigelf and everyone else, the issue has been fixed in 2023.2.0a6.

As soon as the environmental lighting issue has been tracked down (I'm also seeing this in your screenshot) I will update the repository and we will once again have DXR + DLSS in VR in working order.

 

Hi @m0nsky. did you tried custom SRP with DXR in vr directly? The game view in editor has no problem, but it is black screen in oculus 1. I cannot figure out what happened.

I posted 2 thread and no one replayed.
https://forum.unity.com/threads/bla...render-pipelines-core-in-2022-2-16f1.1429318/
https://forum.unity.com/threads/how-to-write-custom-dxr-volumn.1431232/

 

@Tangerinejuice Did you try 2023.2.0a6 or higher?

 

@m0nsky I tried 2023.1.0b10. I did not notice there is version 'a' and 'b'. I would download version 'a' for test.

By the way, the raytracingAccelerationStructure Api changes a lot. I have not figured out dxr srp in lastest version 'b' yet.

 

Another intersting thing is urp and hdrp both use srp package, but they all works fine with oculus. I guess, I should add some magical code for it to work properly in my custom srp.

 

I'm talking about 2023.2, not 2023.1.

 

@m0nsky I test 2023.2.0a14 on rtx 2070 super.

• 2023.2.0a14 create new project with vr template
  • switch to dx12
    • lots of error, black screen
  • add core pacakge
    • no error
  • import hdrp with dxr
    • no error
• 2023.2.0a14 create new project without template
  • switch to dx12
    • no error
  • switch to dx12 and openxr
    • lots of error, black screeen

I uploaded the error screen shot.

How to use frame debugger for oculus? The render result in 2023.1.0 is right and the frame debugger shows the steps of edtior render.
Is it possbile that unity did not stream the result to oculus?