Question Possible to make the Oculus Passthrough API work with the OpenXR plugin?

Does anybody know if it is possible to make the Oculus Passthrough API work with the OpenXR plugin? It seems to work when ONLY the Oculus plugin is enabled in XR plugin management, but when either OpenXR or both is enabled, the background simply turns black.

I am trying to use the hand-tracking feature from XR Toolkit 2.3.0, which requires the OpenXR plugin to be enabled, and ideally, I would combine this with the Oculus Passthrough API.

 

Any Luck?

 

Hi, I can't seem to get it working.

I'm using the Oculus Integration v50 linked by you. I have the OVR Manager attached to the 'Complete XR Origin Hands Set Up' prefab on the XR Origin along with a OVR Passthrough Layer attached. The Main Camera environment set to solid colour (0,0,0,0) . I get the following error: Failed To Initialize Insight Passthrough. Passthrough will be unavailable. Error Failure_OperationFailed. Accompanied by black screen in both the playtime in the editor and on build. So i try to go through and fix all outstanding problems listed in the Oculus tab in Project Settings, The Outstanding issues: Oculus must be added to the XR Plugin active loaders. and Recommended items: Unity's OpenXR Plugin is not recommended when using the OculusSDK, please use Oculus XR Plug-in instead.

If i go through and fix all issues it removes OpenXR out of the plug in management and the hand tracking doesn't work.. I then get to a point where Unity completely crashes because its using both OpenXR and Oculus backends.

This has been hell to try and get working and I'm being forced to choose one or the other (I cannot use full Oculus Integration OVR Rig due to scripting conflicts)

Any help would be much appreciated.

 

Can confirm does not seem to work, just fails with Error Failure_OperationFailed (either in my own attempts, or the sample project).

 

Any Updates would be greatly appreciated!

 

Hi, Lucasperlee and I solved this issue using Unity 2022.2.11f1, V50 Integration asset, OpenXR 1.7.0, Unity XR Hands 1.1.0 and Unity XR Interaction toolkit 2.3.0 (XRI). Here is a tutorial on how we did we did it:

1. Download and install V50 Integration asset, the Open XR, Unity XR Hands and XRI, as shown in the image below (V50 integration won’t appear in package manager):
   
2. Go to the Oculus tab in the project settings and apply all issues except these two (there’s no need to install the oculus XR plug in in package);
3. If your project already has a custom android manifest you need to make a backup and exclude it from the project. In oculus project setup tool fix the issue related to android manifest to generate a new one. Now manually merge your backup with the generated one.
4. In your new merged manifest add “android:exported=”true”” in the manifest activity line
5. Check OpenXR in XR plugin Management.
6. Use these configurations in the Open XR tab in XR Plug-in Management.
7. Check the project validation tab in XR plugin management and fix everything that appears there.
   
   
8. If the project validation tab didn’t add “USE_INPUT_SYSTEM_POSE_CONTROL” in the scripting define symbols, add it.
   
9. Open a new scene and add the Complete XR Origin Hands Set Up prefab from the XRI package samples(You can unpack and create your own prefab if you’d like).
   
   
10. In the Main camera inside the Complete XR Origin Hands Set Up Hierarchy, change clear flags to solid color and background to transparent black (0,0,0,0).
    
    
11. In XR Origin in the Complete XR Origin Hands Set Up hierarchy add OVR manager with these configurations:
    
    
12. Still in XR Origin add OVR Passthrough Layer with these configurations:
    
    
13. You are ready to build the project

 

I am Lucasperlee and I approve this message

 

@ericprovencher , Interested in that!

 

I just built v51 and it worked fine with the exact same configuration as mentionned above.

 

v53 and same configuration here. If I enable passthrough, I can't see hands

 

I tried this method to use passthrough and that works, but for some reason it is breaking switching between controllers and hands. You can see the hands if you put your controllers down, but if you try to use controllers the rig won't switch to show controllers. Is anyone else running into this issue? Is there a way to solve this?

 

Hello, I've followed the steps laid out by androanHeinen, and passthrough works. However my controllers aren't tracking, they're not even showing up in the connected devices on the input debugger. The head is tracking fine. Does anyone know why that may be?

 

Were you able to resolve this? I'm also having this issue after following the same steps. HMD is tracking via openxr and input system, and I'm also getting passthrough, but the controllers are not tracking anymore.

 

The solution posted above seems to be working for me in Builds. In Play Mode, I have not had any success yet even though I have set up passthrough over link. Can anyone confirm that the above works in Play Mode? I get the following:

[OVRPlugin] [Insight MR] Initializing passthrough failed with error -1004. (arvr\projects\integrations\OVRPlugin\Src\Util\InsightMrManager.cpp:803)

Specs

• Unity 2021.3.23f1
• XRI 2.3.2
• XR Plug In Management 4.3.3
• XR Hands 1.1.0
• Oculus XR Plugin 3.3.0
• Oculus Integration 54.1

 

Let me put an end to this discussion!

OpenXR itself does not support being used in conjunction with Oculus Interaction. To address the need of "wanting to use OpenXR for passthrough on Quest", Unity and Meta have jointly launched the Meta OpenXR feature plugin! (Based on things done with AR Foundation)
(https://blog.unity.com/engine-platform/get-started-developing-for-quest-3-with-unity)

Although it's still in its early stages, it's already in a usable state! Remove Oculus Interaction, everyone no longer needs to find a way to resolve the conflict between OpenXR and Oculus.

 

hey folks not trying to hijack the thread, but if you just wanted passthrough and not hand tracking, just controller tracking, is there perhaps a method posted for that? or maybe some tips on how to alter the directions posted in that screencap tutorial earlier? also i'm the only member of the team with a Quest 1, and i'm wondering if any of the integrations support it at all? i'm on V55. i like the previous post on the Meta OpenXR plugin but i'm willing to bet there will be no support for Quest 1 with that. i'm planning on getting a Quest 3 when it comes out but i need to limp along for now. any help appreciated!

 

+1 to what @HelckOne mentioned above. My current understand is that's the future of how Unity is planning to support this. This forum post might be helpful for anyone looking at this in the future: https://forum.unity.com/threads/introducing-meta-quest-support-in-ar-foundation.1450966/

 

Isnt a way to do it without depending on que Oculus Plugin? ie. OVR and so, so it works on other Passthrough-handtracking capable headsets that are not from Meta? we try to use the as least platform exclusive code as possible

 

Yes you can @Extrys , using OpenXR + Unity XR Interaction toolkit AND Meta or Pico OpenXR plugins (not sure for HTC's VIVE XR Elite though)

• If you wanna use Meta passthrough with OpenXR :

Add Meta OpenXR package (com.unity.xr.meta-openxr) and check the capabilities in OpenXR settings but also in the submenu Meta Quest Support options.



• If you wanna use Pico 4 passthrough with OpenXR :

Add from disk the "just released" package PICO Unity OpenXR 1.1.0
from Developers portal and make sure to follow these few steps to enable the right capabilities at start or whenever you want with a script.

WARNING : I had many troubles using this new OpenXR plugin.
Like the broken ray interactor in MR (which works in the last package with Meta thanks to XRI's team), but also the hands renderer and the XR Origin Hands.

You can still use PICO Unity Integration SDK which works with XRI (passthrough + controllers & hands).



I got to say Unity's team is making a great work with OpenXR and HMDs brands to standardize XR developement.

 

@Yann_de_Couessin what version is that? In the OpenXR Feature Group, for Meta Quest, I only have "AR Foundation: Meta Quest Features" and "Meta Quest Support". I'm on package Meta OpenXR Feature version 0.1.2

 

@ericprovencher have you managed to make Passthrough work in Unity via Link cable/Air link? Or does the game need to be exported to the headtset for the Passthrough to work?

Edit : I see now that it's mentioned on GitHub that Link doesn't currently work, thank you!

 

Thank you, this is very relevant to my use case! Just to clarify for my own sanity:

Assuming I want to develop something that works for both the Quest and Pico, I can create an app that works for both just using OpenXR by checking the "Meta Quest Support" checkbox and downloading and installing the "PICO Unity OpenXR SDK". This just supports basic OpenXR features like controller tracking and button input.

Previously, to access Quest-specific features like passthrough, you would need to install the "Oculus Integration SDK" which was built on OpenXR, but not compatible with Unity OpenXR features. However now, on newer versions of Unity, it seems like there's a package you can install from the Unity Registry titled "Unity OpenXR Meta" that allows you to use Quest-specific features using Unity's OpenXR. Is that all correct to say?

 

yes @m0nkeybl1tz exactly. Just one thing : the passthrough won't work from the editor for now. You need to build the project on the Quest.

 

@Yann_de_Couessin great info so far. So for further clarification, is passthrough method supported for Pico without the full pico integration as well. I am interested if this method works for hands and overlays too.

 

Hey Everyone
I'm trying to get passthrough working on the Quest 2 using the Unity OpenXR Meta package and the AR camera manager script on my Main Camera. The Camera has a solid color set to black (0 opacity) but still I cant see the passthrough when I build (not running in editor) to quest 2. Anyone has the same issue or might now what is still missing from the project?

I am using unity XR and XRI packages for the interactions.

I went to all official documentation but sadly this is still quite new and there is not that much out there yet.
Any help much appreciated!




Google ARCore is not enabled in the plugin providers since I read on official documentation that that should not be the case. If I enable it the game does the screen stays black and the game does not start up

 

Thanks for the quick reply!

I have edited my post with the XR plugin settings that I currently have.
Can you have a look as I think everything should be correct.

I am testing a build and not over link play mode

 

Not sure if it matters but I am running Unity on version 2023.2.0b16?

 

@AGeens Hi. Do you have any progress in passthrough issue? Because I have same problem (

 

You could try the Mixed reality template with Unity 2022.3!
I had no problems with the Passthrough with the Template.
Cheers J

 

Would you mind sharing a link to the template?

I just installed the XR SDK All-in-One plugin and updated my project but the passthrough is now black. This wasn't happening with the legacy plugin, I am trying to compare the two projects but I can't see any differnce, just that the passthough in the new Oculus plugin doesn't work.

 

Sorry to revisit this but I was hoping to clarify: do you need the Oculus plugin to enable passthrough, or can you do it with OpenXR + Unity Meta OpenXR?

 

I'm wondering this too, I've achieved passthrough on a number of projects, but only through the Oculus XR plugin (using an XRPassthroughLayer, and an OVRManager configured to allow passthrough) rather than OpenXR. I have an existing project that uses OpenXR that I want to add passthrough functionality to, but don't seem to be able to without switching from OpenXR to Oculus, and thereby breaking everything that's been designed to work with OpenXR

Is there a way to get passthrough working on an Oculus/Meta headset using just OpenXR? Or to somehow run both OpenXR and Oculus XR plugins at the same time? (OpenXR for input, and Oculus for passthrough)

Microsoft have their Open XR MR kit, but looks like it's set up to work with Hololense rather than Oculus, which makes sense..

 

There's nothing in the docs about HDR and Postprocessing breaking the passthrough layer. I'm having issues with this and Valem Tutorials (Youtube) says this is a bug that hopefully is getting fixed. Might be related to other users' issues.

 

Hi @ericprovencher Do you have any news on the availability of operation with the link on a Windows PC?
Thanks

 

Hi @ericprovencher, just wanted to ping the thread. Any news of supporting Meta 3 passthrough via Quest link cable?

 

Are you asking about passthru when in editor over link? I can confirm it works, but (at least on my computer) its pretty janky. The camera views are all distored and what not. But I can at least see my logic works (but then I disable it in editor and just leave it black so I don't get a headache)

 

I think it only works in editor link with Oculus plugin, not OpenXR. The distorted view I think happens when you recenter your playspace. If you never do that, it looks like normal passthrough, albeit lower resolution than on device.

 

Thanks for this. I followed this but I end up with nothing but blackness. If I set it to overlay then I only see my passthrough, but underlay comes through as just black. Could it be because of URP?

 

For anyone else looking. Following this tutorial got passthrough working for me in Overlay mode. Passthrough transparency with MRTK2 and 3 on Quest 2/Pro - DotNetByExample - The Next Generation (localjoost.github.io)

The next one from him in underlay mode did not work for me. Likely because of his tutorial is using MRTK while I am using XRTK.
Full underlay passthrough transparency with MRTK3 on Quest 2/Pro - DotNetByExample - The Next Generation (localjoost.github.io)

@ericprovencher any idea why I can't seem to get underlay working in the build with XR Origin? Black Screen.

 

When i set it to underlay mode: