Unity native VR/XR: Scene Loading, Async, Custom Loading Screen

Hi,

I am trying once again to use only native Unity for our next VR project (up to now, we've always had to fall back to plugins, OVR and SteamVR for some reason or other). I am very happy with the new APIs for positions, buttons, and haptics. One thing I have not seen yet is the following:

Loading a more complex scene (yes, even asynchronously) consistently leads to a situation where no frames are sent to the VR compositor, causing it to fall back into the compositor screen, which looks bad. There are at least two solutions:

• Fixing asynchronous scene loading to always send frames (that would be so cool!), or
  
• use the same workaround as in the OVR and the SteamVR plugin, which is explicitly setting an in-headset, static loading screen:

In OVR:

Code (CSharp):

1. OVRPlugin.SetOverlayQuad(isOverlayInsteadOfUnderlay, isHeadLocked, nativeTexture, nativeTexture, IntPtr.Zero, pose.flipZ().ToPosef(), scale.ToVector3f());

In SteamVR:

Code (CSharp):

1. OpenVR.Overlay.CreateOverlay(key, overlayName, ref loadingScreenOverlayHandle);

This works quite well but unfortunately, only with the plug-ins. Any chance either of these solutions or a completely new one is available / planned / drafted / thought about / already dismissed / buried in coal / rediscovered / served for dinner for native Unity XR support? ;-)

Philip

 

@StayTalm_Unity might I tempt you into looking into this? ;-)

 

@StayTalm_Unity Thanks! it would be really great to know what the take of the team is on this one, so if you can get someone to answer it would be very much appreciated. It would be really cool to be able to get rid of all the platform-specific stuff.

Philip

 

Thanks @StayTalm_Unity , again, for your very helpful answers. Just to clarify, what exactly is on the roadmap you mentioned? A way to set a static loading screen, or having the async loader always sending frames?

And second, what would you recommend we do in the meantime? Use the platform-specific API, or go some different way (everything in one scene, for example, or asset streaming,...or?)

Philip

 

I would really like this as well as I'm not wanting to import the SteamVR asset into my project as it just adds another layer to it.

 

@plmx By the way plmx...

Code (CSharp):

1. OpenVR.Overlay.CreateOverlay(key, overlayName, ref loadingScreenOverlayHandle);

How do you go about using this code? Can this be used to just put a black screen instead of the compositor?

 

That piece of code is part of the OpenVR/SteamVR plugin available on the asset store. There is an example in the asset which shows you how to use it. Basically, you need a Texture2D in your project with the correct import settings to which you'll need to retrieve a handle that you pass into that function.

The overlay function can be used to prevent anything (including the compositor) from showing, so yes, by just using a black texture you'll get a black screen.

Philip

 

Oh damn then that's exactly what I need, thank you so much plmx. I have had the asset installed. So the part that I haven't learned how to do is the 'retrieve a handle' part. Getting the texture in is no problem.

 

@plmx Two things.

Firstly, for what you want can't you rip the code directly from the Steam VR Unity plugin, leaving out the parts of the code you don't need so that it's more lightweight without all the extra stuff? (isn't OpenVR within the Steam VR Unity asset a single file?)

Secondly, is this the method I'm looking for to retrieve the handle to a texture?:

Code (CSharp):

1. Texture.GetNativeTexturePtr

EDIT: I think I found it. After you use CreateOverlay you use SetOverlayTexture.

Code (CSharp):

1. var texture = new Texture_t();
2.                         texture.handle = renderTexture.GetNativeTexturePtr();
3.                         texture.eType = SteamVR.instance.textureType;
4.                         texture.eColorSpace = EColorSpace.Auto;
5.                         overlay.SetOverlayTexture(myOverlayHandleGoesHere, ref texture);

 

Regarding your first point: You can get rid of most of the elements in the SteamVR plugin, but not all of it; besides the DLL access point (openvr_api) I seem to recall that some of the basic classes were useful (at least SteamVR.cs). You'll have to check which code is required for your overlay functionality, you can strip out the rest.

Regarding your second point: Yes, that seems about right. You can control some more aspects of the overlay such as alpha, size, etc. by calling more functions, but the basics seem to be there.

One major gotcha I recall is that the textures must be set to a certain format for this, I think it was Sprite mode and the RGBA 32 bit format in the texture import settings. Otherwise you'll just get black, which again, in your case, doesn't matter. ;-)

 

@plmx I feel like the best solution would be to figure out how to cut down the Steam VR Asset store plugin as much as possible. Personally, I can't stand how any of these VR packages like the Oculus Utilities or Steam VR plague you about 'automatically setting your settings to be ideal'. What if you end up implementing both the OVR and the OpenVR apis.

There should be advanced cut down versions of the asset store plugins to accommodate advanced users. In reality, there are gonna be things inevitably about creating VR games that aren't plug and play no matter what you do.

Thank you very much for the solution gonna test it all out tonight and tell you how it goes.

 

@plmx Wow, NOW I see what you're saying. My OpenVR app was working great, and I even had the Steam VR asset plugin installed.

But...now all of the sudden when I playtest the application it says "It looks like you haven't generated actions for SteamVR yet".

Well, I DON'T WANT TO GENERATE ANY ACTIONS FOR STEAMVR! If I'm to make my application to work with as many VR devices as possible, I don't want to be adding all these OVR assets and SteamVR assets and Vive and this and that form the asset store and bulk up the project. Not Unity devs fault of course, just want to stick with OpenVR and try exporting as well to Oculus Quest in the future. All this auto-setup stuff is killing me. With that said, very thankful for how Unity has implemented the apis into Unity, but it was all working perfectly just this morning and now this. All just to get the Steam compositor to not show when the application hangs.

The problem is now, I want to be able to generate actions but want to be able to then REVERSE that process, which hopefully is possible without Steam VR putting all this extra crap into my program, so I see exactly why @plmx wanted to just use the native Unity to accomplish this without using OVR or the Steam VR asset.

I'm making a project in where I have literally a single bool called 'VRMode' that makes the application VR, so I can develop and deploy to multiple platforms as fast as possible...

 

Okay so here is your cookie cutter solution for if you don't want to have the whole mess of the Steam API in your project creating it's inputs and it's key files or whatever it does. I've cut the Steam API for loading levels only down into two files that you can find here:

https://tinyurl.com/y57vo42g - Open VR API
https://tinyurl.com/yy4l4p3v - SteamVR_LoadLevel simplified and with SteamVR and SteamVR_Settings mashed into it, with modded code so that it just displays a black screen over the compositor so you never see the compositor.

Please test this and see if this works for you, it should just have it cut to black instead of showing the compositor.

You still have to use this of course:

Code (CSharp):

1. SteamVR_LoadLevel(string sceneName);

Test it like this:

Code (CSharp):

1.  
2. using Valve.VR;
3. using UnityEngine;
4. using System.Collections;
5.  
6. public class switchscene : MonoBehaviour
7. {
8.     IEnumerator Start()
9.     {
10.         yield return new WaitForSeconds(1);
11.         Debug.Log("3");
12.         yield return new WaitForSeconds(1);
13.         Debug.Log("2");
14.         yield return new WaitForSeconds(1);
15.         Debug.Log("1");
16.         yield return new WaitForSeconds(1);
17.         Debug.Log("LoadLevel");
18.         //Remember that this is also async:
19.         SteamVR_LoadLevel.Begin("MY SCENE NAME GOES HERE",false,0f,0, 0, 0, 1);
20.     }
21. }

That's for loading levels...well..unfortunately, I don't think there is a solution for if your application drops in frames. I'm going to see if I can just detect it and then intentionally myself kick it into the compositor before it can automatically do it so that I can create the overlay myself. Otherwise, I guess you just can't have fps drops no matter what, everything must absolutely be completely asynchronous as much as possible.

However...there MAY be a way thanks to this little number included in the Unity API:

Code (CSharp):

1. XRStats.TryGetDroppedFrameCount

I think I read that Steam VR will kick you into the compositor when it detects just 10 dropped frames. It might be possible to pre-maturely kick it into the compositor before those 10 dropped frames and in that way set up the black screen ahead of time....but I doubt it's possible to be honest because Steam VR does it automatically. Not sure.

I'm really hoping as I test it that the OpenVR API file included in the Steam API doesn't actually override Unity OpenVR's input settings, as literally when I made my project OpenVR compatible, set up my inputs, everything was working until I imported the Steam API in with all it's extra mess. Will report back here to see if that's the case.

EDIT: The Steam VR Skybox is supposed to be able to do it but it doesn't seem to work. Here's the code cut down. I created a texture 256x256, named it "BLACK.png" and put it inside a folder I named 'Resources/System/'. I also made it uncompressed and turned off mipmaps.

Code (CSharp):

1. Texture blackTex = (Texture)Resources.Load("System/BLACK");
2.  
3.             var compositor = OpenVR.Compositor;
4.  
5.             if (compositor != null)
6.             {
7.                 var handles = new Texture[] {
8.                     blackTex,
9.                     blackTex,
10.                     blackTex,
11.                     blackTex,
12.                     blackTex,
13.                     blackTex
14.                 };
15.                 var textures = new Texture_t[6];
16.                 for (int i = 0; i < 6; i++)
17.                 {
18.                     textures[i].handle = (handles[i] != null) ? handles[i].GetNativeTexturePtr() : System.IntPtr.Zero;
19.                     textures[i].eType = SteamVR.instance.textureType;
20.                     textures[i].eColorSpace = EColorSpace.Auto;
21.                 }
22.                 var error = compositor.SetSkyboxOverride(textures);
23.                 if (error != EVRCompositorError.None)
24.                 {
25.                     Debug.LogError("<b>[SteamVR]</b> Failed to set skybox override with error: " + error);
26.                     if (error == EVRCompositorError.TextureIsOnWrongDevice)
27.                         Debug.Log("<b>[SteamVR]</b> Set your graphics driver to use the same video card as the headset is plugged into for Unity.");
28.                     else if (error == EVRCompositorError.TextureUsesUnsupportedFormat)
29.                         Debug.Log("<b>[SteamVR]</b> Ensure skybox textures are not compressed and have no mipmaps.");
30.                 }
31.             }

The only other thing I could suggest and the obvious thing like you've suggested is to just make everything completely and utterly asynchronous. I have a pretty bulky project so it's a complicated issue, but for anyone designing a VR game that's not a port it's probably not gonna be such a big deal to design their project around being asynchronous to begin with.

The problem is, SceneManager.LoadAsync is dropping frames for me so there's no way like you said I can't use the Steam VR plugin. In my humble opinion Steam should seriously just have an option for developers like a bool that just shows black for the compositor. Same thing with the OVR plugin. Also, there need to be minimalistic versions of these assets without all the added extra stuff that some developers prefer to do themselves. We're programmers, having less drag and drop can actually make development much faster.

Personally for porting my own project, I make use of Resources.Load all over the place and I think in like 50,000 lines of code only use an IEnumerator like once. There are two places in particular where it lags, so it's gonna be a pain in the ass getting it all async because I have to make more states.

For anyone reading this be warned about VR development, design async.

EDIT: And here's my final update to this...basically, I pinned down the program kicking me into the compositor to a single line of code, and it's literally just pouring one structure/class into another. It's not Resources.Load and there's no way to actually make the process asynchronous. Great...I'm gonna try writing to valve about it, for the time being it just seems like data heavy games will just have this happen to them with no way out.

 

Any update on the addition of this functionality?

I noticed the logic sort of already exists with the Unity default splash screen in VR?

 

Also am looking for a solution for cross platform compositor layers

 

At this point, I'd say it's just made out to be really tough. In reality, Steam should have just made it where you could have the screen go black or something.

The only tip I would have, is that you absolutely need to have everything in your VR application asynchronous at all times, never never never have one part that locks up, because then it spits you out into the Steam compositor.

 

And so, we are in 2020, and still the one thing that every VR application ever built in Unity has in common is significantly underdeveloped.

Anyone have an update on how to do this right?

 

I'd like an update on this as well. I've gotten it to work with the old OVRCameraRig, but I've made many tries with the new XR Rig and just can't get the overlay to work with it.

 

Bump

 

It's funny, I'm again deciding that the only solution is single-scene design. Just make the whole game in one scene, and activate things as appropriate. Really unfortunate as scenes make things *way* easier but it's certainly not the end of the world, and there's plenty of ways to keep it organized still.