Different content in each eye

Hi all,

We've been using the standard Oculus plugin for a while, and aside from the juddering problems caused by the most recent oculus runtimes, we've not had any issue with it.

We're keen to move to using Unity 5.1, but one thing we've come up against whilst looking to move is that there are no longer 2 cameras in the editor view. As such, situations where we're playing different content to both eyes (stereo video for example) no longer work.

Is there an obvious way to get around this that we're missing at the moment? There must still be a way to talk to the different eye cameras independently?

Thanks in advance for any help!

SB

 

I think it would be very useful if this was available as a Unity function, so that we could detect in OnPostRender which eye was actually rendering.

 

@thep3000 so in the case of stereo video, we would create 2 shaders, one for left eye one for right eye? And in the shader code we would essentially turn rendering on/off based on the uniform constant? I guess there are also other approaches depending on the details. Just trying to understand how we would use this shader level approach. Currently, I just use layers and set the culling mask on each camera as described here...
http://bernieroehl.com/360stereoinunity/

 

Hey thep3000,

Maybe...

We produce cross platform applications on Oculus, iOS and Android (in various forms), a chunk of which are stereo rendered movies. The format of these movies is L/R eyes top and bottom split. We then map these to spheres in our scene. With the old Oculus SDK, we simply changed the layers the cameras could see so that each eye saw the right section of the video on the appropriate sphere.

Would the shader allow us to do something similar? We really don't want to be hacking through all our old code to change this to a completely different methodology...

 

We need higher level. We need layers with Left and Right. This is how we do stereo movies. We have 1 video file with side by side frames, we have 2 objects in the same place, one for each eye, they have offset UVs.
We use different video decoding shades across 4 platforms. We are not going to hack 4 different 3rd party sharers to support stereo rendering flags.

 

I also need higher level. Some way to draw different elements to each eye, whether by moving their position for each eye, or a layer, or something. I did try just changing the positions of an object in the prerender, but this did not seem to create a difference between the two eyes.

 

@thep3000 Reiterating what others here have said. Our approach is also to use separate layers for each eye. So being able to set the culling mask for the different eye cameras would be ideal. Using shaders would only be partially successful for us.

 

Today I tried to port our main VR application to Unity 5.1. I couldn't find the rendermask option for each eye and hit this topic on Google.

I hope this can be integrated really soon, as we are one of the biggest companies in the Netherlands delivering a VR application for real estate companies and now our application can't be used with 5.1 since it relies mostly on cullingmask per eye. Currently I'll keep using 4.6 until rendermask is possible for each eye in the 5.x cycle.
But good to see that your guys notices it and are working on a solution.

 

@JDMulti, You can still do this with 5.x ... you just have to stick with your third party integration instead of using the new built-in VR support.

 

Yes I know. but the whole point of the change to 5.x is because of the default VR support. But you're also right, as 5.x is possible to use but then with sdk's.

 

Except that according to Oculus the old integration support for Unity is now legacy and will be discontinued at some point in the future. So workarounds such as that will not be working for long.

This situation is exactly what I feared with a 'first party' support for VR being integrated into Unity. Its a completely closed off system, meaning without using individual product SDK's and writing your own wrappers to Unity, we are now at the whims of the Unity release/patch schedule to fix over-sights such as not supporting stereoscopic video. Worse with VR being such a fast paced and mostly not standardized I foresee Unity VR easily lagging behind hardware and software updates, not to mention impossible for developers to address bugs themselves ( for example last years mess up with chromatic offsets in one of Oculus's releases, where I beleive people eventualy hex-edited dll's to fix).

This is one area where I wished UT had made the first party support a pure plugin and released the source on BitBucket like they've done with the Unity UI. That at least would allow developers some degree to fix things themselves.

 

Before that I hope they implemented the support for culling masks per eye. At the moment we really rely on this feature for our VR application that sells like crazy and I can't think about not having this before support of the SDK's drop. It would be a total disaster. The crisis in Europe did hit us hard and VR is pulling us out of this situation, sounds strange but it really does, without it.... no idea.

Is it true that features in patch releases are not on the Unity Roadmap? Would be nice to have a tab for patches as well besides the major big releases. Just an idea.

 

I expect it will work long enough to hold people over until Unity gets support for this built in. But my point was that you don't have to switch back to 4.6 as the same old integration still works in 5.x.

 

Oh I agree with your point, but mine was that its not going to be too long before the legacy installation is too outdated in itself for it to be practical as a workaround for issues like this and thus wishing that Unity VR support would be opened up.

 

We had a strategy for rendering one view to the Oculus and another view to the main monitor. We achieved this by rendering an ultra-wide view so half of it appears on your desktop, and half on the VR headset. It appears you can no longer do this with the UnityVR integration. UnityVR takes over all aspects of your camera. Can we get support for achieving our goal? Here's an article I wrote after GGJ2015 which explains our setup and how we achieved it.

 

Another great use case that an increasing number of games and experiences are exploring. Whilst it could be argued in some cases using networked machines ( or even just networked Unity instances on same machine) would solve this issue, in the example shown where so much content/data is clearly shared by both views it would definitely be nice to get native support for such a feature so it can run on a single machine.

However i'd argue rather than supporting the old style ( though extended monitor usage is still quite important) perhaps instead a better solution would be too leverage the already supported 'VR.VRSettings-showDeviceView'? In this case instead of showing the left eye view the function could be enhanced/overloaded to accommodate a camera reference or rendertexture, thus allowing developers to display any content on the monitor screen?

 

Thanks!

 

But in my Unity editor, when I check for updates in help menu,it shows "current version 5.1.2 f1 is up to date", then how to get the 5.1.2 p1 version? Thanks a lot!

 

@Ed unity:Thanks for you answer and hard work to fix this issue by unity staff!

 

I'm glad this is being worked on. I've been integrating 3D video of late and not have control over the eyes has made my job impossible in Unity 5.1.2.f1. Thankfully there is aldreay a patch and we're in good shape!

 

Been trying out this feature today. Seems to work when I push play although Unity will crash when I stop playing in editor. Also standalone apps crash on close. Seems to get fixed when I add a third camera that renders to both cameras. Still experimenting though.

 

On the off chance that anyone is interested, here's a component and shader which renders different content (red and blue fill in this case) to each eye. Hope it helps.

Attach the script to the Camera Rig's centerEyeAnchor and make sure that the shader is included in your project.

**Note** that I haven't used this with the patch mentioned earlier in this thread; this was the solution I came up with before the issue was officially addressed.

 

Could you clarify this for me? Am I creating a new camera, naming it centerEyeAnchor, adding the script to it, before adding the shader to a plane texture within the scene? I'm specifically looking to use this with stereo video.

Many thanks!

 

If you have added the Oculus OVRCameraRig to your scene, you'll see that it has a couple of sub-objects in the hierarchy, one of which is named "CenterEyeAnchor." You'll attach the script to that.

The script will automatically load the shader - you don't have to create a material, etc. If the script is attached to the CenterEyeAnchor - again, part of the OVRCameraRig, not something you'd create - everything "Should Just Work."

This is supposed to be a *VERY* basic example of showing different content for each eye.

 

If you want to use this for 3D video, and have a texture for each eye available, just Graphics.Blit() the appropriate eye texture in OnRenderImage() - you don't even need the shader. All you need to know is which eye is rendering. Note that, while Update() is called once per frame, OnPreRender() and OnRenderImage() are called twice per frame - once per eye. The example script I provided tracks which eye is currently being rendered with the m_currentEye private field. 0 == Left, 1 == right.

 

Thanks for the support! This is incredibly useful for integrating stereo video into CG scenes, similar to what Kite and Lightning are doing for their VR projects.

 

Okay, I misunderstood. If you already have a quad or something in your scene, and you just want to texture it differently for each eye, then you were pretty close:

Attach my script to the CenterEyeAnchor and, in the OnPreRender event, set the texture based on which eye is rendering. My note about Graphics.Blit() was based on the assumption that you wanted your video to occupy the whole screen.

I hadn't seen the Kite and Lightning stuff before - looks pretty slick.

 

With the Native dual camera setup in unity, is it possible to have a CenterEyeAnchor like solution? Because when I put both seperate cameras in unity in an empty game object, I don't know how to track the center between the two eyes. With the Oculus rig, this was easy to do, with the native unity dual camera setup I have no clue.

Someone have experience with this?

 

I haven't messed with the "native dual camera setup." However, it should be pretty simple to do this.

The OVR Camera rig dynamically creates a camera for each eye at runtime. For each frame, the engine:

1. Calls Update() for the scene
2. Calls OnPreRender(), OnRenderImage() for each eye.

My script just tracks which eye is currently being rendered, and adjusts rendering based on that. The variable "m_currentEye" indicates which eye is being rendered. In OnPreRender(), the line

m_currentEye = 1 - m_currentEye;

toggles the value of m_currentEye (0, 1, 0, 1, ...). That should be enough information for you to determine what to render and to which camera. In my case, I'm using a very simple shader that just fills the texture with red or blue.

I imagine that the native dual camera setup works in a similar fashion - two *Render() calls for each *Update(). If so, this script should serve as a usable starting point.

I'm attaching another version with the OVR references removed, since it wasn't being used in this case anyway, and the shader. Shader is unchanged from previous post.

If this isn't clear, feel free to email me (david lively at gmail, no spaces).

 

I am new in Unity and found this topic from Google. The version of Unity is updating so fast which made some information I searched on the internet to be completely useless. Even though the information was just a year ago.

 

I found this thread after looking for ways to render stereoscopic textures. I would be happy with accessing the eye index in the shader (as @thep3000 suggested back in June). That would be great for performance too. Will it appear any time soon?

 

Is there support for enable and disable headtracking else then pulling out the USB cable of the Oculus tracking camera? I can't find anything else then InputTracking.Recenter() to be used in update function, but I guess that's not a good idea.

 

This may be helpful:

https://forums.oculus.com/viewtopic.php?t=1792

 

I haven't seen built-in support for this, but see my example script and shader above. It's pretty easy to track which eye is rendering yourself. (Really - it's 5-10 lines of code and a global shader uniform.)

 

Yup, that's basically what I ended up doing. It would be better with a shader variable though, as then we'd only need one camera, and Unity only has to do frustum culling once.

 

One thing that I noticed: if you're running in the editor with the game view open, you may get three renders per update instead of two.

 

Update: just found that when moving head too quickly, the matrix value is not stable to indicate which eye is rendering.

Maybe it is documented elsewhere, but for the moment, I can not find any definitive answer on this.

The first step will be, find out which eye is rendering. @davidlively 's script give me a hint, and after adding log output and playing a lot with them, there is a simple way.

First, hook the onPreRender event of the CameraRig game object, then, check the "worldToCameraMatrix" member of the camera object transferred in, which is a Matrix4x4;

When rendering left eye,the matrix's member m03, will be a negative value; and vice vesa.

 

Hi thep3000. Currently I am using the two cameras with different Culling Masks and Target Eyes solution. However, I would like to profit from Unity's VR stereo optimisations. Is there plans to support this, for example by having a Culling Mask for each eye or some other solution?

Also, if I use davidlively solution (shader global int), will I still get the optimisations?

 

I've successfully have a stereo 360 video playing on the GearVR using the two cameras w/ culling mask solution described earlier. However, I notice quite a bit of ghosting/cross-talk around some edges that I do not see for the same video using the Oculus native video player. Any suggestions on how to diagnose/debug this issue? Both of my cameras are at 0,0,0 with each set to one specific eye, culling out all but the one sphere playing the 360 video. I'm tiling my video using the 0.5 tiling (and 0.5 offset for the under video)

 

Hi , I've created a stereo 360 video player for Gear VR using culling mask and 2 spheres with 0.5 offset for 1 video. I'm using the plug in Easy Movie Texture. It's not working for 4k videos on my Samsung s6, this way it will only play video on one sphere and on the other looks black. So I had to use 2 videos 2048x1024 for each sphere but it plays asynchronously, does anyone had the change to do something similar? Thanks.

 

I'm seeing the same issue. Did you ever figure this out?

 

In case anyone encounters this issue, it turned out to be having the eye images swapped.

 

Is there any way to render different content to each eye in Unity 5.6 native VR modes, such as GearVR, Daydream, Cardboard?

 

Yes there is, if you create a VR dual camera setup, you're able to use cullingmasks to mask certain content in certain camera's. You only need to set per camera which eye is being rendered. However notice for now that it's not recommend to use Vulkan API, as it has it flaws at the moment and will be fixed soon.

If you want some help with this, I could help you out. However, you don't need a shader to have that working as what you described.