Reflection Probes - Box Projection looks flat in VR

Hi all, I am working on a VR-project right now and am heavily utilizing reflection probes for my environments. The box projection method works well so far, at least when testing non-vr. But in VR-mode, the box-projection does not work as it should.

Both eyes seem to somehow see the same box-reflection image, therefor completely wiping out all depth that it is supposed to have. So when, e.g., looking into a mirror, the reflection does not look like a reflection (3D) - but more like a photo OF the reflection (2D) hanging on the wall, like the reflection is just a texture laying on top of anything - making it look very very weird. I hope I get the idea accross. (I would post a picture but of course that wouldnt help XD)

Does anybody know if that is an known issue or if I am doing something wrong? I am on unity 2022.1.0b10.2818 atm. Any help or feedback would be highly appreciated

cheers,
ANB_Seth

 

Any reason you're still on a beta of 2022.1? Because it's not a breaking change to move to the released version of 2022.1, just less bugs.

That will maybe not fix but should start the investigation from the best place.

 

thanks for the tip, no not a reason not to update, just wanted to let some time pass since I did break some stuff on an older project by updating (probably by mistakes on my end) and wanted to get some other stuff done in the project first.

However, since it seemd to be a blatantly obvious bug to still be around at all by now, I was and am having my doubts that a simple little update would fix that. I was hoping more of a settings issue I am just not seeing. But I will try updating later today anyway and report back.

 

ok I did upgrade to 2022.1.8f1 ... no change in the matter. As far as I can see the only available "newer" build is an "alpha" version, which I would not want to try atm. I also doubt that it would bring the change I need.

 

I would say it is working as intended A reflection probe without box-projection basically implies that the reflected light rays come from "infinity", meaning all depth information is lost. When correcting the reflection probe via something like a box-projection, the light rays come from the walls of this box..which also does not preserve depth information - instead of infinity, the rays now come from the walls of this box.

 

Well, not to grind too much into technicalities, but the box projection does change perspective depending on the angle of where I am looking at it, thus "simulating" depth, which is what it is there for in the first place. Now, all I would want is that this "perspective angle change" does also apply to the two positions/angles of my 2 VR-Cam-Eyes, which it does not. It seems to be only calculated and rendered once (left or right eye, I am not sure). There just has to be a way to fix that.

 

Box Projection "simulates" depth only insofar as - just like its name implies - projecting the cubemap image onto a three-dimensional box (as opposed to "regular" reflection probes, which essentially use an infinite sphere to project the image). This means that while the projection does anchor the image to a 3D space (thus allowing your perspective to change as you move the camera around), it is not actually providing any depth information about the image - in fact, it is impossible to recreate depth information from a single flat image.

As for VR specifically, sampling a box Projection does, in fact, use different angles for each eye! But just like looking at a flat painting (a projection of a 3D image onto a plane) confers depth information about the plane but not the image, looking at a Box Projection reflection (a projection of a 3D image onto a box) confers depth information about the box but not the cubemap image. If both eyes truly were using the same angle to sample the cubemap, the reflection would look infinitely far away (since the only time our brain sees two near-identical images for each eye is when looking at very far away objects). If the reflection looks flat but grounded, then the stereoscopic sampling is doing its job.

 

Ok, thanks for the explanation, but the stereoscopic sampling kinda is only doing "half" its job XD ...While the reflection image is projected correctly on top a the surface for each eye, it is only "one" reflection-image perspective that is being projected, not two as there should be, one for each eye. And that simply is not the way box projected reflections are supposed to work in VR.

Half-Life Alyx for example features the exact same box projection method (cubemap reflections) and they do work the only way it makes sense: each eye gets its own box-reflection image calculated based on the eye position and then rendered correctly on top of a surface and thereby simulating depth. In unity (or at least on my setup - still not sure where the error lies) it skips the first part of the job, so its only fetching one image ... which, granted, is then layered correctly on top of the surface for each eye.

And its not like the engine doesnt have all the info needed. If I where to make a screenshot of the eg. left eye once, then move slightly to the right and make a screenshot of the left eye again, then combine these two into an stereoscopic image, then this would get me the result I want (and need). Now I just need unity to do this correctly in the first place.

 

Ah, I see what you meen. The fact of the matter is that a Reflection Probe can only capture an image from a single point of view. Even if you sample that box-projected cubemap from different points of view, the result will still be flat since the original was only captured monoscopically.

I have never heard of any stereoscopic cubemap projection though. The issue really is that you would need to create new cubemaps every frame based on current eye positions, which would be prohibitively expensive as far as I know. The whole point of using a box projection was to bake a cubemap once and approximate the shift in image perspective by projecting it onto a box, so having to resample the cubemap(s) every frame in different positions kinda defeats the purpose. Having two arbitrary probes won't work either even if their image/sampling point is different - anything that doesn't align correctly with the eye's actual POV will inevitably look wrong, and Probes capture the scene from their POV, not the camera's.

Are you sure Alyx isn't using Planar Reflection instead? That is what you are describing in your last paragraph really, not Box Projection. You'd essentially render the reflection as seen from one eye and then the other, and merge them to sample a stereoscopic reflection with correct depth and everything. But that's done with cameras, not Reflection Probes (and in URP you'll have to code it yourself).

You could also try a hybrid approach like I did a while back in an older project: use a Box Projection for the room/area's boundaries for cheap (tho less accurate) far-off reflections, and overlay a Planar Reflection of only your dynamic/nearby objects. But a Box Projected Reflection Probe in Unity will definitely not give you any depth information about the reflected image - at least not without a bunch of custom work on your end.

 

Bumping this old thread as I would like to find a solution for this as well. There should not be any need of recalulating the data for the reflection probe, regardless of if it's a box or regular projection.

What we need is a way to sample the reflection based on not just the center eye, but different for the left and right eyes. Imagine you have a big sphere (the box projection), then the angle you look at it would be different for your left and right eye. In a 2D game this would be equal to moving the camera 54mm to the right and then back for each frame to capture each eye.

According to this old post, it was calculated different based on if it was forward or defered rendering:
https://communityforums.atmeta.com/t5/Unity-VR-Development/Reflection-Probes-in-VR/td-p/358725

There are some planar reflection examples, like this paid plugin: https://assetstore.unity.com/packag...effects/mirrors-and-reflections-for-vr-212493
but can't find anything about stereoscopic reflection probes in Unity.

I managed to create a working example in babylonjs if anyone are interested in how it should look (works in chrome with an open OpenXR runtime)
https://playground.babylonjs.com/#F41V6N#1360

So this is clearly something that is possible (even Godot has this out of the box)

Here is the related merge request that made that happen:
https://github.com/BabylonJS/Babylon.js/pull/10872/commits/51b2e07d5f8c86eb11102fe70973fc76072f7cea

 

Has anyone experienced some flickering lines artifacts while using box projection reflexion probes in Unity 2022 LTS URP? Does not happpen in the editor but in the headset (PICO 4)