Effects in AR

I am currently working on a project in AR (Hololens) where we need to create a distortion effect that affects the real world around it. Currently the object's distortion is only visible when another game object is behind it. The project is proprietary so I can't share any more details about the application. Essentially i have a 3d object with a glass shader that distorts anything behind it. But when placed in the scene with no other game objects, its just invisible. Is there any way I can have this distort/affect the real world behind it? I tested in Unity and then with VR and it looked/acted great. It wasn't until I transferred it to AR that I'm having an issue. This is my first AR project so and advice or help would be appreciated.

 

I believe the issue is that the only thing that the shader can distort is whatever is being rendered behind it; since the HoloLens do not render anything behind your object because the real world is already visible separately from the device due to the glasses' transparency, there are no pixels the shader can use to distort the image.

I can see two ways to workaround this:
1- Get what the device cameras see and display it as a texture behind your object.
2- Fake a distortion effect with lots of overlays of animated semi-transparent waves that could deceive the user, probably making use of the spatial mapping feature to get more accurate data of how this effect would behave in front of these objects shapes.

Hope that helps!

 

Alright, so essentially, without a forward facing camera, there is no way to achieve this? I understand that the virtual object distorts pixels rendered behind it but since the real world isn't tracked there aren't pixels to distort. So probably stacking distorted waves would have the same effect I'm currently getting....not visible. Right?

 

If you use the same shader, yes, nothing will be shown. What I was suggesting was to draw a simple animated effect on top of the real world, standard shaders that support transparency.

On the other hand, it seems that you can get what the camera sees with this plugin, so it would be a matter of only drawing this texture behind your object with the distortion shader for it to work as intended.

 

Hi @arybak89,

Another option you could consider is using the camera on the HoloLens to take a picture of the real world and then render that to a full-screen quad which could serve as your scene's backdrop. Then you could distort the backdrop. However, keep in mind that using the HoloLens camera for this operation comes at a performance cost.
https://docs.unity3d.com/Manual/windowsholographic-photocapture.html

https://forum.unity.com/threads/holographic-photo-blending-with-photocapture.416023/#post-2726485