360 photo or video with Depth

Hi,
I'm a complete newbie in Unity, so bear with me! Here's what I'm trying to accomplish. We've all seen 360 video, and even "3D" stereo video is underwhelming after a while, mainly because the whole world is stuck to your head- you can't move around at all. I want to experiment with projecting the 360 image onto a sphere and using a depth map to displace the sphere, giving the image some geometry.

We've shot 360 video footage on the Kandao Obsidian R camera. Their software allows us to output a 360 depth map based on the images, so that part is taken care of.

I've managed to get the displacement setup in Cinema 4D, which I'm very comfortable with, and it looks promising. I can't figure out, however, how to do this in Unity. I've read about displacement verses heightmaps and I'm just a bit confused on how to do it. Any pointers would be greatly appreciated!
P.S. Right now I'm working with a still image, but ultimately I want this to work with video/image sequence. Is this possible?

 

I'm wondering the same. Would love to hear some solutions on working with 360 video in Unity.