Depth map calculation from Perception

Hello, I am investigating using Perception for segmentation and depth map calculation simultaneously in a similar fashion to this paper: https://arxiv.org/abs/1910.00541. I was wondering if is possible to get ground truth for depth maps from objects in frame in the simulation? I've probably explained this poorly, but hopefully someone will understand what I mean

 

Hello,

I'm struggling to get depth to work with this package. My project is using URP. I can get he camera grab script to work, but I need depth images. I've found a script called Depth Grab, but haven't found any instructions for the script. The output I am getting from it is a weird yellow color. It appears like the camera clipping plane is short. I don't see any settings letting me set the clipping plane distance, which would be important for translating the image values into depths.

 

Update on my issue here. My issue using depth grab is weirder than I thought. I've found that actually what is happening is that Depth Grab is generating an image made up of only objects which have a transparent material. That transparent material is captured in full color, suggesting that there is no depth capture happening at all. In the image below you can see I have set one of my materials to transparent (its supposed to be yellow shiny paint). That is the only thing that appears in the image, which is supposed to be a black and white depth image.

 

I found a solution, though I don't know why it works. The issue was the skybox. In the camera component I switched the background type for skybox to default. Suddenly, everything works fine. So my problem is solve. I am curious to know why this fixes things though.

 

@d_pepley I am no expert, but I am going to work with Perception Tutorials shortly... I appreciate you sharing your experience.

I know that rendering each frame --processes things in a specific order -- based upon the render pipeline that is selected. Perception Tutorial says we can use URP or HDRP, but I want to work in URP like you are doing.

In URP, the Base Camera is necessary to set the environment rendering, which is where we find the Skybox toggle... per this link Camera component reference | Universal RP | 10.3.2 (unity3d.com) If it is not set with a skybox or default color...things might get weird.

But you can stack cameras, adding Overlay cameras...which are easy to control. See that link for more info. I found them very useful in the Machine Learning Tutorials I have done...

In ML, the Raycast from an Overlay camera was used to detect distance and clipping was easy to control.
Those tutorials might allow you to explore scripting and settings on cameras and renders. ( I enjoyed the Hummingbird Tutorial)

Also noted this blog Understanding the Importance of Using Multiple Cameras in Unity (theknightsofunity.com)

Best Regards.

 

I think I have determined why the transparent objects don't work. I believe it has to do with how transparent objects don't write to the Z-Buffer. As a result, they must just rendered normally in whatever process is running the depth image.

 

@d_pepley How did you do it? I am only getting a white image. My Configuration is below. Kindly help.

 

My best guess is its your clipping planes. I'd guess your far plane should be something much smaller (depends on your scene of course) so try messing with the near far range. Also I'd lower short to like 0.001.

You are using 8bit here for depth, so for your range you have set (0.3 to 100m). The value of each greyscale color is then (100-0.3)/256. That's fairly low precision. 16bit and 32bit can help improve precision, but you will need a way to translate the raw image into something visible.

 

@d_pepley could you please share the camera Grab script? Where did you find it? I am unable to either find that or the Unity Simulation Capture Package @RuiyuZhang_Unity mentioned above.

I have an interior scene and I am trying to generate the depth map as I fly through. I will save these frames.

Thank you for the help.

 

Hi,

any updates on this? Will the perception package have a depth camera labeler? When?
I think it is a useful feature in many robotics applications.

Thank you