Question Inference on camera image with HDRP

Hello!

I'm trying to use Barracuda to perform a custom pass on the camera image with the HDRP (think object detection running on top of the rendered image as a post-process step for the usecase).

Right now, I am able to run a convolution network in my custom pass (right now, I'm using a dummy random solo convolution for my tests), and store the output in a render texture that I'm displaying on screen using a raw image (see code below). However, I've two issues with this code:

- I'm unable to properly pass the camera color buffer. When the input texture is replaced with
ctx.cameraColorBuffer.rt, the output image is no longer correct (all the pixels have the same value).

- Same thing for the output, when I try to overwrite the camera color buffer with the tensor content, the screen image is not modified at all, like if the custom pass wasn't executed.

Current working code:

Code (CSharp):

1.     protected override void Execute(CustomPassContext ctx)
2.     {
3.         Tensor input = new Tensor(inputTexture, 3); // inputTexture is a Texture2D image that I set in the inspector
4.         m_worker.Execute(input);
5.         Tensor O = m_worker.PeekOutput();
6.         O.ToRenderTexture(outputRenderTexture); // outputRenderTexture is a render texture linked to a raw image for display purpose
7.         input.Dispose();
8.     }

What I tried:

Code (CSharp):

1.     protected override void Execute(CustomPassContext ctx)
2.     {
3.         Tensor input = new Tensor(ctx.cameraColorBuffer.rt, 3);
4.         m_worker.Execute(input);
5.         Tensor O = m_worker.PeekOutput();
6.         O.ToRenderTexture(ctx.cameraColorBuffer.rt);
7.         input.Dispose();
8.     }

If anyone has an idea, I'm not even sure this is possible, but that would be an interesting usecase.

(PS : my code also produces some warnings for some buffer created nowhere apparently, but I'm less concerned about that)

Code (CSharp):

1. Found unreferenced, but undisposed Tensor data which might lead to GPU resource leak: (GPU:untitled#860292400 (n:1, h:1080, w:1920, c:3) buffer: UnityEngine.ComputeBuffer created at: )
2. UnityEngine.Debug:LogWarning (object)
3. Unity.Barracuda.D:LogWarning (object) (at Library/PackageCache/com.unity.barracuda@3.0.0/Barracuda/Runtime/Core/Internals/Debug.cs:72)
4. Unity.Barracuda.ComputeTensorData:Finalize () (at Library/PackageCache/com.unity.barracuda@3.0.0/Barracuda/Runtime/Core/Backends/BarracudaReferenceCompute.cs:121)

 

Thank you for your answer @fguinier !

> verify that ctx.cameraColorBuffer.rt actually contain the data you want (for example rendertexture inspector)

I just did that with the following code, and the camera color image is correctly displayed in the raw image linked to the render texture, indicating the data are indeed what I want.

ctx.cmd.Blit(ctx.cameraColorBuffer, outputRenderTexture, RTHandles.rtHandleProperties.rtHandleScale, Vector2.zero, 0, 0);

> verify ctx.cameraColorBuffer.rt format & resolution match your expectaction

Using the following logs, I can see that the ctx.cameraColorBuffer.rt is a ARGBHalf texture with format R16G16B16A16_SFloat of size 1920x1080. The size is correct, but I'm not sure what I should do with the format information. Could this be the issue as the tensor expects float32 data? Is there a way to specify data format when creating the tensor?

Code (CSharp):

1. Debug.Log($"Format: {ctx.cameraColorBuffer.rt.format}");
2. Debug.Log($"Graphics Format: {ctx.cameraColorBuffer.rt.graphicsFormat}");
3. Debug.Log($"Width: {ctx.cameraColorBuffer.rt.width}");
4. Debug.Log($"Height: {ctx.cameraColorBuffer.rt.height}");

> mix both codepath above (from texture2d to HDRP, vs from HDRP to custom RT)

I've tried to mix stuff to see what works and what doesn't. Basically, using the ctx.cameraColorBuffer either for tensor creation or output ToRenderTexture doesn't work. However, it works if I set a render texture as the ToRenderTextuer target and then ctx.cmd.Blit it onto the ctx.cameraColorBuffer.

I don't really understand why, but I guess this is a workaround for the output at least (at the cost of an extra copy).

Code (CSharp):

1. Tensor input = new Tensor(inputTexture, 3);
2. m_worker.Execute(input);
3. Tensor O = m_worker.PeekOutput();
4. O.ToRenderTexture(outputRenderTexture);
5. ctx.cmd.Blit(outputRenderTexture, ctx.cameraColorBuffer, RTHandles.rtHandleProperties.rtHandleScale, Vector2.zero, 0, 0);
6. input.Dispose();

Error if trying to do O.ToRenderTexture(ctx.cameraColorBuffer) directly :

Compute shader (TextureUtils): Property (Otex2D) at kernel index (2): Attempting to bind texture as UAV but the texture wasn't created with the UAV usage flag set!