Question Linear render texture from video player returns unexpected values

Hi,
I encode grayscale PNGs containing disparity/depth information in 8-bit as a H.264 video and need to fetch the grayscale value as accurately as possible in my custom shader. For the playback I use the default Unity video player in the render texture mode.

My problem is that the grayscale values from the render texture are far off from the original values. I assume it's some kind of linear/gamma color space issue although I set the RenderTextureReadWrite parameter of my render texture to linear.

I made a simple grayscale video (see attachment) for testing purpose to visualize my problem. The video contains five grayscale areas (0, 32, 64, 128 & 255) which I map to color values in my shader to check the grayscale value from the texture fetching. Magenta means incorrect/unexpected values:


Can somebody please help me to understand what happens here? I appreciate any help.

Thanks in advance.

---

How I encode the PNGs with FFmpeg:

Code (CSharp):

1. ffmpeg.exe -r 30 -f image2 -i frame_%04d.png -dst_range 1 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -color_range pc -vcodec libx264 -pix_fmt yuv420p -preset slow -tune animation -x264-params qp=1 unity_grayscale_video_avc_qp1.mp4

How I create the render texture and video player:

Code (CSharp):

1. using UnityEngine;
2.  
3. public class VideoTest : MonoBehaviour
4. {
5.     public string VideoFile;
6.  
7.     private UnityEngine.Video.VideoPlayer VideoPlayer;
8.     private RenderTexture VideoTexture;
9.  
10.     int VideoWidth = 640;
11.     int VideoHeight = 480;
12.  
13.     void Start()
14.     {
15.         // RenderTexture
16.         VideoTexture = new RenderTexture(VideoWidth, VideoHeight, 0, RenderTextureFormat.ARGB32, RenderTextureReadWrite.Linear);
17.         VideoTexture.filterMode = FilterMode.Point;
18.         VideoTexture.wrapMode = TextureWrapMode.Clamp;
19.         VideoTexture.Create();
20.  
21.         // VideoPlayer
22.         VideoPlayer = gameObject.AddComponent<UnityEngine.Video.VideoPlayer>();
23.         VideoPlayer.playOnAwake = false;
24.  
25.         VideoPlayer.renderMode = UnityEngine.Video.VideoRenderMode.RenderTexture;
26.         VideoPlayer.targetTexture = VideoTexture;
27.  
28.         VideoPlayer.url = VideoFile;
29.         VideoPlayer.frame = 0;
30.         VideoPlayer.isLooping = true;
31.  
32.         VideoPlayer.prepareCompleted += PreparedFinished;
33.  
34.         VideoPlayer.Prepare();
35.         VideoPlayer.Play();
36.     }
37.  
38.     void PreparedFinished(UnityEngine.Video.VideoPlayer vp)
39.     {
40.         // Create mesh (one vertex per pixel) with custom material
41.         // ...
42.     }
43. }

My custom shader for testing purpose:

Code (CSharp):

1. Shader "Custom/VideoRendering" {
2.     Properties {
3.         _VideoTexture ("Video texture", 2D) = "white" {}
4.     }
5.     SubShader {
6.         Pass {
7.             CGPROGRAM
8.             #pragma vertex vert
9.             #pragma fragment frag
10.            
11.             #include "UnityCG.cginc"
12.            
13.             uniform sampler2D _VideoTexture;
14.  
15.             struct v2f {
16.                 float4 position : SV_POSITION;
17.                 float2 uv : TEXCOORD0;
18.             };
19.  
20.  
21.             v2f vert(float4 vertex:POSITION, float2 uv:TEXCOORD0) {
22.                 v2f o;
23.                 o.position = UnityObjectToClipPos(vertex);
24.                 o.uv = uv;
25.                 return o;
26.             }
27.  
28.             fixed4 frag(v2f i) : SV_Target {
29.                
30.                 float4 color = tex2D(_VideoTexture, i.uv);
31.  
32.                 float grayValue = color.x;
33.                 grayValue *= 255.0f;
34.                
35.                 if (grayValue >= 0.0f && grayValue <= 1.0f) { // 0 -> black
36.                     color = float4(0.0f, 0.0f, 0.0f, 1.0f);
37.                 }
38.                 else if (grayValue >= 31.0f && grayValue <= 33.0f) { // 32 -> red
39.                     color = float4(1.0f, 0.0f, 0.0f, 1.0f);
40.                 }
41.                 else if (grayValue >= 63.0f && grayValue <= 65.0f) { // 64 -> blue
42.                     color = float4(0.0f, 1.0f, 0.0f, 1.0f);
43.                 }
44.                 else if (grayValue >= 127.0f && grayValue <= 129.0f) { // 128 -> green
45.                     color = float4(0.0f, 0.0f, 1.0f, 1.0f);
46.                 }
47.                 else if (grayValue >= 254.0f && grayValue <= 256.0f) { // 255 -> white
48.                     color = float4(1.0f, 1.0f, 1.0f, 1.0f);
49.                 }
50.                 else { // not found -> magenta
51.                     color = float4(1.0f, 0.0f, 1.0f, 1.0f);
52.                 }
53.  
54.                 return color;
55.             }
56.             ENDCG
57.         }
58.     }
59. }

 

Thank you very much for your response. I'm using the UR pipeline with linear color space.

I added the gamma to linear conversion to my shader:

Code (CSharp):

1. float4 color = tex2D(_VideoTexture, i.uv);
2. color = float4(GammaToLinearSpace(color.rgb), color.a);
3.  
4. float grayValue = color.x;
5. grayValue *= 255.0f;
6.  
7. // ...

This is what I get now:


As you can see the grayscale areas of 32 & 64 are becoming black now, which means their corresponding values are pulled down to zero using the GammaToLinearSpace function. I think, that a gamma curve is being applied here using this function (which does not make any sense to me).

Out of curiosity I also tried the LinearToGammaSpace function:


This result surprised me even more because it's closer to my expectation (except for grayscale area 32 which is still off). But I don't understand the way how Unity handles the color space transformations here.

Do I misunderstand something here and is this a correct behaviour of Unity?

 

Ok, I got it now. Thanks for the explanation.

The offset of ~3 for the red area (grayscale 32) is still huge in my opinion (and too inaccurate for my use case). I also tried the more precise conversion function LinearToGammaSpaceExact, but without any improvement. It seems like Unity is already using the approximation functions beforehand, so no chance for me to get better results.

After comparing the results of the exact and approximate conversion function, I agree with you, that it is indeed a precision issue especially for RGB values below ~70:


For now I don't see any other solution than switching completely to the global gamma color space without losing accuracy.

 

Unfortunately not. I'm not using the VideoClip object because I need to be able to load the Video dynamically as URL outside from the project's assets folder.

 

Thanks for showing this little hack. I will definitely try this.