Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. We have updated the language to the Editor Terms based on feedback from our employees and community. Learn more.
    Dismiss Notice

Question Linear render texture from video player returns unexpected values

Discussion in 'Audio & Video' started by jas789, Jun 24, 2022.

  1. jas789

    jas789

    Joined:
    Jun 24, 2022
    Posts:
    5
    Hi,
    I encode grayscale PNGs containing disparity/depth information in 8-bit as a H.264 video and need to fetch the grayscale value as accurately as possible in my custom shader. For the playback I use the default Unity video player in the render texture mode.

    My problem is that the grayscale values from the render texture are far off from the original values. I assume it's some kind of linear/gamma color space issue although I set the RenderTextureReadWrite parameter of my render texture to linear.

    I made a simple grayscale video (see attachment) for testing purpose to visualize my problem. The video contains five grayscale areas (0, 32, 64, 128 & 255) which I map to color values in my shader to check the grayscale value from the texture fetching. Magenta means incorrect/unexpected values:
    unity_video_player_grayscale_mapping.jpg

    Can somebody please help me to understand what happens here? I appreciate any help.

    Thanks in advance.

    ---

    How I encode the PNGs with FFmpeg:
    Code (CSharp):
    1. ffmpeg.exe -r 30 -f image2 -i frame_%04d.png -dst_range 1 -color_primaries bt709 -color_trc bt709 -colorspace bt709 -color_range pc -vcodec libx264 -pix_fmt yuv420p -preset slow -tune animation -x264-params qp=1 unity_grayscale_video_avc_qp1.mp4

    How I create the render texture and video player:
    Code (CSharp):
    1. using UnityEngine;
    2.  
    3. public class VideoTest : MonoBehaviour
    4. {
    5.     public string VideoFile;
    6.  
    7.     private UnityEngine.Video.VideoPlayer VideoPlayer;
    8.     private RenderTexture VideoTexture;
    9.  
    10.     int VideoWidth = 640;
    11.     int VideoHeight = 480;
    12.  
    13.     void Start()
    14.     {
    15.         // RenderTexture
    16.         VideoTexture = new RenderTexture(VideoWidth, VideoHeight, 0, RenderTextureFormat.ARGB32, RenderTextureReadWrite.Linear);
    17.         VideoTexture.filterMode = FilterMode.Point;
    18.         VideoTexture.wrapMode = TextureWrapMode.Clamp;
    19.         VideoTexture.Create();
    20.  
    21.         // VideoPlayer
    22.         VideoPlayer = gameObject.AddComponent<UnityEngine.Video.VideoPlayer>();
    23.         VideoPlayer.playOnAwake = false;
    24.  
    25.         VideoPlayer.renderMode = UnityEngine.Video.VideoRenderMode.RenderTexture;
    26.         VideoPlayer.targetTexture = VideoTexture;
    27.  
    28.         VideoPlayer.url = VideoFile;
    29.         VideoPlayer.frame = 0;
    30.         VideoPlayer.isLooping = true;
    31.  
    32.         VideoPlayer.prepareCompleted += PreparedFinished;
    33.  
    34.         VideoPlayer.Prepare();
    35.         VideoPlayer.Play();
    36.     }
    37.  
    38.     void PreparedFinished(UnityEngine.Video.VideoPlayer vp)
    39.     {
    40.         // Create mesh (one vertex per pixel) with custom material
    41.         // ...
    42.     }
    43. }

    My custom shader for testing purpose:
    Code (CSharp):
    1. Shader "Custom/VideoRendering" {
    2.     Properties {
    3.         _VideoTexture ("Video texture", 2D) = "white" {}
    4.     }
    5.     SubShader {
    6.         Pass {
    7.             CGPROGRAM
    8.             #pragma vertex vert
    9.             #pragma fragment frag
    10.            
    11.             #include "UnityCG.cginc"
    12.            
    13.             uniform sampler2D _VideoTexture;
    14.  
    15.             struct v2f {
    16.                 float4 position : SV_POSITION;
    17.                 float2 uv : TEXCOORD0;
    18.             };
    19.  
    20.  
    21.             v2f vert(float4 vertex:POSITION, float2 uv:TEXCOORD0) {
    22.                 v2f o;
    23.                 o.position = UnityObjectToClipPos(vertex);
    24.                 o.uv = uv;
    25.                 return o;
    26.             }
    27.  
    28.             fixed4 frag(v2f i) : SV_Target {
    29.                
    30.                 float4 color = tex2D(_VideoTexture, i.uv);
    31.  
    32.                 float grayValue = color.x;
    33.                 grayValue *= 255.0f;
    34.                
    35.                 if (grayValue >= 0.0f && grayValue <= 1.0f) { // 0 -> black
    36.                     color = float4(0.0f, 0.0f, 0.0f, 1.0f);
    37.                 }
    38.                 else if (grayValue >= 31.0f && grayValue <= 33.0f) { // 32 -> red
    39.                     color = float4(1.0f, 0.0f, 0.0f, 1.0f);
    40.                 }
    41.                 else if (grayValue >= 63.0f && grayValue <= 65.0f) { // 64 -> blue
    42.                     color = float4(0.0f, 1.0f, 0.0f, 1.0f);
    43.                 }
    44.                 else if (grayValue >= 127.0f && grayValue <= 129.0f) { // 128 -> green
    45.                     color = float4(0.0f, 0.0f, 1.0f, 1.0f);
    46.                 }
    47.                 else if (grayValue >= 254.0f && grayValue <= 256.0f) { // 255 -> white
    48.                     color = float4(1.0f, 1.0f, 1.0f, 1.0f);
    49.                 }
    50.                 else { // not found -> magenta
    51.                     color = float4(1.0f, 0.0f, 1.0f, 1.0f);
    52.                 }
    53.  
    54.                 return color;
    55.             }
    56.             ENDCG
    57.         }
    58.     }
    59. }
     

    Attached Files:

  2. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    Hi. Thank you for the clear explanation. I don't see any issue with your code or shader. I created a clip using your command and using a colour picker, I confirmed that FFplay had the same colour. In the Editor, I checked the thumbnail for the colour, and it was still the same. I tried with everything together and I got this result
    upload_2022-6-27_17-44-8.png
    This is the code to create my RenderTexture
    Code (CSharp):
    1. rendTex = new CustomRenderTexture(rendTexWidth, rendTexHeight, RenderTextureFormat.ARGB32, RenderTextureReadWrite.Linear);
    2. rendTex.initializationMode = CustomRenderTextureUpdateMode.OnDemand;
    3. rendTex.initializationSource = CustomRenderTextureInitializationSource.TextureAndColor;
    4. rendTex.Create();
    I am not on URP or HDRP but I was in gamma space and when I switched to linear I got this
    upload_2022-6-27_18-1-22.png
    It seems to be indeed an issue with the color space. I am not sure yet if this is an issue on our side.
     
    Last edited: Jun 28, 2022
  3. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    Ok we do a conversion between Linear and Gamma.
    Code (CSharp):
    1.         inline fixed4 AdjustForColorSpace(fixed4 color)
    2.         {
    3. #if defined(UNITY_COLORSPACE_GAMMA) || !defined(ADJUST_TO_LINEARSPACE)
    4.             return color;
    5. #else
    6.             return fixed4(GammaToLinearSpace(color.rgb), color.a);
    7. #endif
    8.         }
    So you probably can do a conversion back or tune your value for linear space.
     
    Last edited: Jun 28, 2022
  4. jas789

    jas789

    Joined:
    Jun 24, 2022
    Posts:
    5
    Thank you very much for your response. I'm using the UR pipeline with linear color space.

    I added the gamma to linear conversion to my shader:
    Code (CSharp):
    1. float4 color = tex2D(_VideoTexture, i.uv);
    2. color = float4(GammaToLinearSpace(color.rgb), color.a);
    3.  
    4. float grayValue = color.x;
    5. grayValue *= 255.0f;
    6.  
    7. // ...
    This is what I get now:
    unity_video_player_gamma_to_linear.jpg

    As you can see the grayscale areas of 32 & 64 are becoming black now, which means their corresponding values are pulled down to zero using the GammaToLinearSpace function. I think, that a gamma curve is being applied here using this function (which does not make any sense to me).

    Out of curiosity I also tried the LinearToGammaSpace function:
    unity_video_player_linear_to_gamma.jpg

    This result surprised me even more because it's closer to my expectation (except for grayscale area 32 which is still off). But I don't understand the way how Unity handles the color space transformations here.

    Do I misunderstand something here and is this a correct behaviour of Unity?
     
  5. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    This makes sense because we are already doing the GammaToLinearSpace on our side so you should not do it another time. On the second image, it works because we convert the video from Gamma -> Linear and you get it back from Linear -> Gamma. So you get back the original color. I am not sure why you don't get the red color though. I assume this is an approximation and because of that you lose a little bit of colour. It seems the threshold is now 29 for red.
     
  6. jas789

    jas789

    Joined:
    Jun 24, 2022
    Posts:
    5
    Ok, I got it now. Thanks for the explanation.

    The offset of ~3 for the red area (grayscale 32) is still huge in my opinion (and too inaccurate for my use case). I also tried the more precise conversion function LinearToGammaSpaceExact, but without any improvement. It seems like Unity is already using the approximation functions beforehand, so no chance for me to get better results.

    After comparing the results of the exact and approximate conversion function, I agree with you, that it is indeed a precision issue especially for RGB values below ~70:
    unity_color_space_exact_vs_approximate.png

    For now I don't see any other solution than switching completely to the global gamma color space without losing accuracy.
     
  7. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    You can otherwise disable the linear colour conversion if this is an issue. In the inspector, while selecting the VideoClip, you can uncheck sRGB, and we will not do the conversion (I know, it is not the best name). Wish it helps!
    upload_2022-6-29_21-16-33.png
     
  8. jas789

    jas789

    Joined:
    Jun 24, 2022
    Posts:
    5
    Unfortunately not. I'm not using the VideoClip object because I need to be able to load the Video dynamically as URL outside from the project's assets folder.
     
  9. The_Island

    The_Island

    Unity Technologies

    Joined:
    Jun 1, 2021
    Posts:
    502
    Oh, I see. You can override an internal shader with your own version if you really need it. To do that, you just need to create a shader with the same name has the internal shader. You can get the source on the download page. Click on "Download" > "Build in shaders" for the Editor version you want and you will get the source of all the internal shaders.
    https://unity3d.com/get-unity/download/archive
    upload_2022-6-30_13-50-48.png
    In the folder, search for VideoDecode.shader. Open the shader and edit AdjustForColorSpace to look like this
    Code (CSharp):
    1. inline fixed4 AdjustForColorSpace(fixed4 color)
    2. {
    3.     return color;
    4. }
    Drag and drop the shader in your project and restart the Editor. And voila, your shader should override our own shader. It means thought you will have to update the shader every time you do a major update but it is a way of doing it. Btw, if you do this don't forget to change the shader inside the graphics settings for the new shader. Otherwise, the shader will not be inside your build.
    upload_2022-6-30_13-56-31.png
     
    Last edited: Jul 1, 2022
  10. jas789

    jas789

    Joined:
    Jun 24, 2022
    Posts:
    5
    Thanks for showing this little hack. I will definitely try this.