How to use Graphics.Blit() more efficiently?

I need help with packing a full hd (1920 x 1080) or any other resolution or aspect ratio texture inside a square render texture having resolution 1024 x 1024. The fitting inside part needs to be done runtime as the input texture can be of any aspect ratio and resolution, but the output render texture has to be 1024 x 1024.

Something like this:


Or this...



I have managed to work up a messy solution using Graphics.Blit() but it is very resource intensive and tedious to maintain and extend. I am looking for a better solution using Graphics.Blit(), if it could be done, with lesser line of code and higher efficiency.

My approach works with an input texture, a temporary render texture and output render texture. The code is below:

Code (CSharp):

1. //Create output render texture
2. RenderTexture outputRenderTexture = new RenderTexture(1024, 1024, 24);
3. tempRenderTexture.Create();
4.  
5. //Read input texture from a file path
6. Texture2D inputTexture = new Texture2D(1, 1);
7. var fileData = File.ReadAllBytes(filePath);
8. inputTexture.LoadImage(fileData);
9.  
10. //Get aspect ratio of input texture
11. float aspectRatio = (float)inputTexture.width / (float)inputTexture.height;
12.  
13. //Calculate width and height of a temporary render texture for scaling the input texture such that it could fit inside the output render texture
14. int targetTexWidth = 0, targetTexHeight = 0;
15. if (inputTexture.width > inputTexture.height)
16. {
17.    targetTexWidth = outputRenderTexture.width;
18.    targetTexHeight = (int)(targetTexWidth / aspectRatio);
19. }
20. else
21. {
22.    targetTexHeight = outputRenderTexture.height;
23.    targetTexWidth = (int)(targetTexHeight * aspectRatio);
24. }
25. if (targetTexWidth > outputRenderTexture.width)
26. {
27.    targetTexWidth = outputRenderTexture.width;
28.    targetTexHeight = (int)(targetTexWidth / aspectRatio);
29. }
30. else if (targetTexHeight > outputRenderTexture.height)
31. {
32.    targetTexHeight = outputRenderTexture.height;
33.    targetTexWidth = (int)(targetTexHeight * aspectRatio);
34. }
35.  
36. RenderTexture tempRenderTexture = new RenderTexture(targetTexWidth, targetTexHeight, 24);
37. tempRenderTexture.Create();
38.  
39. Graphics.Blit(inputTexture, tempRenderTexture);
40.  
41. var oldRt = RenderTexture.active;
42. RenderTexture.active = tempRenderTexture;
43. inputTexture = new Texture2D(tempRenderTexture.width, tempRenderTexture.height, TextureFormat.RGB24, false);
44. inputTexture.ReadPixels(new Rect(0, 0, tempRenderTexture.width, tempRenderTexture.height), 0, 0);
45. inputTexture.Apply();
46.  
47. RenderTexture.active = oldRt;
48.  
49. Color c;
50.  
51. //Create new intermediary texture with output render texture dimensions for painting the pixels of the input texture at the center
52. Texture2D finalTexture = new Texture2D(outputRenderTexture.width, outputRenderTexture.height);
53. finalTexture.wrapMode = TextureWrapMode.Clamp;
54.  
55. //make a loop to paint the image black
56. for (int i = 0; i < finalTexture.width; i++)
57. {
58.    for (int j = 0; j < finalTexture.height; j++)
59.    {
60.        finalTexture.SetPixel(i, j, Color.black);
61.    }
62. }
63. finalTexture.Apply();
64.  
65. // copy all the pixels from the input texture into the middle of the intermediary texture
66. int startX = 0, startY = 0;
67.  
68. if (finalTexture.width > inputTexture.width)
69. {
70.    startX = (finalTexture.width - inputTexture.width) / 2;
71. }
72. if (finalTexture.height > inputTexture.height)
73. {
74.    startY = (finalTexture.height - inputTexture.height) / 2;
75. }
76.  
77. int pixelX = 0, pixelY = 0;
78. for (int i = startX; i < inputTexture.width + startX; i++)
79. {
80.    pixelY = 0;
81.    for (int j = startY; j < inputTexture.height + startY; j++)
82.    {
83.        c = inputTexture.GetPixel(pixelX, pixelY);
84.        finalTexture.SetPixel(i, j, c);
85.        pixelY++;
86.    }
87.    pixelX++;
88. }
89. finalTexture.Apply();
90.  
91. // Finally blit the intermediary texture to ouput render texture.
92. Graphics.Blit(finalTexture, outputRenderTexture);
93.  
94. // Remove unused textures from memory
95. Destroy(inputTexture);
96. Destroy(tempRenderTexture);
97. Destroy(finalTexture);

Please tell me if there is any easier or more efficient way to do this.

Thanks in advance.

 

I doubt the expensive part is the blit, it's probably all the work you are doing on the CPU to create that intermediary texture.

You need to transfer that logic into a shader that you then use to Blit with. Pass the aspect to the shader, then offset and multiply the UVs so that the aspect remains correct and just directly blit from the starting image to the output image.

 

Does that need a custom shader or can it be done using any of the standard shaders?

P.S. - This is my first time with Blit(), so some reference code or material will help. Thanks.

 

You'll need to write a shader.

I don't have time to do that now, hopefully I'll remember / have time tomorrow.

 

Thanks for the quick response @AcidArrow.

I tried couple of other ways to do this, but all in vain. I tried doing Blit directly on the render texture(RT) with the input texture, but it gets stretched to the aspect of the RT. I tried to tweak the scale and offset value in the Blit, but it seemed to repeat the texture from the edge pixels of the input texture.

I need to extend the behaviour of fitting the input texture inside the RT after this. I need to add the Fit Outside, Fit Horizontally, and Fit Vertically on the top of Fitting it inside on a RT. I can work up some more messier logic and try to go at it with a switch statement, but I fear I might end up with something highly unmanageable at the end.

I saw Unity's VideoPlayer do it quite efficiently on an RT and that's multiple frames per second. I need that behaviour and efficiency but for just one texture. I don't know if Unity's VideoPlayer is using a custom shader internally, but can I do that without using a custom shader just like Unity is doing in their VideoPlayer?

If custom shader is the only way to go than, I am at your mercy. I don't know how to go about the approach you suggested. Really appreciate your inputs and hope I could get some more help on this. Thanks again.

 

I am sure the inefficiency isn't in

Graphics.Blit()

but in your attempt to set all outside pixels to black, using

finalTexture.SetPixel()

. Setting individual pixels is ridiculously slow.

Fill the finalTexture with a solid black texture before you blit the image onto it.

There are a couple ways to do that.

* Unity has a pretty crappy built-in property called

Texture2D.blackTexture

which gives you a little black (but transparent) texture; you can then scale that to the desired size, rather than create an un-initialized texture of the desired size.

* You can use

finalTexture.SetPixels()

to set a block of pixels all at once, to a

Color.black

value (which is thankfully opaque black).

Honestly, it would be great if Unity gave a built-in that initialized a solid texture for you in a competent way. Asking C# to work with pixels is, as Spock would say, using bear skins and flint knives.

 

I hear you. I tried using Texture2D.blackTexture before working with SetPixels to clear the image, but it somehow didn't work in my Unity version (2021.1.16). I used it like this, but it didn't make black texture at all:

Code (CSharp):

1. Texture2D finalTexture = Texture2D.blackTexture;
2. finalTexture.Resize(userMediaButton.targetTexture.width,
3.    userMediaButton.targetTexture.height);
4. finalTexture.wrapMode = TextureWrapMode.Clamp;
5. finalTexture.Apply();

This would always give the final texture as white. Is this correct way of doing it or was I doing it wrong?

Edit:
I also tried the other alternative you mentioned using the SetPixels method. But same result. It does not paint it black.

I tried it like this:

Code (CSharp):

1. Texture2D finalTexture = new Texture2D(1, 1);
2. finalTexture.wrapMode = TextureWrapMode.Clamp;
3. finalTexture.SetPixels(new Color[] { Color.black });
4. finalTexture.Apply();
5. finalTexture.Resize(userMediaButton.targetTexture.width,
6.     userMediaButton.targetTexture.height);
7. finalTexture.Apply();

 

I didn't do these, I just covered the original use case (of fitting any image into a square), but hopefully the shader is understandable enough that you can create new ones for the missing use cases.

I won't have time during the week, but maybe ping me next weekend if you still have problems.

Anyway. I'm attaching the Unity package, but I'll also paste the script and the shader here.

Script is super simple:

Code (CSharp):

1. using UnityEngine;
2.  
3. public class FitImageToSquare : MonoBehaviour {
4.  
5.     public Texture2D inputTexture;
6.     public Material matWithFitShader;
7.     public int targetSize = 1024;
8.     public RenderTexture rt;
9.  
10.     public void FitIt() {
11.  
12.         float aspect = (float)inputTexture.width / (float)inputTexture.height;
13.  
14.         matWithFitShader.SetFloat("_Aspect", aspect);
15.  
16.         rt  = new RenderTexture(targetSize, targetSize, 0, RenderTextureFormat.ARGB32);
17.  
18.         Graphics.Blit(inputTexture, rt, matWithFitShader);
19.     }
20. }

Just passing the aspect to the shader (through the material), then blitting with it.

The shader is:

Code (CSharp):

1. Shader "FitImageToSquare" {
2.     Properties {
3.         _MainTex ("Dont use this, Used by code", 2D) = "white" {}
4.     }
5.  
6.     SubShader {
7.         Pass {
8.             CGPROGRAM
9.             #pragma vertex vert
10.             #pragma fragment SimpleBlit
11.  
12.             #include "UnityCG.cginc"
13.  
14.             float _Aspect;
15.             UNITY_DECLARE_TEX2D(_MainTex);
16.             float4 _MainTex_TexelSize;
17.  
18.             struct v2f_yo {
19.                 float4 pos : SV_POSITION;
20.                 float2 uv : TEXCOORD0;
21.             };
22.  
23.             v2f_yo vert(appdata_img v) {
24.                 v2f_yo o;
25.                 o.pos = UnityObjectToClipPos(v.vertex);
26.  
27.                 #if UNITY_UV_STARTS_AT_TOP
28.                 if (_MainTex_TexelSize.y < 0)
29.                 v.texcoord.y = 1.0 - v.texcoord.y;
30.                 #endif
31.  
32.                 if (_Aspect > 1.0) {
33.                     o.uv = v.texcoord * float2(1.0, _Aspect) - float2(0.0, (_Aspect - 1.0) * 0.5);
34.                 } else {
35.                     o.uv = v.texcoord * float2(1.0 / _Aspect, 1.0) - float2((1.0 / _Aspect - 1.0) * 0.5, 0.0);
36.                 }
37.  
38.                 return o;
39.             }
40.  
41.             fixed4 SimpleBlit (v2f_yo i) : SV_Target {
42.                 if (i.uv.x < 0.0 || i.uv.x > 1.0 || i.uv.y < 0.0 || i.uv.y > 1.0) {
43.                     return 0;
44.                 }
45.                 else{
46.                     return UNITY_SAMPLE_TEX2D(_MainTex, i.uv);
47.                 }
48.             }
49.             ENDCG
50.         }
51.     }
52.     Fallback Off
53. }

This part:

if (_Aspect > 1.0) {
o.uv = v.texcoord * float2(1.0, _Aspect) - float2(0.0, (_Aspect - 1.0) * 0.5);
} else {
o.uv = v.texcoord * float2(1.0 / _Aspect, 1.0) - float2((1.0 / _Aspect - 1.0) * 0.5, 0.0);
}

Is doing the aspect correction. Everything else is pretty much boilerplate.

 

You sir, are awesome! Your shader based implementation effectively reduced the code to just few lines and was way faster than my code.

But now, I will need to delve into Shaders in order to be able to extend your script. Thank you for the sample code! Really appreciate it.

Can this be done using only a c# script (without a shader) but with the efficiency and execution speed of your shader?

 

Nah. GPUs are just that much more efficient when it comes to image manipulations like this. This is simply the way to go if you care about efficiency.

Anyway, I added the capability to fill. Here's the updated package.

 

I understand, thanks again for the new sample. It works like a charm.

 

@AcidArrow thanks for your help I needed to do something similar and your blit shader helped me out a lot.