What are the alternatives for ReadPixels or how to optimise its performance?

Hey guys! I'm trying to create a simple game about colors and I need to read some points on the screen and check the color these points are hovering. As you can see in the image below the background is b&w and the patterns will change and distort during the game (as crazy as possible). The player has two points and items only one point, this means I will have a maximum of 5 positions that I need to know the color at a time.



The problem: It works exactly as I want using a RenderTexture from Camera or OnImageRender where I can use ReadPixels from the Texture2D to get the colors, but it costs too much processing for a mobile game.

Failed solutions:

1 - I tried to select smaller sections of the RenderTexture so it wouldn't need to read all the pixels from RenderTexture, this helped but not enough. Even when using 1px by 1px Rects to select exact pixels. The problem is calling "ReadPixels" multiple times is too expensive, it is better to call it once.

2 - I tried to select the exact area needed (which is the player size) with only one "ReadPixels", still too expensive.

3 - I tried to use the "ReadPixels" method on a different thread using Job System or AsyncOperation. I get this exception: "UnityException: get_isReadable can only be called from the main thread". Read Pixels can only be called from the main thread.

Colliders:

This is currently the best solution I found and I will do this if there are no other solutions. Using colliders for one color will solve part of the problem. It takes more time to set up colliders depending on the patterns but this is not the main problem which is the organic patterns and distortions.

 

Unity isn't really set up to do fiddly pixel stuff like this. The obvious optimizations are going to be what you already tried: only do the ReadPixels once per frame (or once per when the underlying image changes), then read from there.

Another alternative is to make a separate texture simultaneously using a separate RenderTexture, but one that is much lower-resolution than the insane high resolution of modern mobile devices.

You should be able to easily do a ReadPixels on something like a 320x480 image, and the user's finger doesn't even have that much resolution in any case. You would never show this low resolution image (except perhaps for debugging), but rather just use it for "picking" the colors.

 

I recently was in a thread similar to this... and by recently I mean like April:
https://forum.unity.com/threads/drawing-application-with-unity.667735/#post-4469914

I suggested using 'GetRawTextureData', this returns a NativeArray that allows reading and writing (and I presume use the jobs system) directly to the texture in RAM. Instead of copying around RAM to modify it then applying back.

From docs:

It still requires calling apply at the end to upload your changes into graphics memory.

https://docs.unity3d.com/ScriptReference/Texture2D.GetRawTextureData.html

 

This is an interesting solution, but I don't have a Texture2D to use. When I get the texture from Camera it is a RenderTexture type, the method that I need to create a Texture2D is precisely the method that is lowering my performance. So, I don't think this will solve my problem specifically, but this is interesting for someone that have an instance of Texture2D.

 

Calling ReadPixels will stall the CPU/GPU pipeline which is slow. The alternative solution is to use AsyncGPUReadback instead but at the expense of a few frames latency (depending on the size of the data).

You can request a subset of data however, so if you calculate the positions on the CPU and feed that into the request(s) you'll be able to minimise the data to just the pixels that you need.

 

If you have a maximum of 5 position I'd assume using GetPixle (singular, https://docs.unity3d.com/ScriptReference/Texture2D.GetPixel.html) is the way to go, it's a lot slower if you need to look at the whole image (at any decent resolution) but in this case (again, I'd assume) it's gonna be faster by not dragging around all the other useless information.

 

You can't call GetPixel(s) on a RenderTexture.

 

This is probably the best solution, at most you will get about 4 frames latency which as long as you are running at 60fps will not be noticeable to the human eye for the most part (unless it happens multiple times in a row).

To be honest I would combine this with @Kurt-Dekker idea of having a lower resolution render texure that you use for selecting the pixels. But you will lose accuracy, so try at various resolutions. You should be fine using at least a 1/2 size render texture to what the actual screen is, but really can probably go right down to like 1/8 size or something

EDIT: another option would be to use a quad or something with a mesh renderer for the bit your trying to read and apply your pattern/color as a texture. Instead of using render textures, raycast and get the triangle hit, you can then get the color out from there

Something like what is done in this thread: https://forum.unity.com/threads/trying-to-get-color-of-a-pixel-on-texture-with-raycasting.608431/

This could be far faster

 

I've followed your solution and I manage to make the game run at 60fps in mid to high devices. But in mid to low devices is still problematic.

What I did:

- The Camera is now using a custom target Render Texture
- This Render Texture has a downscale parameter (I divided the width and height by 10 each) about 100 times fewer pixels.
- The ReadPixels is extracting for this lower resolution Render Texture.
- The ReadPixels occurs every 3 frames. (This actually did help because I was running the game by 20 fps and now 60 fps, meaning that I have the same results but with smoother rendering).

I've lost some precision but It is not noticeable for the player.
I will try other solutions you guys suggested to me, but this is an improvement.

Thanks, guys!

 

Have you tried AsyncGPUReadback suggested by grizzly? I'm curious on it's performance on low end devices

 

One problem with AsyncGPUReadback is that it does not work on OpenGL ES (and I presume some other platforms)
I am looking into this problem as well, Getting a Texture2D from a renderTexture just requires a call to
Graphics.CopyTexture on platforms that support it (which seems to be most)

 

@DavidSWu I'm trying this new solution it seems to be working better.
We didn't try AsyncGPUReadback yet, but this is promising.
Thank you!

 

I find that calling ReadPixels in a coroutine after yield return new WaitForEndOfFrame(); speeds up the performance on an Android 15x compared to calling it in Update or LateUpdate.