How to get pixel values from a texture as floats?

Hey all,

I'm trying to get pixel values from a texture and place them inside a Numpy array.

Code (CSharp):

1.     public static NDArray ConvertSnapshotToNDArray(Texture2D snap)
2.     {
3.        
4.         int width = snap.width;
5.         int height = snap.height;
6.  
7.         NDArray newArray = np.zeros((1, snap.width, snap.height, 1));
8.         for (int x = 0; x < width; x++)
9.         {
10.             for (int y = 0; y < height; y++)
11.             {
12.                 var pixel = snap.GetPixel(x, y);
13.                 newArray[0, x, y, 0] = pixel.grayscale;
14.             }
15.         }
16.  
17.         return newArray;

This current method is very slow so instead, I'm trying to use GetPixels() to get all the pixels, however, they are in Color format so I need to convert each one to grayscale in the loops.

My new method is this:

Code (CSharp):

1.     public static NDArray ConvertSnapshotToNDArray(Texture2D snap)
2.     {
3.        
4.         int width = snap.width;
5.         int height = snap.height;
6.  
7.         NDArray newArray = np.zeros((1, snap.width, snap.height, 1));
8.  
9.         Color[] pixels = snap.GetPixels();
10.         int index = 0;
11.  
12.         for (int x = 0; x < width; x++)
13.         {
14.             for (int y = 0; y < height; y++)
15.             {
16.                 //var pixel = snap.GetPixel(x, y);
17.                 //newArray[0, x, y, 0] = pixel[0];
18.                 newArray[0, x, y, 0] = pixels[index].grayscale;
19.                 index++;
20.             }
21.         }
22.  
23.         return newArray;

This is faster but is also still too slow. Is it possible to place the grayscale values from the texture2D automatically? Or without having to use loops?

 

Why do you need to put it in a Numpy array?

 

You could try (this is a bad hack, and you may go to hell for it) to cast the Pixel map as a one-dimensional Array of float, and Access each Pixel by calculating the offset (y * width + x) * 4 - or even iterate by steps of 4 (or 3 if you have an RGB texture). This Kind of low.level hack *will* break in the future, though.

 

The snapshot will be fed into an ML model. Numpy arrays are the only format that I've gotten to work as imports thus far.

 

Its not about array dimension, its about how many pixels there are.

1024 * 1024 = 1,048,576 pixels. That's over one million iterations, trying to do that in one frame isn't feasible.

I would try off loading the processing onto another thread to help speed things up, or a coroutine.

You will have to get the pixels first and then move that to another thread for processing, or Unity will complain about the call to GetPixels in another thread.

 

Yeah, It doesn't sound like I have many other options tbh. I'm gonna try Vector4 as an input format for my model. If that works, I might be able to convert the values much easier, since apparently they're implicitly converted.

 

Agreed. But that wasn't the OP's question. Where did the single frame requirement come from?

 

Any call to any method will happen in one frame cause it will block the rest of the code from running until its done. Unless its a coroutine or on a different thread.