Question Does ImageConversion.EncodeNativeArrayToJPG(...) return a writeonly array?

Hey,

when I try to access data from my NativeArray<byte> returned by ImageConversion.EncodeNativeArrayToJPG(...), I get runtime errors stating "InvalidOperationException: The UNKNOWN_OBJECT_TYPE has been declared as [WriteOnly] in the job, but you are reading from it.". I am using this function inside a job's execute function.

An easy example that leads to this behaviour is the following:

Code (CSharp):

1. NativeArray<byte> encodedDataNative = ImageConversion.EncodeNativeArrayToJPG(Input, GraphicsFormat, (uint)Width, (uint)Height, (uint)RowBytes, Quality);
2. byte b = encodedDataNative[0]; // creates the InvalidOperationException
3. NativeArray<byte>.Copy(encodedDataNative, Result, encodedDataNative.Length); // also creates the InvalidOperationException (and potentially other errors that I haven't yet reached)

Is there a way around this so that I can access the data inside the native array. I don't want to use the .toArray() function to avoid creating a new byte array that creates garbage. I don't want to use byte[] encodedData = ImageConversion.EncodeArrayToJPG(...) for the same reason.

 

The documentation says that: "Write this data to disk to get the data in the JPG file format". I personally never worked with screenshots, less so in DOTS, but this implies to me that it's really just meant for saving the image to your drive. So whatever you intend to accomplish by reading the data is probably meant to be handled before converting it to a JPG encoded representation.

What do you attempt to do?

 

My intent is to have a performant way of (1) creating a screenshot of the main camera ((2) and get it from the gpu to the cpu), (3) convert it to jpg, and send this data to another application via local network. The sending application is running on a portable VR headset, which means that performance is very important.

To do this, I (1) perform a fullscreen blit (I use the URP), (2) use an async readback request to draw the texture information from gpu to cpu, and (3) use the job system to convert the raw texture data to jpg data.
So far, all three components behave as expected. But now the optimization begins, and the first thing I want to do is get rid of GC as much as possible.

Currently, I use ImageConversion.EncodeArrayToJPG(...) in the job's execute function because I needed a proof of concept asap. This allocates a byte[] array which eventually causes the GC to do it's job and potentially create a lag spike. But, if I instead used ImageConversion.EncodeNativeArrayToJPG(...) to fill a NativeArray<byte>, I would be able to ommit creating the new byte[] array that is created automatically in ImageConversion.EncodeArrayToJPG(...). This is a benefit since I can just dispose the NativeArray created by EncodeNativeArrayToJPG(...) at the end of the job and thus not create any garbage.

Here is a simplified snippet of my code:

Code (CSharp):

1. // called from the main thread
2. private void EncodeAndSendData(NativeArray<byte> textureData)
3.         {
4.             EncoderJob encoderJob = new EncoderJob();
5.             NativeArray<byte> jobResult = new NativeArray<byte>(textureData.Length, Allocator.TempJob);
6.             NativeArray<int> jobEncodedLength = new NativeArray<int>(sizeof(int) * 1, Allocator.TempJob);
7.        
8.             // textureData was created before this method was called and contains a copy of the async data
9.             encoderJob.Input = textureData;
10.             encoderJob.Result = jobResult;
11.             encoderJob.EncodedDataLength = jobEncodedLength;
12.             // ... some more variable initialization
13.        
14.             JobHandle jobHandle = encoderJob.Schedule();
15.             jobHandle.Complete(); // calling jobHandle.complete() is just a temporary solution for the proof of concept
16.        
17.             int actualJpgDataLength = encoderJob.EncodedDataLength[0];
18.             byte[] bytesToSendViaNetwork = new byte[actualJpgDataLength + 4];
19.             // we reserve the first 4 bytes of the byte[] array to encode the jpgs resolution in the form of two short values (but I omit the code here)
20.        
21.             // copy bytes from job's native array into final byte[] array with an offset of 4
22.             for (int i = 0; i < actualJpgDataLength; i++)
23.                 bytesToSendViaNetwork[i + 4] = encoderJob.Result[i];
24.            
25.             // dispose all native arrays used in the process
26.             colorData.Dispose();
27.             jobResult.Dispose();
28.             jobEncodedLength.Dispose();
29.        
30.             // finally, propagate the final byte array to the network layer
31.             SendData(rpcRawData);
32.         }

Code (CSharp):

1. private struct EncoderJob : IJob
2.         {
3.             [ReadOnly] public NativeArray<byte> Input;
4.             public NativeArray<byte> Result;
5.             public NativeArray<int> EncodedDataLength;
6.             // ... some more variables
7.          
8.             public void Execute()
9.             {
10.                 // these three lines work perfectly fine, but create garbage
11.                 byte[] encodedData = ImageConversion.EncodeArrayToJPG(Input.ToArray(), GraphicsFormat, (uint)Width, (uint)Height, (uint)RowBytes, Quality);
12.                 EncodedDataLength[0] = encodedData.Length;
13.                 NativeArray<byte>.Copy(encodedData, Result, encodedData.Length);
14.  
15.                 // this fails due to the InvalidOperationException. If it did work, it wouldn't create garbage
16.                 NativeArray<byte> encodedDataNative = ImageConversion.EncodeNativeArrayToJPG(Input, GraphicsFormat, (uint)Width, (uint)Height, (uint)RowBytes, Quality);
17.                 EncodedDataLength[0] = encodedDataNative.Length;
18.                 NativeArray<byte>.Copy(encodedDataNative, Result, encodedDataNative.Length);
19.                 encodedDataNative.Dispose();
20.             }
21.         }

As a side note: Since I call JobHandle.complete() in the main thread, the main thread is stalled when the GC clears the byte[] arrays created in the worker threads, but I actually don't know how the GC behaves if I let the worker do it's work without calling JobHandle.complete() and just periodically check for the workers completion status (... which is what I ultimatively plan to do). I hope that only the worker thread is stalled than, which would solve the problem for the most part, but maybe the main thread is stalled too while the GC is collecting?

 

Hey Chris, did you manage to solve this ? I'm currently facing the exact same problem while trying to achieve the same thing

 

Yes, I wrote my solution in one of my other threads. Here it is: https://forum.unity.com/threads/con...ting-garbage-impossible.1464521/#post-9188624

 

Well, that's almost exactly what I did implement because I couldn't find any other way.
Thanks for your reply