Question Convert to JPG in Job without creating garbage impossible?

Hey,

I am trying to convert texture data to jpg data inside a job once per frame. To do that, I currently use ImageConversion.EncodeArrayToJPG(...). But that creates a byte[] array, which eventually has to be garbage collected and that creates huge lag spikes. We are talking about screenshots with a resolution of about 1000x1000, which results in roughly 30kb of jpg data, give or take. This makes the garbage collector spike with ~5-10ms more than once per second. Does anyone have an idea how to circumvent this?

I have already tried ImageConversion.EncodeNativeArrayToJPG(...), which returns a native array, but with no luck. The returned native array is readonly, which renders it completely useless inside a job, since I cannot read it's data at all. Do I miss something with the usage of EncodeNativeArrayToJPG?

Here is my current code:

Code (CSharp):

1.  
2. public void Execute()
3. {
4.     // approach using a byte[] array (creates a lot of garbage)
5.     byte[] encodedData = ImageConversion.EncodeArrayToJPG(Input.ToArray(), GraphicsFormat, (uint)Width, (uint)Height, (uint)RowBytes, Quality);
6.     EncodedDataLength[0] = encodedData.Length;
7.     NativeArray<byte>.Copy(encodedData, Result, encodedData.Length);
8.  
9.     // this approach doesn't work because the returned native is readonly:
10.     NativeArray<byte> encodedDataNative = ImageConversion.EncodeNativeArrayToJPG(Input, GraphicsFormat, (uint)Width, (uint)Height, (uint)RowBytes, Quality);
11.     EncodedDataLength[0] = encodedDataNative.Length;
12.     NativeArray<byte>.Copy(encodedDataNative, Result, encodedDataNative.Length); // this creates an error stating that the encodedDataNative is readonly, which seems to make it completely useless?
13.     encodedDataNative.Dispose();
14. }
15.  

 

If that is giving you an error, then that's probably a Unity bug. Try using Result.CopyFrom or encodedDataNative.CopyTo instead. If you need to shrink the result, use Result.GetSubArray.

 

Sadly, all of those result in the same readonly exception. Can you confirm, that this is not the intended behaviour? Because if so, I can file a bug report.

 

I am using Unity 2022.3.0f1 btw

 

I use CopyFrom a lot in my own code where the src NativeArray is [ReadOnly].

At this point, I'm suspicious you are doing something wrong that isn't evident by the code snippets or paraphrasing of the error message.

 

Oh god, sorry, I made a mistake in my original post and my answer. The NativeArray returned by ImageConversion.EncodeNativeArrayToJPG(...) is not [readonly], it is [writeonly]. My previous sentence "The returned native array is readonly ((<- actually writeonly)), which renders it completely useless inside a job, since I cannot read it's data at all." should now actually make sense.

That's why I am struggling. Using ImageConversion.EncodeNativeArrayToJPG would be perfect since I can dispose the array at some point, but I cannot use it because it is writeonly when called inside a job. It is not writeonly when called from the main thread by the way.

Any ideas on how to solve the issue?

 

Oh. That bug. First off, report that. That's just stupid.

Second, the workaround usually involves enabling unsafe code and using GetUnsafePtr() and UnsafeUtility.MemCpy() to get the data where you want it.

 

Thank you, it works! For anyone stumbling over this thread and having the same problem, here is my solution:

Code (CSharp):

1.  
2. BurstCompile]
3. private struct EncoderJob : IJob
4. {
5.     [ReadOnly] public NativeArray<byte> Input;
6.     [WriteOnly] public NativeArray<byte> Result;
7.     [WriteOnly] public NativeArray<int> EncodedDataLength;
8.  
9.     public int Width;
10.     public int Height;
11.     public int RowBytes;
12.     public int Quality;
13.     public GraphicsFormat GraphicsFormat;
14.  
15.     public unsafe void Execute()
16.     {
17.          NativeArray<byte> encodedDataNative = ImageConversion.EncodeNativeArrayToJPG(Input, GraphicsFormat, (uint)Width, (uint)Height, (uint)RowBytes, Quality);
18.          EncodedDataLength[0] = encodedDataNative.Length;
19.          void* internalPtr = NativeArrayUnsafeUtility.GetUnsafeBufferPointerWithoutChecks(encodedDataNative);
20.          void* outputPtr = NativeArrayUnsafeUtility.GetUnsafeBufferPointerWithoutChecks<byte>(Result);
21.          UnsafeUtility.MemCpy(outputPtr, internalPtr, encodedDataNative.Length * UnsafeUtility.SizeOf<byte>());
22.          encodedDataNative.Dispose();
23.     }
24. }

Code (CSharp):

1.  
2. private void StartEncoderJob(NativeArray<byte> textureData)
3. {
4.     // prepare job
5.     EncoderJob job = new EncoderJob();
6.     NativeArray<byte> jpgDataPool = GetPooledJpgDataArray(textureData.Length);
7.     NativeArray<int> jpgDataLength = GetPooledJpgLengthArray();
8.  
9.     // textureData was created before this method was called and contains a copy of the async result data
10.     job.Input = textureData;
11.     job.Result = jpgDataPool;
12.     job.EncodedDataLength = jpgDataLength;
13.     job.Width = _currentWidth;
14.     job.Height = _currentHeight;
15.     job.RowBytes = 0;
16.     job.Quality = _encodingQuality;
17.     job.GraphicsFormat = _renderTexture.graphicsFormat;
18.  
19.     JobHandle handle = job.Schedule();
20.     PendingJob pendingJob = GetPooledPendingJob();
21.     pendingJob.Handle = handle;
22.     pendingJob.Job = job;
23.     pendingJob.CreationTime = Time.time;
24.     _pendingJobs.Add(pendingJob);
25. }

Make sure you have "allow unsafe code" enabled in your player settings AND if you are using assemblies, you will have to tick the "allow unsafe code" checkbox in them as well.

I got the code from this thread. Note, that I do not call the NativeArray.Resize function since I know that Result is definetely bigger than encodedDataNative. I do NOT guarantee safety of this code, but if I ever find a problem with it, I will keep this thread here updated.

 

Hi @LB_Chris
Without disclosing any NDA related stuff, could you please share what kind of application you are building that require 1000x1000 size screenshots every seconds? Its interesting to hear how unity is being used in things unheard of.
Thanks

 

Hey, I cannot say too much, but what I can say that it a feature is related to a non-critical medical application. Having high resolution is not a necessity, but we want to check out how far we can go without affecting the applications actual purpose. A resolution half or a third of 1000x1000 might suffice, but even for that it's not an option to encode on the main thread since our ressources are quite limited.

 

Thanks, Unity being used in medical is great