Using the job system to... save images?

This is an idea I had when I saw just how capable the new job system is, but I'm wondering if this is the best way to handle things. I've tried numerous ways to capture screenshots in the past, including the reportedly fastest methods. Unfortunately, all of them implement a degree of lag which I consider unacceptable. I'll explain what I'm trying to do:

1. When the player presses the "take photo" button and they have enough space left on their roll, I blit from the camera target texture to an array of rendertextures.
2. If the player develops their photos, I loop through the array, converting rendertextures to texture2Ds.
3. I save each texture2D to a special photo directory.

As the photos are 1620x1080 in size, you can probably imagine how slow this gets. A full roll tends to take around 3-4 seconds to develop. Seeing as photos are automatically developed when the player saves their game (or vice versa), this makes the saving process take a while.

So, in comes the job system. The idea that I could find a way to thread all this seems like it'd be absolutely ideal, so I dove right in, following the guides as best I could to make a working system. Unfortunately, I've hit an impasse, or possibly several.

My first attempt was simple enough. I'll try and start with only barely modified code. With that in mind, I tried the following:

Code (CSharp):

1.     struct DevelopPhotosJob : IJob
2.     {
3.         public RenderTexture[] photoTextures;
4.  
5.         public string photoFolderJob;
6.  
7.         Texture2D undevelopedPhoto;
8.  
9.         public void Execute()
10.         {
11.             for (int i = 0; i < photoTextures.Length; i++)
12.             {
13.                 RenderTexture.active = photoTextures[i];
14.  
15.                 undevelopedPhoto = new Texture2D(photoTextures[i].width, photoTextures[i].height, TextureFormat.RGB24, false);
16.                 undevelopedPhoto.ReadPixels(new Rect(0f, 0f, photoTextures[i].width, photoTextures[i].height), 0, 0);
17.                 undevelopedPhoto.alphaIsTransparency = false;
18.                 undevelopedPhoto.Apply();
19.  
20.                 byte[] bytes = undevelopedPhoto.EncodeToPNG();
21.  
22.                 Destroy(undevelopedPhoto);
23.  
24.                 File.WriteAllBytes(photoFolderJob + "/Photos/" + (i + 1).ToString("00") + ".png", bytes);
25.             }
26.         }
27.     }

And I would call the job as such:

Code (CSharp):

1.     private void Update()
2.     {
3.         if (player.GetButtonDown("RightBumper"))
4.         {
5.             TakePhoto();
6.         }
7.  
8.         //if (player.GetButtonDown("LeftBumper"))
9.         //{
10.         //    DevelopPhotos();
11.         //}
12.  
13.         if (player.GetButtonDown("LeftBumper"))
14.         {
15.             var job = new DevelopPhotosJob()
16.             {
17.                 photoTextures = undevelopedPhotos,
18.                 photoFolderJob = photoFolder
19.             };
20.  
21.             JobHandle jobHandle = job.Schedule();
22.             jobHandle.Complete();
23.         }
24.     }

While this compiles, it gives me the error "InvalidOperationException: DevelopPhotosJob.photoTextures is not a value type. Job structs may not contain any reference types."

Okay, I figure that if I can't send that array through, I can at least send a single rendertexture, right? Unfortunately, there was no joy there either, as that gave me the same error. So then I decided "well, maybe I can use a NativeArray and get around the issue that way. Unfortunately, you can't actually make a native array of RenderTextures because you can't use anything but non-nullable value types.

So is there any way to do what I'm trying to accomplish or am I barking up the wrong tree? I'm just trying to find a way to improve the speed of developing photos and saving the game that doesn't completely lock up the game for several seconds.

 

Ya this is kind of an annoying limitation of the job system. I like how jobs work in how you have fine grained control over scheduling and completing, but I don't see how jobs themselves as structs was a good trade off against making them easier to use for general purpose stuff. I would rather see jobs just have a struct data container for job data, leave the job itself as a reference type.

 

Can't you just use array of pixels instead of textures?

 

To do that, I'd need to convert from a RenderTexture to a Texture2D and run through the apply process, which makes up a majority portion of the lag.

 

Can you provide us a TakePhoto() function?

 

Can do.

Code (CSharp):

1.     void TakePhoto()
2.     {
3.         if (totalPhotos < photoRollSize)
4.         {
5.             Graphics.Blit(cam.targetTexture, undevelopedPhotos[totalPhotos]);
6.  
7.             totalPhotos++;
8.         }
9.         else
10.         {
11.             Debug.Log("Photo roll full!");
12.         }
13.     }

Cam is simply the Camera associated with the script.

 

Check out Keijiro's Async GPU Readback example here:
https://github.com/keijiro/AsyncCaptureTest

 

Getting the images isn't the problem, it's the Texture2D.Apply() call and the file saving process.

 

Try Graphics.CopyTexture() instead of Graphics.Blit(), and use Texture2D as target. But still you need to GetPixels outside a job.

I think you can skip alpha part with heavy Texture2D.Apply() just by using good texture format but I don't see any API for "alphaIsTransparency" so don't know really.

 

Oops, should have checked first. The array is what kills it.

 

Well now I've encountered a new problem. How do I pass a string into a job? Using a public string throws up an error, and I can't use a nativearray because strings are a nullable type.

 

string is a not a structure, is always been class, jobs can work only with structures (if I remember correctly from Joakim's presentations on 2017-2018 Keynotes)

 

It should be possible to create a Native*-style structure that wraps a char* (or larger depending on your needs), as the native types themselves are just wrappers around pointers at their core, and there are examples of how to roll your own in the github docs.

I wouldn't be surprised if we eventually get a NativeString and NativeWideString that do this (or some enterprising soul makes them), but that smells of re-inventing the wheel.

 

You can't use a char* either. NativeArray<char> might work but System.Char is not a blittable type so you may need to goto byte. Unless you are manipulating string inside your System, you could pool strings and reference them by an ID then process strings in the main loop. It depends on your need how that would work.

For the filename, I would just pass the integer in then use basically the same File.WriteAllBytes line with the given integer.

 

That works fine, provided I don't want to use Application.DataPath, which I unfortunately do. Trying to call that during a job just results in a "must be called from main thread" error.

 

Seems like most of the Unity APIs and C# APIs need to support Job system. If not, it is a lot of work to do. lol

 

I expect that more and more will be. In one of the other threads a unity guy said that rendering in jobs will be supported in the future (no clue eta). It's still very early in the ECS/Jobs life to expect so much of it to be transferred over already.

 

Ok, this is a bit painful, but it works. In short, I use Encoding.ASCII.GetBytes(string) to convert to a byte array then use NativeArray.CopyFrom(byte[]) to fill it in. NativeArray.ToArray() and Encoding.ASCII.GetString(byte[]) do the reverse.

Create an empty GameObject and attach this script. Watch the script in the inspector and run the scene. As you edit "Test String" then "Result String" is updated by going through a Job.

Code (csharp):

1.  
2. public struct MyJob : IJob
3. {
4.     public NativeArray<byte> inStr;
5.     public NativeArray<byte> outStr;
6.  
7.     public void Execute()
8.     {
9.         // Can we use strings inside a job?
10.         string s = Encoding.ASCII.GetString(inStr.ToArray());
11.         outStr.CopyFrom(Encoding.ASCII.GetBytes(s));
12.     }
13. }
14.  
15. public class JobTestBehaviour : MonoBehaviour
16. {
17.     public string testString = "Hello, World!";
18.     public string resultString = "";
19.  
20.     private void Update()
21.     {
22.         MyJob jobData = new MyJob();
23.         jobData.outStr = new NativeArray<byte>(testString.Length, Allocator.Temp);
24.         jobData.inStr = new NativeArray<byte>(testString.Length, Allocator.Temp);
25.         jobData.inStr.CopyFrom(Encoding.ASCII.GetBytes(testString));
26.  
27.         JobHandle handle = jobData.Schedule();
28.         handle.Complete();
29.         resultString = Encoding.ASCII.GetString(jobData.outStr.ToArray());
30.  
31.         jobData.inStr.Dispose();
32.         jobData.outStr.Dispose();
33.     }
34. }
35.  

 

Well, it's not exactly pretty, but it does work! I had a feeling something like this might be the case but I couldn't figure out how I was supposed to encode everything.

 

@Murfilod was your conclusion that your “develop photo job” won’t work in the new job system? Sounds like your last comment was that you figured out how to pass a string.

 

how about using IntPtr to replace reference in job system?

 

I'm trying to download images from the url and I tried using webrequest in IJob. Unity throws error and I cannot use it. Can you guys help me out or point me in the right direction.

 

Do not use jobs for the web requests.

 

I need to download 7-10 images from the server and I want it to happen in the background. In Start I get the urls and the download the image(for now) but it takes some time to load the images in viewer makes it look like app is not responding.

 

Use coroutines to avoid hanging main thread. Or use Start as a coroutine.
Jobs do not support any kind of requests (yet?).

 

We have coroutines to download url in a loop and after completion we are displaying images. Its takes sometime to download all the images and until its blank screen with BG. I was hoping is there any other way load images in parallel.

 

Theoretically, you could try using default C# sockets in a background thread. Although it won't be easy to sync the data from the background thread to main thread.

 

It will be easy. With a non-blocking queue and Span<T> https://github.com/dave-hillier/disruptor-unity3d/blob/master/DisruptorUnity3d/Assets/RingBuffer.cs

 

I'm building for the webgl and may in future we'll do it for the android and ios. Will this work for webgl? How do I use it to download images from url?

 

If it's webgl then you're screwed. Only option left is coroutines, as Sockets class is prohibited on it.

On Android / IOS it should work (in theory).

 

Isn't UnityWebRequest / Addressable already start a thread to download? Do multiple uwr.SendWebRequest() at once then collect all UnityWebRequestAsyncOperation. Do something while they download then when you need it wait for all collected AO.

 

Unless webgl. Then it's one thread anyway.

 

Did you ever figure out how to take images?

 

Hi, I posted some image screenshot code here https://forum.unity.com/threads/res...ide-down-and-not-correctly-positioned.848230/
It doesn't use jobs, but it is async, and uses nativearray for speed. You can always push the bytes onto a component, using a blittable array for processing, but i'm just saving it to a file atm. I couldn't work out how to write to file in a job though, that was my original goal!