Question Custom serialisation of NativeArray

Hi. I have a project where I need to allocate a potentially large array of floats, which needs to be serialised as part of a ScriptableObject.

Example:

Code (CSharp):

1. public class MyClass : ScriptableObject
2. {
3.         [SerializeField]
4.         public float[] data;
5. }

I want to use this array as input to a Job. However, since Jobs don't work with managed arrays I'd need to convert it to a NativeArray first. On memory-constrained platforms this may be a problem, so I'd like to avoid the extra memory allocation if possible. How can I do this?

I was thinking I could replace the float[] managed array with a NativeArray and pass it directly to the job. However, AFAIK there doesn't seem to be a nice way of doing custom serialisation in Unity? Except maybe implementing OnBeforeSerialize and convert it to a managed array that built-in serialisation can handle, but that has the same issue. I feel like I'm missing something though..

Questions:
- Is there really no way of doing custom serialisation in Unity, so I could serialise/deserialise the NativeArray myself?
- Alternatively: Can I somehow access the underlying data of the managed array and pass it to a job without copying to a new array? (unsafe code, I would assume)

 

Unity has a rather unknown Serialization package that is no longer in preview since 2022.2 (though previews worked fine for me).

It‘s pretty trivial to write an adapter for binary serialization of NativeArray.
It is tremendously difficult to do anything with Json in that package however,so be sure tostay clear of json and use binary which is straightforward.

See: https://docs.unity3d.com/Packages/com.unity.serialization@3.1/manual/index.html

 

'Custom' serialisation in assets is implemented via the ISerializationCallbackReceiver interface: https://docs.unity3d.com/ScriptReference/ISerializationCallbackReceiver.html

Note that this how you extend Unity's serialisation, not replace it. Unity can never serialise types it hasn't been built to serialise natively, you can only turn non-Unity serialisable data into Unity serialisable data and back again with the aforementioned interface.

It's not as restrictive as you think, it's exactly how the Odin serialiser works, and it works very well.

Though in this context, a lazily initialised property is probably a better choice.

 

Thanks for the suggestion! I didn't know about this package. However, unless I've missed something this seems to be a general purpose serialisation package, and not something that lets you modify how ScriptableObjects or MonoBeahaviours are serialised?

I'd like to serialise my data array with my ScriptableObject .. Of course, I could use this to serialize to JSON or byte array, and then have Unity serialise that, but that wouldn't be more efficient than converting to an array type that Unity can serialise.

 

Thanks for the answer!

Yes, I've done that for some other types. But since these arrays can potentially be very large I want to avoid doing this, as users may run out of memory on more memory constrained platforms (WebGL, mobile..).

I think its quite restrictive tbh.. Other engines let you inject code into the serialisation/deserialisation pipeline (

Serialize(FArchive&)

in UE4). I might be missing something, but I haven't found any ways of doing this in Unity.

 

If you're dealing with volumes of data that large, then just the general re-serialisation of the asset in the editor and during domain reloads is going to slow down your editor experience. It's the kind of situation where this data should probably be serialised into a binary asset and only deserialised and reserialised on demand when needed.

I don't think there is, otherwise tools like Odin would probably be using that to handle serialisation instead. A lot of this stuff is handled on the internal side we don't have access to, so it's probably a 'source code access' thing, until some API that lets us do so comes into existence.

Gotta remember that Unity's serialisation is the way it is because it needs to be fast. Any custom serialisation is added overhead. Even Odin says to use its serialisation as little as possible for that reason.

 

Unity's serialization isn't very suited for storing big amounts of binary data. Even byte arrays get awkwardly encoded in Unity's text format and end up taking up much more disk space than necessary (and take longer to read and process).

You're better off putting your data into a binary asset. You can use the

.bytes

extension, which gets imported as a TextAsset but which provides TextAsset.GetData to get the data as a

NativeArray<T>

. Then you can reference the asset in your scriptable object.

With some editor scripting you could also make the TextAsset a sub-asset of your scriptable object or make a custom importer that imports a custom binary file type and generates the TextAsset and scriptable object automatically.

 

Yes, you're right. It might not be ideal to serialise this as a part of a ScriptableObject. Though, storing it as a binary file wouldn't be that straightforward either, since I'm using scripted importers to allow users to configure these assets (change import settings). I'll look into what's the best way to do what I want.

 

Ohh, this might actually work quite well!
I'm using scripted importers to manage my assets, so I suppose I could have them generate and reference a `TextAsset` then? I had heard about TextAssets before, but didn't know I could use them like this.. Thanks! I'll give it a try

Update: Though, on second thought I also need to allow my SerializedObject to be serialized with the scene (though I don't recommend that to the users of my plugin), so not sure how I could get this to work in that case.. I'll look more into it though! Thanks

 

Be aware that any managed object, like the managed float[] will need to be garbage collected once it is no longer referenced. Any object larger than 85kB ends up on the Large Object Heap. This could have a noticeable impact on performance. I believe I've heard that Unity's current garbage collector is non-generatonal, but I'm not sure what impact that really has on very large objects that will have to be collected. My thinking tends to lean toward spiney199's suggestions, and that maybe the data should be broken up into chunks and streamed into the native array piece by piece if it really needs to be that large. What is the upper threshold you expect on number of floats in one array?

 

Hm, yeah. The TextAsset should be serialized in the scene together with the scriptable object if both are not persistent assets. But you can only create TextAssets by script using strings, which could make filling them with the right data difficult.

I now realize that the constructor issue also prevents some of the editor scripting, since you have to write out a

.bytes

file and let Unity import it. Not sure if you can convert your bytes to a string, pass it to Unity and have it returned as a NativeArray without any corruption.

 

Oh, that's a good point.

There's no upper threshold, except the maximum dimension of a Texture3D (will downscale if needed). I've seen datasets larger than 512x512x512. (this is a volume rendering plugin btw, so the data are 3-dimensional datasets).
It would maybe be better to use a list of (flattened) 2D array instead (so 512 arrays of size 512x512), but I would still like to avoid storing them two places - in a managed arrays and native arrays...

 

Oh yes, that could be an issue too. I'll look into it! Also, I'm not sure how I'd load it into a NativeArray without first going through the byte[] array? Maybe a binary file stored in StreamingAssets is the way to go.. There doesn't really seem to be any nice and simple solutions for this, but I'll do some testing and share my findings here (if I get anywhere with it, haha). Thanks for your suggestion!

 

Hey Matias! I also had a use case where I needed to store a `NativeArray` as a field of a ScriptableObject, and used Adrian's suggestion of using a ".bytes" file and referencing that in a TextAsset. This process feels brittle, however, because I'll have an extra `.bytes` file in my project for each NativeArray that I want to store. I tried to add this `.bytes` file as a subasset of my ScriptableObject, but wasn't able to get that to work (more on that later).

Here's a sample of what I got working:

Code (CSharp):

1. public ExampleContainer : ScriptableObject
2. {
3.     public TextAsset exampleFloatDataAsset;
4.  
5.     public NativeArray<float> GetExampleFloatData() => exampleFloatDataAsset.GetData<float>();
6.  
7.     public void SaveExampleData(NativeArray<float> floatData)
8.     {
9.         var byteArray = GetRawBytes(floatData);
10.  
11.         var localFileName = "/data.bytes";
12.         var absolutePath = $"{Application.dataPath}{localFileName}";
13.         var projectRelativePath = $"Assets{localFileName}";
14.  
15.         //create a "data.bytes" with our float data
16.         File.WriteAllBytes(absolutePath, byteArray);
17.        
18.         // import "data.bytes" as a TextAsset"
19.         AssetDatabase.ImportAsset(projectRelativePath);
20.         var loadedAsset = AssetDatabase.LoadAssetAtPath<TextAsset>(projectRelativePath);
21.  
22.         exampleFloatDataAsset = loadedAsset;
23.  
24.         // some of these calls might not be necessary
25.         AssetDatabase.Refresh();
26.      
27.         EditorUtility.SetDirty(this);
28.      
29.         AssetDatabase.SaveAssetIfDirty(this);
30.     }
31.  
32.     // taken from: https://gist.github.com/asus4/a19118e04a0682d65cffe5c08911a498
33.     private static byte[] GetRawBytes<T>(NativeArray<T> arr) where T : struct
34.     {
35.         var slice = new NativeSlice<T>(arr).SliceConvert<byte>();
36.         var bytes = new byte[slice.Length];
37.         slice.CopyTo(bytes);
38.         return bytes;
39.     }
40. }
41.  

My attempt at saving it as a sub-asset was to Instantiate a clone of the loadedAsset, Add this clone as a subasset to my ScriptableObject, and then delete the ".bytes" file asset. This approach worked for me in the Editor, but would fail if I ever reloaded the Editor or at Runtime.

Code (CSharp):

1.  
2. // this approach doesn't work when reloading the editor, or at runtime
3.  
4. // ...
5. var loadedAsset = AssetDatabase.LoadAssetAtPath<TextAsset>(projectRelativePath);
6. var clone = Instantiate(loadedAsset);
7.  
8. AssetDatabase.AddObjectToAsset(clone, objectPath);
9.  
10. // deleting the created main bytes asset
11. AssetDatabase.DeleteAsset(projectRelativePath);
12.  
13. exampleFloatDataAsset = clone;
14.  
15. // ...
16.  

I'm curious if there is a way to make this work with subassets, or if there are other, cleaner approaches to storing NativeArray's in assets.

 

The Unity Serialization package helps: https://docs.unity3d.com/Packages/com.unity.serialization@3.1/manual/index.html

If native collections are still not supported, they're almost trivial to implement.

In any case, it would be cleaner to just have Unity serialize the byte array as you don't have to mingle with files at all:

Code (CSharp):

1. public ExampleContainer : ScriptableObject
2. {
3.     [SerializeField, HideInInspector] private byte[] serializedArray;

Then when serializing, just assign the resulting byte array to this field or a similar field to any other serializable type. I've done this in the past to implement a simple and straightforward undo/redo for a 3d grid cell editor.