TECHNICAL ISSUE: Very slow serialization

hi,

I don't know if this is a silly point or not, but this is what I'm experiencing.

I'm writing an editor extension where the user can check an option to process their data into another form, this option creates about 4.6k of "real" garbage, but in the next editor frame ~2.1MB of garbage is created in ~32k calls, and this can take up to 4s.

I got around this issue by making that processed data non-serialized (which is not really ideal, but I can live with it).

Is there a way around this?
Anyone else experienced this?

Thanks.

 

Probably worth trying binary serialisation if you aren't already - it's much smaller. Alternatively try a lightweight serialiser like this: https://forum.unity.com/threads/bin...inary-serialization-library-for-unity.495707/

 

I'll look into that, thanks. I just think it's strange that the serialization would be that inefficient.

 

So about 67 bytes per 'call', and 125 microseconds per call.

I don't know what a "call" necessarily is. But that doesn't sound like a whole lot of overhead IMO. 67 bytes of garbage isn't much... that's this sentence:

"Hello world, my name is lordofduct; I am a member of Unity forums."

Just that string alone would create that much garbage. And considering that most serialization is converting data into text (usually, binary serializers doesn't). Strings are a good comparison of that. And converting to text is not a simple task. I mean really consider it... what goes into converting a number into its string?

11101110000100

How do you convert that into: "15236"?

What steps do you have to do for that? Like actually do in processor terms? And that's just a single 32-bit integer.

...

Now what I'm wondering is. Why 32k calls? What is a call? Can that be reduced? How much data is being processed? Is this happening regularly through out process causing repeated interruption or does it just happen every once in a while like when clicking a 'process' button, or loading a new scene in the editor?

Lots of things take time in the editor. If every once in a while the editor has to process some data, that's no big deal. I wouldn't worry about it. Developers are used to have to process large amounts of data regularly, and know that it takes time.

The places things like garbage and slow-down should be a concern is at runtime. When a 'player' is playing the game. You don't want to stall out for 4 seconds while in the middle of a heated battle.

But editor time? Pfff, whatever.

 

My actual application is that I'm taking a polygon as described by an array of Vector3s, and processing that in user-defined ways (Catmull-Rom interpolation, shatter, etc.), my issue comes up when I enable or disable one of those modifiers. The actual processing takes 3~10ms, based on the size of the dataset, but after my processing is done Unity hooks for ~4seconds+ serializing that data.

That's completely nuts. It's an action a user can perform quite often (think of it like turning layers on/off when comping something in Photoshop), it's like expecting a MeshRenderer to hook the editor for 4 seconds when you turn it off. To me, that's a problem.

 

For your edge case, that can be a problem.

But it's also a logical fallout of what is going on. Serialization is not a cheap/easy task, it takes time. Especially on a large data set, like a huge set of vectors.

If you need the serialization, then there's nothing else you can really do about it. Unless you off loaded serialization to your own system and did the work when you deemed it necessary.

Otherwise, I'd just put a warning message in the editor letting users know that modifying that data can cause Unity to recalculate for a while and to use it sparingly.

In the same respect, I don't know what your dataset even looks like, or your editor script. So who knows, maybe there are optimizations that can be made. But as far as I know, it's just a really big array of vectors. (how many are we talking, cause that sounds like a lot)

 

If you put the data in a ScriptableObject - either as an asset or bundled in the scene with whatever object's storing the data - you can mark it with PreferBinarySerialization. That both speeds up the serialization process, and massively reduces the asset's size on disk.