Best Option for Slow Functions, so they dont freeze the main thread? (Async? Coroutine? Threading?)

Hi there,


I have an issue and ive tried using Coroutines, AsyncOperations, and Tasks and now System.Threading.Thread to try to move the operation to a different thread, so that it does not stop the main thread, and thus times out network connections and disconnects the client.

Basicly, World data is serialized and sent to the client,

This information is only Serialized Once on the server when generated, and stored, to send to clients, as needed, so the problem does not exist server side, its simply "Part of the loading process"

The Client Receives the data, and begins deserializing it. This operation Takes some time, 1 - 2 minutes on a good cpu.... you may ask what im sending, that much data. two reasons, 1) to test the limits and 2) Big world :\

What is the best way to de-serialize this data, and not freeze the main thread?
System.Threading, Dosnt seem to freeze unity editor, but the clients still timeout disconnect
Tasks Seem to freeze it but i could be using them incorrectly.
And ASyncronic operations seem to freeze unity aswell as Coroutines.


Any ideas? Thanks in advanced

Heres some Example Code im using

Code (CSharp):

1.        
2. public void ImportFromStreamThreaded(Stream inputStream)
3.         {
4.             BinaryFormatter bf = SerializationSurrogateUtil.GetBinaryFormatter();
5.             var exportData = (ExportData)bf.Deserialize(inputStream);
6.             PersistentVegetationCellList = exportData.PersistentVegetationCellList;
7.             PersistentVegetationInstanceInfoList = exportData.PersistentVegetationInstanceInfoList;
8.             PersistentVegetationInstanceSourceList = exportData.PersistentVegetationInstanceSourceList;
9.             inputStream.Position = 0;
10.         }
11.  
12.  
13.  
14.     System.Threading.Thread myThread;
15.     AwesomeTechnologies.Vegetation.PersistentStorage.PersistentVegetationStoragePackage t;
16.     public void ImportVegetation()
17.     {
18.         thisarray = GetComponent<ZacksNetworkTerrain>().StoragePackageByteArray;
19.         t = ScriptableObject.CreateInstance<StoragePackage>();
20.         myThread = new System.Threading.Thread(Op);
21.         myThread.Start();
22.  
23.     }
24.     public bool Imported = false;
25.     byte[] thisarray;
26.     public void Op()
27.     {
28.         byte[] array = thisarray;
29.  
30.         if (array != null)
31.         {
32.             Debug.Log("Starting Import....");
33.             System.IO.Stream stream = new System.IO.MemoryStream(array);
34.             t.ImportFromStreamThreaded(stream);
35.             Imported = true;
36.             Debug.Log("Ending Import");
37.            
38.         }
39.         else
40.         {
41.             Debug.Log("Array is null");
42.            
43.         }
44.     }

 

You 100% want to be doing network operations on a thread that is not the main thread. That means Tasks or System.Threading, or a networking library that puts requests on a background thread automatically (which will use System.Threading under the hood eventually).

If you're getting a networking error like this then your first bottleneck actually has nothing to do with deserialization, it's that the data transfer is taking too long. You have a few ways to resolve this particular problem:

• Get a faster network connection between your client and server
• Increase the client-side request timeout (and check your server request timeout as well if that applies)
• Transfer less data. This could be done by excluding irrelevant data, breaking the data into smaller chunks that you request one at a time, through compression, through a more compact over-the-wire data format, or some combination of those.

After you are able to reliably get the data copied into your game's memory, then you can worry about deserialization.

 

Thanks for the reply. I can actually see my data come through and complete, and be stored on the client, and the client starts deserializing the main package, Within That package is a byte array for Vegetation. When i try to Import Via Deserialzation the vegetation with the code above, ( Op() Method )

Ive attached a Log File

Notice The Timestamp from 2:03:30 to 2:06:25 and then once it stops its freeze (Presumably because the deserialization) the client disconnects.

A couple things.... Yes Im sending like 50 megs via Cmd/Target RPC, so i can test my end result, my plan was to come back and rework the sending of the data to be faster, because im bound by like a 1KB buffer with unity... If you have ANY Suggestion on how to send this amount of data easily, i would love a URL :\ but for now i just wanted to get it sending to prove i could actually do it on client end.

I can try to generate an island with just a couple items, to make this much smaller for testing, infact i just thought of this, -_-

However based on the Log Below, im Certain the client is receiving all data... i can see it come in 1KB at a time....
Infact, my test prior to this was all the TerrainData, and i am able to successfully Re-Create the Terrain with all Splatmap and textures on the client side.... So i moved on to vegetation...

No, i cannot use Unitys, build in vegetation system with its Tree Prototypes and detail within the terrainData Object... It does not have the scalability i require and integration with other plugins i am using (VegetationStudio Pro, MapMagic, And alot more...)

Im open to any suggestions, always willing to learn, Thank you very much in Advanced

Zack

 

Well i thank you for your help, as some times talking it out helps

Part of my problem was yes the speed of the transfer starting the timeout process, after minimizing what i was sending from 50MB to about 1.5 mb, i could fix the other problems occuring, and i finnaly got a good test.

My initial tests run on CMD/RCP as stated above, but i have a working Async socket connection that i will integrate to remove the CMD/RCP,

Thanks you

 

Whenever someone uses the BinaryFormatter I like to add this security recommendation that Microsoft itself put up. You generally should avoid it especially when you want to use it in production in a large scale project as it's an inherently insecure class. I know that it's easy to use and one of the most powerful serializers (as it can serialize almost anything by default). Though security becomes a more and more important topic in game and application development in general.