BinaryFormatter Serialization saved file Broken when change from .NET 3.5 to .NET 4.x

My system mac Unity version 2018.3.4f1

The save file is saved by Scripting Runtime Version .NET 3.5. But if switch the Scripting Runtime Version .NET 4.x, the file cannot loaded. It detects the format is not correct.

What I found, using BinaryFormatter Serialization to save data, use Scripting Runtime Version .NET 3.5 can only open .NET 3.5 file, while using .NET 4.x only can open .NET 4.x Serialization file.

When using hex editor to open the two files. There is many different. The main is .net 3.5 file the opening is "Assembly-CSharp", but the .net 4.x file it opening is "FAssembly-CSharp"

I have a main problem, I am using serialization to save player game data, how do I upgrade the runtime to .net 4.x, but able to deserialize the .net 3.5 file?

Unity is deprecated .net 3.5, we need to find the way to read the serialization file when upgrade to .net 4.x

 

If unity moved to a different implementation of the .net framework for 4.x support (which I bet they did), this is definitely a possibility since the assemblies are different and may have different class/struct layouts in binary format.

This is what 'version lock' is all about. If your game is already released you usually tend to version lock that release and not move to new versions.

Unity still has .net 3.5 in Unity 2018.x, it's only deprecated. Deprecated does not mean "removed", it means it's "going to be removed in later versions. Specifically it'll be removed in 2019.x. As this article states:
https://forum.unity.com/threads/net-3-5-runtime-has-been-deprecated-in-unity-2018-3.601384/

As I said, if this is an already released title, you should get on the LTS (long term support) version of Unity 2018 and stay there in .net 3.5 for this game specifically.

...

If this is an unreleased game... well, who cares if the save file is popped. Delete and start over. It's an unreleased game.

 

This is why its always best to use human readable serialized file formats.

You could also build a converter pipeline. Open a file with the old binary formatter. Convert it to human readable. Then save again with the new binary formatter. (Or leave it as human readable, because that's better).

 

Use convert pipeline is one solution, but I am not way to check is it all users have upgrade to new version? If there is a user have not upgrade to new version, or upgraded to new version but have not open game let me to convert the saved file. Once I change the .Net framework to 4.x. They will lost the save file.

Convert the save file to .net independent file first, and then wait 2 years to not update the .Net version is one solution, but not a good idea. I want to find a method, let me can load the old .Net version file on new .Net version runtime.

Any idea?

 

The general idea is that you ship versions of the game which can open both files. Attempt to open the file with the newest version of the format. If that fails, attempt again with the old format. When you save, use the new format, so everyone's saved game gets updated when next they play it.

The trick is figuring out how to get one build to be able to open both versions of the data. I've not tried something like this, but can a .NET application use a DLL compiled with an older .NET version? If they build to compatible IL this might be a possibility, I'm not sure.

Human readable formats are often both big and slow, though.

That said, they do avoid dependencies like this one, though, and they are a much safer alternative for any kind of cross-platform data.

If it's unreleased then presumably giving people a converter tool would be acceptable? It could potentially even be hosted on a server somewhere so that it's transparent to players as long as they're online.

 

Your suggest is not possible, it is Unity problem. When you using .Net 3.5, you are no way to save as .net 4.x format. When you switch to .NET 4.x, it can not backward readable to .NET 3.5 format. I hope Unity can give a suggest how to do.

 

The real question is why would you change this many things under a released game? It's usually a very bad idea.

With that said: one solution is to build an application in unity which does only the conversion (so minimal app), build with 3.5 .NET, just load the save and convert to an intermediate format. Then the updated game app can read the temp file and save in the new format.

I would really advise against human readable formats when it comes to serialized game saves. (Depends on the save file size of course) It usually is super-slow and super-resource-hungry. Reading and parsing a giant string is painful in every language where the string objects are considered immutable. Although if you think this special circumstance will not be a rare event in the future, you can consider.

 

Something important about formatting changed between 3.5 and 4.x.
The framework is now using the current culture for most operation, whereas it was using the standard one previously.
I noticed it because I am in France, and we write "2,3" instead of "2.3", all my savegames were using US format ("2.3") and the files couldn't be read anymore.

Of course, it is more likely to hit harder text serialization rather than binary serialization, but it still may be the origin of your problem.
I fixed it by forcing the use of the US culture for all the threads at the very start of the game.

 

If you want your current users to still be playing the game years down the line, you'll almost certainly need to add features and updates somewhere along the line to keep them engaged.

Also, especially where mobile apps are concerned, as new devices and versions of the OS are released, updates will sometimes be required to fix crashes with new devices. In addition, sometimes particular bugs can only be fixed by building with a newer version of Unity, if the bug is in Unity rather than your own code.

As long as proper testing is done beforehand, making changes to an already released game doesn't need to be a problem, but I wouldn't recommend it without good reason.

Edit: Also from time to time Google Play change the minimum requirements for uploading apps (including updates to existing apps), which means that fixing bugs in an older app may require a larger update than expected.

 

Mind you, when I say "unreleased", I mean that no one would have the game. It's unreleased.

As opposed to early-release or something.

 

Totally agree, this is why I talked about 'version-lock'.

And I mean version-lock of the engine, not version-lock of your product. You can still develop for it, just don't move forward on the version of the engine. LTS versions are always really good for this.

The behaviour is common in the development world. For example here in my office (enterprise software) we still use VS2013 because of version-lock. We plan to move to the latest version in the next year or so, but have been on version-lock for quite some time due to the state of our product. We don't want to introduce unknown bugs to the system.

Our games Prototype Mansion and Garden Variety Body Horror are also on version lock as well. I plan to push to LTS in the next month, but they will remain on LTS for the release of our 3rd game in that trilogy. We won't be moving to newer versions of Unity until our title following that. There's nothing we can't do in LTS currently. I mean sure we might like some features in 4.x, but we don't "need" those features. I've been putting up with .net 3.5 for years despite in other jobs I do I'm often on the latest version of .net (with the exception of this enterprise product I'm working on right now).

I use json all the time for my save files.

Of course don't load the whole file in as a string, that would definitely chew up memory. So I use StreamReader/TextReader to stream the file.

Like here I use a TextReader and StringBuilder to reduce memory consumption (and gc) while parsing in json:
https://github.com/lordofduct/space...erialization/Serialization/Json/JsonReader.cs
(of course you can't nil all of it out, memory consumption is going to happen)

And a TextWriter when writing (though there's not a whole lot of string parsing when writing... pretty straight forward):
https://github.com/lordofduct/space...erialization/Serialization/Json/JsonWriter.cs

With all that said, it still is a fatter file and slower to parse than binary. But I barely notice a difference in game. But my games aren't that massive. My save files are only a few hundred lines at most.

 

I haven't said anything about the game itself. I said this about the tool you use to develop such game. But see the detailed explanation at @lordofduct he nicely explained it while I was sleeping.

This is why I mentioned that it depends on the size of the save. Sometimes it does not worth the development time to do one way or another. So yeah it really depends what and how much you read/write and how often and in what situations.

 

Any Unity guys there? Can reply something?

 

I found that after change the framework version, it is only unable to read the "Dictionary" data, other data can success Deserialize.

 

I think it should be bugs of Unity.
The Dictionary data can success deserialize when change the .net version, if the Dictionary data is not initialize or contain data. ie. if the dictionary data cont != 0 or not initialize the "BinaryFormatter Serialization" will not broken.

Therefore, it must the bugs of Unity.

 

Has anyone found a solution to this yet?

I've tried loading in the .net 3.5 DLL to do the loading part when it fails in the 4.0 version, but haven't gotten anywhere with that.

My other solution is to find a 3rd party (de)serializer that can load in the 3.5 version and then convert it to its own format. I think this is the more likely one to actually work, but I haven't found a library yet that can handle this.

 

Not a Unity bug. It is expected behavior. It is a bug in your save system that you inadvertently included a dependency on the output of BinaryFormatter serialization never changing between .net versions.

If I were writing a solution for this here's what I would do:

1) Create a separate application with .net 3.5 which takes a command line switch directing it to old save files
-- This application reads in the old save files and then saves them in your own text format, with no dependency on .net versions (so no BinaryFormatter)
-- Make sure you force a specific culture setting when outputting things like numbers to the text file save
2) Add old save file detection to your game, either try to read the save file and check for failure, or open the save file and check for something that gives it away as an old file
3) When an old save file is detected, you launch the application from step 1
4) Check for new text format saves from the step 1 application, and when seen your game converts them to your current format
-- Make sure you force the same culture when reading the text file that you forced when saving it
-- I'd suggest your current format to not be dependent on .net versions, so maybe you use the text file format as your current save format, but that's up to you

 

How can you explain, the deserialize will success, if it has data?
I have a solution, I alway write a dummy data in to the dictionary variable. The serialization and deserialization will no problem.

 

I don't understand your question. If you're having no problem, then what was the point of this thread? I thought the point was you are having a problem.

All I was saying is the source of your problem isn't a Unity bug.

 

Did you find any workaround?

 

Add dummy data, anything is okay. The dictionary is not empty will be fine.

 

I cannot add anything into the .net 3.5 save file and the dictionary object is created during the deserialization. It's not clear at all what you're saying...

 

OK, it looks a bit silly but, as Oscar-Tsang said, at load time it raises an exception in OnDeserialization only for dictionaries that were empty at the save moment. Here's what I did in order to catch and ignore the exception:

1. Before the save object deserialization, I set a binder to the binaryFormatter:

Code (CSharp):

1. binaryFormatter.Binder = new SaveDataDeserializationBinder();
2. data = (SaveData)binaryFormatter.Deserialize(file);

2. In the binder I redirect the Dictionary type to a custom NoExceptionDictionary type:

Code (CSharp):

1. class SaveDataDeserializationBinder : SerializationBinder
2. {
3.     public override Type BindToType(string assemblyName, string typeName)
4.     {
5.         if (typeName.StartsWith("System.Collections.Generic.Dictionary"))
6.         {
7.             typeName = typeName.Replace("System.Collections.Generic.Dictionary", "NoExceptionDictionary");
8.             assemblyName = Assembly.GetExecutingAssembly().GetName().Name;
9.         }
10.  
11.         return Type.GetType(String.Format("{0}, {1}", typeName, assemblyName));
12.     }
13. }

3. NoExceptionDictionary.OnDeserialization catches the exception raised by Dictionary.OnDeserialization:

Code (CSharp):

1. [System.Serializable]
2. public class NoExceptionDictionary<TKey, TValue> : Dictionary<TKey, TValue>
3. {
4.     public NoExceptionDictionary() : base() {}
5.     public NoExceptionDictionary(SerializationInfo info, StreamingContext context) : base(info, context) {}
6.  
7.     public override void OnDeserialization(object sender)
8.     {
9.         try
10.         {
11.             base.OnDeserialization(sender);
12.         }
13.         catch
14.         {
15.         }
16.     }
17. }

As the exception did occur only for empty dictionaries, the old save files seem to load complete with these changes.

 

I'm having the exact same issue. I'm updating my game to Unity 2019.2, and it's unable to deserialize save games generated with the current release version of my game, built on Unity 2018.

The error I'm receiving is: “SerializationException: The keys for this dictionary are missing.” The save file is fine, as it deserializes without issue in the Unity 2018 build of my game.

I tried the SerializationBinder solution that @cmihalache put together, and unfortunately, it gave me the same error. Any ideas?

 

I managed to solve this with a hacky workaround, and will definitely be using a different serialization method in the future. Thanks to @cmihalache for getting me started down this road!

I created a SerializationBinder that switches the Dictionary objects in my data to a new class, OldSaveDataDictionary:

Code (CSharp):

1. class OldSaveDataDeserializationBinder : SerializationBinder
2.     {
3.         public override Type BindToType (string assemblyName, string typeName)
4.         {
5.             if (typeName.StartsWith("System.Collections.Generic.Dictionary"))
6.             {
7.                 return typeof(OldSaveDataDictionary<string, OldSaveDataDictionary<string, object>>);
8.             }
9.             else
10.             {
11.                 return Type.GetType(String.Format("{0}, {1}", typeName, assemblyName));
12.             }        
13.         }
14.     }

OldSaveDataDictionary then enumerates through the KeyValuePairs, and passes the data on to my internal Dictionary. A slightly simplified version:

Code (CSharp):

1. [System.Serializable]
2.     public class OldSaveDataDictionary<TKey, TValue> : Dictionary<TKey, TValue>
3.     {
4.         public OldSaveDataDictionary() : base() {}
5.         public OldSaveDataDictionary(SerializationInfo info, StreamingContext context) : base(info, context)
6.         {
7.             var enumerator = info.GetEnumerator();
8.             while (enumerator.MoveNext())
9.             {
10.                 var current = enumerator.Current;
11.                 if (current.Name == "KeyValuePairs" && current.ObjectType == typeof(KeyValuePair<string, object>[]))
12.                 {
13.                     KeyValuePair<string, object>[] keyValueArray = (KeyValuePair<string, object>[])current.Value;
14.                     Dictionary<string, object> currDictionary = GetRoomStateDictionary();
15.                     for (int i=0; i < keyValueArray.Length; ++i)
16.                     {
17.                         currDictionary[keyValueArray[i].Key] = keyValueArray[i].Value;
18.                     }
19.                 }
20.             }
21.         }
22.    
23.         public override void OnDeserialization(object sender)
24.         {
25.             try
26.             {
27.                 base.OnDeserialization(sender);
28.             }
29.             catch {}
30.         }
31.     }

This worked perfectly. Moral of the story: Don’t serialize persistent data with BinaryFormatter, as it can change from version to version, making the data difficult to read.

Thanks!

- David

 

Expanding on @BrainAndBrain's solution. It looks the problem had something to do with the type info in the binary file. For the dictionary, just returning the dictionary type and generic parameters type seems to make everything work. This is the SerializationBinder I ended up with:

Code (CSharp):

1. public class OlderDotNetVersionMigrationBinder : SerializationBinder
2. {
3.     public override Type BindToType(string assemblyName, string typeName)
4.     {
5.         if (typeName.StartsWith("System.Collections.Generic.Dictionary"))
6.         {
7.             var readType = Type.GetType(typeName);
8.  
9.             return typeof(Dictionary<,>).MakeGenericType(readType.GenericTypeArguments);
10.         }
11.  
12.         return Type.GetType(String.Format("{0}, {1}", typeName, assemblyName));
13.     }
14. }

Using this binder I did not need to create a new serializable class.

 

@WallisKelsey That's much more elegant! Thanks for sharing!

 

Thanks for sharing your fixes! I didn't get any of your snippets to work out of the box, but this worked for me, for some reason:

Code (CSharp):

1. public class OlderDotNetVersionMigrationBinder : SerializationBinder
2.     {
3.     public override Type BindToType(string assemblyName, string typeName)
4.         {
5.         if (typeName.StartsWith("System.Collections.Generic.Dictionary"))
6.             {
7.             var readType = Type.GetType(typeName);
8.  
9.             return typeof(OldSaveDataDictionary<,>).MakeGenericType(readType.GenericTypeArguments);
10.             }
11.         else
12.             {
13.             return Type.GetType(String.Format("{0}, {1}", typeName, assemblyName));
14.             }
15.         }
16.  
17.     }
18.  
19. [System.Serializable]
20. public class OldSaveDataDictionary<TKey, TValue> : Dictionary<TKey, TValue>
21.     {
22.     public OldSaveDataDictionary() : base() { }
23.     public OldSaveDataDictionary(SerializationInfo info, StreamingContext context) : base(info, context)
24.         {
25.         }
26.  
27.     public override void OnDeserialization(object sender)
28.         {
29.         try
30.             {
31.             base.OnDeserialization(sender);
32.             }
33.         catch { }
34.         }
35.     }

I don't know how the dictionaries are actually deserialized properly, even though the constructor is empty, but all the data is there.

 

Based on what you have left, Unity Editor works normally.

However, if you make a build and check it on the Android phone, the error as below occurs.
=========================================

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ExecutionEngineException: Attempting to call method 'NoExceptionDictionary`2[[System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]::.ctor' for which no ahead of time (AOT) code was generated.

==========================================
Is there any solution?

 

I want to know how you solved this problem T-T
I faced same problem now.

If you solved it.
Please tell me how to do.