BinaryFormatter Serialization saved file Broken when change from .NET 3.5 to .NET 4.x

My system mac Unity version 2018.3.4f1

The save file is saved by Scripting Runtime Version .NET 3.5. But if switch the Scripting Runtime Version .NET 4.x, the file cannot loaded. It detects the format is not correct.

What I found, using BinaryFormatter Serialization to save data, use Scripting Runtime Version .NET 3.5 can only open .NET 3.5 file, while using .NET 4.x only can open .NET 4.x Serialization file.

When using hex editor to open the two files. There is many different. The main is .net 3.5 file the opening is "Assembly-CSharp", but the .net 4.x file it opening is "FAssembly-CSharp"

I have a main problem, I am using serialization to save player game data, how do I upgrade the runtime to .net 4.x, but able to deserialize the .net 3.5 file?

Unity is deprecated .net 3.5, we need to find the way to read the serialization file when upgrade to .net 4.x

 

If unity moved to a different implementation of the .net framework for 4.x support (which I bet they did), this is definitely a possibility since the assemblies are different and may have different class/struct layouts in binary format.

This is what 'version lock' is all about. If your game is already released you usually tend to version lock that release and not move to new versions.

Unity still has .net 3.5 in Unity 2018.x, it's only deprecated. Deprecated does not mean "removed", it means it's "going to be removed in later versions. Specifically it'll be removed in 2019.x. As this article states:
https://forum.unity.com/threads/net-3-5-runtime-has-been-deprecated-in-unity-2018-3.601384/

As I said, if this is an already released title, you should get on the LTS (long term support) version of Unity 2018 and stay there in .net 3.5 for this game specifically.

...

If this is an unreleased game... well, who cares if the save file is popped. Delete and start over. It's an unreleased game.

 

This is why its always best to use human readable serialized file formats.

You could also build a converter pipeline. Open a file with the old binary formatter. Convert it to human readable. Then save again with the new binary formatter. (Or leave it as human readable, because that's better).

 

Use convert pipeline is one solution, but I am not way to check is it all users have upgrade to new version? If there is a user have not upgrade to new version, or upgraded to new version but have not open game let me to convert the saved file. Once I change the .Net framework to 4.x. They will lost the save file.

Convert the save file to .net independent file first, and then wait 2 years to not update the .Net version is one solution, but not a good idea. I want to find a method, let me can load the old .Net version file on new .Net version runtime.

Any idea?

 

The general idea is that you ship versions of the game which can open both files. Attempt to open the file with the newest version of the format. If that fails, attempt again with the old format. When you save, use the new format, so everyone's saved game gets updated when next they play it.

The trick is figuring out how to get one build to be able to open both versions of the data. I've not tried something like this, but can a .NET application use a DLL compiled with an older .NET version? If they build to compatible IL this might be a possibility, I'm not sure.

Human readable formats are often both big and slow, though.

That said, they do avoid dependencies like this one, though, and they are a much safer alternative for any kind of cross-platform data.

If it's unreleased then presumably giving people a converter tool would be acceptable? It could potentially even be hosted on a server somewhere so that it's transparent to players as long as they're online.

 

Your suggest is not possible, it is Unity problem. When you using .Net 3.5, you are no way to save as .net 4.x format. When you switch to .NET 4.x, it can not backward readable to .NET 3.5 format. I hope Unity can give a suggest how to do.

 

The real question is why would you change this many things under a released game? It's usually a very bad idea.

With that said: one solution is to build an application in unity which does only the conversion (so minimal app), build with 3.5 .NET, just load the save and convert to an intermediate format. Then the updated game app can read the temp file and save in the new format.

I would really advise against human readable formats when it comes to serialized game saves. (Depends on the save file size of course) It usually is super-slow and super-resource-hungry. Reading and parsing a giant string is painful in every language where the string objects are considered immutable. Although if you think this special circumstance will not be a rare event in the future, you can consider.

 

Something important about formatting changed between 3.5 and 4.x.
The framework is now using the current culture for most operation, whereas it was using the standard one previously.
I noticed it because I am in France, and we write "2,3" instead of "2.3", all my savegames were using US format ("2.3") and the files couldn't be read anymore.

Of course, it is more likely to hit harder text serialization rather than binary serialization, but it still may be the origin of your problem.
I fixed it by forcing the use of the US culture for all the threads at the very start of the game.

 

If you want your current users to still be playing the game years down the line, you'll almost certainly need to add features and updates somewhere along the line to keep them engaged.

Also, especially where mobile apps are concerned, as new devices and versions of the OS are released, updates will sometimes be required to fix crashes with new devices. In addition, sometimes particular bugs can only be fixed by building with a newer version of Unity, if the bug is in Unity rather than your own code.

As long as proper testing is done beforehand, making changes to an already released game doesn't need to be a problem, but I wouldn't recommend it without good reason.

Edit: Also from time to time Google Play change the minimum requirements for uploading apps (including updates to existing apps), which means that fixing bugs in an older app may require a larger update than expected.

 

Mind you, when I say "unreleased", I mean that no one would have the game. It's unreleased.

As opposed to early-release or something.

 

Totally agree, this is why I talked about 'version-lock'.

And I mean version-lock of the engine, not version-lock of your product. You can still develop for it, just don't move forward on the version of the engine. LTS versions are always really good for this.

The behaviour is common in the development world. For example here in my office (enterprise software) we still use VS2013 because of version-lock. We plan to move to the latest version in the next year or so, but have been on version-lock for quite some time due to the state of our product. We don't want to introduce unknown bugs to the system.

Our games Prototype Mansion and Garden Variety Body Horror are also on version lock as well. I plan to push to LTS in the next month, but they will remain on LTS for the release of our 3rd game in that trilogy. We won't be moving to newer versions of Unity until our title following that. There's nothing we can't do in LTS currently. I mean sure we might like some features in 4.x, but we don't "need" those features. I've been putting up with .net 3.5 for years despite in other jobs I do I'm often on the latest version of .net (with the exception of this enterprise product I'm working on right now).

I use json all the time for my save files.

Of course don't load the whole file in as a string, that would definitely chew up memory. So I use StreamReader/TextReader to stream the file.

Like here I use a TextReader and StringBuilder to reduce memory consumption (and gc) while parsing in json:
https://github.com/lordofduct/space...erialization/Serialization/Json/JsonReader.cs
(of course you can't nil all of it out, memory consumption is going to happen)

And a TextWriter when writing (though there's not a whole lot of string parsing when writing... pretty straight forward):
https://github.com/lordofduct/space...erialization/Serialization/Json/JsonWriter.cs

With all that said, it still is a fatter file and slower to parse than binary. But I barely notice a difference in game. But my games aren't that massive. My save files are only a few hundred lines at most.

 

I haven't said anything about the game itself. I said this about the tool you use to develop such game. But see the detailed explanation at @lordofduct he nicely explained it while I was sleeping.

This is why I mentioned that it depends on the size of the save. Sometimes it does not worth the development time to do one way or another. So yeah it really depends what and how much you read/write and how often and in what situations.

 

Any Unity guys there? Can reply something?

 

I found that after change the framework version, it is only unable to read the "Dictionary" data, other data can success Deserialize.

 

I think it should be bugs of Unity.
The Dictionary data can success deserialize when change the .net version, if the Dictionary data is not initialize or contain data. ie. if the dictionary data cont != 0 or not initialize the "BinaryFormatter Serialization" will not broken.

Therefore, it must the bugs of Unity.

 

Has anyone found a solution to this yet?

I've tried loading in the .net 3.5 DLL to do the loading part when it fails in the 4.0 version, but haven't gotten anywhere with that.

My other solution is to find a 3rd party (de)serializer that can load in the 3.5 version and then convert it to its own format. I think this is the more likely one to actually work, but I haven't found a library yet that can handle this.

 

Not a Unity bug. It is expected behavior. It is a bug in your save system that you inadvertently included a dependency on the output of BinaryFormatter serialization never changing between .net versions.

If I were writing a solution for this here's what I would do:

1) Create a separate application with .net 3.5 which takes a command line switch directing it to old save files
-- This application reads in the old save files and then saves them in your own text format, with no dependency on .net versions (so no BinaryFormatter)
-- Make sure you force a specific culture setting when outputting things like numbers to the text file save
2) Add old save file detection to your game, either try to read the save file and check for failure, or open the save file and check for something that gives it away as an old file
3) When an old save file is detected, you launch the application from step 1
4) Check for new text format saves from the step 1 application, and when seen your game converts them to your current format
-- Make sure you force the same culture when reading the text file that you forced when saving it
-- I'd suggest your current format to not be dependent on .net versions, so maybe you use the text file format as your current save format, but that's up to you

 

How can you explain, the deserialize will success, if it has data?
I have a solution, I alway write a dummy data in to the dictionary variable. The serialization and deserialization will no problem.

 

I don't understand your question. If you're having no problem, then what was the point of this thread? I thought the point was you are having a problem.

All I was saying is the source of your problem isn't a Unity bug.

 

Did you find any workaround?

 

Add dummy data, anything is okay. The dictionary is not empty will be fine.

 

I cannot add anything into the .net 3.5 save file and the dictionary object is created during the deserialization. It's not clear at all what you're saying...

 

OK, it looks a bit silly but, as Oscar-Tsang said, at load time it raises an exception in OnDeserialization only for dictionaries that were empty at the save moment. Here's what I did in order to catch and ignore the exception:

1. Before the save object deserialization, I set a binder to the binaryFormatter:

Code (CSharp):

1. binaryFormatter.Binder = new SaveDataDeserializationBinder();
2. data = (SaveData)binaryFormatter.Deserialize(file);

2. In the binder I redirect the Dictionary type to a custom NoExceptionDictionary type:

Code (CSharp):

1. class SaveDataDeserializationBinder : SerializationBinder
2. {
3.     public override Type BindToType(string assemblyName, string typeName)
4.     {
5.         if (typeName.StartsWith("System.Collections.Generic.Dictionary"))
6.         {
7.             typeName = typeName.Replace("System.Collections.Generic.Dictionary", "NoExceptionDictionary");
8.             assemblyName = Assembly.GetExecutingAssembly().GetName().Name;
9.         }
10.  
11.         return Type.GetType(String.Format("{0}, {1}", typeName, assemblyName));
12.     }
13. }

3. NoExceptionDictionary.OnDeserialization catches the exception raised by Dictionary.OnDeserialization:

Code (CSharp):

1. [System.Serializable]
2. public class NoExceptionDictionary<TKey, TValue> : Dictionary<TKey, TValue>
3. {
4.     public NoExceptionDictionary() : base() {}
5.     public NoExceptionDictionary(SerializationInfo info, StreamingContext context) : base(info, context) {}
6.  
7.     public override void OnDeserialization(object sender)
8.     {
9.         try
10.         {
11.             base.OnDeserialization(sender);
12.         }
13.         catch
14.         {
15.         }
16.     }
17. }

As the exception did occur only for empty dictionaries, the old save files seem to load complete with these changes.

 

I'm having the exact same issue. I'm updating my game to Unity 2019.2, and it's unable to deserialize save games generated with the current release version of my game, built on Unity 2018.

The error I'm receiving is: “SerializationException: The keys for this dictionary are missing.” The save file is fine, as it deserializes without issue in the Unity 2018 build of my game.

I tried the SerializationBinder solution that @cmihalache put together, and unfortunately, it gave me the same error. Any ideas?

 

I managed to solve this with a hacky workaround, and will definitely be using a different serialization method in the future. Thanks to @cmihalache for getting me started down this road!

I created a SerializationBinder that switches the Dictionary objects in my data to a new class, OldSaveDataDictionary:

Code (CSharp):

1. class OldSaveDataDeserializationBinder : SerializationBinder
2.     {
3.         public override Type BindToType (string assemblyName, string typeName)
4.         {
5.             if (typeName.StartsWith("System.Collections.Generic.Dictionary"))
6.             {
7.                 return typeof(OldSaveDataDictionary<string, OldSaveDataDictionary<string, object>>);
8.             }
9.             else
10.             {
11.                 return Type.GetType(String.Format("{0}, {1}", typeName, assemblyName));
12.             }        
13.         }
14.     }

OldSaveDataDictionary then enumerates through the KeyValuePairs, and passes the data on to my internal Dictionary. A slightly simplified version:

Code (CSharp):

1. [System.Serializable]
2.     public class OldSaveDataDictionary<TKey, TValue> : Dictionary<TKey, TValue>
3.     {
4.         public OldSaveDataDictionary() : base() {}
5.         public OldSaveDataDictionary(SerializationInfo info, StreamingContext context) : base(info, context)
6.         {
7.             var enumerator = info.GetEnumerator();
8.             while (enumerator.MoveNext())
9.             {
10.                 var current = enumerator.Current;
11.                 if (current.Name == "KeyValuePairs" && current.ObjectType == typeof(KeyValuePair<string, object>[]))
12.                 {
13.                     KeyValuePair<string, object>[] keyValueArray = (KeyValuePair<string, object>[])current.Value;
14.                     Dictionary<string, object> currDictionary = GetRoomStateDictionary();
15.                     for (int i=0; i < keyValueArray.Length; ++i)
16.                     {
17.                         currDictionary[keyValueArray[i].Key] = keyValueArray[i].Value;
18.                     }
19.                 }
20.             }
21.         }
22.    
23.         public override void OnDeserialization(object sender)
24.         {
25.             try
26.             {
27.                 base.OnDeserialization(sender);
28.             }
29.             catch {}
30.         }
31.     }

This worked perfectly. Moral of the story: Don’t serialize persistent data with BinaryFormatter, as it can change from version to version, making the data difficult to read.

Thanks!

- David

 

Expanding on @BrainAndBrain's solution. It looks the problem had something to do with the type info in the binary file. For the dictionary, just returning the dictionary type and generic parameters type seems to make everything work. This is the SerializationBinder I ended up with:

Code (CSharp):

1. public class OlderDotNetVersionMigrationBinder : SerializationBinder
2. {
3.     public override Type BindToType(string assemblyName, string typeName)
4.     {
5.         if (typeName.StartsWith("System.Collections.Generic.Dictionary"))
6.         {
7.             var readType = Type.GetType(typeName);
8.  
9.             return typeof(Dictionary<,>).MakeGenericType(readType.GenericTypeArguments);
10.         }
11.  
12.         return Type.GetType(String.Format("{0}, {1}", typeName, assemblyName));
13.     }
14. }

Using this binder I did not need to create a new serializable class.

 

@WallisKelsey That's much more elegant! Thanks for sharing!

 

Thanks for sharing your fixes! I didn't get any of your snippets to work out of the box, but this worked for me, for some reason:

Code (CSharp):

1. public class OlderDotNetVersionMigrationBinder : SerializationBinder
2.     {
3.     public override Type BindToType(string assemblyName, string typeName)
4.         {
5.         if (typeName.StartsWith("System.Collections.Generic.Dictionary"))
6.             {
7.             var readType = Type.GetType(typeName);
8.  
9.             return typeof(OldSaveDataDictionary<,>).MakeGenericType(readType.GenericTypeArguments);
10.             }
11.         else
12.             {
13.             return Type.GetType(String.Format("{0}, {1}", typeName, assemblyName));
14.             }
15.         }
16.  
17.     }
18.  
19. [System.Serializable]
20. public class OldSaveDataDictionary<TKey, TValue> : Dictionary<TKey, TValue>
21.     {
22.     public OldSaveDataDictionary() : base() { }
23.     public OldSaveDataDictionary(SerializationInfo info, StreamingContext context) : base(info, context)
24.         {
25.         }
26.  
27.     public override void OnDeserialization(object sender)
28.         {
29.         try
30.             {
31.             base.OnDeserialization(sender);
32.             }
33.         catch { }
34.         }
35.     }

I don't know how the dictionaries are actually deserialized properly, even though the constructor is empty, but all the data is there.

 

Based on what you have left, Unity Editor works normally.

However, if you make a build and check it on the Android phone, the error as below occurs.
=========================================

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ExecutionEngineException: Attempting to call method 'NoExceptionDictionary`2[[System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.Int32, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]]::.ctor' for which no ahead of time (AOT) code was generated.

==========================================
Is there any solution?

 

I want to know how you solved this problem T-T
I faced same problem now.

If you solved it.
Please tell me how to do.

 

Anyone figure out the cause of this exception?

Thanks!

 

That exception basically means a type could not be found to create an instance.

The reason this is happening is that builds to Android use an AOT compilation approach (hence the AOT in the error). This goes for old school AOT and new school IL2CPP.

See, when you target a mono/.net build you get the mono/.net runtime. This is a JIT compiled runtime that can generate types at runtime. Generics (the <T> stuff) takes advantage of this.

AOT/IL2CPP on the other hand doesn't come with the JIT compiler/runtime setup. Instead the compiler compiles everything ahead of time.

The problem is, how does it know what types to compile? Well it looks through your code and figures out all the types you've referenced and compiles it. For generic classes it susses that out from all the <T> that are written explicitly... like List<int> means we need to compile a IntList class.

Unfortunately the code y'all are using does this:

Code (csharp):

1.     public override Type BindToType(string assemblyName, string typeName)
2.         {
3.         if (typeName.StartsWith("System.Collections.Generic.Dictionary"))
4.             {
5.             var readType = Type.GetType(typeName);
6.             return typeof(OldSaveDataDictionary<,>).MakeGenericType(readType.GenericTypeArguments);
7.             }
8.         else
9.             {
10.             return Type.GetType(String.Format("{0}, {1}", typeName, assemblyName));
11.             }
12.         }
13.     }

Note that:

Code (csharp):

1. return typeof(OldSaveDataDictionary<,>).MakeGenericType(readType.GenericTypeArguments);

The compiler can't suss out what those GenericTypeArguments are at compile time. It's dependent on runtime.

What you'd need to do is make sure somewhere in your code a OldSaveDataDictionary<TKey,TValue> of your necessary type combinations are declared explicitly somewhere. This way the compiler knows to include them.

NOTE - this also means if you have code stripping turned on for IL2CPP it might strip them if you don't actually use that type explicitly in code despite (what "use" means is highly dependent on how Unity implements their stripping...).

...

TLDR;

The code y'all are using relies on runtime reflection of types that your AOT/IL2CPP compilation doesn't know exist at runtime and thusly aren't available at runtime.

 

The error occurred exactly where you said.
Is there any way to solve the error even if it takes a lot of time?
Even if it is not a magic code, if it can be solved through hard coding, I want to solve it.

 

You need to create a concrete version of that class somewhere in your code.

This should be as simple as having some function written somewhere that just declares variables of each type you need to ensure exists. As long as that function is compiled, then each variable's type will need to be compiled too, thusly creating a concrete instance of that generic type.

 

Thank you for answer. I tried to do what you said, but I updated the editor version after hearing that it was resolved in Unity 2022.1 or later in another thread. And problem solved.
Thank you very much.

 

Lock to a specific engine version is not the ideal fix for this. The ideal fix are not to use binary formatters. They are even depricated in newer versions of .NET because of security reasons. Use JSON or custom binary format.

Also I think it's generally a bad idea to lock to a specific version of unity. A game potentially lives and are maintained for many years and unity does not always back port critical fixes. Entire major versions of unity can stop working on specific target platforms etc, etc.

It's good practice to have a strategy in place for updating unity as a continuous effort.

We usually wait a good while into a LTS until we move because unity is notourious for unstable builds even on LTS. We are for example still on 2020 LTS but have test branches on 2021 LTS.

 

Didn't claim it was ideal. I even comment in this thread how I personally use json, and that it's better.

As for your opinions about version lock, they're opinions, we can disagree. Heck, we sort of actually agree since you even say you "wait a good while into an LTS"... which is the point of LTS... so you can version lock to it. Locking to a version doesn't require indefinite version locking. It means I can ride an LTS for a long period of time, saving the extra effort of any unfortunate refactoring that may arise moving to a newer feature version.

Or as you put it "wait a good while into an LTS".

 

Your initial post sounded like you should never move. That's bad advice in my opinion. But good we cleared that out so that anyone reading this doesn't get the false idea

 

You responded to a nearly 4 year old post, ignored the multiple other posts from me in the nearly 4 year old thread, a comment that is well to short to properly surmise the breadth of my opinion of the tangential topic to OPs question.