What does Undo.RecordObject record, exactly?

Recently I've come across a strange error whenever I perform an Undo operation on my custom data structure:

Assertion failed on expression: 'newOffset >= 0 && newOffset < (int)data.size()'

I have written my own serializable dictionary classes. They worked fine before, but ever since I changed the field (of one of these) to

[SerializeReference]

, I'm getting this error whenever I Undo a modification to it.

In addition, this error won't occur if I use

Undo.RegisterCompleteObjectUndo

instead of

Undo.RecordObject

.

This prompted me to probe at exactly what's happening during an Undo operation. From the docs, we know that

Undo.RecordObject

will wait until the end of the frame to calculate a "diff", which will be applied when Undo is performed.

What is this diff, and how is this calculated/applied? From observing

ISerializationCallbackReceiver

, it looks like during an undo operation, all objects that are affected are serialized, the diff is applied, then all objects are deserialized. The error above happens when the diffs are applied.

So the question is: it looks like Undo.RecordObject records diff on a SerializedObject. But how is it calculated, exactly? How can we make sure a custom class won't break this diff calculation, so we can use

Undo.RecordObject

instead of

Undo.RegisterCompleteObjectUndo

?

 

Are there any prefabs involved here?

There are some bugs where Undo.RecordObject and EditorUtility.SetDirty doesn't properly handle recording changes to prefab instances. Here's a snippet from our codebase:

Code (csharp):

1.  
2. // Some prefab overrides are not handled by Undo.RecordObject or EditorUtility.SetDirty.
3. // eg. adding an item to an array/list on a prefab instance updates that the instance
4. // has a different array count than the prefab, but not any data about the added thing
5. private static void HandlePrefabInstance(GameObject gameObject) {
6.     if (PrefabUtility.IsPartOfAnyPrefab(gameObject)) {
7.         PrefabUtility.RecordPrefabInstancePropertyModifications(gameObject);
8.     }
9. }
10.  

In general, my experience is that Unity straight up refuses to maintain Undo.RecordObject at all. We do all of work with serialized data through SerializedObject/SerializedProperty, since the alternative is an endless stream of S*** not working. So I'd suggest rewriting your serialized dictionary to use that instead.

 

Unfortunately none of it is in prefabs. Writing it entirely in SerializedProperty is probably a good idea. I'm not sure if it's possible (without a lot of work) though, since it's supposedly a generic dictionary.

I remember reading some of your other posts while researching about writing a generic dictionary though - what did you end up doing, if you don't mind me asking?

 

There's two parts to the SerializedDictionary - the editor and the ISerializationCallbackReceiver implementation. The ISerializationCallbackReceiver code doesn't need any dirtying, so that's probably not what you're asking about.


For the drawer, we use SerializedProperty. The Dictionary is simply:

Code (csharp):

1.  
2. public abstract class SerializableDictionary<TKey, TValue> : ISerializationCallbackReceiver, IDictionary<TKey, TValue> {
3.  
4.     #region Serialization
5.  
6.     [SerializeField]
7.     private TKey[] keys;
8.     [SerializeField]
9.     private TValue[] values;
10.  
11.  

And the drawer iterates the SerializedProperty for keys and values and draws the elements. There's a bunch of complexity in our code, but I think that's all optimization and dealing with keys and values that have complex drawers, so I think you can get away with this for most cases:

Code (csharp):

1. var keysProp = dictionaryProp.FindPropertyRelative("keys");
2. var valuesProp = dictionaryProp.FindPropertyRelative("keys");
3.  
4. for (int i = 0; i < keysProp.arraySize; i++) {
5.     EditorGUI.PropertyField(rect, keysProp.GetArrayElementAt(i);
6.     // move rect to second column
7.     EditorGUI.PropertyField(rect, valuesProp.GetArrayElementAt(i);
8.     // move rect back to first column and down
9. }

 

Actually, I'm writing an editor tool so I'm manually recording the ScriptableObject that contains the dictionary throwing the error dirty. My serialized dictionary is mostly the same as yours, except that it uses a list of another serializable custom class KeyValuePair (to prevent any possibility of a mismatched number of keys and values).

I'm having trouble identifying the exact cause of this bug, all I know that it's a combination of my custom dictionary serialization,

SerializeReferenceAttribute

, and

Undo.RecordObject

. Would be really nice if there's more doc on what Undo does under the hood :/