m_Modification doesn't clear old references when the prefab is modified

I've got a bunch of prefabs with a script on them that used to have a field that's been removed. Some of the prefab instances have modifications to that field. I thought that modifications to removed fields would automatically be removed when the prefab instance got reserialized due to something else changing, but it's not happening.

The reason this is especially bad is that the field in question is a ScriptableObject, and that SO was stored as a part of the scene. Unity cleans out such SO's when they're not referenced anymore. SInce the modification to that field stays there, the SO won't get cleared out.
In addition to all of this, we have deleted that SO's type. This means that each of these SO's are causing these warnings to appear in the console on startup:

Code (csharp):

1.  
2. The referenced script (Unknown) on this Behaviour is missing!
3. The referenced script on this Behaviour (Game Object '<null>') is missing!

In the scene file, this looks like this. I've got a prefab instance with this in it:

Code (csharp):

1.  
2. --- !u!1001 &1435652222
3. PrefabInstance:
4.   m_ObjectHideFlags: 0
5.   serializedVersion: 2
6.   m_Modification:
7.     ...
8.     - target: {fileID: 114238998614804306, guid: ff4165164015226499e68b7cb5fdc74e,
9.         type: 3}
10.       propertyPath: navLocation
11.       value:
12.       objectReference: {fileID: 218451484}

The "navLocation" field on the script has been deleted. This is one of several fields who's overrides are not getting removed, but this is the one causing the most problems.

Further down the file is this line:

Code (csharp):

1.  
2. --- !u!114 &218451484
3. MonoBehaviour:
4.   m_ObjectHideFlags: 0
5.   m_CorrespondingSourceObject: {fileID: 0}
6.   m_PrefabInstance: {fileID: 0}
7.   m_PrefabAsset: {fileID: 0}
8.   m_GameObject: {fileID: 0}
9.   m_Enabled: 1
10.   m_EditorHideFlags: 0
11.   m_Script: {fileID: 11500000, guid: 35134fd8a34d47be92f33e58371449fc, type: 3}
12.   m_Name: Krathaven 9_Location
13.   m_EditorClassIdentifier:

35134fd8a34d47be92f33e58371449fc references a deleted script.

Things I have tried to clean this up:
- Make changes to the prefab instance. In my experience, that causes the prefab instance to reserialize, Which usually fixes things like this, but no dice
- Reserialize the entire scene asset, using AssetDatabase.ForceReserializeAssets. That doesn't work either.

- Delete the prefab instance, enter play mode, exit play mode, Undo.
This half-solves the issue, as the SO is removed from the scene file. The prefab modification is still there, but now with

objectReference: {fileID: 0}

Any advice? Is this a bug? What's supposed to make prefab modifications that are not relevant anymore be cleared?

 

It's not a bug, but we're aware the design has some issues.

In general, Unity is very conservative with regards to deleting user data. For example, if you setup properties on a Material and later change the shader to a different one that doesn't have those properties, we don't delete the properties, and if you switch the shader back, your saved values are still there. This principle is used throughout lots of data in Unity.

For this Prefab case, say you remove that script property, but later revert that via version control or other means. Your old data is now back in the state you left it in. This is frequently convenient. The unused overrides data is still stripped from builds, since builds don't has any concept of overrides at all, so it doesn't cause overhead in that sense. But it does cause issues sometimes, and I think the fact that dependencies are still tracked for unused overrides is one such issue.

There are a few cases where we do remove unused overrides. For examples, if you have overrides on a component, but then remove that component (and applies it), then the overrides for that component are removed as well. But in more indirect cases, we don't remove them.

Having tooling to allow cleaning up unused overrides is one of the things that are up for consideration for future improvements we'll work on.

 

In our case, this is really bad, because the ScriptableObject type that the removed field is referencing has been removed from the project. Since Unity only removes ScriptableObjects that are a part of the scene when there's no scripts removing them, those are permanently stuck there, and are puking warning messages into the console on startup.

The warning messages, together with the bloat of thousands of unused fields and ScriptableObjects cause our project size to bloat, and the warning messages also causes entering play mode to take a lot longer, since warning messages are slow. They also make it super hard to find real warning messages that we care about.


Unity also has a long-standing issue where you cannot delete nested assets who's type is not known. If these were ScriptableObjects who's type were known, I could probably fix this by doing LoadAllAssetsAtPath for the scene file, and deleting the instances I wanted gone. But in this instance that becomes impossible.


The warning message is also really, really hard to track down, as they look like this:

Code (csharp):

1. The referenced script (Unknown) on this Behaviour is missing!

and:

Code (csharp):

1. The referenced script on this Behaviour (Game Object '<null>') is missing!

There's no context for any of these messages, and they're wrong, as the referenced script is not a Behaviour, and not attached to any GameObject. I had to spend hours binary search deleting the objects in the scene in order to even understand what's going on.


I think I can fix this, by creating a new, empty ScriptableObject, assigning it the GUID of the deleted ScriptableObject, and then search for instances of that to delete. That's really, really hacky, and requires quite a lot of in-depth understanding of how Unity works - I think a lot of users would just be stuck with not being able to use warning messages for the rest of their project.

 

Yeah it sounds frustrating.

I forgot to be more clear that I meant that the fact that overrides are not removed is not a bug, but the fact that it causes referenced objects to be kept in the scene might still be. We'd appreciate a bug report on this.

It sounds like there's also an unrelated issue with some of our warning messages that are not really related to Prefabs, but feel free to file bug reports about that too of course.

 

I've reported both; 1191638 and 1191643.

 

Hi!

I eventually got both bugs back as "by design", with mentions that a tool to clear the outdated references might be in the pipeline.

We just ran into this issue again in a different manner. We have some assets that used to be directly linked, but are now loaded from an asset bundle. So we went from:

Code (csharp):

1. public class Foo : MonoBehaviour {
2.     public Bar bar; // Bar is an asset file
3. }

To this:

Code (csharp):

1. public class Foo : MonoBehaviour {
2.     public string barID;
3.     private Bar bar;
4.  
5.     void Start() {
6.         var = GetFromAssetBundle(barID);
7.     }
8. }

The problem here is that Foo is on prefab instances, and prefab instances has old m_Modification's that reference bar. This causes the Bar assets to be included both in the asset bundle, and in the game build itself, due to there being a reference from the scene the Foo instance is in, to the Bar asset.

That has to be an actual bug, right? We're getting an asset included in a build, without having any editable references to that asset - all of our references are outdated m_Modification's that we can't really touch.

If I create a bug report showing this, any chance it'll be handled? The m_Modification hanging around forever is sketchy, but fine. That that causes files that shouldn't be included in builds to be included is pretty bad.

 

Hey, thanks. If you want to, I can sit down and create a repro project for exactly that issue and create a bug report.

 

I made a tool to cherry pick and clean up trash modifications.
Maybe this can help others too.
http://wiki.unity3d.com/index.php/PrefabDataCleaner

 

Hey @steinbitglis
This looks great. We had - I believe - the same issue (https://forum.unity.com/threads/removing-dangling-dependencies.892948/#post-5865292)

So I tried your tool . Maybe I'm missing something, but it doesn not show me anything to clean up in my scene.

In the test project attached is a "test" script which exposed a
"PrefabDependency" field, which then got removed.
The assignment however persists on the Cylinder prefab (causing a "hidden dependency"):

PrefabDependency: {fileID: 611505733756841739, guid: 9d8d10dd541e6d94799fc2a7344510b4,
type: 3}

I was hoping this is something your tool would find and clean out, or am I mistaken?

Thanks in advance!

seb

 

In the project you've posted here, there is no prefab instance override data for the field that you have removed.
If you assign some data to the field (and save)., then remove it ... then the window will pick it up and suggest removal.

 

By the way, a similar issue exists for MonoScript default data,
http://wiki.unity3d.com/index.php/MonoScriptCleaner

 

The cleaner script is awesome - but it would be good if it supported more than just prefabs in scenes!

But also... please fix this Unity!

 

Prefab instances in the project (nested prefab children) are also supported. Are there other things that need this?

 

@runevision What are the plans to fix this issue as I now have several prefab modifications in a scene that no longer need to be there? Is there an API call that can be made to remove currently invalid modifications?

 

You would have to ask Steen who last commented on it here, since I'm not aware of the status:
https://forum.unity.com/threads/m_m...n-the-prefab-is-modified.761219/#post-5397249
He's currently on vacation though.

In the mean time it looks like several people have found this workaround tool by steinbitglis useful:
https://forum.unity.com/threads/m_m...n-the-prefab-is-modified.761219/#post-5436957

 

Thanks for the reply @runevision. I took a look at the tool above, but it was easier to just manually edit the file and remove what was needed for now.

@SteenLund It would still be nice if there was a least an API call to clean a scene / prefab of invalid modifications. In our case we were left with a large orphaned array of assets still referenced from m_Modifications that I assume would still be loaded while in the editor.

 

Thanks!

I'll look into if we're using the old API or the new one - I think it's a bit of both? We're manually getting dependencies recursively in order to make some decisions, which is where I think this kicks in.

 

That's exactly what we do!

Do you know how old the bug is? For the project where we use it, it's considerably more work for us to update to 2020.2 (from 2018.4) than to work around the bug (since it's already worked around )

 

Unity seems to just have deleted the tool, along with everything on the wiki. I hope I'm wrong about this, so please correct me if I am.

Not sure if I'll be able to get it back. Hopefully I have it somewhere on an old desktop installation or something.

 

I found a snapshot of the code on the web archive. No idea if it was the latest version.
MonoScriptCleaner: https://web.archive.org/web/20200806232116/http://wiki.unity3d.com/index.php/MonoScriptCleaner
PrefabDataCleaner: https://web.archive.org/web/20210830034634/https://wiki.unity3d.com/index.php/PrefabDataCleaner

 

Thanks @SteenLund

 

Since I see this is a long going thread I add some more here:

This happens also on AnimatorControllers: swapping out an animation for another doesn't remove the dependency with the first one.

This is extremely bad when creating addressables as it creates dependency cascades that are super hard to track down :S

 

You have my permission to repurpose and use the mentioned prefab and monoscript tools in any way you wish.

 

Hey! I have the issue being very similar in this thread. It is all about wrong references getting into AssetBundles.

In my case I have a template prefab with some png assets references. Then I put this prefab into scene and rewrite all assets references in the context of the scene. Save the scene, create AssetBundle and in AssetBundle Browser window I see that all assets with the origin of the prefab are packed in the Bundle as well. The same issue if override Prefab Variant at all and just put it into the scene.
@SteenLund can you have a look please? And for me @steinbitglis tools do not work. My version of Unity - 2019.4.13f1 (the same is on 2019.4.31f1 and 2020.3.21f1)

 

@SteenLund in my case I put scene in asset bundle and in scene I have prefab with all assets references changed to the new assets. So prefab has "template_1.png" reference. But in scene it has "final_1.png" reference. Is it the expected behaviour that original prefab referenced assets get packed as well as new references into the bundle?

And the second very similar case - prefab A has reference to "template-1.png". I create prefab variant - A-variant from it and rewrite references in variant to "final_1.png". Then I put this variant in the scene. And asset bundle of the scene contains both new and original references from the original prefab.

It seems in both cases to be a bug breaking prefabs logic in the case of Assets Management. And in first case it is subjective, but in second it is really strange. If needed, can create a sample project.

 

I opened a case around the asset bundles issue a few months ago, but it was closed as an expected behavior. (Case 1347495): Asset Bundle incorrect dependencies.

I have to say this is a major framework problem. It's causing ~20x increase in my scene loading times, turning my testers away from the game. I traced it down to loads of incorrect asset bundle dependencies.

I resurfaced the case above, and will post any updates I get here. But it's worth doing a broader review of the bigger problem: what's everything that stale references might bloat? Scene size? Asset bundles? Anything else?

 

Try to use this tool. Maybe it will help you to understand your Atlas Bundle situation better https://docs.unity3d.com/Manual/AssetBundles-Browser.html

 

Thanks a lot Valerii_MA. Indeed, this tool has helped me a lot in spotting dependencies, the correct and the incorrect ones. It lets me know what objects cause depencies between bundles (the dependent objects, and the ones they depend on).

 

Ahh we ran into this today too. Our Asset Bundles are huge and it turned out its because we have some mega array thats serialized in many of our prefabs which simply doesn't exist anymore. There doesn't seem to be any way to remove them in Unity, so I've been going thru our prefabs in a text editor to remove things from the modified data block.

The tool above doesn't seem to work. Are there other solutions?

 

We've just come across this of prefab modifications not getting deleted when the field which is being modified has been removed and the scene is dirtied and re-saved.

It does surprise me that it is "by design", because Unity removes data from deleted fields in MonoBehaviours and ScriptableObjects. The argument that it is because Unity is very conservative in deleting user data seems inconsistent because of this.

Also, the given example says that using version control to revert changes which reintroduces a deleted field and, because Unity didn't cleanout the old property modifications, you still have your old data. However, this completely overlooks the fact you have just used version control to revert changes - so, you could revert the prefab changes as well to restore your modifications to the deleted field.

This is a problem for bloating assets with now unused data. In our case specifically it also killed the inspector performance because there was a lot of data stored in the prefab modifications. This, obviously, was a bad design so we changed it and stored the data elsewhere. However, the data remained which meant our fix didn't realise the inspector performance gain again until manual editing of asset files.

All that being said, it sounds like this isn't going to be changed so I have made a simple bit of code which cleans prefab instance property modifications on scene objects. This was the exact use case we needed and as such it may not do exactly what you need, but you should be able to modify if required.

Code (CSharp):

1. using UnityEditor;
2. using UnityEngine;
3.  
4. namespace utils {
5.     public class PrefabModificationCleaner {
6.         [MenuItem("Utils/Remove old prefab overrides")]
7.         private static void Fix() {
8.             var go = Selection.activeGameObject;
9.             if (go == null) {
10.                 Debug.LogError("You must select a GameObject");
11.                 return;
12.             }
13.  
14.             if (!PrefabUtility.IsPartOfPrefabInstance(go)) {
15.                 Debug.LogError("You must select a Prefab instance");
16.                 return;
17.             }
18.  
19.             var propertyModifications = PrefabUtility.GetPropertyModifications(go);
20.             var changed = false;
21.      
22.             for(var i = 0; i < propertyModifications.Length; i++) {
23.                 var propertyModification = propertyModifications[i];
24.                 var so = new SerializedObject(propertyModification.target);
25.                 var prop = so.FindProperty(propertyModification.propertyPath);
26.                 if (prop == null) {
27.                     Debug.Log($"{propertyModification.propertyPath} doesn't exist any more, removing it!");
28.                     ArrayUtility.RemoveAt(ref propertyModifications, i--);
29.                     changed = true;
30.                 }
31.             }
32.  
33.             if (changed) {
34.                 Undo.RecordObject(go, "Remove unused prefab property modifications");
35.                 PrefabUtility.SetPropertyModifications(go, propertyModifications);
36.             }
37.         }
38.     }
39. }

 

This looks great, however it doesn't help the many many people on 2021 who don't have options to upgrade.

Could Unity release code which we can import into older versions which provides the same functionality?

Cheers!

 

@Peter_Olsted or @Mads-Nyholm, is there a way to perform this sort of operation from code? It would be useful for our refactoring tools. I'm looking at PrefabUtility and everything seems to be internal.

EDIT: If you'd rather keep this internal, it could be nice to have this "Remove Unused Overrides" option on all selected prefab assets (in the Project window). Right now, is seems we can only do this in scenes.

 

This is quite a bad problem still. If you have a prefab created from a rig (as a variant) and say override materials a few times, the old material references stick and there's no way to purge them that actually works. Furthermore, AssetDatabase.GetDependencies(path, false) returns the phantom dependencies and they even get baked into asset bundles (even when using the new Scriptable Build Pipeline).

So if you create a model prefab (variant) and change materials a few times you are screwed. No way could anyone open every single model prefab variant by hand in the prefab editor and constantly remove overrides. Things like these should never ever be serialised with the data, no other system in Unity works that way, and AssetDatabase also incorrectly reports these dependencies.

What's even worse is that when a model variant is opened in the prefab editor and the "unused" overrides are removed, Unity often removes the current overrides and not the history... so in our case you end up with no visible materials, but still have the incorrect dependencies.

The entire override system is bad and it shouldn't work this way (it doesn't work in any way atm).

@SteenLund

 

I'd like to run the tool to remove unused overrides on every prefab in our project. Is there an API call for that? No way could anyone spend tens of hours pressing that button in the editor.

This has to change, when a change is made REMOVE THE PREVIOUS OVERRIDE! Isn't that obvious? No other system in Unity works this way.

 

Thanks! Do you think it's a bug that prefab (variants) created from models retain the override values when setting material/blend shapes? Changing materials a few times makes AssetDatabase.GetAssetDependencies return the old values. Also, when the Remove overrides editor is run it removes incorrect overrides, it actually clears the CURRENT material and leaves the old ones. If this was fixed this wouldn't be an issue.

Thanks again.

 

Prefab Variant m_Modifications clear old dependencies