Search Unity

Data Upgrade Script Data Upgrade

Discussion in 'Experimental Scripting Previews' started by superpig, Aug 25, 2016.

  1. Breyer

    Breyer

    Joined:
    Nov 10, 2012
    Posts:
    412
    Very interesting feature. However i dont like the fact that every time you change data layout, you have to reimplement lost fields in upgrader with exactly same type and name without optional way to restore these infos inside Upgrade call.

    Well, disciplined programmers surely will change upgrader immediately after every layout change but would be nice if there was a safeguard. Could be possible to implement something like this?

    Code (CSharp):
    1. public void Upgrade(ref ProjectileLauncher launcher, int version, SerializedProperty[] lostFields) //or oldFields if hard/impossible to filter out not lost fields
    i believe this would enable querying lost fields' name and type and even get their value before layout change.
     
  2. tsweeney

    tsweeney

    Joined:
    Oct 14, 2013
    Posts:
    4
    I heard about this at Unite and am excited by this feature and what I've been reading in the discussion.
    1. I will vote as well for a "force this process" API call which definitively updates the serialized data it can (by path or guid is fine) (as well as doing any other serialization changes). I can think of several projects where we would have benefited from having this API even without this data upgrade feature. Specifically when updating Unity versions or data fields, we'd run into the issue where files would be "changed" by re-serialization on artists/designer machines, but the change would be meaningless to that individual, so they wouldn't know exactly what to do with those changes.
    2. One problem we have is guid changes, almost always caused by VCS .meta file mangling. We've also had times where needing to convert MonoBehaviour/ScriptableObject references between a dll and loose scripts was needed. Is that a potential extension or a tool that could be added to this, or as a utility to repair that sort of damage in lieu of text file search/replace across the project? (Bonus points if there was some sort of API that could let us know if Unity had *ever* seen a file with a particular guid, and what it knew about that file)
    3. Is there any recourse here if member names are the same between versions of the data? Ex: Previously a field of type Sprite was called "image" and later it is changed so that the field is of a scriptable object type instead, but it remains named "image". If you put a field in the updater called "image" of the Sprite type, would that get the contents of the property "image" in the serialized if it was a Sprite, or would it just not be able to do anything useful in that case?
    4. Perhaps it's present somewhere in this thread, but is there an order of precedence between FormerlySerializedAs and the updater field names, when it's trying to pass along data into fields?
     
  3. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    FWIW, we're already working on the 'force' API and will be landing it first - it should solve the 'trickle' of changes across assets and metafiles in general.

    It'll be a very simple API, with an accompanying enum:

    Code (csharp):
    1.  
    2. [Flags]
    3. enum ForceReserializeAssetsOptions
    4. {
    5.     AssetsOnly= 1 << 0,
    6.     MetadataOnly= 1 << 1,
    7.     AssetsAndMetadata= AssetsOnly | MetadataOnly
    8. }
    9.  
    10. public static void AssetDatabase.ForceReserializeAssets(string[] assetPaths = null, ForceReserializeAssetsOptions options = ForceReserializeAssetsOptions.AssetsAndMetadata);
    11.  
    You give it an array of asset paths (or null for 'all assets'), and specify whether you want to operate on the assets, the .meta files, or both. So you should have total control over exactly which files you want to do at which point - or you can just call the function with the default values to reserialize the whole project right now.
     
  4. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    It's a known problem, but not something I'm going to bring into scope for this specific feature. There's a few things happening around remapping/redirecting objects, so it'll get a bit complex to stick my hands in that area right now as well.

    I think it will work, yes. The serialized data just contains the assetGUID/fileID of the target object, it doesn't contain any type information; and all we do in the upgrader code is re-run the deserializer on the same data but into your upgrader object. So when the data is read for the ScriptableObject, we discover the object is a Sprite, try and fail to cast it, and so end up putting null into the object (or a 'type mismatch' value in the Inspector) - but when we read again for the upgrader object, we can assign the Sprite object to the Sprite field on the upgrader just fine. It doesn't matter that the name was 'successfully' matched in the target object already - you can repeat any of the fields in the Upgrader if you need to.

    I've not tested, but I believe that the FormerlySerializedAs logic will be applied when matching upgrader fields to the data in exactly the same way as when matching fields on the target object.
     
  5. ZimM

    ZimM

    Joined:
    Dec 24, 2012
    Posts:
    945
    On a related note. In current versions of Unity, I use my own JSON serialization system to achieve the same goal - support for data upgrade. One of the biggest problems is saving/loading references to nested assets. In my case, Sprites that are within a Texture asset (you know, the little triangle fold out on the Texture that). There is no public API for getting the FileID, except for undocumented Unsupported.GetLocalIdentifierInFile method, which is something I'd like not to rely on. Also, there is no API allowing to get the reference to a Sprite asset even if I have both the GUID of the Texture asset and the FileID of the Sprite. This results in some horribly inefficient code like this:
    Code (CSharp):
    1. public static Object GetAssetDatabaseObject(string guid, int localIdentifier) {
    2.     string assetPath = AssetDatabase.GUIDToAssetPath(guid);
    3.     if (String.IsNullOrEmpty(assetPath))
    4.         return null;
    5.  
    6.     // This actually loads the whole Texture in the memory,
    7.     // even if I am really only interested in Sprites
    8.     Object[] objects = AssetDatabase.LoadAllAssetsAtPath(assetPath);
    9.     if (objects == null || objects.Length == 0)
    10.         return null;
    11.  
    12.     for (int i = 0; i < objects.Length; i++) {
    13.         Object obj = objects[i];
    14.         int objectLocalId = obj.GetLocalIdentifierInFile();
    15.         if (objectLocalId == localIdentifier)
    16.             return obj;
    17.     }
    18.  
    19.     return null;
    20. }
    So... While you guys are at it, perhaps you could add an official API to get the FileID, and to get a reference to an Object by its GUID and FileID combination? That would at least help developers to implement their own data upgrade system.

    Thanks.
     
  6. tsweeney

    tsweeney

    Joined:
    Oct 14, 2013
    Posts:
    4
    Have you tried abusing the recent changes to EditorJsonUtility? I think it will be able to translate between a (manually generated) string with guid and fileid into an actual object reference.
     
  7. TowerOfBricks

    TowerOfBricks

    Joined:
    Oct 20, 2007
    Posts:
    961
    Any news on this? It would be really nice for this to be supported out of the box.
     
  8. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    It's still on my list. We're doing an API for 'force upgrading' assets first (as requested both in this thread, and by a number of customers directly through other channels).
     
  9. ZimM

    ZimM

    Joined:
    Dec 24, 2012
    Posts:
    945
    Having to call EditorJsonUtility just to get an object reference is an overkill. After all, EditorJsonUtility does this via some internal API, so why not just expose the single missing piece as a public API?
     
  10. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    Hiya folks. Just wanted to let you know that we finished the ForceReserializeAssets API, and it'll be in 2017.3 (assuming nothing goes dramatically wrong).

    The new API makes Unity load and re-save the assets you request, so that any changes in data format or upgraders etc can be flushed to disk for you to commit to VCS immediately. You can control which assets are processed, and whether to process assets, .meta files, or both. It also takes care not to include any dirty changes from assets or scenes that you happen to have open at the time.

    Now that this is done, we can get back to the original data upgrader functionality; we can still have the upgraders be defined in a 'lazy' way like originally proposed here, and those of you who want to get everything upgraded so you can delete your upgraders can just call ForceReserializeAssets to apply them to your entire project immediately.
     
    ScottyB, AlkisFortuneFish and Baste like this.
  11. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    3,863
    Just gave the version from the 2017.3 beta a go.

    Our first commit on this project is from mid March, and the Assets folder has 6969 files in it, totaling at 2.27 GB. After running ForceReserializeAssets, 4199 of those files were changed. Looking over the data, you've added an externalObjects field to .meta files, probably in 2017.3, so that means that every single .meta file has been changed.

    After removing meta files from the equation, and rounding a bit, we have just under 3500 files, and about 750 of them were changed by running ForceReserializeAssets - over a fifth! The first commit was with editor version 5.5.1f1, so it's jumped quite a few versions.

    Question - when we open an old project in a new version of the editor, Unity goes through all the files to check which ones needs to be updated already. Could we perhaps get the option to also force reserialization when that happens? I'd be happy with a command line option to do that.
    Now (afaik) we have to open the project, wait for Unity to parse all files to check which one needs to be updated, and then run ForceReserializeAssets to essentially have Unity do the same thing again, just this time rewriting all files instead of just those where it's absolutely necessary. Having just one parse-all-files round would be nice.
     
  12. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    It doesn't (or shouldn't). Opening a project should never touch your assets on disk - unless you've got Editor code which is doing it, but Unity itself should never be doing that. If you're seeing otherwise, it's a bug...
     
  13. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    3,863
    I'm talking about the "Your project was last opened with a different setup of Unity (...) this may require re-import" thing. I was under the impression that it checked through files to figure out which ones to change. It does take a very long time, and it shows an asset import progress bar, but maybe I'm wrong?
     
  14. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    Ah, I see. No, we don't actually scan the Assets folder to figure out what needs updating - at least not any more than in any asset refresh (where most of the work is done by comparing timestamps).

    What we do do is check the versions of all the Importer objects - TextureImporter, ModelImporter, etc - against the version numbers that we saved into the AssetDatabase for each asset. If we've bumped the version number on an individual importer - or on any AssetPostprocessors that may affect that asset - then the check fails, and we reimport the asset. It's common in a major release that the version number on some importer changed, which is why you will usually see some reimporting happening, but if the version number didn't change (as it usually doesn't between e.g. minor releases) then we shouldn't actually be hitting the files on disk.

    The other thing we do is to scan any files that used the DefaultImporter - the fallback importer we use for unrecognised file types - to see whether we can now find a different importer type to use. This is so that when we add support for new file types, they're automatically picked up. But this still doesn't (usually) involve touching the asset file on disk - most importers determine whether a file is supported based on the file extension alone.

    What we do do, which you could say is a little unfortunate, is that when we are reimporting an asset, we unload all the objects we created during the import process, and then when you're calling ForceReserializeAssets we're reloading them again. However, I think in practice this is not going to make a very big difference, because we have to regularly unload objects during ForceReserializeAssets anyway (to prevent memory usage from getting stupidly high thanks to loading every single object in your project into memory, etc). So if the loaded objects happen to be in the first set of objects to be reserialized, it might save a little time, but if they're not then I think they'd get unloaded anyway.
     
    Baste likes this.
  15. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    3,863
    More feedback on the ForceReserializeAssets API:

    - Again, it's very good! Thank you!
    - Running the method seems to update the timeCreated field on .meta files? I ran it once, commited everything, and ran it again. Quite a few files seems to have had their timeCreated field updated to when I ran the command. Is that intentional?
    - There's something going on with rotation values. When running the reserializer twice, I get changes in some prefabs and scenes that looks like this:

    rotation.png

    Which is slightly annoying.

    All in all, when running it the second time, I got 90 changed files, most of which had updated timestamps or minor changes to floating point values.
     
  16. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    It looks like we've actually removed this field in 2018.1, but I'll see if we can improve the behaviour for a 2017.3 patch release.

    Ah, indeed. Could you possibly file a bug report with some of the (pre-force-reserialize) files that you're seeing this on? My guess is that this is not going to be a bug with ForceReserializeAssets specifically but just with stability of serialization in general, but ForceReserializeAssets makes it nice and obvious and testable.
     
  17. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    Hmm.. on further examination I can't repro this. Here's the test I'm using:

    Code (csharp):
    1.  
    2.     [Test]
    3.     public void ReserializingMetaFileDoesNotUpdateTimeCreatedField()
    4.     {
    5.         var mat = new Material(Shader.Find("Standard"));
    6.         AssetDatabase.CreateAsset(mat, "Assets/Test.mat");
    7.         AssetDatabase.SaveAssets();
    8.  
    9.         var timeCreatedBeforeReserialize = int.Parse(Regex.Match(File.ReadAllText("Assets/Test.mat.meta"), "timeCreated: (?<timeCreated>\\d+)").Groups["timeCreated"].Value);
    10.  
    11.         // Wait a couple of seconds so that the current time has changed
    12.         System.Threading.Thread.Sleep(new TimeSpan(0, 0, 2));
    13.  
    14.         AssetDatabase.ForceReserializeAssets(new[]{"Assets/Test.mat"}, ForceReserializeAssetsOptions.ReserializeMetadata);
    15.  
    16.         var timeCreatedAfterReserialize = int.Parse(Regex.Match(File.ReadAllText("Assets/Test.mat.meta"), "timeCreated: (?<timeCreated>\\d+)").Groups["timeCreated"].Value);
    17.  
    18.         Assert.AreEqual(timeCreatedBeforeReserialize, timeCreatedAfterReserialize);
    19.     }
    20.  
    Can you think of what might be different about your usage scenario? Maybe it's particular asset types or similar? Perhaps you could copy the test into your own project and poke around at it, see if you can make it fail?
     
  18. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    OK, I also tried crafting a prefab (EDIT: and a scene) with the data visible in your screenshot, and it's reserializing stably too.

    I've got tests ready to go, just help me out with assets that will make them fail :)
     
    Last edited: Jan 3, 2018
  19. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    3,863
    I'll see if I can figure out something. Note that the time created field wasn't updated on everything, but instead on... idk, about a hundred assets (out of thousands).

    I'll run the reserializer on all the files over lunch, and then try to apply it to a single file, see what happens.
     
  20. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    3,863
    Okay, I've got a repro case. How do you want me to send it? It's a smallish project, where both timeCreated and m_LocalRotation gets changed for some files when I run reserialize all a second time.
     
  21. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    Best to use the bug reporter (so that I get system info too). Could you file it and let me know the case number?
     
  22. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    3,863
    Case 984239
     
  23. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    Awesome, thank you - it's reproducing locally for me now. Digging in to find out why my tests aren't reproducing it...
     
    Baste likes this.
  24. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    Both the 'timeCreated' issue and the small variations in m_LocalRotation should be fixed in 2017.3.0p4.
     
    Xarbrough and Baste like this.
  25. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    3,863
    that m_localRotation issue must be old. Euler to Quaternion to Euler issues, or?
     
  26. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    485
    It didn't even register in my head as an issue, because we get precision differences in the serialisation practically every time we change major(-ish) Unity version, very much including m_LocalRotation. I guess I should be a little more trigger happy with bug reports!
     
  27. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    In-memory precision not precisely matching on-disk precision, combined with normalising the quaternion on load. The result was that certain quaternions would oscillate around 'stable' normalised values. I've changed it so that it skips normalisation if the quaternion is already within epsilon of being normalised.
     
  28. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    372
    Thanks for working on this feature!
     
  29. Kiupe

    Kiupe

    Joined:
    Feb 1, 2013
    Posts:
    516
    Hello,

    The link in the first page is for Unity 5.4.0f3. Where newer version can be found ?

    Thanks
     
  30. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    3,863
    Any update on the progress of this? I'm working on a fairly large open source project where I'll want to change how the data's layed out in the backend in ways that [FormerlySerializedAs] won't be enough to handle. The data upgrader interface would be great to have in this situation.

    I assume you're all really busy with jobs and ecs and stabilizing the new .NET an all that, so I'm not expecting anything in the near future. I'm just wondering if this is something that's still worked on and could be expected within a couple of major releases.
     
    BinaryCats, Can-Baycay and MMind like this.
  31. BinaryCats

    BinaryCats

    Joined:
    Feb 8, 2016
    Posts:
    164
    Well this is a bummer, I got all excited for upgrading our project to improve workflow, merging and a bunch of stuff. Now I realise the build is just for 5.4 o_O

    While I am here, I will throw my hat into the ring and give a description of how I was expecting this feature to work. I hoped unity would give us a hook that worked the same/similar way to [FormerlySerializedAs]. The attribute would provide the (previous) Serialized data and the fieldinfo of the field. That way we could manipulate the data in the way we need to fit the new fieldtype.


    What we will probably end up doing is doing it through property drawers - find the new/old type through property relatives, manipulate the data, same new type data.

    i.e. we want to change string fields to ScriptableObjects
    Code (CSharp):
    1. [hideinInspector]//added
    2.  
    3. string old
    4.  
    5. [ChangeThis]
    6. SomeData new;
    Code (CSharp):
    1. Change This property Drawerer
    2.    if objectreference == null
    3.       Then Get old field string value
    4.       make new SomeData using that value
    5.       set new value
    but I guess that may not fit everyone's use case


    I hope unity addresses this issue soon
     
  32. Xarbrough

    Xarbrough

    Joined:
    Dec 11, 2014
    Posts:
    372
    I know this thread has fallen off the edge of the world, but with every new Unity project I still wish there was a feature like this. We constantly refactor our components and there is no way around changing data and types while iterating on gameplay features.

    The simplest case of renaming a field is handled alright by the FormerlySerializedAs attribute, but often we also switch types or between a single reference to an array of references and so on.
     
    Can-Baycay likes this.
  33. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    3,863
    There's a somewhat hacky way of doing this that we use when this is necessary. It only works for ScriptableObjects, though, a MonoBehaviour solution would be much more involved (you'd have to manually iterate all prefabs and scenes). Anyway, recently posted this to a discord channel, so I'll copy it over here:

    Code (csharp):
    1.  
    2. // starting with this SO, where you want myData to be an int - ie. "0" to 0
    3.  
    4. public class TestScript : ScriptableObject {
    5.     public string myData;
    6. }
    7.  
    8. // change to this:
    9.  
    10. public class TestScript : ScriptableObject, ISerializationCallbackReceiver {
    11.     public string myData;
    12.  
    13.     public int myNewData;
    14.  
    15.     public void OnBeforeSerialize() {
    16.         if (int.TryParse(myData, out int intVal))
    17.             myNewData = intVal;
    18.     }
    19.  
    20.     public void OnAfterDeserialize() { }
    21.  
    22.     [MenuItem("Temp/Transfer data")]
    23.     public static void TransferData() {
    24.         var allInstances = AssetDatabase.FindAssets("t:TestScript").Select(AssetDatabase.GUIDToAssetPath).ToArray();
    25.         AssetDatabase.ForceReserializeAssets(allInstances);
    26.     }
    27. }
    28.  
    29. // call TransferData, save project
    30. // change to this:
    31.  
    32. public class TestScript : ScriptableObject {
    33.     [FormerlySerializedAs("myNewData")]
    34.     public int myData;
    35.  
    36.     [MenuItem("Temp/Transfer data")]
    37.     public static void TransferData() {
    38.         var allInstances = AssetDatabase.FindAssets("t:TestScript").Select(AssetDatabase.GUIDToAssetPath).ToArray();
    39.         AssetDatabase.ForceReserializeAssets(allInstances);
    40.     }
    41. }
    42.  
    43. // call TransferData, save project
    44. // change to this:
    45.  
    46. public class TestScript : ScriptableObject {
    47.     public int myData;
    48. }
     
  34. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    3,863
    @superpig, I read through this thread again and came over this post:

    I never thanked you properly for that answer, so thank you! It did bring up a question, though.

    I recently wrote an asset post processor that edits animations for some very specific fbx. files. That works great, but I'm a bit unsure how I should make those changes get picked up when team members pull changes to the post processor. Right now I write a value to assetImporter.userData to make the file get reimported, but that feels dirty?

    As you wrote in that post, and according to the documentation, you check the version of the post processor, and if that changes, "assets associated with this asset postprocessor will be reimported".

    The problem is that I don't really know what "assets associated with this asset postprocessor" means. Does it have to do with what methods the asset post processor implements? You wrote "AssetPostprocessors that may affect that asset".

    I tried to override GetVersion for my post processor and set it to 1, but that didn't cause any assets to get reimported
     
  35. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    Try setting it to 2. I'm pretty sure that 1 is the default version value used for postprocessors that don't override GetVersion.
     
    Baste likes this.
  36. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    3,863
    Tried. base.GetVersion() is 0. Setting it to any other int value (2, 3, 560909834, uint.maxValue) doesn't have any effect either.
     
  37. superpig

    superpig

    Quis aedificabit ipsos aedificatores? Unity Technologies

    Joined:
    Jan 16, 2011
    Posts:
    4,154
    OK, current theory from the DCC team is that we might not be checking all the AssetPostprocessor methods that we're supposed to be checking, in order to figure out whether your AssetPostprocessor affects models or not. Could you file a bug so they can take a look?
     
    Baste likes this.
  38. Baste

    Baste

    Joined:
    Jan 24, 2013
    Posts:
    3,863
    #1153110

    Although, solving that bug won't really make it useful. I've got an asset post processor that's supposed to target a very small set of .fbx files (files that need animation transitions to be stepped). I don't really want to reimport every single .fbx file in the project. That takes, what, 30 minutes?
    Ideally, we would be able to associate asset post processors with certain folders, or use other kinds of filters that are cheap to check, but reduce the number of assets they work on.