Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

Performance Improvements To Text Serialization In 2019.3

Discussion in '2019.3 Alpha' started by harryr, Apr 15, 2019.

  1. harryr

    harryr

    Unity Technologies

    Joined:
    Nov 14, 2017
    Posts:
    22
    Hey all!

    Alongside many other exciting changes Unity 2019.3 also includes some large improvements to our text serialization backend.

    What's Changed?
    Unity uses YAML as a text serialization format, up until 2019.3 we've been using an open source library called libYAML.
    libYAML has served us well but for speed and maintainability reasons we've been working on a custom internal library called UnityYAML.

    2019.3a1 brings about the full switch to this custom library.

    Why does this mean for you?
    In theory the switch should be completely transparent from a usability point of view.

    From our internal testing the switch over to UnityYAML has resulted in a 30-40% speed up in scene loading times and a ~13% speed up when importing prefabs using text serialization.

    You should see a speed up on the reading and writing of any text serialized documents, these are just some common examples.

    We'd be very interested in hearing the improvements you see in your project from this change.

    What should you look out for?
    We've worked hard to test and QA this new improvement, but as always things may slip through the cracks.

    Please file a bug report according to these instructions if you experience any of the following:
    • Version control noise:
    For example, a text serialized asset changing in a way where the data is the same, but simply moved around a bit due to the new writer.
    We have done thorough testing for this, but it's worth keeping an eye out.​
    • Data loss during serialization
    We'll be tracking new incoming bugs that we think could be related.

    If you are creating your own YAML files
    This has never been officially supported but If you're currently externally creating YAML files then you might need to update your tooling to make sure the format is correct.

    We also want to give you this forum thread as a place to voice any questions or ask about any related issues you've found.
     
  2. steego

    steego

    Joined:
    Jul 15, 2010
    Posts:
    902
    Is it, or are there any plans to make it available for the runtime? Would love to have YAML available as an option to JSON for serialization.
     
  3. harryr

    harryr

    Unity Technologies

    Joined:
    Nov 14, 2017
    Posts:
    22
    Currently this is a C++ library integrated into the editor, we don't have any plans to expose it via a C# api I'm afraid.
     
  4. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,892
    The scenes in my current project open so much faster in the editor now, it was immediately noticeable from the moment I opened the first scene.

    Kudos to the team who worked on the serialization improvements!
     
    Ryiah, Alverik, harryr and 5 others like this.
  5. 479813005

    479813005

    Joined:
    Mar 18, 2015
    Posts:
    39
    Can support serialize dictionary?
     
  6. Bastienre4

    Bastienre4

    Joined:
    Jul 8, 2014
    Posts:
    44
    It's a very good news! Thanks for the hard work!
     
  7. goran_okomotive

    goran_okomotive

    Joined:
    Apr 26, 2017
    Posts:
    26
    I also realized a speed-up on my powerful machine by opening a set of scenes. In fact, opening all 43 scenes of FAR: Lone Sails at once in the editor was 42% faster in Unity 2019.3 than in 2018.4. Thumbs up!
     
    recursive, Seb-1814, harryr and 2 others like this.
  8. vertxxyz

    vertxxyz

    Joined:
    Oct 29, 2014
    Posts:
    35
    Would this speed up animation importing times too?
    I know in the past when I was doing mocap work I made all projects binary because selecting any assets that contained the animations would hang the editor for ~30 seconds as they loaded
     
  9. DouglasPotesta

    DouglasPotesta

    Joined:
    Nov 6, 2014
    Posts:
    70
    When I saw the title I thought this was improvements for serialization of string fields. Lol
    Awesome work!
    Was this done purely for improvement on loading scenes and prefabs?
    Will this have effects on Unity collaborate?
    Will there be an option for legacy text serialization if we are running in to issues?
     
  10. tonycoculuzzi

    tonycoculuzzi

    Joined:
    Jun 2, 2011
    Posts:
    193
    Exciting! Though I can't help but think this would've been a great time to address the gripes most people have with the current serialization system, like the inability to to serialize dictionaries. Still, looking forward to the changes!
     
  11. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,892
    If you have "Asset Serialization Mode = Force Text" turned on in the Project Editor Settings, many asset types are saved as text format and thus it affects more than just scenes and prefabs.

    There is an option in the Project Editor Settings, it's called "Asset Pipeline (experimental)".
     
  12. harryr

    harryr

    Unity Technologies

    Joined:
    Nov 14, 2017
    Posts:
    22
    This change was focused on optimizing the current system and as such doesn't allow C# dictionaries to be serialized.

    If the animation files are stored as YAML .assets then you should definitely see a speed up

    This change should improve the performance of all systems that load and save text serialized data, scenes and prefabs are just very common use cases.

    The only changes for collaborate would be improvements to importing times when pulling down new assets.

    Currently there's no option to revert back to the old system, as of 2019.3 UnityYAML is the default when using text serialization.

    Yeah I can see that, as mentioned above this project was mainly focused on optimizations.
    I'd personally love to see serializable dictionaries but it's out of the scope of this project.
     
  13. vertxxyz

    vertxxyz

    Joined:
    Oct 29, 2014
    Posts:
    35
    Looks like they are, that's awesome news then - text serialisation is so much nicer for source control :)
    I haven't got a project to this version yet, but you've clearly done fantastic work here!
    (Though I still imagine loading text-serialized mocap data will still be slow as the anim files are so damn verbose)
     
  14. Jes28

    Jes28

    Joined:
    Sep 3, 2012
    Posts:
    359
    Few bugs I have noticed when working with version control on old system.

    - Sometimes serialized references was serialized in one line, sometimes in two making difference for source control:

    m_SkyboxMaterial: {fileID: 10304, guid: 0000000000000000f000000000000000, type: 0}

    m_SkyboxMaterial: {fileID: 10304, guid: 0000000000000000f000000000000000,
    type: 0}
    suggest just always serialize it into exactly one line.

    Order of serialized objects sometimes was changed.
    Suggesting just always sort it by uid so every chunk of data will always be in same place no matter what happens.


    Hope that this issues will be adressed with new serializer :)
     
    dadude123 likes this.
  15. harryr

    harryr

    Unity Technologies

    Joined:
    Nov 14, 2017
    Posts:
    22
    This happens when the line passes over roughly 80 characters and is behavior we match with the old writer.
    If we were to update the logic to only write one line we'd actually end up generating a bunch of version control noise fixing up existing two line references.
     
  16. Jes28

    Jes28

    Joined:
    Sep 3, 2012
    Posts:
    359
    Can you please add Project option to make it serialize in 1 line and may be make it default for new projects.

    Line length often changes from commit to commit and actually create noise in source control all the time :)
     
  17. harryr

    harryr

    Unity Technologies

    Joined:
    Nov 14, 2017
    Posts:
    22
    We can definitely look into this, great idea! :)
     
    Bedtime, SugoiDev, Deozaan and 3 others like this.
  18. DavidNLN

    DavidNLN

    Joined:
    Sep 27, 2018
    Posts:
    53
    Very interesting, we have moved to flatbuffers for most of our data as unity wasn't able to load scenes fast enough, maybe we can now move some of the smaller stuff back for easier usage ^^
     
    GameDevCouple_I likes this.
  19. EvOne

    EvOne

    Joined:
    Jan 29, 2016
    Posts:
    57
    Wow! 0_o -What a fast this Unity 2019.3.a7 !!! :):):)
    -With beta 2019.2 - just can be no comparison!
    This is just amazing! :) I installed beta 2019.2 to be able to work with 2D animation and other preview packs and after working with it a little bit, I realized that most likely my notebook will simply not be able to work normally with any new versions of Unity... :(

    But with 2019.3.a7 - all again "Flies"! :))) It was completely unexpected ... :)

    -Big Big BIG THANKS to the developers! :D

    P.S. Sorry for Google-translate... :rolleyes:
     
    GameDevCouple_I likes this.
  20. jashan

    jashan

    Joined:
    Mar 9, 2007
    Posts:
    3,085
    Hi @harryr , this sounds like a really awesome improvement! I'm not sure this is the right place / scope, but it might be:

    One thing that is bothering me a lot is that during Unity updates, on the actual update, only a few changes are applied in the actual project files (i.e. everything under Assets). However, there are a lot of changes that occur in the YAML files (I'm assuming it's the YAML files, the two types of assets I see this most with are scenes and prefabs), and those changes seem to be then written into the files whenever the file is opened and a change is made, causing a lot of noise unrelated to the actual changes that we as developers do in those files.

    While I can total see how this will usually improve the update experience by not having to iterate over all the files on every update, it really does mess up version control because after each Unity update (and that seems to also include minor releases), the changes end up in version control distributed over several days, whenever any file that has such changes is touched.

    I believe this is a case where both approaches are needed because it's like that some people will prefer the current approach, with quicker project updates. But other people, like me, would prefer to have every change caused by a Unity update to be in source control in my "Updated Unity to 2019.x.y" commit, separate from my actual development changes.

    Such an option would most likely be best put into Preferences / General, next to "Compress Assets on Import" (because it's a very similar decision to be made), and could be called something like "Immediately Update Asset Layout on Unity Update" or maybe "Update all YAML-Files on Unity Update".

    You might be doing this during the 2019.3 update already ... or convert to Asset Database v2 "on-the-fly" which would also make finding serialization bugs much harder during the alpha. But from my experience, finding actual serialization bugs would be pretty difficult due to the many "Unity version update" changes we have when going from one version to the next (e.g. 2019.2 to 2019.3).
     
  21. Peter77

    Peter77

    Joined:
    Jun 12, 2013
    Posts:
    3,892
    When I update Unity, I call
    AssetDatabase.ForceReserializeAssets
    after the update to make sure Unity writes all assets in the new format to disk, so I don't run into this problem. Then submit/push all changed files to version control.

    https://docs.unity3d.com/ScriptReference/AssetDatabase.ForceReserializeAssets.html
     
    Sylmerria, Jes28, elcionap and 3 others like this.
  22. jashan

    jashan

    Joined:
    Mar 9, 2007
    Posts:
    3,085
    Ah, very cool, that'll solve this for me. Thank you!
     
    Peter77 and harryr like this.
  23. harryr

    harryr

    Unity Technologies

    Joined:
    Nov 14, 2017
    Posts:
    22
    Hi @jashan, thanks for the feedback!
    @Peter77 beat me to it! The ForceReserializeAssets api was added for the exact reason :)
     
    Peter77 and jashan like this.
  24. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    552
    Oooo, how did I miss that! We have long had a "Set Dirty" context menu, when updating I select all assets, wait ages, right click, wait ages, Set Dirty, then save, just to achieve that.