Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice
  2. Ever participated in one our Game Jams? Want pointers on your project? Our Evangelists will be available on Friday to give feedback. Come share your games with us!
    Dismiss Notice

Performance Improvements To Text Serialization In 2019.3

Discussion in '2019.3 Beta' started by harryr, Apr 15, 2019.

  1. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    608
    @superpig it's toggled on and off with an identical name when repeatedly calling "AssetDatabase.ForceReserializeAssets" (and commiting the changes inbetween calls). The version number did only change on the first call.
     
  2. AlkisFortuneFish

    AlkisFortuneFish

    Joined:
    Apr 26, 2013
    Posts:
    788
    It does work, it's just *hugely* finicky to configure correctly and rather neglected in general. So, agreeing with the general sentiment but we have had it working in our studio.
     
    Ramobo likes this.
  3. Lorash

    Lorash

    Joined:
    May 6, 2019
    Posts:
    215
    Nothing. I was being sarcastic. :)
     
    Ramobo likes this.
  4. Lorash

    Lorash

    Joined:
    May 6, 2019
    Posts:
    215
    These conventions apply to your public stuff, no underscores simply means to not do
    public int Something_Like_This;
    . There is deliberately no convention for private members.
     
  5. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    608
    @harryr @superpig why I was still not able to reproduce this in an fresh project (seems the reordering happens with certain complexity, haven't figured out yet which that is), to give you an overview of the massive amount of that going on:

    Repeatedly forcing reserialization a project with ~5000 files results in 202 of those having reorderings, all of them either .unity or .prefab. Some other files have options toggled.

    m_UseShadowmask toggling happens as can be seen here (that's all changes for this file). No m_ShadowMaskMode and no m_UseShadowMask here, that file underwent reserialization on that exact Unity version already multiple times.
    upload_2019-12-4_16-23-36.png

    Reorderings all over the place:
    upload_2019-12-4_16-25-20.png

    Another peculiar case: .psd.meta files have the pSDShowRemoveMatteOption toggled on reserialization (again, those files underwent reserialization earlier on the same Unity version)
    upload_2019-12-4_16-27-12.png
     
    phobos2077 and Peter77 like this.
  6. OndrejP

    OndrejP

    Joined:
    Jul 19, 2017
    Posts:
    91
    Jes28 likes this.
  7. OndrejP

    OndrejP

    Joined:
    Jul 19, 2017
    Posts:
    91
    Looking forward to single-line serialization, it's definitely step into right direction, it makes VCS merging easier.
     
  8. Jes28

    Jes28

    Joined:
    Sep 3, 2012
    Posts:
    585
    Does anybody know when one line serialization will arrive?
     
  9. harryr

    harryr

    Unity Technologies

    Joined:
    Nov 14, 2017
    Posts:
    37
    The new one line serialization setting will be in 2020.1, the default is on for all new projects and off for existing ones.

    Screenshot 2020-01-02 at 11.12.16.png
     
  10. Jes28

    Jes28

    Joined:
    Sep 3, 2012
    Posts:
    585
    Nice to hear :)

    Will it be backported to Unity 2019?
     
    OndrejP, BTStone and Peter77 like this.
  11. harryr

    harryr

    Unity Technologies

    Joined:
    Nov 14, 2017
    Posts:
    37
    Apologies, no plans to backport as this is a new feature and 2019.3 is now in a stabilizing phase before release :)
     
  12. Deozaan

    Deozaan

    Joined:
    Oct 27, 2010
    Posts:
    666
    So it will be at least March 2021 before people sticking to LTS releases will see it. :(
     
    Lorash and sand_lantern like this.
  13. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    608
    One more piece of information about the random file reordering issues that are causing us tons of woe & git conflicts were non should be: not only Scenes and Prefabs exhibit that, but Timelines as well.

    @harryr
     
  14. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    608
    @harryr Would you be interested in a larger project (~50 GB) that exhibits most of the serialization reordering etc. issues I outlined here?
     
    Peter77 likes this.
  15. harryr

    harryr

    Unity Technologies

    Joined:
    Nov 14, 2017
    Posts:
    37
    Apologies for the delay, larger projects are always welcome for helping us test at scale!
    You mentioned not being able to repro on new projects earlier in the thread, is this ~50GB project the smallest one you've been able to repro on? If not, would I be able to get the smaller project as a starting point?

    Could you also confirm again which versions you're seeing this on?
     
  16. harryr

    harryr

    Unity Technologies

    Joined:
    Nov 14, 2017
    Posts:
    37
    Unfortunately that's currently the case, we often have to balance putting new features in vs stabilizing the product and feature cutoff had already passed for 2019.3 when this feature landed in 2020.1.
     
  17. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    608
    This is running reserialization on a project that was updated from 2019.3.0b11 (where it was fully reserialized) to 2019.3.0f6. First run shows hundreds of files with changed ordering, subsequent runs show dozens. Subsequent runs sometimes produce changes in different files on each run, it's not the same files always changing. Feels very non-deterministic. I would expect that running it a couple times would "solidify" at some point when all files have been reserialized to a new version, but that's not the case.
    So far I have identified the following things happening in that project:
    1. m_UseShadowMask toggles between 0 and 1 every reserialization
    2. some transform orders (m_RootOrder) are swapped in Prefabs on every reserialization
    3. fileID reordering in prefabs on reserialization, but not every time, settles after multiple attempts
    4. on first run, thousands of reorderings in scenes, prefabs
    How can I get you that larger project?

    ...

    For a smaller one - seems my GlobalGameJam project from this weekend exhibits some weird things but most notably reordering in a scene on every reserialization.
    You can find the source here https://github.com/soraryu/ggj-2020, commits look like this: upload_2020-2-3_14-44-7.png
    If you go back to "added reserialization script" and then run "Menu/pfc/ForceAssetdatabase Upgrade" you should be able to see what happens. Pass #1 (expectedly) has a lot of changes, with some of them being very weird (e.g. "HQ-Sphere.FBX.meta" changing globalScale). Pass #2 has a weird change of line endings in MainScene.unity. All passes have a reordering change in QuestHands.unity.
    Additionally, I get the other issue of QualitySettings "antiAliasing" changing from 4 (which I wanted to have) to 0 (no idea why). Interestingly, this change happens on Unity exit.
     
  18. harryr

    harryr

    Unity Technologies

    Joined:
    Nov 14, 2017
    Posts:
    37
    Can you set up a bug for these issues? Once that's complete I should be able to get you instructions on how to send the full project :)

    That's great, thanks!
     
  19. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    608
    Here's a stub bug report: (Case 1217235) Asset Serialization Reordering
     
    harryr likes this.
  20. Lorash

    Lorash

    Joined:
    May 6, 2019
    Posts:
    215
    It's great to see that in spirit of this, shader graphs are serialized in a format that's second only to JSONx: pretty-printed(!!!) JSONs stuffed into strings that are then serialized in a second layer of JSON ending up with nice virtually-unmergeable lines like
    Code (JavaScript):
    1. "JSONnodeData": "{\n    \"m_GuidSerialized\": \"58edd4b4-d64c-4e24-93df-e185782a1ad8\",\n    \"m_GroupGuidSerialized\": \"00000000-0000-0000-0000-000000000000\",\n    \"m_Name\": \"Lit Master\",\n    \"m_NodeVersion\": 0,\n    \"m_DrawState\": {\n        \"m_Expanded\": true,\n        \"m_Position\": {\n            \"serializedVersion\": \"2\",\n            \"x\": 123.0,\n            \"y\": -159.0,\n            \"width\": 200.0,\n            \"height\": 341.0\n        }\n    },\n    \"m_SerializableSlots\": [\n        {\n            \"typeInfo\": {\n                \"fullName\": \"UnityEditor.ShaderGraph.PositionMaterialSlot\"\n            },\n            \"JSONnodeData\": \"{\\n    \\\"m_Id\\\": 0,\\n    \\\"m_DisplayName\\\": \\\"Vertex Position\\\",\\n    \\\"m_SlotType\\\": 0,\\n    \\\"m_Priority\\\": 2147483647,\\n    \\\"m_Hidden\\\": false,\\n    \\\"m_ShaderOutputName\\\": \\\"Vertex Position\\\",\\n    \\\"m_StageCapability\\\": 1,\\n    \\\"m_Value\\\": {\\n        \\\"x\\\": 0.0,\\n        \\\"y\\\": 0.0,\\n        \\\"z\\\": 0.0\\n    },\\n    \\\"m_DefaultValue\\\": {\\n        \\\"x\\\": 0.0,\\n        \\\"y\\\": 0.0,\\n        \\\"z\\\": 0.0\\n    },\\n    \\\"m_Labels\\\": [\\n        \\\"X\\\",\\n        \\\"Y\\\",\\n        \\\"Z\\\"\\n    ],\\n    \\\"m_Space\\\": 0\\n}\"\n        },\n        {\n            \"typeInfo\": {\n                \"fullName\": \"UnityEditor.ShaderGraph.NormalMaterialSlot\"\n            },\n            \"JSONnodeData\": \"{\\n    \\\"m_Id\\\": 33,\\n    \\\"m_DisplayName\\\": \\\"Vertex Normal\\\",\\n    \\\"m_SlotType\\\": 0,\\n    \\\"m_Priority\\\": 2147483647,\\n    \\\"m_Hidden\\\": false,\\n    \\\"m_ShaderOutputName\\\": \\\"Vertex Normal\\\",\\n    \\\"m_StageCapability\\\": 1,\\n    \\\"m_Value\\\": {\\n        \\\"x\\\": 0.0,\\n        \\\"y\\\": 0.0,\\n        \\\"z\\\": 0.0\\n    },\\n    \\\"m_DefaultValue\\\": {\\n        \\\"x\\\": 0.0,\\n        \\\"y\\\": 0.0,\\n        \\\"z\\\": 0.0\\n    },\\n    \\\"m_Labels\\\": [\\n        \\\"X\\\",\\n        \\\"Y\\\",\\n        \\\"Z\\\"\\n    ],\\n    \\\"m_Space\\\": 0\\n}\"\n        },\n        {\n            \"typeInfo\": {\n                \"fullName\": \"UnityEditor.ShaderGraph.TangentMaterialSlot\"\n            },\n            \"JSONnodeData\": \"{\\n    \\\"m_Id\\\": 34,\\n    \\\"m_DisplayName\\\": \\\"Vertex Tangent\\\",\\n    \\\"m_SlotType\\\": 0,\\n    \\\"m_Priority\\\": 2147483647,\\n    \\\"m_Hidden\\\": false,\\n    \\\"m_ShaderOutputName\\\": \\\"Vertex Tangent\\\",\\n    \\\"m_StageCapability\\\": 1,\\n    \\\"m_Value\\\": {\\n        \\\"x\\\": 0.0,\\n        \\\"y\\\": 0.0,\\n        \\\"z\\\": 0.0\\n    },\\n    \\\"m_DefaultValue\\\": {\\n        \\\"x\\\": 0.0,\\n        \\\"y\\\": 0.0,\\n        \\\"z\\\": 0.0\\n    },\\n    \\\"m_Labels\\\": [\\n        \\\"X\\\",\\n        \\\"Y\\\",\\n        \\\"Z\\\"\\n    ],\\n    \\\"m_Space\\\": 0\\n}\"\n        },\n        {\n            \"typeInfo\": {\n                \"fullName\": \"UnityEditor.ShaderGraph.ColorRGBMaterialSlot\"\n            },\n            \"JSONnodeData\": \"{\\n    \\\"m_Id\\\": 1,\\n    \\\"m_DisplayName\\\": \\\"BaseColor\\\",\\n    \\\"m_SlotType\\\": 0,\\n    \\\"m_Priority\\\": 2147483647,\\n    \\\"m_Hidden\\\": false,\\n    \\\"m_ShaderOutputName\\\": \\\"Albedo\\\",\\n    \\\"m_StageCapability\\\": 2,\\n    \\\"m_Value\\\": {\\n        \\\"x\\\": 0.7353569269180298,\\n        \\\"y\\\": 0.7353569269180298,\\n        \\\"z\\\": 0.7353569269180298\\n    },\\n    \\\"m_DefaultValue\\\": {\\n        \\\"x\\\": 0.0,\\n        \\\"y\\\": 0.0,\\n        \\\"z\\\": 0.0\\n    },\\n    \\\"m_Labels\\\": [\\n        \\\"X\\\",\\n        \\\"Y\\\",\\n        \\\"Z\\\"\\n    ],\\n    \\\"m_ColorMode\\\": 0\\n}\"\n        },\n        {\n            \"typeInfo\": {\n                \"fullName\": \"UnityEditor.ShaderGraph.NormalMaterialSlot\"\n            },\n            \"JSONnodeData\": \"{\\n    \\\"m_Id\\\": 2,\\n    \\\"m_DisplayName\\\": \\\"Normal\\\",\\n    \\\"m_SlotType\\\": 0,\\n    \\\"m_Priority\\\": 2147483647,\\n    \\\"m_Hidden\\\": false,\\n    \\\"m_ShaderOutputName\\\": \\\"Normal\\\",\\n    \\\"m_StageCapability\\\": 2,\\n    \\\"m_Value\\\": {\\n        \\\"x\\\": 0.0,\\n        \\\"y\\\": 0.0,\\n        \\\"z\\\": 0.0\\n    },\\n    \\\"m_DefaultValue\\\": {\\n        \\\"x\\\": 0.0,\\n        \\\"y\\\": 0.0,\\n        \\\"z\\\": 0.0\\n    },\\n    \\\"m_Labels\\\": [\\n        \\\"X\\\",\\n        \\\"Y\\\",\\n        \\\"Z\\\"\\n    ],\\n    \\\"m_Space\\\": 3\\n}\"\n        },\n        {\n            \"typeInfo\": {\n                \"fullName\": \"UnityEditor.ShaderGraph.NormalMaterialSlot\"\n            },\n            \"JSONnodeData\": \"{\\n    \\\"m_Id\\\": 3,\\n    \\\"m_DisplayName\\\": \\\"BentNormal\\\",\\n    \\\"m_SlotType\\\": 0,\\n    \\\"m_Priority\\\": 2147483647,\\n    \\\"m_Hidden\\\": false,\\n    \\\"m_ShaderOutputName\\\": \\\"BentNormal\\\",\\n    \\\"m_StageCapability\\\": 2,\\n    \\\"m_Value\\\": {\\n        \\\"x\\\": 0.0,\\n        \\\"y\\\": 0.0,\\n        \\\"z\\\": 0.0\\n    },\\n    \\\"m_DefaultValue\\\": {\\n        \\\"x\\\": 0.0,\\n        \\\"y\\\": 0.0,\\n        \\\"z\\\": 0.0\\n    },\\n    \\\"m_Labels\\\": [\\n        \\\"X\\\",\\n        \\\"Y\\\",\\n        \\\"Z\\\"\\n    ],\\n    \\\"m_Space\\\": 3\\n}\"\n        },\n        {\n            \"typeInfo\": {\n                \"fullName\": \"UnityEditor.ShaderGraph.Vector1MaterialSlot\"\n            },\n            \"JSONnodeData\": \"{\\n    \\\"m_Id\\\": 11,\\n    \\\"m_DisplayName\\\": \\\"CoatMask\\\",\\n    \\\"m_SlotType\\\": 0,\\n    \\\"m_Priority\\\": 2147483647,\\n    \\\"m_Hidden\\\": false,\\n    \\\"m_ShaderOutputName\\\": \\\"CoatMask\\\",\\n    \\\"m_StageCapability\\\": 2,\\n    \\\"m_Value\\\": 0.0,\\n    \\\"m_DefaultValue\\\": 0.0,\\n    \\\"m_Labels\\\": [\\n        \\\"X\\\"\\n    ]\\n}\"\n        },\n        {\n            \"typeInfo\": {\n                \"fullName\": \"UnityEditor.ShaderGraph.Vector1MaterialSlot\"\n            },\n            \"JSONnodeData\": \"{\\n    \\\"m_Id\\\": 12,\\n    \\\"m_DisplayName\\\": \\\"Metallic\\\",\\n    \\\"m_SlotType\\\": 0,\\n    \\\"m_Priority\\\": 2147483647,\\n    \\\"m_Hidden\\\": false,\\n    \\\"m_ShaderOutputName\\\": \\\"Metallic\\\",\\n    \\\"m_StageCapability\\\": 2,\\n    \\\"m_Value\\\": 0.0,\\n    \\\"m_DefaultValue\\\": 0.0,\\n    \\\"m_Labels\\\": [\\n        \\\"X\\\"\\n    ]\\n}\"\n        },\n        {\n            \"typeInfo\": {\n                \"fullName\": \"UnityEditor.ShaderGraph.Vector1MaterialSlot\"\n            },\n            \"JSONnodeData\": \"{\\n    \\\"m_Id\\\": 14,\\n    \\\"m_DisplayName\\\": \\\"Smoothness\\\",\\n    \\\"m_SlotType\\\": 0,\\n    \\\"m_Priority\\\": 2147483647,\\n    \\\"m_Hidden\\\": false,\\n    \\\"m_ShaderOutputName\\\": \\\"Smoothness\\\",\\n    \\\"m_StageCapability\\\": 2,\\n    \\\"m_Value\\\": 0.5,\\n    \\\"m_DefaultValue\\\": 0.5,\\n    \\\"m_Labels\\\": [\\n        \\\"X\\\"\\n    ]\\n}\"\n        },\n        {\n            \"typeInfo\": {\n                \"fullName\": \"UnityEditor.ShaderGraph.Vector1MaterialSlot\"\n            },\n            \"JSONnodeData\": \"{\\n    \\\"m_Id\\\": 15,\\n    \\\"m_DisplayName\\\": \\\"AmbientOcclusion\\\",\\n    \\\"m_SlotType\\\": 0,\\n    \\\"m_Priority\\\": 2147483647,\\n    \\\"m_Hidden\\\": false,\\n    \\\"m_ShaderOutputName\\\": \\\"Occlusion\\\",\\n    \\\"m_StageCapability\\\": 2,\\n    \\\"m_Value\\\": 1.0,\\n    \\\"m_DefaultValue\\\": 1.0,\\n    \\\"m_Labels\\\": [\\n        \\\"X\\\"\\n    ]\\n}\"\n        },\n        {\n            \"typeInfo\": {\n                \"fullName\": \"UnityEditor.ShaderGraph.ColorRGBMaterialSlot\"\n            },\n            \"JSONnodeData\": \"{\\n    \\\"m_Id\\\": 13,\\n    \\\"m_DisplayName\\\": \\\"Emission\\\",\\n    \\\"m_SlotType\\\": 0,\\n    \\\"m_Priority\\\": 2147483647,\\n    \\\"m_Hidden\\\": false,\\n    \\\"m_ShaderOutputName\\\": \\\"Emission\\\",\\n    \\\"m_StageCapability\\\": 2,\\n    \\\"m_Value\\\": {\\n        \\\"x\\\": 0.0,\\n        \\\"y\\\": 0.0,\\n        \\\"z\\\": 0.0\\n    },\\n    \\\"m_DefaultValue\\\": {\\n        \\\"x\\\": 0.0,\\n        \\\"y\\\": 0.0,\\n        \\\"z\\\": 0.0\\n    },\\n    \\\"m_Labels\\\": [\\n        \\\"X\\\",\\n        \\\"Y\\\",\\n        \\\"Z\\\"\\n    ],\\n    \\\"m_ColorMode\\\": 1\\n}\"\n        },\n        {\n            \"typeInfo\": {\n                \"fullName\": \"UnityEditor.ShaderGraph.Vector1MaterialSlot\"\n            },\n            \"JSONnodeData\": \"{\\n    \\\"m_Id\\\": 16,\\n    \\\"m_DisplayName\\\": \\\"Alpha\\\",\\n    \\\"m_SlotType\\\": 0,\\n    \\\"m_Priority\\\": 2147483647,\\n    \\\"m_Hidden\\\": false,\\n    \\\"m_ShaderOutputName\\\": \\\"Alpha\\\",\\n    \\\"m_StageCapability\\\": 2,\\n    \\\"m_Value\\\": 1.0,\\n    \\\"m_DefaultValue\\\": 1.0,\\n    \\\"m_Labels\\\": [\\n        \\\"X\\\"\\n    ]\\n}\"\n        }\n    ],\n    \"m_Precision\": 0,\n    \"m_PreviewExpanded\": true,\n    \"m_CustomColors\": {\n        \"m_SerializableColors\": []\n    },\n    \"m_SerializableSubShaders\": [\n        {\n            \"typeInfo\": {\n                \"fullName\": \"UnityEditor.Rendering.HighDefinition.HDLitSubShader\"\n            },\n            \"JSONnodeData\": \"{}\"\n        }\n    ],\n    \"m_SurfaceType\": 0,\n    \"m_AlphaMode\": 0,\n    \"m_RenderingPass\": 1,\n    \"m_BlendPreserveSpecular\": true,\n    \"m_TransparencyFog\": true,\n    \"m_DrawBeforeRefraction\": false,\n    \"m_RefractionModel\": 0,\n    \"m_Distortion\": false,\n    \"m_DistortionMode\": 0,\n    \"m_DistortionDepthTest\": true,\n    \"m_AlphaTest\": false,\n    \"m_AlphaTestDepthPrepass\": false,\n    \"m_AlphaTestDepthPostpass\": false,\n    \"m_TransparentWritesMotionVec\": false,\n    \"m_AlphaTestShadow\": false,\n    \"m_BackThenFrontRendering\": false,\n    \"m_SortPriority\": 0,\n    \"m_DoubleSidedMode\": 0,\n    \"m_MaterialType\": 0,\n    \"m_SSSTransmission\": true,\n    \"m_ReceiveDecals\": true,\n    \"m_ReceivesSSR\": true,\n    \"m_AddPrecomputedVelocity\": false,\n    \"m_EnergyConservingSpecular\": true,\n    \"m_SpecularAA\": false,\n    \"m_SpecularAAScreenSpaceVariance\": 0.0,\n    \"m_SpecularAAThreshold\": 0.0,\n    \"m_SpecularOcclusionMode\": 0,\n    \"m_DiffusionProfile\": 0,\n    \"m_overrideBakedGI\": false,\n    \"m_depthOffset\": false,\n    \"m_DOTSInstancing\": false,\n    \"m_ZWrite\": false,\n    \"m_transparentCullMode\": 2,\n    \"m_ZTest\": 4,\n    \"m_SupportLodCrossFade\": false,\n    \"m_MaterialNeedsUpdateHash\": 11\n}"
    I couldn't even make this up if I tried.
     
    Last edited: Feb 12, 2020
  21. Lorash

    Lorash

    Joined:
    May 6, 2019
    Posts:
    215
    I always get version control noise like this one but I accepted it as a fact of life when working with Unity (I did absolutely nothing to the object in question, these rounding errors just... happen):

    upload_2020-2-12_11-26-17.png

    However at least these are limited to individual lines that I can easily not commit, unlike that monstrosity in my previous message.
     
  22. FritzsHero

    FritzsHero

    Joined:
    Mar 22, 2018
    Posts:
    7
    Here is a problem with the HDRP and YAML:
    Case 1223630
     
  23. fherbst

    fherbst

    Joined:
    Jun 24, 2012
    Posts:
    608
    @harryr found another case; this is just with an Animator with RigBuilder / AnimationRigging package.
    Serialization order gets changed on every scene save.

    upload_2020-4-3_14-24-43.png
     
  24. Lorash

    Lorash

    Joined:
    May 6, 2019
    Posts:
    215
    The VFX graph has a lot of version control noise for no real changes whatsoever - case 1233799 with an example file
     
unityunity