TextMesh Pro TMPro Dynamic Font Asset constantly changes in source control

Hi everyone,

The problem is that font's atlas texture is a sub-asset of the font asset itself. This makes sense for static fonts.
But with dynamic fonts I don't want to store this atlas on the disk, why would I?

The whole point of dynamic fonts is to constantly change, according to glyphs used in the game. And this doesn't work well with any source control system. It's very annoying behavior that annihilates all pros of dynamic fonts over static ones.

I would like to make a feature request to provide some flag that will allow to not to store dynamic atlas on disk.
Something like 'ClearDynamicDataOnBuild'.

 

The font asset "Clear Dynamic Data on Build" option also clears this data when the editor is closed in the latest preview releases which for Unity 2020.3 or newer is version 3.2.0-pre.2.

In terms of not making this data serializable, that is a challenge in the sense that some users do want this data to be persistent in the Editor.

The majority of those users run full dynamic during development since it is more convenient and until they get a better understanding of their glyph coverage. For this use case, they want the data to be serialized and remain persistent until they are ready to convert some of their assets to Static and finalize their fallback structure. Some users also do ship dynamic font assets partially filled which provides some performance benefits as those existing characters and glyphs won't need to be added to the texture at runtime.

Having said all of the above, I am not opposed to providing another option to not serialize the data at all. I just need to find the time to look into it. In the meantime, if you are not using the latest release, please check if the modified "Clear Dynamic Data on Build" option where the data is flushed when closing the editor might work for you.

 

This means that before each commit I will have to close the editor, which is even worse than discarding changes in the git

 

@Stephan_B

I am facing the same problem.

Even if I have "ClearDynamicDataOnBuild" enabled when I close the editor, it still seems to be reset optimized for the Asset for the open scene. Ideally, the contents of Atllas are empty except in Play mode.

Is this my misconfiguration?

 

Yeah this makes version control of this asset even more confusing.

 

Any update or workaround for this? We have integrated TMP into a large legacy project, which is now causing font diffs for every dev that touches the project's new TMP content. We don't plan to include atlases in builds (we are already toeing the size limit), which makes these atlases only used in the editor and unnecessary to commit.

Short of writing a script that traverses every scene and prefab to render all the TMP text in the project and then commit it all in one go, I'm not sure what I can do currently to prevent these diffs.

Also, even if we commit all the changes, if the atlas is cleared when the editor closes due to "Clear Dynamic Data on Build", doesn't that start the issue all over again? I don't think that option should do that when the editor closes, or there should be an explicit option for clearing on editor close.

 

The workaround we're using is to

git update-index --assume-unchanged

on the font files. However, this is a gigantic pain in the ass when we do want to commit changes to the font files, just not to the atlases, since the data is all stored within one file.

 

If we think about it, font atlas texture should never be present on the disc, even if this is a static atlas.
In nutshell, FontAsses is a collection of settings that we want to serialize, save and share(in git for example). While the atlas texture itself is just a generative product of those settings. Atlas texture could (and in my opinion should) be stored in the Library folder, not in the project's assets.

AND there should be some command for "baking" atlas texture into the actual project file if user is willing to.

But again, storing the atlas texture itself does not make much sense. Especially for dynamic atlases.

So are you the solo dev on the TMPro package? Cuz if it was open source we could make some pull requests to fix common annoyances in the asset everyone love to use.

 

Very much agree that this (without investigating it much, admittedly) does seem like generated data that shouldn't exist in version control.

 

Please fix this, its a very annoying issue. If an atlas is fully dynamic please let us choose to not serialize any characters/glyphs/kerning/whatever else.

 

Is there an update on this issue? We'd like it to be purely dynamic and to never be saved to disk.

 

This is a problem with addressables as it dirties the asset and thus the bundle that the font asset resides in. It means when we push out content updates that the players will have to download the bundle containing our fonts every time again and again for no reason. It is costing us $$$ money serving data.

 

The majority of those users run full dynamic during development since it is more convenient and until they get a better understanding of their glyph coverage. For this use case, they want the data to be serialized and remain persistent until they are ready to convert some of their assets to Static and finalize their fallback structure. Some users also do ship dynamic font assets partially filled which provides some performance benefits as those existing characters and glyphs won't need to be added to the texture at runtime.

While this appears to be a noble goal, this needs to be a feature a user opts into not the default setting of TMP-generated fonts. The vast majority of users aren't going to know that TMP Pro can even do this or what is changing session to session. It's a bit bonkers to just have it functioning like this by default.

I'm with the 57 other users who think this is annoying and needs to be changed. I wasted 20 mins of my life trying to figure out why these were constantly changing for my team members.

@Stephan_B let's get a response here. It's been too long.

 

I most certainly see and understand how this can be an issue for many users. This issue was brought up by the team whose project I am currently reviewing in my new role with the consulting team at Unity.

I agree this functionality should be an option to control whether changes to font assets are persistent in the Editor or not. I did investigate this in the past but could not find any clear solution at that time.

Poke @HugoBD-Unity for visibility.

 

It was glossed over but I'd like to bring up again that this causes the addressable group that the font is located in to be dirty every time as well and requires redownload. An option is to put the dynamic font into its own bundle to minimise download size but still, it's gross.

Another option seems to be to generate the tmp asset at game startup and add it to the fallback list. Thus the font only exists in memory and there is no disk asset to manage.

However this option requires poking the fallback list on the primary font, which dirties it, so we're back to square 1.

Also the workflow here is messy, as you have the now manage the life cycle of this in-memory tmp asset and ensure things work correctly during runtime, edit play mode, but also in the editor when working with prefabs outside of play mode.

So anyway, just another +1 for an option on the tmp text asset to have it completely in memory and managed by tmp, no writes to disk, atlas generated on demand every time.

 

Do you have "Clear Dynamic Data on Build" enabled on those dynamic font assets?

In version 3.2.0-pre.x, this option also clears the dynamic data when the editor is closed. This option also resets the atlas texture back to size 0 (1 x 1 in newer releases). As such, dynamic font assets with Clear Dynamic Data on Build enabled should not take any space in builds (they should be kb in size / tiny).

So during an Editor session, dynamic font assets will pollute the repo but after the editor is closed, that should not be the case other than Font Feature data being added which doesn't get cleared since that is very expensive to repopulate at runtime.

P.S. Still agree that we need a better way to handle this. The above is just additional information.

 

Dynamic atlas generation is nice, but very annoying for source control when the text is being set dynamically in playmode. Would be very nice if the glyphtable would not be serialized depending on user input. Something like a HideFlag could potentially solve this issue.

 

As voter 73 of the people who are "annoyed with how dynamic fonts work with source control", I figured out some kind of workaround.

Put the following code as "OnSaveProcessor.cs" into the Editor folder

Code (CSharp):

1. using System.Linq;
2. using UnityEditor;
3.  
4. public class OnSaveProcessor : AssetModificationProcessor {
5.     public static string[] OnWillSaveAssets(string[] paths) {
6.         return paths.Where(p => !p.EndsWith("SDF.asset")).ToArray();
7.     }
8. }

The idea is to prevent saving the font asset once it was created and configured initially. The coding above is a very basic example that will remove any asset that ends with "SDF" (as font atlasses normally do) from the about-to-save-list.

Of course, you can get more advanced by filtering with a more specific name or maybe try to get and validate the assets properties by its file path. But you get the general approach

 

I want to understand why the ScaleRaioA saved property from debug settings for the material in the font asset is changed. What changed that? Who changed that? Why did that happen? There were no new chars, no new generation of the font, so what is the reason ?

 

Thanks for the push in the right direction @WongKit! I just made this script to auto-clear dynamic font data before any gets saved. Seems to fully fix the issue for me.

Code (CSharp):

1.  
2. // TextMeshPro dynamic font assets have a very annoying habit of saving their dynamically generated binary data in the
3. // same text file as their configuration data. This causes massive headaches for version control.
4. //
5. // This script addresses the above issue. It runs whenever any assets in the project are about to be saved. If any of
6. // those assets are a TMP dynamic font asset, they will have their dynamically generated data cleared before they are
7. // saved, which prevents that data from ever polluting the version control.
8. //
9. // For more information, see this thread: https://forum.unity.com/threads/tmpro-dynamic-font-asset-constantly-changes-in-source-control.1227831/
10.  
11.  
12. #if UNITY_EDITOR
13.  
14. using TMPro;
15. using UnityEditor;
16.  
17. internal class DynamicFontAssetAutoClear : AssetModificationProcessor
18. {
19.     static string[] OnWillSaveAssets(string[] paths)
20.     {
21.         foreach (string path in paths)
22.         {
23.             if (AssetDatabase.LoadAssetAtPath(path, typeof(TMP_FontAsset)) is TMP_FontAsset fontAsset)
24.             {
25.                 if (fontAsset.atlasPopulationMode == AtlasPopulationMode.Dynamic)
26.                 {
27.                     //Debug.Log("Clearing font asset data at " + path);
28.                     fontAsset.ClearFontAssetData(setAtlasSizeToZero: true);
29.                 }
30.             }
31.         }
32.  
33.         return paths;
34.     }
35. }
36.  
37. #endif
38.  

 

Wow, this is a way better handling of the issue. Thank you for sharing this snippet!

 

Thanks @cxode !
Thats awesome!

I quickly assembled a github UPM package with some improvements and editor test:
https://github.com/STARasGAMES/tmpro-dynamic-data-cleaner

That way you can just install the package instead of finding the right place for the script.


Anyways, here is my version of the script:

Code (CSharp):

1. // TextMeshPro dynamic font assets have a very annoying habit of saving their dynamically generated binary data in the
2. // same text file as their configuration data. This causes massive headaches for version control.
3. //
4. // This script addresses the above issue. It runs whenever any assets in the project are about to be saved. If any of
5. // those assets are a TMP dynamic font asset, they will have their dynamically generated data cleared before they are
6. // saved, which prevents that data from ever polluting the version control.
7. //
8. // For more information, see this post by @cxode: https://forum.unity.com/threads/tmpro-dynamic-font-asset-constantly-changes-in-source-control.1227831/#post-8934711
9.  
10. using System;
11. using TMPro;
12. using UnityEditor;
13. using UnityEngine;
14.  
15. namespace TMProDynamicDataCleaner.Editor
16. {
17.     internal class DynamicFontAssetAutoCleaner : UnityEditor.AssetModificationProcessor
18.     {
19.         static string[] OnWillSaveAssets(string[] paths)
20.         {
21.             try
22.             {
23.                 foreach (string path in paths)
24.                 {
25.                     var assetType = AssetDatabase.GetMainAssetTypeAtPath(path);
26.                    
27.                     // GetMainAssetTypeAtPath() sometimes returns null, for example, when path leads to .meta file
28.                     if (assetType == null)
29.                         continue;
30.                    
31.                     // TMP_FontAsset is not marked as sealed class, so also checking for subclasses just in case
32.                     if (assetType != typeof(TMP_FontAsset) && assetType.IsSubclassOf(typeof(TMP_FontAsset)) == false)
33.                         continue;
34.  
35.                     // Loading the asset only when we sure it is a font asset
36.                     var fontAsset = AssetDatabase.LoadMainAssetAtPath(path) as TMP_FontAsset;
37.  
38.                     // Theoretically this case is not possible due to asset type check above, but to be on the safe side check for null
39.                     if (fontAsset == null)
40.                         continue;
41.  
42.                     if (fontAsset.atlasPopulationMode != AtlasPopulationMode.Dynamic)
43.                         continue;
44.  
45.                     // Debug.Log("Clearing font asset data at " + path);
46.                     fontAsset.ClearFontAssetData(setAtlasSizeToZero: true);
47.                 }
48.             }
49.             catch (Exception e)
50.             {
51.                 Debug.LogException(e);
52.                 Debug.LogError("Something went wrong while clearing dynamic font data. For more info look at log message above.");
53.             }
54.  
55.             return paths;
56.         }
57.     }
58. }

 

Nice, thanks for posting your improvements, sharing is caring the try/catch is a really good call, we don't want important data to fail to be saved because of an issue with this script. Although, maybe it would be better inside the foreach loop?

Is there a specific reason you chose to switch out

if (AssetDatabase.LoadAssetAtPath(path, typeof(TMP_FontAsset)) is TMP_FontAsset fontAsset)

for the more verbose solution? The former performs a null check, a type check, and works with subclasses, all in one line.

 

Yes, you're right. That way we can provide log with a path to the font asset that caused issue.

There are two reasons
1. It started with the assumption that calling LoadAssetAtPath() on all provided paths is not efficient. And another assumption is that GetMainAssetTypeAtPath() is faster and doesn't call LoadAssetAtPath() internally.
On big projects this might be noticeable.
Must admit, I have no proof of my assumptions, but I have an argument against When Unity saves asset to disk it means this asset is already in memory and LoadAsset should not actually load data from disk and have any performance issues (in theory).

2. With try/catch block there is another layer of code nesting that looks messy, that's why I inverted all `if` statements to fail early. And I don't know how to elegantly invert `asset is TMP_FontAsset fontAsset` without `!` operator. `is not` operator not present in Unity 2020.3.

 

Hello,

had the same issue with some SDF fonts changing seemingly arbitrarely and it turned out to be this issue here. The AssetModificationProcessor solution fixed it for us. It's an accetable fix for issue that shouldn't exist.

When investigating the issue, I realized the root cause was the rather common mistake of mixing static asset data with dynamic runtime data. I can see how the TMP_FontAsset has evolved into the current form where dynamic atlasing was shoehorned into the existing static implementation. Unfortunately it doesn't play nicely and AssetDatabase assets get modified when running the game (in editor or in actual player build while latter doesn't really matter).

 

You kinda answered your own question there. AssetModificationProcessor is called for those assets that are being saved, and thus already in memory. LoadAssetAtPath should be relatively cheap.

You could optimize this a bit further by adding this first thing in the loop. All TMP_FontAssets are ScriptableObjects, and thus have the ".asset" extension:

Code (CSharp):

1. if (!path.EndsWith(".asset"))
2. {
3.    continue;
4. }

Rest of my processor looks like this:

Code (CSharp):

1. if (!typeof(TMP_FontAsset).IsAssignableFrom(AssetDatabase.GetMainAssetTypeAtPath(path)))
2.                 {
3.                     continue;
4.                 }
5.  
6.                 var font = AssetDatabase.LoadAssetAtPath<TMP_FontAsset>(path);
7.  
8.                 if (font.atlasPopulationMode == AtlasPopulationMode.Dynamic)
9.                 {
10.                     font.ClearFontAssetData(true);
11.                 }

What's wrong with

if (!(asset is TMP_FontAsset fontAsset)) { ... }

, or actually using the positive branch there without !

 

Actually...

unityObject is type

does not actually perform a proper null check. You can have deleted object of the type, and invoking `is` to it will happily give you true.

 

Ugh, this terminology is so confusing. For the purposes of this discussion, let's refer to

x is null

as "True Null Check" and

x == null

as "Is Destroyed Check". And note that doing an Is Destroyed Check also does a True Null Check.

So, the thing is,

AssetDatabase.LoadAssetAtPath

will never return a destroyed object, therefore an Is Destroyed Check is unnecessary. We'd want to have one if the loaded assets were being cached and referred back to over time, but within the scope of

OnWillSaveAssets

, we'll never have to deal with a destroyed object.

However,

AssetDatabase.LoadAssetAtPath

DOES sometimes return true

null

, and therefore a True Null Check is necessary.

 

Thanks for the suggestions in this thread. Our project had this problem and your solutions worked great.

Here's our version. The idea is that by default we don't want team members who play in the Editor to change the dynamic font asset. But we do want to be able to pre-load the dynamic font with common glyphs so that UI is a bit faster to load at runtime.

So we have an EditorPref that can be toggled on/off that allows dynamic fonts to be saved. By default it's off, but when it's time to pre-load the dynamic font atlas, one of us can toggle it on and play the game to add glyphs to the atlas.

Code (CSharp):

1.  
2. public class DynamicFontSaveProcessor : AssetModificationProcessor
3. {
4.     private const string MenuNameAllowSavingDynamicFonts = "MyProjectTools/Allow saving dynamic fonts";
5.     public const string AllowSavingDynamicFontsPref = "AllowSavingDynamicFonts";
6.  
7.     private static string[] OnWillSaveAssets(string[] paths)
8.     {
9.         if (EditorPrefs.GetBool(AllowSavingDynamicFontsPref))
10.         {
11.             return paths;
12.         }
13.         else
14.         {
15.             return paths.Where(path => !IsDynamicFontAsset(path)).ToArray();
16.         }
17.     }
18.  
19.     private static bool IsDynamicFontAsset(string assetPath)
20.     {
21.         var assetType = AssetDatabase.GetMainAssetTypeAtPath(assetPath);
22.         if (!typeof(TMP_FontAsset).IsAssignableFrom(assetType)) return false;
23.  
24.         var fontAsset = AssetDatabase.LoadAssetAtPath<TMP_FontAsset>(assetPath);
25.         if (fontAsset == null) return false;
26.  
27.         if (fontAsset.atlasPopulationMode != AtlasPopulationMode.Dynamic) return false;
28.  
29.         return true;
30.     }
31.  
32.     [MenuItem(MenuNameAllowSavingDynamicFonts)]
33.     private static void MenuAllowSavingDynamicFonts()
34.     {
35.         EditorPrefs.SetBool(AllowSavingDynamicFontsPref, !EditorPrefs.GetBool(AllowSavingDynamicFontsPref));
36.     }
37.  
38.     [MenuItem(MenuNameAllowSavingDynamicFonts, isValidateFunction: true)]
39.     private static bool MenuAllowSavingDynamicFontsValidate()
40.     {
41.         Menu.SetChecked(MenuNameAllowSavingDynamicFonts, EditorPrefs.GetBool(AllowSavingDynamicFontsPref));
42.         return true;
43.     }
44. }
45.  

 

Love that this issue is getting a fix. I'm trying to use this package though and I'm getting this error

[Package Manager Window] Error adding package: https://github.com/STARasGAMES/tmpro-dynamic-data-cleaner.git.

Unable to add package [https://github.com/STARasGAMES/tmpro-dynamic-data-cleaner.git]:

  [https://github.com/STARasGAMES/tmpro-dynamic-data-cleaner.git] does not point to a valid package. No package manifest was found.

UnityEditor.EditorApplication:Internal_CallUpdateFunctions ()

 

Please, look at the installation chapter:
https://github.com/STARasGAMES/tmpro-dynamic-data-cleaner#installation

 

Thanks much! Making it rain @WongKit , @cxode , @MUGIK

 

Firstly, thanks for putting in the time and effort into making this a UPM compatible package.

Unfortunately, the asset file changes are still being generated for me. Is there an initial commit that needs to happen before this begins taking effect, or should it work immediately?

I'm using Unity 2022.3.14f1 (PC, targeting standalone).

Additionally, enabling the Test Runner by adding the "testables" attribute results in a compiler error.

Code (CSharp):

1. Library\PackageCache\com.starasgames.tmpro-dynamic-data-cleaner@3a36076214\Tests\Editor\DynamicFontAssetAutoCleanerTests.cs(38,13): error CS0122: 'TMP_FontAsset_CreationMenu' is inaccessible due to its protection level

It seems TMP_FontAsset_CreationMenu is not a _public_ static class.

I tried using a manual menu item call to achieve the same effect, but that doesn't work.

e.g.

Code (CSharp):

1.             EditorApplication.ExecuteMenuItem("Assets/Create/TextMeshPro/FontAsset/Font Asset Variant");

Happy to post this as an issue on GitHub if that's preferred.

 

Yes, you need one additional commit. Most likely git stores version of the asset with a glyph information, that's why when there is no more glyph info git identifies this as a change. After committing this "empty" font asset it should no more appear in local changes.

We're using SourceTree and sometimes font assets still manage to appear in local changes, but changes disappear when I try to stage them. I guess that either Git or SourceTree uses some FileModified event from the OS to mark file as dirty, but when you stage this file it runs another check and realizes that there are no actual changes in a file.

Originally, package was developed for 2020.3. Maybe they changed API.