TextMesh Pro TMPro Dynamic Font Asset constantly changes in source control

Hi everyone,

The problem is that font's atlas texture is a sub-asset of the font asset itself. This makes sense for static fonts.
But with dynamic fonts I don't want to store this atlas on the disk, why would I?

The whole point of dynamic fonts is to constantly change, according to glyphs used in the game. And this doesn't work well with any source control system. It's very annoying behavior that annihilates all pros of dynamic fonts over static ones.

I would like to make a feature request to provide some flag that will allow to not to store dynamic atlas on disk.
Something like 'ClearDynamicDataOnBuild'.

 

This means that before each commit I will have to close the editor, which is even worse than discarding changes in the git

 

@Stephan_B

I am facing the same problem.

Even if I have "ClearDynamicDataOnBuild" enabled when I close the editor, it still seems to be reset optimized for the Asset for the open scene. Ideally, the contents of Atllas are empty except in Play mode.

Is this my misconfiguration?

 

Yeah this makes version control of this asset even more confusing.

 

Any update or workaround for this? We have integrated TMP into a large legacy project, which is now causing font diffs for every dev that touches the project's new TMP content. We don't plan to include atlases in builds (we are already toeing the size limit), which makes these atlases only used in the editor and unnecessary to commit.

Short of writing a script that traverses every scene and prefab to render all the TMP text in the project and then commit it all in one go, I'm not sure what I can do currently to prevent these diffs.

Also, even if we commit all the changes, if the atlas is cleared when the editor closes due to "Clear Dynamic Data on Build", doesn't that start the issue all over again? I don't think that option should do that when the editor closes, or there should be an explicit option for clearing on editor close.

 

The workaround we're using is to

git update-index --assume-unchanged

on the font files. However, this is a gigantic pain in the ass when we do want to commit changes to the font files, just not to the atlases, since the data is all stored within one file.

 

If we think about it, font atlas texture should never be present on the disc, even if this is a static atlas.
In nutshell, FontAsses is a collection of settings that we want to serialize, save and share(in git for example). While the atlas texture itself is just a generative product of those settings. Atlas texture could (and in my opinion should) be stored in the Library folder, not in the project's assets.

AND there should be some command for "baking" atlas texture into the actual project file if user is willing to.

But again, storing the atlas texture itself does not make much sense. Especially for dynamic atlases.

So are you the solo dev on the TMPro package? Cuz if it was open source we could make some pull requests to fix common annoyances in the asset everyone love to use.

 

Very much agree that this (without investigating it much, admittedly) does seem like generated data that shouldn't exist in version control.

 

Please fix this, its a very annoying issue. If an atlas is fully dynamic please let us choose to not serialize any characters/glyphs/kerning/whatever else.

 

Is there an update on this issue? We'd like it to be purely dynamic and to never be saved to disk.

 

This is a problem with addressables as it dirties the asset and thus the bundle that the font asset resides in. It means when we push out content updates that the players will have to download the bundle containing our fonts every time again and again for no reason. It is costing us $$$ money serving data.

 

The majority of those users run full dynamic during development since it is more convenient and until they get a better understanding of their glyph coverage. For this use case, they want the data to be serialized and remain persistent until they are ready to convert some of their assets to Static and finalize their fallback structure. Some users also do ship dynamic font assets partially filled which provides some performance benefits as those existing characters and glyphs won't need to be added to the texture at runtime.

While this appears to be a noble goal, this needs to be a feature a user opts into not the default setting of TMP-generated fonts. The vast majority of users aren't going to know that TMP Pro can even do this or what is changing session to session. It's a bit bonkers to just have it functioning like this by default.

I'm with the 57 other users who think this is annoying and needs to be changed. I wasted 20 mins of my life trying to figure out why these were constantly changing for my team members.

@Stephan_B let's get a response here. It's been too long.

 

It was glossed over but I'd like to bring up again that this causes the addressable group that the font is located in to be dirty every time as well and requires redownload. An option is to put the dynamic font into its own bundle to minimise download size but still, it's gross.

Another option seems to be to generate the tmp asset at game startup and add it to the fallback list. Thus the font only exists in memory and there is no disk asset to manage.

However this option requires poking the fallback list on the primary font, which dirties it, so we're back to square 1.

Also the workflow here is messy, as you have the now manage the life cycle of this in-memory tmp asset and ensure things work correctly during runtime, edit play mode, but also in the editor when working with prefabs outside of play mode.

So anyway, just another +1 for an option on the tmp text asset to have it completely in memory and managed by tmp, no writes to disk, atlas generated on demand every time.

 

Dynamic atlas generation is nice, but very annoying for source control when the text is being set dynamically in playmode. Would be very nice if the glyphtable would not be serialized depending on user input. Something like a HideFlag could potentially solve this issue.

 

As voter 73 of the people who are "annoyed with how dynamic fonts work with source control", I figured out some kind of workaround.

Put the following code as "OnSaveProcessor.cs" into the Editor folder

Code (CSharp):

1. using System.Linq;
2. using UnityEditor;
3.  
4. public class OnSaveProcessor : AssetModificationProcessor {
5.     public static string[] OnWillSaveAssets(string[] paths) {
6.         return paths.Where(p => !p.EndsWith("SDF.asset")).ToArray();
7.     }
8. }

The idea is to prevent saving the font asset once it was created and configured initially. The coding above is a very basic example that will remove any asset that ends with "SDF" (as font atlasses normally do) from the about-to-save-list.

Of course, you can get more advanced by filtering with a more specific name or maybe try to get and validate the assets properties by its file path. But you get the general approach

 

I want to understand why the ScaleRaioA saved property from debug settings for the material in the font asset is changed. What changed that? Who changed that? Why did that happen? There were no new chars, no new generation of the font, so what is the reason ?

 

Thanks for the push in the right direction @WongKit! I just made this script to auto-clear dynamic font data before any gets saved. Seems to fully fix the issue for me.

Code (CSharp):

1.  
2. // TextMeshPro dynamic font assets have a very annoying habit of saving their dynamically generated binary data in the
3. // same text file as their configuration data. This causes massive headaches for version control.
4. //
5. // This script addresses the above issue. It runs whenever any assets in the project are about to be saved. If any of
6. // those assets are a TMP dynamic font asset, they will have their dynamically generated data cleared before they are
7. // saved, which prevents that data from ever polluting the version control.
8. //
9. // For more information, see this thread: https://forum.unity.com/threads/tmpro-dynamic-font-asset-constantly-changes-in-source-control.1227831/
10.  
11.  
12. #if UNITY_EDITOR
13.  
14. using TMPro;
15. using UnityEditor;
16.  
17. internal class DynamicFontAssetAutoClear : AssetModificationProcessor
18. {
19.     static string[] OnWillSaveAssets(string[] paths)
20.     {
21.         foreach (string path in paths)
22.         {
23.             if (AssetDatabase.LoadAssetAtPath(path, typeof(TMP_FontAsset)) is TMP_FontAsset fontAsset)
24.             {
25.                 if (fontAsset.atlasPopulationMode == AtlasPopulationMode.Dynamic)
26.                 {
27.                     //Debug.Log("Clearing font asset data at " + path);
28.                     fontAsset.ClearFontAssetData(setAtlasSizeToZero: true);
29.                 }
30.             }
31.         }
32.  
33.         return paths;
34.     }
35. }
36.  
37. #endif
38.  

 

Wow, this is a way better handling of the issue. Thank you for sharing this snippet!

 

Thanks @cxode !
Thats awesome!

I quickly assembled a github UPM package with some improvements and editor test:
https://github.com/STARasGAMES/tmpro-dynamic-data-cleaner

That way you can just install the package instead of finding the right place for the script.


Anyways, here is my version of the script:

Code (CSharp):

1. // TextMeshPro dynamic font assets have a very annoying habit of saving their dynamically generated binary data in the
2. // same text file as their configuration data. This causes massive headaches for version control.
3. //
4. // This script addresses the above issue. It runs whenever any assets in the project are about to be saved. If any of
5. // those assets are a TMP dynamic font asset, they will have their dynamically generated data cleared before they are
6. // saved, which prevents that data from ever polluting the version control.
7. //
8. // For more information, see this post by @cxode: https://forum.unity.com/threads/tmpro-dynamic-font-asset-constantly-changes-in-source-control.1227831/#post-8934711
9.  
10. using System;
11. using TMPro;
12. using UnityEditor;
13. using UnityEngine;
14.  
15. namespace TMProDynamicDataCleaner.Editor
16. {
17.     internal class DynamicFontAssetAutoCleaner : UnityEditor.AssetModificationProcessor
18.     {
19.         static string[] OnWillSaveAssets(string[] paths)
20.         {
21.             try
22.             {
23.                 foreach (string path in paths)
24.                 {
25.                     var assetType = AssetDatabase.GetMainAssetTypeAtPath(path);
26.                    
27.                     // GetMainAssetTypeAtPath() sometimes returns null, for example, when path leads to .meta file
28.                     if (assetType == null)
29.                         continue;
30.                    
31.                     // TMP_FontAsset is not marked as sealed class, so also checking for subclasses just in case
32.                     if (assetType != typeof(TMP_FontAsset) && assetType.IsSubclassOf(typeof(TMP_FontAsset)) == false)
33.                         continue;
34.  
35.                     // Loading the asset only when we sure it is a font asset
36.                     var fontAsset = AssetDatabase.LoadMainAssetAtPath(path) as TMP_FontAsset;
37.  
38.                     // Theoretically this case is not possible due to asset type check above, but to be on the safe side check for null
39.                     if (fontAsset == null)
40.                         continue;
41.  
42.                     if (fontAsset.atlasPopulationMode != AtlasPopulationMode.Dynamic)
43.                         continue;
44.  
45.                     // Debug.Log("Clearing font asset data at " + path);
46.                     fontAsset.ClearFontAssetData(setAtlasSizeToZero: true);
47.                 }
48.             }
49.             catch (Exception e)
50.             {
51.                 Debug.LogException(e);
52.                 Debug.LogError("Something went wrong while clearing dynamic font data. For more info look at log message above.");
53.             }
54.  
55.             return paths;
56.         }
57.     }
58. }

 

Nice, thanks for posting your improvements, sharing is caring the try/catch is a really good call, we don't want important data to fail to be saved because of an issue with this script. Although, maybe it would be better inside the foreach loop?

Is there a specific reason you chose to switch out

if (AssetDatabase.LoadAssetAtPath(path, typeof(TMP_FontAsset)) is TMP_FontAsset fontAsset)

for the more verbose solution? The former performs a null check, a type check, and works with subclasses, all in one line.

 

Yes, you're right. That way we can provide log with a path to the font asset that caused issue.

There are two reasons
1. It started with the assumption that calling LoadAssetAtPath() on all provided paths is not efficient. And another assumption is that GetMainAssetTypeAtPath() is faster and doesn't call LoadAssetAtPath() internally.
On big projects this might be noticeable.
Must admit, I have no proof of my assumptions, but I have an argument against When Unity saves asset to disk it means this asset is already in memory and LoadAsset should not actually load data from disk and have any performance issues (in theory).

2. With try/catch block there is another layer of code nesting that looks messy, that's why I inverted all `if` statements to fail early. And I don't know how to elegantly invert `asset is TMP_FontAsset fontAsset` without `!` operator. `is not` operator not present in Unity 2020.3.