TextMesh Pro Version 1.4.1-Preview 1 with Dynamic SDF for Unity 2018.3 now available!

At long last, the wait is over!

Today, I am pleased to announce the release of the first preview of TextMesh Pro version 1.4.0 for Unity 2018.3 with support for Dynamic SDF.

One of the most requested feature since the initial release of TMP has been support for Dynamic SDF which is now available in this new release.

Dynamic SDF makes it possible to add glyphs and characters to font assets at runtime thus significantly enhancing and simplifying workflows for handling localization and languages like CJK with large character sets. This new functionality is supported throughout TMP and its fallback system.

In order to support this new feature, structural changes to Font Assets and Sprite Asset have been necessary and will require existing font assets and sprite assets to be upgraded. This upgrade should be automatically handled by the system as these resources are loaded / accessed in your projects. Although I have been testing this process using dozens of user projects where everything worked fine, I am releasing this new version as a preview as I need your help in further testing and making sure this process works as expected. With that said ...

Steps to Upgrade to this new release
(1) Backup your project first. This is a preview release so let's not take any chances.
(2) Open this backup of your project in Unity 2018.3.
(3) Close any scene that may already be opened in Unity. Although the asset migration process should still work. It is preferable to close any opened scenes.
(4) Open Package Manager UI via the "Window - Package Manager" menu and as per the image below select the latest preview version available for version 1.4.0.



(5) Start opening scenes to see if everything works as expected. If not please report these issues in this thread and as usual I'll get those addressed.

(6) If your project contains many large font assets where many of them happen to be assigned as fallbacks in the TMP Settings, opening the TMP Settings which would force all these font assets to converted as a group might appear to lock up the editor. So if you run into this, instead of accessing the TMP Settings, just select each of these font assets one at a time in the project tab. Wait for each of them to be converted before selecting the next one.

This should be it in terms of the migration.

Next is getting familiar with the new Dynamic SDF system and added flexibility it will offer and new potential workflows in regards to managing these resources. I have already published the following video which covers a big chunk of the new Dynamic System and Fallback.

The following new video covers the upgrade process as well as provides insight on the structural changes to Font Assets and Sprite Assets. The video also covers some new additional functionality that had not previously been shown / discussed.

FAQ
Question 1
Can the content of a font asset be cleared / reset?

Yes. In the Editor, you can use the Context Menu option Reset which will clear the Character, Glyph and Glyph Adjustment Tables as well as reset the atlas texture back to size 0.

Via scripting you can use the following function.

Code (csharp):

1. public void ClearFontAssetData(bool setAtlasSizeToZero = false)

Question 2
I am trying to switch a previously created font asset to dynamic mode and getting a warning about the atlas texture not being readable. How do I set the texture to readable?

In the font asset inspector, double click on the Font Atlas to navigate to the atlas texture. Then switch the inspector to debug mode where the Is Readable flag will now be exposed.



Question 3
Can a font asset be created at runtime?

Yes. The font asset can be created using the following function.

Code (csharp):

1. public static TMP_FontAsset CreateFontAsset(Font font, int samplingPointSize, int atlasPadding, GlyphRenderMode renderMode, int atlasWidth, int atlasHeight, AtlasPopulationMode atlasPopulationMode = AtlasPopulationMode.Dynamic)

As more questions or suggestions come up, I'll add those to the FAQ.

Feedback
Please report any issues in this thread. Feel free to ask questions if anything is unclear. I am here and available to help.

 

Glad it is working well and you are most welcome

In terms of the question, the challenge with overwriting a specific glyph is that glyphs come in all shapes and sizes and as such the new glyph could be bigger than the one it is replacing. So doing this could be tricky.

Having said that, Font Assets can now be Dynamic in terms of adding glyphs and characters to them at runtime but Font Asset themselves can be created at runtime.

Code (csharp):

1.  
2. // Create a new font asset at runtime.
3. m_DynamicFontAsset = TMP_FontAsset.CreateFontAsset(SourceFontFile, 90, 9, GlyphRenderMode.SDFAA, 1024, 1024, AtlasPopulationMode.Dynamic);
4.  
5. // Force some characters to be added to the font asset. Ie. as opposed to some text object forcing those to be added).
6. m_DynamicFontAsset.TryAddCharacters("1234");
7.  
8. //  Assign newly created font asset to some text object.
9. m_TextComponent.font = m_DynamicFontAsset;
10.  
11. // Set the text on the object which in turn will force these to be added to this new font asset dynamically.
12. m_TextComponent.text = "ABCD";
13.  

Given the above code and assuming those characters were available in the font file, the font asset and atlas texture would end up with "1234" and "ABCD" in it.

You can also clear the font asset data using the following. This clears the glyph, character and glyph adjustment tables as well as atlas texture (reset to 0 X 0 in size) but maintains all the relevant settings so you can use it normally and resume adding characters to it.

Code (csharp):

1. m_DynamicFontAsset.ClearFontAssetData();

Between all of these, you can create new font assets, pre-populate them with some characters. Reset them to clear out their characters and all of that at runtime.

 

Most of the time this issue is the result of the scale / tweening not doing uniform scaling on the X, Y and Z. For certain, a scale of zero on the Y of the text object would cause the behavior you describe.

Note that I did change the handling of the SDF scaling so if the issue persists, I'll take a look at it and if you could please provide some simple repro project / scene that I could use to reproduce the behavior which usually greatly speeds up the time it takes to resolve this issues.

 

It does indeed. I use it myself when I need to reset / clear a font asset for testing dynamically adding characters in builds / on devices.

Now just to make sure I don't overlook anything with respect to your use case, what was your thought with regards to doing it one character at a time?

P.S. Since it is already 4:00 on my end, I am going to go sleep but I'll follow up tomorrow.

 

Thank you. I'll take a look tomorrow when I wake up.

Update - I know what the issue is and will have a fix / code change you can make tomorrow.

 

@Binouze Thanks for the time you took to prepare the Repro project as it allowed me to uncover another issue which I would not have otherwise.

The following changes will be necessary to resolve the issue.



Let me know if this resolves the issue. If so these changes will be included in the new preview release.

Special Note: As of Unity 2018.3, packages are additionally cached on a per project basis where these local cached version are located in the project folder in "Library/PackageCache/..." which is where these changes should be done for the given project.

 

By the time I got around to adding the ability to retrieve Glyph Adjustment Data (Kerning) dynamically, it became clear that I was going to need to make changes to the FontEngine and bindings which unfortunately are part of Unity itself and not part of the TMP package. As such this will require a Unity update which I should be able to get approved for 2019.1 but unlikely for 2018.3 given focus now is all on stability.

As per my TODO note, I am fully aware of the shortcomings of that current implementation and not happy about it but had limited options if I wanted this functionality short term.

Having said that, I could add new settings to the TMP Settings where trying to retrieve Glyph Adjustment Data dynamically at runtime could be enabled / disabled.

For some languages Kerning is pretty common but for CJK not as much so maybe this could be a viable short term solution.

P.S. Can you give me an idea of the number of glyphs your font asset contains?

 

How many glyphs are you planning to add / does your font asset contain?

For any given project / per language set, you are likely to only end up using a very small subset of those 60,000 glyphs.

My recommendations remain to have your Primary font assets (for any given language or groups of languages) contain the known characters / glyphs in the project which are the characters found in your the your titles, menus, ui, etc. This font asset should be prepared ahead of time just like before and be static.

Then to handle user input and all other potential characters, assign to your primary static font asset a fallback that is dynamic and there to catch all other (unknown / not already part of the project) characters.

This will provide for best performance and quality for all the known characters / text in the project. The primary would already contain the Glyph Adjustment Data (Kerning) since it is static and prepared ahead of time. Then for all other characters, where dynamic fallbacks are used there will be a higher performance overhead but your likely to only have to handle a small number of characters and furthermore given humans don't type that fast relative to computer processing speed, the performance overhead will not be noticeable by any users.

Although you could rely on the dynamic system to handle all characters without preparing / using any static font assets, I don't think that's the best strategy for managing these resources.

P.S. Let me know if you feel adding the ability to disable dynamic retrieval of Glyph Adjustment Data in the TMP Settings would be useful?

 

Fix an issue where creating Font Assets at runtime were not getting their version number set correctly to version 1.1.0.

This change will be included in the next preview release but just in case, you wish to experiment with font asset creation at runtime, here are the relevant changes.



Special Note: As of Unity 2018.3, packages are additionally cached on a per project basis where these local cached version are located in the project folder in "Library/PackageCache/..." which is where these changes should be done for the given project.

 

Adding a new character and glyph to a font asset will result in an allocation of 64 bytes for the glyph and 40 bytes for the character. There is also another allocation of 76 bytes for an internal list which I will try to re-work / remove. So in the next release version 1.4.0-preview.2 (which I'll release this week) the one time allocation per character & glyph added to a font asset will be 180 bytes and if I can do away with the internal list that will reduce it by 76 bytes.

I have already added the ability to disable retrieving Glyph Adjustment Data (Kerning Pairs) dynamically. This option will be in the TMP Settings. I have already re-worked / removed the use of .ToArray() which was responsible for the bulk of those allocations. The revising implementation will be in another TMP package release which I should be able to get into 2019.1 and will do my best to get in 2018.3. Regardless I'll look for some alternative for 2018.3 just in case.

Allocations per Glyph Pair Adjustment record is 64 bytes. So if you add a character & glyph that has 10 pairs then it will allocate 640 bytes.

Unless you are adding new characters, glyphs and adjustment pairs there should be no allocations.

 

Glyph Adjustment Table Changes
In preparation for the addition of support for OpenType Font Features, the Glyph Adjustment Table presentation and data structure has been changed as seen below.



The updated Glyph Adjustment Table now shows the glyphs for each of the Glyph Pair Adjustment Records. Glyphs will only be shown if they are already present in the Glyph Table.

Individual pairs now reference the glyph index as opposed to unicode characters. When adding new adjustment pairs, the Character or its UTF16 or UTF32 representation can be used to lookup the glyph index for the given character.

Note that it is now possible to define / add GlyphPairAdjustmentRecords for glyphs that are not already present in the Glyph Table. This can occurs when the new "Get Font Features at Runtime" option is enabled in the TMP Settings.



When this option is enabled and as new glyphs are added to the Glyph Table, Glyph Pair Adjustment Records for those new glyphs will be automatically added to the Glyph Adjustment Table.

Font Assets that contain existing legacy kerning data will be automatically updated to the new format when these font assets are first loaded in the Editor or at Runtime (if somehow these font assets never ended up being loaded in the Editor during development or testing).

This new change will be included in the next preview release of TextMesh Pro package. This new release will be version 1.4.0-preview.2 which I am in the final stages of testing.

 

As characters and indirectly glyphs are requested, TMP will search (like it did before) the primary font asset and potential fallbacks. In this process if any font asset is set to Dynamic, TMP will try to add the requested character / glyph to it but if it is unable to do so, it simply moves to the next fallback.

One potential way to handle an ever growing list of characters / glyphs would be to create (ahead of time) several fallbacks that are all empty with the relevant generation settings and to assign them to this primary font asset or wherever in the fallback chain you want them.

Assuming you ended up creating 10 empty fallback font assets with atlas texture size of 1024 x 1024, one of the issue would have been all those textures being allocated but as it turns out, one of the last changes / improvements I made before releasing this new version was to have the atlas texture of these newly created / empty font assets uninitialized / of size zero until you try to add the first character. As such, this makes creating / using several empty font assets very efficient.

Furthermore, it is also possible to create font assets at runtime which won't be too useful in this case yet given I have not exposed any API to get callbacks to allow you to manage / create new font assets as needed but something that would enable the dynamic system to create font assets on demand or additional atlas textures which font assets are already setup internally to handle. Regardless, the ability to either add additional atlas textures per font asset or to create fallback font assets on demand is certainly all possible.

So short term, go with creating whatever number of empty fallbacks you believe will be needed. Keep in mind that these fallbacks can have different sampling point size and padding but still need to have the same ratio of sampling to point size as the primary (reason for that is explained in that video and in many others as well).

 

As far as I know Addressables do not currently allow resources to be load synchronously which is an issue when the text object needs its font asset in order to display text. Until then we still need to use the resources folder.

Having said that, I am likely to add a callback like OnFontAssetRequest, OnSpriteAssetRequest, etc. which users would be able to use to implement their own loading handling of resources like from Asset Bundles and perhaps Addressables given you would have some level of control over how these resources are loaded.

 

New preview release is now available. Release is version 1.4.0-preview.2a.

It addresses the issues reported thus far along with the changes to the Glyph Adjustment Table I referenced in the post above.

See the ChangeLog for more details.

 

Not without more information but let's first check if you get the same behavior using 1.4.0-preview.2a.

If you still get the issue, you could either submit a bug report with the project and then provide me with the case # or if easier check if you are able to reproduce the behavior in some new project / simple scene which you could provide me to look at.

P.S. If I was to take a wild guess, it would be extracting of Kerning data at runtime which in 1.4.0-preview.1b results in way too much allocation which should be much better in 1.4.0-preview.2a.

 

This is expected behavior in the Editor / Inspector as non-persistent / scene only assets cannot be referenced by persistent assets. Having said that, this behavior does not prevent the dynamic creation of font assets and their assignment to any other font assets (persistent or not) at runtime.

This was an issue which I have now fixed. I also added the following overload to this function which will return an array that contains the unicode characters that could not be added to the font asset.

Code (csharp):

1. public bool TryAddCharacters(uint[] unicodes, out uint[] missingUnicodes)

It currently does not.

If the function returns false, then you know not all characters were added. As per my previous comment above, I add an overload to this function that returns the unicode characters that could not be added.

Also added similar overload to the other TryAddCharacters function

Code (csharp):

1. public bool TryAddCharacters(string characters, out string missingCharacters)

Out of curiosity, how many unique characters does your largest translation file contain? It might make sense to use (1) static font asset that contains all these characters for each language / groups of languages and then simply rely on one dynamic font asset to capture remaining characters potentially coming from user input.

 

Since the material and atlas texture are sub assets, the Unity rename function doesn't work on sub objects.

How did you change that name initially? Was the font asset named that way?

 

TMP package version 1.4.0-preview.3a has been released.

This release includes updated Distance Field and Mobile Distance Field shaders to support single pass stereo rendering.

Includes a few minor fixes.

 

It is included in preview.3a.

 

It is important to keep in mind that changes to font assets and their atlas textures in the Editor are persistent. However, that is not the case in builds and as such these changes are not persistent between play sessions resulting in dynamic font assets and fallbacks resetting to their original state for each play session. This should minimize / limit the number of glyphs ultimately being added / needed.

The following function will allow you to clear a font asset / fallback. In your case, I would not reset the atlas size to zero as you expect to resume adding glyphs into it right away.

Code (csharp):

1. public void ClearFontAssetData(bool setAtlasSizeToZero = false)

To test this in the Editor, I did override the Context Menu Reset option on the font assets which uses the above function. An important part of that however if a callback (in the Editor only) which is as follows:

Code (csharp):

1.  
2. // Called in this Reset function
3. TMPro_EventManager.ON_FONT_PROPERTY_CHANGED(true, fontAsset);
4.  
5. // Implementation in the TMPro_Private.cs and TMPro_UGUI_Private.cs
6. void ON_FONT_PROPERTY_CHANGED(bool isChanged, TMP_FontAsset font)

Although this Callback is not available at runtime, you can review the source code to see what it actually calls on the text object. BTW: the UpdateMeshPadding() part can be ignore in this case as the font asset data like point size or padding values would not be changes and this call would not be necessary.

Assuming you have a list of text objects in your scene, you could implement you own method to force an update of the text objects. You could use ForceMeshUpdate() which pretty much does the same thing as the callback but without checking if the text object is using the font asset that was just reset which is something you should handle.

Assuming you had a bunch of text objects on screen with text, doing this reset followed by forcing the text objects to update would result in whatever visible glyphs being added again. Ie. no need to figure out manually what should be added or not. Again, for testing create a few text objects that contain some text and then use the Context Menu Reset on the Font Asset and you will see whatever characters / glyphs being used added back in the atlas right away whereas those no longer used not being added.

I guess at some point I could add a runtime callback but until then you can implement your own manager to force an update on text objects that are using the font asset / fallback that were just reset.

 

Are you using a static font atlas that contains the known text in the project? That would reduce the amount of potential glyphs needed to be added.

 

What version of Unity and TextMesh Pro are you upgrading from?

The line of code giving this error is simply

Code (csharp):

1. return Resources.Load<TMP_Settings>("TMP Settings");

so maybe to package cache or possibly this file got corrupted.

Have you modified your TMP Settings? If not, re-import the TMP Essential Resources.

 

I just installed Unity 2018.3.8f1.

Created a new project and made sure it was using TMP version 1.3.0. Imported the TMP Essential Resources (from 1.3.0), then upgraded to 1.4.0-preview.3a and no errors.

Also tested, upgrading to 1.4.0-preview.3a and the importing the TMP Essential Resources and now errors in that case either.

See if you get the same behavior when creating an empty project and then upgrading TMP in that project to 1.4.0-preview.3a and then importing the updated TMP Essential Resources from that release.

BTW: The function signature and return type is as follows:

Code (csharp):

1. static UnityEngine.Object GetTMPSettings()

so that should be fine.

Let me know what happens testing with a new / empty project. If the behavior is the same, then delete the package cache from "...\AppData\Local\Unity\cache"

 

The fallback would only be necessary to handle characters not already present in the Primary and coming from user input. If there is no user input then a fallback would not be necessary.

Having said that, since Fallback font asset atlas textures are not sized until you add characters to them, there is no downside to including an empty fallback just in case somehow your Primary font asset was missing some character.