Character not found although present, TMPro + Chinese + Localization

Hi,

so, we have a fairly large project which up to now only used English. We've now added Localization with the new Unity Localization package, which worked fine so far. As part of our languages, we have Chinese, Korean and Japanese.

I've added the locales and the text. However, when I now switch the locale to Chinese, I get hundreds of errors about characters not being found. The errors look like this:

Code (CSharp):

1. The character with Unicode value \u8865 was not found in the [ConthraxSb-Regular-Tweaked SDF dynamic] font asset or any potential fallbacks. It was replaced by Unicode character \u25A1 in text object [RegularTitle].
2. UnityEngine.Debug:LogWarning (object,UnityEngine.Object)
3. TMPro.TextMeshProUGUI:SetArraySizes (TMPro.TMP_Text/UnicodeChar[]) (at Library/PackageCache/com.unity.textmeshpro@2.1.4/Scripts/Runtime/TMPro_UGUI_Private.cs:1273)
4. TMPro.TMP_Text:ParseInputText () (at Library/PackageCache/com.unity.textmeshpro@2.1.4/Scripts/Runtime/TMP_Text.cs:1927)
5. TMPro.TextMeshProUGUI:OnPreRenderCanvas () (at Library/PackageCache/com.unity.textmeshpro@2.1.4/Scripts/Runtime/TMPro_UGUI_Private.cs:1634)
6. TMPro.TextMeshProUGUI:Rebuild (UnityEngine.UI.CanvasUpdate) (at Library/PackageCache/com.unity.textmeshpro@2.1.4/Scripts/Runtime/TextMeshProUGUI.cs:216)
7. UnityEngine.Canvas:SendWillRenderCanvases ()
8.  

The text setup is that we have a regular language (ConthraxSb) with Dynamic setup, and fallbacks to the entire Noto Sans family (CJK- sc, -jp, -kr). I've checked, the characters which can't be found as per the above error message are definitely there in the font files (these are the original .otf files from Google). The funny thing is that *some* characters are found, but most are not.

I've tried finding out what the cause is:

• I removed the ConthraxSb font and directly used the CJK-sc file (in a Dynamic font asset), which did not work either.
  
• I changed the CJK-sc asset to "static", used a non-localized TextMeshPro UI element and copied some characters in and rebuild the atlas, this did not work either (i.e. the characters were not added automatically to the list).
  
• I then manually added the character hex codes to the table, rebuild the atlas, and now I get the character (so something seems to be going wrong even when not using a Dynamic Font, but also in the auto-detection of characters for static texture atlasing?).

I've then tried to re-create a small test project with the same errors and I was unable to; everything there worked flawlessly. I tried rebuilding the entire library on our big project, but this did not change anything. The error happens both when starting from the editor and when switching languages in the built version. We have multiple scenes and it happens in all of them.

It seems to me that somewhere, somehow, something goes wrong with the logic which should decide which characters to add the the map, both with Dynamic fonts and with Static-fonts-based-on-Text-In-Scene.

I know you'd like a small test project for checking what goes wrong here, but so far I wasn't able to create one.

However, my questions are: Do you have any ideas/instructions on what I might do in my big project to somehow figure our where it's going wrong? Are there any more caches/libs I might delete? Can I rebuild something else? I.e. how would I debug this?

Thank you,

Philip

 

Those are warnings to inform you that the font file and font assets that you are using do not contain those characters and glyphs. These warnings can be disabled in the TMP Settings.

Can you post an image of your NotoSans Dynamic Font Asset inspector where I can see the Generation Settings, Fallback Font Asset list and other tables collapsed (only looking for counts on those).

Please also post an image of the atlas texture of your font asset.

Lastly, which version of the TMP package are you using in the project?

P.S. The following two videos provide a pretty solid overview of dynamic font assets, fallbacks and localization for CJK. Perhaps reviewing these might shed some light on some missed setting or something.

 

Hi,

thank you for your answer, and for the videos, which helped me solve the issue. If I am not mistaken, characters added in the editor will never be removed from a Dynamic Atlas, so if "Multi-Atlas Textures" is not ticked, after a certain while there is no more space and no more characters can be added, which results in the "Character was not found" message. At least, I think this was my issue, because when I reset the font asset and ticked Multi-Atlas Textures, the issue resolved itself.

This is on 2019.4.21f1, TMPro 2.1.1.

A suggestion: Although the error message correctly says "Character... was not found in ...font asset", I took to reading this as "was not found in ...font" (because Dynamic), and was looking at length as to why the character was not part of that font. Certainly my fault, but perhaps there could be some additional error message that an atlas is full and nothing more can be added? Would've certainly helped me.

Thanks,

Philip

 

That is correct.

Changes to Dynamic Font Assets in the Editor are persistent unless you have "Clear Dynamic Data on Build" set in the Generation Settings of the font asset.

Clear Dynamic Data on Build is a new option added in subsequent releases of the TMP package where upon creating a build or closing the editor, dynamic font asset data is reset / cleared.

Dynamic Font Assets can be manually reset via the context menu Reset option in the Font Asset Inspector.

Resetting / clearing this data also results in the Atlas Texture of Dynamic Font Assets to be set to size zero which greatly reduced build size. The atlas texture of dynamic font assets is resized to their defined size in the Generation Settings when the first glyph is added to them.

Multi Atlas Texture results in additional atlas textures being created as needed when the first atlas texture is full. This does result in (1) additional draw call per atlas texture.

Having dynamic data persistent in the Editor along with Multi Atlas Texture enabled makes it possible for a dynamic font asset to be capable of handling and displaying every single glyph contained in the source font file while enabling you at anytime to look at the font asset to get a sense of the glyph coverage and settings to evaluate if they are optimum for your project.

Good suggestion. I'll take a look at revising the message to make it more informative.

 

Thanks for the added explanation, this makes sense.

(Re Error Message: Another idea would be a warning or something on the font asset itself which shows whether characters have been requested which did not fit in).

Thanks again!

Philip

 

exactly what I've been looking for! thanks mate!