Search Unity

  1. Click here to receive a gift with your purchase of Unity Pro or Unity Enterprise.
    Dismiss Notice
  2. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

TextMesh Pro Korean, Japanese and Chinese Localization issues

Discussion in 'UGUI & TextMesh Pro' started by mrCharli3, Apr 18, 2019.

  1. mrCharli3

    mrCharli3

    Joined:
    Mar 22, 2017
    Posts:
    628
    Hi,

    I have downloaded fonts that support the Korean language, I apply the font to my TMP Text and google the translation for, lets say, "hello".

    English: "Hello"
    Korean: "여보세요"

    However, if I paste the Korean into my TMP Text, the signs are just squares.
    If I try to type normally on my keyboard, Korean signs show up, but they stack on top of each other in the Textfield.

    How do I solve this?

    EDIT: Works fine using normal Unity Text, so something directly related to TMP
     
    Last edited: Apr 18, 2019
  2. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    You need to either create a font asset and include these characters specifically or create a font asset the is set to Dynamic mode which is a new feature available in version 1.4.0 of TextMesh Pro for Unity 2018.3 or 2.0.0 for Unity 2019.1

    See the following post about the Dynamic SDF system as well as this additional video and preview of the fallback and dynamic system.
     
  3. mrCharli3

    mrCharli3

    Joined:
    Mar 22, 2017
    Posts:
    628
    I have plenty of font assets, I've used TMP for a while now. But the asset I generate with Korean characters does not work.I am copy+pasting this into my TMP Text with this font:

    여보세요

    That just turns into squares. However if I type manually on my keyboard the signs show up, but they stack on top of each other.

    The actual .tff work fine in Normal UI Text, just the TMP Asset that does not work properly in TMP Text.

     
    Last edited: Apr 18, 2019
  4. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    Can you provide a link to the font file?

    Most likely the font file is missing those characters in which case UI Text would use some other system font to replace those missing characters.
     
  5. mrCharli3

    mrCharli3

    Joined:
    Mar 22, 2017
    Posts:
    628
    https://fonts.google.com/specimen/Sunflower

    However, I do see that "characters packed" is "100/306", how do I make sure all chars get packed?
     
  6. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    Thank you for providing a reference to the source font file.

    As per the image below, the source font file does appear to contain the following characters "여보세요"

    upload_2019-4-18_14-8-11.png

    However, in your original image, you have the Extended ASCII set selected and as such these Korean characters would be excluded as seen below.

    upload_2019-4-18_14-8-57.png

    In the latest version of TMP which is version 1.4.0 for Unity 2018.3 or 2.0.0 for Unity 2019.1 which now include the new Dynamic SDF system, the Font Asset Creator now provide more informative feedback which includes the list of Characters Included, Missing and Excluded because they could not fit into the given atlas size based on the settings provided.

    My recommendations for handling this, would be to use the latest version of TMP if you are using Unity 2018.3 with the new Dynamic SDF system and to created a static font asset that contains Extended ASCII and then a single fallback that is dynamic which will be assigned to the primary static font asset. This will result in the Korean characters and glyph being added to this dynamic fallback.

    For example, the LiberationSans font asset included in the updated TMP Essential Resources with version 1.4.0 of TMP use this setup where the LiberationSans is static but also has a dynamic fallback assigned to it.
     
    Mauri likes this.
  7. mrCharli3

    mrCharli3

    Joined:
    Mar 22, 2017
    Posts:
    628
    Thanks,

    Is there any way to do this that does not require 2018.3?
    Upgrading from 2018.2 is not an option, broke my project and just spent an entire day restoring it.

    EDIT: I watched this video


    According to this if I set to auto-size it should just generate an asset with all signs in the provided font, but it's not? I'm only getting the "standard" alphabetic characters and numbers, no Korean signs. But you said the font I linked does have those characters. So I'm confused. In your image it looks like uve generated an asset that contains the signs.
     
    Last edited: Apr 19, 2019
  8. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    You need to include the unicode ranges for those other languages. You would define those custom ranges using the Unicode Hex option or you could use a text file that contains all the know text as well.

    P.S. Since the new Dynamic SDF system is very powerful and would be very useful / make your life easier here, what part of your project broken when trying to upgrade to 2018.3?
     
  9. mrCharli3

    mrCharli3

    Joined:
    Mar 22, 2017
    Posts:
    628
    Everything broke, literally could not even open a scene. So I am not going thru that again since I'm almost done with my game. Do you know any good link to a place that has these type of text files or hex-codes stored?

    So what am I missing here, because it is not working:

     
  10. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    What specifically broke?

    Going from 2018.2 to 2018.3 should be pretty smooth. The only issue you would run into with version 1.4.0 of TMP is the need to switch the Scripting Runtime API to .Net 4.x (which is simply to do and will eventually be required as the old .Net 3.5 has been deprecated anyway) but if that was the case, I already reverted that change for version 1.4.1-preview.1 which is available via the package manager.

    In terms of a good place to get information about the Unicode ranges used for any given language, Unicode.Org is the best place.

    For handling localization, the new Dynamic SDF system makes this process so much easier and if the issue with the upgrade was due to something else, the please submit a bug report with the included project as simply upgrading from 2018.2 to 2018.3 should not blow up your project and is something we should look at. If submitting a bug report is too much trouble I certainly understand, I am simply offering to take a look at this to help you out and invest the time myself to figure out why you ran into such issues. The new Dynamic system is very powerful so just trying to also make your life easier in that respect as well.
     
  11. mrCharli3

    mrCharli3

    Joined:
    Mar 22, 2017
    Posts:
    628
    I appriciate your help!

    Nothing that has to do with TMP broke, every prefab I had was "missing", and honestly do not have the time to deal with that for a project that is 95% done. TMP is really the only reason I wanted to upgrade anyway.

    Looking at my screenshot from my last message, can you see if Im missing something or should that work? Im perfectly happy skipping the dynamic part since I have like a total of 20 rows of text in my game. Spending so much time on something that is 0.1% of my game.

    EDIT:

    Solved by using another font, no idea how you made it work with that one.
     
    Last edited: Apr 20, 2019
  12. Giantbean

    Giantbean

    Joined:
    Dec 13, 2012
    Posts:
    86
    I know I am late to this and surely you have solved the original issue but I wanted to point out a link that may help with future projects moving from an 2018.2 to 2018.3 The broken prefabs are likely caused by a compile error stopping a build. If you fix all compiling errors and re-import the prefabs they should work. If you have no compiling errors I would say delete your manifest.json before updating the scene so it can rebuild with the new settings as needed. And of course always back up your scene before updating to a new version.

    Back to the original question of importing oriental fonts I too am confused and have questions about how this is done.

    I am trying to add Japanese fonts which is hard as there is Hiragana ( 3040 - 309f), Katakana ( 30a0 - 30ff), Kanji (
    Unicode code points regex: [\x3400-\x4DB5\x4E00-\x9FCB\xF900-\xFA6A] Unicode block property regex: \p{Han} ...Or at least some of these.) and KJC unified ideograms ( 4e00 - 9faf ...Likely not all of these are needed but I am unsure which ones I will need ) as well as Japanese style punctuation ( 3000 - 303f). Some text blocks would also need full width roman characters ( ff00 - ffef)

    As you can see that is a lot of Unicode to try and make a character set out of. Now I could take all the translated text I have and put it into one block of text removing any duplicates and then create a font atlas based on that (I have no idea how I would then implement this atlas. Do I past all the characters into TMP Font Asset Creator>Custom Character List? and if so what do I put in the Select Font Asset attribute as my only options are LiberationsSans SDF and Fallback. Whats more how do I add more characters if I need them later? Once made do I still past the kanji into the textmesh pro text field as I cant actually type it ) It seems all of these questions should be resolved already as I am using unity 2019.3.8 with TMP 2.0.1 so I should be able to use the Fallback LiberationsSans SDF Atlas to generate whats needed as seen here
    . Trouble is this is not working.

    I put in " 前方フェアリング " and get back " _...☐ "

    Any ideas or guidance would be appreciated.
    Thank you.
     
  13. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    This is likely due to your font asset not containing the requested glyphs. What font file are you using?

    Please make sure you are using version 2.1.0-preview.8 of the TMP package for Unity 2019.3.

    Make sure that you are using a font file that contains those Japanese characters.

    Make sure you also create a dynamic font asset using the context menu "Create - TextMeshPro - Font Asset" or by selecting the font file and using "CTRL + SHIFT + F12".

    Depending on the stage of your project, at some point it will likely makes sense to switch some font assets from dynamic to static but while you are developing, just using dynamic font assets with multi atlas texture enabled as well. This is enabled in the Font Asset Inspector - Generation Settings. This is all new functionality that did not exist when the video was created.
     
  14. Giantbean

    Giantbean

    Joined:
    Dec 13, 2012
    Posts:
    86
    Thank you very much. The "CTRL + SHIFT + F12" was very help full in turning KozGoPr6N-Regular into an atlas material the font asset creator then makes roman characters as mentioned earlier and a duplicate font set to dynamic was then able to add the characters as shown in the video link. Thank you so much for your help!

    Greatly appreciated.
     
    Stephan_B likes this.
  15. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    My solution was I reverted back to regular unity text for all my UI, and I used the "Outline" component to add an outline to my text so it showed up against the background. The outline was the only real reason I was using Text Mesh Pro over Unity text, and the regular unity text is working with Chinese, Japanese, Korean, Thai and more by default. It may depend what TMP effects you need, but if you just need an outline then regular Unity text with the outline component works really well:)
     
  16. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    Creating and using Dynamic Font Assets and / or combination of Static and Dynamic font assets should ensure everything works as expected provide the font file you select does contain the CJK characters you are trying to display. You can also use the Multi Atlas Texture support feature which will ensure the dynamic font asset can grow as needed to essentially support displaying every single character in that font file which is something the Legacy text system cannot do. No one would ever need to display every single character but if you wanted to, you could.

    The Outline component increases the geometry count of the text by 5 X and will impact performance.

    P.S. I really need to update these videos to cover all the new features and much simplified workflow which is

    (1) Import the CJK font file in Unity.
    (2) Select the font file and press CTRL-SHIFT-F12 and voila, you now have a new dynamic font asset.
    (3) Optional - Enable multi atlas texture in the font asset inspector and you can now add every single character in that font file and get all the goodies from SDF like Outlines, Shadows, etc.
     
    Last edited: Apr 16, 2020
    protopop likes this.
  17. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    Thanks:) I could try this out.

    You can switch languages dynamically in my game so I need a font file that would handle all of them. With regular unity text no matter which font I choose all the languages work by default. I’m guessing for Chinese, Japanese and Korean characters it falls back to some internal font or something because the CJK characters look different than the font I use in that text field.

    Do you know maybe a good free google font or font i could test text mesh pro out with to see if it’d work with a dynamic language switch? I’m in the process of converting all my text back to regular but if I can find a font that works with TMP I can try that out before I go any further

    77B096D3-8B87-4C9F-93FE-A833DB4A6196.jpeg
     
  18. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    I have another question.

    How is unity regular text able to display characters from every language?

    Is it that every major platform - iOS, Android, Mac and pc - contain Asian characters in their system fonts and unity text switches to these when an Asian character is displayed? (So when Unity displays an Asian character not included in a font file it’s sctuslly not using the font itself in that case?)

    Is this how unity is able to display so many Asian characters without using a huge font file?
     
  19. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    The Noto font family from Google does support every single language. There are many different versions of the font that can be chosen based on needs. Here is a link to this Noto font family. For CJK, I use the Noto CJK font which has good glyph coverage.

    The legacy text system ends up using "some" system font available on the given platform. Most platform include a set of fonts with coverage for CJK but this varies greatly between platforms in terms of the fonts themselves and this font selection can also change overtime. Font selection can also vary between devices; mostly on Android.

    The ability to use system fonts is nice to save on build size but this also means your application ends up looking different on each platform which for some developer / designer is a big "No No!". In my opinion, unless you have build size limitations, you should be using your own fonts which gives you ultimate control over the visual design, quality and QA.

    In terms of build size, most developer get around this by using AssetBundles, Addressables, etc. where based on language selection, different payloads will be download.

    Note that when it comes to localization, most people initially think about text and fonts but localization most often involves other resources like materials, images, animation, audio, colors, scenes, scripts, etc. Besides using different resources based on language selection, object settings and properties may also need to be modified which is why many studio use some type of localization tool to track how language selection affects these resources, their settings and to ultimately manage this process. Fonts are just a part of this overall localization process.
     
    Last edited: Apr 16, 2020
    celikomerdev and protopop like this.
  20. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    Thanks - I was thinking the system fonts worked something like this, and now I know:)

    Ill give the Noto a try with Text Mesh Pro.

    Im a solo developer creating open worlds, so minimizing pipeline steps is very important. The only localization needed in my games is in the text, and I want to maintain a single executable so as long as I can get either TMP to work with my csv file in all languages or use the default unity text I should be ok. But its good to know about what AA Indies and AAA developers setup when working with localization so I have a better idea of what can be involved. Thank you.
     
  21. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    By the way, is it possible To just set Text Mesh Pro to fallback to system fonts whenever an unavailable language is used, the same way Unity regular text does?
     
  22. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    It is currently possible to create font assets from system fonts but the "automatic fallback to a system font" is not currently available but most certainly something that I want to provide in some form but...

    I am not a fan of the "Fire & Forget" approach where the system magically falls back to some system font as this hides from the developer what is actually going on behind the scene. Ie. that whatever font you selected is missing some characters which you might only discover after shipping your product.

    You also have the case where in your dev environment (Windows for example) whatever font ends up being used works fine from a visual and metric point of view with the primary font you selected but on some platform because some other system font is used, you end up with some other font that does work well with your primary. Again something you uncover after shipping. Ie. something you would have instantly notice in the Editor when the missing glyph character is displayed where you would be able to do something about it.

    So the challenge is ...do I do magic stuff in the background that hides the fact your primary font is missing coverage for some languages which I agree is convenient but could be very costly to uncover after shipping.
     
    protopop likes this.
  23. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    I guess you could do both. Like in my case fallback would be great - simple and easy to vet because I go through all my screens in my built game anyways on my iOS devices. But I get what you mean about offering a more complex but safer version. I guess the real issue is there's no standard yet so we have to compromise somewhere - complexity for safety or simplicity for unknowns. Maybe one day Earth will have a unified language we can all speak - that'll probably make game dev, and all development - a lot easier;)
     
  24. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    I installed NotoCondensedMedium. Im getting white boxes instead of Chinese characters, but I am sure I am missing a step. European and Russian text shows up fine with this

    This is a screenshot of what I did. Do I need to do something special? Like check something to allow for Asian characters?

    Screen Shot 2020-04-16 at 4.13.55 PM.png Screen Shot 2020-04-16 at 4.08.53 PM.png
     
  25. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    The reason why the glyphs don't show up is simply because this NotoSerif-Condensed font does not contain the CJK characters.

    The following NotoSerifSC-Regular for Chinese Simplified (SC) should work. There is another version for Chinese Traditional (TC). Here is a link to the combined NotoSerifCJK.

    In the end, it comes down to picking the right font with the appropriate glyph coverage for the given language.

    Should developers have to worry and learn about this?

    Let me answer with another question.

    Should developers have to worry and learn about Garbage Collection?

    As game developers, mostly indies, we have to learn about all of these topics. Like efficient modeling, light baking, efficient coding, etc. We don't need to becomes masters but we certainly need to learn enough to become proficient.

    P.S. I use tools like High-Logic Font Creator or similar tools to look at font files to see exactly what characters and features they contain.

    Often users rely on some font viewer which most of the time does glyph substitution leading them to believe they contain character they actually do not contain.
     
    Last edited: Apr 16, 2020
    protopop likes this.
  26. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    There's a lot of great advice here - thanks!

    I used the NotoSerifCJK - thank you for the link:)

    Im not seeing Chinese characters in my TMP text. Is there something in my screenshot that's incorrect?

    I think maybe its the Character sequence or something? I feel like im just missing a step or something.

    Screen Shot 2020-04-17 at 12.13.03 AM.png

    Screen Shot 2020-04-17 at 12.12.56 AM.png

    BTW this is the game I'll be using it in. It's out on iOS and there are players around the world, and many of them are asking for translations. Players have been really nice helping me to translate words.

     
  27. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    How are you creating the Font Asset?

    Looks like you are still using the Font Asset Creator which is fine to create static font assets when you know exactly what characters you need. However, until you have a list of all the known characters used in the project, broken out per language or groups of language, just create and use a dynamic primary font asset created from this NotoSerifCJK as per the following steps.

    (1) Import the NotoSerifCJK font file in the project. You have already done this step.
    (2) Select the font file and press CTRL-SHIFT-F12. The new dynamic font asset should be created.
    (3) Select the newly created dynamic font asset and enable Multi Atlas Texture in the Generation Settings
    (4) Assign this font asset to the relevant text objects.

    Then start typing away and all the characters should show up.

    In the Editor while you are developing the project, all changes made to this font asset will be persistent. This font asset will grow as needed and might end up using multiple atlas textures which is fine while you are developing the project. As you get closer to release and have a list of all the known characters used in the project broken out per language or language groups, you will be regenerating / creating new font assets to now use a combination of static and dynamic font assets but while you are developing, just run in full dynamic mode.

    This is a short summary of the new information and revised workflow that I will be covering in the update Font Asset Creation video.
     
    protopop likes this.
  28. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    Here is a video that I created tonight to help provide better insight on the font asset creation process and workflow.



    P.S. I am still working on a more polished video. This is the quick / rough version to help you out.
     
    protopop likes this.
  29. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    Screen Shot 2020-04-17 at 3.43.01 PM.png

    It worked:)

    I can't believe you made an entire 20 minute on the fly video like that. It was very clear and helpful, and Im super impressed at how you covered everything - you have a mind like a steel trap Stephan.

    Whats nice is that my game loads translations from a csv file at runtime. It looks like all I have to do is run the game, change the language and open up each screen with text once to fill up the atlas. It works really well.

    I think especially if you have a known set of text and high quality assurance goals this is the way to go.

    For my case though I am continually adding in new languages - I added Thai last night - and I think its too much complexity for me as an individual to find a font that covers Thai and more as well. The regular unity text is handling all the languages and I may have more coming. I think for the future if TMP could fall back if the font isn't included that would solve my particular case issue. But I have to say the TMP is very tempting because of the simple ways to style it and the low draw calls.

    I think for now I need to stick to Unity UI text because of my specific case but I do recommend people watch your tutorial because now I feel like I understand very clearly how dynamic Text mesh pro works whereas before I was very unclear. You really did an amazing job with that and I can see you're as passionate about helping us out as we are about making games. So thank you very much.

    One last question, for the outline component, when you say its bad for performance, is that only because it creates more vertices/geometry? or is there also some kind of scripting/update performance hit happening every frame. Im asking because I can afford to have it on the text in each screen, theres; only 6 to 20 texts on each screen and only one screen canvas is enabled at a time. But if there is some kind of update that is continually making the outline component hamper performance even when the canvas is disabled then I need to think about that.
     
  30. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    Forgot something very important... The legacy text system and UI Text does not support proper line breaking for Chinese, Japanese and Korean text which TMP does.

    Again, do not be fooled into believing UI Text will be able to magically handle this in the background. Its ability to display Thai is related to Windows as an operating system including over 200 fonts where at least one of them includes support for Thai. This may not be the case on your target platforms / devices. Then you have to understand that the Thai font used when testing on Windows is likely to be different per platform and devices. The font that the legacy text system ends up picking is somewhat random in the sense that it is likely to pick the first font it encounter (alphabetically that contains the character you seek).

    Then you have the limitation of the legacy system using a single atlas texture where the atlas will need to contain a copy of each character and for each point size. This quickly fills up the atlas texture where in no time it ends up being 4096 x 4096. This issue is strictly related to the amount of text and point sizes being used. If all the text is the same point size and the number of characters displayed at any given time is small, then this will not be an issue.

    In terms of adding support for new languages each day. It comes down to picking a font for the given language of the day. In the case of Thai, this task is trivial given the NotoSans and NotoSerif font family includes font for pretty much all languages, the process consists of (1) Downloading the Thai NotoSerif font (2) Unzipping the NotoSerifThai-Regular.ttf font in your project. (3) Create a dynamic font asset using the new font file by simply selecting it and using CTRL-SHIFT-F12. (4) Lastly assign it as a fallback to your Master / Primary NotoSerif font asset that contains all the Latin characters.

    On the next day you would repeat the process for the new language of the day. After a bunch of days, the Fallback list of your Master / Primary NotoSerif font asset would contain a reference to each of those language specific dynamic font assets.

    Using the above approach enables you to continue development and adding support for any given language. It allows you in the Editor to make sure all of your text is displayed correctly and guarantees everything will look the same on all platforms and devices since you picked and control the entire process. During development, you only use dynamic font assets which you will later switch to a combination of static with dynamic fallbacks. (Will create a new video to cover this specifically)

    The underlying geometry will be 5 times larger which impacts rendering with the addition of extra processing as every time the text changes, this Outline component needs to make 4 copies and offset those. In the end, expect about 5X performance difference between UI Text with Outline component vs. SDF text with an Outline. Adding the shadow component will further impact all of this whereas adding a Shadow with SDF doesn't change anything.

    The main drawback with the above is having to include these font files in the project which will increase build size. There are ways to get around this by using AssetBundles or Addressables. Font assets can also be created at runtime using system fonts but I still have work to do on that before I am satisfied with it.

    ... now working on the video.
     
    Last edited: Apr 17, 2020
    protopop likes this.
  31. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    You make a really good case for Text mesh pro:) I'm really torn.

    Im going to try it again with the Thai fallback you mentioned. If the created font assets aren't too large mb wise and it works maybe I will stick with TMP because it looks nice and as you said it makes sure that everything will display as is. Thanks so much for all the information about this. I am sure it can't be easy but hopefully this will help more people.

    In my project for now I'm going to keep both Unity UI text and TextMesh Pro text until I figure this out.

    You were very helpful explaining the system fonts: I checked to make sure and both Android and IOS support the system fonts im using with UI text including Thai. For anyone targeting modern mobile id say it is pretty safe, plus I think for example most people who speak Thai will have Thai as a system font installed on their device. All my text is static so there's no typing anywhere and its all the same point size so that's ok, and I don't use any shadows on my text, just the outline.

    One other issue I noticed though is with Unity UI text I can use any font + the system fallbacks so I have more control. With the TMP I think this impossible too? I guess I have to use my non-asian font as before, then create fallbacks just for cjk and Thai fonts in the Noto fonts (or any cjk/thai font). Like basically we can have as many fallbacks as needed, memory permitting.

    Small note: Im on Mac and it's CONTROL+SHIFT+FN+12 to generate the dynamic font asset.

    PS that's cool about the TMP line breaking
     
  32. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    I will have the new video available in the next 30 minutes which will cover what I described above.
     
    protopop likes this.
  33. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    I think I get what you were talking about before

    If I want to use
    • Itim.ttf gogle font for all European languages
    • Noto for everything else
    I create a static font asset for Itim
    I create 2 fallback dynamic assets for CJK and Thai, using the only cjk (no euro) and Thai (no euro)
    I fill up the dynamic CJK/Thai assets with my game text by playing it at runtime and opening each screen with that languages text (tried this, works)

    Then when I run it, it will display Itim for euro and fallbacks for the others.

    Ill try it:)
     
  34. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    During development, it can be dynamic to keep things simple but before releasing your project you would want to regenerate it as static where it would include all the Latin characters used in the project.

    Correct. These additional dynamic font assets, should be assigned as fallback to your Primary / Master font asset created from "Itim.ttf".

    Then when you need to add support for an additional language you repeat the process you did for you CJK and Thai.
     
    protopop likes this.
  35. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    Here is the other video ...

     
    protopop likes this.
  36. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    Thank you - im watching it now:)

    Is this the right font? Its something called a .ttc font and its 24 mb. Is this the correct files size (it seems really big). The Thai font is only like 500kb. Is it just because CJK has so many letters in the font?

    Screen Shot 2020-04-18 at 7.56.50 PM.png
     
  37. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    Wow. Stephan your tutorial was fascinating and really well spoken. Do you create all these without writing down what you're going to say first? I don't know how you did this and these seem like finished polished tutorial to me, not on the fly temporary ones. And when you are writing in character codes for omegas by memory - ! You really know fonts.

    Text Mesh Pro with fallback languages is definitely more complex to learn. But once you watch this it all becomes really clear. I have it working in CJK and Thai and im glad you took the time to explain local and global fallbacks and search order. I think it was important for me us to know that because it opens up a lot of possibilities.

    I have some questions:

    To fill up my dynamic atlas, Instead of running my game, switching toeach the 4 asian langauges and opening up each option screen, can i just export my 4 language texts from my spread sheet to a text file and import that into my atlas? Like feed unity a text file with all my CJK and Thai letters and use that. It would save time

    The outline on my CJK and Thai letters is thicker than the European text. Im guessing this is font specific. Can I change the outline to be thinner on just my CJK and Thai characters? Or do I have to experiment with differ fonts? Like maybe if I use a CJK light font instead the outline will more closely match my Itim font outlines.

    Screen Shot 2020-04-18 at 8.32.36 PM.png

    Will it save memory to remove default sprite asset from tmp settings if i dont use emojis? Do you think its worth it?

    Do you think is it better /more efficient to use separate Chinese, Korean and Japanese fonts as fallbacks? instead of one big CJK font that contains all 3?

    Should I leave my European item main font as static? And after I fill up my fallback dynamic atlases, can I switch the dropdown too static before I build it. or just leave it dynamic?

    Small maybe bug - when i added the CJK fallback the direction switched right to left. I found and unchecked the RTL checkmark and fixed this, but was unsure if something was going on here. Looks ok though.

    can i check the font SDF atlas size without building the game and looking at the build log? I think my only remaining concern is that the atlas sizes will be too heavy memory wise and crash my mobile game.

    Can i replace liberation sans from The TMP SETTINGS asset safely? Willl this break anything in the editor? Like does Unity editor UI use Liberaton or something?

    why not use static for my CJK and Thai fonts? because it's too heavy memory wise? Like would stain mean it imports ALL the characters even if I don't use some?

    Does the font sdf atlas size affect performance and memory or just build size?

    Im going to test this out and check how the memory performs but if it all checks out ill stick with Text Mesh Pro:)

    I think you did absolutely great work here.
     
  38. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    One of the options available in the Font Asset Creator to define what characters will be included in a font asset is the Characters from File as seen below.

    upload_2020-4-18_19-25-24.png

    This option allows you to specific a .txt file that contains the characters you wish to include in this static font asset. This is the option that many studio use to create their static font asset as they get further into the development cycle and being switching from running full dynamic to a combination of static font assets that contain all the known characters for a given language or groups of languages combined with dynamic font assets to handle user input for those same languages or groups.

    So until you get further along (as well) in the development process, I would continue to run full dynamic with multi atlas texture enabled and would not try to fill those dynamic font assets as they will fill up gradually as you continue to work on the project.

    Now if you really wanted to fill them up with characters, you could (1) use the Font Asset Creator with the "Characters from File" option and manually specifying the Sampling Point Size and Padding to maintain a consistent Sampling Point to Padding Ratio. Font Assets created in the Font Asset Creator are static by default but this can be changed in the Font Asset Inspector Generation Settings. Keep in mind that if these are full, there is no point to switching them to dynamic.

    This is where the Sampling Point Size to Padding Ratio becomes important. In order for the Outline, Underlay and other visual treatment to remain consistent across your primary and fallbacks, they have to use the same Sampling Point Size to Padding Ratio. They can use different Sampling Point Size and padding but the ratio must be the same. For instance, if the primary is using 90 sampling point size with padding of 9 for a ratio of 10%, the fallbacks could be using a sampling point size of 60 with padding of 6 for the same 10%. This would ensure visual consistency while also saving texture space since the dynamic fallback is using a smaller point size. Most of the time since the characters contained in these dynamic font asset comes from user input which is usually smaller on screen, it is totally fine to reduce the sampling point size.

    The default sprite asset is pretty small but you can safely replace it or remove it.

    It all depends on your deployment strategy along with the number of known characters you will end with for those languages. For instance, if you are able to fit all the known characters for these 3 languages in the same atlas texture at good sampling point size / quality which is no larger than 2048 x 2048, then that is desirable. If you could fit each language in their own 1024 x 1024 combined with a dynamic fallback again at good sampling point size and quality that is even better. Keeping those separate might be easier to maintain. But this part is very subjective and sort of up to you.

    You can switch from static to dynamic as needed. However, if the atlas texture is full, it doesn't make much sense to switch to dynamic since the atlas is full anyway. However, as I said above, I would run full dynamic until it is time to make the switch and you have all languages and text / known characters nailed down.

    It is likely that you inadvertently clicked that RTL option as font assets on their own have no impact on this. But enabling RTL will reverse the text direction.

    The atlas size will remain exactly what you specific in the Generation Settings which by default is 1024 x 1024. Multi atlas texture will create additional one as needed but you can easily see this in the inspector.

    Make sure you don't use a sampling point size that is greater than 90 with padding of 9 as larger than this is rarely needed for Latin text. CJK characters can still look very good at 40 sampling point size with padding of 4 to maintain this 10% ratio.

    Yes. Just make sure you replace it and don't leave the field empty. A Default Font Asset is needed.
    You will at some point and for each language or group of language end up with a Static Primary font asset that contains all the known characters for that language or groups of languages. You will pair this static font asset with a dynamic font asset that is using the same Sampling Point Size to Padding ratio as fallback to handle user input for those languages. This will be the same for Latin or CJK. Static Primary coupled with Dynamic Fallback where at ship time, this fallback will be empty with its atlas at size 0 x 0.

    On the graphic size, it is more efficient to read from a smaller texture than larger texture. This varies greatly by platform / device. I would not use anything larger than 2048 x 2048. The smaller the better. What will determine this is ultimately the number of known characters for the given language or group.

    Thank you.

    P.S. I will probably create another video to cover the above.
     
    protopop likes this.
  39. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    Screen Shot 2020-04-20 at 5.51.14 PM.png Screen Shot 2020-04-20 at 5.49.47 PM.png

    OK thanks to your help I have almost finished using Text Mesh pro for my CJK and Thai translations - thank you:)

    Sometimes I notice square boxes as I open up screens - this is usually fixed by changing(lowering) the point size and padding. Im guessing its an artifact of using dynamic assets as I fill up the project.

    Ive exported a FallbackCJK.txt and a FallbackThai.txt files that contain all the CJK letters and Thai letters I use. I set up my translations in Google Sheets so its easy for me to add new text as needed and to export these sheets, as well as my master csv right into my unity project: https://bit.ly/3bnknkg

    Im still a bit unclear on how to import the Fallback.txt files I created. Right now I have my Static Itim font main TMP asset, with fallbacks to Dynamic CJK and Thai Noto san serif fonts (I switched from serif to sand serif because it think I looks cleaner o screen)

    Whats the process to import my text files into the CJK and Thai assets?

    Do I first switch them to static, because if I'm importing all the used characters I don't need dynamic, correct? And then to I open the FONT CREATOR and or other I guess I click UPDATE FONT ASSET at the top of my newly static font assets, and in the font creator panel import the link to my text file, regenerate and that's it? Then ill have a static font atlas with all my characters (and I can play around with the point size and padding keeping the 10% - thanks for that tip works great - in mind)
     
  40. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    This is likely related to Shader Compilation as a result of the following being enabled.

    upload_2020-4-20_16-55-23.png

    Disable Async Shader Compilation and these squares will disappear.

    When you need to update any of your static font assets, you should use the Font Asset Creator with the "Characters from File" option using you exported FallbackCJK.txt. Be sure to use the same ratio of Sampling Point Size to Padding.

    Repeat the process for Thai.

    You will create and assign an empty dynamic fallback your static CJK and Thai font assets to handle any potential characters you might have overlooked or that may come from user input,

    Assuming the static ltim is your primary / master, you will then assign the static CJK and static Thai to it as fallback.

    P.S. Like I mentioned a few times already, you don't need to worry about creating those static font assets until you are getting closer to release. During development you can run with each of those on dynamic mode with multi texture enabled. Then, at some point, you can look to convert to the static setup described above.
     
    Last edited: Apr 21, 2020
    protopop likes this.
  41. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    B80999B6-D125-473B-B6CD-B440BE56998C.png

    BB67EEA6-31E8-412B-9B99-F647C8693E98.png

    It’s working

    Polish is missing a few letters so I’ll try and track that down, they must be missing form the atlas but I think I can figure out what they are and inspect the font t see if they exist

    I’m all translated so I will try out the static fonts. So I guess I’ll have my itim main font asset, with two static fallbacks for cjk and Thai, and two additional dynamic empty fallbacks for cjk and Thai in case anything is missed

    The Russian seems to use the cjk note although I thought it would use the itim but I guess Cyrillic is not included. I’ll investigate

    I think you’re right about me accidentally clicking the rtl before

    I’m going to play around with formatting and I’m going to put the same itim in my text mesh pro settings asset to replace liberation because I don’t use that font and I can see it taking up lots of space in the build. Can I delete the resources font folder too with liberation in it (I won’t touch the shaders etc in the resources folder, just the liberation font) so it’s not included in the build. If I do that I guess I could just put itim and the four fallbacks in the tmp settings asset and use that in my tmp fields instead.

    So a bit more clean up but it’s working on my test devices and I’m really excited to share this next update with my player because half are world wide and they are asking for the game in their native languages, so that will be a treat.

    Thank you so much for explaining all of this because I was ready to drop my text mesh pro and this was a really interesting learning experience. I think for me the most important message that would have convinced me is to know that uo can add fallback fonts in tmp for Asian languages and it’s pretty much as simple as dragging created assets into a few slots ( I mean it’s more than that but that was the big part)l I can see across the forum all the hard work yo have done to see tmp through and answer peoples questions, and I think it’s one of the most helpful experiences I’ve had with someone from unity in a while, and I hope these videos and this thread help others.

    I might still have some questions but I’m gonna go clean this up and get it ready thank you Stephan
     
  42. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    If that font asset is static, it can't add to it. The easiest way to check if Cyrillic is in that font file is to quickly create a dynamic one (just for testing) assign it to some text object and start typing Cyrillic characters.

    Just move the LiberationSans font asset and its fallback outside the resources folder in some Fonts folder. This way, you will still have it for testing but it won't be included in builds.

    You are most welcome and thank you as well for providing feedback and all the replies which in the end help document a lot of this for other users :)

    Always feel free to post / reach out to me if you need assistance. I am here to help :)
     
  43. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    Its going really well. One issue though is a few missing letters in Polish

    śćŚ were the ones I found

    I tried typing it into several fonts in a tmp text field, including liberation, item, arial etc and they all brought ups squares. My csv file imports correctly and I can see when the game is running the śćŚ letters are in the component text box correctly, but square are in the ui tmp text field.

    I found this page: https://answers.unity.com/questions/801337/polish-letters-in-gui-saving-with-utf8-is-not-work.html that says to encode the file (my csv presumably) as utf16 instead of 8. I can't figure out how to export utf16 from google sheets, so I opened my csv in text edit and saved it as utf16 but it was the same problem.

    The rest is working in all my languages - I have my static item master, and the 2 dymamic cjk and Thai fallback atlases. Is there a way I can paste just the missing letters somewhere without messing things up

    EDIT: I noticed that different polish letters show or don't show depending on wether I have Small Caps enabled (I do on all my tmp components) or disabled.

    EDIT 2: I found this page https://www.quora.com/What-is-the-best-Google-Font-for-the-Polish-language. which says I might need to include something called Extended Latin Set - if this isn't available in Google Fonts Itim, maybe I can include it as a fall back from somewhere?
     
    Last edited: Apr 22, 2020
  44. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    OK I think I solved it

    I added a 3rd 512x512 dynamic fallback.

    I downloaded Noto Sans Display - the NOTO family seems to be the most comprehensive for European fonts

    I installed just the Noto Sans Display Condensed medium

    It seems to pick up the few missing characters

    Im going to fully test in on all devices. I can probably change this atlas size to like 128x128 or 256x256 since its just for a few characters, plus I want to save memory

    Basically the trick seems to be making sure you have enough fonts in your TMP stack to cover all you bases.

    I might also see about just replacing my Itim static master font with Noto and scrapping Noto as a fallback, depending on if the look of the Noto Sans is ok. That way id just be using 3 font atlases instead of 4 (although if I do shrink the Noto to 128x128 that should take up too much memory I'm guessing)

    Screen Shot 2020-04-22 at 4.47.26 PM.png
     
  45. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    OK im just posting my progress in case this helps anyone, because it's all kinds of interesting the world of fonts and atlases:)

    It seems like my TMP fields use my item font and just substitute the Noto sans fallbacks where there are missing letters. The difference is very small so I think it looks OK.

    Screen Shot 2020-04-22 at 4.55.45 PM.png

    I shrank my fallback Noto Sans im using for the missing Polish characters to 128x128 and they all fit. There was no need for a larger atlas so that saves memory

    Screen Shot 2020-04-22 at 4.55.53 PM.png

    I looked on the Google Fonts page for my main font and found out Itim Regular has a bunch of Thai glyphs already in it. So I looked at my Noto Thai fallback and its almost empty. Think I can also shrink this one to 128x128

    Screen Shot 2020-04-22 at 4.56.41 PM.png

    My CJK atlas is about 1mb and about half full with all the characters accounted for, so I might be able to half this with a slightly smaller point size/padding (10% ratio).

    Screen Shot 2020-04-22 at 4.56.16 PM.png

    This whole dynamic atlas thing is really neat as long as you pay attention. Im not sure what everyone did before this.
     
  46. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    You are making great progress and thank you for provide / documenting your efforts as this will indeed prove beneficial to other users.

    I will be preparing another video on a topic you are likely to run into shortly which is font metric normalization. See the following post to peek at what that is ;)

    Fortunately, this is something that is also easily addressed but nonetheless something that we eventually need to learn about. Like I said before "fonts come in all shapes and sizes" and that font designers do not always follow guidelines and rules.
     
    protopop likes this.
  47. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    I my build the CJK font is about 17mb

    17.9 mb 2.1% Assets/_Custom/Fonts/_Main/NotoSansCJK-Regular.ttc/NotoSansCJK-Regular.ttc

    Can I remove the .ttc file once the atlas is made if I make it static? Or do we always need the font file in the build, even if we won't be typing in any new letters (my game has no text inputs) - or maybe its best practice to leave it in just in case?

    Or do you think there is a smaller ttf file or something I can use?

    Im not sure but is the .ttc file a collection of different font wrights? Like is there maybe just a NOTO CJK regular.ttf or .otf I can use that's smaller?

    Anything to save some memory:)
     
  48. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    If the font asset is static, the font file will not be included in the build automatically. You do not need to take any action other then setting it to Static. If this font asset has a fallback that is Dynamic then the font file will be included because of the fallback.

    There are different and smaller font files out there. It is always a question of finding one with the visual design and glyph coverage for your needs.

    TTC files tend to be larger because they are essentially multiple font file within one.
     
    protopop likes this.
  49. protopop

    protopop

    Joined:
    May 19, 2009
    Posts:
    1,405
    I launched Wilderless 1.2 today with 14 languages using Text Mesh Pro:)



    Thanks for all your help. I was ready to give it up and you helped me turn it around.
     
    IndieFist and Stephan_B like this.
  50. Stephan_B

    Stephan_B

    Unity Technologies

    Joined:
    Feb 26, 2017
    Posts:
    5,085
    Congratulations and glad I could be of assistance :)
     
    IndieFist likes this.
unityunity