TextMesh Pro Needs Unicode support

TextMesh Pro does support full UTF32 / Unicode. When you create your font assets, you have to include the characters that you wish to use. For Asian languages whose character sets are larger, you might also need to create a few fallback font assets.

See the Font Asset Creation video which covers how to include characters for any languages as well as important options like "Characters from File".

You should also watch the video about the Font Fallback as well as this one to allow combining symbols in the same object such as FontAwesome.

Lastly make sure that you are familiar with Material Presets as this is also an important part of using TMP and avoiding resource duplication.

Here is some additional information that I wrote previously about handling CJK and localization in general.

***************************​

In terms of how to handle the mapping of these characters, here are my suggestions.

I recommend creating a Primary SDF Font Asset which will contain all the known / used Chinese characters in the project. By known I mean those used in your menu and text components but not those who might potentially come from user input. This will result in an SDF Font Asset which most likely contains less than 1000 characters. (P.S. I would actually love to know what you end up with in terms of character count.) To input the list of Characters, I always use "Characters from File" since the text used in your project should already exist in some text file (encoded as Unicode).

Next, I would create 3 additional Fallback font assets which would contain the remaining 8105 characters not already present in your Primary Font Asset. The first Fallback would contain the first 3500 from the Table of General Standard Chinese Characters minus those already in your Primary. The 2nd would contain the 3000 minus again those in the Primary and lastly the third would contain then 1805 minus those in the Primary.

This will give you (1) Primary SDF Font Asset to which you will assign these (3) Fallback SDF Font Assets.

When creating these Fallback Font Assets, the sampling point size and padding and texture resolution do not need to be the same.

In order for the visual appearance of things like Outline, Shadow, etc to be consistent, you have to maintain the same Ratio of Sampling Point Size to Padding. So if the Primary is using a Sampling Point Size of 120 with padding of 10. Then you Fallbacks could be using a Sampling of 60 with padding of 5. You can control the sampling point size by instead of using Auto Sizing on Point Size, you set a value manually.

So for all the known text, I usually maximize the sampling point size since I know these characters are contained in my project and I want them to always look great. However, for the Fallbacks where only a few might be used in the context of user input which might not be visible on screen much, I use lower quality settings which allows me to save on texture size / resources.​

 

This apply to all platforms.

Freetype which Unity and TMP use to raster glyphs, does not provide access to the GPOS and GSUB tables which contain "Font Features" which includes among other things ligatures, diacritical marks, glyph substitutions, kerning, etc.

Although most languages use some font features like kerning and some other stuff, languages like Arabic, Thai, Bengali, etc rely heavily on these features. Asian languages like most Latin languages don't rely on that as much. Regardless, support for Font Features is planned for the Integrated version of TMP.

TMP currently supports the full range of Unicode. UTF16 characters can be accessed with \u03A9 (2 hex paris) while UTF32 is \U0001F600 (4 hex pairs).

Strings in C# are 16 bit so you also have to use \u or \U or surrogate pairs to access UTF32 characters.

Although we can access the full Unicode range in strings or editor input field using the information above, the Text Input Field relies on the Event Class in Unity which is required to process keyboard input. Currently and depending on the platform, UTF32 input doesn't always work. Some additional work will be required to update classes like the Event Class to make sure we get the correct Unicode input on all platforms.

 

I am sure it will get added at some point as besides supporting current TMP users, this is my primary focus.

 

This is something that the author(s) of UPersian could certainly explore.

 

TMP has basic RTL support but does not currently support glyph re-ordering which UPersian does.

It looks like UPersian could be used like the old Arabic asset as described on the TMP user forum. See the following thread / post.

Native support for glyph re-ordering as well as OpenType font features is planned for the new text system that will eventually replace TextMesh Pro.

 

Can you provide me with a link to this font file?

 

Thank you for providing the source font file.

Here are the settings that I used to create a font asset that contains these glyphs.



This is the hex character sequence that I entered

A4,2E85,2E89,2E8D,2E96,2E98,2EA1,2EA3,2EA8,2EAD,2EB9,2EBE,2EC2,2ECF,4491,4EBC,5315,20086,20087,20089,201A2,27607

This font only contains 25 glyphs as you can see in the image below.

 

When trying to reference a UTF16 or UTF32 character in the text, you need to use \uFFFF for UTF16 and \UFF00FF00 for UTF32 which is the standard conversion for how to reference those in strings in C#.

BTW: This information is covered in the Font Asset Creation video.

 

I just downloaded this Lobster font from Google and it does contain those Cyrillic characters as seen in the image below.



Maybe the font you downloaded has some issue. See if you get the same results from the version available from Google (just in case you got it from somewhere else).

In my example, I used an SDF mode but I also testing with Hinted Smooth and also got the correct results.

 

Liberation Sans is the only font included with TMP that includes Cyrillic characters.

In terms of determining what glyphs are present in a font file, the Font Asset Creator can be relied upon as it is accurate. In the current version, it doesn't report if a glyph was not included because it is missing or could not be packed in the Atlas but in the image of the updated version I posted, you will now be able to tell which are missing and which were not packed because they simply didn't fit in the atlas.

BTW: Google fonts is a very reliable place to get fonts. They also clearly list the licensing terms which is so important.

 

The khmer language makes use diacritical marks which isn't fully support (yet) in TextMesh Pro. So although it is possible to include all the characters and diacritical marks in the font asset by using the Font Asset Creator, the marks will not get positioned correctly.

It appears the Khmer language uses mostly single diacritical marks and although the Font Asset Creator won't extract the positional data to properly position these marks, it should be possible to manually create Glyph Adjustment Pairs in the Font Asset inspector to properly position them.

Take a look / play around with this feature which should allow you to setup this marks for the various character / mark combinations.

 

The new preview / release of the TextMesh Pro package version 1.4.0-preview.1b for Unity 2018.3 is now available.

This new release includes the new Dynamic SDF system which will improve the workflow for handling localization. Please see the top sticky post and video about this new release / topic.

 

Most likely the issue is related to OpenType font features like Diacritical Marks, Ligatures, etc. not being support (yet) by TMP.

For me to confirm this, please provide an example of the text you are using and images showing the current results vs. expected results.

 

Please see the following updated videos.