Search Unity

TextMesh Pro How to include basic unicode symbols like "♫"?

Discussion in 'UGUI & TextMesh Pro' started by MrLucid72, Jan 13, 2018.

  1. MrLucid72

    MrLucid72

    Joined:
    Jan 12, 2016
    Posts:
    983
    Let's say I want this symbol (\u266B). When I currently type it with my font, it just shows up as blank. If I paste the symbol itself, it's also blank. If I try any of the default fonts, it's blank.

    So, what do I do?
     
  2. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    The first step is to choose a source font file (.ttf, .otf or .ttc) which includes the glyphs or in this case musical notes you wish to use. One commonly used font is "Bravura" which uses the SIL Open Font License.

    After importing this source font file in Unity, you will need to create a Font Asset and select which characters you would like to include in this font asset. There are several ways to define / select the set of characters to add to a font asset. Please see the following video which covers the Font Asset creation process if you are not already familiar with it.

    Once you have created a font asset that includes the characters you need, you can access UTF16 characters by using \uFF00 (2 hex pairs) or \UFF00FF00 (4 hex pairs) for UTF32 characters.

    Since special font files that include symbols, emojis or musical notes usually map these characters in different unicode ranges, I recommend you get some type of font file viewer like High-Logic Font Creator or FontForge which will allow you to look at the content of the font file to know where these characters are mapped and to allow you to know what character ranges / sets to include in your font asset.
     
    umbrella-crash likes this.
  3. MrLucid72

    MrLucid72

    Joined:
    Jan 12, 2016
    Posts:
    983
    Whew, this is complex just to get some Unicode working: How come the standard Unity Text allows me to add such a music note so freely, but TMP does not? ♫
     
  4. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    The standard UI.Text uses dynamic fonts which at runtime will look if the character exists in the font file and if it does, it will fetch the character data and raster this glyph and then add it into the font atlas texture. Now this is certainly convenient but it does result in performance overhead which can be pretty significant especially when targeting mobile devices.

    So although having to create your font assets ahead of time, determining what characters to add to them, what sampling point size to use and padding, etc requires additional effort and time, it provides for the best possible performance and quality from the point of view of knowing exactly how your text and material presets will look on any devices.

    There are many aspects of game development that require we invest time to achieve the quality of products we seek which includes performance as well. We spent time writing good an efficient code. We spend time creating efficient geometry for our models. We bake lightmaps to get great visuals without the performance of real time lights. We bake navmeshs, etc... Although text may not be perceived as cool as this other stuff, it is still important and will be seen by all your users. So investing time to learn and do any of these things efficiently is important even for text. Plus do you really want to spend you CPU or GPU resources to be adding glyphs into font atlas textures at runtime or use those CPU or GPU resources to do cooler stuff like animations, AI, particles, etc.

    Now having said all of that .. the new Text System which will eventually replace TMP will include a hybrid dynamic font system where although it will be capable of dynamically populating font assets at runtime (which will still result in performance overhead), it will make it possible to populate these font assets while developing or testing your game in the Editor and to make these additions to the font assets persistent. This way by the time you are ready to release your game or project, the bulk if not all the characters you use in your game will have been already added into these font assets thus resulting in zero performance overhead at runtime. Then for those characters which you might have missed, the system will still be able to add those at runtime but given we will be talking about very few characters where most likely they will come from user input which is slow (humans don't type that fast) the performance overhead will be masked / hidden and not really noticeable by your users.
     
  5. dadude123

    dadude123

    Joined:
    Feb 26, 2014
    Posts:
    789
    I'm interested in that new "text system" as you call it.
    And I was wondering if you could tell us a little more about it.

    What features are planned for it? What are its goals?
    What are the features that are currently in development regarding that?
     
  6. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    I am have made a few posts about it already but some of the main major new features are support for OpenType Font Features which although a single line item in description is at the core of complex text layout and essential for proper handling of non Latin based languages. A hybrid dynamic font system as described in my previous post is another. There are of course several minor new features along with looking to bring performance and whatever other improvements I can.

    Lastly, while developing the new text system, I remain mindful of the transition from the current text system to the new one as to try to make that as smooth as possible without having to give up new functionality and features.
     
    dadude123 likes this.
  7. MrLucid72

    MrLucid72

    Joined:
    Jan 12, 2016
    Posts:
    983
    The correct answer was: "Use the backup fonts feature", I found somewhere online. o_O because text has such a minimal impact on desktop performance, this was the answer I was looking for -- unless you haven't finished your research, yet~ which I wouldn't blame you; TMP is loaded with features.

    Since Unity is taking over, and Unity is made to be simple, you may want to find the core Unicode and have a checkbox to enable x unicode, Chinese characters, Russian characters, etc in a supported options box. That would REALLY make things more simple and keep the performance than Googling the heck out of what unicode # ranges we should add, which makes my brain hurt ;)
     
  8. Brian_Q

    Brian_Q

    Joined:
    Nov 17, 2017
    Posts:
    6
    I´m looking for a font that supports a black square (U+25A0) . The fonts that come with Text Mesh Pro don´t seem to have them. I´ve generated Font Atlas for each of them and even included the extended ASCII variation. That Black Box was not included in the missing elements list, so I assumed it would be able to print it, but it doesn´t seem to have them. How can I find out if a font has the black square character and how can I access it for my project?
     
    MrLucid72 likes this.
  9. MrLucid72

    MrLucid72

    Joined:
    Jan 12, 2016
    Posts:
    983
    Im also wanting to know this. The tutorial from the original creator made it sound like we should intuitively know every unicode.... Code... Haha. But even if we know the code ranges, the font has to support this, right?

    Now that Unity owns it, shouldn't there be an easier way to handle supported Unicode or alt languages like Chinese? Super confusing :(
     
  10. Brian_Q

    Brian_Q

    Joined:
    Nov 17, 2017
    Posts:
    6
    I figured out a way. When you go to Font Asset Creator, choose Custom Characters under the Character Type drop down. There you can import an entire font series (so you at least have the basic characters) and then just copy and paste all the other characters you would want. When you create the Font it will have what you need.
     
  11. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    The quickest way to determine if a given glyph like U+25A0 is included in a particular font is to use some external font viewer like High-Logic FontCreator or BMFont. Once you know a particular glyph or sets of glyphs are contained within a given font, you can then process to importing it into Unity and use the TMP Font Asset Creator to create a font asset that includes it.

    Having said that, you can also do this easily from within the TMP Font Asset Creator which requires you import every font file you which to test to see if the glyph is present.

    Once a source font file has been imported into Unity, open the TMP Font Asset Creator and select this source font file. Then select either Character Set option "Unicode Range (Hex) as I have done in the image below. If the glyph shows up then you know it is included in this font file.

    upload_2018-4-16_11-46-51.png

    In the image above, I specified a single glyph which was "25A0". A range of glyphs can be specific using "25A0-25FF" which is the unicode range for Geometric Shapes.

    Groups of ranges can be specified by using "2500-257F,25A0-25FF". This would include the Drawing Box unicode group and the Geometric Shapes.

    The Unicode values for all characters and languages are listed http://unicode.org/charts/

    This process will get easier as I am planning on adding the ability to view glyphs contained in a font file directly in the Font Asset Creator. Then with the addition of the hybrid dynamic font system, you will be able to add these dynamically in your font assets.
     
    MrLucid72 likes this.
  12. MrLucid72

    MrLucid72

    Joined:
    Jan 12, 2016
    Posts:
    983
    Our hero!

    About the current process: This seems pretty straightforward if wanting to add like one Unicode character at a time, but what would we do if we want to add full traditional Chinese support, for example? No idea where to even begin.
     
  13. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    Characters / Glyphs for all languages are defined in the Unicode standards. Here is a link to the Unicode Chart which contains where everything is mapped in the Unicode.

    To add more than one character, you would select options like Unicode Range or Unicode Hex Range or Custom Characters or Characters from File where this text file contains all the characters you wish to include in the font asset.

    Characters from File is what most studios use as they already use text files that contain all the text for their project which is usually created and used in the process of localizing / translating the text in their project. They usually have text files for each language or groups of languages. As a result, they can easily use the Latin text file(s) to generate their font assets for Latin based languages. Then the Simplified and Traditional Chinese for their Primary and fallback font assets for Chinese, etc.
     
  14. samlabs_lea

    samlabs_lea

    Joined:
    May 1, 2018
    Posts:
    7
    If this approach is taken; how do you dynamically swap font assets based upon language?
     
  15. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    To manage language resources, many uses tool like I2 Localization. Others have their own solutions to manage / change font asset based on language selection.
     
  16. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    1,084
    Is this new Text System in place yet?
     
  17. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    That is not the new Text System but new version of the TMP package for Unity 2018.3 with Dynamic SDF support should be available within the next 7 to 10 days.
     
  18. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    1,084
    Does that interpolate what accented characters might look like or depends on the original ttf to have the accented character?
     
  19. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    Characters with accents come in two different forms. The first is individual glyphs for each of the accented characters. The 2nd is using Diacritical Marks (the accents) which are positioned appropriately for each glyph and diacritical mark combination. Diacritical Marks and their positioning is an OpenType feature which will not be supported in the next release but planned for a subsequent release of TMP.

    The new dynamic system is able to dynamically fetch all of these glyphs. However, the Diacritical mark / ability to extract that from the font file or ability to define then in the font asset is what is not available yet.
     
    ina likes this.
  20. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    1,084
    I'm still using the old TMP and for fonts that include the diacritical mark glyphs, it seems to work fine. Just not for the more interesting fonts that don't include the non-English characters...

    Looking forward to when the diacritical marks would be added!
     
  21. Cleverlie

    Cleverlie

    Joined:
    Dec 23, 2013
    Posts:
    219
    Hi @Stephan_B here is an idea, I don't know if it would be too complex to implement, but it would be nice to have something like the <sprite=blabla> tag, but something like:

    "this is an icon <icon="fas fa-address-card">"

    something similar to what you do in web development, instead of using unicode numbers which may be cryptic, we could have a table somewhere or dictionary which translates "fas fa-address-card" to "\uf2bb"

    this should be just a shorthand to writing

    "this is an icon \uf2bb"

    and nothing else, I'm already using FontAwesome as a fallback font asset to my main font (roboto for example), so I can insert icons in the middle of any text, but sometimes the icons you want to use don't come from a creator that provides a cheatsheet with the unicode values, or even worse, they don't even come in a font file, so you have to take the SVG files and create a custom font, and assign arbitrary unicode values to each icon, inside a range that you know doesn't intersect with the range of other popular fonts.

    what do you think? is this something that adds value to tmp? or is it even possible to implement?

    so far TMP is the only way I have to insert resolution independent icons in my Unity-made apps, the SVG importer is still too inmature and quirky, and doesn't come with all the nice features TMP has.
     
  22. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    Characters and Sprites can be access via the C# Escape Characters for UTF16 "\u" and UTF32 "\U".

    Sprites can also be access via index <sprite index=0> or by name <sprite name="fas fa-address-card">.

    Are you saying that you would like to be able to lookup characters by name just like you can lookup text sprites by name?

    In version 1.4 or newer of TMP, Characters have a unicode value and point to a glyph index. Characters do not have names like sprites do. So just to make sure I understand the request, are you asking for the ability to define a name for characters?
     
  23. MrLucid72

    MrLucid72

    Joined:
    Jan 12, 2016
    Posts:
    983
    When he says "icon", I think he's talking about a Unicode symbol like "♫" -- I always forget the cryptic unicode patterns, too.

    However, I resolve this with styles. For example, for a bullet, I turned to styles to have "<style=bullet></style>" and I can forget the Unicode sequence. It's a bit bloaty, but styles seem to already do the trick. *Maybe* it'll work without </style>, but it triggers my OCD to not close it ;D
     
  24. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    Sadly this cryptic Unicode is how all characters are mapped and referenced in font files ;)

    It is also sad that I can reference more and more characters by their unicode values :eek:
     
  25. Cleverlie

    Cleverlie

    Joined:
    Dec 23, 2013
    Posts:
    219
    my current workflow is this:

    I have my main font file Roboto, and I configure as fallback the FontAwesome font, and second fallback a custom font with Icons made in svg and put together in this custom font created with Fontello.

    so TMP will fallback automatically to FontAwesome Icons, and then if the unicode value is not there, it falls back to my custom icons.

    my idea is this: have a custom asset where we can define a dictionary where keys are just strings, and values are unicode values and in which font to look for them.

    then TMP as part of it's regular parsing could check for tags like <icon="my custom icon">, checkout that "my custom icon" string in the custom asset dictionary and replace the initial string just like this:

    "hi, this is an icon: <icon="my custom icon">"
    parsed internally as
    "hi, this is an icon: <font="whatever font we defined in the custom asset">/uf340</font>"

    this way not only we can forget about unicode values, but we can define any number of icons and avoid clashing of unicode ranges between icon fonts, because the system will look for the icon in a specified font.

    the only work the user has to do once is to define the dictionary that maps strings to icons in specific fonts.

    this is just a nice to have, I can live with just putting my icon fonts as fallbacks of my main font and resort to a cheatsheet of unicodes everytime until I memorize them, I'm only worried about unicode range clashing when you use multiple icon fonts from different sources, including custom ones.
     
  26. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    Like I mentioned before, the Unicode value is the unique key to identify characters and sprites. So if you have font assets and sprite assets that contain the same Unicode values, TMP will use the first occurrence of the Unicode value. This is why you can assign fallback to primary font assets vs. global fallback in the TMP Settings. This is also why you can assign a local sprite asset to a text object vs. global sprite asset in the TMP Settings.

    Now in terms of an alternative way to access these Unicode values, as @MrLucid72 stated, you could use the style tag where the would be something like this:

    upload_2019-10-21_13-26-22.png

    You would be able to access this icon #1 by using "<style="Icon #1"></style>"

    Alternatively, I could add the ability to define / name characters in font assets just like we can name sprites in sprite assets.

    This would require coming up with some new rich tag name.

    <cname="Name of Character to lookup"> could work.

    I don't want to use <icon> simply because you might be looking up a character which is not always an icon.

    I considered using & but that has the potential of getting messy on the parsing side since that is used in HTML / XML but not in C# strings.
     
    Cleverlie likes this.
  27. Cleverlie

    Cleverlie

    Joined:
    Dec 23, 2013
    Posts:
    219
    hi Stephan, thanks for the answer, yes I know this might get things complicated, I was just brainstorming, and of course the <icon> tag was just the first thing that came to my mind but it could be something more generic like <symbol="foo">

    ultimately, this is just an idea to replace the need of putting literal hexa codes and start using more friendly and easier to remember strings, and the implementation of the so called "custom asset" that I talked about earlier was to avoid unicode values clashing, since as you said, there are unicode values that are used and shared in many icon fonts, but for a good start it would be a nice addition to add names to glyphs just as it works with sprites, I don't know if that may raise even more problems or not.

    thanks for your quick support and taking your time to read the users, TMP is one of very few unity features that have actual day to day support and user input and feedback from the forums.
     
  28. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    Adding the ability to name characters just like sprites makes sense.

    We just need to figure out a good way to provide access to this feature whether it be a tag or something like &lt which is used in HTML but could potentially greatly complicate the parsing and is not something supported in C#. Adding a tag like <cname="name of glyph"> would be easy. I just don't like <cname> ... <icon> is too specific ... open to suggestions here ;)

    You are most welcome and thank you for taking the time to provide feedback, explore ideas and features and most importantly for using Unity and TMP :)
     
  29. VladimirGordeev

    VladimirGordeev

    Joined:
    May 16, 2017
    Posts:
    11
    Where "TMP package for Unity 2018.3 with Dynamic SDF" ?
    Promised a year ago!
     
  30. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    Updated releases of TextMesh Pro are exclusively available via the Unity Package Manager. Version 1.4.0 of TextMesh Pro with support for Dynamic SDF was released back in January 2019.
     
  31. jGate99

    jGate99

    Joined:
    Oct 22, 2013
    Posts:
    1,938
    @Stephan_B,
    How can i on runtime provide an icon , when i say runtime i meant i have already created atlas with 3 icons.
    But i set icon on runtime tmpro.text = "\uIconCode"
    but it simply shows the text or code instead of showing actual icon
     
  32. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    In order for the glyph / icon to be added to a font asset at runtime (assuming it is not already present in the font asset), the font asset must be using Dynamic mode. The glyph / icon must exist in the font file.

    The \uFF00 or 2 hex pairs to access UTF16 unicode values. If the icon is mapped in the UTF32 range then you must use \UFF00FF00 or 4 hex pairs.

    What is the font file you are using?

    Are you sure the icon exists in this font file?

    Can you link the source font file and unicode value of this icon?
     
    jGate99 likes this.
  33. jGate99

    jGate99

    Joined:
    Oct 22, 2013
    Posts:
    1,938
    Sorry, icon is already in the generated asset and appears if i type in text area like \uFFetc
    but lets say i have button where i want to show "add" or "edit" icon, both already embed in font asset file generated by TMPRO. All i need is set that value to text property on runtime

    var addICON = "\\uFFetc1"
    var editICon = "\\uFFetc2"

    What happens is when i do this tmpro.text = add ? addIcon : editIcon;

    so i see is actual unicode text \uFFetc instead of icon
     
  34. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    When you use two slash like "\\" you are telling C# to escape the character sequence. As such it will display as \uFF00.

    You have to use string addIcon = "\uFF00";
     
  35. Francois_Prins

    Francois_Prins

    Joined:
    Jan 21, 2020
    Posts:
    2
    @Stephan_B

    I have a similar issue to the above.

    I would like to find out if it is possible to use Alt + codes to type special characters in Unity using TextMeshPro?


    In other programs I can press ALT+136 to get an "ê" or ALT+130 for "é", ALT+0150 for "–", ALT+0215 for "×", etc.


    However, in Unity the moment I press ALT it toggles between the different tools (Hand tool, move, rotate, etc.)

    Is there a setting in Unity I can change to allow me to type these special characters using ALT+ codes?

    Thank you,
    F
     
  36. Stephan_B

    Stephan_B

    Joined:
    Feb 26, 2017
    Posts:
    6,595
    The ability to use ALT + decimal value of the Unicode (expressed in hex values) is not supported in the Unity Editor.

    However, you can use the C# standard character escape sequence for UTF16 and UTF32 characters. For UTF16, the sequence is \u00FF (2 Hex pairs) and \U00FF00FF (4 hex pairs) for UTF32.
     
  37. Francois_Prins

    Francois_Prins

    Joined:
    Jan 21, 2020
    Posts:
    2
    Thank you @Stephan_B

    I am fairly new to Unity and C#, so please excuse any dumb questions.

    Is there a list of all the characters and their corresponding hex codes that can be typed this way? I did a search, but can only find the following:

    Escape Sequence Represents
    \a
    Bell (alert)
    \b Backspace
    \f Form feed
    \n New line
    \r Carriage return
    \t Horizontal tab
    \v Vertical tab
    \' Single quotation mark
    \" Double quotation mark
    \\ Backslash
    \? Literal question mark
    \ ooo ASCII character in octal notation
    \x hh ASCII character in hexadecimal notation
    \x hhhh Unicode character in hexadecimal notation if this escape sequence is used in a wide-character constant or a Unicode string literal.

    https://docs.microsoft.com/en-us/cpp/c-language/escape-sequences?view=vs-2019

    Or is this not the best way to type special characters using this method?

    I know that one can copy the required character from the Character Map and then paste it in Unity, but I trying to find a more efficient way by typing them directly in Unity.

    Thank you,
    F
     
  38. pKallv

    pKallv

    Joined:
    Mar 2, 2014
    Posts:
    1,189
    I tried the legacy Text and there it is all supported