TextMesh Pro TMP: Breaking Chinese Bugs for Traditional ZhuYin ("BoPoMoFo" - Taiwan Locale)

@stephan Still bugged in 2020.1.9f1 Following the same steps as the 2019 post way above:

EDIT: I can't even post bopomofo in Unity forum -__-"


So here's a screenshot of the post:

 

...?


Eh? A bit off-topic, but hindering the flow to this bug report: Why are Chinese characters banned completely here? They didn't used to be (I just quoted an above one, so sometime in 2020 it got banned).

EDIT 1: Testing simplified... 国
EDIT 2: So simplified Chinese works, but only traditional and Bopomofo (Taiwan) characters are banned here. This seems explicitly added since it worked fine in the 2019 bug report. Oof.
EDIT 3: Japanese has shared Kanji - I wonder if Unicode characters can decipher the origins. Testing... 日本. Zhtw: <banned>
EDIT 4: I can confirm that Japanese kanji is detected separately from a dupe traditional Chinese combo, implying that Bopomofo and traditional Chinese banning is intentional：

(Perhaps something to do with the Tencent deal last year, I'd imagine - if that is true, that is disheartening)

 

I just tested the above in 2020.1.9f1 and it appears to be working as expected.



There is another issue which is Editor related where IME appears to get disabled on focus loss. Maybe when you tested the above, the input field lost focus?

In terms of the ban words, I would suggest directing this question to a forum moderator as I have no idea on my end.

 

Yea - part of the repro steps was to submit then type again [same with the original 2019 issue]. The first time you type it is fine (always good)~ the actual bug was typing a 2nd time via onFinished (or whatever it's called) likely unfocuses and refocuses.

 

Let me see if I can reproduce the behavior. Just so I understand the sequence, the user types in the bottom input field and presses enter. The text is then copied / set in one of the output input field above. And then when you re focus and type again in the bottom input field the issue occurs. Correct?

Are you able to reproduce the behavior without having to use a specific script that uses OnSubmit()?

 

Minimally:

1. In a new scene, drop a TMP_Input
2. Set a Unicode-friendly font
3. Play and type jp6 with bopomofo, resulting in "文"
4. Click outside the Input
5. The IME immediately goes back to English (even before you click back in - you can see in the Windows bar).

Yep, seems like you're right - has to do with unfocusing: The IME switches back to English after unfocusing an input. Expected: Remains the same. Any chance of a quick fix I could embed?

 

I quick fix should be possible for the TMP Input Field but will take longer for the text fields in the Unity Editor.

I'll provide an update as soon as I have a fix for the input field.

 

How goes it?

 

Had a chance to take a closer look over the weekend and the issue is in the Editor as using "Input.imeCompositionMode = IMECompositionMode.On" doesn't re-enable IME once it has been disabled.

I have reached out to the Input and Platform teams for their insight. I'll also take another look later this week once the next TMP release is out.

 

Glad to hear you worked on it before the weekend: How'd it go?

 

Apologies to drive you crazy, but any movement on this? There are thousands of players waiting for this fix initially posted in 2019 -- I promised some big things based on the hopes of the last big patch that it was fixed~

By chance, is there a quick n dirty fix I could apply found from your research since Oct 27? I really need to deliver this: We're losing momentum over this bug (especially after hyping up a fix that did not arrive); surely we're not the only one. Thanks!

 

This particular issue has been in Unity since 5.x and part of the Editor / native code base. This issue was never fixed previously.

This issue has to do with the IME being disabled when focus is lost in any Editor text fields as well as input fields.

So whenever focus is lost, the IME is disabled. On Windows you can manually re-enable IME by pressing CTRL + Spacebar.

I am still looking for a fix on this issue.

 

Does the new IME in Unity expose expanded control? Is there a way to catch input onDone (or whatever it was called) to get a current IME val, then add some kind of internal callback onDoneFinish to set IME back to the tracked value?

If we can detect / set the IME, some type of watchdog could potentially work.

Current:
1. onDone
2. Custom onDone code, if handled.
3. Windows IME switches

New:
1. onDone
2. Custom onDone code, if handled.
3. Watchdog gets current IME
4. Windows IME switches
5. Watchdog restores IME.

Or with the new IME system, are there any callbacks on IME change? Could cancel if a flag is active for an input submission happening (check singleton).

 

This is a Windows API / IME issue where when the context changes (which for instance happens when focus goes from an Inspector to the Scene View or any other window in Unity) and the re-associated to the new context, the IME gets disabled somehow (by windows).

The following might prove useful, which seems to indicate that association with a new context should only be done when the WM_IME_SETCONTENT message is issued otherwise the association might result in the IME being disabled by default.

 

I believe I have finally found a solution to this IME state issue. This will require more testing but seems to work fine now

Since this fix is in Unity itself, we will need to wait for those new versions of Unity to be released. I'll provide an update once this change has landed and in which specific point release of Unity it will be contained in.

 

That's wonderful news!

 

Just a weekly follow up

 

Still waiting on these changes to land in their respective versions of Unity. As per my last post, I'll provide an update once I know in which point release these will be included.

 

Unity 2019.4 fix will be included in Unity 2019.4.16f1.

Fix will also be available in 2018.4, 2020.1, 2020.2.0b14, 2021.1.0a8.

 

Excellent! I will look forward to the 2020.1 one

 

Which 2020.1 release should have the fix? I noticed there wasn't a sub ver. Thanks!

 

Changes were just merged for 2018.4 and 2020.1 banches.

This fix will be included in Unity 2018.4.30f1 and Unity 2020.1.16f1.

P.S. I believe this will be the last release of 2020.1 so I would suggest moving to 2020.2 which will be the LTS release.

 

a bit off topic, but do you feel a move from 2020.1 to 2020.2 is generally pretty smooth?

 

Generally speaking, you are more likely to encounter issues when moving to a new major release like 2019 to 2020 then moving to a new point release like 2020.1 to 2020.2. That is not to say that you won't run into any issues but it should be smoother. Just keep in mind that 2020.2 is still in beta but near final release.

From a TMP point of view, the same package is used for 2020.1 and 2020.2. This might not be the same for other packages.

The reason I brought this up is because 2020.1 is near the end of its life cycle whereas 2020.2 being the LTS release (once out of beta) will continue to get updates for the next two years or more years. So perhaps wait for 2020.2 to be out of beta and the official LTS release.

 

Is TMP 3.0.1 (July 27) correct for Unity 2020.1.16? I remember you said it's an IME fix for Unity -- not TMP -- so no need for a new TMP version? Just making sure!

 

Latest release is 3.0.3 but the fix is indeed contained in Unity 2020.1.16.

 

After upgrading to 3.0.3 and Unity 2020.1.16, I immediately had this bug:



http://recordit.co/NncUx5QlFP

In our loading screen tips text, the 2nd word keeps poofing or partially poofing, for some reason.

If I move the rect around, it's still there. If I change the text, it'll temporarily fix it - I move the rect, and the bug is back.

Bug occurs in both editor and standalone.

 

I have seen a similar report but was not able to reproduce. This issue is likely related to an effort to reduce memory overhead where some of the internal text processing structures are now shared between text objects. This should be a simple fix once I can reproduce it.

I'll try to reproduce it today and let you know what I come up with.

 

@MrLucid72 Please give the following change a try and let me know if that resolves the issue.

In the TMPro_UGUI_Private.cs in the OnPreRenderCanvas() function around line 1616, make the following change which will force the text input to always be parsed when any of the text properties are changed.

Code (csharp):

1.  
2. // Reparse the text if the input has changed or text was truncated.
3. if (true) // m_isInputParsingRequired || m_isTextTruncated)
4. {
5.     #if TMP_PROFILE_ON
6.     Profiler.BeginSample("TMP - ParseInputText()");
7.     #endif
8.  
9.  

This is a temporary change while I review / consider if we should always re-parse which potentially adds performance overhead when a text property is changed which in theory would not require re-parsing of the text. For instance, changing the vertex color does not require re-parsing. The trade-off is lower memory overhead as some of the internal data structures can be shared between all text objects.

 

The bright side? I think you guys fixed it. However, we had to downgrade back to 2020.1.9 (from 2020.1.16) because of a MAJOR breaking bug involving loading async - I have no clue what actually happens, but only upgrading Unity triggers desyncs on async load levels. The only pattern I could find was for older rigs (with updated gfx card drivers and on win10).

It was actually such a bad bug that it botched our free-to-play revamp and we had to delay (all the testers had modern PCs - only found out later).

I'm just so frustrated at Unity right now and the lack of QA in their stable releases... I know it's not your fault, but I am beyond upset at the amount of effort and loss I've had to go through just to get this IME fix. Since it wasn't backported to Unity 2020.1.9 (still very new -- how come this wasn't patched in?), I no longer have access to the fix.

 

What is the issue you are observing related to loading async? Once I have more information, I can check if a bug was already submitted on this and poke the relevant folks to figure out when they expect it fixed.

In terms of the IME issue, the fix landed in 2020.1.16f1. Unfortunately, we can't go back to patch previously released versions of Unity like 2020.1.9f1. My apologies again for not getting to this IME fix sooner.

So sorry again. Like I said before, I am going to be getting a dedicated QA for text. In terms of the async issue, once I get more information from you, I'll try to track down whoever handles those issues which and hope that it can get resolved quickly.

Since 2020.2.0f1 was just released where 2020.2 will become the LTS, I curious if that Async issue is present there. Once I have more information on this issue, I can check to see if it is present in 2020.2.

 

If I were to guess, it likely exists there since .16 is so close to the LTS. However, I'm sure they're more likely to fix it if it exists here, once found.

I have run out of time to keep hopeful for this bug fix after a year of trying to do everything in my power to assist. The irony that it's fixed but beyond my reach is almost as comical as it is hair-tearing. My only hope is that someone confirms the async scene loading bug exists, then confirmed the bug was fixed in the LTS. The poor-lucked Chinese speaking community will have to deal with this bug until then.

For a bug reported in 2018, it seems like a reasonable request to ask for a backport the fix to a Unity version only 1 or 2 months old if there's a breaking bug preventing an upgrade. Is there anyone I can contact about this? I pay for a sub and this is blocking: Would this make a difference?

Either way, thanks for going out of your way.

 

What is this Async issue so I can test to see if it is still present in 2020.2?

 

It's hard to tell since my resources expired, but all I know is this:

* Players would constantly be stuck in limbo between scenes due to some sort of async scene loading change.

* Sometimes scenes wouldn't preload correctly or have an issue preloading more than once, so users would be stuck in scenes.

* Seemed to mostly happen (if not exclusively) to OLDER PCs. Not ancient, but middle aged and lower. Middle aged happened 20 percent of the time. Old old happened 100 percent. Don't test with a modern PC: Find a crusty laptop with integrated gfx. Repro was in win64 standalone. Maybe old rig can repro in editor, but not sure. I feel editor handles async scene loading differently.

* We narrowed it down to some sort of async scene loading issue that had a breaking change. It could be infinite loading, or an issue with callbacks when a scene is loading/unloading.... Something significant changed that was breaking. Simply reverting back with no changes fixed all issues; no minor version should have breaking changes (or the usual Unity case of not enough QA and rush a release in a beta-like state): Breaks the core rules of Updating 101.