[RELEASED] UI Accessibility Plugin - UAP V1.0

Hi,

1. I got Accessibility Plugin(UAP) from Asset Store for my 2D android app,which has five buttons: PlayButton, QuitButton, GameOverText, Play Again, and QuitButton ).

2. I attached UAP resources( see them in parenthesis) to the buttons as follows(File hierarchy):

0 Menu
o Canvas ( Accessible UI Group Root and Accessibility Manager)
-- BackGround ( ----- )
-- MainMenu ( ----- )
---- PlayButton ( Accessible Button )
---- QuitButton ( Accessible Button)

0 Main
o Canvas ( Accessible UI Group Root and Accessibility Manager)
-- GameOverText ( Accessible Button )
---- Play Again ( ----- )
---- Quit ( Accessible Button )

3. The Issue is as follows :
0 Whenever I put on play button on Unity Editor, I get following message.
" MissingReferenceException: The Object of type 'AudioSource' has been destroyed but you are
still trying to access it."

0 Only "PlayButton" speaks. The other buttons are mute.

I tried to solve this issue, but failed.
It will be very appreciated when you give me instructions to resolve this issue.

Best Regards,

Ps: I have not read out the entire UAP documentation. I followed only the process of "Quick Start Guide( 7 Steps)"

 

Hi,
the issue comes from the Accessibility Manager component. There can/should only be one of them in the scene at any point in time, and I wouldn't recommend attaching that component to one of the Canvases in your scene.
You only need to attach the Accessible UI Group Root to the Canvas.

The workflow is that you add the Accessibility Manager prefab from the UAP/Resources folder into the scene. There's also a menu option to do this:
Tools --> UAP Accessibility --> Add Accessibility Manager to Scene

Then it should work. If there is more than one Accessibility Manager in the scene, the duplicate ones will destroy and remove themselves. If you attach them to GameObjects that you need otherwise, then these GameObjects will be deleted - which is probably causing your issues.

 

Just became aware of this plugin and it looks interesting. Does it help with mouse free operation (using tab key to move between 2D UI elements and <enter> to select?

 

Yes, that is how the plugin operates on desktop platforms. It also reads out the text on the UI elements and gives hint on how to interact with them (for example how to change the value of a slider with the arrow keys).

 

Thanks - we'll check it out!

 

Is this the only way to use accessibility in unity?

 

As far as I am aware, at the time of writing, this it is the only plugin that provides screen reader functionality, yes.

However, you can also create a self-voicing game, instead. There are a few games that go this route. The game "Frequency Missing" is made in Unity and does an excellent job at this in my opinion.
It might end up being more work, since you will not only have to record all text and all menu names in your game as audio files (or use a text-to-speech plugin). But you will need to code a way to navigate your menus, read out the appropriate texts, play the sound files, and of course play the game with swipes or other means that are suitable for a blind player.

It depends on how many menus you have, whether you want your game translated in different languages etc. If you're not sure which is the best option for you, you can post some more info about your game here or write me an email (support@metalpopgames.com) and I can make you a better recommendation.

 

Hi guys,
When I add Accessibility Manager to the scene and change second canvas rendering mode to Screen Space - Camera, UAP start not working.
How can i fix it?

 

Hi bbproger,
the plugin only has one limitation when working with a Canvas in screen space - camera mode. The general accessibility should work fine, just the touch explore isn't detecting the UI element under the finger correctly.
In other words, the usual navigation with swipes, the voice announcements etc all should work as expected. Just searching the screen with the finger while the plugin reads out what's under the fingertip is currently not yet supported (I am working on adding that as well, though).

I just did another quick test to confirm that everything is working as I described it (in the latest version of the plugin)
Is this the behaviour you're seeing? If it isn't, please let me know, since there might be something else wrong.

 

Just purchased the plugin. Thank you for creating it, it sounds like exactly what we need. After importing it I got 9 errors. All related to Debug. Example:

Assets/UAP/Scripts/Core/UAP_AccessibilityManager.cs(4106,9): error CS0117: `Debug' does not contain a definition for `LogWarning'

Suggestions on how to proceed?

P.S. Never mind I think I found the issue. We have Unity.Debug overloaded with our own class. I will add LogWarning to it.

P.P.S Yup that was it. Working fine now. Sorry about the false alarm.

 

Awesome, glad you found the issue.

 

Hi,
It seems the plugin is using Time.deltaTime when checking for single taps, swipes and touch exploring. Is there a particular reason for using Time.deltaTime instead of Time.unscaledDeltaTime? Currently it renders UAP unusable during gameplay sections where Time.timeScale is being altered, as deltaTime will also slow down / speed up during those sections. For example pausing the game via setting Time.timeScale to 0 will make it impossible to navigate with accessibility plugin during paused state.

We are developing an audio game set to release September 1st, and would like to know if this is intended behavior or not.

 

Hi!
Yoy are 100% correct, and I recently noticed this myself, too. Not using the unscaled time was an oversight. This has already been fixed in the latest update to the plugin. This new version isn't live in the Asset Store just yet, because I'm currently working on updating the documentation with the new features.

If you want to adjust the code in the meantime, it is safe to change all the references to deltaTime to the unscaled version. And the next official update will include this as well.

 

Hello @mikrima I'm working on a multi-platform app using UAP, but I seem to be having a bit of an issue with Android. Currently if you have TalkBack enabled, my app asks you to disable TalkBack. If you leave it enabled you can use the app using two-finger gestures, but not one, which is quite different from how it behaves on iOS, where when VoiceOver is enabled you still get one finger gestures. Are there any functional work-arounds for this behavior or is the only available route to return to the windows shade or settings and disable TalkBack if you want one finger gestures?

 

Hi JVaughan,
when TalkBack is enabled, it forcefully grabs the single-finger input and doesn't forward it to the app at all. It is a constant source of pain. There is no workaround for this, unfortunately.
VoiceOver is different, because it allows the app to take the single-finger input even when it is active.

However, there's no need to navigate to the Settings menu. TalkBack can simply be paused by pressing the up and down volume keys at the same time for 3 seconds. TalkBack users usually know this shortcut well, and won't have much trouble with it at all.

 

Hey @mikrima! I'm having an issue with Android. When I launch my app, it says "Game Mode" or something to that extent. I never get the pop up prompting the user to turn off TalkBack. Am I missing something?

 

Hi @TarahPeltzWork,
Have you maybe started the app on Android before and disabled Accessibility mode? The plugin will save when it was turned off, so that it doesn't bother users again in the future. Some Android phones incorrectly report whether TalkBack is running, so this is a necessity.
Does the shortcut to turn on accessibility work for you? (three finger triple tap on the screen - TalkBack must be paused, otherwise Unity doesn't receive the input from the touch)

 

Yes, so our workaround right now is to launch the app with accessibility mode turned on and then turn off accessibility mode. The app remains accessible, which is what we want. However, I thought there was supposed to be a pop up of some sort prompting the user to turn off TalkBack.
Unfortunately, the three finger triple tap doesn't work for me. I'm on a Samsung Galaxy S9+.

 

The “Enabled after app install” checkbox only decides whether the app should start with the plugin and accessibility enabled directly after install, on first app start. On all subsequent app starts, the checkbox is no longer relevant, and the plugin will restore to whatever state it was in last. As an example – if the accessibility plugin was disabled the last time the app was running, it will stay disabled. If it was enabled, it will turn on again.


The behavior on Android and iOS is slightly different, in that on iOS the plugin can automatically turn on or off when it detects VoiceOver running. This is not possible on Android, because some Android phones incorrectly report TalkBack as active, even when it isn’t. Turning on accessibility mode automatically will upset and confuse users and possibly lead to bad reviews.


Yes, TalkBack needs to be paused/suspended/turned off during play, since Android otherwise doesn’t send the touch input to the app itself (this is a restriction in the operating system and the plugin cannot get around it). There is no visible popup telling users to pause TalkBack, it is just a voice announcement.

The triple tap will only work if TalkBack is off, because the app doesn't even receive the touch input otherwise.

If this isn’t playing, something must be going wrong in the app setup or possibly there’s a bug in the plugin. If you can send me a reproduction case/steps, I’ll be happy to investigate.

 

Hey, and congrads on the plugin!! Developing for groups with sense problems should be the norm!!
On that issue, I would like to know if there is a way to control the "Pitch" and Gender of the TTS (just like you do for the speech rate) and add it to the Settings of my app.

 

Hi Jahnbt,

no, the pitch and gender cannot be controlled, at least not at the moment. I understand it can make sense for a game to request a different voice or gender, for example to read out text from AI players or tutorial and story characters.

Support for a different pitch is already on my roadmap (but not ETA on this at all). The gender is a different matter. On most platforms, the plugin will make use of the native screen reader, such as TalkBack, VoiceOver or NVDA. In these cases, the user selected voice will be used. Working around that and requesting a different gendered voice is possible, but not as straight forward. That's why it's not a priority right now and I don't know if or when support for this would be added.

 

Thank you for the fast response.
Another question. What is the status on Unity + Talkback? I found your post about not being possible to deactivate it through Unity. Do I have to detect if Talkback is running and suggest users to manually deactivate it? Or is there a way for these two to co-exist?

 

Yesnokindof.
There's still no way around having to pause TalkBack (manually) while a Unity game is running. However, the plugin does the detection for you and it will also play a message to the player asking them to please suspend TalkBack during play. So you won't have to do anything on your end.

 

Hello! I've been working on integrating UAP into an existing project, and the process has been very straightforward and simple. Currently, the only issue I'm running into is the app was originally set up with Screen Space - Camera canvases so the touch explore is not working. Do you have any updates on when this will be fixed, or is it still in development? Thanks!

 

Hi @RhogarDelmirev,
unfortunately the fix isn't yet complete. I scheduled it to be included in the plugin update after this upcoming next one.
Could you please let me know how critical this is for you and what your timeline is? If it's urgent, I can try and look into it sooner.

 

Thanks for the response mikrima! I was able to convert the canvases that required accessibility integration into Screen Space - Overlay mode. We are rolling out accessibility functionality over time, so this solution works will work for our needs. Because of this, getting the Screen Space - Camera functionality its not especially urgent, although it is nice to know that a solution is coming in a future update.

 

Hey @mikrima I was originally going to ask about the WWW warning I get from UAP, but I did some looking around and saw that change and some awesome updates were made but are not released yet.

So as a follow up question: Do you have an expected/wanted release time frame for the 1.1.0 update?

 

Hi, nice plugin, thanks for your work. Just one question.

Can I remove the initial instructions? They sound first time a user loads the app.


Greetings.

 

Is there an option we have to turn on to get this to happen? It doesn't seem to play a message to the player for suspending TalkBack

It works perfectly for iOS/VoiceOver, though -- thanks!

Edit: Saw that there was an update, and everything seems to work now. Thanks again!

 

Hey @mikrima ,

I wrote some time ago (as User: Jahnbt) about implementing SetPitch in our app. The thing is that the way you made it is difficult to use another plugin to set the Pitch for the TTS system your plugin use. So we need to find a solution. Will you be willing to work together so that we add one small function to the plugin for pitch setting?

I guess if we had access to your Android Studio Project for the plugin setting a new function would be pretty easy to implement.

Please contact us so we can find a solution.

 

Great Plugin. We are using version 1.06 and working on integrating it into our games.

1) We are using the function AP_AccessibilityManager.Say which says the player is allowed to interrupt the TTS. How do we test this? The platform is Windows. What do we do to interrupt the TTS the way a blind player would?

2) A possible bug in the template we fixed internally in Update() in UAP\Scripts\Core\UAP_AccessibilityManager.cs

if (m_CurrentElementHasSoleFocus)
{
// Clopas added null check
if (m_CurrentItem != null && m_CurrentItem.m_Type == AccessibleUIGroupRoot.EUIElement.ETextEdit)

 

Hey Mikrima,

We love your plugin. You bridged a huge gap for accessibility in terms of Unity applications. Keep up the good work! While you work on the "Out of box WebGL TTS," I would recommend looking into a work around to utilize the out of browser, platform based TTS like NVDA.

 

Hi @mikrima,

Love your plugin so far! Is there a way to tie the Accessibility_Manager into Unity's AudioMixer so that when I use the
UAP_AccessibilityManager.Say() function it will duck any background audio to make sure players can hear what's being said?

In this unity3d video, it shows how I can use one mixer to Send information to another to Duck the volume.

I'm currently tweaking the volume down for the audio mixer before invoking Say(), and using
while (UAP_AccessibilityManager.IsSpeaking()) to wait until it's done to ramp the background audio back up.

 

Hi @mikrima,

I implemented your SDK and it works fine on android but on iOS it doesn't.
I know that on android the user has to disable talkback to be able to use the app but there is nothing mentioned about something similar to this for iOS but the app doesn't work when the voice-over is active. I've tested the app and disabled the voice-over then it works fine.
should I add something like what we have on android for informing the user to disable the talkback to the iOS version too?

 

Hi, fantastic plugin!

Works great in editor but I have errors when I make a windows build.

DllNotFoundException: nvdaControllerClient - should I manually include the dll?

Solved: doesn't support 64bit, use 32 instead to fix missing dll.

 

I'm getting a "ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index" error in the UAP_AccessibilityManager.cs when my app changes scenes back to the main menu.

if I uncomment lines 12 & 13 (that I added), I no longer get the error on line 16. Is this going to affect your plugin?

Using Version 1.10 of UAP with Unity 2019.2.21f1

Code (CSharp):

1. int containerIndex = m_ActiveContainerIndex;
2.       if (containerIndex < 0)
3.          containerIndex = 0;
4.  
5.       for (int i = 0; i < m_ActiveContainers.Count; ++i)
6.       {
7.          int index = i + containerIndex;
8.  
9.          if (index >= m_ActiveContainers.Count)
10.             index = index - m_ActiveContainers.Count;
11.        
12.         //if (index >= m_ActiveContainers.Count)
13.          //index = m_ActiveContainers.Count - 1;
14.  
15.          if (m_ActiveContainers.Count > 0)
16.             m_CurrentItem = m_ActiveContainers[index].GetCurrentElement(m_CyclicMenus);
17.          else
18.             m_CurrentItem = null;
19.          if (m_CurrentItem == null)
20.          {
21.             Log("Nothing selected");
22.             if (makeSureItemIsSelected)
23.             {
24.                DecrementUIElement();
25.                UpdateCurrentItem(selectionSource);
26.                return;
27.             }
28.          }
29.          if (m_CurrentItem != null)
30.          {
31.             m_ActiveContainerIndex = index;
32.  
33.             if (m_PreviousItem != m_CurrentItem && ((m_PreviousItem == null || m_CurrentItem == null) || (m_PreviousItem.m_Object != m_CurrentItem.m_Object)))
34.             {
35.                Log("Selected item was updated" + ((m_CurrentItem != null) ? " to " + m_CurrentItem.m_Object.GetTextToRead() : " NONE") + " (Frame: " + Time.frameCount + ") + previous item: " + (m_PreviousItem != null ? m_PreviousItem.m_Object.GetTextToRead() : "null"));
36.                if (isEnabled)
37.                {
38.                   if (m_PreviousItem != null && m_PreviousItem.m_Object != null)
39.                      m_PreviousItem.m_Object.HoverHighlight(false, selectionSource);
40.                   if (m_CurrentItem != null && m_CurrentItem.m_Object != null)
41.                      m_CurrentItem.m_Object.HoverHighlight(true, selectionSource);
42.                }
43.             }
44.  
45.             m_PreviousItem = m_CurrentItem;
46.             return;
47.          }

 

我在使用中文的时候没办法识别，英文久能正常是识别，在Unity上面测试的时候是正常的，但是打包到手机上不会朗读是怎么回事？

 

Looks like this plugin works with UI Dropdown, but not Dropdown - TextMesh Pro

 

hello, I like your plugin, and I want to know does it work with Arabic or not?

 

I'm trying to fully understand how this works.

I'm looking to set it up as a WebGL.

I was under the impression the build would be read by the users screen reader but in practice it instead seems to just start reading it on load. My current system does not have any screen reader so I'm not sure where the audio is coming from or why it's enabling the screen reader assets when I have the "auto-enable if screen reader detected". I assumed it would mean nothing would be enabled unless the reader was detected but that is not my experience. I've had to fully disable it and rebuild because my system doesn't need the screen reader.

Basically I'm trying to create something that uses the screen reader when needed and doesn't when it's not needed but I'm not seeing how to get it to do that correctly.

 

Great plugin, just a question, how can I change the speed of the speech?

 

there hasn't been any response from the developer in this thread since last year, is this plugin still supported, and is there an ETA for v1.10?

 

There hasn't been much from the devs in the forum unfortunately, but the asset was update in January of this year.
From what I can see in my project it looks like what is in the Change Log for 1.10 is out because I have build to web with working TTS through Google, and my project is using TextMeshPro with the text being read without issue(s) that I can see.

 

Is anybody using this with the latest version of NGUI -- if so, does it work fine? (The docs say it's meant for NGUI 3.11.2 but that requires the NGUI Pro license to obtain....)

 

We've got this set up in a project (which has uGUI), works great. However, when we build to WebGL and upload that game to our website, we can’t get it to work.

MacOS: I’ve enabled VoiceOver and when I load the page, it shows a tip: "You are currently on web content. To enter the web area, press Control-Option-Shift-Down Arrow.” But that doesn’t do anything at all. I can’t figure out how to enable the voiceover and tabbing etc. (I”ve also tried Cmd-Option-Shift-Down Arrow” since it’s Mac.)

Windows 10: I’ve enabled “Narrator” which I guess is the built-in screen reader now…but it doesn’t so anything at all, doesn’t even show a tip like MacOS does. (Narrator is working, reading everything else, including a text link on the WebGL page, but it seems oblivious to the WebGL content itself.)

So then I did a build which has UAP forced on for all users, but still I couldn't get any UAP functionality to work. EDIT: That was on Mac Safari. On IE, it does work properly when forced on.

Anyone know what we might be doing wrong?

 

Have been using UAP for a while now, however when I port an app to Android from Windows I find the audio is set to a very high rate when speaking text to speech. Is there a way to adjust the speech speed/rate? I have look at Accessibility Manager scripts but I still can't find any working solution.

 

Hi grobm!
The only issue we've had with unusual speaking rates on Android was on rather old versions of Android which use different scaling for the speech rate setting.

However, the speaking rate can be modified using the static function 'SetSpeechRate'
UAP_AcceessibilityManager.SetSpeechRate( [1..100] );

Note that this setting will not affect iOS VoiceOver, which will always use the user's personal VoiceOver speech settings.

 

Hi @mikrima , this plugin is not working when I deploy to an iOS device…but works great in the editor. I’ve turned voiceover on and have the plugin set to turn on if voiceover is detected.

When I start my game, the green UAP UI highlight goes around the first speakable element and the voice prompt begins reading it…however Voiceover is turned off (and the plugin is only set to turn on if Voiceover is turned on) on my phone and when I attempt to navigate the screen nothing happens, I cannot navigate anywhere in the app.

When I run the game in iOS, I see the following errors when I attempt to navigate the app in Xcode’s console:

SendMessage: object UAP_IOSGestures does not have receiver for function OnSwipeRightCallback!
SendMessage: object UAP_IOSGestures does not have receiver for function OnSwipeDownCallback!
SendMessage: object UAP_IOSGestures does not have receiver for function OnSwipeDownCallback!

Long story short…the plugin turns on when it’s supposed to be off and I cannot navigate anywhere in the app at all.

I’ve also tested with voiceover turned on and that does not make any difference.

Again this all works perfectly in the editor and only gives me trouble when I install my app on iOS. My game works perfectly if I disable UAP, so it seems the issue lies in the plugin.

@mikrima could you please help point me in the right direction?

 

Hi Iriley3!
Could you tell me which version of iOS you're using on the iOS test device? I doubt that this is an issue, but in the past, beta versions of iOS have messed up the gesture recognition. This was for example the case when Apple first released iOS 13 beta.

It sounds like the script that is supposed to catch the gestures isn't working correctly. There's a good chance that it crashed earlier somewhere upon initialization. If this is the case, it is possible that this is also responsible for the plugin turning on when it shouldn't.
Could you verify that the file ios_gestures.cs is present, and contains the supposed missing OnSwipeRightCallback etc functions?
The plugin creates a GameObject with the name UAP_IOSGestures at runtime, and attaches the ios_gestures.cs script. (UAP_AccessibilityManager.cs line 2036ff, function EnableAccessibility())
From the error message, either the GameObject or the script seem to be deleted at a later point.
Do you remember if you made any modifications to the plugin?

Could you please use one of the included example scenes, build to iOS and test if the navigation works as expected there?

 

Sent the message below this to @mikrima via email. Adding here just in case it may be helpful to anyone else:

I’m using an iPhone X, running iOS 13.5.1 and building with Xcode 11.3.1.

The iOS_gestures.cs script is present and it does contain the missing OnSwipeCallback functions. I’ve also added some debug logs to the functions to see if they are triggered. I added one Log on the “SetUAP” function and it fires correctly. I’ve also added logs to each OnSwipeCallback, however none of them seem to be running when a swipe is made. Below is an example of what I added to each OnSwipeCallback function in the iOSGestures.cs script to see where the issue is lying (The Debug.Log items are what I added):

Code (CSharp):

1. void OnSwipeRightCallback(string fingerCount)
2. {
3.    if (m_DebugOutputLabel != null)
4.    {
5.       m_DebugOutputLabel.text += "Swipe Right detected - " + fingerCount + "\n";
6.    }
7.    int fingers = 1;
8.    int.TryParse(fingerCount, out fingers);
9.    if (m_UAP != null && UAP_AccessibilityManager.IsEnabled())
10.    {
11.       m_UAP.OnSwipe(UAP_AccessibilityManager.ESDirection.ERight, fingers);
12.       Debug.Log("gestures: right swipe completed");
13.    }
14.    Debug.Log("gestures: right swipe attempted");
15. }

The UAP_IOSGestures GameObject does instantiate at runtime and does have the iOSGestures.cs script attached to it. I also added a debug log to run when the object is created and can confirm that it does instantiate when on the phone.

For testing purposes, I tried completely redownloading the plugin and running a fresh version, but found that the issue still remained. After that, the only modifications I made were the addition of debug logs to some functions in the UAP_AccessibilityManager.cs and iOSGestures.cs scripts to verify that the items you suggested previously are running.

I just tried building the “Unity UI Navigation Example” to my iPhone and am finding the same issue with the same Xcode console messages (SendMessage: object UAP_IOSGestures does not have receiver for function OnSwipeRightCallback!).

Thanks again for such an awesome plugin! I really appreciate your help!