Search Unity

  1. Unity Asset Manager is now available in public beta. Try it out now and join the conversation here in the forums.
    Dismiss Notice
  2. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  3. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

Bug Cannot access classes within the Accessibility module

Discussion in '2023.2 Beta' started by harry_unity459, Aug 29, 2023.

  1. harry_unity459

    harry_unity459

    Joined:
    Mar 11, 2023
    Posts:
    8
    Hi all

    I am developing apps on behalf of Special Effect and aiming for Switch Access to be able scan through in game buttons, as well as enabling use with TalkBack and VoiceOver.

    I am unable to access any class within UnityEngine.Accessibility.

    This code:
    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4. using UnityEngine.Accessibility;
    5.  
    6. public class TestingAccessibility : MonoBehaviour
    7. {
    8.     AccessibilityHierarchy accessibilityHierarchy;
    9. }
    10.  
    Gives the following error:

    Assets\TestingAccessibility.cs(8,5): error CS1069: The type name 'AccessibilityHierarchy' could not be found in the namespace 'UnityEngine.Accessibility'. This type has been forwarded to assembly 'UnityEngine.AccessibilityModule, Version=0.0.0.0, Culture=neutral, PublicKeyToken=null' Enable the built in package 'Accessibility' in the Package Manager window to fix this error.

    However, there is no package called Accessibility visible in the package manager.

    upload_2023-8-29_10-31-52.png

    In the above example I have attempted to use the AccessibilityHierarchy class, but the same error is given when attempting to access any class.

    I'm probably doing something daft, but any support would be much appreciated!

    Thanks
    Harry
     
  2. bianca-stana

    bianca-stana

    Unity Technologies

    Joined:
    Jan 26, 2021
    Posts:
    14
    Hello Harry,

    This is because the Accessibility module was not initially listed and added to projects by default. To add the module to your project, you need to manually add it to Packages/manifest.json:

    "com.unity.modules.accessibility": "1.0.0",


    The module will be listed in the Package Manager and added by default to new projects starting with Unity 2023.2.0b8.

    Please note that the current set of Accessibility APIs does not fully support switch access, as they primarily focus on mobile screen reader support (TalkBack and VoiceOver). However, we are actively working on improving and extending them.

    I hope this helps! We are excited to see that our Accessibility APIs are already being used. :)

    All the best,
    Bianca
     
    Last edited: Aug 31, 2023
    JuliaP_Unity and karl_jones like this.
  3. harry_unity459

    harry_unity459

    Joined:
    Mar 11, 2023
    Posts:
    8
    Hi Bianca

    Thanks very much for the speedy reply. That's interesting and good to know. I can now access those classes successfully.

    I'm now running this code to get my bearings:

    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4. using UnityEngine.Accessibility;
    5.  
    6. public class Test : MonoBehaviour
    7. {
    8.     Color[] colors = { Color.red, Color.green, Color.blue };
    9.  
    10.     [SerializeField]
    11.     GameObject canvas, buttonOne, buttonTwo;
    12.  
    13.     public AccessibilityNode buttonOneNode, buttonTwoNode;
    14.  
    15.  
    16.     public AccessibilityHierarchy accessibilityHierarchy;
    17.  
    18.     void Awake()
    19.     {
    20.         accessibilityHierarchy = new AccessibilityHierarchy();
    21.         buttonOneNode = accessibilityHierarchy.AddNode("button one", buttonOneNode);
    22.         buttonTwoNode = accessibilityHierarchy.AddNode("button two", buttonTwoNode);
    23.      
    24.         Debug.Log(buttonOneNode.frame);
    25.         Debug.Log(buttonTwoNode.frame);
    26.  
    27.         buttonOneNode.frame = buttonOne.GetComponent<RectTransform>().rect;
    28.         buttonTwoNode.frame = buttonTwo.GetComponent<RectTransform>().rect;
    29.  
    30.         buttonOneNode.role = AccessibilityRole.Button;
    31.         buttonTwoNode.role = AccessibilityRole.Button;
    32.  
    33.         foreach (var node in accessibilityHierarchy.rootNodes)
    34.         {
    35.             Debug.Log(node);
    36.             node.isActive = true;
    37.         }
    38.  
    39.         Debug.Log(buttonOneNode.frame);
    40.         Debug.Log(buttonTwoNode.frame);
    41.         Debug.Log(buttonOneNode.role);
    42.         Debug.Log(buttonTwoNode.role);
    43.  
    44.  
    45.         accessibilityHierarchy.RefreshNodeFrames();
    46.  
    47.         Debug.Log("active hierarchy: " + AssistiveSupport.activeHierarchy);
    48.     }
    49.  
    50.     // Update is called once per frame
    51.     void Update()
    52.     {
    53.      
    54.     }
    55. }
    But the active hierarchy is returning empty, and I can see no active hierarchies during runtime or editing:
    upload_2023-8-29_12-29-44.png

    So I suppose my follow up question is, what's the intended workflow of adding nodes?

    My hope re:switch access was that once AndroidOS knows the position of each interactable item it can then use that to scan between them, as it needs that information for TalkBack anyway. Is there any Switch Access functionality, is that down the line, or possible but yet to be tested?

    Thanks
    Harry
     
    Last edited: Aug 29, 2023
  4. bianca-stana

    bianca-stana

    Unity Technologies

    Joined:
    Jan 26, 2021
    Posts:
    14
    For cashing purposes, you can have multiple hierarchies at once (for example, one for each screen), but only one can be active at a time. Thus, once the hierarchy is created, it needs to be set as the active one using AssistiveSupport.activeHierarchy. This has to be done both initially and when the screen reader is activated (you can listen to this using AssistiveSupport.screenReaderStatusChanged) because AssistiveSupport.activeHierarchy is set to
    null
    when the screen reader is deactivated. Additionally, the active hierarchy should be set after calling WaitForEndOfFrame() like below:
    Code (CSharp):
    1. IEnumerator SetAccessibilityHierarchy()
    2. {
    3.     yield return new WaitForEndOfFrame();
    4.  
    5.     AssistiveSupport.activeHierarchy = accessibilityHierarchy;
    6. }
    We plan to publish a sample project soon to make it easier to understand how the screen reader support APIs should be used and what the workflow should look like.

    Regarding switch access: What you are saying sounds right. Switch access may work using the current set of Accessibility APIs. However, please note that we did not test it and cannot guarantee full functionality, as it is possible that additional configuration may be required beyond what our current APIs offer.
     
    Last edited: Aug 31, 2023
  5. harry_unity459

    harry_unity459

    Joined:
    Mar 11, 2023
    Posts:
    8
    Hi Bianca.

    Thanks again for getting back to me. Having done some testing I can get TalkBack going, but no luck on Switch Access. Do you have a vague idea of when that functionality may be coming, or any ideas on what else I might be able to try?

    Thanks
    Harry
     
  6. bianca-stana

    bianca-stana

    Unity Technologies

    Joined:
    Jan 26, 2021
    Posts:
    14
    Hi Harry,

    I have quickly tested Switch Access on one of our internal Android sample projects. Basic functionality appears to work for accessibility nodes that are set to listen to the AccessibilityNode.selected callback. Note that this callback should not be triggered by the application itself, as it comes from the operating system based on the user's actions (we are currently working on making it an event). Also, this callback is linked to AccessibilityNodeInfo.ACTION_CLICK, and there is currently no API available for AccessibilityNodeInfo.ACTION_LONG_CLICK.

    Unfortunately, we do not have a date for when we will be able to fully support switch access.

    Let me know if you have any further questions or concerns!
     
  7. harry_unity459

    harry_unity459

    Joined:
    Mar 11, 2023
    Posts:
    8
    Hi Bianca

    That's really helpful, thanks.

    After adding that, the blue box is appearing, but only in the top left corner. I have tried different means of anchoring to no avail. I can see that the width/height/x/y are being set on the .frame correctly from the logs. Here's my code so far:

    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4. using UnityEngine.Accessibility;
    5.  
    6. public class Test : MonoBehaviour
    7. {
    8.     Color[] colors = { Color.red, Color.green, Color.blue };
    9.  
    10.     [SerializeField]
    11.     GameObject canvas, buttonOne, buttonTwo, buttonThree;
    12.    
    13.     public AccessibilityNode buttonOneNode, buttonTwoNode, buttonThreeNode;
    14.    
    15.     public AccessibilityHierarchy accessibilityHierarchy;
    16.  
    17.     void Awake()
    18.     {
    19.         accessibilityHierarchy = new AccessibilityHierarchy();
    20.         buttonOneNode = accessibilityHierarchy.AddNode("button one", buttonOneNode);
    21.         buttonTwoNode = accessibilityHierarchy.AddNode("button two", buttonTwoNode);
    22.         buttonThreeNode = accessibilityHierarchy.AddNode("button three", buttonThreeNode);
    23.  
    24.         Debug.Log(buttonOneNode.frame);
    25.         Debug.Log(buttonTwoNode.frame);
    26.         Debug.Log(buttonThreeNode.frame);
    27.  
    28.         buttonOneNode.frameGetter = GetButtonOneFrame;
    29.         buttonTwoNode.frameGetter = GetButtonTwoFrame;
    30.         buttonThreeNode.frameGetter = GetButtonThreeFrame;
    31.  
    32.         buttonOneNode.role = AccessibilityRole.Button;
    33.         buttonTwoNode.role = AccessibilityRole.Button;
    34.         buttonThreeNode.role = AccessibilityRole.Button;
    35.  
    36.         foreach (var node in accessibilityHierarchy.rootNodes)
    37.         {
    38.             node.isActive = true;
    39.         }
    40.  
    41.         buttonOneNode.selected = LogOne;
    42.         buttonTwoNode.selected = LogTwo;
    43.         buttonThreeNode.selected = LogThree;
    44.  
    45.         AssistiveSupport.activeHierarchy = accessibilityHierarchy;
    46.  
    47.         StartCoroutine(SetAccessibilityHierarchy());
    48.  
    49.         AssistiveSupport.screenReaderStatusChanged += OnScreenReaderStatusChanged;
    50.         accessibilityHierarchy.RefreshNodeFrames();
    51.     }
    52.  
    53.     bool LogOne()
    54.     {
    55.         Debug.Log("Log One");
    56.         Debug.Log(buttonOneNode.frame);
    57.         return true;
    58.     }
    59.  
    60.     bool LogTwo()
    61.     {
    62.         Debug.Log("Log Two");
    63.         Debug.Log(buttonTwoNode.frame);
    64.         return true;
    65.     }
    66.  
    67.     bool LogThree()
    68.     {
    69.         Debug.Log("Log Three");
    70.         Debug.Log(buttonThreeNode.frame);
    71.         return true;
    72.     }
    73.  
    74.     void OnScreenReaderStatusChanged(bool isActive)
    75.     {
    76.         Debug.Log("Screen reader status changed: " + isActive);
    77.         StartCoroutine(SetAccessibilityHierarchy());
    78.     }
    79.  
    80.     IEnumerator SetAccessibilityHierarchy()
    81.     {
    82.         yield return new WaitForEndOfFrame();
    83.  
    84.         AssistiveSupport.activeHierarchy = accessibilityHierarchy;
    85.        
    86.         Debug.Log("active hierarchy: " + AssistiveSupport.activeHierarchy);
    87.         buttonOneNode.frame = buttonOne.GetComponent<RectTransform>().rect;
    88.         buttonTwoNode.frame = buttonTwo.GetComponent<RectTransform>().rect;
    89.         buttonThreeNode.frame = buttonThree.GetComponent<RectTransform>().rect;
    90.     }
    91.  
    92.     Rect GetButtonOneFrame()
    93.     {
    94.         Debug.Log("GetButtonOneFrame");
    95.         return buttonOne.GetComponent<RectTransform>().rect;
    96.     }
    97.  
    98.     Rect GetButtonTwoFrame()
    99.     {
    100.         Debug.Log("GetButtonTwoFrame");
    101.         return buttonTwo.GetComponent<RectTransform>().rect;
    102.     }
    103.  
    104.     Rect GetButtonThreeFrame()
    105.     {
    106.         Debug.Log("GetButtonThreeFrame");
    107.         return buttonThree.GetComponent<RectTransform>().rect;
    108.     }
    109. }
    What is used in your test project to set the .frame correctly?

    Thanks again,
    Harry
     
  8. bianca-stana

    bianca-stana

    Unity Technologies

    Joined:
    Jan 26, 2021
    Posts:
    14
    Hi Harry,

    Make sure that the node frames are always up-to-date and that you applied the correct scaling from world coordinates to screen coordinates. We recommend using AccessibilityNode.frameGetter rather than AccessibilityNode.frame so that the frames are automatically recalculated. If AccessibilityNode.frame is not explicitly set, it will retrieve its value from AccessibilityNode.frameGetter.
    Code (CSharp):
    1. node.frameGetter = () => GetFrame(gameObject.GetComponent<RectTransform>());
    2.  
    3. ...
    4.  
    5. Rect GetFrame(RectTransform rectTransform)
    6. {
    7.     var canvas = rectTransform.GetComponentInParent<Canvas>();
    8.     var camera = canvas.renderMode == RenderMode.ScreenSpaceOverlay ? null : canvas.worldCamera;
    9.     var worldCorners = new Vector3[4];
    10.     var screenCorners = new Vector3[4];
    11.  
    12.     rectTransform.GetWorldCorners(worldCorners);
    13.  
    14.     for (var i = 0; i < worldCorners.Length; i++)
    15.     {
    16.         screenCorners[i] = RectTransformUtility.WorldToScreenPoint(camera, worldCorners[i]);
    17.     }
    18.  
    19.     GetMinMaxX(screenCorners, out var minX, out var maxX); // custom method that outputs the minimum and maximum y value in a vector
    20.     GetMinMaxY(screenCorners, out var minY, out var maxY); // custom method that outputs the minimum and maximum x value in a vector
    21.  
    22.     return new Rect(minX, Screen.height - maxY, maxX - minX, maxY - minY);
    23. }
    AccessibilityHierarchy.RefreshNodeFrames is not necessarily needed. It is a convenience method that updates the AccessibilityNode.frame of all nodes in the accessibility hierarchy based on AccessibilityNode.frameGetter and then calls AssistiveSupport.notificationDispatcher.SendLayoutChanged.

    Hope this helps!
     
    Last edited: Sep 14, 2023
  9. harry_unity459

    harry_unity459

    Joined:
    Mar 11, 2023
    Posts:
    8
    Hi Bianca

    That's tremendously helpful. This is now working :D. There's something slightly off about the frames on the Google pixel 6a, but I think that's to do with the resolution reported by the phone. Thanks again, this is a real win for us.

    Harry
     
  10. bianca-stana

    bianca-stana

    Unity Technologies

    Joined:
    Jan 26, 2021
    Posts:
    14
    Awesome! I am genuinely happy that it works and glad to have been of assistance. :)

    For any further questions or feedback regarding our Accessibility APIs, please do not hesitate to post on our new Accessibility Forum.

    Good luck with your project!
     
  11. harry_unity459

    harry_unity459

    Joined:
    Mar 11, 2023
    Posts:
    8
    Hi Bianca

    I've now moved on to iOS, using the same code as for Android, Switch Control automatically goes to the equivalent of point scan. Do you have any insights into how to get it to recognise the buttons and therefore be scannable? In this case, VoiceOver does not seem to recognise the buttons either.

    The code is a little harder to show now it's implemented with our WebGL support. But you can think of switches as an array of buttons. As previously stated this is working correctly on Android.


    Code (CSharp):
    1. #else    
    2.             // makes use of Unity accessibility API FOR iOS and Android
    3.             accessibilityHierarchy.Clear();
    4.  
    5.             foreach (Switchable s in _switches){
    6.                 Rect testRect = GetFrame(s.GetComponent<RectTransform>());
    7.                 AccessibilityNode tempNode = null;
    8.                 if (!s.gameObject.activeInHierarchy || s.disabled || accessibilityHierarchy.TryGetNodeAt(testRect.x, testRect.y, out tempNode)) continue;
    9.                 AccessibilityNode node = accessibilityHierarchy.AddNode(s.gameObject.name, null);
    10.                 node.isActive = true;
    11.                 node.role = AccessibilityRole.Button;
    12.                 node.frameGetter = () => GetFrame(s.GetComponent<RectTransform>());
    13.                 node.selected += () => Log(s.gameObject.name, s.GetComponent<StandardAccessibleButton>());
    14.                 Debug.Log("Added node: " + node.label);
    15.             }
    16.  
    17.             accessibilityHierarchy.RefreshNodeFrames();
    18.  
    19.             StartCoroutine(SetAccessibilityHierarchy());
    20.             AssistiveSupport.screenReaderStatusChanged += OnScreenReaderStatusChanged;
    21. #endif
    Thanks
    Harry
     
  12. harry_unity450

    harry_unity450

    Joined:
    May 25, 2023
    Posts:
    3
    Giving this a little bump :)
     
  13. bianca-stana

    bianca-stana

    Unity Technologies

    Joined:
    Jan 26, 2021
    Posts:
    14
    Hi Harry,

    It took me a bit of time to look into this. Due to how the iOS support is implemented, the accessibility hierarchy is only applied if VoiceOver is active.

    As previously mentioned, the current set of Accessibility APIs is designed to enable screen reader support. Switch access on Android is just an unintended side effect.

    I have discussed switch access with my team, and we would greatly appreciate it if you could submit a feature request on our UI roadmap.
     
    Last edited: Sep 14, 2023
  14. bianca-stana

    bianca-stana

    Unity Technologies

    Joined:
    Jan 26, 2021
    Posts:
    14
    Regarding VoiceOver not recognising your buttons – unfortunately, I cannot determine the cause of this issue based on the information provided. However, if you could submit a bug report through the Unity Bug Reporter and include a project with the relevant code, we would be happy to investigate further.
     
    JuliaP_Unity likes this.
  15. harry_unity459

    harry_unity459

    Joined:
    Mar 11, 2023
    Posts:
    8
    Hi Bianca

    I'm running into an issue where-in the scan pattern used by the Switch Access App on a Samsung tablet is not adhering to the hierarchy displayed within Unity.

    Samsung Tablet using Switch Access scan pattern


    The hierarchy is -
    25: "MuteSFXButton"
    26: "ReduceSFXButton"
    27: "IncreaseSFXButton"
    28: "MaximumSFXButton"
    ..followed by the next row in the correct order.

    They are all root nodes.

    On the Google Pixel 6a however, these nodes are scanned in the correct order using Switch Access:


    Samsung's own Switch feature does not recognise any nodes at all.

    Is this is a known issue, or is there some way around this?

    Thanks again,
    Harry
     
  16. bianca-stana

    bianca-stana

    Unity Technologies

    Joined:
    Jan 26, 2021
    Posts:
    14
    Hi Harry,

    As stated previously, the current set of Accessibility APIs focuses on screen reader support; switch access is just an unintended side effect and might not always work as expected.

    Does your issue reproduce with the screen reader as well?
     
  17. harry_unity459

    harry_unity459

    Joined:
    Mar 11, 2023
    Posts:
    8
    Hi Bianca, it did indeed reproduce with the screen reader, in testing that I realised what I'd done. The Google Pixel 6a phone apparently does it's own grouping for Switch Access that was obfuscating the underlying issue.

    Thanks
    Harry
     
  18. bianca-stana

    bianca-stana

    Unity Technologies

    Joined:
    Jan 26, 2021
    Posts:
    14
    Hi Harry,

    The screen reader navigates the accessibility hierarchy in a Depth First Search (DFS) order, so you should be able to determine the navigation order through how you structure your hierarchy. If all nodes are root nodes, the screen reader will navigate them in their order.

    If the screen reader does not respect this order, please submit a bug report through the Unity Bug Reporter so we can investigate the issue.

    I hope this helps!
     
  19. harry_unity450

    harry_unity450

    Joined:
    May 25, 2023
    Posts:
    3
  20. bianca-stana

    bianca-stana

    Unity Technologies

    Joined:
    Jan 26, 2021
    Posts:
    14
    Hi Harry,

    Thank you for letting me know. That's fantastic news! I will share them with my team. :)