Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Hololens app gets stuck during KeywordRecognizer constructor when in a noisy environment

Discussion in 'VR' started by EdgarSantos, Mar 22, 2019.

  1. EdgarSantos

    EdgarSantos

    Joined:
    Nov 11, 2013
    Posts:
    27
    For some time now, we noticed that when we do demos in noisy environments, our app takes a very long time to startup (e.g. sometimes more than 2 minutes). This is not acceptable in most situations (factories, tech fairs).

    After lots of investigation on our side, we managed to consistently reproduce the issue with a very simple application (just Unity, no MRT, no Vuforia, etc) and found out the issue by debugging it.

    Basically, if there's noise (e.g. factory noises, background music) during the KeywordRecognizer instantiation, the app gets stuck until the noise disappears (or gets low enough).
    This happens, at least, with Unity 2017.4 LTS and Unity 2018.3.

    We reproduce the issue by having some loud music playing near the device during startup.
    The app will get stuck not rendering anything. If you stop the music and silence returns, the app continues and
    starts rendering again.

    I submitted a bug report about this but still didn't get any answer.

    Is there any way we can mitigate the issue in the meantime?
    Our demo situations are suffering a lot because of this issue and not having voice commands is out of the question.

    Thank you
     
  2. timke

    timke

    Joined:
    Nov 30, 2017
    Posts:
    407
    Hi,

    Thank you for posting this as we've had a couple other customers reporting very similar issues. From our investigation, we've identified this as a Microsoft bug within Windows speech recognition system on HoloLens. Have your HoloLens devices been updated to RS4 release? We believe this update introduced the bug/regression.

    However, the "noisy environment" aspect is new information for us, which we'll pass on to Microsoft, so thank you for identifying this in your investigation.

    The work around is to directly activate the Windows SpeechRecognizer COM object on a background thread. While this doesn't fix the problem (initialization will still take a long time), it prevents Unity's main thread from being blocked so update and rendering can continue.

    Here's some sample code to implement this work around:
    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4. using UnityEngine.UI;
    5. using UnityEngine.Windows.Speech;
    6.  
    7. public class testscript : MonoBehaviour {
    8.     bool initialized = false;
    9.     bool directInitComplete = false;
    10.  
    11.     public Text text;
    12.     // Use this for initialization
    13.     void Start () {
    14.         InitializeWindowsSpeechDirectly();
    15.     }
    16.  
    17.     // Update is called once per frame
    18.     void Update () {
    19.         if (!initialized && directInitComplete)
    20.         {
    21.             Debug.Log("Creating KeywordRecognizer!");
    22.  
    23.             float timeStart = Time.realtimeSinceStartup;
    24.             var r = new KeywordRecognizer(new string[] { "test-keyword" }, ConfidenceLevel.Low);
    25.             float totalTime = Time.realtimeSinceStartup - timeStart;
    26.             Debug.Log("new KeywordRecognizer took " + totalTime + "s");
    27.  
    28.             if (text != null) text.text = "new KeywordRecognizer took " + totalTime + "s";
    29.  
    30.             r.OnPhraseRecognized += (args) =>
    31.             {
    32.                 Debug.Log("Phrase recognized!");
    33.             };
    34.  
    35.             Debug.Log("Starting KeywordRecognizer!");
    36.             r.Start();
    37.  
    38.             initialized = true;
    39.  
    40.             Debug.Log("Finished initialization!");
    41.         }
    42.     }
    43.  
    44. #if ENABLE_WINMD_SUPPORT
    45.     Windows.Media.SpeechRecognition.SpeechRecognizer _recognizer;
    46. #endif
    47.  
    48.     private void InitializeWindowsSpeechDirectly()
    49.     {
    50. #if ENABLE_WINMD_SUPPORT
    51.  
    52.         Windows.System.Threading.ThreadPool.RunAsync((workItem) =>
    53.         {
    54.             // WinRT APIs will throw exceptions
    55.             try
    56.             {
    57.                 var recogonizer = new Windows.Media.SpeechRecognition.SpeechRecognizer();
    58.  
    59.                 _recognizer = recogonizer;
    60.                 directInitComplete = true;
    61.             }
    62.             catch { };
    63.         });
    64. #else
    65.         directInitComplete = true;
    66. #endif
    67.  
    68.     }
    69. }
    Note this is basically all Unity does; it's a very light wrapper around the Windows API.

    Thank you again for the new information, this helps explain why only some HoloLens users are encountering this problem.