Search Unity

  1. Unity 2020.1 has been released.
    Dismiss Notice
  2. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Audio Lightning fast Voice commands for Android and iOS (on-device)

Discussion in 'Audio & Video' started by trungnq97, Jul 28, 2020.

  1. trungnq97

    trungnq97

    Joined:
    Oct 31, 2019
    Posts:
    4
    Dear all,
    Recently I am fascinated by voice technology. My kids (4 and 7) are fans of Google Voice search. It's so fun to ask Google about all kinds of things they are curious about :D

    So I thought it would be cool to use voices to control games! I mean using only short commands like jump, gun, fire, heal, drop, left, right, stop, etc. The commands will need to be recognized extremely fast.

    I did a bit of search on the asset store to find if there is any asset that does this. But I couldn't find one that fits my needs. There are assets that wrap around cloud-based or OS-based speech recognition services. They are great for their use-cases but they could be slow, bandwidth consuming, battery consuming, do not provide a smooth user-experience, could be expensive to scale if cloud-based speech recognition services are required [1].

    So I thought of building a lightning fast on-device voice commands embedded directly inside games. It works everywhere (Android, iOS :D) with zero dependency on OSes speech recognition, internet connections. Scaling as big and as fast as you want when your games attract hundreds of thousands of users. Everything is done on-device, fast and efficient.

    Would be great to hear your thoughts. How should the library be designed to be easy to use and to fit most games. What voice commands will you want first for your games?

    Cheers!

    [1] https://developer.apple.com/documentation/speech/sfspeechrecognizer#2364809
     
  2. Mark_01

    Mark_01

    Joined:
    Mar 31, 2016
    Posts:
    470
    Hi, I am not entirely sure what you are looking for, but I am guessing you what this done on the phone.
    I have Zero coding skills so I can not help ( sorry ) .. But I think it is a cool idea, after a couple of days thinking on
    it in the back of my mind,, I did a search on the store for speech to text. So I am guessing what you could try is
    Use the speech to text .. then the code would see the text " jump " and then it would do the action like
    Speech to text jump .. then jump means move the object in the Y axis .. If i have the thought right.

    https://assetstore.unity.com/packag...d-text-chat-148188?q=speech to text&orderBy=1
     
  3. trungnq97

    trungnq97

    Joined:
    Oct 31, 2019
    Posts:
    4
    Hi Mark_01,
    I am planing to implement a voice command recognition for Unity. The system should be tiny, fast and accurate. It should be cross-platforms (Android and iOS for now). It should not depend on OSes (like Android, iOS) to do speech to text recognition. It can work offline on devices, no internet connection required.

    Speech to text (STT) can be a good start. But STT systems might not be the best fit for the use-case I mentioned. If STT systems use deep neural networks, they tend to be quite big (1GB memory required) [1]. So accurate STTs tend to be hosted on clouds. To make STTs smaller, accuracy is often sacrificed.

    Interestingly, for games, we might need just a small set of predefined commands (20-30 commands for example). We could build a tiny (less than 1MB) voice command recognition system that has characteristics/features I mentioned above.

    I would like to hear thoughts about the idea, different use-cases (for different types of games) and any set of commands that can be practically used. If the idea makes sense and can be useful, I will start implementing it.

    [1] DeepSpeech, opensource automatic speech recognition (ASR) engine from Mozzila. Mozzila did a great job to reduce memory size from 1.8GB to 84MB. https://hacks.mozilla.org/2019/12/deepspeech-0-6-mozillas-speech-to-text-engine/
     
    Mark_01 likes this.
  4. trungnq97

    trungnq97

    Joined:
    Oct 31, 2019
    Posts:
    4
    Mark_01 likes this.
unityunity