Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

Word Detection - Verbal Commands

Discussion in 'Made With Unity' started by theylovegames, Sep 23, 2012.

  1. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    Yes. There's a detection event with a score. Some of the examples pause the detection for a little bit after each detected word. Using the callback you can do anything.
     
  2. kaplica

    kaplica

    Joined:
    Feb 20, 2014
    Posts:
    87
    Awesome, that's what I needed. Will buy:)

    P.S - Any special discounts for students? :p
     
  3. Avinash

    Avinash

    Joined:
    Mar 25, 2010
    Posts:
    7
    few queries..
    • does it support unity 5?
    • how many languages and accents does it support ?
     
  4. bluesyhfang

    bluesyhfang

    Joined:
    Apr 4, 2014
    Posts:
    8
    Hi theylovegames,

    Thanks for your great job. This asset is the best one I can find for voice recognition for Unity!

    I am working on a project that involves using voice commands to control a camera. Is this possible using your tools? Do you have any similar demos?

    Many thanks in advance!

    Bluesyhfang
     
  5. kaplica

    kaplica

    Joined:
    Feb 20, 2014
    Posts:
    87
    Would you be able to give any example on the word detection thing? I'm not quite sure how to work it out. And the example scene for word detection is a little bit different from what I want.
     
  6. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
  7. jrDev

    jrDev

    Joined:
    Nov 5, 2008
    Posts:
    1,644
    Hello,

    I am interested in your package, one thing I am not clear about is if this detects words that I prerecorded so other people can use it as soon as they open the app and it will be recognized when they say a word. What I am trying to say is this something like the OpenEars plugin but a different way of doing it?

    Thanks,
    jrDev
     
  8. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
  9. FarhanJohari

    FarhanJohari

    Joined:
    May 25, 2015
    Posts:
    6
    is this verbal commands support mobile?
     
  10. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    Yes this package supports mobile iOS and Android.

    I've also expanded the package to work with the Emotiv brain headset.


     
  11. Igor-Frank

    Igor-Frank

    Joined:
    Feb 10, 2015
    Posts:
    2
    Hi there,

    I'm interesting in this package. I did try web samples but it doesn't work as it should i think. First I recorded all the commands with set. And after that when I tried to use it, it doesn't understand me at all. Every word that I said was randomly understand as completely different. So for example if I said "left" it understands "right", "up", "down".

    To be short, if this cannot be extremely optimized than it is not useful for us. Is there a option to get a demo version of asset just to test it if suit our needs?

    Thank you in advance...

    Regards,
    Igor
     
  12. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    Hi Foriero,

    A community contribution! That's great!

    The trouble with waveform matching is that it's a sum of the containing frequencies which makes frequency detection difficult to extract. FFT gives you the probability for a set of frequencies.

    In the end it's all data that can contribute to the scoring and detection. I'm looking into machine learning algorithms to improve the matching. As with all things it's a learning process. A nice feature of the package is that it's full source to extend.

    I will gladly take contributions and add to the next beta.

    +1 (community contribution award)

    Thanks,

    ~Tim Graupmann
     
  13. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    Hi Igor.

    The package is full source. So there's not a demo. I sent you a voucher over PM so you can try it out.
     
  14. jaibe_13

    jaibe_13

    Joined:
    Jan 17, 2016
    Posts:
    1
    i would like to test your plug in verbal commands but my friend already bought it . and he find a problem which is when you hide the gui (game object Bool false and put it in a button) the command stucks and wont trigger the command . but when the gui visible is true , that time the command will execute with out saying the command ?. . is it possible to hide the gui text and still make the command work ? i really need this asset .
     
  15. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    Hi Jaibe, check your inbox as I've sent a PM with a voucher! Let us know how it goes!
     
  16. OlliIllustrator

    OlliIllustrator

    Joined:
    Nov 1, 2013
    Posts:
    48
    Hi, I am new to this asset.
    Just first testing it out after purchase, want to do some tests with VR in the near future.
    One thing that so far nobody has mentioned: seems Unity 5.3 breaks all the demos, the interfaces do not show up (oh how I hate Unity' s indifference towards backwards compatibilty with even point releases ).
    There are also errors regarding the goat mp3 files.
    Installed 5.2 back and mostly ran fine technically, though I sometimes find the demos to be very hard to train and there is problems with the play button of the audio---

    Main question:
    What I have not yet understood properly yet is:
    how do you use the "set" buttons correctly?

    Do you activate them before talking or after talking or while talking?
    My results vary in such a manner that I cannot really tell what is the correct way yet.

    Also in Unity 5.2 while everything else mostly seems to be working, the audio clip play functions does not seem to work at all (in the web demo everything worked, so I assume that the package needs an overhaul for the newer versions.)
    Do you have any schedule for when Unity 5.3 compatibility will be done?

    Oh, and the goat scene seems to be missing stuff, the demo does not work even in 5.2.


    Keep up the good work, this is an fascinating asset with lots of potential.
     
  17. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    HI OlliIllustrator,

    I moved the goat media files to a different package to reduce the size. So if you really need them, there's a link in the docs.
    https://github.com/tgraupmann/TAGENIGMA-Docs/blob/master/Word-Detection.md

    https://github.com/tgraupmann/TAGEN...ata/WordDetection_GoatMediaClips.unitypackage

    I have a 5.3 beta that I can send to you. Just shoot me a link. support at theylovegames.com

    The webplayer demos need to be reexported for WebGL since the web player is being retired.

    The demo buttons are push to talk. So you press the button and hold, say the word, and then release.

    Thanks,

    ~Tim Graupmann
     
  18. vandcurrent

    vandcurrent

    Joined:
    Oct 27, 2015
    Posts:
    7
    hi there,
    i'm totaly newbie at unity, i had already bought your plugin. in my case, i want to make an android application which can use a voice command to show the information of the model. so for example, i said chair, and there will an information about the chair in text. can u help me?

    i already running your example 17, and theres no save and load for any voice i already input. and how my aplication automaticaly choose the mic from android.

    thanks before.
     
  19. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    The same examples should work on Android. Your Android app has a data folder which you can save the profiles in. Microphone selection is the same as PC. However, if Unity can't detect your Microphone it might not be listed. What device are you using?
     
  20. vandcurrent

    vandcurrent

    Joined:
    Oct 27, 2015
    Posts:
    7
    i'm using nexus 5, and i didnt built into apk yet. just try it at unity.

    can i simply put code from example 4 to example 17, the code is for save and load profile?
     
  21. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    Yes the code from each example should be interchangeable, given that you match the field and parameter names.
     
  22. vandcurrent

    vandcurrent

    Joined:
    Oct 27, 2015
    Posts:
    7
    hi there,
    i've been try to built an apk app, but my saved didn't include to the apk package.
    so i must set the voice command from the begining if i install the apk to the new phone.

    theres a way that if i built an apk and my saved profile can be loaded automatically?
    and how auto select the mic? so if i start the app, i just can use the voice command without any setting about noise etc.
    so if someone use my apk he didnt must set any voice command.

    thanks
     
  23. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    You can load your profile in the Start event of a MonoBehaviour script. You can put the profile into a Resources or SteamingAssets folder.
    http://docs.unity3d.com/Manual/StreamingAssets.html

    You could put the profile into an AssetBundle.

    The profile is just an AudioClip and getting the data from the AudioClip.

    You could even put the profile in "Assets/Plugins/assets" so it gets bundled into the APK.

    You should be able to unzip the APK and see your profile in the "assets" folder.
     
  24. vandcurrent

    vandcurrent

    Joined:
    Oct 27, 2015
    Posts:
    7
    i'll try to put it into assets folder, but how about auto select microphone?
    so i didn't have to click the microphone list at the begining.

    thanks b4
     
  25. vandcurrent

    vandcurrent

    Joined:
    Oct 27, 2015
    Posts:
    7
    ah, i already found it. its in examples.cs, just comment the gui script.
     
  26. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
  27. vandcurrent

    vandcurrent

    Joined:
    Oct 27, 2015
    Posts:
    7
    did my code is right to use assets stream

    load profile:
    try
    {
    string spath = "jar:file://" + Application.dataPath + "!/assets/";
    string path = string.Format(spath + "{0}_{1}.profile", GetType().Name, command);
    //string path = string.Format("Assets/Plugins/Assets/{0}_{1}.profile", GetType().Name, command);
    if (File.Exists(path))
    {
    using (
    FileStream fs = File.Open(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite)
    )
    {
    using (BinaryReader br = new BinaryReader(fs))
    {
    AudioWordDetection.LoadWord(br, details);
    //Debug.Log(string.Format("Loaded profile: {0}", path));
    details.Label = command;
    }
    }
    }
    else
    {
    Debug.Log(string.Format("Profile not available for: {0}", path));
    }
    }

    save profile:
    string fpath = string.Format("{0}_{1}.profile", GetType().Name, details.Label);
    string path = System.IO.Path.Combine(Application.streamingAssetsPath + "/" + fpath);
    string result = "";
    IEnumerator Example() {
    if (filePath.Contains("://")) {
    WWW www = new WWW(path);
    yield return www;
    result = www.text;
    }
    }

    in the load profile no code error, but in save profile there's an error CS1525, CS8025, and CS 1027 at IEnumerator.

    NB : i have using system.Collection;
     
  28. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    Reading via WWW is going to work. I don't think you'll be able to write to that path on iOS? You'll have to find the iOS documentation of how to get write access to the file system or data folder for iOS?
     
  29. vandcurrent

    vandcurrent

    Joined:
    Oct 27, 2015
    Posts:
    7
    no, i didn't wrote on iOS, i wrote it on android, in the url u gave, there's a code to make a steramingAssets used:
    path = "jar:file://" + Application.dataPath + "!/assets/";

    and to access it using streamingAssetsPath which code is:
    using UnityEngine;
    using System.Collections;

    public class ExampleClass : MonoBehaviour {
    public string filePath = System.IO.Path.Combine(Application.streamingAssetsPath, "MyFile");
    public string result = "";
    IEnumerator Example() {
    if (filePath.Contains("://")) {
    WWW www = new WWW(filePath);
    yield return www;
    result = www.text;
    } else
    result = System.IO.File.ReadAllText(filePath);
    }
    }

    and when i implemented the streamingAssetsPath there's alaways error.
    in my opinion, the streamingAssetsPath can't be used at filestream proccess. i'm i right?
     
  30. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    It's likely you can write to `Application.dataPath` but you will need to add write permissions to your android manifest.
     
  31. andresgv12

    andresgv12

    Joined:
    Feb 25, 2016
    Posts:
    1
    Hey I have a problem, when I run an example the mic selection it says count=1 the microphones doesn't appear on the selection only the message "count=1"

    Please help
     
  32. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    I bet this has to do with your Microphone name being reported as an empty string. If you email support@theylovegames.com I'll send you the latest beta which fixes this issue.
     
  33. Nana_Nasiri

    Nana_Nasiri

    Joined:
    Jan 9, 2016
    Posts:
    2
    Is there any way that I can use Word Detection to drive a character controller for free in my game?
     
  34. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    Email support@theylovegames.com and I'll grant you a voucher code. I have a limit of 12 vouchers that I can generate per year. So there is a limited supply.
     
    Last edited: Mar 2, 2016
  35. oOoANDOSoOo

    oOoANDOSoOo

    Joined:
    Jun 17, 2013
    Posts:
    28
    Hi, great asset. i modified the current release to try and get it working with the latest version of Unity (5.3.4), but I can see here in the forums that you have a Beta of the next release. Is it possible to get a copy of that so i can check it out, we've just started moving onto Voice Detection on our game and I really need to check if this asset we bought will hold up for our needs.

    Cheers,
    -Andy.
     
  36. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    Hi Andy. Email support@theylovegames.com to get a copy of the beta.

    I've been refactoring for a simple API and redoing all the examples in the new Unity GUI and examples in WebGL. I'll have a new beta available in the next couple weeks in Unity 5.3.4.
     
  37. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    Last edited: Apr 6, 2016
  38. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    In the meantime, I can update each example to use the new Unity GUI and rebuild for Web Player. Because it's Web Player, it probably only works in Firefox.

    WordDetection - Example 1 - Detection Mic
    http://theylovegames.com/WordDetection_1_1.html

    This example captures raw wave data from the Microphone and displays the Spectrum.
     
  39. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    Here's a preview of the example code with the new API.



    WordDetection - Example 1 - Detection Mic
    http://theylovegames.com/WordDetection_1_1.html

    Code (CSharp):
    1. using UnityEngine;
    2.  
    3. namespace UnityWordDetection
    4. {
    5.     /// <summary>
    6.     /// Display two graphs
    7.     /// 1) Display the microphone wave data
    8.     /// 2) Display the spectrum analysis of the wave data
    9.     /// </summary>
    10.     public class Example : MonoBehaviour
    11.     {
    12.         /// <summary>
    13.         /// The number of cubes to use in each graph
    14.         /// </summary>
    15.         const int PLOT_SIZE = 512;
    16.  
    17.         /// <summary>
    18.         /// The Y position of the wave graph
    19.         /// </summary>
    20.         const int POSITION_GRAPH_WAVE_Y = -100;
    21.  
    22.         /// <summary>
    23.         /// The Y position of the spectrum graph
    24.         /// </summary>
    25.         const int POSITION_GRAPH_SPECTRUM_Y = 200;
    26.  
    27.         /// <summary>
    28.         /// The scale for cubes of the wave graph
    29.         /// </summary>
    30.         const float SCALE_GRAPH_WAVE = 500f;
    31.  
    32.         /// <summary>
    33.         /// The scale for cubes of the spectrum graph
    34.         /// </summary>
    35.         const float SCALE_GRAPH_SPECTRUM = 10f;
    36.  
    37.         /// <summary>
    38.         /// The color bias for the wave graph
    39.         /// </summary>
    40.         const float COLOR_BIAS_GRAPH_WAVE = 10f;
    41.  
    42.         /// <summary>
    43.         /// The color bias for the spectrum graph
    44.         /// </summary>
    45.         const float COLOR_BIAS_GRAPH_SPECTRUM = 0.1f;
    46.  
    47.         /// <summary>
    48.         /// A graph of wave data
    49.         /// </summary>
    50.         private Graph _mGraphWavePlot = new Graph();
    51.  
    52.         /// <summary>
    53.         /// A graph of spectrum data
    54.         /// </summary>
    55.         private Graph _mGraphSpectrum = new Graph();
    56.  
    57.         /// <summary>
    58.         // The data set holds the wave and spectrum data
    59.         /// </summary>
    60.         private MicrophoneDataSet _mMicrophoneDataSet = new MicrophoneDataSet();
    61.  
    62.         /// <summary>
    63.         /// Set up the graphs to display wave and spectrum data
    64.         /// </summary>
    65.         void Awake()
    66.         {
    67.             // create the wave graph
    68.             _mGraphWavePlot.Init(transform, PLOT_SIZE, POSITION_GRAPH_WAVE_Y);
    69.  
    70.             // create the spectrum graph
    71.             _mGraphSpectrum.Init(transform, PLOT_SIZE, POSITION_GRAPH_SPECTRUM_Y);
    72.         }
    73.  
    74.         /// <summary>
    75.         /// Display microphone data on a fixed update
    76.         /// </summary>
    77.         void FixedUpdate()
    78.         {
    79.             if (null == DetectionMicrophone.Instance._mWordDetectionMicrophone)
    80.             {
    81.                 // microphone hasn't been selected
    82.                 return;
    83.             }
    84.  
    85.             // read the microphone data
    86.             DetectionMicrophone.Instance.GetData(_mMicrophoneDataSet);
    87.  
    88.             // draw the wave
    89.             _mGraphWavePlot.DrawWave(_mMicrophoneDataSet, SCALE_GRAPH_WAVE, Color.red, COLOR_BIAS_GRAPH_WAVE, false);
    90.  
    91.             // draw the spectrum
    92.             _mGraphSpectrum.DrawSpectrum(_mMicrophoneDataSet, SCALE_GRAPH_SPECTRUM, Color.green, COLOR_BIAS_GRAPH_SPECTRUM, true);
    93.         }
    94.     }
    95. }
    96.  
     
    Last edited: Apr 8, 2016
  40. mcmount

    mcmount

    Joined:
    Nov 15, 2015
    Posts:
    3
    Hi, I just bought the plugin and I'm having problems with the demos. Step-by-step for the demos (atleast one) would be good, for example mentioning how the buttons work.
    Anyway, I'm running the demo "example 7" and figured out you have to push and hold when teaching the words. The problem is, if other words has been recorded, and you record new command holding the button, the previously recorded buttons activate as well for a very short time (presumably assuming I'm saying those words also). This makes it impossible to run the demo with own words. I'm running Unity 5.3.4p1, windows 10.

    Is there something I'm doing wrong?
     
  41. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    Thanks for the suggestion! I'm in the process of refactoring and redoing all the examples in the new GUI. Adding pitch detection (ignoring octaves) sounds like an excellent feature.
     
  42. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    In the later examples, there is push to talk where detection is suspended while recording a new word. I'm in the process of refactoring everything to make the API easier and to add step by step tutorials.
     
  43. Ahsan-Khan-Unity

    Ahsan-Khan-Unity

    Joined:
    Jun 10, 2016
    Posts:
    2
    please make a detail demo on how its works. got stuck at select the microphone count=0;
     

    Attached Files:

  44. Ahsan-Khan-Unity

    Ahsan-Khan-Unity

    Joined:
    Jun 10, 2016
    Posts:
    2
    can you give me your email?
     
  45. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    Feel free to email support@theylovegames.com and I'll send you the update beta which might fix your issue. Sometimes Microphones have a blank name.
     
  46. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    799
    I'm a bit confused over the use case of this - does this work out of the box for new end users or does it require that the user do a voice training mode to record their own voice saying each command?
     
  47. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    The end user needs to record a word for each voice command (training). The package is designed to work offline without an Internet connection.
     
  48. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    799
    Has this been tested to work on iOS and Android?
     
  49. theylovegames

    theylovegames

    Joined:
    Aug 18, 2012
    Posts:
    172
    Both are supported.
     
  50. Ice_Time

    Ice_Time

    Joined:
    Jun 29, 2016
    Posts:
    3
    Can you write a document to explain the parameters,for example "CaptureTime","SampleRate","Threshold" and so on.If we change the value,what will happen.
    Thank you very much