Question Trying to make a character move the mouth(blenshape) only when there is sound in audioclip

Hello guys,

I hope you all are doing great!

I created a script that get audioSorce.clip samples to make the NPC move their mouth through blenshape only when the audioclip level is above certain range. The target of this game is WebGL. This is for lipsynch purpose. Here is the script:

Code (CSharp):

1. using System.Collections;
2. using UnityEngine;
3.  
4. public class DetectSilenceInClip : MonoBehaviour
5. {
6.     public SkinnedMeshRenderer meshRend;
7.     public int blendshapeIndex = 0;
8.     [Range(20.0f, 100.0f)]
9.     public float minBlend = 30.0f, maxBlend = 50.0f;
10.  
11.     public AudioSource audioSource;
12.     [Tooltip("How low the sound should be in order to be considered silence")]
13.     public float silenceThreshold = 0.05f;
14.     [Tooltip("How long the silence should be to be considered a pause in the sound")]
15.     public float silencePeriodThreshold = 0.03f;
16.  
17.     float[] fSamples = new float[1024];
18.     bool isPlaying = false;
19.     bool inside = false;
20.     bool talking = true;
21.     private float timeSinceLastNonSilentData = 0.0f;
22.  
23.  
24.     private void Update()
25.     {
26.         isPlaying = audioSource.isPlaying;
27.     }
28.  
29.     private void OnEnable()
30.     {
31.         EventManager.StartListening(EventsInProject.EnterCollider, OnEnter);
32.         EventManager.StartListening(EventsInProject.ExitCollider, OnExit);
33.         EventManager.StartListening(EventsInProject.LessonIsDone, OnLessonDone);
34.         EventManager.StartListening(EventsInProject.NextLessonContent, OnNextLesson);
35.     }
36.  
37.     private void OnDisable()
38.     {
39.         EventManager.StopListening(EventsInProject.EnterCollider, OnEnter);
40.         EventManager.StopListening(EventsInProject.ExitCollider, OnExit);
41.         EventManager.StopListening(EventsInProject.LessonIsDone, OnLessonDone);
42.         EventManager.StopListening(EventsInProject.NextLessonContent, OnNextLesson);
43.     }
44.  
45.     private void OnEnter()
46.     {
47.         inside = true;
48.         StartCoroutine(AnalyzeAudioData());
49.     }
50.  
51.     private void OnExit()
52.     {
53.         inside = false;
54.         StopCoroutine(AnalyzeAudioData());
55.         ResetBlendshapes();
56.     }
57.  
58.     void OnNextLesson()
59.     {
60.  
61.     }
62.  
63.     IEnumerator AnalyzeAudioData()
64.     {
65.         while (inside)
66.         {
67.             if (isPlaying)
68.             {
69.                 audioSource.clip.GetData(fSamples, audioSource.timeSamples);
70.  
71.                     for (int i = 0; i < fSamples.Length - 1; i++)
72.                     {
73.                     if (Mathf.Abs(fSamples[i]) <= silenceThreshold)
74.                     {
75.                         timeSinceLastNonSilentData += Time.deltaTime;
76.                         if (timeSinceLastNonSilentData >= silencePeriodThreshold)
77.                         {
78.                             if (talking)
79.                             {
80.                                 talking = false;
81.                                 Talk(false);
82.                                 ResetBlendshapes();
83.                             }
84.                         }
85.                     }
86.                     else
87.                     {
88.                         if (!talking)
89.                         {
90.                             talking = true;
91.                             Talk(true);
92.                             timeSinceLastNonSilentData = 0.0f;
93.                         }
94.                     }
95.                 }
96.             }
97.  
98.             yield return null;
99.         }
100.     }
101.  
102.  
103.     void OnLessonDone()
104.     {
105.         ResetBlendshapes();
106.     }
107.  
108.     void Talk(bool open)
109.     {
110.         if (open)
111.         {
112.             meshRend.SetBlendShapeWeight(blendshapeIndex, UnityEngine.Random.Range(minBlend, maxBlend));
113.         }
114.         else
115.         {
116.             ResetBlendshapes();
117.         }
118.     }
119.  
120.     void ResetBlendshapes()
121.     {
122.         meshRend.SetBlendShapeWeight(blendshapeIndex, 0.0f);
123.     }
124. }
125.  

This script works perfectly in the Editor when I play the game. But when I build in WebGL, The game freezes and shows me these errors:



Does anyone have any ideas about what could be wrong in my script that crashes the WebGL app?

Any help would be very much appreciated!

Regards,
Carlos

 

Looks like a memory limit issue. WebGL is very limited on that front - profile the app or strip down assets until it runs.

 

Hello KamilCSPS,

Thank you for your input. Debugged this many times, and the crash points to this line of code in the script:

audioSource.clip.GetData(fSamples, audioSource.timeSamples);

The problem seam to be is more like fSamples length is different to audioSource timeSamples. Creating an out of bound access error.

Regards,
Carlos

 

Sounds like a bug then.

I would suggest you file a bug report and then tag unityruba (very active Unity dev on the WebGL forums) to take a quick look.

 

I really appreciate you input KamilCSPS, but after all my research I do not believe this is a bug, but that the issue is that I am using the wrong way to do this for WebGL. That is why I show my script and errors. Perhaps some other developer has found the correct way to do this for WebGL and can shine a light on what the correct way to achieve this in WebGL is. The idea is that my character has a single blenshape openning the mouth. And it should activate this blenshape only when the audio clip sample is above certain level, or range, of value how ever you want to call it.

Regards,
Carlos

 

The fact that it works in-editor but not when exported points to a bug or else it would had been documented by Unity.

At the very least, Unity tends to gives warnings about out-of-platform usage of something.

In any case, a Unity Dev will help you either way - and a bug will better scope the out of platform usage in their docs. I see it as a win-win. Notwithstanding someone else coming along to help out of course.