WebGL Lipsyncing

Hello all,

I'm currently in the processes of porting our project over to WebGL. I've hit a couple snags along the way to say the least, but the largest so far is getting our game's lip-syncing working. We use a very simple algorithm to calculate the lip-syncing at run time, similar to the answer here. The problem is that in WebGL the function AudioSource.GetSpectrumData() isn't supported.

I've started trying to replicate the function by using the Web Audio API with no success yet. I've started with writing the following native JavaScript to try and get the frequency data.

Is this the right approach to be taking? Can I get the same sample data that GetSpectrumData gives me from the Web Audio API? Please let me know if I'm completely wrong with this, I know little to nothing emscripten or any kind of audio engineering.

Code (JavaScript):

1. var LipsyncJS = {
2.     GetLipsyncSamples: function(array, size) {
3.         var analyzer = null;
4.         var source = null;
5.      
6.         try {
7.             var buffer = new Uint8Array(Module.HEAPU8.buffer, array, Float32Array.BYTES_PER_ELEMENT * size);
8.             buffer = new Float32Array(buffer.buffer, buffer.byteOffset, size);
9.      
10.             if(typeof WEBAudio != 'undefined' && WEBAudio.audioInstances.length > 1) {
11.                 source = WEBAudio.audioInstances[1].source;
12.                 analyzer = source.context.createAnalyser();
13.                 source.connect(analyzer);
14.              
15.                 analyzer.getFloatTimeDomainData(buffer);
16.             }
17.         }
18.         catch(e) {
19.             console.log("Failed to get lipsync sample data" + e);
20.         }
21.         finally {
22.             if(analyzer != null && source != null) {
23.                 source.disconnect(analyzer);
24.             }
25.         }
26.     }
27. };
28.  
29. mergeInto(LibraryManager.library, LipsyncJS);

Thanks a lot!

 

For those of you curious and want to get GetSpectrumData working on WebGL, I was able to solve this. All be it, with a slightly more complicated JavaScript than before. It doesn't return carbon copy what GetSpectrumData does, but with some tweaking you can get the data the right way. Unfortunately, like I said before, I'm terrible with all things audio and a lot of this is over my head. I imagine this would be fairly easy to build into the Unity engine, but for now here's what I used:

EDITED: 4/25/16

Spoiler: JSLib Code

Code (JavaScript):

1.  
2. /**
3.  * Functions and logic to get lipsync data from a currently playing source, by setting up an AnalyserNode to watch
4.  * a source in the Web Audio API. This will not work in browsers that do not use the Web Audio API (IE).
5.  */
6. var LipsyncJS = {
7.     /** Contains all the currently running analyzers */
8.     $analyzers: {},
9.    
10.     /**
11.     * Call to start a sampler and analyze the stream, close the stream later to prevent memory leaks
12.     * @param namePtr string pointer to the name of the analyzer to start (to reference when getting a sample or closing)
13.     * @param duration the duration of the audio clip to find (clips can't be found by name)
14.     * @param bufferSize the size of the analyzer buffer to use for sampling
15.     * @returns true if the analyzer was setup, false if otherwise
16.     */
17.     StartLipSampling: function(namePtr, duration, bufferSize) {
18.         //the amount of error acceptable between the audio instance and the duration provided
19.         var acceptableDisantce = 0.075;
20.        
21.         //the name of the analyzer to start
22.         var name = Pointer_stringify(namePtr);
23.        
24.         //final analyzer and audio source
25.         var analyzer = null;
26.           var source = null;
27.          
28.           try {        
29.               //if the audio instances can be found
30.               if(typeof WEBAudio != 'undefined' && WEBAudio.audioInstances.length > 1) {
31.                   //iterate through each instance and look for sources
32.                 for(var i=WEBAudio.audioInstances.length-1; i>=0; i--) {
33.                     if(WEBAudio.audioInstances[i] != null) {
34.                         var pSource = WEBAudio.audioInstances[i].source;
35.                        
36.                         //if the instance has a source and a duration and is in the margin of error then it's our source
37.                         if(pSource != null && pSource.buffer != null && Math.abs(pSource.buffer.duration - duration) < acceptableDisantce) {
38.                             source = pSource;
39.                             break;
40.                         }
41.                     }
42.                 }
43.                
44.                 //if no source found
45.                 if(source == null) {
46.                     return false;
47.                 }
48.                
49.                 //create an analyzer and connect it to the source to start getting frequency data
50.                   analyzer = source.context.createAnalyser();
51.                 analyzer.fftSize = bufferSize * 2;
52.                   source.connect(analyzer);
53.                
54.                   //save the analyzer in the analyzers list
55.                 analyzers[name] = {analyzer:analyzer, source:source};
56.                
57.                 return true;
58.               }
59.           }
60.           catch(e) {
61.               console.log("Failed to connect analyser to source " + e);
62.            
63.               //disconnect if an error
64.             if(analyzer != null && source != null) {
65.                   source.disconnect(analyzer);
66.               }
67.           }
68.        
69.         return false;
70.     },
71.    
72.     /**
73.     * Close an analyzer that is currently looking at a source
74.     * @param namePtr string pointer to the name of the analyzer to close
75.     * @returns true if the analyzer was found and closed
76.     */
77.     CloseLipSampling: function(namePtr) {
78.         //find the analyzer
79.         var name = Pointer_stringify(namePtr);
80.         var analyzerObj = analyzers[name];
81.        
82.         //if the analyzer is found disconnect it from the source and delete it from the running analyzers
83.         if(analyzerObj != null) {
84.             try {
85.                 analyzerObj.source.disconnect(analyzerObj.analyzer);
86.                 delete analyzers[name];
87.                 return true;
88.             }
89.             catch(e) { console.log("Failed to disconnect analyser " + name + " from source " + e); }
90.         }
91.        
92.         return false;
93.     },
94.    
95.     /**
96.     * Get the current frequency data from a running analyzer
97.     * @param namePtr string pointer to the name of the analyzer to use
98.     * @param bufferPtr float[] pointer to the array to populate with the frequency data
99.     * @param bufferSize the length of the float[] buffer
100.     * @returns true if the samples were retrieved without error, false if otherwise
101.     */
102.     GetLipSamples: function(namePtr, bufferPtr, bufferSize) {
103.           try {
104.               //get a float array from the emscripten buffer pointer
105.               var buffer = new Uint8Array(Module.HEAPU8.buffer, bufferPtr, Float32Array.BYTES_PER_ELEMENT * bufferSize);
106.               buffer = new Float32Array(buffer.buffer, buffer.byteOffset, bufferSize);
107.            
108.               //get the analyzer
109.             var name = Pointer_stringify(namePtr);
110.             var analyzerObj = analyzers[name];
111.            
112.             if(analyzerObj == null) {
113.                 console.log("Could not find analyzer " + name + " to get lipsync data for");
114.                 return false;
115.             }
116.            
117.             //get the frequency data (similar to AudioSource.GetSpectrumData in Unity)
118.             analyzerObj.analyzer.getFloatTimeDomainData(buffer);
119.            
120.             return true;
121.           }
122.           catch(e) {
123.               console.log("Failed to get lipsync sample data " + e);
124.           }
125.        
126.         return false;
127.     }
128. };
129.  
130. //add to the js scope
131. autoAddDeps(LipsyncJS, '$analyzers');
132. mergeInto(LibraryManager.library, LipsyncJS);
133.  

It work's kinda like this:

Spoiler: CSharp Code

Code (CSharp):

1.  
2. public class Lipsync : MonoBehaviour {
3. //only works in WebGL it is important to not use in the editor as the js native code can't be called in Editor
4. #if UNITY_WEBGL && !UNITY_EDITOR
5.  
6.     [DllImport("__Internal")]
7.     private static extern bool StartLipSampling(string name, float duration, int bufferSize);
8.  
9.     [DllImport("__Internal")]
10.     private static extern bool CloseLipSampling(string name);
11.  
12.     [DllImport("__Internal")]
13.     private static extern bool GetLipSamples(string name, float[] freqData, int size);
14.  
15.     private AudioSource source;
16.  
17.     private float[] freqData = new float[256];
18.  
19.     private void Update() {
20.         //if starting
21.         StartLipSampling(name, source.clip.length, 256);
22.      
23.         //if audio stopped
24.         CloseLipSampling(name);
25.      
26.         //when getting the audio data (kinda like GetSpectrumData)
27.         GetLipSamples(name, freqData, freqData.length);
28.     }
29. #endif
30. }
31.  

 

Thanks for the solution. I was looking for a workaround for GetSpectrumData to work in my WebGl app which is converted from webplayer.
I'm new to JS. What does this below 2 lines of code do?
autoAddDeps(LipsyncJS, '$analyzers');
mergeInto(LibraryManager.library, LipsyncJS);

Also how do you pass the data back to unity?

Thanks!

 

The lines:

Code (JavaScript):

1. autoAddDeps(LipsyncJS, '$analyzers');
2. mergeInto(LibraryManager.library, LipsyncJS);

I believe are for telling the compiler to include the code in the WebGL JavaScript file after compilation and it links the code so that you can call the functions from C#.

The DllImport's in my C# code above link directly with the functions of the same name in the JavaScript. Whatever those JavaScript functions return (depending on the data type) is returned to Unity.

Take a look at this, it's pretty helpful little guide. Also, Jonas created a little library for websockets that I found a good example.

 

Hey, Picky! I am really interested in getting Webgl Lipsync working on my game. Is there any way you can tell me how to implement this, or make a guide? I'm totally clueless about this stuff.

 

Yeah I was totally clueless when I started too. Lipsyncing is an absolute pain in any platform. I'll try and help as best I can.

Take a look at my second post, it has most of the code that you'll need to run it. I'd also like to note: if you're considering doing high quality lipsync, this is not the way to go. You'll probably, want to use some pre-parser like the ones described here. This is a quick and dirty lipsync, where quality isn't really an issue. I could try and make a sample project at some point, but I don't have a ton of time for that unfortunately. Here's the basic routine that you should follow:

• Refer to this guide on how to setup a WebGL plugin.
• Refer to this on how to setup this type of lipsyncing. Your code will look something like this (NOTE: I just wrote this quick I didn't actually test it and offer no guarantees on it actually working):

Spoiler: C# Code

Code (CSharp):

1. public class Lipsync : MonoBehaviour {
2. //only works in WebGL it is important to not use in the editor as the js native code can't be called in Editor
3. #if UNITY_WEBGL && !UNITY_EDITOR
4.     /// <summary>The number of samples to take at a time</summary>
5.     public const int NUM_SAMPLES = 1024;
6.  
7.     /// <summary>The maximum frequency</summary>
8.     public const float MAX_FREQUENCY = 24000;
9.  
10.     [Tooltip("The mouth object to move")]
11.     public GameObject mouth;
12.  
13.     [Tooltip("The audio source to monitor")]
14.     public AudioSource source = null;
15.  
16.     [Tooltip("The low range of the frequency to limit to")]
17.     public float fLow = 200;
18.  
19.     [Tooltip("The high range of the frequency to limit to")]
20.     public float fHigh = 800;
21.  
22.     public float volume = 40;
23.  
24.     /// <summary>The frequency data from every update (not used after update, but kept around to be more memory efficent)</summary>
25.     private float[] freqData;
26.  
27.     private float y0;
28.  
29.     private void Start() {
30.         y0 = mouth.transform.position.y;
31.         freqData = new float[NUM_SAMPLES];
32.         source.Play();
33.      
34.         //start sampling (this probably won't work, you might have to wait a couple frames before it becomes available)
35.         StartLipSampling(name, source.clip.length, NUM_SAMPLES);
36.     }
37.  
38.     private float BandVol() {
39.         // limit low...
40.         float fLow = Mathf.Clamp(fLow, 20, MAX_FREQUENCY);
41.  
42.         // and high frequencies
43.         float fHigh = Mathf.Clamp(fHigh, fLow, MAX_FREQUENCY);
44.  
45.         int n1 = (int) Mathf.Floor(fLow * NUM_SAMPLES / MAX_FREQUENCY);
46.         int n2 = (int) Mathf.Floor(fHigh * NUM_SAMPLES / MAX_FREQUENCY);
47.         float sum = 0;
48.  
49.         // average the volumes of frequencies fLow to fHigh
50.         for(int i = n1; i <= n2; i++) {
51.             sum += Math.Abs(freqData[i]);
52.         }
53.  
54.         return sum / (n2 - n1 + 1);
55.     }
56.  
57.     private void Update() {
58.         //get the samples at this frame
59.         GetLipSamples(name, freqData, freqData.Length)
60.          
61.         //move the object
62.         mouth.transform.position.y = y0 + BandVol() * volume;
63.     }
64.  
65.     /// <summary>Close on destroy</summary>
66.     private void OnDestroy() {
67.         CloseLipSampling(name);
68.     }
69.     [DllImport("__Internal")]
70.     private static extern bool StartLipSampling(string name, float duration, int bufferSize);
71.     [DllImport("__Internal")]
72.     private static extern bool CloseLipSampling(string name);
73.     [DllImport("__Internal")]
74.     private static extern bool GetLipSamples(string name, float[] freqData, int size);
75.     private AudioSource source;
76.     private float[] freqData = new float[256];
77.     private void Update() {
78.         //if starting
79.         StartLipSampling(name, source.clip.length, 256);
80.    
81.         //if audio stopped
82.         CloseLipSampling(name);
83.    
84.         //when getting the audio data (kinda like GetSpectrumData)
85.         GetLipSamples(name, freqData, freqData.length);
86.     }
87. #endif
88. }

• Setup a model with a mouth that has two or more bones to move apart and use some of the above code to calculate it. The above code just move a single GameObject up and down.

If you have any specific problems I can try and help, but I don't have the time to create an entire guide / framework.

-Picky

 

Appreciate the code sample. I am trying to use this currently and the system is unable to find the analyzer: "Failed to connect analyser to source TypeError: Cannot read property 'source' of null; Could not find analyzer MentorAudioSource to get lipsync data for". Is there any caveats that haven't been documented? What exactly is name parameter? It looks like its the gameObject name, or is it the name of the clip? Also, does the clip have to be in StreamingAssets/ or is it okay if it's played internally?

A little background:
My script component gets audioSource from another object IE SoundManager, so not attached to called, specifically. I have tried the following

in Start:

Code (csharp):

1.  
2. audioSource = soundEngine.mentorAudioSource.GetComponent<AudioSource>();
3.  

1. StartLipSampling(audioSource.gameObject.name, audioSource.clip.length, numSamples);
2. StartLipSampling(name, audioSource.clip.length, numSamples);
3. StartLipSampling(audioSource.clip.name, audioSource.clip.length, numSamples);
4. StartLipSampling(soundEngine.mentorAudioSource.name, audioSource.clip.length, numSamples);

None of which seem to work, really. Audio plays fine but nothing on the backend. However, with 4, I do get something back as his mouth does move for first .5 sec and then nothing. However, still reporting the same errors as above. Appreciate any insight you can provide. =D

 

The name is just a way to distinguish one analyzer from another. So if you're running multiple lipsyncs you can manage them.

Did you use the updated jslib in the first post? I updated it to account for that null error last time I posted. You may have to wait a few frames before starting the sampling, it can take a a little bit to show up in the audio instances. Make sure that StartLipSampling returns true before you start attempting to get samples.

 

@Picky Salamander, thank you. That seemed to do the trick. I was yielding a frame and then checking previously, however, like you said, you have to wait a good deal longer. I essentially wait until I receive a true back from the call. Thanks for the reply.

 

Dude, you are the greatest! I'm going to check this out and see if I can make some magic happen. Thanks a lot! (P.S, if you made this a Unity Asset I know some people, like myself, would totally buy it.)