Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

SALSA Lipsync Suite - lip-sync, emote, head, eye, and eyelid control system.

Discussion in 'Assets and Asset Store' started by Crazy-Minnow-Studio, Apr 23, 2014.

  1. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Hello chasehek,

    Please remove [Eyelashes] from [Other SkinnedMeshRenderers] in the [CM_FuseSync] inspector. Does this resolve the issue?

    Thanks,
    Michael
     
  2. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838


    After the launch of the IBM Watson Unity SDK, we were curious about how the text-to-speech service might work with SALSA to deliver real-time text-to-speech-based lip-sync. This can already be achieved with the wonderful RT-Voice asset from Crosstales, but we take pride in SALSA working with as many solutions as possible to make our customers options as broad as possible. After signing up for a free account and having a look at the documentation, we found the IBM Watson text-to-speech service simple to use and a great pairing with SALSA lipsync.

    There is no special handling required to make the IBM Watson text-to-speech service work with SALSA, simply output the generated AudioClip to the same AudioSource component that SALSA references and you're good to go. Check our blog post for IBM Watson Unity SDK sample code.

    https://crazyminnowstudio.com/posts/ibm-watson-and-salsa-lipsync/
     
  3. Firlefanz73

    Firlefanz73

    Joined:
    Apr 2, 2015
    Posts:
    1,050
    Sounds interesting. Are there sample Video Clips?
     
  4. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Not yet, but we'll get a demonstration video posted when time permits.
     
  5. Yaakov_Shahak

    Yaakov_Shahak

    Joined:
    Oct 15, 2017
    Posts:
    3
    Hi. We are using Salsa in conjunction with morph3d and we do the setup with the one-click component. Everything is working great as excepted, but we have an issue with the 'RandomEyes3D' component.
    The issue is that we have a scene with 2 cameras for two modes. one camera is assigned to Ar mode (with Vuforia), and the main one is assign to 3d mode. We have 2 avatars on the scene, one for each mode. And we toggeling between the two modes.
    The issue is that, if the toggle between two modes is made in a moment of blinking (with RandomEyes3D), then when we switch back, the avatar is stuck in the blinking position forever, means his eyelids is closed.
    We tried a lot of approaches for that, but for vain. Can there is any hints about how we can fix that?
     
  6. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Hello Yaakov_Shahak,

    Is it possible to put together a minimal example project that contains only enough to reproduce the symptom, and send it or share it, along with your SALSA invoice number, to assetsupport@crazyminnow.com?
     
  7. chong_dev

    chong_dev

    Joined:
    Feb 28, 2018
    Posts:
    1
    Hi
    I have purchased it now but it is not showing "Crazy Minnow Studio" on Window after I imported package.
    What is the reason?
     

    Attached Files:

  8. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Hello chong_dev,

    The "Crazy Minnow Studio" menu is under the "Component" menu. If you are looking to use one of our add-on's, be sure to download and install the applicable add-on from our website first.

    https://crazyminnowstudio.com/unity-3d/lip-sync-salsa/downloads/
     
  9. chasehek

    chasehek

    Joined:
    Feb 4, 2017
    Posts:
    2
    No, it not resolved problem(



    It affects only on nose and brows, but not blink_left/right
     
    Last edited: Feb 28, 2018
  10. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Hi chasehek,

    Please verify that on [CM_FuseSync] you removed the [Eyelashes] SkinnedMeshRenderer from the [Other SkinnedMeshRenderer] field, but the [Eyelashes] field is still populated with the [Eyelashes] SkinnedMeshRenderer. I tried to recreate the eyelash issue with a Fuse character in a test project, but have been unable to recreate the issue. Is this a Mixamo or Adobe Fuse character? If Adobe, please send your model to assetsupport@crazyminnow.com.
     
  11. Yaakov_Shahak

    Yaakov_Shahak

    Joined:
    Oct 15, 2017
    Posts:
    3
    Well I tried to minimize it but the folder is still 800Mb with Morph stuff, and when I tried to just to export the scene, Morph didn't work well with that.
     
  12. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Hi Yaakov_Shahak,

    I understand, it's sometimes difficult to isolate in a large project. Please contact us at assetsupport@crazyminnow.com, include your SALSA invoice number, and reference to your username on this thread so we know who the email is from. We'll work out another way assist you.

    Thanks,
    Michael
     
  13. sunseeker1988

    sunseeker1988

    Joined:
    Jul 10, 2014
    Posts:
    386
    Hi, the CM_Autodesk sync script is reverting the saySmall shapes, sayMedium Shapes and SayLarge shapes back to normal values upon playing the game in the editor. How can I prevent this? salsa.JPG
     
  14. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Hello sunseeker1988,

    Sorry about that, I believe we discovered and patched this bug in our Fuse 1-click and the fix didn't make it into the Autodesk 1-click. We'll get it updated, tested, and uploaded soon.

    Michael
     
    sunseeker1988 likes this.
  15. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Hello sunseeker1988,

    We have posted v2.6.0 of our Autodesk 1-click setup. The bug that was resetting the [CM_AutodeskSync.cs] BlendShape map values on start has been fixed. This version also includes our new [CM_AutodeskLOD.cs] script, that is added during the 1-click setup, that automatically finds all available LOD levels and provides a public function to switch LOD's. See the [Examples/Scripts/LODTester.cs] file to learn how to use it.

    https://crazyminnowstudio.com/posts...er-generator-with-salsa-with-randomeyes-v1.2/

    Michael
     
    Gregorik and sunseeker1988 like this.
  16. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    sunseeker1988 likes this.
  17. shredingskin

    shredingskin

    Joined:
    Nov 7, 2012
    Posts:
    145
    I'm using an instantiated MCS player from the resources folder, and every time that open the project I get an out of bounds error:
    Array index (4) is out of bounds (size=0)
    UnityEngine.SkinnedMeshRenderer:SetBlendShapeWeight(Int32, Single)
    CrazyMinnow.SALSA.MCS.CM_MCSSync:LateUpdate() (at Assets/Tools/Crazy Minnow Studio/SALSA with RandomEyes/Third Party Support/MCS/CM_MCSSync.cs:161)

    Any idea how to solve this ?
    I know it's mostly a MCS problem, but doesn't hurts to ask here.
     
  18. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Hello shredingskin,

    We have seen issue with some of the MCS character accessory meshes not being correctly built to handle multiple LOD's, especially a few of the hair meshes. Please test the character without accessories to see if the issue is resolved. If it is, check for the availability of multiple LOD's on each accessory to track down the problematic mesh.

    Thanks,
    Michael
     
    Last edited: Mar 24, 2018
  19. Alex-3D

    Alex-3D

    Joined:
    May 21, 2013
    Posts:
    78
    Michael,

    You know that accessories here have nothing to do with. I did send you simplest project with naked characters with a such problem 21 feb 2018 (email theme Errors when creating salsed mcs character from prefab). This is a great incompability between Salsa and MCS.
    I wrote you. You wrote The MCS character in this project is corrupt or did not import fully.
    I wrote Morph3D - they keep silence more than a month.

    Thanks
    Alexander
     
    Last edited: Mar 24, 2018
  20. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838

    Hi Alex-3D,

    I'm sorry you feel that way. We can't speak for other asset providers, but we have been supporting SALSA, and all of the add-on's we've built, for four years and make great effort to respond and resolve issues as quickly as possible. MCS has changed a lot over the years, and we've always updated our 1-click add-on in a timely manner upon discovering changes within the MCS system or bugs in our code.

    As mentioned in email, we tested your scene and were able to recreate the same errors your were getting, but the errors were inconsistent so we created a new project, imported the MCS male, SALSA, MCS 1-click setup, the animations from your project, and created an animated, SALSA enabled, MCS male prefab and instantiated it at run-time, and the errors were gone. Since you did not respond to that email, we assumed re-importing the MCS character worked for you as well. If you're still having trouble we're happy help..

    Michael
     
  21. Alex-3D

    Alex-3D

    Joined:
    May 21, 2013
    Posts:
    78
    shredingskin,
    If you change just one MCS morph, create prefab and use this prefab in scene (or instantiate it from resources as it figured out), you can get a such error (or can not get). It seems unsolvable until Morph3D wakes up and do something intelligible.

    Michael,
    Despite the fact that this is an Morph3D's fault, this breaks only Salsa. Body morphs do not disappear and characters looks as it should, all scripts are executing. But Salsa's inner blendshape structure became periodically broken and character can begin to smile instead of blinking. Seriously. I prevent of this by calling CM_MCSSync.Initialize() by any case (Start, changing LOD ect). Custom shapes can't be used at all as they can appear shuffled too at any moment. It does not care Morph3D. Leed them. Your position is stronger! Salsa works with any characters and is the only fit lipsync asset on the store. On the other hand nobody needs realistic 50K tris MCS character without face animations and lipsyncing.

    Thanks
    Alexander
     
  22. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    When you activate a BlendShape on an MCS character, the BlendShape is added to the end of the BlendShape array with index = BlendShapeList.Length-1. However, since MCS is adding the BlendShapes to the FBX file directly instead of a duplicate mesh asset file as we do with MorphMixer, the BlendShapes are all lost when you close and open the project. Once you open the project, the MCS system adds all the previously active BlendShapes back to the mesh, not in the original order you selected, but in a fixed order they determine. This means that if you add or remove BlendShapes after MCS 1-click setup is complete, then close/open the project, the index order may change and it may be necessary to call CM_MCSSync.Initialize() to remap the indexes. You can actually do the same thing for custom shapes using the following RandomEyes3D public function:

    AutoLinkCustomShapes(bool excludeEyeShapes, Salsa3D excludeSalsaShapes);

    This won't remap groups since they are setup in a custom way by the user. You could script your group setups but it would be easier to just build your own groups using only custom shapes by their default name.

    If MCS duplicated their meshes first as Unity mesh asset files (as we do with MorphMixer), then wrote the BlendShapes to the mesh asset file instead, Blendshape indexes would not be wiped and re-added each time you open a project, and therefore the BlendShape indexes wouldn't change. We expressed this concern to the MCS team a year ago. We'll touch base with them again to see if they're in a better position to investigate options.

    Michael
     
  23. Gregorik

    Gregorik

    Joined:
    Dec 11, 2014
    Posts:
    331
    Hi @Crazy-Minnow-Studio. I'm having trouble reconciliating Autodesk Character Creator blendshapes with Unity. (And with Adventure Creator's Shapeable component in particular.) I don't yet own Salsa 1.5, but I'd like to ask if there's a good workflow regarding Salsa+Autodesk blendshapes, or perhaps even Salsa+AdventureCreator+Autodesk blendshapes. And if your new TextToLipsync function works OK with Autodesk Creator blendshapes. Thanks.
     
  24. shredingskin

    shredingskin

    Joined:
    Nov 7, 2012
    Posts:
    145
    Now it seems the problem has magically solved, but yeah, the MCS doesn't handle very well instantiated prefabs from the resources (it also has problems with changed materials to the character).

    It had a lot of potential and seemingly cheap, but there are a lot of hazzle for what I'm doing, I'm now trying to replace characters with Fuse (since it's free and works ok).
     
  25. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Hi Gregorik,

    We offer a 1-click setup for Autodesk characters, and recently released an update that also exposes LOD switching.

    https://crazyminnowstudio.com/posts...er-generator-with-salsa-with-randomeyes-v1.2/

    When using Adventure Creator and Salsa3D there is no special handling necessary. Simply make characters play dialogue using Adventure Creators documentation, and SALSA enabled characters will automatically perform lipsync. Our TextSync text-to-lipsync asset can be used to to approximate lipsync to Adventure Creator dialogue that does not have audio using our TextSync AC Listener, which automatically hooks into the speech events in Adventure Creator.

    https://crazyminnowstudio.com/posts/latest-version-of-adventure-creator-supports-salsa/

    https://crazyminnowstudio.com/posts/salsa-text-to-lipsync-using-textsync/

    I have not used Adventure Creator's Shapeable component, we'll have to look into that.

    Michael
     
    Gregorik likes this.
  26. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Hi Gregorik,

    Adventure Creator's Shapeable component is AC's way of driving BlendShape's. When using SALSA for lipsync, this is all handled automatically, and for facial expressions, our way is to use RandomEyes Groups, along with our Adventure Creator custom actions. You can read more about our AC Actions add-on using the link below.

    https://crazyminnowstudio.com/posts/latest-version-of-adventure-creator-supports-salsa/
    • Re3D_SetGroup
      • After setting up facial expression groups in RandomEyes3D, use this to activate/deactivate an expression group.
    • Re3D_SetGroupDuration
      • After setting up facial expression groups in RandomEyes3D, use this to activate an expression group for a specified duration.
    Michael
     
    Gregorik likes this.
  27. Gregorik

    Gregorik

    Joined:
    Dec 11, 2014
    Posts:
    331
    Wow. This officially makes Salsa a better choice than the competition, particularly for adventure games. ;)
     
    Crazy-Minnow-Studio likes this.
  28. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    I just realized we never posted here that we have a couple of Timeline for SALSA actions available in our SALSA downloads area. One Timeline action creates a track specifically for TextSync users to control TextSync dialogue switching, complete with dynamically responding Timeline track clip names and durations.

    The second Timeline track-type binds and controls SALSA-usable AudioSources. AudioClip clips added to the track also dynamically assume the name of the AudioClip as well as the duration. NOTE: You currently cannot drag-n-drop AudioClips to the SALSA AudioSource track -- it will create a Unity AudioSource track that you cannot bind SALSA to. Watch the video below for information on how to use the add-on.

    There are two videos on our Youtube channel demonstrating this add-on's use. The part 2 video is the most current and is probably the only one you need to watch. Part 1 was released specifically for TextSync and was an old version without dynamic response in the clips.

    Enjoy!
    Darrin
     
    Gregorik likes this.
  29. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    If you're doing a project in WebGL, check out how you can use SALSA lip-sync in WebGL with our new Amplitude asset, and free AmplitudeSALSA add-on.

     
  30. Dorian-Dowse

    Dorian-Dowse

    Joined:
    Jun 2, 2016
    Posts:
    94
    Hi. I'm using a Daz 3D character with Salsa and I'm having trouble with the jaw. I've run the one click set up for Daz characters and I'm using the example CM_Ethan_Demo script to run the character. Everything works except for the jaw. The lips move but the jaw seems welded in place. I've tried all sorts of variations with no success. The Ethan demo has a moving jaw. The video example of Daz and Salsa seems to have a moving jaw. Mine doesn't.

    I'm using: Salsa 1.5.5, DAZ3D 1-Click v3.7.0, Unity 2017.3.0f3 and Daz3D 4.10.
     
  31. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Hello Dorian-Dowse,

    You can unlink the jaw bone from the character's avatar using the following steps:
    • Select the character in the project folder.
    • In the Inspector, on the Rig tab, the Animation Type should be set to Humanoid.
    • Select the Configure button.
    • On the Mapping tab, select the Head, and unlink the jaw bone.
    • Click the Done button.
    Avatar_Jaw.png

    Michael
     
  32. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,487
    I'm trying the UMA 2 1-click package, but I get the following error after importing:

    Assets/Crazy Minnow Studio/SALSA with RandomEyes/Third Party Support/UMA 2/Editor/CM_UmaSyncEditor.cs(39,147): error CS0246: The type or namespace name `UMADynamicAvatar' could not be found. Are you missing `UMA' using directive?

    I added the using UMA statement to the script and it got past the error. Just thought you should know.
     
  33. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Hi magique,

    Thanks for letting us know. Can you confirm if you downloaded the UMA-DCS package [UMA DCS 1-Click v1.6.0]? It sounds like you may be using the deprecated [UMA2 1-Click v1.5.0].

    Thanks,
    Michael
     
  34. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,487
    I don't remember. I thought I got the latest. Anyway, I'm not going to use the UMA now because of some issues.

    I am trying MCS now, but am seeing a strange issue. When the character blinks, the hair around the eyes also moves. Any ideas on how to fix that?
     
  35. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    MCS added a BlendShape called [eCTRLEyesClosed] to the hair mesh, which is the same shape that is used to blink the eyes on the body mesh. When setting it's value to 100, you can see that it pulls the corner of the bangs down. This means that the CM_MCSSync script is working as intended, and synchronizing BlendShapes across multiple meshes. That being said, I don't know why MCS added hair movement to the blink BlendShape. With synchronization added to the CM_MCSSync script, the only way to avoid this is to remove that Blendshape, or add a check during synchronization to see if the SkinnedMeshRenderer is the hair mesh and skip synchronization for that mesh. I don't know if there are any valid BlendShapes on MCS hair meshes, but this change would mess those up if there are. Since this is the second time this issue has come up, maybe we'll just include a hair sync exclusion property in the inspector and default to excluded since the issue still exists in the model.

    Michael
     
  36. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,487
    I have no clue how to remove a blend shape. Can that be done in the Unity editor?
     
  37. Dorian-Dowse

    Dorian-Dowse

    Joined:
    Jun 2, 2016
    Posts:
    94
    Oh I did this already. It's still not working. The blend shapes are working, but I'm monitoring the x rotation on the jaw while Salsa is running and there is no movement. If I monitor the eyes, I can see the rotations working. The jaw though is still.

    I just tried it all again with a standard genesis 2 female following the instructions on your web site to the letter. Still no jaw actions. If I compare this to the "taking spider" video you can clearly see the jaw moving in the video. In mine only the lips are moving.
     
  38. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    You can do it through the MCS API, but we're including this in our next MCS 1-click update.

    We had an idea for a more robust way to deal with all the index changes, and these new rogue hair BlendShapes in MCS characters. The CM_MCSSync script now includes an index verification function that runs whenever the CM_MCSSync inspector gets focus, as well as on start. This allows it to adapt to index changes on the fly, and seems to work really well.

    To deal with the rogue hair Blendshapes, we've included a section in the inspector for adding string names. Any MCS SkinnedMeshRenderer that contains these names within their own name, will have their BlendShapes removed on start. It's non-destructive since it's done on start and not serialized or saved to the editor.

    We've implemented the update in the MCS 1-click add-on and it seems to address both issues quite well. After a bit more testing, we'll get the update published for download.

    @Alex-3D, let us know if you're interested in testing this update.

    Michael
     
    magique likes this.
  39. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Hi Dorian-Dowse,

    Our DAZ 1-click setup does not use the jaw bone at all, it only uses Blendshapes for speech. Please be sure to follow the export instructions on our DAZ blog page to ensure all the necessary BlendShapes are included in the export.

    https://crazyminnowstudio.com/posts/using-daz3d-characters-with-salsa-with-randomeyes/

    The CM_DazSync inspector should look like this, but with whichever supported model generation your'e using.
    DAZ_1-click.png
     
  40. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838


    Our MCS 1-click setup version 2.1.0 is now available for download.
    • Added a new function that automatically remaps BlendShpes indexes to try to address the MCS moving target indexes (like self healing).
    • Added [Remove BlendShapes] section to the inspector and pre-populated it with a couple hair options to avoid rogue hair movement blendshapes linked to blinking.
    • Added a jaw bone link and sync, and a new default BlendShape map for more dynamic lip-sync.
    https://crazyminnowstudio.com/posts/using-mcs-characters-with-salsa-lipsync/

     
    magique likes this.
  41. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
  42. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Our micInput-lite add-on has been updated (v1.7.0-beta) with an experimental change that could reduce microphone input lag/delay by up to 10x for "automatic startup"-enabled projects. This change is marked as beta since we have not been able to test it on many projects, but wanted to get it out to everyone as quickly as possible. The change is very minimal and non-invasive so should be safe for any project. If you use this and it works great for you, please send us an email or post a comment here and let us know how happy you are! ;)

    Thanks to Zane for bringing it to our attention!

    One more thing! We also moved the Component add menu item to Crazy Minnow Studio > SALSA > Addons > micInput

    Thanks,
    Darrin
     
  43. magique

    magique

    Joined:
    May 2, 2014
    Posts:
    3,487
    Thanks for the new MCS 2.1 1-click. It works great and no more hair issues when blinking.
     
    Crazy-Minnow-Studio likes this.
  44. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    SALSA 2.0 progress report -- it is taking us a little while, but we have made huge progress the past couple of weeks. Check out the devblog!
     
    wetcircuit likes this.
  45. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    wetcircuit likes this.
  46. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Our Unity Timeline integration package has been split into two packages:

    • SALSA Audio (for AudioClip-driven projects)
    • SALSA TextSync (for text-to-lipsync projects)
    The original package contained both solutions, but caused namespace errors if you were not using TextSync. Both packages are available on the SALSA downloads page.

    Blog post here.
     
  47. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Our Unity Timeline for SALSA Audio add-on has been updated to include two new fixes:
    • Disabled design-time playback of audio clips. v0.3.1
    • Fix for intermittent back-to-back duplicate playback of an audio clip. v0.3.2
    You can pick up the new add-on package v0.3.2 on our downloads page.
     
  48. xibanya

    xibanya

    Joined:
    Nov 26, 2016
    Posts:
    10
    Hi there! I use FMOD studio as my audio middleware. Is there a way to point SALSA at my FMOD assets? Alternatively, is there a way to capture the lip sync animation for an audio file and play that animation back later without needing to use Unity's studio event system? All the solutions I've come up with seem to involve having each audio file doubled up and that would bloat the project considerably.

    E: I sorta figured it out on my own, here's my solution in case it helps anyone else. basically,

    Code (CSharp):
    1.  
    2. using System;
    3. using System.Linq;
    4. using UnityEngine;
    5. using FMODUnity;
    6. using FMOD.Studio;
    7. using CrazyMinnow.SALSA;
    8. using System.Runtime.InteropServices;
    9.  
    10.  public class SoundStarter : MonoBehaviour
    11.  {
    12. public bool is3d = false;
    13. public Transform target3d;
    14.  [EventRef]
    15. public string soundPath;
    16. public Salsa3D salsa;
    17. private EventInstance ev;
    18. private FMOD.DSP fft;
    19.  
    20. private void Start()
    21.     {
    22.         ev = Sound.CreateEvent(soundPath);
    23.         if (is3d) ev.set3DAttributes(FMODUnity.RuntimeUtils.To3DAttributes(target3d));
    24.  
    25.         FMODUnity.RuntimeManager.LowlevelSystem.createDSPByType(FMOD.DSP_TYPE.FFT, out fft);
    26.         FMOD.ChannelGroup channelGroup;
    27.         FMODUnity.RuntimeManager.LowlevelSystem.getMasterChannelGroup(out channelGroup);
    28.         channelGroup.addDSP(FMOD.CHANNELCONTROL_DSP_INDEX.HEAD, fft);
    29.  
    30.         ev.start();
    31.     }
    32.  
    33. private void Update()
    34.     {
    35.             IntPtr unmanagedData;
    36.             uint length;
    37.             fft.getParameterData((int)FMOD.DSP_FFT.SPECTRUMDATA, out unmanagedData, out length);
    38.             FMOD.DSP_PARAMETER_FFT fftData = (FMOD.DSP_PARAMETER_FFT)Marshal.PtrToStructure(unmanagedData, typeof(FMOD.DSP_PARAMETER_FFT));
    39.             var spectrum = fftData.spectrum;
    40.  
    41.             if (fftData.numchannels > 0)
    42.             {
    43.                 salsa.average = spectrum[0].Skip(10).Take(100).Average() * 5;
    44.             }
    45.     }
    46. }
    47.  
    as for why I'm skipping 10, taking 100, averaging that then multiplying that by 5 to get the amplitude, well uh, haha things just seemed to work OK when I did that, I'm not actually sure the best way to get the amplitude once I have the spectrum data so if anybody knows, please tell me!

    EE: ah, I was skipping because I had a drone in the background of the scene and was getting false positives when sampling the spectrum data from the master channel. I suspect that if I sampled only the channel the event instance was playing on, I'd get a much more accurate lipsync. If I get this to work I'll update the script.
     
    Last edited: May 7, 2018
    Crazy-Minnow-Studio likes this.
  49. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838
    Hello! Writing to SALSA's 'average' field is a great way to leverage SALSA's triggering and animation engine while driving the values from another source. In fact, we use this same method for SALSA TextSync and it is exactly why SALSA was modified to allow this behavior. Good luck on your project!
     
  50. Crazy-Minnow-Studio

    Crazy-Minnow-Studio

    Joined:
    Mar 22, 2014
    Posts:
    838