Search Unity

[Face Capture] Anyone got it to work with VRoid Studio characters with HANA_Tool?

Discussion in 'Film, TV, Animation & Cinematics Previews' started by akent99, Jul 12, 2021.

  1. akent99

    akent99

    Joined:
    Jan 14, 2018
    Posts:
    142
    I have managed to get the app to connect to the server, but I have failed to get the components to do anything (rotate head, adjust blendshapes etc). Has anyone got it going with a VRoid Studio character? (VRoid Studio is free.)

    I used HANA_Tool to add the 52 ARKit "Perfect Sync" blendshapes. (Costs a few dollars on Booth.)

    VRoid Studio characters have a root object with child Face and Body objects.

    upload_2021-7-11_20-56-33.png

    The facial blendshapes are all in the Face child. HANA_Tool is available on Booth and adds the ARKit blendshapes to the basic ones that VRoid Studio adds to characters. The names are "close" to the Unity names. (The top names are the default names used by VRoid Studio, the ones below it are the extra ones added by HANA_Tool.)

    upload_2021-7-11_20-57-18.png

    I am not sure where to put the face actor component or how to test it. There are sliders like "how close a match the blendshape name should be", but that is not very helpful to work out what to set it to (no value I have tried worked). I tried putting the ARKit Face Actor component on both the root game object and on the Face object, but neither worked.

    There is no feedback when things are not working. I don't know if I picked the wrong things in the mapper.

    upload_2021-7-11_21-1-13.png

    Just wondering if someone had it working with VRoid Studio characters already before I invest more time.
     
  2. ScottSewellUnity

    ScottSewellUnity

    Unity Technologies

    Joined:
    Jan 29, 2020
    Posts:
    8
    @akent99
    Add just one FaceActor component to the "Sam-v11-2 Prefab Variant" object, and assign the mapper asset to it.

    Once you have added the face actor component to the prefab, you should be able to use "Add Renderer" button in the Face Mapper inspector to add mappings for meshes that have blend shapes. It should show a list of all Skinned Mesh Renderers with at least one blend shape that are a child of the transform with the FaceActor component.

    If for some reason the "Add Renderers" option doesn't have any meshes, you can try enabling debug mode on the inspector (right click on the Inspector window title), add an element to the "Maps" array, and set the "Path" property of the new element to ''Face" in your case (or any path to a valid skinned mesh renderer relative to the face actor transform). Then disable debug mode on the inspector, and there should be a renderer you can configure blend shapes mappings for.

    The rest of the mapper configuration looks fine to me.

    For a full description of the face capture mapper setup, you can use the docs here:
    https://docs.unity3d.com/Packages/com.unity.live-capture@1.0/manual/ref-component-arkit-default-face-mapper.html
     
  3. akent99

    akent99

    Joined:
    Jan 14, 2018
    Posts:
    142
    I had missed the Add Renderers button click (oops!). After clicking it mapped the blendshapes correctly...

    upload_2021-7-12_23-34-51.png

    ... but it still does not work. I start the app (play mode), I have a connection, the app shows as being connected in the app and in Unity. But the character model does not move.

    Further, in the Face Actor component when in and not in Play mode, changing numbers for the blendshapes has no effect. Suggestion for docs - describe how to test if things are working. Head rotation etc is not working either. But I am not sure if dragging the draggers is supposed to work or not. Also the blendshape values does not list a unit - is it 0 to 1 or 0 to 100 etc?

    Going to the Face behavior, I can change the blendshape values there and they have an immediate effect, so I know the blendshape works.

    upload_2021-7-12_23-47-50.png

    upload_2021-7-12_23-48-43.png

    I also created a Take Track in a timeline, clicked the Start Recording and Stop Recording buttons. No movement of the character.

    I turned off all the other components on the character to make sure no collision (no difference).

    I don't think its blendshape specific since head rotation is not working either.

    Any other tools to help debug mappings? Thanks!
     
    Last edited: Jul 13, 2021
  4. ScottSewellUnity

    ScottSewellUnity

    Unity Technologies

    Joined:
    Jan 29, 2020
    Posts:
    8
    Currently there isn't a very convenient way to test your FaceMapper configuration before connecting the app, but we have plans to improve this. For now, select the FaceActor component and assign the FaceMapper that corresponds to the character rig. In the inspector Debug Mode, ensure that the "Blend Shapes Enabled" toggle is on. Then, you should be able to pose the face using the "Current Pose" field on the actor component. The blend shapes here range from 0-1, as that is what ARKit uses for the value ranges.

    Here are the steps to get recording once you've set up the character with a FaceActor component in a fresh scene:

    1) Add a TakeRecorder to the scene using the "Menu/GameObject/Live Capture/Take Recorder" option.
    2) In the new Take Recorder component, click on the + button from the Capture Devices list and add an ARKit Face Device. This adds a new child gameobject to the take recorder with the Face Device component.
    3) In the new Face Device, assign the FaceActor component you want to control.
    4) Connect the app to the editor.
    5) Assign the connected device as the "Client Device" on the FaceDevice component. It is usually auto-assigned by default.

    After that, you should be able to record takes, and switch to preview mode in the TakeRecorder and select a take to play back the take.

    You are right that the docs for the face capture setup could be improved, I'll make some changes to clarify things.
     
    Last edited: Jul 13, 2021
  5. akent99

    akent99

    Joined:
    Jan 14, 2018
    Posts:
    142
    Yes, the above is what I was doing. I tried to repeat, but I failed to get it to go. It found all the blendshapes in the mapper, but if I change any of the Current Pose settings, nothing happens to the character in the scene. Also head rotation etc does not work (so I don't think its a blendshapes issue).

    I may try again later on a fresh project. Maybe my character is doing something... funky. Note I can get other trackers like VSeeFace to work. Normally I would crack open the C# files and add some print statements in it to try and track it down, but not this time.

    I do note that some other programs have not liked the VRoid characters either - something about the avatar I recall. The VRM package to import the character creates the avatar or something, so some tools complain because its not from a FBX file (something like that).
     
unityunity