[Face Capture] Anyone got it to work with VRoid Studio characters with HANA_Tool?

I have managed to get the app to connect to the server, but I have failed to get the components to do anything (rotate head, adjust blendshapes etc). Has anyone got it going with a VRoid Studio character? (VRoid Studio is free.)

I used HANA_Tool to add the 52 ARKit "Perfect Sync" blendshapes. (Costs a few dollars on Booth.)

VRoid Studio characters have a root object with child Face and Body objects.



The facial blendshapes are all in the Face child. HANA_Tool is available on Booth and adds the ARKit blendshapes to the basic ones that VRoid Studio adds to characters. The names are "close" to the Unity names. (The top names are the default names used by VRoid Studio, the ones below it are the extra ones added by HANA_Tool.)



I am not sure where to put the face actor component or how to test it. There are sliders like "how close a match the blendshape name should be", but that is not very helpful to work out what to set it to (no value I have tried worked). I tried putting the ARKit Face Actor component on both the root game object and on the Face object, but neither worked.

There is no feedback when things are not working. I don't know if I picked the wrong things in the mapper.



Just wondering if someone had it working with VRoid Studio characters already before I invest more time.

 

I had missed the Add Renderers button click (oops!). After clicking it mapped the blendshapes correctly...



... but it still does not work. I start the app (play mode), I have a connection, the app shows as being connected in the app and in Unity. But the character model does not move.

Further, in the Face Actor component when in and not in Play mode, changing numbers for the blendshapes has no effect. Suggestion for docs - describe how to test if things are working. Head rotation etc is not working either. But I am not sure if dragging the draggers is supposed to work or not. Also the blendshape values does not list a unit - is it 0 to 1 or 0 to 100 etc?

Going to the Face behavior, I can change the blendshape values there and they have an immediate effect, so I know the blendshape works.





I also created a Take Track in a timeline, clicked the Start Recording and Stop Recording buttons. No movement of the character.

I turned off all the other components on the character to make sure no collision (no difference).

I don't think its blendshape specific since head rotation is not working either.

Any other tools to help debug mappings? Thanks!

 

Yes, the above is what I was doing. I tried to repeat, but I failed to get it to go. It found all the blendshapes in the mapper, but if I change any of the Current Pose settings, nothing happens to the character in the scene. Also head rotation etc does not work (so I don't think its a blendshapes issue).

I may try again later on a fresh project. Maybe my character is doing something... funky. Note I can get other trackers like VSeeFace to work. Normally I would crack open the C# files and add some print statements in it to try and track it down, but not this time.

I do note that some other programs have not liked the VRoid characters either - something about the avatar I recall. The VRM package to import the character creates the avatar or something, so some tools complain because its not from a FBX file (something like that).

 

Hello! I know this thread's a bit old, but in case you're still working on this (i.e. getting Perfect Sync blend shapes working for VRoid models in Unity, using Unity's Live Capture package), just wanted to share that I was able to successfully get it working!

The software versions I was using:
- VRoid Studio: 1.10.0
- Unity: 2021.3.8f1
- 3tenePRO with FaceForge: V3

The tutorials I followed, start to finish:
- Add the 52 blend shapes to my VRoid model. (This tutorial uses FaceForge instead of HANA Tool; it's slightly pricier at 3,300 yen (~23 USD), but worked like a charm for me.)
- Import my VRoid model into Unity.
- Tweak any Blend Shape clips, if needed. (Only need to watch the next ~1 minute from the linked timestamp. In my case, some of the VRoid-specific clips needed fixing (the A/E/I/O/U ones), but all the 52 Perfect Sync clips were fine as is.)
- Install the Unity Face Capture package. (Also consulted some additional Unity documentation to get my network requirements set up correctly, and to install the companion Unity Face Capture iOS app.)
- Set up the Unity Scene to use Face Capture. (Wanted to note, there was one step here that kind of threw me off, but probably just because I'm a Unity noob. For the step "Creating an ARKit Face Device > 3. In the newly created ARKit Face Device component, assign a Face Actor to the Actor field.", I made the mistake of dragging over my .prefab asset (from the Assets file explorer window) instead of the actor in the scene (from the Hierarchy window).)

And with that, it worked for me!

Lastly, for reference I've attached a zip folder of screenshots for my Unity setup (specifically, the ARKit Face Actor, Face Mapper, and Face Device; the Take Recorder; the prefab blend shapes and blend shape clips; and the scene Hierarchy).

 

Thank you for sharing! I do plan to come back to this, but still battling with my upgrade to HDRP and finding a good mix of settings to use overall. E.g.

- first half, strange lighting from below on face I have not worked out yet; second half, hair bounce at start of each shot. I am just using animation clips for facial expression poses and blend between them so Live Capture has not been my top priority. (Other problems to solve first!)

So I would love to hear any other insights or experiences you have along your way!