Search Unity

Question MARS and blendshape?

Discussion in 'Unity MARS' started by xhkong, Nov 19, 2020.

  1. xhkong

    xhkong

    Joined:
    Jul 2, 2020
    Posts:
    26
    I'm new to MARS.

    In MARS, is there a way for me to use similar blendshape model as in original AR Foundation's FaceManager to replace a user's head?
     
    xhkong, Nov 19, 2020
    #1

  2. jmunozarUTech

    jmunozarUTech

    Unity Technologies

    Joined:
    Jun 1, 2020
    Posts:
    297
    jmunozarUTech, Nov 24, 2020
    #2

  3. xhkong

    xhkong

    Joined:
    Jul 2, 2020
    Posts:
    26
    Thanks for getting back on this. I looked through the thread you mentioned. I'm trying to migrate my AR Foundation project into MARS. In the original project with AR foundation, I'm able to use ARFaceManager to set a face mesh and use blendshape visualizer to replace user face with a 3d model blendshape. Since MARS do not expose neither blendshapes nor facemesh, I'm trying to figure out how to have MARS and ARFoundations/ARFaceManager work side by side? I tried a ARSession object and MARS session object in my scene side by side, but it doesn't work as they seem to conflict each other.

    Can you provide some guidance on this?
     
    xhkong, Nov 29, 2020
    #3

  4. CiaranWills

    CiaranWills

    Unity Technologies

    Joined:
    Apr 24, 2020
    Posts:
    199
    MARS and ARFoundation should be able to co-exist in the same scene - what sort of issues are you seeing?

    MARS does expose the face mesh if the underlying platform provides one, but it will likely involve a bit more scripting than using ARFaceManager; The IMRFace structure contains the mesh as well as the landmarks and expressions.

    When you say blendshapes are you talking about blendshapes from ARKit or your own that you are applying to the mesh?
     
    CiaranWills, Nov 30, 2020
    #4

  5. xhkong

    xhkong

    Joined:
    Jul 2, 2020
    Posts:
    26
    I'm trying to achieve 2 things here. 1. Applying a custom texture to a face mesh and add a custom blendshape face object like animoji. Both I could do through ARFoundation's ARFaceManager, but not available from MARS.

    When I add a ARSession object, ARFaceManager to a MARS project, the FaceMesh prefab registered in ARFaceManager won't attached to the user's face in live video.
     
    xhkong, Dec 1, 2020
    #5

  6. xhkong

    xhkong

    Joined:
    Jul 2, 2020
    Posts:
    26
    Can you share the proper way to setup a scene with both MARS and ARFoundation Objects? For example, where do I put ARCameraManager to make it not conflict with MARS camera?
     
    xhkong, Dec 1, 2020
    #6

  7. CiaranWills

    CiaranWills

    Unity Technologies

    Joined:
    Apr 24, 2020
    Posts:
    199
    You can attach the ARCameraManager and the MARSCamera to the same camera in your scene; MARS should figure it all out. And if you have an existing AR Session Origin then you can put the MARS Session on there. Depending on whether you are starting from a MARS-exclusive or AR Foundation session this may happen automatically.

    I was just doing some testing and with both I got the AR Face Manager prefab appearing (on an ARCore device). Can you tell me what platform you are using, and also what version of the AR Foundation package you have?
     
    CiaranWills, Dec 2, 2020
    #7

  8. xhkong

    xhkong

    Joined:
    Jul 2, 2020
    Posts:
    26
    I'm using the ARFoundation 4.1.1 and testing on an iPhone X
     
    xhkong, Dec 2, 2020
    #8

  9. xhkong

    xhkong

    Joined:
    Jul 2, 2020
    Posts:
    26
    I attached ARCamera to MARS Camera object together with MARSCamera, if MARSCamera object is enabled, ARFace object is not showing up, nor does the MARS prefabs in the scene.
     
    xhkong, Dec 2, 2020
    #9

  10. CiaranWills

    CiaranWills

    Unity Technologies

    Joined:
    Apr 24, 2020
    Posts:
    199
    This is what my scene looks like when I have this working:
    upload_2020-12-3_15-51-17.png
    And on Main Camera:
    upload_2020-12-3_15-52-3.png

    Is this similar to your setup? If you can share your scene that would be helpful to figure out what's happening.
     
    CiaranWills, Dec 3, 2020
    #10

  11. xhkong

    xhkong

    Joined:
    Jul 2, 2020
    Posts:
    26
    This is a simple scene that I created from scratch. The Default ARFace prefab did show up in the camera. However, the Glasses Prefab that I attached to the nose is not there. If I move it around a bit, it would show up in the camera view but not anchored to the face landmark as it should. If I remove all the ARFoundation related objects, the glasses object would anchor as expected.



    upload_2020-12-4_3-58-29.png

    upload_2020-12-4_4-33-0.png
     
    xhkong, Dec 4, 2020
    #11

  12. xhkong

    xhkong

    Joined:
    Jul 2, 2020
    Posts:
    26
    Please let me know if you can reproduce this issue. If not I can upload my sample project and share a link.
     
    xhkong, Dec 4, 2020
    #12

  13. CiaranWills

    CiaranWills

    Unity Technologies

    Joined:
    Apr 24, 2020
    Posts:
    199
    Thanks - if you are able to share your project that would be helpful.

    Sorry for the delay!
     
    CiaranWills, Dec 18, 2020
    #13