Search Unity

  1. We are migrating the Unity Forums to Unity Discussions. On July 12, the Unity Forums will become read-only. On July 15, Unity Discussions will become read-only until July 18, when the new design and the migrated forum contents will go live. Read our full announcement for more information and let us know if you have any questions.

Bug SkinnedMeshRendererConversion, Bindposes, and Reparenting

Discussion in 'Graphics for ECS' started by DreamingImLatios, Nov 16, 2021.

  1. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,349
    Alright. I'm really stumped on this one.

    So the Hybrid Renderer currently converts skinned mesh renderers by taking the mesh and material, and creating a new entity for it reparented to the root bone with and identity LocalToParent.

    This suggests to me that the bindpose of the mesh is computed relative to the root bone regardless of the transforms of the renderer in the broader hierarchy. Given that the bindpose calculation is engine-side, I just want to confirm if this is really the case?
     
  2. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,349
    Alright. So I figured out what is going on, and I believe this is a bug. The behavior that would make more sense is as follows:
    If the SMR is attached to or the child of the root, leave it where it is in the hierarchy and use existing logic, as animations to the root will affect the whole RenderMesh.

    If the SMR is independent to the bone hierarchy, then each SkinMatrix needs to be computed by computing the local-to-(root's parent), rather than the local-to-root. But then the RenderMesh doesn't have to be reparented and so it can preserve its transform offsets.

    And lastly, if the SMR is attached way down the bone hierarchy, that is nonsensical and should throw an error.

    But reparenting currently can break prefabs with transform offsets on the SMRs independent of the bone hierarchy.
     
  3. Jebtor

    Jebtor

    Unity Technologies

    Joined:
    Apr 18, 2018
    Posts:
    117
    Hey, thanks for your post. Could you expand on how it breaks prefabs with transform offsets on the SMR? Also what version of Dots are you on?
     
  4. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,349
    I'm on latest public release. For reference, this isn't me running into an issue as much as it was me encountering a discrepancy between the Hybrid Renderer and a custom solution I am working on. However, while it is pretty rare, there are cases where having a transform offset be applied to the mesh separate from its skeleton is useful. I believe (but haven't tested) if you were to have an animated character where the smr is a sibling to the root bone, then duplicated the smr, moved it along the z axis, and scaled and offset it such that it was a "big clone with matching movements", I suspect this would work correctly in GOs but completely break with the Hybrid Renderer. (It definitely wouldn't work in the Hybrid Renderer 0.11.) I also believe this may be the cause for this issue: https://forum.unity.com/threads/ava...onverted-during-authoring-conversion.1112773/
     
  5. Jebtor

    Jebtor

    Unity Technologies

    Joined:
    Apr 18, 2018
    Posts:
    117
    To start some general information on SkinnedMeshRenderer (SMR) conversion. It is quite complex and does not have a one to one relation during conversion. As you mention the conversion system adds a render entity for each submesh / material assigned to the SMR. On top of that another entity is created that holds the animated data (skin matrices and blend shape weights). This is generally the entity other systems interface with. A prime example would be animation updating the pose.

    This last entity is converted in place. The render entities, however, can end up in different places in the hierarchy as you observe. This depends on the root bone field on the SMR. If it assigned we reparent, if not the entities are converted in place too.
    The root bone is important for (amongst of things) the render bounds of the renderer. When assigned the render bounds of the SMR will follow this transform instead of the transform the SMR is attached to. The latter is always the case for MeshRenderers. In order to mimic this behavior for SMR in Dots we have two options. Either we have some magic that handles the virtual reparenting. Or we 'reparent' during conversion. We have chosen for the latter as it reduces complexity. Above all it allows Hybrid Renderer to treat MeshRenderers and SMR the same. In other words Hybrid Renderer is not aware if a mesh is skinned or not. The entities are the exact same, skinned entities simply have a few extra components that are touched by systems earlier in the pipeline.

    I would like to better understand the problems you foresee. A concrete case would help. Every vertex in a skinned mesh is bound to a bone. So the SMR transform itself should not influence the end result (given that the SMR transform is not part of the skeleton). If I understand the described case correctly only the SMR is getting duplicated, both SMRs share the same skeleton and thus will give the same mesh? In order to have a big brother you would need to duplicate the skeleton too and scale that.
     
  6. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,349
    Alright. So I just tried this with Game Objects, and it turns out that Game Objects exhibit the same breakage issue. So the Hybrid Renderer is correct in that it matches Game Object behavior. However, this is a gross misunderstanding of skinning across the entirety of Unity (and you aren't the only ones which I will explain in a bit).

    To understand the real problem, you need a prefab structure with a hierarchy somewhat like what I have:
    • Character (with Animator)
      • Body (SMR)
      • Skeleton Transform (skeleton ref space)
        • Hips (root bone)
          • Spine (bone)
    Now let's assume that Body has the identity transform. If you modify that local transform of the Body in the editor, the transform moves, but the actual skinned mesh stays put. Now, reset that transform back to identity, and add a cube as a child to the Body. Now modify the local transform of the Body again. Notice how the cube moves will the skinned mesh stays put? The transform relationship is completely broken. This is fundamentally incorrect. The skinning process should only affect the raw mesh data, and have nothing to do with the local-to-world transform. Yet here we see it completely nullify the local-to-world transform for the skinned mesh.

    The reason this likely happens is two-fold. First, Unity does not correctly differentiate between the root bone and the skeleton reference space (the parent of the root bone or world space if the root bone doesn't have a parent). And second, DCCs also exhibit a misunderstanding by making the skinned mesh renderer either attached to or a child of the root bone. This is nonsensical, and it completely defeats the purpose of the root bone actually being a bone with a bind pose and vertex influences. But it is easy to correct for by multiplying the bind pose by the local-to-root to get the skin matrix instead of multiplying by the local-to-skeletonReferenceSpace which would be the correct transformation. And really, DCCs should be putting the skinned mesh objects as children to the reference space. I know of at least one DCC that does that, and usually certain objects need to have identity transforms in order for Unity to import it with correct results.

    The problem here is that Unity forces all skinned meshes to behave similarly to the nonsensical interpretation by discarding the mesh's transform and treating it as a child of the root bone.
     
  7. Jebtor

    Jebtor

    Unity Technologies

    Joined:
    Apr 18, 2018
    Posts:
    117
    Isn't this exactly the behavior you could achieve if you set up the skinned entities yourself? Either by writing your own conversion system or creating the entities at runtime. When doing this you would not reparent the render entities into the hierarchy, but instead keep them in place. Do note that a lot of the components and classes are marked as internal since the mesh deformations in Dots are still experimental.

    The SkinnedMeshRenderer conversion we supply out of the box should mimic the behavior of GameObjects. The reparenting recreates the behavior without forcing the behavior you describe on anyone that opts out of the conversion system.
     
  8. DreamingImLatios

    DreamingImLatios

    Joined:
    Jun 3, 2017
    Posts:
    4,349
    Yes. I am doing that in my solution, in which I rewrote skinning from the ground up: https://github.com/Dreaming381/Kinemation-Skinning-Prototype

    I fully understand what is happening now. But for a long time, Unity's behavior has made it difficult to debug other skinned mesh import issues. It wasn't until I understood what the Hybrid Renderer conversion was doing and why that I understand the cause of many past frustrations. To be clear, there were other issues in those imports. But this behavior obfuscated them.
     
    hugokostic likes this.