Search Unity

  1. We are migrating the Unity Forums to Unity Discussions by the end of July. Read our announcement for more information and let us know if you have any questions.
    Dismiss Notice
  2. Dismiss Notice

Question How to access the bone positions used for skinning in a vertex or fragment shader?

Discussion in 'General Graphics' started by KageKirin, Aug 20, 2021.

  1. KageKirin

    KageKirin

    Joined:
    Jan 22, 2014
    Posts:
    60
    Hi,

    I'm looking for a way to access the per-vertex bone positions and weights in either the vertex shader, or ideally directly in the fragment shader.

    Long story short, I want to compute a spheroid normal from the vertex position to its associated bone(s) as center.

    How can I get the per-vertex bone info in the shader? I suppose the data can be accessed through some magic variable name.
    If the above is not possible, what would actually be the best way to get the data?

    Best regards.
     
    littledwarf likes this.
  2. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,401
    This is not data that's exposed to Unity shaders by default. Skinned meshes are pre-transformed either on the CPU or GPU prior to the shader getting access to them, so data about the bone weights or bone positions are stripped from the meshes before they're ever seen.

    The "best" way to do this is to implement mesh skinning on your own from scratch... which also means not using Unity's built in skinned mesh renderer component, or animation systems. But at least then you'll have access to all the skinning data you're looking for.

    Yeah, not exactly simple.

    The next option would be to copy the bone indices and weights into an unused UV set on your mesh via c#, and then iterate over all of the bone game objects and copy their transforms into an array to pass to your material at runtime via another c# script. If you only care about the bone with the highest weight, then you only need that one bone's index. But you'll still need to manually store it in some unused component somewhere.

    https://docs.unity3d.com/ScriptReference/Mesh.GetBoneWeights.html
    https://docs.unity3d.com/ScriptReference/Mesh.SetUVs.html
    https://docs.unity3d.com/ScriptReference/SkinnedMeshRenderer-bones.html

    Lastly, you will need to do this in the vertex shader and not the fragment shader. The reason is fairly straightforward. The fragment shader gets an interpolated value from the vertex shader, and if you try to pass the singular bone index from the vertex to the fragment you'll get an interpolated "index" that is just the index being treated as a floating point value which means it's entirely useless as an index once you get it in the fragment shader. You might get a value of "20" in the fragment shader, but that value might be because it was 20 on all 3 vertices of the triangle or if all 3 vertices have completely different indices that happen to interpolate to 20 at the current pixel. You could use
    nointerpolation
    or an
    integer
    interpolator which would keep the value constant across the entire triangle, but then you have a new issue which is you won't really know beforehand which vertex it's going to use across the entire triangle (and it can change on different platforms), and there's no way to smoothly blend between different normals.

    There are ways around this, but they're complicated, and in this case just passing the one normal value is likely more than good enough for your use case.
     
    DylanF likes this.
  3. KageKirin

    KageKirin

    Joined:
    Jan 22, 2014
    Posts:
    60
    Hi and thanks a lot.

    Let's skip the option of hand-rolling my own animation system, as it's a bit beyond scope.

    So, I can get the bones, and respective weights using the above methods.
    Am I right to assume the bones come in the correct order?

    Actually, I found the example code here pretty useful: https://docs.unity3d.com/ScriptReference/Mesh.GetBonesPerVertex.html

    As for getting the interpolated position, I was planning on doing this in the vertex shader, and only transmit said interpolated position so I can use a simple varying/interpolator and a per-pixel adjusted position on the fragment shader.

    You recommend using Mesh.SetUVs(). Is there anything, with respects to lightmaps/GI etc., I should take into account when selection the UV slot?
     
  4. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,401
    I would assume / hope so, but *shrug* I have no idea for sure.

    Unity doesn't support light maps on skinned meshes (anymore), so nope.
     
  5. KageKirin

    KageKirin

    Joined:
    Jan 22, 2014
    Posts:
    60
    The bone order actually seems correct.

    @bgolus: Thanks for your help. I actually managed to implement this today (see attachment). It's not exactly optimized, nor performant for that matter, but the basic idea works.

    Obviously, I could optimize the computational part using Burst and Unity.Mathematics, but I guess uploading vertex data every frame is not recommended.
    -> What would a more optimal solution look like? Perform the calculation in compute shader and directly write into the vertex buffer? Upload the bone transforms/positions into a SSBO (shader storage buffer) and access from vertex shader?
    How would I do either of this?

    Cheers.
     

    Attached Files:

  6. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,401
    The more optimal solution would be to encode the bone indices & weights into the vertex data, and update the bone transforms in a vector array or structured buffer you assign onto the material every frame.

    You could also have the indices and weights in a structured buffer you read from using the vertex ID instead of modifying the mesh at all. That extra indirection might not be very fast on mobile hardware, but should be fine on desktop & consoles.
     
  7. KageKirin

    KageKirin

    Joined:
    Jan 22, 2014
    Posts:
    60
    > The more optimal solution would be to encode the bone indices & weights into the vertex data, and update the bone transforms in a vector array or structured buffer you assign onto the material every frame.

    Good, this is exactly what I started doing.

    In the long run, I might just ask the artists to directly add the bone indices/weights as UVs directly, instead of patching them in, but for now this seems fine.

    Question about StructuredBuffer: do I have to pass them as GraphicsBuffer or ComputeBuffer to the material?
     
  8. KageKirin

    KageKirin

    Joined:
    Jan 22, 2014
    Posts:
    60
    I'm getting this:
    Code (CSharp):
    1. Metal: Vertex shader missing buffer binding at index 3 (BonePositions / Debug/BufferSpheroidNormal)
    2.  
    No idea what I'm doing wrong.

    Attachments: current WIP code.
     

    Attached Files:

  9. bgolus

    bgolus

    Joined:
    Dec 7, 2012
    Posts:
    12,401
    Generally you want to use a
    ComputeBuffer
    . A
    GraphicsBuffer
    is mainly for letting you access built in vertex data arrays.

    Some general thoughts:
    You probably want to use a material property block to set the buffer on the renderer rather than modifying the materials directly. Create a
    MaterialPropertyBlock
    in
    Start()
    and reuse it in
    Update()
    .
    https://docs.unity3d.com/ScriptReference/Renderer.SetPropertyBlock.html

    Don't create a new buffer every
    Update()
    . You also only need to create one for each renderer component once in
    Start()
    and update the data inside each. You also only really need to assign it once, apart from issues where buffers can become "detached" when alt-tabbing or changing resolutions. But otherwise just updating the data in the buffer with
    SetData()
    will update the shader.

    Calling
    buffer.Dispose()
    means you're destroying the buffer and flushing it from the GPU. So yeah, it's missing, because you're nuking it.
     
  10. KageKirin

    KageKirin

    Joined:
    Jan 22, 2014
    Posts:
    60
    Thanks for the reply.

    After a bit more work, I have got something working quite ok-ish.

    Now, I'm looking how to make this using ECS/DOTS, or at least more generic using some Controller/Manager class.
     

    Attached Files:

  11. KageKirin

    KageKirin

    Joined:
    Jan 22, 2014
    Posts:
    60

    Attached Files:

  12. KageKirin

    KageKirin

    Joined:
    Jan 22, 2014
    Posts:
    60
    So, question time: I update the UpdateBonePositionsJobController.cs to use the job system b/c figures, copying the data could be done in parallel in a burst-compiled job.
    (Also this would be the first step to move to ECS if DOTS were working correctly in 2021.1, but let's leave this issue aside for now.)

    The attached code compiles, but I get the following runtime error:


    InvalidOperationException: UpdateBonePositionsJob.transformAccessArray.m_TransformArray uses unsafe Pointers which is not allowed. Unsafe Pointers can lead to crashes and no safety against race conditions can be provided.
    If you really need to use unsafe pointers, you can disable this check using [NativeDisableUnsafePtrRestriction].
    Unity.Jobs.LowLevel.Unsafe.JobsUtility.ScheduleParallelForTransform (Unity.Jobs.LowLevel.Unsafe.JobsUtility+JobScheduleParameters& parameters, System.IntPtr transfromAccesssArray) <0x16c7a3010 + 0x00072> in <dc812d4b5c74494fae1251d76d294592>:0
    UnityEngine.Jobs.IJobParallelForTransformExtensions.Schedule[T] (T jobData, UnityEngine.Jobs.TransformAccessArray transforms, Unity.Jobs.JobHandle dependsOn) (at /Users/bokken/buildslave/unity/build/Runtime/Jobs/Managed/IJobParallelForTransform.cs:86)
    UpdateBonePositionsJobController.Update () (at Assets/Scripts/Jobs.v1/UpdateBonePositionsJobController.cs:68)

     

    Attached Files:

  13. KageKirin

    KageKirin

    Joined:
    Jan 22, 2014
    Posts:
    60
    Gotcha, my bad.

    The IJobParallelForTransform can't hold a TransformAccessArray, it seems. The attached version now has no runtime errors.

    Questions, though:

    1. TransformAccessArray: do I need to set transforms each frame (in Update()) before launching the job, or does it contain the transforms as a C#-typical list of references to the actual data, i.e. once updated through the animation process, the data is the same?
    2. The NativeArray<float3> (_bonePositions in my case) that I give the job each frame, is it just a reference/pointer to the same data or do I need to copy it back/set directly from the job data for ComputeBuffer.SetData() ?

    Cheers and thanks for the help.
     

    Attached Files:

    Last edited: Oct 2, 2021
  14. KageKirin

    KageKirin

    Joined:
    Jan 22, 2014
    Posts:
    60
    Additional questions about
    GraphicsBuffers
    and
    ComputeBuffers

    From the doc, I take it I can set named GraphicsBuffers per MaterialPropertyBlock. Does that mean, I could set fixed vertex attributes (i.e. bone indices and weights) through a GraphicsBuffer attached to a MaterialPropertyBlock, which in turn can be set per SkinnedMeshRenderer, meaning I would not have to modify the mesh?
    Would this idea work, or if not, what would work?


    Additional questions about
    MaterialPropertyBlock.SetConstantBuffer()
    and
    MaterialPropertyBlock.SetBuffer()

    What's the difference? What kind of parameters are set differently? What does each function entail to in the engine code?

    Cheers and thanks in advance for the help.
     
  15. DylanF

    DylanF

    Joined:
    Jun 25, 2013
    Posts:
    56
    Thanks for the info - this is working pretty well for me. There's some little issues, but they may be blending discrepancies. This is the per-bone matrix I'm sending to shadergraph for rotating stored deformation vectors:
    Code (CSharp):
    1. (this.transform.worldToLocalMatrix * skinnedMeshRenderer.bones[i].localToWorldMatrix).inverse;
    EDIT: if you ended up here, you may be looking for this solution:
    https://forum.unity.com/threads/is-...rtices-of-a-skinned-mesh.771806/#post-5138297
     
    Last edited: Jul 20, 2023