Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice

Question How to handle advanced sprite swapping with a 2D skeletal rig?

Discussion in '2D Experimental Preview' started by ibbitzzz, Nov 28, 2020.

  1. ibbitzzz

    ibbitzzz

    Joined:
    Jun 15, 2018
    Posts:
    9
    Hi all,

    I'm looking to create a skeletal rig for my game that can dynamically swap out various parts based on its rotation/context. For reference, here are some things that I want my rig to be able to do the following:
    • Swap out hands (open hand, fist, back of hand, pointing)
    • Use a different head depending on what direction the player is looking
    • Change mouths for lip-syncing/expressions
    From what I understand, this can likely be accomplished using Sprite Libraries. However, all of the use cases I've seen for sprite swapping have been to change the appearance of limbs while keeping the same visual more or less (eg: people using it to show damage or for changing outfits). This means the different parts are generally on different sprites. For my case, the parts I want to swap out would most likely be part of the same sprite sheet, as they're all part of the same state.

    So far, this process seems to be the closest to what I'm looking for, but the process seems very clunky and I can't skin the whole sprite at once.



    I imagine no matter what route I take I'll need to do some scripting, but I want to make the process as simple and maintainable as possible. I just don't know if this is the least painful approach.
     
  2. Ted_Wikman

    Ted_Wikman

    Unity Technologies

    Joined:
    Oct 7, 2019
    Posts:
    791
    Hello @ibbitzzz,
    The creator of the video you linked is working in .png files, which works, but can be a bit clunky. If you are able to, I would suggest using our PSB flow (as described here). This way you keep all the sprites in one (.psb) file and it will be laid out in the same way as inside Photoshop. If you have multiple parts that you want to switch between, you can put them in different layers inside Photoshop and inside Unity, you only work with one rig.

    To swap the parts in a more dynamic way, you will need to write some scripts to get the outcome you are after.

    Let me know if you have any further questions or thoughts
     
  3. Rafarel

    Rafarel

    Joined:
    Jul 21, 2017
    Posts:
    199
  4. ibbitz

    ibbitz

    Joined:
    Sep 8, 2020
    Posts:
    8
    Hello all, thanks for your replies so far!

    Yep, this is what I'm trying to accomplish. The video I posted above also uses Sprite Library Assets for this. I'd like to be able to use this to make a detailed rig, but am getting hung up on the details for how to implement such a level of detail.

    To help explain what I need my rigs to be capable of I made a small MSPaint diagram.

    upload_2020-12-3_14-24-28.png
    As you could guess there's a different number of torsos, arms, hands, mouths, etc. So the way I see it it's not possible to just make a rig for the front view, side view, etc. Additionally, there's plenty of animations that require swapping between these variants, so the only solution as I see it is to make all parts swappable within the same rig.

    Taking a look at your provided resources, I did have followup questions:
    • When using the PSD Importer, is it essentially doing the same thing that the video is doing to create a rig, just using the layer info to position? Or are there other things that happen behind the scenes?
    • Is PSD Importer really required to import these as a rig/generate the prefab? It seems like a weird design choice to give tool-agnostic projects a worse experience and prevent them from being able to define default poses.
    • How would sprite swapping even work for foreshortened limbs
    • I may need to do some squash & stretch for certain animations to help with their fluidity, are there already known ways of accomplishing this? The simplest approach is probably to change the part's scale, but can I do something like enlongating an arm?
    Would you be able to point me in the right direction for this? Not really sure where to start with updating bones, rebinding poses (assuming that's what is needed to accomplish this kind of rig)
     
    Last edited: Dec 3, 2020
  5. Ted_Wikman

    Ted_Wikman

    Unity Technologies

    Joined:
    Oct 7, 2019
    Posts:
    791
    Hello @ibbitz ,
    Thanks for that detailed write-up and the image to clarify what you are trying to accomplish. What you are going for is quite a complex setup. I am however excited to hear more about your take on solving the tasks.

    Onto your questions:
    • When using the PSD Importer, is it essentially doing the same thing that the video is doing to create a rig, just using the layer info to position? Or are there other things that happen behind the scenes?
    The PSD Importers job is to turn a PSB file into one asset file which contains all the relevant information that was exported from Photoshop. The product produced is a Model Prefab (the same format as if you import an FBX into Unity), which contains GameObjects for each layer defined in the PSB file with SpriteRenderers and SpriteSkin components and their positions set to correspond to the positions in the PSB file. The Model Prefab also contains a Sprite Atlas with all the separate textures laid out to maximise atlas usage. Note that it does not generate a rig, but rather sets the asset up for easy rig creation.
    • Is PSD Importer really required to import these as a rig/generate the prefab? It seems like a weird design choice to give tool-agnostic projects a worse experience and prevent them from being able to define default poses.
    The PSD Importer is not required, but advised. As your video showed, you can use the 2D Animation toolset without PSD Importer, but it will not be as convenient as using PSD Importer with PSB files. I agree with you that having as much of a DCC tool agnostic flow as possible is the preferred way to go, however, we have to start somewhere and Photoshop with PSB files was the starting point we selected.
    • How would sprite swapping even work for foreshortened limbs
    To foreshorten limbs in the most basic way, you can scale the bones skinned to the limbs. If you then sprite swap, the sprite will be swapped out, but the scale of the bones will stay, giving the new sprite the same deformation, if it was skinned in the same way as the previous sprite. Note that foreshortening limbs tends to also include some faked rotation accomplished by techniques like FFD, which you can do in 2D Animation, but is not as convenient as other DCC tools which wholeheartedly support FFD.
    • I may need to do some squash & stretch for certain animations to help with their fluidity, are there already known ways of accomplishing this? The simplest approach is probably to change the part's scale, but can I do something like enlongating an arm?
    Yes, you can accomplish squash and stretch by scaling the bones. I came across this video the other day where a developer used the 2D Animation toolset with Physics2D to get some pretty neat squash and stretch effects.

    In closing, I think you need to do a few experiments with the tools. Break the task you are trying to accomplish down into smaller tasks/experiments to see how the tools available for you can accomplish what you set out to do. Every developer will work and solve development challenges in different ways, and I am excited to hear how you approach these tasks.
     
    ibbitz likes this.
  6. ibbitz

    ibbitz

    Joined:
    Sep 8, 2020
    Posts:
    8
    Thanks for your detailed writeup Ted, this answers a lot of confusion points for me! :)

    Totally fair point. As a fellow software engineer, I understand it's probably a product decision & it gave the simplest workflow for 99% of use cases. Hope to see more as the project is fleshed out!

    As far as design goes, my initial thoughts would be to define a few scripts to manage rotations/connections:
    • AnimationRigPart
      • Manages rotation along the axis of the part's root bone
      • Maps each SpriteLibrary sprite to a rotation value
      • Traverses Transform hierarchy to determine actual rotation
        • eg: If torso is rotated, limbs should also rotate to match
      • On Update/OnValidate change sprite to match rotation
    • TorsoAnimationRig
      • Subclass of AnimationRigPart
      • Maps limb locations to a SpriteSkin bone & an offset for each sprite in library
        • Unsure of best way to do this; Quaternion against bone vector or lock it to a point on the mesh?
      • Uses rotation to update the render order/depth of each limb's sprite
    • LimbAnimationRig
      • Subclass of AnimationRigPart
      • Handles foreshortening using trig functions to scale bones (?)
      • Blends IK constraints of the limb based on rotation
      • Uses rotation to update the depth of each child sprite (eg: if forearm is in front of bicep)
    With something like this, I think I would be able to just animate the rotation values of each AnimationRigPart and the scripts could dynamically adjust the sprites as needed. Other things like hands or mouths and such I can probably just change the sprite manually. For now, squash and stretch can probably just be accomplished via scaling & if I need to bring in FFD then I can probably explore it more then.

    I guess just 2 other questions stem from this:
    • For positioning limbs on a deformed torso sprite, is there an ideal approach? I'm guessing that since the geometry weights affect deformation it's not the most straightforward answer.
    • I can see that bones are accessible through the SpriteSkin, but is there a way to access/modify the mesh procedurally?
     
  7. Ted_Wikman

    Ted_Wikman

    Unity Technologies

    Joined:
    Oct 7, 2019
    Posts:
    791
    Thanks for sharing the outline of your design. I think it sounds interesting to connect the rotation of the bones with sprite swap. For extreme pose changes (facing left to facing forward), you might want to consider having two separate rigs and characters to swap between, since the bones will most likely be very different in layout and scale.

    For positioning limbs on a deformed torso sprite, is there an ideal approach? I'm guessing that since the geometry weights affect deformation it's not the most straightforward answer
    No, I don't think there is an ideal approach and is more up to a per implementation basis. Note though that if you scale one of your bones non uniformly, the child bones will be skewed, since the child bones lives in the parent bones transform.

    I can see that bones are accessible through the SpriteSkin, but is there a way to access/modify the mesh procedurally?
    You can read but not write vertex position data with the current SpriteSkin API. Out of curiosity, what would you like to do with the mesh data?
     
  8. ibbitz

    ibbitz

    Joined:
    Sep 8, 2020
    Posts:
    8
    My thought was that by using the mesh data that I could effectively lean on the existing SpriteSkin logic to handle limb repositioning. Given a starting position for the limb, I could find the triangle that the point is in (or at least closest to), and adjust the position as the mesh (and consequently the triangle) deforms.
    One other idea I had that was more of a stretch would be to pull the verticies closer/farther from a bone for more control over squash/stretch, but given that vertices are not writeable its not critical.
     
  9. Ted_Wikman

    Ted_Wikman

    Unity Technologies

    Joined:
    Oct 7, 2019
    Posts:
    791
    I see, thanks for sharing
     
  10. tonytopper

    tonytopper

    Joined:
    Jun 25, 2018
    Posts:
    116
    One thing that could help developers tackle this problem is clarifying how the Sprite Resolver works with the Sprite Skin, bones, weights, meshes, etc.

    I want to swap hands, feet, etc, much like the OP was hoping to do. But after getting the sprite swapping up and running it's unclear how the bones from the different sprite skins work together.
     
  11. Ted_Wikman

    Ted_Wikman

    Unity Technologies

    Joined:
    Oct 7, 2019
    Posts:
    791
    Thanks for the insight @tonytopper. We have a section dedicated to Sprite Swapping in our 2D Animation documentation, but it seems like you would like some deeper understanding into what happens when a Sprite is changed.

    So first things first, the Sprite Resolver component is not much more than a tool to change the SpriteRenderer.sprite property. It doesn't have any links to the Sprite Skin system, and doesn't care if a Sprite has bones or not. Its only mission is to be able to change the SpriteRenderer.sprite property.

    Sprite Skin is where the magic is at. OnEnable and on every frame, Sprite Skin checks if the Sprite Renderer has changed its sprite property or not. If the sprite property is changed, a few things happens:
    • The vertex buffer inside the Sprite Renderer is reset to its static undeformed form.
    • The Sprite Skin caches the new Sprites GUID, UVs, vertex positions, tangents (if any), bone weights and bind poses.
    Once the Sprite Skin reaches its Deform call, it will use the newly cached data to deform and then pass a deformed vertex buffer down into the Sprite Renderer to be rendered on screen.

    Note that we do not, by default, update the list of bones (Transforms) being used to drive the deformation when the Sprite has been changed. This is because in its current form, there is a slight CPU and GC Alloc overhead if we were to update the list of bones every time the Sprite is changed. However, if you would like to have the list of bones updated as well, you can enable the Auto Rebind flag, which will enable the Sprite Skin to look for bones (Transforms) in the hierarchy and update its internal list of bones. This is useful if you have two Sprites with different bone structures that you would like to swap between, where both bone structures are present in the hierarchy.

    Hopefully this explanation helps to clarify a bit what is going on behind the scenes when a Sprite is changed.
     
  12. tonytopper

    tonytopper

    Joined:
    Jun 25, 2018
    Posts:
    116
    Thank you @Ted_Wikman for the quick reply. This definitely helps me get my head around things better.

    As I experiment, it seems having the exact same skeleton is the limiting factor I wasn't expecting. And within that having the same bone list. Some sort of skeleton fragment system might be nice. I might have 20 different hands but just 3 different heads and 100 different bottoms. It might also be nice to have some sort of X, Y flipping feature available in the resolver.

    It sounds like for sprites that get swapped often you don't want Auto Rebind turned on if you can avoid it anyway.

    I am noticing that my bone bindings break in multiple ways in the editor if I have a different bone list and swap the sprites using the Sprite Resolver.

    upload_2021-12-3_13-17-48.png

    upload_2021-12-3_13-18-18.png

    You can see it doesn't map the bones as anticipated. It looks like the list of bones needs to be ordered the same. Is that right?

    The Entry_0 sprite ends up not being influenced by any of the bone positions, which I am guessing is because there are "unassigned references".

    I started my experimentation with an extreme example here so I could figure this out by using a bikini bottom versus a long dress. I was anticipating something would break. Recommendations on how to handle this would be great.

    Would it be better to just make the bone list match even if a particular sprite isn't weighted by that bone?

    I had copied and pasted the bone rig. Next, I am going to try to use the "Main Skeleton" field to directly reference the skeleton that way.
     
  13. tonytopper

    tonytopper

    Joined:
    Jun 25, 2018
    Posts:
    116
    Managed to fix this issue by making the bone list the same and in the same order. So even though the bikini bottom doesn't use the tibia bone at all, we keep it in the list of influences anyway.

    Is there any issue with this?
     
  14. Ted_Wikman

    Ted_Wikman

    Unity Technologies

    Joined:
    Oct 7, 2019
    Posts:
    791
    So, we will only scan the bone hierarchy from the Root Bone and downwards. If you are swapping between two bone hierarchies, you need to help the Sprite Skin by assigning the correct Root Bone when switching between hierarchies.

    If you have two Sprites which uses the same Bone hierarchy, but one of the Sprites only uses a subset of the bones in the hierarchy, it will cause an issue when switching.

    I.e. we have the bone hierarchy:
    A
    -- B
    ---- C
    ------ D
    And the first Sprite uses A -> D, but the second Sprite only uses A, B and D (leaving out C), this will cause an issue. Make sure the Sprites knows about all the bones from its lowest child to its root, even if it doesn’t register any weight to the bone. This is so that we can correctly map the skeleton hierarchy with the transform hierarchy in the scene.

    Seems like you managed to find a solution in your last message, but let me know if there are still any unanswered questions left.

    We are currently looking into simplifying this flow in a future version of the package, so these conversations have been great at highlighting how complex the current system is, and the general need to remove some of its quirks.
     
    tonytopper likes this.
  15. jsip

    jsip

    Joined:
    Jan 25, 2015
    Posts:
    6
    I think the easiest and most intuitive way to handle this would be allowing us to map multiple sprites to a single bone in the Sprite Editor - or just make sure the SpriteRenderer component is attached to each bone.

    If a sprite is to have a mesh - that's good - but meshes only go so far in animation and the ability to swap sprites on a skeleton for different body part perspectives easily is a huge must-have WITHOUT jumping through hoops setting up complete alternative rigs, using library components, etc - it's not intuitive and has a LOT of drawbacks.

    You can work around this by simply slicing up your sprite sheet, adding your bones, then dragging the individual sprites from the pack as objects, adding the Sprite Skin and assembling the object yourself which will then have a SpriteRenderer component attached.

    In other applications like Spine2D, sprites on bones are treated as attachments so you can swap sprites as needed as you go through for different hand gestures, perspectives, etc. As it stands it's incredibly cumbersome to do this one little thing in Unity while I feel the overall bone system and mesh is faster and cleaner inside of Unity. I can rig characters faster but this is where everything tends to fall apart and is very much so needed for serious artists.

    It seems like the current swap system was more for swapping entire characters to use the same skeleton that all have similar shapes and implementing the Libraries for single items rather than just seeing a SpriteRenderer attached to each bone is a bit weird.

    We are able to reposition bones and tweak skeletons during animations so for simple humanoid stuff I don't even see a real use-case for entire copy/paste swaps with libraries as it is far easier to just swap all the sprites on each bone (if the renderer component is attached) than it is to re-rig an entire skeleton with a different skin and use the cumbersome Library feature.

    Making a SpriteRenderer component accessible on each bone/limb would be an easier (and faster) approach.

    Just my .02
     
    prokhorvlg likes this.
  16. prokhorvlg

    prokhorvlg

    Joined:
    Sep 12, 2021
    Posts:
    6
    Just wanted to express my agreement on all points with jsip.

    Replacing individual sprites on a rig during runtime has been a pain, especially as my project has multiple rigs with shared parts and animations. It would be a godsend if there was a way to just slap on an individual sprite replacement without having to make sure it conforms to the bone structure of the rig it's going onto!

    Otherwise, the 2d animation tools have been fantastic, but this feature certainly needs work. Please let me know if anything addressing this is in the works; I would prefer to delay working on this aspect of my project until there's an easier way to swap out individual sprites.
     
  17. Ted_Wikman

    Ted_Wikman

    Unity Technologies

    Joined:
    Oct 7, 2019
    Posts:
    791
    Thanks for the feedback @jsip & @prokhorvlg.
    First off, to give some context on the structure we currently have. One of the things we try to push for, tool-design wise, is to not force users into fixed structures/convoluted setups in order to have a component work. The setup we have for Sprite Skin right now is that you pull out the graphics into the scene, add a Sprite Skin component to the graphics and can then generate the bone structure in order to make the component work as intended. This flow is very easy to learn, and require a very minimal setup in order to see the result.

    Going the bone/skeleton first approach forces users to do more work before the result can be seen and tested. Once the skeleton is updated, the hierarchy the Sprite Renderer is attached to would also have to be updated, in order for the Sprite Skin components to work properly.

    This is not to say that we shouldn't look into and improve the current system though! It is great to know where the tools currently fail you, and where we should focus some effort in improving it.

    We do not have any work planned for this as of right now, so do continue with your current implementation plans.

    A question back to you two, this Sprite Renderer you would like to swap the Sprite in, is it being deformed? or does it only have one bone attached to it? or maybe even 0 bones and just moving according to the movements of a parent transform (e.g. rigid attachment in the hand, like a bottle or a sword)?
     
    prokhorvlg likes this.
  18. prokhorvlg

    prokhorvlg

    Joined:
    Sep 12, 2021
    Posts:
    6
    Appreciate the response!

    In my case, the character is a robot so there is no deformation of the sprite, but it is a part of the robot and has a single bone linked to it. To be more specific about my use case; I'm looking to implement a number of shared sprites for hand/arm components that I can swap out based on the specific situation (holding a weapon, leaning against wall, balled up in a fist, relaxed, etc).
     
  19. Ted_Wikman

    Ted_Wikman

    Unity Technologies

    Joined:
    Oct 7, 2019
    Posts:
    791
    Thanks for sharing!

    If you only have one bone associated with the Sprite, the current 2D Animation solution should perform well. You mentioned that you have multiple rigs and you share parts (Sprites?) between them. Is this where our current system causes issues? If so, what kind of issues are you seeing there?
     
  20. tonytopper

    tonytopper

    Joined:
    Jun 25, 2018
    Posts:
    116
    Given that the thread's title is "How to handle advanced sprite swapping with a 2D skeletal rig?", I am surprised that no one has mentioned pivot points yet. Manually setting each sprite's pivot point in the editor isn't scalable for me.

    And I can't find a way to set the sprite pivot via script using SpriteAlignment: https://docs.unity3d.com/2021.2/Documentation/ScriptReference/SpriteAlignment.html

    This would make rigging with sprite swapping in mind much easier in my opinion. Writing the code to set the pivot using a Vector2 relative to the original texture seems like code I'd rather not have to write myself given that Unity must have that code somewhere. But that's the only way I can figure it out. That or creating a
    TextureImporterSettings
    https://docs.unity3d.com/ScriptReference/TextureImporterSettings.html

    But I am worried that will muck with the PSD Importer.
     
  21. Ted_Wikman

    Ted_Wikman

    Unity Technologies

    Joined:
    Oct 7, 2019
    Posts:
    791
  22. tonytopper

    tonytopper

    Joined:
    Jun 25, 2018
    Posts:
    116
    In my case, I want to use different SpriteAlignments and even custom ones. It just depends on what I am processing. For instance, an all "ear" sprites might be center right but all "hair" sprites could be top center. All "hand" sprites might be custom x=30, y=5