Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice
  3. Dismiss Notice

Question The Anime Toolbox

Discussion in 'Virtual Production' started by Harsh-NJ, Apr 4, 2023.

  1. Harsh-NJ

    Harsh-NJ

    Joined:
    May 1, 2020
    Posts:
    315
    Hey everyone!
    Came to know about the Anime Toolbox and related new packages (toonshader, visual compositor, material switch, that streaming images thing) and the MIKONOTE Sample, after I halted using my laptop since October.
    It is really awesome cause I want to make anime shorts using Unity. The tools seem promising, but the internet currently lacks resources on them.
    And, Is Anime Toolbox a package or just a set of (these) packages?


    Just wanted to know if anyone here (if any official) can recommend a suitable workflow for using these tools, and especially about the visual compositor as it looks horrible in the MIKONOTE Sample. I really want to get started making Anime in Unity

    Regards,
    Harsh Narayan Jha
     
    sindharta_at_unity likes this.
  2. akent99

    akent99

    Joined:
    Jan 14, 2018
    Posts:
    588
    I have not used the Anime Toolbox, but play with animation in Unity. If you don't have any luck with it, there are other "toon" shaders like RealToon and Poiyomi, plus the Unity Sequences package can be used for stringing multiple Timelines together for a longer episode. But you have to work out a fair bit on your own...
     
  3. akent99

    akent99

    Joined:
    Jan 14, 2018
    Posts:
    588
    Okay, wow! I had not looked at the Visual Compositor before. Complete with Stable Diffusion integration! It looks like a pretty powerful tool (which also usually means pretty complex). I cheat - I just use the game view and record animation that way. Much easier to start off.

    It looks like the compositor allows you to create your final footage from a layered stack of assets, with different visual effects applied per object. The benefit of this over just recording the game view as I understand it is you can control lighting, color, and blur etc much better per layer. I find this hard to do via the Game view window - e.g. making the background darker I sometimes resort to black fog. Instead I think this would bring in the different parts of the final shot, with the ability to apply different effects to them. E.g. make the character in the foreground black as a silhouette. Great visual control if you have the artistic skills to leverage it.

    So the Visual Compositor gives you all those controls inside Unity, but with a lot more complexity that goes along with it. The readme made the point it was set up by a professional compositor, someone who lives doing that work, so it was pretty advanced - not a basic example.

    If you wanted an easier tool to learn from, you could look at Shoost for the composition steps. https://www.patreon.com/posts/62585593 - Use Unity for animating the character, then compose the layers (with effects) using Shoost. For me, I have stuck to just recording the Game view - so its more like recording a cutscene in a game, and living with the restrictions. But I am not doing cel shading - I fell in love with all the HDRP lighting effects which did not seem to fit well with cel shading.

    I would love to hear how you go with it! Interesting stuff. (Sorry if you know all of the above - it was new to me that it was available!)
     
    Last edited: Apr 4, 2023
    Harsh-NJ likes this.
  4. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    49
    Hello and thank you for your interests in Anime Toolbox.

    AnimeToolbox is a set of the packages that you mentioned, and in addition to "Mikonote Anime Toolbox Sample", we also have a couple of simpler projects that we release periodically on Github (under Assets of each release).
    In the future, we'd like to integrate them as a part of templates in Unity Hub, so that's why these projects has "template" in their names.

    upload_2023-4-5_16-16-29.png

    You can find more information on these projects on our Github Wiki page.

    Can you please elaborate what you mean by this ?

    In addition, we have also written quite a couple of tutorials of Anime Toolbox on note.com, although unfortunately they are only available in Japanese at the moment.
     
    akent99 and Harsh-NJ like this.
  5. Harsh-NJ

    Harsh-NJ

    Joined:
    May 1, 2020
    Posts:
    315
    I had guess it’s functionality from the work compositor, but didn’t know all of that.

    btw we met again @akent99 ! How are you doing with your web comics? I am very glad to say that The Cycle got selected in 3 festivals so far.
     
    akent99 likes this.
  6. Harsh-NJ

    Harsh-NJ

    Joined:
    May 1, 2020
    Posts:
    315
    Thanks for the response!

    First I want to say I am fully interested in the new Anime tools for Unity combined with the previously released Sequencer they would be much more powerful.

    That’s great!

    The MIKONOTE Sample is a rather big one, made by a real studio and the Graph is certainly made by a good artists. It is a good resource for learning, but not for getting started. Some smaller tutorials/projects/guides will do the job for beginners. I shall try the templates you linked.

    what I wanted to know is a simple workflow in order to get my character and animations rendered into an anime episode. What can I think of (after having made an animation short with sequencer) is to get my characters and animations into Unity…. Arange them into timeline+sequencer…. And then what? What further functionalities will Anime Toolbox provide?

    plus some tutorials on the Visual Compositor would do the job right!

    All the best to the anime team at unity!
     
    akent99 likes this.
  7. akent99

    akent99

    Joined:
    Jan 14, 2018
    Posts:
    588
    Wow! That is fantastic news! Congratulations! I have a tendency to tinker more than I should. Still trying to clean up my props and locations after upgrading to HDRP (most of the grasses disappeared and I did not notice until recently!). Also been playing with some of the speech synthesis work going on (https://extra-ordinary.tv/2023/03/02/emotional-speech-synthesis-for-animation-is-here/) and more recently exploring using AI for using screenplay text to create Timeline contents. Early days. I have seen a few projects doing similar work (e.g. building a location). Interesting stuff.
     
    Harsh-NJ likes this.
  8. akent99

    akent99

    Joined:
    Jan 14, 2018
    Posts:
    588
    Thank you also for sharing all that information. I would also be interested to understand more about how it uses Timeline, and in particular is that use compatible with the Unity Sequences package, which would seem a logical combination. Do you have an alternative strategy? Or are they just not compatible? (I use the parts from the Sequences package I like - I don't use all of it.)

    I saw "storyboard 0.71 preview" as well on the list of packages. I created an extended custom version of Storyboard that allowed me to put text over an image and then used that for speech bubbles. Another potential use for the AnimeToolbox is creating comics with static poses rather than videos with animation. I created a little library of speech bubble images (speaking, thinking, etc) with tails pointing in different directions then reused those images adding the text over the top of the image (the text inside the bubble) where the text was in the Timeline clip. It was much faster to use than creating a new image per speech bubble for me. So if you were taking feature requests for the Storyboard package, that would be mine - allow text as well as images. (My code is here if of any interest - quick and dirty, but it was useful for me.)
     
    sindharta_at_unity likes this.
  9. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    49
    Yes, VisualCompositor works with Sequences
    We have an internal project that uses both of this, and you can view the results on Youtube.




    Unfortunately, we are not ready to share this project publicly yet because it's using paid assets.

    That makes sense. I was wondering if the graph appeared garbled or out of order for you, because we did have this problem in earlier versions, but it seems that it's not the case here.

    Similar to what @akent99 has pointed out, VisualCompositor allows us to
    1. separate objects into different "layers" to be rendered: for example, character layer, BG layer, etc
    2. apply 2D transforms or effects differently to these layers
    3. and blend them together after
    As an example, this will allow us to remove 3D perspective camera effects when rendering multiple cameras. In the following picture, you can see that these characters (Unity-chan) face slightly different directions because we are using a 3D perspective camera.

    upload_2023-4-6_11-14-22.jpeg

    But what we usually want in an Anime cut, is something like this where characters face exactly the same direction.

    upload_2023-4-6_11-15-7.jpeg

    To achieve this effect, we need to
    1. Put all objects in the same position in the editor (refer to the screenshot below taken from one of our Japanese tutorials)
    2. Create a RenderingNode in VisualCompositor
    3. Select what objects to render (and lights to use) in the RenderingNode via the objects (and lights) port.
    4. Apply TransformNode to move the the position of each object in 2D way
     
    Last edited: Apr 6, 2023
    akent99 and Harsh-NJ like this.
  10. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    49
    Some other tricks that we can do are:

    1. Create and apply this flowing effect, which we call ryuhai (Japanese tutorial link).



    2. Add cut in effects (Japanese tutorial link)



    Unfortunately, we only have these tutorials in Japanese, so English speakers will have to rely on translation tools at the moment.

    Our users in Japan have also experienced with VisualCompositor and came up with interesting ideas/results. Here are some that I'd like to share here:
    1. Blend text in front/behind a character easily (by Maron_VTuber).
    2. Webcomic (by the same person)
    3. Using mathematical nodes to animate one plane and blend them together (by samezi)
    4. And finally, a short fan film (by VtuberCrane)

     
    Last edited: Apr 6, 2023
    Harsh-NJ likes this.
  11. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    49
    Thank you. We'll take a look at your code and consider this.
     
  12. akent99

    akent99

    Joined:
    Jan 14, 2018
    Posts:
    588
    I am sure it could be done nicer. Here is a quick example of the result in case of interest.
     
  13. Harsh-NJ

    Harsh-NJ

    Joined:
    May 1, 2020
    Posts:
    315
    Today I went through the articles on note.com (obviously translated them to English) and learnt a lot. All I wanted is to get a head start with the Visual Compositor and how to use it, so that I won’t wander into the black screen without the knowledge of any nodes. Now I know what each node does and the kind of “correct workflow” that should be employed.

    I will soon post a step by step workflow of using the whole Toolbox on I get my hands on it and make something cool with it.

    thanks very much!
     
    tsukimi and sindharta_at_unity like this.
  14. sindharta_at_unity

    sindharta_at_unity

    Unity Technologies

    Joined:
    Jul 4, 2019
    Posts:
    49
    Might be a bit late, but we've just released a video tutorial showcasing how to perform real-time image compositing using VisualCompositor.
    The tutorial audio is in Japanese, but we provide English subtitles for non-Japanese speakers.



    If you have any feedback, please feel free to write it. We'd love to hear it.
     
    newguy123 likes this.
  15. newguy123

    newguy123

    Joined:
    Aug 22, 2018
    Posts:
    1,278
    Looking good!
    Cant wait to try it out!
     
    sindharta_at_unity likes this.