Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

Will Unity support Realtime Compositing in the near future?

Discussion in 'Film, TV, Animation & Cinematics' started by Brainslum, Aug 5, 2019.

  1. Brainslum

    Brainslum

    Joined:
    Jan 10, 2017
    Posts:
    48
    Hi,

    I am a virtual production creator based in Los Angeles. I think Unity has a few edges over Unreal in terms of virtual production such as volumetric filmmaking support and Cinemachine. However, real time compositing is the most reliable real time production workflow or its final image quality and speed. This area Unity lacks thrid party or in house support.

    By realtime compositing, I am referencing Zero Density Reality and support for Camera focus/aperture tracking devices such as Red Stype, Halide FX from Lightcraft and NCAM(this company has depth tracking from what I have heard).

    However, after testing devices and plugins, I figured Stype is the only one supporting Unity right now, and their development kit is rather obsolete: noot sure about HDRP, but their plugin is purely based on camera class API, instead of taking advantages of the cinemachine. It just requires headache change and maintenance for it to work with the recent Unity features.

    So do you guys know any stable third party developer for Realtime Compositing, that is up to date with the recent Unity features?
    Also, something like Zero Density Reality would be really appreciated in the futre.

    If Unity has goals in Film/TV. I think a stable real time compositing solution is a must and will seal the deal when it comes to filmmaking in real time.

    Thanks!
     
  2. Brainslum

    Brainslum

    Joined:
    Jan 10, 2017
    Posts:
    48
    hi again,

    would love to hear some reponse about this.

    thabks!
     
  3. OldLegWig

    OldLegWig

    Joined:
    Mar 18, 2018
    Posts:
    92
    What advantages would cinemachine add if you are wanting to composite based on the movement and lens of a real camera?
     
  4. Brainslum

    Brainslum

    Joined:
    Jan 10, 2017
    Posts:
    48
    Hmhmmm can someone give insight on whether unity has compatible comp device??????????

    Reason cinemachine would be nice is that I may want to transition a shot from dolly to tracked
     
  5. Matjio

    Matjio

    Unity Technologies

    Joined:
    Dec 1, 2014
    Posts:
    32
    Hi @Brainslum,

    Sorry for the delay, I was off for holidays.

    We have been focused in the past months on the pipeline (Python, Shotgun, OpentimelineIO, Recorder, Alembic, USD, Renderpasses) and unfortunately could not start focusing yet on Video IO and tracking.

    We are still definitely considering better off the shelf support for virtual production and the various features that we need to build for this usage, but we just had no bandwidth yet.

    However we have made a few things which could hopefully help:
    -Unity Japan has worked on a first POC for video IO with BlackMagic card (Decklink): it is available here: https://github.com/keijiro/Klinker (open source and could very probably be adapted to AJA).
    -Our video team has been working with a few studios to propose a way to have precise frame rate in Unity to do genlock: https://blogs.unity3d.com/2019/06/03/precise-framerates-in-unity/
    -we have worked at HackWeek on a Virtual Studio POC to start experimenting with the different pieces of tools that we would need to build. We would like to clean up and release some of this work as soon as we have bandwidth to do so.

    Also note that Solidanim+école Melies + INRIA have been working on very interesting solutions to do virtual production with Unity including Lens distortion, tracking with Vive pro, and control of zoom and focus: https://twitter.com/solidanim/status/1136242249732370433

    Re. Plugins with NCam and Stype I was not aware of the issues. The best would be to contact the representatives to ask if they have an updated version of their plugins. They are usually supportive of their customers using any engine. I am very happy to help if there is any issue on the way.

    DM me if you need introductions.

    Best,
    Mathieu
     
  6. Brainslum

    Brainslum

    Joined:
    Jan 10, 2017
    Posts:
    48
    Thanks for the exlanation!

    Is there a time estimate on when virtual production would be available in Unity? A documentation about how to set all of these up would be awesome. Unreal Engine has a quite detailed documentation and for Unity, we really just need something official to feel secure going forward doing virtual productions.

    Personally, I would love to see Startracker from mo-sys, or Ncam having a solid supported plugin in Unity. Most of the studios use these two systems. We really feel like it is time for Unity to have official support in virtual productions because its really blowing up.

    Also, is livelink to Maya possible to do in Unity?

     
  7. Brainslum

    Brainslum

    Joined:
    Jan 10, 2017
    Posts:
    48

    Hi,

    I have a few further questions:

    1. It seems like there is a problem with genlock and the time in unity? I am not sure I thoroughly understand that part. But does that mean when I use off-the-shelf products like Startracker, it is not working the same way as Startracker + Brainstorm or UE4?
    2. As mentioned earlier, is there a rough estimate when we can see a fully fleshed Real Time Compositing Virtual Production workflow and use it in production? (maybe a sample project from Unity)

    A few suggestions I would love to make:
    1.It seems like the film industry loves to use the ultimatte + brainstorm + real time rendering workflow, it has become a standard now. So we really need unity to be working with brainstorm
    2.We also really need official support for Aja and Blackmagic, the git link is amazing, but is there a chance that can be a officially supported package, through package manager?
    3.Mosys startracker is another almost industry standard for mid tier virtual production, we would really love Unity to be working with them and have a plugin supported officially.. instead of from unity developers
    4.Essentially, we really feel like Unity should have a package targeted at Virtual Production.

    I know these sound demanding, when I was at Unite LA, many of my colleagues love what Unity is doing.
    But with Unreal Engine 4.20+, they have been adding supports officially, from Epics team:
    1.to bring Mo-sys plugin + tons of virtual production features
    2.Epic team support for Aja and Blackmagic
    3.Maya Livelink,
    4.Integration with Brainstorm, which make the workflow possible.
    5.They also released the Virtual Studio Project, a project like Sherman, where you are given an officially set up project to do virtual production. None of these we have to experiment on our own.

    I dont want to be mentioning Epics like a dick, I hate the UI of UE4 and many other aspects. However, when it comes to Film and TV, we don't have the freedom to experiement on our own. The industry needs reliable and stable official solutions, instead of relying on 3rd parties and hacking our own. I just really want to see Unity brings a matured solution, because anyways Film and TV is centered around industry professionals and not indie developers
     
    Spookytooth3D and LostPanda like this.
  8. Spookytooth3D

    Spookytooth3D

    Joined:
    Oct 31, 2013
    Posts:
    73
    Hi @Brainslum,

    I have been working on virtual production tools in Unity, on the VR/AR side of things. I know that Unity has been working hard to solve this issue, but to your point, the market visibility is currently in UE4's court. They have done a great job of winning the hearts and minds of the VFX pipeline from executive producers all the way down to vfx sups, and previs companies like Halon, Third Floor, and Proof. I am currently porting my AR based virtual camera system over to UE4 just to have both platforms to offer in my toolset. What makes this tough is having Second Unit Directors look at my system, and ask "So, now we take that into Unreal?". To them, Unreal is the buzzword. That's really what needs to change.

    I love the Unity development environment for the reason that it is easily extensible. I've been developing with it for over a decade, and the custom package setup is quite the moving target. Presets or a template project could resolve that, I guess. But it is frustrating working as a visualization supervisor, peddling Unity, while at the same time, feeling a bit left out of the loop. Maybe that's my own fault. Is there a white paper from Unity with a basic layout of how they would approach all of this?

    Unity is a tool originally designed to democratize game development, and they have done an amazing job at transforming that industry, and I do believe all is not lost. Unity still has the opportunity to get a foothold in the industry, and have made some progress in doing so by acquiring Digital Monarch for this purpose. Those guys are industry super pros, and have deep knowledge of visual effects pipelines, and how all of this should be working in terms of mo-cap stages and the larger visual effects production setups. But yes, it would be better to leverage the industry standard tools to integrate Unity in the same way they have UE4 into the Black Magic toolset, or Mo-Sys. And maybe they are having these conversations with these companies as well. Who knows? I know Unity can do this as they are larger than they've ever been with a lot of smart guys from industry.

    Time will tell and I am still hopeful that they will get some traction. If we can help with that process with a little inclusion, even better.

    Cheers.

     
    Dirrogate, aka_anoop and BCFEGAmes like this.
  9. Dirrogate

    Dirrogate

    Joined:
    Feb 18, 2014
    Posts:
    120
    almost makes me wonder what Unity are doing since this thread is nearing 1 year old and nothing yet on the Virtual Production front, except patches thrown up on Github, sadly.

    Implementations such as UnityRecorder, have short comings like not being able to 'record' values of camera focus/aperture etc.
    I think it's up to the community to cobble up solutions, meanwhile some product manager at Unity is getting paid to actually do this and is lax on the job.

    Here's a Facebook group that hopefully knowledgeable and passionate people such as @Brainslum , @Spookytooth3D and others find useful and contribute to: https://www.facebook.com/groups/892348284544650/

    @Matjio May I request you to bring this up at the next Unity in-house meeting please?
     
    LostPanda likes this.
unityunity