Will Unity support Realtime Compositing in the near future?

Hi,

I am a virtual production creator based in Los Angeles. I think Unity has a few edges over Unreal in terms of virtual production such as volumetric filmmaking support and Cinemachine. However, real time compositing is the most reliable real time production workflow or its final image quality and speed. This area Unity lacks thrid party or in house support.

By realtime compositing, I am referencing Zero Density Reality and support for Camera focus/aperture tracking devices such as Red Stype, Halide FX from Lightcraft and NCAM(this company has depth tracking from what I have heard).

However, after testing devices and plugins, I figured Stype is the only one supporting Unity right now, and their development kit is rather obsolete: noot sure about HDRP, but their plugin is purely based on camera class API, instead of taking advantages of the cinemachine. It just requires headache change and maintenance for it to work with the recent Unity features.

So do you guys know any stable third party developer for Realtime Compositing, that is up to date with the recent Unity features?
Also, something like Zero Density Reality would be really appreciated in the futre.

If Unity has goals in Film/TV. I think a stable real time compositing solution is a must and will seal the deal when it comes to filmmaking in real time.

Thanks!

 

hi again,

would love to hear some reponse about this.

thabks!

 

What advantages would cinemachine add if you are wanting to composite based on the movement and lens of a real camera?

 

Hmhmmm can someone give insight on whether unity has compatible comp device??????????

Reason cinemachine would be nice is that I may want to transition a shot from dolly to tracked

 

Thanks for the exlanation!

Is there a time estimate on when virtual production would be available in Unity? A documentation about how to set all of these up would be awesome. Unreal Engine has a quite detailed documentation and for Unity, we really just need something official to feel secure going forward doing virtual productions.

Personally, I would love to see Startracker from mo-sys, or Ncam having a solid supported plugin in Unity. Most of the studios use these two systems. We really feel like it is time for Unity to have official support in virtual productions because its really blowing up.

Also, is livelink to Maya possible to do in Unity?

 

Hi,

I have a few further questions:

1. It seems like there is a problem with genlock and the time in unity? I am not sure I thoroughly understand that part. But does that mean when I use off-the-shelf products like Startracker, it is not working the same way as Startracker + Brainstorm or UE4?
2. As mentioned earlier, is there a rough estimate when we can see a fully fleshed Real Time Compositing Virtual Production workflow and use it in production? (maybe a sample project from Unity)

A few suggestions I would love to make:
1.It seems like the film industry loves to use the ultimatte + brainstorm + real time rendering workflow, it has become a standard now. So we really need unity to be working with brainstorm
2.We also really need official support for Aja and Blackmagic, the git link is amazing, but is there a chance that can be a officially supported package, through package manager?
3.Mosys startracker is another almost industry standard for mid tier virtual production, we would really love Unity to be working with them and have a plugin supported officially.. instead of from unity developers
4.Essentially, we really feel like Unity should have a package targeted at Virtual Production.

I know these sound demanding, when I was at Unite LA, many of my colleagues love what Unity is doing.
But with Unreal Engine 4.20+, they have been adding supports officially, from Epics team:
1.to bring Mo-sys plugin + tons of virtual production features
2.Epic team support for Aja and Blackmagic
3.Maya Livelink,
4.Integration with Brainstorm, which make the workflow possible.
5.They also released the Virtual Studio Project, a project like Sherman, where you are given an officially set up project to do virtual production. None of these we have to experiment on our own.

I dont want to be mentioning Epics like a dick, I hate the UI of UE4 and many other aspects. However, when it comes to Film and TV, we don't have the freedom to experiement on our own. The industry needs reliable and stable official solutions, instead of relying on 3rd parties and hacking our own. I just really want to see Unity brings a matured solution, because anyways Film and TV is centered around industry professionals and not indie developers

 

Hi @Brainslum,

I have been working on virtual production tools in Unity, on the VR/AR side of things. I know that Unity has been working hard to solve this issue, but to your point, the market visibility is currently in UE4's court. They have done a great job of winning the hearts and minds of the VFX pipeline from executive producers all the way down to vfx sups, and previs companies like Halon, Third Floor, and Proof. I am currently porting my AR based virtual camera system over to UE4 just to have both platforms to offer in my toolset. What makes this tough is having Second Unit Directors look at my system, and ask "So, now we take that into Unreal?". To them, Unreal is the buzzword. That's really what needs to change.

I love the Unity development environment for the reason that it is easily extensible. I've been developing with it for over a decade, and the custom package setup is quite the moving target. Presets or a template project could resolve that, I guess. But it is frustrating working as a visualization supervisor, peddling Unity, while at the same time, feeling a bit left out of the loop. Maybe that's my own fault. Is there a white paper from Unity with a basic layout of how they would approach all of this?

Unity is a tool originally designed to democratize game development, and they have done an amazing job at transforming that industry, and I do believe all is not lost. Unity still has the opportunity to get a foothold in the industry, and have made some progress in doing so by acquiring Digital Monarch for this purpose. Those guys are industry super pros, and have deep knowledge of visual effects pipelines, and how all of this should be working in terms of mo-cap stages and the larger visual effects production setups. But yes, it would be better to leverage the industry standard tools to integrate Unity in the same way they have UE4 into the Black Magic toolset, or Mo-Sys. And maybe they are having these conversations with these companies as well. Who knows? I know Unity can do this as they are larger than they've ever been with a lot of smart guys from industry.

Time will tell and I am still hopeful that they will get some traction. If we can help with that process with a little inclusion, even better.

Cheers.

 

almost makes me wonder what Unity are doing since this thread is nearing 1 year old and nothing yet on the Virtual Production front, except patches thrown up on Github, sadly.

Implementations such as UnityRecorder, have short comings like not being able to 'record' values of camera focus/aperture etc.
I think it's up to the community to cobble up solutions, meanwhile some product manager at Unity is getting paid to actually do this and is lax on the job.

Here's a Facebook group that hopefully knowledgeable and passionate people such as @Brainslum , @Spookytooth3D and others find useful and contribute to: https://www.facebook.com/groups/892348284544650/

@Matjio May I request you to bring this up at the next Unity in-house meeting please?

 

Hi there - we have started integrating MoSys Startracker - feel free to reach out if interested. Cheers Allan@thevolumecph.dk

 

Thank you for the information you provided.

 

Hey Unity what's the status of virtual production R&D? Can't see anything related on the roadmap. UE is pushing hard in this area, have you marked the battle as lost already?