Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Unite 2023 Registrations are now LIVE! We are thrilled to announce that Unite 2023 Registration is now live and open for all!
    Dismiss Notice
New Forum User Notice Unite 2023 Registrations are now LIVE!

Question High Res 360 video approaches and optimisations

Discussion in 'Audio & Video' started by paulinventome, Dec 3, 2021.

  1. paulinventome

    paulinventome

    Joined:
    Mar 8, 2019
    Posts:
    4
    Hi All,

    Relatively new to Unity but long time developer.

    I am looking at whether I can use Unity in a CAVE environment where i have 270 degrees of view.

    I need to play 360 video source across that 270 degrees of view. I don't have a floor screen and the ceiling is partly there so the focus is front, left and right.

    I need about 6K on each wall, which would need 24k res all around.

    I was thinking that mapping video to a skybox could be efficient as a skybox won't have any lighting calculations and i would assume it would be culled quickly for the parts that cannot be seen.

    But i am wondering whether there are issues with the size of the textures being used in these calculations or whether i should be mapping or projecting the video source onto geometry in the scene.

    I am assuming the build in renderer or the URP would be most efficient?

    If i take a single movie file i wonder if i could map different quarters of the movie file for the 4 cubic surfaces i am really looking for?

    Has anything done anything like this, is there anything i can read up on?

    Kindest
    Paul
     
    paulinventome, Dec 3, 2021
    #1

  2. DominiqueLrx

    DominiqueLrx

    Unity Technologies

    Joined:
    Dec 14, 2016
    Posts:
    256
    Hi Paul!

    Unity has a builtin panoramic shader that is meant to be used with the skybox, a bit like what you are describing. However, it is meant to be used in 180 and 360 degrees spheres, or a with a 6-face cube. Therefore it seems you'll have to write your own shader in order to intercept each rectangle in the video source (and produce black on non-used planes). You could also set the VideoPlayer to the RenderTexture render mode and then adjust the tiling/offset of each of the planes on which you want to map the result, if this strategy works for you.

    This being said, the resolution you're describing will probably not fit in any of the video formats supported by Unity. Even though the H.264/H.265 specification does support higher than 4K, actual implementations typically support around 4K as a max (e.g.: on Windows, this would typically be 4096x2304). Some Windows systems have GPU-specific deciders that will let you go up to 8K, but it's safer not to rely on this. You could try having four 4K movies, each using its own VideoPlayer, that you'd keep synchronized, but you may perceive some desync once in a while (how often depends on whether the content moves really fast or not).

    If you choose to go with webm files using the vp8 codec, then you can fit a much higher resolution in there: the theoretical limit is 16384x16384. But as you go higher, you'll find that there may be performance bottlenecks that will prevent the VideoPlayer to maintain a useful frame rate; this will depend on the system you run this on.

    Finally, there is the issue of texture size. If we go with the size you have in mind (4 rectangles of 6K, which is 4 times 6144x3160, or 12288 x 6320. This is probably beyond the limit of anything that's reasonable to expect. Even though the individual objects you are targetting may be using a 6K texture, the VideoPlayer will need at some point a single texture that contains the full rectangle, and this may not work reliably everywhere.

    So: you'll have to experiment within the indications/warnings I've enumerated above. If you provide us with information about the platform and content you intend to use, as well as with the results of your experiments, maybe we can guide you a little further. This scenario is clearly "on the edge", but maybe you'll find acceptable compromises.

    Hope this helps, let us know how it goes!

    Dominique
    A/V developer at Unity.
     
    DominiqueLrx, Dec 6, 2021
    #2
    paulinventome likes this.

  3. paulinventome

    paulinventome

    Joined:
    Mar 8, 2019
    Posts:
    4
    Dominique,

    Thank you so much for replying. Here are some more details.

    Firstly i am hoping to achieve this off a single workstation/server rather than having to go down the clustering route which brings it's own complexities, not to mention costs.

    In terms of hardware, it's an open book right now, it would be custom built for this application so part of what i am trying to do is to understand the bottlenecks. I can easily set up PCI4 based NVMe RAID card, i can use Threadripper or Epyc for PCI Lanes and cores and i can use A5000 cards. So on bandwidth/storage/CPU/GPU i can focus on what might work the best.

    Q: Are there any guidelines on where i might see these bottlenecks and is there a good way in Unity to monitor this? The profiler adds it's own overhead so i don't know that is the best route - do i watch with a task manager to see how everything is going?

    So i can also use the HAP or Notch codec as well, these appear to support the higher res formats. I've found that delivering a 360 equirectangular is very wasteful in my scenario because i have no floor, back and the ceiling is lower res. So really i should be focused on finding my source movie limit and then chopping up what is 'projected' onto the front, left and right with a lower res for the ceiling. I believe a 16k x 2k for the main walls would give me pixel for pixel. So as you say it looks like I might need to write a custom shader - are there any source examples for something like the projection shader or panoramic shader?

    Q: Is it better to focus on a single source video and take different segments from that. Or is it better to use multiple smaller movies? I am assuming a single one would be most efficient?

    In terms of my specific content it would be useful for me to take an external video feed from a decklink card at 4K and map that onto a 2D surface ontop of the background being played - sound feasible?

    A single GPU would give me 4 outputs at 5K each. But i know that Unity supports 8 displays

    Q: If i used multiple A5000 cards, well 2, then how does Unity use the cards? If i have say 3 and 3 outputs connected (6 in total) would this actually work? Is there any configuration with multiple GPUs that would work best for me?

    And then finally

    Q: What is the most efficient rendering pipeline for this? Built in or URP? (I assume HDRP has a higher overhead). As in going forward, should i target URP?

    I really, really appreciate any advice, will save me a lot of testing!

    Kindest
    Paul
     
    paulinventome, Dec 7, 2021
    #3