Search Unity

[VirtualCamera] Feedback

Discussion in 'Film, TV, Animation & Cinematics' started by GeniusKoala, May 18, 2021.

  1. GeniusKoala

    GeniusKoala

    Joined:
    Oct 13, 2017
    Posts:
    72
    Hey!

    First of all you did a good job witht this virtual camera system. It runs really smoothly on my iPhone11. I hoped for this system for a long time now and here it is! It allow us for plenty of uses and we are thrilling to explore all of them!

    I tried the Victual Camera Actor, not the Cinemachine one (not yet!) and here are my feedbacks so far :

    What goes good

    - Streaming quality is top
    - The device reacts really smoothly and Unity accordingly updates the scene
    - Movements, rotations, height difference (kneeling for example) and zooming work good so far
    - The interface for recording work good enough with the timer
    - Everything is really responsive. Preview mode runs the take quickly and it's exactly similar to my movements

    What goes less good

    - When I enable preview from Unity Editor, I can't quit it from my iPhone11.
    - The zoom slider is too much near the top of my iPhone11. When I want to decrease the zoom, it drags the notification bar too.
    - When the connection is lost, changing "Connect" to "Reconnect" would be great. At least, "Disconnect" and "Connect" is really quick so it's not a big deal. Just UX feedbacks!
    - The Unity Project needs to be on focus to steam data to my iPhone11.

    New features ?

    - I thought maybe having the shot list directly on my iPhone11 would be great. At the moment I can only see the last take. I think it would add more capabilities to the director directly on stage and not to depend on the computer.

    I think this system will change a lot how we can record our shots! I would be pleased to ask my studio to test it in real conditions with some of our directors. We will send you more feedbacks!

    Thanks again for your job!
     
  2. bradweiers

    bradweiers

    Unity Technologies

    Joined:
    Nov 3, 2015
    Posts:
    59
    Thanks for your feedback @GeniusKoala

    We are aware of this bug and a fix is coming shortly.

    Can you let us know which iPhone you're using? We'll get a fix out as soon as possible.

    Right now this is a limitation of how the Editor works to preserve CPU cycles when the Editor is in the background. We are in discussions about a long-term change that could help accommodate this but I don't have a timeline for you.

    We're currently working on the app UI for this. We're mostly just figuring out how to make the UX nice :)

    Please do and let us know! Thanks again for the thoughtful feedback.
     
    GeniusKoala likes this.
  3. GeniusKoala

    GeniusKoala

    Joined:
    Oct 13, 2017
    Posts:
    72
    Thanks for the reply! I read after on the documentation that the UI was based on iPad for the moment so it's ok if it's not optimal for iPhones. I am testing on my iPhone11 (classic model 64Gb) on iOS 14.4.

    Maybe having shortcuts for different zooms levels would be an interesting feature. Also the slider for the height is too close of the slider of the zoom on my iPhone11 at least!

    If the app most of all aims iPads I would understand since the screen is larger. So far technically it's working great on my iPhone11. We should invest in an iPad too.
     
  4. HIBIKI_entertainment

    HIBIKI_entertainment

    Joined:
    Dec 4, 2018
    Posts:
    203
    Super functional! I'm extremely happy about this.

    I'm impressed with the initial approaches with the system organisation too, even if streamlining the process will get better over time.
    I can't be the only one who flipped my phone in the air to see if it can handle it too :eek:

    • Is there going to be support for the DoFCoC debugger when using the Continuous AF mode? ( the last option) currently, it's just mostly white so I cant see how it's reacting in the scene
    • regarding loss of location for fast movements, is there any recommendation in terms of real-world movements like back, forward and side-to-side?

      overall this is an amazing start :)
     
  5. ventrap

    ventrap

    Unity Technologies

    Joined:
    May 30, 2017
    Posts:
    8
    Thanks for the feedback !
    I didn't try it ! but now that you're talking about it ! ha ha !


    - not sure what debugger option you're talking about but did you try to display the Focus Plane on the App ? It helps visualize the depth at which you're focusing... then on the Editor side, on the Virtual Camera Device GameObject, there are a few parameters that you can play with to customize the Focus Plane the way you want. (In the Focus Plane Renderer component)
    - You're saying that when you do fast movement, it's losing the location ? I tried on the iPad pro app and it's pretty solid, it's using the gyroscope too I think in ARkit to keep track of the positionning...
    Here are a few ideas when you want to record "strange" motion:
    - use the iterative process to record takes (first Position, then defining the first one as Base, record only Rotation, etc...)
    - use the Joysticks
    - play with the Motion Axis Scale to decrease or increase the effect of the real world motion
    - play with the Joystick sensibility to increase or decrease the speed at which you're moving in the 3d scene when using the joystick
     
    HIBIKI_entertainment likes this.
  6. HIBIKI_entertainment

    HIBIKI_entertainment

    Joined:
    Dec 4, 2018
    Posts:
    203
    ha, it's just one of those things to do.

    Focus Grid.png
    ah, nice! This was the perfect resolution for my issue there, I was using the Renderpipeline debugger doi!
    RP DOF debugger.png

    haha yes now this is as you mention below the 'strange movements' solution, it was my intention to overlay these kinds of things, so thank you for that.
    in this instance, I was merely testing robustness.

    You'll have to take my word that I'm not moving it here and im not that bad of a camera person to actually consider this valid during production :D


    But just in case it does pop up elsewhere, this was on an iPhone XR fyi


    Thank you for all the brilliant responses.i really appreciate it.

    :cool:
     
  7. bradweiers

    bradweiers

    Unity Technologies

    Joined:
    Nov 3, 2015
    Posts:
    59
    Even though the iPad is probably the ideal device given the form factor, we're working making the iPhone version as good as possible. Keep the feedback coming!
     
    pauldrummond and GeniusKoala like this.
  8. GMonkey1780

    GMonkey1780

    Joined:
    Nov 30, 2019
    Posts:
    1
    Good morning
    I am new on this - Download the iPhone App.
    I am having this problem - Unity see my iPhone and my iPhone is connected to the Virtual Camera server
    But on the iPhone screen I don't see the only a black environment with wireframe please help or suggestion
     
  9. bradweiers

    bradweiers

    Unity Technologies

    Joined:
    Nov 3, 2015
    Posts:
    59
    Hi @GMonkey1780. Just a few questions:
    • macOS or Windows?
    • What GPU?
    • What Unity version?
    • Which iPhone model?
    • Which iPhone iOS version?
    • Which app version? (you can find this in TestFlight)
    • Which Live Capture package version? (you can find this in the Package Manager)
     
  10. KazooR

    KazooR

    Unity Technologies

    Joined:
    Oct 25, 2020
    Posts:
    2
    Hello.
    I'm trying this on my iPhone 12 max...
    I was able to connect without any stress at all when I used it on my Mac.
    It is extremely comfortable.

    However, I tried to run it on Windows 10 because of the harsh rendering.
    In Windows 10, I am unable to connect.
    In the same Wifi environment and the iPhone 12MAX side also recognizes the Windows machine in the server search.
    I have also run the FireWall config.

    Is there any other place I should look?
     
  11. ScottSewellUnity

    ScottSewellUnity

    Unity Technologies

    Joined:
    Jan 29, 2020
    Posts:
    8
    @KazooR
    You can try to manually enter the IP of the server to connect to instead. That can help in scenarios where the computer has many interfaces available and the client doesn't pick the correct one to automatically connect to when using server discovery. Also, ensure you are not on a VPN if possible.
     
  12. KazooR

    KazooR

    Unity Technologies

    Joined:
    Oct 25, 2020
    Posts:
    2
    Thank you for your advice.
    I was able to get connected by turning off all Windows Defender.
    I followed the steps and did the Firewall config, but in the end it was best to force it off.
    The behavior is very impressive, with great response.
     
    Head-Trip likes this.
  13. satishgoda

    satishgoda

    Joined:
    Aug 30, 2017
    Posts:
    1
    Just started tinkering with the vcam app today. I started with my iPhone 11 Pro Max and then iPad 11 Pro. I am collecting notex on UI and UX and will share here in couple of weeks. Looking forward to the updates too. (Is there a roadmap for vcam and facial capture development)?
     
  14. marc_tanenbaum

    marc_tanenbaum

    Unity Technologies

    Joined:
    Oct 22, 2014
    Posts:
    554
    Hi @satishgoda and welcome to the forum!

    We'll appreciate the notes when you have them.

    As for a roadmap, we have an internal one, of course, but haven't made it public yet. There will definitely be a public one before too long. Mostly, I just need to get some pieces together to make it happen.
     
  15. GeniusKoala

    GeniusKoala

    Joined:
    Oct 13, 2017
    Posts:
    72
    Hey it's been quite a long time! We are going to use the Virtual Live Camera for some shots of our upcoming short. We will also use motion capture (XSens + ManusVR gloves)! We all are excited!

    I noticed sometimes it says that the port 9000 is already used by another program or another Unity instances. I think it's a bug since I only have one Unity opened instance. I just have to close and open my instance again and it works but I can't offer to make it a lot of times on the recording with the director.

    Also I am testing the features and I would know if it was possible to snapshot videos ? I guess for the moment it records the Transform of the camera to play it again. If we want to see our shot after the recording which every director always wants on a recording stage I would also need to snapshot a video of our scene, not only the camera Transform. I think on iOS I can record the screen and since the Unity scene is streamed on the phone it could make it for the moment. It would make it easier to record either on the phone or on the PC the video of the Game View.

    I will let you know how things are going!

    Thanks for making great tools for us!
     
  16. HIBIKI_entertainment

    HIBIKI_entertainment

    Joined:
    Dec 4, 2018
    Posts:
    203

    In terms of snapshot you're referring to a live feed to the director of both the unity editor scene and the phone's screen?
     
    GeniusKoala likes this.
  17. HIBIKI_entertainment

    HIBIKI_entertainment

    Joined:
    Dec 4, 2018
    Posts:
    203
    Sorry for my short response, I had to read this a couple of times to see if I understood correctly ( still pretty early for me).

    In general your shots are saved as a take so you can reference these back in the shot component, ( I can't remember exactly what game object it's in for the system) you'll see a list of them as the timeline takes though.

    You can also always create a rendered out video on every timeline play as well, (you can add the unity renderer to timeline and it'll record whatever length it's set to)

    That could be saved in a network location for the director to scrub back and forth independently if they wish, a little bit of a communication error chance there but doable, this can also be exported as images and multi frame images too.

    Hope that's kind of in the ball park of what you were thinking.
     
    GeniusKoala likes this.
  18. GeniusKoala

    GeniusKoala

    Joined:
    Oct 13, 2017
    Posts:
    72
    Thanks for both of your answers! I’m sorry if I was not clear. Actually I thought about it during my tests. Since we use motion capture and the virtual camera we need to record both of them a the same time. Is it possible to change the behavior of the « record » button of the iPhone app to add additional actions like start the motion capture recording? I will need to customize the package. To review a shot after a recording I need to playback both the camera and the motion capture performance animation and both need to be record at the same time to stay synchronized.
     
  19. Sergi_Valls

    Sergi_Valls

    Unity Technologies

    Joined:
    Dec 2, 2016
    Posts:
    204
    Hi GeniusKoala,.

    We are aware of that issue, but it is hard to reproduce. Maybe disabling assembly reloads on enter playmode can help. We will keep investigating. Meanwhile you can try using a different port to avoid restarting Unity.

    We are considering solutions for capturing the Take as a video file. Recording both on the client and the server would be useful indeed.

    Cheers!
     
  20. Sergi_Valls

    Sergi_Valls

    Unity Technologies

    Joined:
    Dec 2, 2016
    Posts:
    204
    We can add an event to the TakeRecorder where you could subscribe triggered when the recording state changes. Would that unblock you?
    We also have in our roadmap creating a generic device that can record any hierarchy in the scene. This is useful when using devices that are not integrated yet into live-capture and still be able to record them and create takes.
     
  21. GeniusKoala

    GeniusKoala

    Joined:
    Oct 13, 2017
    Posts:
    72
    Yes I think it could make the job! I was wondering how much could we modify the client app on the iOS device? Is that possible to build our own app to customize our experience?
     
  22. Grimreaper358

    Grimreaper358

    Joined:
    Apr 8, 2013
    Posts:
    765
    Will there be a feature/option to record the position of the device/camera as well instead of rotation only and moving with the onscreen joystick? Something like a VR Virtual Camera setup, where you can move and that's recorded as a positional movement.

    This is basically one of the main things I'd like to see improved. This will also greatly improve the way you can record a lot of shots.

    Example



    There's also an issue I'm having where I'm getting a small lag every two or three seconds.
     
    GeniusKoala likes this.
  23. bradweiers

    bradweiers

    Unity Technologies

    Joined:
    Nov 3, 2015
    Posts:
    59
    We definitely do record the position as well. Maybe you have it disabled in the "Channels" on your VirtualCameraDevice?

    A couple of things to try:
    1. Make sure you are in play mode. The virtual camera works in edit mode but it's not recommended for performance or visual quality.
    2. Close all other running applications and also close all Editor panels except for the game view and maybe one inspector. Each panel in the editor has a performance cost.
    3. Lower the video streaming settings in your user preferences.
    Let me know if that works please!
     
    Last edited: Oct 6, 2021
    GeniusKoala likes this.
  24. Grimreaper358

    Grimreaper358

    Joined:
    Apr 8, 2013
    Posts:
    765
    Using it in Playmode is working smoothly.

    • The only other issue is sometimes the scene rendering isn't showing on the device and just the default Grid Room but the tracking still works. Even after restarting the app multiple times, it doesn't show.




    For the position recording, it's only being done one way right now and is what I was referring to. Currently, you need to use the onscreen Joystick (Left Side - To move around the scene) and slider (Right Side - To move up and Down)

    So basically being able to capture movement without using the joysticks.


    Being able to move the device in the real world and that movement is captured as shown in the video I posted before. When he moves the phone up and down, left and right, forward and backward, that movement is recorded along with rotation with no need for on-screen controls to try and achieve the same movements. These controls are great to get certain shots and working in a confined space but for something more dynamic it's not so great. Having both ways using on-screen controls and actually moving in real space to capture position/pacing will basically solve this.


    Examples - Using only the on-screen joystick will take a lot more time and effort to try and get something similar to these examples.

    Timestamp - 5:20



    Timestamp - 2:33



    Timestamp - 0:30
     
    akent99 likes this.
  25. HIBIKI_entertainment

    HIBIKI_entertainment

    Joined:
    Dec 4, 2018
    Posts:
    203
    Hrmm
    We can freely walk around and it tracks.
    Video ( lag is just screen cap)
    Less lag

    you also have tacking scalars in case you're working within smaller environments and want larger movements. i have the joystick turned off.
    double check both the app and editor settings, to ensure you have the scalars set to 1:1 ( default) and have the position rotation etcetc button all enabled, they are by default. ( impressed) i can't remember off the top of my head but i think i remember seeing someone mention that the position needed to be set to local and not world in the app and editor.
    but i can check our test scene if you need.

    i do like your input there with two joystick input. maybe this could be integrated with the new input system.
     
    Last edited: Oct 8, 2021
    Grimreaper358 and GeniusKoala like this.
  26. Grimreaper358

    Grimreaper358

    Joined:
    Apr 8, 2013
    Posts:
    765
    Oh sweet, There's definitely something wrong on my end then. I'll check things again and make sure.
     
    HIBIKI_entertainment likes this.
unityunity