Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

Video Unity AOV Recorder update

Discussion in 'Audio & Video' started by marief_unity, Jul 26, 2019.

  1. marief_unity

    marief_unity

    Unity Technologies

    Joined:
    Oct 9, 2018
    Posts:
    120
    Hey there !

    AOVRecorder 0.1.0-preview-3 has been officially released ! This is a Recorder plugin that relies on High Definition Render pipeline.

    AOVRecorderWindow.png

    AOVs (Arbitrary output variables) - sometimes called render passes or render layers - are data that are being calculated to produce the beauty pass (=the final render).
    AOVs examples: depth, normals, albedo...
    Being able to export them separately allows more options at compositing step.

    aovs.png

    Please note, this is a first preview version (beta). There are known issues and limited settings for now :
    • First frame (0) sometimes is black
    • Active Camera target is not supported by HDRP
    • Depth data is incorrect if the scene distance is large with HDRP version < 6.8
    • PNG and JPEG files are written as linear file instead of sRGB.
    • Simultaneous recording of the beauty pass and other render passes if TAA is enabled will generate artifacts
    • Recorder issue: Timeline recording of the beauty pass will record black with HDRP

    Also : it is highly recommended to use this plugin with the latest version of HDRP possible.

    See documentation.

    To install it, open Package Manager window, make sure you enable "show preview packages" advanced option and install com.unity.aovrecorder.

    Please let us know how testing goes. Any feedback is more than welcome :)

    Thanks !
     
    Selzier likes this.
  2. Mad_Banny

    Mad_Banny

    Joined:
    Jun 25, 2018
    Posts:
    4
    Hi!
    Here is my feedback:

    Unity 2019.2, HDRP 6.9.1

    Code (CSharp):
    1. NullReferenceException: Object reference not set to an instance of an object
    2. UnityEditor.Recorder.Input.AOVCameraAOVRequestAPIInput.EnableAOVCapture (UnityEditor.Recorder.RecordingSession session, UnityEngine.Camera cam) (at Library/PackageCache/com.unity.aovrecorder@0.1.0-preview.3/Editor/Sources/Recorders/AOVRecorder/AOVCameraInput.cs:43)
    3. UnityEditor.Recorder.Input.AOVCameraAOVRequestAPIInput.NewFrameStarting (UnityEditor.Recorder.RecordingSession session) (at Library/PackageCache/com.unity.aovrecorder@0.1.0-preview.3/Editor/Sources/Recorders/AOVRecorder/AOVCameraInput.cs:138)
    4. UnityEditor.Recorder.Recorder.SignalInputsOfStage (UnityEditor.Recorder.ERecordingSessionStage stage, UnityEditor.Recorder.RecordingSession session) (at Library/PackageCache/com.unity.recorder@2.0.2-preview.3/Editor/Sources/Recorder.cs:160)
    5. UnityEditor.Recorder.RecordingSession.PrepareNewFrame () (at Library/PackageCache/com.unity.recorder@2.0.2-preview.3/Editor/Sources/RecordingSession.cs:184)
    6. UnityEditor.Recorder.RecorderComponent:Update()
    7.  
    Code (CSharp):
    1. NullReferenceException: Object reference not set to an instance of an object
    2. UnityEditor.Recorder.Input.AOVCameraAOVRequestAPIInput.DisableAOVCapture (UnityEditor.Recorder.RecordingSession session) (at Library/PackageCache/com.unity.aovrecorder@0.1.0-preview.3/Editor/Sources/Recorders/AOVRecorder/AOVCameraInput.cs:127)
    3. UnityEditor.Recorder.Input.AOVCameraAOVRequestAPIInput.FrameDone (UnityEditor.Recorder.RecordingSession session) (at Library/PackageCache/com.unity.aovrecorder@0.1.0-preview.3/Editor/Sources/Recorders/AOVRecorder/AOVCameraInput.cs:150)
    4. UnityEditor.Recorder.Recorder.SignalInputsOfStage (UnityEditor.Recorder.ERecordingSessionStage stage, UnityEditor.Recorder.RecordingSession session) (at Library/PackageCache/com.unity.recorder@2.0.2-preview.3/Editor/Sources/Recorder.cs:168)
    5. UnityEditor.Recorder.RecordingSession.RecordFrame () (at Library/PackageCache/com.unity.recorder@2.0.2-preview.3/Editor/Sources/RecordingSession.cs:134)
    6.  
     

    Attached Files:

  3. jean_unity_film_tv

    jean_unity_film_tv

    Unity Technologies

    Joined:
    Jan 31, 2018
    Posts:
    23
    Hi Mad_Banny,

    Sorry to see that you are having problems.
    Here are a few questions for trying to isolate the issue.
    Are you using the AOVRecorder with an HDRP scene? The AOV Recorder will only record AOVs from an HDRP scene.
    How do you start the recording? With the recorder window or with a record clip on the timeline?
    What are your recording options?
    What is your camera input? Are you trying to record the main camera or a tagged camera?

    Would it be possible to zip and upload you project for us to investigate and try to reproduce the issue here?
    Thanks,

    Jean
     
  4. Mad_Banny

    Mad_Banny

    Joined:
    Jun 25, 2018
    Posts:
    4
    1. Yes, I use it with HDRP scene.
    2. I tried both window and timeline.
    3. I tried all recording options.
    I guess this is due to RP asset settings
     
  5. Mad_Banny

    Mad_Banny

    Joined:
    Jun 25, 2018
    Posts:
    4
    Tried it in the default scene and it worked
     
  6. jean_unity_film_tv

    jean_unity_film_tv

    Unity Technologies

    Joined:
    Jan 31, 2018
    Posts:
    23
    Glad to see that it worked for you with the default scene, I will try it here with your render pipeline asset to see if I can reproduce your issue.
     
  7. jean_unity_film_tv

    jean_unity_film_tv

    Unity Technologies

    Joined:
    Jan 31, 2018
    Posts:
    23
    Hi Mad_Banny,
    To give you an update, I wasn't able to reproduce your issues with the AOV Recorder using the HQ_Deffered_HDRenderPipelineAsset that you provided. If you can provide a small project that would reproduce the problems or a list of steps that would be very useful.

    Thanks,

    Jean
     
  8. mahenkumar

    mahenkumar

    Joined:
    Oct 8, 2014
    Posts:
    10
    I can't find this option in my Unity 2019.2. I am using all the required versions.
    Is it only for pro Unity users ?
     

    Attached Files:

  9. marief_unity

    marief_unity

    Unity Technologies

    Joined:
    Oct 9, 2018
    Posts:
    120
    Hey @mahenkumar !

    Did you install the package via package manager ?
    AOVRecorder_packman.png

    If so, do you have errors in the console ?

    Thanks !

    ps: AOV Recorder is available for everyone.
     
  10. xCyborg

    xCyborg

    Joined:
    Oct 4, 2010
    Posts:
    527
    I do hope you intend to quickly integrate that with Unity Recorder, also to extend support of AOVs to LWRP and even standard pipeline.
    Please tell me you don't intend to keep it separate.
     
  11. marief_unity

    marief_unity

    Unity Technologies

    Joined:
    Oct 9, 2018
    Posts:
    120
    Hey @xCyborg

    AOVRecorder is a plugin of Recorder, so it is actually part of Recorder. We choose to make it a separate package because it highly depends on High Definition RP (it's using HDRP new AOV Request API) and we didn't want this dependency to burden Recorder, there are a lot of users that have no interest in AOVs.

    For now, there is no AOV API in LWRP, but when there will be, the plan is to indeed allow AOVRecorder to use it.

    There is no plan to make AOV Recorder work with Legacy Render Pipeline.
     
  12. xCyborg

    xCyborg

    Joined:
    Oct 4, 2010
    Posts:
    527
    @marief_unity Well, as far as I'm concerned, it's another fragmented duplicate package.
     
  13. andrzej_

    andrzej_

    Joined:
    Dec 2, 2016
    Posts:
    33
    Could we at least move the option to add AOV to the import window of the Recorder (just like samples) with a huge !warning! text about HDRP support only? Making it a separate package (but still the same tool and GUI element to interact with), seems a bit redundant.
    One thing that would be very useful for anyone working in video production would be to have 'masks', similar to cryptomattes or multimasks? Color coding by material/entity/layer ... This is a must for compositing.
     
  14. marief_unity

    marief_unity

    Unity Technologies

    Joined:
    Oct 9, 2018
    Posts:
    120
    Hey @andrzej_,

    AOVRecorder as a separate package :

    I understand your concern but we made a choice considering :
    - AOVRecorder is a tool that not everyone is interested in
    - HDRP is a required dependency BUT we can't impose that dependency to all Recorder users
    - Package management is still in development, for example conditional dependency remains a too recent feature (we support until 2018.4)
    - And we know by experience (including in Recorder) using reflection causes more issues in the long term.

    That said, 2 things will happen that could be disruptive of this decision:
    - AOV Request API being ported in URP (no ETA)
    - 2018.4 being "deprecated" enough so we can freeze the Recorder dev on it (not happening in a near future, but will happen for sure)

    Until that, it is what it is. We remain confident we chose the best solution possible considering the current parameters, even with its obvious disadvantages.

    Cryptomatte :
    Adding ObjectID / Cryptomatte passes is indeed in our scope. I can't provide you an ETA though.

    Thanks for your feedback !
     
  15. Solovykh

    Solovykh

    Joined:
    Aug 28, 2017
    Posts:
    27
    Hey, I'm trying to write my own custom recorder. However, all the recorders in the Recorder packages are marked as "internal". How did you guys manage to have AOVRecorder inherit from BaseTextureRecorder which is internal?
     
  16. cv_dg

    cv_dg

    Joined:
    Sep 19, 2019
    Posts:
    3
    We are trying to use the AOVRecorderSettings in the scripted capture method. The ImageRecorderSettings work just fine for us, but the one thing we can't seem to figure out is where to set the AOVGType in the settings. We want to be able to record out the alpha for a matte. Does anyone know how to do this?

    Unity 2019.1.9f1
    Unity AOV Recorder preview.5 - 0.1.0
     
  17. cv_dg

    cv_dg

    Joined:
    Sep 19, 2019
    Posts:
    3
    Looks like this is the AOVGSelection which is internal and defaults to AOVGType.Beauty.
     
  18. cv_dg

    cv_dg

    Joined:
    Sep 19, 2019
    Posts:
    3
    The other thing with this is even though I set the AOVCameraInputSettings, it isn't setting the targetCamera in AOVCameraInput.RenderAndReadbackAOVCapture.

    Code (CSharp):
    1. aovRecorder.imageInputSettings = new AOVCameraInputSettings
    2. {
    3.     cameraTag = "MainCamera",
    4.     source = ImageSource.MainCamera,
    5.     outputWidth = 384,
    6.     outputHeight = 216,
    7.     captureUI = false
    8. };
    So I had to force it to targetCamera = Camera.main so that that it would let me run without a nullreferenceexception.

    Code (CSharp):
    1. if (aovRecorderSettings != null)
    2. {
    3.     targetCamera = Camera.main; // Added to stop NullReferenceException
    4.     targetCamera.Render();
    5.     RenderTexture.active = outputRT;
    6.     readbackTexture.ReadPixels(new Rect(0, 0, outputWidth, outputHeight), 0, 0, false);
    7.     readbackTexture.Apply();
    8.     RenderTexture.active = null;
    9. }
    I am also seeing that the first frame is always missing as shown in the Known Issues log.
     
  19. dschmuck

    dschmuck

    Joined:
    Nov 27, 2017
    Posts:
    8
    Hi!
    Really great that you guys added this package. We are using Unity to render and still do a traditional composting workflow as a final step after unity, so any additional passes we can get out of unity are greatly appreciated!

    We still have a bunch of questions, we hope you can help us with!
    1. will multichannel exrs be eventually supported? For our comp workflows it just makes everything more conveniant
    2. currently there only seem to be the commonMaterialProperties available as render passes (plus depth)
      It would be fantastic to have access to the other information you can view in the debug view like the LightingModes for example to get proper diffuse lighting and specular lighting passes.
      I tried to expand the current package to be able to render these Lighting passes by using "LightingProperty" and "SetDebugLightingMode", but I couldn't quite get it to run (it alway rendered the beauty instead of the expected lighting pass)
    3. will MotionVector pass be included soon? we want to add motion Blur in Comp!
    4. How would you go about adding more complex custom render passes?
      I am currently implementing an object/material/asset ID solution that depends on shader overrides. Currently it is done using an additional render process after rendering beauty and the other passes.
      I would love to include this process directly into the "main" render process and render it to its own pass.
      Do you have any suggestions on how to get that to work?
    5. Lastly the ID passes rely on metadata that should be written into the header of the exr. Currently this is done in a separate follow-up process. Is it somehow possible to write medata into the exrs when rendering with aov recorder or are you guys thinking about adding that?
    I hope you guys can give us some answers.
    Thanks for all the good work for making films with unity!!
     
  20. jean_unity_film_tv

    jean_unity_film_tv

    Unity Technologies

    Joined:
    Jan 31, 2018
    Posts:
    23
    Hi @dschmuck,

    See my answers below after >>>

    Hi!
    Really great that you guys added this package. We are using Unity to render and still do a traditional composting workflow as a final step after unity, so any additional passes we can get out of unity are greatly appreciated!
    We still have a bunch of questions, we hope you can help us with!
    will multichannel exrs be eventually supported? For our comp workflows it just makes everything more conveniant
    >>>
    Yes we are planning to add multichannels exr support to the AOV Recorder.


    currently there only seem to be the commonMaterialProperties available as render passes (plus depth)
    It would be fantastic to have access to the other information you can view in the debug view like the LightingModes for example to get proper diffuse lighting and specular lighting passes.
    I tried to expand the current package to be able to render these Lighting passes by using "LightingProperty" and "SetDebugLightingMode", but I couldn't quite get it to run (it alway rendered the beauty instead of the expected lighting pass)
    will MotionVector pass be included soon? we want to add motion Blur in Comp!
    >>>
    Also, yes we are planning to make more AOVs types available like the motion vector pass.
    In HDRP version < 6.7 the AOV recorder is using the debug api for creating the AOVs, from HDRP version 6.7 and above we are using a new AOVRequest API. This allow us to make requests for the AOVs that we want to gather and record.

    How would you go about adding more complex custom render passes?
    I am currently implementing an object/material/asset ID solution that depends on shader overrides. Currently it is done using an additional render process after rendering beauty and the other passes.
    I would love to include this process directly into the "main" render process and render it to its own pass.
    Do you have any suggestions on how to get that to work?
    >>>To create an object ID pass using the new HDRP Custom Pass might allow you to do what you want.
    https://docs.unity3d.com/Packages/c...endering.HighDefinition.CustomPassDrawer.html


    Lastly the ID passes rely on metadata that should be written into the header of the exr. Currently this is done in a separate follow-up process. Is it somehow possible to write medata into the exrs when rendering with aov recorder or are you guys thinking about adding that?
    I hope you guys can give us some answers.
    >>> Currently this is not possible. But I can see how this could be very useful.


    Thanks for all the good work for making films with unity!!
    >>> You are welcome!
     
    dschmuck likes this.
  21. Solovykh

    Solovykh

    Joined:
    Aug 28, 2017
    Posts:
    27
    Figured out the answer to my question. In the base Unity Recorder package, under Editor/Sources there's an AssemblyInfo.cs file that exposes the unity recorder's internals to the AOV recorder
     
  22. marief_unity

    marief_unity

    Unity Technologies

    Joined:
    Oct 9, 2018
    Posts:
    120
    Hey @Solovykh,

    Sorry for the late answer, indeed ! Yes, for now we use this not-so-great InternalsVisibleTo.
    But we are actually not far from releasing a new version of Recorder with a wider Public API to allow people to create new Recorders.

    Thanks !
     
  23. andrzej_

    andrzej_

    Joined:
    Dec 2, 2016
    Posts:
    33
    @marief_unity
    Just noticed that AOV recorder doesn't work with raytracing (only beauty and depth pass work, but no alpha, or normal for example).
     
  24. marief_unity

    marief_unity

    Unity Technologies

    Joined:
    Oct 9, 2018
    Posts:
    120
    Hi @andrzej_,

    Did you set-up your Ray-Tracing scene from scratch ?
    When you say it does not work: do you have errors, black images, wrong output ?
    What versions of Unity / HD RP ?

    I gave a try on my side with SmallRoomRayTracing sample + HDRP 7.1.5 + Unity 2019.3.0b11 : beauty is black and other passes contain the beauty o_O.
    So there is definitively something off, I'm going to log an issue on our side to investigate in depth.

    Note though that I tried starting from HDRP template and convert my project in Ray Tracing mode, and it was working (meaning beauty, depth, normal, alpha export had the corresponding passes).

    Anyways, thanks for reporting the issue, it is always greatly appreciated.