Search Unity

  1. New Unity Live Help updates. Check them out here!

    Dismiss Notice

NatCorder - Video Recording API

Discussion in 'Assets and Asset Store' started by Lanre, Nov 18, 2017.

  1. jrmgx

    jrmgx

    Joined:
    Oct 21, 2016
    Posts:
    14
    I understood that, I was wondering how I can prevent this, probably using some sort of other setup than CameraInput I guess?
     
  2. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    Doing this is not trivial, as you will likely have to make your camera render to an off-screen render texture then blit that render texture to the screen. All sorts of issues come up with this approach, so I don't recommend it.
     
  3. jrmgx

    jrmgx

    Joined:
    Oct 21, 2016
    Posts:
    14
    Ho noes, I though it would have been easy.
    Anyway how do you think I can prevent the double rendering then?
     
  4. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    You can't, except you implement what I have described above.
     
  5. jrmgx

    jrmgx

    Joined:
    Oct 21, 2016
    Posts:
    14
    Ok thank you for your help :)
     
    Lanre likes this.
  6. mayurasodariya33

    mayurasodariya33

    Joined:
    May 8, 2019
    Posts:
    2
    Is there any way to improve video render quality while recording video with unity camera?(in android)
    Here is my current project graphics setting.
    Case 1) When I use only 1 render which is ==> Vulkan render, I am getting 0 device support while uploading aab file in play store(Google console)

    Case 2) When I use below graphic setting(Find in attachment) I am getting low clarity on rendering

    please guild me how project setting should be while using Vulkan render :)
     

    Attached Files:

  7. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    Figuring out the device filter support on Google Play has nothing to do with NatCorder; a google search would be more helpful for figuring that out. As for quality, if you are getting a low quality from the recording, then consider increasing the recording bitrate (see the MP4Recorder constructor docs).
     
  8. mindmovies

    mindmovies

    Joined:
    May 9, 2019
    Posts:
    6
    Unity 2019.3.2f1
    NatCorder 1.7.0
    Unity editor crashes (or deployed build on WIndows 10) everytime
    when I call recorder.FinishWriting(); without any logs or traces to find the cause.
    Please , help!
     
  9. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    Can you share your recording code?
     
  10. mindmovies

    mindmovies

    Joined:
    May 9, 2019
    Posts:
    6
    Sure , but i did lots of attempts and still experiencing this crash. At first its been crashing just when I start recording, but with this attempt I reused CameraInput class like this and now only crashes at the end of movie while calling videoRecorder.FinishWriting(). Also my recording is not frame based , I pass timestamp manually (my target is to record scene views very fast , like making 30fps movie of 2 minutes in 10 seconds (unlimited unity player fps, but not all frames used for recording for other project reasons). Works fine on older Unity and Natcorde versions)

    Code (CSharp):
    1. using NatCorder;
    2. using NatCorder.Clocks;
    3. using System.Collections;
    4. using System.Collections.Generic;
    5. using UnityEngine;
    6. using UnityEngine.UI;
    7. using UnityEngine.Events;
    8. using NatCorder.Internal;
    9. using UnityEngine.Rendering;
    10.  
    11. [System.Serializable]
    12. public class MyStringEvent : UnityEvent<int, string>
    13. {
    14. }
    15.  
    16. public class VideoRec : MonoBehaviour
    17. {
    18.  
    19.     public const int NEW_VIDEO = 1;
    20.     public const int STOPPED_RECORDING = 2;
    21.  
    22.     public Camera mainCamera;
    23.  
    24.     public RenderTexture CameraTexture;
    25.  
    26.     private MP4Recorder videoRecorder;
    27.  
    28.     public MyStringEvent m_RecorderEvent = new MyStringEvent();
    29.  
    30.     private RenderTexture frameBuffer;
    31.     private Texture2D readbackBuffer;
    32.     private byte[] pixelBuffer;
    33.  
    34.     public VideoRec()
    35.     {
    36.      
    37.     }
    38.  
    39.     public void StartRecording()
    40.     {
    41.         Debug.Log("mainCamera resolution " + mainCamera.pixelWidth + " " + mainCamera.pixelHeight);
    42.         Debug.Log("framerate " + Config.VIDEO_FRAMERATE);
    43.         mainCamera.forceIntoRenderTexture = true;
    44.      
    45.         videoRecorder = new MP4Recorder(mainCamera.pixelWidth, mainCamera.pixelHeight, Config.VIDEO_FRAMERATE);
    46.  
    47.         var frameDescriptor = new RenderTextureDescriptor(videoRecorder.frameSize.width, videoRecorder.frameSize.height, RenderTextureFormat.ARGB32, 24);
    48.         frameDescriptor.sRGB = true;
    49.         this.frameBuffer = RenderTexture.GetTemporary(frameDescriptor);
    50.  
    51.         this.readbackBuffer = SystemInfo.supportsAsyncGPUReadback ? null : new Texture2D(frameBuffer.width, frameBuffer.height, TextureFormat.RGBA32, false, false);
    52.         this.pixelBuffer = new byte[frameBuffer.width * frameBuffer.height * 4];
    53.  
    54.     }
    55.  
    56.     public async void StopRecordingAsync()
    57.     {
    58.         m_RecorderEvent.Invoke(STOPPED_RECORDING, null);
    59.      
    60.         if (videoRecorder != null)
    61.         {
    62.             string result = await videoRecorder.FinishWriting();
    63.             OnRecording(result);
    64.             videoRecorder = null;
    65.          
    66.         }        
    67.     }
    68.  
    69.     // Update is called once per frame
    70.     public IEnumerator Tick(long timeStamp)
    71.     {    
    72.         if (videoRecorder != null && mainCamera.activeTexture != null)
    73.         {
    74.             var endOfFrame = new WaitForEndOfFrame();
    75.             while (true)
    76.             {
    77.                 // Check frame index
    78.                 yield return endOfFrame;
    79.  
    80.  
    81.                 var prevTarget = mainCamera.targetTexture;
    82.                 mainCamera.targetTexture = this.frameBuffer;
    83.  
    84.                 mainCamera.Render();
    85.                 mainCamera.targetTexture = prevTarget;
    86.  
    87.                 if (SystemInfo.supportsAsyncGPUReadback)
    88.                     AsyncGPUReadback.Request(frameBuffer, 0, request =>
    89.                     {
    90.                         if (pixelBuffer != null)
    91.                         {
    92.                             request.GetData<byte>().CopyTo(pixelBuffer);
    93.                             videoRecorder.CommitFrame(pixelBuffer, timeStamp);
    94.                         }
    95.                     });
    96.                 else
    97.                 {
    98.                     var prevActive = RenderTexture.active;
    99.                     RenderTexture.active = frameBuffer;
    100.                     readbackBuffer.ReadPixels(new Rect(0, 0, frameBuffer.width, frameBuffer.height), 0, 0, false);
    101.                     readbackBuffer.GetRawTextureData<byte>().CopyTo(pixelBuffer);
    102.                     videoRecorder.CommitFrame(pixelBuffer, timeStamp);
    103.                     RenderTexture.active = prevActive;
    104.                 }
    105.             }
    106.  
    107.         }
    108.     }
    109.  
    110.     void OnRecording(string path)
    111.     {    
    112.         m_RecorderEvent.Invoke(NEW_VIDEO, path);    
    113.     }
    114. }
     
    Last edited: Feb 20, 2020
  11. BakuDigital

    BakuDigital

    Joined:
    May 9, 2013
    Posts:
    18
    Damn! It's problematic for us... Any way to get it working on 2018 ? Maybe I missed a post... Thanks!
     
  12. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    Your code is prone to crashing when recording is stopped. You need to make sure that the moment FinishWriting is called, there will be other calls to the recorder. Your Tick method can still try to commit frames to the recorder while it is finishing, which is almost guaranteed to cause a crash. Notice how `videoRecorder` will only become `null` after the recorder has finished writing. This will cause `Tick` to keep trying to commit frames while the recorder is finishing, which will cause the crash.
     
    mindmovies likes this.
  13. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    Nope. This is a Unity issue, and Unity fixed it in 2019.2.
     
  14. mayurasodariya33

    mayurasodariya33

    Joined:
    May 8, 2019
    Posts:
    2
    Hello,

    I am getting very low fps while recording video through natcorder,
    1) Without natcorder 29-32 fps.
    2) While recording video getting around 9-11 fps.
    Is anything else could I do to increase fps on android device.!
     
  15. mindmovies

    mindmovies

    Joined:
    May 9, 2019
    Posts:
    6
    Thank you very much for noticing. Now its working without crash.
     
    Lanre likes this.
  16. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    There are many variables to look at. The first is how complicated your app's rendering is before recording. It your app's rendering takes a lot of CPU and GPU time, then adding recording will naturally reduce your frame rate. Also, the higher your recording resolution, the more work the device has to do so consider reducing the resolution.
     
  17. Yode_group

    Yode_group

    Joined:
    Apr 5, 2016
    Posts:
    12
    Hello, I encountered a problem with your plugin on some devices.

    Initially, in our project, it was noticed on the Honor 8X device, during recording with some chance the sound ceases to be recorded. The sound is simply interrupted, mainly in the beginning of the recording.
    At the end of the video, the sound breaks off https://yadi.sk/i/3jHtaKOOUDpPng

    The problem repeated on Unity 2019.1.14 and 2019.3.1.

    In a clean draft of the same versions, the device has an asynchronous video and audio recording, a delay of about 2 seconds goes to the audio track. And each time increases.

    And also a bug was sometimes noticed with the fact that the audio track from the previous record was superimposed on a new one, but on the previous one it was not at all and continues like this, the previous audio track is sewn into the video.

    The video recorded the work with Xiaomi mi 9 SE and example project ReplayCam
    https://onedrive.live.com/?authkey=!AA2DfCbkYAPv8Es&cid=CF3AE80FA057CEB9&id=CF3AE80FA057CEB9!36242&parId=root&o=OneUp

    Also on some devices, on the video you can see, at the end of the recording, a black frame with noise



    Sorry for my English, thanks!
     
    Last edited: Feb 25, 2020
  18. JonathanFri

    JonathanFri

    Joined:
    Jul 3, 2019
    Posts:
    3
    Hey, whats the current code to pause/resume recording?
    Before it was RealtimeClock.Paused = true and disposing audio+ video.
    However this property isn't there anymore.
    Whats the replacement?
     
  19. Nicky_Casey

    Nicky_Casey

    Joined:
    Aug 16, 2018
    Posts:
    1
    Hi there,

    Are x86 architectures still supported? When I build an apk it is missing the libNatCorder and libNatRender for x86 but they are present for armeabi-v7a.
     
  20. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    These issues all revolve around Unity's Microphone API, which is known to produce glitchy audio. We developed our own microphone API, NatMic, because of issues like this. Check it out.
     
  21. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    The property is still there. Nothing happened to it.
     
  22. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    None of the NatSuite API's support x86 on Android. This is in line with Unity dropping support for it.
     
  23. Makio64

    Makio64

    Joined:
    Nov 9, 2014
    Posts:
    11
    Hello, I experience a first black frame on iPhone 11 when I record but the texture isn't black

    Relevent code :
    Code (CSharp):
    1.     public void StartRecording () {
    2.         bool haveSound = false;
    3.         if(Microphone.devices.Length > 0){
    4.             audioSource = GetComponent<AudioSource>();
    5.             audioSource.clip = Microphone.Start(Microphone.devices[0], true, 60, 48000);
    6.             audioSource.Play();
    7.             haveSound = true;
    8.         }
    9.         if(haveSound){
    10.             _videoRecorder = new HEVCRecorder(Screen.width, Screen.height, 60, 48000, 1, OnRecording, 5909760*4, 1);
    11.         } else {
    12.             _videoRecorder = new HEVCRecorder(Screen.width, Screen.height, 60, 0, 0, OnRecording, 5909760*4, 1);
    13.         }
    14.         _clock = new RealtimeClock();
    15.         _ARCamera.Render();
    16.         _cameraInput = new CameraInput(_videoRecorder, _clock, _ARCamera);
    17.         running = true;
    18.     }
     
  24. Edisyo

    Edisyo

    Joined:
    Dec 11, 2017
    Posts:
    9
    Hi!

    How to prevent cameras from switching, if I am recording with different camera (for excluding UI elements)? When I press button and hold to record, it goes from main camera to recording camera. Is it possible record in background aka so on screen i still see main camera, but recording from other camera?
     
  25. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    Upgrade to NatCorder 1.7.0 and try this again. Let me know if the issue persists.
     
  26. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    This might be a bug. Can you share screenshots of both cameras in the inspector? And can you share your recording code?
     
  27. Makio64

    Makio64

    Joined:
    Nov 9, 2014
    Posts:
    11
    Thanks but the 1.6.6 is the latest available in the asset store manager in Unity 2020.1.0a17
     
  28. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    That's weird. I recommend sending a report to Unity. NatCorder will work with Unity 2018.3+.
     
  29. Makio64

    Makio64

    Joined:
    Nov 9, 2014
    Posts:
    11
    I update to the latest version of unity and was able to use the 1.7, the black frame is gone ! thanks :)

    Im using the latest unity : 2020.1.0a24 & the 1.7 Natcoder ,

    also here the relevant code, the biggest change here is on the use of the new AudioInput :
    Code (CSharp):
    1.  
    2. public void StartRecording () {
    3. bool haveSound = false;
    4. if(Microphone.devices.Length > 0){
    5. audioSource = GetComponent<AudioSource>();
    6. audioSource.loop = true;
    7. audioSource.clip = Microphone.Start(Microphone.devices[0], true, 30, AudioSettings.outputSampleRate);
    8. audioSource.Play();
    9. haveSound = true;
    10. }
    11. if(haveSound){
    12. _videoRecorder = new HEVCRecorder(Screen.width, Screen.height, 30, 48000, 1, 5909760, 1);
    13. } else {
    14. _videoRecorder = new HEVCRecorder(Screen.width, Screen.height, 30, 0, 0, 5909760, 1);
    15. }
    16. _clock = new RealtimeClock();
    17. _ARCamera.Render();
    18. _cameraInput = new CameraInput(_videoRecorder, _clock, _ARCamera);
    19. audioInput = haveSound ? new AudioInput(_videoRecorder, _clock, audioSource, true) : null;
    20.  
    21. running = true;
    22. }
     
    Last edited: Feb 27, 2020
  30. Makio64

    Makio64

    Joined:
    Nov 9, 2014
    Posts:
    11
    Spoke too fast sorry, there is now a memory problem after fews seconds the max memories is reach
     

    Attached Files:

  31. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    Can you share the logs, and can you share screeshots of the app's frame time with and without recording? Such memory behaviour is impossible with NatCorder alone.
     
  32. Makio64

    Makio64

    Joined:
    Nov 9, 2014
    Posts:
    11
    The peak start immediately when I start recording, I edit my previous post with the relevant code and I didnt change anything else in the app except the update of Unity and Natcorder to 1.7 from 1.6.6, before I could easily register 3minutes on Natcorder with sounds and without memory problem.
     
  33. Makio64

    Makio64

    Joined:
    Nov 9, 2014
    Posts:
    11
    Logs with Natcorder :
    2020-02-27 11:03:29.894204+0100 appreality[726:106615] NatCorder: Prepared HEVCRecorder video encoder at resolution 1125x2436@30.000000Hz with average bitrate 5909760 and keyframe interval 1s

    2020-02-27 11:03:29.896958+0100 appreality[726:106615] NatCorder: Prepared HEVCRecorder audio encoder for 1 channels at 48000Hz

    WARNING -> applicationDidReceiveMemoryWarning()

    WARNING -> applicationDidReceiveMemoryWarning()

    2020-02-27 11:03:34.037718+0100 appreality[726:107284] [ServicesDaemonManager] interruptionHandler is called. -[FontServicesDaemonManager connection]_block_invoke

    Crash memory almost always on the GPU Async request ( see join screenshot )

    Also join the screenshot of memory when natcorder isnt use, nothing to signal, the memory app is stable ( like with the v1.6.6 )

    Tested on iphone11 latest iOS, latest xcode, latest unity, v1.7.
     

    Attached Files:

  34. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    Your recording resolution is extremely high, so it makes sense that the async readback requests pile up in memory and make it explode. Record at 1280x720 or 1920x1080 instead. You have to pay attention to your app's frame time (CPU and GPU, in Xcode). If the GPU can't keep up, pending requests will accumulate in memory and your app will crash.
     
  35. Makio64

    Makio64

    Joined:
    Nov 9, 2014
    Posts:
    11
    I agree its high but it didnt crash after fews minuts on the v1.6.6, its like the gpu isnt debounce on the new v1.7
     
  36. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    At some point in 1.6 we were using Texture2D.ReadPixels to read pixel data from the GPU. This has very stable memory performance, as it blocks until the data is ready. The problem is that it blocks until the data is ready, giving worse performance. Metal supports AsyncGPUReadback, so we use it. Either way, I strongly recommend reducing that recording resolution. It is way too high.
     
  37. Makio64

    Makio64

    Joined:
    Nov 9, 2014
    Posts:
    11
    In the v1.6.6 it was already using the asyncGPUReadback and was recording at the same resolution ( I agree I should put a limit at 1080x1920 ) and that's actually the main reason I decide to use NatCorder, as you said ReadPixels is not good at all. I just wonder whats the breaking change on the core API of NatCorder leading to such a recession in terms of performance and stability from the v1.6.6 to the v1.7?
     
  38. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    Nothing changed on iOS, macOS, or Windows between 1.6 and 1.7. Again, profile your app for its performance characteristics as that will tell you what to do.
     
  39. Makio64

    Makio64

    Joined:
    Nov 9, 2014
    Posts:
    11
    I did more test and even when the recording is finished in the ReplayCam the memory is never getting lower, shoudnt you work with small buffer to limit the usage in the ram and write in async on the disk ?
     
  40. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    We don't keep any buffers at all, which is why I'm saying the memory behaviour you are seeing looks spurious. I cannot reproduce this behaviour with the ReplayCam example, which is always a good place to start.
     
  41. Makio64

    Makio64

    Joined:
    Nov 9, 2014
    Posts:
    11
    I revert Unity to the 2020.1.0a17 and its now working!!
     
    Lanre likes this.
  42. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    So it must've been a Unity memory leak. I recommend filing a bug report with them anyway. In any case, this sounds good.
     
  43. bhtruong93

    bhtruong93

    Joined:
    Jul 12, 2017
    Posts:
    8
  44. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    The leak you faced is not of the same order as the one the OP mentioned, so it isn't the same issue. The OP was seeing memory explode immediately recording was started. I haven't tried to reproduce this leak with 1.7.0. I will do so and get back to you.
     
  45. DaniilGalahow

    DaniilGalahow

    Joined:
    Dec 2, 2013
    Posts:
    9
    Hello!
    I have trouble with recording video with sound through combination MP4Recorder+CameraInput/AudioInput. All made like in readme:

    Code (CSharp):
    1. public class VideoRecorderBehaviour : MonoBehaviour
    2.     {
    3.         [SerializeField]
    4.         Camera camera = null;
    5.         [SerializeField]
    6.         AudioListener microphone = null;
    7.         [SerializeField]
    8.         Text recordButtonText = null;
    9.  
    10.         [SerializeField]
    11.         int sampleRate = 44100;
    12.         [SerializeField]
    13.         int channelsCount = 2;
    14.  
    15.      
    16.         IMediaRecorder recorder;
    17.         IClock clock;
    18.         CameraInput cameraInput;
    19.         AudioInput audioInput;
    20.      
    21.         bool isRecording = false;
    22.  
    23.         void Start()
    24.         {
    25.             isRecording = false;
    26.             recordButtonText.text = "Record";
    27.         }
    28.  
    29.  
    30.         void StartRecording()
    31.         {
    32.             clock = new RealtimeClock();
    33.             recorder = new MP4Recorder(640, 480, 30, sampleRate, channelsCount);
    34.             cameraInput = new CameraInput(recorder, clock, camera);
    35.             audioInput = new AudioInput(recorder, clock, microphone);
    36.             isRecording = true;
    37.         }
    38.  
    39.  
    40.         void StopRecording()
    41.         {
    42.             cameraInput.Dispose();
    43.             audioInput.Dispose();
    44.             recorder.FinishWriting();
    45.             recorder = null;
    46.             isRecording = false;
    47.         }
    48.  
    49.  
    50.         public void OnButtonClick()
    51.         {
    52.             if(!isRecording)
    53.             {
    54.                 StartRecording();
    55.                 recordButtonText.text = "Stop";
    56.             }
    57.             else
    58.             {
    59.                 StopRecording();
    60.                 recordButtonText.text = "Record";
    61.             }
    62.         }
    63.     }
    But when I tried to record video with audio I got a corrupted .mp4 file. If I commenting out audio input recording, all works well (of course, without sound). But I need an audio in file. Where can be an error?
     
  46. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    If you are recording Unity audio (recording from an AudioListener or AudioSource) then your sample rate and channel count MUST match those of Unity Engine. See the ReplayCam example for how we get the proper sample rate and channel count to pass to the recorder.
     
  47. DaniilGalahow

    DaniilGalahow

    Joined:
    Dec 2, 2013
    Posts:
    9
    Thank you, but it was no use. I changed string 33 to
    recorder = new MP4Recorder(1280, 720, 30, AudioSettings.outputSampleRate, (int)AudioSettings.speakerMode);
    , but still not working.

    I am recording sound from AudioListener, attached to another gameobject (not same to which my script attached). I also tried different sample rate and channels, and with some rates it records video but without audio, with other rates it records corrupted files, like in my case. (Maybe this info can be useful to resolve the trouble).
     
  48. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    Can you share one such corrupted file? What device and OS version are you recording on?
     
  49. DaniilGalahow

    DaniilGalahow

    Joined:
    Dec 2, 2013
    Posts:
    9
    Yes, I can. I'll send to you a Google Drive link in private message.

    I'm using PC with Win7 as workstation. App in development is for Android and iOS. Unity version is 2019.3.2f1. I tried to record only on Win7 yet.
     
  50. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,960
    NatCorder doesn't support Windows 7. It requires Windows 10 64-bit. This is mentioned on the Asset Store page and in the README.
     
unityunity