Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

NatCorder - Video Recording API

Discussion in 'Assets and Asset Store' started by Lanre, Nov 18, 2017.

  1. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    Can you send me a repro project? The bitrate setting doesn't have any effect on memory usage during recording.
     
  2. Regone

    Regone

    Joined:
    Aug 1, 2014
    Posts:
    26
    As I was failing to reproduce the problem in a clean new project with the same configuration, build options I had results which were waaaaa...aaaaaay better than what I had in my App's project and it looks as expected.

    Then I started stripping down the entire project and came to realise the culprit is an outline Asset for some weird reason...

    I guess the issue is SOLVED and I'm sorry for the inconvenience I may have caused.

    But there is one last issue which I'm trying to figure out - When shooting a video despite if there is anything "performance heavy" eveything looks stuttery when recording and the result video file has 24fps even though I've manually set it to 60fps, am I using it wrong?
    ( new VideoFormat(videoWidth, videoHeight,60);)

    Disclaimer: Checked the fps with vlc from UnityEditor recording.
     
    Lanre likes this.
  3. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    This depends on whether your system can handle reading back from the recording render textures at 60FPS (which it should), and whether you are actually committing frames at 60FPS (set your Application.targetFramerate to 60).
     
  4. siamon

    siamon

    Joined:
    Apr 14, 2015
    Posts:
    7
    Hey, just checking is this fix live already - i can only see 1.4.1 as current version on the assetstore
     
  5. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    The package is pending Asset Store review.
     
    siamon likes this.
  6. Regone

    Regone

    Joined:
    Aug 1, 2014
    Posts:
    26
    Yeah, I see a fps difference depending on what I look at only while recording:(, but how come the iO'S native screen-capture method (from the dashboard) is capturing it smoothly?
     
  7. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    iOS's native screen recorder has access to the screen's backbuffer (pixel data already sitting in system memory). NatCorder on the other hand isn't a screen recorder, so we must first readback pixel data from the GPU which is where the performance cost comes from.
     
    Regone likes this.
  8. texnoline21

    texnoline21

    Joined:
    Feb 6, 2016
    Posts:
    10
    Good day.
    I apologize for my bad english.
    Thanks for updating NatCorder. Everything works fine. Because of the poor quality of the recording from the microphone in Unity, NatMik was purchased. Unfortunately I can not understand how to save sound and video in one file. All I managed to achieve is to save the video and audio as separate files. I would be grateful for your help.

    Code (CSharp):
    1. namespace NatMicU.Examples
    2. {
    3.     using UnityEngine;
    4.     using Core;
    5.     using Core.Recorders;
    6.     using NatCorder.Clocks;
    7.     using NatCorder.Inputs;
    8.     using NatCorder;
    9.     using NatShareU;
    10.  
    11.     public class VideoRecord : MonoBehaviour
    12.     {
    13.         public AudioSource audioSource;
    14.         private IRecorder recorder;
    15.  
    16.         public string localpatch;
    17.  
    18.         private IClock clock;
    19.  
    20.         [Header("Recording")]
    21.         public int videoWidth = 1280;
    22.         public int videoHeight = 720;
    23.  
    24.         private MP4Recorder videoRecorder;
    25.         private IClock recordingClock;
    26.         private CameraInput cameraInput;
    27.  
    28.         public void StartRecord()
    29.         {
    30.             var microphoneFormat = Format.Default;
    31.             var device = Device.Default;
    32.            
    33.             // Create a WAV recorder to record the audio to a file
    34.             recorder = new WAVRecorder(microphoneFormat, OnAudioFile);
    35.             recorder.StartRecording();
    36.  
    37.             // Start recording
    38.             NatMic.StartRecording(device, microphoneFormat, OnSampleBuffer);
    39.            
    40.             //var audioFormat = new AudioFormat(microphoneFormat.sampleRate, microphoneFormat.channelCount);
    41.             recordingClock = new RealtimeClock();
    42.             videoRecorder = new MP4Recorder(videoWidth, videoHeight, 30, 0, 0, OnVideo, 1320000);
    43.            
    44.             // Create recording inputs
    45.             cameraInput = CameraInput.Create(videoRecorder, Camera.main, recordingClock);
    46.         }
    47.  
    48.         public void StopRecord()
    49.         {
    50.             // Stop the recording inputs
    51.             cameraInput.Dispose();
    52.             // Stop recording the WAV file and dispose the recorder
    53.             recorder.Dispose();
    54.             // Stop recording
    55.             NatMic.StopRecording();
    56.             videoRecorder.Dispose();
    57.         }
    58.  
    59.         private void OnSampleBuffer(float[] sampleBuffer, long timestamp)
    60.         {
    61.             // Commit the sample buffer to the WAV recorder
    62.             recorder.CommitSamples(sampleBuffer, timestamp);
    63.         }
    64.  
    65.         private void OnAudioFile(string path)
    66.         {
    67.             // Log the path
    68.             Debug.Log("Saved recording to: " + path);
    69.         }
    70.  
    71.         private void OnVideo(string path)
    72.         {
    73.             Debug.Log("Saved recording to: " + path);
    74.             localpatch = path;
    75.  
    76.             // Playback the video
    77. #if UNITY_EDITOR
    78.  
    79. #elif UNITY_IOS
    80.            
    81. #elif UNITY_ANDROID
    82.            
    83. #endif
    84.         }
    85.  
    86.         // Share video
    87.         public void OnShare()
    88.         {
    89.             NatShare.Share(localpatch);
    90.         }
    91.  
    92.     }
    93. }
     
  9. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    Check out the NatMic README.md for how to use NatMic and NatCorder together. Also check out this example project.
     
  10. texnoline21

    texnoline21

    Joined:
    Feb 6, 2016
    Posts:
    10
    Unfortunately, the information in README, as well as the example of the project, became outdated after the release of NatCorder 1.5
     
  11. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    For the most part, the integration is the same. Where you call `NatCorder.SomeFunction`, replace it with `videoRecorder.SomeFunction`. You need to send the audio data from NatMic (in the OnSampleBuffer function) to the NatCorder recorder. Also, when creating the video recorder, the sample rate and channel count must match those of the microphone format when you start NatMic. You are currently passing in zero.
     
  12. specular12

    specular12

    Joined:
    Jan 28, 2014
    Posts:
    8
    Hello, I updated to 1.5 from 1.3f3 in an existing project, and I am getting the error:

    The type or namespace name `Docs' could not be found. Are you missing an assembly reference?

    for the files: NatCorder.cs, NatCorderTypes.cs, AudioRecorder.cs, and CameraRecorder.cs

    On all the files, the "using Docs;" namespace is apparently invalid. I checked the changlog but I don't happen to see an input name change. The project is using Unity 2018.3. Any help would be greatly appreciated! Thanks
     
  13. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    Delete NatCorder in your project before importing the update from the Asset Store. You have stale sources.
     
    ina likes this.
  14. texnoline21

    texnoline21

    Joined:
    Feb 6, 2016
    Posts:
    10
    I am not a programmer, alas. Unfortunately, without working examples, NatMik and NatCorder are useless to me. Individually, they work great, but unfortunately I could not understand how to get them to work together.
     
  15. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    This isn't much of an impediment to using both API's. The API's are designed to be extremely simple. There is a ton (and I can't emphasize this enough) of complexity we are hiding by providing API's with only 3 or 4 functions. The very least you can do is to try to get an understanding of what these 3 or 4 functions do.
    First, read through the README.md's of both API's. Get a basic understanding of how they work and ask me for clarification if there is anything that doesn't seem clear. Trying to just piece together something from example code is not a sustainable way to learn new API's.

    When you call NatMic.StartRecording, you pass in a function to receive the audio data that is being streamed from the microphone (this is your OnSampleBuffer method). In this method, you want to commit the audio data to NatCorder for encoding to the MP4 that is being recorded. So call `videoRecorder.CommitSamples` passing in the sample buffer. As for the timestamp to pass to CommitSamples, use `clock.Timestamp` instead of the timestamp that is provided by NatMic (this detail is explained in the NatMic README).
     
  16. seneka123

    seneka123

    Joined:
    Jun 16, 2013
    Posts:
    30
    Hello! Please share the asset with me! =)
    My email address: seneka38@gmail.com
     
  17. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    We are no more offering the API for beta testing. Instead, grab it from the Asset Store.
     
  18. Regone

    Regone

    Joined:
    Aug 1, 2014
    Posts:
    26
    Heya :)
    I can't find any equivalent in MP4Recorder for Videoformat's fields such as .height .width. frames

    example:
    Code (CSharp):
    1.  
    2. var videoFormat = new VideoFormat( (int)(((float)videoHeight /Screen.height)*Screen.width), videoHeight,videoFPS);
    3.  
    4. Debug.LogFormat("Starting video recoding in {0}x{1}@{2}fps - {3}bps ({4}kb/s)", videoFormat.width, videoFormat.height, videoFormat.framerate, videoFormat.bitrate, videoFormat.bitrate* 0.000125f);
    I know there have been changes in the API but is there something im missing that I can get the information out of later?
     
  19. seneka123

    seneka123

    Joined:
    Jun 16, 2013
    Posts:
    30
    Hello! How to specify your own path to save the video file?
     
  20. seneka123

    seneka123

    Joined:
    Jun 16, 2013
    Posts:
    30
    In my application, the landscape orientation of the screen is prohibited, I independently determine the orientation of the phone in space, and on this basis I need to record video in portrait mode, or in landscape, but since There is no automatic (Unity) change in the orientation of the screen, then in the landscape orientation of the screen, the video simply stretches along the width. I change the resolution of the video stream, but the video continues to record in the vertical orientation of the screen. Is it possible to manually rotate the orientation of the video for recording?
     
  21. DigitalBeach

    DigitalBeach

    Joined:
    Jan 17, 2015
    Posts:
    29
    We also have need of being able to specify the filename for the saved video. I don't really need to save the full path but want to change the name of the file generated. I don't see any way to do this.
     
  22. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    The MP4Recorder doesn't expose the width and height that are passed in. Since you specify the width and height when constructing the MP4Recorder in the first place, you might as well save those values for later use.
     
  23. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    This is not exposed for good reason. If you need the recording to be in some location, with some name, or both, use the System.IO.File.Move API to do so.
     
  24. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    You can't change the resolution of a video (no media format I know of has variable dimensions). Beyond that, you can write a custom CameraInput script to use a shader to rotate the frame texture.
     
  25. alek123

    alek123

    Joined:
    Dec 9, 2012
    Posts:
    3
    Hey Lanre I just update today a NatCorder from previous API version to current one 1.5. But not sure where is a problem but previous version on same resolution and same file format record about 54FPS but current one on same settings record only 5.7FPS.

    Current code for new version (1.5)
    Code (CSharp):
    1.   var width = 1280;
    2.         var height = 720;
    3.  
    4.         _recordingClock = new RealtimeClock();
    5.         _videoRecorder = new MP4Recorder(
    6.             width,
    7.             height,
    8.             30,
    9.              0,
    10.               0,
    11.             OnReplay
    12.         );
    13.  
    14.      
    15.        
    16.         _cameraInput = CameraInput.Create(_videoRecorder, Camera.main, _recordingClock);
    17.        
    18.  
    19.  
    20.         yield return new WaitForSeconds(15f);
    21.  
    22.         StopRecording();
    23.  

    Code for previous version :



    Code (CSharp):
    1.  
    2.         var framerate = _container == Container.GIF ? 10 : 30;
    3.         var videoFormat = new VideoFormat(width, (int)height, framerate);
    4.         _recordingClock = new RealtimeClock();
    5.         NatCorder.StartRecording(_container, videoFormat, new AudioFormat(), OnReplay);
    6.         if(_videoRecorder ==null)
    7.         _videoRecorder = CameraRecorder.Create(Camera.main, _recordingClock);
    8.         if (_container == Container.GIF)
    9.             _videoRecorder.recordEveryNthFrame = 5;
    10.  
    11.  
     
  26. Skjalg

    Skjalg

    Joined:
    May 25, 2009
    Posts:
    151
    I'm having troubles with performance. I'm attaching some profiler screenshots here for you to take a look at. It seems like the culprit is the Texture2D.GetRawTextureData, even on iOS that you claim shouldn't generate GC memory when committing frames.

    Mac: Skjermbilde 2019-03-20 kl. 11.43.33.png

    iPhone 6 Plus
    Skjermbilde 2019-03-20 kl. 12.46.30.png

    I'm using the latest 1.5 NatCorder.

    Initializing Recording Code:
    Code (CSharp):
    1.  
    2. public static void StartRecording()
    3. {
    4.             var framerate = LowEndDevice.IsLowEndDevice() ? 24 : 30;
    5.             tonemapping.recordEveryNthFrame = Mathf.RoundToInt(Application.targetFrameRate / (float)framerate);
    6.             recordingClock = new NatCorder.Clocks.RealtimeClock();
    7.             //0,4618226600985222
    8.             //Video Format
    9.             var width = LowEndDevice.IsLowEndDevice() ? 720 : 1280;
    10.             var height = Mathf.CeilToInt((width * (Screen.height / (float) Screen.width)));
    11.             //bit rate is same as Everyplay
    12.             var bitrate = 3072000 ; // natcorder default is 960 * 540 * 11.4f
    13.             var keyFrameInterval = 3;
    14.        
    15.             //Audio Format
    16.             var sampleRate = AudioSettings.outputSampleRate;
    17.             var channelCount = (int)AudioSettings.speakerMode; // Stereo = 2
    18.        
    19.             videoRecorder = new NatCorder.MP4Recorder(width, height, framerate, sampleRate, channelCount, OnRecordingDone, bitrate, keyFrameInterval);
    20.             tonemapping.Clock = recordingClock;
    21.             tonemapping.Recorder = videoRecorder;
    22.             SkateSnd.Clock = recordingClock;
    23.             SkateSnd.Recorder = videoRecorder;
    24.         }
    25.  
    Video Recording Code:
    Code (CSharp):
    1.  
    2. public void OnPostRender()
    3. {
    4.             if (Recorder != null && frameCount++ % Mathf.Clamp(recordEveryNthFrame, 1, 8) == 0)
    5.  
    6.             {
    7.                 var encoderFrame = Recorder.AcquireFrame();
    8.                 Graphics.Blit(_destinationRenderTexture, encoderFrame, material, renderPass);
    9.                 Recorder.CommitFrame(encoderFrame, Clock.Timestamp);
    10.             }
    11. RenderTexture.ReleaseTemporary(_destinationRenderTexture);
    12. }
    13.  
    Audio recording code
    Code (CSharp):
    1.         private void OnAudioFilterRead (float[] data, int channels)
    2.         {
    3.             if (ReplayManager.IsSupported() && ReplayManager.IsRecordingSupported() && ReplayManager.IsRecording())
    4.             {
    5.                 Recorder.CommitSamples(data, Clock.Timestamp);
    6.             }
    7.         }
    I've also tested this on iPhone 5 with the same results. The GC Alloc and subsequent GC hit brings down the fps from a steady 30 to 5.
     
    Last edited: Mar 20, 2019
  27. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    What device and OS version does this happen on?
     
  28. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    So there are a few observations to make here. The first is that we changed how NatCorder works on iOS such that it will make memory consumption a lot more predictable (thus preventing memory crashes). The consequence of doing this is GC.

    The second observation is that the allocation doesn't have any effect on performance whatsoever. As you can see, 'Time ms' is 0.00. What has a performance cost is reading back pixel data from the committed RenderTextures, which is an operation we simply can't avoid. On all supported platforms except Android, the native NatCorder implementation expects raw pixel data, so we must do a readback from your committed RenderTextures at some point.

    The last observation is that we also can't avoid the GC. Reason is that Texture2D.GetRawTextureData will always allocate and doesn't provide an overload for copying into a client-provided buffer.
     
  29. alek123

    alek123

    Joined:
    Dec 9, 2012
    Posts:
    3
    That was in Editor (2018.3.7f1 - WINDOWS 10 64bit)
     
  30. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    Can you share a screenshot of the profiler? Check where the massive frame time is coming from.
     
  31. Skjalg

    Skjalg

    Joined:
    May 25, 2009
    Posts:
    151
    What about the AsyncGPUReadback method mentioned earlier? It was in preview mode 6 months ago, but is fully supported and stable right now in 2018.3.

    Will you take advantage of that for 2018.3 and up to miniminze the GC? Right now Natcorder is unusable as it allocates 1.1mb of GC every frame it commits, which makes the GC run every other frame, which brings the fps down to about 6-7 from 30. I can understand that you want to support older unity versions, but thats what #if UNITY_2018_3_OR_NEWER defines are for.

    Just to note, for comparison, Everyplay on the other hand only costs me a few frames (down to 27-28 from 30) - but since that is deprecated and the hudless feature stopped working in 2018 I really want NatCorder to be a viable option.
     
  32. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    AsyncGPUReadback makes no difference. If anything, it'll make memory usage a lot more unpredictable (NatCorder already did this before, but we had to move away from it). The readback is an expensive operation. Whether sync or async, it takes time.
    This is false as shown in the profiler screenshots you shared. The GC collection is for the lowest generation, and takes 0.00ms (as shown in your screenshots). GC isn't a bottleneck at all; the bottleneck is in the readback which is unavoidable as I've explained.
    Everyplay doesn't need to do a readback because it has access to the screen's backbuffer in system memory. NatCorder is not a screen recorder, so none of that applies to us. Once again, the bottleneck you are seeing is in the readback. GC has nothing to do with it, as seen in the profiler screenshots you shared.
     
    Skjalg likes this.
  33. Skjalg

    Skjalg

    Joined:
    May 25, 2009
    Posts:
    151
    You're absolutely right about the screenshot just showing the GC.Allocation. (not collection as you say) But as I previously mentioned, this allocation of 1.1mb makes the Garbage Collector run (collection) every other frame (which means it runs in the next frame, which is not shown in the screenshot as that is not the actual culprit, but rather the symptom). If you pay close attention to the sinus-curve looking brown graph there, the gc goes up then down then up then down every other frame. Because GC runs, which takes up far more ms than anything else.
    upload_2019-3-20_18-46-32.png

    upload_2019-3-20_18-47-29.png
     
  34. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    How much time in ms are those GC peaks? With an allocation like this, the only option is to keep running GC on every frame (so as to avoid large hiccups). Again, this is all unavoidable for us (and we've asked Unity to revise their API's to make it more bearable, but to no avail).
     
  35. lowfrq

    lowfrq

    Joined:
    Nov 14, 2018
    Posts:
    3
    Hello, Lanre) How can i return prev version (1.4) of plugin. Bc in new version needs to refactor code, but i have no time now. Thanks!
     
  36. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    The refactor will take you less than a minute. Simply follow the README.md and use the example code for guidance.
     
  37. korovamilkbar

    korovamilkbar

    Joined:
    Oct 2, 2014
    Posts:
    1
    Hi,
    Are you guys planning to add a feature for pausing record session?
    I want to pause recording when user opens pause menu.
     
  38. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    You can already 'pause' recording. Simply stop committing frames for recording for the duration of the pause. Check out this article.
     
  39. whuaso79

    whuaso79

    Joined:
    Apr 20, 2013
    Posts:
    2
    Anyone having trouble making the Replay Cam sample to work (with record microphone enabled) on android devices?
    I've tried in a Pixel 3 and a Galaxy S7 and it freezes as soon as I start recording.

    Thanks
     
  40. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    Can you upload the full logs from logcat in a .txt file?
     
  41. whuaso79

    whuaso79

    Joined:
    Apr 20, 2013
    Posts:
    2
    Full log and summary
     

    Attached Files:

    Last edited: Mar 23, 2019
  42. gabrieltva

    gabrieltva

    Joined:
    Oct 28, 2015
    Posts:
    5

    Hello,

    I Have the same problem here. When will start record, the script consumes much more CPU and the device turn very slow.
    Look the screenshot of the Unity Profiler:
    Captura de Tela 2019-03-22 às 16.55.20.png

    The version of plugin is 1.5.0. But the 1.4v do the same problem.

    Any way to solve that?

    Thanks,
    Gabriel Stafoca
     
    Last edited: Mar 22, 2019
    whuaso79 likes this.
  43. Skjalg

    Skjalg

    Joined:
    May 25, 2009
    Posts:
    151
    You need to find another plugin to record your screen with.
    Here's a neat graph to give you some pointers on what to test next:
     
    whuaso79 and gabrieltva like this.
  44. gabrieltva

    gabrieltva

    Joined:
    Oct 28, 2015
    Posts:
    5
    Nice.

    I used the Everyplay in Android, but in iOS, can I get the path of the video recorded?
    I need to get the path, play there on screen, and upload to the server.
     
  45. Skjalg

    Skjalg

    Joined:
    May 25, 2009
    Posts:
    151
    Yeah Everyplay works fine on iOS, but only if you record the entire screen as the hudless recording feature doesn't work any more. It has a callback, same as Natcorder that gives you a path to the recorded video and you and you can take it from there.
     
    gabrieltva likes this.
  46. gabrieltva

    gabrieltva

    Joined:
    Oct 28, 2015
    Posts:
    5

    Thank you. But I need to record just a canvas or a camera and the video need to be defined in a determined resolution. Not the screen resolution. :/
     
  47. Skjalg

    Skjalg

    Joined:
    May 25, 2009
    Posts:
    151
    Aha, then I suggest you take a look at Next Gen Recorder
     
  48. gabrieltva

    gabrieltva

    Joined:
    Oct 28, 2015
    Posts:
    5
    Unfortunately, this plugin doesn't have support to android.
    I need both, iOS and Android.
     
    Last edited: Mar 23, 2019
    whuaso79 likes this.
  49. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    You seem to be erroring on the microphone not being able to start:
    Code (CSharp):
    1. E Unity   : Starting microphone failed: "An error occured that wasn't supposed to.  Contact support. " (33)
    Check that you have granted the app microphone permissions.
     
  50. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,616
    Are you recording on Windows? If so, are you recording with audio? There is a memory leak that affects Windows currently (the fix is coming in the next minor update).