NatCorder - Video Recording API

Ho noes, I though it would have been easy.
Anyway how do you think I can prevent the double rendering then?

 

You can't, except you implement what I have described above.

 

Ok thank you for your help

 

Is there any way to improve video render quality while recording video with unity camera?(in android)
Here is my current project graphics setting.
Case 1) When I use only 1 render which is ==> Vulkan render, I am getting 0 device support while uploading aab file in play store(Google console)

Case 2) When I use below graphic setting(Find in attachment) I am getting low clarity on rendering

please guild me how project setting should be while using Vulkan render

 

Figuring out the device filter support on Google Play has nothing to do with NatCorder; a google search would be more helpful for figuring that out. As for quality, if you are getting a low quality from the recording, then consider increasing the recording bitrate (see the MP4Recorder constructor docs).

 

Unity 2019.3.2f1
NatCorder 1.7.0
Unity editor crashes (or deployed build on WIndows 10) everytime
when I call recorder.FinishWriting(); without any logs or traces to find the cause.
Please , help!

 

Can you share your recording code?

 

Sure , but i did lots of attempts and still experiencing this crash. At first its been crashing just when I start recording, but with this attempt I reused CameraInput class like this and now only crashes at the end of movie while calling videoRecorder.FinishWriting(). Also my recording is not frame based , I pass timestamp manually (my target is to record scene views very fast , like making 30fps movie of 2 minutes in 10 seconds (unlimited unity player fps, but not all frames used for recording for other project reasons). Works fine on older Unity and Natcorde versions)

Code (CSharp):

1. using NatCorder;
2. using NatCorder.Clocks;
3. using System.Collections;
4. using System.Collections.Generic;
5. using UnityEngine;
6. using UnityEngine.UI;
7. using UnityEngine.Events;
8. using NatCorder.Internal;
9. using UnityEngine.Rendering;
10.  
11. [System.Serializable]
12. public class MyStringEvent : UnityEvent<int, string>
13. {
14. }
15.  
16. public class VideoRec : MonoBehaviour
17. {
18.  
19.     public const int NEW_VIDEO = 1;
20.     public const int STOPPED_RECORDING = 2;
21.  
22.     public Camera mainCamera;
23.  
24.     public RenderTexture CameraTexture;
25.  
26.     private MP4Recorder videoRecorder;
27.  
28.     public MyStringEvent m_RecorderEvent = new MyStringEvent();
29.  
30.     private RenderTexture frameBuffer;
31.     private Texture2D readbackBuffer;
32.     private byte[] pixelBuffer;
33.  
34.     public VideoRec()
35.     {
36.      
37.     }
38.  
39.     public void StartRecording()
40.     {
41.         Debug.Log("mainCamera resolution " + mainCamera.pixelWidth + " " + mainCamera.pixelHeight);
42.         Debug.Log("framerate " + Config.VIDEO_FRAMERATE);
43.         mainCamera.forceIntoRenderTexture = true;
44.      
45.         videoRecorder = new MP4Recorder(mainCamera.pixelWidth, mainCamera.pixelHeight, Config.VIDEO_FRAMERATE);
46.  
47.         var frameDescriptor = new RenderTextureDescriptor(videoRecorder.frameSize.width, videoRecorder.frameSize.height, RenderTextureFormat.ARGB32, 24);
48.         frameDescriptor.sRGB = true;
49.         this.frameBuffer = RenderTexture.GetTemporary(frameDescriptor);
50.  
51.         this.readbackBuffer = SystemInfo.supportsAsyncGPUReadback ? null : new Texture2D(frameBuffer.width, frameBuffer.height, TextureFormat.RGBA32, false, false);
52.         this.pixelBuffer = new byte[frameBuffer.width * frameBuffer.height * 4];
53.  
54.     }
55.  
56.     public async void StopRecordingAsync()
57.     {
58.         m_RecorderEvent.Invoke(STOPPED_RECORDING, null);
59.      
60.         if (videoRecorder != null)
61.         {
62.             string result = await videoRecorder.FinishWriting();
63.             OnRecording(result);
64.             videoRecorder = null;
65.          
66.         }        
67.     }
68.  
69.     // Update is called once per frame
70.     public IEnumerator Tick(long timeStamp)
71.     {    
72.         if (videoRecorder != null && mainCamera.activeTexture != null)
73.         {
74.             var endOfFrame = new WaitForEndOfFrame();
75.             while (true)
76.             {
77.                 // Check frame index
78.                 yield return endOfFrame;
79.  
80.  
81.                 var prevTarget = mainCamera.targetTexture;
82.                 mainCamera.targetTexture = this.frameBuffer;
83.  
84.                 mainCamera.Render();
85.                 mainCamera.targetTexture = prevTarget;
86.  
87.                 if (SystemInfo.supportsAsyncGPUReadback)
88.                     AsyncGPUReadback.Request(frameBuffer, 0, request =>
89.                     {
90.                         if (pixelBuffer != null)
91.                         {
92.                             request.GetData<byte>().CopyTo(pixelBuffer);
93.                             videoRecorder.CommitFrame(pixelBuffer, timeStamp);
94.                         }
95.                     });
96.                 else
97.                 {
98.                     var prevActive = RenderTexture.active;
99.                     RenderTexture.active = frameBuffer;
100.                     readbackBuffer.ReadPixels(new Rect(0, 0, frameBuffer.width, frameBuffer.height), 0, 0, false);
101.                     readbackBuffer.GetRawTextureData<byte>().CopyTo(pixelBuffer);
102.                     videoRecorder.CommitFrame(pixelBuffer, timeStamp);
103.                     RenderTexture.active = prevActive;
104.                 }
105.             }
106.  
107.         }
108.     }
109.  
110.     void OnRecording(string path)
111.     {    
112.         m_RecorderEvent.Invoke(NEW_VIDEO, path);    
113.     }
114. }

 

Damn! It's problematic for us... Any way to get it working on 2018 ? Maybe I missed a post... Thanks!

 

Your code is prone to crashing when recording is stopped. You need to make sure that the moment FinishWriting is called, there will be other calls to the recorder. Your Tick method can still try to commit frames to the recorder while it is finishing, which is almost guaranteed to cause a crash. Notice how `videoRecorder` will only become `null` after the recorder has finished writing. This will cause `Tick` to keep trying to commit frames while the recorder is finishing, which will cause the crash.

 

Nope. This is a Unity issue, and Unity fixed it in 2019.2.

 

Hello,

I am getting very low fps while recording video through natcorder,
1) Without natcorder 29-32 fps.
2) While recording video getting around 9-11 fps.
Is anything else could I do to increase fps on android device.!

 

Thank you very much for noticing. Now its working without crash.

 

There are many variables to look at. The first is how complicated your app's rendering is before recording. It your app's rendering takes a lot of CPU and GPU time, then adding recording will naturally reduce your frame rate. Also, the higher your recording resolution, the more work the device has to do so consider reducing the resolution.

 

Hello, I encountered a problem with your plugin on some devices.

Initially, in our project, it was noticed on the Honor 8X device, during recording with some chance the sound ceases to be recorded. The sound is simply interrupted, mainly in the beginning of the recording.
At the end of the video, the sound breaks off https://yadi.sk/i/3jHtaKOOUDpPng

The problem repeated on Unity 2019.1.14 and 2019.3.1.

In a clean draft of the same versions, the device has an asynchronous video and audio recording, a delay of about 2 seconds goes to the audio track. And each time increases.

And also a bug was sometimes noticed with the fact that the audio track from the previous record was superimposed on a new one, but on the previous one it was not at all and continues like this, the previous audio track is sewn into the video.

The video recorded the work with Xiaomi mi 9 SE and example project ReplayCam
https://onedrive.live.com/?authkey=!AA2DfCbkYAPv8Es&cid=CF3AE80FA057CEB9&id=CF3AE80FA057CEB9!36242&parId=root&o=OneUp

Also on some devices, on the video you can see, at the end of the recording, a black frame with noise



Sorry for my English, thanks!

 

Hey, whats the current code to pause/resume recording?
Before it was RealtimeClock.Paused = true and disposing audio+ video.
However this property isn't there anymore.
Whats the replacement?

 

Hi there,

Are x86 architectures still supported? When I build an apk it is missing the libNatCorder and libNatRender for x86 but they are present for armeabi-v7a.

 

These issues all revolve around Unity's Microphone API, which is known to produce glitchy audio. We developed our own microphone API, NatMic, because of issues like this. Check it out.

 

The property is still there. Nothing happened to it.

 

None of the NatSuite API's support x86 on Android. This is in line with Unity dropping support for it.

 

Hello, I experience a first black frame on iPhone 11 when I record but the texture isn't black

Relevent code :

Code (CSharp):

1.     public void StartRecording () {
2.         bool haveSound = false;
3.         if(Microphone.devices.Length > 0){
4.             audioSource = GetComponent<AudioSource>();
5.             audioSource.clip = Microphone.Start(Microphone.devices[0], true, 60, 48000);
6.             audioSource.Play();
7.             haveSound = true;
8.         }
9.         if(haveSound){
10.             _videoRecorder = new HEVCRecorder(Screen.width, Screen.height, 60, 48000, 1, OnRecording, 5909760*4, 1);
11.         } else {
12.             _videoRecorder = new HEVCRecorder(Screen.width, Screen.height, 60, 0, 0, OnRecording, 5909760*4, 1);
13.         }
14.         _clock = new RealtimeClock();
15.         _ARCamera.Render();
16.         _cameraInput = new CameraInput(_videoRecorder, _clock, _ARCamera);
17.         running = true;
18.     }

 

Hi!

How to prevent cameras from switching, if I am recording with different camera (for excluding UI elements)? When I press button and hold to record, it goes from main camera to recording camera. Is it possible record in background aka so on screen i still see main camera, but recording from other camera?

 

Upgrade to NatCorder 1.7.0 and try this again. Let me know if the issue persists.

 

This might be a bug. Can you share screenshots of both cameras in the inspector? And can you share your recording code?

 

Thanks but the 1.6.6 is the latest available in the asset store manager in Unity 2020.1.0a17

 

That's weird. I recommend sending a report to Unity. NatCorder will work with Unity 2018.3+.

 

I update to the latest version of unity and was able to use the 1.7, the black frame is gone ! thanks

Im using the latest unity : 2020.1.0a24 & the 1.7 Natcoder ,

also here the relevant code, the biggest change here is on the use of the new AudioInput :

Code (CSharp):

1.  
2. public void StartRecording () {
3. bool haveSound = false;
4. if(Microphone.devices.Length > 0){
5. audioSource = GetComponent<AudioSource>();
6. audioSource.loop = true;
7. audioSource.clip = Microphone.Start(Microphone.devices[0], true, 30, AudioSettings.outputSampleRate);
8. audioSource.Play();
9. haveSound = true;
10. }
11. if(haveSound){
12. _videoRecorder = new HEVCRecorder(Screen.width, Screen.height, 30, 48000, 1, 5909760, 1);
13. } else {
14. _videoRecorder = new HEVCRecorder(Screen.width, Screen.height, 30, 0, 0, 5909760, 1);
15. }
16. _clock = new RealtimeClock();
17. _ARCamera.Render();
18. _cameraInput = new CameraInput(_videoRecorder, _clock, _ARCamera);
19. audioInput = haveSound ? new AudioInput(_videoRecorder, _clock, audioSource, true) : null;
20.  
21. running = true;
22. }

 

Spoke too fast sorry, there is now a memory problem after fews seconds the max memories is reach

 

Can you share the logs, and can you share screeshots of the app's frame time with and without recording? Such memory behaviour is impossible with NatCorder alone.

 

The peak start immediately when I start recording, I edit my previous post with the relevant code and I didnt change anything else in the app except the update of Unity and Natcorder to 1.7 from 1.6.6, before I could easily register 3minutes on Natcorder with sounds and without memory problem.

 

Logs with Natcorder :
2020-02-27 11:03:29.894204+0100 appreality[726:106615] NatCorder: Prepared HEVCRecorder video encoder at resolution 1125x2436@30.000000Hz with average bitrate 5909760 and keyframe interval 1s

2020-02-27 11:03:29.896958+0100 appreality[726:106615] NatCorder: Prepared HEVCRecorder audio encoder for 1 channels at 48000Hz

WARNING -> applicationDidReceiveMemoryWarning()

WARNING -> applicationDidReceiveMemoryWarning()

2020-02-27 11:03:34.037718+0100 appreality[726:107284] [ServicesDaemonManager] interruptionHandler is called. -[FontServicesDaemonManager connection]_block_invoke

Crash memory almost always on the GPU Async request ( see join screenshot )

Also join the screenshot of memory when natcorder isnt use, nothing to signal, the memory app is stable ( like with the v1.6.6 )

Tested on iphone11 latest iOS, latest xcode, latest unity, v1.7.

 

Your recording resolution is extremely high, so it makes sense that the async readback requests pile up in memory and make it explode. Record at 1280x720 or 1920x1080 instead. You have to pay attention to your app's frame time (CPU and GPU, in Xcode). If the GPU can't keep up, pending requests will accumulate in memory and your app will crash.

 

I agree its high but it didnt crash after fews minuts on the v1.6.6, its like the gpu isnt debounce on the new v1.7

 

At some point in 1.6 we were using Texture2D.ReadPixels to read pixel data from the GPU. This has very stable memory performance, as it blocks until the data is ready. The problem is that it blocks until the data is ready, giving worse performance. Metal supports AsyncGPUReadback, so we use it. Either way, I strongly recommend reducing that recording resolution. It is way too high.

 

In the v1.6.6 it was already using the asyncGPUReadback and was recording at the same resolution ( I agree I should put a limit at 1080x1920 ) and that's actually the main reason I decide to use NatCorder, as you said ReadPixels is not good at all. I just wonder whats the breaking change on the core API of NatCorder leading to such a recession in terms of performance and stability from the v1.6.6 to the v1.7?

 

Nothing changed on iOS, macOS, or Windows between 1.6 and 1.7. Again, profile your app for its performance characteristics as that will tell you what to do.

 

I did more test and even when the recording is finished in the ReplayCam the memory is never getting lower, shoudnt you work with small buffer to limit the usage in the ram and write in async on the disk ?

 

We don't keep any buffers at all, which is why I'm saying the memory behaviour you are seeing looks spurious. I cannot reproduce this behaviour with the ReplayCam example, which is always a good place to start.

 

I revert Unity to the 2020.1.0a17 and its now working!!

 

So it must've been a Unity memory leak. I recommend filing a bug report with them anyway. In any case, this sounds good.

 

What Unity version were you using that had the memory leak?

I'm using 2018.4.14f1 and I'm experiencing it. @Lanre you were able to verify this
https://forum.unity.com/threads/natcorder-video-recording-api.505146/page-50#post-5264069

 

The leak you faced is not of the same order as the one the OP mentioned, so it isn't the same issue. The OP was seeing memory explode immediately recording was started. I haven't tried to reproduce this leak with 1.7.0. I will do so and get back to you.

 

Hello!
I have trouble with recording video with sound through combination MP4Recorder+CameraInput/AudioInput. All made like in readme:

Code (CSharp):

1. public class VideoRecorderBehaviour : MonoBehaviour
2.     {
3.         [SerializeField]
4.         Camera camera = null;
5.         [SerializeField]
6.         AudioListener microphone = null;
7.         [SerializeField]
8.         Text recordButtonText = null;
9.  
10.         [SerializeField]
11.         int sampleRate = 44100;
12.         [SerializeField]
13.         int channelsCount = 2;
14.  
15.      
16.         IMediaRecorder recorder;
17.         IClock clock;
18.         CameraInput cameraInput;
19.         AudioInput audioInput;
20.      
21.         bool isRecording = false;
22.  
23.         void Start()
24.         {
25.             isRecording = false;
26.             recordButtonText.text = "Record";
27.         }
28.  
29.  
30.         void StartRecording()
31.         {
32.             clock = new RealtimeClock();
33.             recorder = new MP4Recorder(640, 480, 30, sampleRate, channelsCount);
34.             cameraInput = new CameraInput(recorder, clock, camera);
35.             audioInput = new AudioInput(recorder, clock, microphone);
36.             isRecording = true;
37.         }
38.  
39.  
40.         void StopRecording()
41.         {
42.             cameraInput.Dispose();
43.             audioInput.Dispose();
44.             recorder.FinishWriting();
45.             recorder = null;
46.             isRecording = false;
47.         }
48.  
49.  
50.         public void OnButtonClick()
51.         {
52.             if(!isRecording)
53.             {
54.                 StartRecording();
55.                 recordButtonText.text = "Stop";
56.             }
57.             else
58.             {
59.                 StopRecording();
60.                 recordButtonText.text = "Record";
61.             }
62.         }
63.     }

But when I tried to record video with audio I got a corrupted .mp4 file. If I commenting out audio input recording, all works well (of course, without sound). But I need an audio in file. Where can be an error?

 

If you are recording Unity audio (recording from an AudioListener or AudioSource) then your sample rate and channel count MUST match those of Unity Engine. See the ReplayCam example for how we get the proper sample rate and channel count to pass to the recorder.

 

Thank you, but it was no use. I changed string 33 to

recorder = new MP4Recorder(1280, 720, 30, AudioSettings.outputSampleRate, (int)AudioSettings.speakerMode);

, but still not working.

I am recording sound from AudioListener, attached to another gameobject (not same to which my script attached). I also tried different sample rate and channels, and with some rates it records video but without audio, with other rates it records corrupted files, like in my case. (Maybe this info can be useful to resolve the trouble).

 

Can you share one such corrupted file? What device and OS version are you recording on?

 

Yes, I can. I'll send to you a Google Drive link in private message.

I'm using PC with Win7 as workstation. App in development is for Android and iOS. Unity version is 2019.3.2f1. I tried to record only on Win7 yet.

 

NatCorder doesn't support Windows 7. It requires Windows 10 64-bit. This is mentioned on the Asset Store page and in the README.

 

Uh, OK. Thank you. I thought it wolud run in Editor without issues on Win7 too. OK, then I'll test app directly on Android.

 

I am about to purchase your asset to replace Everyplay and now Ultimate Replay Kit. I am moving away from Ultimate Replay Kit because when I start to record on Android it pops up a dialogue that asks for permission. This happens once per launch which is too much.

Does NatRecorder have a similar need to pop up a permission dialogue? Thank you.