NatCorder - Video Recording API

Also, I bought NatDevice to add reliable audio to my recordings but the documentation is incomplete on that subject:



Basically, how do I go about adding the microphone device's audio with NatDevice's API to the NatCorder MP4Recorder? I bet that's what most people want to do no? The AudioInput class does not seem to be able to deal with NatDevice's API.

Looking forward to hear from you!

Thank you!!

 

Using AsyncGPUReadback i get a red tint on the rendered video on MacOS in 2020.2.2f1. It's not a real problem i just swicthed to not using Async instead. Maybe you can fix it. Thank you for the amazing recorder!

 

The backtrace mentions Unity, not NatCorder. I would recommend filing a bug report with Unity.

 

I'm working on an example that integrates both, but the integration is pretty simple. For the camera, you can follow what ReplayCam does, except instead of using WebCamTexture for the preview, you use NatDevice (see the docs). For the microphone, see the docs.

 

The performance hit you are seeing is because of pixel buffer readbacks from the GPU. This all happens in Unity, before NatCorder is involved.

I'm working on a free extension library for NatCorder that will include an optimization for performing async readbacks on OpenGL ES for Android.

Oops I didn't realize this was incomplete. I'll fix that today. You can look at the same page on NatCorder's documentation.

 

This is a Unity bug, so I recommend filing a bug report with Unity. There isn't any fix we can write; it has to be fixed upstream.

 

Hi Lanre! Thanks for the quick response! Always appreciated!

The whole point of my application is to record videos for social media so the framerate has to be 30 fps while recording otherwise the app is kinda pointless!

Is there an ETA on the free extension library, everyone who does AR and wants to be recording video would definitely benefit from that which is probably a lot of people that buy the asset is my guess?

Here's the profiler where I highlighted the culprit in the profiler. (the section at the beginning is without recording, and the section at the right is while recording) (Here's the full data of the profiler)

Is that all because of the Pixel buffer readbacks from the GPU you think?

Is there anything I can do on my end to alleviate that?





On IOS, I do see the EarlyUpdate.UpdateAsyncReadbackManager taking around 22% of CPU usage and it goes from 60 fps to 45 fps which is still above the 30 fps mark so that's good.

 

This depends on a few more things, like your app's CPU and GPU frame time without recording and what rendering API is being used. See the first link I shared in my last post for a full explanation. I'll try to have the library up next week.

Yes, readbacks are very (very) expensive. If you're interested in further reading, see this post by Google. There isn't much you can do to improve this except say reducing your recording resolution. Other alternatives involve writing native rendering code, which is complicated if you aren't familiar with native graphics programming.

This is the cost incurred by Unity's AsyncGPUReadback implementation. A proper implementation shouldn't have that high a cost, but I don't know how they implemented theirs so I can't say.

 

Thank you for the detailed answer Lanre!

Ok great! I'll try your library once it's up and running!

Interesting read, the Google post!

I tried using HD instead of using FHD on Android and it actually didn't change too much in terms of fps!

I also had an issue where the sound was not matching the image in Adobe Premiere Pro but I realized that the reason was that the MP4recorder generates a VFR video and that Adobe Premiere Pro does not automatically correct the clips generated by the MP4recorder as it does with other VFR videos. I had to transcode it first in a software called "Handbrake" to a Constant Framerate video for it to work.

Maybe it has to do with the fact that the clips generated by the MP4recorder on my Samsung S7 Edge have the extension .3gp for some reason instead of .mp4! Your documentation says it should be a .mp4 file :
https://docs.natsuite.io/natcorder/api/mp4recorder :
"NatCorder includes the MP4Recorder for recording *.mp4 video files. This is currently the most common video format, and as such, practically every device in the world has hardware encoding support for it."

I did have an idea to fix my problem though. I saw that we can do offline recording using Natcorder. What if I just save the frames image/audio information in an array and then when the recording is done, I would just start an offline recording in a background thread? (And give the user a progress bar indicator or something)

Would that work? I saw another app that does that called "Snaappy" on Google Play and it produces a .mp4 at the end too!

 

You can use a fixed interval clock to guarantee a constant frame rate.

On some Android devices, copying the video to the gallery will append a .3gp extension in the file name, even though the file is an MP4. If you look at your app's private documents directory where NatCorder saves videos, you'll see that it's an MP4 file. And if you transfer the 3gp file to your computer and inspect it with mediainfo, you'll also see that it's still an MP4 file. You can completely ignore the 3gp extension.

I don't think any of this is necessary. Also, offline recording requires special care because you can't store too many frames in memory before you exhaust RAM and your app crashes with an OOM. Finally, offline recording doesn't support audio; only video. The better paradigm for 'offline recording' for audio and video is to record both separately (a video-only MP4 and audio-only AAC or WAV), then simply combine them without transcoding. NatCorder doesn't provide any facilities for doing this, so you'll need something like FFmpeg.

 

Thank you for the quick response again!

I tried commenting the RealtimeClock and using a FixedIntervalClock instead and I'm still getting a variable frame rate if I inspect the video file in MediaInfo. The video is much slower though than with the RealtimeClock but the fps is better though at around 28 fps but it went down to 6 fps like you can see below. I was expecting a constant 30 fps no matter the fps of the app.

Code (CSharp):

1. //var clock = new RealtimeClock();
2. var clock = new FixedIntervalClock(30);

Ah ok I'll ignore the 3gp extension then! It's definitely still an MP4 !

Alright, I won't go that route if you don't think it necessary! I guess I was trying to postpone what could cause fps drops during the recording to later but it seems like that's just too complicated for nothing!

So, I think the best route for me right now is to stick with 720p with Android and use the real-time clock since the FixedIntervalClock is not really working for me unless I'm missing something. And then, when you're done with the free library, I'll give it a shot to see if it will help with the performance issues on Android with AR Foundation.

The only other thing I wish would work is for Adobe Premiere Pro to recognize the recorded clip as VFR:
https://premierebro.com/blog/premiere-pro-1201-update-variable-frame-rate-and-new-features

Here's a clip recorded with the app Snaappy (AR app on Google Play) and Adobe Premiere Pro does recognize the VFR correctly and compensate for it (Maybe missing some metadata?)
https://we.tl/t-oTiboH8apD
(I have attached the media info of the snappy video and the video I took with NatCorder too)

This what I get for the snappy video (Variable Frame Rate Detected) within Adobe Premiere Pro but I'm not getting it with the videos recorded with NatCorder.

 

This is weird. If you use a fixed interval clock, the video should have a constant frame rate. Can you share your recording code? I should note that even though the video will be at 30FPS, the passage of time won't be constant (because your app isn't actually rendering at 30FPS).

Can you share your full recording code? I can't quite tell what's going on.

Also, I think I realize why you're getting a variable frame rate with the fixed interval clock. When using a fixed interval clock, you should not share the same clock across both video and audio streams. The audio stream should keep using a RealtimeClock.

 

@aholla did you ever figure this out? I just had the same issue for the first time.

 

On iOS, if recording fails, the culprit is usually either that the device is running low on storage space; or the recording timestamps are not strictly monotonic.

 

Hey @Lanre I'm not getting any audio either on mac or any handheld devices (android, IOS). Video recording is all good. Am I missing something for audio ?
Here's my code

Audio in the scene is coming from VideoPlayer which is set to "Direct"

Code (CSharp):

1. public void StartRecording()
2.     {
3.         Vector2 recordRes = RecordConstants.GetValidRecordRes();
4.  
5.         int frameRate = 30;
6.         recordingClock = new RealtimeClock();
7.         recorder = new MP4Recorder(
8.             (int)recordRes.x,
9.             (int)recordRes.y,
10.             frameRate,
11.             AudioSettings.outputSampleRate,
12.             (int)AudioSettings.speakerMode
13.         );
14.  
15.         // RecordCams[1] assigned from Inspector
16.         RecordCams[0] = GetVideoCamera();
17.  
18.         // Create recording inputs
19.         cameraInput = new CameraInput(recorder, recordingClock, RecordCams);
20.         audioInput = new AudioInput(recorder, recordingClock, RecordCams[1].gameObject.Getcomponent<AudioListener>());
21.  
22.     }
23.  
24.     public async void StopRecording()
25.     {
26.         audioInput?.Dispose();
27.         cameraInput.Dispose();
28.         var path = await recorder.FinishWriting();
29.  
30.         Debug.Log($"Saved recording to: {path}");
31.  
32.        }

 

NatCorder can't record direct audio from a VideoPlayer because direct audio doesn't go through Unity. You're trying to record audio from the scene's audio listener, so whatever audio you play has to be heard by the audio listener. Change the audio mode to go through an audio source that gets played in the scene.

 

Yeah I got that part figured out now. The issue is I download AR assets from server which has a VideoPlayer attached to it. This asset has its VideoPlayer's audio output set to "Direct" and changing this output dynamically to "AudioSource" doesn't work. It only works when VideoPlayer gameobject is not dynamically created but already part of the scene and has it's audio output set to "AudioSource"

 

You might want to file that as a bug with Unity.

 

Hi Lanre,
NatCorder is really cool API and I purchased the Pro version. Just meet an issue when I try to test using my IPhone via XCode using your exmamples (webCam and replyCam). I can record the video and replay after I 'touch up'. But I can not find the video in my IPhone photo Gallery. When I test in Unity, it is working and video clip could be stored in the Unity project folder path. Many thanks

 

Hey there. NatCorder never saves recordings to the camera roll; it always saves recordings to the app's private documents directory.

You can use a native gallery plugin like NatShare to copy the video to the camera roll.

 

Hi Lanre,
I´m working on a project that record screen with video player component, I set the audio output mode to audio source and attached the Recorder audio source. The issue is when I start recording the audio in the app stops. The recorded video has the audio as espected but I need that the audio on the application never stops. How can I solve this issue?

 

There is a `mute` flag in the `AudioInput` constructor. Make sure it is set to `false`.

 

Hi Lanre,
I am using Natcorder in ARCore. However, I was struggling with a lot of rapid performance during video recording, and then I found the link.
https://github.com/natsuite/NatCorder-Performance-Extensions
Are the NatCorder Performance Extensions on that link currently unavailable?

 

NCPX is still under development, and using it will require NatCorder 1.8.0 which I'll be submitting to the Asset Store tomorrow.

 

Been looking thru the demos and docs and I can't figure out how to record just a specific GO such as the preview raw image without picking up other UI GO on the canvas. Is this possible?

 

Thanks for the answer!
We are always using assets well. Please thrive~

 

If you are trying to record a specific (non-UI) game object, create a camera and put your object in a layer mask that only this camera can see, then record just that camera. If you are trying to record a specific UI object, create a canvas with that object, then set it to "screen-space camera" and use the camera you created, then record just that camera.

 

Lanre, thanks for the reply. I had been working it that way but missed a couple additional settings that made your suggestions work perfect. For the additional settings,, check out the Answer on this link:

https://stackoverflow.com/questions/53480971/unity2d-layering-multiple-canvas

 

Hello, I am interested in your platforms.

In order to record an AR Foundation scene with microphone comments by the user, and save it directly on the Android/iOS gallery, I need both NatCorder (cam video) and NatDevice (mic audio), and also use NatShare (from github) to save it on the devices gallery (Android/iOS). Is this correct?

If so, are there any conditions for acquiring both together, or the only option is buying them separetely?

Thanks.

 

This is correct. NatCorder will record the scene in AR (see this example project). You can then use NatDevice to add reliable microphone audio (see the docs). And finally, you can use NatShare to copy the recorded video to the camera roll (see the docs).

Not currently. They aren't bundled together; instead you have to purchase them separately.

 

Thanks for the response.

Another quick question: are there any way to record the video without the GUI layout, to remove the record button from the video?

 

Hi,

I'm very frustrated using this product. I never had so many unity crashes trying to using this library. You really need to work on stability.

I'm trying to do the easy task of recording from camera with audio at 1280x720@30fps but, between the crashes, I got only bad MP4 files with wrong duration, different frame rate from what I ask, out of sync audio/video.

I have bought both packages, probably I'm doing something wrongs, but the library crash brutally too frequently.

Can You provide an example to record both camera and audio that works?

 

The easiest way to remove UI from a recording is to place it in an Overlay canvas, so that it doesn't get seen during recording.

 

Overwhelming chances are that you're doing something that leads to a crash. If you use NatCorder properly, you could only ever get a managed exception when recording is stopped, which you can intercept with a try-catch statement.

Do you face crashes and similar problems when recording in the ReplayCam example? The video frame rate is controlled by the frame timestamps (i.e, your app's actual, and often variable, frame rate), not by the numbers you pass into the constructor of the (MP4/HEVC) recorder. Audio syncing issues will likely come from Unity's Microphone API, which is why I strongly recommend using NatDevice in a production environment. Unity's Mic API has known glitches.

Have you tried the included ReplayCam example?

 

Well-designed software doesn't crash for wrong calls, at least throws an exception.

ReplayCam example is not applicable: is not working with NatSuite.Devices and the result video is not in sync with audio.

Can you provide a simple working example to record video from a native camera device in sync with a microphone at 1280x720@30fps?

 

Except you're working with native code, in which case C doesn't have exceptions. And the NatCorder Unity API is pretty much a lightweight wrapper for the underlying C library.

I'm working on integration examples but the example for NatDevice and NatCorder is not ready yet. Besides, the documentation already has the necessary code. Can you share your code?

When the integration example is ready, it'll be on GitHub. Before then, feel free to share your code and I can point you to any issues you might have.

 

Code (CSharp):

1. namespace NatSuite.Examples {
2.  
3.     using UnityEngine;
4.     using System.Collections;
5.     using Recorders;
6.     using Recorders.Clocks;
7.     using Recorders.Inputs;
8.     using NatSuite.Devices;
9.     using static NatSuite.Devices.MediaDeviceQuery;
10.  
11.     public class CameraRecorderTest : MonoBehaviour {
12.  
13.         [Header(@"Recording")]
14.         public int videoWidth = 1280;
15.         public int videoHeight = 720;
16.         public int frameRate = 30;
17.         public int bitrates = 2_000_000;
18.  
19.         private IMediaRecorder videoRecorder;
20.  
21.         bool isRecording = false;
22.  
23.         MediaDeviceQuery cameraDevices;
24.         IAudioDevice audioDevice;
25.         ICameraDevice cameraDevice;
26.  
27.         public MeshRenderer cameraRenderer;
28.  
29.         Texture2D cameraTexture;
30.         IClock clock;
31.  
32.         public async void Start()
33.         {
34.             Application.targetFrameRate = frameRate;
35.             QualitySettings.vSyncCount = 1;
36.  
37.             if (!await MediaDeviceQuery.RequestPermissions<CameraDevice>())
38.             {
39.                 Debug.LogError("User did not grant camera permissions");
40.                 return;
41.             }
42.             // Request mic permissions
43.             if (!await MediaDeviceQuery.RequestPermissions<AudioDevice>())
44.             {
45.                 Debug.LogError("User did not grant microphone permissions");
46.                 return;
47.             }
48.  
49.             var deviceQuery = new MediaDeviceQuery(MediaDeviceQuery.Criteria.AudioDevice);
50.             // Get the device
51.             audioDevice = deviceQuery.currentDevice as AudioDevice;
52.  
53.             cameraDevices = new MediaDeviceQuery(Criteria.GenericCameraDevice);
54.             // Start camera preview
55.  
56.             Debug.LogFormat("Audio device {0} (available:{1})", audioDevice, deviceQuery.devices.Length);
57.  
58.             cameraDevice = cameraDevices.currentDevice as ICameraDevice;
59.             Debug.LogFormat("Camera device {0}  (available:{1})", cameraDevice, cameraDevices.devices.Length);
60.  
61.             if (cameraDevice != null)
62.             {
63.                 cameraDevice.previewResolution = (videoWidth, videoHeight);
64.                 cameraRenderer.material.mainTexture = cameraTexture = await cameraDevice.StartRunning();
65.                 cameraRenderer.transform.localScale = new Vector3((9f / cameraTexture.height) * cameraTexture.width, 9, 1);
66.             }
67.  
68.         }
69.  
70.  
71.         public void StartRecording ()
72.         {
73.             var sampleRate = audioDevice.sampleRate;
74.             var channelCount = audioDevice.channelCount;
75.             clock = new RealtimeClock();
76.            
77.             videoRecorder = new MP4Recorder(videoWidth, videoHeight, frameRate: frameRate, sampleRate, channelCount,bitrate: bitrates);
78.             audioDevice.StartRunning(videoRecorder.CommitSamples);
79.             isRecording = true;
80.         }
81.  
82.         private void Update()
83.         {
84.             if (isRecording && videoRecorder!=null)
85.             {
86.                 videoRecorder.CommitFrame(cameraTexture.GetPixels32(), clock.timestamp);
87.             }
88.         }
89.  
90.         public async void StopRecording ()
91.         {
92.             isRecording = false;
93.             if (audioDevice!=null) audioDevice.StopRunning();
94.  
95.             var path = await videoRecorder.FinishWriting();
96.             // Playback recording
97.             Debug.Log($"Saved recording to: {path}");
98.             Handheld.PlayFullScreenMovie($"file://{path}");
99.          
100.         }
101.  
102.         private void OnDestroy()
103.         {
104.             if (cameraDevice != null) cameraDevice.StopRunning();
105.         }
106.     }
107.     }

 

Hi, I have noticed from the example code on git for AR here, and also from other example sources, that the base resolution for all the examles is the HD resolution (1280x720), and sometimes there is just a correction for the aspect ratio been used.

I had problems with larger resolutions in other libs I have tested, that ended crashing the devices.

I have a concern here, does the mp4 recorder support larger resolutions, such as native screen sizes for devices with Full HD, 2k, 4k screen resolution?

 

So a few issues in your code:

• You request a 1280x720 camera preview, but then start recording with the resolution you requested. The key detail to keep in mind is that the numbers you pass to the camera device are merely requested values. The NatDevice docs mention this. When the camera preview starts, you have to get the actual preview resolution using `cameraDevice.previewResolution` or by checking the dimensions of the preview texture (which is the correct way to go when recording, since it respects orientation). From here, you must pass in these values to the MP4Recorder constructor. You're lucky that recording isn't crashing immediately because just about every camera supports a 1280x720 camera preview. If you used a different resolution, it would cause a segmentation fault and hard-crash because there's no way to NatCorder to catch this.

• When you start streaming the microphone to the recorder, the recorder is using the system timestamps (from NatDevice) instead of the timestamps from the recording clock (see the docs on this). This is likely why your audio and video are out of sync. You need to change that line of code to this:

Code (CSharp):

1. audioDevice.StartRunning((sampleBuffer, timestamp) => {
2.     // Use the recording clock timestamp, NOT the device's timestamp
3.     videoRecorder.CommitSamples(sampleBuffer, clock.timestamp);
4. });

This should fix the issues you're facing.

 

Support for high resolutions depends on the device you're running on. The reason all examples start at 1280x720 is because of performance considerations. As the recording resolution increases, the system falls under more load. And given that Unity apps can be very heavy on hardware, without recording, adding recording could put the system under a lot more load. So it is always recommended to start from 1280x720 as a reference resolution, then increase it as you profile your app across devices and see if you can maintain reasonable performance.

 

@Lanre Thank you so much for developing this incredible recording plugin. My app has a minimum Android API level of 21. Will there be anyway for NatCorder 1.7.4 to work on Android API level 21? I am using Unity 2019.4.22. Thank you again for this unique plugin.

 

Hey there. There isn't a way to run NatCorder on older versions of Android unfortunately. Also the next update will bump up the requirements to Android API level 24 and iOS 13.

 

I kindly suggest you explain it in the Recorder section, there aren't any clues that video size must be the exact size of camera device size (https://docs.natsuite.io/natcorder/api/mp4recorder).

So, if camera preview is not 1280x720, how I can record a video to 1280x720?

 

That's up to you depending on any considerations your app has. If you require 1280x720 recording resolution, then you have to come up with a way to 'squeeze' the camera preview into a frame that is exactly that size. You can use shaders and GPU rendering for this, but again it all depends on what approach is best for you.

 

Probably, the best way is to record a camera that looks at the preview camera texture. I've just tried it but I change the strategy after some crash. I'm going to check the previous code, probably the error on the sound recording was causing the crash.

Thanks.

 

I bother you with another request for help. I have a scene with a NatCamera. The preview texture is used as material texture. When the app loses focus, I have a crash. How can I avoid it?

Thanks.

 

You have to stop any devices and recorders you have running before the app loses focus. You can do this using Unity's OnApplicationFocus method. And when the app regains focus, restart the devices.

 

@Lanre, what is the best way to programmatically delete a video generated by NatCorder? Thank you for your awesome plugin.

 

Hey @Mr-Mechanical the System.IO.File API is what I'd use. Deciding when to delete videos, and keeping track of them, is where you'd need to put some thought. You could walk the persistent data path directory; or better yet always record to the same file name so you don't get tons of videos.

 

Hi Lanre!
I have a question about the example ReplayCam.
I put a Sprite in the scene and the sprite is always move, when I start record ,the sprite paused.but the video also record the Sprite path,When i End record,the Sprite begin move. how can i fix it.? Thanks.