NatCorder - Video Recording API

No, NatCorder doesn't use replay kit (or the Android equivalent). It works directly with the hardware encoders, so it doesn't ask for the user's permission.

 

I'm updating to the latest version of the plugin - why was RenderTextureInput removed?
Do you have any examples using a render texture for recording?

 

Because Unity already provides all the functionality you need. See `Texture2D.ReadPixels` and `AsyncGPUReadback`. We use these functions in `CameraInput`.

 

I have two cameras that I use in my gameplay. During my gameplay the I often switch to the secondary reaction camera when an event happens. Is there a way in NatCorder to switch to a different camera in the middle of a replay recording session? Currently it stays on my main gameplay cam and ignores the switch.

This is a legacy game that is not using Cinemachine which would definitely make my life easier.

Thanks!

 

Destroy your current CameraInput then create a new one for the new camera. Make sure to keep the same clock as before.

 

@Lanre does Natcorder work with URP?

 

@Lanre I'm using NatCroder with Vuforia and noticed that the fps drops drastically both in iPhone11pro and iPad6. Is the video resolution have any affect to framerates?

 

Yup.

 

The recording resolution is the most important performance factor when recording. Make sure to profile your app at different recording resolutions to get a good understanding of its performance characteristics.

 

It seems I have almost everything working with Natcorder, but am hoping for some guidance with sharing and playback. Currently I can playback the video using the Handheld class and I know you have NatShare that works with NatCorder. Is there a way to playback the video and from that playback view invoke share functionality? Or do I have to close the playback view and have a separate interface?

Thanks

 

You'll have to do this in separate steps. Handheld.PlayFullscreenMovie suspends Unity Engine and puts a video player view on top of the app. While Unity is suspended, you cannot call any sharing code.

 

Hi,

I repurposed your ReplayCam script to record video with game audio. I have two questions regarding this. Whenever I start recording, the audio in the game mutes but the recorded video has the audio without any issues. Is this expected behavior?
Second question is whether we can record both game audio and microphone with the video? Or only one source is possible?

Here is the code:

Code (CSharp):

1. namespace NatCorder.Examples {
2.  
3.     using UnityEngine;
4.     using System.Collections;
5.     using Clocks;
6.     using Inputs;
7.     //using NatShare;
8.  
9.  
10.     public class ReplayCam : MonoBehaviour {
11.  
12.         [Header("Recording")]
13.         public int videoWidth = 1280;
14.         public int videoHeight = 720;
15.         public bool recordMicrophone;
16.         string existingMediaPath;
17.         public AudioSource playMusic;
18.  
19.         private IMediaRecorder videoRecorder;
20.         private CameraInput cameraInput;
21.         private AudioInput audioInput;
22.         private AudioSource microphoneSource;
23.         private string path;
24.  
25.         //private IEnumerator Start()
26.        // {
27.             // Start microphone
28.             //microphoneSource = gameObject.AddComponent<AudioSource>();
29.             //microphoneSource.mute =
30.             //microphoneSource.loop = true;
31.             //microphoneSource.bypassEffects =
32.             //microphoneSource.bypassListenerEffects = false;
33.             //microphoneSource.clip = Microphone.Start(null, true, 30, AudioSettings.outputSampleRate);
34.             ////microphoneSource.clip = 'BB-Tune.wav';
35.  
36.             //yield return new WaitUntil(() => Microphone.GetPosition(null) > 0);
37.             //microphoneSource.Play();
38.        // }
39.  
40.         private void OnDestroy () {
41.             // Stop microphone
42.             //microphoneSource.Stop();
43.             //Microphone.End(null);
44.         }
45.  
46.         public void StartRecording () {
47.             // Start recording
48.             var frameRate = 30;
49.             Debug.Log("");
50.             var sampleRate = recordMicrophone ? AudioSettings.outputSampleRate : 0;
51.             var channelCount = recordMicrophone ? (int)AudioSettings.speakerMode : 0;
52.             var recordingClock = new RealtimeClock();
53.             videoRecorder = new MP4Recorder(
54.                 videoWidth,
55.                 videoHeight,
56.                 frameRate,
57.                 sampleRate,
58.                 channelCount,
59.                 recordingPath => {
60.                     Debug.Log($"Saved recording to: {recordingPath}");
61.                     path = recordingPath;
62.                     existingMediaPath = path;
63.                     var prefix = Application.platform == RuntimePlatform.IPhonePlayer ? "file://" : "";
64.                     Handheld.PlayFullScreenMovie($"{prefix}{recordingPath}");
65.                     SaveVideo(existingMediaPath);
66.                  
67.                 }
68.  
69.             );
70.             // Create recording inputs
71.             cameraInput = new CameraInput(videoRecorder, recordingClock, Camera.main);
72.             audioInput = recordMicrophone ? new AudioInput(videoRecorder, recordingClock, playMusic, true) : null;
73.             // Unmute microphone
74.             //microphoneSource.mute = audioInput == null;
75.         }
76.  
77.         public void StopRecording () {
78.             // Stop recording
79.             audioInput?.Dispose();
80.             cameraInput.Dispose();
81.             videoRecorder.Dispose();
82.  
83.             //using (var payload = new SavePayload())
84.             //{
85.             //    payload.AddMedia(path);
86.             //    Debug.Log("User Video Saved");
87.             //}
88.             // Mute microphone
89.             //microphoneSource.mute = true;
90.          
91.          
92.         }
93.  
94.         public void SaveVideo(string existingMediaPath)
95.         {
96.             //Debug.Log("Triggered"+ existingMediaPath);
97.             NativeGallery.SaveVideoToGallery(existingMediaPath, "AMR", "AMRGallery", null);
98.         }
99.     }
100. }

 

ReplayCam's AudioInput is created to mute the audio of the audio source it records from. You'll want to change that.

You'll have to figure out a way to mix the audio before sending the final mixed audio to NatCorder. You can't just create multiple audio inputs.

 

Thanks for your answers. Will definitely try it.

 

Hi Lanre,

Did you manage to solve this issue with playing video on chrome browser?

We faced the same problem that others had before, but could not find an answer.
However, we noticed that the videos recorded from Windows and iOS play successfully.

 

Excuse me, I bought your Asset, but I cannot find the APIs documentation.
Can you please provide a link to the docs?

thanks!

 

We haven't gotten a fix for this unfortunately. It is originating from MediaCodec, which is Android's hardware encoder API.

 

There is a Docs folder in the NatCorder directory. That contains the full API reference.

 

Can I record AR and UI canvas together? I used CameraInput but it record only AR, and I can't find ScreenInput.

 

Make sure that your UI canvas is not in Overlay rendering mode. The README mentions this detail.

 

This is a very critical issue for us. This problem should disappear when using the encoder in .webm format.

How difficult is it to implement from your side? We are ready to pay for this work.

 

We will not be implementing WEBM support. If it is necessary for your application, then your best bet is to write a native plugin that does this.

 

Thank you for creating an amazing asset.
I am using NatCorder 1.6.1, NatMic 1.3.3.

Everything is playing well now.
But, There is just one problem.
This is a very urgent issue as we are currently serving an app / web.

What you shot in the app should play in Chrome.

There is an issue in which videos recorded on certain AOS devices cannot be played when played on Chrome. It plays well everywhere else. It is not playing only on Chrome.

There seems to be a problem with the audio codec.
If you disable audio-related code, it will play well in Chrome as well.

If modification is difficult in a short period of time, is there an alternative to modify using a different encoding asset?

I will wait for your answer.

 

There is one more question.

We are using Unity version 2018.4.17f1.

In order to install the latest version of Natcoder, it is necessary to replace it with 2019 or higher.

I would like to know the highest version of natcoder that can be used as 2018.4.17f1 version.

 

Hi @Lordmin and @Yode_group,

I have been using NatCorder since the very initial versions and I have been facing this very issue since version 1.6. Let me save you guys some time with what I have found till now for this.

First of all, @Lordmin, you were correct to identify that without the audio, the video works fine on chrome. I had done quite a bit of to and fro with the Chrome browser dev team and @Lanre for this and found out that the problem lies with the muxing of audio and video especially in case of Android device's media codec where the very first(or some of the initial) audio packet in the final output video is corrupted. You can check this with FFmpeg by running

ffmpeg -err_detect explode -i <file> -acodec pcm_f32le -vn -f wav -y /dev/null

Now, the default video player of the chrome browser (HTML5 video player) is a little stringent in this case and blocks payback even if there is a single bad packet. In this case, the bad one is the very first audio packet or one of the initial ones so it does not start playback at all.

I had raised a bug with the chromium team here, to try to convince them to let the video play like other browsers do by dropping bad packets, but they seem to be strict about the behavior and asked me to fix the video instead.

Now, the problem we all are in here is that the video recorded in NatCorder uses the default media codec of Android to mux the audio and video. And the issue of corrupt audio packet arising from the default media codec itself is seemingly difficult to overcome as the audio packet turns corrupt after muxing in the media codec. So the problem arises where NatCorder does have much control over.

So, how to solve/work around it?
Well, I could think of 3 alternatives.

The first option is to convince the chrome browser's dev team to drop bad packets and allow playing such a video like all other browsers and almost all video players do. The reason I am putting this in the list of solutions is that the videos are muxed using the default codecs of Android itself and the video player demands a perfect video without a single bad packet. So indirectly we might not have been in this situation if the Android's default media codec would have worked the way Chrome's dev team would want it to do.

Second and more approachable in case of video streaming on browsers is transcoding the final output using encoders like FFmpeg. It works and the video plays in the chrome browser once transcoded. If video streaming is what you guys want then you can also transcode the video to use HLS or MPEG-DASH to have faster and adaptive streaming.

The third alternative is NatCorder uses another codec instead of the default one to mux audio and video in Android specifically. A good one is VLC but its a choice based on the asset's architecture and up to @Lanre.

I hope it helps.

 

Hi,

I have implemented the recording feature and it records without any hitch. But while recording there is noticeable stutter and it's not as smooth. What are some recommended settings for getting best performance.
I am recording at 720p but even if I reduce the resolution to 480p the performance remains the same.

Other question is regarding offline recording. What exactly is this feature and how to achieve this. I didn't see anything about this in the documentation. Sorry if this is already answered.

 

This is not possible. We use the official platform-specific API for encoding.

This would be the best fix imo. But the Chrome team seems to be quite stubborn. On my end, I can only follow the official MediaCode API spec. Trying to do any fancy magic to inspect the AAC packets and somehow modify them is simply impractical.

Be careful with FFmpeg due to licensing.

I can't use a different encoder; NatCorder is limited to whatever the official media API is for the specific operating system. Using a third party encoder opens up a nasty can of licensing and dependency issues that makes it not worth building and maintaining on my end.

 

You can use NatCorder 1.7.0 with Unity 2018.3+. The Asset Store says 2019 because that's the version used to upload the package.

 

Are you recording on Android? If so, make sure that your app is rendering with Vulkan instead of OpenGL ES.

This means recording not in realtime--think transcoding applications. I'm working on a project with a bunch of examples, but here's one that illustrates transcoding with NatReader and NatCorder. When doing offline recording, there is a crucial detail to note: timestamps. You must make sure that you timestamps are not based on real time. For this purpose, NatCorder includes the FixedIntervalClock which generates evenly spaced out timestamps.

 

Yes, that's right. I am using it on a ARCORE enabled app.
In the player setting, under graphics API I have put Vulkan on top and OpenGL in the bottom and enable automatic Graphics API. Is that the correct way to go? When I disable automatic graphics API and keep the same order (Vulkan then OPENGL) the app doesn't load on my test phone (One Plus 3t).

Thanks for the info, I will try it out.

 

It looks like ARCore doesn't support Vulkan. In this case, you'll have to stick with OpenGL ES.

 

Getting Errors in Xcode 11.3 for NatCodrer 1.6.6

Undefined symbols for architecture arm64:

"_NCIsRecording", referenced from:

_NatCorderBridge_IsRecording_mB457DA41B1F46A303F67C99CEFD77169BF720929 in Bulk_Assembly-CSharp_3.o

_NatCorderOSX_get_IsRecording_mAEB076C2FC3834F8660C86A5AD248FE590BE9798 in Bulk_Assembly-CSharp_3.o

_NatCorderOSX_CommitSamples_m89ED6FDD0A7C5CF5D9102625C352686B78F0C0D2 in Bulk_Assembly-CSharp_3.o

_U3CU3Ec__DisplayClass13_0_U3CCommitFrameU3Eb__1_m71398D161044D80FB353000DDEA2481B3EB7E0E9 in Bulk_Assembly-CSharp_3.o

_NatCorderiOS_get_IsRecording_mEBD164B680E1238A9AFFD06ABCF24920429759C4 in Bulk_Assembly-CSharp_3.o

_NatCorderiOS_CommitSamples_m2A3B678A8ACD58E3067EE85924A98529045215BA in Bulk_Assembly-CSharp_3.o

"_NCInitialize", referenced from:

_NatCorderBridge_Initialize_mD2FDB5BF13E73BC12AD24018A6D9ECE359749422 in Bulk_Assembly-CSharp_3.o

ld: symbol(s) not found for architecture arm64

clang: error: linker command failed with exit code 1 (use -v to see invocation)

And
Undefined symbols for architecture arm64:

"_NCIsRecording", referenced from:

_NatCorderBridge_IsRecording_mB457DA41B1F46A303F67C99CEFD77169BF720929 in Bulk_Assembly-CSharp_3.o

_NatCorderOSX_get_IsRecording_mAEB076C2FC3834F8660C86A5AD248FE590BE9798 in Bulk_Assembly-CSharp_3.o

_NatCorderOSX_CommitSamples_m89ED6FDD0A7C5CF5D9102625C352686B78F0C0D2 in Bulk_Assembly-CSharp_3.o

_U3CU3Ec__DisplayClass13_0_U3CCommitFrameU3Eb__1_m71398D161044D80FB353000DDEA2481B3EB7E0E9 in Bulk_Assembly-CSharp_3.o

_NatCorderiOS_get_IsRecording_mEBD164B680E1238A9AFFD06ABCF24920429759C4 in Bulk_Assembly-CSharp_3.o

_NatCorderiOS_CommitSamples_m2A3B678A8ACD58E3067EE85924A98529045215BA in Bulk_Assembly-CSharp_3.o

"_NCInitialize", referenced from:

_NatCorderBridge_Initialize_mD2FDB5BF13E73BC12AD24018A6D9ECE359749422 in Bulk_Assembly-CSharp_3.o

ld: symbol(s) not found for architecture arm64

clang: error: linker command failed with exit code 1 (use -v to see invocation)

 

Those symbols are obsolete. Delete NatCorder in your project and import NatCorder 1.7.0 from the Asset Store.

 

I am Using 2018.4.14 Natcorder 1.7.0
On iPhone XR(iOS 13.3.1) and iPhone 6s (iOS 12.2)
Natcorder not working properly
i.e
I have a playback animation to record as an mp4. I played animation and recorded it, Its recorded animation and gave mp4 but while it's recording I'm unable to see playback animation. With the previous version of Natcorder it's working fine but the new version of Natcorder I'm unable to see playback animation. It's just giving an mp4 file.

 

I'm a bit confused. What is this playback animation you are referring to? Is it a video? Is a Unity animation (moving game objects in the scene)? Recording will have no effect on what is happening in Unity Engine, so chances are that there is some issue in your code.

 

I read it all, and I must say that it's scarce to say the least.....I would not call it a documentation, but rather a list of classes.

However, I'm struggling to understand why my MP4Recorder completely ignores the framerate parameter. The resulting mp4 file always has a variable framerate corresponding to my program's frame rate....Even if the program's framerate is higher than the desired mp4 framerate.....

 

See this article.

 

Hi Lenre!

Which one is best Configuration for recording video with high quality and lower video size.Right now I am using

width = 720, height = 1280, framerate = 30, bitrate = 3584000, keyframeInterval = 3, sampleRate = 48000, channelCount = 2

Output Video : Around 9MB - 12MB duration 30s and quality little bit low.

We need around 5MB - 7MB duration 30s.so sharing video in social media its Better!!

 

Here I'm attaching links for videos those shows whats my issue
1)The first video shows video recording when I press share and after the clear parental gate (using the old plugin)
it's working exactly how I want

2)) The Second video shows video recording when I press share and after the clear parental gate (using Natcorder 1.7.0)
Its getting struck while recoding

both giving same mp4 file but while doing the record with new plugin its looks like app got struck(The same thing happened with macOS Catalina also)

 

Not anymore?

 

I'm still a bit confused. Are you still recording with NatCorder when you press share? If so, then recording is expected to fail if the app goes out of focus. If your issue is that the app is paused when the sharing UI pops up, then this has nothing to do with NatCorder.

 

Increasing quality means increasing the file size. You practically can't increase one and decrease the other. You are already modding the bitrate, so there isn't anything else to modify.

 

Check out the open PR.

 

I'm trying to output a GIF usign GIFRecorder.
I made an animation that spins an object 360° and makes the gif. It is supposed to loop seamlessly, but there is a pause at the loop point. If I edit the gif in another program, the delay between frames is constant but the last frame has a longer delay. If I manually edit it to be equal to the other frames, and make a new GIF, the loop is perfect.

Is there something I can change to obtain it directly in NatCorder? Or is it a bug?

Thank you

 

You have to make sure that you are committing frames at exact intervals, such that your 360 degree rotation can be factored into the frame difference. It likely isn't a NatCorder bug because NatCorder never drops frames; every frame you commit will make it to the final recording.

 

There is no frame dropping.

Frames are committed with an fixed increment on clock, not based on their time. BTW, if you inspect the gif that is output, all frames are equally spaced, but the last one has a longer delay factored in. The delay property of last frame only has a role when you loop the gif. Maybe its delay is set somehow at a fixed value in NatCorder because, well, there is no subsequent frame on which to calculate the delay?

 

The timestamps are actually ignored for the GIF recorder. The frame duration is what is used to space out all frames that are committed. What platform does this issue happen on?

 

Windows.

I opened the gif in a hex editor. The application extension block seems to have a mistake on the first frame.
The first frame has:
21 F9 04 01 00 00 00
all the others have
21 F9 04 00 03 00 00

I highlighted in Bold the delay time of the frame.
Changing the first to
21 F9 04 01 03 00 00
The loop becomes smooth.

You may want to change this in NatCorder.

 

Thanks for the detailed info. I'll check it out.

 

Hi @Lanre im having a headache with a problem with NatCorder... today it was working correctly, then nothing happened and it suddenly stop working, i mean Editor crash every time i want to start recording a video, but i didn´t make any change to the code for recording, the only thing i made was update cinemachine to the last version, but i even tried removing it, and the problem persist, with any camera. And because of the Editor crash... i don´t event know where to start to find the problem. After countless restarts, i´ve tried the examples, and the same problem, updated to the last version... and the problem persist... how can i even know what is happening without a log?