NatCorder - Video Recording API

The S3 is a pretty old phone, so old in fact that it shouldn't be able to run NatCorder (the max update for the S3 is Android 4.4, and NatCorder requires Android 6+). In any case, the device is running out of memory. The next update will improve memory stability.

 

Need your help. I use ARCore, Netcorder, NatShare in callback OnRecord i want to get NatShare.GetThumbnail(Path); When i found mark and instantiate model, i started record after some time i stoped record and have error. All pemision are granted

 

One more problem. Video resolution 1440*2560. But so bad quality...

 

Thank you for your reply. My bad, the device is not the Samsung S3 but the Samsung Tab S3. Nonetheless, I was able to solve the crash, by limiting the number of times "CommitFrame" is called to no more than the video frame rate:

Code (CSharp):

1.  
2. protected virtual float LastFrameRecordedTimeInSeconds { get; set; }
3. protected virtual float RecordingTimeIntervalSeconds { get; set; } = 1/30f;
4.  
5. void LateUpdate ()
6. {
7.     if (IsRecording)
8.     {
9.         if (Time.unscaledTime > LastFrameRecordedTimeInSeconds + RecordingTimeIntervalSeconds)
10.         {
11.             var renderFrame = MediaRecorder.AcquireFrame();
12.             Graphics.Blit(WebCamTexture, renderFrame);
13.             MediaRecorder.CommitFrame(renderFrame, RecordingClock.Timestamp);
14.  
15.             LastFrameRecordedTimeInSeconds = Time.unscaledTime;
16.         }
17.     }
18. }
19.  

1920x1080 and 1280x720 at 30fps now record fine, no crash at all

 

This means that for whatever reason, Android can't retrieve a frame for the video. Check the NatShare source for how this happens.

Increase your recording bitrate.

 

Latest NatCorder issues -
The type or namespace name `Docs' could not be found. Are you missing an assembly reference?

 

What is a new way to check if natcorder is recording? .IsRecording no longer seems to work?

 

m4p recorder != null

 

I use default int videoBitrate = (int)(960 * 540 * 11.4f). how much increase?

 

Hey Lanre!
Loving everything - been using it for a few months. I'm building an easy way to record clips into one video. One thing I've noticed is that when I pause - record - pause - record - on ios there is a .5 - 1 second long gap at the beginning of the resume where audio is recorded (from NatMic) but the same video frame is being used. This results in a video that looks like it has a "hitch" in it during the edit points. My pause and resume code is:

Code (CSharp):

1.         public void PauseRecording()
2.         {
3.             _clock.Pause();
4.             _cameraInput.Dispose();
5.             _audService.StopRecording();
6.         }
7.  
8.         public void ResumeRecording()
9.         {
10.             _clock.Resume();
11.             _cameraInput = new CameraInput(_videoRecorder, _clock, App.instance.MainCamera);
12.             _audService.StartRecording();
13.         }

Am I doing anything incorrectly? Is there any way to fix it?

My hunch is that the clock and audio resuming is instant, whereas the CameraInput has a short warmup time. This means that during the warmup time - the previous frame is being written to the mp4 along with fresh audio - resulting in the strange hitching I'm seeing.

If no good workaround - my other thought is to save each "resume" is a different clip and stitching them together at the end - so I'm not totally dead in the water if there's no fix.

Thanks,
-Chris

 

You shouldn't have any code that uses the Docs namespace. It is an internal namespace.

When you create a recorder, you are recording. And when you dispose of it, you are no longer recording. It's a simplified paradigm.

 

You can follow the YouTube bitrate guidelines.

 

What is your `_audService` doing? One thing to check is that you are using the same recording clock when committing video and audio frames (this way pausing the clock will be reflected across both video and audio tracks). The hitch you see is characteristic of one track having continuous time while another track is paused.

 

Thanks for the quick reply! I'll answer your question (though it may not be relevant - more info below).

Starting recording (included magic numbers for clarity):

Code (CSharp):

1.          
2.         public void StartRecording()
3.         {
4.             int MAX_VIDEO_RES = 1920;
5.             int samplerate = 44100;
6.             int channelcount = 2;
7.             float width = App.instance.MainCamera.scaledPixelWidth;
8.             float height = App.instance.MainCamera.scaledPixelHeight;
9.             float ratio = width / height;
10.             int vidwidth = ratio > 1 ? MAX_VIDEO_RES : (int)(MAX_VIDEO_RES * ratio);
11.             int vidheight = ratio > 1 ? (int)(MAX_VIDEO_RES * ratio) : MAX_VIDEO_RES;
12.             _clock = new RealtimeClock();
13.             _videoRecorder = new MP4Recorder(vidwidth, vidheight, 30, samplerate, channelcount, OnVideo);
14.             _cameraInput = new CameraInput(_videoRecorder, _clock, App.instance.MainCamera);
15.             _audService = new AudioService(samplerate, channelcount, _videoRecorder, _clock);
16.             _audService.StartRecording();
17.         }

The audio service:

Code (CSharp):

1.         public AudioService(int samplerate, int channelcount, MP4Recorder recorder, RealtimeClock clock)
2.         {
3.             _samplerate = samplerate;
4.             _channelcount = channelcount;
5.             videoRecorder = recorder;
6.             _clock = clock;
7.         }
8.  
9.         public void StartRecording()
10.         {
11.             audioDevice = AudioDevice.Devices[0];
12.             audioDevice.StartRecording(_samplerate, _channelcount, this);
13.         }
14.  
15.         public void ResumeRecording()
16.         {
17.             audioDevice.StartRecording(_samplerate, _channelcount, this);
18.         }
19.  
20.         public void StopRecording()
21.         {
22.             if (audioDevice.IsRecording)
23.             {
24.                 audioDevice.StopRecording();
25.             }
26.         }
27.  
28.         // Invoked by audio device with new audio data
29.         public void OnSampleBuffer(float[] sampleBuffer, int sampleRate, int channelCount, long timestamp)
30.         {
31.             // Send sample buffers directly to NatCorder for recording
32.             videoRecorder.CommitSamples(sampleBuffer, _clock.Timestamp);
33.  
34.         }
35.  

As I've started implementing the "stitch" method - it actually looks like the hitch is once I begin any recording (not just upon resume) and happens during the first .5 seconds of recording.

Best way to describe it is if you assume I am recording at 30 fps, NatCorder will save frames 1, 5, 17, 18, 19, 20, 21, etc......

It drops frames 2-4 and 6-16.

Potentially it is from a frame drop of the app itself - but the only thing I am doing is OnClick calling that StartRecording() function. Nothing else fancy going on except for the video recorder. This is on an iPhone X on iOS 12.? (updating to 12.3 just in case).

It could also potentially be triggering a garbage collection? I'm on Unity 2018.3.0f2

 

Ah I see. This is from NatMic. We've fixed this bug on iOS in the upcoming update. A better solution is to simply have your audio service keep the microphone running throughout the duration of the pause, and not send samples to the video recorder.

 

Cool. Looking forward to the update. I'll see if running the mic helps things in the meantime.

 

How i can make fade in, in end of video??

 

Hi. We purchased your asset and have next trouble. When building a project with a scene "Replay" example on an Android device (Samsung tab A2), after 10-15 seconds of recording, the application crashes. How can this problem be solved ?
IOS build with same scene is work fine.

 

NatCorder doesn't have any video editing capabilities like this.

 

What is your recording resolution? That is usually the first thing to look at. The device might not be able to handle recording at that size.

 

All settings is default, resolution 1280x720.

"The device might not be able to handle recording at that size."

Testable device is far from the worst that a potential user may have.
How to determine which resolution is fine for a particular device ?

 

Ah that's weird. Can you upload the full logs from logcat in a .txt file? A crash like this usually occurs when recording at resolutions higher than 1920x1080.

 

For reference I found this site had a useful formula for calculating the bitrate, similar to the one in NatCorder.

It should be noted that the values they give for K need to be multiplied by 1000 or 1024 for use in NatCorder, I guess using different units? This formula comes pretty close to the recommendations on YouTube if you use high = r = 1 or normal = r = 0.5.

However for my specific case i found the resulting bitrate was still too low, but then my project is pretty much a worst case situation for encoding as every frame is radically different from the previous one and there is a gradient in the background. I would assume for more usual cases the values from that formula would work well.

 

Been using Natcorder for a week or so and so far its been a really good experience. I would love to see more options in terms of encoding options for both MP4 and GIF, but its certainly usable. For example its odd that you can only have a maximum of 1 keyframe every second for MP4, though i'm unsure if that is a limitation of the specification or NatCorder? For GIF it would be nice to have the option for not dithering or maybe choosing different dither types.

Crash Bug
Anyway all was going well with NatCorder until today, when I tried out exporting to GIF and started have Unity hard crash when stopping the recordings. After a little bit of digging I discovered the cause was a bug in my code that meant I was calling StopRecording() multiple times. So I assume that calling cameraInput.Dispose() and/or gifRecorder.Dispose() was the culprit.

Obviously this was a case of my own bug, but it would be nice and I think expected if the plugin could offer some protection and ensure that whatever it is doing in its dispose method is not executed multiple times or if its already been disposed. Certainly in a case like this were a simple one line bug meant Unity would hard crash to the desktop with no crash log adding some protection from user mistakes would be beneficial.

 

I have error(( Device Asus Google Nexus 7 16GB (2nd gen). Android 6.0.1

Code (CSharp):

1.  
2. private void StartRecord()
3.         {
4.             recordingClock = new RealtimeClock();
5.             videoRecorder = new MP4Recorder(
6.                 Screen.width,
7.                 Screen.height,
8.                 30,
9.                 0,
10.                 0,
11.                 OnRecording
12.             );
13.             cameraInput = new CameraInput(videoRecorder,recordingClock,Camera.main);
14.             _recording = true;
15.             SetButtonrecordState();
16.         }
17.  

Code (CSharp):

1.  
2. private void StopRecord()
3.         {
4.             cameraInput.Dispose();
5.             videoRecorder.Dispose();
6.             _recording = false;
7.             SetButtonrecordState();
8.         }
9.  

 

1. Log with "unity" filter
2. Another try after app is run and crashed without filter

 

The MP4Recorder uses bits per second.

This is a limitation of the Android platform, propagated up to the front-end API. All other platforms allow setting a keyframe interval in frames, whereas Android specifies it in seconds.

This is done on purpose. We use Floyd Steinberg dithering on all platforms as it has the best visual quality.

I disagree with this. The Dispose pattern on recorders is the closest we come to RAII. You create a recorder, use it, and dispose it. When you dispose an object, there is an expectation that you also dereference the object, as you can no longer use it. This is why the IDisposable interface in C# doesn't offer an `IsDisposed` boolean property (for NatCorder to protect against you calling Dispose twice, we would implicitly have to implement such a property, which goes against the paradigm). All you have to keep in mind is that when you dispose an IDisposable, dereference immediately after.

 

Take the full logs (no filtering) from logcat and upload it in a .txt file. The full logs will contain the exact reason for the crash.

 

does plugin supported 16:10 recording? Because i decrease aspect ration and crash dont happing again

 

Yes, but the device might not, and there isn't any way to know. This is a common problem on Android.

 

Interesting, any chance of getting keyframe by frames exposed for non-Android devices?

It may have the overall best quality, but not always. Its also odd that it seems to want to dither a perfect white background, is that normal? Maybe what I really need here is a way of indicating a static background color or something, I think there is a means of doing that in GIF specifications? Something for the future perhaps?

Well as I said, i made a simple error, in copy and pasting from my MP4 recorder code to a GIF recorder code I omitted setting a flag to false so that StopRecording wouldn't be repeatedly called. I guess really what is needed is to have both cameraInput and gifRecorder set to null and use a condition null check before disposing. In whcih case it would have been nice to have had that done in the examples in the package since thats were the original code came from.

 

Google Pixel 3a
Android version 9
API Level 24
ReplayCam script

05-21 17:48:28.497 29608 29624 E Unity   : AndroidJavaException: java.lang.ClassNotFoundException: com.yusufolokoba.natcorder.MP4Recorder



05-21 17:48:28.497 29608 29624 E Unity   : java.lang.ClassNotFoundException: com.yusufolokoba.natcorder.MP4Recorder



05-21 17:48:28.497 29608 29624 E Unity   : at java.lang.Class.classForName(Native Method)



05-21 17:48:28.497 29608 29624 E Unity   : at java.lang.Class.forName(Class.java:453)



05-21 17:48:28.497 29608 29624 E Unity   : at java.lang.Class.forName(Class.java:378)



05-21 17:48:28.497 29608 29624 E Unity   : at com.unity3d.player.UnityPlayer.nativeRender(Native Method)



05-21 17:48:28.497 29608 29624 E Unity   : at com.unity3d.player.UnityPlayer.c(Unknown Source:0)



05-21 17:48:28.497 29608 29624 E Unity   : at com.unity3d.player.UnityPlayer$e$1.handleMessage(Unknown Source:88)



05-21 17:48:28.497 29608 29624 E Unity   : at android.os.Handler.dispatchMessage(Handler.java:102)



05-21 17:48:28.497 29608 29624 E Unity   : at android.os.Looper.loop(Looper.java:193)



05-21 17:48:28.497 29608 29624 E Unity   : at com.unity3d.player.UnityPlayer$e.run(Unknown Source:20)



05-21 17:48:28.497 29608 29624 E Unity   : Caused by: java.lang.ClassNotFoundException: Didn't find class "com.yusufolokoba.natcorder.MP4Recorder" on path: DexPathList[[zip file "/data/app/com.kevaid.doodlecam-xlTR7LbvX17qMNMqTc_tzw==/base.apk"],nativeLibraryDirectories=[/data/app/com.kevaid.doodlecam-xlTR7LbvX17qMNMqTc_tzw==/lib/arm, /data/app/com.kevaid.doodlecam-xlTR7LbvX17qMNMqTc_tzw==/base.apk!/lib/arme

 

hi，I use this asset to recorder mp4 files, it's right in iOS, but it looks darker in Android . In my upload files，I write RenderTexture to png Files，I found color is right， but mp4 color always darker, can you help me?

 

Unfortunately this isn't possible. When crafting a cross-platform API, you're usually restricted to the greatest common denominator of all supported platforms. Any deviation would result in an awkward API.

Hm what platform does this occur on? Sounds like a bug.

You shouldn't keep a flag of whether you are recording or not. One very easy way of avoiding gotchas in code is to keep as little canonical information as possible, and derive as much as you can from that set. If you have some `videoRecorder` member variable that is null, you can always infer that you are not recording (the Dispose paradigm lets you make this assertion, provided you follow the paradigm). And similarly, if the `videoRecorder` variable is not null, then it must be the case that you are recording. The rest of the logic will flow from this.

The examples don't do any checks because some of the preconditions are guaranteed by the UI, not by code. In all our examples, we use our custom recording button. When the button's touch-down event is triggered, recording is started; and when the touch-up event is triggered, recording is stopped. It is impossible for the touch up event to be triggered without a preceding touch down event, so our example code can always bank on this precondition. It makes life easy

 

Hm this is weird. Try deleting NatCorder in your project and reimporting it from the Asset Store. Once you do that, upload the logs from logcat in a .txt file.

 

You must be using linear rendering. NatCorder only supports linear rendering on macOS and Windows; it doesn't support linear rendering on any other platform (although it works on iOS, as you have seen).

 

That was on Windows 10 - though i'll need to double check it was just a pure white background camera and that no other gameObjects were affecting the screen. I'll let you know if I confirm it, but only got a week left on client project before install date so it will be low priory.

Great point and i'm kicking myself for not doing this in the first place. That's what you get for being under pressure and crunching with a week before project install. You are correct it does simplify and add robustness to the code, however one aspect I'm unsure about is the safety of setting videoRecorder to null once its disposed.

One aspect is at the back of my mind is that in Unity, in some specific cases null != null - e.g. Unity Blog. I guess I could just not have videoRecorder as MonoBehaviour fields and that should address this concern.

However the larger issue concerns what happens when you perform StopRecording, where you are expected to do videoRecorder.dispose, but I'm unsure if its safe to immediately set videoRecorder to null? My understanding is that dispose will result in the video being finalized and written to disk, which can take several frames/ms to perform, prior to then issuing its callback mechanism back to Unity. My concern is setting videoRecorder to null might kill that process before it has a chance to finish?

Would the safest approach be to set videoRecorder to null in the callback, but then you are left with the case of videoRecorder not being null, but disposed, which makes validation awkward again.

Any thoughts on this?

 

Yes，We use linear color space. Is it because of the need for openGL3.0 on Android and only Metal in iOS ? Can NatCorder supports linear on Android and iOS ?

 

Is there a plan to support Linear on Android and iOS ?

 

Ah, this one is out of our control then. On all platforms expect Android, we use platform-provided quantization and dithering API's. On Android, we use our own implementation.

Why? This is the proper usage, and this is what the examples do.

This is pretty interesting (I had no clue), but it only applies to types that derive from UnityEngine.Object (looks like they just overrode the `==` operator). It doesn't affect NatCorder types, because none of our public classes derive from UnityEngine.Object/MonoBehaviour.

Yes, it is safe to immediately dereference the recorder after setting it to null. The recorder doesn't get garbage collected until the recording callback has been invoked from the native layer, so there's nothing to worry about.

Call Dispose then immediately dereference the recorder. NatCorder will handle the rest

 

We don't support linear rendering on mobile platforms for technical reasons, especially because of Android. We've seen that linear rendering is unstable on Android, so we can't really guarantee proper functionality with it.

 

Hi Lanre, thanks for a great plugin, it's proved very useful. I have a question which isn't directly related to NatCorder, but involves it and I was hoping you might be able to help with.

I'm working on the project at the moment where one part involves the user choosing some options, and then a video is created based on the options they have chosen (think something like choosing an animated character, then choosing an environment, and then we generate a video of those characters animated in that environment). I can do this using Natcorder recording at a normal speed, however I was wondering if you knew if there was anyway it would be possible to record the scene faster than 1:1 playback time, especially on mobile?

For example, say I'm recording a 1 minute video of the characters in the environment, I would like to be able to record a 30 fps video with Time.timeScale = 4 and the Natcorder Clock running 4 times the speed, so that the actual recording only takes 15 seconds but the output video is actually 1 minute. I have managed to get this to work to an extent on desktop (where I measure the time taken to render the previous frame and use that to update both Time.timeScale and a multiplier on the clock), and can successfully record at roughly 2-3x speed (i.e. record a 30 second video in 10-15 seconds). However whenever I run it on mobile, the maximum speed I can render frames at seems to be locked at 30 frames per second. I found this thread which states that rendering on iOS is tied to VSync, which makes sense when rendering to the screen, but this camera only ever renders to the Natcorder RenderTexture so VSync and screen tearing shouldn't even happen. Unfortunately all my searching has just taken me to various things on optimising your game for mobile, which isn't really what I need as the scenes are very simple and the limitation is really Unity not animating and rendering the frames fast enough.

Any ideas?

Thanks

 

@Lanre

Hi there! I’m having a rather critical issue with NatCorder on Android.

Below is a simplified version of my script. It attempts to make an 8 second recording from the camera and audiolistener and then plays it back. This works on Mac and iOS, but on Android the app freezes and then crashes every time, either right when recording begins or a second or so in. I’ve tried a number of configurations and have not been able to make recording work on Android.

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using NatCorder.Inputs;
5. using NatCorder;
6. using NatCorder.Clocks;
7.  
8. #if UNITY_EDITOR
9. using UnityEditor;
10. #endif
11.  
12. public class RecordingScript : MonoBehaviour
13. {
14.   AudioListener myAudioListener;
15.   Camera myCamera;
16.  
17.   private MP4Recorder videoRecorder;
18.   private IClock clock;
19.   bool done;
20.   string filename;
21.  
22.   const int VIDEO_WIDTH = 1280;
23.   const int VIDEO_HEIGHT = 720;
24.  
25.   // Start is called before the first frame update
26.   void Start()
27.   {
28.     myAudioListener = GetComponent<AudioListener>();
29.     myCamera = GetComponent<Camera>();
30.  
31.     StartCoroutine(RecordRoutine());
32.   }
33.  
34.   IEnumerator RecordRoutine()
35.   {
36.     yield return new WaitForSeconds(1f);
37.  
38.     Debug.Log("Beginning recording.");
39.  
40.     clock = new RealtimeClock();
41.     videoRecorder = new MP4Recorder(VIDEO_WIDTH, VIDEO_HEIGHT, 30, AudioSettings.outputSampleRate, (int)AudioSettings.speakerMode, OnRecording);
42.     AudioInput audioInput = new AudioInput(videoRecorder, clock, myAudioListener);
43.     CameraInput cameraInput = new CameraInput(videoRecorder, clock, myCamera);
44.  
45.     yield return new WaitForSeconds(8f);
46.  
47.     Debug.Log("Ending recording.");
48.  
49.     audioInput.Dispose();
50.     cameraInput.Dispose();
51.  
52.     // Stop recording
53.     videoRecorder.Dispose();
54.  
55.     while (string.IsNullOrEmpty(filename))
56.     {
57.       yield return null;
58.     }
59.   }
60.  
61.   void OnRecording(string path)
62.   {
63.     Debug.Log("Saved recording to: " + path);
64.     // Playback the video
65. #if UNITY_EDITOR
66.     EditorUtility.OpenWithDefaultApp(path);
67. #elif UNITY_IOS
68.         Handheld.PlayFullScreenMovie("file://" + path);
69. #elif UNITY_ANDROID
70.                 Handheld.PlayFullScreenMovie(path);
71. #endif
72.   }
73. }
74.  

Since almost no configuration is working for me on Android I have to wonder if some critical change exists in Unity or Android which breaks NatCorder?

If it helps I can send you my complete demo project to reproduce the issue.

Also attached is the log output before the freeze. I don’t see any information about the error itself there, but it may mean more to you.


Unity 2019.1.3f1
NatCorder 1.5.1
Android version: 9
Test device: Pixel 3

 

NatCorder is capable of offline recording (basically, recording in a for-loop), but this is really only intended for standalone platforms where there's enough memory to hold a bunch of frames in RAM till the encoder can eat them all up. You can try it, but you have to be especially cautious of your memory consumption. The key you need is to manually 'push' your game forward in time, then render to NatCorder at the end of each timestep. The Time.timeScale trick won't work because you are increasing the speed of the game without increasing how often each of those frames gets rendered to the screen (this will be locked to Vsync on mobile platforms). If you can tie all your game movements to a simple `Game.StepForward()` function, your recording code will look something like this:

Code (CSharp):

1. for (;;) {
2.     // Step the game forward in time
3.     game.StepForward();
4.     // Record the current frame
5.     // Make sure to use a FixedIntervalClock for this (uses like this is what it was made for)
6.     NatCorder.Acquire + Camera.Render + NatCorder.Commit
7. }

 

Can you upload the full, unfiltered logs from logcat in a .txt attachment?

 

Thanks for the information. Since I'm mainly targeting mobile in this case I would guess it's probably not possible - I did follow a similar pattern on one attempt and on iOS it very quickly used up all the available memory and the app crashed. I think it's probably best to just work the UX around this happening as a background task. Cheers!

 

Hey @Lanre, we're trying to use NatCorder on LWRP (2018.3.7). Everything seems to work when using the ReplayCam example on device, but when trying to integrate into an ARFoundation project with LWRP, every recorded video ends up as black for the duration of the recording. Can you offer any insight?

Thanks!

 

I think I've gotten a report of an issue like this in the past. Apparently it had to do with how ARFoundation handles rendering in Unity. Check this out.

 

Here you go.

 

Looks like you app is using Vulkan. NatCorder only supports OpenGL ES3.