NatCorder - Video Recording API

You could consider writing a native plugin to perform the concatenation. I don't have any plans for building an FFmpeg wrapper.

 

Yes, I wrote NatCorder. Why do you ask?

 

Did you try out the iOS library I shared? If so, can you confirm that you are no longer losing frames?

 

When I try to record video with audio, I get an invalid mp4 file (that's 80kb long). If I set audioChannels to zero, it records just fine. Here is my code:

Code (CSharp):

1.         mediaRecorder = new MP4Recorder(filename, Screen.width, Screen.height, 30, 44100, 2, recordingEnded, 5909760*2,3);
2.         // Create a camera input to record the main camera
3.         cameraInput = new CameraInput(mediaRecorder, clock, MainCamera);
4.         // Create an audio input to record the scene's AudioListener
5.         audioInput = new AudioInput(mediaRecorder, clock, Audio);

Any ideas?

 

What device is this happening on? What is your `Audio` variable? Also, it looks like you have modified MP4Recorder (I highly, highly recommend against this). I also recommend never recording at screen resolution since screen resolutions can be pretty high, and you shouldn't be hard coding the sample rate and channel count when recording from Unity. Get the values from Unity itself (see AudioSettings class).

 

Im using ReplayCam example.

I record a video using the Android camera, the path gets saved. Then i record a second video, that path gets saved.

I want to append the second video to the first video. So I decide to start recording from a texture, play the first video to a texture, then play the second video to the texture, then stop recording.

This should record both videos into one video, appending the second video to the end of the first.

But when i stop recording it crashes. Do you have any idea what im doing wrong?

Here is my code below. The code before this is the same as came with the package.

After it records the second video, i call the PlayAndRecord(firstPath,secondPath) below.

It goes thru that whole function and then crashes after it leaves StopWholeRecord, you can see from the crash log below at the bottom.

Thanks

Code (CSharp):

1.  
2.     public IEnumerator PlayAndRecord(string path1,string path2)
3.         {
4.             Debug.Log("PLAY AND RECORD:v1="+path1 + " v2="+path2);
5.             yield return new WaitForEndOfFrame();
6.             yield return new WaitForSeconds(0.5f);
7.             yield return new WaitForEndOfFrame();
8.  
9.             StartWholeRecord();                  // START RECORDING
10.             yield return new WaitForEndOfFrame();
11.             yield return new WaitForSeconds(0.5f);
12.             yield return new WaitForEndOfFrame();
13.             Debug.Log("RECORD STARTED");
14.  
15.             PlayVideo(path1);                  // PLAY BACK THE FIRST VIDEO
16.             yield return new WaitForEndOfFrame();
17.             yield return new WaitForSeconds(0.5f);
18.             yield return new WaitForEndOfFrame();
19.             Debug.Log("FIRST VIDEO DONE");
20.  
21.             PlayVideo(path2);              // PLAY BACK THE SECOND VIDEO
22.             yield return new WaitForEndOfFrame();
23.             yield return new WaitForSeconds(0.5f);
24.             yield return new WaitForEndOfFrame();
25.             Debug.Log("SECOND VIDEO DONE");
26.  
27.             StopWholeRecord();      // STOP RECORDING
28.         }
29.  
30.         public void StartWholeRecord()
31.         {
32.             Debug.Log("START WHOLE VIDEO RECORD");
33.             recordingClock = new RealtimeClock();
34.             videoRecorder = new MP4Recorder(
35.                 videoWidth,
36.                 videoHeight,
37.                 30,
38.                 recordMicrophone ? AudioSettings.outputSampleRate : 0,
39.                 recordMicrophone ? (int)AudioSettings.speakerMode : 0,
40.                 OnRenderedWholeVideo
41.             );
42.             // Create recording inputs
43.             renderInput = new RenderTextureInput(videoRecorder, recordingClock);
44.             if (recordMicrophone) {
45.                 StartMicrophone();
46.                 audioInput = new AudioInput(videoRecorder, recordingClock, microphoneSource, true);
47.             }
48.         }
49.  
50.         public void PlayVideo(string path)
51.         {
52.             Debug.Log("PLAY VIDEO IN:" + path);
53. #if UNITY_EDITOR
54.             EditorUtility.OpenWithDefaultApp(path);
55. #elif UNITY_IOS
56.             Handheld.PlayFullScreenMovie("file://" + path);
57. #elif UNITY_ANDROID
58.             Handheld.PlayFullScreenMovie(path);
59. #endif
60.             Debug.Log("PLAY VIDEO OUT:" + path);
61.         }
62.  
63.         public void StopWholeRecord()
64.         {
65.             Debug.Log("STOP WHOLE VIDEO RECORD IN");
66.             // Stop the recording inputs
67.             if (recordMicrophone) {
68.                 Debug.Log("stopping mic...");
69.                 StopMicrophone();
70.                 Debug.Log("...mic stopped");
71.                 audioInput.Dispose();
72.                 Debug.Log("...audio disposed");
73.             }
74.             Debug.Log("disposing renderer...");
75.             renderInput.Dispose();
76.             Debug.Log("...renderer disposed");
77.             // Stop recording
78.             videoRecorder.Dispose();
79.             Debug.Log("STOP WHOLE VIDEO RECORD OUT");
80.         }
81.  
82.         private void OnRenderedWholeVideo(string path)
83.         {
84.             Debug.Log("WHOLE VIDEO RENDERED:" + path);
85.             lastPath = path;
86.             currentPath = null;
87.         }

(Filename: ./Runtime/Export/Debug.bindings.h Line: 45)
08-25 20:17:20.652 15045-15060/? I/Unity: STOP WHOLE VIDEO RECORD OUT

(Filename: ./Runtime/Export/Debug.bindings.h Line: 45)
08-25 20:17:20.661 15045-15251/? V/Unity: NatRender: Released GLRenderContext
08-25 20:17:20.661 15045-15250/? V/Unity: NatCorder: MP4 video encoder changed output format: {csd-1=java.nio.HeapByteBuffer[pos=0 lim=8 cap=8], mime=video/avc, frame-rate=30, width=720, height=1280, color-standard=1, color-range=2, bitrate=5909760, csd-0=java.nio.HeapByteBuffer[pos=0 lim=21 cap=21], color-transfer=3, max-bitrate=5909760}
08-25 20:17:20.669 15045-15250/? V/Unity: NatCorder: MP4 video encoder encountered EOS
08-25 20:17:20.704 15045-15250/? E/Unity: NatCorder Error: Failed to stop MP4Recorder
java.lang.IllegalStateException: Failed to stop the muxer
at android.media.MediaMuxer.nativeStop(Native Method)
at android.media.MediaMuxer.stop(MediaMuxer.java:254)
at com.yusufolokoba.natcorder.MP4Recorder$10.run(MP4Recorder.java:287)
at java.lang.Thread.run(Thread.java:761)

 

Greetings, there are two topics that I would like to ask about.

The first is regarding if there is any update with the audio problems experienced when recording with audio resulting in corrupted files, alongside with the editor crashing?

My other concern is better understood with the attached image. As you can see, there is a green line appearing in the bottom when recording, I've made several tests and can't find out what is causing it. Do you have any insight regarding this issue?

Thanks for your help.

 

You can't use Handheld.PlayFullscreenMovie to playback the videos because Unity Engine gets suspended, and uses a native view to play the video. To be able to record with NatCorder, you have to playback the videos in Unity, using a VideoPlayer or something similar. The error you get is because you start and stop recording without ever recording any frames (since Unity is paused while Handheld.PlayFullscreenMovie plays the video).

 

What platform are you seeing this on?

 

can i record ar 720*1280 or 1080*1920??

 

Yes. The orientation doesn't matter, so you can record at wxh or hxw.

 

1. Audio holds the (only) audioListener in the scene
2. I just changed how MP4Recorder handles the filename
3. I tried using AudioSettings , same results (80kb file)
4. I tried reducing resolution, same results

 

I attached a Unity VideoPlayer to the preview GameObject in the ReplayCam demo, and its working now. Thanks!

 

What device is this happening on?

 

Any news about the black frames we get on the Samsung S7 Edge when using ARCore and not Vuforia?

Thanks,

 

i use simple code and where i start recording i have 1-1.5sec app freezing. Device - Samsung A30

Code (CSharp):

1.  
2. clock = new RealtimeClock();
3. videoRecorder = new MP4Recorder(1080,1920,30,0,0,OnRecordStop);
4. cameraInput = new CameraInput(videoRecorder,clock,Camera.main);
5. audioInput = new AudioInput(videoRecorder,clock,Camera.main.GetComponent<AudioListener>());
6.  

 

Can you email me? I'd like to send an updated Android library with a potential fix.

 

Can you upload the full, unfiltered logs from logcat in a .txt file?

EDIT: Also you are creating an audio input even though you have not specified a sample rate or channel count.

 

Hi,
I'm using NatCorder and NatShare. Both seem to work well. Could you tell me if you know how to compress the recorded gifs ( whithout modifying dimensions or frame skip ?)
Thanks a lot

 

I don't think this is possible. We don't offer any 'compression' parameters.

 

Hi,

We are using Natcorder in an app that is using EasyAR SDK for basic image target AR. The problem is that on iOS, using Metal, Natcorder is capturing the video feed upside down, but the 3D gameobjects that are displayed on the image target are displaying correctly. Using OpenGLES3 doesn't show this problem.
We think that it is related with EasyAR using CommandBuffer BeforeForwardOpaque on the Unity Camera to render the device camera feed and probably is also related to platform-specific rendering differences mentioned in Unity manual.
Any idea how can we fix it?

 

This is an EasyAR issue. NatCorder doesn't have an inversion issue (if it did, it would also occur in the included examples). Also, NatCorder doesn't support OpenGL ES3, are you using the latest version?

CameraInput renders the camera to a RenderTexture that gets committed (the normal off-screen rendering technique in Unity). So it seems that EasyAR could be incorrectly inverting the view. I recommend reaching out to their support.

 

I'm using Natcorder for an internal company tool and some of my OSX users are experiencing flickering in the recorded videos. This manifests as rectangular patches of blue or yellow that appear on single frames.

I'm using the MP4 Recorder with the realtime clock and the easy Camera Input and Audio Input setup similar to your examples.

Code (CSharp):

1.      
2.       public void StartRecording (string fileName = "ExportRecording")
3.       {
4.           _filename = fileName;
5.           // Create a recording clock
6.           _recordingClock = new RealtimeClock();
7.           // Start recording
8.           _mediaRecorder = new MP4Recorder(_videoWidth,_videoHeight, 30, AudioSettings.outputSampleRate, (int) AudioSettings.speakerMode, OnRecordingComplete, 5909760);
9.           // Create a camera input to record the main camera
10.           _cameraInput = new CameraInput(_mediaRecorder, _recordingClock, _recordingCamera);
11.           // Create an audio input to record the scene's AudioListener
12.           _audioInput = new AudioInput(_mediaRecorder, _recordingClock, _recordingListener);
13.       }
14.      
15.       public void StopRecording () {
16.           // Destroy the recording inputs
17.           _cameraInput.Dispose();
18.           _audioInput.Dispose();
19.           // Stop recording
20.           _mediaRecorder.Dispose();
21.           _mediaRecorder = null;
22.       }
23.  

 

Can you upload one such frame/video?

 

I will have to check with management. I will probably not be able to post publicly, but I could potentially email you.

 

Windows

What about "there is any update with the audio problems experienced when recording with audio resulting in corrupted files, alongside with the editor crashing"?

 

Ok, I'll make a test today.

 

how i can check if my device support 1920*1080 recording?

 

Hello.

I am using NatCorder 1.6.1 in Unity 2018.3.5f1.
Whats the support status for Multiple Cameras?

I was successfully recording multiple cameras with a script found on github from the author, which worked fine on old NatCorder, but now it's not compatible.

CameraInput constructor now directly support an array of cameras. But I have an ISSUE: SECOND CAMERA FLICKERS in the final video. It seems like there is a racing condition when saving the camera's frames. The cameras depth property should be used to guarantee the rendering order.

We are in a hurry to update our app. We were forced to update NatCorder due to the Android 64bits error.
Thanks,
Pablo

 

I have emailed a video to you at the email address listed on the Natcorder Asset Store page

 

This is not true when using a Vuforia ARCamera rig. When using Vuforia, the recorded video is squashed or stretched to match the device aspect ratio. Just try it with your ReplayCam but swap the MainCamera for a Vuforia ARCamera. Vuforia probably adjusts the projectionMatrix and not the FOV. if you're doing some adjustment using FOV that's probably the cause.

Also, the editor freezing after a second play is really starting to slow me up. Any idea what is causing this?

 

Hi,
started using your asset today. I ran into a problem on my Mac that seems to be Metal related. When doing a recording from inside the editor using your test scenes the recordings do work, but the colours are odd, like it is missing a channel and have some weird purple filter. When disabling Metal and using OpenGL 4 the colours turn out to be correct. Any ideas why? Possible setting? This does not seem to happen with the WebCam example, but with the other 2 examples on Unity 2019.

Another thing I noticed is that I very often can only run the editor once, and do a successful recording. When I stop the play mode in the editor and start it again, the editor just hangs. This happens on 2019 and 2018 versions of Unity, on Metal and Open GL. If I don't do a recording, but just press play, stop, press play again, it always works. I can also record several clips after another, but once I stop, I can't restart again.

The builts of the test scenes on iOS and Android work fine. So that is great!

A minor thing, not directly related to your code, is when the WebCamTexture is created without adding a frame rate I get like 1 frame per second for the web cam view. This is fixed by adding a frame rate when creating the WebCamTexture, e.g.

Code (CSharp):

1. cameraTexture = new WebCamTexture(cameraName, 1280, 720, 30);

Thanks,
Luciano

 

@Lanre,

I'm having issues with video playback when recording on Pixel 2XL and 3XL. I'm using the Unity VideoPlayer Component to stream a recorded 720p video with no audio. The playback is choppy and stutters. I know the VideoPlayer is out of your scope, but have you tested recording playback using the Unity VideoPlayer? When I save the video to Gallery, it plays back fine from there. I have tried different framerates, resolutions, bitrates, and keyframeIntervals but nothing has made the stutter go away.

 

I happens both on my Desktop and Android device (S10). Unity version is 2019.1

 

I don't believe I said there was an update. I likely said I was working on an update that would address these issues. As for the green line issue, what resolution are you recording at? Artifacts like this are usually because of the encoder itself (I've come across a similar issue on Android).

 

On paper, any device capable of running NatCorder (according to our minimum requirements) is also capable of recording at 1920x1080@30FPS. As for runtime checking, it depends on things like how much CPU/GPU is available and how much memory is free--all things that we can't slap into a boolean variable.

 

So I solved the flicker very forcefully, seems like their is a race condition CommitFrame vs render of other cameras not pretty but here is my solution

Code (CSharp):

1.         private IEnumerator OnFrame () {
2.             var yielder = new WaitForEndOfFrame();
3.             for (;;) {
4.                 // Check frame index
5.                 yield return yielder;
6.                 var recordFrame = frameCount++ % (frameSkip + 1) == 0;
7.                 if (recordFrame) {
8.                     // Acquire frame
9.                     var framebuffer = RenderTexture.GetTemporary(frameDescriptor);
10.                     // Render every camera
11.                     for (var i = 0; i < cameras.Length; i++) {
12.                         var prevActive = RenderTexture.active;
13.                         var prevTarget = cameras[i].targetTexture;
14.                         cameras[i].targetTexture = framebuffer;
15.                         cameras[i].Render();
16.                         cameras[i].targetTexture = prevTarget;
17.                         RenderTexture.active = prevActive;
18.                     }
19.                     // Commit frame
20.                     yield return yielder;
21.                     yield return yielder;
22.                     yield return yielder;
23.  
24.                     frameInput.CommitFrame(framebuffer);
25.                     RenderTexture.ReleaseTemporary(framebuffer);
26.                 }
27.             }
28.         }
29.  

 

The class you sent me works! No more black frames on the Samsung S7 Edge using ARCore. Thanks.

 

The MultiCameraInput example code and CameraInput do the exact same thing. If you're getting flickering, then check your depth order. Also note that you have to pass in the cameras to CameraInput in the correct order, from deepest depth to shallowest depth.

 

We aren't doing anything with the FOV. We render the camera to a RenderTexture, and Unity automatically sets up the camera's aspect based on this. A simple scheme is to fix either the width or height, then compute the other using a fixed aspect (the screen aspect for instance).

 

No clue yet. I'm not able to reproduce it deterministically on macOS, but am able to do so on Win10. I can't seem to isolate it, though it seems that when recording without using a RenderTexture (like the WebCam example), there might be a hang but it isn't indefinite; it lasts only a few seconds. I'm going to file a bug report with Unity as this one is out of my hands (nothing in my code should have any influence on how the Editor works).

 

That's really weird. Can you share a screenshot of the tint? This likely has to do with Unity's AsyncGPUReadback implementation. What version of Unity are you using?

A lot of devs are facing this issue, me included. I'm looking for what the problem might be.

Thanks! I've updated the code with this revision.

 

If it plays back properly on other video players, then you've pretty much isolated it to VideoPlayer. I recommend filing a bug report and trying AVPro. I'm also considering writing a video player API because VideoPlayer is quirky and AVPro is prohibitively expensive. Would anyone be interested in this?

 

Does the same thing happen with the ReplayCam example? If so, can you share your full code?

 

NatCorder1.6 Unity 2018.2.7f1 Android 6.0
At Android 6.0, the recorded mp4 file can't be opened, and copied to Win10, prompts wrong format when opening. Here are the mp4 file and log.

I find someone has report this bug, and I test with NatCorder1.61, app crashes when startRecording.

 

This happens in Unity 2019.2.2f1 on latest version of MacOS 10.14.6. When running inside the editor all colours are shown correctly. In Unity 2018 the colours are also fine, my guess it also defaults to Metal then, though not sure now.

 

I got your email. Upgrade to 1.6.1 on the Asset Store (delete and reimport it from the Asset Store). Also, NatCorder 1.6 requires Unity 2018.3, so you need to upgrade your version of Unity.

 

Looks like a Unity bug. I'll file a bug report. @Alexey thoughts on this?

 

Already communicated through email.. but my video flickering issue was resolved by removing the Metal renderer from the list of renderers for the build.

 

NatCorder 1.61 Unity 2018.3 Android6 crash