NatCorder - Video Recording API

Yup, just commit a raw pixel buffer.

 

We've pushed a silent update that fixes this issue. Delete and reimport NatCorder into your project. You can then re-enable multithreaded rendering and everything should work just fine.

 

Thank you for the fix, it works great on Android now but the recording is not working on the Editor anymore, it fires up the callback, opens up the native videoplayer, but throws an error, when I go back to unity and I click somewhere else, the Unity Editor freezes and becomes unresponsive, until I have to kill it with the Task Manager.

The video file that gets saved is also empty.

Unity Version: 2019.1.6f1

 

Hello.​

I've been experiencing the following issue with NatCorder:
I've set up a "Camera Recorder" to record videos in a similar way to how it's done in the "ReplayCam" example scene provided with the asset. Everything works as expected in the editor, but I get mixed results whenever I build for Android. The videos are saved and can be played properly when the application runs on an Xperia Z3+ (Android 7.1.1), but a video filled with black frames is recorded when it runs on a Samsung Galaxy S7 Edge. The audio is recorded properly, though, so I'm not sure of what the cause of the issue might be. I've only tested the application with these two devices, plus another Samsung phone in which the same problem occurs.

Here's the method that starts the recording on a button press:

Code (CSharp):

1.  
2. public void StartRecording(Camera appCamera, CameraPreviewer cameraPreviewer)
3.         {
4.             recordingClock = new RealtimeClock();
5.             videoRecorder = new MP4Recorder(
6.                 (int)cameraPreviewer.RequestedResolution.x,
7.                 (int)cameraPreviewer.RequestedResolution.y,
8.                 cameraPreviewer.RequestedFPS,
9.                 recordMicrophone ? AudioSettings.outputSampleRate : 0,
10.                 recordMicrophone ? (int)AudioSettings.speakerMode : 0,
11.                 OnRecordingFinish
12.             );
13.          
14.             cameraInput = new CameraInput(videoRecorder, recordingClock, appCamera);
15.             if (recordMicrophone)
16.             {
17.                 StartMicrophone();
18.                 audioInput = new AudioInput(videoRecorder, recordingClock, microphoneSource, true);
19.             }
20.         }

The "OnRecordingFinish" callback contains a call to the following method:

Code (CSharp):

1. void PlayPreviewVideo(string path)
2.         {
3.             #if UNITY_EDITOR
4.                 Application.OpenURL(path);
5.             #endif
6.             #if UNITY_ANDROID
7.                 Handheld.PlayFullScreenMovie(path);
8.             #endif
9.             #if UNITY_IOS
10.                 Handheld.PlayFullScreenMovie("file://" + path);
11.             #endif
12.         }

Both the preview and the saved video contain black frames on the mentioned Samsung devices.

I'd really appreciate if you could provide me some feedback. Thank you.

 

How do we convert an AR camera to your Camera Input? Camera.main is accepted as a Unitycam. Can you give an example?
using Wikitude;

private WikitudeCamera Cam;

 

What OS are you running on?

 

Disable Vulkan in Player Settings (Graphics API). If the video is black, then it's usually because you're using Vulkan which NatCorder doesn't support.

 

Apparently a WikitudeCamera correponds to a hardware camera. CameraInput works with game cameras.

 

Exactly my point. I'm not sure how someone would access the feed and get this to work with Augmented Reality Cameras. Not sure how to modify the Replay script to work as you mentioned others have done. I'm sure the SDKs don't allow access to the feed. The video recorded always shows as black for me since the SDK takes over the actual Unity Camera.

I may have found some direction.
https://support.wikitude.com/support/discussions/topics/5000085296

 

Yeah this is all specific to Wikitude and out of my domain. If you have a pixel buffer (byte[], Color32[]), commit it directly to the recorder. If you have a RenderTexture, use a RenderTextureInput to commit it.

 

Same issue on a Samsung S7 Edge. Not all frames are black and only OpenGLES3 is used. It's a project using ARCore. It used to work well on the same device with Vuforia.

 

Sorry I left that out, Windows 10.

 

Can you email me one such recording?

 

Hey, I just updated to 1.6.1 and all looks good in editor (windows), but in build (ARM64) on android (Samsung A70) it fails as soon as I start recording. Any ideas on how to fix this, I've included the code I use to call the recording and the log.

Code (CSharp):

1.     public void StartRecordingVideo()
2.     {
3.         if (!HasPermission())
4.             return;
5.  
6.         ShowRecordingUI();
7.  
8.         clock = new RealtimeClock();
9.         // Start recording
10.         mediaRecorder = new MP4Recorder(Screen.width, Screen.height, 10, 0, 0, OnVideoRecordingComplete);
11.         // Create a camera input to record the main camera
12.         cameraInput = new CameraInput(mediaRecorder, clock, mainCamera);
13. #if NATCORDER_15
14.         cameraInput.recordEveryNthFrame = 3;
15. #elif NATCORDER_16
16.         cameraInput.frameSkip = 3;
17. #endif
18.         WaitForPlaybackAndSave();
19.     }

 

Sent you an email with a video.

 

I don't seem to be seeing the issue on Android

 

Your recording resolution is odd (810x1743). I highly (highly) recommend sticking with a 'standard' resolution like 720x1280 because most Android devices will silently fail on some non-standard resolutions. And having an odd number in your resolution is a cardinal sin.

 

@Lanre

Using Unity 2019.1.14f, Natcorder 1.6.1, and NatMic 1.3.1: I've got one bug, one question, and a couple suggestions.

Bug: On Android, I can only get the latest version to record at Screen.width x Screen.height or 720 x1280. Any other value, throws:

Code (CSharp):

1. AndroidJavaException: android.media.MediaCodec$CodecException: Error 0xfffffc0e

I am trying to apply a downscale multiplier to the Screen resolution for my recordings. This worked in previous versions. The problem with using 720 x 1280 is that it doesn't match the aspect ratio of the device.

Question: Where's a working example of using Natcorder and NatMic that records both scene and microphone? I can't seem to find a current example. I've had this working in the past but the api has changed so much, those example are invalid. Without any clear documentation, I tried to use a MixerDevice to mix a VirtualDevice and the microphone from AudioDevice.GetDevices() (code below). However, this crashes on an Android S10. (Haven't tested on iOS).

Code (CSharp):

1. microphoneDevice = AudioDevice.GetDevices()[0];
2. unitysceneDevice = new VirtualDevice(FindObjectOfType<AudioListener>());
3. MixerDevice mixerDevice = new MixerDevice(microphoneDevice, unitysceneDevice);
4. mixerDevice.StartRecording(sampleRate, channelCount, this);

Suggestion 1: Lanre, you spend a lot of time supporting people on this forum and by email. We all really appreciate your commitment to supporting your product. I also appreciate the bare-bones simplicity of your code. That said, we need feature complete implementations so that we don't have to dissect your work just to do something simple. what might seem intuitive and clear to you, might not be to us. You would probably cut your support time in half if you provided some new up-to-date examples that are more feature-complete and better documented than ReplayCam. That would include examples for recording both scene and microphone, for recording at down-scaled resolutions, for recording and different bit-rates/sample rates, recording multiple cameras, and whatever other implementations people ask about the most.

Suggestion 2: There appears to be a number of known issues that aren't documented anywhere. For example, the Pixel 3 crashing when recording audio has been around for a long time but people keep asking why it's crashing on Pixel 3. Also, the Multithreading thing got me. Maybe you should add a known issues section in your change log.

Just trying to make your support life easier so you can continue to make us more great stuff.

 

Greetings, is there an update on the error where the editor crashed after recording? And also I'd like to know if there is a solution for the buggy videos when recording with audio caused corrupted files. Thanks for your help and apologies in advance if it was solved before, I was out of town and didn't have access to the forums.

 

Hello.
Thanks for the reply; I have just tried disabling Vulkan from the "Graphics API" options, and that did the trick for me.

 

This is not a bug in our implementation. When scaling your display resolution, you must guarantee that your resolution is even (width and height must not be odd numbers). In older versions of NatCorder, we used to automatically round up the width and height to the nearest even numbers, but we can't do so anymore now that recorders accept raw pixel data.

We've had a number of issues with the MixerDevice relating to timing and synchronization, so we had to downplay the MixerDevice for the time being. In any case, email me the crash logs from the S10 because it should not be crashing.

This is fair feedback which I appreciate. The issue I've consistently run into is that a lot of developers simply copy example code verbatim, without regard for the differences between the example and their intended goal. This is one of the reasons why the front-end API is extremely simple; so that it won't take rocket science to customize it to one's needs. I can't actually create example code that integrates another API since said API is likely sold separately. Recording at different resolutions and bitrates is as simple as changing a single parameter in the MP4Recorder constructor. That alone doesn't warrant an entirely different example scene. In fact, the ReplayCam example allows the user to set the resolution right in the editor. Similarly, recording multiple game cameras is as simple as passing more cameras into the CameraInput constructor.

Not necessarily. I believe this issue has been fixed implicitly because I haven't gotten any reports of it since the 1.6.1 update.

Likewise there was a silent update with the fix for this. Now, multithreaded rendering doesn't crash recording any longer.

I would totally do this except that known issues usually pop out after the update.

I very much appreciate the feedback. I'm hoping to fill any gaps in developers knowledge by simply answering questions like this one.

 

Not yet. I'll check out it this weekend.

 

which folder is this: "app's private documents directory" ?
i cant find it

i will also check out NatShare of course, but first lets see that it is copied the first time.

 

This is a perfect example of why we need more documentation, change logs, and examples. This is a breaking change that is undocumented and a best practice that is only mentioned here on a 40+ page forum post. The example should be updated to explain this issue and show a proper usage. Your examples shouldn't allow the user to put it in a broken state by putting in bad resolutions. Also, probably 95% of use cases are recording the whole screen so they need to adjust the resolution to match that aspect ratio. Your example should be something like...

Code (CSharp):

1. // NOTE: Most recording codecs tend to prefer a standard resolution like 720 x 1280.
2. // Also, resolutions MUST be composed of even number resolutions.
3. // The following defines a resolution clamped to 720 on the short side and
4. // and uses the screen aspect ratio to determine the long side length.
5.  
6. // Get the screen dimensions
7. int screenShortSide = Mathf.Min(Screen.width, Screen.height);
8. int screenLongSide = Mathf.Max(Screen.width, Screen.height);
9.  
10. // Define the video short side to a fixed value known for the best performance
11. int videoShortSide = 720;
12.  
13. // Clamp recorder long side to be scaled by the aspect ratio of the screen.
14. int videoLongSide = Mathf.FloorToInt((float)screenLongSide * (videoShortSide / (float)screenShortSide));
15.  
16. // Ensure that the resolution number is even
17. if (videoLongSide % 2 != 0) videoLongSide++;
18.  
19. // Define the final resolution for recording base on device orientation.
20. int videoWidth = Screen.width < Screen.height ? videoShortSide : videoLongSide;
21. int videoHeight = Screen.width < Screen.height ? videoLongSide : videoShortSide;

Does this mean there is currently no way to record both Scene audio and Microphone? That's a pretty big deal that should be clearly explained in a "Known issues" log. You should poll users on their use cases. I would guess most people that want to record microphone also want to record Scene audio.

All the more reason that your examples should be more error-proof, support more real-world use cases, and have clear documentation about what is happening in the code.

I can understand that. With the one exception of APIs you sell. If NatMic is primarily intended for use with Natcorder we need to understand how to use the two together in a meaningful way. I've looked at the NatMicCorder example several times but it doesn't answer all my questions. In particular, how to record both scene and mic audio.

Well, let me be the first to tell you the bug still exists in 1.6.1. If I set recordingAudio = true in the following example on Pixel 3, the app crashes as soon as the MP4Recorder is created. The crash will occur even if I don't commit any audio data.

Code (CSharp):

1. var sampleRate = recordingAudio ? AudioSettings.outputSampleRate : 0;
2. var channelCount = recordingAudio ? (int)AudioSettings.speakerMode : 0;

If you have to mention that a fix has been made, the update probably shouldn't be silent. Multithreaded Rendering is enabled by default in Unity so probably a sizable amount of users have the broken version or Natcorder installed and don't know there is a fix available.

And we very much appreciate the support you provide. I guess the issue I'm having is that this forum post is sooo long and covers so many issues (many of which are for older versions) that it's pretty much impossible to find a current description of the best practices that we need to get them most out of your product.

 

@Lanre I'm trying to use NatCorder for a specific use case but it's still failing in an odd way.

My 1 FPS animation works fine in editor, doesn't work on iOS. Instead, the FPS is anywhere from 0.25s to 0.33s per frame and the frame count is always 3 or 4 unless I add >15 frames. For example, if I add 9 frames, it's still only 3 frames in the final video.

Code (CSharp):

1. FixedIntervalClock f = new FixedIntervalClock(1);
2.                 MP4Recorder frameCollector = new MP4Recorder(
3.                     width,
4.                     height,
5.                     1,
6.                     0,
7.                     0,
8.                     (string path) => {
9.                         Debug.Log("saving video file");
10.                     }
11.                 );
12.  
13.                 foreach (Texture2D image in images) {
14.                     frameCollector.CommitFrame(image.GetPixels32(), f.Timestamp);
15.                     f.Tick();
16.                 }
17.  
18.                 frameCollector.Dispose();

 

That's the entire point. The directories are not meant to be publicly viewable. Sometimes, it is Application.persistentDataPath.

Not too sure what you mean here. NatCorder will save the video to the app's private folder. You can then use NatShare to 'copy' it to the camera roll.

 

This specific issue is more nuanced. For instance, creating a GIFRecorder with odd resolutions should work perfectly, and creating an MP4Recorder with an odd resolution might work on iOS and macOS, but not Android. So we really can't blankly say that people shouldn't record at resolutions that are odd. I'll add a note in the README that resolutions should preferably be even.

The choice to use a width and height explicitly was very much deliberate. Video sharing has very conventional resolutions, like 640x480, 1280x720, and 1920x1080. Developers should stick with this convention, and the CameraInput script makes this effortless because it records the camera view by mimicking a virtual display with the recording resolution. So there is absolutely no constraint that says you must record at a resolution that uses the exact same aspect ratio as the screen. CameraInput works the exact same as resizing the game window in the editor to some new size.

A few things here: first, MixerDevice is still available in NatMic; second, this has nothing to do with NatCorder, it is specific to NatMic.

Also a fair point, but as I've said before, the difference between recording a microphone and recording scene+microphone audio amounts to just three lines, or one depending on how concise you like to code. If there was an example for every single case, then all the example code would be terribly redundant. And trying to make the code less redundant will make the code more complicated for developers. We're betting on simplicity, and so far it has worked extremely well. NatCam was the first API I published, and I tried to do a lot of the things you suggested, but that resulted in tons of code that developers were having issues with when trying to customize or repurpose. Taking inspiration from Apple's API's, we started moving towards concise, modular API's that very neatly plug in to one another, and with that developers have been able to extract the full power and versatility of our API's. Reducing cognitive load has been the best thing we've done with regards to our API's. I'm very much open to making examples separate from the API's, so I would open some public repos on my GitHub that illustrate certain things.

Yikes! Can you email me the unfiltered logs from logcat? This is a critical issue that has to be fixed ASAP.

This is a good point. I'll reupload the build with a new version number (along with a fix for the editor issue).

Email me! I'm pretty responsive over email (I don't bite, I promise). Also, I just wanna say a massive thank you for the detailed feedback. Feedback like yours is what has allowed me to improve the API's so much over the years. Feel free to bounce any ideas for open-source/example projects you'd like to see. I personally want to start making a move to more open-source projects. In fact, I've been working on an open-source video decoder.

 

Hm this can only happen if the encoder isn't ready for more data. Currently, the recorder will simply silently fail to record that frame. I will update the code so that the call to `CommitFrame` blocks until the encoder can consume more data.

 

Ah, that makes a lot of sense. Do you think I can get a beta version of this when it's ready?

Thank you,
Aidan

 

Just so I can implement this and move on, can you please provide the simple 1-3 line of code for recording both scene audio and microphone. Preferably two examples, one for Unity Microphone and one for NatMic. I have spent hours and hours of time trying to do something that is "obvious" and "simple" to you but I cannot get it to work.

 

I have a similar issue on iOS and macOS Editor. On Android and Windows Editor everything works pretty fine. Thank you.

 

Same here. I got an audio source to play, but really I need the audio listener to record all the sound taking place in the scene. This plugin changed completely. I've been trying to get my app back to basic functionality.

 

Hi, my temporary solution is to put the CommitFrame logic inside a while loop with a

Code (CSharp):

1. return yield new WaitForFixedUpdate()

inside an IEnumerator so each frame of video at least gets a cycle to process. Definitely not a guaranteed fix, probably effected by memory usage, but works for now.

 

I'm very new to C# so I'm having difficulty working out what the documentation example script is missing.
I just want to record the game camera to an MP4.

Any help would be very very appreciated!

Code (CSharp):

1. IClock recordingClock;
2. IMediaRecorder mediaRecorder;
3. CameraInput cameraInput;
4. AudioInput audioInput;
5.  
6. void StartRecording () {
7.     // Create a recording clock
8.     recordingClock = new RealtimeClock();
9.     // Start recording
10.     mediaRecorder = new ...;
11.     // Create a camera input to record the main camera
12.     cameraInput = new CameraInput(mediaRecorder, recordingClock, Camera.main);
13.     // Create an audio input to record the scene's AudioListener
14.     audioInput = new AudioInput(mediaRecorder, recordingClock, audioListener);
15. }
16.  
17. void StopRecording () {
18.     // Destroy the recording inputs
19.     cameraInput.Dispose();
20.     audioInput.Dispose();
21.     // Stop recording
22.     mediaRecorder.Dispose();
23.     mediaRecorder = null;
24. }

 

Try this build. Place the library in NatCorder > Plugins > iOS.

 

Change this:

Code (CSharp):

1. audioDevice = AudioDevice.GetDevices()[0];

To this:

Code (CSharp):

1. audioDevice = new MixerDevice(AudioDevice.GetDevices()[0], new VirtualDevice(someAudioListener));

If you are facing any issues, upload logs. This conversation belongs to the NatMic thread.

 

Try the lib I just uploaded. The encoder will probably not need 16ms/33ms to process frames; it should be much faster. No need for adding a delay.

 

The only line you need to change is the `mediaRecorder = new ...`. Simply construct an MP4Recorder (see the README, documentation, example code).

 

Ah ok. So I will not be able to see the file anyway, because it is in the persistent folder?
Ok, so anyway, I have to use natshare (or something similar to copy the file to the gallery?

Got ya.

 

we buyed your asset. Does it support x64 android? If no, when it will?

 

Yes it does.

 

i download last version, create new project import plugin, import vuforia, import natshare. Create simple script, build and have error on start recording

Code (CSharp):

1.  
2. public void StopRecording()
3.     {
4.         cameraInput.Dispose();
5.         audioInput.Dispose();
6.         mediaRecorder.Dispose();
7.         mediaRecorder = null;
8.     }
9.  
10.     public void StartRecording()
11.     {
12.         Debug.Log("123");
13.         // Create a recording clock
14.         recordingClock = new RealtimeClock();
15.         // Start recording
16.         mediaRecorder = new MP4Recorder(Screen.width, Screen.height, 30, 0, 0, OnRecordStop);
17.         // Create a camera input to record the main camera
18.         cameraInput = new CameraInput(mediaRecorder,recordingClock,Camera.main);
19.         // Create an audio input to record the scene's AudioListener
20.         audioInput = new AudioInput(mediaRecorder,recordingClock,Camera.main.GetComponent<AudioListener>());
21.     }
22.  
23.     private void OnRecordStop(string obj)
24.     {
25.         var thumbnail = NatShare.GetThumbnail(obj,1);
26.  
27.         PreviewTexture.texture = thumbnail;
28.     }

 

The hardware encoder failed to start, probably because your screen resolution is too high. Never record at screen resolution. Reduce it to 1280x720 or 1920x1080.

 

Hello I want to buy your NatCorder because i want to be able to record video from phone cam and audio from phone mic, and combine them into the same mp4 file.

While doing that show a preview of the live video from the phone camera onto a texture so the user can see what they are recording.

Can NatCorder do that? Can you show the code on how to do this.

After that, i want to play back the audio and video (that i just recorded) to the screen, while doing that record the screen itself along with the audio that is being played back, all into a second video.

Can NatCorder do all that?

 

The included ReplayCam example does exactly this.

You need to load and play the video (perhaps using Unity's VideoPlayer). Once you do this, pretty much the same code from ReplayCam will work (ReplayCam records whatever is on screen, so if you display a VideoPlayer instead of the WebCamTexture preview, you're all set).

 

Hello,

I bought your NatCorder from Unity asset store.

Im trying to record video/audio from phone camera where the user can record small sections of audio/video at a time, pausing after each section, being able to review playback that section, then, recording another section, being able to playback review that section, and continuing to do that. Then at the end they will have all the little sections combined into a single video.

I know your system is able to pause and resume recording. But can it pause, play back the recorded portion, then resume recording (appending new recorded video to previously recorded).

Another thing is the user has to also be able to re-record the last recorded section over again if they dont like it.

Like this...

1. Start recording first section (video/audio)
2. Pause/stop recording and review (playback what they just recorded)
3. Resume recording a new section
4. Pause/stop recording (playback only that last section they just recorded)
5. If they dont like that last portion, they can re-record it
6. Finally combine all sections into one video.

Is there a way to do that? Maybe i have to create multiple MP4Recorders (a new one after each pause) then combine them all at the end?

 

You can't pause a recording, "review it", then resume recording. You have to entirely stop recording to have access to the recording. Hence you are going to be recording several distinct videos, each corresponding to a section. You can then combine them (perhaps using FFmpeg). NatCorder doesn't do video post-processing, so it can't combine videos.

 

What if i create two instances of the recorder, a full recorder, and a section recorder. The full recorder stores the whole video, and the section recorder records the little sections.

Once the user records and is satisfied with a recorded section of video, I push that section to the full recorder (without ending the full recorder), then record a new section.

We have to have access to all the video/audio/timestamps samples in the section recorder to be able to push those samples to the full recorder.

When the user satisfy with that section, push appending that it to the full recorder. Keep recording and reviewing each section and pushing to the full recorder until the end, stopping the full recorder, getting access to the whole video without using FFMPEG (which doesnt work in Unity).

is this possible?

 

A recorder cannot, by definition, record 'sections'. There is a 1:1 mapping between what you consider a section and a recorder.

You can't 'push' frames from a recorder to another. Recorders can only record frames, not load them and give to something else.

Say you have `n` sections, your only option is to record those `n` sections with `n` recorders (given the 1:1 mapping) and get `n` section videos. You must then figure out how to concatenate those `n` sections into 1 final video, which is where FFmpeg comes in. NatCorder does not do any post-processing; it simply records to a video file and that's it.

 

I cant use FFMEG because there is none for Unity anymore, this is why i was trying to find a work around because i need to concate the videos.

There was one old 32-bit FFMPEG in the unity asset store but its no longer being actively developed and updated. So since Android is going full 64 bit, it will no longer work.

There is one 64-bit FFMPEG for Android and iOS, but it has to be converted to work with Unity, if you could do that im sure you could make money off it.

https://github.com/tanersener/mobile-ffmpeg

Maybe another way would be to simply play the individual recorded video segments one after the other to a texture, and then while they are playing, record them all from the texture with NatCorder creating a new video of all of them concatenated.

Is there sample code i can use that NatCorder records Audio/Video from a texture instead of the camera?