NatCorder - Video Recording API

Hi Lanre,

We have a home technology in native that manages a camera to render line drawing in post-process and the application that uses this techno was developed with Natcorder 1.5.1. We need to upgrade Unity to 2019 and we got the last version of Natcorder but we trying to replace the RenderTexture that method "AcquireFrame" returns but I try to figure out how this RenderTexture is coming. Because the code you see below works with our techno but we are stuck to find a replacement.

Before:
// m_Recorder is a MP4Recorder instance.

Code (CSharp):

1. var frame = m_Recorder.AcquireFrame();
2. if (frame != null)
3. {
4.     Graphics.Blit(m_NatcorderCam.targetTexture, frame);
5.     m_CurrentClip.Frames++;
6.     m_Recorder.CommitFrame(frame, m_RecordingClock.Timestamp);
7. }
8.  

It's very appreciated if you can explain how AcquireFrame was working to find a way to replace it.

We tried many ways found on this thread but we have five cameras in this project. If we can know what represents the "frame" returned by m_Recorder.AcquireFrame(); We could resolve it faster.

Thank you

 

If ReplayCam works, then I recommend building off it. I'm not sure why your code doesn't work, but you are likely making an error somewhere. I also recommend going through the documentation as it will inform you of any errors you might run into.

Recording is a heavy process, so you have to make sure that you are optimizing your app's rendering. See the documentation for tips. On Android, async readbacks might not be supported, especially if you are rendering with OpenGL ES instead of Vulkan. Again, the documentation discusses this.

NatCorder cannot record audio that is playing directly to the device's output. You have to play the audio through an audio source in the scene, then use an AudioInput to record the audio coming from it.

 

If you want to record frames from a RenderTexture, you must first readback the pixel data from the GPU. Then send the pixel data to the recorder. See the documentation for more info.

 

Yeah, I tried it but still no good. I think the key is the RenderTexture given back from AcquireFrame or AcquireFrame do something we miss it. The last person that working on it has updated to 1.6.0 and reverted to 1.5.1 because of that.

 

Oh it looks like you are asking about an updated version of NatCorder. Upgrade to NatCorder 1.7 on the Asset Store. I don't provide support for outdated versions of the API.

 

I've updated Natcorder from 1.5.1 to 1.7.3, I'm was saying someone in my enterprise tried to update from 1.5.1 to 1.6.0 last year but the fact they lost the AcquireFrame method force him to reverts the code because he doesn't find a way to fix without AcquireFrame. Now it's me that has the same problem to update from 1.5.1 to 1.7.3.

I see the native code for our techno is completely write for OpenGLES2, I think it's a part of the problem. I remember you said Natcorder doesn't support OpenGLES2.

 

There is no longer an AcquireFrame method. You can either create a RenderTexture or get a temporary one (see Unity's documentation). When you want to commit the frame in the RenderTexture, you need to perform a readback of the pixel data from the GPU. See the documentation on this.

 

Yeah, I tried it.

I see the native code for our techno is completely write for OpenGLES2, I think it's a part of the problem. I remember you said Natcorder doesn't support OpenGLES2.

 

NatCorder is now completely agnostic to the graphics API being used. It will run on OpenGL ES2 as normal. If you have a problem, please be very specific as to what it is. Share any errors, logs, code you have that might be helpful in resolving it.

 

Hi could I get help recording audio?

I want to record from the Audio source on ios. I followed this guide: https://medium.com/natsuite/natcorder-tutorial-audio-workflows-1cfce15fb86a

But the audio playing gets 'stuck' when i start recording and their is no audio in my recorded video.

Here is the code for the recorder:

Code (CSharp):

1.         // Set the recorder inputs
2.         recorder = new MP4Recorder(_videoWidth, _videoHeight, frameRate, AudioSettings.outputSampleRate, (int)AudioSettings.speakerMode);
3.  
4.         // Clock handles the frames.
5.         var clock = new RealtimeClock();
6.  
7.         // Create recording inputs
8.         cameraInput = new CameraInput(recorder, clock, new Camera[] { _cam, _camFilters, _watermark });
9.  
10.         _audioInput.Init(recorder);

And Here is the code for the 'AudioInput' that is attached to the audio listener in the scene:

Code (CSharp):

1.  
2.     public IMediaRecorder _recorder;    // Initialize this somewhere
3.     IClock clock;               // Also initialize this somewhere
4.  
5.     void OnAudioFilterRead(float[] sampleBuffer, int channels) {
6.         // Send to NatCorder for encoding
7.         _recorder?.CommitSamples(sampleBuffer, clock.timestamp);
8.     }
9.  
10.     public void Init(IMediaRecorder recorder) {
11.         _recorder = recorder;
12.     }

Thanks for the help!

 

Hey there. First off, use NatCorder's AudioInput class, instead of using your own. Beyond that, make sure that all the audio sources in your scene have the "Bypass Listener Effects" setting disabled.

 

Hi, yes i used the audiosource in a video, but also in this case the audio gliched a lot, all the audio of the video start and stop, it is probably because a fps drop.
the scene have only a video an replyCal Script for recording.

I already tried with multitread and it do solve unfortunately,

Sorry for all this fast posts I had a delivery, for now my boss told me to skip about this.
In the meanwhile thanks a lot he help

 

I'm not sure I fully understand the issue here. If you are facing audio glitching, then check that the issue is not coming from the VideoPlayer.

You already mentioned that ReplayCam works for you, so check that the issues aren't coming from the video player.

What device and OS version are you testing on? You need to make sure that your device supports asynchronous readbacks. You can use the Unity Profiler to see where all the time is being spent.

 

Hi Thanks,

I have done the above and now things are moving forward. However my fial video does not have any audio in it. Could you please help me figure it out. Also if there a simple demo to capture the screen and the audio? I am playing a music clip from a AudioSource.

Code (CSharp):

1.         // Clock handles the frames.
2.         var clock = new RealtimeClock();
3.  
4.         // Audio
5.         // Say we have an `AudioListener` we want to record
6.         // Create a recorder
7.         var sampleRate = AudioSettings.outputSampleRate;
8.         var channelCount = (int)AudioSettings.speakerMode; // Stereo = 2
9.         _audioRecorder = new WAVRecorder(sampleRate, channelCount);
10.  
11.         // And use an `AudioInput` to record game audio
12.         _audioInput = new AudioInput(_audioRecorder, clock, _audioSource);
13.  
14.         // Set the recorder inputs
15.         _videoRecorder = new MP4Recorder(_videoWidth, _videoHeight, frameRate, AudioSettings.outputSampleRate, (int)AudioSettings.speakerMode);
16.  
17.         // Create recording inputs
18.         _cameraInput = new CameraInput(_videoRecorder, clock, new Camera[] { _cam, _camFilters, _watermark });

 

You are sending your audio to the WAVRecorder, so no audio is being sent to the video recorder.

 

Ah thanks! How do i send the audio to the MP4 recoder? Or how to I record the screen and the audio at the same time?

 

When you create the audio input, pass in your video recorder instead of the WAV recorder.

 

Thanks, That's it!

 

hmm We trying to add record function on our ARCore base ARApp.
our testing on editor is mp4 file quality is okay. good.
( Editor + Instant Preview of ARCore. )

but mp4 file have sound but screen is stopped on first frame when we install apk on phone.

this png is our project setting.
Have to change something?

 

Can you share your recording code along with the full runtime logs from logcat (in a .txt attachment).

 

Sorry for my bad english maybe I not write all correct
The problem occurs only while I am recording, and the profile show me a very high usage on the android device... (and the video lag a LOT)

I mentioned that ReplayCam works, but only in the demo...
If i put a videoplayer in the same scene i get the problem of all the video that suffer of terrible lag.

The solution is a business to client solution, so I can not create a solution that works only with some devices.
I am testing with an android device Galaxy A20e with Android version 10.

I have not the problem in the editor, but as I said the solution have to work with different devices.
With some AR requirements I can not use Vulcan as graphics and multirendering do not help

Let me know if you can help me

 

Hmm my co-worker edited many so now can't work.
So I just made simple copy and run.
eh? this time... video working. but no sound..
( maybe some of plugin trouble with your on our main project. but why no sound on this time?? )
I will ask again after more many test.
thanks

 

Not being able to use Vulkan is a major performance killer on Android. I'm aware that ARCore requires OpenGL ES. I'm looking into writing an async readback plugin for OpenGL ES since Unity doesn't support it. Hopefully, that should greatly reduce the performance cost.

 

That will be awesome.
I have found this repo. But the performance is not quite good.
Also found this in assetstore. I have not tried it yet because it looks quite suspicious and a bit expensive.

 

@Lanre
Hello
I need to record over and over again on Android.
After each record I finish it.
But after seven or eight repetitions, the frame rate drops sharply.
And finally it crashes.

 

@Lanre Could you recommend anything to improve performance on Android?

We are building an AR app, on IOS everything is perfect. On Android we cant use Vulkan and the performance is much slower.

I am using an mp4 recorder with 3 input cameras.

Do you have any tips?

- Would reducing the number of camera captures make significant improvements?
- Would removing the audio capture improve much?
- Does the recorder dimensions have a big impact?
- Would the GIf recorder be any better than the mp4 recorder?
- etc..

Thanks,

Adam

 

Can you share the full logs from logcat in a .txt attachment?

 

The main problem stems from not being able to use the AsyncGPUReadback class. I'm looking into writing a free plugin to provide this functionality on Android.

 

Amazing, sounds like this would solve a lot of issues.

Thanks.

 

When recording videos on Android at a resolution smaller than the screen size the UI doesn't resize correctly with Unity 2019.4.15f (LTS), but does work correctly with 2019.4.14f. Can you confirm that you see this as well? Do you have any work arounds that you could suggest? I tried calling Canvas.ForceUpdateCanvases() in the OnFrame of CameraInput but that doesn't seem to help.

Thanks.

 

This sounds like a Unity bug. I recommend filing a bug report with Unity. I can't think of anyway to go around this since it's coming from Unity.

 

Making some progress on implementing async readbacks on Android + OpenGL ES. If you use ARCore/ARFoundation, be excited:


This should give you Vulkan-level performance, so no more massive frame rate hits when you record in your AR app.

 

@Lanre


The problem was solved
I "new recorder" twice in a row.

Thanks for the reply

 

hello,
well love the Natcoder....
is there a simple script example how to move the file after recording to desktop?

this doesn't work so far... i'm on Mac OS X

Code (CSharp):

1. string pathDesktop = System.Environment.GetFolderPath (System.Environment.SpecialFolder.Desktop);
2.             System.IO.File.Move ($"file://{path}", pathDesktop);

also how do i'll record webcam video together with webcam audio in a mp4 file?
got only Video work proper...

a simple example would be nice...
many thx
M.

 

Don't add "file://". Just pass in the path as-is.

I don't think Unity supports streaming audio with the camera preview.

 

Hi! I just got the Asset and I'm kind of new to this. I've been trying it out for a while but I can't get it to work. I copy-pasted the script in the Getting Started Documentation and added it to an Empty GameObject on my scene and Unity just crashes on me every time. I tried doing the MP4 Recorder with the same results.

Also trying out the Scene examples, crashes my Unity after some seconds of running (I press the white button but nothing happens, I don't see any new files showing up on the root of my project)

I'm using Unity 2019.4.11f and Building configurations are set to Android

If you need more info about the crashes let me know and I'll see how I can get those, but what I would really like is just a script to record the whole scene from the Main Camera for a few seconds and then stop.

 

I'm not too sure why you are getting consistent crashes. Can you share the editor logs or crash report in an attachment? Just to confirm, the ReplayCam example scene is also crashing for you? In ReplayCam, you are supposed to press and hold the record button (like Snapchat). But even then, you should not be getting any crashes.

 

Sure, let me do that and I'll get back to you. After using the ReplayCam button where should I see the record? In the root path of the project?

 

I believe ReplayCam will log the path to the recorded video. In the editor it should be at the project root.

 

Hi again! After getting ReplayCam to work correctly I'm using that script as template to write mine. And here is where it crashes. The idea for the script is very simple, just control the recording from the Unity Inspector clicking on the bools startRecording and stopRecording. Of course this is just to test out but still when I try to start the recording, it fails miserably, probably I'm doing something wrong so here's the script and thank you very much.

Code (CSharp):

1. using UnityEngine;
2. using System.Collections;
3. using NatSuite.Recorders;
4. using NatSuite.Recorders.Clocks;
5. using NatSuite.Recorders.Inputs;
6.  
7. public class ReplayTest : MonoBehaviour {
8.  
9.     [Header(@"Recording")]
10.     public int videoWidth = 1280;
11.     public int videoHeight = 720;
12.     public bool recordMicrophone;
13.  
14.     public bool startRecording;
15.     public bool stopRecording;
16.     private IMediaRecorder recorder;
17.     private CameraInput cameraInput;
18.     private AudioInput audioInput;
19.     private AudioSource microphoneSource;
20.  
21.     private IEnumerator Start () {
22.         // Start microphone
23.         startRecording = false;
24.         stopRecording = false;
25.         microphoneSource = gameObject.AddComponent<AudioSource>();
26.         microphoneSource.mute =
27.         microphoneSource.loop = true;
28.         microphoneSource.bypassEffects =
29.         microphoneSource.bypassListenerEffects = false;
30.         microphoneSource.clip = Microphone.Start(null, true, 10, AudioSettings.outputSampleRate);
31.         yield return new WaitUntil(() => Microphone.GetPosition(null) > 0);
32.         microphoneSource.Play();
33.     }
34.  
35.     private void OnDestroy () {
36.         // Stop microphone
37.         microphoneSource.Stop();
38.         Microphone.End(null);
39.     }
40.  
41.     public void StartRecording () {
42.         // Start recording
43.         startRecording = false;
44.         var frameRate = 30;
45.         var sampleRate = recordMicrophone ? AudioSettings.outputSampleRate : 0;
46.         var channelCount = recordMicrophone ? (int)AudioSettings.speakerMode : 0;
47.         var clock = new RealtimeClock();
48.         recorder = new MP4Recorder(videoWidth, videoHeight, frameRate, sampleRate, channelCount);
49.         // Create recording inputs
50.         cameraInput = new CameraInput(recorder, clock, Camera.main);
51.         audioInput = recordMicrophone ? new AudioInput(recorder, clock, microphoneSource, true) : null;
52.         // Unmute microphone
53.         microphoneSource.mute = audioInput == null;
54.     }
55.  
56.     public async void StopRecording () {
57.         // Mute microphone
58.         microphoneSource.mute = true;
59.         // Stop recording
60.         audioInput?.Dispose();
61.         cameraInput.Dispose();
62.         var path = await recorder.FinishWriting();
63.         // Playback recording
64.         Debug.Log($"Saved recording to: {path}");
65.         Handheld.PlayFullScreenMovie($"file://{path}");
66.     }
67.        
68.     void Update () {
69.         Debug.Log("Updating");
70.         if (startRecording) {
71.             StartRecording();
72.         }
73.         if (stopRecording) {
74.             StopRecording();
75.         }
76.     }
77. }
78.  

Just in case I'm attaching this to a Empty GameObject on the scene and as long as I dont press startRecording it doesn't crash.

Thanks a lot again!

 

I don't recommend starting and stopping recording in Update. Using booleans to know when to start and stop can cause issues down the line. Without the logs or a stack trace, I can't tell what is causing the crash. If ReplayCam works correctly, then the issue likely comes from something in your code.

 

Hi, ive run into an issue that I think is a bug.

I am recording multiple cameras (x3). One is an AR Camera, and the other two are UI layers, one with a hidden camera that is disabled which I am using for a watermark.

All was working fine until I tried reducing the capture size. Initially I was using Screen.Size. But now I am capping it at 720p and using the screen ratio to calulate the other length.

This works fine for the AR camera. the video comes out at a lower resolution. The problem is my UI layer is not resized and now apears much larger. The watermark used to be small inte bottom corner, now it is to big to fit in the view.

I would expect it to capture it as it appears on the screen but it doesnt.

Make sense?

 

It sounds related to this issue. It is a Unity issue, so I recommend filing a bug report with Unity.

 

Hiim also getting an ios on ios where the "Recorder failed to finish writing"

Code (CSharp):

1. Non-fatal Exception: Exception
2. 0  ???                            0x0 Throw (System.Runtime.ExceptionServices.ExceptionDispatchInfo)
3. 1  ???                            0x0 ctor (System.Collections.Concurrent.ConcurrentDictionary`2+Tables[TKey,TValue].)
4. 2  ???                            0x0 get_Current (VideoPlayerManager+<PlayVideo>d__8.System.Collections.IEnumerator)
5. 3  ???                            0x0 CreateCacheableTask[TResult] (System.Runtime.CompilerServices.AsyncTaskCache)
6. 4  ???                            0x0 StopRecording (VideoRecording)

Is there anything I can do about this?

 

Can you share the full logs? I need to see everything from before your first recording.

 

So this is from a live project an these are captured logs. its all i have. But i have quite a lot of the same thing.

 

Can I ask about the core.jar in the plugins folder? I had to remove this to enable firebase to work but now the video capture doesnt work.

The core.jar is incompartible with firebase. Is there a way to fix this? It creates duplicates.

 

Can you reproduce it locally and send me the full logs? Without it, I can't tell what the issue might be. Also make sure to share your code.

You should be able to remove the core-rc library if you see duplicates.

 

Is there any reason why the get thumbnail functionality was removed? This missing feature is really painful^^
I am trying to adjust the ReplayCam script in a way that it takes a Screenshot if the button is pressed shortly, but the ScreenCapture class from Unity is a pain when using UI.

 

NatCorder never had a get thumbnail function. You are likely referring to the function in NatShare, which didn't quite fit into the social sharing idea (it was more of a lone utility function). Trying to hotwire the example code is cumbersome and unnecessary.

What you'll need to do is to render your specific camera(s) to a RenderTexture, then readback the pixel data from that RenderTexture into a Texture2D. You can then use that Texture2D for any purpose you choose, including encoding to a JPEG or PNG.