NatCorder - Video Recording API

This isn't garbage memory. NatCorder needs to get the pixel data to send to the hardware encoder. Doing so requires calling Texture2D.GetRawTextureData. And unfortunately, this function will always return a new byte array. There is no way to have it fill an existing array. So there is really no way around it right now.

 

Thanks, I feared that answer.

Another question, then: since I cannot appreciate any real benefit in performances with this new update, is there a way to disable its newly introduced optimizations to keep being compatibile with Android API level 18? According to Unity, switching to Marshmallow as a minimum is approximately a 25% market share loss.

 

The garbage collection only affects standalone and WebGL platforms. On iOS and Android, we don't generate GC memory when committing frames. Switching to Marshmellow on Android affects both MP4 recording performance and GIF recording quality. There isn't a way to switch off the changes that have been made; you'll have to stick with an older version of the API.

 

Yeah I know its very similar to ReplayCam, and it looks redundant to someone who knows the API very well. But you have to realize that most people when downloading your asset for the first time will not read the readme, they will go straigth to the examples folder and look for an example scene that matches their need. And most peope are not looking for a webcamera recording solution, but a video game recording solution - and your example doesn't show that part very well (even though thats kind of what happens).

 

As I said the profiler just shows that a long time is spent in NatCorderBridge.EncodeFrame

 

Did you have a chance to try AsyncGPUReadback? It's asyncronous and also it relies on NativeArray<T>, so you can dispose them immediately after use.

 

I totally understand this, it's something I've done in the past. But with NatCorder, we specifically need to discourage this. Video recording isn't a straight-forward process; it's pretty complicated on different platforms. We've gone great strides to simplify it all to just five functions, but there are still quirks here and there. As a result, we absolutely 100% need developers to check out the README.md, at the very least.

 

Committing shouldn't take this long. What is your recording resolution? And what AudioFormat are you passing into StartRecording? This might be the encoder choking on a misconfiguration.

 

This is ideal, but it was introduced in Unity 2018.1 and stable in 2018.3. A lot of devs use older versions of Unity, and can't upgrade because of dependencies or breaking changes.

 

I totally understand that. Maybe it could be a good idea to implement this more efficient alternative as well, using conditional compilation to enable backward compatibility.

Aside from that: do you have any suggestion to improve realtime recording? I've smashed my head against it for several days, but results are not good enough. Both on PC and Android the framerate is way too low.
I tried to overcome it by recording frames on a fixed framerate: video smoothness is perfect, but I'm not able to get audio in sync, even pitching master channel and stretching audio samples in OnAudioFilterRead doesn't work.

Framerate in Unity Editor is over 200fps (1080p), but when I start recording it doesn't reach 10fps (720p); some recording configurations also make Unity to crash after 2 or 3 seconds.
I tried using camera.Render(), to Blit from camera.activeTexture, to assign Natcorder's RenderTexture to camera in OnPreRender().. I also tried calling GC.Collect periodically in a background thread.
Nothing works.

 

Hi, I've been using the NatCorder plugin in my project to record gameplay in a 2D scene. This is working correctly in the Editor and in Android, however, when used in iOS, the devices crash because of excessive memory usage.

When profiling the app through Unity, it can be seen that ManagedHead.UsedSize is using huge amounts of memory when the recording process is started, along with System.ExecutablesAndDlls (As seen in the screenshot)

How can this behavior be prevented? In the screenshot it can also be seen that garbage collection is never triggered, which could be a factor in the excesive memory usage.

Thanks for your help.

 

Hi Lanre,

We're seeing a repeatable crash with NatCorder 1.4.0 and NatMic 1.2.0 running on a Samsung S6 with Android 7.0:

2019-02-08 19:27:45.235 11110-12155/com.xxx E/AndroidRuntime: FATAL EXCEPTION: NatCorder Audio Encoding Thread
Process: com.xxx, PID: 11110
java.lang.Error: FATAL EXCEPTION [NatCorder Audio Encoding Thread]
Unity version : 2018.3.0f2
Device model : samsung SM-G920F
Device fingerprint: samsung/zerofltexx/zeroflte:7.0/NRD90M/G920FXXU6ERF5:user/release-keys

Caused by: java.lang.IllegalStateException: Can't write, muxer is not started
at android.media.MediaMuxer.writeSampleData(MediaMuxer.java:496)
at com.yusufolokoba.natcorder.MP4Recorder$4.onOutputBufferAvailable(MP4Recorder.java:261)
at android.media.MediaCodec$EventHandler.handleCallback(MediaCodec.java:1662)
at android.media.MediaCodec$EventHandler.handleMessage(MediaCodec.java:1609)
at android.os.Handler.dispatchMessage(Handler.java:102)
at android.os.Looper.loop(Looper.java:154)
at android.os.HandlerThread.run(HandlerThread.java:61)

The same issue isn't present on Samsung S9 with Android 8.0.

Please do you have any thoughts on how we can work around this? At present, we are only recording mic audio, not mixing with game audio as that causes different unwanted side-effects.

When recording game audio with Format.Unity and passing a listener to NatMic.StartRecording, this is the error we receive:

2019-02-11 11:29:06.576 11186-11867/com.xxx E/AndroidRuntime: FATAL EXCEPTION: NatMic Microphone Thread
Process: com.xxx, PID: 11186
java.lang.Error: FATAL EXCEPTION [NatMic Microphone Thread]
Unity version : 2018.3.0f2
Device model : samsung SM-G920F
Device fingerprint: samsung/zerofltexx/zeroflte:7.0/NRD90M/G920FXXU6ERF5:user/release-keys

Caused by: java.lang.ArrayIndexOutOfBoundsException: length=1920; index=1920
at com.yusufolokoba.natmic.NatMic.run(NatMic.java:91)
at java.lang.Thread.run(Thread.java:762)

 

First rule of thumb is to reduce the recording resolution. What resolution are you recording at?

 

NatCorder doesn't accrue GC memory on iOS. What it does accrue is texture memory because it tries not to block the Unity main and render threads. I am working on a minor update that should improve this behaviour. What resolution are you recording at?

 

Can you send me the full logs from logcat in a txt file?

I've received a report of this. I am looking into it. I plan to publish a minor NatMic update with the fix tomorrow.

 

Currently I'm recording at two resolutions, 1334x750 (16:9) in an iPhone 6 and 1366x1024 (4:3) in an iPad 4.

I'm using an older version (1.3f2) of the plugin since updating would cause us to loose a big part of the market, specially in Android.

What would you recommend to avoid this issue? If possible without updating.

Thanks for your help.

 

720p

 

Hi Lanre, in the new version of the plugin i get the following exception and its being spammed

E/Unity: AndroidJavaException: java.lang.ClassNotFoundException: com.yusufolokoba.natcorder.Bridge
java.lang.ClassNotFoundException: com.yusufolokoba.natcorder.Bridge
at java.lang.Class.classForName(Native Method)
at java.lang.Class.forName(Class.java:453)
at java.lang.Class.forName(Class.java:378)
Caused by: java.lang.ClassNotFoundException: com.yusufolokoba.natcorder.Bridge
at java.lang.Class.classForName(Native Method)
at java.lang.BootClassLoader.findClass(ClassLoader.java:1355)
at java.lang.BootClassLoader.loadClass(ClassLoader.java:1415)
at java.lang.ClassLoader.loadClass(ClassLoader.java:312)
at java.lang.Class.classForName(Native Method)
at java.lang.Class.forName(Class.java:453)
at java.lang.Class.forName(Class.java:378)
Caused by: java.lang.NoClassDefFoundError: Class not found using the boot class loader; no stack trace available
at UnityEngine.AndroidJNISafe.CheckException () [0x00091] in <004fc436a9154f7fab4df9679445af6c>:0
at UnityEngine.AndroidJNISafe.CallStaticObjectMethod (System.IntPtr clazz, System.IntPtr methodID, UnityEngine.jvalue[] args) [0x00011] in <004fc436a9154f7fab4df9679445af6c>:0
at UnityEng


it's being called whenever a audio is commited

 

Hi, I'm facing a problem on iOS below:

Code (CSharp):

1. Undefined symbols for architecture arm64:
2.   "_NRGPUFenceSync", referenced from:
3.       _MTLFence__ctor_mD945145DE46BAB61848931056A28BE7947144BA0 in Bulk_Assembly-CSharp_2.o
4.       _MTLFence_GPUFenceSync_m2EE0F4FB66C2A60DFF29C433E46B16F3671B2451 in Bulk_Assembly-CSharp_2.o
5. ld: symbol(s) not found for architecture arm64
6.  

I'm both using NatCam and NatCorder, so I've removed NatCam/libNatRender.a.


Here's my import setting for NatCorder/libNatRender.a
and I'm currently using 2018.3

 

If your app runs at 60FPS, then you could set the CameraRecorder.recordEveryNthFrame to 2 to record 30FPS videos. This alleviates memory pressure a great deal.

 

Can you try deleting and re-downloading NatCorder from the Asset Store? We published a silent update that resolved the missing symbol issues.

 

The app is running at 30 FPS in the mentioned low end devices. We identified the issue in the framePool not being able to free memory fast enough, so to prevent crashes we stop commiting frames to the pool for some milliseconds as to alleviate pressure. However, is there any solution to prevent such an excesive memory usage?

 

In Natcorder, whenever you use AudioRecorder, we have to use CameraRecorder? Can we use AudioRecorder with greyworld scene logic?

 

NatCorder works by providing the client RenderTextures on which to render. NatCorder has to continuously allocate RenderTextures, and when those RenderTextures have been used, they are freed. It is this allocation that causes memory pressure to build up. There is no way to go around it.

 

You don't have to use an AudioRecorder with an CameraRecorder. You can use an AudioRecorder while manually committing video frames.

 

i get this crash if i use AudioRecorder while manually commiting video frames.

--------- beginning of crash
E/AndroidRuntime: FATAL EXCEPTION: NatCorder Audio Encoding Thread
java.lang.IllegalStateException: Can't write, muxer is not started
at android.media.MediaMuxer.writeSampleData(MediaMuxer.java:697)
at com.yusufolokoba.natcorder.MP4Recorder$4.onOutputBufferAvailable(MP4Recorder.java:261)
at android.media.MediaCodec$EventHandler.handleCallback(MediaCodec.java:1674)
at android.media.MediaCodec$EventHandler.handleMessage(MediaCodec.java:1621)
at android.os.Handler.dispatchMessage(Handler.java:105)
at android.os.Looper.loop(Looper.java:164)
at android.os.HandlerThread.run(HandlerThread.java:65)

 

Yes, I downloaded your plugin just before I wrote the issue.
I also checked the version which is 1.4.0

 

Hi! Just curious is NatCam Core the only one deprecated? NatCam is still current, for device camera access?

 

Fix for this coming next week.

 

Can you email me your NatCorder folder?

 

NatCam Core and NatCam Pro were merged into one API: NatCam. NatCam is still your one stop shop for all things device camera control.

 

Sure, Please check your gmail

 

Hi,
I wanted to record the screen video, but I used it
NatCorder.StartRecording（Container.MP4，VideoFormat.Screen，AudioFormat.None，OnRecording）;
The recorded video size is 0KB

I refer to it here :
https://olokobayusuf.github.io/NatCorder-Docs/
Can the natcorder record only one camera?

Or is there a problem with the second parameter setting?Because your API doesn't have that parameter there
NatCorder.CommitFrame(frame,(long)Time.deltaTime);

 

After calling StartRecording, you need to actually push frames to NatCorder. Check out the README.md for more info.

 

Hi @Lanre, do you have any more suggestions on that? Thanks

 

I don't think there's much more you can do. The readback is really what eats up performance. I am considering adding support for AsyncGPUReadback. I think this is the only thing that can improve memory consumption, albeit at a memory cost as is the case on iOS.

 

I'm having a similar problem to Skjalg. This is in the Unity Editor (windows). when I record frames with NatCorder, I get about 2-5 seconds of smooth recording, and then the framerate tanks and each frame is about 1 second long. The culprit is inside EncodeFrame() as you can see below. The recording is at 1080p, 30fps.

in the screenshow you can see the few seconds of OK performance (medium blue spikes to the right), and then the complete change

 

I uploaded the project file to the forum as an example on how to use NatCorder. Its easily reproducable with that. I think you possibly even downloaded it on your own machine and looked at the code I used

If you can't reproduce it on your end then it is possible this is a hardware problem? I'm on Windows 10 Pro N
Itel Core i7-6700HQ @ 2.6Ghz with a nvidia GeForce GTX 908M (asus laptop).

I've attached a gif of how it looks like to record something in the test scene I uploaded earlier with the settings
Container: MP4
Audio Format: Unity
Size: 1280x720

Here's also a screenshot of how it looks. The first green spike is when I start recording. The big purple block is after a few seconds of recording when things suddenly starts taking 1000ms



Edit:
It looks like this might be a unity bug as people are having similar problems with the Unity Recorder https://forum.unity.com/threads/unity-recorder-update.509458/page-7#post-4084444

 

Same issue here: sometimes it works smooth, sometimes after a couple of seconds its starts taking 2000+ms per frame (almost identical profiler outcome).
Unity 2018.3.2f1.

 

I have a question:

Is it possible to fetch the recorded video and upload it to a server, and delete the file from the documents folder?
It is to record a webcam chat between people, and it needs to be saved for legal purposes.
So the recorded stream needs to go to a server and not to the local filesystem.

Is this possible with the NatCorder plugin?

(and is there any extra permissions necessary?)

 

And does this happen on Windows or macOS? The one report of a similar issue I had turned out to be because the developer wasn't committing audio frames even though they had specified an audio format in StartRecording. This seems to be specific to Windows. I'll try to reproduce it.

 

When recording is stopped, NatCorder will invoke a callback you provide with the path to the recorded video.

This is up to you (NatCorder doesn't do anything apart from video recording), but there are API's for uploading raw data to remote endpoints.

You can use the System.IO.File API's for this.

NatCorder can only record to the local file system, not to a stream.

 

I implemented the AsyncReadback alternative: it works smoothly and GC.Alloc passed from 3.5 MB per frame to 200 b.

Main issue is still there, though: profiler reports that every frame a new Texture is created and kept, and as soon as it reaches a certain threshold (1~3 seconds) each frame starts taking 2000+ms and eventually Unity hangs/crashes.

I'm not creating any texture.. it seems like internally Natcorder allocate new temporary RenderTextures but somehow never release them properly.

EDIT:
I missed this post

I'll investigate more in this direction.

 

Hi @Lanre I'm starting to use NatCorder in a project and it is an impressive asset! I love the simplicity of the API and how everything just works.

There is just one thing that I haven't figured out. Is there a way to define where do I want the videos to be located? I have searched in the docs and code and it seems to me that there is no way. Can you please confirm this? - Thanks. Juan P.

Edit: Nevermind. I just found the answer in the forum. Thanks anyway!

 

I confirm: ensuring correct CommitSamples calls or disabling audio recording at all doesn't make any difference: texture memory increases frame by frame and after a second or two it slows down or crashes.

EDIT: Texture usage increment can be experienced on Android as well

 

This is on Windows 10. Here's the code I use to start the recording:

Code (CSharp):

1. NatCorderU.Core.VideoFormat vidFmt = new NatCorderU.Core.VideoFormat(1920, 1080, 30, 10 * 1024 * 1024);
2. NatCorderU.Core.NatCorder.StartRecording(NatCorderU.Core.Container.MP4,
3.                                                  vidFmt,
4.                                                  NatCorderU.Core.AudioFormat.Unity,
5.                                                  _gotRecording);

and capturing a frame looks like this:

Code (CSharp):

1. RenderTexture frameTex = NatCorderU.Core.NatCorder.AcquireFrame();
2. Graphics.Blit(ReplayMenuRef.GameRTT, frameTex);
3. NatCorderU.Core.NatCorder.CommitFrame(frameTex, timestamp);
4.  

GameRTT is a RenderTexture that my main game camera is rendering to.

 

It actually looks

This is probably it, because the video's I do get to record which are 1-2s long doesn't contain any audio

 

What is this AsyncReadback method?

 

It's another approach to GPU memory read that Unity recently made available.
You can try my simple implementation by replacing the last two methods in BlockingContext.cs with these:

Code (CSharp):

1. public override void Readback (RenderTexture frame, ReadbackDelegate handler) {
2.             if (synchronizationHint == SynchronizationHint.LessMemory)
3.                 ReadbackAsync(frame, handler);
4.             else
5.                 dispatcher.Dispatch(() => dispatcher.Dispatch(() => ReadbackAsync(frame, handler)));
6.         }
7.         #endregion
8.  
9.  
10.         #region --Operations--
11.  
12.         private void ReadbackAsync (RenderTexture frame, ReadbackDelegate handler) {
13.             // Null checking
14.             if (!frame)
15.             {
16.                 handler(IntPtr.Zero);
17.                 return;
18.             }
19.  
20.             // Readback
21.             AsyncGPUReadback.Request(frame, 0, (AsyncGPUReadbackRequest request) =>
22.             {
23.                 NativeArray<byte> array = request.GetData<byte>();
24.                 unsafe
25.                 {
26.                     IntPtr ptr = (IntPtr)Unity.Collections.LowLevel.Unsafe.NativeArrayUnsafeUtility.GetUnsafePtr(array); // void* -> IntPtr explicit conversion.
27.                     handler(ptr);
28.                 }
29.             });
30.         }
31.  
32.         #endregion

Please note that it's extremely simplified, and it could lead to some crash with longer recordings (basically, StopRecording could be executed while some frames are still waiting to be processed.. I've a workaround for it, if you need it, just ask)

 

@Lanre, I think I found the culprit.

In your latest update, in iOS implementation (used on Windows) you switched from getting/releasing a temporary RenderTexture to creating a new one. For some reasons, Release() isn't enough to free them up, and those newly created RenderTextures are piling up frame by frame and eating all system memory (about 1GB/s).

Changing to GetTemporary instead of Create totally removes the issue.
Is there any reason Create should be preferred over GetTemporary (maybe on iOS)?