NatCorder - Video Recording API

Thanks!

Question... is it really only from Unity version 2018.3.2?
Our app is on 2018.2.20... not backward compatible?

Best, Sander

 

Sounds good.

This actually makes sense. So because NatCorder is designed to never block (for performance), we must create a new, independent RenderTexture every time AcquireFrame is called. This RenderTexture can't be disposed until it has been 'encoded'. In the next NatCorder update, this will happen immediately CommitFrame is called on macOS, Windows, and WebGL. On Android, this doesn't happen until a while after, because the texture has to be sent to NatCorder's encoding thread where the frame is copied directly to the encoder's Surface. On iOS, things are still up in the air (balancing performance and memory consumption is one heck of a challenge). If the encoder can't keep up, then a backlog of textures starts to fill up memory, adding memory pressure. This is what manifests itself as a massive slowdown.

Nope. NatCorder doesn't use temporary RT's because their lifetime is pretty ambiguous from the docs. And they will always get released, but not immediately because they have to be 'encoded' first.

Yup, this is expected behaviour. Ideally, the best thing would be for NatCorder to use only one RenderTexture for all calls to CommitFrame. The only problem with this is that it breaks support for offline recording (recording in a for-loop), and rendering is asynchronous so we would have to completely synchronize before allowing another frame to be committed, which will murder performance.

 

Where are you committing audio frames? Are you creating an AudioRecorder?

 

Unity added this as a way to perform a pixel data readback from a RenderTexture without having to block (readbacks are expensive because the GPU DMA is really only optimized for transfers in one direction: from sysmem to vmem).

 

On Metal, using GetTemporary introduced tearing. Now that we're doing something different, it'll be good to revisit it. Thanks for bringing this to my attention!

 

NatCorder will work with older Unity versions. The package was simply uploaded with the latest version of Unity.

 

All that makes sense, and thank you for your detailed explanation.

I think I didn't explained the issue well enough, though.

What I am experiencing, both on Windows and Android, is that RenderTextures actually gets never released.
Since rendering frame one, RenderTextures count is always increasing by one, every single frame, until everything explode. It's quite unlikely that a PC that can afford rendering 300+ fps cannot encode a single 720p frame in 3000-4000ms.
I tried to understand what was blocking them from being released, without any luck, until I discovered that I could workaround this issue by using temporary RenderTargets, so I stopped investigating further.

 

How are you computing the render texture count? If you are using GetNativeTexturePtr, it doesn't matter whether the pointers are monotonically increasing. The graphics driver could decide to return the same pointer as a previously released texture (as is usually the case on OpenGL ES), or it can just return a forever-incrementing pointer.

 

Hi, on Mac I am getting a DllNotFoundException when using NatCorder

DllNotFoundException: NatCorder
NatCorderU.Core.Platforms.NatCorderOSX..ctor () (at Assets/NatCorder/Plugins/Managed/Platforms/NatCorderOSX.cs:42)
NatCorderU.Core.NatCorder..cctor () (at Assets/NatCorder/Plugins/Managed/NatCorder.cs:111)
Rethrow as TypeInitializationException: The type initializer for 'NatCorderU.Core.NatCorder' threw an exception.

So right now my status is:
Windows: Works fine for 2 seconds, then I get 1fps. If I do stop before the two seconds though, I get a video without sound and the video is verticall flipped when playing it back with the Video Player component in Unity.
Mac: DllNotFoundException spam
iOS: Video playback is vertically flipped here as well, and memory increases every second until the game crashes after a 10-15s video. (Low memory warning is printed right before the crash).

I haven't tried Android yet, but I'm starting to want my money back as this is product is not usable anywhere in its current state.

 

I'm relying on Profiler

 

Update to NatCorder 1.4.0, and make sure you are running macOS 10.13 at least.

Are you using the render pipelines? Can you share your recording code?

 

I figured it out, it happens when I don't use a AudioRecorder, but still set the AudioFormat to be AudioFormat.Unity. I believe this is something you should fix in your API. It would probably be better to pass in the audiorecording class into the Videorecord class instead of having an enum to tell the system whether or not it is created because that is too losely coupled.

As for the vertically flipped video playback, it doesn't happen in the simple GameCam scene setup which I uploaded earlier (where you can also see this bug with the audio recording not being there creating this stuttering effect, because it actually lacks the code to create the audio recorder).
https://forum.unity.com/threads/natcorder-video-recording-api.505146/page-26#post-4174678

As for those that need an example on how to use the VideoRecorder for recording gameplay and and the game sounds and not the webcam and microphone, heres an updated gamecam script:

Code (CSharp):

1. using System;
2. using NatCorderU.Core;
3. using NatCorderU.Core.Clocks;
4. using NatCorderU.Core.Recorders;
5. using UnityEngine;
6. using UnityEngine.Events;
7.  
8. namespace NatCorderU.Examples
9. {
10.     public class GameCam : MonoBehaviour
11.     {
12.         [Serializable]
13.         public class VideoRecordedEvent : UnityEvent<string>
14.         {
15.         }
16.  
17.         [Serializable]
18.         public enum SelectableAudioFormat
19.         {
20.             None,
21.             Unity,
22.         }
23.      
24.         [Header("Recording"), SerializeField]
25.         private Container container = Container.MP4;
26.         [SerializeField]
27.         private Camera recordingCamera;
28.  
29.         [Header("Audio Format"), SerializeField]
30.         private SelectableAudioFormat audioFormat = SelectableAudioFormat.Unity;
31.      
32.         [Header("videoRecorded"), SerializeField]
33.         private VideoRecordedEvent videoRecorded = new VideoRecordedEvent();
34.      
35.         private CameraRecorder videoRecorder;
36.         private AudioRecorder audioRecorder;
37.         private IClock recordingClock;
38.  
39.         public void StartRecording () {
40.             // Create recording configurations // Clamp video width to 720
41.             var width = 720;
42.             var height = width * Screen.height / Screen.width;
43.             var framerate = container == Container.GIF ? 10 : 30;
44.             var videoFormat = new VideoFormat(width, (int)height, framerate);
45.             // Create a recording clock for generating timestamps
46.             recordingClock = new RealtimeClock();
47.             // Start recording
48.             var aFormat = audioFormat == SelectableAudioFormat.None ? AudioFormat.None : AudioFormat.Unity;
49.             NatCorder.StartRecording(container, videoFormat, aFormat, OnReplay);
50.             if (recordingCamera == null)
51.             {
52.                 recordingCamera = Camera.main;
53.             }
54.             videoRecorder = CameraRecorder.Create(recordingCamera, recordingClock);
55.          
56.             //Commenting out this line here will make the video stutter really badly after 2 seconds if you've set it to use AudioFormat.Unity
57.             audioRecorder = AudioRecorder.Create(FindObjectOfType<AudioListener>(), recordingClock);
58.             // If recording GIF, skip a few frames to give a real GIF look
59.             if (container == Container.GIF)
60.                 videoRecorder.recordEveryNthFrame = 5;
61.         }
62.  
63.         public void StopRecording () {
64.             // Stop the recording
65.             videoRecorder.Dispose();
66.  
67.             if (audioRecorder != null)
68.             {
69.                 audioRecorder.Dispose();
70.             }
71.             NatCorder.StopRecording();
72.         }
73.  
74.         void OnReplay (string path) {
75.             Debug.Log("Saved recording to: "+path);
76.          
77.             videoRecorded.Invoke(path);
78.         }
79.     }
80. }

 

But theres still the DLLNotFoundException happening on MacOSX (Sierra)

 

Figured out what was wrong with the flipped video. An older video player component (MediaPlayerCtrl) had flipped the RawImage uv coordinates in the y.

I'm also experiencing a weird bug where natcorder.isRecording is returning true when I press play in the unity editor after having recompiled my scripts. Does it really retain that state between sessions?

 

When recording is started, we have to provide information like video resolution and audio layout to the encoders up-front, which is reflected in StartRecording. This has nothing to do with how the actual video and audio frames are recorded. And none of these are enums.

So what could be causing it? What is your rendering setup like?

As I mentioned, NatCorder requires macOS 10.13 (High Sierra). Sierra is 10.12.

Cool.

Not at all. Are you calling StartRecording in some other script? And are you using the latest version from the Asset Store (this is critical)?

 

Hi guys,

I'm not sure what I'm doing wrong, but I can't seem to make Natcorder record my microphone on Android. It works okay on iOS, interestingly. On Android, whenever I try to make a recording with the microphone switched on, it just freezes. I checked the code and there seems to be a loop in the ReplayCam script that is never left if the microphone can't be accessed:

while (Microphone.GetPosition(null) <= 0) ;

I altered the script to check for the microphone for two seconds and just continue if it can't be accessed after that, with the result that now it can record video, but still has no access to sound. I did all this first with a slightly older version of Natcorder. When I updated to see whether this would fix the issue, I was surprised to find out that the newest version still has this exact same line, meaning that it still freezes the application indefinitely. For some reason, the new version also requires a higher Android version and turns my recordings by 90 degrees. I don't care which version I get to work in the end, but this issue has been bugging me for days now, and I have to resolve it soon.

Has anybody else experienced something like this? Is there a solution?

Thanks in advance,
Michael

 

This is a good observation. The expectation is that the 'Record Microphone' flag will only be enabled on devices that actually have a microphone, so that loop should be exited once the mic has started.

This sounds like a serious bug. The recording is rotated by 90 degrees? Does this happen with the ReplayCam example.

Haven't had any report of this. Make sure that your microphone's AudioSource doesn't have either 'Bypass Effects' or 'Bypass Listener Effects' enabled.

 

Well, my device is a smartphone, so it clearly doesn't lack a microphone. I checked and the Android manifest also contains the microphone use privilege. It just seems that the microphone can't be found or used.

I think I spoke too soon here. Apparently the issue isn't turning, but that the latest version of ReplayCam seems to be preset with dimensions that make it record more than the screen is actually showing. I'll try to describe the problem more accurate tomorrow. For the time being, microphone support is a more pressing issue for me.

I checked, and neither option is enabled.

 

Hello there,

I'm trying to use NatCorder with NatCam on iOS but I'm having some build issues and was wondering what the correct method to use both assets together was? I had to delete "libNatRender.a" from NatCorder's Plugins/iOS folder to prevent a Unity build error, however in XCode this results in a linker error:

Code (CSharp):

1. Undefined symbols for architecture arm64:
2.   "_NRMTLReadback", referenced from:
3.       _MTLContext_Readback_m12C3FBFD3E3FBED6C434C252C5295A1133DE92A0 in Bulk_Assembly-CSharp_0.o
4.       _MTLContext_Readback_mEEEDD7AD0F437D69AD4406403B2AAEE7B56F8BCA in Bulk_Assembly-CSharp_0.o

Is there a way to get the two assets to work together?

Thanks!

Edit: Seems to be the same issue as this user but with a different symbol (_NRMTLReadback instead of _NRGPUFenceSync) - I've tried deleting & redownloading/reimporting both assets without any joy.

Edit 2: Actually I get exactly the same error as the other user if I delete NatCam's libNatRender.a instead of the one in NatCorder:

Code (CSharp):

1. Undefined symbols for architecture arm64:
2.   "_NRGPUFenceSync", referenced from:
3.       _MTLFence__ctor_mD945145DE46BAB61848931056A28BE7947144BA0 in Bulk_Assembly-CSharp_0.o
4.       _MTLFence_GPUFenceSync_m2EE0F4FB66C2A60DFF29C433E46B16F3671B2451 in Bulk_Assembly-CSharp_0.o

I don't suppose there's a quick fix for this yet?

 

This would make it a bug in Unity's Microphone API then.

 

What version of NatCorder are you using? Download 1.4 from the Asset Store. The name of the symbol should be "_NRMTLReadback". Also, make sure to delete all the NatRender sources inside NatCam, not in NatCorder. This includes libNatRender.a and all the C# sources under NatCam > Plugins > Managed > NatRender.

 

Ah, thanks for this - deleting the NatRender folder from NatCam fixed the Xcode build error.

Unfortunately though, I now have another issue to report - whenever the recording of a GIF finishes, I get the following error:


This is building for iOS 11, and running on an iPad 2017 (9.7") with iOS 11.2.

This also happens if I make an empty project in Unity 2018.3.3f1 and build the Giffy example from NatCorder.

 

you were right, this issue was related to using the unity audio type, but not submitting any audio samples. thank you for the help!

it might be worth it to add some kind of debug log warning or something to help users debug this issue by themselves.

 

Hi we are using NatCorder to record gameplay videos for sharing and replay purposes in our game. It is a fast action game that runs on 60 fps on iOS and Android.

We managed to record and share the videos however on iOS (haven't tested on Android yet) we see some weird glitching during gameplay if recording is on. Reading the forums we thought it was the tearing effect that was mentioned a few posts above, however we turned off Metal on iOS and it still happens.

We record 1280x720 resolution video with sound on iPhone 6. We use Unity 2018.3.6f and updated NatCorder yesterday. If you like I can send you an email containing an example screen capture of the tearing effect along with our recording code.

Can you also tell us if it is possible to share the videos along with a specified text and maybe an app link.

We would appreciate any help on this issue.

 

I've just tried porting my Unity application to iOS and I'm getting an error:
ld: symbol(s) not found for archeitecture arm64.
clang: error: linker command failed with exit code 1 (use -v to see invocation)

And it's accompanied by a lot of references to NC and NatCorder method calls.
I'm not a mac expert but I would really appreciate any help. Does it need any configuration setup like the AndroidManifest?

 

I having trouble running the plugin on android,
I am using the replaycam scene and the build keeps crashing on android as soon I start recording

 

This is a known issue. The fix is coming in the next minor update.

 

Unfortunately, we can't do this. We can't dictate when the developer should and should not commit audio samples. We simply document the expectations and expect the developer to follow them.

 

Weird glitching in the recording? Or on the screen?

Are you using the latest version of NatCorder? NatCorder only supports Metal on iOS, so you shouldn't be able to disable Metal and still record.

NatShare, our social sharing API, doesn't support sharing both a video and text at the same time. You'll have to either modify NatShare to support this or use another sharing plugin. NatCorder doesn't have any sharing functionality; it is solely a video recorder.

 

Can you provide a screenshot of the error in Xcode? What specific symbol is not found?

 

Can you upload the full logs from logcat in a txt file? The logs will always contain the reason for any crash.

 

Ok, thank you - do you've any idea yet how long it might be until the update/fix is live? Are we talking days or weeks?

 

How do I get those logs? the build is running on an android device, I dont think It generates any log on the device
Thanks

 

Asset store review usually takes a few days.

 

Logcat.

 

Launch NatCorder on Android and
This message come out

2019-02-27 10:39:17.473 32016-32056/? E/Unity: NullReferenceException
at (wrapper managed-to-native) UnityEngine.RenderTexture.Release(UnityEngine.RenderTexture)
at NatCorderU.Core.Platforms.NatCorderAndroid+<>c__DisplayClass18_0.<onEncode>b__0 () [0x00000] in <4910071b4fe8420daa2883f7525bff0e>:0
at NatRenderU.Dispatch.MainDispatch.Update () [0x00058] in <4910071b4fe8420daa2883f7525bff0e>:0
at (wrapper delegate-invoke) <Module>.invoke_void()
at NatRenderU.Dispatch.DispatchUtility+<OnFrame>d__14.MoveNext () [0x00045] in <4910071b4fe8420daa2883f7525bff0e>:0
at UnityEngine.SetupCoroutine.InvokeMoveNext (System.Collections.IEnumerator enumerator, System.IntPtr returnValueAddress) [0x00027] in <c651486033b34c6aa0857cb621ef91a5>:0

what am i missing?

 

This is the paste of the error: https://pastebin.com/7UMsLJVe

It's also got errors from 8th Wall but I don't think it's really anything specific. Do I need to add extra frameworks? How do I do that?

UPDATE: I tried a few more things and redownloaded my packages and reimported them and restarted and what I now got is this: https://pastebin.com/PZx3eY7J

 

im interested to test

 

Can you upload the full logs from logcat in a .txt file? You shouldn't be getting this error.

 

It looks like you're using an older version of either NatCam or NatCorder. Upgrade to the latest versions on the Asset Store. That missing symbol hasn't existed for quite a while now.

 

We aren't beta testing anymore. Grab NatCorder from the Asset Store.

 

Hi! We have troubles with your last NatCorder plugin implementation. We are using NatCorder for RawImage (WebCamTexture) recording on Android. And there are many exceptions connected to "natcorder.Bridge" [java.lang.ClassNotFoundException: com.yusufolokoba.natcorder.Bridge]. Each recording frame = one exception. After several seconds of record app crashes without any reason in log. Some ideas??

 

Hi Lanre,

Your API is fantastic and I’m successfully recording ARKit sessions and saving them to my camera roll using NatShare!

One question I have is that I’m seeing aliasing, and a lack of sharpness compared to video recorded with the native iOS camera app. Is this a fundamental limitation of the way Unity works when recording to a RenderTexture? Here’s a comparison video using the ReplayCam example Unity scene with a bitrate of 40000000, framerate of 30 and resolution to 1920x1080:

https://drive.google.com/file/d/19ZZDiyXqCdWJ70YKSYTGYcGSG820jYXS/view?usp=sharing

The results are worse when recording an ARKit session - same Natcorder settings as above:



When it comes to NatCorder, is native camera quality even possible? I know video recording was removed from NatCam but was this originally able to access native quality frames from the camera as video? Do you have any tips on maximising quality with NatCorder?

Thanks and you've done a smashing job on the API - it's really easy to implement and I'm really enjoying using it.

Owain

 

Are you recording with audio? This is a known issue that will be fixed in the next minor update.

 

NatCorder doesn't have anything to do with the camera at all. Native camera quality will depend on the service providing camera frames--ARKit or WebCamTexture. Can you compare screenshots between the native camera and your app (screenshot of the app itself, not a recording of the app). The same quality difference should be reflected there.

Thank you!

 

Thanks. That worked. I think it's a problem with how third party packages are handled in Unity. It's a huge issue if you update your package and deprecate something, deleting older files, and when I try to update, old files remain unless I initially delete everything.

But why is the Screen struct gone? Where do I get default values?

 

Yes, we are recording with audio, but audio is not so important for us, we try to disable it.

EDIT: No [java.lang.ClassNotFoundException: com.yusufolokoba.natcorder.Bridge] exceptions after disabling audio recording. Thanks! But there is still another one mistake or bug -> it's not possible to record video on Android more than e.g. 20-30s (tested in example scene "ReplayCam"). After this time, app crashes everytime No reason in log, nothing. We tried it on 4 different Android devices -> the same result = crash..

EDIT2: Crash problem is probably connected with memory - after ~30s there is not enough RAM so the system stop the app activity. Any idea how to solve this?

 

Yup, that's why I always recommend deleting the API from your project before importing an updated build.

You're probably referring to VideoFormat.Screen. Don't use it as it creates needlessly high-res videos (that can add a lot of memory pressure to recording). Instead, create a VideoFormat with a preferred size.

 

What is your recording resolution?

 

Recording resolution is 720x1280 px. We tryied to move down the bitrate and the crash is now after ~1 minute..