NatCorder - Video Recording API

Hi There,
Is the Beta test still open? Can I join?

Thanks

 

Hi Lanre, is this update already on review for the Asset Store or still to be released?

 

Hi there. The beta isn't available anymore as NatCorder is now available on the Asset Store.

 

It has been submitted for review. I expect Unity to approve it within the next few days.

 

The update is live! Here is the NatCorder 1.1 changelog:

+ We have added a native macOS backend! The NatCorder recording API is now fully supported on macOS.
+ We have also added a native Windows backend! This backend is still experimental so it should not be used in production builds.
+ The Standalone backend (using FFmpeg) has been deprecated because we have added native Windows and macOS implementations.
+ We have significantly improved recording stability on Android especially for GPU-bound games.
+ Added support for different Unity audio DSP latency modes.
+ Added `sampleCount` property in `IAudioSource` interface.
+ Fixed crash when `StartRecording` is called on iOS running OpenGL ES2 or ES3.
+ Fixed rare crash on Android when recording with audio.
+ Fixed audio-video timing discrepancies on Android.
+ Fixed video tearing on Android when app does not use multithreaded rendering.
+ Fixed `FileUriExposedException` when `Sharing.Share` is called on Android 24 or newer.
+ Fixed `Sharing.GetThumbnail` not working on iOS.
+ Fixed `Sharing.SaveToCameraRoll` failing when permission is requested and approved on iOS.
+ Fixed `Sharing.SaveToCameraRoll` not working on Android.
+ Fixed rare crash on Android when a very short video (less than 1 second) is recorded.
+ Fixed build failing due to missing symbols for Sharing library on iOS.
+ Improved on microphone audio stuttering in ReplayCam example by adding minimal `Microphone` API in `NatCorderU.Extensions` namespace.
+ Refactored `Configuration.Default` to `Configuration.Screen`.

 

Hi Lanre, for some reason Unity thinks that I'm posting spam when I reply to our PMs, and won't actually let me reply, so I am replying here:

That was exciting news! I downloaded the update a little while ago and have been playing with it. If I build your ReplayCam scene alone, it works fine on my android device (pixel 2 xl). If I take the record button, preview, and replay cam object/script and move it into the scene for my app, it makes my app run choppy still (but less so than before) and after I record I only see a black screen. Is there some way I can resolve these issues? Thanks

 

Hey there. Can you email me a repro project? Do you have multithreaded rendering enabled in Player Settings? This is almost a requirement to make recording go smoothly. Regarding the black screen, can you also email me the full logs from logcat? I need to verify that your setup is correct.

 

Hi Lanre.
Congratulations on the release of new version!
I'm testing NatCorder 1.1 microphone recording with ReplayCam example.
When I record multiple times, only the first voice will be recorded in video.
It seems 'Extensions.Microphone' is keeping first recorded AudioClip.
Did you ever seen this issue?

 

PERPETUAL_CLIP is defined by default.
Therefore, previous 'clip' is not set to null and is reused when I call StartRecording.
I don't understand why reuse previous AudioClip.

Spoiler: Extensions.Microphone source code

Code (CSharp):

1. public static AudioClip StartRecording (string deviceName = null) {
2.     #if !UNITY_WEBGL || UNITY_EDITOR // No `Microphone` API on WebGL :(
3.     clip = clip ?? UnityEngine.Microphone.Start(deviceName, true, 60, 48000);
4.     while (UnityEngine.Microphone.GetPosition(deviceName) <= 0) ;
5.     #endif
6.     return clip;
7. }
8. public static void StopRecording (string deviceName = null) {
9.     #if !PERPETUAL_CLIP && (!UNITY_WEBGL || UNITY_EDITOR)
10.     UnityEngine.Microphone.End(deviceName);
11.     clip = null;
12.     #endif
13. }

 

We were testing this for microphone stuttering. You can simply comment out the PERPETUAL_CLIP definition and it should be fine. When the definition is enabled, NatCorder will keep the microphone running in the background. The reason why it plays audio from a previous time is because you also have to set the `AudioSource.timeSamples` to `Microphone.GetPosition(deviceName)` (this only applies when PERPETUAL_CLIP is enabled). We have added a fix to this on our end.

 

Hi, I purchased this asset today and have been playing with example scenes.
Generally, good work!
I'm trying to create my custom scene by modifying with ReplayCam scene and actually I already have some questions.

1. the volume of recorded audio is really low.How can I adjust this? I enabled "Record Microphone Audio" in inspector and other settings are default.
2. I cannot find recorded videos in my device. How can I save video in cameraroll?

asset ver: NatCorder 1.1f1
device: iPhoneX
os: iOS 11.2.6

Thanks!

 

This might be affected by your device volume. Also, make sure that the AudioSource in your scene has its volume set to full and that it is spatially close to the AudioListener in the scene. This is because Unity's OnAudioFilterRead provides audio samples from the 'viewpoint' of the AudioListener, not the AudioSource. So you have to account for volume attenuation.

Videos are saved to the app's private documents directory. To add them to the camera roll, use the Sharing.SaveToCameraRoll function (in the NatCorderU.Extensions namespace). Check the README.md file and the docs for more info.

 

Thank you for quick response.
I found out video saved in camera roll has enough volume. Only video in playback played with Handheld.PlayFullScreenMovie(path) has very low even after moving AudioSource closer to AudioListener. I will try to find the cause.
Anyway, your comment was helpful. Will play with this deeper tomorrow, thanks!

 

That sounds good. Maybe it's a Unity bug.

 

Hi Lanre,

Just saw NatCorder and it looks great.

I'm wondering if there a recording option to record the last n-seconds of gameplay? (like the Xbox Live record functions)

- Wes

 

So just making sure I understand:

• StartRecording with AudioListener for in-game audio.
• StartRecording with AudioSource (clip from NatCorderU.Extensions.Microphone.StartRecording) and audioIsRecordOnly true for microphone audio.

But how do I record the microphone AND the in-game audio at the same time?

 

Hi Wes. This isn't possible. NatCorder records directly to a file. One option you could explore is after recording a video, trim the video leaving only the last 8 seconds.

 

Yes, this is correct.

Recording from an AudioSource in the scene is not limited to just the Microphone. You can use it to record any particular AudioSource. The reason why we have the `audioIsRecordOnly` flag is because we usually don't want the microphone audio to play back in the scene (we just want to record it).

You will have to do some manual mixing! You will need to create a component that gets the microphone audio (using the OnAudioFilterRead callback on an AudioSource playing the mic audio) and saves it into some sample buffer; then mute the microphone audio so that the mic isn't heard in the scene; then add another component to the AudioListener that also implements OnAudioFilterRead for capturing scene audio; then mix the two sample buffers together and commit the result to NatCorder (using NatCorder.CommitSamples).

 

Hi Lanre,

First of all, great job at making this asset, I love it! Save me a lot of time tackling video recording problem.
I'm trying to record video with certain aspect ratio (say my camera render fullscreen 9:16 and I want to record a portion of it like 4:5), I use

Replay.StartRecording(worldCamera, new Configuration(width, height), OnReplay)

and it's actually output the video's resolution I wanted but the content stretched out to fit that resolution.
So my question is, is it possible to record video with the resolution I want without stretching its content?

 

Thank you @Lanre!
The new release has stopped all of my crashes. I still can not find the videos in my Camera Roll on Android though.
I am using Sharing.SaveToCameraRoll(this.pathToReplay);

 

Hi, i used your updated plugins.

-I tried to record a voice via microphone, its good and it works. Then, I tried to add a sample in-game sound when a button is pressed, the in-game sound is fine when i click the button but when I record the in-game sound, it becomes very different(The normal sound is something like a "swoosh-sound", but in the replay video, it became "TSAAAAA-sound" ), this is when i'm using Samsung S7 Edge. I tried it with another device(HUAWEI P9), and in that device, it's just fine. Any idea why it's like that?

-I tried to record an in-game sound sample without microphone by unchecking the RecordMicrophoneAudio in ReplayCam.cs, but it seems i can't hear any in-game sound in the video when it is being replayed and the file also don't have the in-game sound.

 

Have been using this plugin on iOS for about a week and a half with no issues. Building an AR app using Vuforia 7 and ground plane tracking. Most of my testing has been on the 6S and iPhone X and I have not had issues. However on the iPhone 7 Plus after a few seconds of recording I can see the framerate of the application drop drastically and xcode will throw the following error:

Code (csharp):

1. 2018-03-28 15:43:45.660075-0300 gowpoc[1526:1117525] -[MTLRenderPipelineDescriptorInternal validateWithDevice:], line 2400: error 'No valid pixelFormats set.'
2. -[MTLRenderPipelineDescriptorInternal validateWithDevice:]:2400: failed assertion `No valid pixelFormats set.'

has anyone else ran into similar issues?

 

This is a fair observation. Because the `Replay` API uses Graphics.Blit, which basically just draws a fullscreen quad, a difference in aspect ratio will result in stretching. To resolve this, your two options are to: make sure that the aspects match; or modify ReplayCam's VideoRecorder component to blit with a material that corrects for the aspect difference (I'm exploring this option for the next update).

 

Can you email me with your code? What Android device and OS do you use?

 

I'm not sure why this could be happening. It will be useful if you can send me the video.

The ReplayCam example doesn't record game audio. It either records microphone audio (when the flag is enabled), or no audio at all.

 

Can you email me the full logs from Xcode?

EDIT: I did some digging and it looks like a Unity bug. See this and this.

 

Thank you for the reply. I am using a Samsung Galaxy S7 Edge running Android 7.0

 

Hi Lanre,

I sent you email about screen overlapping issue on iOS.
Please check your email and reply me.

 

Hello,
I'm trying to use the NatCorder start and stop recording. It works pretty fine on Unity but when I launch it on ipad mini I got this error :

No more log or error about NatCorder between start and stop, so is there any way to know why NatCorder didn't start record ? Is there any known issue about NatCorder on ipad ?

Have a good day

 

Got it. I'll respond soon.

 

Can you enable verbose mode (NatCorder.Verbose = true), start and stop recording, and send me the logs? This might be happening because the managed layer isn't properly forwarding calls to the iOS implementation. It should be a trivial thing to fix. Shoot me an email with the logs and we'll go from there.

 

Thanks for your reply.
Found the answer, it was really weird and completely my fault. I use a personnal tool to build my unity projet to ios, and it erase the NatCorder's build variables. So it creates this weird issue...

 

That sounds good. Thanks for letting me know.

 

Hi @Lanre

My app crashes when I try to record an AR scene with NatCorder. The device is iPad Pro 12.9", with iOS 11.2.6.

Here's the log:

Code (CSharp):

1. NatCorder: Initialized NatCorder 1.1 iOS backend
2. UnityEngine.EventSystems.ExecuteEvents:GetEventHandler(GameObject)
3. UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
4. UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
5. UnityEngine.Debug:LogErrorFormat(Object, String, Object[])
6. UnityEngine.Logger:Log(LogType, Object)
7. Newtonsoft.Json.Utilities.CollectionWrapper`1:System.Collections.IList.Insert(Int32, Object)
8. UnityEngine.Debug:Log(Object)
9. NatCorderU.Core.Platforms.NatCorderiOS:.ctor()
10. NatCorderU.Core.NatCorder:.cctor()
11. UnityEngine.EventSystems.ExecuteEvents:GetEventHandler(GameObject)
12. NatCorderU.Core.Replay:StartRecording(Camera, Configuration, VideoCallback, IAudioSource)
13. SurrealKit.MainSceneManager:StartRecording()
14. UnityEngine.EventSystems.ExecuteEvents:GetEventHandler(GameObject)
15. System.IComparable`1:CompareTo(T)
16. System.IComparable`1:CompareTo(T)
17. System.IComparable`1:CompareTo(T)
18. (Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)
19.  
20. NatCam Dispatch: Initialized main dispatch
21. UnityEngine.EventSystems.ExecuteEvents:GetEventHandler(GameObject)
22. UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
23. UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
24. UnityEngine.Debug:LogErrorFormat(Object, String, Object[])
25. UnityEngine.Logger:Log(LogType, Object)
26. Newtonsoft.Json.Utilities.CollectionWrapper`1:System.Collections.IList.Insert(Int32, Object)
27. UnityEngine.Debug:Log(Object)
28. NatCamU.Dispatch.MainDispatch:.ctor()
29. NatCorderU.Core.Platforms.NatCorderiOS:StartRecording(Configuration, VideoCallback, IAudioSource)
30. NatCorderU.Core.NatCorder:StartRecording(Configuration, VideoCallback, IAudioSource)
31. NatCorderU.Core.NatCorder:StartRecording(Configuration, VideoCallback, IAudioSource)
32. NatCorderU.Core.Replay:StartRecording(Camera, Configuration, VideoCallback, IAudioSource)
33. SurrealKit.MainSceneManager:StartRecording()
34. UnityEngine.EventSystems.ExecuteEvents:GetEventHandler(GameObject)
35. System.IComparable`1:CompareTo(T)
36. System.IComparable`1:CompareTo(T)
37. System.IComparable`1:CompareTo(T)
38. (Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)
39.  
40. -[MTLDebugCommandBuffer waitUntilCompleted]:103: failed assertion `waitUntilCompleted on uncommitted command buffer'
41. (lldb)

I'll be very glad if you point me in the right direction.

Thanks!
Ahmet

 

What is the memory usage of NatCorder? For example, for a 10 minutes video, does it keep all 10 minutes in memory or does it commit to disk as it records? If it does commit, what is the memory when processing the video when the video ends? IS there an example you could give that would give an idea of memory usage? I would like to use this for a mobile game. Thank you!

 

Thank Lanre, I actually went to the first approach and it worked the way I wanted it to. It would be cool if you can include this feature in the future update

 

Hello,
We compile and run latest version on iOS 10.2, and exception after recording file as:

FileNotFoundException: file:///var/mobile/Containers/Data/Application/5FC9D511-23CB-4232-B6C6-877EAE999BFE/Documents/recording_2018_03_30_15_59_59_584.mov does not exist
at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00000] in <filename unknown>:0
at ProGifWorkerSync.OnVideoRecorded (System.String path) [0x00000] in <filename unknown>:0
at NatCorderU.Core.Platforms.NatCorderiOS+<OnVideo>c__AnonStorey1.<>m__0 () [0x00000] in <filename unknown>:0
at System.Collections.Generic.List`1[T].ForEach (System.Action`1 action) [0x00000] in <filename unknown>:0
at NatCamU.Dispatch.DispatchUtility+<OnFrame>c__Iterator0.MoveNext () [0x00000] in <filename unknown>:0
at UnityEngine.SetupCoroutine.InvokeMoveNext (IEnumerator enumerator, IntPtr returnValueAddress) [0x00000] in <filename unknown>:0
at Omanju.ByteClap.Processors.IFixedLengthElementArrayProcessor`1[TElement].GetElementSize () [0x00000] in <filename unknown>:0
at Omanju.ByteClap.Processors.IFixedLengthElementArrayProcessor`1[TElement].GetElementSize () [0x00000] in <filename unknown>:0
at Omanju.ByteClap.Processors.IFixedLengthElementArrayProcessor`1[TElement].GetElementSize () [0x00000] in <filename unknown>:0
UnityEngine.UnhandledExceptionHandlerrintException(String, Exception)
UnityEngine.UnhandledExceptionHandler:HandleUnhandledException(Object, UnhandledExceptionEventArgs)
Omanju.ByteClap.Processors.IFixedLengthElementArrayProcessor`1:GetElementSize()
Omanju.ByteClap.Processors.IFixedLengthElementArrayProcessor`1:GetElementSize()
Omanju.ByteClap.Processors.IFixedLengthElementArrayProcessor`1:GetElementSize()

It seem recording successfully, when call OnVideo with the recorded path, it is not existing path to video ?

 

hi @Lanre thx for the update. great update.

@ahmetardal , we have the same crash problem...

@Lanre this is pretty important for us, can you prioritize this crash. thx

 

Thanks for this report. I have created a fix for the issue. I can send you the updated native library if you'd like (email me).

 

NatCorder only uses as much memory as there are frames waiting to be encoded by hardware. Typically, this amounts to no more than 25MB, and we are still looking for ways to reduce it. Encoded chunks are committed to device storage as they are produced by the encoders (it is managed by the OS).

 

I think we will include this in the next release (1.1f2).

 

Hi. This exception is generated by your code (you are trying to move the recorded video after recording). The crash is probably happening because on iOS, you have to take special care with the protocol that is prepended to the returned path ("file://"). When using System.IO API's, you must remove the protocol.

 

Shoot me an email. It's a minimal issue and I have a fix for it.

 

@Lanre

Thanks! Great!
I've just emailed you.

 

Responded!

 

Thank you,
We have get localpath on new Uri(path).localPath, then it work
Best regards

 

Sounds good. Thanks for letting me know!

 

Hello,
When will window backend be official ?
Do all platform support 4k recording ?

Best regards

 

Probably in the next release (1.2f1).

Practically, no. Most mobile devices do not have enough computational resources to support 4K recording. On standalone platforms (macOS, Windows), it should be possible.

 

Hello,
On iPhone 6S, it can record at 4k easily, and latest as iPhone 7/8/X much faster. It is very powerful hardware. Please add support for 4k, and we will limit max resolution in our code. for example: on iPhone 6, will allow max 720p, and newer is larger.

Best regards