NatCorder - Video Recording API

Do you have a NatCorder call example?

 

Oops, you're right. It should be created right before StartRecording and passed in. I will be releasing an article about the new release today.

 

Can you email me? It's a huge bummer that ARCore doesn't support multithreaded rendering. I can send you a custom Android library that removes some synchronization, which might improve the app framerate.

 

The encoder configuration and implementation differences are the only thing I can think of when it comes to Win7. From the stack trace, the crash occurs when the video writer is releasing, so I'm not surprised that the video records properly.

The WebGL implementation is agnostic of the containing OS running the browser; it uses the browser's video encoder implementation (see the MediaRecorder API).

The WebGL backend is experimental because it doesn't properly conform to the NatCorder implementation spec. NatCorder backends should always support recording videos with the H.264 codec (producing MP4/MOV files) and should support recording audio with video--both of which NatCorder WebGL does not support. The reason NatCorder WebGL does not support the H.264 codec is because most browsers don't; instead, most browsers support VP8/WEBM. And the reason NatCorder WebGL doesn't support audio recording is that Unity doesn't support the OnAudioFilterRead callback on WebGL, which is the only way to extract game audio for recording. I should note that technically, you can still record a video with audio with WebGL, as you can get the audio data from an AudioClip or procedurally generate it then pass it to NatCorder; you just can't record game audio directly.

 

Hey Lanre,

Great plugin, and thanks for the active ongoing support on here.

We're experimenting with this plugin and finding that it works fine with ARKit on iOS 11.3 iPhone 8+ w/ Unity 2018.1 but as soon as we add our own content, the encoded frames start appearing out of order.

To try to debug, if we add a TextMesh and add it to our render texture and increment it every time we acquire and commit a frame it might appear like (as an example)
1 2 3 4 5 6 7 3 4 10

where frame 8 and 9 are missing and substituted with one of the previous frame in the encoded video. Frame rate in the actual app is running at stable 60 FPS.

I've tried downscaling configuration to 180x320 just for kicks, or changing the record frame rate to 5 FPS and it seems like it helps a little, but even in those extreme cases it happens occasionally.

Not doing anything crazy as far as I can tell:

private static readonly int VIDEO_FPS = 30;



void HandleButtonDown() {

    NatCorder.StartRecording(new Configuration(1080, 1920, VIDEO_FPS),

       OnRecordingComplete, this);

}



void Update() {

    if (NatCorder.IsRecording && (Time.time - lastCommitTime > 1.0f / VIDEO_FPS) {

        Frame frame = NatCorder.AcquireFrame();

        Graphics.Blit(deviceCamTex, frame);

        NatCorder.CommitFrame(frame);

        lastCommitTime = Time.time;

        ++frameCount;

    }

}

Can also confirm the CommitFrame is being called with each frameCount based on logs. I imagine if the hardware is too taxed, it could take longer to write, but committed frames not appearing in the encoded video at all could be a bug (or we're doing something dumb). Do you have any a guidance? I see no one on this thread has seen something like this. Our content is another native plugin that happens on a separate thread (using Metal) that also needs to decode a h264 video so maybe it's some resource contention on the native side?

 

Gotcha. Is there a way to fix that? We don't need to target Windows 7 with this app, but I would love to be able to actually work on the recording side of things in the editor instead of having to deploy to Android to test every little change I make. (Although it would still be nice if there was a way to make it generally work on Windows 7. It still has a huge market share after all.)

Hm. I might be doing something wrong then; my video records under Android, but doesn't under WebGL. It creates a webm blob, but that webm blob is either empty or defective - I get the Chrome video player interface, but it doesn't play when I open its URL in the browser.

I also have two problems on Android:

Firstly, the video quality is quite good, but the sound quality fluctuates between "good" and "stuttering, with annoying audio artifacts". Is that likely a Unity Microphone problem?

Secondly, in my app, I record a video with NatCorder and afterwards I play it back with the Unity VideoPlayer component. Most of the times, that works fine, but sometimes (maybe ~5% of all recordings) the Unity VideoPlayer on Android plays only the video and not the audio component. Here are my findings:

• A "defective" video consistently doesn't work with the VideoPlayer on Android. If I load another video, its audio plays fine; if I then load the "defective" video again, no audio plays again.
  
• It works everywhere else. If I open the video in any of the following things, it will play with both video and audio:
  • An Android video app.
  • Handheld.PlayFullScreenMovie() on Android.
    
  • VLC media player on Windows.
  • The Unity VideoPlayer in the Windows 7 Unity editor.

In summary, the audio part of those "defective" videos is obviously recorded (because it plays everywhere else), but something is obviously different in them (because usually the audio part plays fine in Unity VideoPlayer, except for those "defective" videos).

I assume it's either a really weird encoding problem in NatCorder (because most of the recordings work, but some don't) or a really weird decoder problem in VideoPlayer on Android.

 

Regarding Vuforia + NatCorder
Is Replay only available on iOS via ReplayKit or is it iOS/Android

 

I'd like you to isolate this issue, so don't do decoding and just use the TextMesh as you have in your test above. The only thing that pops out is your blit in Update. In my experience, this is touchy especially on Metal. If you take a look at the VideoRecorder sources (VideoRecorder.cs), we perform the blit in OnRenderImage. You typically want to do the blit in either OnRenderImage or at the end of a frame (`yield return new WaitForEndOfFrame`). Try this and let me know if you still get out-of-order frames.

 

There isn't a fix that we know of. As I mentioned previously, it is likely an implementation detail in the Win7 libs.

Have you tried using a different browser, like Firefox? Also, what version of Chrome?

Unity's Microphone API is not suited for realtime recording, and as such produces weird artifacts in recorded audio. Unfortunately there isn't an alternative Microphone API for Unity. We are considering creating one.

I strongly believe that it is the Unity VideoPlayer at fault here. You can use the mediainfo command line tool to assess a normal video and a 'broken' one. If NatCorder's audio recording is indeed broken, then you won't see any information listed for an audio track.

 

Not at all. The Replay API is just a helper class that builds on the lower-level NatCorder API. As such, it works on all platforms that NatCorder supports.

 

It didn't happen with our own decoder, so I believed it was related to performance but moving that to a coroutine seems to have resolved it. Thanks!

I was going by code snippets in the README file which was acquiring and committing the frames in an Update call, this might be a useful thing to clarify in the README, I see that the VideoRecorder does it the correct way. Get a feeling the black video thumbnail (due to first frame being black) is related to people calling Blit in Update() as well since we don't see this anymore either.

 

Hi Lanre.

I have released a simple example integrating NatCorder, NatShare and OpenCVForUnity.
https://github.com/EnoxSoftware/NatCorderWithOpenCVForUnityExample
It works well even when the camera preview API changed from WebCamTexture to NatCam.

 

Was using NatCorder on another project and for some reason when I was calling NatCorder.StopRecording(); the callback I had set up wasn't working. After looking into in android he was creating the video file but I could open it. So something when wrong after the stopRecording.

Anyhow, I got around and solve it but I'm posting because I wish you guys added a callback for error so this situation could have been more easily handed.

 

What I don't understand is why Windows 8 is the minimum version for NatCorder Is it not possible to use the implementation of the Win7 libs properly? (I'm wondering so insistently because this makes NatCorder useless for desktop games. Any developer targeting Windows/Mac would lose about 32% potential market share by locking into Win8+ judging by those statistics. This is very frustrating because I like NatCorder so far and I want to be able to use it in other projects, and it's not like there's a lot of alternatives.)

It works in Firefox. Sadly the Unity VideoPlayer refuses to play it despite accepting other URLs. If you can't see any reason for that, I guess I'll have to file another Unity bug report.

The Chrome version is 67.0.3396.99 (64 bit).

Ah. That would be fantastic. And you say that it's not suited for realtime recording - is it possible to record the whole audio sample and then add it to the video when the recording is stopped?

You're right. I already got confirmation for my bug report. As far as mediainfo is concerned, the 'broken' video has an audio track (which doesn't surprise me, because it plays everywhere else).

I also have discovered a new Android NatCorder problem. I call NatCorder.StopRecording(), but my videoCallback is never called and the video file is created, but can't be played. I'm attaching two LogCat log examples.

 

You raise a good point about the black frame being related to calling Blit in Update. But the issue of getting artifacts when recording a camera in Update is related solely to recording a camera. Our GreyWorld example records a WebCamTexture in Update and it works perfectly. I'll add a note about this in the README.

 

I gave it a quick scan; It looks great!

 

Can you send us the logs? This should never happen, so I'm interested in knowing what went wrong. The reason why we decided not to add an error callback is that if something went wrong during recording, then 9 times out of 10 the entire app would crash (and your code wouldn't be able to handle it). Since this practically never happens, we didn't add it.

 

I understand. I moved on and don't have the log because I'm running on a deadline here but thanks for the reply.

On another note, I'm struggling a but to get audio from the microphone right. Is there any example of using unity's Microphone class?

This is my basic setup:

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using NatCorderU.Core;
4. using UnityEngine;
5.  
6. /**
7. * We attach this script to an AudioSource/AudioListener
8. * and forward the audio to NatCorder
9. */
10. public class WebcamAudio : MonoBehaviour, IAudioSource
11. {
12.    
13.     // Sample rate is simply Unity's audio sample rate
14.     public int sampleRate { get { return AudioSettings.outputSampleRate; }}
15.    
16.     // Sample count is the number of audio samples per channel in one sample
17.     // buffer that gets committed. This must equal `samples.Length / channels`
18.     // from the `OnAudioFilterRead` function. I hard coded 1024 because this is
19.     // the default value of `samples.Length` with default DSP buffer
20.     // size (in Audio Settings)
21.     public int sampleCount { get { return 1024 / channelCount; }}
22.  
23.     // Channel count is the number of audio channels. I hard code this because
24.     // Unity's default audio output mode is stereo, so two channels
25.     public int channelCount { get { return 2; }}
26.  
27.     private AudioSource _source;
28.  
29.     public void Init()
30.     {
31.         _source = GetComponent<AudioSource>();
32.        
33.     }
34.  
35.     public void StartRecording()
36.     {
37.         var totalTimeInSec = 60 * 10;
38.         _source.clip = Microphone.Start("Built-in Microphone", false, totalTimeInSec, 44100);
39.         _source.Play();
40.     }
41.  
42.     public void StopRecording()
43.     {
44.         _source.Stop();
45.         Microphone.End("Built-in Microphone");
46.     }
47.  
48.     void OnAudioFilterRead (float[] samples, int channels)
49.     {
50.         // Create a timestamp for the audio
51.         long timestamp = (long) AudioSettings.dspTime;
52.         // Send to NatCorder for encoding
53.         NatCorder.CommitSamples(samples, timestamp);
54.     }
55.    
56.     public void Dispose ()
57.     {
58.         // Invoked by NatCorder when recording is stopped
59.         // You might want to destroy the instance here
60.     }
61. }
62.  

I call StartRecording and StopRecording before callings NatCorder's Start and Stop so the mic starts before the video.

I also ahd to cast AudioSettings.dspTime to long since NatCorder's CommitSamples needs it as that.

 

How to record both UI and camera?

 

Check out the AudioRecorder class in NatCorder > Plugins > Managed > Replay. And see how it is used in Replay.cs. If you are recording using Replay.Start/StopRecording, then you don't need to make your own class for picking up game audio; you can simply pass the AudioSource playing the microphone clip to Replay.StartRecording.

 

Make sure your UI canvas is not in Overlay mode, as this will make the canvas invisible to NatCorder.

 

We simply are not able to get it to work on Win7. We have done a lot of searching, trying to find any clues as to the crash and all we can come up with is that it must be an implementation detail in the Win7 libraries (out of our reach). As has been observed, recording itself completes normally. The crash occurs as MediaFoundation is tearing down its resources, which we don't control.

You can record the entire audio sample and commit it to NatCorder, but not after StopRecording is called. With NatCorder, you can commit video and audio frames whenever you want, provided that NatCorder is in the recording state and that your frame timestamps are monotonically increasing; you don't have to commit immediately you get data. Unfortunately, this doesn't protect you from Unity's microphone glitches which happens sporadically.

Sounds good.

What device and OS version are you recording on? Try recording without video, and recording for longer to see if that helps. It seems that the device isn't properly handling the video track's EOS (end of stream).

 

Oh, okay. That's too bad. Thanks for the explanation!

Alright. I'll try this to see if it'll improve the audio quality.

It's a Sony Xperia Z3 Compact with Android 6.0.1. Sorry I didn't make that clear earlier: Most of the time, the recording finishes properly on the device, but sometimes (and indeed so far only when the recorded video is very short, e.g. under 5 seconds) it fails as described.

 

Not exactly sure what's going on here, but we're improving state handling in the next version so it might help.

 

Is there any way I can deal with it for now, except manually implementing some kind of timeout?

I tried to reproduce the problem, and considering that it happened rather often when I recorded my short videos, it's proving to be surprisingly hard. I made 20 videos each with 60 sec, 30 sec, 15 sec and 5 sec, and every single one was successful, except #19 (of 20) of the 60 sec videos (I'm attaching the log of that one).

What happened twice (the first time with that failed #19 of the 60 sec video, the second time while the phone was just sitting there with the app open after I recorded lots of 30 sec videos) was that I got a ton of "concurrent mark sweep GC" messages. The second time that also crashed the phone with a "JNI ERROR (app bug): global reference table overflow". I'm also attaching a log for that. It sounds like some kind of memory problem.

 

Hello,

We noticed that when we are recording with microphone on iOS 11 occurs larsen effect on saved file. Microphone recording played in AudioSource Unity3D is clear without any defects. Have you noticed the same problem?

Thanks!

 

Given the information you've just presented, no. This issue looks pretty unpredictable to me.

The concurrent mark sweep GC message is just the Android runtime performing GC. As for the global reference table overflow issue, it looks like your code is accessing a NatCam device camera on each frame. Can you share your code snippet?

 

Do you have 'Prepare for recording' enabled in Player Settings? Also, do you have the ForceToSpeaker library installed?

 

Hey i have been using natcorder and i am using the replay class to record the footage but the UI is not appearing on the recorded video, is this a thing of the replay? do i have to use other things to record the ui too?
I am recording that has the culling mask to everything.

 

Your UI canvas is probably in Overlay mode. Overlay canvases don't get picked up by normal renders or blits. Check the README for more information.

 

I'm trying to use Natcorder with Unity's new Lightweight Render Pipeline in 2018.2, but it's crashing the app whenever I call Replay.StopRecording() on iOS devices. I'm having trouble debugging because it crashes the Editor as well. Any advice?

 

Can you email me the logs from the editor crash? I expect it to be more descriptive than the iOS logs.

 

You're right; I was a bit worried that it happened more often towards the end of the log, especially without me doing anything in the app, but studying the log I realize that it's not the case. Sorry for the false alarm.

Sure:

Code (CSharp):

1. void LateUpdate()
2. {
3.   if (!NatCorder.IsRecording)
4.     return;
5.  
6.   // FrameCounter gets incremented in NatCam.OnFrame
7.   var currentFrame = webCam.CameraSource.FrameCounter;
8.   if (currentFrame <= previousFrame)
9.     return;
10.  
11.   previousFrame = currentFrame;
12.  
13.   var frame = NatCorder.AcquireFrame();
14.  
15.   // texture is a reference to NatCam.Preview
16.   Graphics.Blit(texture, frame);
17.  
18.   NatCorder.CommitFrame(frame);
19. }

Additionally to that, I have a RawImage that has NatCam.Preview as its texture, and I call NatCam.IsPlaying and
NatCam.Camera.PreviewResolution every frame. (The latter is only actually needed when the PreviewResolution changes because the device rotates, but I'm not sure how to best track that. I assume that Input.deviceRotation could change, but the NatCam.Camera.PreviewResolution could still be the old value for e.g. a frame?)

 

G'day! Just wondering, did you finish up those changes to the audio capture system? I haven't found anything in this thread but a while ago you mentioned there were changes coming soon.
Thanks,
Pete

 

This is probably the cause. Don't manually track NatCam.Camera.PreviewResolution. NatCam will always trigger the OnStart event when the app orientation changes, even if the new resolution is the same as the old one. Also using DeviceCamera.PreviewResolution has the limitation of not respecting app orientation. So if the app is in portrait, you would still get [larger]x[smaller] instead of [smaller]x[larger]. NatCam.Preview.width/height tracks this difference.

 

Hi Pete. We are submitting the new version tomorrow. I have been sick for the past week, so it has really slowed me down. Thankfully, the new version is complete.

 

Thanks! I’ll keep an eye out for it. Look after yourself

 

Alright. Thanks for the info, that helps a lot!

But... why does it crash at all? Is it possible that something isn't freed correctly? Looking at the log, we have lots of references to com.yusufolokoba.natcam.NatCamDevice and almost the same amount to int[] and java.lang.Integer. I can manage without calling DeviceCamera.PreviewResolution regularly, but I'm worrying that there is some kind of reference leak in general and that it would just crash at some point in the future if references aren't freed.

I finally got around to try this, but it seems that on Android after calling NatCorder.StartRecording with an IAudioSource, the app crashes if I don't call NatCorder.CommitSamples regularly.

I'm attaching two logs showing the problem:

1. In the first one, NatCorder.CommitSamples is called in OnAudioFilterRead and a 30 second video is recorded fine.
2. In the second one, NatCorder.CommitSamples is not called anywhere and the app slows down to a crawl after 10 seconds and crashes a few seconds later.

 

When passing a non-primitive object from Java to C through the JNI, the Java runtime has to keep a reference to the object so that the object doesn't wrongly get garbage collected. There is a fixed-size table for storing such references, and overflowing it by accumulating a ton of references will cause a fatal crash.

This is by design. And just to correct your observation, the app will crash if you never call CommitSamples. Calling it even once will prevent this crash. I won't bother you with the explanation of what's actually going on, but you must commit something if you specify that something will be committed. Similarly, if you start recording without committing a single video frame, your app will crash.

 

I see. I have to admit that I don't know much about the topic. From what I found on google when I researched "global reference table overflow", it seems that global references have to be manually released somewhere (e.g. mentioned here). What I especially don't know is who should be doing the releasing. NatCam? Unity? Considering the number of global references to com.yusufolokoba.natcam.NatCamDevice, int[] and java.lang.Integer it seems that over 5000 frames worth of global references weren't released.

Oh! I guess I can work with that. So the way to go would be to commit once as soon as I have the first sample from the microphone, and then everything else just before I stop NatCorder?

 

We might explicitly have to release the references (Unity's AndroidJavaObject:ispose function does this). I just have to check.

Correct; this is the workaround.

 

Lanre, we are trying to record an ARKit experience to an .mp4. We purchased your plugin, and can run the test app and record the camera, but can't seem to get it to work in our map to record the ARKit experience. Ultimately, we want to crop the screen and make it a certain resolution, but first, we need to get it recording the screen with the ARKit face tracking on top. Can you point us in the right direction?
thanks,
Dale

 

Hi Dale. Can you be more descriptive as to how you can't get it to work? What exactly is happening? Can you share logs?

 

Lanre, we just got it to work using the ReplayAPI. Will we be able to crop it so that we are not recording the full iPhoneX screen, but a square 1080 x 1080 image?

 

Also, is there a way to grab a still frame of the result or, even better, stream the result to a PC that is "controlling" the iPhone?

 

Is there a way to tell the video file where to go once it is recorded, or tell it to move to a specific address after it is saved to the default location?

Thanks!

 

To crop the recording, you will need to switch to the lower-level NatCorder API. Using the GreyWorld as a start, you'll need to manually blit each game camera frame to NatCorder's encoder frames (AcquireFrame-Blit-CommitFrame). When blitting, you can use a custom shader to do the cropping. You'd have to write the shader to manipulate the UV's to crop the image.

You can use NatShare to get a thumbnail from the recorded video. There is no way to stream to a PC.

 

You can't define where the video gets saved, but you can use the System.IO.File.Move function to move the file once it's recorded (in the RecordingCallback).

 

Does NatCorder have the zxing or QR code reader implemented as in the previous NatCamera packages?

 

Where could I find reference for this?