NatCorder - Video Recording API

What platform are you running on? Windows? If so, the path separator is a back-slash, not forward slash. To be safe, use `Path.PathSeparator`. Secondly, you are creating a directory with the name and extension of the video (this is the last component of the `temp` array after splitting). And in your File.Move call, the second parameter is the new directory path with the name+extension of the file, not just the new directory. Check out the documentation for File.Move and other functions you are using.

Also, why do you have a `new WaitForSeconds(5)`? If your intuition is to wait for the video file to be there, then this is not necessary because NatCorder guarantees that when the VideoCallback is invoked, the video file is present and writing is complete.

 

Thanks for being so active in responding to user questions! It must be a full-time job for you!!

The error I am currently facing:
When the user presses the save video button... "Sharing.SaveToCameraRoll(_path);" works perfectly on Pixel 1, but instantly crashes the app on IphoneX.

Any thoughts?
Thanks

 

On the iPhone X, it crashed 100% of the time, several seconds into recording with Configuration.Screen. If I reduce the size to 1080 it lags a little but doesn't crash.

There is no crash log generated, all I see in Xcode debug is:

Code (CSharp):

1. Message from debugger: Terminated due to memory issue

So I guess iOS is force quitting it because of memory?

UPDATE: Reducing the video size on the iPhone X will have to be the fix. Trying to record any video above 1080 seems to be a memory issue. Which makes sense, plus 1080 video is perfectly fine.

 

We didn't make any changes to the Android backend in 1.2. Can you make sure that you have 'Multithreaded Rendering' enabled in Player Settings? It makes a world of difference in recording performance.

 

You seem to be using an older version of NatCorder. Please upgrade to 1.2 because we removed the Sharing API and made it into a separate project, NatShare. Make sure to post any sharing-related issue on the NatShare thread.

 

Yes, the iPhone is definitely choking on >1080p. The H.264 codec doesn't do well with resolutions greater than 1920x1080, and since NatCorder doesn't enforce any restrictions (it trusts that the developer is being safe), recording with Configuration.Screen on devices with very high-res displays can be troublesome. I recommend recording at more economical resolutions, like 1280x720, 1600x900, or 1920x1080.

 

I needed to update Unity to the beta 2018.2.0b3 because it had a feature we needed, and I'm now coming across this error which seems to be NatCorder related. I am trying to build for Android, but the error seems to be looking at the iOS plugins.

How do I fix this?

I did make sure to get the latest update from the Asset Store (and deleted the old one before importing it) I actually had to do that with a lot of Assets after I upgraded to this beta (From 2017.2 iirc)

Exception: Unknown CPU architecture for library Assets/NatShare/Plugins/iOS/libNatShare.a
UnityEditor.Android.PostProcessor.Tasks.NativePlugins.ProcessPlugin (UnityEditor.Android.PostProcessor.PostProcessorContext context, System.String pluginPath, System.String pluginTargetCPU)
UnityEditor.Android.PostProcessor.Tasks.NativePlugins.Execute (UnityEditor.Android.PostProcessor.PostProcessorContext context)
UnityEditor.Android.PostProcessor.PostProcessRunner.RunAllTasks (UnityEditor.Android.PostProcessor.PostProcessorContext context)
UnityEditor.Android.PostProcessAndroidPlayer.PostProcess (BuildTarget target, System.String stagingAreaData, System.String stagingArea, System.String playerPackage, System.String installPath, System.String companyName, System.String productName, BuildOptions options, UnityEditor.RuntimeClassRegistry usedClassRegistry, UnityEditor.Build.Reporting.BuildReport report)
UnityEditor.Android.AndroidBuildPostprocessor.PostProcess (BuildPostProcessArgs args, UnityEditor.BuildProperties& outProperties)
UnityEditor.PostprocessBuildPlayer.Postprocess (BuildTargetGroup targetGroup, BuildTarget target, System.String installPath, System.String companyName, System.String productName, Int32 width, Int32 height, BuildOptions options, UnityEditor.RuntimeClassRegistry usedClassRegistry, UnityEditor.Build.Reporting.BuildReport report) (at C:/buildslave/unity/build/Editor/Mono/BuildPipeline/PostprocessBuildPlayer.cs:287)
UnityEngine.GUIUtilityrocessEvent(Int32, IntPtr)

 

It seems that the Android engine is trying to process the iOS library. This is also a NatShare issue, so it should be posted on the NatShare forum thread. Click on libNatShare.a in the editor and make sure that in the inspector, only iOS is selected as a supported platform.

 

Never mind.... I actually clicked on the file it was referring to and as soon as I did I realised I had seen this problem before.

The problem is "Any Platform" was selected and I needed to make sure that only "iOS" was selected.

EDIT: And wow you are fast. You ninja'd me when I had figured it out.

 

I bought the plugin a few weeks ago and I was able to record the screen pretty easily but I have a question. I would like to record a part of the screen instead of the full screen and I can't figure out a way to do that. It says in the asset description that you can capture part of the screen but there isn't any documentation for that. I would really appreciate some guidance. Thank you

 

Hi Lanre, thanks for checking. Multithreaded Rendering did really make a huge difference in performance.

I had it turned off as in the past Vuforia wouldn't work with Multithreaded Rendering in Android, I guess I was wrong with its newest version.

 

Hi again, could you provide an example as a reference because i just cant seem to work this out. I've asked some colleges as well and we cant seem to figure this out. The help is appreciated

 

Hello @Lanre , we have found that the app crashes if the user records a video with less than 1 second (for example, if the player taps the start/stop button quickly). This happens only in Android, it works fine in iOS.

I get the following log in Android:

E/AndroidRuntime(15789): Caused by: java.lang.IllegalStateException: Can't stop due to wrong state.

E/AndroidRuntime(15789):     at android.media.MediaMuxer.stop(MediaMuxer.java:228)

E/AndroidRuntime(15789):     at com.yusufolokoba.natcorder.Recorder.finishEncoder(Recorder.java:74)

E/AndroidRuntime(15789):     at com.yusufolokoba.natcorder.Encoder.run(Encoder.java:117)

E/AndroidRuntime(15789):     at java.lang.Thread.run(Thread.java:818)

Could you give us some insight on this?

Edit: This can be replicated in a new Unity 2017.3 project with the NatCorder's 1.2f1 GreyWorld example Scene. If the user starts then stops very quickly, the app will crash.

 

Barring a fix - do a work around by instead of having the video stop when the user pushes the button, have the button press set a flag to stop and stop on update() instead. Have a counter that starts when the user starts recording (add Time.deltaTime every update) and then stop the recording only after that counter is above one second.

OR make is so you can't stop the recording before one second.

Thanks for the catch - the way our app is designed it might be possible for people with unsteady hands to double-tap the button after triggering the recording - so I'll probably want to disable the button for a second.

 

Yes, I already implemented a workaround that ignores calls to stop the recording if the recording elapsed time is less than 1 second.

However, I believe it is valuable for NatCorder to assess a fix in its Android native library to handle this smoothly.

 

Hello @Lanre , I am also facing some issues with recording and sending the app into the background.

When recording and the app is sent into the background, the recording will fail when put back into foreground. When the app is back into the foreground: in iOS the video is not saved at all, and in Android it keeps recording the last frame before the app goes into background (plus, the audio goes out of sync as it does not keep track for these filler frames in the video).

The following log appears in iOS:

2018-05-07 17:14:41.009154+0200 thor[2385:1544731] NatCorder Error: Failed to record video frame for time 27288043520 due to invalid recorder state: 3

2018-05-07 17:14:41.023487+0200 thor[2385:1544731] NatCorder Error: Failed to record video frame for time 27303395328 due to invalid recorder state: 3

2018-05-07 17:14:41.039789+0200 thor[2385:1544731] NatCorder Error: Failed to record video frame for time 27321442304 due to invalid recorder state: 3

...

If I implement a behaviour to use Replay.PauseRecording() and Replay.ResumeRecording() accordingly, the results are the same in both iOS and Android.

protected void OnApplicationPause(bool isPause) {

      if (!Replay.IsRecording)

        return;



      if (isPause) {

        Replay.PauseRecording();

      }

      else {

        Replay.ResumeRecording();

      }

}

Could you please advice on this?

 

I am experiencing an issue on Android where if a video is playing when NatCorder is recording the video stops rendering in the view. I've been rendering the video to a Raw Image and putting that Raw Image onto a worldspace canvas (because it lines up with an AR object)

Note that the video playback only seems to freeze on the screen. In the recording the video it is showing it as it should be - it just can't seem to record the video and show it on the screen at the same time.

Is there a way to make this work or is it a hard device limit - I have a feeling there may be a technical hurdle of recording video and playing video at the same time.

EDIT: Turns out was a 'me' problem once again. I had been starting the video on update() if it wasn't already playing - basically I needed to set the video and start it somewhere so I lazily did it there. However since the video loops it has a brief period where is isn't playing so it starts it again when it was already looping - causing it to basically load another copy of the movie into memory.

I figured it out when I let it play without recording immediately.

 

Alright, after some time i was able to fix the video file moving issue, I'm still having issues with the audio for the video file.

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using System;
4. using UnityEngine;
5. using UnityEngine.UI;
6. using NatCorderU.Examples;
7. using NatCorderU.Core;
8. using NatCorderU.Extensions;
9. using NatCorderU;
10.  
11. public class AudioGetter : MonoBehaviour, IAudioSource {
12.  
13.  
14.     public int sampleRate { get { return AudioSettings.outputSampleRate; } }
15.  
16.     public int sampleCount { get { return 1024 / channelCount; } }
17.  
18.     public int channelCount { get { return 2; } }
19.  
20.     void OnAudioFilterRead(float[] samples, int channels)
21.     {
22.         long timeStamp = (long)(AudioSettings.dspTime * 1e+9f);
23.         NatCorder.CommitSamples(samples, timeStamp);
24.     }
25.  
26.     public void Dispose()
27.     {
28.      
29.     }
30. }

I used this script to obtain the audio

 

i can't set frame rate.

new Configuration(640, 480, 30);

but always over 60 FPS

unity 2018.1
windows 10 stand alone

 

Hi, and sorry for taking so long to get back to you. To crop out part of the screen, you would have to use a shader that performs the crop by scaling the UV's. You will then use NatCorder.StartRecording and AcquireFrame to manually record frames with the shader. The GreyWorld example provides a foundation for this so check it out.

 

I am looking into this and will get back to you with an update. Sorry for the inconvenience.

 

This happens because the encoder takes time to start, and if it is stopped before its startup is completed, then it will crash. I will look into adding a guard to prevent against the crash.

 

On iOS, the encoder absolutely will fail if the app is sent into the background. There is no public API to prevent this, so you must stop (not pause) recording when the app is going into the background.

 

Make sure you are passing an instance of the AudioGetter into NatCorder.StartRecording/Replay.StartRecording, and make sure that you have the AudioGetter script attached to an AudioSource or AudioListener in the scene.

 

Hi there. Can you be more descriptive? Also please share more code. What is "always over 60 FPS"?

 

Hey @Lanre I have updated to the latest NatCorder and added the latest NatShare. Please can you share an example script combining both for a simple record video, save to the devices gallery function?
I am only able to get the images to appear in my devices gallery if I take a photo using the native camera app as this seems to trigger a refresh. I am a bit stuck.... :-(

 

This is what I have currently:

Code (CSharp):

1. /*
2. *   NatCorder
3. *   Copyright (c) 2017 Yusuf Olokoba
4. */
5.  
6.  
7. namespace NatCorderU.Examples {
8.  
9.     using UnityEngine;
10.     using UnityEngine.UI;
11.     using System.Collections;
12.     using Core;
13.     using Extensions;
14.     using System.IO;
15.  
16.  
17.  
18.     public class ReplayCam2 : MonoBehaviour {
19.  
20.  
21.        
22.         public string path;
23.        
24.  
25.             // Use this for initialization
26.  
27.        
28.         [Header("UI")]
29.         public RawImage preview;
30.         public AspectRatioFitter aspectFitter;
31.  
32.         [Header("Recording")]
33.         public bool recordMicrophoneAudio;
34.         public AudioSource audioSource;
35.  
36.    
37.  
38.         public void StartRecording () {
39.             if (recordMicrophoneAudio) {
40.                 // Start the microphone
41.                 audioSource.clip = Microphone.Start(null, true, 15, 44100);
42.                 audioSource.Play();
43.                 // Start recording with microphone audio
44.                 Replay.StartRecording(Camera.main, new Configuration(640,480,20), OnReplay, audioSource, true);
45.             }
46.             // Start recording without microphone audio
47.             else Replay.StartRecording(Camera.main, new Configuration(640,480,20), OnReplay);
48.         }
49.  
50.         public void StopRecording () {
51.             // Stop the microphone
52.             if (recordMicrophoneAudio) Microphone.End(null);
53.             // Stop recording
54.             Replay.StopRecording();
55.         }
56.  
57.         /*
58.         void OnReplay (string path) {
59.             Sharing.GetThumbnail(path, OnThumbnail, 2f);
60.             this.path = path;
61.             Debug.Log("Saved recording to: "+path);
62.             #if UNITY_IOS || UNITY_ANDROID
63.             // Playback the video
64.             Handheld.PlayFullScreenMovie(path);
65.             #endif
66.         }
67.        
68.         */
69.        
70.         void OnReplay (string path) {
71.            
72.            
73.             //add time and date
74.             const string FMT = "O";
75.         System.DateTime now1 = System.DateTime.Now;
76.         string strDate = now1.ToString(FMT);
77.         Debug.Log(now1);
78.         // Define destination path and file name
79.         string movePath = "/storage/emulated/0/Pictures/m2mvideo"+(strDate)+".mp4";
80.         //string movePath = "/storage/emulated/0/videomm.mp4";
81.         // Move the recording there
82.         File.Move(path, movePath);
83.         // Play the recording from its new location
84.         Handheld.PlayFullScreenMovie(movePath);
85.         //refresh the android gallery
86.     }
87.    
88.        
89.        
90.         void OnThumbnail (Texture2D thumbnail) {}
91.        
92.         //share it
93.         public void OnShare(){Sharing.Share(this.path);}
94.         //save it
95.         public void saveroll()
96.         {
97.             Sharing.SaveToCameraRoll(this.path);
98.         }
99.        
100.        
101.        
102.  
103.  
104.  
105.        
106.  
107.  
108.  
109.  
110.     }
111. }

 

It is getting the videos in the location I want them in but only showing up if I take a photo with the native camera (on android) or if I turn the device off and on again.

 

Is this issue android-specific? It sounds like a NatShare bug (not NatCorder), so open an issue on GitHub or post on the NatShare thread. I'll look into it.

 

Thanks @Lanre! Does the code look ok / passable?

 

Yes, your code looks good. The only thing I spotted was that you never set `this.path` to `movePath` in `OnReplay`. But it seems that you do that elsewhere.

 

I'm seeing seeing videos with black borders, if they are larger than my max video size.

Here is the config I'm passing to StartRecording. If the screen resolution is larger than my max, use max: (My app is portrait)

Code (CSharp):

1. Vector2Int MaxVideoSize = new Vector2Int(1080, 1920);
2. Configuration config = Configuration.Screen;
3.             if (Screen.width > MaxVideoSize.x || Screen.height > MaxVideoSize.y) {
4.                 config = new Configuration(MaxVideoSize.x, MaxVideoSize.y, config.framerate, config.bitrate, config.keyframeInterval);
5.             }

and on iPhone X with a screen resolution of 2436 x 1125, the video looks like this: (green is just to cover content)

 

The photo you attached didn't seem to upload (I can't see it). The black borders are probably because of our aspect ratio fitter shader. The Replay API adds padding in the video when the aspect ratio of the camera is different from the aspect of the final recording. This is done to prevent stretching in the video.

 

Thanks, I'll adjust for device aspect ratio, this was the image

Here is my video aspect clamping coding for anyone needing to do the same thing:

Code (CSharp):

1. public Vector2Int MaxVideoSize = new Vector2Int(1080, 1920);
2.  
3. Configuration config = Configuration.Screen;
4.             if (Screen.width > MaxVideoSize.x || Screen.height > MaxVideoSize.y) {
5.                 Vector2Int videoSize = new Vector2Int(Screen.width, Screen.height);
6.                 float screenRatio = Camera.main.aspect;
7.                 if (videoSize.x > MaxVideoSize.x) {
8.                     // clamp by width
9.                     videoSize.x = Mathf.Min(videoSize.x, MaxVideoSize.x);
10.                     videoSize.y = Mathf.FloorToInt((float) videoSize.x / screenRatio);
11.                 }
12.                 if (videoSize.y > MaxVideoSize.y) {
13.                     // clamp by height
14.                     videoSize.y = Mathf.Min(videoSize.y, MaxVideoSize.y);
15.                     videoSize.x = Mathf.FloorToInt((float) videoSize.y * screenRatio);
16.                 }
17.                 config = new Configuration(videoSize.x, videoSize.y, config.framerate, config.bitrate, config.keyframeInterval);
18.             }

 

Hi! I need to record the stream of 2 webcams simultanely on disk. Is it possible with this plugin?

 

NatCorder is not a camera API, so it doesn't know about webcams. And as far as I know, it is not possible to have more than one camera open with WebCamTextures. NatCorder handles recording textures and game cameras, so if you can get your webcam preview into a texture in Unity, then NatCorder can record it.

 

yes I have 2 webcamtextures running. What I need to know is if natcorder can record two webcamtextures simuntanely

 

You simply have to blit both WebCamTextures to encoder frames. See the GreyWorld example where we record one WebCamTexture. What you would probably want to do is to blit with a custom shader that can record one to the left, and the other to the right.

 

thus it's impossible to obtain two separate mp4 files for each cam at runtime

 

Oh, I didn't know you meant making multiple recordings at once. This is not possible.

 

ok thank you

 

What codec does it use for video recording?
Does it write the video file on disk in realtime?
thx

 

NatCorder uses the H.264/MPEG4-AVC codec for video recording.

Yes, it writes to the disk in realtime.

 

I'm having a similar problem with my app. I have vuforia set up to show various videos using Unity's video player with each video linked to a audiosource. I then have an Audiolistener on the main camera which is passed to the StartRecording function. The Audio is recorded in editor when unity is first opened but stops after a while for no apparent reason.

 

Can you hear the audio being played in the scene?

 

I could.

I managed to also get it to work by moving the AudioListener off of the camera and then passing a single AudioSource to the StartRecording function. This setup still lets me hear during the scene and records each video's sound individually.

This setup allows users to hear only 1 video at a time which is fine for my app but may cause issues if I wanted to listen to multiple audio clips at once.

 

That's weird. If you can hear the audio, then NatCorder should be able to record it. Can you send me a repro and one of the videos that doesn't seem to have audio?

 

Just looking into this myself - just to clarify, to clean up the files, we need to trim the start of the path to remove the file:// for System.IO to work, is that right?

Are there any considerations to make for the other platforms?

 

Yes. This is the case on iOS and macOS.

 

Just adding another comment on memory crashes during recording. Even with lower video sizes, lower-end devices also need the video framerate reduced to avoid memory crashes. The iPhone 6s for example, the lowest device supported for ARKit, will crash in my app unless video is 1280x720 @30fps or less, 60 fps will crash it. I ended up adjusting the video size, framerate, and bitrate per device and what it can handle. So don't just rely on Configuration.Screen.

An awesome addition would be to have a Configuration.Best that would config the best video size, framerate, and bitrate based on the device... *wink* *wink*