NatCorder - Video Recording API

Same problem on Unity2019.2.2 ,and Unity2018.4.2 MAC,

 

Hey There,
We've used this on iOS and it's working great. On android it works quite well except with a few devices we've found where it crashes the app upon start recording.
We're testing it along with the ARCore plugin. But have noticed that recording a video on the Google Pixel XL devices (not the regular devices) gives an index out of bounds when recording the video feed (as if the stream we're popping in to the buffer is too big, or perhaps the buffer was made incorrectly with the large devices). I think it's not worked on Pixel XL 2 and Pixel XL 3.
We don't have regular access to the device that fails (as it's the clients and friends). But I did want to post the logs if it helps you in future updates as it does link to the right function.
Here the Prep for the mp4 encoder on a valid device (Google Pixel 1):
https://pastebin.com/ZnWumS26

Here is a broken log:
Unity log: https://pastebin.com/8vqJvRKB
Full android log: https://pastebin.com/KTgTYgxn

Here is the useful bit of the android crash:

Code (CSharp):

1. 09-05 16:35:16.269 28939 29344 V Unity   : NatCorder: MP4 video encoder changed output format: {max-bitrate=5909760, csd-1=java.nio.HeapByteBuffer[pos=0 lim=8 cap=8], color-transfer=3, mime=video/avc, width=1440, bitrate=5909760, color-range=2, frame-rate=30, color-standard=2, height=2788, csd-0=java.nio.HeapByteBuffer[pos=0 lim=23 cap=23]}
2. 09-05 16:35:16.277 28939 28949 I pera.insa.alof: Background concurrent copying GC freed 509(277KB) AllocSpace objects, 2(30MB) LOS objects, 20% free, 93MB/117MB, paused 7.241ms total 171.706ms
3. 09-05 16:35:16.277 28939 29344 I MPEG4Writer: limits: 4294967295/0 bytes/us, bit rate: -1 bps and the estimated moov size 3191 bytes
4. 09-05 16:35:16.280 28939 29035 E native  : session.cc:519 generic::failed_precondition: VIO is not tracking.
5. 09-05 16:35:16.284 28939 29106 W native  : tracking.cc:3303 Feature descriptors not exist
6. 09-05 16:35:16.284 28939 29106 W native  : box_detector.cc:180 Detection skipped due to empty features or descriptors.
7. 09-05 16:35:16.285 28939 29365 D MPEG4Writer: Video skip non-key frame
8. 09-05 16:35:16.289 28939 29365 I MPEG4Writer: setStartTimestampUs: 67908
9. 09-05 16:35:16.289 28939 29365 I MPEG4Writer: Earliest track starting time: 67908
10. 09-05 16:35:16.299 28939 29006 I native  : distribute.cc:96 No keypoints to prune.
11. 09-05 16:35:16.303 28939 29019 I native  : image_target_detector.cc:233 Input query has too few keypoints.
12. 09-05 16:35:16.314  1192  1648 W InputDispatcher: channel 'b18e950 com.android.chrome/com.google.android.apps.chrome.Main (server)' ~ Consumer closed input channel or an error occurred.  events=0x9
13. 09-05 16:35:16.314  1192  1648 E InputDispatcher: channel 'b18e950 com.android.chrome/com.google.android.apps.chrome.Main (server)' ~ Channel is unrecoverably broken and will be disposed!
14. 09-05 16:35:16.328 28939 29345 E AndroidRuntime: FATAL EXCEPTION: Thread-18
15. 09-05 16:35:16.328 28939 29345 E AndroidRuntime: Process: com.squintopera.insa.aloft, PID: 28939
16. 09-05 16:35:16.328 28939 29345 E AndroidRuntime: java.lang.IllegalArgumentException: bufferInfo must specify a valid buffer offset, size and presentation time
17. 09-05 16:35:16.328 28939 29345 E AndroidRuntime:     at android.media.MediaMuxer.writeSampleData(MediaMuxer.java:670)
18. 09-05 16:35:16.328 28939 29345 E AndroidRuntime:     at com.yusufolokoba.natcorder.MP4Recorder$7.run(MP4Recorder.java:271)
19. 09-05 16:35:16.328 28939 29345 E AndroidRuntime:     at java.lang.Thread.run(Thread.java:764)

It's very strange, we've tried not recording video, recording at half resolution (as it's bigger screen size form factor). It's using the CameraInput with MP4Encoder and we've tried with no-audio but it's the video feed that is the issue.

Code (CSharp):

1.  videoRecorder = new MP4Recorder(width, height, 30, AudioSettings.outputSampleRate, (int)AudioSettings.speakerMode, OnRecording);
2. cameraInput = new CameraInput(videoRecorder, clock, camera);
3. if (audioSource != null)
4. {
5.     audioInput = new AudioInput(videoRecorder, clock, audioSource, true);
6. }

 

I'm currently using the Handheld.PlayFullScreenMovie function and looked at it with xcode.

However, there is no error log, and I clicked on the error message and pointed to that part.

Do you have any idea what the problem is?

Code (CSharp):

1. - (BOOL)loadVideo:(NSURL*)url
2. {
3.     AVURLAsset* asset = [AVURLAsset URLAssetWithURL: url options: nil];
4.     if (!asset)
5.         return NO;
6.  
7.     NSArray* requestedKeys = @[@"tracks", @"playable"];
8.     [asset loadValuesAsynchronouslyForKeys: requestedKeys completionHandler:^{
9.         dispatch_async(dispatch_get_main_queue(), ^{
10.             [self prepareAsset: asset withKeys: requestedKeys];
11.         });
12.     }];
13.     return YES;
14. }

 

as you said "you can still record a video with audio with WebGL, as you can get the audio data from an AudioClip or procedurally generate it then pass it to NatCorder". May I ask how to do it?

 

Stuck with this plugin for a long time. Any help would be much appreciated.
https://support.wikitude.com/support/discussions/topics/5000092795

 

When recording in Landscape Left orientation using the Front Facing Camera on iOS, the video is flipped. How can I fix that?

 

I sent you an updated build over email. Test it out and let me know if the issue persists.

 

Not just yet. Still experimenting cos as far as I can tell it's coming from Unity's preload thread, which doesn't have any direct connection to NatCorder.

 

This is a limitation from Unity because of how Unity renders overlay canvases. This will not be changing anytime soon.

 

Are you trying to record the screen? Does the CameraInput helper script not work? I don't have any experience with Wikitude so I can't provide support for that specifically.

 

WebCamTexture has a `videoRotationAngle` property that you can use to rotate the texture.

 

You have to make sure that your app does not use Vulkan. Do so by disabling "Auto Graphics API" in Player Settings then removing Vulkan from the list.

 

Not sure what is going on here. On iOS, make sure you prepend the path from NatCorder with "file://" before giving it to Handheld.PlayFullscreenMovie.

 

Procedural audio is simply generating audio samples at runtime. How you do it is up to you, but you can pass the generated sample buffers (float[]) to NatCorder recorders using the CommitSamples function. I just noticed that AudioClip.GetData is not supported, so you can't use an AudioClip.

 

I've found the cause of the Editor hanging and have created a fix.

NatCorder 1.6.2 update should be on the Asset Store on Tuesday or Wednesday.

 

Hello,

When i record the screen it comes out blank (blue background is all i see) in the MP4 file. Do you know why?

I looked at the actual MP4 file and its blank, so its not the player.

Here is the code

Code (CSharp):

1.     public void StartRecording()
2.     {
3.         // Start recording
4.         recordingClock = new RealtimeClock();
5.         videoRecorder = new MP4Recorder(
6.             Screen.width,//videoWidth, //Screen.height,//videoWidth,
7.             Screen.height, //videoHeight, //Screen.width, //videoHeight,
8.             30,
9.             recordMicrophone ? AudioSettings.outputSampleRate : 0,
10.             recordMicrophone ? (int)AudioSettings.speakerMode : 0,
11.             OnReplay
12.         );
13.         // Create recording inputs
14.         cameraInput = new CameraInput(videoRecorder, recordingClock, Camera.main);
15.         if (recordMicrophone) {
16.             StartMicrophone();
17.             audioInput = new AudioInput(videoRecorder, recordingClock, microphoneSource, true);
18.         }
19.     }
20.  
21.     private void StartMicrophone()
22.     {
23. #if !UNITY_WEBGL || UNITY_EDITOR // No `Microphone` API on WebGL :(
24.         // Create a microphone clip
25.         microphoneSource.clip = Microphone.Start(null, true, 60, 48000);
26.         while (Microphone.GetPosition(null) <= 0) ;
27.         // Play through audio source
28.         microphoneSource.timeSamples = Microphone.GetPosition(null);
29.         microphoneSource.loop = true;
30.         microphoneSource.Play();
31. #endif
32.     }
33.  
34.     public void StopRecording()
35.     {
36.         // Stop the recording inputs
37.         if (recordMicrophone) {
38.             StopMicrophone();
39.             audioInput.Dispose();
40.         }
41.         cameraInput.Dispose();
42.         // Stop recording
43.         videoRecorder.Dispose();
44.     }
45.  
46.     private void StopMicrophone()
47.     {
48. #if !UNITY_WEBGL || UNITY_EDITOR
49.         Microphone.End(null);
50.         microphoneSource.Stop();
51. #endif
52.     }
53.  
54.     private void OnReplay(string path)
55.     {
56.         currentPath = "file://" + path;
57.         StartCoroutine(PlayVideo(currentPath));
58.     }

Thanks

 

By blank, is the video black or blue? Can you share the logs in a .txt file?

 

It works now, to fix tthe problem set the UI CANVAS to "Screen Space camera", then drag the main camera into the RENDER CAMERA slot in the canvas, then boom it worked.

 

Only add "file://" when passing the path to `Handheld.PlayFullscreenMovie`, not when using System.IO API's.

 

I want to record Texture2D which I am manipulating continuously which some other process is going on in the main camera. But all the tutorials focus recording from either Camera.main or Webcamtexture. Can somebody help me with a code snippet to record from a normal Texture2D and not directly the camera.
I looked at NatCorderU Examples and I am importing NatCorderU.Core for this purpose.

 

It's almost the exact same as with a WebCamTexture. Simply call GetPixels32 then commit the Color32 array (see WebCam example).

 

I'm sorry but can you tell me which example are you referring to? Till now I have followed the two examples given in the NatCorderU package and this one: https://github.com/olokobayusuf/NatMicCorder-Demo/blob/master/Assets/Scripts/NatMicCorder.cs

Let me know if I am missing some examples. Thanks for the prompt reply.

 

It sounds like you are using an older version of NatCorder. Upgrade to 1.6.2 on the Asset Store.

 

Hello Lanre,

Just having some difficulties when creating XCode archives with the following error:

Code (CSharp):

1. Undefined symbols for architecture armv7:
2.   "_NCEncodeSamples", referenced from:
3.       _MediaRecorderBridge_EncodeSamples_mB7B02B4BEE53CF514F7C66B4E0CF659A572E01C0 in Bulk_Assembly-CSharp_1.o
4.       _MediaRecorderiOS_CommitSamples_mAAC4711641ED1E0795BAB028E2363C6A4DC50B9C in Bulk_Assembly-CSharp_1.o
5.   "_NCMTLReadbackReadback", referenced from:
6.       _RenderTextureInput_MTLReadbackReadback_m2CDDDB61FD4D93A5CFC89664913611260BD4287B in Bulk_Assembly-CSharp_1.o
7.       _U3CU3Ec__DisplayClass10_0_U3CCommitiOSU3Eb__1_m771B2A3D6F65CCE0E3031711671A5F474DB8D2D4 in Bulk_Assembly-CSharp_1.o
8.   "_NCCreateMP4Recorder", referenced from:
9.       _MP4Recorder__ctor_m46E223A976C530597C7E729D63BD96132039DC22 in Bulk_Assembly-CSharp_1.o
10.       _MediaRecorderBridge_CreateMP4Recorder_mDBE59CA7E85A824DD5C2AE87D3508609C8CCFAD5 in Bulk_Assembly-CSharp_1.o
11.   "_NCMTLReadbackDispose", referenced from:
12.       _RenderTextureInput_MTLReadbackDispose_m0E5563632A35AE1AB64D04D1F0888F3095D96661 in Bulk_Assembly-CSharp_1.o
13.       _RenderTextureInput_U3CDisposeU3Eb__1_0_m5EE96B70807D5FC0801BE7EC7116D7CCC7FCEE5D in Bulk_Assembly-CSharp_1.o
14.   "_NCMTLReadbackCreate", referenced from:
15.       _RenderTextureInput__ctor_mBB4BB59488E797A138EB12A292BD5C1BDF2B57BE in Bulk_Assembly-CSharp_1.o
16.       _RenderTextureInput_MTLReadbackCreate_mF7F446F09F92470716963C5ED83CA80E025A6B00 in Bulk_Assembly-CSharp_1.o
17.   "_NCStopRecording", referenced from:
18.       _MediaRecorderBridge_StopRecording_mA4CFDBBD6CDFE7576B30729CC56B7B2A211E9C98 in Bulk_Assembly-CSharp_1.o
19.       _MediaRecorderiOS_Dispose_m73BCCDE24383DF120D7252C9E369FDEDE416D46F in Bulk_Assembly-CSharp_1.o
20.   "_NCCreateHEVCRecorder", referenced from:
21.       _HEVCRecorder__ctor_m54BA0C2A6C54D7974358BF27CAE4417D06437DA3 in Bulk_Assembly-CSharp_1.o
22.       _MediaRecorderBridge_CreateHEVCRecorder_m2067600B4D962E6FF050B2D092A6B5FC5EF1AD8A in Bulk_Assembly-CSharp_1.o
23.   "_NCEncodeFrame", referenced from:
24.       _MediaRecorderBridge_EncodeFrame_m11B197C0DB5AE78EAA9083FAA07DAD343698246D in Bulk_Assembly-CSharp_1.o
25.       _MediaRecorderiOS_CommitFrame_m75B4141A18D5A05DF3F18C974252AF91C69947B2 in Bulk_Assembly-CSharp_1.o
26.   "_NCStartRecording", referenced from:
27.       _MediaRecorderBridge_StartRecording_m1647875A066BCDC5F09CB79D1E2BA01F3197E32D in Bulk_Assembly-CSharp_1.o
28.   "_NCCreateGIFRecorder", referenced from:
29.       _GIFRecorder__ctor_mDCADA7399C36E4B0975DD287AA8DBE690C10E0D7 in Bulk_Assembly-CSharp_1.o
30.       _MediaRecorderBridge_CreateGIFRecorder_mE94056AC35C676F2D148BEBFDF8CD61B45B31534 in Bulk_Assembly-CSharp_1.o
31. ld: symbol(s) not found for architecture armv7
32. clang: error: linker command failed with exit code 1 (use -v to see invocation)

Though I am able to build in device without any problems. Did I miss a setup or something?

Thanks in advance

 

NatCorder does not support armv7. You have to remove that architecture from your archive build.

 

Thanks a lot! I was able to get around that with the same version of NatCorder. I have another doubt. While running in Unity, I was able to store the mp4 in the project folder and it was as desired. Now when I run the app on my iOS, I am not sure where the mp4 video is stored. I am new to iPhone development and hence I have no idea. Can you say where is the default save the location of mp4 generated using NatCorder?

 

The video is stored in the app's private documents directory. If you want to copy the video to the camera roll, use our free NatShare API.

 

Hi!
There are problems in order to archive the project in xcode.
Xcode give me this errors:
Undefined symbol: _NCIsRecording
Undefined symbol: _NCInitialize
Undefined symbol: _NatCamRenderDelegate

I check NatCorder lib via lipo -info libNatCorder.a and see that it support only arm64. I set to my architectures in xcode only arm64 and try to archiving, but it is not help me.
I already read everything that was on this topic on the Internet and I don’t know what is the problem.

I use Unity 2019.1.2f, xCode 10.3, NatCorder 1.6.1

 

i made ios build and have crash on start recording. This is all logs

 

Those are very old symbols. Delete and rebuild your Xcode project from Unity (or select the Replace option instead of Append when building from Unity).

 

What device and OS version are you running on?

 

Iphone7, 12.4

 

And does this crash happen every time you record?

 

Yes

 

Can you share the full logs from app start to crash?

 

Little bit later. This my recording code. Maybe here are problem. Also this code work on android correct, no crash

Code (CSharp):

1. _clock = new RealtimeClock();
2.         _videoRecorder = new MP4Recorder(
3.             1080,
4.             1920,
5.             30,
6.             AudioSettings.outputSampleRate,
7.             2,
8.             OnRecordStop
9.         );
10.         _cameraInput = new CameraInput(_videoRecorder,_clock,Camera.main);
11.         _audioInput = new AudioInput(_videoRecorder,_clock,Camera.main.GetComponent<AudioListener>());
12.  

 

There is no problem here. Share the logs when you can.

 

Your app is rendering with OpenGL ES. NatCorder requires Metal on iOS and macOS.

 

I have black screen on vuforia if use metal api(((

 

This is a Vuforia issue. Reach out to their support.

 

Just got this asset and tried to make this work to record gameplay from my app, and in Windows with Unity version 2019.1.0b8, I'm getting this:

Requested RenderTexture with sRGB format. sRGB formats are not supported in gamma mode, fallback to a UNorm format.
UnityEngine.Camera:Render()
NatCorder.Inputs.<OnFrame>d__9:MoveNext() (at Assets/NatCorder/Plugins/Managed/Inputs/CameraInput.cs:84)

Of course, this is for every frame it is trying to record.

My code:

Code (CSharp):

1.     void StartRecording()
2.     {
3.         Debug.Log("====================================STARTING RECORDING===============================");
4.         recordingClock = new RealtimeClock();
5.         AudioListener listener = Camera.main.GetComponent<AudioListener>();
6.         if (listener != null)
7.         {
8.             mediaRecorder = new MP4Recorder(Camera.main.pixelWidth, Camera.main.pixelHeight, 15, AudioSettings.outputSampleRate, (int)AudioSettings.speakerMode, onRecordingComplete);
9.             audioInput = new AudioInput(mediaRecorder, recordingClock, listener);
10.         }
11.         else
12.         {
13.             mediaRecorder = new MP4Recorder(Camera.main.pixelWidth, Camera.main.pixelHeight, 15, 0, 0, onRecordingComplete );
14.         }
15.         cameraInput = new CameraInput(mediaRecorder, recordingClock, Camera.main);
16.  
17.     }
18.  
19.     void StopRecording()
20.     {
21.         Debug.Log("====================================ENDING RECORDING===============================");
22.         // Destroy the recording inputs
23.         cameraInput.Dispose();
24.         audioInput.Dispose();
25.         // Stop recording
26.         mediaRecorder.Dispose();
27.         mediaRecorder = null;      
28.     }
29.  
30.     private void onRecordingComplete (string path) {
31.         Debug.Log("====================Saved recording to: "+path);
32.         // Playback the video
33.         #if UNITY_EDITOR
34.         EditorUtility.OpenWithDefaultApp(path);
35.         #elif UNITY_IOS
36.         Handheld.PlayFullScreenMovie("file://" + path);
37.         #elif UNITY_ANDROID
38.         Handheld.PlayFullScreenMovie(path);
39.         #endif
40.     }  
41.  
42.  
43.  

Any ideas here?

 

Upgrade to NatCorder 1.6.2 on the Asset Store (delete NatCorder in your project before importing the update) then try recording again and let me know if the issue persists.

 

Thanks for the quick response, but I literally purchased it last night, and downloaded 1.6.2. I imported the latest version already.

 

The line number from the error is different from what it should be. In any case, open CameraInput.cs and comment out the line that says `frameDescriptor.sRGB = true`.

 

That quieted the message, but the resulting MP4 is still a 1k file, and it never executed the onRecordingComplete callback.

 

Check that Camera.main is not null. Can you share logs?

 

Camera.main is not null, I've debugged and inspected the values at that point, just to make sure the pixel width/height values were correct, and that it gets the listener as well.

Does the scripting backend matter? I'm using IL2CPP, for a number of reasons, and I notice you are using mono directives.

UPDATE: It seems that IL2CPP should handle the directive (based on my google-fu). My callback is never actually called though when I call StopRecording in the code above.

 

Tried NatCorder 1.6.2 today, Android is working fine. iOS is recording with weird colors, check the image here (the bear's cape should be red). I'm using Unity 2019.2.5f1, Vuforia 8.3.8, iOS 12.4.1, Metal.

I reverted to v1.6.1 and everything is working fine.
I'm also getting 2 warnings for NatCorder/NatShare demo projects. check them here

 

If the callback isn't being invoked, then chances are that no video frames have been recorded. Do the ReplayCam and WebCam examples work for you?