NatCorder - Video Recording API

It's likely related. The issue is that no other device crashes, because only the iPhone 6 has 1GB of RAM. Every other iOS device has more than 1GB of RAM. For now, the best workaround is to detect the iPhone 6 in code and reduce the recording resolution accordingly.

 

Hi,

I experience crash on iPhoneX during recording. I combine NatCorder with NatMic. The crash occurs only on some devices like iPhoneX, iPhone 6s but works well on iPhone 7, 8 Xs, 8+ and so on.

Video Format: 720x1280@30fps

Code for starting recording and commuting sample

public void StartRecording()

    {

        var framerate = 30;

        var videoFormat = new VideoFormat(videoWidth, videoHeight, framerate);

        // Start the microphone with NatMic

        var microphoneFormat = NatMicU.Core.Format.Default;

        NatMicU.Core.NatMic.StartRecording(microphoneFormat, OnSampleBuffer);

        // Start recording with NatCorder

        recordingClock = new RealtimeClock();

        var audioFormat = new AudioFormat(microphoneFormat.sampleRate, microphoneFormat.channelCount);

        NatCorder.StartRecording(Container.MP4, VideoFormat.Screen, audioFormat, OnSave);



        videoRecorder = CameraRecorder.Create(mainCamera, videoFormat, recordingClock);

        videoRecorder.preserveAspectRatio = false;

    }



    // Invoked by NatMic on new microphone events

    void OnSampleBuffer(NatMicU.Core.AudioEvent audioEvent, float[] sampleBuffer, long timestamp, NatMicU.Core.Format format)

    {

        // Send sample buffers directly to NatCorder for recording

        if (audioEvent == NatMicU.Core.AudioEvent.OnSampleBuffer && NatCorder.IsRecording)

            NatCorder.CommitSamples(sampleBuffer, recordingClock.CurrentTimestamp);

    }

Is there something I missing to make it work on all iOS devices?

 

Your code uses VideoFormat.Screen. Can you share a link with an example video? Can you share the logs from Xcode in a .txt file?

 

That was u bug. Thank you! Audio format which we provide is

Code (CSharp):

1. var videoFormat = new VideoFormat(720, 1280, 30);

Logs indicate following problem:
TLTransferContext downloadToBuffer:fromTexture:]
MP4Recorder.m line 226 -[MP4Recorder onRender]
RenderContext.m line 36 __23-[RenderContext start:]_block_invoke

Even though we sometime are able to record than when playing video we get crash.

Also how to manage memory leaks? In our team we track render textures from "acquireFrame" and dispose them manually to avoid memory overflow. If we don't do that than they are left for the whole cycle of life of an app. Could it be a problem?


here's a code to have a buffer of only 20 render textures

Code (CSharp):

1. private void OnFrame () {
2.             if (NatCorder.IsRecording && frameCount++ % recordEveryNthFrame == 0) {
3.                 // Acquire frame, save camera state
4.                 var encoderFrame = NatCorder.AcquireFrame();
5.                 var prevTarget = Camera.targetTexture;
6.                 Camera.targetTexture = encoderFrame;
7.                 // Render
8.                 Camera.Render();
9.                 // Restore camera state, commit frame
10.                 Camera.targetTexture = prevTarget;
11.                 NatCorder.CommitFrame(encoderFrame, Clock.CurrentTimestamp);
12.  
13.                 texturesToReleaseEven.Add(encoderFrame);
14.                 evenTextures++;
15.  
16.                 Clean(false);
17.             }
18.         }
19.  
20. public void Clean(bool force)
21.         {
22.             if (evenTextures > 20)
23.             {
24.                 RenderTexture texture = texturesToReleaseEven[0];
25.                 texture.Release();
26.                 RenderTexture.Destroy(texture);
27.  
28.                 texturesToReleaseEven.RemoveAt(0);
29.                 evenTextures--;
30.             }
31.             if (force)
32.             {
33.                 foreach (RenderTexture texture in texturesToReleaseEven)
34.                 {
35.                     texture.Release();
36.                     RenderTexture.Destroy(texture);
37.                 }
38.             }
39.         }

 

Furthermore, when changed to proper video format there is an silent error which don't crash the app during recording, but crash it when we want to play the video:

[B][Crashlytics] Exception stack trace: NatCamU.Dispatch.MainDispatch.Update ()[/B]



[B]System.Action.Invoke ()[/B]



[B]NatCamU.Dispatch.DispatchUtility+<OnFrame>c__Iterator0.MoveNext ()[/B]



[B]UnityEngine.SetupCoroutine.InvokeMoveNext (IEnumerator enumerator, IntPtr returnValueAddress)



[/B]



[B]NullReferenceException: A null value was found where an object instance was required.[/B]



[B]  at NatCamU.Dispatch.MainDispatch.Update () [0x00000] in <filename unknown>:0 [/B]



[B]  at System.Action.Invoke () [0x00000] in <filename unknown>:0 [/B]



[B]  at NatCamU.Dispatch.DispatchUtility+<OnFrame>c__Iterator0.MoveNext () [0x00000] in <filename unknown>:0 [/B]



[B]  at UnityEngine.SetupCoroutine.InvokeMoveNext (IEnumerator enumerator, IntPtr returnValueAddress) [0x00000] in <filename unknown>:0[/B]

 

hi lanre.

all of sudden my prototype received error DllNotFoundException when start recording use natcorder.
at first i think it's because i tried to implemented natcorder, natcam, and natmic at the same time. so i rollback the code and delete all three pludgins and redownload only natcorder. but the error still remains.

another info:
NatCorder 1.3f2
Unity 2018.2.19f1

any help will be great. thank you.

 

I strongly recommend not to modify NatCorder's internals, and absolutely do not release NatCorder's render textures. The render textures are queued natively until they have been encoded, after which they are sent back to .NET to be released. This is likely the reason behind the exception that is being raised in [MTLTransferContext downloadToBuffer:_]. If you see a build up in memory, that is probably because you are committing frames faster than the encoder than dequeue them. If you app is running at 60FPS but can't actually record at 60FPS, then a buildup will happen. In this case, only record every other frame (effectively creating a 30FPS recording). When using a CameraRecorder, you can do this by setting CameraRecorder.recordEveryNthFrame = 2.

 

Email me with your invoice number and I'll send an updated build that fixes this issue.

 

Not sure if this has been asked before, but is there a way for us to determine where the file is being saved? Can I pass in a folder path and a filename?

Would love to be able to decide where the file is saved, and what the filename is based on dynamic data.

Cheers

 

Videos get saved to the app's private documents directory.

Nope. Instead, you can move the file to your desired directory in the RecordingCallback after recording has completed.

 

Does it work on UWP platform?

 

No, NatCorder doesn't officially support UWP.

 

With Ngui, the picture taken on android phone is horizontal, and the picture taken on computer is ok

 

Use android phone to shoot video, video saved is black, use the computer is normal

 

I presume you mean recorded video, not picture. Can you share a screenshot?

What device are you recording on? Can you share your recording code?

 

Code (CSharp):

1.   public class NatCorderController : MonoBehaviour
2.     {
3.  
4.         public UITexture m_UiTexture;
5.         public UICameraPreview m_Preview;
6.         private float m_GreyNess;
7.         private IClock m_Clock;
8.         private const float m_GreySpeed = 3f;
9.  
10.    
11.         private AudioFormat audioFormat;
12.         private AudioRecorder audioRecorder;
13.         public AudioListener audioListener;
14.         // Use this for initialization
15.         void Start()
16.         {
17.  
18.         }
19.  
20.         // Update is called once per frame
21.         void Update()
22.         {
23.             // Animate the greyness
24.             if (m_Preview.m_CameraTexture && m_UiTexture.mainTexture == m_Preview.m_CameraTexture)
25.             {
26.                 var currentGreyness = m_UiTexture.material.GetFloat("_Greyness");
27.                 var targetGreyness = Mathf.Lerp(currentGreyness, m_GreyNess, m_GreySpeed * Time.deltaTime);
28.                 m_UiTexture.material.SetFloat("_Greyness", targetGreyness);
29.             }
30.             // Record frames
31.             if (NatCorder.IsRecording && m_Preview.m_CameraTexture.didUpdateThisFrame)
32.             {
33.                 // Acquire an encoder frame
34.                 var frame = NatCorder.AcquireFrame();
35.                 // Blit with the preview's greyscale material
36.                 Graphics.Blit(m_Preview.m_CameraTexture, frame, m_UiTexture.material);
37.                 // Commit the encoder frame for encoding
38.                 NatCorder.CommitFrame(frame, m_Clock.CurrentTimestamp);
39.             }
40.         }
41.  
42.         public void StartRecording()
43.         {
44.      
45.             // Become grey
46.             m_GreyNess = 0f;
47.             // If the camera is in a potrait rotation, then we swap the width and height for recording
48.             //bool isPortrait = m_Preview.m_CameraTexture.videoRotationAngle == 90 || m_Preview.m_CameraTexture.videoRotationAngle == 270;
49.             //int recordingWidth = isPortrait ? m_Preview.m_CameraTexture.height : m_Preview.m_CameraTexture.width;
50.             //int recordingHeight = isPortrait ? m_Preview.m_CameraTexture.width : m_Preview.m_CameraTexture.height;
51.             /*var videoFormat = new VideoFormat(recordingWidth, recordingHeight);*/
52.             var videoFormat = new VideoFormat(540, 960);
53.             var audioFormat = new AudioFormat(AudioSettings.outputSampleRate, (int)AudioSettings.speakerMode);
54.             m_Clock = new RealtimeClock();
55.             // Start recording
56.             NatCorder.StartRecording(Container.MP4, videoFormat, audioFormat, OnVideo);
57.             audioRecorder = AudioRecorder.Create(audioListener);
58.         }
59.         public void StopRecording()
60.         {
61.             // Revert to normal color
62.             m_GreyNess = 0f;
63.             // Stop recording
64.             audioRecorder.Dispose();
65.             NatCorder.StopRecording();
66.         }
67.         void OnVideo(string path)
68.         {
69.             Debug.Log("Saved recording to: " + path);
70. #if UNITY_IOS
71.             Handheld.PlayFullScreenMovie("file://" + path);
72. #elif UNITY_ANDROID
73.             Handheld.PlayFullScreenMovie(path);
74. #endif
75.          
76.     }

• The picture saved by the phone is all black
  

 

The picture you attached isn't showing. Also, your code looks good so check that your blitting shader isn't causing problems.

 

Hey,
We use NatCorder to record AR (multiple cameras).
To avoid crashes, I lowered resolution and skip encoding every x frames.

Using this approach, it was basically a game of compiling and tweaking until the recording worked on our target devices.

However, after upgrading to Unity 2018.3, it seems that the memory usage has gone up and the app crashes on some devices.

My ideal solution to get around this would be to limit encoding by how many frames are waiting for enqueue.

Would you consider adding an API to get the current count of frames waiting for enqueue, or to limit the length of the queue (dropping frames if its full)?

Thanks

 

Hi, lanre, how to do that or how to code like that? I am not coder, please be easy. I want record all audio in scene but NOT MICROPHONE

UPDATE
- I did it!! just add the code:
audioRecorder = AudioRecorder.Create(audioListener, recordingClock);

 

Hey Lanre, done. Sorry this took so long!

 

We won't be adding this because it exposes too much (it adds a lot of complexity for developers to worry about). On the topic of making sure that the device can support recording at a given resolution, we've always been forced to make this a responsibility of the developer, not NatCorder.

The issue is that while it makes sense to impose resolution and framerate restrictions on iOS and Android (like Everyplay did), NatCorder isn't designed as a realtime video recorder. It is built to be more versatile, allowing the developer to record either in a for-loop or in realtime. We also can't resort to throttling until the record queue is flushed because the effects are very (very) visible in the recording. There's gonna be a weird jerk/timeskip in the video.

Having to tweak the resolution for different devices is not ideal--I know. Instead, I recommend recording at a 'safe' resolution. This is 1280x720 and 1600x900 on most mid-to-high-end devices. We're also working to drastically improve recording performance so that recording doesn't add any frame time to your game's rendering; and so that frames don't get queued up. We've achieved the former in 1.4.0. The latter is more difficult because we're bottlenecked by the performance of the hardware encoder.

 

Hey guys,

Do you have any idea why my recorded video is coming out with a strange tint to it? I'm using the post processing stack v2 and the realtimeclock with the CameraRecorder that came with the package.

I am also curious about the green streak at the bottom of the image...

Picture bellow for reference:

 

I would be very interested in beta testing this if possible. I have a project where video frames are coming into Unity from an external device at 60fps and only recording 1/4 of the frames, but I need to check every frame coming from the device without blocking.

 

hi lanre.
i want to follow up this issue. i already sent you invoice number and thread post reference to your email. do you have any progress you can share?

 

Hey there. Are you using linear rendering? If so, that'll explain the yellow tint. NatCorder doesn't yet support linear rendering, but we are looking at adding it in the next update.

 

Shoot me an email.

 

Sorry for the late response. I've replied.

 

Do you know when you'll release the next version? I want to render some videos by the end of the month and I don't think I'll be able to settle for gamma color space for this

I'm only planning to use this on Windows platform if you potentially might have a beta build I could use???

Thanks!

 

Shoot me an email then. I have to make some changes on Windows and I'd like to add linear support sooner rather than later. I can send you builds to test.

 

thanks Lanre, the beta eliminate the bug. I will resume the prototype again and explore the plugin.

cheers.

 

Thank you so much! Vulkan was the problem for me on Android too

 

Sounds good. NatCorder doesn't support Vulkan yet.

 

Hi @Lanre

iOS 12 record button causes instant crash. the memory utilized keeps increasing and results in crash.

 

What device, and what recording resolution?

 

Hi,

Great plugin, works well for the MP4 recording.
Now I'm trying to record a GIF, it works on iOS and Windows, but on Android it outputs a gif with weird colors?
Here's an album with the 3 GIF from each platform : https://imgur.com/a/sJCmvBY

Do you have an idea what I'm doing wrong ?

 

Your code is likely not at fault here. I'll look into this. Thanks for the imgur, it's really useful.

 

hi Lanre,

suddenly our apps received DllNotFoundException so we must update the NatCorder to beta version you provided. android version build went well but iOS version build was failed.
hereby the screenshoot of the error:

Spoiler

on this app we only use NatCorder and I make sure the libNatRender.a and libNatCamRenderPipeline.a only one in our project folder.

thanks in advance, and i'll be glad to help if you need anything else to give you more insight on this issue.

 

You need to delete NatCorder from your project before importing an updated build.

 

hi Lanre, this solved the issue. thank you very much. i will take a note of this.

 

Hi Lanre, do you know if this has been tested and confirmed working on Android 18 - 22? Every time I test these versions with a virtual device on Android Studio it crashes but I know these are not the most reliable representations.

 

Yes it has. NatCorder isn't intended to run on the simulator, given that it uses some of the lower-level Android infrastructure that might not be emulated in the simulator.

 

No problem thanks, I suspected as much from the errors just wanted to confirm.

 

In case this is of any use to you I've had 16 unique user crash reports around "com.yusufolokoba.natcorder.mp4.Encoder.drainEncoder", all of these have been for Oppo devices running Android 7.1 or 8.1.

Device list:

F9 (CPH1823)
CPH1725 (CPH1725)
F7 (CPH1819)
CPH1723 (CPH1723)
CPH1859 (CPH1859)
F7 (CPH1821)
A73 (CPH1837)
F9 (CPH1825)

 

Is it a null reference exception? The next version should fix this.

 

Thank you, did you find anything ? Maybe you want some more information from me ?

 

I don't need anything more. This is likely an artifact from our error diffusion dithering implementation.

 

Is this for the Unity Editor only?

Or can one include it in a game so players themselves can record gameplay etc?

 

No it isn't Editor only. NatCorder supports Windows Editor, Windows Standalone, macOS Editor, macOS Standalone, iOS, Android, and WebGL.

 

Hey, I am having exactly same issue described above. I've sent you an email with invoice number as well.

 

hi @Lanre , Natcorder in Unity 2018.3.1f1 doesn't call the callback in the start recording function. It stops calling after 3-5 times. is Natcorder supported in Unity 2018.3.1f1?

Hope the log helps

01-18 14:47:21.616 32641-32660/com.test.testD/Unity: NatCorder: Preparing video encoder with format: {frame-rate=30, bitrate=5909760, height=1280, mime=video/avc, color-format=2130708361, i-frame-interval=3, width=720}

01-18 14:47:21.636 364-937/? E/QC_CORE: OMXCORE: mNumOfNormalThread++, current is 4

    OMXCORE: mNumOfInstance++, current is 4

01-18 14:47:21.636 364-25974/? E/OMX-VENC-720p: set_parameter: metamode is valid for input port only

01-18 14:47:21.636 364-25974/? E/OMXNodeInstance: OMX_SetParameter() failed for StoreMetaDataInBuffers: 0x8000101a

01-18 14:47:21.646 32641-4751/com.test.testE/ACodec: [OMX.qcom.video.encoder.avc] storeMetaDataInBuffers (output) failed w/ err -2147483648

01-18 14:47:21.646 364-7012/? E/OMX-VENC-720p: WARNING: Unsupported Color format [2130708361]

    C2D init is successful

01-18 14:47:21.646 364-938/? E/OMX-VENC-720p: WARNING: Unsupported Color format [2130708361]

01-18 14:47:21.646 536-4694/? E/mm-camera: mct_pipeline_process_set:command=800000e

    mct_pipeline_send_ctrl_events: Send Set Parm events

01-18 14:47:21.656 536-4694/? E/mm-camera: mct_pipeline_process_set:command=800000e

    mct_pipeline_send_ctrl_events: Send Set Parm events

01-18 14:47:21.716 536-4694/? E/mm-camera: mct_pipeline_process_set:command=800000e

    mct_pipeline_send_ctrl_events: Send Set Parm events

01-18 14:47:21.736 1660-1675/? E/MP-Decision: Update arg 1

01-18 14:47:21.756 32641-4755/com.test.test D/Unity: NatCam Rendering: Created ES3 GLRenderContext