Search Unity

  1. We would like to hear your feedback about Unity and our products. Click here for more information.
    Dismiss Notice

NatCorder - Video Recording API

Discussion in 'Assets and Asset Store' started by Lanre, Nov 18, 2017.

  1. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    This has to be an issue on Unity's or Gradle's end. I know that it isn't NatCorder's because the AAR simply has to be universal. It is up to the final APK build to split by ABI.
     
  2. shallwaycn

    shallwaycn

    Joined:
    Apr 3, 2012
    Posts:
    193
    Hello,

    I bought your tool one week ago, and it's great so far, but when we tested it on ios12, iPhone 6. The memory usage is too high and It soon crashes when start recording. our app doesn't use any memories, only a NatCam and some compute shader doing some post effects.

    The strange thing is it works well on iOS 11, after some researches I found it's maybe the Unity's bug which is still pending fix:
    https://forum.unity.com/threads/ios-12-large-memory-usage.558550/

    However we don't know whether they are related or not. We'd be very appreciate If you can take a investigation on the iOS 12 devices.

    ------

    By the way is it possible to buy the source code license? we may use NatCorder and NatCam to develop a commercial app, If we can get the source code it would be very helpful.

    Best Regards

    Shallway
     
  3. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    It's likely related. The issue is that no other device crashes, because only the iPhone 6 has 1GB of RAM. Every other iOS device has more than 1GB of RAM. For now, the best workaround is to detect the iPhone 6 in code and reduce the recording resolution accordingly.
     
  4. stelma

    stelma

    Joined:
    Nov 26, 2018
    Posts:
    3
    Hi,

    I experience crash on iPhoneX during recording. I combine NatCorder with NatMic. The crash occurs only on some devices like iPhoneX, iPhone 6s but works well on iPhone 7, 8 Xs, 8+ and so on.

    Video Format: 720x1280@30fps

    Code for starting recording and commuting sample

    public void StartRecording()
    {
    var framerate = 30;
    var videoFormat = new VideoFormat(videoWidth, videoHeight, framerate);
    // Start the microphone with NatMic
    var microphoneFormat = NatMicU.Core.Format.Default;
    NatMicU.Core.NatMic.StartRecording(microphoneFormat, OnSampleBuffer);
    // Start recording with NatCorder
    recordingClock = new RealtimeClock();
    var audioFormat = new AudioFormat(microphoneFormat.sampleRate, microphoneFormat.channelCount);
    NatCorder.StartRecording(Container.MP4, VideoFormat.Screen, audioFormat, OnSave);

    videoRecorder = CameraRecorder.Create(mainCamera, videoFormat, recordingClock);
    videoRecorder.preserveAspectRatio = false;
    }

    // Invoked by NatMic on new microphone events
    void OnSampleBuffer(NatMicU.Core.AudioEvent audioEvent, float[] sampleBuffer, long timestamp, NatMicU.Core.Format format)
    {
    // Send sample buffers directly to NatCorder for recording
    if (audioEvent == NatMicU.Core.AudioEvent.OnSampleBuffer && NatCorder.IsRecording)
    NatCorder.CommitSamples(sampleBuffer, recordingClock.CurrentTimestamp);
    }


    Is there something I missing to make it work on all iOS devices?
     
  5. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    Your code uses VideoFormat.Screen. Can you share a link with an example video? Can you share the logs from Xcode in a .txt file?
     
  6. stelma

    stelma

    Joined:
    Nov 26, 2018
    Posts:
    3
    That was u bug. Thank you! Audio format which we provide is

    Code (CSharp):
    1. var videoFormat = new VideoFormat(720, 1280, 30);
    Logs indicate following problem:
    TLTransferContext downloadToBuffer:fromTexture:]
    MP4Recorder.m line 226 -[MP4Recorder onRender]
    RenderContext.m line 36 __23-[RenderContext start:]_block_invoke

    Even though we sometime are able to record than when playing video we get crash.

    Also how to manage memory leaks? In our team we track render textures from "acquireFrame" and dispose them manually to avoid memory overflow. If we don't do that than they are left for the whole cycle of life of an app. Could it be a problem?


    here's a code to have a buffer of only 20 render textures
    Code (CSharp):
    1. private void OnFrame () {
    2.             if (NatCorder.IsRecording && frameCount++ % recordEveryNthFrame == 0) {
    3.                 // Acquire frame, save camera state
    4.                 var encoderFrame = NatCorder.AcquireFrame();
    5.                 var prevTarget = Camera.targetTexture;
    6.                 Camera.targetTexture = encoderFrame;
    7.                 // Render
    8.                 Camera.Render();
    9.                 // Restore camera state, commit frame
    10.                 Camera.targetTexture = prevTarget;
    11.                 NatCorder.CommitFrame(encoderFrame, Clock.CurrentTimestamp);
    12.  
    13.                 texturesToReleaseEven.Add(encoderFrame);
    14.                 evenTextures++;
    15.  
    16.                 Clean(false);
    17.             }
    18.         }
    19.  
    20. public void Clean(bool force)
    21.         {
    22.             if (evenTextures > 20)
    23.             {
    24.                 RenderTexture texture = texturesToReleaseEven[0];
    25.                 texture.Release();
    26.                 RenderTexture.Destroy(texture);
    27.  
    28.                 texturesToReleaseEven.RemoveAt(0);
    29.                 evenTextures--;
    30.             }
    31.             if (force)
    32.             {
    33.                 foreach (RenderTexture texture in texturesToReleaseEven)
    34.                 {
    35.                     texture.Release();
    36.                     RenderTexture.Destroy(texture);
    37.                 }
    38.             }
    39.         }
     
  7. stelma

    stelma

    Joined:
    Nov 26, 2018
    Posts:
    3
    Furthermore, when changed to proper video format there is an silent error which don't crash the app during recording, but crash it when we want to play the video:


    [B][Crashlytics] Exception stack trace: NatCamU.Dispatch.MainDispatch.Update ()[/B]

    [B]System.Action.Invoke ()[/B]

    [B]NatCamU.Dispatch.DispatchUtility+<OnFrame>c__Iterator0.MoveNext ()[/B]

    [B]UnityEngine.SetupCoroutine.InvokeMoveNext (IEnumerator enumerator, IntPtr returnValueAddress)

    [/B]

    [B]NullReferenceException: A null value was found where an object instance was required.[/B]

    [B] at NatCamU.Dispatch.MainDispatch.Update () [0x00000] in <filename unknown>:0 [/B]

    [B] at System.Action.Invoke () [0x00000] in <filename unknown>:0 [/B]

    [B] at NatCamU.Dispatch.DispatchUtility+<OnFrame>c__Iterator0.MoveNext () [0x00000] in <filename unknown>:0 [/B]

    [B] at UnityEngine.SetupCoroutine.InvokeMoveNext (IEnumerator enumerator, IntPtr returnValueAddress) [0x00000] in <filename unknown>:0[/B]
     
  8. DennyOctagon

    DennyOctagon

    Joined:
    Jul 12, 2018
    Posts:
    7
    hi lanre.

    all of sudden my prototype received error DllNotFoundException when start recording use natcorder.
    at first i think it's because i tried to implemented natcorder, natcam, and natmic at the same time. so i rollback the code and delete all three pludgins and redownload only natcorder. but the error still remains.

    another info:
    NatCorder 1.3f2
    Unity 2018.2.19f1

    any help will be great. thank you.
     

    Attached Files:

    Last edited: Dec 21, 2018
  9. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    I strongly recommend not to modify NatCorder's internals, and absolutely do not release NatCorder's render textures. The render textures are queued natively until they have been encoded, after which they are sent back to .NET to be released. This is likely the reason behind the exception that is being raised in [MTLTransferContext downloadToBuffer:_]. If you see a build up in memory, that is probably because you are committing frames faster than the encoder than dequeue them. If you app is running at 60FPS but can't actually record at 60FPS, then a buildup will happen. In this case, only record every other frame (effectively creating a 30FPS recording). When using a CameraRecorder, you can do this by setting CameraRecorder.recordEveryNthFrame = 2.
     
  10. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    Email me with your invoice number and I'll send an updated build that fixes this issue.
     
  11. EMOTION-THEORY

    EMOTION-THEORY

    Joined:
    Jul 16, 2013
    Posts:
    65
    Not sure if this has been asked before, but is there a way for us to determine where the file is being saved? Can I pass in a folder path and a filename?

    Would love to be able to decide where the file is saved, and what the filename is based on dynamic data.

    Cheers :)
     
  12. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    Videos get saved to the app's private documents directory.
    Nope. Instead, you can move the file to your desired directory in the RecordingCallback after recording has completed.
     
  13. Zinios

    Zinios

    Joined:
    Nov 7, 2017
    Posts:
    3
    Does it work on UWP platform?
     
  14. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    No, NatCorder doesn't officially support UWP.
     
  15. LiuDaZhuang

    LiuDaZhuang

    Joined:
    Dec 26, 2018
    Posts:
    3
    With Ngui, the picture taken on android phone is horizontal, and the picture taken on computer is ok
     
  16. LiuDaZhuang

    LiuDaZhuang

    Joined:
    Dec 26, 2018
    Posts:
    3
    Use android phone to shoot video, video saved is black, use the computer is normal
     
  17. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    I presume you mean recorded video, not picture. Can you share a screenshot?
    What device are you recording on? Can you share your recording code?
     
  18. LiuDaZhuang

    LiuDaZhuang

    Joined:
    Dec 26, 2018
    Posts:
    3
    Code (CSharp):
    1.   public class NatCorderController : MonoBehaviour
    2.     {
    3.  
    4.         public UITexture m_UiTexture;
    5.         public UICameraPreview m_Preview;
    6.         private float m_GreyNess;
    7.         private IClock m_Clock;
    8.         private const float m_GreySpeed = 3f;
    9.  
    10.    
    11.         private AudioFormat audioFormat;
    12.         private AudioRecorder audioRecorder;
    13.         public AudioListener audioListener;
    14.         // Use this for initialization
    15.         void Start()
    16.         {
    17.  
    18.         }
    19.  
    20.         // Update is called once per frame
    21.         void Update()
    22.         {
    23.             // Animate the greyness
    24.             if (m_Preview.m_CameraTexture && m_UiTexture.mainTexture == m_Preview.m_CameraTexture)
    25.             {
    26.                 var currentGreyness = m_UiTexture.material.GetFloat("_Greyness");
    27.                 var targetGreyness = Mathf.Lerp(currentGreyness, m_GreyNess, m_GreySpeed * Time.deltaTime);
    28.                 m_UiTexture.material.SetFloat("_Greyness", targetGreyness);
    29.             }
    30.             // Record frames
    31.             if (NatCorder.IsRecording && m_Preview.m_CameraTexture.didUpdateThisFrame)
    32.             {
    33.                 // Acquire an encoder frame
    34.                 var frame = NatCorder.AcquireFrame();
    35.                 // Blit with the preview's greyscale material
    36.                 Graphics.Blit(m_Preview.m_CameraTexture, frame, m_UiTexture.material);
    37.                 // Commit the encoder frame for encoding
    38.                 NatCorder.CommitFrame(frame, m_Clock.CurrentTimestamp);
    39.             }
    40.         }
    41.  
    42.         public void StartRecording()
    43.         {
    44.      
    45.             // Become grey
    46.             m_GreyNess = 0f;
    47.             // If the camera is in a potrait rotation, then we swap the width and height for recording
    48.             //bool isPortrait = m_Preview.m_CameraTexture.videoRotationAngle == 90 || m_Preview.m_CameraTexture.videoRotationAngle == 270;
    49.             //int recordingWidth = isPortrait ? m_Preview.m_CameraTexture.height : m_Preview.m_CameraTexture.width;
    50.             //int recordingHeight = isPortrait ? m_Preview.m_CameraTexture.width : m_Preview.m_CameraTexture.height;
    51.             /*var videoFormat = new VideoFormat(recordingWidth, recordingHeight);*/
    52.             var videoFormat = new VideoFormat(540, 960);
    53.             var audioFormat = new AudioFormat(AudioSettings.outputSampleRate, (int)AudioSettings.speakerMode);
    54.             m_Clock = new RealtimeClock();
    55.             // Start recording
    56.             NatCorder.StartRecording(Container.MP4, videoFormat, audioFormat, OnVideo);
    57.             audioRecorder = AudioRecorder.Create(audioListener);
    58.         }
    59.         public void StopRecording()
    60.         {
    61.             // Revert to normal color
    62.             m_GreyNess = 0f;
    63.             // Stop recording
    64.             audioRecorder.Dispose();
    65.             NatCorder.StopRecording();
    66.         }
    67.         void OnVideo(string path)
    68.         {
    69.             Debug.Log("Saved recording to: " + path);
    70. #if UNITY_IOS
    71.             Handheld.PlayFullScreenMovie("file://" + path);
    72. #elif UNITY_ANDROID
    73.             Handheld.PlayFullScreenMovie(path);
    74. #endif
    75.          
    76.     }


    • The picture saved by the phone is all black
     
  19. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    The picture you attached isn't showing. Also, your code looks good so check that your blitting shader isn't causing problems.
     
  20. JonInception

    JonInception

    Joined:
    Feb 25, 2018
    Posts:
    1
    Hey,
    We use NatCorder to record AR (multiple cameras).
    To avoid crashes, I lowered resolution and skip encoding every x frames.

    Using this approach, it was basically a game of compiling and tweaking until the recording worked on our target devices.

    However, after upgrading to Unity 2018.3, it seems that the memory usage has gone up and the app crashes on some devices.

    My ideal solution to get around this would be to limit encoding by how many frames are waiting for enqueue.

    Would you consider adding an API to get the current count of frames waiting for enqueue, or to limit the length of the queue (dropping frames if its full)?

    Thanks
     
  21. nikosurfing

    nikosurfing

    Joined:
    Mar 11, 2014
    Posts:
    29
    Hi, lanre, how to do that or how to code like that? I am not coder, please be easy. I want record all audio in scene but NOT MICROPHONE

    UPDATE
    - I did it!! just add the code:
    audioRecorder = AudioRecorder.Create(audioListener, recordingClock);
     
    Last edited: Jan 2, 2019
    Lanre likes this.
  22. TobiasW

    TobiasW

    Joined:
    Jun 18, 2011
    Posts:
    78
    Hey Lanre, done. Sorry this took so long!
     
    Lanre likes this.
  23. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    We won't be adding this because it exposes too much (it adds a lot of complexity for developers to worry about). On the topic of making sure that the device can support recording at a given resolution, we've always been forced to make this a responsibility of the developer, not NatCorder.

    The issue is that while it makes sense to impose resolution and framerate restrictions on iOS and Android (like Everyplay did), NatCorder isn't designed as a realtime video recorder. It is built to be more versatile, allowing the developer to record either in a for-loop or in realtime. We also can't resort to throttling until the record queue is flushed because the effects are very (very) visible in the recording. There's gonna be a weird jerk/timeskip in the video.

    Having to tweak the resolution for different devices is not ideal--I know. Instead, I recommend recording at a 'safe' resolution. This is 1280x720 and 1600x900 on most mid-to-high-end devices. We're also working to drastically improve recording performance so that recording doesn't add any frame time to your game's rendering; and so that frames don't get queued up. We've achieved the former in 1.4.0. The latter is more difficult because we're bottlenecked by the performance of the hardware encoder.
     
  24. OFFICIAL_bryanw

    OFFICIAL_bryanw

    Joined:
    Jul 6, 2018
    Posts:
    36
    Hey guys,

    Do you have any idea why my recorded video is coming out with a strange tint to it? I'm using the post processing stack v2 and the realtimeclock with the CameraRecorder that came with the package.

    I am also curious about the green streak at the bottom of the image...

    Picture bellow for reference:
    upload_2019-1-3_13-53-14.png
     
  25. sam598

    sam598

    Joined:
    Sep 21, 2014
    Posts:
    48
    I would be very interested in beta testing this if possible. I have a project where video frames are coming into Unity from an external device at 60fps and only recording 1/4 of the frames, but I need to check every frame coming from the device without blocking.
     
  26. DennyOctagon

    DennyOctagon

    Joined:
    Jul 12, 2018
    Posts:
    7
    hi lanre.
    i want to follow up this issue. i already sent you invoice number and thread post reference to your email. do you have any progress you can share?
     
  27. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    Hey there. Are you using linear rendering? If so, that'll explain the yellow tint. NatCorder doesn't yet support linear rendering, but we are looking at adding it in the next update.
     
    OFFICIAL_bryanw likes this.
  28. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    Shoot me an email.
     
  29. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    Sorry for the late response. I've replied.
     
  30. OFFICIAL_bryanw

    OFFICIAL_bryanw

    Joined:
    Jul 6, 2018
    Posts:
    36
    Do you know when you'll release the next version? I want to render some videos by the end of the month and I don't think I'll be able to settle for gamma color space for this

    I'm only planning to use this on Windows platform if you potentially might have a beta build I could use???

    Thanks!
     
  31. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    Shoot me an email then. I have to make some changes on Windows and I'd like to add linear support sooner rather than later. I can send you builds to test.
     
    OFFICIAL_bryanw likes this.
  32. DennyOctagon

    DennyOctagon

    Joined:
    Jul 12, 2018
    Posts:
    7
    thanks Lanre, the beta eliminate the bug. I will resume the prototype again and explore the plugin.

    cheers.
     
    Lanre likes this.
  33. BoraxKid

    BoraxKid

    Joined:
    Dec 1, 2016
    Posts:
    6
    Thank you so much! Vulkan was the problem for me on Android too
     
  34. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    Sounds good. NatCorder doesn't support Vulkan yet.
     
  35. Achoo

    Achoo

    Joined:
    Jun 4, 2015
    Posts:
    8
    Hi @Lanre

    iOS 12 record button causes instant crash. the memory utilized keeps increasing and results in crash.
     
  36. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    What device, and what recording resolution?
     
  37. BoraxKid

    BoraxKid

    Joined:
    Dec 1, 2016
    Posts:
    6
    Hi,

    Great plugin, works well for the MP4 recording.
    Now I'm trying to record a GIF, it works on iOS and Windows, but on Android it outputs a gif with weird colors?
    Here's an album with the 3 GIF from each platform : https://imgur.com/a/sJCmvBY

    Do you have an idea what I'm doing wrong ?
     
  38. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    Your code is likely not at fault here. I'll look into this. Thanks for the imgur, it's really useful.
     
  39. DennyOctagon

    DennyOctagon

    Joined:
    Jul 12, 2018
    Posts:
    7
    hi Lanre,

    suddenly our apps received DllNotFoundException so we must update the NatCorder to beta version you provided. android version build went well but iOS version build was failed.
    hereby the screenshoot of the error:
    Screen Shot 2019-01-10 at 13.54.23.png

    on this app we only use NatCorder and I make sure the libNatRender.a and libNatCamRenderPipeline.a only one in our project folder.

    thanks in advance, and i'll be glad to help if you need anything else to give you more insight on this issue.
     
  40. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    You need to delete NatCorder from your project before importing an updated build.
     
  41. DennyOctagon

    DennyOctagon

    Joined:
    Jul 12, 2018
    Posts:
    7
    hi Lanre, this solved the issue. thank you very much. i will take a note of this.
     
    Lanre likes this.
  42. Geneworm

    Geneworm

    Joined:
    Jun 26, 2017
    Posts:
    17
    Hi Lanre, do you know if this has been tested and confirmed working on Android 18 - 22? Every time I test these versions with a virtual device on Android Studio it crashes but I know these are not the most reliable representations.
     
  43. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    Yes it has. NatCorder isn't intended to run on the simulator, given that it uses some of the lower-level Android infrastructure that might not be emulated in the simulator.
     
  44. Geneworm

    Geneworm

    Joined:
    Jun 26, 2017
    Posts:
    17
    No problem thanks, I suspected as much from the errors just wanted to confirm.
     
    Lanre likes this.
  45. Geneworm

    Geneworm

    Joined:
    Jun 26, 2017
    Posts:
    17
    In case this is of any use to you I've had 16 unique user crash reports around "com.yusufolokoba.natcorder.mp4.Encoder.drainEncoder", all of these have been for Oppo devices running Android 7.1 or 8.1.

    Device list:

    F9 (CPH1823)
    CPH1725 (CPH1725)
    F7 (CPH1819)
    CPH1723 (CPH1723)
    CPH1859 (CPH1859)
    F7 (CPH1821)
    A73 (CPH1837)
    F9 (CPH1825)
     
  46. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    Is it a null reference exception? The next version should fix this.
     
  47. BoraxKid

    BoraxKid

    Joined:
    Dec 1, 2016
    Posts:
    6
    Thank you, did you find anything ? Maybe you want some more information from me ?
     
  48. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    I don't need anything more. This is likely an artifact from our error diffusion dithering implementation.
     
  49. DutchdaveTT

    DutchdaveTT

    Joined:
    May 4, 2018
    Posts:
    12
    Is this for the Unity Editor only?

    Or can one include it in a game so players themselves can record gameplay etc?
     
  50. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,261
    No it isn't Editor only. NatCorder supports Windows Editor, Windows Standalone, macOS Editor, macOS Standalone, iOS, Android, and WebGL.