Search Unity

  1. New Unity Live Help updates. Check them out here!

    Dismiss Notice

NatCorder - Video Recording API

Discussion in 'Assets and Asset Store' started by Lanre, Nov 18, 2017.

  1. VisalDXP

    VisalDXP

    Joined:
    May 16, 2014
    Posts:
    7
    Hi There,
    Is the Beta test still open? Can I join?

    Thanks
     
  2. henriqueranj

    henriqueranj

    Joined:
    Feb 18, 2016
    Posts:
    144
    Hi Lanre, is this update already on review for the Asset Store or still to be released?
     
  3. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    Hi there. The beta isn't available anymore as NatCorder is now available on the Asset Store.
     
  4. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    It has been submitted for review. I expect Unity to approve it within the next few days.
     
  5. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    The update is live! Here is the NatCorder 1.1 changelog:

    + We have added a native macOS backend! The NatCorder recording API is now fully supported on macOS.
    + We have also added a native Windows backend! This backend is still experimental so it should not be used in production builds.
    + The Standalone backend (using FFmpeg) has been deprecated because we have added native Windows and macOS implementations.
    + We have significantly improved recording stability on Android especially for GPU-bound games.
    + Added support for different Unity audio DSP latency modes.
    + Added `sampleCount` property in `IAudioSource` interface.
    + Fixed crash when `StartRecording` is called on iOS running OpenGL ES2 or ES3.
    + Fixed rare crash on Android when recording with audio.
    + Fixed audio-video timing discrepancies on Android.
    + Fixed video tearing on Android when app does not use multithreaded rendering.
    + Fixed `FileUriExposedException` when `Sharing.Share` is called on Android 24 or newer.
    + Fixed `Sharing.GetThumbnail` not working on iOS.
    + Fixed `Sharing.SaveToCameraRoll` failing when permission is requested and approved on iOS.
    + Fixed `Sharing.SaveToCameraRoll` not working on Android.
    + Fixed rare crash on Android when a very short video (less than 1 second) is recorded.
    + Fixed build failing due to missing symbols for Sharing library on iOS.
    + Improved on microphone audio stuttering in ReplayCam example by adding minimal `Microphone` API in `NatCorderU.Extensions` namespace.
    + Refactored `Configuration.Default` to `Configuration.Screen`.
     
    henriqueranj likes this.
  6. nerkderk

    nerkderk

    Joined:
    Oct 24, 2017
    Posts:
    14
    Hi Lanre, for some reason Unity thinks that I'm posting spam when I reply to our PMs, and won't actually let me reply, so I am replying here:

    That was exciting news! I downloaded the update a little while ago and have been playing with it. If I build your ReplayCam scene alone, it works fine on my android device (pixel 2 xl). If I take the record button, preview, and replay cam object/script and move it into the scene for my app, it makes my app run choppy still (but less so than before) and after I record I only see a black screen. Is there some way I can resolve these issues? Thanks
     
  7. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    Hey there. Can you email me a repro project? Do you have multithreaded rendering enabled in Player Settings? This is almost a requirement to make recording go smoothly. Regarding the black screen, can you also email me the full logs from logcat? I need to verify that your setup is correct.
     
  8. HansAppLab

    HansAppLab

    Joined:
    Dec 7, 2017
    Posts:
    6
    Hi Lanre.
    Congratulations on the release of new version!
    I'm testing NatCorder 1.1 microphone recording with ReplayCam example.
    When I record multiple times, only the first voice will be recorded in video.
    It seems 'Extensions.Microphone' is keeping first recorded AudioClip.
    Did you ever seen this issue?
     
  9. HansAppLab

    HansAppLab

    Joined:
    Dec 7, 2017
    Posts:
    6
    PERPETUAL_CLIP is defined by default.
    Therefore, previous 'clip' is not set to null and is reused when I call StartRecording.
    I don't understand why reuse previous AudioClip.
    Code (CSharp):
    1. public static AudioClip StartRecording (string deviceName = null) {
    2.     #if !UNITY_WEBGL || UNITY_EDITOR // No `Microphone` API on WebGL :(
    3.     clip = clip ?? UnityEngine.Microphone.Start(deviceName, true, 60, 48000);
    4.     while (UnityEngine.Microphone.GetPosition(deviceName) <= 0) ;
    5.     #endif
    6.     return clip;
    7. }
    8. public static void StopRecording (string deviceName = null) {
    9.     #if !PERPETUAL_CLIP && (!UNITY_WEBGL || UNITY_EDITOR)
    10.     UnityEngine.Microphone.End(deviceName);
    11.     clip = null;
    12.     #endif
    13. }
     
  10. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    We were testing this for microphone stuttering. You can simply comment out the PERPETUAL_CLIP definition and it should be fine. When the definition is enabled, NatCorder will keep the microphone running in the background. The reason why it plays audio from a previous time is because you also have to set the `AudioSource.timeSamples` to `Microphone.GetPosition(deviceName)` (this only applies when PERPETUAL_CLIP is enabled). We have added a fix to this on our end.
     
  11. seijik42

    seijik42

    Joined:
    Sep 8, 2013
    Posts:
    5
    Hi, I purchased this asset today and have been playing with example scenes.
    Generally, good work!
    I'm trying to create my custom scene by modifying with ReplayCam scene and actually I already have some questions.

    1. the volume of recorded audio is really low.How can I adjust this? I enabled "Record Microphone Audio" in inspector and other settings are default.
    2. I cannot find recorded videos in my device. How can I save video in cameraroll?

    asset ver: NatCorder 1.1f1
    device: iPhoneX
    os: iOS 11.2.6

    Thanks!
     
  12. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    This might be affected by your device volume. Also, make sure that the AudioSource in your scene has its volume set to full and that it is spatially close to the AudioListener in the scene. This is because Unity's OnAudioFilterRead provides audio samples from the 'viewpoint' of the AudioListener, not the AudioSource. So you have to account for volume attenuation.
    Videos are saved to the app's private documents directory. To add them to the camera roll, use the Sharing.SaveToCameraRoll function (in the NatCorderU.Extensions namespace). Check the README.md file and the docs for more info.
     
  13. seijik42

    seijik42

    Joined:
    Sep 8, 2013
    Posts:
    5
    Thank you for quick response.
    I found out video saved in camera roll has enough volume. Only video in playback played with Handheld.PlayFullScreenMovie(path) has very low even after moving AudioSource closer to AudioListener. I will try to find the cause.
    Anyway, your comment was helpful. Will play with this deeper tomorrow, thanks!
     
  14. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    That sounds good. Maybe it's a Unity bug.
     
  15. nikescar

    nikescar

    Joined:
    Nov 16, 2011
    Posts:
    105
    Hi Lanre,

    Just saw NatCorder and it looks great.

    I'm wondering if there a recording option to record the last n-seconds of gameplay? (like the Xbox Live record functions)

    - Wes
     
  16. dustin_red

    dustin_red

    Joined:
    Feb 7, 2018
    Posts:
    46
    So just making sure I understand:
    • StartRecording with AudioListener for in-game audio.
    • StartRecording with AudioSource (clip from NatCorderU.Extensions.Microphone.StartRecording) and audioIsRecordOnly true for microphone audio.

    But how do I record the microphone AND the in-game audio at the same time?
     
  17. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    Hi Wes. This isn't possible. NatCorder records directly to a file. One option you could explore is after recording a video, trim the video leaving only the last 8 seconds.
     
  18. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    Yes, this is correct.
    Recording from an AudioSource in the scene is not limited to just the Microphone. You can use it to record any particular AudioSource. The reason why we have the `audioIsRecordOnly` flag is because we usually don't want the microphone audio to play back in the scene (we just want to record it).
    You will have to do some manual mixing! You will need to create a component that gets the microphone audio (using the OnAudioFilterRead callback on an AudioSource playing the mic audio) and saves it into some sample buffer; then mute the microphone audio so that the mic isn't heard in the scene; then add another component to the AudioListener that also implements OnAudioFilterRead for capturing scene audio; then mix the two sample buffers together and commit the result to NatCorder (using NatCorder.CommitSamples).
     
  19. VisalDXP

    VisalDXP

    Joined:
    May 16, 2014
    Posts:
    7
    Hi Lanre,

    First of all, great job at making this asset, I love it! Save me a lot of time tackling video recording problem.
    I'm trying to record video with certain aspect ratio (say my camera render fullscreen 9:16 and I want to record a portion of it like 4:5), I use
    Replay.StartRecording(worldCamera, new Configuration(width, height), OnReplay)
    and it's actually output the video's resolution I wanted but the content stretched out to fit that resolution.
    So my question is, is it possible to record video with the resolution I want without stretching its content?
     
    kf1 likes this.
  20. mrmdesign

    mrmdesign

    Joined:
    Apr 18, 2013
    Posts:
    239
    Thank you @Lanre!
    The new release has stopped all of my crashes. I still can not find the videos in my Camera Roll on Android though.
    I am using Sharing.SaveToCameraRoll(this.pathToReplay);
     
  21. jbvobling

    jbvobling

    Joined:
    Sep 26, 2017
    Posts:
    27
    Hi, i used your updated plugins.

    -I tried to record a voice via microphone, its good and it works. Then, I tried to add a sample in-game sound when a button is pressed, the in-game sound is fine when i click the button but when I record the in-game sound, it becomes very different(The normal sound is something like a "swoosh-sound", but in the replay video, it became "TSAAAAA-sound" ), this is when i'm using Samsung S7 Edge. I tried it with another device(HUAWEI P9), and in that device, it's just fine. Any idea why it's like that?

    -I tried to record an in-game sound sample without microphone by unchecking the RecordMicrophoneAudio in ReplayCam.cs, but it seems i can't hear any in-game sound in the video when it is being replayed and the file also don't have the in-game sound.
     
  22. jofa2442

    jofa2442

    Joined:
    Mar 28, 2018
    Posts:
    1
    Have been using this plugin on iOS for about a week and a half with no issues. Building an AR app using Vuforia 7 and ground plane tracking. Most of my testing has been on the 6S and iPhone X and I have not had issues. However on the iPhone 7 Plus after a few seconds of recording I can see the framerate of the application drop drastically and xcode will throw the following error:


    Code (csharp):
    1. 2018-03-28 15:43:45.660075-0300 gowpoc[1526:1117525] -[MTLRenderPipelineDescriptorInternal validateWithDevice:], line 2400: error 'No valid pixelFormats set.'
    2. -[MTLRenderPipelineDescriptorInternal validateWithDevice:]:2400: failed assertion `No valid pixelFormats set.'
    has anyone else ran into similar issues?
     
  23. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    This is a fair observation. Because the `Replay` API uses Graphics.Blit, which basically just draws a fullscreen quad, a difference in aspect ratio will result in stretching. To resolve this, your two options are to: make sure that the aspects match; or modify ReplayCam's VideoRecorder component to blit with a material that corrects for the aspect difference (I'm exploring this option for the next update).
     
  24. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    Can you email me with your code? What Android device and OS do you use?
     
  25. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    I'm not sure why this could be happening. It will be useful if you can send me the video.
    The ReplayCam example doesn't record game audio. It either records microphone audio (when the flag is enabled), or no audio at all.
     
  26. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    Can you email me the full logs from Xcode?

    EDIT: I did some digging and it looks like a Unity bug. See this and this.
     
  27. mrmdesign

    mrmdesign

    Joined:
    Apr 18, 2013
    Posts:
    239
    Thank you for the reply. I am using a Samsung Galaxy S7 Edge running Android 7.0
     
    Lanre likes this.
  28. HansAppLab

    HansAppLab

    Joined:
    Dec 7, 2017
    Posts:
    6
    Hi Lanre,

    I sent you email about screen overlapping issue on iOS.
    Please check your email and reply me.
     
  29. mressier

    mressier

    Joined:
    Mar 29, 2018
    Posts:
    2
    Hello,
    I'm trying to use the NatCorder start and stop recording. It works pretty fine on Unity but when I launch it on ipad mini I got this error :
    No more log or error about NatCorder between start and stop, so is there any way to know why NatCorder didn't start record ? Is there any known issue about NatCorder on ipad ?

    Have a good day :)
     
  30. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    Got it. I'll respond soon.
     
  31. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    Can you enable verbose mode (NatCorder.Verbose = true), start and stop recording, and send me the logs? This might be happening because the managed layer isn't properly forwarding calls to the iOS implementation. It should be a trivial thing to fix. Shoot me an email with the logs and we'll go from there.
     
  32. mressier

    mressier

    Joined:
    Mar 29, 2018
    Posts:
    2
    Thanks for your reply.
    Found the answer, it was really weird and completely my fault. I use a personnal tool to build my unity projet to ios, and it erase the NatCorder's build variables. So it creates this weird issue...
     
  33. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    That sounds good. Thanks for letting me know.
     
  34. ahmetardal

    ahmetardal

    Joined:
    Mar 15, 2018
    Posts:
    3
    Hi @Lanre

    My app crashes when I try to record an AR scene with NatCorder. The device is iPad Pro 12.9", with iOS 11.2.6.

    Here's the log:
    Code (CSharp):
    1. NatCorder: Initialized NatCorder 1.1 iOS backend
    2. UnityEngine.EventSystems.ExecuteEvents:GetEventHandler(GameObject)
    3. UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
    4. UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
    5. UnityEngine.Debug:LogErrorFormat(Object, String, Object[])
    6. UnityEngine.Logger:Log(LogType, Object)
    7. Newtonsoft.Json.Utilities.CollectionWrapper`1:System.Collections.IList.Insert(Int32, Object)
    8. UnityEngine.Debug:Log(Object)
    9. NatCorderU.Core.Platforms.NatCorderiOS:.ctor()
    10. NatCorderU.Core.NatCorder:.cctor()
    11. UnityEngine.EventSystems.ExecuteEvents:GetEventHandler(GameObject)
    12. NatCorderU.Core.Replay:StartRecording(Camera, Configuration, VideoCallback, IAudioSource)
    13. SurrealKit.MainSceneManager:StartRecording()
    14. UnityEngine.EventSystems.ExecuteEvents:GetEventHandler(GameObject)
    15. System.IComparable`1:CompareTo(T)
    16. System.IComparable`1:CompareTo(T)
    17. System.IComparable`1:CompareTo(T)
    18. (Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)
    19.  
    20. NatCam Dispatch: Initialized main dispatch
    21. UnityEngine.EventSystems.ExecuteEvents:GetEventHandler(GameObject)
    22. UnityEngine.DebugLogHandler:Internal_Log(LogType, String, Object)
    23. UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
    24. UnityEngine.Debug:LogErrorFormat(Object, String, Object[])
    25. UnityEngine.Logger:Log(LogType, Object)
    26. Newtonsoft.Json.Utilities.CollectionWrapper`1:System.Collections.IList.Insert(Int32, Object)
    27. UnityEngine.Debug:Log(Object)
    28. NatCamU.Dispatch.MainDispatch:.ctor()
    29. NatCorderU.Core.Platforms.NatCorderiOS:StartRecording(Configuration, VideoCallback, IAudioSource)
    30. NatCorderU.Core.NatCorder:StartRecording(Configuration, VideoCallback, IAudioSource)
    31. NatCorderU.Core.NatCorder:StartRecording(Configuration, VideoCallback, IAudioSource)
    32. NatCorderU.Core.Replay:StartRecording(Camera, Configuration, VideoCallback, IAudioSource)
    33. SurrealKit.MainSceneManager:StartRecording()
    34. UnityEngine.EventSystems.ExecuteEvents:GetEventHandler(GameObject)
    35. System.IComparable`1:CompareTo(T)
    36. System.IComparable`1:CompareTo(T)
    37. System.IComparable`1:CompareTo(T)
    38. (Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)
    39.  
    40. -[MTLDebugCommandBuffer waitUntilCompleted]:103: failed assertion `waitUntilCompleted on uncommitted command buffer'
    41. (lldb)
    I'll be very glad if you point me in the right direction.

    Thanks!
    Ahmet
     

    Attached Files:

  35. nikescar

    nikescar

    Joined:
    Nov 16, 2011
    Posts:
    105
    What is the memory usage of NatCorder? For example, for a 10 minutes video, does it keep all 10 minutes in memory or does it commit to disk as it records? If it does commit, what is the memory when processing the video when the video ends? IS there an example you could give that would give an idea of memory usage? I would like to use this for a mobile game. Thank you!
     
  36. VisalDXP

    VisalDXP

    Joined:
    May 16, 2014
    Posts:
    7
    Thank Lanre, I actually went to the first approach and it worked the way I wanted it to. It would be cool if you can include this feature in the future update
     
  37. hdxpmc

    hdxpmc

    Joined:
    May 1, 2016
    Posts:
    26
    Hello,
    We compile and run latest version on iOS 10.2, and exception after recording file as:

    FileNotFoundException: file:///var/mobile/Containers/Data/Application/5FC9D511-23CB-4232-B6C6-877EAE999BFE/Documents/recording_2018_03_30_15_59_59_584.mov does not exist
    at System.IO.File.Move (System.String sourceFileName, System.String destFileName) [0x00000] in <filename unknown>:0
    at ProGifWorkerSync.OnVideoRecorded (System.String path) [0x00000] in <filename unknown>:0
    at NatCorderU.Core.Platforms.NatCorderiOS+<OnVideo>c__AnonStorey1.<>m__0 () [0x00000] in <filename unknown>:0
    at System.Collections.Generic.List`1[T].ForEach (System.Action`1 action) [0x00000] in <filename unknown>:0
    at NatCamU.Dispatch.DispatchUtility+<OnFrame>c__Iterator0.MoveNext () [0x00000] in <filename unknown>:0
    at UnityEngine.SetupCoroutine.InvokeMoveNext (IEnumerator enumerator, IntPtr returnValueAddress) [0x00000] in <filename unknown>:0
    at Omanju.ByteClap.Processors.IFixedLengthElementArrayProcessor`1[TElement].GetElementSize () [0x00000] in <filename unknown>:0
    at Omanju.ByteClap.Processors.IFixedLengthElementArrayProcessor`1[TElement].GetElementSize () [0x00000] in <filename unknown>:0
    at Omanju.ByteClap.Processors.IFixedLengthElementArrayProcessor`1[TElement].GetElementSize () [0x00000] in <filename unknown>:0
    UnityEngine.UnhandledExceptionHandler:printException(String, Exception)
    UnityEngine.UnhandledExceptionHandler:HandleUnhandledException(Object, UnhandledExceptionEventArgs)
    Omanju.ByteClap.Processors.IFixedLengthElementArrayProcessor`1:GetElementSize()
    Omanju.ByteClap.Processors.IFixedLengthElementArrayProcessor`1:GetElementSize()
    Omanju.ByteClap.Processors.IFixedLengthElementArrayProcessor`1:GetElementSize()

    It seem recording successfully, when call OnVideo with the recorded path, it is not existing path to video ?
     
    Last edited: Mar 31, 2018
    LouisHong likes this.
  38. sahinboydas

    sahinboydas

    Joined:
    Mar 30, 2018
    Posts:
    1
    hi @Lanre thx for the update. great update.

    @ahmetardal , we have the same crash problem...

    @Lanre this is pretty important for us, can you prioritize this crash. thx
     
    ahmetardal likes this.
  39. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    Thanks for this report. I have created a fix for the issue. I can send you the updated native library if you'd like (email me).
     
    ahmetardal likes this.
  40. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    NatCorder only uses as much memory as there are frames waiting to be encoded by hardware. Typically, this amounts to no more than 25MB, and we are still looking for ways to reduce it. Encoded chunks are committed to device storage as they are produced by the encoders (it is managed by the OS).
     
  41. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    I think we will include this in the next release (1.1f2).
     
  42. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    Hi. This exception is generated by your code (you are trying to move the recorded video after recording). The crash is probably happening because on iOS, you have to take special care with the protocol that is prepended to the returned path ("file://"). When using System.IO API's, you must remove the protocol.
     
  43. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    Shoot me an email. It's a minimal issue and I have a fix for it.
     
  44. ahmetardal

    ahmetardal

    Joined:
    Mar 15, 2018
    Posts:
    3
    @Lanre

    Thanks! Great!
    I've just emailed you.
     
  45. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    Responded!
     
    ahmetardal likes this.
  46. hdxpmc

    hdxpmc

    Joined:
    May 1, 2016
    Posts:
    26
    Thank you,
    We have get localpath on new Uri(path).localPath, then it work
    Best regards
     
  47. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    Sounds good. Thanks for letting me know!
     
  48. hdxpmc

    hdxpmc

    Joined:
    May 1, 2016
    Posts:
    26
    Hello,
    When will window backend be official ?
    Do all platform support 4k recording ?

    Best regards
     
  49. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,962
    Probably in the next release (1.2f1).
    Practically, no. Most mobile devices do not have enough computational resources to support 4K recording. On standalone platforms (macOS, Windows), it should be possible.
     
  50. hdxpmc

    hdxpmc

    Joined:
    May 1, 2016
    Posts:
    26
    Hello,
    On iPhone 6S, it can record at 4k easily, and latest as iPhone 7/8/X much faster. It is very powerful hardware. Please add support for 4k, and we will limit max resolution in our code. for example: on iPhone 6, will allow max 720p, and newer is larger.

    Best regards
     
unityunity