Search Unity

  1. Click here to see what's on sale for the "Best of Super Sale" on the Asset Store
    Dismiss Notice
  2. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice

NatCorder - Video Recording API

Discussion in 'Assets and Asset Store' started by Lanre, Nov 18, 2017.

  1. pjccccc

    pjccccc

    Joined:
    Oct 7, 2015
    Posts:
    43
    [DETAIL]
    It's not possible to record video with Microphone in current version. (I'm not sure, you have already added something in latest beta)

    Assigning AudioClip to AudioSource.clip is not a solution. It makes echo effect
    (Microphone -> Speaker -> MIcrophone -> Speaker
    ......... forever)
     
  2. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    You can record on Windows and macOS (using FFmpeg), not Windows Universal.
    You can share videos using the native sharing UI (provided by iOS/Android).
    No, this is not possible. The native sharing UI simply offers all sharing options to the user.
    Yes. Whether you use NatCam or WebCamTexture, you can record anything that can be rendered to a texture.
     
  3. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    NatCorder doesn't support sharing to specific places, like YouTube. The Share function provided by the Sharing API uses the path to the video to open the native sharing modal for the OS.
     
  4. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    You can't record with an `AudioClip` (we already considered adding this, it poses too many issues). Instead, you should use the `AudioSource` overload and hook up your `AudioClip` to play from that audio source. We have added a boolean in the `StartRecording(..., AudioSource)` overload that will prevent the echo effect. We are designing an example scene that will properly illustrate the workflow for recording a replay with microphone audio.
     
  5. magglemitch

    magglemitch

    Joined:
    Dec 8, 2013
    Posts:
    83
    Loving the plugin so far. Want to set up sharing so I can test, and I've been trying to use the code from the medium article.

    Some things are coming up red as the names don't exist. I'm heading the script with "using NatCorderU.Core;" - was there anything else I should be adding so that it recognizes the sharing namespaces?

    Thanks
     
  6. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    I believe you need to import NatCorderU.Extensions. We might change this before the release version is completed.
     
  7. magglemitch

    magglemitch

    Joined:
    Dec 8, 2013
    Posts:
    83
    Thanks. This added a couple of the names, however, the "DisplayThumbnail" is red, and so is "pathToReplay". Any idea what I add to make those work?
     
  8. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    The gist you are referring to simply uses made-up names to illustrate using the Sharing API. In practice, these will be functions that do things with the video thumbnail and video path respectively.
     
  9. magglemitch

    magglemitch

    Joined:
    Dec 8, 2013
    Posts:
    83
    Oh right - so it doesn't already have sharing set up? Is it to be used with another sharing plugin or something?
     
  10. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    I'm not sure what you mean. The Sharing API exposes the Share and GetThumbnail API's. The gist you are referencing was made up to illustrate how these API's would be used. It isn't actually part of an example or project.
     
  11. magglemitch

    magglemitch

    Joined:
    Dec 8, 2013
    Posts:
    83
    Ah no prob. I was hoping there'd be an example or documentation on how to share the video. I'll mess around with it until I come up with something. Thanks!
     
    Lanre likes this.
  12. smoothblur

    smoothblur

    Joined:
    Mar 21, 2010
    Posts:
    4
    I'd like to get access to the current beta of the NatCorder API. If possible let me know.
     
  13. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    Totally--the beta is still available for testing. Shoot me a PM with your email address and I'll share it with you.
     
  14. pjccccc

    pjccccc

    Joined:
    Oct 7, 2015
    Posts:
    43
    My version doesn't have `StartRecording` with boolean param.

    Did you added it to latest beta? Can you give me again?
     
  15. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    We haven't made another beta build yet; we are still making additions and bug fixes.
     
  16. pjccccc

    pjccccc

    Joined:
    Oct 7, 2015
    Posts:
    43
    If you give me some examples how to recording with mic, it really helps me.

    I already changed all of my code with NatCorder instaed Everyplay.

    Thanks
     
  17. henriqueranj

    henriqueranj

    Joined:
    Feb 18, 2016
    Posts:
    151
    Hello Lanre,

    I tried your Beta build with the sample project and I am experiencing some inconsistent behaviours for iOS:
    - In iPhones it is working fine (iPhone 6s with iOS 11, iPhone 5s with iOS 10, iPhone 5c with iOS 9) - it records a video and properly shows the replay afterwards.
    - In iPads:
    • iPad 5 (Air - A1474) with iOS 11 - sometimes crashes when debugging via XCode with no relevant logs; when it doesn't crash it does not show the video for the replay.
    • iPad 4 (A1458) with iOS 10 - sometimes crashes when debugging via XCode and gives the log mentioned in the "spoiler" bellow; gives same behaviour as the iPad above.
    • iPad Air 2 (A1566) with iOS 11 - works fine, generates a video and shows the replay.
    NatCorder: Initialized NatCorder 1.0 iOS backend
    NatCam Dispatch: Initialized main dispatch
    UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
    UnityEngine.Logger:Log(LogType, Object)
    UnityEngine.Debug:Log(Object)
    NatCorderU.Core.Platforms.NatCorderiOS:StartRecording(Configuration, SaveCallback, IAudioSource)
    NatCorderU.Core.NatCorder:StartRecording(Configuration, SaveCallback)
    NatCorderU.Core.Replay:StartRecording(Camera, Configuration, SaveCallback)
    NatCorderU.Examples.RecordScreen:ToggleRecording()
    NatCorderU.Examples.RecordScreen:Update()

    (Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)
    2018-01-10 14:15:23.870298+0100 natcorderbeta[587:242761] NatCorder Error: Failed to record video frame for time 17779154944 due to invalid recorder state: 3
    (repeats over every frame...)
    2018-01-10 14:15:27.912812+0100 natcorderbeta[587:242761] NatCorder Error: Failed to record video frame for time 22284019712 due to invalid recorder state: 3
    2018-01-10 14:15:27.913229+0100 natcorderbeta[587:242761] NatCorder Logging: Encoder detected EOS marker
    2018-01-10 14:15:27.913672+0100 natcorderbeta[587:242761] NatCorder Logging: Encoder finishing
    Saved recording to: file:///var/mobile/Containers/Data/Application/CA9C806F-FFFC-474A-8236-3E0CC917EB6E/Documents/recording_2018_01_10_14_15_23_339.mov
    UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
    UnityEngine.Logger:Log(LogType, Object)
    UnityEngine.Debug:Log(Object)
    NatCorderU.Examples.RecordScreen:OnReplay(String)
    NatCorderU.Core.Platforms.<OnVideo>c__AnonStorey1:<>m__0()
    System.Collections.Generic.List`1:ForEach(Action`1)
    NatCamU.Dispatch.IDispatch:Update()
    NatCamU.Dispatch.<OnFrame>c__Iterator0:MoveNext()
    UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

    (Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)
    NatCam Dispatch: Released dispatcher
    UnityEngine.DebugLogHandler:LogFormat(LogType, Object, String, Object[])
    UnityEngine.Logger:Log(LogType, Object)
    UnityEngine.Debug:Log(Object)
    NatCamU.Dispatch.IDispatch:Release()
    NatCorderU.Core.Platforms.<OnVideo>c__AnonStorey1:<>m__0()
    System.Collections.Generic.List`1:ForEach(Action`1)
    NatCamU.Dispatch.IDispatch:Update()
    NatCamU.Dispatch.<OnFrame>c__Iterator0:MoveNext()
    UnityEngine.SetupCoroutine:InvokeMoveNext(IEnumerator, IntPtr)

    (Filename: /Users/builduser/buildslave/unity/build/artifacts/generated/common/runtime/DebugBindings.gen.cpp Line: 51)
    [prepareAsset]Error: The operation could not be completed
    An unknown error occurred (-12893)

    Have you ever experienced this?
     
  18. henriqueranj

    henriqueranj

    Joined:
    Feb 18, 2016
    Posts:
    151
    As a follow up to my post, I tried adding Vuforia 7 to the project (with Unity 2017.3.0f3). Due to the default nature of the new Vuforia library, it activates itself on the main camera and shows the camera view in the background of the NatCorder sample scene.

    When I started recording, the app crashes immediately on all devices except for the iPhone 5c with iOS 9. While debugging in XCode, it breaks in an EXC_BAD_ACCESS (code=1, address=0x0) error.

    Could you consider looking into this interaction of NatCorder and Vuforia?
     
  19. robson_depaula

    robson_depaula

    Joined:
    Sep 30, 2014
    Posts:
    37
    I would like to participate in the beta, this looks really interesting.

    Best Regards
     
    Lanre likes this.
  20. leonardo_demartino

    leonardo_demartino

    Joined:
    Jan 5, 2018
    Posts:
    3
    On Android it seems to work ok, but in iOS:
    09.3.1 OK
    10.3.2 CRASH when it tries to start recording EXC_BAD_ACCESS (code=1, address=0x0)
    11.2.1 CRASH when it tries to start recording EXC_BAD_ACCESS (code=1, address=0x0)
     
  21. henriqueranj

    henriqueranj

    Joined:
    Feb 18, 2016
    Posts:
    151
    @leonardo_demartino Did you also have Vuforia active while you tried to record?
     
  22. leonardo_demartino

    leonardo_demartino

    Joined:
    Jan 5, 2018
    Posts:
    3
    Nope, I don't use Vuforia at all.
     
  23. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    We are still building the examples. Sorry for the delay; I've been really busy over the past few days.
     
  24. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    I have come across this sometime in the past, but I wasn't able to find the cause. What recording configuration were you using? Log the configuration width, height, framerate, and bitrate and let me know what it is. The issue might be happening because the encoder doesn't support the configuration.
     
    henriqueranj likes this.
  25. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    This occurs because of native plugin rendering. iOS doesn't support dynamic linking, so you have to manually register a render delegate in native code. The problem is that Unity's API for doing so doesn't account for more than one plugin needing a native render delegate. The fix goes something like this:

    - Build your Xcode project and open it, then open Vuforia's VuforiaNativeRendererController.mm file (not sure if it is still called this, but it should).
    - Open NatCamRenderDispatch.mm and copy the prototype definitions for VuforiaSetGraphicsDevice and VuforiaRenderEvent from the Vuforia controller file into this one.
    - Import VuforiaRenderDelegate.h into NatCamRenderDispatch.mm and in shouldAttachRenderDelegate, set the render delegate to a new instance of VuforiaRenderDelegate (basically copy what is in the same function in VuforiaNativeRendererController.mm).
    - In OnRenderDispatch, invoke VuforiaRenderEvent with passing in the eventID argument.
    - Comment out the entire VuforiaNativeRendererController source file.
     
    henriqueranj likes this.
  26. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    Please email me a screenshot of the stack trace in Xcode and send me the full logs (from app start).
     
  27. henriqueranj

    henriqueranj

    Joined:
    Feb 18, 2016
    Posts:
    151
    I was using the Default Configuration from the NatCorder API. It may have something to do with the Screen.width and Screen.height as the video resolution being too much for these devices. I will try applying specific settings and test it again as soon as I have some time.

    Thank you for these instructions, I will also give them a try when I have some time. Do you foresee a more seamless way for NatCorder to integrate with other plugins that require a native render delegate? I think a lot more people will try to use NatCorder for AR apps.
     
  28. robson_depaula

    robson_depaula

    Joined:
    Sep 30, 2014
    Posts:
    37
    Thanks for the beta package Lanre.

    Integration was a breeze, the performance is great (only tested default quality) and I can confirm it is capable of recording ARCore sessions.

    Only thing I noticed is that the UI in overlay is not recorded which could be ok in case this is not required. Anyway, this is stated in your readme with a possible workaround to achieve this.

    Great job!
     
    Lanre likes this.
  29. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    Yes, do let me know the results.
    Not until Unity updates their native render plugin registration API. I will update the readme with instructions, but the process will not be uniform across different native render plugins because they all register for native rendering in different ways (NatCorder and Vuforia are a good example).
     
  30. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    Perfect!
    This has been a known issue with Unity (I don't think they've addressed it). When the UI canvas mode renders as an overlay, it isn't visible in calls to Graphics::Blit. You have to change the UI canvas render mode to screen space camera or world space.
     
  31. peterahou

    peterahou

    Joined:
    May 19, 2015
    Posts:
    48
    Thanks for sharing the beta.
    I did a really quick and simple first test on iOS with just an animated Mixamo character and the basic recording functionality:
    Code (CSharp):
    1. Replay.StartRecording(Camera.main, Configuration.Default, HandleSessionComplete, audioListener);
    Unfortunately, the output video appears to be quite glitchy with some stuttering and flickering. I used an iPhoneX with iOS 11.1.1 to do the recording. The app was built using Unity 2017.3.0p2 and Xcode 9.2.
    Any ideas what might be causing these visual errors?
    https://drive.google.com/file/d/1y1kxT71wihNJbchobrr_8R0YrzMt02C0/view?usp=sharing

    Edit: tested on an iPhone 6 with the same results.
     
    Last edited: Jan 16, 2018
  32. peterahou

    peterahou

    Joined:
    May 19, 2015
    Posts:
    48
    Ah, I think I found the problem. Even your simple spinning cube in the RecordScreen example scene starts glitching if the framerate is set to 60. Do I need to configure something to make 60fps work glitch free?
    Code (CSharp):
    1.         void Awake()
    2.         {
    3.             Application.targetFrameRate = 60;
    4.         }
     
  33. henriqueranj

    henriqueranj

    Joined:
    Feb 18, 2016
    Posts:
    151
    Hi @peterahou , I had the same effects in Android devices when I was messing about with the bitrate in the video recording configuration. I don't know much yet about the details of bitrate vs framerate vs video resolution, but I would suggest you take a look into that.
     
  34. peterahou

    peterahou

    Joined:
    May 19, 2015
    Posts:
    48
    Thanks for the suggestion :)
    Unfortunately, I don't see any difference – the glitches still appear even when I double the bitrate used by the default configuration and reduce the resolution to 1280x720.
    https://drive.google.com/file/d/1IRo4w6-_791gCU0pFQc6NkTo497tKWZZ/view?usp=sharing
     
    Lanre and henriqueranj like this.
  35. mrmdesign

    mrmdesign

    Joined:
    Apr 18, 2013
    Posts:
    239
    I was planning on buying EasyAR pro for the video recording feature but have not been able to get it to work at all, even though I have a pro trial key.

    NatCorder was up and running in minutes! Great job!
    I have experienced a few crashes on Android and it usually happens on the second recording. I will try tweaking a few things and am sure I will figure it out.

    The good news is that it works with EasyAR basic which means it will be very useful for people wanting to record image tracking AR.

    What price are you planning to put on it when it is released? (apologies if you have already answered this)
     
  36. peterahou

    peterahou

    Joined:
    May 19, 2015
    Posts:
    48
    So, just a quick follow up. It appears that I can only get a glitch free recording if I force the framerate to 30. It's not enough to use a Configuration with the framerate parameter set to 30 making the recorder run at 30fps, I have to set Application.targetFrameRate to make everything run at 30fps.
    Is anyone else using iOS and able to get different results?
     
  37. mrmdesign

    mrmdesign

    Joined:
    Apr 18, 2013
    Posts:
    239
    Further testing: As soon as I hit record in my EasyAR powered app it does start feeling sluggish. There is definite stuttering when I move my android phone that isn't there when I am not recording. If I leave it recording for too long (30 seconds) it crashes the app.
     
  38. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    That's a good find. I have opened an issue for this and I will try to reproduce the issue.
    I've noted this. This looks like a bug in both the Replay API and NatCorder.
     
  39. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    Bitrate shouldn't affect this. The issue comes around because Unity isn't completing the call to Blit on the frame before NatCorder encodes the frame.
     
  40. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    Can you email me the full logs from logcat with the crash? It will be very useful in debugging.
    We haven't settled on a price yet. I'll make sure to post once we do.
     
  41. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    This makes sense with our current suspicions. I'll work on a fix and follow up with you for testing.
     
  42. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    What device are you testing on? Are you always running AR, even when you aren't recording? Regarding the crash after a while, send me the logs for that.
     
  43. mrmdesign

    mrmdesign

    Joined:
    Apr 18, 2013
    Posts:
    239
    I am using a Galaxy S7 Edge to test on and I am running the AR constantly.
     
  44. peterahou

    peterahou

    Joined:
    May 19, 2015
    Posts:
    48
    I decided to move on to the sharing API and see if I could get it to work on iOS. Unfortunately, it doesn't seem to work at all. Has anyone had any luck with this?

    Edit: Ah, my bad. Looking at the source I see that this has not been implemented yet :)
     
    Last edited: Jan 18, 2018
    Lanre likes this.
  45. mrmdesign

    mrmdesign

    Joined:
    Apr 18, 2013
    Posts:
    239
    Quick update: If I turn MSAA off on the recording camera, it no longer crashes and its as smooth as butter!

    I spoke to soon. It seemed smoother but I may have imagined it as the stuttering is still there if I move the phone too quickly and it still crashes after a while. I will send a crash log.
     
    Last edited: Jan 18, 2018
  46. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,445
    I've had similar reports of MSAA and stuttering. Do send me the logs.
     
  47. peterahou

    peterahou

    Joined:
    May 19, 2015
    Posts:
    48
    Well, I'm seeing issues with stuttering and glitches both with and without MSAA. Also, it appears that my animated character moves faster in the video than it does in the realtime scene when I play it back using Unity's built in video player. In fact, the last frame of the video is displayed for a couple of seconds at the end of the video, suggesting some framerate mismatch or something like that. This happens at both 30 and 60fps.
     
  48. Mr-Oliv

    Mr-Oliv

    Joined:
    Sep 14, 2012
    Posts:
    27
    Hi! Thanks for putting together this asset!
    Just tried exporting the example project (the rotating cube), but the app freezes when I double tap. Is there something I need to do when exporting to make it work?
    I'm on a Google Pixel, latest Android.
    Thanks

    NullReferenceException: Object reference not set to an instance of an object: at NatCorderU.Core.Replay+CameraRecorder.OnRenderImage (UnityEngine.RenderTexture src, UnityEngine.RenderTexture dst) [0x00000] in <filename unknown>:0
    (Filename: Line: -1)
    NatCorder Error: Cannot acquire frame when NatCorder is not recording
     
  49. aRayon

    aRayon

    Joined:
    Oct 27, 2016
    Posts:
    4
    Hi all, I would love to participate in the BETA program. Recording video and audio within our APPs is a challenge that I have had to face many times and never with a 100% success... Thank you very much for working on this!
     
  50. zhoujinhao2599

    zhoujinhao2599

    Joined:
    Oct 23, 2017
    Posts:
    2
unityunity