Search Unity

  1. Unity 2019.2 is now released.
    Dismiss Notice

NatCorder - Video Recording API

Discussion in 'Assets and Asset Store' started by Lanre, Nov 18, 2017.

  1. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    Your sample rate and channel count arguments (when creating the MP4 recorder) are wrong. They should match that of the Unity audio engine (check AudioSettings).
     
  2. AndreaSpace1

    AndreaSpace1

    Joined:
    Dec 22, 2017
    Posts:
    2
    Hi, I bought the plugin in order to record MP4 videos on Android using the device camera. Testing the NatCorder in an empty project (only NatCorder, Unity WebCamTexture and a bit of UI to start and stop the recording, no audio), on Samsung S3 device:

    - When recording in 1920x1080 30fps no audio, the application crashes after ~7 seconds of recording.
    - When recording in 1280x720 30fps no audio, it crashes after ~30 seconds.
    - Recording at 640x360 30fps seems fine, it went on for about 10 minutes without crashing (after 10 minutes the recording was manually stopped).

    Both crashes produced the same logcat, which is attached in this post. Is there any way to record in 720p / 1080p without it crashing?

    Here the Unity code:

    Code (CSharp):
    1.  
    2. using NatCorder;
    3. using NatCorder.Clocks;
    4. using System;
    5. using UnityEngine;
    6. using UnityEngine.UI;
    7.  
    8. public class SceneController : MonoBehaviour {
    9.  
    10.     public Text FeedbackText;
    11.     public Text VideoPathText;
    12.  
    13.     protected virtual bool IsRecording { get; set; }
    14.    
    15.     protected virtual WebCamTexture WebCamTexture { get; set; }
    16.     protected virtual IClock RecordingClock { get; set; }
    17.     protected virtual IMediaRecorder MediaRecorder { get; set; }
    18.     protected virtual string VideoPath { get; set; }
    19.  
    20.     protected virtual int RecordingResolutionWidth { get; set; }
    21.     protected virtual int RecordingResolutionHeight { get; set; }
    22.  
    23.     void Awake()
    24.     {
    25.         SetRecordingResolution(640, 360);
    26.     }
    27.  
    28.     void LateUpdate ()
    29.     {
    30.         if (IsRecording)
    31.         {
    32.             var renderFrame = MediaRecorder.AcquireFrame();
    33.             Graphics.Blit(WebCamTexture, renderFrame);
    34.             MediaRecorder.CommitFrame(renderFrame, RecordingClock.Timestamp);
    35.         }
    36.     }
    37.  
    38.     public void DidTapOn360()
    39.     {
    40.         SetRecordingResolution(640, 360);
    41.     }
    42.  
    43.     public void DidTapOn480()
    44.     {
    45.         SetRecordingResolution(854, 480);
    46.     }
    47.  
    48.     public void DidTapOn720()
    49.     {
    50.         SetRecordingResolution(1280, 720);
    51.     }
    52.  
    53.     public void DidTapOn1080()
    54.     {
    55.         SetRecordingResolution(1920, 1080);
    56.     }
    57.  
    58.     public void DidTapOnStart()
    59.     {
    60.         Debug.Log($"{nameof(SceneController)}.{nameof(DidTapOnStart)}");
    61.  
    62.         if (!IsRecording)
    63.         {
    64.             try
    65.             {
    66.                 WebCamTexture = new WebCamTexture();
    67.                 WebCamTexture.Play();
    68.  
    69.                 RecordingClock = new RealtimeClock();
    70.  
    71.                 MediaRecorder = new MP4Recorder(RecordingResolutionWidth, RecordingResolutionHeight, 30, 0, 0, OnVideoRecorded);
    72.  
    73.                 IsRecording = true;
    74.             }
    75.             catch (Exception e)
    76.             {
    77.                 Debug.LogError($"{nameof(SceneController)}.{nameof(DidTapOnStart)}: ({e})");
    78.             }
    79.         }
    80.  
    81.         UpdateFeedbackText();
    82.     }
    83.  
    84.     public void DidTapOnStop()
    85.     {
    86.         Debug.Log($"{nameof(SceneController)}.{nameof(DidTapOnStop)}: {nameof(VideoPath)} ({VideoPath})");
    87.  
    88.         if (IsRecording)
    89.         {
    90.             IsRecording = false;
    91.  
    92.             try
    93.             {
    94.                 MediaRecorder.Dispose();
    95.                 MediaRecorder = null;
    96.  
    97.                 RecordingClock = null;
    98.  
    99.                 WebCamTexture.Stop();
    100.                 Destroy(WebCamTexture);
    101.                 WebCamTexture = null;
    102.             }
    103.             catch (Exception e)
    104.             {
    105.                 Debug.LogError($"{nameof(SceneController)}.{nameof(DidTapOnStop)}: ({e})");
    106.             }
    107.         }
    108.  
    109.         UpdateFeedbackText();
    110.     }
    111.  
    112.     protected virtual void SetRecordingResolution(int width, int height)
    113.     {
    114.         if (!IsRecording)
    115.         {
    116.             RecordingResolutionWidth = width;
    117.             RecordingResolutionHeight = height;
    118.         }
    119.     }
    120.  
    121.     protected virtual void OnVideoRecorded(string videoPath)
    122.     {
    123.         Debug.Log($"{nameof(SceneController)}.{nameof(OnVideoRecorded)}");
    124.  
    125.         VideoPath = videoPath;
    126.         VideoPathText.text = VideoPath;
    127.     }
    128.  
    129.     protected virtual void UpdateFeedbackText()
    130.     {
    131.         FeedbackText.text = IsRecording ? $"Recording at {RecordingResolutionWidth}x{RecordingResolutionHeight}" : "Stopped";
    132.     }
    133. }
    134.  
     

    Attached Files:

  3. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    The S3 is a pretty old phone, so old in fact that it shouldn't be able to run NatCorder (the max update for the S3 is Android 4.4, and NatCorder requires Android 6+). In any case, the device is running out of memory. The next update will improve memory stability.
     
  4. glutvort

    glutvort

    Joined:
    Apr 13, 2019
    Posts:
    31
    Need your help. I use ARCore, Netcorder, NatShare in callback OnRecord i want to get NatShare.GetThumbnail(Path); When i found mark and instantiate model, i started record after some time i stoped record and have error. All pemision are granted
     

    Attached Files:

  5. glutvort

    glutvort

    Joined:
    Apr 13, 2019
    Posts:
    31
    One more problem. Video resolution 1440*2560. But so bad quality...
     

    Attached Files:

  6. AndreaSpace1

    AndreaSpace1

    Joined:
    Dec 22, 2017
    Posts:
    2
    Thank you for your reply. My bad, the device is not the Samsung S3 but the Samsung Tab S3. Nonetheless, I was able to solve the crash, by limiting the number of times "CommitFrame" is called to no more than the video frame rate:

    Code (CSharp):
    1.  
    2. protected virtual float LastFrameRecordedTimeInSeconds { get; set; }
    3. protected virtual float RecordingTimeIntervalSeconds { get; set; } = 1/30f;
    4.  
    5. void LateUpdate ()
    6. {
    7.     if (IsRecording)
    8.     {
    9.         if (Time.unscaledTime > LastFrameRecordedTimeInSeconds + RecordingTimeIntervalSeconds)
    10.         {
    11.             var renderFrame = MediaRecorder.AcquireFrame();
    12.             Graphics.Blit(WebCamTexture, renderFrame);
    13.             MediaRecorder.CommitFrame(renderFrame, RecordingClock.Timestamp);
    14.  
    15.             LastFrameRecordedTimeInSeconds = Time.unscaledTime;
    16.         }
    17.     }
    18. }
    19.  
    1920x1080 and 1280x720 at 30fps now record fine, no crash at all
     
    Lanre likes this.
  7. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    This means that for whatever reason, Android can't retrieve a frame for the video. Check the NatShare source for how this happens.
    Increase your recording bitrate.
     
    glutvort likes this.
  8. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    829
    Latest NatCorder issues -
    The type or namespace name `Docs' could not be found. Are you missing an assembly reference?
     
  9. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    829
    What is a new way to check if natcorder is recording? .IsRecording no longer seems to work?
     
  10. glutvort

    glutvort

    Joined:
    Apr 13, 2019
    Posts:
    31
    m4p recorder != null
     
    Lanre likes this.
  11. glutvort

    glutvort

    Joined:
    Apr 13, 2019
    Posts:
    31
    I use default int videoBitrate = (int)(960 * 540 * 11.4f). how much increase?
     
  12. distastee

    distastee

    Joined:
    Mar 25, 2014
    Posts:
    37
    Hey Lanre!
    Loving everything - been using it for a few months. I'm building an easy way to record clips into one video. One thing I've noticed is that when I pause - record - pause - record - on ios there is a .5 - 1 second long gap at the beginning of the resume where audio is recorded (from NatMic) but the same video frame is being used. This results in a video that looks like it has a "hitch" in it during the edit points. My pause and resume code is:

    Code (CSharp):
    1.         public void PauseRecording()
    2.         {
    3.             _clock.Pause();
    4.             _cameraInput.Dispose();
    5.             _audService.StopRecording();
    6.         }
    7.  
    8.         public void ResumeRecording()
    9.         {
    10.             _clock.Resume();
    11.             _cameraInput = new CameraInput(_videoRecorder, _clock, App.instance.MainCamera);
    12.             _audService.StartRecording();
    13.         }
    Am I doing anything incorrectly? Is there any way to fix it?

    My hunch is that the clock and audio resuming is instant, whereas the CameraInput has a short warmup time. This means that during the warmup time - the previous frame is being written to the mp4 along with fresh audio - resulting in the strange hitching I'm seeing.

    If no good workaround - my other thought is to save each "resume" is a different clip and stitching them together at the end - so I'm not totally dead in the water if there's no fix.

    Thanks,
    -Chris
     
  13. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    You shouldn't have any code that uses the Docs namespace. It is an internal namespace.
    When you create a recorder, you are recording. And when you dispose of it, you are no longer recording. It's a simplified paradigm.
     
  14. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    You can follow the YouTube bitrate guidelines.
     
  15. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    What is your `_audService` doing? One thing to check is that you are using the same recording clock when committing video and audio frames (this way pausing the clock will be reflected across both video and audio tracks). The hitch you see is characteristic of one track having continuous time while another track is paused.
     
  16. distastee

    distastee

    Joined:
    Mar 25, 2014
    Posts:
    37
    Thanks for the quick reply! I'll answer your question (though it may not be relevant - more info below).

    Starting recording (included magic numbers for clarity):

    Code (CSharp):
    1.          
    2.         public void StartRecording()
    3.         {
    4.             int MAX_VIDEO_RES = 1920;
    5.             int samplerate = 44100;
    6.             int channelcount = 2;
    7.             float width = App.instance.MainCamera.scaledPixelWidth;
    8.             float height = App.instance.MainCamera.scaledPixelHeight;
    9.             float ratio = width / height;
    10.             int vidwidth = ratio > 1 ? MAX_VIDEO_RES : (int)(MAX_VIDEO_RES * ratio);
    11.             int vidheight = ratio > 1 ? (int)(MAX_VIDEO_RES * ratio) : MAX_VIDEO_RES;
    12.             _clock = new RealtimeClock();
    13.             _videoRecorder = new MP4Recorder(vidwidth, vidheight, 30, samplerate, channelcount, OnVideo);
    14.             _cameraInput = new CameraInput(_videoRecorder, _clock, App.instance.MainCamera);
    15.             _audService = new AudioService(samplerate, channelcount, _videoRecorder, _clock);
    16.             _audService.StartRecording();
    17.         }

    The audio service:
    Code (CSharp):
    1.         public AudioService(int samplerate, int channelcount, MP4Recorder recorder, RealtimeClock clock)
    2.         {
    3.             _samplerate = samplerate;
    4.             _channelcount = channelcount;
    5.             videoRecorder = recorder;
    6.             _clock = clock;
    7.         }
    8.  
    9.         public void StartRecording()
    10.         {
    11.             audioDevice = AudioDevice.Devices[0];
    12.             audioDevice.StartRecording(_samplerate, _channelcount, this);
    13.         }
    14.  
    15.         public void ResumeRecording()
    16.         {
    17.             audioDevice.StartRecording(_samplerate, _channelcount, this);
    18.         }
    19.  
    20.         public void StopRecording()
    21.         {
    22.             if (audioDevice.IsRecording)
    23.             {
    24.                 audioDevice.StopRecording();
    25.             }
    26.         }
    27.  
    28.         // Invoked by audio device with new audio data
    29.         public void OnSampleBuffer(float[] sampleBuffer, int sampleRate, int channelCount, long timestamp)
    30.         {
    31.             // Send sample buffers directly to NatCorder for recording
    32.             videoRecorder.CommitSamples(sampleBuffer, _clock.Timestamp);
    33.  
    34.         }
    35.  
    As I've started implementing the "stitch" method - it actually looks like the hitch is once I begin any recording (not just upon resume) and happens during the first .5 seconds of recording.

    Best way to describe it is if you assume I am recording at 30 fps, NatCorder will save frames 1, 5, 17, 18, 19, 20, 21, etc......

    It drops frames 2-4 and 6-16.

    Potentially it is from a frame drop of the app itself - but the only thing I am doing is OnClick calling that StartRecording() function. Nothing else fancy going on except for the video recorder. This is on an iPhone X on iOS 12.? (updating to 12.3 just in case).

    It could also potentially be triggering a garbage collection? I'm on Unity 2018.3.0f2
     
    Last edited: May 18, 2019
  17. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    Ah I see. This is from NatMic. We've fixed this bug on iOS in the upcoming update. A better solution is to simply have your audio service keep the microphone running throughout the duration of the pause, and not send samples to the video recorder.
     
  18. distastee

    distastee

    Joined:
    Mar 25, 2014
    Posts:
    37
    Cool. Looking forward to the update. I'll see if running the mic helps things in the meantime.
     
    Lanre likes this.
  19. glutvort

    glutvort

    Joined:
    Apr 13, 2019
    Posts:
    31
    How i can make fade in, in end of video??
     
  20. mr-satru

    mr-satru

    Joined:
    Aug 7, 2014
    Posts:
    20
    Hi. We purchased your asset and have next trouble. When building a project with a scene "Replay" example on an Android device (Samsung tab A2), after 10-15 seconds of recording, the application crashes. How can this problem be solved ?
    IOS build with same scene is work fine.
     
  21. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    NatCorder doesn't have any video editing capabilities like this.
     
  22. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    What is your recording resolution? That is usually the first thing to look at. The device might not be able to handle recording at that size.
     
  23. mr-satru

    mr-satru

    Joined:
    Aug 7, 2014
    Posts:
    20
    All settings is default, resolution 1280x720.

    "The device might not be able to handle recording at that size."

    Testable device is far from the worst that a potential user may have.
    How to determine which resolution is fine for a particular device ?
     
  24. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    Ah that's weird. Can you upload the full logs from logcat in a .txt file? A crash like this usually occurs when recording at resolutions higher than 1920x1080.
     
  25. Noisecrime

    Noisecrime

    Joined:
    Apr 7, 2010
    Posts:
    1,515
    For reference I found this site had a useful formula for calculating the bitrate, similar to the one in NatCorder.

    It should be noted that the values they give for K need to be multiplied by 1000 or 1024 for use in NatCorder, I guess using different units? This formula comes pretty close to the recommendations on YouTube if you use high = r = 1 or normal = r = 0.5.

    However for my specific case i found the resulting bitrate was still too low, but then my project is pretty much a worst case situation for encoding as every frame is radically different from the previous one and there is a gradient in the background. I would assume for more usual cases the values from that formula would work well.
     
    Lanre likes this.
  26. Noisecrime

    Noisecrime

    Joined:
    Apr 7, 2010
    Posts:
    1,515
    Been using Natcorder for a week or so and so far its been a really good experience. I would love to see more options in terms of encoding options for both MP4 and GIF, but its certainly usable. For example its odd that you can only have a maximum of 1 keyframe every second for MP4, though i'm unsure if that is a limitation of the specification or NatCorder? For GIF it would be nice to have the option for not dithering or maybe choosing different dither types.

    Crash Bug
    Anyway all was going well with NatCorder until today, when I tried out exporting to GIF and started have Unity hard crash when stopping the recordings. After a little bit of digging I discovered the cause was a bug in my code that meant I was calling StopRecording() multiple times. So I assume that calling cameraInput.Dispose() and/or gifRecorder.Dispose() was the culprit.

    Obviously this was a case of my own bug, but it would be nice and I think expected if the plugin could offer some protection and ensure that whatever it is doing in its dispose method is not executed multiple times or if its already been disposed. Certainly in a case like this were a simple one line bug meant Unity would hard crash to the desktop with no crash log adding some protection from user mistakes would be beneficial.
     
  27. glutvort

    glutvort

    Joined:
    Apr 13, 2019
    Posts:
    31
    I have error(( Device Asus Google Nexus 7 16GB (2nd gen). Android 6.0.1
    Code (CSharp):
    1.  
    2. private void StartRecord()
    3.         {
    4.             recordingClock = new RealtimeClock();
    5.             videoRecorder = new MP4Recorder(
    6.                 Screen.width,
    7.                 Screen.height,
    8.                 30,
    9.                 0,
    10.                 0,
    11.                 OnRecording
    12.             );
    13.             cameraInput = new CameraInput(videoRecorder,recordingClock,Camera.main);
    14.             _recording = true;
    15.             SetButtonrecordState();
    16.         }
    17.  
    Code (CSharp):
    1.  
    2. private void StopRecord()
    3.         {
    4.             cameraInput.Dispose();
    5.             videoRecorder.Dispose();
    6.             _recording = false;
    7.             SetButtonrecordState();
    8.         }
    9.  
     

    Attached Files:

    Last edited: May 21, 2019
  28. mr-satru

    mr-satru

    Joined:
    Aug 7, 2014
    Posts:
    20
    1. Log with "unity" filter
    2. Another try after app is run and crashed without filter
     

    Attached Files:

    Last edited: May 21, 2019
    Lanre likes this.
  29. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    The MP4Recorder uses bits per second.
    This is a limitation of the Android platform, propagated up to the front-end API. All other platforms allow setting a keyframe interval in frames, whereas Android specifies it in seconds.
    This is done on purpose. We use Floyd Steinberg dithering on all platforms as it has the best visual quality.
    I disagree with this. The Dispose pattern on recorders is the closest we come to RAII. You create a recorder, use it, and dispose it. When you dispose an object, there is an expectation that you also dereference the object, as you can no longer use it. This is why the IDisposable interface in C# doesn't offer an `IsDisposed` boolean property (for NatCorder to protect against you calling Dispose twice, we would implicitly have to implement such a property, which goes against the paradigm). All you have to keep in mind is that when you dispose an IDisposable, dereference immediately after.
     
    Noisecrime likes this.
  30. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    Take the full logs (no filtering) from logcat and upload it in a .txt file. The full logs will contain the exact reason for the crash.
     
  31. glutvort

    glutvort

    Joined:
    Apr 13, 2019
    Posts:
    31
    does plugin supported 16:10 recording? Because i decrease aspect ration and crash dont happing again
     
  32. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    Yes, but the device might not, and there isn't any way to know. This is a common problem on Android.
     
  33. Noisecrime

    Noisecrime

    Joined:
    Apr 7, 2010
    Posts:
    1,515
    Interesting, any chance of getting keyframe by frames exposed for non-Android devices?

    It may have the overall best quality, but not always. Its also odd that it seems to want to dither a perfect white background, is that normal? Maybe what I really need here is a way of indicating a static background color or something, I think there is a means of doing that in GIF specifications? Something for the future perhaps?

    Well as I said, i made a simple error, in copy and pasting from my MP4 recorder code to a GIF recorder code I omitted setting a flag to false so that StopRecording wouldn't be repeatedly called. I guess really what is needed is to have both cameraInput and gifRecorder set to null and use a condition null check before disposing. In whcih case it would have been nice to have had that done in the examples in the package since thats were the original code came from.
     
  34. Aidan-Wolf

    Aidan-Wolf

    Joined:
    Jan 6, 2014
    Posts:
    50
    Google Pixel 3a
    Android version 9
    API Level 24
    ReplayCam script


    05-21 17:48:28.497 29608 29624 E Unity : AndroidJavaException: java.lang.ClassNotFoundException: com.yusufolokoba.natcorder.MP4Recorder

    05-21 17:48:28.497 29608 29624 E Unity : java.lang.ClassNotFoundException: com.yusufolokoba.natcorder.MP4Recorder

    05-21 17:48:28.497 29608 29624 E Unity : at java.lang.Class.classForName(Native Method)

    05-21 17:48:28.497 29608 29624 E Unity : at java.lang.Class.forName(Class.java:453)

    05-21 17:48:28.497 29608 29624 E Unity : at java.lang.Class.forName(Class.java:378)

    05-21 17:48:28.497 29608 29624 E Unity : at com.unity3d.player.UnityPlayer.nativeRender(Native Method)

    05-21 17:48:28.497 29608 29624 E Unity : at com.unity3d.player.UnityPlayer.c(Unknown Source:0)

    05-21 17:48:28.497 29608 29624 E Unity : at com.unity3d.player.UnityPlayer$e$1.handleMessage(Unknown Source:88)

    05-21 17:48:28.497 29608 29624 E Unity : at android.os.Handler.dispatchMessage(Handler.java:102)

    05-21 17:48:28.497 29608 29624 E Unity : at android.os.Looper.loop(Looper.java:193)

    05-21 17:48:28.497 29608 29624 E Unity : at com.unity3d.player.UnityPlayer$e.run(Unknown Source:20)

    05-21 17:48:28.497 29608 29624 E Unity : Caused by: java.lang.ClassNotFoundException: Didn't find class "com.yusufolokoba.natcorder.MP4Recorder" on path: DexPathList[[zip file "/data/app/com.kevaid.doodlecam-xlTR7LbvX17qMNMqTc_tzw==/base.apk"],nativeLibraryDirectories=[/data/app/com.kevaid.doodlecam-xlTR7LbvX17qMNMqTc_tzw==/lib/arm, /data/app/com.kevaid.doodlecam-xlTR7LbvX17qMNMqTc_tzw==/base.apk!/lib/arme

     
  35. summer_zc

    summer_zc

    Joined:
    Dec 14, 2018
    Posts:
    4
    hi,I use this asset to recorder mp4 files, it's right in iOS, but it looks darker in Android . In my upload files,I write RenderTexture to png Files,I found color is right, but mp4 color always darker, can you help me?
     

    Attached Files:

  36. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    Unfortunately this isn't possible. When crafting a cross-platform API, you're usually restricted to the greatest common denominator of all supported platforms. Any deviation would result in an awkward API.
    Hm what platform does this occur on? Sounds like a bug.
    You shouldn't keep a flag of whether you are recording or not. One very easy way of avoiding gotchas in code is to keep as little canonical information as possible, and derive as much as you can from that set. If you have some `videoRecorder` member variable that is null, you can always infer that you are not recording (the Dispose paradigm lets you make this assertion, provided you follow the paradigm). And similarly, if the `videoRecorder` variable is not null, then it must be the case that you are recording. The rest of the logic will flow from this.

    The examples don't do any checks because some of the preconditions are guaranteed by the UI, not by code. In all our examples, we use our custom recording button. When the button's touch-down event is triggered, recording is started; and when the touch-up event is triggered, recording is stopped. It is impossible for the touch up event to be triggered without a preceding touch down event, so our example code can always bank on this precondition. It makes life easy ;)
     
    Noisecrime likes this.
  37. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    Hm this is weird. Try deleting NatCorder in your project and reimporting it from the Asset Store. Once you do that, upload the logs from logcat in a .txt file.
     
    Aidan-Wolf likes this.
  38. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    You must be using linear rendering. NatCorder only supports linear rendering on macOS and Windows; it doesn't support linear rendering on any other platform (although it works on iOS, as you have seen).
     
  39. Noisecrime

    Noisecrime

    Joined:
    Apr 7, 2010
    Posts:
    1,515
    That was on Windows 10 - though i'll need to double check it was just a pure white background camera and that no other gameObjects were affecting the screen. I'll let you know if I confirm it, but only got a week left on client project before install date so it will be low priory.

    Great point and i'm kicking myself for not doing this in the first place. That's what you get for being under pressure and crunching with a week before project install. You are correct it does simplify and add robustness to the code, however one aspect I'm unsure about is the safety of setting videoRecorder to null once its disposed.

    One aspect is at the back of my mind is that in Unity, in some specific cases null != null - e.g. Unity Blog. I guess I could just not have videoRecorder as MonoBehaviour fields and that should address this concern.

    However the larger issue concerns what happens when you perform StopRecording, where you are expected to do videoRecorder.dispose, but I'm unsure if its safe to immediately set videoRecorder to null? My understanding is that dispose will result in the video being finalized and written to disk, which can take several frames/ms to perform, prior to then issuing its callback mechanism back to Unity. My concern is setting videoRecorder to null might kill that process before it has a chance to finish?

    Would the safest approach be to set videoRecorder to null in the callback, but then you are left with the case of videoRecorder not being null, but disposed, which makes validation awkward again.

    Any thoughts on this?
     
  40. summer_zc

    summer_zc

    Joined:
    Dec 14, 2018
    Posts:
    4
    Yes,We use linear color space. Is it because of the need for openGL3.0 on Android and only Metal in iOS ? Can NatCorder supports linear on Android and iOS ?
     
  41. summer_zc

    summer_zc

    Joined:
    Dec 14, 2018
    Posts:
    4
    Is there a plan to support Linear on Android and iOS ?
     
  42. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    Ah, this one is out of our control then. On all platforms expect Android, we use platform-provided quantization and dithering API's. On Android, we use our own implementation.
    Why? This is the proper usage, and this is what the examples do.
    This is pretty interesting (I had no clue), but it only applies to types that derive from UnityEngine.Object (looks like they just overrode the `==` operator). It doesn't affect NatCorder types, because none of our public classes derive from UnityEngine.Object/MonoBehaviour.
    Yes, it is safe to immediately dereference the recorder after setting it to null. The recorder doesn't get garbage collected until the recording callback has been invoked from the native layer, so there's nothing to worry about.
    Call Dispose then immediately dereference the recorder. NatCorder will handle the rest ;)
     
    Noisecrime likes this.
  43. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    We don't support linear rendering on mobile platforms for technical reasons, especially because of Android. We've seen that linear rendering is unstable on Android, so we can't really guarantee proper functionality with it.
     
  44. adrian-clark

    adrian-clark

    Joined:
    Aug 9, 2018
    Posts:
    3
    Hi Lanre, thanks for a great plugin, it's proved very useful. I have a question which isn't directly related to NatCorder, but involves it and I was hoping you might be able to help with.

    I'm working on the project at the moment where one part involves the user choosing some options, and then a video is created based on the options they have chosen (think something like choosing an animated character, then choosing an environment, and then we generate a video of those characters animated in that environment). I can do this using Natcorder recording at a normal speed, however I was wondering if you knew if there was anyway it would be possible to record the scene faster than 1:1 playback time, especially on mobile?

    For example, say I'm recording a 1 minute video of the characters in the environment, I would like to be able to record a 30 fps video with Time.timeScale = 4 and the Natcorder Clock running 4 times the speed, so that the actual recording only takes 15 seconds but the output video is actually 1 minute. I have managed to get this to work to an extent on desktop (where I measure the time taken to render the previous frame and use that to update both Time.timeScale and a multiplier on the clock), and can successfully record at roughly 2-3x speed (i.e. record a 30 second video in 10-15 seconds). However whenever I run it on mobile, the maximum speed I can render frames at seems to be locked at 30 frames per second. I found this thread which states that rendering on iOS is tied to VSync, which makes sense when rendering to the screen, but this camera only ever renders to the Natcorder RenderTexture so VSync and screen tearing shouldn't even happen. Unfortunately all my searching has just taken me to various things on optimising your game for mobile, which isn't really what I need as the scenes are very simple and the limitation is really Unity not animating and rendering the frames fast enough.

    Any ideas?

    Thanks
     
  45. Jason-RT-Bond

    Jason-RT-Bond

    Joined:
    Mar 26, 2012
    Posts:
    53
    @Lanre

    Hi there! I’m having a rather critical issue with NatCorder on Android.

    Below is a simplified version of my script. It attempts to make an 8 second recording from the camera and audiolistener and then plays it back. This works on Mac and iOS, but on Android the app freezes and then crashes every time, either right when recording begins or a second or so in. I’ve tried a number of configurations and have not been able to make recording work on Android.

    Code (CSharp):
    1. using System.Collections;
    2. using System.Collections.Generic;
    3. using UnityEngine;
    4. using NatCorder.Inputs;
    5. using NatCorder;
    6. using NatCorder.Clocks;
    7.  
    8. #if UNITY_EDITOR
    9. using UnityEditor;
    10. #endif
    11.  
    12. public class RecordingScript : MonoBehaviour
    13. {
    14.   AudioListener myAudioListener;
    15.   Camera myCamera;
    16.  
    17.   private MP4Recorder videoRecorder;
    18.   private IClock clock;
    19.   bool done;
    20.   string filename;
    21.  
    22.   const int VIDEO_WIDTH = 1280;
    23.   const int VIDEO_HEIGHT = 720;
    24.  
    25.   // Start is called before the first frame update
    26.   void Start()
    27.   {
    28.     myAudioListener = GetComponent<AudioListener>();
    29.     myCamera = GetComponent<Camera>();
    30.  
    31.     StartCoroutine(RecordRoutine());
    32.   }
    33.  
    34.   IEnumerator RecordRoutine()
    35.   {
    36.     yield return new WaitForSeconds(1f);
    37.  
    38.     Debug.Log("Beginning recording.");
    39.  
    40.     clock = new RealtimeClock();
    41.     videoRecorder = new MP4Recorder(VIDEO_WIDTH, VIDEO_HEIGHT, 30, AudioSettings.outputSampleRate, (int)AudioSettings.speakerMode, OnRecording);
    42.     AudioInput audioInput = new AudioInput(videoRecorder, clock, myAudioListener);
    43.     CameraInput cameraInput = new CameraInput(videoRecorder, clock, myCamera);
    44.  
    45.     yield return new WaitForSeconds(8f);
    46.  
    47.     Debug.Log("Ending recording.");
    48.  
    49.     audioInput.Dispose();
    50.     cameraInput.Dispose();
    51.  
    52.     // Stop recording
    53.     videoRecorder.Dispose();
    54.  
    55.     while (string.IsNullOrEmpty(filename))
    56.     {
    57.       yield return null;
    58.     }
    59.   }
    60.  
    61.   void OnRecording(string path)
    62.   {
    63.     Debug.Log("Saved recording to: " + path);
    64.     // Playback the video
    65. #if UNITY_EDITOR
    66.     EditorUtility.OpenWithDefaultApp(path);
    67. #elif UNITY_IOS
    68.         Handheld.PlayFullScreenMovie("file://" + path);
    69. #elif UNITY_ANDROID
    70.                 Handheld.PlayFullScreenMovie(path);
    71. #endif
    72.   }
    73. }
    74.  
    Since almost no configuration is working for me on Android I have to wonder if some critical change exists in Unity or Android which breaks NatCorder?

    If it helps I can send you my complete demo project to reproduce the issue.

    Also attached is the log output before the freeze. I don’t see any information about the error itself there, but it may mean more to you.


    Unity 2019.1.3f1
    NatCorder 1.5.1
    Android version: 9
    Test device: Pixel 3
     

    Attached Files:

  46. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    NatCorder is capable of offline recording (basically, recording in a for-loop), but this is really only intended for standalone platforms where there's enough memory to hold a bunch of frames in RAM till the encoder can eat them all up. You can try it, but you have to be especially cautious of your memory consumption. The key you need is to manually 'push' your game forward in time, then render to NatCorder at the end of each timestep. The Time.timeScale trick won't work because you are increasing the speed of the game without increasing how often each of those frames gets rendered to the screen (this will be locked to Vsync on mobile platforms). If you can tie all your game movements to a simple `Game.StepForward()` function, your recording code will look something like this:
    Code (CSharp):
    1. for (;;) {
    2.     // Step the game forward in time
    3.     game.StepForward();
    4.     // Record the current frame
    5.     // Make sure to use a FixedIntervalClock for this (uses like this is what it was made for)
    6.     NatCorder.Acquire + Camera.Render + NatCorder.Commit
    7. }
     
  47. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    Can you upload the full, unfiltered logs from logcat in a .txt attachment?
     
  48. adrian-clark

    adrian-clark

    Joined:
    Aug 9, 2018
    Posts:
    3

    Thanks for the information. Since I'm mainly targeting mobile in this case I would guess it's probably not possible - I did follow a similar pattern on one attempt and on iOS it very quickly used up all the available memory and the app crashed. I think it's probably best to just work the UX around this happening as a background task. Cheers!
     
    Lanre likes this.
  49. aobjects

    aobjects

    Joined:
    Jan 16, 2016
    Posts:
    9
    Hey @Lanre, we're trying to use NatCorder on LWRP (2018.3.7). Everything seems to work when using the ReplayCam example on device, but when trying to integrate into an ARFoundation project with LWRP, every recorded video ends up as black for the duration of the recording. Can you offer any insight?

    Thanks!
     
  50. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,602
    I think I've gotten a report of an issue like this in the past. Apparently it had to do with how ARFoundation handles rendering in Unity. Check this out.