Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

NatCorder - Video Recording API

Discussion in 'Assets and Asset Store' started by Lanre, Nov 18, 2017.

  1. chovington

    chovington

    Joined:
    Apr 2, 2015
    Posts:
    4
    Hi Lanre,

    We have a home technology in native that manages a camera to render line drawing in post-process and the application that uses this techno was developed with Natcorder 1.5.1. We need to upgrade Unity to 2019 and we got the last version of Natcorder but we trying to replace the RenderTexture that method "AcquireFrame" returns but I try to figure out how this RenderTexture is coming. Because the code you see below works with our techno but we are stuck to find a replacement.

    Before:
    // m_Recorder is a MP4Recorder instance.
    Code (CSharp):
    1. var frame = m_Recorder.AcquireFrame();
    2. if (frame != null)
    3. {
    4.     Graphics.Blit(m_NatcorderCam.targetTexture, frame);
    5.     m_CurrentClip.Frames++;
    6.     m_Recorder.CommitFrame(frame, m_RecordingClock.Timestamp);
    7. }
    8.  
    It's very appreciated if you can explain how AcquireFrame was working to find a way to replace it.

    We tried many ways found on this thread but we have five cameras in this project. If we can know what represents the "frame" returned by m_Recorder.AcquireFrame(); We could resolve it faster.

    Thank you
     
  2. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    If ReplayCam works, then I recommend building off it. I'm not sure why your code doesn't work, but you are likely making an error somewhere. I also recommend going through the documentation as it will inform you of any errors you might run into.
    Recording is a heavy process, so you have to make sure that you are optimizing your app's rendering. See the documentation for tips. On Android, async readbacks might not be supported, especially if you are rendering with OpenGL ES instead of Vulkan. Again, the documentation discusses this.
    NatCorder cannot record audio that is playing directly to the device's output. You have to play the audio through an audio source in the scene, then use an AudioInput to record the audio coming from it.
     
  3. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    If you want to record frames from a RenderTexture, you must first readback the pixel data from the GPU. Then send the pixel data to the recorder. See the documentation for more info.
     
  4. chovington

    chovington

    Joined:
    Apr 2, 2015
    Posts:
    4
    Yeah, I tried it but still no good. I think the key is the RenderTexture given back from AcquireFrame or AcquireFrame do something we miss it. The last person that working on it has updated to 1.6.0 and reverted to 1.5.1 because of that.
     
  5. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Oh it looks like you are asking about an updated version of NatCorder. Upgrade to NatCorder 1.7 on the Asset Store. I don't provide support for outdated versions of the API.
     
  6. chovington

    chovington

    Joined:
    Apr 2, 2015
    Posts:
    4
    I've updated Natcorder from 1.5.1 to 1.7.3, I'm was saying someone in my enterprise tried to update from 1.5.1 to 1.6.0 last year but the fact they lost the AcquireFrame method force him to reverts the code because he doesn't find a way to fix without AcquireFrame. Now it's me that has the same problem to update from 1.5.1 to 1.7.3.

    I see the native code for our techno is completely write for OpenGLES2, I think it's a part of the problem. I remember you said Natcorder doesn't support OpenGLES2.
     
    Last edited: Nov 25, 2020
  7. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    There is no longer an AcquireFrame method. You can either create a RenderTexture or get a temporary one (see Unity's documentation). When you want to commit the frame in the RenderTexture, you need to perform a readback of the pixel data from the GPU. See the documentation on this.
     
  8. chovington

    chovington

    Joined:
    Apr 2, 2015
    Posts:
    4
    Yeah, I tried it.

    I see the native code for our techno is completely write for OpenGLES2, I think it's a part of the problem. I remember you said Natcorder doesn't support OpenGLES2.
     
  9. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    NatCorder is now completely agnostic to the graphics API being used. It will run on OpenGL ES2 as normal. If you have a problem, please be very specific as to what it is. Share any errors, logs, code you have that might be helpful in resolving it.
     
  10. aholla

    aholla

    Joined:
    May 27, 2014
    Posts:
    81
    Hi could I get help recording audio?

    I want to record from the Audio source on ios. I followed this guide: https://medium.com/natsuite/natcorder-tutorial-audio-workflows-1cfce15fb86a

    But the audio playing gets 'stuck' when i start recording and their is no audio in my recorded video.

    Here is the code for the recorder:
    Code (CSharp):
    1.         // Set the recorder inputs
    2.         recorder = new MP4Recorder(_videoWidth, _videoHeight, frameRate, AudioSettings.outputSampleRate, (int)AudioSettings.speakerMode);
    3.  
    4.         // Clock handles the frames.
    5.         var clock = new RealtimeClock();
    6.  
    7.         // Create recording inputs
    8.         cameraInput = new CameraInput(recorder, clock, new Camera[] { _cam, _camFilters, _watermark });
    9.  
    10.         _audioInput.Init(recorder);
    And Here is the code for the 'AudioInput' that is attached to the audio listener in the scene:
    Code (CSharp):
    1.  
    2.     public IMediaRecorder _recorder;    // Initialize this somewhere
    3.     IClock clock;               // Also initialize this somewhere
    4.  
    5.     void OnAudioFilterRead(float[] sampleBuffer, int channels) {
    6.         // Send to NatCorder for encoding
    7.         _recorder?.CommitSamples(sampleBuffer, clock.timestamp);
    8.     }
    9.  
    10.     public void Init(IMediaRecorder recorder) {
    11.         _recorder = recorder;
    12.     }
    Thanks for the help!
     
  11. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Hey there. First off, use NatCorder's AudioInput class, instead of using your own. Beyond that, make sure that all the audio sources in your scene have the "Bypass Listener Effects" setting disabled.
     
  12. Ikaro88

    Ikaro88

    Joined:
    Jun 6, 2016
    Posts:
    302
    Hi, yes i used the audiosource in a video, but also in this case the audio gliched a lot, all the audio of the video start and stop, it is probably because a fps drop.
    the scene have only a video an replyCal Script for recording.

    I already tried with multitread and it do solve unfortunately,

    Sorry for all this fast posts I had a delivery, for now my boss told me to skip about this.
    In the meanwhile thanks a lot he help
     
    Last edited: Nov 26, 2020
  13. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    I'm not sure I fully understand the issue here. If you are facing audio glitching, then check that the issue is not coming from the VideoPlayer.
    You already mentioned that ReplayCam works for you, so check that the issues aren't coming from the video player.
    What device and OS version are you testing on? You need to make sure that your device supports asynchronous readbacks. You can use the Unity Profiler to see where all the time is being spent.
     
  14. aholla

    aholla

    Joined:
    May 27, 2014
    Posts:
    81
    Hi Thanks,

    I have done the above and now things are moving forward. However my fial video does not have any audio in it. Could you please help me figure it out. Also if there a simple demo to capture the screen and the audio? I am playing a music clip from a AudioSource.

    Code (CSharp):
    1.         // Clock handles the frames.
    2.         var clock = new RealtimeClock();
    3.  
    4.         // Audio
    5.         // Say we have an `AudioListener` we want to record
    6.         // Create a recorder
    7.         var sampleRate = AudioSettings.outputSampleRate;
    8.         var channelCount = (int)AudioSettings.speakerMode; // Stereo = 2
    9.         _audioRecorder = new WAVRecorder(sampleRate, channelCount);
    10.  
    11.         // And use an `AudioInput` to record game audio
    12.         _audioInput = new AudioInput(_audioRecorder, clock, _audioSource);
    13.  
    14.         // Set the recorder inputs
    15.         _videoRecorder = new MP4Recorder(_videoWidth, _videoHeight, frameRate, AudioSettings.outputSampleRate, (int)AudioSettings.speakerMode);
    16.  
    17.         // Create recording inputs
    18.         _cameraInput = new CameraInput(_videoRecorder, clock, new Camera[] { _cam, _camFilters, _watermark });
     
  15. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    You are sending your audio to the WAVRecorder, so no audio is being sent to the video recorder.
     
  16. aholla

    aholla

    Joined:
    May 27, 2014
    Posts:
    81
    Ah thanks! How do i send the audio to the MP4 recoder? Or how to I record the screen and the audio at the same time?
     
  17. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    When you create the audio input, pass in your video recorder instead of the WAV recorder.
     
  18. aholla

    aholla

    Joined:
    May 27, 2014
    Posts:
    81
    Thanks, That's it!
     
    Lanre likes this.
  19. Interfo

    Interfo

    Joined:
    Dec 11, 2017
    Posts:
    3
    hmm We trying to add record function on our ARCore base ARApp.
    our testing on editor is mp4 file quality is okay. good.
    ( Editor + Instant Preview of ARCore. )

    but mp4 file have sound but screen is stopped on first frame when we install apk on phone.

    this png is our project setting.
    Have to change something?
     

    Attached Files:

  20. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Can you share your recording code along with the full runtime logs from logcat (in a .txt attachment).
     
  21. Ikaro88

    Ikaro88

    Joined:
    Jun 6, 2016
    Posts:
    302
    Sorry for my bad english maybe I not write all correct
    The problem occurs only while I am recording, and the profile show me a very high usage on the android device... (and the video lag a LOT)

    I mentioned that ReplayCam works, but only in the demo...
    If i put a videoplayer in the same scene i get the problem of all the video that suffer of terrible lag.

    The solution is a business to client solution, so I can not create a solution that works only with some devices.
    I am testing with an android device Galaxy A20e with Android version 10.

    I have not the problem in the editor, but as I said the solution have to work with different devices.
    With some AR requirements I can not use Vulcan as graphics and multirendering do not help

    Let me know if you can help me
     
  22. Interfo

    Interfo

    Joined:
    Dec 11, 2017
    Posts:
    3
    Hmm my co-worker edited many so now can't work.
    So I just made simple copy and run.
    eh? this time... video working. but no sound..
    ( maybe some of plugin trouble with your on our main project. but why no sound on this time?? )
    I will ask again after more many test.
    thanks :)
     
    Lanre likes this.
  23. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Not being able to use Vulkan is a major performance killer on Android. I'm aware that ARCore requires OpenGL ES. I'm looking into writing an async readback plugin for OpenGL ES since Unity doesn't support it. Hopefully, that should greatly reduce the performance cost.
     
    N7Nico, Ikaro88 and jiraphatK like this.
  24. jiraphatK

    jiraphatK

    Joined:
    Sep 29, 2018
    Posts:
    307
    That will be awesome.
    I have found this repo. But the performance is not quite good.
    Also found this in assetstore. I have not tried it yet because it looks quite suspicious and a bit expensive.
     
    Lanre likes this.
  25. ghasedak3411

    ghasedak3411

    Joined:
    Aug 25, 2015
    Posts:
    23
    @Lanre
    Hello
    I need to record over and over again on Android.
    After each record I finish it.
    But after seven or eight repetitions, the frame rate drops sharply.
    And finally it crashes.
     
  26. aholla

    aholla

    Joined:
    May 27, 2014
    Posts:
    81
    @Lanre Could you recommend anything to improve performance on Android?

    We are building an AR app, on IOS everything is perfect. On Android we cant use Vulkan and the performance is much slower.

    I am using an mp4 recorder with 3 input cameras.

    Do you have any tips?

    - Would reducing the number of camera captures make significant improvements?
    - Would removing the audio capture improve much?
    - Does the recorder dimensions have a big impact?
    - Would the GIf recorder be any better than the mp4 recorder?
    - etc..

    Thanks,

    Adam
     
  27. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Can you share the full logs from logcat in a .txt attachment?
     
  28. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    The main problem stems from not being able to use the AsyncGPUReadback class. I'm looking into writing a free plugin to provide this functionality on Android.
     
  29. aholla

    aholla

    Joined:
    May 27, 2014
    Posts:
    81
    Amazing, sounds like this would solve a lot of issues.

    Thanks.
     
    Lanre likes this.
  30. infosekr

    infosekr

    Joined:
    Jul 12, 2013
    Posts:
    46
    When recording videos on Android at a resolution smaller than the screen size the UI doesn't resize correctly with Unity 2019.4.15f (LTS), but does work correctly with 2019.4.14f. Can you confirm that you see this as well? Do you have any work arounds that you could suggest? I tried calling Canvas.ForceUpdateCanvases() in the OnFrame of CameraInput but that doesn't seem to help.

    Thanks.
     
    Last edited: Dec 2, 2020
  31. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    This sounds like a Unity bug. I recommend filing a bug report with Unity. I can't think of anyway to go around this since it's coming from Unity.
     
  32. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Making some progress on implementing async readbacks on Android + OpenGL ES. If you use ARCore/ARFoundation, be excited:
    Screen Shot 2020-12-01 at 22.40.55.png

    This should give you Vulkan-level performance, so no more massive frame rate hits when you record in your AR app.
     
    danUnity, aholla and jiraphatK like this.
  33. ghasedak3411

    ghasedak3411

    Joined:
    Aug 25, 2015
    Posts:
    23
    @Lanre


    The problem was solved
    I "new recorder" twice in a row.

    Thanks for the reply
     
    Lanre likes this.
  34. Muckel

    Muckel

    Joined:
    Mar 26, 2009
    Posts:
    471
    hello,
    well love the Natcoder....
    is there a simple script example how to move the file after recording to desktop?

    this doesn't work so far... i'm on Mac OS X
    Code (CSharp):
    1. string pathDesktop = System.Environment.GetFolderPath (System.Environment.SpecialFolder.Desktop);
    2.             System.IO.File.Move ($"file://{path}", pathDesktop);
    also how do i'll record webcam video together with webcam audio in a mp4 file?
    got only Video work proper...

    a simple example would be nice...
    many thx
    M.
     
  35. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Don't add "file://". Just pass in the path as-is.
    I don't think Unity supports streaming audio with the camera preview.
     
    Muckel likes this.
  36. alexanderromero360

    alexanderromero360

    Joined:
    Nov 28, 2020
    Posts:
    4
    Hi! I just got the Asset and I'm kind of new to this. I've been trying it out for a while but I can't get it to work. I copy-pasted the script in the Getting Started Documentation and added it to an Empty GameObject on my scene and Unity just crashes on me every time. I tried doing the MP4 Recorder with the same results.

    Also trying out the Scene examples, crashes my Unity after some seconds of running (I press the white button but nothing happens, I don't see any new files showing up on the root of my project)

    I'm using Unity 2019.4.11f and Building configurations are set to Android

    If you need more info about the crashes let me know and I'll see how I can get those, but what I would really like is just a script to record the whole scene from the Main Camera for a few seconds and then stop.
     
  37. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    I'm not too sure why you are getting consistent crashes. Can you share the editor logs or crash report in an attachment? Just to confirm, the ReplayCam example scene is also crashing for you? In ReplayCam, you are supposed to press and hold the record button (like Snapchat). But even then, you should not be getting any crashes.
     
  38. alexanderromero360

    alexanderromero360

    Joined:
    Nov 28, 2020
    Posts:
    4
    Sure, let me do that and I'll get back to you. After using the ReplayCam button where should I see the record? In the root path of the project?
     
  39. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    I believe ReplayCam will log the path to the recorded video. In the editor it should be at the project root.
     
  40. alexanderromero360

    alexanderromero360

    Joined:
    Nov 28, 2020
    Posts:
    4
    Hi again! After getting ReplayCam to work correctly I'm using that script as template to write mine. And here is where it crashes. The idea for the script is very simple, just control the recording from the Unity Inspector clicking on the bools startRecording and stopRecording. Of course this is just to test out but still when I try to start the recording, it fails miserably, probably I'm doing something wrong so here's the script and thank you very much.

    Code (CSharp):
    1. using UnityEngine;
    2. using System.Collections;
    3. using NatSuite.Recorders;
    4. using NatSuite.Recorders.Clocks;
    5. using NatSuite.Recorders.Inputs;
    6.  
    7. public class ReplayTest : MonoBehaviour {
    8.  
    9.     [Header(@"Recording")]
    10.     public int videoWidth = 1280;
    11.     public int videoHeight = 720;
    12.     public bool recordMicrophone;
    13.  
    14.     public bool startRecording;
    15.     public bool stopRecording;
    16.     private IMediaRecorder recorder;
    17.     private CameraInput cameraInput;
    18.     private AudioInput audioInput;
    19.     private AudioSource microphoneSource;
    20.  
    21.     private IEnumerator Start () {
    22.         // Start microphone
    23.         startRecording = false;
    24.         stopRecording = false;
    25.         microphoneSource = gameObject.AddComponent<AudioSource>();
    26.         microphoneSource.mute =
    27.         microphoneSource.loop = true;
    28.         microphoneSource.bypassEffects =
    29.         microphoneSource.bypassListenerEffects = false;
    30.         microphoneSource.clip = Microphone.Start(null, true, 10, AudioSettings.outputSampleRate);
    31.         yield return new WaitUntil(() => Microphone.GetPosition(null) > 0);
    32.         microphoneSource.Play();
    33.     }
    34.  
    35.     private void OnDestroy () {
    36.         // Stop microphone
    37.         microphoneSource.Stop();
    38.         Microphone.End(null);
    39.     }
    40.  
    41.     public void StartRecording () {
    42.         // Start recording
    43.         startRecording = false;
    44.         var frameRate = 30;
    45.         var sampleRate = recordMicrophone ? AudioSettings.outputSampleRate : 0;
    46.         var channelCount = recordMicrophone ? (int)AudioSettings.speakerMode : 0;
    47.         var clock = new RealtimeClock();
    48.         recorder = new MP4Recorder(videoWidth, videoHeight, frameRate, sampleRate, channelCount);
    49.         // Create recording inputs
    50.         cameraInput = new CameraInput(recorder, clock, Camera.main);
    51.         audioInput = recordMicrophone ? new AudioInput(recorder, clock, microphoneSource, true) : null;
    52.         // Unmute microphone
    53.         microphoneSource.mute = audioInput == null;
    54.     }
    55.  
    56.     public async void StopRecording () {
    57.         // Mute microphone
    58.         microphoneSource.mute = true;
    59.         // Stop recording
    60.         audioInput?.Dispose();
    61.         cameraInput.Dispose();
    62.         var path = await recorder.FinishWriting();
    63.         // Playback recording
    64.         Debug.Log($"Saved recording to: {path}");
    65.         Handheld.PlayFullScreenMovie($"file://{path}");
    66.     }
    67.        
    68.     void Update () {
    69.         Debug.Log("Updating");
    70.         if (startRecording) {
    71.             StartRecording();
    72.         }
    73.         if (stopRecording) {
    74.             StopRecording();
    75.         }
    76.     }
    77. }
    78.  
    Just in case I'm attaching this to a Empty GameObject on the scene and as long as I dont press startRecording it doesn't crash.

    Thanks a lot again!
     
  41. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    I don't recommend starting and stopping recording in Update. Using booleans to know when to start and stop can cause issues down the line. Without the logs or a stack trace, I can't tell what is causing the crash. If ReplayCam works correctly, then the issue likely comes from something in your code.
     
  42. aholla

    aholla

    Joined:
    May 27, 2014
    Posts:
    81
    Hi, ive run into an issue that I think is a bug.

    I am recording multiple cameras (x3). One is an AR Camera, and the other two are UI layers, one with a hidden camera that is disabled which I am using for a watermark.

    All was working fine until I tried reducing the capture size. Initially I was using Screen.Size. But now I am capping it at 720p and using the screen ratio to calulate the other length.

    This works fine for the AR camera. the video comes out at a lower resolution. The problem is my UI layer is not resized and now apears much larger. The watermark used to be small inte bottom corner, now it is to big to fit in the view.

    I would expect it to capture it as it appears on the screen but it doesnt.

    Make sense?
     
  43. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    It sounds related to this issue. It is a Unity issue, so I recommend filing a bug report with Unity.
     
  44. aholla

    aholla

    Joined:
    May 27, 2014
    Posts:
    81
    Hiim also getting an ios on ios where the "Recorder failed to finish writing"
    Code (CSharp):
    1. Non-fatal Exception: Exception
    2. 0  ???                            0x0 Throw (System.Runtime.ExceptionServices.ExceptionDispatchInfo)
    3. 1  ???                            0x0 ctor (System.Collections.Concurrent.ConcurrentDictionary`2+Tables[TKey,TValue].)
    4. 2  ???                            0x0 get_Current (VideoPlayerManager+<PlayVideo>d__8.System.Collections.IEnumerator)
    5. 3  ???                            0x0 CreateCacheableTask[TResult] (System.Runtime.CompilerServices.AsyncTaskCache)
    6. 4  ???                            0x0 StopRecording (VideoRecording)
    Is there anything I can do about this?
     
  45. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Can you share the full logs? I need to see everything from before your first recording.
     
  46. aholla

    aholla

    Joined:
    May 27, 2014
    Posts:
    81

    So this is from a live project an these are captured logs. its all i have. But i have quite a lot of the same thing.
     
  47. aholla

    aholla

    Joined:
    May 27, 2014
    Posts:
    81
    Can I ask about the core.jar in the plugins folder? I had to remove this to enable firebase to work but now the video capture doesnt work.

    The core.jar is incompartible with firebase. Is there a way to fix this? It creates duplicates.
     
  48. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Can you reproduce it locally and send me the full logs? Without it, I can't tell what the issue might be. Also make sure to share your code.
    You should be able to remove the core-rc library if you see duplicates.
     
    aholla likes this.
  49. GoX

    GoX

    Joined:
    Mar 3, 2020
    Posts:
    1
    Is there any reason why the get thumbnail functionality was removed? This missing feature is really painful^^
    I am trying to adjust the ReplayCam script in a way that it takes a Screenshot if the button is pressed shortly, but the ScreenCapture class from Unity is a pain when using UI.
     
  50. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    NatCorder never had a get thumbnail function. You are likely referring to the function in NatShare, which didn't quite fit into the social sharing idea (it was more of a lone utility function). Trying to hotwire the example code is cumbersome and unnecessary.

    What you'll need to do is to render your specific camera(s) to a RenderTexture, then readback the pixel data from that RenderTexture into a Texture2D. You can then use that Texture2D for any purpose you choose, including encoding to a JPEG or PNG.