Search Unity

NatCorder - Video Recording API

Discussion in 'Assets and Asset Store' started by Lanre, Nov 18, 2017.

  1. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    This would be implicitly fixed in the next update.
    Also not too sure what could be going on here. Can you check the profiler for any clues? Where is all that frame time being spent?
     
  2. Menion-Leah

    Menion-Leah

    Joined:
    Nov 5, 2014
    Posts:
    79
    Hello @Lanre

    I'm trying to use Natcorder to add a "record replay" feature to the game I'm currently working on.
    I just imported the recently released version, hoping it could fix some issues I'm experiencing, but unfortunately it's not the case.

    Basically, when enabling video recording the framerate suffers from very frequent garbage collection: in fact, for every frame Natcorder compression is allocating 3.5 MB of garbage. Also texture memory is skyrocketing, and Texture count is increasing by 1 each frame. I'm not creating any Texture2D or RenderTexture by myself, only letting my Cameras to render over RenderTexture obtained by .AcquireFrame().

    Here you can see a profiler screenshot addressing this issue; I'm on Windows 10, Unity 2018.3.2f1, and also recording on Android (OnePlus 3T, 1280x720) is very choppy and sometimes it crashes.

    https://imgur.com/QixkiQB


    Thank you for your support
     
  3. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    This isn't garbage memory. NatCorder needs to get the pixel data to send to the hardware encoder. Doing so requires calling Texture2D.GetRawTextureData. And unfortunately, this function will always return a new byte array. There is no way to have it fill an existing array. So there is really no way around it right now.
     
  4. Menion-Leah

    Menion-Leah

    Joined:
    Nov 5, 2014
    Posts:
    79
    Thanks, I feared that answer.

    Another question, then: since I cannot appreciate any real benefit in performances with this new update, is there a way to disable its newly introduced optimizations to keep being compatibile with Android API level 18? According to Unity, switching to Marshmallow as a minimum is approximately a 25% market share loss.
     
  5. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    The garbage collection only affects standalone and WebGL platforms. On iOS and Android, we don't generate GC memory when committing frames. Switching to Marshmellow on Android affects both MP4 recording performance and GIF recording quality. There isn't a way to switch off the changes that have been made; you'll have to stick with an older version of the API.
     
  6. Skjalg

    Skjalg

    Joined:
    May 25, 2009
    Posts:
    150
    Yeah I know its very similar to ReplayCam, and it looks redundant to someone who knows the API very well. But you have to realize that most people when downloading your asset for the first time will not read the readme, they will go straigth to the examples folder and look for an example scene that matches their need. And most peope are not looking for a webcamera recording solution, but a video game recording solution - and your example doesn't show that part very well (even though thats kind of what happens).
     
    mills and Menion-Leah like this.
  7. Skjalg

    Skjalg

    Joined:
    May 25, 2009
    Posts:
    150
    As I said the profiler just shows that a long time is spent in NatCorderBridge.EncodeFrame
    upload_2019-2-8_13-33-41.png
     
  8. Menion-Leah

    Menion-Leah

    Joined:
    Nov 5, 2014
    Posts:
    79
    Did you have a chance to try AsyncGPUReadback? It's asyncronous and also it relies on NativeArray<T>, so you can dispose them immediately after use.
     
  9. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    I totally understand this, it's something I've done in the past. But with NatCorder, we specifically need to discourage this. Video recording isn't a straight-forward process; it's pretty complicated on different platforms. We've gone great strides to simplify it all to just five functions, but there are still quirks here and there. As a result, we absolutely 100% need developers to check out the README.md, at the very least.
     
  10. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    Committing shouldn't take this long. What is your recording resolution? And what AudioFormat are you passing into StartRecording? This might be the encoder choking on a misconfiguration.
     
  11. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    This is ideal, but it was introduced in Unity 2018.1 and stable in 2018.3. A lot of devs use older versions of Unity, and can't upgrade because of dependencies or breaking changes.
     
  12. Menion-Leah

    Menion-Leah

    Joined:
    Nov 5, 2014
    Posts:
    79
    I totally understand that. Maybe it could be a good idea to implement this more efficient alternative as well, using conditional compilation to enable backward compatibility.

    Aside from that: do you have any suggestion to improve realtime recording? I've smashed my head against it for several days, but results are not good enough. Both on PC and Android the framerate is way too low.
    I tried to overcome it by recording frames on a fixed framerate: video smoothness is perfect, but I'm not able to get audio in sync, even pitching master channel and stretching audio samples in OnAudioFilterRead doesn't work.

    Framerate in Unity Editor is over 200fps (1080p), but when I start recording it doesn't reach 10fps (720p); some recording configurations also make Unity to crash after 2 or 3 seconds.
    I tried using camera.Render(), to Blit from camera.activeTexture, to assign Natcorder's RenderTexture to camera in OnPreRender().. I also tried calling GC.Collect periodically in a background thread.
    Nothing works.
     
  13. Orloffyeah

    Orloffyeah

    Joined:
    Dec 9, 2012
    Posts:
    4
    Hi, I've been using the NatCorder plugin in my project to record gameplay in a 2D scene. This is working correctly in the Editor and in Android, however, when used in iOS, the devices crash because of excessive memory usage.

    When profiling the app through Unity, it can be seen that ManagedHead.UsedSize is using huge amounts of memory when the recording process is started, along with System.ExecutablesAndDlls (As seen in the screenshot)

    How can this behavior be prevented? In the screenshot it can also be seen that garbage collection is never triggered, which could be a factor in the excesive memory usage.

    Thanks for your help.

    upload_2019-2-11_0-49-51.png
     
  14. Acco

    Acco

    Joined:
    Nov 19, 2014
    Posts:
    1
    Hi Lanre,

    We're seeing a repeatable crash with NatCorder 1.4.0 and NatMic 1.2.0 running on a Samsung S6 with Android 7.0:

    2019-02-08 19:27:45.235 11110-12155/com.xxx E/AndroidRuntime: FATAL EXCEPTION: NatCorder Audio Encoding Thread
    Process: com.xxx, PID: 11110
    java.lang.Error: FATAL EXCEPTION [NatCorder Audio Encoding Thread]
    Unity version : 2018.3.0f2
    Device model : samsung SM-G920F
    Device fingerprint: samsung/zerofltexx/zeroflte:7.0/NRD90M/G920FXXU6ERF5:user/release-keys

    Caused by: java.lang.IllegalStateException: Can't write, muxer is not started
    at android.media.MediaMuxer.writeSampleData(MediaMuxer.java:496)
    at com.yusufolokoba.natcorder.MP4Recorder$4.onOutputBufferAvailable(MP4Recorder.java:261)
    at android.media.MediaCodec$EventHandler.handleCallback(MediaCodec.java:1662)
    at android.media.MediaCodec$EventHandler.handleMessage(MediaCodec.java:1609)
    at android.os.Handler.dispatchMessage(Handler.java:102)
    at android.os.Looper.loop(Looper.java:154)
    at android.os.HandlerThread.run(HandlerThread.java:61)

    The same issue isn't present on Samsung S9 with Android 8.0.

    Please do you have any thoughts on how we can work around this? At present, we are only recording mic audio, not mixing with game audio as that causes different unwanted side-effects.

    When recording game audio with Format.Unity and passing a listener to NatMic.StartRecording, this is the error we receive:

    2019-02-11 11:29:06.576 11186-11867/com.xxx E/AndroidRuntime: FATAL EXCEPTION: NatMic Microphone Thread
    Process: com.xxx, PID: 11186
    java.lang.Error: FATAL EXCEPTION [NatMic Microphone Thread]
    Unity version : 2018.3.0f2
    Device model : samsung SM-G920F
    Device fingerprint: samsung/zerofltexx/zeroflte:7.0/NRD90M/G920FXXU6ERF5:user/release-keys

    Caused by: java.lang.ArrayIndexOutOfBoundsException: length=1920; index=1920
    at com.yusufolokoba.natmic.NatMic.run(NatMic.java:91)
    at java.lang.Thread.run(Thread.java:762)
     
    Last edited: Feb 11, 2019
    Achoo likes this.
  15. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    First rule of thumb is to reduce the recording resolution. What resolution are you recording at?
     
  16. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    NatCorder doesn't accrue GC memory on iOS. What it does accrue is texture memory because it tries not to block the Unity main and render threads. I am working on a minor update that should improve this behaviour. What resolution are you recording at?
     
  17. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    Can you send me the full logs from logcat in a txt file?
    I've received a report of this. I am looking into it. I plan to publish a minor NatMic update with the fix tomorrow.
     
  18. Orloffyeah

    Orloffyeah

    Joined:
    Dec 9, 2012
    Posts:
    4
    Currently I'm recording at two resolutions, 1334x750 (16:9) in an iPhone 6 and 1366x1024 (4:3) in an iPad 4.

    I'm using an older version (1.3f2) of the plugin since updating would cause us to loose a big part of the market, specially in Android.

    What would you recommend to avoid this issue? If possible without updating.

    Thanks for your help.
     
  19. Menion-Leah

    Menion-Leah

    Joined:
    Nov 5, 2014
    Posts:
    79
    720p

     
  20. Achoo

    Achoo

    Joined:
    Jun 4, 2015
    Posts:
    8
    Hi Lanre, in the new version of the plugin i get the following exception and its being spammed

    E/Unity: AndroidJavaException: java.lang.ClassNotFoundException: com.yusufolokoba.natcorder.Bridge
    java.lang.ClassNotFoundException: com.yusufolokoba.natcorder.Bridge
    at java.lang.Class.classForName(Native Method)
    at java.lang.Class.forName(Class.java:453)
    at java.lang.Class.forName(Class.java:378)
    Caused by: java.lang.ClassNotFoundException: com.yusufolokoba.natcorder.Bridge
    at java.lang.Class.classForName(Native Method)
    at java.lang.BootClassLoader.findClass(ClassLoader.java:1355)
    at java.lang.BootClassLoader.loadClass(ClassLoader.java:1415)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:312)
    at java.lang.Class.classForName(Native Method)
    at java.lang.Class.forName(Class.java:453)
    at java.lang.Class.forName(Class.java:378)
    Caused by: java.lang.NoClassDefFoundError: Class not found using the boot class loader; no stack trace available
    at UnityEngine.AndroidJNISafe.CheckException () [0x00091] in <004fc436a9154f7fab4df9679445af6c>:0
    at UnityEngine.AndroidJNISafe.CallStaticObjectMethod (System.IntPtr clazz, System.IntPtr methodID, UnityEngine.jvalue[] args) [0x00011] in <004fc436a9154f7fab4df9679445af6c>:0
    at UnityEng


    it's being called whenever a audio is commited
     
    Last edited: Feb 13, 2019
  21. pjccccc

    pjccccc

    Joined:
    Oct 7, 2015
    Posts:
    41
    Hi, I'm facing a problem on iOS below:

    Code (CSharp):
    1. Undefined symbols for architecture arm64:
    2.   "_NRGPUFenceSync", referenced from:
    3.       _MTLFence__ctor_mD945145DE46BAB61848931056A28BE7947144BA0 in Bulk_Assembly-CSharp_2.o
    4.       _MTLFence_GPUFenceSync_m2EE0F4FB66C2A60DFF29C433E46B16F3671B2451 in Bulk_Assembly-CSharp_2.o
    5. ld: symbol(s) not found for architecture arm64
    6.  
    I'm both using NatCam and NatCorder, so I've removed NatCam/libNatRender.a.


    Here's my import setting for NatCorder/libNatRender.a
    and I'm currently using 2018.3

    스크린샷 2019-02-13 오후 10.56.10.png
     
  22. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    If your app runs at 60FPS, then you could set the CameraRecorder.recordEveryNthFrame to 2 to record 30FPS videos. This alleviates memory pressure a great deal.
     
  23. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    Can you try deleting and re-downloading NatCorder from the Asset Store? We published a silent update that resolved the missing symbol issues.
     
  24. Orloffyeah

    Orloffyeah

    Joined:
    Dec 9, 2012
    Posts:
    4
    The app is running at 30 FPS in the mentioned low end devices. We identified the issue in the framePool not being able to free memory fast enough, so to prevent crashes we stop commiting frames to the pool for some milliseconds as to alleviate pressure. However, is there any solution to prevent such an excesive memory usage?
     
  25. Achoo

    Achoo

    Joined:
    Jun 4, 2015
    Posts:
    8
    In Natcorder, whenever you use AudioRecorder, we have to use CameraRecorder? Can we use AudioRecorder with greyworld scene logic?
     
  26. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    NatCorder works by providing the client RenderTextures on which to render. NatCorder has to continuously allocate RenderTextures, and when those RenderTextures have been used, they are freed. It is this allocation that causes memory pressure to build up. There is no way to go around it.
     
  27. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    You don't have to use an AudioRecorder with an CameraRecorder. You can use an AudioRecorder while manually committing video frames.
     
  28. Achoo

    Achoo

    Joined:
    Jun 4, 2015
    Posts:
    8
    i get this crash if i use AudioRecorder while manually commiting video frames.

    --------- beginning of crash
    E/AndroidRuntime: FATAL EXCEPTION: NatCorder Audio Encoding Thread
    java.lang.IllegalStateException: Can't write, muxer is not started
    at android.media.MediaMuxer.writeSampleData(MediaMuxer.java:697)
    at com.yusufolokoba.natcorder.MP4Recorder$4.onOutputBufferAvailable(MP4Recorder.java:261)
    at android.media.MediaCodec$EventHandler.handleCallback(MediaCodec.java:1674)
    at android.media.MediaCodec$EventHandler.handleMessage(MediaCodec.java:1621)
    at android.os.Handler.dispatchMessage(Handler.java:105)
    at android.os.Looper.loop(Looper.java:164)
    at android.os.HandlerThread.run(HandlerThread.java:65)
     
  29. pjccccc

    pjccccc

    Joined:
    Oct 7, 2015
    Posts:
    41
    Yes, I downloaded your plugin just before I wrote the issue.
    I also checked the version which is 1.4.0
     
  30. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    811
    Hi! Just curious is NatCam Core the only one deprecated? NatCam is still current, for device camera access?
     
  31. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    Fix for this coming next week.
     
    Achoo likes this.
  32. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    Can you email me your NatCorder folder?
     
  33. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    NatCam Core and NatCam Pro were merged into one API: NatCam. NatCam is still your one stop shop for all things device camera control.
     
  34. pjccccc

    pjccccc

    Joined:
    Oct 7, 2015
    Posts:
    41
    Sure, Please check your gmail
     
    Lanre likes this.
  35. QianXiYa

    QianXiYa

    Joined:
    Nov 1, 2017
    Posts:
    27
    Hi,
    I wanted to record the screen video, but I used it
    NatCorder.StartRecording(Container.MP4,VideoFormat.Screen,AudioFormat.None,OnRecording);
    The recorded video size is 0KB

    I refer to it here :
    https://olokobayusuf.github.io/NatCorder-Docs/
    Can the natcorder record only one camera?

    Or is there a problem with the second parameter setting?Because your API doesn't have that parameter there
    NatCorder.CommitFrame(frame,(long)Time.deltaTime);
     
  36. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    After calling StartRecording, you need to actually push frames to NatCorder. Check out the README.md for more info.
     
  37. Menion-Leah

    Menion-Leah

    Joined:
    Nov 5, 2014
    Posts:
    79
    Hi @Lanre, do you have any more suggestions on that? Thanks
     
  38. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    I don't think there's much more you can do. The readback is really what eats up performance. I am considering adding support for AsyncGPUReadback. I think this is the only thing that can improve memory consumption, albeit at a memory cost as is the case on iOS.
     
  39. walaber_entertainment

    walaber_entertainment

    Joined:
    Jul 10, 2017
    Posts:
    14
  40. Skjalg

    Skjalg

    Joined:
    May 25, 2009
    Posts:
    150
    I uploaded the project file to the forum as an example on how to use NatCorder. Its easily reproducable with that. I think you possibly even downloaded it on your own machine and looked at the code I used :)

    If you can't reproduce it on your end then it is possible this is a hardware problem? I'm on Windows 10 Pro N
    Itel Core i7-6700HQ @ 2.6Ghz with a nvidia GeForce GTX 908M (asus laptop).

    I've attached a gif of how it looks like to record something in the test scene I uploaded earlier with the settings
    Container: MP4
    Audio Format: Unity
    Size: 1280x720

    Here's also a screenshot of how it looks. The first green spike is when I start recording. The big purple block is after a few seconds of recording when things suddenly starts taking 1000ms

    upload_2019-2-18_11-42-39.png

    Edit:
    It looks like this might be a unity bug as people are having similar problems with the Unity Recorder https://forum.unity.com/threads/unity-recorder-update.509458/page-7#post-4084444
     

    Attached Files:

    Last edited: Feb 18, 2019
  41. Menion-Leah

    Menion-Leah

    Joined:
    Nov 5, 2014
    Posts:
    79

    Same issue here: sometimes it works smooth, sometimes after a couple of seconds its starts taking 2000+ms per frame (almost identical profiler outcome).
    Unity 2018.3.2f1.
     
    Lanre likes this.
  42. sentoplene

    sentoplene

    Joined:
    Oct 2, 2018
    Posts:
    2
    I have a question:

    Is it possible to fetch the recorded video and upload it to a server, and delete the file from the documents folder?
    It is to record a webcam chat between people, and it needs to be saved for legal purposes.
    So the recorded stream needs to go to a server and not to the local filesystem.

    Is this possible with the NatCorder plugin?

    (and is there any extra permissions necessary?)
     
  43. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    And does this happen on Windows or macOS? The one report of a similar issue I had turned out to be because the developer wasn't committing audio frames even though they had specified an audio format in StartRecording. This seems to be specific to Windows. I'll try to reproduce it.
     
  44. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    2,458
    When recording is stopped, NatCorder will invoke a callback you provide with the path to the recorded video.
    This is up to you (NatCorder doesn't do anything apart from video recording), but there are API's for uploading raw data to remote endpoints.
    You can use the System.IO.File API's for this.
    NatCorder can only record to the local file system, not to a stream.
     
  45. Menion-Leah

    Menion-Leah

    Joined:
    Nov 5, 2014
    Posts:
    79
    I implemented the AsyncReadback alternative: it works smoothly and GC.Alloc passed from 3.5 MB per frame to 200 b.

    Main issue is still there, though: profiler reports that every frame a new Texture is created and kept, and as soon as it reaches a certain threshold (1~3 seconds) each frame starts taking 2000+ms and eventually Unity hangs/crashes.

    I'm not creating any texture.. it seems like internally Natcorder allocate new temporary RenderTextures but somehow never release them properly.

    EDIT:
    I missed this post

    I'll investigate more in this direction.
     
  46. MiniBeatBoy

    MiniBeatBoy

    Joined:
    Sep 23, 2013
    Posts:
    5
    Hi @Lanre I'm starting to use NatCorder in a project and it is an impressive asset! I love the simplicity of the API and how everything just works.

    There is just one thing that I haven't figured out. Is there a way to define where do I want the videos to be located? I have searched in the docs and code and it seems to me that there is no way. Can you please confirm this? - Thanks. Juan P.

    Edit: Nevermind. I just found the answer in the forum. Thanks anyway!
     
    Last edited: Feb 18, 2019
    Lanre likes this.
  47. Menion-Leah

    Menion-Leah

    Joined:
    Nov 5, 2014
    Posts:
    79
    I confirm: ensuring correct CommitSamples calls or disabling audio recording at all doesn't make any difference: texture memory increases frame by frame and after a second or two it slows down or crashes.

    EDIT: Texture usage increment can be experienced on Android as well
     
    Last edited: Feb 18, 2019
  48. walaber_entertainment

    walaber_entertainment

    Joined:
    Jul 10, 2017
    Posts:
    14
    This is on Windows 10. Here's the code I use to start the recording:
    Code (CSharp):
    1. NatCorderU.Core.VideoFormat vidFmt = new NatCorderU.Core.VideoFormat(1920, 1080, 30, 10 * 1024 * 1024);
    2. NatCorderU.Core.NatCorder.StartRecording(NatCorderU.Core.Container.MP4,
    3.                                                  vidFmt,
    4.                                                  NatCorderU.Core.AudioFormat.Unity,
    5.                                                  _gotRecording);
    and capturing a frame looks like this:
    Code (CSharp):
    1. RenderTexture frameTex = NatCorderU.Core.NatCorder.AcquireFrame();
    2. Graphics.Blit(ReplayMenuRef.GameRTT, frameTex);
    3. NatCorderU.Core.NatCorder.CommitFrame(frameTex, timestamp);
    4.  
    GameRTT is a RenderTexture that my main game camera is rendering to.
     
  49. Skjalg

    Skjalg

    Joined:
    May 25, 2009
    Posts:
    150
    It actually looks
    This is probably it, because the video's I do get to record which are 1-2s long doesn't contain any audio
     
  50. Skjalg

    Skjalg

    Joined:
    May 25, 2009
    Posts:
    150
    What is this AsyncReadback method?