NatCorder - Video Recording API

I am working on a mobile video editor app that takes short videos and then stitches them together in a user specified order after recording them. Is there any way to combine 2 mp4 videos with natcorder? Thank you so much for any help.

 

Just finished testing on all devices and received reports from the team...it works great now!
Thanks for your help!

 

NatCorder only handles video encoding. You would have to get frames from the videos then commit them in whatever order the user chooses. That aspect is out of scope for NaCorder. There is an open-source video decoding API in the NatSuite Framework, but it is pre-release. See a usage example with NatCorder.

 

Could I collect Texture2d arrays instead of mp4 clips and commit the frames in an offline approach? Would this be performant enough? Thank you for the help.

 

The example code in the page I linked does exactly what you describe. Just don't fill up memory.

 

If my videos are only up to 1 minute long. Would you suggest to save the texture2d to disk (serializableobject) or just use memory?

 

Save it to a video, perhaps using NatCorder. Then decode it with VideoPlayer or NatReader and commit it to a final composition using NatCorder. The MP4 container has already been designed for efficient compression to file; no need to reinvent the wheel with Texture2D I/O.

 

I am thinking of recording some gameplay for the stores, can it be recorded inside the Editor?

 

If you want to record cinematic gameplay, I recommend using Unity's editor-only recorder package. But if you want pure gameplay in Play mode, same as would be in the app on a target device, then NatCorder works in the macOS and Windows editors the same way it does on mobile platforms--no code changes necessary.

 

recorder = new HEVCRecorder(mScreenWidth, mScreenHeight, 30);

erorr log :
2020/09/23 18:09:36.271 14314 15161 Error ACodec Unable to instantiate a encoder for type 'video/hevc' with err 0xfffffffe.

 

On Android, the HEVCRecorder requires Android N or newer. What device and OS version is this error happening on?

 

API level 24 ?
https://developer.android.com/guide/topics/media/media-formats

In this link, does this feature work for Android 5 and later? Why is that?

 

That page is not updated. You should be looking at the Encoder column, which doesn't have a dot indicating support. HEVC encoding support was added in API level 24. Also, I don't know what version of Android you are on right now but the next NatCorder update will require Android API level 23 as a minimum.

 

I did this test on Android 6 , api level 23
When is the release date of this update?

 

No release date yet. But yeah, you can't use the HEVC recorder unless you are running API level 24 or newer.

 

Would storing the series of clips as MP4 and then decoding them and reencoding them into 1 clip add extra compression?

 

Whatever quality loss you face won't be negligible. You can also increase the bitrate on the recorder. See the docs for more info.

 

Thanks

 

Hi

I'm using Natsuite to record a scene that includes speechbubbles.
The speechbubble is a 2D canvas, rendered in worldspace. (Canvas rendermode is set to worldspace)
Speechbubbles are placed in "speechbubble layer". The speechbubble is a couple of sprites and a TMPro text label
When I set main camera to render the speechbubbles, I am able to correctly render them out also in video output with NatSuite. So, so far, so good.

However: I want the speechbubbles to be rendered on top of everything else. I therefore add a camera to the stack of the main camera: This camera clears depth buffer, and renders everything on top of main camera. This looks fine in game, but the speechbubbles themselves are not rendered to video.
The really weird thing is, if I put other stuff in the layer rendered by the overlay cam, this comes out fine in video render.

So: TLR Using URP and a camera with another camera in overlay stack, 2D canvas elements are not rendered in the overlay camera, even if those elements are in worldspace. Yet, the elements are rendered just fine if they are in the main camera.

 

Are you sure that your speech bubbles are being rendered by the overlay camera? Since your speech bubbles are in a specific layer, go to the main camera and ensure that it cannot see that layer. That's the only reason I can think of without having more info. Can you post screenshots of the inspectors for the main camera, overlay camera, and canvas?

 

yes. I'm sure. And I agree it's weird.

Here are the settings:
Main cam:


Main cam cullling



Overlay cam:


(Overlay cam is a child of main cam of course.)
Overlay cam culling


Speechbubble canvas :



Now, if I run this, my speechbubbles are rendered on top of everything, even though they are worldspace objects, exactly like I want. If I render to video, they are not included.

If I include speechbubble layer in maincam culling, and exclude from overlaycam culling mask, they ARE rendered to video, but of course not on top of everything. If I include something else, e.g. actors that are just 3D objects in worldspace, they are rendered on top of everything, AND included in video render. So, my guess is, that there is something going wrong in the specific case of something rendered by a canvas renderer in worldspace mode, and only when this is rendered in an overlay camera in a camera stack.

 

For a sanity check, can you set your overlay camera to render to a RenderTexture (create one in your project files, then assign it in the inspector). Then display that render texture on a UI panel where you can see it. Do the speech bubbles show up there?

 

I mean .. sure, but an easier sanity check I've made, is to runtime enable and disable the overlay cam. With the overlay cam active, the speechbubbles are rendered just fine, but with the overlay inactive, no speechbubbles are rendered (in gameview). This leads me to conclude that speechbubbles must be rendered by the overlay cam, and *only* by the overlay cam.

Are you unable to reproduce the issue?

Try creating a scene with some 3D objects in default layer, some objects in a layer layerX and a canvas rendering in worldspace, with some 2D stuff in layerX: Create a camera( URP) with layermask excluding layerX. add a camera in camerastack which excludes everything except layerX. run game: All objects should be shown. export to video using NatSuite: All objects, except stuff from canvas should export. (including 3D objects in layerX)

That at least is what I'm seeing. I'm on Unity 2019.4.0f1 LTS I will try to update to latest 2019 LTS and report back

 

I have updated to 209.4.11f. Problem remains.

I have solved my own problem by alternate means, so the issue is less pronounced for me now. But it would still be nice to get it fixed. Thanks for your quick and helpful replies!

 

This check isn't helpful because you've already confirmed that the speech bubbles are being rendered by the overlay camera. My sanity check is to see that it is being rendered to texture, which is how CameraInput is able to record game cameras. You should render your overlay camera to a RenderTexture that you display, so you can see whether the bubbles are present or not. I wouldn't expect them to be visible, and if that's the case then you'll have to forward this to Unity as a bug.

 

OK that makes sense. I'll try, and report back to you. But I'm guessing this is Unity crapping out

 

Uhm ... overlay cameras do not have the same render to texture property that basecam has. So. No. I can't do the check you're asking for. I don't know how anyway.

 

Ah damn. Okay I recommend sending a repro to Unity. In the meantime it sounds like you found a workaround.

 

Hi,

We have issue when trying to build xcode to device. Xcode 11.7, Unity 2019.4, Natcorder 1.7.3. The error stated that it doesn't found symbol for arm67.
I have attached sceenshot of the error please get back to us as soon as possible.

Thanks

 

These symbols are from an outdated version of NatCorder. Delete NatCorder in your project (specifically, any "NatSuite" and "NatCorder" folders). Then reimport it from the Asset Store and do a clean iOS build (don't Append).

 

yes , When is the release date of this update?

 

There isn't a set release date yet. I'm still working on improvements and fixes. Is there a specific change or bug fix you are waiting for?

 

Yes, this is your answer:

That page is not updated. You should be looking at the Encoder column, which doesn't have a dot indicating support. HEVC encoding support was added in API level 24. Also, I don't know what version of Android you are on right now but the next NatCorder update will require Android API level 23 as a minimum.

 

I'm not sure about the misunderstanding, so let me clarify:

• The HEVCRecorder requires API level 24 on Android.
• NatCorder, even though it currently requires API level 21, can only successfully create the HEVCRecorder on devices running API level 24 or newer.
• In the next NatCorder update, we are bumping up the minimum required version of Android from API level 21 to API level 23.
• Even with this increase of the minimum required version, NatCorder will still require API level 24 or newer to be able to successfully create a HEVC recorder.

 

I am really enjoying NatCorder, it's an amazing plugin. Everything works great.

I am curious how do you detect the maximum recording framerate for the specific phone? Also, is there a way to detect what framerate the mp4 file was recorded in?

Thank you so much for your help.

 

The frame rate that you specify to the recorder is merely a hint; it has no bearing on the actual frame rate of the video. The actual frame rate depends on the spacing between timestamps on consecutive frames. And I'm not sure your second question has anything to do with NatCorder. On macOS, I use `mediainfo`.

 

Thank you for your response.

In my use case, I use offline recording but the synthesized timesteps don't match the actual framerate of the video recorded. Is there a way to fix this?

 

Can you share your recording code along with one such video? What clock are you using?

 

Thank you for your help. Here is the code doing the offline recording/processing. Unity forum doesn't support uploading video so I will edit and share a youtube link later today. I am using the FixedIntervalClock for creating mp4s. The video is about 2x the speed it should be.

Code (CSharp):

1.  
2.  public async void Record(float time, float recordingSpeed = 1)
3.     {
4.         VideoContainer.Clip clip = new VideoContainer.Clip();
5.         clip.frames = new List<Color32[]>();
6.         VideoContainer.Instance.clips.Add(clip);
7.         numRecordingFrames = (int)(time * 60f);
8.         recording = true;
9.         for (int i = 0; i < numRecordingFrames; i++)
10.         {
11.             recordingTime += 1/60f;
12.             VideoContainer.Instance.clips[VideoContainer.Instance.clips.Count - 1].frames.Add(activeCameraTexture.GetPixels32());
13.             await Task.Delay(16);
14.         }
15.         recording = false;
16.     }
17.  

Code (CSharp):

1. public class VideoContainer : Singleton<VideoContainer>
2. {
3.     // (Optional) Prevent non-singleton constructor use.
4.     protected VideoContainer() { clips = new List<Clip>(); }
5.  
6.     // Then add whatever code to the class you need as you normally would.
7.     public struct Clip
8.     {
9.         public List<Color32[]> frames;
10.     }
11.     public List<Clip> clips;
12.     public string video;
13. }

Code (CSharp):

1. public async void ProcessVideo()
2.     {
3.         EndRecording();
4.         var clock = new FixedIntervalClock(60);
5.         var recorder = new MP4Recorder(activeCameraTexture.width, activeCameraTexture.height, 60);
6.         processing = true;
7.         VideoContainer.Instance.video = await Task.Run(() =>
8.         {
9.             for (int i = 0; i < VideoContainer.Instance.clips.Count; i++)
10.             {
11.                 for (int f = 0; f < VideoContainer.Instance.clips[i].frames.Count; f++)
12.                 {
13.                     recorder.CommitFrame(VideoContainer.Instance.clips[i].frames[f], clock.timestamp);
14.                 }
15.             }
16.             return recorder.FinishWriting();
17.         });
18.         processing = false;
19.     }

 

The issue is likely as a result of capturing frames from a WebCamTexture. The web cam texture might not be running at 60FPS, and using `await Task.Delay(16)` will likely skip two frames instead of one if your app is running at 60FPS (I recommend using a coroutine and WaitForEndOfFrame, as this is actually guaranteed to run in step with Unity's update loop, whereas Task.Delay is not).

Run mediainfo on your video; it is likely going to say a constant frame rate of 60FPS, reflecting the clock you use to record, so it isn't NatCorder at fault here. Also, YouTube transcodes uploads so I don't recommend uploading to YouTube.

 

When opening the project in Xcode I have this error:

ARC Semantic Issue Group
/Users/Rafael/Documents/Projetos/NSF_XcodeProject/Classes/Unity/MetalHelper.mm:300:45: No known instance method for selector 'presentDrawable:afterMinimumDuration:'

 

This has nothing to do with NatCorder. It is a Unity class, so you'll want to file a bug report with Unity.

 

Hi! I had a question before buying.

So, I have to make a "Record your Reaction" feature. Basically, get the frontal camera feed while the user uses the app, record it, and then share it. Also, camera feed should be shown on screen while recording.

Can I use NatCorder for this requierement?

If I can, what should I do, record the part of the screen that has the RawTexture showing the camera feed, or pass the WebCam texture directly to NatCorder?

Thanks!

 

You can use NatCorder to record textures or the screen. How you choose to do it depends on your app's design. I should note that there is no concept like passing a WebCamTexture directly to NatCorder. I recommend going through the docs to get a good sense of how recording works.

 

What device and OS version does this happen on? Can you reimport NatShare from the Asset Store and try again?

 

Hello, I have 2 errors when building, thank you for the help.

Code (CSharp):

1. Assets/NatSuite/Examples/WebCam/WebCam.cs(43,13): error CS0103: The name 'Handheld' does not exist in the current context
2.  
3. Assets/NatSuite/Examples/ReplayCam/ReplayCam.cs(67,13): error CS0103: The name 'Handheld' does not exist in the current context

 

What platform are you building for? You might have to comment out those lines in the respective example scripts.

 

I'm building for macOS. May I just delete the examples folder to resolve this?

 

Just comment out those lines. No need to throw away the entire examples folder.

 

I've finally come to my senses and I've decided to use mp4 clips instead of storing the uncompressed video in memory (this takes too much memory). What is the best way to add NatReader to a NatCorder project? Should I just download the code and import it into the existing project, or does it require special project settings?

Thank you so much for the help.