NatCorder - Video Recording API

Ok. Thats work now. Great performance. Thanks.

 

Hi, has anyone ever record a video with Unity's microphone as audio input? (Unity's default Microphone Class)
We are encountering this weird popping and cracking sound in IOS
It only occurs on IOS. The problem comes up when Force IOS Speakers when recording in Project Setting is true.
We need this setting to true because otherwise, IOS will lower game sound when we started the Microphone.
(We want both mic and game sound)

https://drive.google.com/drive/folders/1Qdz3CaAWLeKsLO4mRt8rp9sF0JXZGkc0?usp=sharing
Here's a script that we use to record audio. It worked fine in Android and Editor.


We use AudioSource input.

 

Hi,
I am trying to record the AR feed using ARFoundation. The problem I am facing is that if the AR object is too far from the ARcamera, the Preview screen doesn't render any AR object. When the AR object is too close to the camera, the AR objects gets stuck to the camera view. Has anyone facing the problem?

I am using Unity 2019.4.6

 

This is a big reason why we introduced NatMic, our native microphone API. It has since been succeeded by NatDevice, which supports both the camera and microphone.

 

I haven't faced this issue before, but check your camera's clipping distances.

 

Hello, how do you record just the ar camera using ar foundation and URP? I've been trying different options with stacking cameras but the video either is black or renders everything

 

hi,
What could this error be for?

 

@Lanre Would be good to know if you can think of a way this can be done? Would be useful in our project.

 

Hi,
I'm trying to find a definitive example of recording game audio and microphone audio using NatDevice and NatCorder.
With the code below i get stuttering mic audio. So i figured its perhaps a samplerate issue. But i'm unsure as to what to change.

You've mentioned in the docs not to use multiple Inputs of the same type. So i'm unsure if audioDevice.StartRunning is considered another input to the AudioInput.

I'm currently looking to see if the issue is related to the sampleRates being different between the device(mic) and the output sampleRate. I'm unsure as to what i should do here.

EDIT: So i disabled the _audioInput and instead of stuttering i got super fast playback. I think i read that thats clock related somewhere....

FIXED:
My continued search found an example in the forum to use Mixer device here.
https://forum.unity.com/threads/natdevice-media-device-api.374690/page-46#post-5648608

phew!

Would be great to have this in the help docs.

[Additional fix]
Further searching fixed an issue with iOS. https://forum.unity.com/threads/natdevice-media-device-api.374690/page-50
Destination array was not long enough. Check destIndex and length, and the array's lower bounds
Edit MixerDevice.cs to increase buffer size .
var copyBuffer = new float[16384];

Code (CSharp):

1.  
2.         // Microphone audio.
3.         var query = new MediaDeviceQuery(MediaDeviceQuery.Criteria.AudioDevice);
4.         audioDevice = query.currentDevice as IAudioDevice;
5.  
6.  
7.         Debug.Log("SampleRate: Device: "+audioDevice.sampleRate);
8.         Debug.Log("SampleRate: AudioSettings: "+AudioSettings.outputSampleRate);
9.        /// SampleRate: Device: 44100
10.        /// SampleRate: AudioSettings: 48000
11.  
12.  
13.         // Recorder setyop
14.         var clock = new RealtimeClock();
15.         recorder = new MP4Recorder((int) prod.x, (int) prod.y, 30, AudioSettings.outputSampleRate, 2);
16.         cameraInput = new CameraInput(recorder, clock, Camera.main);
17.  
18.         /// Game + Mic Audio
19.         _audioInput = new AudioInput(recorder, clock, Camera.main.gameObject.GetComponent<AudioListener>());
20.         audioDevice.StartRunning((sampleBuffer, timestamp) => recorder.CommitSamples(sampleBuffer, clock.timestamp));
21.        

Code (CSharp):

1. // Microphone audio.
2.         var query = new MediaDeviceQuery(MediaDeviceQuery.Criteria.AudioDevice);
3.         var microphone = query.currentDevice as IAudioDevice;
4.         audioDevice = new MixerDevice(microphone, Camera.main.gameObject.GetComponent<AudioListener>());
5.  
6.         Debug.Log("SampleRate: Device: "+audioDevice.sampleRate);
7.         Debug.Log("SampleRate: AudioSettings: "+AudioSettings.outputSampleRate);
8.         // Recorder setyop
9.         var clock = new RealtimeClock();
10.         recorder = new MP4Recorder((int) prod.x, (int) prod.y, 30, AudioSettings.outputSampleRate, 2);
11.         cameraInput = new CameraInput(recorder, clock, Camera.main);
12.  
13.         /// Game + Mic Audio
14.         //_audioInput = new AudioInput(recorder, clock, Camera.main.gameObject.GetComponent<AudioListener>());
15.         audioDevice.StartRunning((sampleBuffer, timestamp) => recorder.CommitSamples(sampleBuffer, clock.timestamp));

 

Hi Again,

The Black screen on on the Blackview android happens sometimes but on the Samsung S9 I have an issue that appears very often.
The camera seems to halt when the device comes back from being switched "off" (pause).
The logcat (attached here) states :

CameraService: CameraService::connect evicting conflicting client for camera ID 0

As I call (to test) a query.Advance(); I get this just after the "conflicting"

NatSuite: NatDevice Error: Camera 0 failed to stop preview

Steps to reproduce : run the app with the camera preview playing; Press the "on/off" button of the phone so the phone goes to sleep. Wait a bit and press the on/off button again, the phone will unpause (in my case ask the passcode) and then show the app again... with the camera not moving anymore. When I than call the "StopRunning" the script crashes on the failed to stop preview.

Thanks!

 

I haven't used ARF and URP together, so I'll have to check this and get back to you. It seems that the camera stacking feature might interfere with a standard render-to-texture.

 

This means that for some reason, the hardware codec is throwing an error. It is impossible to tell why this happens from the exception that is logged.

 

The challenge here is in mixing audio, while avoiding drifting. The MixerDevice implementation is a reference implementation, but it still falls into the synchronization issue. It will be removed in the next NatDevice update.

Basically, don't commit video from more than one source; and don't commit audio from more than one source. You can think of recording as a FIFO queue: the frames you put in, in the sequence you put them in, "come out" in the final recording. If you recorded say audio using both an audio device and an audio input, your "queue" will have two disparate audio streams sliced together. Think of this like a flickering video. What you want to do is to mix the audio from both sources, then commit them.

The MixerDevice implementation is a reference; it isn't intended to be used in production. It is being removed as a result. Mixing two audio streams is very much an open problem.

 

Looks like you posted on the wrong thread. This thread is for NatCorder issues; you'll want to follow up on the NatDevice thread.

It looks like there are two consecutive attempts to open the camera. I'll try to reproduce it on my end.

 

It happens 1/3 up to 1/5 procedures of putting the app to sleep. I do not re-initialize the camera. Only on initial "Start()" of the app

 

Totally with you on the one input per media.

Whilst it the mizer was sort of working, it basically needed a bit more testing around the buffer lengths which were throwig up errors.. So i switched out to just using the mic for now. I wouldn't necessarily remove Mixer unless you have an alternative. With some comments that its untested/supported and needs work, i actually felt it helped me understand the overall process of writing to the media streams. Someone might find it usefull is all.

The main issue i'm facing at the moment is with the stutter when initiating the MP4Recorder. I'm trying different resolutions to see if that helps. I'm wondering if starting in a coroutine would help.

 

I have a problem with the android app crashing when I tap the phone screen after I start recording. It works on iOS, but does not work when I build the APK and put it on my android device. please advise thank you!

Device: Doogee X10S runs Android 8
Android API Level target: 30
Unity version: 2019.4.2f
NatCorder version: 1.7.3

Code:

Code (CSharp):

1.     public IEnumerator StartRecording ()
2.     {
3.         if (recordingState != RecordingState.Stopped)
4.         {
5.             Debug.Log("Attempting to start recording when already active");
6.         }
7.         Debug.Log("Start Recording");
8.         yield return InitializeWithPermissions();
9.         if (!permissionsFailed)
10.         {
11.             // Start recording
12.             var frameRate = 30;
13.             var sampleRate = recordMicrophone ? AudioSettings.outputSampleRate : 0;
14.             var channelCount = recordMicrophone ? (int)AudioSettings.speakerMode : 0;
15.             clock = new RealtimeClock();
16.             recorder = new HEVCRecorder(Screen.width, Screen.height, frameRate, sampleRate, channelCount);
17.             CreateInputs();
18.         }
19.         else
20.         {
21.             Debug.Log("failed to get permissions to record");
22.         }
23.     }
24. private void CreateInputs()
25.     {
26.         audioInput = recordMicrophone ? new AudioInput(recorder, clock, microphoneSource, true) : null;
27.         // Unmute microphone
28.         microphoneSource.mute = audioInput == null;
29.         // configure camera inputs
30.         cameraInput = new CameraInput(recorder, clock, cameras.ToArray());
31.         recordingState = RecordingState.Active;
32.     }

Crash Log:
2020-11-07 16:25:28.023 5379-5379/? E/InputEventReceiver: Exception dispatching input event.
2020-11-07 16:25:28.023 5379-5379/? E/MessageQueue-JNI: Exception in MessageQueue callback: handleReceiveCallback
2020-11-07 16:25:28.026 5379-5379/? E/AndroidRuntime: FATAL EXCEPTION: main
Process: com.letsbramble.bramble, PID: 5379
java.lang.Error: *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
Version '2019.4.2f1 (20b4642a3455)', Build type 'Release', Scripting Backend 'il2cpp', CPU 'armeabi-v7a'
Build fingerprint: 'DOOGEE/X10S/X10S:8.1.0/O11019/20180717:user/release-keys'
Revision: '0'
ABI: 'arm'
Timestamp: 2020-11-07 03:25:13-0500
pid: 5379, tid: 5440, name: Thread-19 >>> com.letsbramble.bramble <<<
uid: 10133
signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr --------
r0 00000000 r1 00001540 r2 00000006 r3 00000008
r4 00001503 r5 00001540 r6 89481fcc r7 0000010c
r8 00000000 r9 89482019 r10 0000000a r11 89482018
ip 7f7fd420 sp 89481fb8 lr a9762a5b pc a975c48c

backtrace:
#00 pc 0001a48c /system/lib/libc.so (abort+64) (BuildId: 6f041c4782cb6c6ffbcea36ea218aaba)
#01 pc 00360b3f /system/lib/libart.so (art::Runtime::Abort(char const*)+262) (BuildId: 04658dfd580f58ea5c3323d4dc975b2e)
#02 pc 0042145d /system/lib/libart.so (android::base::LogMessage::~LogMessage()+452) (BuildId: 04658dfd580f58ea5c3323d4dc975b2e)
#03 pc 0024f981 /system/lib/libart.so (art::JavaVMExt::JniAbort(char const*, char const*)+1232) (BuildId: 04658dfd580f58ea5c3323d4dc975b2e)
#04 pc 0024fb77 /system/lib/libart.so (art::JavaVMExt::JniAbortF(char const*, char const*, ...)+66) (BuildId: 04658dfd580f58ea5c3323d4dc975b2e)
#05 pc 00283287 /system/lib/libart.so (art::JNI::GetObjectClass(_JNIEnv*, _jobject*)+670) (BuildId: 04658dfd580f58ea5c3323d4dc975b2e)
#06 pc 00000f87 /data/app/com.letsbramble.bramble-yjB9vZqHnTIlbJ1ni-1XAQ==/lib/arm/libNatCorder.so (NCCommitSamples+70) (BuildId: 2567ae6a57fad542c249dc2cc296d227ffcfe4c3)
#07 pc 010fb088 /data/app/com.letsbramble.bramble-yjB9vZqHnTIlbJ1ni-1XAQ==/lib/arm/libil2cpp.so (BuildId: cdfa55d4fc734abd8c7e61c2a595291c87af6fb7)
#08 pc 010fb778 /data/app/com.letsbramble.bramble-yjB9vZqHnTIlbJ1ni-1XAQ==/lib/arm/libil2cpp.so (BuildId: cdfa55d4fc734abd8c7e61c2a595291c87af6fb7)
#09 pc 010f9cd4 /data/app/com.letsbramble.bramble-yjB9vZqHnTIlbJ1ni-1XAQ==/lib/arm/libil2cpp.so (BuildId: cdfa55d4fc734abd8c7e61c2a595291c87af6fb7)
#10 pc 010fa228 /data/app/com.letsbramble.bramble-yjB9vZqHnTIlbJ1ni-1XAQ==/lib/arm/libil2cpp.so (BuildId: cdfa55d4fc734abd8c7e61c2a595291c87af6fb7)
#11 pc 00da6c88 /data/app/com.letsbramble.bramble-yjB9vZqHnTIlbJ1ni-1XAQ==/lib/arm/libil2cpp.so (BuildId: cdfa55d4fc734abd8c7e61c2a595291c87af6fb7)
#12 pc 0033c45c /data/app/com.letsbramble.bramble-yjB9vZqHnTIlbJ1ni-1XAQ==/lib/arm/libil2cpp.so (BuildId: cdfa55d4fc734abd8c7e61c2a595291c87af6fb7)
#13 pc 002707fc /data/app/com.letsbramble.bramble-yjB9vZqHnTIlbJ1ni-1XAQ==/lib/arm/libil2cpp.so (BuildId: cdfa55d4fc734abd8c7e61c2a595291c87af6fb7)
#14 pc 00228641 /data/app/com.letsbramble.bramble-yjB9vZqHnTIlbJ1ni-1XAQ==/lib/arm/libunity.so (BuildId: 3a891f7d3dec7a88ee0697a98eee5125e85cced4)
#15 pc 00231c8f /data/app/com.letsbramble.bramble-yjB9vZqHnTIlbJ1ni-1XAQ==/lib/arm/libunity.so (BuildId: 3a891f7d3dec7a88ee0697a98eee5125e85cced4)
#16 pc 0011d031 /data/app/com.letsbramble.bramble-yjB9vZqHnTIlbJ1ni-1XAQ==/lib/arm/libunity.so (BuildId: 3a891f7d3dec7a88ee0697a98eee5125e85cced4)
#17 pc 00875e0b /data/app/com.letsbramble.bramble-yjB9vZqHnTIlbJ1ni-1XAQ==/lib/arm/libunity.so (BuildId: 3a891f7d3dec7a88ee0697a98eee5125e85cced4)

at libc.abort(abort:64)
at libart.art::Runtime::Abort(char const*)(Abort:262)
at libart.android::base::LogMessage::~LogMessage()(~LogMessage:452)
at libart.art::JavaVMExt::JniAbort(char const*, char const*)(JniAbort:1232)
at libart.art::JavaVMExt::JniAbortF(char const*, char const*, ...)(JniAbortF:66)
at libart.art::JNI::GetObjectClass(_JNIEnv*, _jobject*)(GetObjectClass:670)
at libNatCorder.NCCommitSamples(NCCommitSamples:70)
at libil2cpp.0x10fb088(Native Method)
at libil2cpp.0x10fb778(Native Method)
at libil2cpp.0x10f9cd4(Native Method)

 

ARCore 1.18 + unity3d 2020.1.3f
Can't record mp4 of ARcore camera.
how to?
Your ReplayCam class saving a video file.
but recorded video image is just stopped when run app android 10
on PC working well(instant preview).
only problem in android app.

 

I'll move it to GitHub gists.

Make sure you're using the latest version of NatCorder.

 

Can you be more descriptive? I don't understand exactly what your setup is. What does ReplayCam have to do with you recording with ARCore?

 

Can you share the full, unfiltered logs from app start to crash in a .txt attachment? The crash is likely either because you are trying to record at screen resolution which is highly discouraged, or because your device doesn't have a HEVC encoder.

 

Hi there!

I just installed NatShare and NatCorder from Asset Store to my project. I'm making an AR filter app for iOS & Android. My app crashes when saving the video to the user's camera roll. I use:

microphoneSource.mute = true;
audioInput?.Dispose();
cameraInput.Dispose();
var path = await recorder.FinishWriting();

Debug.Log($"Saved recording to: {path}");

var payload = new SavePayload(Application.productName);
payload.AddMedia(path);

await payload.Commit(); // Here the app crashes

When testing it in Unity I got an error:

DllNotFoundException: NatShare
NatSuite.Sharing.SavePayload+<>c__DisplayClass0_0.<.ctor>b__0 (NatSuite.Sharing.Internal.Bridge+CompletionHandler callback, System.IntPtr context) (at Assets/NatSuite/Plugins/Managed/Sharing/SavePayload.cs:22)
NatSuite.Sharing.Internal.NativePayload..ctor (System.Func`3[T1,T2,TResult] payloadCreator) (at Assets/NatSuite/Plugins/Managed/Sharing/Internal/NativePayload.cs:21)
NatSuite.Sharing.SavePayload..ctor (System.String album) (at Assets/NatSuite/Plugins/Managed/Sharing/SavePayload.cs:22)
CameraManager+<get_pathAsync>d__36.MoveNext () (at Assets/Assets/Scripts/CameraManager.cs:304)
--- End of stack trace from previous location where exception was thrown ---
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () (at <fb001e01371b4adca20013e0ac763896>:0)
System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__6_0 (System.Object state) (at <fb001e01371b4adca20013e0ac763896>:0)
UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at <a979f18d51af41179d12b797e8c5be14>:0)
UnityEngine.UnitySynchronizationContext:ExecuteTasks()

Help!

 

Sharing won't work in the Editor. See this issue.

 

For sure it won't work in Editor, but in Android build it crash. The app closes without any logs, the crash happends in the commit line. I saw this issue previously and find out something from the source code, how's that?

 

Can you share the full unfiltered logs from logcat in a .txt attachment? Does your Android device have an x86 CPU? NatSuite API's only support armeabi-v7a and arm64-v8a architectures on Android.

 

I attached the log file. I'm using a Samsung Galaxy Note 8.

I need the app to save the recorded video to the camera roll. I'm sure the app has the user permissions.

 

I need the logs from app start to crash. These logs don't contain any information pertinent to this issue. Can you retake them?

 

@Lanre
hi

Do you have a tutorial for using native mode?

public void CommitFrame (IntPtr nativeBuffer, long timestamp)

 

No there isn't a tutorial. You should not use this method unless you know exactly what you are doing.

 

Same issue here, I can´t save the video to gallery, the app in android crashes and closes in the commit line.

I need help please.

 

I need to see the logs. Please provide the logs in a .txt attachment.

 

Don´t worry man, i have solved this issue using the nativeGallery plugin:

https://github.com/yasirkula/UnityNativeGallery

NatCorder and nativeGallery result in a very great combination.

Thanks and Regards!

 

It would be great if your plugin has support for setting video metadata such as "Rotation", "Author", "Title" etc before recording.

I have an issue where we need to output video in landscape mode in our portrait-locked app.
My solution right now is Graphics.Blit() with rotate shader every frame which did not look like a good idea considering it add even more command to the already stressed GPU (from re-rendering the scene) and we HAVE to read pixels after bliting which will cause sync point. No, we can not use Vulkan because we are developing an AR app.

I have tried using https://github.com/tanersener/mobile-ffmpeg to rotate video metadata after finish recording and it worked like a charm. There's no overhead when recording and it's super fast. Sadly, we can't use this in our product because of the LGPL licensing issue.

 

NatCorder won't be adding support for this because of sparse support and the engineering effort. Android doesn't support adding a metadata track until API level 26. And Windows' SinkWriter doesn't support writing metadata.

This is pretty fair; working from Unity you really don't have many options. Without Vulkan, you can't use async readbacks. You could simply transpose the pixel buffer on the CPU in a worker thread to free up the GPU. If you use unsafe code with pointers, you should get close to the speed you'd have in C/C++.

As far as I can tell, and from a compatibility standpoint, writing the metadata after the MP4 has been written is your best option. I recommend looking into a way to do this that doesn't rely on FFmpeg.

 

@lanrell Hi, Recording in Mp4 the screen with an HDR Camera as CameraInput in a URP doesnt loooks to work, the color are totally incorrect, is there a way to fix it ? PS : the recording integrated to the iphone record the right color.

- Tested on iPhone 12 pro
- URP 8.2
- Latest Natcorder Version
- HDR pipeline rendering ( HDR camera )
- Unity 2020.2.0b5
- DisplayP3 color profile

Thanks in advance !

 

Hi Lanre, yes the problem was fixed after changing to MP4, thank you

 

@lanrell Any Idea how to record the HDR colors ? Thanks!

 

I am too having problems using URP. The post-processing is not captured at all by the recording: bloom, HDR colors, etc.

Also it doesn't record any of the stacked Overlay cameras, the only solution was to set the Canvas to screen space Camera and link the Base camera to it, but that's not a good solution, it breaks other stuff not related to NatCorder. Is there a way to record other cameras when using URP in the current version?

 

It is possible stop device audio while recording?
Or record only some audiosource and others not?
Or add multiple audiosiurce mutated to the recording?

 

Also after a lot of work seems not work the async rentertexture

https://docs.natsuite.io/natcorder/workflows/recording-rendertextures

 

Is there any root issue that might cause very slight desync between audio and video recording? No error are being generated or the like, just a minor drift in the output file (like maybe a tenth of a second after a minute of recording).

 

My app just freeze everything after 4 seconds, with a script copied from the example...

It is because you use use 2 audio channels on mp4 recorder!, but in the demo it works..


this line of code

Code (CSharp):

1.             recorder = new MP4Recorder(
2. Screen.width,                           // Video width
3. Screen.height,                            // Video height
4. 30,                             // Video framerate
5. AudioSettings.outputSampleRate, // Unity's audio sample rate
6. 2  // Unity's audio channel count
7.  
8.  
9. );
10.      

or

Code (CSharp):

1.             recorder = new MP4Recorder(
2. Screen.width,                           // Video width
3. Screen.height,                            // Video height
4. 30,                             // Video framerate
5. AudioSettings.outputSampleRate, // Unity's audio sample rate
6. (int)AudioSettings.speakerMode  // Unity's audio channel count
7.  
8.  
9. );
10.      

totally crash my app after 4 seconds of recordings
but this one:

Code (CSharp):

1.             recorder = new MP4Recorder(
2. Screen.width,                           // Video width
3. Screen.height,                            // Video height
4. 30,                             // Video framerate
5. AudioSettings.outputSampleRate, // Unity's audio sample rate
6. 0  // Unity's audio channel count
7.  
8.  
9. );
10.      

works but not let me to commit audio that is the reason why my company buy this for


another strange behavour

Example:

this class work

Code (CSharp):

1. /*
2. *   NatCorder
3. *   Copyright (c) 2020 Yusuf Olokoba
4. */
5.  
6. namespace NatSuite.Examples {
7.  
8.     using UnityEngine;
9.     using UnityEngine.UI;
10.     using System.Collections;
11.     using Recorders;
12.     using Recorders.Clocks;
13.     using Recorders.Inputs;
14.     using UnityEngine.Rendering;
15.  
16.     public class WebCam1 : MonoBehaviour {
17.  
18.         [Header(@"UI")]
19.  
20.         public MP4Recorder recorder;
21.         public IClock clock;
22.         public bool recording;
23.         private Color32[] pixelBuffer;
24.         private AudioSource microphoneSource;
25.         public static WebCam1 instance;
26.         public RenderTexture rt;
27.         public Texture2D TestTexture;
28.         #region --Recording State--
29.         CameraInput cameraInput;
30.         public bool SimpleRecording;
31.         public void StartRecording () {
32.    
33.             clock = new RealtimeClock();
34.  
35.             recorder = new MP4Recorder(
36. Screen.width,                           // Video width
37. Screen.height,                            // Video height
38. 30,                             // Video framerate
39. AudioSettings.outputSampleRate, // Unity's audio sample rate
40. 2  // Unity's audio channel count
41.  
42.  
43. );
44.             // Create recording inputs
45.             if(SimpleRecording)
46.             cameraInput = new CameraInput(recorder, clock, Camera.main);
47.             recording = true;
48.         }
49.  
50.         public async void StopRecording () {
51.             if(SimpleRecording)
52.             cameraInput.Dispose();
53.  
54.             // Stop recording
55.             recording = false;
56.             var path = await recorder.FinishWriting();
57.             // Playback recording
58.             Debug.Log($"Saved recording to: {path}");
59.             Handheld.PlayFullScreenMovie($"file://{path}");
60.         }
61.         #endregion
62.  
63.  
64.         #region --Operations--
65.  
66.         IEnumerator Start () {
67.             instance = this;
68.             yield return Application.RequestUserAuthorization(UserAuthorization.WebCam);
69.             if (!Application.HasUserAuthorization(UserAuthorization.WebCam))
70.                 yield break;
71.         }
72.  
73.         void Update () {
74.             // Record frames from the webcam
75.             if(!SimpleRecording)
76.                 if (recording)
77.                 {
78.                  
79.                         Debug.Log(rt.width);
80.                         Debug.Log(rt.height);
81.  
82.                         Texture2D myTexture2D = new Texture2D(rt.width, rt.height);
83.                            RenderTexture.active = rt;
84.                            myTexture2D.ReadPixels(new Rect(0, 0, rt.width, rt.height), 0, 0);
85.                            myTexture2D.Apply();
86.                            Color32[] fanculo = myTexture2D.GetPixels32();
87.                            recorder.CommitFrame(fanculo, clock.timestamp);
88.                 }
89.             }
90.  
91.         private IEnumerator InitMicrophone()
92.         {
93.             microphoneSource = gameObject.AddComponent<AudioSource>();
94.             microphoneSource.mute =
95.             microphoneSource.loop = true;
96.             microphoneSource.bypassEffects =
97.             microphoneSource.bypassListenerEffects = false;
98.             microphoneSource.clip = Microphone.Start(null, true, 10, AudioSettings.outputSampleRate);
99.             yield return new WaitUntil(() => Microphone.GetPosition(null) > 0);
100.             microphoneSource.Play();
101.         }
102.         #endregion
103.     }
104. }

this one cause the freeze of the app, BUT ARE THE SAME CLASS!!!!

Code (CSharp):

1. *
2. *   NatCorder
3. *   Copyright (c) 2020 Yusuf Olokoba
4. */
5.  
6. namespace NatSuite.Examples {
7.  
8.     using UnityEngine;
9.     using UnityEngine.UI;
10.     using System.Collections;
11.     using Recorders;
12.     using Recorders.Clocks;
13.     using Recorders.Inputs;
14.     using UnityEngine.Rendering;
15.  
16.     public class WebCam2 : MonoBehaviour {
17.  
18.         [Header(@"UI")]
19.  
20.         public MP4Recorder recorder;
21.         public IClock clock;
22.         public bool recording;
23.         private Color32[] pixelBuffer;
24.         private AudioSource microphoneSource;
25.         public static WebCam2 instance;
26.         public RenderTexture rt;
27.         public Texture2D TestTexture;
28.         #region --Recording State--
29.         CameraInput cameraInput;
30.         public bool SimpleRecording;
31.         public void StartRecording () {
32.    
33.             clock = new RealtimeClock();
34.  
35.             recorder = new MP4Recorder(
36. Screen.width,                           // Video width
37. Screen.height,                            // Video height
38. 30,                             // Video framerate
39. AudioSettings.outputSampleRate, // Unity's audio sample rate
40. 2  // Unity's audio channel count
41.  
42.  
43. );
44.             // Create recording inputs
45.             if(SimpleRecording)
46.             cameraInput = new CameraInput(recorder, clock, Camera.main);
47.             recording = true;
48.         }
49.  
50.         public async void StopRecording () {
51.             if(SimpleRecording)
52.             cameraInput.Dispose();
53.  
54.             // Stop recording
55.             recording = false;
56.             var path = await recorder.FinishWriting();
57.             // Playback recording
58.             Debug.Log($"Saved recording to: {path}");
59.             Handheld.PlayFullScreenMovie($"file://{path}");
60.         }
61.         #endregion
62.  
63.  
64.         #region --Operations--
65.  
66.         IEnumerator Start () {
67.             instance = this;
68.             yield return Application.RequestUserAuthorization(UserAuthorization.WebCam);
69.             if (!Application.HasUserAuthorization(UserAuthorization.WebCam))
70.                 yield break;
71.         }
72.  
73.         void Update () {
74.             // Record frames from the webcam
75.             if(!SimpleRecording)
76.                 if (recording)
77.                 {
78.                  
79.                         Debug.Log(rt.width);
80.                         Debug.Log(rt.height);
81.  
82.                         Texture2D myTexture2D = new Texture2D(rt.width, rt.height);
83.                            RenderTexture.active = rt;
84.                            myTexture2D.ReadPixels(new Rect(0, 0, rt.width, rt.height), 0, 0);
85.                            myTexture2D.Apply();
86.                            Color32[] fanculo = myTexture2D.GetPixels32();
87.                            recorder.CommitFrame(fanculo, clock.timestamp);
88.                 }
89.             }
90.  
91.         private IEnumerator InitMicrophone()
92.         {
93.             microphoneSource = gameObject.AddComponent<AudioSource>();
94.             microphoneSource.mute =
95.             microphoneSource.loop = true;
96.             microphoneSource.bypassEffects =
97.             microphoneSource.bypassListenerEffects = false;
98.             microphoneSource.clip = Microphone.Start(null, true, 10, AudioSettings.outputSampleRate);
99.             yield return new WaitUntil(() => Microphone.GetPosition(null) > 0);
100.             microphoneSource.Play();
101.         }
102.         #endregion
103.     }
104. }

It is useless in this way, waste 1 week of my time and money I am REALLY disapponinted


This sdk create also more problem on my app.....

 

NatCorder does not support HDR rendering at this time.

 

I'm not sure what you mean by stopping device audio while recording. If you want to 'mute' the audio while recording, then you have to send zeros in the audio buffer to the recorder. Recording some audio sources and not others is not trivial; you have to manually mix the audio from the audio sources before sending them to the recorded. I'm not sure what you are asking with your third question.

 

I'm not aware of any drifting that happens. What device does this happen on? Can you share your recording code?

 

You are specifying an audio format (passing in non-zero numbers for the sample rate and channel count) without actually committing any audio (no audio input, no recorder.CommitSamples). On Windows, this always results in a hang. Don't specify an audio format if you are not committing any audio. Read the documentation because all these notes are listed.

 

I forgot to mention that I commit the audio with another script

this is used for the example of webcam2

Code (CSharp):

1. void OnAudioFilterRead(float[] data, int channels)
2.     {
3.         if(record)
4.         if (data.Length > 0)
5.                 {
6.                     if(WebCam2.instance!=null)
7.                     if (WebCam2.instance.recording)
8.                     {
9.                         WebCam2.instance.recorder.CommitSamples(data, WebCam2.instance.clock.timestamp);
10.              
11.                     }
12.                 }
13.             if (mute)
14.         {
15.             System.Array.Clear(data, 0, data.Length);
16.         }
17.     }

And sometimes it also cause some problem with the camera (is a AR project)

btw I also have a delay in the michropone input too in the scene with the mic recorder, can i compensate it with removing some seconds from the clock.timestamp while recording?

Could be the unity version? 2019.4.11f?

 

Does the ReplayCam example work for you?

 

I have also to mention that the framerate DROP A LOT on recording on android, and also on android it crash

and as you told me i read the documentation but seems also the async not work

https://docs.natsuite.io/natcorder/workflows/recording-rendertextures

if i put this

Code (CSharp):

1. AsyncGPUReadback.Request(renderTexture, 0, request => {
2.     // Once complete, access the data container
3.     var nativeArray = request.GetData<byte>()
4.     // And commit the pixel buffer
5.     recorder.CommitFrame(nativeArray.ToArray());
6. });

in the update it not record correctly, why?

 

UPDATE:
the reply is...unfortunately not.....

When i try to recorder a videoplayer with external audiosource all the audio get cracked...