NatDevice - Media Device API

The aspect ratio here is 1.778. This is different from the size of the recording, which is controlled by the preview size.

This is also controlled by the preview, so you must set the camera's preview framerate to 30.

You can set this with the Configuration struct that is passed in to NatCam.StartRecording:

Code (CSharp):

1. NatCam.StartRecording(
2.     new Configuration(bitrate, keyframeInterval, recordAudio?), // Create a recording configuration
3.     OnVideo // Callback
4. );

 

Thank you for your prompt reply.

Please explain the variables of keyframeInterval.

 

It is the number of seconds between keyframes. Keyframes are frames that aren't interpolated. This is a good thread than explains it.

 

I have errors when running and building as windows standalone using three different cameras. Does this only support smartphone? Using the Core version.

Code (CSharp):

1.  
2. Could not connect pins - RenderStream()
3. UnityEngine.WebCamTexture:Play()
4. NatCamU.Core.Platforms.NatCamLegacy:Play() (at Assets/NatCam/Core/Plugins/Managed/Platforms/Legacy/NatCamLegacy.cs:92)
5. NatCamU.Core.NatCam:Play(DeviceCamera) (at Assets/NatCam/Core/Plugins/Managed/NatCam.cs:81)
6. NatCamU.Core.NatCamBehaviour:Start() (at Assets/NatCam/Core/Plugins/Managed/NatCamBehaviour.cs:47)
7.  

Edit:
From MSDN, similar errors to this are caused by missing directshow filters, so installing the codec pack fixed it.

 

Hi,
I'm using the last version of Natcam on Unity 2017.3.0f3 and I've got some issues.
When I try to Release the camera and Play it later, my app crashes on Android with an exception : AndroidJavaException: java.lang.NullPointerException: Attempt to invoke virtual method 'void com.yusufolokoba.natcam.rendering.GLRenderContext.stop()' on a null object reference
I don't have this exception on Galaxy S8 and on Editor.
Am I doing something wrong ?

Thanks a lot for your attention.

 

Good catch. I looked into it and saw that it is a DirectShow API issue.

 

Can you email me the logs? What device are you running on? It seems that you are calling NatCam.Release() multiple times.

 

Hi,

I am using NatCam.Play( DeviceCamera.RearCamera), I have some issues on this

1. on iOS it will request for microphone permission, can I disable this because I'm using the camera to take photo only
2. when I exit the page, I call NatCam.StopRecording() and NatCam.Release (). on iOS the device volume control is not function, I am not able to change the volume of the background music

 

Try deleting your current version of NatCam and reimporting it from the Asset Store. We have fixed this issue.

On iOS, when the camera is running, it usually doesn't allow the volume buttons to change volume. You can verify this in the stock iOS camera app.

 

on iOS, after I close the camera screen and the camera is not running, but the volume button is not function.

 

I have opened an issue for this. I will look into it.

 

Hello Lanre,
I just spotted that videos recorded from Iphone were in .mov
I did all my pipeline on .mp4 format :/ is there a way to save in .mp4 instead ?

Thank you very much !
Seb

 

The videos are already encoded in with the MPEG4 format. You can use the .mov file like any other .mp4 file.

 

Thank you Lanre.

In fact I have this code which is working great on Android, but I get an error on IOS :

Code (CSharp):

1.  
2. private void OnVideo (string path)
3. {
4.     NatCam.Pause();
5.  
6.     string videoPath = newPath+"video1"+".mp4";
7.    
8.     //This prints "KO"
9.     if (System.IO.File.Exists(path))
10.         {
11.             Debug.Log ("OK");
12.         }
13.     else
14.         {
15.             Debug.Log ("KO");
16.         }
17.  
18.     System.IO.File.Move (path, videoPath);     //Error => FileNotFoundException : file does not exist
19. }
20.  

Do you have any idea why it's not working ?

 

iOS file paths are prepended with the 'file://' protocol. I don't believe the System.IO API's support this, so you might need to remove the prefix.

 

Thank you Lanre, any update on this issue?

 

Hi Lanre,

It seems that we can't get a proper Color32 array by casting the Preview texture to a Texture2D and calling GetPixels32() (i'm trying to detect a QRCode here), is that right ?

cheers

 

We were not able to reproduce this issue. You can confirm this by starting a new project in Unity, importing NatCam, and building the ReplayCam example for iOS.

 

You are correct. If you would like to get the preview data, you need NatCam Pro. You can then call NatCam.PreviewFrame and use GetPixels32() on the resulting texture.

 

Hi, I'm still using the 1.5 version. Just a quick question, how do I get the name of the picture I capture? I get the path, but not the file name.

 

Do you mean a video? NatCorder doesn't necessarily assign a name, you can simply check the path for what the video is called. It is usually in the form ''video_[date and time].mp4"

 

What about pictures? I see a string of numbers, and it doesn't seem like date and time. Like 1519726336944.jpg

 

NatCam doesn't save pictures to the device (and does not return the path). NatCam.CapturePhoto simply returns the Texture2D directly to you. This is why I thought you were referring to recorded videos. Are you sure you are using NatCam 1.5? Can you share your photo capture code?

 

Hmmmm. I guess then it's not 1.5. Sorry about that. It's the version where NatCam Extended wasn't deprecated yet, is instead of returning the Texture2D, it would save the picture into the directory preselected.

This is what I have

NatCam.SavePhoto(photoTaken, NatCamU.Extended.SaveMode.SaveToPhotoGallery, Orientation.Rotation_270, (mode, path) => GameObject.Find ("Path").GetComponent<UnityEngine.UI.Text> ().text = "Photo saved with "+mode+" to path: "+path);

 

You can check the README.md file or Changelog.md file for what version of NatCam you are using.

To get the file name, simply get the last token of the path tokenized by the "/" character. That is, split the path string by "/" and the last token will be the name+extension of the file.

 

Is there a way to use this at the same time as iOS face tracking? Just tried it and it seems ARKit and NatCam are fighting for the front camera sensor.

 

Not at all. OSes have the restriction that only one process has exclusive access to the device cameras. Hence no two API's can perform operations on the device cameras. You cannot use NatCam while ARKit is currently running and vice versa.

 

Makes perfect sense, thanks.

 

Hi, Lanre. Thanks for this wonderful asset you provide.

I've got a clarity-focused suggestion that of course you can take or leave. Since "High", "Medium", and "Low" are quite subjective descriptors for resolution (that will seem inaccurate years later as standard resolutions increase), renaming the ResolutionPreset enum to something like the following would be much clearer, IMO:

[CoreDoc(61)] public enum ResolutionPreset : byte {
[CoreDoc(62)] r1280x720 = 0,
[CoreDoc(63)] r1920x1080 = 1,
[CoreDoc(64)] Highest = 2,
[CoreDoc(65)] r640x480 = 3,
[CoreDoc(66)] Lowest = 4,
}

 

This is a fair point. Thanks for the suggestion; I have made the change.

 

Great to hear. Thanks.

Lanre, have you considered NatCam's current (or future) compatibility with ARKit and ARCore? I see in the top post here that someone had trouble with an integration involving ARKit and NatCam. I have not yet tried myself, though it's highly likely I will.

I expect that for many of us NatCam users, ARKit and ARCore will be something we want or even need to integrate into our projects. Any knowledge you can share of current compatibility or planned compatibility would be helpful to hear.

 

NatCam and ARKit/ARCore cannot run at the same time (but can be used at separate times throughout the app). The reason for this is that only one process/service can have exclusive access to the device cameras. This is an OS-level restriction imposed by Android and iOS.

 

I'm thinking on replacing WebCamTexture by NatCam. I'm choosing between Core and Pro versions. Camera functionality is not crucial for my app, so all I want is - frame preview, maybe with some focus control / AF features, with possibility to take shot on click. I'm even not saving photo, just using shot texture for image recognition.

So normally NatCam Core looks like sufficient solution for my purposes. However description of Pro package tells me that Core have some limitations, related to preview / texture grab functionality. So should Core really be enough for the purpose described?

P.S. I'm also interested in using portrait camera mode without need to rotate image/texture manually, is it possible?

 

If you want to be able to capture high-res photos (you mentioned "taking shot on click"), then NatCam Core would be sufficient. But if you want the preview data or preview frame on demand (like taking a 'screenshot'), then you need NatCam Pro.

On Android (only), photos are not oriented upright (because of performance considerations), but there is a function to do the rotation for you. NatCam will always provide captured photos with an orientation flag, which you can pass in to the utility function (included in NatCam) to do the rotation for you.

 

I'm a bit confused here. Isn't "taking shot on click" merely a taking "preview frame" on demand by user click action? Or you mean, with Core I may take full res photo data, but not a resized preview one? But if you mean that Core may only take photo without preview, it doesn't make much sense to user, isn't it?

Ok, let me describe issue a bit more detailed. I have my own photo screen layout, where I want to take 3:4 photo (it's like 4:3, but in portrait mode). So I made layout with AspectRatioFitter, that will make my preview frame as large as possible (depending on mobile screen size, it may be limited by width, but sometimes on height, because of additional controls). But without manually applying rotation, preview data from WebCam applied to that texture shows as highly distorted (because it's original 4:3 shrinked to 3:4 sprite).

In the same time, when I apply rotation to sprite according to WebCam info, it shows image properly, but in a smaller way. Because actually AspectRatioFitter layout is not taking into account texture rotation, of course, so its larger side is equal to screen width, not a smaller one.

So, that's all what I want - is a possibility to say to library "hey, I want to show portrait photo preview with desired portrait aspect to this one texture, also aligned vertically". Is that possible?

 

NatCam Core and Pro give you the camera preview in a Texture object (note that this is not a Texture2D, as it does not have any data in system memory, only on the GPU). NatCam Core and Pro also offer a CapturePhoto function which take a full-res photo from the camera (applying things like flash and so on), and return that to you in a Texture2D (so that you can get the pixel data using GetPixels). What NatCam Pro offers that NatCam Core doesn't is the ability to get the preview pixel data as the preview is being streamed. Let me know if any of this isn't clear.

I see. Firstly, NatCam does not handle displaying photos at all; it simply gives you the photo as a Texture2D and you are free to do anything you want with it. Hence, you will still need to manually set the aspect ratio on the AspectRatioFitter and display the photo upright (NatCam also has a component for using a rotation material to display textures upright). The MiniCam example included in NatCam does these things.

I am looking into making the photos on Android always upright so that it is one less thing that developers have to worry about.

 

Good news! Following from your questions, I decided to investigate ways to quickly rotate the captured photo before passing it to the client (C#). I was able to find a satisfactory method, meaning that now you can expect all photos to be upright when returned by NatCam.
__
To all NatCam devs:
This change is pretty significant. Firstly, it means that we are changing the signature of the photo callback to return only the captured photo (you don't need to worry about Orientation anymore). Second, it means that we are deprecating the Orientation struct (I'm personally very happy to see this finally go, it added a lot of complexity in handling photo textures). Third, it means that the NatCamPanel component is not needed anymore, so it is being deprecated. Fourth, it means that the Utilities.RotateImage functions are getting deprecated too since they won't be needed. Fifth, it means your life will now be made so much easier because there's significantly less that you have to worry about when it comes to captured photos.

@Zdreni , thank you!

 

I seem to be affected by this driver bug as well, on an Asus Zenfone AR. I have the preview resolution set to 640x480 and the screen was barely updating. I set the ReadablePreview flag to false as you recommend and the preview becomes smooth. However, I need the preview data to send to OpenCV for my application. Is there any other workaround allowing me to get the preview data? Thanks.

 

The workaround for a device like this is to use Graphics.Blit with Texture2D.ReadPixels and Texture2D.GetRawTextureData/GetPixels32 to download the pixel data from the GPU. Blit the NatCam preview to a temporary render texture; set the RT active and readback into a Texture2D (using ReadPixels), then access the pixel data in the texture.

 

Hi Lanre, I would like to buy your asset for recording an AR scene with a facetracking plugin called Visage.
Is there a working video/example of an AR app recording camera feed + different unity cameras? (for example in the recorded video we need to exclude the UI camera, but not the 3D models being displayed on another unity camera).

I know there is NatCorder the would make it simpler, but my client constrained the unity version to unity 5.6.4p4, and your plugin just supports 2017.1 and greater versions.
Thanks in advance!

 

Hi Alfred. It sounds like you want NatCorder, not NatCam. NatCam is strictly a camera API. NatCorder will work with Unity 5.3+. The package was simply submitted with Unity 2017.1, which is why the Asset Store says it supports that version and newer.

 

Hi Lanre, Two questions
1. Is it possible to do two streams one from front camera and one from rear camera in the same scene on Android? the way I am trying right now is have two RawImages in separate Canvases and apply ReplayCam component to each. Currently both RawImages only show either rear or front camera stream
2. With just the camera on (single stream, not taking any photos or making any recording) I am hitting an error in libmono.so followed by app crash after few minutes

Multiple streams, is it currently possible? Best way to share logs?

Thanks

 

Mobile devices do not allow running more than one camera at any given point in time. I responded to your email.

 

Can NatCam save photos to the ios camera roll or would we need to implement that separately?

 

NatCam doesn't support saving to the device gallery. You can implement it yourself or use a plugin that can do it.

 

Hello Lanre,

Where can i find the OpenCV for Unity integration with NatCam tutorial ???

 

We don't have a tutorial yet. We are considering making a slight change that will not require us to have an OpenCV API (instead, developers can use the `Utils` functions that come with OpenCVForUnity).

 

Hi, Lanre. Thanks for this wonderful asset you provide.

I just published a repository of examples of integration between NatCam and OpenCVForUnity.
When I tested on some mobile devices, it seems to work well.
https://github.com/EnoxSoftware/NatCamWithOpenCVForUnityExample

 

Hey EnoxSoftware! Thank you so much for this. We have some changes planned that should further improve the speed of NatCam with OpenCV (specifically in flipping the preview buffer). I will open PR's on this repo once we have the changes live. Thanks for your amazing asset and the support you provide!

 

Hello Lanre, I deployed an app with the previous version of NatCam and a friend told me that my app is instant crashing when he uses the natCam part.

He has a Samsung Galaxy A5 (Android 7.0). Do you heard issue about this model ? I can't have a log because he is far away. I will investigate on my side.

Seb