NatDevice - Media Device API

It seems like you are not using the latest version of the Android SDK. Update it and try building again. If it doesn't work, do this:

Code (CSharp):

1. Rename NatCam.aar in Assets>NatCam>Plugins>Android to NatCam.zip.
2. Extract NatCam.zip.
3. In the unzipped folder, open AndroidManifest.xml.
4. Change the targetSdkVersion in the uses-sdk tag to your SDK version (yours is 22).
5. Rezip the file.
6. Rename from .zip to .aar

 

Hi, love the plugin so far and see it offers face recognition and tracking. Any plans on Natcam having any other kind of image recognition/tracking? Such as something to print out and paste on the walls that will allow you to trigger a virtual object based on distance and orientation? That would be amazing! Not to mention justify a higher asking price.

 

Thank you very much for your help, eventually it works.

Maybe it will help someone:
1) First of all, I've updated Android SDK, but it wasn't help, I still had the same error
2) Second, I made suggested changes in AndroidManifest.xml which located in NatCam.aar.
And then I got a different error:
Compiling Error: Major Version 52 is newer than 51....
3) I've conducted some research on the internet and came up with conclusion that it is the java version issue.
4) Despite I have java version 1.8, I've downloaded the new one: 'build 1.8.0_92-b14' then reboot my Mac after the installation, but the error doesn't go anywhere.
5) Last step, I just deleted 'android-24' folder from android sdk and voila, it's work!

 

Hi @christougher. Unfortunately, this won't be possible for NatCam. This goes beyond the scope of a camera API and enters the realm of augmented reality. We could consider creating a separate asset to do this. I will keep this post updated as we decide what to do.

 

Ahh, great that you found a fix. We'll stick with this then. Thank you!

 

I was faced with a new problem:
I'm using the portrait orientation, and I set it up in Unity player settings.
So, when I run the app and rotate the phone to landscape the image on the screen from the camera is acting weird, it is distorted.

And it seems like the camera is trying to stay in portrait orientation, I don't know how to explain it, just take a look at the image:

How can I fix it, are there any settings which I have to take into account?

 

I face new problem when use both Vuforia and NatCam.
Scene 1: I put Vuforia.
Scene 2: I put NatCam.
My flow: Load scene 1 -> load scene 2 -> load scene 1
When I reload scene 1 from scene 2, Vuforia camera appear black screen.
I delete NatCam from scene 2 everything is ok.
I try to disable NamCam before I reload scene 1 but it not works.
It's seem NatCam change some settings that conflict Vuforia

 

What device are you running on? And do you rotate the app to landscape (using AutoRotation in player settings), or do you simply rotate the device physically? I have faced this issue several times when developing NatCam.

 

Make sure you call NatCam.Release() when you close scene 2 and open scene 1. Note that NatCam and Vuforia are currently incompatible, so ideally you should not be using both of them together.

 

I've used iPhone 6 plus and I simply rotated the device physically.
And you know what, the picture on the screen seems to be distorted even if I rotated the phone on 20-45 degrees angle. I could make a photo if you want.

At the end NatCam doesn't work smoothly on iphone 6 and i decide to use unity WebCamTexture on IOS devices because it works much faster and without glitches ((.
Maybe the next version of NatCam will work better on IOS and then I will use it.
As for now I going to use it only for Android.

 

Do you have NatCamPreviewScaler enabled? If so, what is its scale mode? Not properly scaling the preview will cause this behaviour.

This is very much debatable. NatCam and WebCamTexture on iOS essentially do the exact same thing, except that NatCam gives you much more control over the camera. Concerning the smoothness of the preview, this could be because NatCam gives a higher resolution preview than WebCamTexture (NatCam allows up to 4K preview on iOS which is completely beyond WebCamTexture); or it could be because you are running a readable preview which starts to have an effect on performance for very high resolution previews. Overall, WebCamTexture and NatCam running side-by-side with the same resolution will provide the exact same performance and smoothness.

 

Hi,

I can use NatCam in my app to record videos with the smartphone camera and share them on facebook?

Thanks

 

Hi. No, NatCam does not support video recording yet. We might add it in 1.4.

 

@Lanre There are examples of code or tutorial to understand how to integrate facial recognition? (Perhaps with the insertion of a virtual object by using the extrapolated coordinates by the algorithm)

 

Hi @D3m0n . Currently, there isn't any example up for facial recognition. I have been very behind on video tutorials and tutorial projects. Next week, I will be releasing several NatCam Snippets episodes including Face tracking, and I will upload the project to GitHub. Sorry for the wait.

 

Hi @Lanre . I'm having the same problems Denis is having with the NatCam plugin on iOS. The camera feed on screen is distorting/warping just like the screenshot he supplied a few days ago. To reproduce what we're seeing, create a new project. Import the NatCam plugin. Open the demo scene. Build and Run. I have an urgent deadline that was actually last week Friday, but I haven't been able to resolve this issue. Can you please assist me with this issue urgently? It works great on Android.

I'm using Unity 5.3.4p3, and ran the app on my iPhone 6s.

 

Hi @Brent_The_3D_Guy do you have a NatCamPreviewScaler attached to your RawImage with a ScaleMode other than None? Denis' problem looked like UV stretching as a result of different aspect ratios of the camera preview and the preview UI panel.

I'll try to reproduce the issue on my SE. What resolution or resolution preset are you using?

EDIT: I can confirm that it is because the preview UI panel is not being scaled correctly. Check the ScaleMode of NatCamPreviewScaler.

 

Hi NatCam Dudes/Dudettes,

I see you offer support for a whole range of barcodes (https://abfa31284539dde9981c6be9d12.../host/0B5rPOCTjzOsgYVdhZkFaZmxzakE/index.html)

but, I am wondering if NatCam supports the ITF (interleaved 2 of 5) format?
(info: https://en.wikipedia.org/wiki/Interleaved_2_of_5 )

I have included an image of the type of barcode I would need to scan.
https://www.dropbox.com/s/iy9znlegix7bodf/test.jpg?dl=0

Thank you for the info.

Kind regards
Wim

 

The demo scene bundled with the NatCam plugin (Minigram) already has a "NatCamPreviewScaler" on the RawImage. It was on "Fixed height variable width" by default, but I get distortion no matter what Scale Mode I choose. It's as if the NatCamPreviewScaler is not scaling properly. i.e. The camera feed is always displayed as a square if I rotate my phone to landscape from portrait. "Fill" does not fill when orientation changes. I would expect this "Fill" setting to fill the entire screen with the image with the proper aspect ratio, but I still only get a square image in the centre of the screen. Even if I disable AutoRotation be setting my "Default Orientation" to Portrait only, the image will appear fine in Portrait, but rotating the device will result in a distorted texture. When rotating the phone to landscape, the image should then not rotate, but it does. It's as if the image is still in portrait mode, but rotated 90 degrees.

Do you have any other suggestion I can try? I need to be able to use this feature in both landscape and portrait modes.

 

NatCam 1.3 does not support ITF, but 1.4b0 (the 1.4 beta I'm working on) does. Here's an APK that scans APK codes. You can purchase NatCam from the Asset Store, email me your invoice no, and I'll send you 1.4b0.
^The ITF code you attached.

 

Hmm. Can you post a picture of Fill mode? And I haven't tested AutoRotation, but it will probably look weird because NatCam orients the preview based on the app orientation. So if you are changing rotation using AutoRotation, the preview will rotate and flip in front of the user. I personally don't advise using AutoRotation with a camera preview, instead manually rotate your UI to fit (and keep the app's orientation constant) like the stick iOS camera app does. The weirdness when you run in portrait only sounds very much like PreviewScaler. To be 100% sure, do this:

Code (CSharp):

1. Create a fullscreen UI panel (stretch the canvas/panel across the screen).
2. Remove the NatCamPreviewScaler or set its ScaleMode to None (so that no scaling occurs).
3. Run NatCam with either HD or FullHD resolution.
4. Deploy on your device and see if the weirdness remains.

The point of this test is that the aspect ratio of your screen (16:9 on every iPhone that isn't older than a 5) and of the camera preview (HD at 720p or FullHD at 1080p has aspect 16:9) will be the same (which removes the need for a NatCamPreviewScaler). Theoretically, the weirdness would not be present. Let me know the results of this test.

 

Great, thanks! Maybe just one last question. I'm having problems with camera focus when I try to get close to objects with the standard webcamtexture setup. Will NatCam be able to do "Macro-focus"?

Thanks again!

Wim

 

Not out of the box; Macro focus isn't supported. I could edit the sources for you to make autofocus use macro focus mode instead.

 

Everyone experiencing the weird behaviour with NatCamPreviewScaler, here is an updated version of the component. Let me know if the issue still persists for you. To be able to use this updated version, you must add "FillView" to the ScaleMode enum in NatCamProperties.cs.

Please note that camera apps should not use AutoRotation. This interferes with the way camera application corrects the preview data orientation. This is also VERY BAD for UX (user experience). Like other camera apps do, manually handle orientation changes by adjusting the UI instead of rotating the entire app. See the stock iOS camera app or the stock Android camera app on how it is done.

 

Does anyone know how the Unity certification program work? I'm available to do freelance jobs and I'm looking where devs like me can offer our services.

 

Hi @Lanre , thanks so much for your feedback. I'll do those tests as soon as I get a chance. I'm currently resizing the RawImage UI element manually through code, which is working well now. The only reason I was using the Autorotation, was because I was using another plugin to then take and share a screenshot. Without changing the orientation, the image would, obviously, be shared at 90 off the correct orientation if the photo was taken in landscape. I agree with you that the view should be fixed to either portrait or landscape. I'll read up on how to rotate my Texture2D screenshot before posting, based on the device orientation

Thanks for your speedy support! It really is a great plugin and is working well now on my side, although with autorotation currently. I'll correct it if theres still time or in the next update.

 

You're welcome! Try the new NatCamPreviewScaler I uploaded.

 

Hey @Lanre, thanks for all the work you're doing once again! The feature set is great, and it's more stable and easy to use now too.

I've been fussing a bunch about performance of the VR app I'm working on. Getting the 60 frames per second target on Gear VR will be paramount to its success, of course. Even though NatCam is much (muuuch) faster than WebCamTexture, I keep coming back to NatCam as the major source of frametime.

On a Galaxy S7, using Unity 5.3.5f1, with multithreaded rendering enabled, I get an average of 40 frames per second on the Minigram demo app. The same goes for my custom app. Not quite where I'd need it to be.

I don't technically need the camera itself to run at 60fps right now, but the app itself should.

I've been profiling and checking out the managed side of the code (without multithreaded rendering enabled), and it looks like roughly all the frametime is spent on:

Code (csharp):

1. NatCamNativeInterface.cs
2. 270: GL.IssuePluginEvent(NatCamNativeCallback(), request);

This uses up about 15ms in this configuration, which is pretty much our entire frame budget for both CPU and GPU time. This is probably where all the heavy lifting is done? Anyway, since it goes into native code that's where my search ended. This is better than anything I can get with WebCamTexture, but it's still not great.

I spotted the DispatchMode.Asynchronous option and gave it a shot like so:

Code (csharp):

1. NatCamNativeInterface.cs
2. 115: Dispatch = NatCamDispatch.Prepare(DispatchMode.Asynchronous, listener);

But it seems that feature is still under development, it currently crashes errors due to resource creation off the render thread. It this supposed to execute the heavy lifting outside the render thread? I would *love* that, as most of the S7's processor's other cores are currently not used intensively.

Since you have a far better understanding of the performance aspect, can you tell me if there's anything I'm doing wrong, or whether what I'm asking for is just plain infeasible? Thanks.

 

Hi @Tinus . I have another report of this, so I'll be looking into it

NatCam bottleneck

This perplexes me because no heavy lifting is done. NatCam is basically drawing a quad--two triangles. I will certainly get to the bottom of this, but it is difficult to imagine that a quad slows down everything. One possible culprit is the evil, but necessary, glFinish() when NatCam submits the draw call to GL to prevent tearing. But still, it's just a quad.

Not so fast. NatCamDispatch's functionality is complete; it is actually a pretty neat utility for threading you can use for non-NatCam purposes. The issue here is Unity: Unity's entire API is not thread-safe (except Debug which is just Debug) and it enforces the invocation thread requirement. Even if GL.IssuePluginEvent could be called on a separate thread, it's just a quad being drawn

I will be profiling NatCam to see where the bottleneck originates from, and to see if I can fix it or find a workaround. I'll update this thread so you don't have to do anything, just post if you have any questions or suggestions.

EDIT: I have reproduced the issue. Now to profile and find the bottleneck.

 

@Lanre Phew, glad to hear you were able to reproduce it, that's a weight off my shoulders. I did indeed recognise NatcamDispatcher as a usable threading system, I imagine you'll be using it later for non-UnityAPI operations then, in addition to the coroutine system for Unity calls? We've started using multithreaded job/task systems like this in our games recently so it's always interesting to hear how other devs build their threading systems around Unity.

Anyway, if it's just drawing a quad something's definitely up. Looking forward to your finds, thanks again!

 

Hello @Lanre

We bought NatCam and it's been working good up to now. One question, how can I get the camera focus point (or field of view), something being mention here http://stackoverflow.com/questions/3261776/determine-angle-of-view-of-smartphone-camera ?

Thanks.

 

Hi @thienhaflash NatCam allows you to get the field of view of each camera:

Code (CSharp):

1. DeviceCamera.HorizontalFOV;
2. DeviceCamera.VerticalFOV;

See the included documentation from the Unity editor under NatCam>Scripting Reference and go to the DeviceCamera class.

Currently, NatCam does not allow you to get the focus point. Instead, it allows you to set the focus point.

 

Yes, you can use NatCamDispatch for offloading workload to a separate thread but also requesting work to be executed on the main (Unity) thread. This is what DispatchMode implies--async for separate thread, sync for Unity thread. I use NatCamDispatch in a game I'm working on. It was modeled after Objective-C's Grand Central Dispatch. So you could create a worker-thread dispatcher like so:

Code (CSharp):

1. NatCamDispatch dispatch = NatCamDispatch.Prepare(DispatchMode.Asynchronous, this, 60); //'this' ref is a MonoBehaviour //60 updates per second if using DispatchContinuous

And queue an operation to be executed:

Code (CSharp):

1. dispatcher.Dispatch(() => Debug.Log("Hello from worker thread"));

Or have an Update loop on a separate thread:

Code (CSharp):

1. dispatcher.DispatchContinuous(ThreadedUpdate);
2.  
3. private void ThreadedUpdate () { //Do stuff..
4.  
5. }

Then release the dispatcher (and thread if async) once you're done:

Code (CSharp):

1. NatCamDispatch.Release(dispatch);

NatCam comes with goodies

 

Any new tutorials out yet?

 

I'll be releasing several videos today. I have recorded all but two videos in the NatCam Snippets series, but am not sure if I can record the remaining two today. I also won't release a gap (like going from episode 5 to episode 7). So sometime this afternoon, about 4 new snippets videos will be added on YouTube. I will also post here.

 

The two videos yet to be completed are the one on face tracking and the Unitygram app (which will encompass everything learnt in all the Snippets episodes). Unfortunately, we just discovered a fault in the Android face detection pipeline. It works, but is greatly inhibited by device/app orientation. We will be rolling out a fix in 1.3f2 which will be reviewed by Unity in the next couple of days.

Regarding NatCam Snippets, we would go ahead and release all the episodes excluding face tracking. As a result, the culmination of all the Snippets which would be Unitygram would exclude face tracking. Once the update is live, we will release a tuto for face tracking. Sorry for the inconvenience. Thank you!

 

Hi Lanre, I would need a little extra help in order to save correctly the photo with frame.EncodeToJPG(80); in Minigram.cs

I have roughly the code below in order to save the image to disk :

Code (CSharp):

1.  
2.     public void CapturePhoto () {
3.         //Divert control if we are checking the captured photo
4.         if (checkIco.gameObject.activeInHierarchy) {
5.            WriteImageToDisk(); // <= the function is here
6.             OnCheckedPhoto();
7.             //Terminate this control chain
8.             return;
9.         }
10.         //Capture photo with our callback
11.         NatCam.CapturePhoto(OnCapturedPhoto);
12.     }
13.  
14. public void WriteImageToDisk()
15. {
16. pictureName "toto.jpg";
17. frame = NatCam.PreviewFrame;
18. Debug.Log("Frame is "+frame.width+"x"+frame.height);
19.  
20. //On transforme la frame en bytes
21. byte[] byt = frame.EncodeToJPG(80);
22. //On detruit la frame en mémoire
23. Destroy(frame);
24.  
25. string cheminCompletImage = Application.persistentDataPath+"/Gallerie/" +pictureName;
26.  
27. //On enregistre l'image par rapport au répertoire en cours (et avec le bon slash)
28. File.WriteAllBytes (cheminCompletImage, byt);
29. }
30.  
31.  

The issue I have it that "NatCam.PreviewFrame" seems to be the current last frame (not the frame taken from the preview).

To sum up :

1. I take a picture of a table (for instance)
2. The preview of the table appears
3. I rotate the smartphone out of the table to see the ground.
4. I push the validate button (wich executes WriteImageToDisk() )
5. The saved image will not contains the table but the ground.

So where is the right place to launch my WriteImageToDisk() function ?

If I try to put it directly inside :

Code (CSharp):

1.     void OnCapturedPhoto (Texture2D photo) {
2.    WriteImageToDisk();
3.     }

I get an error :
"Coroutine couldn't be started "because the game object Preview is inactive"".

Then if I remove :
"Destroy(frame);" from function "WriteImageToDisk()"...
=> it seems to work but... I'm not sure it's the right way to do it ?

Maybe I have to "Destroy(frame);" later at the preview choice ?
Could you tell me the cleanest way to do it ?

Thanks a lot in advance !
Seb

 

Hi Seb!

This is correct behaviour. If you want to save the picture you took, simply send the photo Texture2D that is passed to OnCapturedPhoto to your WriteImageToDisk method. PreviewFrame returns the current frame of the preview, in other words, what the camera is looking at in that exact moment. The photo that is sent to your OnCapturedPhoto method however is the photo that was captured when you clicked the capture button. It sounds like this is the texture you want to save. Also, have you considered using NatCam.SaveToPhotos/Natcam.PhotoSaveMode?

Concerning the error you get saying that the coroutine couldn't be started, it doesn't look to be related to anything you are doing (at least in the code you posted). There is no coroutine being started anywhere from what I can see.

 

Perfect Lanre, I understand better the mechanism. Moreover you fixed the "streching issue" in portrait mode... Awesome

Well I don't use the "PhotoSaveMode.SaveToPhotos" because I don't want the photo to be in the gallery but rather in "Application.persistentDataPath".

Thanks !
Seb

 

I will be posting 9 more episodes of NatCam Snippets within the next two hours. Stay tuned!

 

NatCam Snippets:

The playlist is up. The last two videos are still being uploaded by YouTube.

 

Oh, and here's the NatCam LUT unitypackage. It is a free NatCam extension that allows you to use 1D Look Up Textures to color-grade the camera preview.



NatCam LUT 1.0

EDIT: NatCam LUT will be uploaded to the Asset Store (so that you can get notified of updates).

 

Is there an example of how OpenCV for Unity plugin is supported using natcam?

 

Yes, there is a code sample in the documentation. Go to Classes>NatCam>PreviewMatrix. I will be making a NatCam Snippets episode on this later today.

 

I had to suspend working on the performance bottleneck to produce the NatCam snippets episodes. I'm now working on the bottleneck. I'll keep you updated.

EDIT: Bottleneck has been found. Fixing it will be... interesting. It's an Android API that is absolutely necessary/unavoidable for NatCam to work. Let's see what we can do...

EDIT EDIT: Fixing this bottleneck is quite difficult. I have managed to go from 43FPS to 55FPS average while running a HD preview on an old Galaxy S3. I'll try my best to get to the magic number: 60FPS.

EDIT EDIT EDIT: I just hit 60FPS with multithreaded rendering on. I'll try to squeeze out as much performance without multithreaded rendering (not that anyone actually doesn't use multithreaded rendering, but I might be able to improve the workaround by doing so).

EDIT EDIT EDIT EDIT: I have managed to slice NatCam's rendering pipeline time on the S3 for a HD preview from about 17ms to 4ms. NatCam 1.4b0 now runs at 60FPS smoothly in Unity. Let me know if you would like to profile it yourself. Email me at olokobayusuf@gmail.com.

NOTE that all times reported are with multithreaded rendering off. Now, with multithreaded rendering enabled, NatCam spends less than 0.20ms updating its rendering pipeline on average.

 

Can PreviewMatrix be used for mobile devices? I used it for standalone (Mac OSX) first and it worked fine but when I build the same code for IOS the image got white (NatCam.PreviewMatrix was null).

I got a following error message on xcode.


NullReferenceException: A null value was found where an object instance was required.

at NatCamU.NatCam.get_PreviewMatrix () [0x00000] in <filename unknown>:0


(Filename: currently not available on il2cpp Line: -1)

 

Make sure the preview type is set to Readable.

 

Hey there, I'm very interested in the face detection features that look like they are just about ready. Looking at your example image above though, I see that there is potentially rect data for the detected faces, but not landmark data... I know from briefly looking at the Android API (and I think iOS) that this is an option that can be turned on when you call the face detection- like landmarks = true or something if I remember, is this an option you have implemented or could?

I actually made a face detection thing myself a while back based on LiuLiu's CCV face detection for javascript- basically because including OpenCV in you app seems to mean that it ends up being like 100 meg!

If your app could give me the landmark data I could integrate that really easily with my existing processing code, and you'd have a sale! I'm guessing the code isn't open source? I could probably add it myself for you...

Let me know what the score is here

 

Oh no, just saw that this doesn't work with Vuforia- thats bad for me. Whats the situation there?

 

Android supports detecting landmarks (specifically, the google vision API) but iOS does not. We don't release functionality specific to only one platform, so NatCam doesn't support landmarks.

NatCam's source isn't open source, but its sources are open to NatCam developers. There's a difference between open source and sources being open for modification; I believe you mean the latter. You could add such functionality yourself, but you would have to modify NatCam's native metadata detection pipeline substantially to marshal facial landmark data.

Currently, there isn't support for Vuforia. I did email them a few weeks back and they mentioned that they would have their developers reach out to me. I haven't heard from them since then. Vuforia and NatCam must be used exclusively because only one process is ever allowed access to the camera daemon. NatCam's API is accessible from the native layer, and it has a native pipeline for passing preview data to just about any layer. Vuforia's SDK on the other hand is completely closed. See here. This makes interop impossible except they decide to do something about it.