NatDevice - Media Device API

Hi @NKidd can you email me the logs for your Android device using OpenCV? And also test that it is because of NatCam by disabling the OpenCV face tracker code and simply running a readable preview. If the readable preview runs fine, then you are bottlenecked by the OpenCV calculations.

 

Hi @Alvaro_OneTangoStudio. Adding a rendering pipeline to NatCam iOS is what is taking so long. However, I am suspending that and moving to Android. Perhaps later today, the beta will be released.

 

That would be great, actually!

 

@Tinus Totally tangent, but Volo Airsport looks really cool.

 

Hi guys. NatCam v1.2 is very pressing for us and for you but some issues will require a lot more time than we thought. On Android, camera switching is very problematic, as some resolutions reported by the camera itself don't work. We are facing several other issues and have decided to delay the new version to fix these.

It's much better to take time to develop something stable, than rush and push out crap.

Sorry for the huge inconvenience. But know that I'm working round the clock to resolve these issues. We have decided to rebuild the entire codebase. v1.2b3 will still be released this evening, but with the camera switching bugs still present.

 

You guys are doing great work, really. Started a freelance gig that requires real-time webcam view in Gear VR, scared myself silly when I say the performance of Unity's built-in webcam texture, then found you guys had pretty much just released this plugin to address this very issue. Saved my bacon, huge boon for this project.

The code is easy to use, looks clean, and I hugely respect that you include the native source code as well in case anyone needs it.

This will have been asked earlier, but how do I try the beta releases? Quite excited to try early fixes for the OpenCV matrix support to see if it works on our target hardware, even if I can't use it in production yet.

Oh, and thanks for the Volo compliment!

 

Hi @Tinus. Email me at olokobayusuf@gmail.com.

We are rebuilding NatCam from ground up to be cleaner, more performant, and more economical with memory. We are moving the API to a more OOP scheme. We are also adding better support for the device cameras, instead of limiting devs to DeviceCamera.FrontCamera and DeviceCamera.RearCamera.

 

@Lanre Ok, you are building it again from scratch, but do you have a schedule? I think is important for everyone to give us one so we can managed our project based on it. Otherwise this turns into a bad choice in terms of software investment.

 

@Alvaro_OneTangoStudio Thursday is the date slated for the next beta. I will post an explanation of the new API, the API map, and upload documentation either tomorrow or Monday.

 

Hi, I purchased the asset yesterday and am not happy with the result. I created a new project with only the asset imported. Then I opened the example scene. What I want to do is turn on the torch during preview.
To do so, in the script Unitygram, I added:
NatCam.TorchMode=Switch.On;
I tried putting it before and after NatCam.Play(); Result was the same in every case: the torch did not lit up.
Furthermore, the image is very slow, sluggish, not at all as fast as the realtime webcamTexture from Unity.
I tried this on two iPhones, a 5s and a 6s Plus. The first one (5s) was less sluggish than the iPhone 6s plus, which is strange, because the s6 plus is a faster device.
Since I am not satified at all, I would like to receive a refund.

 

Hi @LucasUylings . I apologize for the inconvenience. The sluggish preview on iOS has been fixed; I will be releasing a beta today with a lot of fixes. If you would like to get the new beta, or go ahead with the refund, please email me at olokobayusuf@gmail.com.

I wonder why the torch doesn't light up. Can you send me the logs?

 

NatCam v1.2b4a is complete. It is not production ready, but features the new API and some new features. Email me at olokobayusuf@gmail.com if you would like to test it out.

 

Lanre, Alvaro here CEO from One Tango Studio. My team have been trying to reach you by email to get the new api and had no response. We urgently need the working api. Right now we have the whole development paused because of it. I need faster and better support for a 50usd api. Looking forward to a solution.

 

Hi, short question to NatCam: What is best way to display the real framerate of the camera preview? Thanx!

 

Hi @Alvaro_OneTangoStudio . I sent you an email and am yet to receive a response.

@cbrack You might look into calculating time deltas in the NatCam OnPreviewUpdate event.

 

Thank you @Lanre for your plugin it was excactly what we needed for our app. I am though having a few problems and hope you can help me. My app is all about taking photos so I am starting the camera all the time. I am though not sure what is the best way to use NatCam. Below are a few ways:

1) Should I use the NatCam.Initialize() every time I start the camera and NatCam.Stop() at the very end. This works fine on iOS but NatCam.Stop() crashes my app on Android (tried on 2 different samsung devices).

2) Should I just call NatCam.Initialize() in GameObject.Start() just once and then call NatCam.Pause() at the very end. This results in a scrambled "second photo", that is the first time I use the camera it is ok but the next time and after that my photo will be scrambled (see attached photo). This is the same behavior on both iOS and Android.

3) Am I missing somthing. Should I be clearing the NatCam.Preview or maybe using NatCam.PreviewFrame

I have been experimenting with differenent ways for two days with little luck.
Please advice as soon as you can.

 

@gudjong I replied your email

 

I am getting the same issue as gudjong. I'm trying to implement a 'flip camera' button and I tried just recalling NatCam.SetActiveCamera but the image that was taken was scrambled and the image preview was of the previous camera as well as being flipped upside down.

Then I just tried recalling everything that was in the Unitygram script after stopping the NatCam and just ended up with crashing.

I should mention I'm running on Samsung Pro Tab 4 Android Tablet on JellyBean

 

Nevermind I figured it out, I needed to use Pause instead out outright stopping it and then not recall the initialize.

Ok new question, when I flip the camera the preview image that was being shown on the former camera flips upside down sometimes before switching to the camera I am flipping to. Thoughts?

 

@voporak5 I believe this issue has been fixed in the v1.2 beta. Can you email me at olokobayusuf@gmail.com to get sent a copy of the beta? Concerning the camera preview flipping, what platform are you running on? Please email me so that we may investigate further.

 

Sounds good I emailed you

 

@voporak5 replied

 

Hello guys.

I would like to request beta testers for NatCam v1.2.

We are preparing for release (or submission to be more precise) and we would like to undergo extensive testing. Please email me at olokobayusuf@gmail.com. The new API will be more beneficial to more specialized usages of NatCam, and we have fixed a significant majority of bugs that eluded us in NatCam v1.0 and NatCam v1.1.

Here is a list of features and fixes we have so far:
+ Completely rebuilt API to have a more object-oriented programming pattern, and to be much cleaner.
+ Immediate native-to-managed callbacks, deprecated UnitySendMessage for MonoPInvokeCallback with function pointers.
+ Added NativePreviewCallback and NatCamNativeInterface.EnableComponentUpdate() to get access to the raw luma (Y) and chroma (UV) buffers from the camera.
+ Added the NatCam rendering pipeline to NatCam iOS.
+ Added NatCamNativeInterface.DisableRenderingPipeline() to disable NatCam's native rendering pipeline.
+ Camera preview is now accessible through NatCam.Preview and returns a Texture (not Texture2D).
+ Camera switching on iOS and Android is now stable and responsive.
+ Added SetFramerate() in the DeviceCamera class and FrameratePreset enum.
+ Added HorizontalFOV, VerticalFOV, IsTorchSupported, IsFlashSupported, and IsZoomSupported in the DeviceCamera class.
+ Added DeviceCamera.Cameras list of cameras on the device, not just DeviceCamera.Front/RearCamera.
+ Added ability to specify NatCam interface (native or fallback) and NatCamInterface enum.
+ Added verboseMode switch to NatCam.Initialize() for debugging.
+ Added NatCam.RequestBarcodeDetection(), BarcodeRequest struct.
+ Deprecated DeviceCamera.SupportedResolutions and Resolution struct. Use ResolutionPreset instead.
+ Deprecated CapturePhoto overloads for NatCam.CapturePhoto(params PhotoCaptureCallback[] callbacks).
+ Removed '#if CALLIGRAPHY_DOC_GEN_MODE' conditional directives making code much cleaner.
+ Fix camera preview lagging on iOS.
+ OpenCV PreviewMatrix now updates from the native pixel buffer. This gives some memory savings and performance increase.
+ Captured photo is now the highest resolution that the camera supports by default.
+ Captured photo is now RGB24 format. This means you can use Get/SetPixels(s), EncodeToJPG/PNG, and Apply.
+ Preview is now RGBA32 format on both iOS and Android.
+ Fixed "Error Creating Optimization Context" when using Readable preview on Galaxy S6 and Galaxy Tab.
+ Fixed rare scan line jitter when NatCam corrects padding with Readable preview on some Android devices.
+ Added ALLOCATE_NEW_PHOTO_TEXTURES macro for optimizing memory usage.
+ Added ALLOCATE_NEW_FRAME_TEXTURES macro for optimizing memory usage.
+ Fixed Android crash on Stop().
+ Fixed error when LoadLevel is called--something along the lines of "SendMessage: NatCamHelper not found".
+ Added editor-serialized variables for Unitygram example. Now, you can set Unitygram's variables from the editor instead of code.
+ Automatically link required frameworks on iOS.
+ Deprecated OPTIMIZATION_USE_NATIVE_BUFFER macro, and as a result, direct support for Unity 5.2 has been stopped.
+ NatCamPreviewUIPanelScaler and NatCamPreviewUIPanelZoomer have been renamed to NatCamPreviewScaler and NatCamPreviewZoomer respectively.
+ Deprecated NatCam.Stop() for NatCam.Release().
+ Fixed NatCam.Release() (formerly 'Stop') being ignored when NatCam.Pause() is called immediately before.
+ Fixed NatCamPreviewScaler not correctly scaling HD and FullHD preview on iOS.
+ Fixed rotation and skewing when using CapturedPhoto with OpenCV.
+ Memory leak when calling Release() (formerly named 'Stop') and Initialize() several times on iOS has been fixed.
+ Completely rebuilt the documentation.
+ Added Easter Eggs on iOS.

 

Is there an example on how to implement MR code reading? T need to demonstrate to a client the ability to read a bar code and save the data to a string for use in the app. The example file that comes with the asset is pretty bare on demonstrating functions such as this.

 

Hi @Vern_Shurtz Can you email me at olokobayusuf@gmail.com? I'd like to send you the v1.2 beta. The new beta features a new API for barcode detection. And I will be implementing a camera app example for the package with barcode detection.

 

Email sent. Thank you for your awesome support!!

 

Hi. I am sure I am missing something very simple, but no matter what I try I can't get the barcode event to fire....

I have tried giant 8.5 x 11 inch printouts of perfectly clear QR codes, EAN codes, etc and nothing seems to register

The docs says to make sure MRDetection is on but I see no such variable to set

I did use NatCam.DetectMachineReadableCode as per the documentation

void Start () {
//Initialize NatCam
NatCam.Initialize(PreviewType.Readable);
//Settherearcamera active
NatCam.SetActiveCamera(DeviceCamera.RearCamera);
//SettheresolutiontoHD, 720p
NatCam.SetResolution(ResolutionPreset.HD);
//Startthecamera preview
NatCam.Play();
//Setthefocusmodeto autofocus
NatCam.FocusMode = FocusMode.HybridFocus;
//Turnonautomatictouch detection
NatCam.DetectTouchesForFocus = Switch.On;
//SettheRawImage'stexturetothecamera preview
previewUIPanelRawImage.texture = NatCam.Preview;



NatCam.DetectMachineReadableCode (OnDetectedISBN,MachineReadableCodeFormat.Any);
}

//Updateiscalledonceper frame
void Update () {

}

void OnDetectedISBN (List<MachineReadableCode> codes) {
//Logwhatwe found

Debug.Log("FoundISBNcodes.Thefirstcodeis" + codes[0].ToString());
}

 

Hi @epit . In NatCam.Initialize, you must specify that you want MR detection. Like so:

Code (CSharp):

1. NatCam.Initialize(PrepareForMachineReadableCodeDetection:true);

If you would like to try the new Unitygram camera app sample with barcode detection, email me at olokobayusuf@gmail.com.

 

Hi. I wrote a script to detect qr code but it does not seem to work.

Code (CSharp):

1.  
2.     void Start()
3.     {
4.    
5.         //Initialize NatCam
6.         NatCam.Initialize(PreviewType.Readable,PhotoSaveMode.DoNotSave,true);
7.         //Set the rear camera active
8.         NatCam.SetActiveCamera(DeviceCamera.RearCamera);
9.         //Set the resolution to HD, 720p
10.         NatCam.SetResolution(ResolutionPreset.HD);
11.         //Start the camera preview
12.         NatCam.Play();
13.  
14.         NatCam.DetectMachineReadableCode(NatCam_OnFoundMRCodeEvent, MachineReadableCodeFormat.QR);
15.         //Set the focus mode to autofocus
16.         NatCam.FocusMode = FocusMode.HybridFocus;
17.         //Turn on automatic touch detection
18.         NatCam.DetectTouchesForFocus = Switch.On;
19.         //Set the RawImage's texture to the camera preview
20.  
21.  
22.         previewUIPanelRawImage.texture = NatCam.Preview;
23.     }
24.  
25.  
26.     void NatCam_OnFoundMRCodeEvent(List<MachineReadableCode> result)
27.     {
28.         Debug.Log("Found "+result[0].data);
29.     }

 

Hi @sunnysun Your code looks good, although I must note that if all you need to do is barcode detection, you don't need a readable preview. Are you running on iOS and Android? And what device is this on? Please email me at olokobayusuf@gmail.com.

 

v1.2 is getting closer to release.

 

Hi Lanre, I have the Beta 1.2 running and everything is working well for detecting barcodes and sending the captured information to my logic. By the way I am using Playmaker for the logic since my C# is a bit lacking. The last thing I need to be able to do is have the user turn the light(Torch) on in order to read the barcode easier. I would like to do something simple such as call a TorchOn or TorchOff method such as below. Would something like this work?

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.UI;
3. using System.Collections.Generic;
4. using NatCamU;
5.  
6. public class CamTorch : MonoBehaviour {
7.  
8.  
9.     public void TorchOn () {
10.  
11.         NatCam.ActiveCamera.TorchMode = true;
12.  
13.     }
14.  
15.     public void TorchOff () {
16.  
17.         NatCam.ActiveCamera.TorchMode = false;
18.  
19.         }
20.  
21. }

 

@Vern_Shurtz you're close:

Code (CSharp):

1. NatCam.ActiveCamera.TorchMode = Switch.On;

 

Ah... Silly me.

 

Hi,

Tried to build the NatCam Demo (clean new project, windows version of unity, ios export) on iOS with XCode 6.2 and got a linker error:

ld: warning: directory not found for option '-L/Users/XXX/Desktop/NatCam_iOS/LibrariesNatCam/Plugins/iOS'
ld: library not found for -lNatCam
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Did the same thing with the mac version of unity and got the following linker errors:

ld: warning: arm64 function not 4-byte aligned: ltmp0 from /Users/XXX/Documents/Unity/NatCamDemo/NatCamIOS/Libraries/libiPhone-lib.a(PLCrashAsyncThread_current-5565F35D16E4818B.o)
ld: warning: arm64 function not 4-byte aligned: _plcrash_async_thread_state_current from /Users/XXX/Documents/Unity/NatCamDemo/NatCamIOS/Libraries/libiPhone-lib.a(PLCrashAsyncThread_current-5565F35D16E4818B.o)
Undefined symbols for architecture arm64:
"_ALAssetsGroupPropertyName", referenced from:
___30-[NatCam SaveImageToAppAlbum:]_block_invoke267 in libNatCam.a(NatCam.o)
"_vImageHorizontalReflect_ARGB8888", referenced from:
-[NatCam RotateBuffer:Width:Height:Size:] in libNatCam.a(NatCam.o)
"_OBJC_CLASS_$_ALAssetsLibrary", referenced from:
objc-class-ref in libNatCam.a(NatCam.o)
"_vImageRotate90_ARGB8888", referenced from:
-[NatCam RotateBuffer:Width:Height:Size:] in libNatCam.a(NatCam.o)
"_AVCaptureSessionPreset3840x2160", referenced from:
-[NatCam RetrieveCameraInfo] in libNatCam.a(NatCam.o)
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

armv7 does not work at all.


What I'm doing wrong here?

Thanx!

 

Hi @cbrack

Is it possible to build for iOS on Windows? I didn't know it changed. Anyway, what version of NatCam did you build with? v1.1 or v1.2b4? Try to build to a separate folder, or the same folder using the Replace option instead of Append.

On Mac however, you must add Accelerate.framework and AssetsLibrary.framework to your project. This has been fixed in v1.2.

EDIT: From the linker errors, you must be using v1.1.

EDIT: For the first error, your library search path seems to be wrong.

First, it seems like it should be 'Users/XXX/Desktop/Project_Name/Assets.
Also, LibrariesNatCam should probably be Assets/NatCam.

 

Thank you very much for your answer! I made it to compile and link on the mac with your hints. Unfortunately the application crashes (running on an iPad mini) in NatCam RetrieveCameraInfo.

I'm using NatCam v1.1.

Do you have any tip for me? Shall I use the v1.2(b)? Thanx!

 

Hi @cbrack this is a bug in v1.1 when running on iOS 8.0 and under. I have emailed you NatCam v1.2b4c. I will email you NatCam v1.1 with the fix for iOS later today.

 

Hi guys. v1.2 has been submitted for review. Now we wait.

 

NatCam v1.2 is out!

 

Hello guys, does anyone know how to change the active camera at run-time because my app hangs when I try to do it. BTW I do it changing NatCam.SetActiveCamera(DeviceCamera.RearCamera) to NatCam.SetActiveCamera(DeviceCamera.FrontCamera);

 

Hi @BenYari your code is correct for v1.1:

Code (CSharp):

1. NatCam.SetActiveCamera(DeviceCamera.FrontCamera);

And for v1.2:

Code (CSharp):

1. NatCam.ActiveCamera = DeviceCamera.FrontCamera;

The crash when switching cameras is one of the important bugs we addressed.

 

So I just noticed that I forgot to change the barcode detection line in Unitygram from:

Code (CSharp):

1. //Create a barcode detection request
2. BarcodeRequest request = new BarcodeRequest(OnDetectBarcodes, BarcodeFormat.EAN_13 | BarcodeFormat.QR, !continuousDetection); //DEBUG //detectionFormat

to:

Code (CSharp):

1. //Create a barcode detection request
2. BarcodeRequest request = new BarcodeRequest(OnDetectBarcodes, detectionFormat, !continuousDetection);

So don't be surprised if the Unitygram example can only detect QR and EAN_13 codes. Change it!

 

Lanre - this plugin looks very promising. I just bought it. I'm trying to use it with the also excellent OpenCV asset from EnoxSoftware but I'm having a rough go (at least with your new release, I haven't tried the old). I think one of the problems is that the documentation is out of date. Can you post the "Hello World" of NatCam + OpenCV somewhere for me? I'm pretty confused how to use the previewMatrix, and even how to properly initialize the scene now (the PDF included in the current release still has 1.1 syntax that no longer works). Thanks for your help.

 

Hi @iPhoneCoder . Thank you! I do apologize for the documentation on OpenCV being scanty; we haven't had much time to delve into it. To access NatCam.PreviewMatrix, you must uncomment:

Code (CSharp):

1. //#define OPENCV_DEVELOPER_MODE

at the top of NatCam.cs, NatCamNativeInterface.cs, and NatCamFallbackInterface.cs. Once you do this, you will be able to access the preview matrix by calling:

Code (CSharp):

1. NatCam.PreviewMatrix

To initialize a NatCam session, it is advised to use the UnitygramBase class. You can either use as-is, or subclass it to extend its functionality. In this case, we must subclass it to perform our OpenCV functions. Here is my implementation:

Code (CSharp):

1. using UnityEngine;
2. using UnityEngine.UI;
3. using NatCamU;
4. using OpenCVForUnity;
5.  
6. public class OpenCVDemo : UnitygramBase { //UnitygramBase inherits from MonoBehaviour, so it is a component in the Editor
7.  
8. public RawImage rawImage;
9. Texture2D texture;
10. Color32[] colors;
11.  
12. public override void Start () {
13.     //Register our OpenCV init code for when the preview starts
14.     //Note that when this is called , PreviewMatrix is null
15.     NatCam.OnPreviewStart += InitializeOpenCV;
16.     //Register for the frame updates
17.     NatCam.OnPreviewUpdate += DrawOpenCV;
18.     //Start UnitygramBase
19.     base.Start();
20. }
21.  
22. public void InitializeOpenCV () {
23.     //Initialize our color buffer
24.     colors = new Color32[NatCam.Preview.width *  NatCam.Preview.height];
25.     //Initialize our destination texture
26.     texture = new Texture2D (NatCam.Preview.width, NatCam.Preview.height, TextureFormat.RGBA32, false);
27.     //Set our UI Panel's RawImage texture to our destination texture
28.     rawImage.texture = texture;
29. }
30.  
31. public void DrawOpenCV () {
32.     //Draw a red line on our image
33.     Imgproc.line (NatCam.PreviewMatrix, new Point (0, 0), new Point (NatCam.PreviewMatrix.cols (), NatCam.PreviewMatrix.rows ()), new Scalar (255, 0, 0, 255), 4);
34.     //Update our destination texture with the line drawn above
35.     Utils.matToTexture2D (NatCam.PreviewMatrix, texture, colors);
36. }
37. }

EDIT: Fixed wrong parameters in Texture2D constructor. Also fixed missing namespace includes.
EDIT: Added missing OpnCVForUnity namespace include.

 

Really appreciate the ultra fast support. Are you using Unity 5.3.x with this code? In my current environment, Unity for Mac, 5.3.4f1, I cannot compile that code as a C# file. First it choked on RawImage so I added:
using UnityEngine;
using UnityEngine.UI;

But then I hit: OpenCVDemo.cs(25,127): error CS1502: The best overloaded method match for `UnityEngine.Texture2D.Texture2D(int, int, UnityEngine.TextureFormat, bool)' has some invalid arguments.

I'm a little out of practice in Unity (been working purely in native Obj-C this year), but I usually don't get quite this lost. Thanks again for the guidance.

 

Oops. Fixed it. The Texture2D ctor is supposed to be Preview.width/height instead of PreviewMatrix.width/height. I try my best to provide instant support. You can reach me at just about anytime.

 

Great, that sorted the Texture2D complaint. I also had to add using OpenCVForUnity; to the namespace includes. It compiles now. I'll let you know if I'm still lost.

 

Great!

 

Hi @Lanre,
I'm developing an image editing-like app. The Idea is very simple: i want to take image of an object, write a short description on it before saving it to the device. Right now i'm using vuforia camera and screencapture function to do this, but unfortunately the resolution of the image saved is poor, compared to image taken directly with device camera.

I don't really know whther the problem caused by the vuforia camera or the screenshot quality.
Could your plugin be the solution for my app? do you provide trial version maybe?