NatDevice - Media Device API

Hi Lanre,
we have just upgraded to version 1.6 NatCam professional on our newly launched app. But the plugin creates a lot of crashes on Android phones: Galaxy J7, Galaxy A5, Galaxy S6, Galaxy S7 Edge. Any feedback on that?
thanks,
Amy

 

Hi Amy. Can you email me more information about the crashes? Can you send me logs?

 

Hello Lanre,

i don't see the updated version yet, can you please publish it because we really want to further test this.

Best!

 

We are waiting for Unity to approve it. I will let you know when it is on the Asset Store. You will not get a notification because it is the same version number (1.6f1).

 

Thanks Lanre, will it be mentioned in the Release Log?

 

Hi Lanre,

We're having some issues with camera preview on iOS (NatCam Core). On some devices (iPad Air 2) the preview image is zoomed in - on other devices it works fine.

NatCam

System Camera

We've followed the tutorial on Medium, so I think the code is alright:

Code (csharp):

1.  
2.     void Start(){
3.         InitCam();
4.    }
5.  
6.     public void InitCam(){
7.      
8.         NatCam.Play(DeviceCamera.RearCamera);
9.      
10.          NatCam.OnStart += OnStart;            
11.  
12.  
13.     }
14.     void OnStart()
15.     {
16.         NatCam.Camera.ZoomRatio = 1;
17.        
18.         preview.Apply(NatCam.Preview, Orientation.Rotation_0, NatCamU.Core.ScaleMode.Fill);
19.  
20.     }
21.  

Any ideas?
- Piotr

 

No, there won't be any changes to the release changelog on the Asset Store (but the Changelog.md that comes in NatCam has been updated).

The update is now live on the Asset Store.

 

Try using a different scale mode when calling NatCamPreview.Apply.

 

I'll do that. But what is strange the preview differs on different devices of the same type (the zoom level is different on two different iPads 2). Will let you know how it worked out.

 

Hi Lanre,

I've just updated Natcam Pro from the assetStore and I notice that
Natcam.PreviewBuffer (ref Color32[] colors, out int width, out int height) method have disappeared.
I was using it to manage QRcode scanning by sending the Color32 array to ZXing.

Does the method moved ? or is there a equivalent to get the Color32 array with the update ?

 

We removed it because it posed several platform-specific issues. What you can do instead is call PreviewFrame with a Texture2D and then get the pixels with Texture2D.GetPixels32.

 

Ok. I try to call Natcam.PreviewFrame instead but I get this error :

This is the function where I call PreviewFrame method :

Code (CSharp):

1. public Color32[] GetTex(int w, int h)
2. {
3.     Color32[] colors = new Color32[w * h];
4.     Texture2D frame = new Texture2D(w, h);
5.  
6.     NatCam.PreviewFrame(ref frame);
7.  
8.     colors = frame.GetPixels32();
9.     Texture2D.Destroy(frame);
10.    
11.     return colors;
12. }

Lanre, do you have any idea I could have done wrong ?
I'm running this on the editor (Unity 2017.1f1, for iOS).

 

Hi Lanre,

Natcam Core consumer.

Part 1:
I'm using something pretty similar to MiniCam script, some code below, on an iPad. Running the app in Landscape Left orientation only. I was having some weirdness when applying the captured photo to the NatCamPreview panel. I tried a bunch of different scale modes but in the end I've noticed that the image coming in to the callback has letter-boxing in it (black bars top and bottom). It's a landscape image in the middle third of portrait texture.

What it looks like is that the upright/portrait texture is being allocated as 720x1280 but the actual photo is landscape and being scaled to width 720 and written into the middle section pixels of the portrait texture. I've written the "photo" texture directly to file and looked at it and it's 720x1280 with black top, landscape photo, black bottom. Don't have the tablet here, but will post saved photos when I get my hands on the device next.

Part 2:
Also, given that iOS photos are always returned as Upright images with Orientation set to 0 since 1.6f1, how do you recommend dealing with displaying them to the user in their intended orientation?

Some code that probably won't help much

Code (CSharp):

1.  
2.     public override void CapturePhoto()
3.     {
4.         // Divert control if we are checking the captured photo
5.         NatCam.CapturePhoto(OnPhoto);
6.     }
7.     protected override void OnPhoto(Texture2D photo, Orientation orientation)
8.     {
9.         // Cache the photo
10.         this.photo = photo;
11.         this.orientation = orientation;
12.         // Display the photo
13.         Debug.LogFormat("Display Photo [{0}, {1}]", photo.width, photo.height);
14.         panel.Apply(photo, 0, NatCamU.Core.ScaleMode.ScaleWidth);
15.         // stop tracking focus gestures
16.         focuser.StopTracking();
17.         // Disable the controls
18.         controlBar.gameObject.SetActive(false);
19.         // Enable the Confirm
20.         confirmBar.gameObject.SetActive(true);
21.     }

 

Code from the natcam doc says:

Code (CSharp):

1. void SaveScreenshot () {
2.     // Declare the frame texture
3.     Texture2D frame;
4.     // Get the preview frame
5.     NatCam.PreviewFrame(ref frame);
6.     // Encode it to JPEG
7.     byte[] jpg = frame.EncodeToJPG();
8.     // Save it
9.     System.IO.WriteAllBytes("screenshot.jpg", jpg);
10.     // Free the texture and its memory // Note that this is very important to avoid a memory leak
11.     Texture2D.Destroy(frame);
12. }

It creates a handle to the 'frame' texture. Maybe the PreviewFrame function does a null check before allocating a correctly sized texture for you. Try doing the same as the ref doc:

Code (CSharp):

1. public Color32[] GetPreviewFramePixels()
2. {
3.     Texture2D frame; // no need for = new Texture2D(...
4.     NatCam.PreviewFrame(ref frame); // cus 'frame' gets created inside this function and returned via reference
5.     Color32[] colors = frame.GetPixels32(); /* no need for = new Color32[w * h] the 'frame' texture knows its w/h, this causes 'colors' to reference 'frame's pixel array(copy) and 'new Color32[w * h]' will get Garbage Collected 'cus it has no more friends */
6.     Texture2D.Destroy(frame);
7.     return colors;
8. }

 

Just a few issues:

^ This is correct. PreviewFrame will create the texture if it is null (so that it will set the correct dimensions and format). As a result, you should not create the texture yourself.

But also, you are adding memory pressure because PreviewFrame must create a new Texture2D each time you pass in 'null'. Instead, keep your frame texture as an instance variable and pass it in. This way, the first time you pass it in, it is null and NatCam will create the texture. But in subsequent calls, it is not null, and NatCam will update its contents as opposed to recreating a new Texture2D.

You also create a `colors` array, then immediately dereference it when you reassign the variable to frame.GetPixels32(). You are allocating unnecessary memory that GC will have to be cleaned up at some point. And because the buffer is pretty big, your app will experience a noticeable hiccup when GC starts cleaning. So only declare the colors array when you initialize it to `frame.GetPixels32()`.

Let me know if you face any further issues or have a question.

 

There should be no letterboxing in the actual photo that is returned by NatCam. I will check this and get back to you.

Display the texture as-is (and pass in Orientation.Rotation_0 or 0 if using NatCamPreview.Apply). I am looking into the letterboxing as it will probably be the cause of any issues in displaying the texture.

EDIT: I just ran MiniCam in landscape left on an iPhone SE. It worked perfectly. Then I tried landscape right, and I was definitely able to reproduce the issue. Thanks for bringing this to my attention; I have opened an issue.

EDIT: It was a trivial oversight on my part. I'll roll out an update this week. Thanks again for bringing this to my attention!

 

Lanre, any news on this ? I would like to do the migration but it is safe ?

 

I never received the crash logs, and I have not had any similar reports from other developers (and I am unable to reproduce any crash on our Galaxy S devices), so I cannot say that 1.6 is not stable. If I get any reports with enough information, then be rest assured that I will roll out a fix as soon as I can.

 

scamg, Lanre, thanks for your lights

 

Hi Lanre,

I'm getting several linker errors when building on Xcode, I'll paste here only one:
duplicate symbol _OBJC_IVAR_$_NatCamPro.audioOutput in: /PROJECT_PATH/Libraries/NatCam/Core/Plugins/iOS/libNatCamCore.a(NatCamPro.o)
/PROJECT_PATH/Libraries/NatCam/Pro/Plugins/iOS/libNatCamPro.a(NatCamPro.o)

Notes:
- In the project I use Firebase and Cross Platform Native Plugin
- I just updated to newest version
- When I delete NatCam/Pro folder it run perfectly

 

I just noticed this. I have attached the updated NatCamCore.a static library. Place it in NatCam>Core>Plugins>iOS. I will roll out a fix on the Asset Store next week.

 

Thanks Lanre, you're the man!

 

Hi Lanre, I was hoping you could answer a couple of questions about the OpenCV implementation of NatCam.

First question - in your VisionCam example, on every device I've tested so far the matrix in OnMatrix() is upside down, and on some devices flipped horizontally as well. Currently I'm dealing with this by using

Code (CSharp):

1. OpenCVForUnity.Core.rotate(matrix, matrix, 1);

but it seems expensive to be doing this on every frame. Is there a better solution?

My second question - could you please give an example of how to manually set the preview resolution when using OpenCVBehaviour?

 

Good morning Lanre, I have a little question.

I'm trying to test miniCam in FullHD mode :
NatCam.Camera.SetPhotoResolution (ResolutionPreset.FullHD);

When taking a photo with the rear camera, the photo resolution is 1920X1080 and the weight is 1Mb.
But when taking a photo with the front camera the resolution is 4608X3456 and the weight is 8Mb (which is way too heavy).

I would like to limit the size to 1920X1080 with the front camera (if the mobile phone is able to reach this resolution).
Is it possible to do so ?

I have a OnePlus 5

Have a nice day,
Seb

 

This is because OpenCV uses an inverted coordinate system. Y-0 is the top of the image, whereas in OpenGL (and so NatCam), Y-0 is the bottom of the image. There are ways to get around the cost of Core.flip, but it largely depends on what you are doing. The fastest possible thing you can do, if your implementation permits you, is to do all your OpenCV processing upside down (keeping the matrix as it is), flushing it to a texture (which will still be upside down), and then use a material/shader to display the image upright. This is similar to what the NatCamPreview component does.

You can set it from the editor (OpenCVBehaviour derives from NatCamBehaviour), but if you want to set it from code, then you must override the Start method and manually set the preview resolution there (then call base.Start so that NatCamBehaviour will continue with the NatCam setup)

Code (CSharp):

1. public override void Start () {
2.     previewResolution = ResolutionPreset....;
3.     base.Start();
4. }

 

Hi Seb. When you call NatCam.Camera.SetPhotoResolution, have you made sure that NatCam.Camera is set to DeviceCamera.FrontCamera? You can directly set the photo resolution of the camera using DeviceCamera.FrontCamera.SetPhotoResolution.

 

Thank you very much Lanre for your helpful response. My project involves using object detection on a trained dataset, which would mean I’d need to retrain it with every image upside down - that could be an option!

Otherwise is using flip/rotate what you’d recommend? My frame rate is good but on a phone the frame sometimes ‘flickers’ from rotated to non rotated, as if some frames are being skipped. Do you know if there’s a way to avoid this?

Thanks again!

Edit: Sorry I’m logged in to my personal account - that’s a reply to the question (from Colenso) above!

 

I don't think this is viable, but you are better suited to make the decision knowing the extent of change that would need to be made.

If you want absolute performance, you have a few options (in descending order):
1. GPU acceleration. You can blit the current preview frame to a RenderTexture with a material/shader that flips the preview. In native code, you can then bind the RenderTexture (get its native handle using GetNativeTexturePtr) to an FBO and perform a readback using a Pixel Buffer Objects (PBO), or two. This is what NatCam's FastRead module (now integrated into the NatCam Render Pipeline) does. If you're running on iOS, then you can use CoreVideo Texture Caches instead.
2. SIMD. If you're on iOS, it's pretty simple: use Accelerate.framework's vImage geometry API's. If you're on Android, then it's not so easy. You would have to write ARM NEON that flips the image, processing on a batch of pixels 'per instruction'. I believe the NDK automatically transpiles ARM NEON to x86 SSE if you are building for the x86 ABI. If not, then you would also have to write x86 SSE.
3. memcpy. It is pretty safe to assume that memcpy is optimized for SIMD (because it is), so you can copy rows to a target buffer, in reverse order. The result will be an image that is vertically flipped. If you want to flip horizontally, then this will not help (you would have to reverse each individual row, and there's no standard library API that I know of that does this). Using the memcpy approach is linear with the number of rows in the image (the image height).

 

Thank you Lanre

 

Hi Larne, I have the latest pro version. I am trying to run ReplayCam example on Android 8.0 (Nexus 5x device), and the preview is always a pink screen. Any idea what might be happening? [Mini Cam example runs fine]

Also, when I run the same (ReplayCam) on PC, the preview is fine but I get "Cannot record video because implementation does not support recording" error.

Can you please take look at the attached log? Really appreciate you help

 

I'm not sure what the problem could be, especially since MiniCam works. Can you attach the logs from app start?

The NatCam Legacy backend, which is currently what runs on all platforms except iOS and Android, does not support recording.

 

Thanks for your reply Larne,

Sure, np, just need it to work for Android for now

 

Apparently you are running from the front camera. Does the MiniCam example work with the front camera?

 

Yes, with MiniCam I can both switch front/back cameras and take photo. Any other suggestions? Should it work on Android 8.0 without issues? Should I try to find some other device that for sure works? I am trying to run the ReplayCam version that comes with the package, is there some other version that I can try? Thanks

 

I don't believe this has anything to do with Android 8.0, since you say that MiniCam works. This is perplexing because up until you start recording, ReplayCam should behave the exact same as MiniCam. The logs also show no apparent errors.

I took a look at the screenshot and noticed that there is no UI at all. The button that comes with ReplayCam is not present. This might not be a NatCam issue; it might be an issue with Unity or your current Unity version. Let's continue this conversation over email, I'm much more responsive over email.

 

I have a weird thing going on with NatCam. The camera work perfectly, but I also need to show a 3D model on screen, but when the camera is active, the model will flicker. Can someone help me?

 

Have you updated to NatCam 1.6? We fixed this issue in 1.6.

 

Oh it was fixed? I saw that in 1.6, extended and so the save picture function was gone, so I'm still using 1.5

 

Yes it was, and we did deprecate NatCam Extended.

 

Is there any way I can solve the flickering problem without updating to 1.6?

 

Hi Lanre,

thanks for the NatCam Plugin. I bought it today, because I needed higher resolutions.

I'am testing my app on a iPad 2 (1024 × 768). If I set the resolution to highest, it gets 1280x720. But the Screen is 4:3 and if I have a 16:9 camera resolution the camera is stretched.

Code (CSharp):

1. NatCam.Camera.SetPreviewResolution(ResolutionPreset.Highest);
2.  
3. ...
4.  
5. camPreview.Apply(NatCam.Preview, Orientation.Rotation_0, NatCamU.Core.UI.ScaleMode.None);

I need a flexible resolution setting for every device.

How can I get the exact possible resolution withe the same screen ratio? Or how can I know, which resolution the camera on the current system supports?

By the way, is there no unity remote support for this camera system? I have to build every time.

 

Enable multithreaded rendering in Player Settings. That should fix it.

 

Different devices support different resolutions, so there is no way to enforce a single resolution. You should set your scale mode to something other than None so that it scales properly. With NatCam, there is no way to get supported resolutions on a device. You simply set a resolution, and if the device supports it, then NatCam will set it. If not, NatCam will set the closest supported resolution.

No there isn't. Unity Remote is proprietary by Unity Technologies.

 

Thank you that solver the problem.

 

Hello there - I got NatCam Pro today and I'm having issues getting it to build for iOS using Unity 2017.1.1f1 on a mac. If I make a new blank project, import only NatCam as an asset and try to build one of its example scenes, the build fails during the last stage of AOT cross compile. The error message given is rather unhelpful:

* Assertion at ..\mono\mini\aot-compiler.c:2228, condition `method->flags & METHOD_ATTRIBUTE_STATIC' not met

Has anyone any idea where I'm going wrong? Thanks

Edit: This is also happening when I try to build for iOS using my windows PC (same error). I've also tried using IL2CPP for the backend instead of mono with no joy.

Edit: Found the solution - for Xcode/iOS, you can only build using IL2CPP with Architecture set to ARM64. ARMv7 will cause the issue above

 

Thanks. I solved it with the AspectRatioFitter component, where I set the ratio of the camera resolution. The UI RawImage scales by himself according to canvas...

 

Hi there,

the official NatCam Core description in the asset store says support for Windows is scheduled for the next major release. As we are looking for a webcam solution for windows: Is there a chance it will arrive until March 2018?

Our goal is to show a live webcam stream in a high resolution (e. g. 1920x1080) at 30 fps on a vanilla Windows 10 tablet (newest atom processor). The WebcamTexture class of unity currently gives us only 5 fps and if there is a chance NatCam could do more here we would be happy to purchase it!

 

Hi Lanre,

I'm having an issue with the NatCam and Android Oreo 8.0 (Nexus 6P), I tested the same scene (MiniCam) with another device 7.0 and worked well.

I attached a part of the log. Since all initialization messages seems to be the same on both devices.

The problem is that a Java exception occurs and the photo isn't captured.

 

Hi Lanre, sorry for the late response. I was focussing on other projects which had a higher priority. I have updated the NatCam plugin to the latest version 1.6f1 and tested the MiniCam scene.

These are the results on a Xiaomi Mi5s and s SamsungS5 Neo.

Xiaomi Mi5s live feed:

Xiaomi Mi5s captured picture:


Samsung S5 Neo live feed:

Sansung S5 Neo captured picture:


Is it possible this occurs due to the camera sensor uses its full size to capture a picture, and uses a cropped version of it to give a live feed?

If you need some more information, feel free to ask and I'll answer as fast as I can, so we can both solve this problem

Kind regards

 

I can't conclusively say, but chances are that we will have a NatCam 2.0 beta by then.