NatDevice - Media Device API

Considering using NatCam - which looks like a very nice plugin - and noticed all the links in the asset store go to 502s on google drive. Can you point me to any online documentation that exists? Thanks.

 

I'm struggling to get the right approach for instantiating and destroying NatCam. I have a prefab with a camera manager behavior on it, subclasses off of Unitygram. When I want to use the camera, I instantiate the prefab, call initialize on NatCam and off I go. Later, I am done with the camera, and I call release and destroy the prefab. If I then try to create the prefab again to use the camera, I crash during initialization. Is this the right approach to use with instantiating and releasing?

Crash results:
2016-09-27 15:45:07.642245 bookflixhd[2506:1046695] NatCam Native: Registered Callbacks

2016-09-27 15:45:07.642355 bookflixhd[2506:1046695] NatCam: Inspected Device: Found 2 cameras

/BuildRoot/Library/Caches/com.apple.xbs/Sources/Metal/Metal-85.83/ToolsLayers/Debug/MTLDebugRenderCommandEncoder.mm:2682: failed assertion `incorrect type of texture (MTLTextureTypeCube) bound at texture binding at index 0 (expect MTLTextureType2D) for _MainTex[0].'

Message from debugger: Terminated due to signal 6

 

Hi Chris. This error can only happen if the WebCamTexture that NatCam maintains is null:

Code (CSharp):

1. if (!supportedDevice) {
2.    Ext.Warn("Current device has no cameras");
3.    return;
4. }
5. Preview.Pause();

And Preview is only ever null in two scenarios: NatCam.Play has not been called; and NatCam.Release was called.
Check your state management and consider sending me your code for analysis.

 

Google took down their webpage hosting feature with Google Drive. Here is the NatCam 1.3f2 documentation zip file.

 

Not at all. NatCam is not bound to the state of the active scene (or game objects like prefabs). You should not call NatCam.Initialize/Release unless you need to. Starting it up and tearing it down is expensive. Check NatCam.isInitialized before calling NatCam.Initialize() and do not call NatCam.Release unless you know you will never need to use the camera for the rest of the applications runtime (which in most cases is never). One exception to this rule is when you need another plugin to use the cameras, like Vuforia. In this case, calling NatCam.Release is your only option.

The crash you posted regarding the MTLTexture does not seem related to NatCam. NatCam does not use Metal for anything now. Check that you don't have another thing causing this crash.

 

Hi guys. Here's a little update:

- NatCam 1.4 development is undergoing some delays because we have been facing some issues on the betas. We expect the most recent beta (beta 2) to be ready by Wednesday.
- We are dropping the NatCam.SavePhoto(Texture2D, SaveMode) (previously called NatCam.SaveToPhotos(Texture2D, PhotoSaveMode)) API. We are doing this to ensure uniformity as we progress to the NatCam 2.x release cycle (other platforms won't have constructs like Gallery and Album).
- We are really excited for the 2.x release cycle. Currently, the only reliable webcam solution on desktop OSes is AVPro which is currently over twice the price of NatCam.
- We are planning a website for NatCam (NatCam.io). Do you have any suggestions for what should be on it? The documentation is a must. We also want to see if we can add a forum. If you are a web developer or know a web developer, please email me at olokobayusuf@gmail.com. We would prefer to contract this out.

 

the metal crash does seem to be occuring in natcam - it happens on line 507 of NatCamRendering.mm

[encoder setRenderPipelineState:mtlPipelineState];

Changing the pixel format in line 144 gets the crash to stop but causes capture to not work.

edit:did a test project with a newly imported and updated natcam - no dice on reproducing the error.

edit2: Did further testing. I'm only experiencing this in our app on the iPhone 7 in iOS 10. I don't have any other devices that I've updated to iOS 10 so I can't say if this is an iOS 10 issue or an iPhone 7 issue. I haven't bothered adding the right keys to new test project to run at all in iOS 10 (crashes on trying to access camera without right plist keys) so I don't know whether a clean project with the plugin imported will work or not.

 

The NatCamRendering error you stumbled upon was from NatCam 1.2. We have deprecated the rendering pipeline on iOS since NatCam 1.3. The crash on iOS 10 is because of the NSCameraUsageDescription key. We have fixed this in NatCam 1.4b1, but you can add the key manually in Xcode.

 

Hi Lanre,

Stubling into some problems with natcam 1.3f4 on Android:
- Camera preview seems like stretched border pixels. Once I take a picture it turns out file... just the live cam preview that is strangely stretched. (Assetstore version does not have this. Only the update "to fix crash on exit bug" you sent me.)

Also you mentioned that in the 1.4 release, the camera will aways take pictures in landscape left mode, even if app is locked to portrait orientation. I could really use that now already

Any solutions for this?

Kind regards,

Wim Wouters

 

I replied to your email.

 

Any news on vuforia?

 

No

Not at all. They never got back to me.

 

@johnfrog I saw an email about a PM you sent me. Please email me instead. Regarding the out of memory issue, change the NatCam.ActiveCamera's photo resolution to something like ResolutionPreset.FullHD using DeviceCamera.SetPhotoResolution. Regarding Android N, 1.3 (and version below) are incompatible. We have fixed this in 1.4.

 

Yet another minor update regarding NatCam 1.4, we will be removing the NatCam.PhotoSaveMode API. We are stripping the API to core, exclusive camera functionality in order to prepare for the NatCam 2.x release cycle. This particular change means that NatCam will no more allow you to save your captured photos to the device gallery or albums. There are several plugins on the Asset Store that would allow you to do this.

 

That is disappointing, this is one of the reasons why I bought NatCam. I've not got it into my live project yet and already about to loose some functionality I paid for. I could have used another plugin that just does camera view that is cheaper and now I feel I have wasted all this time getting it working.

I think a lot of people here might agree too.

What is the situation on asset refunds if features advertised are removed?

 

Based on what you are using NatCam for, you can keep using NatCam 1.3. Based on our conversations, you don't seem to need any of the new features in NatCam 1.4.

Not at all. We've been getting complaints of the API being bloated, packing lots of extra features. We have to take care of this before adding more platforms in NatCam 2.x.

We rarely issue refunds. This is because developers already have access to the API and access to updates on the Asset Store. This would be unfair to us as nothing would stop devs from getting the API, getting a refund, and keeping the API.

 

Well the list of upcoming fixes in 1.4 that you posted in August seem like things I would want fixed
https://forum.unity3d.com/threads/natcam-webcam-api.374690/page-9#post-2768107

Even on 25th Sept there was no mention of saving being removed
https://forum.unity3d.com/threads/natcam-webcam-api.374690/page-10#post-2800306

quoting change log...

NatCam 1.4b0:
+ Greatly imrpoved speed of capturing photos especially on Android.
+ Added SaveMode enum.
+ Added SaveMode.SaveToAppDocuments to save photos and videos privately.
+ Added NatCam.OnPhotoSave and NatCam.OnVideoSave events to get the path of saved media.

So now as of a few days ago you are removing saving. Yes for me personally it is bloated, I don't need barcodes or face detection, but others might. Just as others might want saving that they have paid for.

If this is definitely going ahead I will ask Unity about refunds and if they can remove the asset from my account, as it is false advertising to sell a product based on a set of features and then remove those features a few days later based on the above.

Why don't you support the 1.x release cycle with the set of features people paid for, then release 2.0 with additional features but without saving, therefore not advertising that 2.0 does image saving encouraging people to buy it?

 

This is not at all an advertising stunt. In NatCam 1.4b1, the feature is almost completely implemented. The feature was instituted on request of a developer since NatCam 1.3f1. We have dropped it in 1.4b2. You are emphasizing the save feature as if it encompasses the entirety of the NatCam API.

We are doing exactly that--much needed features like a fluid fast camera preview, photo capture, barcode scanning, face detection and tracking, camera flash, exposure, torch, zoom, and so much more.

The NatCam 2.x release cycle will not be seeing new features, but new platforms. Before we do this, we must make the final 1.x release have all the core features that will be transferred to 2.x. After 1.4, we are doing a complete rebuild of the API to modularize it for supporting more platforms.

I do not advertise NatCam so that people can buy it for its image saving feature. Doing that is completely senseless. It is a camera API, not an image utility. Besides, code for doing this already exists in 1.3. And on a last note, you can stick to 1.3 since saving is the most important feature for you.

 

NatCam 1.3f4 has been submitted to the Asset Store. Here is the full changelog for NatCam 1.3:

NatCam 1.3f4

FEATURES:
+ Added universal barcode detection support. Now, barcodes can be detected in the editor and all other platforms.
+ Added Exposure control with DeviceCamera.SetExposure(float), DeviceCamera.MinExposureBias and DeviceCamera.MaxExposureBias.
+ Added ExposureMode enum and DeviceCamera.ExposureMode.
+ Added Face detection with Face struct and NatCam.RequestFace().
+ Added DeviceCamera.SetPhotoResolution()+ overloads.
+ Added DeviceCamera.ActivePhotoResolution.
+ Added a new, low-cost rendering pipeline on Android.
+ Removed rendering pipeline on iOS. This has increased performance especially for GPU-bound applications.
+ Reduced rendering pipeline memory usage on Android.
+ Removed NATCAM_DEVELOPER_MODE and component buffer access (Y and UV buffers)from rendering pipeline.
+ Fixed hanging and crashing on Samsung Galaxy S4, Nexus 1, and other Android devices with PowerVR SGX540 family GPU's.
+ Barcode detection now supports Unicode characters.
+ Made camera switching faster on Android.
+ Reimplemented NatCamPreviewScaler to be more stable.
+ Added NatCam.SaveToPhotos for saving Texture2D to the gallery or app album.
+ Added necessary checks and error logging for google_play_services when detecting barcodes and faces on Android.
+ Fixed bug on Android where camera must be manually focused before autofocus starts.
+ Fixed bug where NatCam becomes unresponsive when there is no camera in the scene.
+ Removed Verbose switchfrom NatCam.Initialize(), it is now a member variable (NatCam.Verbose =...).
+ Deprecated BarcodeDetection switchin NatCam.Initialize()for MetadataDetection (which now includes faces).
+ Changed OnDetectedBarcode delegate template to take single barcode instead of list.
+ Fixed barcodes not being detected when the preview is resumed after calling Pause on Android.
+ Fixed FormatException when scanning some barcodes.
+ Fixed bug where NatCam.ExecuteOnPreviewStart invokes immediately after switching cameras.
+ Fixed bug where NatCam.PreviewMatrix isnull when the OnPreviewStart eventis broadcast.
+ Fixed bug where NatCamPreviewScaler will incorrectly stretch UI panel on iOS.
+ Renamed Unitygram example to Minigram.

CHANGES FROM 1.3f2:
+ On Android, the native preview update and preview data dimensions now respects the orientation of the app.
+ On Android, NatCam.PreviewFrame is no more skewed depending on app orientation.
+ Added timeStamp integer field to Barcode and Face.
+ Added SaveMode enum.
+ Fixed preview incorrectly rotating when app isusingfixed orientation on iOS.
+ Fixed null reference exception when NatCam.Release is called on the fallback interface.
+ Fixed faces not being detected properly by front camera on Android.
+ Fixed face positions not being in viewport coordinates on Android.
+ Fixed face positions being inverted on the Y axis
+ Fixed bug wehere photos would not be captured after Pause() was called.
+ Fixed bug on iOS where DeviceCamera.VerticalFOV returned horizontal FOV.
+ Fixed rare crash when calling NatCam.Release()on Android.
+ Fixed compiler errors in NatCamNativeInterface.UpdatePhoto().
+ Changed Android minimum SDK version to API level 15.
+ Deprecated PhotoSaveMode enum.
+ Exposed google-play-services.jar to prevent build errors on duplicate copies.

CHANGES FROM 1.3f1:
+ Added ScaleMode.FillView so that NatCamPreviewScaler fills a RectTransform.
+ Fixed faces not being detected on Android.
+ Fixed rare crash immediately app is suspended on Android.
+ Fixed NatCamPreviewScaler scaling incorrectly on some devices.
+ Removed NatCamPreviewScaler.Apply + overloads. It now scales properly for any texture assigned to the graphic.

 

Hi there. Ended up getting NatCam. First off - well done! A much better alternative to webcam texture. However, I'm seeing an issue where the photo dimensions I get back are not as expected/requested. We're using NatCam to allow users to take still photos from within our application. We have tried two approaches. In both cases we initialize it with Full HD for the preview, and then, for Photo Resolution we request either "highest" or, as an experiment, we passed in the screen.width/height. In both cases, on iPad, we get back a photo whose aspect ratio is not the aspect ratio of the screen (on iphone, it seems to work fine). What we want is the highest resolution that has the screens aspect ratio. Here's an example of the resolutions requested, and what we get:
iPad air 2 - ios 9.2
:Initialize) Setting active cam to take photo at 2048,1536
:OnPreviewStarted) Preview started. Natcam photo res: (3264.0, 1836.0)
:OnCapture) Screen vs Original Photo: (2048,1536) vs (3264,1836)

The resolution of a normal photo taken with the iPad's camera is 3264,2448.
We just also tried setting the preview resolution to values that match the screen aspect ratio. and then requesting a photo that matched screen resolution. No luck. We continue to experiment, but there seems to be some under the hood relationship between the preview aspect ratio and the photo aspect ratio. Trying to figure out the best way to let users see as high quality preview as we can, then get a higher quality still that matches that preview.

thanks in advance for any guidance you can provide.

 

Does anyone else have an issue when saving the photo that they are rotated? (1.3) I have a landscape app and the images saved are all rotated 90 degrees clockwise. Thanks

 

I updated to natcam 1.3 and the metal crashes stopped. I am however having a new issue where if you point the phone down at the ground (maybe < ~15 degrees) the image is rendered incorrectly.

Here is a link to a capture showing the bug with a project that is nothing more than importing natcam and using the example minigram project.

https://www.dropbox.com/s/22j8f250ao4oshg/natcam bug.mp4?dl=0

 

On iOS, cameras use 'formats', each with a specific preview size and a few supported, specific photo sizes. NatCam has to search for the one closest to what the developer wants (with DeviceCamera.SetResolution/SetPhotoResolution). NatCam gives precedence to the preview resolution. You can't get around this issue. Why exactly are you trying to preserve the aspect ratio? You can simply use the NatCamPreviewScaler if it is scaling you are worried about.

 

I emailed you.

 

@abryden You need this fix.

 

@Cyborn I saw that you sent me a message. Please email me or post on this thread because it is hard to track on the Unity website (I don't see any message here). Both crashes you faced have been fixed in 1.3f4 which will be released soon (waiting to be reviewed by Unity). You can fix the NullReferenceException by adding this line:

Code (CSharp):

1. if (Dispatch == null) return;

at the top of NatCamNativeInterface.Render (int).

 

After NatCam 1.4, we will be releasing NatCam 1.5 then segueing into the 2.x release cycle. It will feature a more accessible, modularized API. More news to come.

@coshea saving photos will be reintroduced in 1.5.

 

Hi

Emailed you back about photo rotation.

So will there be no saving in 1.4 update?

It is great that you are keeping saving in 1.5 but I was going to suggest an alternative, why don't you take that native texture2d saving code and spin it into its own asset for $5. Make the asset literally save a texture to the device and let you set the album name, that is it. Personally I'd be happy to pay for that (as other assets dont let you pick album name) and keep using natcam for image capture.

Chris

 

Nope. It has been removed in 1.4. not to worry though, we've already started working on 1.5 so there won't be much of a hiatus between both releases.

We'd rather not make an asset that isn't providing a much-needed solution. It isn't in high demand.

No worries. NatCam 1.5 will be a very interesting release. More info to follow soon

 

Capturing photos on Android in 1.4 is now instantaneous . All thanks to dropping the image rotation and going for per-use-case transformation. NatCam 1.4b2 has only one significant issue left before we go into testing video capture and then release it. Stay tuned!

 

Thanks for the response. Using preview scaler for the preview, and then using our matrix scaling to scale down the still photo is what we ended up doing. The only remaining issue is that on some android devices, the still photo ends up being the same aspect, but showing slightly more content, then the preview.

 

This seems to depend on how the device camera takes photos. There isn't anything I can do in the camera API to control this behaviour. Implementing a workaround for this might be challenging; I suggest simply taking a screenshot of the preview using NatCam.PreviewFrame instead.

 

NatCam 1.3f4 is live.

NatCam 1.4b2 will soon be ready for testing. We are trying to wrap up video recording. I will be asking for testers once it is completed.

 

I tried this fix and the new version (just posted to asset store) and I'm still experiencing the problem in the following video: (made a new one to show both orientations are experiencing the issue). I tried this both with a fresh 1.3f4 and a fresh 1.3f4 with the file you linked dropped in.

https://www.dropbox.com/s/o2dkldh97ad00hv/natcam bug 5.mp4?dl=0

Any ideas?

 

What is the app orientation you have in Player Settings?

 

autorotation - all allowed

 

Then this is intended behaviour. When your app supports all orientations, the camera has no way of knowing which one is the 'reference' orientation to use for the preview, so it will use whatever the current device orientation is. Autorotation for camera apps is bad UX practice. You should lock the rotation and instead manually handle device orientation events in your UI (like how the stock iOS and Android camera apps behave).

 

Having a transition occur in the camera when no transition occurred in the UI is correct behavior? All I want is for the current orientation of the camera to match the current UI orientation. This does for the most part occur but when you point it at a low angle (one that does not trigger a orientation change) the camera changes orientation anyway.

edit: I could be wrong but I don't think that the following showing up in the example app using default settings in unity is correct:

http://imgur.com/wXUbMm3

edit: I realize that for a maximally smooth experience it is ideal to not rotate the camera when the device orientation changes. However this introduces other issues in the workflow of an app.

 

Based on what you described, this sounds like a bug we addressed in 1.4. Does the change occur during face up or face down? If so, let me know.

Sounds like the face up/down issue.

UX is very important. Prioritize this always.

 

Only during face up/down. Thanks!

I agree on prioritizing UX - we're trying to find a good way to achieve the same effect as the native camera app but are having trouble getting it to work with our other screens.

 

Please email me at olokobayusuf@gmail.com. I will work on the fix and send it to you. Unity takes forever to review an update so we might just keep it and release 1.4 when ready.

 

I have the same issue as BublarGroupAB.

Started using NatCam this morning for the first time, seems to work fine but returning to home, or if the phone goes to sleep, resuming the app results in the preview not updating.

Everything else is running as normal but no image updates...

Phone is a Galaxy S7 Edge.

As a sanity check I built the example MiniGram and that is the same.

 

This happens on some Android devices. We have fixed it in 1.4b2. We will be releasing the fix with 1.4.

Correct. Also, barcode and face detection would work. The issue occurs because of the camera preview not being rendered, but the camera is still running.

Can you paste the full stack trace here? I haven't encountered this issue.

 

Is there any fix we can do to solve this Larne? Rather than waiting for 1.4b2 that is?

I have just downlaoded your UnityGram apk off the youtube vid and that works without any issues.

 

I'm afraid not. It requires a native fix (and even I don't have the native sources for NatCam 1.3f4).

Although it was built with an earlier version of NatCam, the Unitygram APK should behave the same way. I'm a bit perplexed that it doesn't.

 

Any workarounds?

I have tried calling Release then re-initialising NatCam as per the docs but that results in a white texture (presumably NatCam.Preview is null at this point).

 

Do you call ExecuteOnPreviewStart and set the preview texture again? If so, can you post your suspend logic and the logs?

 

Hi guys. We are trying to wrap up with NatCam 1.4. There is one significant constraint with video recording on Android: the minimum API level requirement has been bumped up to API level 18 (Android 4.3 Jelly Bean). This restriction will apply even if you are not using video recording.

In NatCam 1.5, the restriction will be somewhat lifted; it will only exist for those who will be recording videos. More info about NatCam 1.5 to come; it's going to be a huge release, the crescendo of the 1.x release cycle.

 

@Lanre Hi, Where's the video snippet (on your youtube) for face-tracking?
Which one is better for a simple faces-tracking (with a better camera view ofcourse)? OpenCV + Natcam OR simply just using your NatCam face-tracking support. Targeted for android.
And which version is good for the above respective answer? 1.3 or 1.4?

Just send you an email.

oh, great plugin btw

EDIT: the exact openCV sample I would like to integrate with NatCam is WebCamTextureAsyncDetectFaceSample.cs, it works with WebCamTextureToMatHelper.cs. I searched around the forum and found somebody here: https://forum.unity3d.com/threads/released-opencv-for-unity.277080/page-17#post-2777711 post example based on your script NatCamTextureToMatHelper.cs. But I still can't get it working. Always get NullReferenceException using Imgproc.cvtColor

 

I've just purchased Natcam, very nice plugin I must say.
The problem I've encountered is that after I built the app to Android, when I switched to another scene, and then back the scene with Natcam, the Natcam preview is blank, but if I click the capture button, it will still show the captured photo.If I clicked again, preview is back online.
I am using SceneManager.LoadScene to switch scenes, and this problem doesn't exist in Unity on my Mac, just on the Android built for my Galaxy S6.
I am new to Unity, should I add any script to resume the preview after the switch?

Thanks