NatDevice - Media Device API

Awesome! Any idea how long this usually takes?

 

I think about 10 days, probably a little over that.

 

How much will this be?

 

Fifty dollars.

 

So it will be released somewhere next week? I currently have a full implementation of barcode recognition using OpenCV and WebCamTexture, but it's indeed not as performant as I'd like it to be (have only tested with Samsung Galaxy S6), and I'm only using 1280x720 as resolution. Looking forward to this package!

 

@2Elemental It should. NatCam's barcode reading functionality is based off native implementations so there's no added layer (which means more time-consuming processing). In fact, scanning EAN_13 barcodes (ISBN codes on books) was the entire reason why we started developing NatCam.

 

That's nice to hear. In fact, it's not actually the barcode scanning that is not performant, as we can currently encode and decode pretty much every barcode, but it's the WebCamTexture that is laggy. Our barcode recognition using OpenCV only happens every 250ms, so a high FPS is not required. Does NatCam also support encoding of barcodes, or only decoding?

 

NatCam supports decoding barcodes only. Encoding barcodes is not a part of the native camera API's

 

OK, and which barcodes and/or QR does it support for decoding?

 

• QR
• EAN_13 (and ISBN)
• EAN_8
• Data Matrix
• PDF 417
• Code 129
• Code 93
• Code 39

 

Great, I'll have to see which one is the most performant then, but this is looking good

 

Any news on the release?

 

Still waiting for Unity

 

Hi Lanre, very excited to give this a spin on my Samsung S6 (SM-G920V)/Android 5.1.1 !!

I tried the Natcamagram apk linked in the thread above, and found that the Preview mode doesn't appear to be fully functioning--the screen is color sensitive, but does not display a high-resolution preview. It does display the high resolution bmp after holding the button for a few seconds.

Image 702: static image
Image 701: color sensitive Preview; no resolution

Is this the expected behavior, and do you have a sample with the Preview mode enabled?

 

Hi @sstadelman We might have fixed the issue (but didn't update Natcamagram). I'll PM you for further investigation. Thank you!

 

NatCam is currently on hold because we want to make a comprehensive scripting reference (documentation) for ease of use. Thus, there'll be a delay of about 6 days.

 

I need this so bad! I swear I will buy it as soon as it appears on asset store. Are you accepting donations allowing me to try your code?

 

@Lanre Do you have performance times for different resolutions on different devices?
We have an application where we need the fastest possible camera frame rate at a lower resolution
either 720p or even 480p which we want to apply a blob detection pipeline using the NEON processor.
Will this be possible using NatCam? We dont even need the texture available in Unity although the scene rendering (which is done in Unity) is dependent on the result of our pipeline which is why we need a Unity plugin. What is the fastest frame rate at lower resolutions on an iPhone 6 and a Note 4 or Galaxy S6?

 

Hi there, I got the same result as sstadelman on a Galaxy Note 5 with NatCamagram. Static resolution was fine, by default though things were just a blur. It reminded me of my efforts with Webcamtexture when I tried to read the resolution of the texture before the capture had started, ending up with 16 * 16 whatever I did.

 

Hi @aaronw NatCam will default to the highest frame rate on Android.

On iOS however, the construct is different. The frame rate is dependent on the session preset (AVFoundation's AVCaptureSession doesn't allow you specify resolution or frame rate) and is usually (in our tests) 30FPS or 60FPS (depending on the device).

On both platforms, you are ensured a smooth preview always. I should note that if you wish to do further processing on the preview texture (on the CPU), you should use the readable preview (this supplies the preview data instead of you trying to perform a readback from the GPU which is notoriously slow).

Supplying a desired frame rate is on the roadmap (for version 2.0)

EDIT: Corrected "highest resolution" to "highest frame rate".

 

Hi @grahamwell care to be a tester? We believe we fixed the issue after I posted Natcamagram but we would have to test to find out.

 

Also, I will be posting a link to NatCam's documentation in a few days. To create it, we are developing a package called Calligraphy that easily creates API documentation for your C# API's.

 

@Lanre from your response to @aaronw, it looks like there are two options for accessing the preview texture:
(1): 'view'-only (GPU-only), does not permit processing or OpenCV access, etc.
(2): 'read'-access (CPU), permits processing, OpenCV access, etc.

If so, this is very good--it definitely seems that we'll be in the (1) camp.

 

Hi Stan @sstadelman. This is 100% correct. A brief history lesson:

Originally, NatCam got the preview frame from the device camera and uploaded it directly to the GPU (this was especially crucial on Android where we needed fluid frame rates, and to perform a transformation on the textures--processes that proved all too slow on the CPU). But due to the persistence of the OpenCV community, we had to make the textures accessible to the CPU (we needed a way to perform a readback from the GPU to the CPU). On iOS, this was relatively easy whereas on Android, it was hell--literally. Anyone familiar with OpenGL programming is aware of the notorious glReadPixels. After several days of nerve-racking research and coding, we finally implemented a very performant way of doing performing a readback at little-to-no cost (available on GLES2, GLES3, and GLES3.1AEP, which account for ALL Android devices that run Unity).

NatCam's Initialize method has a parameter that specifies whether the preview should be readable or not. Although the cost is very little, it is there. That's why we didn't just create NatCam will readable previews all the time.

As a prominent graphics programming expert and stackoverflow-ist said (paraphrased):

 

Makes perfect sense. We're doing an AR app on Android that is *not* reliant on bar code or image processing for camera positioning. (In fact, we're replacing the Vuforia AR camera, since we no longer need their feature-set). So, we'll probably be opting for the non-readable parameter value.

 

@sstadelman Interesting. I'm curious as to how that works. Did you receive my email?

 

Yes, installed and it is working as expected. Looks quite sharp! There is a slight motion lag as you pan very quickly, but I benchmarked that behavior against the native Android device Camera app, (and the native iOS Camera) and it's comparable in Unity to the Native apps.
One thing which will be interesting is how to manage the autofocus behavior for a GearVR headset app. It hasn't been a massive issue for us thus far with the WebCamTexture (since everything is pretty fuzzy) but with the higher resolution, you might notice it a little more. Will share some research on what we see when we put it in the Gear.

 

@sstadelman That's great.

When you quick-pan the camera, I'm sure that the sensor (or its driver) skips frames. I'm almost 100% sure that that's why.

 

Hi @Lanre, I've been following this thread since I've started looking for a solution for the ever slow Webcamtexture. I have a conceptual app running with Vuforia on the GearVR. I'd like to substitute Vuforia with Natcam as I'm currently only using the passthrough camera feature. Vuforia has been clocking reasonable good speed on the Note4, but I'd like to see faster frame updates, and less jitter.

I looked for the .apk to test, but I can't seem to find it. Please advise?

Thank you for the awesome work.

 

I have a note4 and s6 with GearVR. I would love to implement NatCam and use the preview camera feature. Let me know when its up or if I can test out implementation!

 

NatCam is almost submitted. The wait has been on a new package we have been developing called Calligraphy, which helps us quickly generate a scripting reference right from the editor. Calligraphy is almost done, and when it is, I'll generate NatCam's documentation and upload it, then submit it for review.

 

NatCam support for Android IL2CPP is no more marked experimental as all the APK's we've built and sent out were built with the IL2CPP backend, and we have not seen any issues with it.

 

Any update on the release? Can't wait to play with this.

 

NatCam will be released next week. I am performing some last minute changes and developing the scripting reference generator.

 

@Lanre That's great news!

 

Check out NatCam's scripting reference here!.

EDIT: Google Drive's hosting is pretty neat.

 

Very cool!! Any news on release in the Asset Store?

 

@Vern_Shurtz Thank you! The plan is to wrap up in the next 5 hours and send it back to Unity. I'll keep this thread updated.

 

Any news about on release in the Asset Store? Or a way to get the plugin outside the store? Or a beta?

(I have a deadline and from what I read, NatCam will help me a lot, heh)

 

Finally finished! Waiting on Unity.

 

Excellent.

 

It's Out!!!! Get it.

 

Please make sure to add the source code other ways it is impossible to use it for any production system

 

Hi @Cromfeli. We plan to do so in the next minor release. We did not include native sources because we need to make some changes and improvements. We will be finished with these late next week. Stay tuned!

 

Can you post an update when the sources are included?

 

@SioHio I sure will. We will be including all native sources except Android C++ (the native native layer). We have to perform an optimization for OpenGLES3 devices that won't make NatCam v1.1.

We will also be adding a new delegate to provide access to the native pixel data, and access to the preview's pixel data in memory. This way, you can send the IntPtr to your own native plugins for operations (don't forget to memcpy).

Also, we are working to provide much tighter and stable OpenCV support with EnoxSoftware, the great people responsible for OpenCVForUnity. AR is also a new and upcoming focus.

 

I hope you maintain clean separation between different modules. Rather on the level of disabling/enabling those modules that one needs. I would not like your plugin to become hairy monster

 

@Cromfeli We sure will! The source files and instructions will be in a zip folder, not interfering with anything. Overall, we aren't making any changes where we enable something one person needs.

 

Hi, putting this on my list of things to watch

Few questions:
What does it mean "except Android C++", are you referring to the original android source code from Google, or some code you wrote yourself in c++?

All the images in the asset store are fullscreen, the texture access isn't implemented yet?
If it has, you may want to add some different preview.

Is there still a lot of difference when you use the texture compared to webcamtexture or a lot of the gain is lost?

 

Hi @00christian00

-By Android C++, I'm referring to the C++ code we wrote ourselves. It is very extensive, and will require special instructions for handling once we include it. We aren't including it in v1.1 because we have to investigate whether we can squeeze out more performance on OpenGLES3 devices when performing a readback by using Pixel Buffer Objects.

-Oh I didn't think of that. It seems those images are misleading. The preview is returned as a Texture2D (you can check the documentation under Classes>NatCam>Preview. To ensure compatibility across unsupported platforms (non-iOS and non-Android platforms), there is also PreviewTexture which is a Texture (either a Texture2D on supported platforms, or a WebCamTexture on unsupported platforms, downcast to a Texture). I have noted this, and I will add better images. Thank you.

-The first difference you see is the crisp resolution (if you specify one, like FullHD or HighestReoslution). The next is the speed. It. Is. Fast. Much faster than WebCamTexture on a significant number of devices. And of course, all the other features are icing on the cake.