NatDevice - Media Device API

Hi NatCam folks. Been using your plugin for a while and could use some advice. We have an app that lets users take photos and then do stuff with them. We want to show a preview, then when they take the photo we want the still camera to do the taking, and we want the highest res of that still camera (which we then downscale). We also want to know the FoV of the *still* camera photo we got.

Problem 1: we ask for highest res of the still camera, but what we get is the highest res in an HD aspect ratio. So, we expect to get, e.g. 3264,2446 res (which is the normal still camera res on ipad) but after requesting "highest" for photo resolution, we get 3264.0, 1836.0 - which is an HD aspect ratio (the height is smaller as you can see). Is there a way to ensure that NatCam will give us a photo that is, truly, the highest res & proper aspect ratio for the still camera?

Problem 2: assuming we solve #1 - our preview will now be out of sync with our still (not just in res, but in aspect ratio). Is there a way to show someone a preview ala the preview in the native camera? Even if it's at a lower res - what we want is to present the right field of view and aspect ratio.

Problem 3: Related to the above - Natcam lets us ask for the active camera field of view - but I suspect what it tells us is the video camera Fov. Is there a way to get the Still camera FoV?

Thanks in advance for any guidance you can provide.

 

Related question (maybe): Where can I get NatCam 1.5 and would that solve these issues?

 

This is determined by the device's AVCaptureDeviceFormat. The camera can only have one active format at a time, and each format has a different preview and photo resolution. NatCam will pick a format giving most relevance to the preview resolution, then framerate, then photo resolution. Getting the highest photo resolution with a certain preview resolution is practically hit-or-miss. In my tests, I have found that a HD preview (720p) usually allows the photo resolution to be maximum.

You can write a custom UI panel or preview mesh scaler component to properly show the photo.

The native camera API's do not offer any distinction between preview and photo FOV's.

NatCam 1.5 is not ready although it is close to completion. The system for picking resolutions remains the same as in 1.3 and 1.4.

 

NatCam 1.5 is complete. We will be submitting it to Asset Store review tomorrow. We decided to skip beta testing since it builds off the stability of 1.4b2 and 1.3f4. As a result, I will NOT be handing out the build to developers myself. The build will be distributed on the Asset Store.

 

Thanks for your reply! What is not clear to me, still, is how the aspect ratio could ever be made to be the aspect ratio of the still camera. Am I right that what you're saying below amounts to saying that while we could get the highest photo res, the aspect ratio of the photo will remain HD (and thus, to fit onto a standard ipad, we will end up cropping/not getting all the FoV of the photo)? In my experiments this seemed to be the case. That is, no numbers passed into PreviewRes/PhotoRes would result in a non hd aspect ratio.

 

This depends on the supported formats of the exact device camera on the exact iPad.

No. On my iPhone SE, I have a HD preview but have a photo resolution that has a 1.33 aspect. This completely depends on the device camera's AVCaptureFormat. It depends on the preview resolution since NatCam gives preview resolution highest precedence. One way I could imagine you finding the highest photo resolution is this:

Code (CSharp):

1. Resolution highestPreview, highestPhoto;
2. Resolution[] standardPreviews = {All the standard preview resolutions you wish to search through};
3. foreach (resolution in standardPreviews) {
4.      // Set the preview resolution
5.      NatCam.ActiveCamera.SetResolution(resolution);
6.      // Set the photo resolution
7.      NatCam.ActiveCamera.SetPhotoResolution(ResolutionPreset.Highest);
8.      // Check the photo resolution
9.      Resolution currentPhotoRes = NatCam.ActiveCamera.PhotoResolution;
10.      // Check if it is the highest
11.      if (currentPhotoRes > highestPhoto) {
12.           // Set
13.           highestPhoto = currentPhotoRes;
14.           highestPreview = resolution;[/INDENT]
15.      }
16. }
17. // Set
18. NatCam.ActiveCamera.SetResolution(highestPreview);

This is specific to the device and to the current format (which is dependent on the preview resolution you set).

 

Hey guys, Vuforia is still off?

 

NatCam and Vuforia are incompatible because of Vuforia's lack of a way to provide preview data.

 

I have a question about the upcoming video capture support: will I be able to filter the recorded video? I'm working on a game right now that calls for the ability to record black & white footage, so I'd want to not just affect the preview, but the recorded video as well.

I actually rolled my own filtered Unity video capture for iOS using CoreImage filters, but there are a couple of performance issues that hit after a few minutes, no doubt due to my sloppy coding, so I was thinking about evaluating NatCam as a possible solution, especially if it would get me Android support as well.

Thanks.

 

No. This is currently not supported and is not planned for the future. Using custom shaders on the camera preview would require implementing those shaders natively, drawing the preview with them, and performing a fast readback from the GPU before sending the filtered frame to the hardware encoders. It is extremely complicated and is out of scope for a camera API like NatCam.

That sounds like a memory issue. You might also consider using OpenGL ES or Metal with a CVOpenGLESTextureCache/CVMetalTextureCache. This is the fastest, most scalable, but most complicated solution.

On Android, you would need to look into the MediaCodec and MediaMuxer API's.

 

NatCam 1.5 Professional is out!

Current NatCam developers have automatically been transfered onto the Professional Spec.

Go on and play with video recording

 

I'm working in an AR game and I'd like to implement NatCam to use the preview camera feature. Is it possible to test it before upgrade? I'd need to test the performance in my game. Thanks in advance!

 

Hi. What AR SDK does your project use?

 

I don't use any AR sdk. In my game the player picks up multiple geolocated objects in the city and I project the telephone webcam in a plane as a WebCamTexture, but it appears very lagged. I've read that NatCam can do it faster without lage and I want to test it.

 

Here is the example APK.

 

@Lanre When I install the apk in my android (Xperia Z) the app crashed in the first screen

 

Just updated NatCam to latest version in my project and having a few bizarre errors, hoping you can help me figure out what could be missing...

Assets/NatCam/Extended/Plugins/Managed/Platforms/Android/NatCamAndroid.cs(92,38): error CS0117: `AndroidJNI' does not contain a definition for `ToByteArray

Assets/NatCam/Extended/Plugins/Managed/Platforms/Android/NatCamAndroid.cs(94,45): error CS1061: Type `jvalue' does not contain a definition for `i' and no extension method `i' of type `jvalue' could be found (are you missing a using directive or an assembly reference?)

Assets/NatCam/Extended/Plugins/Managed/Platforms/Android/NatCamAndroid.cs(96,24): error CS0117: `AndroidJNI' does not contain a definition for `CallStaticVoidMethod'

obviously something is weird with Unity and not Natcam. Any idea what the heck is going on? This is Unity 5.3.6f1 but it imports just fine in a different project with the same version of Unity.

Thx
Chris

 

Trying to build on Unity 5.5 using latest NatCam 1.5 and just shows an empty screen. Any ideas? Cant see any errors in the logs.

 

We discovered numerous reports of this issue on Android. We are working to fix it and push out an update in 1-2 days. I'll re-upload the APK and let you know.

 

I am not sure as to what would cause this issue. I'll see what I can find.

 

We are working on fixing this. I'll be emailing updates in 1-2 days from now.

 

We are working to fix this.

 

Hi guys.

Please wait 1-2 days as we fix the issues with NatCam Pro 1.5. In the mean time, I can send you 1.4b2.

 

I've reverted to natcam 1.3 which worked fine for me. In the meanwhile I'm completely uninstalling Unity and reinstalling as I've encountered weird problems like this before with some of Unityengine's own API and other basic functions not being available.

 

Sounds good. I'm working as fast as possible to resolve this. Sorry for the inconvenience.

 

Hi there!

Your latest build is crashing on android /latest Unity. It shows a frame of the camera then closes.
We tested the included demoscene.

Leo

 

We're working on resolving all the issues now.

 

All the fixes have been implemented.

I will be submitting NatCam Pro 1.5f2 tomorrow for review.

EDIT:
We received more reports of bugs so we had to continue working on it.

 

My app has been crashed following below logs with the latest Natcam.
Is there any duplicated issue for this?
The old version(i don't remember, but I downloaded maybe mid of Sep.) works fine.
Thanks for your kindly response in advance.

12-13 21:25:05.662 I 286 DEBUG signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 00000000
12-13 21:25:05.702 I 280 Vold [LGE][VOLD][NetlinkHandler.cpp][onEvent()] subsys:cpu, action:0
12-13 21:25:05.712 D 293 QCamera2HWI static void qcamera::QCamera2HardwareInterface::metadata_stream_cb_routine(mm_camera_super_buf_t*, qcamera::QCameraStream*, void*): photosolid_user_real_gain = 11.541289
12-13 21:25:05.802 I 286 DEBUG r0 00008ce0 r1 6cdf6a20 r2 00000000 r3 00000000
12-13 21:25:05.802 I 286 DEBUG r4 7707d788 r5 00000000 r6 7692e008 r7 7692e00c
12-13 21:25:05.802 I 286 DEBUG r8 7692e010 r9 7692e014 sl 00384000 fp 6cdf6abc
12-13 21:25:05.802 I 286 DEBUG ip 00000000 sp 6cdf6a58 lr 75fd9230 pc 00000000 cpsr 400d0010
12-13 21:25:05.802 I 286 DEBUG d0 0000000000000000 d1 0000000000000000
12-13 21:25:05.802 I 286 DEBUG d2 0000000000000000 d3 0000000000000000
12-13 21:25:05.802 I 286 DEBUG d4 0000000000000000 d5 0000000000000000
12-13 21:25:05.802 I 286 DEBUG d6 0000000000000000 d7 0000000000000000
12-13 21:25:05.802 I 286 DEBUG d8 0000000000000000 d9 0000000000000000
12-13 21:25:05.802 I 286 DEBUG d10 0000000000000000 d11 0000000000000000
12-13 21:25:05.802 I 286 DEBUG d12 0000000000000000 d13 0000000000000000
12-13 21:25:05.802 I 286 DEBUG d14 0000000000000000 d15 0000000000000000
12-13 21:25:05.802 I 286 DEBUG d16 00008d4000000000 d17 6e8640080000000b
12-13 21:25:05.802 I 286 DEBUG d18 773cec28773cec28 d19 0000000000000002
12-13 21:25:05.802 I 286 DEBUG d20 429d5ee000000001 d21 6ad1843b00000000
12-13 21:25:05.802 I 286 DEBUG d22 0000000600000000 d23 0000000100000001
12-13 21:25:05.802 I 286 DEBUG d24 00000000bf8b848f d25 3eb21b833eb21d2e
12-13 21:25:05.802 I 286 DEBUG d26 3fc66a7d00000000 d27 0000000000000000
12-13 21:25:05.802 I 286 DEBUG d28 00000000becf110b d29 bf7002e6bf700526
12-13 21:25:05.802 I 286 DEBUG d30 41b9bf6abeccc65c d31 c2184585c21a0983
12-13 21:25:05.802 I 286 DEBUG scr 6800001b
12-13 21:25:05.802 I 286 DEBUG
12-13 21:25:05.802 I 286 DEBUG backtrace:
12-13 21:25:05.802 I 286 DEBUG #00 pc 00000000 <unknown>
12-13 21:25:05.802 I 286 DEBUG #01 pc 0000422c /data/app-lib/com.handroid.cameragunfight-1/libNatCamFastRead.so (NatCamFastReadGLES3::Read(unsigned char**, int*, int*, int*)+92)
12-13 21:25:05.802 I 286 DEBUG #02 pc 00000e50 /data/app-lib/com.handroid.cameragunfight-1/libNatCamProfessional.so (Java_com_yusufolokoba_natcamprofessional_NatCamProfessional_UpdateFastReadContext+76)
12-13 21:25:05.802 I 286 DEBUG #03 pc 000204cc /system/lib/libdvm.so (dvmPlatformInvoke+112)
12-13 21:25:05.802 I 286 DEBUG #04 pc 000512af /system/lib/libdvm.so (dvmCallJNIMethod(unsigned int const*, JValue*, Method const*, Thread*)+398)
12-13 21:25:05.802 I 286 DEBUG #05 pc 00052d15 /system/lib/libdvm.so (dvmResolveNativeMethod(unsigned int const*, JValue*, Method const*, Thread*)+184)
12-13 21:25:05.802 I 286 DEBUG #06 pc 00029960 /system/lib/libdvm.so
12-13 21:25:05.802 I 286 DEBUG #07 pc 00030ec8 /system/lib/libdvm.so (dvmMterpStd(Thread*)+76)
12-13 21:25:05.802 I 286 DEBUG #08 pc 0002e560 /system/lib/libdvm.so (dvmInterpret(Thread*, Method const*, JValue*)+184)
12-13 21:25:05.802 I 286 DEBUG #09 pc 00063795 /system/lib/libdvm.so (dvmCallMethodV(Thread*, Method const*, Object*, bool, JValue*, std::__va_list)+336)
12-13 21:25:05.802 I 286 DEBUG #10 pc 0004ce97 /system/lib/libdvm.so
12-13 21:25:05.802 I 286 DEBUG #11 pc 00002eec /data/app-lib/com.handroid.cameragunfight-1/libNatCamRenderDispatch.so (_JNIEnv::CallStaticVoidMethod(_jclass*, _jmethodID*, ...)+36)
12-13 21:25:05.802 I 286 DEBUG #12 pc 000030e0 /data/app-lib/com.handroid.cameragunfight-1/libNatCamRenderDispatch.so (NatCamReferencer::Invoke()+60)
12-13 21:25:05.802 I 286 DEBUG #13 pc 003757d8 /data/app-lib/com.handroid.cameragunfight-1/libunity.so
12-13 21:25:05.802 I 286 DEBUG #14 pc 00007ec4 <unknown>

 

We are working to introduce a global boolean to control whether the microphone should be used. It will control whether audio recording should be performed thereby controlling whether permissions will be asked.

 

Since NatCam 1.4, we stopped all pre-processing on the captured photo to make photo capture as fast as possible (the speed gains were tremendous). Now, all photos will be returned in the natural orientation of the camera (landscape left). I talked more about this issue in this post.

 

This has been fixed. Email me your invoice number and I will send you the fix.

 

Dear Yusuf,
I'm interested in purchasing the NatCam plugin and it looks like what I need right now.

But I just wanted to confirm that it uses camera api2 on Android, specifically I need that to make slow motion video recordings (120-240fps). Is that possible with the current version of the plugin?

Thank you in advance!

 

NatCam uses camera1 (android.hardware.camera) to maintain support for the majority of Android devices. Slow motion video is not supported.

 

Hi,

I'm using the latest version of NatCam, however, it still crashes when I call

Code (CSharp):

1. NatCam.Release()

on Android.

I'm using code to setup NatCam and call the release method in the parent GameObject OnDestroy() method. Should I be doing this, or is there a better way to release NatCam?

EDIT: Also, is there anything I need to do before calling Release()? Should as pausing NatCam first?

Any help would be great.

Thanks in advance.

 

You don't need to call Release when you stop using NatCam. Pause will suffice. When closing the app, NatCam will automatically call Release.

Not at all. Simply call Release.

Can you email me the crash logs from your device? My email address is olokobayusuf@gmail.com.

 

Hi Yusuf,
I'm using the latest version of NatCam, however, there is a bug in the Scanner scene.

Internal_GetWidth can only be called from the main thread.
Constructors and field initializers will be executed from the loading thread when loading a scene.
Don't use this function in the constructor or field initializers, instead move initialization code to the Awake or Start function.
UnityEngine.Texture:get_width()
NatCamU.Core.Platforms.NatCamLegacy:<MetadataUpdate>m__A() (at Assets/NatCam/Extended/Plugins/Managed/Platforms/Legacy/NatCamLegacy.cs:69)
NatCamU.Core.Utilities.INatCamDispatch:<Update>m__13(Action) (at Assets/NatCam/Core/Plugins/Managed/Utilities/INatCamDispatch.cs:61)
System.Collections.Generic.List`1:ForEach(Action`1)
NatCamU.Core.Utilities.INatCamDispatch:Update() (at Assets/NatCam/Core/Plugins/Managed/Utilities/INatCamDispatch.cs:61)
NatCamU.Core.Utilities.NatCamConcurrentDispatch:UpdateConcurrent() (at Assets/NatCam/Core/Plugins/Managed/Utilities/NatCamConcurrentDispatch.cs:59)

 

I just bought your asset, when I build I have a gray screen, impossible to have a video return, what have I done wrong? (Unity 5.5.0f3, android 6.0.1 galaxy s7)

I'm trying your script Assets\NatCam\Professional\Examples\GreyCam with no add-on and no modification

 

Hi. I have fixed this in 1.5f2. This build is almost complete; we just have to resolve one issue on Android.

 

There is a bug in 1.5f1 that causes this to happen. We are almost done with the fix in a new build, 1.5f2. We will complete this build and publish it very soon.

 

I am interested in your asset. Can you tell me if it will work for me...

I need to display (render to texture) the device camera within Unity for an AR project. I need render texture to display in the scene, NOT as part of the UI/Canvas. Full-screen regardless or device orientation (portrait/landscape).

Can NatCam do that? If it will work (yay, that's great), how bloated will the asset make my project (if at all)?

Thanks.

 

Hi. What AR SDK are you using? NatCam cannot run alongside Vuforia or ARToolkit. If these don't apply to you, then you can blit the NatCam preview texture directly to the screen (or simply use the UI system or a fullscreen quad). I strongly advise using the UI system because it is easier to maintain and extend.

I presume you wish to do this:

Code (CSharp):

1. camera.targetTexture = null; // Done somewhere else
2.  
3. void OnPostRender () {
4.      Graphics.Blit(NatCam.Preview, null); // Blit directly to the screen
5. }

This is completely fine.

Blitting directly to the screen would avert these issues completely. But you will have to write a shader to correct for the aspect ratio difference between the screen and the camera (so as to avoid squishing).

Yup.

NatCam is very minimalist. You can browse the file tree on the asset store page.

 

They don't as I'm trying to avoid using ARToolkit and Vuforia, which is hopefully where your asset comes in.

Will the blit to the screen still allow me to "see" 3D objects in the Unity scene, or is the camera (and any UI/Canvas) all I will see?

fullscreen quad? Is this something available in your asset as a prefab or something? I believe this is what I'm currently working with and having all kinds of issues trying to get the portrait/landscape issues fixed. Also seeing flipped (mirror) issues with the webcamTexture that I'm using for render texture. Add to these issues are the aspect ratio issues of multiple devices.

I'm not sure what I want to do, but this seems perfect! I'm not too worried about the shader part; hopefully this won't turn out to be a problem later. Seems pretty straight-forward. As long as I can see the camera feed and 3d objects.

 

Sounds good. But I should note that NatCam does not come with an AR tracking engine. This is where Vuforia and ARToolkit would be helpful. If you just need free-floating objects, then NatCam will be ideal.

Depending on how you do it, this is possible. You would have to use two cameras and blit twice to overlay scene objects onto the background. This is why I advise using a game object or the UI system.

No. NatCam only comes with helper components that work with the UI system. However, I have attached a ViewportScaler script and a ScalePlane script that utilizes it.

 

Thanks for the quick response!

Okay it sounds like I have two choices with your asset...

1. Use setup with two blits
2. Use a game object with webcamtexture (along with viewport scaler)

If I use #1 then the portrait/landscape orientation isn't an issue, but what about performance on mobile with 2 blits, would I still get 30 fps with in an empty scene?

If I use #2, what NatCam asset feature would I be utilizing here? This is how I'm currently setup.

Last question is regarding the C# source for iOS, will I be provided with this per documention? There seems to be some confusion about exactly what source you actually provide.

Thanks again!!

 

NatCam's preview texture or WebCamTexture.

On recent devices, yes. But double-blitting is usually more expensive than having one extra draw call.

It doesn't sound like you need NatCam's functionality here, except you want more camera functionality (like focusing, zooming, exposure, torch, and so on) and higher resolution previews that are much faster than WebCamTexture.

C# sources are provided. The native sources however are not; if you get NatCam Pro, then I can send you the native source projects for iOS and Android.

 

I don't need the camera focus/zoom/etc features, however I'm curious about the "higher res" and "faster" previews which you mentioned. I'm pretty sure this is something I will want/need for my app.

I didn't realize there was a NatCam and NatCam Pro (with source). I only see NatCam Pro in the asset store. How to I get NatCam (non pro)?

Edit: I think I misunderstood. You're saying NatCam Pro comes with c# source and Native "projects" (not source) for iOS and Android.
Thanks you guys are awesome!

 

Unity is yet to publish NatCam Core and NatCam Extended.

Pro comes with all C# sources (so does Core and Extended). But only Pro gives access to the native projects (native sources) by requesting them from me.

 

Have you fixed the example version that you sent me? I need to test NatCam in my game before upgrade. Thanks!

 

Yup. Sorry I forgot to send you the link to the APK. The package itself is almost done.

 

Thanks for the great support. I may not need all of the features, but this asset (and your excellent support) is worth it.

Just purchased!!

I'm sure I'll have more questions once I get going on this.

 

Do you have an example Unity scene showing the preview on a game object? Should I just add natCam to a plane along with the the ScalePlane script?

 

No I don't. But you can create a script that inherits from NatCamBehaviour and have a material or renderer on which to apply the NatCam preview texture when the preview starts. Like so:

Code (CSharp):

1. public Renderer objectRenderer;
2.  
3. public override void OnPreviewStart () {
4.      // Set the preview texture onto the material
5.      objectRenderer.material.mainTexture = NatCam.Preview;
6. }