NatDevice - Media Device API

Hi, does it support Unity 2018.3?

 

Yup, it does.

 

Okay. I'm experiencing a problem where I switch cameras during run-time (meaning the preview is already running) and the preview freezes on several android devices. But this doesn't happen on 2.1 so I guess it's best to wait for that.

 

I've had some delays but I should finish the update in the next two days.

 

@Lanre
Simplifiy API makes it a great plugin.
Im trying to export an android studio project from Unity 2018.3 but i get following error
please advise

(UnityEditor.Android.PostProcessor.PostProcessorContext context) (at <c20938b27270486da108dbf5415cc818>:0)
UnityEditor.Android.PostProcessor.PostProcessRunner.RunAllTasks (UnityEditor.Android.PostProcessor.PostProcessorContext context) (at <c20938b27270486da108dbf5415cc818>:0)
UnityEditor.Android.PostProcessAndroidPlayer.PostProcess (UnityEditor.BuildTarget target, System.String stagingAreaData, System.String stagingArea, System.String playerPackage, System.String installPath, System.String companyName, System.String productName, UnityEditor.BuildOptions options, UnityEditor.RuntimeClassRegistry usedClassRegistry, UnityEditor.Build.Reporting.BuildReport report) (at <c20938b27270486da108dbf5415cc818>:0)
UnityEditor.Android.AndroidBuildPostprocessor.PostProcess (UnityEditor.Modules.BuildPostProcessArgs args, UnityEditor.BuildProperties& outProperties) (at <c20938b27270486da108dbf5415cc818>:0)
UnityEditor.PostprocessBuildPlayer.Postprocess (UnityEditor.BuildTargetGroup targetGroup, UnityEditor.BuildTarget target, System.String installPath, System.String companyName, System.String productName, System.Int32 width, System.Int32 height, UnityEditor.BuildOptions options, UnityEditor.RuntimeClassRegistry usedClassRegistry, UnityEditor.Build.Reporting.BuildReport report) (at /Users/builduser/buildslave/unity/build/Editor/Mono/BuildPipeline/PostprocessBuildPlayer.cs:286)

 

The error log isn't full, but it is probably because Unity is trying to add the iOS library to the Android build. Navigate to libNatCan.a in the Editor and in the Inspector, make sure that the plugin applies only to iOS and nothing else.

 

Looks like this was the issue, does it support all android architectures?
x86, arm7, arm64bit

 

Yes, it supports armv7, arm64, and x86.

 

Thanks for prompt reply,

When i run camera code, i get this error in log

Code (CSharp):

1. [Exception] DllNotFoundException: Unable to load DLL 'NatCam': The specified module could not be found.
2.     NatCamU.Core.Platforms.NatCamBridge.RegisterCoreCallbacks (NatCamU.Core.Platforms.NatCamBridge+StartCallback startCallback, NatCamU.Core.Platforms.NatCamBridge+PreviewCallback previewCallback, NatCamU.Core.Platforms.NatCamBridge+PhotoCallback photoCallback, System.String context) (at <00000000000000000000000000000000>:0)
3.     NatCamU.Core.Platforms.NatCamAndroid..ctor () (at <00000000000000000000000000000000>:0)
4.     NatCamU.Core.NatCam..cctor () (at <00000000000000000000000000000000>:0)
5.     Arrow.Plugins.System.Camera.ANatCam.get_IsPlaying () (at <00000000000000000000000000000000>:0)
6.     ScreenPhotoConverter.OnCellSizeChangeHandler () (at <00000000000000000000000000000000>:0)
7.     UnityEngine.UI.Slider.set_wholeNumbers (System.Boolean value) (at <00000000000000000000000000000000>:0)
8.     ScreenPhotoConverter.Awake () (at <00000000000000000000000000000000>:0)
9.     ScreenPhotoConverter.ToggleScreen (System.Boolean show) (at <00000000000000000000000000000000>:0)
10.     Arrow.Components.AScreenNavigator.ShowScreenAt (System.Int32 index) (at <00000000000000000000000000000000>:0)
11.     Arrow.Utils.Permissions.AUtilsPermissions+<RequestUserPermission>d__0.MoveNext () (at <00000000000000000000000000000000>:0)
12.     MEC.Timing.Update () (at <00000000000000000000000000000000>:0)
13.     Rethrow as TypeInitializationException: The type initializer for 'NatCamU.Core.NatCam' threw an exception.
14.     Arrow.Plugins.System.Camera.ANatCam.get_IsPlaying () (at <00000000000000000000000000000000>:0)
15.  
16.    
17. [Exception] NullReferenceException: Object reference not set to an instance of an object.
18.     NatCamU.Core.NatCam.get_IsPlaying () (at <00000000000000000000000000000000>:0)

Please do note that im building this project by exporting to Android Studio with custom Gradle file. Let me know if this has to do something with it

Thanks

 

Can you verify that libNatCam.so is present in the generated project?

 

Thanks, its now working

 

Perfect!

 

Hey there,
i'm having an issue with the Preview and Photo. The resolution from the preview is different from the photo. I tried it on different devices and im still having the same problem.
I'm using the highest quality.
Any help?

 

So in Conjunction to Brian_Dean, Here is the code in CallBack. I have observed different cropping on both sides on different devices.

Code (CSharp):

1. Texture2d photox;
2. private void OnPhoto(Texture2D photo)
3.     {
4.         try
5.         {
6.      
7.            this.photox = photo;
8.             photoTaken.SetActive(true);
9.             print("Took Photo");
10.          
11.  
12.          
13.             byte[] bytes = this.photox.EncodeToJPG();
14.             //Write out the JPG.
15.             int timeStamp = System.DateTime.UtcNow.Millisecond;
16.             int currentPhotoCount = PlayerPrefs.GetString("AllPhotos", "").Split(',').Length;
17. #if !UNITY_EDITOR
18.      NativeGallery.SaveImageToGallery(bytes,"Gallery","PhotoMerce {0}.jpg",null);
19.         print("File Saved to gallery ");
20.      //   pathOfPhoto= filePath;
21.  
22. #else
23.             string filePath = Application.persistentDataPath + "/Shot-" + currentPhotoCount + " " + timeStamp + ".jpg";
24.             File.WriteAllBytes(filePath, bytes);
25.             print("File Saved to System" + filePath);
26.             PlayerPrefs.SetString("AllPhotos", PlayerPrefs.GetString("AllPhotos", "") + "," + filePath);
27.             print("AllPhotos " + PlayerPrefs.GetString("AllPhotos", ""));
28.             pathOfPhoto = filePath;
29. #endif
30.             isSaving = false;
31.             photoTaken.SetActive(true);
32.          
33.      
34.          photoTaken.GetComponent<Image>().sprite = Sprite.Create(photox, new Rect(0, 0, photox.width, photox.height), new Vector2(0.5f, 0.5f));
35.  
36.         }
37.         catch (Exception e)
38.         {
39.             Debug.Log("Caught Exception while SavingPhoto" + e.Message);
40.         }
41.     }

 

I have used NatCam.CapturePhoto(OnPhoto); to get the photo. If we use CaptureFrame the Flash won't work. And image saved is 1920x1080 only.

 

Did you set both

Code (CSharp):

1.      
2.             NatCam.Camera.PreviewResolution = CameraResolution._1920x1080;
3.             NatCam.Camera.PhotoResolution = CameraResolution._1920x1080;

 

This isn't a problem; it is how the device cameras work. You should not rely on the aspect ratios of the preview and photo being the same because on most devices and cameras, they never will.

 

You shouldn't depend on both aspect ratios matching. In the MiniCam example for instance, we have to rescale the display RawImage when we present the photo so that it doesn't look squished (and we repeat this process when we resume the preview).

 

Thanks for getting back. This does not have anything to do with MiniCam. And the problem is not with the Preview. Preview looks fine. And CaptureFrame also works fine. But CapturePhoto is giving image which looks zoomed in or cropped. And I am not talking about photo which i am previewing, but the photo which I am saving on disk from the Texture2d returned from CapturePhoto.
Let me know if I am doing anything wrong here.

 

When you say the photo is zoomed in, then that might be something else. But it is expected behaviour for the photo to have a different field of view from the preview (since it has different resolution). For all intents and purposes, the preview and photo resolution should be considered as being completely independent of the other.

 

Wait, I didn't get what you mean by "we have to rescale the display RawImage when we present the photo so that it doesn't look squished (and we repeat this process when we resume the preview)."
Do you mean to say we have to kinda Fake the Preview? by Scaling it out? Can you help us out here?

 

Hmm, but why zoomed in? if zoomed out that would be actually ok. But zoomed in is causing loss of image.
Can we have the same FOV in Preview?

 

I was explaining how we handle the preview and photo aspect difference in the MiniCam example. In MiniCam, we have a single RawImage that displays the preview, and when a photo is captured, the photo. When we display one or the other, we have to properly scale the RawImage so that it doesn't look squished.

 

It's zoomed in because the preview FOV is always an upper bound (I have not come across a single camera where this is not the case). As I said, there is no way to guarantee the same aspect ratio (and by implication, FOV) in the preview and photo resolutions. It is entirely up to the device camera, and most device cameras don't respect this.

Your app logic should not have to rely on both preview and photo aspects/FOV's being the same.

 

I just tried on a Huawei P10 and the images from the normal phone camera is 2240x3968 and the app takes photos in 3968x2976.
This phone gives lot of difference. But I did check the images in multiple different phones it does give some accurate results most of the times. And I suppose setting the Preview resolution to Highest isn't using full Resolution and it uses 1920x1080. Is this true?
Sorry I am being so much into getting to know the Asset, its just I will have better understanding as to what to do in case.

 

You can explicitly set the photo resolution to (3968, 2240) in your code, but this will only apply to the P10. This is why it's difficult to try to control the photo resolution.

NatCam doesn't clamp the preview resolution. If you get 1920x1080 on the device, then that is the maximum that the device supports.

No worries. Check out some of the medium articles for more info.

 

Perfect, this is what I was looking for. I will try out this. Thanks for help.

 

Also taking photos at highest is taking up lot of memory which causes app to crash on some devices. Is this a known issue?

 

So in my case when do i use below code?
// Release the photo texture
Texture2D.Destroy(photo);
In the callback function?

 

Destroy the photo texture once you are done with it. It depends on what you do with the photo texture, but once you aren't using it any longer, destroy it.

 

Hello @Lanre
I'm still waiting for the fix, and we are stuck with this problem... Could you please tell me if there is a workaround or something while waiting for the new fix ?
Thank you,

Regards

 

Currently there is no workaround. If you can't wait, email me your invoice number and I'll share my current build of the upcoming version with you.

 

Hei,

Just to inform that we are experiencing ,multiple issues lately.
The camera is set on and off in our application.
Many devices will fail on second attempt to start the camera and result is just a still blurry image. I saw it on S6 and Lenovo Phab at least. My guess is an exception when starting camera on OnEnable since all the UI that should be disable right after it, remains on screen.
Second issue, the camera switch will also freeze the image on Android.

I'll try and see if I can get some debug on those.

Regards

 

The update which fixes these issues will be submitted later today.

 

How long does it generally take before the update appears in the Asset Store?

 

Anywhere between 2 days and 2 weeks (typically on the lower end)

 

Hi,

Just went through the latest release that came today, well, it is not any better.
I assume you don't have many devices to try on because it fails to work properly on many of them.

So far:

Huawei y6 2018, green line on right side, then crashes (MiniCam example crashes too)
Nokia 6.1, green line, image refocuses on capture and result is blurry.
Samsung J7 2017, refocuses on capture and result is blurry.
Nexus 5X, green line on edge (image is upside down but this is Gfx API issue)
Galaxy S6 refocuses on capture and result is blurry.

It worked fine on some though:
Galaxy S7 and S8
Lenovo Phab

Also the zoom seems to not affect the result...First it is quite slow to respond and when capturing, it will discard the zoom and claim the whole image. It was previously working.
Here is the method:

Code (CSharp):

1. public void SetZoom(float ratio)
2. {
3.         float value = Mathf.Clamp(1f, ratio, NatCam.Camera.MaxZoomRatio);
4.         NatCam.Camera.ZoomRatio = slider.value = value;
5. }

 

We test on Samsung Galaxy devices. S3 up to S7.

The green line on the edge sounds like padding bytes. What is the preview resolution? As for the crash, can you upload full logs from logcat in a .txt file?

I'm aware of the refocusing before photo capture. It's somewhat sporadic. I am working on a fix.

Your zoom method is incorrect. You are clamping the value of 1.0 between [ratio, NatCam.Camera.MaxZoomRatio]. You want to clamp the ratio between [1.0, NatCam.Camera.MaxZoomRatio] (replace first and second arguments to Clamp).

 

Hi,

The funny part about the Huawei is that the logs were not showing much wrong if I remember right. I was using the monitor.bat for it and despite freezing the image, logs would not show anything.

I’ll have to check about the zoom method coz as I wrote it above it should not work at all.
Though the same code before was working fine on iOS, zooming and capturing the right image but now it zooms but capture the full image.

 

NatCam 2.1 is Out.

And it's only going to get better from here on out, as we are going to be focusing on performance and deep integration with computer vision and virtual reality solutions.

 

Oh hold on a moment. It sounds like you want zoom to affect the photo, not just the preview. In this case, then you've found a bug! I totally missed this.

 

How can I upgrade to the newest version? The asset store seems to indicate that I have to buy it again.

Never mind, I figured it out.

 

Go to your download section and try importing it from there.

Usually it has worked but this time I had to delete and import again as there was some duplicate errors.

 

I actually have no resolution explicitly set. I was about to use NatCam.Camera.PhotoResolution = new Vector2Int(1280, 720); but it is currently commented out.

It may be that the focusing issue is on my side actually, I'll check it out. I may have missed it coz it works fine on some devices and used to work on all. Still you mentioned that you are aware of the issue so not sure.
The possible culprit code:

Code (CSharp):

1.    
2.     protected virtual void OnFrame()
3.     {
4.         foreach (var touch in Input.touches)
5.         {
6.           //  NatCam.Camera.FocusPoint = new Vector2(touch.position.x / Screen.width, touch.position.y / Screen.height);
7.         }
8.     }

On any given tap, it will try trigger to reset the focus point including when tapping the capture button.
I'll try without it if it helps.

Btw, SoftabiityGroup and me are the same person, just personal and pro account...

 

I faced this issue sometimes during testing. Apparently dealing with focus right before a capture is more nuanced with the new API.

Yeah I thought so.

 

On Nexus 6P, there is a large semi-transparent green bar at the top of the camera preview with the new version.
It actually looks like the colors are shifted down by the same amount. That is, it's as if the black+white part of the image were separated from the colors, and all the colors were shifted down by 100 pixels or something.
Attached a couple of screenshots.

edit: It works fine on Samsung Note 8/9

 

Ok so now I have removed all kind of manual focus code, it seems to work fine. At least the S6 was failing and is now working. The autofocus setting is a bit slow to register but that'll do it anyway. It just tends to stare blurry for a couple of seconds and then starts to focus.

So now, I am left with the green line as shown in the previous post, they only appears on the preview and at least one of our device (Huawei) simply crashed even with MiniCam example.

I'll run some tests on Tuesday to see if I can get some logs about the Huawei.

Cheers

 

Can you upload the full logs in a .txt file?

 

What can I do to gather useful information about the green bar/color shifting issue I showed above?

 

If you have Android Sdk, you can find the folder, then tools and then monitor.bat.
Keep your phone usb connected.
Create a Unity tag so it filters out all the Android massive debug log dump.