NatDevice - Media Device API

The problem was in development build you sent me. Reverting to Asset store version also fixes the android IAP memory leak

 

Can you send me crash logs?

 

I already sent it to you, but u just ignored it, I had to fix your asset by myself, and still it is not compatible with ad networks plugins

 

Hm sorry about this, I have some emails I haven't gotten to yet, I'll check them out. What is the issue with ad networks?

 

Hello!
We're using CapturePhoto and we got a report that the camera and then app starts to hang after 4-5 captures on a Samsung Galaxy J6+.
Have you ever heard or are able to test this?

 

Hey there. Are you releasing the photo texture when you no longer need it? You need to call `Texture2D.Destroy` on the texture.

 

Yes, I destroy the texture. Maybe the problem is that the user is spaming captures. I've added measures to prevent this now.
Something else. On Android, I try to set the photoResolution to the preview resolution but it doesn't seem to work.

This is how I do it:

Code (CSharp):

1. var previewTexture = await natCamera.StartRunning();
2.  
3. currentPhotoWidth = previewTexture.width;
4. currentPhotoHeight = previewTexture.height;
5.  
6. natCamera.photoResolution = (currentPhotoWidth, currentPhotoHeight);

Any ideas?

outputed preview res is 720x1280 and capture photo is 1458x2592 which gets reported as:

Code (CSharp):

1. NatSuite NatDevice: CameraDevice 0 photo resolution set to 2592x1458

Should I set the photoResolution before StartRunning? If so, how can I get the preview resolution beforehand?
edit: Doesn't make a difference.

 

Hm okay sounds good, let me know if you still run into the issue.

The camera isn't guaranteed to support the preview resolution, photo resolution, or frame rate that you set. So instead, NatDevice will set that property to a supported value that is closest to what you requested.

Yes, always set the preview, photo resolution, and frame rate before you call StartRunning.

 

> Resolutions are always specified in landscape format, meaning that the width is always larger than the height.

Thanks for this info. It's working now!

 

Hello again!

I'm not able to get Auto-Focus on iOS devices right. I had to implement tap-to-focus in the meanwhile.
What's the intended solution for auto focus?

 

Hey there, can you be more descriptive? Autofocus is the default, provided you have not set `cameraDevice.focusLock = true`.

 

Hi again. So, I did what you said for the resolution. Unfortunately I'm not able to fix the resolution problem. Lets fix the resolution and then talk about other problems. The resolution I chose in the code is same as a photo that was taken with the native camera app on iPhone using back camera.

I'm attaching both my code and the log from Xcode. It seems the resolution that I'm setting is being ignored. I also tried your MiniCam.CS code, and the resolution is still same as mine and not the native camera app.

Let me know what you think. Thanks!

 

The photo resolution has to be set before the camera preview is started. This rule also applies to the preview resolution and frame rate. And given how the iOS camera works, you have to 'search' for the max photo resolution by looping through some pre-defined *preview* resolutions (try 480p, 720p, 1080p) and then trying to set the photo resolution you want. Once you set the photo resolution, get its value immediately after to see what the camera actually chose. This loop will give you the preview resolution that supports the max photo resolution, based on how iOS works.

 

I tried what you said. but I'm still getting lower resolution. can you please write a sample code for how to set the resolution? The resolution I'm setting is same as the native camera (4032x3024)! it seems the only aspect ratio that works is 16:9. and still I'm not getting same res in that ratio!

 

Can you share your code? We will use that as a starting point.

 

Hello @Lanre , we are having an issue with NatDevice 1.0.2 on iOS. Specifically the issue presents itself on an iPad device where the preview image is smaller then the actual photo taken. The issue does not happen with an iPhone or Android phone. We are not able to confirm if this is an issue on an Android tablet at this time.

We used Unity 2019.4.12 and the iOS device was running 14.5, This issue occurs with both the front and back cameras.
For our application the preview and resulting photo must match. It does match on a phone but does not match on the iPad.

You may replicate this issue by starting Minicam, lining up the bottom of the preview view-port through the camera with some identifiable mark. Take the photo and then compare where the mark appears vs where it appeared during the preview.

Thank you for your prompt attention to this issue.

 

Hi @Lanre - many thanks for your work on NatDevice - it all works great but I'm having an odd issue.

When using the built-in microphone on iOS, if I enable echo cancellation, I'm seeing continuous error logs containing the following:

Code (csharp):

1. AUBuffer.h:61:GetBufferList: EXCEPTION (-1) [mPtrState == kPtrsInvalid is false]: ""

This only occurs when echo cancellation is enabled. It doesn't appear to cause any functional difference, and echo cancellation actually works very well, but the constant error logging is definitely a concern and I expect will probably cause the app to fail App Store review.

I'm using Unity 2019.4.17f1, Xcode 12.5, targeting iOS 12.1.

My test device is an iPhone 7 Plus running iOS 14.6.

Anecdotally I've seen some people online suggest this error is related to AudioUnits not being set up correctly; I'm not doing anything with AudioUnits myself so I'm wondering whether you've seen this issue or could advise on how best to go about getting this sorted?

Many thanks.

 

Hey there, this is not an issue or a bug if I understand correctly. It sounds like the preview aspect ratio is different from the photo aspect ratio. If this is what you're referring to, then this is correct. The photo resolution isn't guaranteed to have the same aspect as the preview resolution, and in most cases it won't have the same aspect. Your app shouldn't expect the aspect ratios to match.

 

As far as I understand, the error is innocuous. I've faced it myself. It does relate to AudioUnits, but from what I see online, a 'fix' would require modifying Unity's internal audio setup, which is out of reach for NatDevice. I don't know that App Review will deny the app because of the log.

EDIT: It's worth noting that the next NatDevice update will require iOS 13.

EDIT EDIT: I just looked at the NatDevice 1.1.0 code, and it uses a different API that might fix this issue. The update should be on the Asset Store in the next few days.

 

Exellent - thanks for the info @Lanre - will keep an eye on the Asset Store for the update

 

Hi @Lanre , I am not sure I follow.

When using the iOS native camera app or Natdevice on my iPhone, the preview and final captured image look identical.

When using the iOS native camera app on my iPad, the preview and final captured image look identical.

When using the Natdevice on my iPad, the preview and final captured image are not identical.

How do we achieve the same output with Natdevice on our iPad's?

Glenn

 

What do you mean by "identical"? Same aspect ratio? If so, you can try setting the exact preview and photo resolution on the camera device. But your app shouldn't expect the aspect ratios to match.

When describing issues like this, try to be as descriptive as possible. Including screenshots and any other useful information would ensure that I understand what you're describing.

 

Hi @Lanre, In my first post I documented how the issue could be duplicated. Did you follow my instructions?

 

My iPad is currently damaged so I can't test it. Can you share your images?

 

Did you ever have a fix for this too?

 

@Lanre I have uploaded a zip file containing video clips of me using my iPhone and then my iPad with Minicam app.

Thanks

 

The aspect ratio of the photo is smaller (thinner) than that of the preview. Again, your app shouldn't be predicated on the aspect ratio of the photo having a specific value.

As for the image being displayed in the MiniCam example, just set the aspect mode on the photo panel to 'Envelope Parent'. That way, the photo will cover the entire screen.

 

Our app is not dependant on the aspect ratio of the photo having a specific value, we simply need to have the preview and the actual photo be the same. The built in cameras on both the phone and the ipad do this automatically. We needed to understand why natdevice and the minicam example are handling this differently.

I will try this but I am not sure this solves the issue as the actual photo taken by the camera is larger then what is previewed in the window.

 

If you know the photo resolution you want, you can always set it:

Code (CSharp):

1. cameraDevice.photoResolution = (1920, 1080);

NatDevice will set a supported resolution that is closest to what you request.

 

Hey @Lanre

Loving the plugin so far, saved me a bunch of time.

I am having an issue with iOs resolution;

I set the preview + photo resolution (1920 x 1080) before starting the camera, now the specified resolution works on Android but on iOs it defaults to 2376 x 4224, the issue is that we are uploading these photos, so the file size is massive.

Code (CSharp):

1.  
2. m_device = m_query.currentDevice as CameraDevice;
3.            
4. var currentPhotoWidth = 1920;
5. var currentPhotoHeight = 1080;
6.            
7. m_device.previewResolution = (currentPhotoWidth, currentPhotoHeight);
8. m_device.photoResolution = (currentPhotoWidth, currentPhotoHeight);
9.            
10. var previewTexture = await m_device.StartRunning();

Is there a way to fix this?

If it is a case of iOs not supporting this resolution, how do I dynamically work out what resolution it can use?

Perhaps if there's no way to set the correct resolution on iOs, there are ways to compress the file itself.

Regards,
Ian

 

On iOS, camera devices have discrete 'formats', each one with exactly one preview resolution and a small number (3-5) of photo resolutions. NatDevice gives precedence to the preview resolution, so when you set a photo resolution, it looks in the small set of supported photo resolutions for the res that is closest to what you request (as opposed to changing the current format).

You could either upload a preview frame (instead of the photo); or write a native plugin to encode the image to HEIC in hopes of better compression.

 

Hi @Lanre, so based on Thompson11's code with a request of 1920 x 1080 you are indicating that 2376 x 4224 was the closest one found? Do you have an API call to return the available resolutions?

 

Yup. Each discrete format only supports a very small set of photo resolutions, typically between 3 and 5.

Nope, and as I've explained before, on iOS, the photo resolutions depend on the current preview resolution. The cost of exposing this far outweighs any benefits in doing so.

 

Does anyone here or @Lanre know what those 3-5 resolutions are?

I don't mind defaulting to one of them, I just need it to be a smaller size.

Regards,
Ian

 

I mentioned this earlier, but the supported photo resolutions depend on the current preview resolution, so it's not a fixed set. If you want the smallest photo resolution, set a photo resolution like (0, 0). NatDevice will find the supported resolution that is closest to what you request.

 

@Lanre, in your MiniCam example your AspectRatioFitter has a value of .5117188 in the ratio field. How did you arrive at that value?

 

Code (CSharp):

1. // Display preview texture
2. rawImage.texture = previewTexture;
3. aspectFitter.aspectRatio = (float)previewTexture.width / previewTexture.height;

MiniCam dynamically sets the aspect ratio to match the preview texture:

 

camera_settings.device.flashSupported is working, but

FlashMode is no work

My phone is

HUAWEI Mate40 Pro

HUAWEI Mate30 Pro

 

Hey there, can you share your code and the unfiltered logs from logcat in a .txt attachment?

 

Code (CSharp):

1.   public void ToggleFlashMode()
2.   {
3.     Debug.Log("ToggleFlashMode");
4.     // Only `CameraDevice` supports setting focus point, not `ICameraDevice`
5.     // if (query.currentDevice is CameraDevice device)
6.     // {
7.     //   device.flashMode = device.flashMode == FlashMode.On ? FlashMode.Off : FlashMode.On;
8.     //   flashIcon.color = device.flashMode == FlashMode.On ? Color.white : Color.gray;
9.     // }else{
10.     //   Debug.Log("NO work ToggleFlashMode");
11.     // }
12.     // Check if flash is supported
13.     if (!camera_settings.device.flashSupported)
14.     {
15.       scanning = false;
16.       StopButton.SetActive(true);
17.       camera_settings.device.StopRunning();
18.       camera_settings.device = null;
19.       Debug.Log("flash is not supported");
20.       QR_Text.GetComponent<Text>().text = "no work ";
21.       flashIcon.color = Color.gray;
22.       return;
23.     }
24.     // Toggle
25.     if (camera_settings.device.flashMode == FlashMode.On)
26.     {
27.       camera_settings.device.flashMode = FlashMode.Off;
28.       flashIcon.color = Color.gray;
29.     }
30.     else
31.     {
32.       camera_settings.device.flashMode = FlashMode.On;
33.       flashIcon.color = Color.white;
34.     }
35.   }

this is my code , how can get to .txt for my phone?

 

Here's an article on how to use logcat. Does the flash also not work when you use the MiniCam example?

 

Hi @Lanre

We are running a simple script to display a preview texture. Really similar to the example you provide. What is the expected time between the begining of the Start() function and the preview texture being displayed ?

Currently on Android (Samsung SM-T500), with the last version of NatDevice (1.1.0) it takes around 4 seconds. It seems way too long, and might be longer than the previous versions.

Is there any way to improve this?

Here is the code we use :

Code (CSharp):

1.  
2. private async void Start()
3. {
4.     // Request camera permissions
5.     if (!await MediaDeviceQuery.RequestPermissions<CameraDevice>())
6.     {
7.         Debug.LogError("User did not grant camera permissions");
8.         return;
9.     }
10.     // Create a device query for device cameras
11.     query = new MediaDeviceQuery(MediaDeviceCriteria.CameraDevice);
12.     // Start camera preview
13.     CameraDevice device = query.current as CameraDevice;
14.     Texture2D previewTexture = await device.StartRunning();
15.     Debug.Log($"Started camera preview with resolution {previewTexture.width}x{previewTexture.height}");
16.     // Display preview texture
17.     rawImage.texture = previewTexture;
18.     aspectFitter.aspectRatio = (float)previewTexture.width / previewTexture.height;
19. }
20.  

 

Hey there. Unfortunately, there's no way to speed this up. The process for opening the camera and starting the preview is async on Android, and we have no control over how soon the camera system will invoke our callbacks to get things going. On older devices, this can take some time as you've seen.

 

Hi @Lanre,

We are using NatDevice version 1.0.2 to do some pixel calculation from the camera input on iOS mobile devices (iPhones/iPad):

Texture2D previewTexture = await cameraDevice.StartRunning();

We observe a strange bug: if we are using Bluetooth earphones/speakers, at the very moment when the camera activates itself, the playing background music gets interrupted for a short duration (not problematic), then resumes with very degraded audio quality (problematic).
We do not ask for microphone permission, nor do we include or use any microphone lib/functionality.
Also, the music quality is normal prior to the camera activation. We don't have this issue when using wired (jack) headphones or external speakers.

Do you have any clue of what is happening?

 

Hm this sounds like a bug with iOS, especially because it doesn't happen with wired headphones or external speakers. Also, the likely culprit here is creating the media device query, not starting the camera preview. When the device query is created, NatDevice updates the AVAudioSession options to allow bluetooth audio devices to be discovered.

 

Thank you for your answer. As you can imagine, we use a media device query in order to retrieve the phone's camera. Is it possible to do so without using a media device query, or to prevent the AVAudioSession update mechanism, if it might help solving the problem?

 

Hi @Lanre,

I see in the MediaDeviceQuery constructor this statement

devices.AddRange(AudioDevices());

, is it the line responsible for updating the AVAudioSession options?
Since we don't use NatDevice to interact with microphones or audio devices, can I just skip this line to check if it solves the issue?

The code block I'm talking about:

Code (CSharp):

1.         /// <summary>
2.         /// Create a media device query.
3.         /// </summary>
4.         /// <param name="criterion">Criterion that devices should meet.</param>
5.         public MediaDeviceQuery (Predicate<IMediaDevice> criterion = null) {
6.             // Get media devices
7.             var devices = new List<IMediaDevice>();
8.             switch (Application.platform) {
9.                 case RuntimePlatform.Android: goto case RuntimePlatform.IPhonePlayer;
10.                 case RuntimePlatform.IPhonePlayer: devices.AddRange(AudioDevices()); devices.AddRange(CameraDevices()); break;
11.                 case RuntimePlatform.OSXEditor: goto case RuntimePlatform.WindowsPlayer;
12.                 case RuntimePlatform.OSXPlayer: goto case RuntimePlatform.WindowsPlayer;
13.                 case RuntimePlatform.WindowsEditor: goto case RuntimePlatform.WindowsPlayer;
14.                 case RuntimePlatform.WindowsPlayer: devices.AddRange(AudioDevices()); devices.AddRange(WebCamDevices()); break;
15.                 default: devices.AddRange(WebCamDevices()); break;
16.             }
17.             // Filter by provided criterion
18.             this.devices = devices.Where(device => criterion != null ? criterion(device) : true).ToArray();
19.         }

 

Yup that's correct. You can comment out that piece of code, and NatDevice won't find audio devices (and won't modify the AVAudioSession).

 

Thank you, I'll try it out and let you know if it solves the issue!

 

Hi @Lanre,
Commenting this line solved the issue, thank you again for your help!