Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

NatDevice - Media Device API

Discussion in 'Assets and Asset Store' started by Lanre, Dec 17, 2015.

?

Should we add exposure controls in v1.3? This means dropping support for iOS 7

Poll closed Jun 10, 2016.
  1. Yes

    9 vote(s)
    75.0%
  2. No

    3 vote(s)
    25.0%
  1. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Hey there. I got your email. Can you share the full logs from logcat in a .txt attachment? Is your app using autorotation, or is it fixed to portrait in Player Settings?
     
  2. Anlasa

    Anlasa

    Joined:
    May 27, 2013
    Posts:
    2
    Hi,

    I've just imported NatDevice to a new project, I played the MiniCam example but RawImage remain blank and when I press any button (Photo, Toggle flash and Switch camera) next error appears:

    NullReferenceException: Object reference not set to an instance of an object
    NatSuite.Examples.MiniCam+<SwitchCamera>d__9.MoveNext () (at Assets/NatSuite/Examples/MiniCam/MiniCam.cs:75)

    I also created a new scene with the HelloCam script from Getting started guide, but it didn't work too (it didn't show any errors but neither the camera preview).

    I'm using Unity 2019.4.1f1

    Thanks,
     
  3. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Make sure that you have granted permissions to the app. Check the logs for clues.
     
  4. AdminOh

    AdminOh

    Joined:
    Feb 11, 2016
    Posts:
    23
    Hello, it was actually an error in our project. We didn't fully checked our code and prefabs after migrating from webcamtexture to your plugin. We have now fixed the issue.
     
    Lanre likes this.
  5. musicdeveloper

    musicdeveloper

    Joined:
    Oct 16, 2019
    Posts:
    68
    Hello Lanre,

    After updating NatDevice to Version 1.0.2, my app's Windows users started reporting that they can no longer connect to Audio Inputs.

    After investigating, it looks like the following line of code stalls and does not return a value on Windows Standalone.

    Code (CSharp):
    1. async void GetMicPermissions()
    2.     {
    3.         // Request mic permissions
    4.         if (await MediaDeviceQuery.RequestPermissions<AudioDevice>())
    5.         {

    Perhaps this could be caused by the changes to the way Permissions are handled in the new update?
    By the way, I'm extremely grateful that that was fixed for MacOS iOS!

    This is an urgent problem for my app - the only immediate solution would seem to be to roll back NatDevice - but then that's going to cause the MacOS / iOS problem to reappear.

    My email is: visualmusicdesign@gmail.com

    Thank you!
     
  6. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    I just sent you an email with the fix. Thanks for bring this to my attention.
     
    musicdeveloper likes this.
  7. xharkx

    xharkx

    Joined:
    May 17, 2017
    Posts:
    26
    How can I change previewResolution?
    I always get preview resolution 1920x1080. I can not change.

    I modified your example scene code.
    Minicam.cs

    var device = query.currentDevice as ICameraDevice;
    device.previewResolution = (1920, 1440);
    Debug.Log($"Set camera preview with resolution {device.previewResolution.width}x{device.previewResolution.height}");
    //Output->1440x1080
    var previewTexture = await device.StartRunning();
    Debug.Log($"Started camera preview with resolution {previewTexture.width}x{previewTexture.height}");
    //Output->1920x1080


    Unity2019.3.2.f1
    NatDevice Version 1.0.2
    iphone 7 or iphone 11 pro
     
  8. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Can you share the logs from Xcode?
     
  9. MasterControlProgram

    MasterControlProgram

    Joined:
    Apr 18, 2015
    Posts:
    48
    Was trying to track down some memory leaks in my project and noticed that I was collecting "NatDevice WebCameraDevice Helper" game objects in DontDestroyOnLoad objects. It would create a new one every time I started the camera but never destroyed when I stopped the camera.

    Looked in a bit and the StopRunning() function in WebCameraDevice.cs is destroying the Component and not the object itself. I suspect the fix is changing like 79 to:

    Code (CSharp):
    1. WebCameraDeviceAttachment.Destroy(attachment.gameObject);
     
  10. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Thanks for this. I've made the change.
     
  11. vcasas

    vcasas

    Joined:
    Oct 6, 2017
    Posts:
    7
    Hi, I am making a computer vision app and I need to access in c++ the intptr of the frames preview in real time. What is the fastest way using this plugin?
     
  12. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Access the preview texture's raw data:
    Code (CSharp):
    1. var pixelBuffer = (IntPtr)Unity.Collections.LowLevel.Unsafe.NativeArrayUnsafeUtility.GetUnsafePtr(previewTexture.GetRawTextureData<byte>());
     
  13. AdminOh

    AdminOh

    Joined:
    Feb 11, 2016
    Posts:
    23
    Hello @Lanre

    We have tested your plugin on a new tablet and it seems to have some issues. The tablet is a Samsung Galaxy Tab A T580. The script running is pretty simple and is mostly taken from one of your example : https://pastebin.com/27aGybg0

    After a while, and without doing anything, the framerate gradually slow down, making the "CapturePhoto" button almost unusable (the CapturePhoto function seem to take a while to execute).

    One of the most bizarre thing is that the framerate goes up when I physically hide the camera view. When obstructing the physical camera, there is an error that appears in logcat (!).

    Here are two differents logs from logcat :
    - The first one when nothing is obstructing the camera https://pastebin.com/EhZ2vgUP
    - The second one when something is obstructing the camera https://pastebin.com/vpi9rUD7

    I hope this is enough information. If not, I'll gladly send anything you might need to help us solve this issue.

    Thanks!
     
  14. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    How are you calling `CapturePhoto`, and are you destroying the photo texture when you no longer need it?
    As far as I can tell this error is device-specific. What seems to be going on is that the device is falsely reporting that there is a new camera frame, so when NatDevice receives that report and tries to use the frame, the frame is missing (hence the null reference). Have you tried running on a different, newer device?
     
  15. AdminOh

    AdminOh

    Joined:
    Feb 11, 2016
    Posts:
    23
    The texture is indeed destroyed when it is not needed anymore. Here is a snippet of our code that is called when pressing the photo button : https://pastebin.com/SzHZG6xr
    - "capturePhotoIsRunning" seems to be needed so we dont call the function while it is running.
    - "Destroy(savedCameraTexture);" cleans up the preview texture if it was not properly destroyed.
    - "#if UNITY_EDITOR", allows us to capture a picture in the editor, as CapturePhoto only works on devices.

    We have tried on another device (a Samsung T550), that is older, and it seems to work better (even though the camera preview framerate is definitely lower than when using the native android camera app).

    The T580 is not a really old device (2016) and we unfortunately can't change the model that is used for this application.
     
  16. andvasconcel

    andvasconcel

    Joined:
    Jan 16, 2018
    Posts:
    2
    Hello @Lanre , when I open "NatSuite MiniCam scene" on unity editor, an error occurred.


    Code (CSharp):
    1.  
    2. NullReferenceException: Object reference not set to an instance of an object
    3. NatSuite.Examples.MiniCam+<SwitchCamera>d__9.MoveNext () (at Assets/NatSuite/Examples/MiniCam/MiniCam.cs:75)
    4. --- End of stack trace from previous location where exception was thrown ---
    5. System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () (at <fb001e01371b4adca20013e0ac763896>:0)
    6. System.Runtime.CompilerServices.AsyncMethodBuilderCore+<>c.<ThrowAsync>b__6_0 (System.Object state) (at <fb001e01371b4adca20013e0ac763896>:0)
    7. UnityEngine.UnitySynchronizationContext+WorkRequest.Invoke () (at <3dc54541a2574ac7826a004a212a4332>:0)
    8. UnityEngine.UnitySynchronizationContext:ExecuteTasks()
    9.  

    When I generate APK and install, works perfectly.


    I use windows10 - 64bits - Unity2019.4.1f1 and Visual studio community 2019 (up to date).

    Please, help me.
     
  17. andvasconcel

    andvasconcel

    Joined:
    Jan 16, 2018
    Posts:
    2

    Hello,
    I discovered the bug. When I used "Simulator tab" instead "Game tab", the crash occurred.
     
    Lanre likes this.
  18. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Have you tried reducing the preview resolution? This could be one reason why the frame rate goes up when the camera view is hidden.
    The reason why the NatDevice frame rate is lower than the native camera app is that unlike the native camera app which can keep the camera preview entirely on the GPU (for display only), NatDevice has to readback the preview data from the GPU on every frame (to support computer vision use cases, and Vulkan). This operation is expensive, but it practically unnoticeable on newer devices with better performance.
     
  19. leavittx

    leavittx

    Joined:
    Dec 27, 2013
    Posts:
    176
    Hi! Is it possible to change web cam settings (like auto/manual white balance) on Windows/OS X platforms?
    E.g. on windows the dialog looks like this usually:
     
  20. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    NatDevice does not have a native camera implementation on standalone platforms; it falls back to WebCamTexture. Besides NatDevice does not offer any controls for the image visual quality (brightness, contrast, hue, and so on); it only offers controls for the camera (focus, zoom, exposure, and so on).
     
  21. AdminOh

    AdminOh

    Joined:
    Feb 11, 2016
    Posts:
    23
    Updating our OS seem to have improved the behaviour, the log is cleaner. But we still observe a framerate drop the longer the camera is activated. Less than before, but still observable. And when the framerate is lower, the "CapturePhoto" is also less reponsive (sometimes we have to tap multiple tap on the button).


    We also have another issue when trying to implement the flash functionality. We retrieved the code from you MiniCam example :

    Code (CSharp):
    1.  
    2. public void ToggleFlashMode()
    3. {
    4.     // Only `CameraDevice` supports setting focus point, not `ICameraDevice`
    5.     if (deviceQuery.currentDevice is CameraDevice device)
    6.     {
    7.         device.flashMode = device.flashMode == FlashMode.On ? FlashMode.Off : FlashMode.On;
    8.         flashIcon.color = device.flashMode == FlashMode.On ? Color.white : Color.gray;
    9.     }
    10. }
    The toggle work properly, but the physical flash does not always work when calling CapturePhoto. Here is the behaviour we observed:
    - After toggling ON/OFF the flash, the icon is in the proper color (thus, "device.flashMode", is properly assigned).
    - Calling Minicam.FocusCamera() always turn on the flash when device.flashMode = FlashMode.On.
    - Calling CapturePhoto() does not turn on the flash, UNLESS FocusCamera has been called before.

    First example:
    - Toggling on the flash.
    - Calling FocusCamera (= first flash).
    - Calling CapturePhoto (=second flash).
    - Calling CapturePhoto a second time (=no flash).

    Second example:
    - Toggling on the flash.
    - Calling CapturePhoto (=no flash).

    Please let us know if you can help us fix this and if you need more information.
     
    Lanre likes this.
  22. LostPanda

    LostPanda

    Joined:
    Apr 5, 2013
    Posts:
    173
    I remember AVPro Live Camera can do.
     
    leavittx and Lanre like this.
  23. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Are you able to reproduce this on other devices? What device are you testing on?
    I'll look into this. Interestingly enough I can't reproduce it on my Samsung Galaxy device.
     
  24. AdminOh

    AdminOh

    Joined:
    Feb 11, 2016
    Posts:
    23
    It is happening on the Samsung T580. Our Samsung T550 seem to have overall better performance, but we might need to run it for a longer period of time before seeing issues.

    And this issue is happening on the Samsung T580 (the T550 doesn't have a flash so we could not test it here).
     
  25. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Are you able to profile the app running on the T580? I'm curious to know whether it has something to do with memory consumption, though I have no reason to believe there's a memory leak coming from NatDevice. If you can make sure to use the Android Studio profiler, not the Unity profiler.
     
  26. Liviuss

    Liviuss

    Joined:
    Apr 2, 2014
    Posts:
    101
    Hi, we just upgraded from NatCam to NatDevice and we are trying to build the MiniCam scene for UWP. We don't get any errors bu application is not requesting permission. We use unity 2020.1 and win10x64. With NatCam it requested permissions. In console we don't have any errors. Any advice? Thanks.
    P.S. Permissions are set in publishing section and also are present in VS solution.
     
  27. Liviuss

    Liviuss

    Joined:
    Apr 2, 2014
    Posts:
    101
    Just an update, by switching to Unity WebCamTexture with:
    Code (CSharp):
    1. IEnumerator Start()
    2.     {
    3.        
    4.         yield return Application.RequestUserAuthorization(UserAuthorization.WebCam);
    5.         if (Application.HasUserAuthorization(UserAuthorization.WebCam))
    6.         {
    7.             Debug.Log("Permission granted");
    8.  
    9.             //Logic here
    10.         }
    11.         else
    12.         {
    13.             Debug.Log("Permission not granted");
    14.         }
    15.  
    16.     }
    It is working fine and ask permission dialog is displayed.
     
  28. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Were you using MediaDeviceQuery.RequestPermissions? NatDevice doesn't support UWP, instead it will fall back to WebCamTexture. I'll add a fix to MediaDeviceQuery to request permissions on UWP.
     
  29. kuailemario

    kuailemario

    Joined:
    Mar 19, 2020
    Posts:
    25
    Hello, may I ask, I apply for camera permission in Android, and then call NatCam.Play, I got
    NatCam.activeCamera = null
    after I have obtained the camera permission.

    How should I reset NatCam’s activeCamera?
     
  30. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    You'll need to upgrade to the latest NatDevice. You're using a far outdated version of NatCam, and NatCam has been deprecated.
     
  31. neomaxdeveloper

    neomaxdeveloper

    Joined:
    Jul 5, 2020
    Posts:
    1
    Even if you enter previewResolution (640, 480) on iOS, 1920,1080 is output after StartRunning (). Can you modify it?
     
  32. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    I have an open issue for this. I'll look into it this weekend and get back to you.
     
  33. neosca

    neosca

    Joined:
    Jul 1, 2014
    Posts:
    11
    @Lanre
    NatDevice.

    Sent email as well.

    The camera zoom functionality is working fine on iOS devices however the same code is not working on Android.

    Earlier I was using NatCam. I thought it might be a problem with the plugin since it is deprecated now, so I purchased NatDevice and am facing the same issue with it as well.

    Code (CSharp):
    1. //I am accessing camera like this -
    2.  
    3.         NatSuite.Devices.CameraDevice natCameraDevice;
    4.  
    5.  
    6.             var criterion = MediaDeviceQuery.Criteria.RearFacing;
    7.             var query = new MediaDeviceQuery(criterion);
    8.             natCameraDevice = query.currentDevice as CameraDevice;
    9.  
    10. //setting the zoom like this.
    11.  
    12.             natCameraDevice.zoomRatio = ZoomSlider.value;
     
  34. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Can you share the logs in a .txt attachment? You'll have to check what the camera's zoom range is.
     
  35. neosca

    neosca

    Joined:
    Jul 1, 2014
    Posts:
    11
    I am debugging the camera zoom range. Minimum is 1 and maximum is 4.
    I have just shared my entire Unity Project with you on email. On the email listed in unity store. this one -- yusuf@natsuite.io
     
  36. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Your code is incorrect. You are using WebCamTexture to start and run the camera, but trying to use NatDevice to control it. You have to use either NatDevice or WebCamTexture--not both. You never actually start the NatDevice camera.
     
  37. neosca

    neosca

    Joined:
    Jul 1, 2014
    Posts:
    11
    I see. Thank you for the explanation. I will use NatDevice instead of webcamtexture.

    Can you please provide an example code snippet of NatCorder (MP4Recorder class) working with NatDevice. I want to record camera video same as I was doing with webcamtexture.
     
  38. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    All you need to do is swap out the WebCamTexture code with NatDevice. There aren't any steps needed to integrate NatCorder and NatDevice; they work in parallel. Simply start the camera and display it on your RawImage. You can follow the NatDevice examples on how to do this.
     
  39. mahna3411

    mahna3411

    Joined:
    Dec 11, 2018
    Posts:
    39
    hello ,

    There is a problem with the camera output via mobile phone below
    Image collected.
    please check it .

    Res Screen : 720 * 1280
    Res Camera : 720 * 1280
    mobile : Samsung galexy j710


    bug.jpg test.jpg
     

    Attached Files:

  40. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Hm that's weird. Can you share the full unfiltered logs from logcat in a .txt attachment?
     
  41. mahna3411

    mahna3411

    Joined:
    Dec 11, 2018
    Posts:
    39
    No unfortunately, this mobile is one of the users and is not with me.
    I can only send the apk file and ask the result.
     
  42. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Does your RawImage have an aspect ratio fitter?
     
  43. mahna3411

    mahna3411

    Joined:
    Dec 11, 2018
    Posts:
    39
    The uploaded image is exactly the settings on the desired mobile.
     

    Attached Files:

  44. mahna3411

    mahna3411

    Joined:
    Dec 11, 2018
    Posts:
    39
    This miniCam Scene , and I have not changed it
     
  45. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    You should set the aspect ratio to whatever the camera preview's aspect is at runtime (use the width and height of the preview texture). Don't hard-code a value in the editor.
     
  46. mahna3411

    mahna3411

    Joined:
    Dec 11, 2018
    Posts:
    39
    yes ,
    And so it is
    previewAspectFitter.aspectRatio = previewTexture.width / (float)previewTexture.height;

    I just wanted to say that the result is this float
     
  47. mahna3411

    mahna3411

    Joined:
    Dec 11, 2018
    Posts:
    39
    If you have doubts about my output, please send an output and I will send you the result on this mobile
     
  48. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    In this case is there anyway you can have your user send you logs? Without logs I have no way of knowing what the problem is.
     
  49. mahna3411

    mahna3411

    Joined:
    Dec 11, 2018
    Posts:
    39
    Do I have to connect this mobile phone to the unity editor with a cable for the log?
     
  50. Lanre

    Lanre

    Joined:
    Dec 26, 2013
    Posts:
    3,975
    Not the Unity editor. You'll need to use the logcat tool that is bundled with the Android SDK.