Search Unity

  1. Unity 6 Preview is now available. To find out what's new, have a look at our Unity 6 Preview blog post.
    Dismiss Notice
  2. Unity is excited to announce that we will be collaborating with TheXPlace for a summer game jam from June 13 - June 19. Learn more.
    Dismiss Notice

[RELEASED] OpenCV for Unity

Discussion in 'Assets and Asset Store' started by EnoxSoftware, Oct 30, 2014.

  1. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,568
    Thank you for your inquiry.

    This problem is a bug in Unity itself.
    https://issuetracker.unity3d.com/is...s32-returns-black-pixels-when-in-webgl-player (Voting for this issue may make fixing this bug a priority.)
    At this time if you use WebCamTexture in your project I would recommend building with Unity2021.
     
  2. Idealisers

    Idealisers

    Joined:
    Jun 26, 2014
    Posts:
    10
  3. tlt

    tlt

    Joined:
    Nov 16, 2011
    Posts:
    12
    Hi,

    I'm wondering if the DNN module supports GPU inference or any form of GPU acceleration. I intend to target the Android platform.
     
  4. Paltr

    Paltr

    Joined:
    Jun 19, 2017
    Posts:
    3
    Hi! Is there some way to create Mat from subrect of WebCamTexture, like Utils.webCamTextureToMat but that accepts the subrect to copy?
     
  5. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,568
    Unfortunately opencv itself does not support GPU acceleration for the Android platform.
     
    tlt likes this.
  6. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,568
    There is no API to directly convert a specific Rect in WebCamTexture to Mat.
    As shown in the following code, it is possible to retrieve the Mat of a specific Rect of the transformed Mat from WebCamTexture.
    Code (CSharp):
    1.         void Update()
    2.         {
    3.             // Check if the web camera is playing and if a new frame was updated
    4.             if (webCamTextureToMatHelper.IsPlaying() && webCamTextureToMatHelper.DidUpdateThisFrame())
    5.             {
    6.                 // Retrieve the current frame as a Mat object
    7.                 Mat rgbaMat = webCamTextureToMatHelper.GetMat();
    8.                 roiMat = new Mat( rgbaMat, new Rect( 10, 10, 200, 300));
    9.  
     
    Paltr likes this.
  7. jibincj

    jibincj

    Joined:
    Mar 28, 2024
    Posts:
    1
    Hi,
    I want to do the following image processing on Hololens 2 camera feed.
    1. Gaussian Blur
    2. color to gray image
    3. Laplacian filter
    4. scharr edge detection
    5. CV threshold to convert to black & white image.
      I am not able to find this features in trial version.
      Its available in paid version?
      Will I get full access to openCV scripts?
     
  8. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,568
    On the UWP platform, OpenCVForUnity does not support modules that depend on the dnn module, but all other modules are supported.
    https://enoxsoftware.com/opencvforunity/documentation/support-modules/
    You can check the API reference to see if the following image processing is supported by OpenCVForUnity.
    https://enoxsoftware.github.io/OpenCVForUnity/doc/html/annotated.html
    Gaussian Blur
    color to gray image
    Laplacian filter
    scharr edge detection
    CV threshold to convert to black & white image.
     
  9. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    1,085
    Is there support for VisionOS Yet?
     
  10. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,568
    OpenCVForUnity support for visionOS is already under development, and while it has been confirmed to work in visionos simulator, I do not have the actual device, so I have not been able to confirm that it works on the actual device.

    If you have Apple Vision Pro, you may be interested in trying the beta version of the visionOS build. If you would like to try, please fill out the form below with your OpenCVForUnity InvoiceNo and I will send you the beta version.
    https://enoxsoftware.com/opencvforunity/contact/other-inquiry/
     
  11. Futurristic

    Futurristic

    Joined:
    Jun 21, 2016
    Posts:
    48
    Hello, Your assets is really very good. I have one query. I want to make a vr app using meta quest 3 with passthrough mode. Does your sdk support that?
    One more query that I want my app to detect fruits and vegetables too with their names, Is that possible ? How do i add feature to detect more objects type too ?
     
  12. Futurristic

    Futurristic

    Joined:
    Jun 21, 2016
    Posts:
    48
    does your plugin supports custom YOLO models ?
     
  13. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,568
    I think it is possible to create a pass-through app that just synthesizes the background by using the UnitySDK functionality in meta quest. However, it is a difficult path to create an app that adds OpenCV image processing because of the limited pixelbuffer acquisition of the camera input video.

    Our published YOLO model example also supports running a training model with custom classes to detect.
     
  14. Olivier-Tache

    Olivier-Tache

    Joined:
    Feb 27, 2015
    Posts:
    21
    Hi @EnoxSoftware , I can't find the Stitcher class in OpenCV For Unity (2.5.9). Am I missing something or is it unsupported at the moment?
     
    Last edited: Apr 5, 2024
  15. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,568
    Olivier-Tache likes this.
  16. Paltr

    Paltr

    Joined:
    Jun 19, 2017
    Posts:
    3
    Hi @EnoxSoftware , I have several questions about VideoWriter:
    0. I'm working under Win 11 and trying to use ffmpeg backend to write a video. I downloaded the latest ffmpeg, added it's bit directory to the PATH and tried to open VideoWriter like this:
    - m_VideoWriter.open(this.VideoFilepath, Videoio.CAP_FFMPEG, VideoWriter.fourcc('a', 'v', 'c', '1'), 30, size)
    - m_VideoWriter.open(this.VideoFilepath, Videoio.CAP_FFMPEG, VideoWriter.fourcc('m', 'p', 'e', 'g'), 30, size)
    but both returned false.
    No messages were found in the Console, Utils.setDebugMode doesn't call callback if set.
    Or ffmpeg must be provided as dll or somehow else?
    1. Are CAP_AVFOUNDATION and CAP_ANDROID implemented? What fourcc must be provided to make them work? Are the backends hardware accelerated?
     
  17. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,568
    Thank you for your inquiry.

    1)Download "OpenCV for Windows Version 4.9.0"(http://opencv.org/downloads.html).
    2)Copy "opencv/build/bin/opencv_videoio_ffmpeg490_64.dll" to "OpenCVForUnity/Plugins" Folder.
    3)Setup "opencv_videoio_ffmpeg490_64.dll" Inspector Settings.
    ffmpeg_dll_setup.PNG
    unnamed.png
    You can use ffmpeg for the backend by setting Videoio.CAP_FFMPEG in the apiPreference as shown in the code below. The ffmpeg backend may allow you to specify more file formats than the default backend.

    VideoWriterExample
    Code (CSharp):
    1. writer.open(savePath, Videoio.CAP_FFMPEG, VideoWriter.fourcc('M', 'J', 'P', 'G'), 30, new Size((int)captureRectPixel.width, (int)captureRectPixel.height));
    2. Debug.Log("writer.getBackendName(): " + writer.getBackendName());
    The open() method of VideoWriter.fourcc('a', 'v', 'c', '1') did not succeed in the project set up using the above procedure, but VideoWriter.fourcc('m', 'p', 'e', 'g') did without problems.
    Code (CSharp):
    1. writer.open(savePath, Videoio.CAP_FFMPEG, VideoWriter.fourcc('m', 'p', 'e', 'g'), 30, new Size((int)captureRectPixel.width, (int)captureRectPixel.height));

    CAP_AVFOUNDATION would work on macOS and iOS platforms, and CAP_ANDROID on Android platforms. Not sure which fourcc each supports.
     
    Paltr likes this.
  18. Futurristic

    Futurristic

    Joined:
    Jun 21, 2016
    Posts:
    48
    Hi, I am setting up ObjectDetectionYOLO scene for meta quest 3 mixed reality mode. When i running the build scene nothing happens. Can you help me how to setup ObjectDetection inside meta quest 3. I am using unity. Thanks!
     
  19. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,568
    As I mentioned in my previous reply, MetaQuest3 does not allow ordinary developers to obtain pixel information from camera images. This restriction seems to be meta's policy from the viewpoint of privacy protection.
    Therefore, it is not easy to develop image processing applications such as object detection that require pixel data from the camera image.
    If you really want to do that, you will need to use the trick of connecting an external camera, as shown in the example on the following page.
    https://qiita.com/vent0908/items/9e13ca393c5c4a9c987
     
  20. Futurristic

    Futurristic

    Joined:
    Jun 21, 2016
    Posts:
    48
    Ok. thanks. And how do we connect external camera with quest device ? Any idea ? And the link https://qiita.com/vent0908/items/9e13ca393c5c4a9c987 you given is not working.....
     
  21. Futurristic

    Futurristic

    Joined:
    Jun 21, 2016
    Posts:
    48
    I found a video on youtube
    who is doing object detection with meta quest device..... How he can achieve that whereas quest camera feed is not available for ordinary users..? Thanks.
     
  22. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,568

    Unfortunately, we do not know the details of that YouTube video.

    Here is the correct URL:
    https://qiita.com/vent0908/items/9e13ca393c5c4a9c9874
     
  23. fordtimelord

    fordtimelord

    Joined:
    Nov 3, 2020
    Posts:
    21
    I was looking into this for the sale, having had it saved for a while, and I see that the playmaker support has disappeared. What happened there? Do the old actions still work? Thanks!
     
  24. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,568
    Since VisualScripting is now supported as a standard feature of Unity, the visual scripting tool supported by OpenCVForUnity has been changed from PlayMaker to VisualScripting. Older versions of "PlayMakerActions for OpenCVforUnity" will not work with the latest version of "OpenCVForUnity".
    https://github.com/EnoxSoftware/VisualScriptingWithOpenCVForUnityExample
     
  25. Falondrian

    Falondrian

    Joined:
    Nov 1, 2017
    Posts:
    74
    How well does this asset work together with ARFoundation 6.0.1 and how noticable is the perf impact of the ar cam texture conversion to the OpenCV format?
     
  26. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,568
    The following projects exist as examples of integrations between our assets and ARFoundation.
    https://github.com/EnoxSoftware/ARFoundationWithOpenCVForUnityExample
    However, this was updated a year ago and has not been tested to work properly with the latest ARFoundation version. Probably not a problem unless the API has changed significantly.
    As for the performance of the camera texture to OpenCVMat conversion, there is a demo app apk file available for you to try.
     
    Falondrian likes this.
  27. theolagendijk

    theolagendijk

    Joined:
    Nov 12, 2014
    Posts:
    118
    @EnoxSoftware Thank you for your great OpenCVForUnity asset and thanks for the PoseSkeletonVisualiser ( Version 2.5.9 and higher ). I was wondering if you could also do a code example of animating the bones of a 3D rigged hand model based on hand pose estimations. So that I could have a virtual 3D hand mimic a real hand. The asset store has a lot of rigged 3D models of hands but it's not immediately clear to me what would be the best approach to taking the MediaPipe's hand pose estimation and using them to drive a 3D hand model. Any thoughts or implementation suggestions?
     
  28. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,568

    Unfortunately, we have not created an example of an animated 3D hand model based on our media pipe hand pose estimation.
    Technically, we believe it is feasible by modifying the script of the following project.

    https://github.com/digital-standard...cuda/blob/master/Assets/Scripts/VNectModel.cs
    https://github.com/digital-standard/ThreeDPoseUnityBarracuda
     
    theolagendijk likes this.
  29. sgmdev

    sgmdev

    Joined:
    Jul 26, 2023
    Posts:
    14
    Hello! Can this asset be used to record video with sound & take pictures with camera?
    I'm working on a mobile game (Android & iOS) where users can record a short video with the camera and/or take pictures and save them to their device. Is this possible?
     
  30. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,568
    Our OpenCVForUnity plug-in mainly provides matrix calculation and image processing functions.
    For camera and video capture, playback and storage on Android/iOS devices, we recommend using the following plug-ins that specialize in such functions.

    VideoKit : https://github.com/natmlx/videokit
    AVPro : https://renderheads.com/products/

    Although old, an example of a video recording and image processing application that links “NatCorder,” the predecessor of “VideoKit,” with OpenCVForUnity exists. We hope this will be helpful.
    https://github.com/EnoxSoftware/NatCorderWithOpenCVForUnityExample
     
    Chris-RH likes this.