Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Join us on March 30, 2023, between 5 am & 1 pm EST, in the Performance Profiling Dev Blitz Day 2023 - Q&A forum and Discord where you can connect with our teams behind the Memory and CPU Profilers.
    Dismiss Notice

[RELEASED] OpenCV for Unity

Discussion in 'Assets and Asset Store' started by EnoxSoftware, Oct 30, 2014.

  1. alexmccraw

    alexmccraw

    Joined:
    Oct 31, 2019
    Posts:
    1
    @EnoxSoftware I can't get the bilateral filter operation to actually work, I'm trying to use it in a realtime image processing pipeline but to verify it was not performing the operation I created an editor script that loads an image, attempts to apply the filter, and saves a copy of the image. I verified that the input and output images are identical, I'm not sure why this operation in particular is failing as I'm using plenty of other Imgproc methods succesfully. Below is my test code as well as the input and output image.

    Code (CSharp):
    1.     [ContextMenu("Test")]
    2.     void Test()
    3.     {
    4.         Mat image = Imgcodecs.imread("[OBFUSCATED]\\cat.jpg");
    5.         Imgproc.bilateralFilter(image, image, 9, 150, 150);
    6.         Imgcodecs.imwrite("[OBFUSCATED]\\bilateral.jpg", image);
    7.     }
    cat.jpg
    Original image

    bilateral.jpg
    Filtered image
     
  2. ibompuis

    ibompuis

    Joined:
    Sep 13, 2012
    Posts:
    99
    Last test, it's a mistery :D:D:D

    Just new photo (not trained) converted to jpg and voila...
    I don't know what happen with all other photo, format ? compression ? or ???
    Image_10-02-2023_à_10.39.jpg
     
  3. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,498
    Just to be sure, I have to ask, are the defects you are currently encountering that only occur on specific models the result of testing the YOLOv5 example (v1.0.2) on a clean project?
    Several of your recent posts seem to be about modified code.
     
  4. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,498
    Curiously, OpenCV seems to generate an internal error if the src and dst of the bilateralFilter function are the same mat.
    It is necessary to temporarily prepare a mat of the same size for output.
    Code (CSharp):
    1.  
    2.     [ContextMenu("Test")]
    3.     void Test()
    4.     {
    5.         Mat image = Imgcodecs.imread("[OBFUSCATED]\\cat.jpg");
    6.         Mat dst = image.clone();
    7.         Imgproc.bilateralFilter(image, dst, 9, 150, 150);
    8.         dst.copyTo(image);
    9.         Imgcodecs.imwrite("[OBFUSCATED]\\bilateral.jpg", image);
    10.     }
    11.  
    This error can be confirmed by enabling debug mode.
    Code (CSharp):
    1.  
    2. //if true, The error log of the Native side OpenCV will be displayed on the Unity Editor Console.
    3. Utils.setDebugMode(true);
    4.  
    5. Imgproc.bilateralFilter(image, image, 9, 150, 150);
    6.  
    7. Utils.setDebugMode(false);
    8. //
    9. imgproc::bilateralFilter_11() : OpenCV(4.7.0-dev) C:\Users\satoo\Desktop\opencv\modules\imgproc\src\bilateral_filter.dispatch.cpp:166: error: (-215:Assertion failed) (src.type() == CV_8UC1 || src.type() == CV_8UC3) && src.data != dst.data in function 'cv::bilateralFilter_8u'
    10. //
    11.  
     
    Last edited: Feb 10, 2023
  5. ibompuis

    ibompuis

    Joined:
    Sep 13, 2012
    Posts:
    99
    Clean project or modified code is same problem with the model, like I said in my previous post all photo used in model not work, only new photo... (my last post with good result is modified code and new photo)
     
  6. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,498
    Hmmm, that is really strange behavior.
    I can't think of a cause or solution...
     
  7. WayneVenter

    WayneVenter

    Joined:
    May 8, 2019
    Posts:
    51
    @ibompuis don't do this
    Code (CSharp):
    1. Imgcodecs.imread(Application.persistentDataPath+"/test.jpg");
    Do this
    Code (CSharp):
    1. Imgcodecs.imread(Path.Combine(Application.persistentDataPath, "test.jpg"));
     
    ibompuis likes this.
  8. gromiczek

    gromiczek

    Joined:
    Jul 19, 2019
    Posts:
    21
    Hi - thanks for all your work and support on OpenCV for Unity!

    I am working with a RealSense 455d camera and have used the RealSense SDK to do some image processing on the camera input and I now want to pass that to the OpenCV of Unity SDK to do perform blob detection. Right now the output from the RealSense pipeline is a binding to a RawImage.texture on a canvas like so:

    texture_binding_in_unity.jpg

    However, now I'd like to send this output to something like your `
    MultiObjectTrackingBasedOnColorExample` script, but I'm unsure how to connect the dots between the `RawImage.texture` I have and the process in your script. It appears that the `
    WebCamTextureToMatHelper` is a key part of this interface, but how could I connect my `RawImage.texture` output to this or some other part of the script? Thanks!
     
  9. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,498
    Unfortunately I don't have a RealSence so I haven't tried it, but I did find the following examples.
    https://qiita.com/UnagiHuman/items/f877da98d7600e3d579c
    According to this example article, integration with OpenCVForUnity seems to be possible.
     
    gromiczek likes this.
  10. Quadropups

    Quadropups

    Joined:
    May 23, 2017
    Posts:
    18
    Hello! Quick question. Does this API allow turn on/off phone flashlight while AR is active?
     
  11. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,498
    OpenCV does not have the ability to control mobile phone flashlight.
    Third party assets must be linked.
     
  12. WayneVenter

    WayneVenter

    Joined:
    May 8, 2019
    Posts:
    51
    @EnoxSoftware hi guys, back on the DNN Object Detection track, thank you for adding Yolov7, but you did not include Yolov5 in the distribution / release version, also you are using .weights file in the Yolov7 example and not ONNX, the main purpose of my original request was to use ONNX files.

    Is the issues that you can't covert the weigts file to ONNX, because the main use case is to take Pytorch .pt files to .ONNX, this works 100% for Yolov5 example you created.

    I did some digging on this and found the following:
    How export to onnx be used by opencv?
    https://github.com/WongKinYiu/yolov7/issues/49

    I also read your notes about ONNX in the example, if you can have a look at this alternative solution as per the above mentioned issue.

    Opencv dnn success run custom yolov7
    1. reparam trained best.pt
    https://github.com/WongKinYiu/yolov7/blob/main/tools/reparameterization.ipynb
    nc = 1 #change with your nc
    model = Model('cfg/deploy/yolov7.yaml', ch=3, nc=nc).to(device)
    for i in range( (nc + 5) * 3):

    2 use branch https://github.com/WongKinYiu/yolov7/tree/u5 to export onnx model
    3 run with the same as yolov5 6.1
    https://github.com/VITA-Alchemy/yolov5_6.0_opencvdnn_python/blob/main/main_dnn.py
     
    Last edited: Mar 1, 2023
  13. valentin-simian

    valentin-simian

    Joined:
    Nov 16, 2012
    Posts:
    49
    Hello, I'm getting an almost black screen from a WebGL build on Chrome. To be specific, the texture shows one frame that was captured, then just goes black. Building HandPoseEstimationExample using SimpleWebServer if it matters.
     
  14. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,498
    There seems to be a bug in Unity 2022.2.0f1 and above where webcamTexture.didUpdateThisFrame only updates the first frame in WebGL build.
    Could you try a build with Unity2021?
     
  15. Rayo3

    Rayo3

    Joined:
    May 21, 2018
    Posts:
    1
    Hello, the SimpleBlobExample doesn't seem to work. In Unity and the OpenGL example I get an error: "error: (-5:Bad argument) minDistBetweenBlobs>0 in function 'validateParameters'". When setting minDistBetweenBlobs=10, minArea=1, maxArea=100000, minCircularity=1, maxCircularity=100000, minInertiaRatio=1, maxInertiaRatio=100000, minConvexity=1, maxConvexity= 100000 in code I don't get an error anymore.
     
  16. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,498
    Thank you very much for reporting. This problem will be fixed in the next version.
     
  17. Zalosath

    Zalosath

    Joined:
    Sep 13, 2014
    Posts:
    660
    Hello, I'm having a problem with hconcat, it works for 2 Mats but not for 3.
    I'm using:
    Code (CSharp):
    1. Size max = new Size(Math.Max(one.size().width, Math.Max(two.size().width, colourMatch.size().width)), Math.Max(one.size().height, Math.Max(two.size().height, colourMatch.size().height)));
    2. Imgproc.resize(one, one, max);
    3. Imgproc.resize(two, two, max);
    4. Imgproc.resize(colourMatch, colourMatch, max);
    5.  
    6. Mat combined = new Mat();
    7. Core.hconcat(new List<Mat>() { one, two, colourMatch }, combined);
    If I don't include colourMatch it works, if I do it returns a Mat that is 0x0 in size, so matToTexture2D fails with GetPixels32 on degenerate image. I can use matToTexture2D on all three Mats (one, two, and colourMatch) individually and it works fine, but after combining it creates a Mat that is 0x0 and therefore causes an error when trying to use matToTexture2D.

    Note that it only fails with colourMatch so it's likely a problem with that Mat, it's created like so:
    Code (CSharp):
    1. Mat mat = new Mat(75, 50, CvType.CV_8UC3);
    2.         for (int x = 0; x < 50; x++)
    3.         {
    4.             for (int y = 0; y < 75; y++)
    5.             {
    6.                 if (x <= 24 && y <= 24)
    7.                 {
    8.                     // top left corner
    9.                     mat.put(y, x, new double[] { color1.r, color1.g, color1.b });
    10.                 }
    11.                 else if (x <= 24 && y <= 49)
    12.                 {
    13.                     // middle left
    14.                     mat.put(y, x, new double[] { color2.r, color2.g, color2.b });
    15.                 }
    16.                 else if (x <= 24 && y <= 74)
    17.                 {
    18.                     // bottom left corner
    19.                     mat.put(y, x, new double[] { color3.r, color3.g, color3.b });
    20.                 }
    21.                 else if (x > 24 && y <= 24)
    22.                 {
    23.                     // top right corner
    24.                     mat.put(y, x, new double[] { color4.r, color4.g, color4.b });
    25.                 }
    26.                 else if (x > 24 && y <= 49)
    27.                 {
    28.                     // middle right
    29.                     mat.put(y, x, new double[] { color5.r, color5.g, color5.b });
    30.                 }
    31.                 else if (x > 24 && y <= 74)
    32.                 {
    33.                     // bottom right corner
    34.                     mat.put(y, x, new double[] { color6.r, color6.g, color6.b });
    35.                 }
    36.             }
    37.         }
    38.         return mat;
    Why is this happening and how do I fix it? Thanks.

    EDIT: Fixed, one and two were 8UC4 and colourMatch was 8UC3.
     
    Last edited: Mar 13, 2023
  18. rstorm000

    rstorm000

    Joined:
    Aug 19, 2010
    Posts:
    227
    What's the expected framerate on the lightweight pose estimation? I'm only getting 15fps on a good computer with the trial version
     
  19. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,498
    Our simple test code did not generate any errors.

    Code (CSharp):
    1.  
    2. using OpenCVForUnity.CoreModule;
    3. using OpenCVForUnity.ImgprocModule;
    4. using OpenCVForUnity.UnityUtils;
    5. using System;
    6. using System.Collections.Generic;
    7. using UnityEngine;
    8.  
    9. public class HconcatTest : MonoBehaviour
    10. {
    11.  
    12.     // Start is called before the first frame update
    13.     void Start()
    14.     {
    15.         //if true, The error log of the Native side OpenCV will be displayed on the Unity Editor Console.
    16.         Utils.setDebugMode(true);
    17.  
    18.  
    19.         Mat one = new Mat(100, 100, CvType.CV_8UC3);
    20.         Mat two = new Mat(200, 200, CvType.CV_8UC3);
    21.         Mat colourMatch = new Mat(300, 300, CvType.CV_8UC3);
    22.  
    23.         //
    24.         Size max = new Size(Math.Max(one.size().width, Math.Max(two.size().width, colourMatch.size().width)), Math.Max(one.size().height, Math.Max(two.size().height, colourMatch.size().height)));
    25.         Imgproc.resize(one, one, max);
    26.         Imgproc.resize(two, two, max);
    27.         Imgproc.resize(colourMatch, colourMatch, max);
    28.  
    29.         Mat combined = new Mat();
    30.         Core.hconcat(new List<Mat>() { one, two, colourMatch }, combined);
    31.         //
    32.  
    33.  
    34.         Debug.Log("one:" + one);
    35.         Debug.Log("two:" + two);
    36.         Debug.Log("colourMatch:" + colourMatch);
    37.  
    38.         Debug.Log("combined:" + combined);
    39.  
    40.  
    41.         Utils.setDebugMode(false);
    42.     }
    43. }
    44.  
    Enabling Utils.setDebugMode(true); may help identify the cause of the error.
     
  20. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,498
    That was about the same FPS on my PC.
    It may be possible to increase the FPS if we find another even lighter model.
    Although not related to OpenCV, a plugin for easy Human Pose Estimation in Unity is available here.
    I think this is faster since it uses Barracuda for inference.
    https://github.com/keijiro/BodyPixBarracuda
     
  21. joaomlo2

    joaomlo2

    Joined:
    Jan 3, 2017
    Posts:
    1
    Hi, I'm currently attempting to implement a WebGL webcam video recorder. The problem I'm having so far happens right on the Editor where for some reason I am unable to save both video (with VideoWriter) and images (with imwrite) to the Application.persistentDataPath folder.
    Saving on other folders, such as Desktop, works completely fine.
    Also, I tested creating a simple text file on that folder using File.WriteAllText and it works correctly, which lead me to believe it has something to do with the plugin itself.

    The code I wrote for the camera operation is as follows:
    Code (CSharp):
    1.     private IEnumerator _CameraOperationCoroutine(){
    2.         VideoWriter writer = new VideoWriter();
    3.         string save_path = Application.persistentDataPath;
    4.         do{
    5.             switch (this.camStatus){
    6.                 case CameraStatus.NoDevice:
    7.                 case CameraStatus.Unauthorized:
    8.                 case CameraStatus.Unavailable:
    9.                     ToggleErrorMessage(true);
    10.                     break;
    11.                 case CameraStatus.Initializing_Feed:
    12.                     _webcamDisplay.material.mainTexture = _webCamTexture;
    13.                     _webCamTexture.Play();
    14.                     this.camStatus = !_webCamTexture.isPlaying ? CameraStatus.Unavailable : CameraStatus.Idle;
    15.                     break;
    16.                 case CameraStatus.Starting_Recording:
    17.                     string filename = DateTime.Now.ToString("ddMMyyyyHHmmssff") + ".mp4";
    18.                     writer = new VideoWriter(Path.Combine(save_path,filename),VideoWriter.fourcc('h','2','6','4'),30, new Size(_webCamTexture.width,_webCamTexture.height));
    19.                     this.camStatus = CameraStatus.Recording;
    20.                     break;
    21.                 case CameraStatus.Recording:
    22.                     if (_webCamTexture.didUpdateThisFrame){
    23.                         Mat frame = new Mat(_webCamTexture.height, _webCamTexture.width, CvType.CV_8UC3);
    24.                         Utils.webCamTextureToMat(_webCamTexture,frame);
    25.                         Imgproc.cvtColor(frame,frame,Imgproc.COLOR_BGR2RGB);
    26.                         writer.write(frame);
    27.                     }
    28.                     break;
    29.                 case CameraStatus.Stopping_Recording:
    30.                     writer.release();
    31.                     string[] files = Directory.GetFiles(Application.persistentDataPath);
    32.                     foreach (string file in files)
    33.                     {
    34.                         Debug.Log("FILE: "+file);
    35.                     }
    36.                     this.camStatus = CameraStatus.Idle;
    37.                     break;
    38.                 case CameraStatus.Idle:
    39.                 default:
    40.                     break;
    41.             }
    42.             yield return null;
    43.         }
    44.         while (this.camStatus != CameraStatus.NoDevice && this.camStatus != CameraStatus.Unauthorized && this.camStatus != CameraStatus.Unavailable);
    45.     }
     
  22. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,498
    When the Imgcodecs.imwrite() method was added to VideoWriterExample, the video and images were saved without problems.
    Windows10
    Unity 2021.1.0f1
    OpenCVForUnity 2.5.3

    Imgcodecs.imwrite(Application.persistentDataPath + "/VideoWriterExample_image"+frameCount+".png", recordingFrameRgbMat);

    https://github.com/EnoxSoftware/Ope...WriterExample/VideoWriterExample.cs#L148-L175
    Code (CSharp):
    1.         void OnPostRender()
    2.         {
    3.             if (isRecording)
    4.             {
    5.                 if (frameCount >= maxframeCount ||
    6.                     recordingFrameRgbMat.width() != Screen.width || recordingFrameRgbMat.height() != Screen.height)
    7.                 {
    8.                     Debug.LogError("Please fix the screen ratio of the Game View to recognize the recording area.");
    9.                     OnRecButtonClick();
    10.                     return;
    11.                 }
    12.  
    13.                 frameCount++;
    14.  
    15.                 // Take screen shot.
    16.                 screenCapture.ReadPixels(new UnityEngine.Rect(0, 0, Screen.width, Screen.height), 0, 0);
    17.                 screenCapture.Apply();
    18.  
    19.                 Utils.texture2DToMat(screenCapture, recordingFrameRgbMat);
    20.                 Imgproc.cvtColor(recordingFrameRgbMat, recordingFrameRgbMat, Imgproc.COLOR_RGB2BGR);
    21.  
    22.                 Imgproc.putText(recordingFrameRgbMat, frameCount.ToString(), new Point(recordingFrameRgbMat.cols() - 70, 30), Imgproc.FONT_HERSHEY_SIMPLEX, 1.0, new Scalar(255, 255, 255), 2, Imgproc.LINE_AA, false);
    23.                 Imgproc.putText(recordingFrameRgbMat, "SavePath:", new Point(5, recordingFrameRgbMat.rows() - 30), Imgproc.FONT_HERSHEY_SIMPLEX, 0.8, new Scalar(0, 0, 255), 2, Imgproc.LINE_AA, false);
    24.                 Imgproc.putText(recordingFrameRgbMat, savePath, new Point(5, recordingFrameRgbMat.rows() - 8), Imgproc.FONT_HERSHEY_SIMPLEX, 0.5, new Scalar(255, 255, 255), 0, Imgproc.LINE_AA, false);
    25.  
    26.                 writer.write(recordingFrameRgbMat);
    27.  
    28.                 Imgcodecs.imwrite(Application.persistentDataPath + "/VideoWriterExample_image"+frameCount+".png", recordingFrameRgbMat);
    29.             }
    30.         }
     
  23. kwkw

    kwkw

    Joined:
    Mar 3, 2014
    Posts:
    10
    Hi, is it possible to use human segmentation as a virtual background like in Zoom?

    i using
    Core.bitwise_not(rgbaMat, rgbaMat); 

    but it turned out to be like the picture I attached below. How do I remove the white masking?
     

    Attached Files:

    Last edited: Mar 20, 2023
  24. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,498
    If you are interested in doing a Zoom-like screen composite, the following KeyFrameGreenScreenExample code may be helpful.
    How to do image compositing using a mask image:
    https://github.com/EnoxSoftware/Ope...5bd4fcc6227ec/Assets/OpenCVForUnity/Examples/ Advanced/KeyFrameGreenScreenExample/KeyFrameGreenScreenExample.cs#L228-L232
     
  25. vetronus

    vetronus

    Joined:
    Jan 21, 2016
    Posts:
    11
    OpenCV for Unity is not working on my M1 Mac. It is giving this error:

    DllNotFoundException: opencvforunity assembly:<unknown assembly> type:<unknown type> member:(null)
    OpenCVForUnity.CoreModule.Mat..ctor (System.Int32 rows, System.Int32 cols, System.Int32 type) (at Assets/OpenCVForUnity/org/opencv/core/Mat.cs:516)
    CamApp.Update () (at Assets/CamApp.cs:23)

    Can someone help me out?
     
  26. EnoxSoftware

    EnoxSoftware

    Joined:
    Oct 29, 2014
    Posts:
    1,498
    M1Mac ( Apple Sillicon ) is supported with OpenCVForUnity 2.4.7 or higher.