Search Unity

  1. We are migrating the Unity Forums to Unity Discussions by the end of July. Read our announcement for more information and let us know if you have any questions.
    Dismiss Notice
  2. Dismiss Notice

ARKit + OpenCVForUnity

Discussion in 'AR' started by ina, Oct 20, 2017.

  1. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    1,089
    Has anyone tried this yet
     
    Blarp likes this.
  2. Blarp

    Blarp

    Joined:
    May 13, 2014
    Posts:
    270
    aximsat and DHein like this.
  3. CarinaMclane

    CarinaMclane

    Joined:
    Mar 12, 2014
    Posts:
    23
    We are using ARKit (and ARCore) with OpenCV - biggest issue is extracting the image buffer from the Webcam Texture to pass to a CV mat without incurring too much of a cost (given whatever you do with OpenCV then adds further compute cost).
     
  4. SunnyChow

    SunnyChow

    Joined:
    Jun 6, 2013
    Posts:
    360
    are you guys talking about OpenCV or OpenCV For Unity? OpenCV For Unity is a plugin on asset store
     
    ina likes this.
  5. ina

    ina

    Joined:
    Nov 15, 2010
    Posts:
    1,089
    the github linked seems to be a custom wrap. opencvforunity is good for prototyping but has a lot of performance issues :-\
     
  6. arcade_jon

    arcade_jon

    Joined:
    Feb 9, 2018
    Posts:
    11
    I've got the same challenge. Am using the ARTextureHandles tp get the IntPtr to the camera feed textures, but my OpenCV Mats render as black. Any ideas welcome! The code is:

    using System;
    using UnityEngine;
    using OpenCVForUnity.CoreModule;
    using OpenCVForUnity.ImgprocModule;
    using OpenCVForUnity.UnityUtils;
    using UnityEngine.UI;
    using UnityEngine.XR.iOS;


    public class ProcessVideo : MonoBehaviour
    {
    Texture2D texture;
    Texture2D videoTextureY;
    RawImage rawImage;
    Resolution currentResolution;
    bool faceDetectionRunning;

    public void StartFaceDetection()
    {
    faceDetectionRunning = true;
    }

    public void StopFaceDetection()
    {
    faceDetectionRunning = false;
    }

    void Start()
    {
    currentResolution = Screen.currentResolution;
    rawImage = gameObject.GetComponent<RawImage>();
    texture = new Texture2D(currentResolution.width, currentResolution.height, TextureFormat.RGBA32, false);
    rawImage.texture = texture;
    }



    void Update()
    {
    if (faceDetectionRunning == true)
    {
    #if !UNITY_EDITOR && UNITY_IOS

    Resolution currentResolution = Screen.currentResolution;

    ARTextureHandles handles = UnityARSessionNativeInterface.GetARSessionNativeInterface().GetARVideoTextureHandles();

    if(handles.TextureY !=null){
    ProcessARVideoTexture((System.IntPtr)handles.TextureY);
    }

    #endif
    }
    }

    private void ProcessARVideoTexture(IntPtr pointerY)
    {
    //Running the following 4 lines results in the correct video texture being applied to the gameObject's RawImage texture, so we know the pointer is right.
    //videoTextureY = Texture2D.CreateExternalTexture(currentResolution.width, currentResolution.height, TextureFormat.RGBA32, false, false, pointerY);
    //videoTextureY.filterMode = FilterMode.Bilinear;
    //videoTextureY.wrapMode = TextureWrapMode.Repeat;
    // rawImage.texture = videoTextureY;

    Size size = new Size(currentResolution.width, currentResolution.height);

    //PROBLEM: The following 12 lines of code render a black texture with some static-like lines on it to the gameObject's RawImage texture. I need the rgbaMat to be the TextureY from the camera feed. Please help!
    using (Mat rgbaMat = new Mat(size, CvType.CV_8UC4, new Scalar(0, 0, 0, 255)))
    {
    using (Mat singleChannelMat = new Mat(size, CvType.CV_8UC1))
    {
    Utils.copyToMat(pointerY, singleChannelMat);

    Imgproc.cvtColor(singleChannelMat, rgbaMat, Imgproc.COLOR_GRAY2RGBA);

    Utils.fastMatToTexture2D(rgbaMat, texture);
    }
    }
    }
    }
     
    ina likes this.