Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

How do I apply Perspective Transformation in a Render Texture?

Discussion in 'Image Effects' started by LucasStraub, May 22, 2016.

  1. LucasStraub

    LucasStraub

    Joined:
    Apr 2, 2015
    Posts:
    5
    Hi everyone,

    Despite my lack of knowledge I'm trying to code an Image Effect that applies Perspective Transformation (See last topic in http://docs.opencv.org/3.1.0/da/d6e/tutorial_py_geometric_transformations.html#gsc.tab=0 to how it works) in a Render Texture.

    I've done a code that accomplish this, but it takes almost 20 seconds (To get Render Texture, apply transformations returning a Texture2D) and I need it to happen every frame.
    Here's the code I've done (I'm using OpenCvSharp to the transformations):
    Code (CSharp):
    1. using UnityEngine;
    2. using System.Collections;
    3. using OpenCvSharp;
    4.  
    5. public class Test2 : MonoBehaviour {
    6.     // Virtual Camera;
    7.     public Camera cam;
    8.  
    9.     // Result Texture - Screen that will be showned to the user;
    10.     Texture2D tex;
    11.     public Texture2D result;
    12.  
    13.     // Edges;
    14.     public Transform[] points;
    15.  
    16.     void OnDrawGizmos(){
    17.         Vector3[] resultPoints = new Vector3[points.Length];
    18.  
    19.         Vector3 screenCenter = cam.transform.forward * cam.farClipPlane + cam.transform.position;
    20.  
    21.         for(int i = 0; i < points.Length; i++) {
    22.             Ray ray = new Ray (cam.transform.position, points[i].position - cam.transform.position);
    23.             Plane plane = new Plane (-cam.transform.forward, screenCenter);
    24.  
    25.             float distance;
    26.             if (plane.Raycast (ray, out distance)) {
    27.                 resultPoints [i] = ray.origin + ray.direction * distance;
    28.  
    29.                 Debug.DrawRay (ray.origin, ray.direction * distance, Color.gray);
    30.  
    31.                 if (i != 0) {
    32.                     Debug.DrawLine (resultPoints [i - 1], resultPoints [i]);
    33.                     if (i == points.Length - 1) {
    34.                         Debug.DrawLine (resultPoints [0], resultPoints [resultPoints.Length - 1]);
    35.                     }
    36.                 }
    37.             }
    38.         }
    39.     }
    40.  
    41.     [ContextMenu("Shot")]
    42.     void Shot(){
    43.         if (points.Length == 4) {
    44.             Vector3[] viewportPoints = new Vector3[4];
    45.             for (int i = 0; i < points.Length; i++)
    46.                 viewportPoints [i] = cam.WorldToViewportPoint (points [i].position);
    47.             ApplyPT (viewportPoints);
    48.         }
    49.     }
    50.  
    51.     void ApplyPT(Vector3[] viewportPoints){
    52.         tex = cam.targetTexture.ToTexture2D ();
    53.  
    54.         CvMat c = tex.ToCvMat();
    55.  
    56.         CvPoint2D32f[] pts1 = new CvPoint2D32f[viewportPoints.Length];
    57.  
    58.         for(int i = 0; i < viewportPoints.Length; i++)
    59.             pts1 [i] = new CvPoint2D32f (viewportPoints[i].x * tex.width, viewportPoints[i].y * tex.height);
    60.        
    61.         CvMat c2 = new CvMat(result.height, result.width, MatrixType.U8C4);
    62.  
    63.         CvPoint2D32f[] pts2    = new CvPoint2D32f[4];
    64.  
    65.         pts2[0] = new CvPoint2D32f (0, 0);
    66.         pts2[3] = new CvPoint2D32f (0, c2.Height);
    67.         pts2[2] = new CvPoint2D32f (c2.Width, c2.Height);
    68.         pts2[1] = new CvPoint2D32f (c2.Width, 0);
    69.  
    70.         CvMat m = Cv.GetPerspectiveTransform (pts1, pts2);
    71.  
    72.         Cv.WarpPerspective (c, c2, m);
    73.         result = c2.ToTexture2D(result);
    74.  
    75.         Debug.Log (Time.realtimeSinceStartup);
    76.     }
    77. }
    I've tried applying this code to OnRenderImage in a Image Effect, but it crashes Unity.

    I don't know much about Shaders, but might be the answer for this. So please if you know the path tell me were to start.

    Thanks for you time.
     
  2. XinYueStudio

    XinYueStudio

    Joined:
    Oct 27, 2015
    Posts:
    2
    Hi LucasStraus ,Don't you has found a way? I have this problem, too
     
  3. kislotaooankit

    kislotaooankit

    Joined:
    Apr 20, 2020
    Posts:
    15
    Sorry I am little late,

    But did you found out the answer ?
     
  4. LucasStraub

    LucasStraub

    Joined:
    Apr 2, 2015
    Posts:
    5
    Not exacly, I end up changing my camera frustum instead of manipulating the image after being captured.

    My problem was something like the image bellow. I would show, ingame, the image relative of what the user should see if its head was inside the game.

    My solution was simular of how Sebastian Lague implemented on its Coding Adventure: Portals video.
    He explains pretty well, but not sure if it will fit for you. (I would share my code, but I lost it somehow :()

    Be free to ask questions if you want
     
  5. kislotaooankit

    kislotaooankit

    Joined:
    Apr 20, 2020
    Posts:
    15

    Hey Lucas Thanks for such detailed reply .. i haven't checked your resources yet but i would like to explain my problem case first.. also i am noob with unity as i started with these things in march only.

    Basically what i want is that i would scan an image(Could be printed on paper) through my Device camera using some AR library like arcore or vuforia etc. and apply some sort of transform such that only required skewed image gets cropped out and is converted into flat 2d image in realtime.

    The way i am doing is by first extracting the 4 corner points of main image/target by initiating 4 empty gameobjects at corners and tracking their transform component.
    upload_2020-6-18_16-41-1.png

    Now i am converting them into screen coordinates to get their placement on canvas.

    upload_2020-6-18_16-48-4.png


    next step is to crop that image with reference to these 4 points and then apply perspective transform to flatten the skewness but i dont know how to crop out ROI from canvas with these four points.

    Or to initiate a second camera and somehow tilt its view wrt to these 4 points like the image in your example...

    if you have any suggestion in my approach or something new pls go ahead .. also i will check your resources too by this weekend

    thank you
     
  6. LucasStraub

    LucasStraub

    Joined:
    Apr 2, 2015
    Posts:
    5
    No problem, turns out I'm a AR developer and I did something similar in the past, it may be easier to help you.

    The easy way is to add a camera inside and facing your Image Target object. On the camera set Target Texture to a new RenderTexture then set Projection to orthograpic, match its Size to the image target borders and set Colling Mask to the same layer that the AR background plane (where the AR camera is rendered) layer only. Also, check if this new camera has a bigger Far Clipping Plane than the AR camera. It should look like this:
    This way the Camera should output to the RenderTexture a real life version of your target image already "cropped".

    This method will work to one Image Target only. If you want to use in multiple Image Targets you can set the Camera position relative to the current Image Target via script or have multiple RenderTextures for each Image Target.

    PS: Do your next step be to paint the drawing in real life and than apply to a 3d object texture in Unity?
     

    Attached Files:

  7. kislotaooankit

    kislotaooankit

    Joined:
    Apr 20, 2020
    Posts:
    15
    Haha you read my mind somewhat ... probably due to that duck image ...

    Yes i would like to do something like that only i had done it with a readymade package you might be knowing as "region capture" but as i mentioned i am new to this AR field and so i wanted to implement it the same by my own.

    However i had badluck finding the resources of how region capture works and code was too complicated for me tbh.

    So i resorted to OpenCV approach as i had some experience in this field ... however it was costly method so i was looking for something similar which you provided

    Thank-you i will try that.
     
    Last edited: Jun 19, 2020