How to use OpenCV camera calibration to set Physical Camera parameters?

I have calibrated data - camera parameters (attached below). How do I use this to set the Physical Camera parameters in Unity to recreate my real life camera?

My confusions are specifically:

1. How to add focal length?

Camera (intrinsic matrix) is the above, which has

• -> a principal point that is usually at the image center
• -> are the focal lengths expressed in pixel units.

The focal length in Unity (I suspect) can't be in pixel units. Is there some way of retrieving focal length in meters from focal length expressed in pixel units?​

2. Is the Sensor Size the same as cx, cy -> principal point?
I notice that if I change my Sensor size, even if my focal length is fixed, the field of view changes, so it seems strange to me. Shouldn't FOV always be fixed for a fixed focal length?

3. What part of the camera parameters from OpenCV correspond to lens shift? Or is that assumed to be some default value?

4. I am guessing the answer to this is No - Can you model distortion parameters into Unity camera?

Source - Open CV : Camera calibration https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html

Thank you!

Camera

Code (csharp):

1.  
2. #YAML File
3. CAMERA Parameters from OpenCV
4. image_width: 0
5. image_height: 1080
6. camera_name: left
7. camera_matrix:
8.   rows: 3
9.   cols: 3
10.   data: [1643.55164017145, 0, 833.842174850277, 0, 1632.81195514379, 657.989554730761, 0, 0, 1]
11. distortion_model: plumb_bob
12. distortion_coefficients:
13.   rows: 1
14.   cols: 5
15.   data: [-0.40444238705588, 0.581618979028972, -0.00497978193873161, 0.00232175743375933, -0.215474790066087]
16. rectification_matrix:
17.   rows: 3
18.   cols: 3
19.   data: [0.9987597352703238, 0.03749814884259891, 0.0327548475211089, -0.03730056012128149, 0.999282144850663, -0.006622929661203378, -0.03297968188735999, 0.00539294131591293, 0.9994414724067496]
20. projection_matrix:
21.   rows: 3
22.   cols: 4
23.   data: [1199.817340845223, 0, 810.3841648101807, 0, 0, 1199.817340845223, 711.4899749755859, 0, 0, 0, 1, 0]

 

Oh this forum, I can upload file. (it is attached. go to the bottom of my post, then you can download pdf)

First, I'd recommend you to read this book from page 153 to 157 (http://cvrs.whu.edu.cn/downloads/ebooks/Multiple%20View%20Geometry%20in%20Computer%20Vision%20(Second%20Edition).pdf I think you already know, but just in case) to understand what is exact K after calibration (which is eqation (6.9) at page 157)

So K you have does not directly contain ether f or px(or py) and this is very important thing to understand first.

I made some note to explain what happens in unity.
After you read the note, you will understand my code :

Code (CSharp):

1.  
2. public class IntrinsicSetup : MonoBehaviour {
3.  
4.  
5.     Camera mainCamera;
6.  
7.     public float f = 35.0f; // f can be arbitrary, as long as sensor_size is resized to to make ax,ay consistient
8.  
9.    // Use this for initialization
10.    void Start () {
11.         mainCamera = gameObject.GetComponent<Camera>();
12.         changeCameraParam();
13.     }
14.  
15.     public void changeCameraParam()
16.     {
17.         string path = "Assets/Resources/Intrinsic.txt";
18.         float ax, ay, sizeX, sizeY;
19.         float x0, y0, shiftX, shiftY;
20.         int width, height;
21.  
22.         string[] lines = File.ReadAllLines(path);
23.         string[] parameters = lines[1].Split(' ');
24.         string[] resolution = lines[3].Split(' ');
25.  
26.         ax = float.Parse(parameters[0]);
27.         ay = float.Parse(parameters[1]);
28.         x0 = float.Parse(parameters[2]);
29.         y0 = float.Parse(parameters[3]);
30.  
31.         width = int.Parse(resolution[0]);
32.         height = int.Parse(resolution[1]);
33.  
34.         sizeX = f * width / ax;
35.         sizeY = f * height / ay;
36.  
37.         //PlayerSettings.defaultScreenWidth = width;
38.         //PlayerSettings.defaultScreenHeight = height;
39.  
40.         shiftX = -(x0 - width / 2.0f) / width;
41.         shiftY = (y0 - height / 2.0f) / height;
42.  
43.         mainCamera.sensorSize = new Vector2(sizeX, sizeY);     // in mm, mx = 1000/x, my = 1000/y
44.         mainCamera.focalLength = f;                            // in mm, ax = f * mx, ay = f * my
45.         mainCamera.lensShift = new Vector2(shiftX, shiftY);    // W/2,H/w for (0,0), 1.0 shift in full W/H in image plane
46.  
47.     }
48. }
49.  

here, you can use your camera_matrix.
now yours is
ax, 0, x0, 0, ay, y0, 0, 0, 1. (so what you have is not fx! its fx*mx=ax)

I didn't try with distortion. So I cannot help you with it. However I saw in some unity documentation of giving some waving effect. Maybe you can give distortion as effect, but I won't be like giving distortion coefficient to reproducing your distortion

 

Thank you so much for this detailed answer! The notes and the resource you provided were really really good.

I have a follow-up question though. In this answer (https://answers.unity.com/questions...camera-from-its-intr.html?childToView=1645305), you mentioned that there is a minus sign in LensShiftX, which I don't see in the note - did you miss that by mistake?

 

A

Ahhhh... that one ... If you see my original answer from other post, you can see I wrote "//somehow this is opposite sign" I have no idea to be honest. I am assuming this is rather caused by convention difference .... , like when it comes to lens shift, which direction it starts & where is origin of image something like these always confuse me

 

Thank you!

 

Hi Sir,

i have same problem. i have the parameter matrix of my camera and i wann visualise it in Unity could u help me please with the program anf thank u

Code (CSharp):

1. [code=CSharp]%YAML:1.0
2. ---
3. calibration_time: "Fri Oct  2 16:43:44 2020"
4. image_width: 640
5. image_height: 480
6. board_width: 9
7. board_height: 6
8. square_size: 5.2000001072883606e-02
9. aspectRatio: 1.
10. flags: 2
11. camera_matrix: !!opencv-matrix
12.    rows: 3
13.    cols: 3
14.    dt: d
15.    data: [ 6.4054031826016183e+02, 0., 3.0338619644214515e+02, 0.,
16.        6.4054031826016183e+02, 2.5601243873490392e+02, 0., 0., 1. ]
17. distortion_coefficients: !!opencv-matrix
18.    rows: 5
19.    cols: 1
20.    dt: d
21.    data: [ -7.3933578764705618e-02, 4.7089239158550916e-01,
22.        5.4172605723194515e-03, -5.6869323277609161e-03,
23.        -1.0417712178987861e+00 ]
24. avg_reprojection_error: 3.5360094587309865e-01
25.  

[/code]