Calculating World to Screen projection manully?

For the reasons unspecified I need to avoid using default Unity's API like WorldToScreenPoint and I need to calculate this manually. Best case scenario would be to assume I only have Vector2/Vector3 functionality and no built-in Matrix stuff. Okay, this is now more of math question and math and me are incompatible.

So, give that we have:

• farZclip
• nearZclip
• fovX
• Camera position in World
• Object position in World
• Aspect ratio
• Screen (0,0) starts at the top left

Is it enough to calculate and figure out where on screen would be object's center (or any point)? if so, Is there any simplistic examples perhaps with test cases that showcase it?

 

No, the information provided is not enough. You need the full camera matrix which includes the orientation / rotation of the camera. Also usually you specify the vertical fov, so I'm not sure what "fovX" represents. The near and far clip planes are not really necessary if you just want to calculate the 2d screen position. Unity's WorldToScreenPoint returns the distance of the point from the camera in worldspace units in the z component (if that's even relevant to your case).

Just to clarify: Unity's Matrix4x4 struct should be thread safe. So using it inside Unity in a seperate thread should work just fine. If that's not your usecase I would assume you need to calculate this outside of Unity? Since a camera in Unity can have different projections that may not be based on a fov, it's usually easier to grab the projection matrix and the camera matrix from the camera and use those. Though if you really need to calculate it manually, you can of course do that.

The relevant information could be found in my matrix crash course where I explained what the values of the projection matrix actually represent. At the end of my answer I've linked this SO question which specifically was about creating the projection matrix based on a fov.

Well, projecting a 3d point onto a 2d surface is not that complicated, however I'm not sure what you have in mind when you say "simplistic". Expressing the necessary math through a matrix is for sure simpler than disecting all the calculations manually. Matrix multiplication is not black magic and Unity even has the reference source of the Matrix4x4 struct online.

edit

I forgot to mention that if you want to know the Screen position you also need the screen resolution (Screen.width and Screen.height). Also screenspace usually starts at the bottom left corner, not the top left. GUI space starts at the top left.

 

I just had some time and here's the condensed math split into the 6 logical steps:

Code (CSharp):

1.  
2.     public static Vector3 WorldToLocal(Vector3 aCamPos, Quaternion aCamRot, Vector3 aPos)
3.     {
4.         return Quaternion.Inverse(aCamRot) * (aPos - aCamPos);
5.     }
6.     public static Vector3 Project(Vector3 aPos, float aFov, float aAspect)
7.     {
8.         float f = 1f / Mathf.Tan(aFov * Mathf.Deg2Rad * 0.5f);
9.         f /= aPos.z;
10.         aPos.x *= f / aAspect;
11.         aPos.y *= f;
12.         return aPos;
13.     }
14.     public static Vector3 ClipSpaceToViewport(Vector3 aPos)
15.     {
16.         aPos.x = aPos.x * 0.5f + 0.5f;
17.         aPos.y = aPos.y * 0.5f + 0.5f;
18.         return aPos;
19.     }
20.  
21.     public static Vector3 WorldToViewport(Vector3 aCamPos, Quaternion aCamRot, float aFov, float aAspect, Vector3 aPos)
22.     {
23.         Vector3 p = WorldToLocal(aCamPos, aCamRot, aPos);
24.         p = Project(p, aFov, aAspect);
25.         return ClipSpaceToViewport(p);
26.     }
27.  
28.     public static Vector3 WorldToScreenPos(Vector3 aCamPos, Quaternion aCamRot, float aFov, float aScrWidth, float aScrHeight, Vector3 aPos)
29.     {
30.         Vector3 p = WorldToViewport(aCamPos, aCamRot, aFov, aScrWidth / aScrHeight, aPos);
31.         p.x *= aScrWidth;
32.         p.y *= aScrHeight;
33.         return p;
34.     }
35.  
36.     public static Vector3 WorldToGUIPos(Vector3 aCamPos, Quaternion aCamRot, float aFov, float aScrWidth, float aScrHeight, Vector3 aPos)
37.     {
38.         Vector3 p = WorldToScreenPos(aCamPos, aCamRot, aFov, aScrWidth, aScrHeight, aPos);
39.         p.y = aScrHeight - p.y;
40.         return p;
41.     }
42.  

As you can see the final version (WorldToGUIPos) that converts a world space position into a GUI space position requires the following:

• The camera position
• The camera rotation
• The camera vertical fov
• The screen width and height
• And finally the worldspace object position you want to project

I've compared the result to Unity's WorldToScreenPoint with the y axis flipped as well (so we're in GUI space) and the result is exactly the same.

Here's some testing code:

Code (CSharp):

1.     public Camera cam;
2.     public Transform point;
3.     void OnGUI()
4.     {
5.         var p = WorldToGUIPos(cam.transform.position, cam.transform.rotation, cam.fieldOfView,Screen.width, Screen.height, point.position);
6.  
7.         var p2 = cam.WorldToScreenPoint(point.position);
8.         p2.y = Screen.height - p2.y;
9.  
10.         GUI.Label(new Rect(10,10,400,200), "p: " + p + "\np2:" + p2);
11.  
12.         GUI.Box(new Rect(p.x - 5, p.y - 5, 10, 10), "");
13.     }
14.  

Just assign the camera to "cam" and some arbitrary object to "point". p and p2 should both result in the same screen space point.

 

Impressive.

Is vertical FOV a must requirement. hFOV is not gonna do?

That's a good point. Is there a way to project it for arbitrary forward axis? Just swap them in place with z or is there a catch?

The structs in general are thread-safe by definition unless you explicitly mutate them. Calling to Unity API isnt

Also this is an icall:

Code (csharp):

1.  
2. [MethodImpl(MethodImplOptions.InternalCall)]
3. private static extern void Inverse_Injected(ref Quaternion rotation, out Quaternion ret);
4.  

 

That's only half true. Yes almost all of the Unity API is protected by an explicit thread check since everything related to UnityEngine.Object derived types have this dual personality and the majority of the gameobjects state lived on the native side. However pretty much all of Unity's struct types are thread safe. Yes certain method implementations are on the native C++ side. However those are pure functions that work on that struct instance only. So they don't rely on any shared information.

That's also not really true. Their nature makes it quite hard to introduce race conditions since whenever you pass a struct around you get a copy of that struct. However structs are not implicitly thread safe somehow. Yes, if you have a pure immutable struct it could be considered thread safe out-of-the-box. Though thread safety is only relevant if two or more threads access the same data at the same time.

So Quaternion.Inverse, Matrix4x4.inverse and all those methods which rely only on the values of the struct itself are thread safe and can be used from other threads. Just those parts of the Unity API which actually deals with shared information on the native side is not only "not thread safe" but is actually restricted to be only usable from the main thread.

Anyways you can calculate the inverse of a quaternion manually by simply inverting the x,y and z components of the quaternion. Generally the inverse of a quaternion is the conjugate divided by the length of the quaternion. Though since we only deal with unit quaternions the length is always 1.0 so we can actually ignore that.

Code (CSharp):

1. public static Quaternion Inverse(Quaternion aQuat)
2. {
3.     return new Quaternion(-aQuat.x, -aQuat.y, -aQuat.z, aQuat.w);
4. }

This is enough for unit quaternions. Though more correct would be:

Code (CSharp):

1. public static Quaternion Inverse(Quaternion aQuat)
2. {
3.     float invLength = 1f / Mathf.Sqrt(Quaternion.Dot(aQuat, aQuat));
4.     return new Quaternion(-aQuat.x * invLength, -aQuat.y * invLength, -aQuat.z * invLength, aQuat.w * invLength);
5. }

However that would be wasteful since, as already mentioned, within Unity and the usage as pure rotations we only deal with unit quaternions.

 

Does it have to be Quaternion, can we use Vector3 representation of rotation instead?

 

@Bunny83 I was struggling to get worldtoscreenpoint to work properly, and was reluctant to try this, but I regret it, because your code works beautifully! Thank you!

All I needed to change was blocking out the y reversal;

 

Are you sure you talk about WorldToScreenPos? because it doesn't have any y reversal in it?. Only WorldToGUIPos does this since GUI space starts at the top left while screenspace starts at the bottom left.

 

No, I was just saying I needed to reverse it in WorldToGUIpos.

I had a different problem for WorldToScreenPos, hard to describe, put the coordinates it was outputting were wrong. Couldn't figure it out, I have it working completely fine in another project, but in this project it's just wildly off...

EDIT [SOLVED]: I solved the WorldToScreenPos problem. It was the result of a broken camera, I think somehow the frustrum was permanently broken by putting in very large or small clipping values. The solution was to delete the existing camera and remake it.

With WorldToScreenPos (brace should be on the Star Destroyer but is significantly off)


With WorldToGUIPos (brace is correctly on the star destroyer's transform point)