Positioning GameObjects based on VisualElement coordinates

I would like to experiment with UI Toolkit at runtime and my plan is to implement a flexible user interface that adapts to the available screen size by using some appropriate PanelSettings. I would also like to dynamically align some GameObjects in the scene based on the coordinates/bounding boxes of some key VisualElements in the visual tree.
There is the GeometryChangedEvent that I could capture to get the bounds of VisualElement and use this to drive the alignment of my GameObjects.
Is this a legitimate way to achieve what I want to do or are there any more clever ways to "connect" UI based elements and scene objects together?
I should mention that I just plan to have 2d scene, so any transformation beetween UI and Camera coordinate system should be trivial.

Edit: I just realized, that the "Answers" section would have been a more appropriate place to raise my question. Sorry for that!

 

This thread shows how to do it. Unfortunately, it only works on your native resolution. https://forum.unity.com/threads/possible-world-position-bug.995668/

If you or any Unity Developer knows how to make it work based on the scaling factor.

 

Thanks for pointing to this thread. Maybe I can find a way to make it work resolution-independent with some values from the Screen class (width, height, orientation, dpi etc.)

 

No problem! Please let me know if you do! I cannot seem to find a scaling factor like UGUI has.

Maybe it hasn’t been implemented yet?

 

Update
So the way that I proposed (catching the OnGeometryChanged event) works like a charm for me!

Steps to do:

1) Register a GeometryChangedEvent listener on the VisualElement that you want to inspect

Code (CSharp):

1. fooElement.RegisterCallback<GeometryChangedEvent>(OnGeometryChanged);

2) In the listener method get the new bounding rectangle of the Element (in screen space) and transform it to world space

Code (CSharp):

1. private void OnGeometryChanged(GeometryChangedEvent evt)
2. {
3.     Rect bounds = evt.newRect;
4.     var pos = Camera.main.ScreenToWorldPoint(new Vector3(bounds.x, Screen.height - bounds.y, 0)); // note the inversion of the y-axis here!
5.     myGameObject.transform.position = new Vector3(pos.x, pos.y, 0);
6. }

With this code I was able to position a GameObject ("myGameObject") in the scene based on the top left corner of a VisualElement ("fooElement"). If I change the Screen or game view then the GameObject will reposition itself and will stay at the top left corner of the VisualElement. I tested that with a nested(!) VisualElement, so my fooElement was already a child of another VisualElement and took only a fractional part of parents space. So it seems, that the bounds that you get via OnGeometryChangedEvent.newRect are already in Screen-space and NOT local to the parent VisualElement.

One thing to keep in mind here is, that like I said before I was only interested in the 2d coordinates and thus I set the z-position of my GameObject to zero.

I noticed something while testing that might confuse other people. When I'm in Play-Mode and resize the game window, then the OnGeometryChanged event seems to fire during the whole resize process. If however I maximize the game window and then resize the whole unity editor window (which in turn also resizes the game window), then the OnGeometryChangedEvent is only fired when the resizing is over (after the mouse button has been released). I also build a windows application and tested the event there (in a resizable window). Result here is that the event is also fired during the whole resize process and not only after the mouse button was released.

So long story short - it seems that at least for my usecase I can use the new runtime ui together with some custom code to reposition some GameObjects based on the layout of my dynamic user interface. Cool!

 

Thanks for posting such a great update!

Unfortunately, it doesn't work for my case

1. I am using experimental.animation and the GeometryChangedEvent doesn't work with this.
2. Using visualElement.contentRect doesn't work.

Do you think this is a bug that should be reported?

 

I have no documentation about contentRect, but it looks to me that this is simply the wrong property!

If I debug the values of contentRect inside of my event handler and compare them to the rectangle, that I get from the event than it looks that they are different. What I think contentRect describes is the inner bounding box of the element. So if the element has a width of 200 and a height of 100 and has a border one each side of 1 pixel then you will get (1, 1, 198, 98) as the contentRect.

But I think there's a solution for your problem. Just use the "layout" property instead! This will give you exactly the rect that you will also get via the OnGeometryChangedEvent. And then you just need to tranform the coordinates like in my code snippet.

 

Ah! Yea that is what I thought the contentRect was, the rect gotten from the event. I will try that, thanks!

EDIT: Unfortunately, this didn't work. Thanks for the info! I posted another thread about this issue and waiting to see what the Unity devs think.

 

This code here should also work with Scale mode "Scale with screen size" and with all match modes (shrink, expand, match with or height). Parts have been borrowed from the PanelSettings class
Keep in mind to use VisualElement.layout instead of GeometryChangedEvent.newRect for your case!

Code (CSharp):

1.    
2. private void OnGeometryChanged(GeometryChangedEvent evt)
3.     {
4.         Rect rect = evt.newRect;
5.  
6.         Vector2 sourcePos = new Vector2(rect.x, rect.y); // top left corner
7.         // Vector2 sourcePos = new Vector2(rect.x + rect.width, rect.y); // top right corner
8.         // Vector2 sourcePos = new Vector2(rect.x, rect.y + rect.height); // bottom left corner
9.         // Vector2 sourcePos = new Vector2(rect.x + rect.width, rect.y + rect.height); // bottom right corner
10.  
11.         float xPos = 0;
12.         float yPos = 0;
13.        
14.         switch (PanelSettings.scaleMode)
15.         {
16.             case PanelScaleModes.ConstantPixelSize:
17.                 xPos = sourcePos.x * PanelSettings.scale;
18.                 yPos = sourcePos.y * PanelSettings.scale;
19.                 break;
20.             case PanelScaleModes.ConstantPhysicalSize:
21.                 float correctedScale = Screen.dpi / PanelSettings.referenceDpi;
22.                 xPos = sourcePos.x * correctedScale;
23.                 yPos = sourcePos.y * correctedScale;
24.                 break;
25.  
26.             case PanelScaleModes.ScaleWithScreenSize:
27.  
28.                 if (PanelSettings.referenceResolution.x * PanelSettings.referenceResolution.y != 0)
29.                 {
30.                     var refSize = (Vector2) PanelSettings.referenceResolution;
31.                     var sizeRatio = new Vector2(Screen.width / refSize.x, Screen.height / refSize.y);
32.  
33.                     var denominator = 0.0f;
34.                     switch (PanelSettings.screenMatchMode)
35.                     {
36.                         case PanelScreenMatchModes.Expand:
37.                             denominator = Mathf.Min(sizeRatio.x, sizeRatio.y);
38.                             break;
39.                         case PanelScreenMatchModes.Shrink:
40.                             denominator = Mathf.Max(sizeRatio.x, sizeRatio.y);
41.                             break;
42.                         default: // PanelScreenMatchModes.MatchWidthOrHeight:
43.                             var widthHeightRatio = Mathf.Clamp01(PanelSettings.match);
44.                             denominator = Mathf.Lerp(sizeRatio.x, sizeRatio.y, widthHeightRatio);
45.                             break;
46.                     }
47.                    
48.                     xPos = sourcePos.x * denominator;
49.                     yPos = sourcePos.y * denominator;
50.                 }
51.  
52.                 break;
53.         }
54.  
55.         float invertedYPos = Screen.height - yPos;
56.         var newPos = Camera.main.ScreenToWorldPoint(new Vector3(xPos, invertedYPos, 0));
57.         cube.transform.position = new Vector3(newPos.x, newPos.y, 0);
58.     }

 

Thanks again for looking into it. It still doesn't work . I will just keep looking into it! Maybe it has to do with my version of UIToolkit vs yours or some other factor. I feel like I am missing something simple when I am using worldBound.center. Some scaling factor like in UGUI that isn't available yet. I really appreciate the code!

 

No need to thank me, I had my personal interests in that topic
I cross my fingers, that you will solve your problem too!
I should mention that I use an orthographic camera and set all z-values to zero.

 

The UIToolkit sample project Dragon Crashers has a bit of code to position a ParticleSystem in 3D space based on the position of a VisualElement:

Code (CSharp):

1.  
2.     // converts a screen position from UI Toolkit to world space
3.     public static Vector3 ScreenPosToWorldPos(this Vector2 screenPos, Camera camera = null, float zDepth = 10f)
4.     {
5.         if (camera == null)
6.             camera = Camera.main;
7.  
8.         if (camera == null)
9.             return Vector2.zero;
10.  
11.         float xPos = screenPos.x;
12.         float yPos = screenPos.y;
13.         Vector3 worldPos = Vector3.zero;
14.  
15.         if (!float.IsNaN(screenPos.x) && !float.IsNaN(screenPos.y))
16.         {
17.             // flip y-coordinate; in UI Toolkit, (0,0) is top-left instead of bottom-left.
18.             yPos = camera.pixelHeight - yPos;
19.  
20.             // convert to world space position using Camera class
21.             Vector3 screenCoord = new Vector3(xPos, yPos, zDepth);
22.             worldPos = camera.ScreenToWorldPoint(screenCoord);
23.         }
24.         return worldPos;
25.     }

 

Could you give some advise? How would you recommend to deal with converting position from screen space to world space when panel has "Scale With Screen Size" scale mode, screen match mode - "Match Width or Height". Plus taking safe area into consideration. I'm having hard time figuring it out