Resolved How to efficiently move a large amount of elements?

My app requires social functions like SMS style messaging and group chat for which in principle UI Toolkit is perfect. UI Toolkit has let me programmatically generate and create/display/organize simple things like chat bubbles very easily on the screen. However, I'm hitting a dead end wall. Once there are any significant number of elements on the screen, performance for simple things like scrolling a window grinds to a halt and it becomes unusable.

I checked the Unity Profiler and I'm getting spikes of 111 draw calls on this demo "chat" window when scrolling. Perhaps this is one for every visual element on the screen (chat bubbles, rows elements for aligning, spacers, text labels, etc.). I haven't counted but it sounds about right. This is not even half as complicated as I need my screen to be.

An SMS chat in Android/iOS for example can have hundreds of bubbles and scroll perfectly. A web browser can have 50+ pages of text/images/CSS and still scroll smoothly. This is what is expected by users and with basic modern technology.

I have looked online and from what I have seen I ought to have <60 draw calls for smooth function. However, I can find nothing on how to improve the draw calls or batch them for UI Toolkit.

The closest I found is this thread which discusses using Sprite Atlases to supposedly drop the number of draw calls: https://forum.unity.com/threads/feedback-wanted-sprite-assets-support.962958/#post-6270672

However, I tried adding a sprite atlas to my project, added to it the sprite (PNG 9-sliced in Unity) that is used as one chat bubble, included it in final project, set sprite atlas to "always" in editor settings, and then re-ran the profiler and no benefit. I was still getting peaks of 111 draw calls. If this is the solution, how do you implement it?

My project is dead in the water without a solution for this. I have put 8+ months into this project already and I need it to work in Unity as it is both a gaming and social app simultaneously. Is there some solution? Or is there some solution that could be imagined? How could this hypothetically be solved? Is there anything custom I could code that would fix it? Or some built in solution I'm missing or not understanding?

Currently just having a bunch of rectangles, rows, chat bubble sprites, and text labels on screen is crippling my top of the line Samsung S21+ and S10 phones. The S21+ can barely keep up and the S10 can't at all. This is obviously ridiculous considering how visually simple the result is and how powerful the devices are. It's not remotely usable.

 

Are you using ListView or ScrollView?

 

Thanks for your replies guys and sorry for my perhaps dramatic thread title. I was a little bit panicked because I thought I was hitting a performance wall and after all this time invested, the notion freaked me out. I am still mildly freaked out though I am encouraged by your replies and engagement.

Unity is 2020.3.21f1 and UI Toolkit is 1.0.0-preview.18.

I am not using a premade ScrollView although that is what I first did when I started with UI Elements in UI Builder. I noticed similar scrolling performance bottlenecks there. Currently, everything in my app is generated programmatically from C# (no UI builder) so I am not using premade things. I am simply moving the style.top of the scrolled element frame by frame based on clicked mouse movements or inertia calculations. This seems actually more efficient due to the simplicity but still yields bottlenecks.

Thanks for the suggestion. I spent my whole evening putting together a test project which demonstrates my basic UI build principle with a scalable number of elements and complexity. I then ran the profiler and some frame rate debugging to see what I was objectively getting.

The performance was slightly better than I thought it was last night, and likely survivable on both devices for my purposes though I don't know how well it will manage on lower tier devices (which would be most cell phones). I can certainly share the project I made to demo the performance. Before I send it, I would appreciate a bit more perspective so perhaps I can test a bit better first or ensure the test project captures the issues best.

From what I see, draw calls when scrolling/moving a parent element are roughly proportional to the number of visual elements moved though not precisely. For example, with about 1000 visual elements (combo of Label and VisualElement objects) on screen and moving I get around 1100 draw calls. With ~100-200 elements I get around 120-140 draw calls.

I can manage >= 60 fps up to around 500 small elements (eg. single word labels, simple flex containers for groups of them) which sounds like plenty. But I am still a bit wary as these phones are both very fast with quad and 8 core processors which might be 2-3 times faster than lower tier phones I am not sure how low other phones will peak.

Both phones degrade so once you get to around 1000 small/simple elements - the fastest S21+ can maintain around 45 fps and the S10 around 30 fps. While this will manage on these two devices I am not sure at this point if this is reasonable or not. If a phone is 1/3 the speed my limit might be 100 elements which is not easy for things like group chats or complex profile pages.

In a group chat for example, every chat message could be 4 or more elements (row to sort, author's small photo with round object to mask or add border, text label for message with sprite BG, report/reply buttons, timestamp, etc). At that rate, with a 500 element limit you can end up with only 100 messages before a top of the line quad or 8 core processor fails to keep up. A lower device might max out at 25-40 messages. This still does not seem quite reasonable to me.

There is one thing I am seeing which stands out in particular. I can't figure out where the insanely high vertex and triangle counts are coming from. When my app is idle with ~1000 simple elements I get ~38K triangles and 122K vertexes. This seems high enough already, but when I scroll, it goes up to an astronomical 45 MILLION VERTEXES PER FRAME.

Where on earth is it rendering 45 million vertexes from? In this example, there are less than 1200 words on screen and maybe 50 additional rectangle visual elements. Even if these are related to the vertexes of the text contours/rendering, it seems evident that multiplying the number of vertexes rendered per frame by 3750x every time the screen moves and trying to calculate 45 MILLION ANYTHING per frame would naturally slow things down, even on the fastest new quad and 8 core processors.

What are these 45 million calculations happening per frame to move ~1000 visual elements on a screen? Why does it need 45 million vertexes per frame to move everything even one pixel?

It seems perhaps if Unity was not trying to do such an insane amount of math for such a simple task the problem might be solved. If these are redundant calculations (which I presume they must be - there can't be 45 million useful vertex calculations of any etiology per frame in such a simple design), then fixing that redundancy would likely solve the issue and give the kind of performance one might expect.

I wonder if perhaps it is being triggered by some recursive cascade from the flex system. Ie. if every time one flex item resizes, it triggers all the others to resize back and forth and it keeps doing this continuously until it "settles." Although this doesn't make sense either as web browsers can arrange websites with many flex elements and not require 45 million calculations per refresh. I have built other systems in C++ using flex based systems that also did not require any similarly crippling amount of math to arrange.

I will be happy to submit my test project so you can take a look. I just wonder if you have any initial further thoughts before I do so that I can make the submission as useful as possible or do any more work on my end for narrowing the bottleneck down further. I will send it within the next day either way.

Thanks again for your interest and help. Honestly Unity is a fantastic design environment and the UI Toolkit is amazing. I have said so many times already on this forum. When I first started with Unity I tried building UI chats with game objects and vertical sorters, etc. It was unworkable just due to all the hierarchies and limitations of organization methods on screen..

UI Toolkit and the Yoga Flex system makes everything so easy to lay out by comparison. It's more like web design which makes sense for this purpose. It's easy to generate things and set their style by C#. There is no comparative system I am aware of that can let you blend such style of UI design and game development. This is what I need for my app. It is essential to what I'm making. But it seems there is something not quite right beneath the surface. I hope we can figure it out so the potential can be reached and I can brig my app to life.

Thanks.

 

While more complicated to use the ListView does not display all the elements but only the visible ones, so in your case it might be worth considering.

 

Is there (or will there be) any cheatsheet-like document/list for performance tips in UI Toolkit, @antoine-unity ? For example (of course it depends on the scenario), in web I'd use translate for simple element moving if I don't need to use margin or top/left/etc. (translate > margins > top/left/etc for performance wise most of the time). But in Unity, I don't know which is the performant way. I'd be nice if we can read some tips from the Unity team

 

Thanks for your updates and suggestions! All very helpful!

I was able to test the optimizations that were suggested. "usageHints.DynamicTransform" and "transform.position.y" applied to the moving element were both very highly effective for improving efficiency. (I tested "GroupTransform" as well and it was less effective in my situation.)

Both DynamicTransform and using "transform" to move things seemed to play a role in the optimization. Each individually helped my test app run smoothly in most cases on my devices. I got the draw count and vertex numbers down to a reasonable level when using both together.

There were two remaining issues I can see:

1) VERTEX SPIKES:
With both optimizations, vertex counts remain in the reasonable zone (tens to hundreds of thousands), but yet can still spike up to the tens or hundreds of millions periodically. Although I understand the exact number of vertices isn't critical, once you get up to tens or hundreds of millions there do seem to be performance issues that manifest, so I would sleep better if these weren't happening.

Both my devices I'm testing on as noted are very advanced and most of the world do not have phones like this. Slower devices would likely feel these spikes more. Editor certainly shows lag when they happen on my older laptop. If my app is stuttering randomly on slower phones when scrolling this can easily ruin the user experience.

I presume this is what you mean mcoted3d when you also said you have seen the spikes on the ScrollView, which also has these same optimizations built in. It would be nice not to have these crazy high spikes at all.

2) INITIAL FRAME DROPS:
With both optimizations, the first split second to 2 seconds of attempted scrolling triggers a massive drop in frame rate still. After this happens, everything becomes perfectly smooth in most cases and my mobile test devices can hold their max refresh rates with almost no deviation except possibly for a few frames here or there during the vertex spikes from #1.

This is evident for half a second or so even at small numbers of elements like ~400 but becomes up to 1-2 seconds of lag when dealing with massive numbers like 16,000 elements (torture test). Amazingly, my quad core S10 can still handle up 16,000 elements after this 1-2 seconds passes perfectly but during those first 1-2 seconds is dead and approximately zeroed out on frame rate.

Something is obviously happening during the first motion that then does not need to happen again after. If the brief heavy system burden of this is unavoidable, it would be nice to have a function to manually/automatically trigger it on startup instead. This way, even if the app takes an extra 1-2 seconds to load, once it is loaded, there will be no extra lags. I presume it might be possible to trigger it to get it out of the way during the first/second update frame with a single pixel motion but this is kind of hacky and I haven't tested it to see if it works.

This would not be an ideal solution either though, because let's say you are trying to allow a user to click through different user profiles or chat groups. If every time a new one loads, you must "stress" the system for 1-2 seconds to the point that the app becomes unusable during that time, even if you have control over when that "stress" happens, it will still be a problem.

The only workaround in that case then might be to trigger the stress for one or two movable elements on start up, then just recycle those same few movable element as parents and load/unload children to it every time you need another chatroom or profile view. Maybe that would make sense and prevent the burden each time. I'm not sure. On the other hand, perhaps changing their children will re-trigger this stress in which case it won't fix anything.

Rough Test Data
Here is my rough data I collected during testing.

Here you can see the occurrence of vertex spikes from issue #1 while both optimizations are applied. With ~1000 elements, it idles around 122K vertices at baseline but shoots up to 41 million periodically:

The following were my results for various test conditions to see what each optimization did (moving element via "top" vs. "transform" and enabling dynamic transform):

With around 1000 visual elements (roughly 20 groups of 50 labels):

1) "Top" motion + no DynamicTransform:
up to 48 million vertices with motion
up to 1100 draw calls
runs terribly on s10/s21 and in Editor

2) "Transform" motion + no DynamicTransform:
up to 48 million vertices with motion
up to 1100 draw calls
runs terribly on s10/s21 and in Editor

3) "Top" motion + DynamicTransform:
up to 46 million vertices with motion (but more typically ~26 million)
up to 1100 draw calls (but more typically 800 draw calls)
drops frame rate initially for half second of motion then runs smoothly on s10/s21
scrolling stutters on laptop in Editor during motion

4) "Transform" motion + DynamicTransform:
typically 120k vertices max during motion, but still with rare spikes to 41 million vertices
draw calls stable at 6
drops frame rate initially for half second of motion then runs smoothly on s10/s21
scrolling stutters on laptop in editor sometimes during spikes

Conclusion
We can see from those above tests both optimizations are necessary to stop the crazy high draw calls and most of the abnormal vertex counts. With both optimizations the draw calls drop massively and vertices remain (mostly) reasonable in motion.

Torture tests with both optimizations (Transform motion and DynamicTransform):

1) With around 4000 visual elements (roughly 40 groups of 100 labels):
motion gives typically ~300K vertices but with spikes up to 157 million
draw calls stable at 13
initially drops frame rate and unable to respond properly for first bit of motion then after 1 second handles 60 fps stable on s10
no obvious glitches in motion on s10 after first second of motion
scrolling does stutter on laptop in editor during vertex spikes

2) With around 16000 visual elements (80 groups of 200 labels):
1 million vertices at rest - unable to run motion on my laptop at all in profiler to see output and i don't want to move project to my faster desktop since I like working on my couch
draw calls stable at 22
unable to move at all for first 1-2 seconds of attempted motion but then able to scroll on s10 stable at 60 fps, with perhaps tiny rare fps drops (likely during vertex spikes) but minimally perceptible on fps readout.

I did not see any point in torturing anything further than that.

PROJECT/BUG SUBMISSION
Again, thank you for sharing those optimizations as they solved 90% of the issue. In case it is any help to debug the last two issues (remaining massive vertex spikes and initial laggy frame rate on first motion) I submitted my simple test project momentarily as a bug report with ID: 1398570_8t8ekpg7gr37416r.

The project is very simple with just one script on one game object which generates an ugly nonsense "user profile" from some random words and basic elements. A debug text to the top panel will show the target, smoothed, and real time frame rates.

There are 2 public fields on the script which control the number of tags (labels) per "section" of the profile and number of sections (groups) so it is easy to scale as per the above tests. I have a few public bools for controlling the various optimizations above as well, set by default as they should be.

The whole thing is just around 250 lines of code (and most of that is styling for the labels etc to make them show reasonably appropriately) so nothing big or hard to look through. The "scroll" mechanism is intentionally very rudimentary and childlike as I wanted to minimize any confounding issues.

Thanks again for your help and feedback. I look forward to any further solutions to the remaining issues.

 

You are right! Thank you for catching that. I didn't notice the vertex spikes were happening from my mouse leaving/entering the window.

This also explains why I was getting a massive drop in frame rate on mobile during first movement of the element. I thought it was movement that was triggering this. However, it is the same issue. On mobile, there is no "mouse cursor", so the first time you click the screen is the first time pointer enters the screen and it was triggering the same issue.

I updated unity as suggested to 2021.2 and it solved it. I get frame drops on the s10 during the initial moments of motion gestures down to the 50s (from 60 fps) with 16,000 elements on screen, but honestly that's an insane amount of elements and if it can handle that it can handle any practical application.

As a point of interest, the cost in terms of vertices and draw counts seems resolved by upgrading as well and applying the unityHints - whether I use "top" or "translate" or "transform" doesn't seem to matter anymore. I have good performance with all.

Is there any reason to still not use "top" if I'm getting good performance with no issues? Whatever you fixed in the upgrade seemed to have fixed this too. If there's any difference I don't see it anymore. The only optimization now needed is unityHints.

Any efficiency improvements are always appreciated of course. I didn't objectively time it but I feel like the Unity upgrade made my new builds sit on black screen for a bit longer after the splash screen than previous. Anyway, as long as it works I'm happy which it does now.

Yeah I was worried I would have to do a ListView type solution but at this point if I can handle thousands of elements at once there is almost no point. It seems almost impossible to overstress in practical use. If I have to, eg on a continuous group chat that could get too long, I would just keep a list of "chat entry VE's" then once the count is too high start destroying the old ones to keep the total count reasonable.

Thanks again so much. To summarize the point of the thread for anyone reading it:

- Upgrade to Unity 2021.2 to avoid rebuilding your entire UI when your mouse first enters on Windows or when you first click the screen on mobile.
- Use transform or translate to move objects, not "top" or other parameters (although doesn't seem to matter much in new version).
- Mark objects to be moved with usageHints.DynamicTransform

Perhaps you can retitle the thread "How do you efficiently move UI Elements?" or something of that nature so it is more useful for others and less inflammatory in tone.

I just have a few more questions. In your document here (https://docs.unity3d.com/ScriptReference/UIElements.UsageHints.DynamicTransform.html) you state: "Please note that the number of VisualElements to which this hint effectively applies can be limited by target platform capabilities. For such platforms, it is recommended to prioritize use of this hint to only the VisualElements with the highest frequency of transformation changes."

I am not asking for any specific quotes or hard data, but I am wondering what you mean by this. For example, if I am building for mobile (Android/iOS), how many elements could I likely have with such a tag on them in simultaneous existence (for example: scroll/chat windows, drag and drop images for re-arranging on a profile similar to an "inventory" system)?

Could you offer any more elaboration on what we are talking about here? Do we mean keep it would likely be good principle to keep it under 5? Under 10? Under 20? Under 50? Under 100?

Is it also about the total number of objects with this tag, or the total with this tag that are moving/transforming at any one moment in time? ie. if you have 50 with this tag, but only one is transforming at a time, is that likely much worse than just having 1 with the tag?

Does it matter if you then set the element with this hint on it to invisible or disabled? Eg.:

- ve.setEnabled(false);
- ve.style.display = DisplayStyle.None;
- ve.style.visibilty = Visibility.hidden;

Also if you apply it to a visual element then destroy that element, I presume that gets removed from the count right? What is the correct method to remove/destroy such a visual element or clear this hint? I have not yet had to do so.

For example, let's say I'm creating a new "movableElement" every time I generate a new chat view and applying DynamicTransform to it. If someone leaves the chat window, I would like to destroy this in their running app so it doesn't go to my count. Is this the right way to wipe that element first of all if I want to 'dispose' of it?

Code (csharp):

1. parentPanel.Remove(movableElement);
2.                 movableElement = null;

Would that get rid of the usageHint allocation or if not how could I? Alternatively, am I best just recycling the movableElement with the usageHint by clearing its content with "movableElement.Clear();" and reusing it for the next chat window?

I feel like recycling is probably best - I can keep say 2-5 of these "usageHint" elements in the background and just keep using them. But I wonder how it would work in both cases.

Lastly: Do I need DynamicTransform on objects for which I'm tweening properties like opacity or backgroundColor? Or is it only worthwhile if I am moving them on the screen?

Thanks again for the help and feel free to also mark the thread "Resolved."

 

Thanks for the clarification on the DynamicTransform limitations and for your help in this thread figuring this all out. I should also add a correction to my prior post:

It seemed moving by "top" no longer triggers the millions of vertices issues in Editor, but it still does trigger it in Android. When I debugged my main app from my Android device through the profiler I could only resolve the millions of vertices with both DynamicTransform AND switching to Translate.

Unfortunately I have just one more weird thing regarding this I have to ask you.

Since I updated to 2021.2 as suggested, for some strange reason the exact same project I sent you in the bug report no longer plays at 120 hz on my s21+. Any version from before the update does.

I even have my main real app which I updated also to 2021.2 and it runs at 120 hz on the S21+. So the throttling/limitation only affects the demo app I made, not my real app.

I don't understand why or what changed. The code for setting the frame rate is the same in both apps:

Code (csharp):

1. QualitySettings.vSyncCount = 0;
2.         Application.targetFrameRate = Screen.currentResolution.refreshRate;

Yet this outputs 120 hz on one app and only 60 hz now on the demo app I sent you (when that was previously running also at 120 hz). I changed nothing else in the demo app.

It is very puzzling. Can you think of any good reason or way to figure out why it is being throttled? Might it be Android throttling it?

I notice since the update this demo project hangs on a black screen for a bit before loading when it didn't before. I wonder if it's trying to load 120 hz then something is telling it not to and it reloads again 60 hz.

I have noticed there is another problem I was going to submit a bug report about and make a second thread regarding, but I now wonder if it's connected. When I set my Android devices to 1440p resolution, I still get maximum 1080 as the output from Screen.width in any Unity projects - ie. again, it seems something is throttling their resolution.

So it seems now this demo app can no longer reach full 120 hz system refresh rate. Just like none of my apps can get to full 1440p system resolution. I am not sure what this represents or how to fix it.

The throttling seems arbitrary because I have the same resolution setting mechanism on two apps and one (my real app) still gets 120 hz but this demo app no longer does.

Thanks for any further thoughts.