Optimizing A Multi-layered Bar Graph That's Always Moving

This is what it looks like: https://video.twimg.com/tweet_video/D3OS8bCUEAAVovz.mp4

So right now every layer of each bar is a RectTransform. Having a few of these graphs on screen is fine, but then I put a handful up and the performance tanks, especially when they are put into a ScrollRect. I feel like I made a lot of optimizations:

• Each bar is only updated when needed, meaning the RectTransform's aren't touched unless motion is required.
• The layers of each bar are comprised of rectangles which are in an object pool, with all graphs sharing the same pool.
• layers are sorted on value change, so each layer of the bar knows exactly what order it should be in.
• No overdraw, layers are drawn starting from the previous layer's position.
• If the graph is not visible, all rectangles are pushed back into the object pool for other graphs to use. Values are still computed though, so once the graph is visible again, it can pull all the needed rectangles from the object pool.

But it's still not enough. I have tried disabling the part of my code that does the rendering, while keeping the part that would update the value of the bars (to see if it was the rendering or all the math going on to calculate bar positions), and the performance impact was pretty small when it came to all the number crunching. I could have hundreds of them and, so long as they weren't rendering anything, at most It'd add 1-1.9ms. Now try to render 10 graphs, each with 20 bars and 2 layers, and suddenly we are adding another 12ms. Normally I'd shy away from optimizations this early on, but that's a massive performance hit for just the UI alone. It hits especially hard when all the bars are moving to the side to make room for a new bar, since most of these graphs are moving in sync. What could I try in this case? Is this one of those rare situations where IMGUI may outshine the new UI, or am I just using the new UI wrong? What other optimizations could I make?