Performance spike on adding an object in VeticalLayoutGroup

Hello there I'm making a chat that can has lots of messages inside and the container for these messages uses VerticalLayoutGroup.
As more messages in VLG as more CPU load on adding new message.
It seems that each time when I'm adding a new message into VLG it calculates something but for all messages and not only for new one.
So is there any way to fix it?

There is next hierarchy:

Canvas
-Scroll View
--Viewport
---Content (Vertical Layout Group, Content Size Fitter)
----Panel (Image) lots of panels like this

 

is it a bug or a normal behaviour?
should I write my own analog of VerticalLayoutGroup?