Unity UI Forcing a GridLayoutGroup to rebuild is not working as expected

I have an inventory system I'm trying to make using a GridLayoutGroup for a quick and dirty prototype. Everything starts up as normal and the inventory is built perfectly fine when I click run. Pressing tab activates the GameObject so that I can see the UI. Still fine. It looks like this:



But then lets say I drag and drop the first white square onto the first gold coin to switch them around.
It does this:



All of the item icons have been pushed off of the item slots (though the item icons switching places still technically works as intended).

What I have going on is a GridLayoutGroup panel with only the item slots, then a separate sibling panel without any kind of layout group that has all of the icons. The icons then have their anchoredPosition set to the respective inventory slot anchoredPosition.

To be able to do that, the inventory slot needs to be in the correct place when the anchoredPositions are set.

Here's the method for updating the entire UI based on a list called 'items' that's already declared and populated:

Code (CSharp):

1. public void UpdateInventoryUI(bool closeAfterUpdate)
2.     {
3.         ClearInventoryUI();
4.  
5.         if (items.Count != 0)
6.         {
7.             int index = 0;
8.             foreach (Item item in items)
9.             {
10.                 GameObject itemIcon = Instantiate(itemTemplate, parentOfInventoryItems.transform);
11.                 GameObject Inventoryslot = Instantiate(itemSlotTemplate, parentOfInventorySlots.transform);
12.                 itemIcon.SetActive(true);
13.                 Inventoryslot.SetActive(true);
14.  
15.                 itemIcon.GetComponent<InventoryItemIcon>().SetData(item, Inventoryslot, this);
16.                 Inventoryslot.GetComponent<InventorySlot>().SetData(index, itemIcon, this);
17.  
18.                 itemIconsList.Add(itemIcon);
19.                 inventorySlotsList.Add(Inventoryslot);
20.  
21.                 index++;
22.             }
23.  
24.             LayoutRebuilder.ForceRebuildLayoutImmediate(parentOfInventorySlots.GetComponent<RectTransform>());
25.  
26.             foreach(GameObject inventorySlot in inventorySlotsList)
27.                 inventorySlot.GetComponent<InventorySlot>().CenterItemIcon();
28.         }
29.  
30.         if (closeAfterUpdate)
31.             gameObject.SetActive(false);
32.     }

The strange part is that this method is used to initialize the UI and it works perfectly as intended. But when used to update it after removing an item or rearranging the order, the entire group of icons gets flung to the end.

To put it into numbers, both the first inventory slot and it's icon are expected to be at (37.5, -37.5) after forcing the layout group to rebuild and recenter, but they are actually at (487.5, -37.5) during this code execution.

Please. I have googled this problem so many different ways and can't find a working solution. I've been on building this UI for 3 days with nothing but problems, and It's just supposed to be a stinkin' prototype. Thank you in advance.