GameObject.SetActive Runtime Speed + Cloth Component?

I just upgraded from Unity 2019.3 to 2019.4.1f1

I'm experiencing some slow down in enabling / disabling a cloth component on objects ~4k polys. Is there a way to speed this up? I swap between objects with the cloth modifier mid gameplay in some cases, so it's not like I can ignore it. It was never noticeable in 2019.3, but in 2019.4 something is slowing it down, aggressively. I'll try any suggestions! Sorry if this has been asked, I had no idea what to search for.

• The object slowing things down is about 4k polys.
• I tried turning off the materials, and scripts on the object. The slow down only occurs when the cloth modifier is turned on and off.
• The slow down occurs even if the skinned mesh renderer is off and even if all the exposed cloth component values are set to zero/false.