Bug Accessing DownloadHandlerAssetBundle.assetBundle causes main thread to lock during parallel load

When loading multiple asset bundles in parallel, sometimes accessing DownloadHandlerAssetBundle.assetBundle property, will cause Unity to lock the main thread, which obviously causes the game to freeze. This only happens on some bundles but not others, even though they are loaded in parallel and there is no dependency between them.

When inspecting via profiler, it shows "Loading.LockPersistentManager" and it could be taking up to 2 (!) seconds. I searched for it but the only thing I could find is related to loading scenes (which isn't happening in my case)

1. The issue seems intermittent. I am unable to reproduce it with just an empty project, so I am not sure I can submit a proper bug report.
2. Making the bundles load in a sequence makes the issue go away (but increases the overall load time)
3. Changing the order in which calls are made, even though they are parallel, changes things a bit (the duration of the lock might change, OR other asset bundle will cause the lock). Removing certain bundles make the lock "move" to another bundle (even if removed one is unrelated to the lock)
4. Removing any LoadAsset calls that may happen in parallel, makes the issue better, however does not solve completely. I was able to reduce the lock time from 2+ seconds down to 30ms by delaying all LoadAsset calls after all bundles are loaded (even from unrelated bundles).
5. Setting DownloadHandlerAssetBundle.autoLoadAssetBundle to true does not seem to solve the issue, however it DOES solve the issue in an empty project, which makes me think something causes it to not work as expected?

Any thoughts or help is appreciated. It's very difficult to build large scale Unity project or stream any content in, when loading assets causes massive performance spikes with no clear path to resolve it without seeing the Unity source code.