Performance Improvements when loading Sprites from AssetBundle

I have the following project structure:

1 Assetbundle with a Sprite Atlas, and all of the icons that make us that Sprite Atlas (a lot, lets say 100+ icons) (Assetbundle A)
1 Assetbundle with some prefabs (Assetbundle B)

Both Assetbundles are in StreamingAssets (but in the future A might be a DLC)

What I need to do in my game, is load a panel (from Assetbundle B), and based on some external config, a subset of icons from Assetbundle A.
I have all of this working, but loading of the icons takes a lot of time, and I'm trying to figure out how to optimize this.

Things I've tried:

1. For each one of the icons, go to the bundle, and get that icon
2. Grab the Sprite Atlas, and then, for each one of the icons, get the sprite I need

I though #2 was going to be much faster, since I would be loading the whole spritesheet from AB -> Memory once, but that wasn't the case. Each 'GetSprite' call took a lot of time.

Having references in my prefab to the sprites is not a feasible solution, because they are a ton of them.

Is there something I'm missing here?

 

@Stephan_B