Sprite Packer in AssetBundles

I'm using the built in Sprite packer with tags in my game. I'm also using Asset bundles. I have created a bundle that references the all Sprites (not the pngs/textures) from a single tag. This creates a bundle with the sprite atlas that also for some reason includes the pngs themselves (I think this is a bug??)

However, the major problem I'm having besides that is that my sprite atlas is being included in EVERY bundle that has an object pointing to one of these sprites. I'm not sure if this is a bug (Unity people swear its supposed to work correctly but I'm calling LIARS)

Does anyone have experience with this on how to make it work??