I should have seen it coming, as Resources has the same issue. To be able to figure out the type of an Asset, Unity loads it. It's dumb. That's what headers are for (or manifests), but clearly not in this case. So if you have an AssetBundle comprising many assets of many types, trying to retrieve only a specific type will make Unity load everything. The solution is to have AssetBundle "per-type" instead.