Using Tilemaps inside Prefabs getting pink sprites

So I'm assuming you should be able to put a Tilemap in a prefab, but whenever I drag my prefab into the Scene (or load on the fly from script) they all come in with pink squares. If I rebuild all the tiles in Awake() it looks fine, but my game has a huge lag spike to do this. I would much rather build and save from editor and not have to rebuild all the tilemaps during run time. And yes I'm still sorting out the trees. : D