Object pooling and many different types of objects

Hi, I'm kind of stumped as to what the best approach would be for having a ton of different types of objects pooled?

For example, lets say I have a lot of plants, but those plants are extremely diverse, ie cacti, palm trees, etc. So lets say I have, in total, 500 different types of plants. The game will be hosted on a server with 8 players, where each player at worst case will be near 800 plants.

For simplicity sake, lets say none of these trees are calling any scripts and are managed by a manager class. On the server, all they have attached to them is a static box collider. While on the client they have an additional spriterenderer attached.

So now I'm wondering - how should I manage something this large scale on the server? At worst case I would need 500 pools with about 800 game objects each, which leads to roughly 400,000 game objects!