Audio Read-ahead with AudioSource.GetSpectrumData()

I'm having issues with reading spectrum data ahead of time.
I'm trying to get level generation ahead of the player for a 3D rhythm game.

I've ended up using AudioSource.GetSpectrumData() and putting a delay effect 100% wet on the fader, so that the listeners audio gets delayed behind the reading from the file.

This works well enough, except it means that a delay of multiple seconds is a necessity at the start of the level, which is far from ideal.

Is there any way I can get spectrum data from a channel that isn't routed to the main out? Or a muted channel?
Even when using AudioSource rather than AudioListener, I'm still getting difficulties where the AudioSource just stops spitting out data when it's routed to a muted channel. Is it even possible to do this with Unity's built-in functions?

I'm avoiding as much as possible using an external FFT library, just to avoid headaches with platform portability.

Any insight would be awesome!