loadLevel + loadLevelAsync = freeze?

Hi,
So right now we're loading the next level with loadLevelAsync when the level begins, so we with at the end of the level can switch level without loading the next level at the end. If we instead want to jump back to the main menu(which is in another scene) we then use the standard loadLevel.

But when having one level loaded with loadLevelAsync and then loading another one with loadLevel Unity completely freezes. I've tried to null the AsyncOperation to somehow unload the loaded level, since I can't find any other method for unloading what you've loaded, but this does not work either.

Anyone encountered this?