VR experience for Oculus and Steam *Without* steamVR plugin

I'm trying to have a single build for both steamVR and Oculus without using the steamVR plugin but I have 2 problems:

1) Oculus is working through steamVR and it is much slower than when I remove steamVR from the project
2) If I put oculus first the camera gets sink on the ground

How does people go about these problems? do you have to do 2 different builds? if you publish to steam, can you specify different builds for different platforms?

Thanks