Any way to ignore postprocessing on lower layer camera?

My top layer camera has the postprocessing dof effect. But I want the DOF to only effect the objects moving forward and back in the space between the background and camera, and I want a sharp image as the background.

I have 2 cameras, one rendering on a layer above the other. Despite having different culling masks separating what the cameras are rendering, the top level camera blurs my crisp background and I'm wondering if there's any trick to only having the dof effect the objects between the background and the cameras.