How to ignore shadows from other layers?

Hi, I have a problem with shadows.

I'm doing some fancy stuff with stencil buffers where I draw 2 scenes at the same time. To handle lighting individually, each scene is also on their own layer and has their own lights. That includes setting the culling masks up in a way that lights only consider elements from the own layer.

But still on surfaces of the one layer I see shadows of the other. That seems to be some screenspace mapping in the end.
Is there a way to make surfaces only receive shadows of lights which defined their own layer in the culling mask?