Culling non shadow caster objects

I have searched along the forum but didn't find anything related.

I've implemented a custom shadow map and I would need to do some culling on non shadow caster objects for the depth map. It would be nice if the camera who renders the depth map could cull objects that have unticked the "Cast Shadows" option in "MeshRenderer" component. There would be the option to search all Renderers in "OnPreCull" camera method and disable those with "Cast Shadows" value on false but it's very inefficient.

Short question: How to cull non caster shadow objects in an easy and performant way?