Modifying the camera frustum breaks all Mouse events in 3D world space

I've been using the effect described on this page https://docs.unity3d.com/Manual/ObliqueFrustum.html to give my 2.5D game an oblique perspective. It's been working great. However, now I am implementing clicking and mouse events on certain game objects and I find that it is not working. Once you modify the camera frustum as described on that page, it breaks all mouse events (not UI obviously), rendering all the handy onMouseDown, etc functions completely useless.

I made my own test using Camera.main.ScreenPointToRay(Input.mousePosition) and I can see that the the ray is not being drawn in the correct spot once you modify the frustum. It is not accounting for the modified frustum. I imagine Unity is using something similar internally as well. I should be able to get around this by writing my own ScreenPointToRay(), but that seems a little sloppy, and it sucks that I won't be able to use any of the built in mouse methods on game objects. Using my own method likely means I'll have to constantly poll to see if the mouse is down in an Update method as well, also a little sloppy.

Not sure if this is known, or if I should report this as a "bug".