Shadow pass shader batched (and didn't reset variables) even when textures are different

I have a custom geometry shader that does vertex displacement (modified from the DX11 grass shader asset). It also has shadows enabled, and due to the geom changes there is a custom shadow casting pass as well.

I have multiple meshes that use the same material that is referencing this shader. Each of them will point to a different texture to inform them of vertex displacement. When I render, I expected the shadows to conform to the actual vertices rendered in the normal passes. However, that's not the case. Looking at the frame debugger, it seems like the shadow passing will enable dynamic batching because something in the pipeline think the materials are the same (they are not and are instanced). Even with dynamic batching off, it will still set the same material on multiple meshes despite the instances.

This changes if I set one of the float variables in one of the instances to be slightly different during runtime (by script or by editor). This led me to think that texture references are not being taken into account during the shadowcaster pass.

Any ideas on how I can get around it without having to create tons of materials or modify random non-texture properties on the fly?