What's the best equivalent of Graphics.DrawMesh but for raycasting?

Suppose I have objects that don't really have transforms because I display them all using Graphics.DrawMesh.

Ideally, I'm looking for something like Graphics.DrawMesh, but for raycasting, where I pass the position, rotation, and a mesh, and then I find out whether the raycast hit it and at what point.

I'm aware of Collider.Raycast which is almost there, but requires I instantiate GameObjects with colliders attached for all my meshes. I was using Graphics.DrawMesh to avoid instantiating GameObjects for everything.

I suppose I could create one Collider and then assign it a new mesh and position each time I need to. But I don't know what sort of performance hit that will cause. I know I only need to test one or two meshes in any single frame.