Make a collider collide with an object but not vice versa?

So I have two gamobjects with colliders, in this case an ammo box and a player, and I want the player to collide with the ammo box, i.e. they have to walk over it rather than through it, and I want the ammo box to still have physics and collide with the floor, but I don't want it to collide with the player, i.e. the player knock it over. As far as I can tell, with the collision matrix and things like Physics.IgnoreCollision you can only make it ignore collision both ways, rather than just one. Does anybody have any idea if this is even something that can be done in unity?