Question Stop rigidbody from sticking to wall and sliding infinitely?

I'm trying to make a VR Rigidbody player, and I don't want the player to stick to walls if they collide with one. I have tried giving the capsule collider a physic material with 0 friction, but then the player infinitely slides across the floor if the player is on a slope.
The exact opposite happens when I un-apply the physics material, where now the player won't slide, but will get stuck to walls.
I've tried messing with Drag in the rigidbody but then the player falls very slow.
How could I go about making it so the player doesn't stick to walls and doesn't infinitely slide?
Any help would be much appreciated, thank you!

Here's my config: