Preserve speed magnitude on collision

Hi,
I am developing a slot car game, with physics handling as realistic as possible. The pad under the car moves within a track, and is able to change track by activating the appropriate lane changes (not visible in the video).

Both the pad and the rails have 0 as friction and bounce parameters.
The pad is a cylinder, and the car is rear-wheel drive.

The problem is that in the pad-binary collision, part of the speed is lost, and I can't find a way to compensate for it.

Inside the OnCollisionStay method, I have the information related to the collision. I would like to be able to apply the speed change here in the direction perpendicular to the normal of the point of impact, and in the direction in which the car is moving.

But in this method I have a list of contact points (so which one should I refer to?), And I have a single value that tells me the impulse that was applied to compensate for the bump.
From these can I deduce the lost speed, or should I calculate it as a simple difference between the current value and the value at the previous FixedUpdate? (but in this case I don't know if the difference is due to the collision, or to the speed change of the car itself)

Do you have any suggestions for me?

Thanks a lot!

Here's a video, on the top-right the speed which decrease on collision:

 

Not what you asked but do you really need the colliders on the track ? It doesn't feel smooth because we see an abrupt turn everytime the car hits a collider. I think it would be more performant to just guide the car with code.

 

I already did with a follow-path mode, but I would like to do it entirely guided by physics.
It can be done.
I was able to find a way to lessen the problem a bit, but it's still not what I'd like it to be.

This one has been done as I would like to do: