Unity Multiplayer Ball game collisions

Hello people!

This is my first post

I have been searching A LOT, I asked people in the Unity and Mirror Discord server, but still couldn't solve this:

I'm creating a game that is simple in it's theory. I have a ball and two teams. Players have to shoot goals by colliding with the ball.
For players I am using NetworkTransform and NetworkRigidbody both having client authority.
For the ball I am currently also using both, but no client authority.

Problem:
The clients can't collide properly as the server can. It just looks unnatural and the forces seem to be much weaker. I was told that NetworkTransform affects this due to permanently sending position updates.

I tried:
Removing NetworkTransform and using a custom written position updating script helped a bit (updated via syncvars every 1000 ticks or so, but as expected it's inaccurate.

My current solution:
Disabling collision detection on the client side and using NetworkTransform. Clients see the collisions too late but at least it kinda works. However this way clients can't really "dribble" the ball since it's mostly inside the player on their screen.

What would be the best way to have okay-ish synced physics in this case? I really don't need a perfect solution.

Thank you kindly!