CharacterController doesn't get pushed by objects?

Okay so I don't know what I am doing wrong, but due to some reason my player with a character controller component isn't getting pushed by objects ( having simple box colliders etc. ). The behaviour is very weird. I have also shared a link to a video where I have demonstrated my issue.

So in the most basic setup, I have -
• Capsule - Player ( Character Controller and a BasicMovement script )
• Cube - Big ( Box Collider )
• Cube - Small ( Box Collider )

So if I just move the player it ofcourse get stopped by these two cubes. But if I move the cubes towards player instead then they either just went straight ahead without colliding , or only the big cube collides. Not the small one. Not sure what is happening wrong here. Here's the code -

BasicMovement script

Code (CSharp):

1. CharacterController characterController;
2. public float speed = 2f;
3. public bool enableGravity; // Just for debugging purpose
4.  
5. private Vector3 velocity;
6.  
7. void Start()
8. {
9.     characterController = GetComponent<CharacterController>();
10. }
11.  
12. void Update()
13. {
14.     Vector2 dir = new Vector2(Input.GetAxis("Horizontal"), Input.GetAxis("Vertical")).normalized;
15.  
16.     if(enableGravity)
17.     {    
18.          // 1 out of 2 cubes collide and move the player
19.          velocity = transform.forward * dir.y + transform.right * dir.x + Vector3.up * -2f;
20.     }
21.     else
22.     {        
23.          // 0 out of 2 cubes collide and player remains stationary
24.          velocity = transform.forward * dir.y + transform.right * dir.x;          
25.     }
26.  
27.     characterController.Move(velocity * speed * Time.deltaTime);
28. }

What I have found is that if I use a velocity on characterController.Move() which has a normal x and z velocity but a 0 on the y axis, then none of the cubes is able to push the player. But if I add some y value to the velocity like -2 then the cube is able to push the player. But only the larger one. The smaller one is still unable to push the player. This is a very weird behaviour, don't know what I am doing wrong.

Here is a video demonstrating my problem -