Question Bullet colliding with the player

I'm creating an FPS and I tried to implement more realistic shots, everything was working fine until I noticed that the bullet seemed to be colliding with something invisible, after some tests I noticed that when I disabled the "Character Controller" of the Player the bullet stopped colliding with the invisible object,
but I couldn't find any way to make the bullet ignore the player, I'm kind of stuck.
NOTE: the strange thing is that the bullet only collides with the player when I'm looking straight ahead, everything works if I look from any other direction.