No angular velocity from initial bullet (physics object) impacts

First off, please take a look at this, particularly the first part where I'm shooting the corner of the floating cube...

I would expect the cube to spin around it's centre of mass from the very first bullet hit - but it just gets pushed away for a while before it finally starts spinning and reacting to the bullet impacts as I would expect.

The shootable objects in the air are have size (1), mass (50), drag (10) and angular drag (0.05), with gravity turned off. There are no constraint checkboxes checked - it has free movement.

Collisions are set to Continuous Dynamic and the physics settings are all at default.

The bullet has a weight of 1 and a velocity of around 200, so it's hitting them with a large amount of force and the collisions are clearly detected.

I can't for the world work out why the floating objects doesn't spin from the first shot.

I've tried more accurate physics settings, increasing/decreasing the mass/drag/friction/bounciness of all the objects involved, different collision settings, but it always just gets pushed for a while before Angular Velocity seems to have any effect.

Can someone let me know what I'm missing please?

 

I'd start by setting the maxAngularVelocity on the cube to something higher than the default:

https://docs.unity3d.com/ScriptReference/Rigidbody-maxAngularVelocity.html

The default seems to be very lower for things like this, so maybe set it very higher to see if that helps. You can set maxAngularVelocity via a script you can add to the object.