How does one set a Child of an Entity to a proper rotation?

So I've been using the parenting system inherent in ECS. I am generating my own objects completely newly, so I am not using the built in converter.

I create a float4x4 matrix and set it up right.

Code (CSharp):

1. float4x4 localMatrix = new float4x4(localRotation, localPosition);

Then I set up the component on the child as a LocalToParent with the following code.

Code (CSharp):

1. EntityManager.AddComponentData<LocalToParent>(entity, new LocalToParent() { Value = localMatrix });

It works fine as long as the rotation is a quaternity.identity. But as soon as I rotate it, I get crazy skewing on the child object. Am I setting up the matrix right? The parent has no scale component on it.

Do I need to do something extra to the localtoworld on the child before applying LocalToParent?