Bug PhysicsDebugDisplay spams NullReferenceExceptions

I'm currently trying to debug an issue with colliders in Unity Physics - but when I add PhysicsDebugDisplay to the relevant components, all I get is the console spammed with:

Code (CSharp):

1. InvalidOperationException: The NativeArray has been deallocated, it is not allowed to access it
2. Unity.Collections.LowLevel.Unsafe.AtomicSafetyHandle.CheckReadAndThrowNoEarlyOut (Unity.Collections.LowLevel.Unsafe.AtomicSafetyHandle handle) (at <2feaf16e80004e0cadae3f2e05f2a3fa>:0)
3. Unity.Collections.NativeArray`1[T].CheckElementReadAccess (System.Int32 index) (at <2feaf16e80004e0cadae3f2e05f2a3fa>:0)
4. Unity.Collections.NativeArray`1[T].get_Item (System.Int32 index) (at <2feaf16e80004e0cadae3f2e05f2a3fa>:0)
5. Unity.Physics.Authoring.DisplayBodyColliders+DrawComponent.OnDrawGizmos () (at C:/GameDev/Packages/com.unity.physics@0.4.1-preview/Unity.Physics.Hybrid/Utilities/DebugDisplay/DisplayCollidersSystem.cs:566)
6. UnityEngine.GUIUtility:ProcessEvent(Int32, IntPtr)

Interestingly, I have seen the DisplayBodyColliders/Draw... before, when I didn't have any PhysicsDebugDisplay(Authoring) components attached to any objects. But those showed up quite unreliably.

What I did see when they did show up was that even though LocalToWorld seemed to have updated correctly (those use CopyTransformFromGameObject, and they are Kinematic), the collider was still shown at the original position (Translation also does not get updated but I believe CopyTransformFromGameObject directly writes to LocalToWorld ... but PhysicsCollider using Translation instead of LocalToWorld could explain the behaviour).