Bug Accessing ARSession nativePtr and setting it's delegate breaks things in ARFoundation in unity

I am building a native iOS plugin in which I want to handle native ARSession events from ARKit,
I pass the ARSession nativePtr to the plugin once the session is initialized in Unity.
Then in my plugin I set the ARSession delegate to the class I have defined in my plugin.
On setting the delegate, I get the the callback fired for session didUpdateFrame, didAddAnchor, didUpdateAnchor, didRemoveAnchor delegate methods, but in Unity things are broken, I don't see the mesh on setting the delegate, the planes get added but they don't get updated,

Any reason why this could be happening.

Can I do something to get the native ARSession delegate callback events from ARKit without breaking things in unity?

 

@tdmowrer ?

 

@todds_unity @DanMillerU3D