Any mesh change in Blender massively screws up Polybrush edited object in Unity (Polybrush unusable)

I'll explain with the pictures posted along with this thread.

1) I texture paint on a Blender file. Looks fine.
2) I go into Blender and change anything. In this pic I simply raised up one of the faces.
3) Now in Unity the object is screwed up immensely. You can see there are holes in the mesh and the textures are painted in ridiculously off ways.

This is incredibly infuriating and makes this absolutely unusable. Please help.

Edit:
4) If I "Delete Vertex Streams" on the object this is the result.