Placing hololens holograms flush with understanding mesh

I'm trying to create a similar effect as in the RoboRaid game. Problem is I can't get my holograms placed correctly.

I've set up a simple scene with a mapping and understanding prefab from the Mixed Reality toolkit. I've set the mapping to nog draw meshes and the understanding draws the green wireframes.

As a simple proof of concept I'm trying to place a scaled down cube (0.1f) flush with a wall in the editor or emulator, but it won't work. I'm doing a QueryTopology_FindLargestWall (tried the object query one too), and then using the result position. I assume this position is the center of the wall, so the position.z should be flush with the wall. So when I'm assigning this position to my cube I subtract half the size from this point (from the Z position).

This never seems to fit. Worst of all in the editor when I move the camera the cube seems to move in and out of the wall. I've adde a worldanchor to te cube, but this does not change anything.

When looking in the emulator this effect does not happen, but it always seems to stick out of the wall a little bit. Anyone have an example in which this should work? I've based my code on the spatial understanding example in the MRTK examples.

 

So I tried some more things and I seem to have figured out what I'm doing wrong but somehow I'm missing the fix.

What I did now was just use the GazeManager to get the hit information when tapping.

Then when placing I apply the rotation and position to the object. I use the hit normal, but with flipped X and Z (to make the object face away from the wall).

And this is where I started running in to problems again. The object is not always place in the position that I targeted.
When trying the same stuff with a Quad it all works, and this is how most examples I could find also work by placing flat objects by drawing the box from scratch.

The problem is that trying to offset the object from the wall works a bit different after the object is rotated.

The issue with the box not being centered on the hit position makes sense when you move the box and thus change the center where it rotates around. And after rotating the axis seem to have shifted and I have to translate the based on the rotation.

So I'm missing something I just can't figure out how to do it.