HoloLens sharing with UWP application for PC

I am trying to build a UWP application for the PC that can manipulate the hologram shown in the HoloLens application. I was pretty sure that this was possible, but I am stuck at the moment. I have this simple application with manipulation options at the moment and I tried to achieve this with the sharing capabilities of the toolkit shown in the 240 Microsoft academy. I figured that this should be possible since both applications are on the UWP. The desktop application and HoloLens application do connect to the same room and session through the SharingService, but I am not sure what to do next.

 

Maybe this can help you: https://mtaulty.com/2017/02/21/windows-10-uwp-hololens-a-simple-two-way-socket-library/

 

I tried that, but it didn't seem to connect. I didn't get any error messages as well, so it was kind of hard to determine what the issue was. I eventually achieved this by using UNet which works pretty well. The only problem is that I want the desktop application to be the server instead of the client, but that doesn't work at the moment. It does work vice versa though.