Remote raycast from tablet in headset

Hello,
I want to achieve a remote raycast. What I mean:

• See the image from the front camera of the Hololens on a tablet
• Touch the tablet screen to highlight parts of the scene inside the Hololens.

The Video part is already done, I just need some advice concerning the raycast.
Somewhere I learned that you can use the Camera.main for this, but after several attempts the ray doesn’t align with the actual viewport.
The screen position I send over network is normalized. I tried reconstructing it using Screen.width and height and also Camera.pixelWidth and pixelHeight and they both come to the same result:
The viewport of the main camera covers only a fraction of the actual camera.
You can see the result in the next photo.
The pink square is a visualisation of the Main.camera frustrum for raycasting.

There’s a bit of an offset on the bottom left that does’t appear at runtime but because of the screen capture of the HL portal. However the top right offset is pretty similar to what I get in the app.

How do you do proper raycasting on Hololens?

 

After some investments, I have found that it was due to my implementation that was incorrect. You effectively need to use Camera.pixelWidth and pixelHeight to get the right viewport.

 

hello, I also meet the question, I transport viewport point on network, then ray cast has mistake, any solutions?