grab non rect texture from screen and map to texture (please read description)

Here is the problem,

I have a drawing on an A4 sheet that I frame with my mobile camera and I track with vuforia
(so I know it's rotation scale and position relative to the camera)

I need to be able to save that drawing onto a texture (with the same aspect ratio as the sheet)
and it would basically stretch the corners to fit into the texture.

I have no idea on how to do this, I know how to grab pixels on screen but I don't know how to transform them
if anyone could please help as I wanted to do this in a shader and not on cpu

 

You Do It With Math!


Okay, so you've got a camera and a tracked object. Presumably you know the corners (ie: you have a simple quad mesh that's aligned to what's visible in the camera). The trick is to render to a render texture using the mesh's UVs as the screen position, and using the camera's projection matrix as a "projector" rather than the actual rendered screen position.

// vertex shader
// clip space xy is in -w to w range, so UVs should
// be in 0.0 to 1.0 range for this to work.
// The z of 0.5 is arbitrary, just a value that's
// between the 0 and 1 of Direct3D and -1 and 1 of
// OpenGL's clip space to ensure it won't get clipped.
o.pos = float4(v.texcoord.xy * 2.0 - 1.0, 0.5, 1.0);

// pass the screen position as calculated from the usual clip space calculation
o.screenPos = ComputeScreenPos(UnityObjectToClipPos(v.vertex));

// fragment shader
float2 screenUV = i.screenPos.xy / i.screenPos.w;
return tex2D(_CameraTex, screenUV);

 

Thank you for taking the time to answer

I have to ask for more support on this (I do hate myself for doing so)
I generally tend to come up with my own solutions but with shaders I really don't know what to do.

would you be so kind to help me write the shader?
I understand the fundamentals and what I should do but I really am clueless here

I don't know how to render to a texture within a shader, nor do I know how to set up the shader properties.

Tell me if I got this right so far,

I need to place a quad oriented like the real life drawing (so that it basically overlaps it entirely), then run this shader on that quad and I would come up with the texture i need?

p.s. I am willing to pay you, now you see how desperate I am =s

 

Roughly, yes. There's a little more setup required from the c# side of things because ...

A shader just renders to whatever the current render target is. By default that's the frame buffer (ie: what appears on your screen), but you can override that to be a render texture instead. The only special thing you'd need to set on the shader would be the mobile camera image that the AR camera is using as the background. I've never used Vuforia, so I have no idea how to go about that exactly.

However it seems like this is something someone out there should have already done at some point, and sure enough they have:
https://github.com/maximrouf/RegionCapture

 

Where do I send the insanely huge pie I am baking for you?

Thank you =)

 

Now the only problem is lens distortion from the real camera, I think I'll just leave it like this and prompt the user to frame the drawing from the top instead of sideways

edit: which apparently doesn't help one bit =(