Question Can someone explain this explanation?

I'm trying to use webcamtexture as the background of my app. I've used a similar method as the OP in the stack overflow thread which resulted in the same issues where the camera is zoomed in too much or there would be letterboxing. Could someone explain where or how the proposed solution should be applied.

https://stackoverflow.com/questions/71353035/how-to-get-live-feed-from-phone-camera-to-unity

"It's simply math issue, which I could share you this script for correct calculation. It's already been used in many projects in many years.

You only need to assign a Quad, and the Quad will be in the far background plane of your main camera.

Similar idea could be applied for UI Rect too. In theory, you should be able to achieve this result(video call testing in C#)."

Code (CSharp):

1. void CalculateBackgroundQuad()
2. {
3.     Camera cam = Camera.main;
4.     ScreenRatio = (float)Screen.width / (float)Screen.height;
5.  
6.     BackgroundQuad.transform.SetParent(cam.transform);
7.     BackgroundQuad.transform.localPosition = new Vector3(0f, 0f, cam.farClipPlane / 2f);
8.  
9.     float videoRotationAngle = webCamTexture.videoRotationAngle;
10.  
11.     BackgroundQuad.transform.localRotation = baseRotation * Quaternion.AngleAxis(webCamTexture.videoRotationAngle, Vector3.forward);
12.  
13.     float distance = cam.farClipPlane / 2f;
14.     float frustumHeight = 2.0f * distance * Mathf.Tan(cam.fieldOfView * 0.5f * Mathf.Deg2Rad);
15.  
16.     BackgroundQuad.transform.localPosition = new Vector3(0f, 0f, distance);
17.     Vector3 QuadScale = new Vector3(1f, frustumHeight, 1f);
18.  
19.     //adjust the scaling for portrait Mode & Landscape Mode
20.     if (videoRotationAngle == 0 || videoRotationAngle == 180)
21.     {
22.         //landscape mode
23.         TextureRatio = (float)(webCamTexture.width) / (float)(webCamTexture.height);
24.         if (ScreenRatio > TextureRatio)
25.         {
26.             float SH = ScreenRatio / TextureRatio;
27.             float TW = TextureRatio * frustumHeight * SH;
28.             float TH = frustumHeight * (webCamTexture.videoVerticallyMirrored ? -1 : 1) * SH;
29.             QuadScale = new Vector3(TW, TH, 1f);
30.         }
31.         else
32.         {
33.             float TW = TextureRatio * frustumHeight;
34.             QuadScale = new Vector3(TW, frustumHeight * (webCamTexture.videoVerticallyMirrored ? -1 : 1), 1f);
35.         }
36.     }
37.     else
38.     {
39.         //portrait mode
40.         TextureRatio = (float)(webCamTexture.height) / (float)(webCamTexture.width);
41.         if (ScreenRatio > TextureRatio)
42.         {
43.             float SH = ScreenRatio / TextureRatio;
44.             float TW = frustumHeight * -1f * SH;
45.             float TH = TW * (webCamTexture.videoVerticallyMirrored ? 1 : -1) * SH;
46.             QuadScale = new Vector3(TW, TH, 1f);
47.         }
48.         else
49.         {
50.             float TW = TextureRatio * frustumHeight;
51.             QuadScale = new Vector3(frustumHeight * -1f, TW * (webCamTexture.videoVerticallyMirrored ? 1 : -1), 1f);
52.         }
53.     }
54.     BackgroundQuad.transform.localScale = QuadScale;
55. }

 

Staring at code is rarely useful. You need to run it and reason about how it is failing and adjust it accordingly.

Here's a good approach you can start with:

What is often happening in these cases is one of the following:

- the code you think is executing is not actually executing at all
- the code is executing far EARLIER or LATER than you think
- the code is executing far LESS OFTEN than you think
- the code is executing far MORE OFTEN than you think
- the code is executing on another GameObject than you think it is
- you're getting an error or warning and you haven't noticed it in the console window

To help gain more insight into your problem, I recommend liberally sprinkling Debug.Log() statements through your code to display information in realtime.

Doing this should help you answer these types of questions:

- is this code even running? which parts are running? how often does it run? what order does it run in?
- what are the values of the variables involved? Are they initialized? Are the values reasonable?
- are you meeting ALL the requirements to receive callbacks such as triggers / colliders (review the documentation)

Knowing this information will help you reason about the behavior you are seeing.

You can also supply a second argument to Debug.Log() and when you click the message, it will highlight the object in scene, such as

Debug.Log("Problem!",this);

If your problem would benefit from in-scene or in-game visualization, Debug.DrawRay() or Debug.DrawLine() can help you visualize things like rays (used in raycasting) or distances.

You can also call Debug.Break() to pause the Editor when certain interesting pieces of code run, and then study the scene manually, looking for all the parts, where they are, what scripts are on them, etc.

You can also call GameObject.CreatePrimitive() to emplace debug-marker-ish objects in the scene at runtime.

You could also just display various important quantities in UI Text elements to watch them change as you play the game.

If you are running a mobile device you can also view the console output. Google for how on your particular mobile target, such as this answer or iOS: https://forum.unity.com/threads/how-to-capturing-device-logs-on-ios.529920/ or this answer for Android: https://forum.unity.com/threads/how-to-capturing-device-logs-on-android.528680/

Another useful approach is to temporarily strip out everything besides what is necessary to prove your issue. This can simplify and isolate compounding effects of other items in your scene or prefab.

Here's an example of putting in a laser-focused Debug.Log() and how that can save you a TON of time wallowing around speculating what might be going wrong:

https://forum.unity.com/threads/coroutine-missing-hint-and-error.1103197/#post-7100494

You must find a way to get the information you need in order to reason about what the problem is.

 

Thanks for the reply. There were a few tips in there that will help. I've tried adapting the function in to my script and created the variables but things like "baseRotation" have me stumped. I'm not understanding how this applies in the context getting video onto the screen. My guess is the webcamtexture is being put on to the quad instead of a canvas. My understanding of unity is limited.

 

I agree, according to my reading of the code you presented. The image is being cast onto a quad. So you are trying to get the web cam feed output on a UI component, as you mentioned canvas? The image being cast to a quad isn't necessarily incorrect, depending on what you're trying to create. If you're making a web conference type of application you can have the quad centered for example, and use the UI as a wrapper around that quad.

 

I'm trying to get the entire background be a live feed of the iPad's back camera. I found something on YouTube which achieves this but the feed will either be zoomed in and fills the screen or have a letterbox. I need to get what the iOS camera app shows.

I have references to a Quad mesh filter and the webcamtexture "background" from my original script. I'm not sure how to connect the two.
Should the CalculateBackgroundQuad function be applied to a Quad game object? How do I get the webcamtexture to project onto the quad? Sorry, I'm basically asking for a tutorial but any information is helpful and appreciated.

 

Code (CSharp):

1. public List<GameObject> photoPlanes = new List<GameObject>();
2.     public int fileCounter;
3.     [SerializeField] private Camera phoneCam;
4.     private PhoneObjectAppear phoneScript;
5.  
6.     private void Start()
7.     {
8.         Debug.Log(Application.persistentDataPath);
9.         phoneScript = GetComponent<PhoneObjectAppear>();
10.         //DeleteFileContents();
11.     }
12.  
13.     private void Update()
14.     {
15.         if(Input.GetKeyDown(KeyCode.P))
16.         {
17.             ApplyPngToPlanes();
18.         }
19.     }
20.  
21.     public void Capture()
22.     {
23.         if (phoneScript.phoneObj.activeSelf)
24.         {
25.             RenderTexture activeRenderTexture = RenderTexture.active;
26.             RenderTexture.active = phoneCam.targetTexture;
27.  
28.             phoneCam.Render();
29.  
30.             Texture2D image = new Texture2D(phoneCam.targetTexture.width, phoneCam.targetTexture.height);
31.             image.ReadPixels(new Rect(0, 0, phoneCam.targetTexture.width, phoneCam.targetTexture.height), 0, 0);
32.             image.Apply();
33.             RenderTexture.active = activeRenderTexture;
34.  
35.             byte[] bytes = image.EncodeToPNG();
36.             Destroy(image);
37.  
38.             string persistentFilePath = Application.persistentDataPath + fileCounter + ".png";
39.             File.WriteAllBytes(persistentFilePath, bytes);
40.             SoundManager.PlaySound(SoundManager.Sound.CameraClick);
41.             fileCounter++;
42.         }
43.     }
44.  
45.     public void ApplyPngToPlanes()
46.     {
47.         for (int i = 0; i < photoPlanes.Count; i++)
48.         {
49.             string persistentFilePath = Application.persistentDataPath + i + ".png";
50.             Texture2D tex = null;
51.             byte[] fileData;
52.  
53.  
54.             if (File.Exists(persistentFilePath))
55.             {
56.                 fileData = File.ReadAllBytes(persistentFilePath);
57.                 tex = new Texture2D(2, 2, TextureFormat.BGRA32, false);
58.                 tex.LoadImage(fileData);
59.             }
60.             photoPlanes[i].GetComponent<Renderer>().material.mainTexture = tex;
61.         }
62.     }
63.  
64.     public void DeleteFileContents()
65.     {
66.  
67.         string persistentFilePath = Application.persistentDataPath;
68.         string[] persistentFileEntries = Directory.GetFiles(persistentFilePath);
69.         foreach (string fileName in persistentFileEntries)
70.         {
71.             File.Delete(fileName);
72.         }
73.     }

Here is what i used for a VR project at work where the user has a VR cellphone and the images they take on the phone get applied to the planes or quads at the end so they can review. It probably has too much but definitely has what you need.

The dataPath stuff is just to save it to the headset or PC. You only need that file stuff if you are trying to save the images in a folder after they are captured.

 

Is there a way for unity to get the native camera resolution from the iPad and set the width and height to that for scaling?

 

Almost definitely, yes