Question Android Get Camera Pixels

Hello,

I'm developing a mini-game using AR Foundation, where I have to track an Image (something will appear), and then a random color will be given and I'll have to search around the (real) world for that color (for example, pointing my camera at the wall to get a whitish color).

So, for the AR part I think it's pretty done, but ... here's the problem, I've been searching and searching for a way to access my android camera and get the data from its pixels, so I could do some math and try to find the closest color to the randomized color.

I've tried to use some examples given by the documentation of the AR Foundation but so far nothing worked... Then I tried to use WebCamTexture... well it was the closest I got... actually worked but only when i was using my WebCam(Computer), as soon as I built and tested it on my Android it didn't work... (at least worked 1 frame..)

So I've set a simple scene with an AR Session and AR Session Origin, and created a simple UI, with 3 texts:
On that gives me the WebCamTexture.isPlaying, one to control how many seconds as passed (I called Frame Counter for some reason...) and one more that would give me the RGBA of the texture (pixel) given by the camera.

As I said it worked if I used my PC Web Camera, every second the RBGA value changed and matched some standard colors (like blue and Black) but.. when I Built the project to my Android that same value only changed at the start of the game, after that, it stayed the same until I closed exe.

The fun part (or not...) is that I first thought the game crashed but then I saw my Counter it was still incrementing the value.

This is my Test code... if someone could help me out figuring how to access the camera the right way I would appreciate it!

Code (CSharp):

1. public class GM : MonoBehaviour {
2.    
3.        
4.         [Header("UI")]
5.         [SerializeField] Text isPlaying;
6.         [SerializeField] Text frameRate;
7.         [SerializeField] Text colorData;
8.    
9.    
10.         [Header("Android Settings")]
11.         [SerializeField] int currentCamIndex = 0;
12.         [SerializeField] WebCamDevice device;
13.         [SerializeField] WebCamTexture webCamTexture;
14.         [SerializeField] bool isDeviceRunning = false;
15.    
16.         [Header("Color Settings")]
17.         [SerializeField] Texture texture;
18.         [SerializeField] Vector3 selectedColorVector;
19.         [SerializeField] Vector3 currentColorVector;
20.         [SerializeField] float r = 0;
21.         [SerializeField] float g = 0;
22.         [SerializeField] float b = 0;
23.    
24.         [Header("Timer Settings")]
25.         [SerializeField] bool enableTimer = false;
26.         [SerializeField] float timeToProcess = 2f;
27.         [SerializeField] float timeElapsedOnProcess = 0f;
28.    
29.         [Header("Developer Settings")]
30.         [SerializeField] bool isRunning = false;
31.         [SerializeField] bool clearData = false;
32.         [SerializeField] int frameCounter = 0;
33.    
34.         private void Start() {
35.    
36.             // Register the Device
37.             RegisterDevice();
38.    
39.             isRunning = true;
40.         }
41.    
42.    
43.         private void RegisterDevice() {
44.             device = WebCamTexture.devices[currentCamIndex];
45.             webCamTexture = new WebCamTexture(device.name);
46.             isPlaying.text = "Is Playing: " + webCamTexture.isPlaying;
47.         }
48.    
49.         private void Update() {
50.            
51.             if(isRunning) {
52.                 if(!isDeviceRunning) {
53.                     webCamTexture.Play();
54.                     isDeviceRunning = true;
55.                 }
56.    
57.                 if(timeElapsedOnProcess >= timeToProcess) {
58.                     timeElapsedOnProcess = 0f;
59.                     DisplayInformation();
60.                 } else {
61.                     timeElapsedOnProcess += Time.deltaTime;
62.                 }
63.    
64.             }
65.         }
66.    
67.         private void DisplayInformation() {
68.    
69.             frameCounter++;
70.             isPlaying.text = "Is Playing: " + webCamTexture.isPlaying;
71.             frameRate.text = "Frame Counter: " + frameCounter.ToString();
72.             if (webCamTexture.didUpdateThisFrame)
73.                 colorData.text = webCamTexture.GetPixel(0, 0).ToString() + " : " + frameCounter.ToString();
74.             else
75.                 colorData.text = " : " + frameCounter.ToString();
76.         }
77.     }

 

https://docs.unity3d.com/Packages/com.unity.xr.arfoundation@4.2/manual/cpu-camera-image.html