Question Get and Set Camera pixels in Unity.

Is there anyway to get AND set the output of a Camera component (or any solution that would mimic this)? For clarity, I want to send literal camera pixel data over a network for my application so that the client would not have to internally process events and would just be viewing them.

I've looked through the Camera API and have not found a way to get/set pixel output. Any reply would be extremely helpful, thanks in advance.

 

https://docs.unity3d.com/ScriptReference/RenderTexture.html might be the path you're looking for.

 

If you want to get the pixel data from the screen, there are several methods. But the main one is using

tex2D.ReadPixels()

to copy data from the currently set render target to a

Texture2D

.

Inversely you can render to the screen by using

Graphics.Blit()

with a Texture2D.

However...

ReadPixels()

is very slow, and sending uncompressed images across a network, even a local network, is also very slow. Hence the reason video compression exists. Unity has a utility for capturing footage in real time (via the Unity Recorder package), but it only works in the editor. And there aren't any tools for sending or playing back those streams over the network. And it's not a trivial thing to get working.

There's a reason services like Parsec exist.

 

After going through the API this doesn't seem to be what I am looking for. It does use what Im looking for though. I want to access the data that the camera is sending to the RenderTexture. Which unfortunately you cannot grab from the RenderTexture itself. It is a step closer though, so thanks either way.

 

This seems to be exactly what I was looking for. And thank you for the heads up about the image compression, I wasn't thinking about it in that way so I would've missed it. It somehow flew over my head that a generic 1600 x 900 screen would generate 1,440,000 pixels.

I will try to see if there is a way to modify and rethink my approach, however this method will probably be scrapped due to the speed. But I appreciate the reply either way.

 

Here's my rough solution which I derived using the comments above (mainly bgolus, thank you!). This takes a grab of the screen (camera), copies RGB (ignoring the alpha) pixel information to a byte array, then sends the array to an external app via tcp every frame. As pointed out above, network code wouldn't support high bandwidth of HD resolution so I limited the screen resolution to 400x400. (Another solution might take the full HD screen resolution and reduce it by taking a pixel, say, every 10 pixels.) A 400x400 resolution is RGB 480,000 bytes.

The following can be attached to any in game object that has a renderer, like a cube.

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using System.Diagnostics;
4. using System.Net;
5. using System.Net.Sockets;
6. using System.Text;
7. using UnityEngine;
8.  
9. public class CubeNet : MonoBehaviour
10. {
11.     // Use this for initialization
12.     TcpListener listener;
13.     string msg;
14.     bool connected = false;
15.     TcpClient client;
16.     NetworkStream ns;
17.  
18.     public Renderer screenGrabRenderer;
19.     private Texture2D destinationTexture;
20.     private Color32[] pixels;
21.     private static byte[] pixelsComp;
22.  
23.     void Start()
24.     {
25.         listener = new TcpListener(IPAddress.Parse("127.0.0.1"), 55001);
26.         listener.Start();
27.         print("is listening");
28.  
29.         screenGrabRenderer = GetComponent<Renderer>();
30.         destinationTexture = new Texture2D(Screen.width, Screen.height, TextureFormat.RGB24, false);
31.         screenGrabRenderer.material.mainTexture = destinationTexture;
32.         Camera.onPostRender += OnPostRenderCallback;
33.         pixelsComp = new byte[480000];
34.     }
35.  
36.     void Update()
37.     {
38.         if (connected == false)
39.         {
40.             if (listener.Pending())
41.             {
42.                 print("socket connected");
43.                 connected = true;
44.                 client = listener.AcceptTcpClient();
45.                 ns = client.GetStream();
46.                 byte[] bytes; bytes = Encoding.ASCII.GetBytes(msg);
47.                 ns.Write(bytes, 0, bytes.Length);
48.             }
49.         }
50.         else
51.         {
52.             byte[] buffer; buffer = new byte[client.ReceiveBufferSize];
53.             int bytesRead; bytesRead = ns.Read(buffer, 0, client.ReceiveBufferSize);
54.             if (bytesRead > 0)
55.             {
56.                 ns.Write(pixelsComp, 0, pixelsComp.Length);
57.             }
58.         }
59.     }
60.  
61.     void OnPostRenderCallback(Camera cam)
62.     {
63.         if (cam == Camera.main)
64.         {
65.             Rect regionToReadFrom = new Rect(0, 0, Screen.width, Screen.height);
66.             destinationTexture.ReadPixels(regionToReadFrom, 0, 0, false);
67.             pixels = destinationTexture.GetPixels32(0);
68.  
69.             //Compile pixel information sequentially in an array
70.             for (int i = 0, j = 0; i < pixelsComp.Length; i++, j++)
71.             {
72.                 pixelsComp[i] = pixels[j].r; i++;
73.                 pixelsComp[i] = pixels[j].g; i++;
74.                 pixelsComp[i] = pixels[j].b;
75.             }
76.  
77.             //System.Array.Reverse(pixels, 0, pixels.Length);
78.             //destinationTexture.SetPixels32(pixels, 0);
79.             //destinationTexture.Apply();
80.         }
81.     }
82.  
83.     // Remove the onPostRender callback
84.     void OnDestroy()
85.     {
86.         Camera.onPostRender -= OnPostRenderCallback;
87.     }
88.  
89. }