sharing camera data

hi. first, i'm sorry about my english.
i'm trying to make a program - using vuforia and webrtc in unity for android(like vuforia-chalk).
i can another device connect and video,audio call(from webRTC), but if i import vuforia, i can't connection other device.
because vuforia occupies the camera first, so webRTC can not access it.
So, i'm trying to send camera view(or Texture? or byte array? i don't know it is corrrect ) to webRTC or vuforia.

here comes the maddness :
1. how do i camera access at the same time from two library(vuforia and webrtc)
1.1. if i can't, how can i access vuforia camera?
1.2. otherwise, how can i send another video data byte array from webrtc to vuforia?

i'm so sorry because I don't think my intentions are properly included and my english is so bad...
thanks to read my question.

Ki Deok

 

Hello @Kideokk,

This cannot be done AFAIK. For most devices the camera is a singleton, so only one process can access at a time.

Vuforia provides APIs to control the camera, so perhaps there is an option for you to disable the camera while Vuforia is running to make it available to another process:

https://library.vuforia.com/content...rence/unity/classVuforia_1_1CameraDevice.html

Vuforia 7.2 SDK does not allow camera frames from an external camera to be injected. This is a feature currently being evaluated for productization.

Thanks,
Vuforia Engine Support

 

thank you answer for my question. and i read your link, i have questions.
i will trying send from vuforia camera data (using CameraDevice Class - GetCameraImage,and translate byte data)
to WebRTC. (i don't know it is available).

first - if i get vuforia camera data and send to WebRTC, 3D object is included?
second - after i will creat AR Pen in UWP environment, have you any reference?(like vuforia - chalk)

 

Hello @Kideokk,

No. These are the raw camera frames, as provided to Vuforia, before anything is rendered.

Unlike the Vuforia SDK, there currently are no sample apps that demonstrate use cases similar to Vuforia Chalk. This is a standalone product (and app) that we sell.

Thanks,
Vuforia Engine Support

 

so, i tried screen capture, it is included 3D object.
here is my screen capture code

Code (CSharp):

1. using System.Collections;
2. using System.Collections.Generic;
3. using UnityEngine;
4. using System.IO;
5.  
6. public class ScreenCapture : MonoBehaviour {
7.  
8.     // 4k = 3840 x 2160   1080p = 1920 x 1080
9.     public int captureWidth = 1920;
10.     public int captureHeight = 1080;
11.  
12.     // optional game object to hide during screenshots (usually your scene canvas hud)
13.     public GameObject hideGameObject;
14.  
15.     // optimize for many screenshots will not destroy any objects so future screenshots will be fast
16.     public bool optimizeForManyScreenshots = true;
17.  
18.     // configure with raw, jpg, png, or ppm (simple raw format)
19.     public enum Format { RAW, JPG, PNG, PPM };
20.     public Format format = Format.PPM;
21.  
22.     // folder to write output (defaults to data path)
23.     public string folder;
24.  
25.     // private vars for screenshot
26.     private Rect rect;
27.     private RenderTexture renderTexture;
28.     private Texture2D screenShot;
29.     private int counter = 0; // image #
30.  
31.     // commands
32.     private bool captureScreenshot = false;
33.     private bool captureVideo = false;
34.  
35.     // create a unique filename using a one-up variable
36.     private string UniqueFilename(int width, int height)
37.     {
38.         // if folder not specified by now use a good default
39.         if (folder == null || folder.Length == 0)
40.         {
41.             folder = Application.dataPath;
42.             if (Application.isEditor)
43.             {
44.                 // put screenshots in folder above asset path so unity doesn't index the files
45.                 var stringPath = folder + "/..";
46.                 folder = Path.GetFullPath(stringPath);
47.             }
48.             folder += "/screenshots";
49.  
50.             // make sure directoroy exists
51.             System.IO.Directory.CreateDirectory(folder);
52.  
53.             // count number of files of specified format in folder
54.             string mask = string.Format("screen_{0}x{1}*.{2}", width, height, format.ToString().ToLower());
55.             counter = Directory.GetFiles(folder, mask, SearchOption.TopDirectoryOnly).Length;
56.         }
57.  
58.         // use width, height, and counter for unique file name
59.         var filename = string.Format("{0}/screen_{1}x{2}_{3}.{4}", folder, width, height, counter, format.ToString().ToLower());
60.  
61.         // up counter for next call
62.         ++counter;
63.  
64.         // return unique filename
65.         return filename;
66.     }
67.  
68.     public void CaptureScreenshot()
69.     {
70.         captureScreenshot = true;
71.     }
72.  
73.     void Update()
74.     {
75.         // check keyboard 'k' for one time screenshot capture and holding down 'v' for continious screenshots
76.         captureScreenshot |= Input.GetKeyDown("k");
77.         captureVideo = Input.GetKey("v");
78.  
79.         if (captureScreenshot || captureVideo)
80.         {
81.             captureScreenshot = false;
82.  
83.             // hide optional game object if set
84.             if (hideGameObject != null) hideGameObject.SetActive(false);
85.  
86.             // create screenshot objects if needed
87.             if (renderTexture == null)
88.             {
89.                 // creates off-screen render texture that can rendered into
90.                 rect = new Rect(0, 0, captureWidth, captureHeight);
91.                 renderTexture = new RenderTexture(captureWidth, captureHeight, 24);
92.                 screenShot = new Texture2D(captureWidth, captureHeight, TextureFormat.RGB24, false);
93.             }
94.  
95.             // get main camera and manually render scene into rt
96.             Camera camera = this.GetComponent<Camera>(); // NOTE: added because there was no reference to camera in original script; must add this script to Camera
97.             camera.targetTexture = renderTexture;
98.             camera.Render();
99.  
100.             // read pixels will read from the currently active render texture so make our offscreen
101.             // render texture active and then read the pixels
102.             RenderTexture.active = renderTexture;
103.             screenShot.ReadPixels(rect, 0, 0);
104.  
105.             // reset active camera texture and render texture
106.             camera.targetTexture = null;
107.             RenderTexture.active = null;
108.  
109.             // get our unique filename
110.             string filename = UniqueFilename((int)rect.width, (int)rect.height);
111.  
112.             // pull in our file header/data bytes for the specified image format (has to be done from main thread)
113.             byte[] fileHeader = null;
114.             byte[] fileData = null;
115.             if (format == Format.RAW)
116.             {
117.                 fileData = screenShot.GetRawTextureData();
118.             }
119.             else if (format == Format.PNG)
120.             {
121.                 fileData = screenShot.EncodeToPNG();
122.             }
123.             else if (format == Format.JPG)
124.             {
125.                 fileData = screenShot.EncodeToJPG();
126.             }
127.             else // ppm
128.             {
129.                 // create a file header for ppm formatted file
130.                 string headerStr = string.Format("P6\n{0} {1}\n255\n", rect.width, rect.height);
131.                 fileHeader = System.Text.Encoding.ASCII.GetBytes(headerStr);
132.                 fileData = screenShot.GetRawTextureData();
133.             }
134.  
135.             // create new thread to save the image to file (only operation that can be done in background)
136.             new System.Threading.Thread(() =>
137.             {
138.                 // create file and write optional header with image bytes
139.                 var f = System.IO.File.Create(filename);
140.                 if (fileHeader != null) f.Write(fileHeader, 0, fileHeader.Length);
141.                 f.Write(fileData, 0, fileData.Length);
142.                 f.Close();
143.                 Debug.Log(string.Format("Wrote screenshot {0} of size {1}", filename, fileData.Length));
144.             }).Start();
145.  
146.             // unhide optional game object if set
147.             if (hideGameObject != null) hideGameObject.SetActive(true);
148.  
149.             // cleanup if needed
150.             if (optimizeForManyScreenshots == false)
151.             {
152.                 Destroy(renderTexture);
153.                 renderTexture = null;
154.                 screenShot = null;
155.             }
156.         }
157.     }
158. }
159.  

and i think that if i translate image to byte array and send to WebRTC camera, can't use it like?
am I in the wrong direction? or Is there any other way?
otherwise, it can't use(using two library : Vuforia and WebRTC)?

 

It's possible but the problem is frame drops.

 

Can you please explain how is this can be achieved? An example or a workflow will be helpful

 

Wikitude, Easy AR , etc