Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

Get WebcamTexture pixel data without using GetPixels32()

Discussion in 'General Graphics' started by Red_Dragon69, Jul 29, 2022.

  1. Red_Dragon69

    Red_Dragon69

    Joined:
    Sep 7, 2015
    Posts:
    113
    Hi,
    I'm currently trying to get WebcamTexture data without using GetPixels32(). I ended up using the following code, which should work, as long as Multithreaded Rendering is turned off in PlayerSettings. But Unity crashes every time and I don't know why :( Has anybody already achieved smth with a similar approach? WebcamTexture itself works, the RawImage is just for checking the obtained pixels. It seems that I made a mistake at Marshal.Copy, but I can't figure it out...

    Code (CSharp):
    1. IntPtr imgBufferPtr = webcamTexture.GetNativeTexturePtr();
    2. Texture2D texture = new Texture2D(webcamTexture.width, webcamTexture.height, TextureFormat.RGB24, false);
    3.  
    4. int bufferSize = webcamTexture.width * webcamTexture.height * 3;
    5. byte[] rawData = new byte[bufferSize];
    6. Marshal.Copy(imgBufferPtr, rawData, 0, bufferSize);
    7. texture.LoadRawTextureData(rawData);
    8. texture.Apply();
    9.  
    10. // display temporary achieved image
    11. rawImage.material.mainTexture = texture;
    Bug Logs:
    Obtained 27 stack frames

    0x00007ffe36c19f49 (mono-2.0-bdwgc) [icall-def.h:824] ves_icall_System_Runtime_InteropServices_Marshal_copy_from_unmanaged_raw

    0x000001da2719edff (Mono JIT Code) (wrapper managed-to-native) System.Runtime.InteropServices.Marshal:copy_from_unmanaged_fixed (intptr,int,System.Array,int,void*)

    0x000001da273cc893 (Mono JIT Code) System.Runtime.InteropServices.Marshal:Copy (intptr,byte[],int,int)
    ....


    Best regards :)
     
  2. Kurt-Dekker

    Kurt-Dekker

    Joined:
    Mar 16, 2013
    Posts:
    38,520
    Using Marshal.Copy is essentially "unprotected" coding. This means you are responsible for 100% of having the right source, destination, count, etc. Otherwise... BOOM.

    When it blows up, usually you copied too much, or you copied data into something that isn't able to be changed.

    Since Marshal.Copy is a highly-overloaded method, I suggest you use named arguments to make sure you are supplying source / destination correctly, etc.
     
    Red_Dragon69 likes this.
  3. Red_Dragon69

    Red_Dragon69

    Joined:
    Sep 7, 2015
    Posts:
    113
    Hi, thank you for your reply.

    I played around a lot and came to the conclusion, that this doesn't work as I intended it. Seems, that the webcamTexture.GetNativeTexturePtr(); returns a pointer indeed, but the data can't be used directly in Unity for further usage. As far as I know now, this approach doesn't bring the data from GPU to CPU within Unity, explaining why the copy process doesn't work, or the returned image is pure noise or byte array is always empty. For this, using GetPixels32() is needed.

    When creating platform specific texture objects outside of Unity and using these textures in Unity Scenes Texture2D.CreateExternalTexture may be the way to go.

    Best regards :)
     
  4. Voxel-Busters

    Voxel-Busters

    Joined:
    Feb 25, 2015
    Posts:
    1,952
    In my quick test too I noticed that the data is pure white image. May be it's just giving garbage when going through GetNativeTexturePtr approach. But if it works, thats something which can be very optimised way to use the data!
     
  5. Bunny83

    Bunny83

    Joined:
    Oct 18, 2010
    Posts:
    3,919
    Have you actually started at the beginning? Reading the actual documentation of GetNativeTexturePtr?
    So it's either a pointer or handle to a certain resource. Which depend on the actual platform and graphics API used.

    The documentation continues with a specific list of things which may be returned:
    Some things you can derive from this right away:
    • Since it is possible for this method to return null if that feature is not supported, you REALLY want to do a null check before you proceed in any way.
    • On most platforms it actually returns a pointer to an actual object, not the raw image data. So you would need to know which API is used in order to do anything meaningful with that value returned by GetNativeTexturePtr.
    • When on an open GL API, the returned value is actually the "name" of the native OpenGL texture. Just in case you didn't know, a "name" in the OGL world is just an integer value. This is not a pointer, memory address or reference but just a number that is generated by Open GL. It might be some sort of index into internal arrays, though this is abstracted away. You can only use this value with OpenGL calls to refer to a certain texture object.
    So the retrival of the actual raw image data can't really be done through this method. Theoretically you can, but only with a lot of extra work. This method is meant to be used in native code rendering plugins where you have direct access to the corresponding graphics API.

    So there's probably no way around GetPixels32. Though you want to use didUpdateThisFrame to limit the amount of times you update your data since the webcam might have a relatively low framerate and copying the same unaltered image every frame would be wasteful.

    The most important thing is that you should cache your Color32 data array and not creating a new one every time you want to update the image.
     
    Red_Dragon69 likes this.
  6. Voxel-Busters

    Voxel-Busters

    Joined:
    Feb 25, 2015
    Posts:
    1,952
    Thanks a lot for jumping into the discussion :).

    Actually I don't mean to use GetNativeTexturePtr directly. I'm trying to use it in a native plugin but not directly passing the texturePtr to native. This is what I did. The below code is specifically written for Android

    Code (CSharp):
    1. if(m_texture2d == null)
    2. {
    3.           //On Android it uses graphics Format 88 (undocumented) which is ARGB_32.
    4.           var texFormat = GraphicsFormatUtility.GetTextureFormat(m_webCamTexture.graphicsFormat);
    5.           m_texture2d = new Texture2D(m_webCamTexture.width, m_webCamTexture.height, texFormat, false);
    6.  
    7. }
    8.                  
    9.  m_texture2d.UpdateExternalTexture(m_webCamTexture.GetNativeTexturePtr());
    10.  var bytes = m_texture2d.GetRawTextureData();    //There will be a copy here, but thats ok for now to test if it's working. Later will replace with NativeArray
    11.  
    12. // Passing the above bytes array to Android native via ByteBuffer wrap
    I'm already using didUpdateThisFrame to trigger the above code to avoid passing redundant data to native.

    Yea, this is going to be hacky, but just curious if we can use the WebCamTexture's native info so that I can avoid rolling my own solution (creating external texture on native and passing to unity) to achieve the same result.

    Of-course using SetPixels looks to be the only way as I see the data isn't as expected on native. On native I see the data seems to be complete grey color :|


    Also, one thing which is strange (I mostly could be doing it wrong) is,
    If I draw the m_texture2d in unity it's drawing fine (after using UpdateExternalTexture), but once sent to native, the data is "CD" for all pixels, which is driving me crazy.
     
    Last edited: Feb 14, 2024
    Bunny83 and Red_Dragon69 like this.
  7. Voxel-Busters

    Voxel-Busters

    Joined:
    Feb 25, 2015
    Posts:
    1,952
    I think it makes sense to consider un-managed data pointed by GetRawTextureData "may not" get updated on calling UpdateExternalTexture. This is because, raw texture data is on cpu side and to facilitate easy texture data modifications so that we can "apply" once at the end of any processing. So UpdateExternalTexture doesn't actually need to update this cpu buffer.

    Anyone confirm this please?
     
  8. Bunny83

    Bunny83

    Joined:
    Oct 18, 2010
    Posts:
    3,919
    Well, I don't think this is documented anywhere. Though from the documentation we can already guess that no readback would happen, why should it ^^. Many textures are usually marked as not readable so the CPU side buffer can actually be freed. In the end, for rendering, you only need the texture on the GPU. According to the documentation, UpdateExternalTexture is essentially just a re-assignment of a different native texture so that the managed object actually refers to a different native texture. It would be strange if that method would actually cause a readback.

    Of course there are cases where a readback is required. You may have more success with Rendering.AsyncGPUReadback.

    Funny enough, I guess the actual webcam data goes through the CPU anyways and Unity will upload it to the GPU. It's a pity they don't provide a more direct access to the camera feed.
     
  9. Voxel-Busters

    Voxel-Busters

    Joined:
    Feb 25, 2015
    Posts:
    1,952
    Yes, right. It's not required to do a load on the cpu once the texture is updated and it's not documented too, so fine. Earlier I expected because I thought "GetRawTextureData" as the name says, may fetch on request, but it's not. But it looks like it just returns the pointer if its loaded earlier, else garbage.

    I will go through the native plugin approach by creating my own camera feed on native as it gives more control and I can save a lot of copy and conversions on the managed side!

    If any one looking for a solution to update a texture2d and want to use it in unity (and don't want to access GetRawTextureData), you can simply do this.

    Code (CSharp):
    1.  
    2. if (m_texture2d == null)
    3. {
    4.     m_texture2d = new Texture2D(webCamTexture.width, webCamTexture.height, TextureFormat.RGBA32, false);
    5. }
    6.  
    7. m_texture2d.UpdateExternalTexture(webCamTexture);
    8.  
    The above code does the required work instead of doing GetPixels and Apply on the texture. This is working as of now.

    I had earlier success with AsyncGPUReadback. So, yea thats the better way to avoid stalling if any. But, couple of devices (older) I tested didn't support.

    @Bunny83 As I got a chance now, would like to say you "thank you". I have read lots of content posted by you on this forum and it is of immense help. Really appreciate for sharing your knowledge :)
     
    Last edited: Feb 16, 2024