Search Unity

  1. Megacity Metro Demo now available. Download now.
    Dismiss Notice
  2. Unity support for visionOS is now available. Learn more in our blog post.
    Dismiss Notice

AVPro Live Camera

Discussion in 'Assets and Asset Store' started by AndrewRH, May 24, 2012.

  1. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    AVPro Live Camera.png
    Latest Version 2.9.1 - Released 17 February 2021

    The AVPro Live Camera plugin for Unity allows you to capture video camera devices (such as webcams) directly into Unity. Unity already has WebCamTexture but it has poor performance on Windows and doesn't support many high-end cameras and capture cards.

    Available at the Unity Asset Store or download the free trial version from the webpage.

    Features:
    • *NEW* Unity 2020 support
    • *NEW* XSplit-VCam and transparent virtual webcam support
    • High performance
    • Easy to use
    • Camera settings exposed
    • Blackmagic DeckLink capture card support (see our AVPro DeckLink plugin for full-featured support)
    • TV-card support
    • Supports Unity 5.6 and above
    • Surpasses Unity WebCamTexture by supporting high-end devices and non-RGB camera modes
    • Works in the editor and also in stand-alone builds
    • 5 demos included
    • NGUI component included
    • No scripting required (for most usage)
    • Windows PC only
    We see this plugin being useful in the following areas:
    • Interactive Installations
    • Serious Games
    • Scientific Research
    • Training and Simulation
    • Computer vision
    • Kiosks
    • Video Apps

    Why not take a look at the website and try the demo?



    We developed this plugin for some interactive installations that required high-end capture card support and have since used it on many projects with webcams.

    We hope this plugin is useful to other people. Please let us know your comments/suggestions. :D

    -RenderHeads
     
    Last edited: Feb 17, 2021
  2. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    The demo has just been updated to allow multiple cameras to stream at once. We also fixed 2 quality bugs with DeckLink capture cards.

    There's already been quite a bit of interest so far, but no action on this forum yet :) Anyone have any opinion to offer? Much call for DeckLink / advanced camera support?

    :)
     
  3. droderick

    droderick

    Joined:
    Aug 25, 2008
    Posts:
    169
    Hi Andrew,
    Looks pretty cool. Couple of questions for you:
    Does the type of camera matter much in terms of video quality?
    Have you tested this much with Ultra HD video?
    Also, is it possible to stack the capture cards on one system?
    Have you toyed around with running this on multiple PC's?
    Genlock capability with more than 1 feed?
    We don't have a specific need for this now, but could see using it in a large environment in pretty cool ways. Nice work! thanks,
    d
     
  4. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Hi droderick, thanks for your interest in the plugin :) To answer your questions:

    Yes, the type of camera usually matters when it comes to quality :) Lower end digital cameras often perform unwanted filters and chroma-subsampling which are best avoided. I also think it's best to get a truely progressive camera as interlacing can be a real pain. Our plugin does support deinterlacing though :)

    We haven't tested it on UHD as we don't have access to such equipment but if you'd like to supply it we can have a look :) The code currently is limited to a frame buffer of 4K however we have only tested it to up hardware that does 1080p.

    I'm not sure what you mean by 'stack' capture cards. Certainly it supports multiple capture cards.

    Currently it doesn't support gen-locking but this is something we are thinking of adding in the future.

    Thanks!
    -Andrew
     
  5. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Hi,

    The asset has just been released on the Asset Store! :)

    We've also updated the downloadable demo which now includes the 3 demo scenes that ship with the asset:

    1) Default Camera Demo


    This is the most basic demo as it requires no scripting at all. The demo simply draws the default
    camera to the screen and is built from drag 'n drop components.

    2) Camera Explorer Demo



    This demo doesn’t use any of the drag ‘n drop components (except the required
    AVProLiveCameraManager component) and demonstrates how easily the a demo can be written
    using a few lines of scripting. The demo is created by a single 120 line script file, most of which is
    GUI related.

    3) Material Mapping Demo



    This demo shows how to integrate AVProLiveCamera into your 3D scene. No scripting is required and
    you can just use either of the 2 included components to map live camera images to your meshes or materials.

    For more information, documentation and to download the demo just visit the web site.

    Keen to hear your thoughts,

    -Andrew
     
  6. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Hi all, I wanted to let everyone know about an exciting new feature we have planned for an upcoming release of AVPro Live Camera: Overlays.

    Overlays allow us to display videos in Unity really FAST and using very little CPU. It works by leveraging the graphics API (DirectX and OpenGL) directly and completely bypassing Unitys texture system. Currently our plugin plays camera strems to Unity textures and it works really well, however there is some extra overhead when using DirectX as we can't update the textures directly like we can in OpenGL.

    The current method of camera streams to Unity textures will remain however we will offer the new Overlay mode as an extra option for those that need faster video playback using DirectX.

    The overlays are 2D rectangles that are drawn on top of the final scene. They can be positioned, alpha blended and have their depth set.

    If you have any comments or questions about this upcoming feature let us know!
     
  7. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Hi! Version 1.2 has just been released and new features include:

    + Added support for devices with crossbars including TV-cards and capture cards with multiple inputs (eg AVerMedia).
    + Faster device enumeration (startup).
    + Added colour to AVProLiveCameraGUIDisplay component
    + Inspector UI improvements.
    + Fixed a texture size bug in OpenGL mode.

    Why not try the updated demo (10MB)?

    The support for TV-card devices was a user requested feature. Is there a feature you would like to see? Let us know.
     
  8. xandeck

    xandeck

    Joined:
    Apr 2, 2009
    Posts:
    563
    We just bought it at Interativa and it works really good, we are really satisfied by the results.
    One question, AndrewRH:
    - Do you know any good way to make the video in FULL HD to run with better FPS?
    Currently it is running under ~15 FPS, using a Core i7 with 8GB Win7 64Bits, with a good 1 GB graphic card.
    With HD resolution it is really good.

    Best,
     
  9. NathanRH

    NathanRH

    Joined:
    Oct 24, 2012
    Posts:
    447
    Hi xandeck,

    AndrewRH is off spreading the good word (real-time graphics that is) so I'll try and help instead.

    First of all could you please provide as much detail about the camera you are capturing from as possible please? Including the video capture format.

    Also, did you try running in OpenGL mode using the -force-opengl switch?

    Cheers,

    Nathan
     
  10. xandeck

    xandeck

    Joined:
    Apr 2, 2009
    Posts:
    563
    Hello Nathan,

    We are capturing from 2 cameras (tests purposes):
    Microsoft LifeCam (HD)
    Logitech HD Pro Webcam C920 (FULL HD)

    I did not try the force-opengl yet (I think)... but I can and let you know..

    Thanks
     
  11. NathanRH

    NathanRH

    Joined:
    Oct 24, 2012
    Posts:
    447
    Hi xandeck,

    You should also verify the incoming framerate from the webcam(s). Not all webcams can actually capture at 30 fps in 1080p - as you increase the resolution, the framerate in general decreases. Many webcams only do 15 fps in 1080p so it might be worth checking that first. Also check the exposure settings - in anything else other than a bright environment, many webcams automatically increase exposure time which can make the video look jerky instead of smooth.

    You can use a utility like VirtualDub to play around with camera settings.

    Cheers,

    Nathan
     
  12. xandeck

    xandeck

    Joined:
    Apr 2, 2009
    Posts:
    563
    Thanks Nathan, we will try your suggestions.

    Cheers
     
  13. cooli2012

    cooli2012

    Joined:
    Oct 25, 2012
    Posts:
    3
    Hi, Nathan
    Whether to support Blackmagic decklink Intensity Shuttle USB3.0?
    It is an USB3.0 external device。
     
  14. NathanRH

    NathanRH

    Joined:
    Oct 24, 2012
    Posts:
    447
  15. Mads Pedersen

    Mads Pedersen

    Joined:
    Sep 27, 2012
    Posts:
    5
    Hi
    You write that DirectX soon will be supported in AV PRO.

    Is there DirectX in the 1.2 demo? If not, do you have a release date?

    I'm working on a project where we have many camera feeds into unity.
    Maybe DirectX could help with the speed.

    By the way, your program works with a Deltacast card I work with.

    Thanks
    Mads
     
  16. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Yes, we also use it with the Intensity Shuttle USB 3.0 and it works well :)
     
  17. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    @Mads Thanks for the info about the Deltacast :)

    All versions work with DirectX and OpenGL, however it works fastest in OpenGL mode. We plan to add a fast 'overlay' mode that'll also run fast in DirectX but only for 2D overlay rendering. We may wait to see whether Unity 4.0 supports DirectX texture access, in which case we'll skip the 'overlay' mode implimentation.
     
  18. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Also a lot of those 'full hd' cameras only support 1080p using strange pixel formats that aren't commonly supported. If you're still having trouble drop us an email (contact@renderheads.com) and we'll try to figure out which pixel format the webcam is using and add support to it for you.
     
  19. xandeck

    xandeck

    Joined:
    Apr 2, 2009
    Posts:
    563
    Hello

    We are trying now to use it with the Blackmagic Decklink. Looks like it reads from driver (because it is showing in Unity GUI test).
    Also drivers are well installed. Do I have to do anything more to make it running? Right now is not running, but because we cannot make the input video to work (we will try with HDMI soon).
     
  20. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    It should work just fine as we've used it with several BlackMagic devices. One thing we found was that it can be tricky working out exactly which mode to set it to. There's no automatic mode selection, instead you have to make sure the mode you set your DeckLink to is the exact mode of the camera you plug in to it. Have a fiddle and let us know how you get on.
     
  21. xandeck

    xandeck

    Joined:
    Apr 2, 2009
    Posts:
    563
    Hello Andrew,

    Yesterday we got no luck trying to setup using the serial component, we tried many R-Y, etc, components and no success.
    But today, we used a HDMI component cable plugged directly into a FHD camera, it worked. And as you said, it was really tricky to get the right setup for the camera I am using, if I put the wrong setup (wrong 1080, or 720 modes) it does not work.

    The sad thing is: what shows in the camera screen also shows in the Unity screen, lol
    I have to find a camera that does not show any kind of HUD in it's screen.

    Thanks for the help.
     
  22. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Yeah, it's annoying that some cameras show their HUD :( I have a little Sony hdr-cx150 and it works nicely and shows no HUD.

    Glad you got it working in the end though.
     
  23. xandeck

    xandeck

    Joined:
    Apr 2, 2009
    Posts:
    563
    Hello again, Andrew.

    Ok, now just passing by to say that everything is working good. We put a very good Full HD camera (I dont have the model here, but it is a really top camera) and everything is running good. After setup the correct resolution/everything, the video runs ok.

    We are making an augmented reality experience for a "Shopping Center" and we are using your solution integrated with our own, mixing Kinect and another minor things. Everything is working good. Our last test will be make it running for more than 12 hours without stop, to see how the PC handles it and if nothing crashes.

    Thanks for the help and I can tell here that this solution is really good, it attends totally what we needed and it works great with Unity.

    Cheers

    EDIT: one last question :) is there any options to auto-mirror the video/picture?
     
    Last edited: Nov 1, 2012
  24. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    I'm pleased it's all working for you now :) There isn't an option to mirror the picture but you can try one of these:
    1) Often in the webcam / capture card settings there is an option to flip the image
    2) Open AVProLiveCameraFormatConverter.cs and look at the Build() method. You'll see there are 2 booleans there for flipping, you can hardcode those or expose them to Unity higher up.

    Hope that helps
     
  25. xandeck

    xandeck

    Joined:
    Apr 2, 2009
    Posts:
    563
    Hello Andrew,

    Thanks for the reply. Actually, it was most a doubt, we solved this easy just typing to negative the X in transform position of our plane. The camera we are using for it does not have a mirror option. But I will take a look on your code ;)

    Best
     
  26. xandeck

    xandeck

    Joined:
    Apr 2, 2009
    Posts:
    563
    Hello again, Andrew...

    Please, do you know if the "AverMedia HD Capture C985 Bus" works with your plugin?
    We finished a prototype project, using that camera and chip I already told you, but with this new one, it is not working (Unity does not even load it).

    THanks in advance
     
  27. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Thanks xandeck - we're chatting to you via email to resolve this issue and figure out why this card isn't supported. Hopefully we'll be able to find out what's going on and fix it :)
     
  28. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Hi All, I just want to announce the newest version of the AVPro Live Camera plugin which features Unity 4.0 support!

    We still support Unity 3.5+ but now with Unity 4.0 we can finally bring you much faster DirectX texture updates because of the new native texture feature. Previously we offered this only when running in OpenGL mode.

    Hope you enjoy using our plugin and we'd like to hear from you!
     
  29. theobject

    theobject

    Joined:
    Oct 6, 2010
    Posts:
    7
    Is there a possibility of capturing live video to disk using your plugin?
     
  30. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Yes that's completely possible. Our plugin only handles the capturing to AVI file and doesn't do any sort of live video streaming though. For live video streaming you may want to look at one of our other plugins.

    Here is an example of a project we made using our plugins to play 2 HD videos (cloud layers), compositing and capture to AVI for YouTube upload:

    http://www.renderheads.com/portfolio/2012/DellIFA/

    Because we were capturing in HD we had to capture to an intermediate format (Lagarith) and transcode to a full compression format (H264 MP4) using a background process before uploading to YouTube.
     
  31. Virtualware

    Virtualware

    Joined:
    Sep 17, 2012
    Posts:
    75
    I'm using a Logitech c920 at full HD resolutions, and in my work pc (intel i5) it's ok, almost 30 fps and no delay, but in the target pc (atom 330) it's not ok.
    Right now, the image is moving fluently during a few seconds followed by a short freeze, and moves again. The freeze rate is proportional to the resolution, for example: in 720p the image is moving during a few seconds; in 1080p the movement lasts less than a second; in 640 or similar there is no problem.
    I understand that the cpu is rather limited, but I think that the problem is that some buffer is being saturated in the camera thread. So what I'm asking for is the possibility to enable frame dropping in that thread, simply reading all frames available in each iteration of the loop instead of reading just one, so the image is still moving.
     
  32. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Which version of Unity are you using, and are you running in DirectX or OpenGL mode? In Unity 3.x in DirectX there is a lot of CPU work that's needed to update the textures. For Unity 3.x I recommend running in OpenGL mode. For Unity 4.0 there is no problem with DirectX.

    Our threading shouldn't cause any pause due to camera buffering etc. Please let me know your Unity and graphics API configuration and I can do some testing here on a netbook.
     
  33. Virtualware

    Virtualware

    Joined:
    Sep 17, 2012
    Posts:
    75
    I'm using Unity 4 with the default rendering in DirectX. I've tried the opengl mode but the results are similar o even worse.
     
  34. Virtualware

    Virtualware

    Joined:
    Sep 17, 2012
    Posts:
    75
    I've made some more tests and it seems it's related to non dedicated graphics memory. I've tested in my laptop with the dedicated graphics card (nvidia 325) and uses 70% of cpu. Same laptop but using integrated graphics and the cpu goes to 100% and the freezes appear.
    Integrated graphics use shared memory which is much slower than dedicated graphics memory, so it looks like is somehow related to the texture update speed. Maybe there are too many updates and the graphics card is not fast enough processing them and that's where the buffering problem occurs?
     
  35. megisto

    megisto

    Joined:
    Dec 25, 2009
    Posts:
    127
    Hello Andrew,

    i am interested in your plugin. Just a question. Is it possibile to send the video back to the output of the video card after been composited in Unity?

    Thank you
     
  36. StephanK

    StephanK

    Joined:
    Jul 10, 2012
    Posts:
    47
    I have one fairly stupid question. How can I save a frame as a png to disk with your plugin? I tried grabbing the render texture from the material that is updated by your component, but somehow i always end up with an empty texture.
     
  37. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Hi StephanK,

    The OutputTexture member is actually a RenderTexture. You can save it to a PNG with something like this:

    RenderTexture stream = (RenderTexture)liveCamera.OutputTexture;
    if (stream)
    {
    Texture2D temp = new Texture2D(stream.width, stream.height, TextureFormat.ARGB, false);
    RenderTexture.active = stream;
    temp.ReadPixels(new Rect(0,0,stream.width, stream.height), 0, 0, false);
    byte[] pngBytes = temp.EncodeToPNG();
    System.IO.File.WriteAllBytes("filename.png", pngBytes);
    RenderTexture.active = null;
    Texture2D.Destroy(temp);
    }

    Hope that helps :)
     
  38. StephanK

    StephanK

    Joined:
    Jul 10, 2012
    Posts:
    47
    Thanks for the reply. I actually was already using the solution you suggested, I just didn't realize that alpha was set to 0. Is there an easy way to get the plugin to set alpha to 1?

    I also have another problem, which seems a bit more serious. We have two high res cameras, an ids ueye 3.0 and a logitech cw900 pro. I was able to use the ids cam with your plugin, but I had problems with the cam config. It seems to do a vertical flip by default. If I deactivate it using the showconfigwindow it works correctly, but if I restart the software it goes back to flipped. I guess there is a way to save these settings somewhere, but I couldn't figure out where and how. Another problem is that the camera crashes after about an hour seemingly for no reason. The programm keeps running but there are no updates for the cam and if I try to restart the device seems to be still locked, so I have to restart windows to get access to it back.

    For those reasons we tried the logitech device as a backup. This could work, but here the problem is that the plugin seems to show only a certain area of the picture, as if the zoom settings were set to a high level. I am using e2esofts VCam software as a reference. The frustum of the camera seems to be a lot wider than what is displayed by your plugin. Unfortunately I need the whole frustum as our application has to cover a broad area. I looked at the config window and there is a zoom setting, but the lowest value allowed seems to be 100 and the result is an image that looks like the lower right (or left? don't remember right now) corner of the full image displayed in vcam. Do you have any suggestions how to solve this problem, or is it a possible bug?

    Speaking of bugs... If I have an active LiveCamera and use your capture plugin the resulting capture is unusable. It is flickering all the time, because every few frames there is a "wrong" frame blended in, that was rendered by a different camera (that is not even running at that time). If I deactivate the gameobject where your LiveCameraManager and the LiveCamera itself are attached everything works fine.
     
  39. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Hi StephanK, long post :)

    First of all - yes, there is a conflict between some of our plugins currently. We only just discovered it and we have a fix now which we'll be putting on the Asset Store any day now.

    Setting the alpha value to 1 is something that I think the pixel format conversion shader should already be doing? You could go into the included YUV shaders and just make sure you set oCol.a = 1 at the end of the shader. I *think* you can also try to set the TextureFormat to RGB24 for the PNG texture - that might work I haven't tested it..

    We don't have any experience with the IDS cameras but that behaviour does some most unusual.. The vertial flip thing - it's possible for us to expose an option in AVProLiveCamera component to enable forced flipping. We already have logic for flipping built into the code (check our AVProLiveCameraFormatConverter.cs Build() function), so we'd just need to expose that to the component higher up. As for the crashing - I have no idea :( Perhaps try running in RGBA mode.

    The zoom issue with the Logitech sounds like an aspect ratio issue. I have a Logitech Pro 9000 and when I select certain display modes with different aspect ratios it seems to crop the resulting image (which is perfectly reasonable). so you just need to make sure the right mode is being selected. I suggest try going through the modes in the Camera Explorer demo.
     
  40. StephanK

    StephanK

    Joined:
    Jul 10, 2012
    Posts:
    47
    Hi Andrew,

    The Logitech problem seems to be solved after we installed the complete Logitech software. The IDS problem seems to be a hardware problem, as we get the same behaviour when routing the camera through 3rd-party software like VCam. I'll look at the conversion shader to see what's going on there.
     
  41. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Good to know! Glad you have it mostly working. As for the shader and the plugin conflicts - we're gonna have an update really soon to fix that. I'll let you know.
     
  42. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Hi All, Version 1.5 has just been released. Changes include:

    + Added Color32 grabber component. This is useful if you want to pass the camera image data to the 3rd party code, for example for augmented reality (AR).
    + Optimised pixel format conversion.
    + Fixed conflict with other AVPro plugins by using unique GL.IssuePluginEvent() (requires forthcoming updates with other plugins too).
    + Fixed some shaders display half the texture
     
  43. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Unity 4.1 was released today and the current release of our plugin doesn't work with it yet. We are working on the issue and should have a new version out in a few hours. Thanks for your patience :)
     
  44. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Please to say that the plugin has now been updated on the Asset Store to support Unity 4.1 and we also fixed a few other issues with other versions of Unity!
     
  45. Hijsalot

    Hijsalot

    Joined:
    Mar 20, 2013
    Posts:
    1
    Andrewww HERO, it works like a boss!
    you made my life a bit more easy

    (it works with my Black Magic extreme 3)
     
  46. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    I'm glad to hear it :) Please feel free to post your comment/rating on the Asset Store :)
     
  47. RicoLastico

    RicoLastico

    Joined:
    Mar 25, 2013
    Posts:
    1
    Hi,

    I'm about to buy your plugin, but just want to be sure it is compatible with the BlackMagic Intensity Pro internal card.

    Thanks
     
  48. BeyondReality

    BeyondReality

    Joined:
    Mar 25, 2013
    Posts:
    7
    Hey AndrewRH,

    We bought your plugin and are realy happy with it. It works like a charm with our BlackMagic DeckLink HD Extreme 3.

    I read in one of your early posts this plugin supports deinterlacing. Where can I find this option?
    Our camera does not have the option to turn it off, so I was hoping for some post effect or variable I can toggle on/off.

    Best regards,

    Sam
     
    Last edited: Mar 25, 2013
  49. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    Yes it is compatible with Blackmagic Intensity Pro
     
  50. AndrewRH

    AndrewRH

    Joined:
    Jan 24, 2012
    Posts:
    2,806
    We haven't added deinterlacing yet but it should be there in the next release.