Search Unity

  1. Good news ✨ We have more Unite Now videos available for you to watch on-demand! Come check them out and ask our experts any questions!
    Dismiss Notice
  2. Ever participated in one our Game Jams? Want pointers on your project? Our Evangelists will be available on Friday to give feedback. Come share your games with us!
    Dismiss Notice

[Demo] Flash webcam assisted Augmented Reality

Discussion in 'Made With Unity' started by alejobrainz, Mar 11, 2009.

  1. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    250
    Hi.

    I was fooling around with the FLAR library yesterday and pulled together a simple proof of concept of integrating Flash and Unity to provide augmented reality.

    Have a look at:
    http://www.zerofractal.com/fractalreality/augmentedReality.html

    Mac users, if you are having problems accessing your webcam, right click on flash, go to settings...camera. The built in camera is usually called USB Video device or something like that.

    This is an initial approach. Now I intend to pass the video feed into a video texture. Any ideas are welcome. I was thinking pushing bitmap data though an external interface into unity, but I don't know if the browser javascript is up to the task, or if the SendMessage() supports arrays at all.

    Crits and comments are welcome!
     
  2. ProtonOne

    ProtonOne

    Joined:
    Mar 8, 2008
    Posts:
    406
    Pretty cool.

    My printer hasn't been used for a while and I'm almost out of ink apparently, so the print out was pretty ugly. But it seemed to track it okay sometimes. And when it did, the scene moved and rotated very nicely.

    I was trying to think of what this could be used for, when I remembered this video:
    http://www.youtube.com/watch?v=Jd3-eiid-Uw

    Yes, I would be geeky enough to glue your logo to my forehead to see a scene in 3d :p
     
  3. wimwouters

    wimwouters

    Joined:
    Jan 7, 2009
    Posts:
    20
    Very coool! Been playing with AR in Director a little too... would be great if you could connect to camera with unity. Processing video images for AR should be faster than using Flash I think...
     
  4. Dakta

    Dakta

    Joined:
    Apr 8, 2008
    Posts:
    492
    I remember w while back someone had something similar, but just Flash. Had a little green guy that jiggled around when you moved the camera.


    But now, with Unity integration, it opens up worlds of possibilities.



    Very cool man! I hope we see something more than a camera control tool out of this, though. What about taking the 3D model from Unity, and blue screening it into the flash video? That would be really cool!

    (Oh, and lame computer framerate issues. I think it was the Unity scene you used. My Web R/C, right?)
     
  5. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    250
    Dakata. I´m using the exact same library as the green guy. I believe we can get full integration in two different directions:

    1. by integrating the flash video feed into unity as a video texture. I´m scribbling on that one. This way I could overlay the 3d content on the video feed.

    2. by developing through opencv ways to feed the video into unity plus building the AR libraries there. This one should be much more efficient, but I believe wonk work on the web, would it?

    As for it being a camera tool it should be far from that. In the demo the camera is actually still. I´m transforming actually the object. I think some kick ass things can grow from here.
     
  6. Dakta

    Dakta

    Joined:
    Apr 8, 2008
    Posts:
    492
    Totally!


    The possibilities for Flash integrated augmented reality (even if all Flash does is get the video feed) with Unity are quite open.


    I see this as on the forefront of the latest game innovation.


    We'll see what people can do with it.
     
  7. RobbieDingo

    RobbieDingo

    Joined:
    Jun 2, 2008
    Posts:
    484
    MacBookPro (Safari and Firefox), can't change the flash camera setting to Built-In as the dropdown list of Cameras is cropped within the webpage frame.

    Would love to check this out though... can you change the page layout?
     

    Attached Files:

  8. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    250
    Done. Gave it 100+ pixels. Thx for your observation.

    I also uploaded a new guide that yields better results.
     
  9. RobbieDingo

    RobbieDingo

    Joined:
    Jun 2, 2008
    Posts:
    484
    @zerofractal, yep that works fine now.

    ... and it is very, very cool!
     
  10. jeffcraighead

    jeffcraighead

    Joined:
    Nov 15, 2006
    Posts:
    740
    +1 for cool!
     
  11. nickavv

    nickavv

    Joined:
    Aug 2, 2006
    Posts:
    1,801
    Very cool. I would be perfectly willing to tape your symbol to my forehead for headtracking btw (hint hint).
     
  12. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    250
  13. motorway

    motorway

    Joined:
    Dec 21, 2008
    Posts:
    47
    What is the bug? I don't see anything on this page. The screen remains black after powering the cam.
     
  14. motorway

    motorway

    Joined:
    Dec 21, 2008
    Posts:
    47
    Finally Flash began to work, but what must appear in Unity window and what I must do?
     
  15. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    250
    if you point the paper to the camera, you should see a house...
     
  16. hk1973

    hk1973

    Joined:
    Oct 14, 2008
    Posts:
    29
    It's impressive!! What is the latest progress?
     
  17. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    250
    I'm stuck in transferring the webcam image from flash to unity on the fly, nor unity supports a wmode = transparent to simply overly unity on top of flash, so I'm now looking at building a Artoolkit based plugin for the stand alone unity projects :S
     
  18. robobobo

    robobobo

    Joined:
    Mar 4, 2009
    Posts:
    35
    Hi there.
    Could you please tell me what are you camera settings in Unity and if you are adjusting the values coming from flash in any way ? I'm trying to set up something similar but having problems matching the position.
    Really appreciate you help !
    Thanks !
     
  19. milkytreat

    milkytreat

    Joined:
    Jan 15, 2007
    Posts:
    267
  20. EducaSoft

    EducaSoft

    Joined:
    Sep 9, 2007
    Posts:
    650
    AR in Flash is indeed possible but not that fast and its an extra step.

    I have heard of a library called OpenCV which should make this kind of stuff possible too.


    The easiest way to do it is of course to use a WiiMote (be it an original or a wiimote where you desoldered the sensor and made your own electronics project with.

    Wiimote can go 200fps for 4 blob tracking.
    Unfortunately, it involves more work then printing out a tracker page.


    I was allready investigating this WiiMote sensor to do for example motion capture. If it can capture 4 blobs at 200 fps, then with clever tricks it should be possible to track 4x4=16 markers at 50fps or 32markers at 25fps.

    Cheap mocap possible ???
     
  21. bloodtiger10

    bloodtiger10

    Joined:
    Nov 9, 2008
    Posts:
    619
    OpenCV was around but I don't know anything about it anymore there was a guy who used it to control boxes with your hands or something like that. Nice job, zerofractal!
     
  22. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    250
    Thanks. The big problem is actually feeding the camera to unity. I tried doing it via javascript but could get 1 512x512 canvas in no less than 3 seconds using AS3...which is not bad to pass 1 textures but simply unacceptable for feeing video. I guess a 320x240 feed should go at about 1 frame each 2 seconds or so...

    The thing about opencv or such is not being to be able to do the webcam interface in a web player due to plugin restrictions. Theoretically you can do it for stand alone games. The same goes to wii motes. We are stating to use wiimotes to do head tracking but.... no web players.

    as for the camera settings, the camera is not actually calibrated, but should be a deal of just sending the transforms and applying them to the unity camera. I'm passing off tracker info into unity through js.

    On flash I did this:

    Code (csharp):
    1.  
    2. private function _onEnterFrame(e:Event = null):void {
    3.     this._capture.bitmapData.draw(this._video);
    4.            
    5.     if (this._detector.detectMarkerLite(this._raster, 80)  this._detector.getConfidence() > 0.5) {
    6.         this._detector.getTransformMatrix(this._resultMat);
    7.         this._baseNode.setTransformMatrix(this._resultMat);
    8.         this._baseNode.visible = true;
    9.         this._renderer.render();
    10.  
    11.         if (ExternalInterface.available) {
    12.             var baseRotation = Matrix3D.matrix2euler(_baseNode.transform);
    13.             ExternalInterface.call("GetUnity().SendMessage", "sandbox", "argPosX", -_baseNode.x); //mirror mode (make + for unmirrored)
    14.             ExternalInterface.call("GetUnity().SendMessage", "sandbox", "argPosY", _baseNode.z);
    15.             ExternalInterface.call("GetUnity().SendMessage", "sandbox", "argPosZ", -_baseNode.y);
    16.             ExternalInterface.call("GetUnity().SendMessage", "sandbox", "argRotX", baseRotation.x);
    17.             ExternalInterface.call("GetUnity().SendMessage", "sandbox", "argRotY", -baseRotation.z); //mirror mode (make + for unmirrored)
    18.             ExternalInterface.call("GetUnity().SendMessage", "sandbox", "argRotZ", -baseRotation.y); //mirror mode (make + for unmirrored)
    19.         }
    20.     }
    21. }
    22.  
    And in unity I did this:

    Code (csharp):
    1.  
    2. var xAng : float = 0;
    3. var yAng : float = 0;
    4. var zAng : float = 0;
    5.  
    6. function argPosX(value){
    7.     transform.position.x = value;
    8. }
    9.  
    10. function argPosY(value){
    11.     transform.position.y = value;
    12. }
    13.  
    14. function argPosZ(value){
    15.     transform.position.z = value;
    16. }
    17.  
    18. function argRotX(value){
    19.     xAng = value;
    20.     transform.rotation.eulerAngles = Vector3(xAng,yAng,zAng);
    21. }
    22.  
    23. function argRotY(value){
    24.     yAng = value;
    25.     transform.rotation.eulerAngles = Vector3(xAng,yAng,zAng);
    26. }
    27.  
    28. function argRotZ(value){
    29.     zAng = value;
    30.     transform.rotation.eulerAngles = Vector3(xAng,yAng,zAng);
    31. }
    32.  
     
  23. robobobo

    robobobo

    Joined:
    Mar 4, 2009
    Posts:
    35
    Thanks a lot for the info. So you are aplying transform to the camera in Unity?? I thought you would apply it to an object an keep the camera stationary.... Interesting.
    I think I got the image feeding working, but the tracking is off. Once I have it figured out , I'll post the code. If you can post the unity file that would be great.
    Thanks again!
     
  24. alejobrainz

    alejobrainz

    Joined:
    Sep 10, 2005
    Posts:
    250
    You aré right!! The transformation is applied to an object. AR is all about the object
     
  25. robobobo

    robobobo

    Joined:
    Mar 4, 2009
    Posts:
    35
    And is your camera sits at 0,0,0 pointing in Z direction ?
     
  26. renanweber

    renanweber

    Joined:
    Sep 4, 2009
    Posts:
    2
    Hi, we're developing an application similar to ZeroFractal's, but it seems more like a workaround to the fact that Unity can't access a camera.

    At first we tried flartoolkit (pv3d), but to achieve a "regular" quality it hugely compromises the performance (getting 3fps with a 4000 triangles model)... and that's why we moved to Unity.
    Well application is running with a far more detailed model (over 30k triangle)... and it's much faster.

    But we still have a problem. It works fine on PC, but on a Mac the axis coordinates seems to be messed up. Flash (FLAR) sends the same transformation matrix but model coordinates on unity are completely different and "unstable" (changing values all the time). Any ideas on how to fix this?

    ps: since this thread is inactive for a while, any news on showing the camera inside Unity?

    Thanks in advance for the help.
     
  27. robobobo

    robobobo

    Joined:
    Mar 4, 2009
    Posts:
    35
    We ran into sort same problem. But it's consistent on pc and mac. Do you transform the angles or axises in any way?
     
  28. renanweber

    renanweber

    Joined:
    Sep 4, 2009
    Posts:
    2
    we figured out how to fix the problem. we're trying to calculate the transform data "manually" on flash side and send a string with it to Unity, at first we had a problem converting the numbers (flash uses a dot for the fraction part of the float, and unity uses comma... no big deal, but you have to replace '.' for ',' ). we believe this should be causing the same inconsistency on mac side (i don't know for sure because i've never used a mac for more the 5min).

    but the important thing, how we solved. just like zerofractal's. using Papervision's Matrix3d functions, and sending it to unity.
     
  29. robobobo

    robobobo

    Joined:
    Mar 4, 2009
    Posts:
    35
    Could you please help me out. I tried to use the same exact code as zero fractal but the axises and seem off.
    Also, what camera settings did you use in Unity ? Did you change "Field of view " settings or did you stick with 60 ? Any way you can post the code ??
     
  30. Tom163

    Tom163

    Joined:
    Nov 30, 2007
    Posts:
    1,040
    Very interesting thing. For the record: I didn't print the guide, but displayed it on my iPhone. Works well as long as there isn't much angle, then it loses it due to reflections.
     
  31. Pedro Thiago

    Pedro Thiago

    Joined:
    Sep 1, 2008
    Posts:
    6
    For the record too:
    We was running out of ink in the printer and I'd start to draw the marker with a pilot. It didn't worked. But while complaining about the not working after that helpless rustic process, the app started to work using my head as a marker.

    That was freaking AWESOME even knowing that is probably a calibration issue. That has inspired us a lot here :wink:
     

    Attached Files:

  32. jedur

    jedur

    Joined:
    Oct 5, 2009
    Posts:
    1
    Hi,
    I'm trying to put an AR app inside the iPhone using Unity, do you thing this is possible?

    I was reading that the frame rate to put the image of the camera as a texture in Unity was not acceptable, have you found a fix to this issue?

    Thanks in advance for your help.
     
  33. jose brandao

    jose brandao

    Joined:
    May 13, 2011
    Posts:
    1
    Hello to all,

    I need call a externall function (javascript) on my unity, i know that in flash i can to use this.

    import flash.external.ExternalInterface, after ExternalInterface.call

    How i to do this on Unity?

    thanks
     
unityunity