Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.

Video Input as Texture, then Interpreting Change

Discussion in 'Scripting' started by joshuaseaver, Jan 18, 2006.

  1. joshuaseaver

    joshuaseaver

    Joined:
    Oct 26, 2005
    Posts:
    28
    So now that I've determined that I can (in theory) port the Myron webcam computer vision system ( http://webcamxtra.sourceforge.net/ ) into unity, and have some Director functions that help me know how to call the Unity plugin I will (hopefully) create, how do I know that Unity will be able to handle the incoming data?

    For instance, using the Director WebCam Xtra (based on Myron), I can find change in an image and draw globs based on the video like so:
    Code (csharp):
    1.  
    2. //NOTE EVERYONE, THIS IS JAVASCRIPT LINGO, NOT UNITY JAVASCRIPT
    3.  
    4. function drawGlobPixels() {
    5.   //draw pixels for each glob
    6.   if (gGlobPixels == 1) {
    7.      var gDrawColorRGB = color(255, 255, 204)
    8.      var gpix = webcamxtra.globPixels();
    9.      for (var i = 1; i <= gpix.count; i++) {
    10.           var tg = gpix[i];
    11.           for (var ii = 1; ii <= tg.count; ii++) {
    12.               _movie.stage.image.setPixel(tg[ii],gDrawColorRGB);
    13.       }
    14.     }
    15.   }
    16. }
    And then using the glob position, I can have a 3D object react to the position of the glob.

    In Unity, can I take that incoming data, and create an XYZ point in space that can interact with the other objects in my gamespace?
     
  2. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Yes you can do all that.
    I looked at the myron source code before and I didn't see anything that would prevent you from making it into a unity plugin.

    I think it would totally rock if you made this plugin.
    I'll gladly help you out if there are any problems in making myron work with unity.
     
  3. joshuaseaver

    joshuaseaver

    Joined:
    Oct 26, 2005
    Posts:
    28
    Cool!
    The first thing I have to do is figure out how to compile it. I have created an XCode Project and have followed the Unity tutorials for creating a plugin. I think I have everything set up correctly, but (not knowing the source code) when I try to build I get one error "storage class specified for field 'start'". (Screen grab attached).

    Not being a C++ person, is this an obvious or easy thing to correct?
     

    Attached Files:

  4. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    The extern "C" around the class seems a bit strange.
    Or try renaming start to some other name.

    Can't think of much more without having it in front of me.
     
  5. joshuaseaver

    joshuaseaver

    Joined:
    Oct 26, 2005
    Posts:
    28
    I added the Extern "C" around the class. I had thought that adding the Extern "C" was a requirement I read in the Unity tutorials about creating a plugin.
     
  6. freyr

    freyr

    Joined:
    Apr 7, 2005
    Posts:
    1,148
    Extern "C" tells the compiler that what you are defining is C code and not C++. C does not have classes, so you are in effect trying to compile C++ code as if it were plain C. This is confusing the compiler causing it to report weird errors.
     
  7. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    In the source there is a file called:
    cexterns.cpp and cexterns.h

    They seem to have started working on a c interface but it is not completed yet.
    They have a few functions exposed, eg.

    Code (csharp):
    1.  
    2. extern "C" int myron_globCount(){
    3.     if(m==0)
    4.             return 0;
    5.     return m->globCount();
    6. }
    7.  

    So basically you have to continue exposing functions with the same pattern.
     
  8. joshuaseaver

    joshuaseaver

    Joined:
    Oct 26, 2005
    Posts:
    28
    ...in both the cexterms.cpp and cexterns.h files?
     
  9. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    In the header file you of course only need to declare the function prototype.
     
  10. joshuaseaver

    joshuaseaver

    Joined:
    Oct 26, 2005
    Posts:
    28
    Right, so I declared the function prototpyes in the header, but now I have to expose the functions for each and every one in the cexterns.cpp...
     
  11. joshuaseaver

    joshuaseaver

    Joined:
    Oct 26, 2005
    Posts:
    28
    So I edited the cextern.h and the cexterns.cpp and I now get a build error. So I guess that's progress anyway.

    The error is when it tries to link to the Unity Plugin.
    It is saying there are undefined symbols:
    _DisposeUserData _EnterMovies etc.

    Any ideas? I will of course re-read about building the plugin in XCode in case I've just missed a step...
     
  12. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Have you linked with all the necessary frameworks.
    Sounds like Quicktime is missing.
     
  13. joshuaseaver

    joshuaseaver

    Joined:
    Oct 26, 2005
    Posts:
    28
    Yep, not added...
     
  14. joshuaseaver

    joshuaseaver

    Joined:
    Oct 26, 2005
    Posts:
    28
    Yeah!
    I added the QT framework and the build was successful. There were 5 warnings, but I can probably track them down...
    Now comes the next step. How do I apply the video input comping through the plugin to a texture in Unity?
     
  15. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Start by reading some basic input from the webcam.

    Eg. read the glob count and fetch them into the C# plugin.
    Read the edges and then move objects around based on the input.



    When you have done that you can get down to textures.
    You have to do that from within the plugin at the moment.
    Just create a dummy file texture in unity and assign it to a material.

    To get to the OpenGL bound texture, use
    Code (csharp):
    1.  
    2. int texID = renderer.material.texture.GetInstanceID();
    3.  
    Then in the plugin you can do what ever you want to that particular texture ID using OpenGL calls. But do yourself a favor and start with the modifying input based on the web cam.
     
  16. joshuaseaver

    joshuaseaver

    Joined:
    Oct 26, 2005
    Posts:
    28
    Okay, I got the plugin into the Assets folder and wrote this little C# script:

    Code (csharp):
    1. using UnityEngine;
    2. using System.Runtime.InteropServices;
    3.  
    4. class MyronScript : MonoBehaviour
    5. {
    6.    // This tells unity to look up the function FooPluginFunction inside the plugin named "PluginName"
    7.    [DllImport ("Myron Unity Plugin")]
    8.    private static extern float myron_start ();
    9.  
    10.    void Awake ()
    11.    {
    12.       int height = 320;
    13.       int width = 240;
    14.       // Calls the FooPluginFunction inside the PluginName plugin
    15.       myron_start(height,width);
    16.    }
    17. }
    but I'm gettting this error from Unity:
     
  17. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    You need to make sure that the C# binding and C binding match exactly.

    The c code looks like this:
    Code (csharp):
    1.  
    2. extern void myron_start(int w,int h);
    3.  
    So the C# side must look like this:
    Code (csharp):
    1.  
    2.  [DllImport ("Myron Unity Plugin")]
    3.  private static extern void myron_start (int w, int h);
    4.  
     
  18. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Do you know what webcams myron supports?
    Does it support an iSight?
     
  19. joshuaseaver

    joshuaseaver

    Joined:
    Oct 26, 2005
    Posts:
    28
    I am using a Panasonic industrial video camera, going through a Firewire bridge, but I also tried it with an iSight and it worked fine. If I remember correctly, think I couldn't get it to go beyond 320*240 resolution.
     
  20. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Nice.

    How far did you get btw?
    Can you post the project if you get something working.
     
  21. klindeman

    klindeman

    Joined:
    Jun 13, 2005
    Posts:
    295
    Yeah, this sounds really cool.
     
  22. Joachim_Ante

    Joachim_Ante

    Unity Technologies

    Joined:
    Mar 16, 2005
    Posts:
    5,203
    Hey AngryAnt, do you happen to have an iSight camera i can borrow?
     
  23. joshuaseaver

    joshuaseaver

    Joined:
    Oct 26, 2005
    Posts:
    28
    Here's where I am. I got it to compile, and I tried to get it to launch from within unity with this code in a file called myron.cs :


    Code (csharp):
    1. using UnityEngine;
    2. using System.Runtime.InteropServices;
    3.  
    4. class MyronScript : MonoBehaviour
    5. {
    6.    // This tells unity to look up the function FooPluginFunction inside the plugin named "PluginName"
    7.    [DllImport ("Myron Unity Plugin")]
    8.    private static extern float myron_start ();
    9.  
    10.    void Awake ()
    11.    {
    12.       int height = 320;
    13.       int width = 240;
    14.       // Calls the FooPluginFunction inside the PluginName plugin
    15.       myron_start(height,width);
    16.    }
    17. }
    18.  
    but I get the error from the Unity console which is confusing:
    I supplied two arguements in the call...

    Here's from the actual myron.cpp code that initializes myron:
    Code (csharp):
    1.  
    2. void Myron::start(int w,int h){
    3.  
    4.    
    5.     UserData mySGVideoSettings = NULL;
    6.     //-- begin user code                                                                                 
    7.     ComponentDescription theDesc;                                                
    8.     //ComponentResult theresult;                                                    
    9.     Component sgCompID ;                                                          
    10.     Rect videoRect;                                                              
    11.     int error;                                                                                                  
    12. #if defined(Macintosh)
    13. ...
    14. ...
    and from the cexterns.cpp file:
    Code (csharp):
    1. extern "C" void myron_start(int w,int h){
    2.     if(m==0){
    3.         m = new Myron();
    4.         m->start(w,h);
    5.     } else {
    6.         m->stop();
    7.         delete m;
    8.         m = new Myron();
    9.         m->start(w,h);
    10.     }
    11. }
    And the camera is on and connected....
     
  24. joshuaseaver

    joshuaseaver

    Joined:
    Oct 26, 2005
    Posts:
    28
    I just noticed that I had inverted width and height...I'll change those but it should still launch one would think since it's just two integers...
     
  25. NicholasFrancis

    NicholasFrancis

    Joined:
    Apr 8, 2005
    Posts:
    1,587
    Don't you need to declare the parameters in your DllImport declaration, like this:

    Code (csharp):
    1.  
    2. [DllImport ("Myron Unity Plugin")]
    3.    private static extern float myron_start (int w, int h);
    4.  
     
  26. joshuaseaver

    joshuaseaver

    Joined:
    Oct 26, 2005
    Posts:
    28
    Thanks! That took care of the error. Now on to seeing if I can start getting motion data from the plugin! :D
     
  27. joshuaseaver

    joshuaseaver

    Joined:
    Oct 26, 2005
    Posts:
    28
    I appologize, I've not really done much in the way of C++, so I'm still having some probs because I'm trying to figure out someone elses code.
    I'm getting a Sytem:EntryPointNotFoundException:myron_start error.

    Going backwards down the chain, I call from the plugin (... represents that code continues):

    Code (csharp):
    1. using UnityEngine;
    2. using System.Collections;
    3. using System.Runtime.InteropServices;
    4.  
    5. public class MyronScript : MonoBehaviour
    6. {
    7.    [DllImport ("Myron Unity Plugin")]  
    8.    static extern void myron_start(int w, int h);
    9.    void Awake ()
    10.    {
    11.      myron_start(320, 240);
    12.    }
    13. }
    And the plugin itself has this at the top of it's cexterns.h file...
    Code (csharp):
    1.  
    2. extern void myron_start(int w,int h);
    3. extern void myron_stop();
    4. extern void myron_update();
    5. extern void myron_settings();
    6. extern unsigned char*myron_version();
    7. extern void myron_findGlobs();
    8. ...
    and this at the top of its cexterns.cpp file...

    Code (csharp):
    1. #include "myron.h"
    2.  
    3. static Myron *m = 0;
    4.  
    5. extern "C" void myron_start(int w,int h){
    6.     if(m==0){
    7.         m = new Myron();
    8.         m->start(w,h);
    9.     } else {
    10.         m->stop();
    11.         delete m;
    12.         m = new Myron();
    13.         m->start(w,h);
    14.     }
    15. }
    16. ...
    17.  
    and this at the start of the myron.h file...

    Code (csharp):
    1. class Myron{
    2.  public:
    3.   Myron();
    4.   ~Myron();
    5.   void start(int w,int h);
    6.   void stop();
    7. ...
    8. }
    And finally, this at the start of the myron.cpp file...
    Code (csharp):
    1.  
    2. #include "myron.h"
    3. #include <math.h>
    4.  
    5.  
    6. Myron::Myron(){
    7. #if defined(Macintosh)
    8. #else
    9.    InitializeQTML(0);
    10. #endif
    11.  
    12. }
    13.  
    14. Myron::~Myron(){
    15. #if defined(Macintosh)
    16. #else
    17.   TerminateQTML();
    18. #endif
    19.  
    20. }
    21.  
    22.  
    23. //-----------------------------------------------------------------------------
    24.  
    25. void Myron::start(int w,int h){
    26. ...
    27. }
    Any pointers as to what I'm not declaring correctly?
     
  28. tsphillips

    tsphillips

    Joined:
    Jan 9, 2006
    Posts:
    359
    I don't know what the problem is, but here's some extra information. For the exception being thrown, see http://msdn.microsoft.com/library/d...stementrypointnotfoundexceptionclasstopic.asp.

    It seems to be looking for an entry method in an assembly, which would imply (I think) that it is not yet trying to invoke the external function. It's as if there should be a C# assembly containing myron_start (maybe generated automatically) that would subsequently invoke the actual external function. This is all guesswork on my part, however, as I don't know the internals of .NET.

    Tom