Search Unity

  1. Welcome to the Unity Forums! Please take the time to read our Code of Conduct to familiarize yourself with the forum rules and how to post constructively.
  2. Unity 2022.2 is now available as the latest Tech release.
    Dismiss Notice
  3. We are making some changes to the DOTS forums.
    Dismiss Notice
  4. Have a look at our Games Focus blog post series which will show what Unity is doing for all game developers – now, next year, and in the future.
    Dismiss Notice

KinectExtras with MsSDK

Discussion in 'Assets and Asset Store' started by roumenf, Jan 14, 2014.

  1. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
  2. darthviper107

    darthviper107

    Joined:
    May 22, 2013
    Posts:
    15
    Does the facetracking give position of the head in relation to the Kinect? I'm trying to do a head tracking project and need to be able to know where the head position relates in Unity compared to the real world.

    Thanks
     
  3. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
  4. darthviper107

    darthviper107

    Joined:
    May 22, 2013
    Posts:
    15
    Ah, I found it. Does the position account for the angle of the sensor?
     
  5. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    If you get the position from KinectManager - yes. It accounts for the tilt angle and height of the sensor above the ground. If you get it from FacetrackingManager - it doesn't, as far as I see. Hmm, I need to check this a bit more detaileded and fix it, if needed. Thank you for pointing out this issue!

    Do you need help on how to transform the head's position, depending on sensor angle and height?
     
  6. darthviper107

    darthviper107

    Joined:
    May 22, 2013
    Posts:
    15
    I think I have it figured out using KinectManager, though it's coming out to a weird value. With the sensor at about 1 meter off the ground the head position ends up lower than the sensor, so when I stand in front of it it shows my head as coming in at 0.7 meters when it should probably be at 1.7 meters.
    I'm trying to get the info this way:
    HeadPosition = KinectManager.Instance.GetJointPosition(player, 3);

    I assume the head is int 3?
     
  7. lmbarns

    lmbarns

    Joined:
    Jul 14, 2011
    Posts:
    1,592
    Love your packages man, they're awesome. Been using them for a year or so now.

    I've used the regular ms kinect package in a couple exhibition/events/trade shows...even got paid more than I charged. Thanks so much!!
     
  8. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Yes, head is int 3, but I'd suggest that you use 'GetJointPosition(userId, (int)KinectWrapper.NuiSkeletonPositionIndex.Head)' instead.
    Otherwise you're right. The height is 1m less as needed. The reason for this is this line in the Start() method of KinectManager: float heightAboveHips = SensorHeight - 1.0f;

    I try to remember what was the reason for subtracting this 1m there, but can't. So, either comment out this subtraction or add 1 meter to the Y-component of the returned head (or other joint) position.
     
  9. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Thank you too, lmbarns! :)
     
  10. Ribosome

    Ribosome

    Joined:
    Jan 15, 2013
    Posts:
    43
    Hi, roumenf, I got problem when trying to use KinectWrapper and the package at the same time:
    I am using this to detect my hands states(grip or release) this is the codes
    Code (csharp):
    1. using UnityEngine;
    2. using System.Collections;
    3.  
    4. public class GripRealeaseDetector : MonoBehaviour {
    5.     public  InteractionManager manager;
    6.     public UILabel tipLbl;
    7.  
    8.     void Awake()
    9.     {
    10.         manager = Camera.main.GetComponent<InteractionManager>();    
    11.     }
    12.  
    13.     void Update()
    14.     {
    15.         if (manager == null) return ;
    16.        
    17.         // if there is a hand grip, try to select the underlying object and start dragging it.
    18.         if (manager.IsLeftHandPrimary())
    19.         {
    20.             // if the left hand is primary, check for left hand grip
    21.             if (manager.GetLeftHandEvent() == InteractionWrapper.InteractionHandEventType.Grip)
    22.             {
    23.                 tipLbl.text = "左拳握拳状态";
    24.             }else if(manager.GetLeftHandEvent()==InteractionWrapper.InteractionHandEventType.Release)
    25.             {
    26.                 tipLbl.text = "左拳松开状态";
    27.             }
    28.         }
    29.         else if (manager.IsRightHandPrimary())
    30.         {
    31.             // if the right hand is primary, check for right hand grip
    32.             if (manager.GetRightHandEvent() == InteractionWrapper.InteractionHandEventType.Grip)
    33.             {
    34.                 tipLbl.text = "右拳握拳状态";
    35.             }
    36.             else if (manager.GetRightHandEvent() == InteractionWrapper.InteractionHandEventType.Release)
    37.             {
    38.                 tipLbl.text = "右拳松开状态";
    39.             }
    40.             else if (manager.GetRightHandEvent() == InteractionWrapper.InteractionHandEventType.None)
    41.             {
    42.                 //tipLbl.text = "右拳none状态";
    43.             }
    44.         }
    45.         else
    46.         {
    47.             tipLbl.text = "No hand primary";
    48.             print("right hand state"+InteractionWrapper.GetRightHandState());
    49.         }      
    50.        
    51.     }  
    52. }
    this script works fine in your example scene, but when I using KinectWrapper to get the those data I need, I always got this output :
    else
    {
    tipLbl.text = "No hand primary";
    print("right hand state"+InteractionWrapper.GetRightHandState());
    }

    so I trace the code to find out where went wrong, and here is the result I got In InteractionManager.cs
    Code (csharp):
    1. void Update ()
    2.  {
    3.   if(!interactionInited)
    4.    return;
    5.  
    6.        
    7.   // update Kinect interaction
    8.   if(InteractionWrapper.KinectInteractionUpdate() == 0)
    9.   {
    10.            
    11.    int lastSkeletonTrackingID = skeletonTrackingID;
    12.    skeletonTrackingID = (int)InteractionWrapper.GetSkeletonTrackingID();
    13.    
    14.    if(skeletonTrackingID != 0)
    15.    {
    16.                 print("UUUUUUUU");      
    the skeletonTrackingID is always 0,
    what 's the problem?InteractionWrapper and kinect wrapper doesn't get along ?
    would you please tell me how to solve this problem? Thanks in advance
     
  11. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Hi Ribosome,

    To integrate Kinect Interaction with Kinect Manager, you need to:
    1. Open KinectScripts/InteractionWrapper.cs and at its beginning uncomment: #define USE_KINECT_MANAGER
    2. Open KinectScripts/KinectWrapper.cs and at its beginning uncomment: #define USE_KINECT_INTERACTION_OR_FACETRACKING
    3. Make sure that both KinectManager and InteractionManager are components of MainCamera or game object in the scene.

    Have you done this?

    Greetings!
     
  12. Ribosome

    Ribosome

    Joined:
    Jan 15, 2013
    Posts:
    43
    thanks for answering , but can't find any thing like #define * in either InteractionWrapper or KinectWrapper
     
  13. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
  14. Traiger

    Traiger

    Joined:
    Dec 22, 2013
    Posts:
    6
    Is there any reason why when I rotate my avatar 180 on the y aixs the arm rotation on the x axis gets rotated?

    To be explicit if I point up the avatar points down and vice versa. Pointing left and right work as expected.
     
  15. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Do not rotate the avatar. Parent it to an object (like UCharCtrlBack, UCharCtrlFront in the example) and rotate this control-object instead. Avatar under the ctrl-object should have 0,0,0 as rotation.
     
  16. ikarusfr

    ikarusfr

    Joined:
    Feb 23, 2014
    Posts:
    3
    Is it possible to integrate KinectBackgroundRemoval to this package?

    I could observe that there is a functionality in the package that simulates the background removal, using the color map and user map, however, the KinectBackgroundRemoval softens the edges and do not lose pixels.

    Please, it would be very important to me your package can use the KinectBackgroundRemoval.

    Thanks
     
  17. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Hello ikarusfr. First of all, as I see this, background removal has nothing to do with KinectExtras, but rather with 'Kinect with MS-SDK'-package. Secondly, as you have already noticed, you can get background removal automatically, if you check both 'Compute User Map'- and 'Compute Color Map'-parameters of KinectManager - a component of MainCamera in the example. KinectBackgroundRemoval-interface is not used at the moment in the wrapper, but if you need it explicitly, you can try to add and use its interface and function. The full source code of the wrapper is out there.
     
  18. ikarusfr

    ikarusfr

    Joined:
    Feb 23, 2014
    Posts:
    3
    Hi, the background removal feature of your package is using the depth data from the Kinect, but this is too noisy to use for a smooth player mask.

    I don't know how to add the KinectBackgroundRemoval-interface to the wrapper.

    It would be a really good feature in your package "Kinect with MS-SDK" add the KinectBackgroundRemoval-interface.

    Regards.
     
  19. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Ok, I'll put this feature into my todo-list.
    Greetings!
     
  20. karljj1

    karljj1

    Joined:
    Feb 17, 2011
    Posts:
    440
    Hi,

    We recently bought the KinectExtras asset which is great so far.

    We are using the InteractionWrapper to track each hand and treat them like cursors which can click, grip and drag etc. however the customer would like to have 2 users interacting simultaneously. I have noticed the InteractionWrapper only seems to track a single user. Is it possible for it to track 2, so that we could have 4 hands on screen performing grip/release etc?

    I know this can be done if we use the KinectManager however it does not seem to have the same features as the InteractionManager.
    Is it possible to adjust the interaction manager to allow for 2 users, perhaps have 2 InteractionManagers one per skeleton?

    Thanks

    Karl
     
  21. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Hi Karl,

    At the moment only one skeleton can be tracked by the InteractionManager. I suppose this behaviour could be modified, but it will require some coding time and a lot of tests. It will require modification of the native wrapper too. If you want to have this done quickly and you have some C# and CPP knowledge, I can send you the sources of the dll and some instructions, so you could do the modifications it by yourself. Otherwise, I'll try do it, but at the moment can't say exactly how long it will take. Please send me an e-mail, if you want to get the sources.

    Greetings,
    Rumen
     
  22. karljj1

    karljj1

    Joined:
    Feb 17, 2011
    Posts:
    440
    Thanks Rumen.

    I will take you up on your offer for the source code, its something I need to have working in the next few days :sad:

    I have sent you an email.

    Regards

    Karl
     
  23. Livecesar

    Livecesar

    Joined:
    Apr 2, 2014
    Posts:
    1
    Hello,

    I'm using the speech recognition demo, but I would like to change the language recognition (english by default - 1033 \ SpeechManager), for exmaple to spanish (1034), but I'm stuck, am I doing rigth changing that value?, what else do I have to change?.

    Thanks

    Cesar
     
  24. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
  25. Seycho

    Seycho

    Joined:
    Jun 3, 2014
    Posts:
    1
    Hello ,

    I want to use two kinects at the same time but i don't know how to do it or even if it's possible . Can you tell me wether it's possible or not and maybe give me clues in order to do it?

    thanks you by advance !

    Seycho
     
    Last edited: Jun 3, 2014
  26. RicardoFig

    RicardoFig

    Joined:
    Jul 1, 2014
    Posts:
    2
    Hello

    I just bought this package, I want to know how to display the camera video as fullscreen, I observed that using facetrackmanager you put in partially in the screen, how can I change the position of the video to be full screen? the idea is to put avatars in a real world (camera video) and interact with the human detected by kinect
     
  27. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
  28. BillCartauld

    BillCartauld

    Joined:
    Sep 3, 2014
    Posts:
    1
    Does this SDK do live viseme recognition? I am interested in writing an app (for windows phones) that can turn a phone conversation live into an animated movie. :)
     
  29. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Hi Bill, KinectExtras is for desktops and laptops. Send me an e-mail, if you want to try it.
    Please excuse my stupidity, but what does 'live viseme recognition' mean?
     
  30. Kolovrat

    Kolovrat

    Joined:
    May 5, 2014
    Posts:
    17
    Is it possible to create solid interactive depth map with this? Something like this:
     
  31. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    As to me, you need to convert the depth image from 2D to 3D and add a collider to the user's depth mesh. But I think I already answered this question by e-mail.
     
    Kolovrat likes this.
  32. Kolovrat

    Kolovrat

    Joined:
    May 5, 2014
    Posts:
    17
    Thanks Rumen,
    yes you have answered on email. I posted here before sending you an email.
     
  33. kornel-l-varga

    kornel-l-varga

    Joined:
    Jan 18, 2013
    Posts:
    43
    heey everyone, Marry Xmas!

    I would like to ask if this pack is for V1 or V2 of Kinect?
    plus do you guys know how could I get the position where the kinect tracks from? What I mean is the relative space of the Kinect Sensor in relation with my built joints. For coordination issues it would be great if I could dynamically change in VR space where the Kinect track from.

    greetings,
    K
     
  34. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Merry Christmas and Happy 2015!
    This package is for Kinect v1 only. Most of the KinectExtras managers and demos are already part of the v2-package "Kinect v2 with MS-SDK".
    To measure the distance you can use the head's position from FacetrackingManager or integrate the KinectExtras with "Kinect with MS-SDK" (instructions are here: http://rfilkov.com/2013/12/17/kinectextras-with-mssdk/) and use the public functions of KinectManager.
     
  35. zhangfan

    zhangfan

    Joined:
    Apr 10, 2013
    Posts:
    12
    Could i init the KinectInteraction and BackgroundRemoval at the same time.
    now i have problem that when i execute the function InitBackgroundRemoval and InitKinectInteraction,the backgroundRemoval display with just a white plane,and do not work.
    Thanks!
     
  36. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    I tried it and got background removal working, but interaction - not any more. Anyway, there is an issue and I've put it in in my todo-list.
     
  37. Attreyu

    Attreyu

    Joined:
    Mar 5, 2015
    Posts:
    5
    Maybe not the best topic to post this question, but is it possible to substract the background using Kinect V2 and replace it with a virtual scene, made in unity, while keeping the user not changed, as in exactly how he appears in reality ?

    It's for a very simple app where "the player/avatar" is yourself, for example dancing on a virtual theatre scene, where near you there are little animated animals and toys, made in Unity, so the whole background (theatre scene in this case) is virtual, including the lighting, except yourself.
     
  38. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Hi, there is a separate thread for the K2-asset. The background removal is possible. There is a special background removal demo included in the asset. You can also combine it with the 3d objects in your scene. For examples, see the overlay-demo, depth-collider demo or this scene, created by Ricardo Salazar:


    On the other hand, if you want to put the player's image on a 3d humanoid model, you can get the cut-out texture of the user, but as I'm not a designer, can't tell you how to apply it to the model without distortions. Also, keep in mind that background removal is a resource-hungry process. It will eat a lot the scene performance, especially in hi-res (color camera resolution). Hope this info helps.
     
  39. J2014

    J2014

    Joined:
    Oct 20, 2015
    Posts:
    6
    Hi!For face tracking multiple users,how do i to make it?
     
  40. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Face tracking of multiple users is not supported for Kinect v1.
     
  41. J2014

    J2014

    Joined:
    Oct 20, 2015
    Posts:
    6
    Thank u for requesting.
    For tracking multiple users, determine the faces you want to track, e.g. by instancing IFTFaceTracker and calling DetectFaces. Create additional instances of IFTFaceTracker for each extra face you want to track.
    i see it from msdn.i need use kinect v2 to achieve face tracking of multiple users?
     
  42. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Kinect v1 can track maximum two users. What I meant is that the native wrapper in KinectExtras tracks one face only. Kinect v2 can track up to six faces (and users).
     
  43. Anagashi

    Anagashi

    Joined:
    Dec 3, 2012
    Posts:
    9
    Hi, I just purchased this version of the Kinect Plugins. and my current kinect SDK and runtime is 1.8, and followed your instruction on importing this version of kinect plugin.

    After I imported this plugin, my Kinect unable to detect me.

    At the beginning it does show DLL error, then it is gone.

    (I am using Window 10 and Unity 5)

    Please reply ASAP, thanks.

    Edit: I kinda solved the problem, and hope that it won't appear again.
     
    Last edited: Oct 26, 2015
    roumenf likes this.
  44. J2014

    J2014

    Joined:
    Oct 20, 2015
    Posts:
    6
    thank you.
     
  45. Anagashi

    Anagashi

    Joined:
    Dec 3, 2012
    Posts:
    9
    It is possible to use facetracking manager to find the position of the eyes?

    I look at the demo and can't figured out how it works.

    Please teach me, thank you.
     
  46. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    Hmm, maybe, but not as directly as in K2-facetracking, as far as I remember. I think you can use the model vertices (the FacetrackingManager.Instance.GetFaceModelVertex()-function), to locate the indices of the eyes in the face model.
     
    Anagashi likes this.
  47. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    I don't quite understand what happened and how exactly you solved it, but congratulations anyway. If the problem appears again, zip the project and send it to me via WeTransfer, so I could take a look. Also, please mind I don't work at weekends and holidays.
     
    Anagashi likes this.
  48. jimdavis2001

    jimdavis2001

    Joined:
    Dec 11, 2015
    Posts:
    1
    I have successfully used "Kinect with MS-SDK" from the Asset Store, to use a Kinect v1 with Unity 5.
    However, I want to try face tracking, and if I'm understanding correctly, this requires the "KinectExtras with MS-SDK" package, also from RF Solutions.
    I was able to get "Kinect with MS-SDK" from the Asset Store but "KinectExtras with MS-SDK" appears to be no longer available.
    Is there still some way to get "KinectExtras with MS-SDK"? Or was it deprecated because there might be some better or alternative way of doing this now?
    I realize most people have probably moved on to the Kinect v2 by now, and in fact I have one as well... but I have to use Windows 7 for what I'm doing, which means I need to use the Kinect v1.
    Thank you!
     
  49. roumenf

    roumenf

    Joined:
    Dec 12, 2012
    Posts:
    635
    I deprecated the KinectExtras, because it is fully integrated in the Kinect-v2 asset. The K2-package works with Kinect v1, too. But if you specifically need the KinectExtras-package, for instance to integrate it in your current project, you may contact me by e-mail, and I'll send it to you. No problem.
     
  50. Smiaro

    Smiaro

    Joined:
    Jul 22, 2013
    Posts:
    19