KinectExtras with MsSDK

Aimed to provide support to "KinectExtras with MS-SDK"-package users - https://www.assetstore.unity3d.com/#/content/10492 Please post your questions below.

 

Does the facetracking give position of the head in relation to the Kinect? I'm trying to do a head tracking project and need to be able to know where the head position relates in Unity compared to the real world.

Thanks

 

Hi Darthviper,

The short answer is: Yes. Although, you can get the head position of the tracked player also from KinectManager ("Kinect with MS-SDK" package - https://www.assetstore.unity3d.com/#/content/7747).

Greetings!

 

Ah, I found it. Does the position account for the angle of the sensor?

 

If you get the position from KinectManager - yes. It accounts for the tilt angle and height of the sensor above the ground. If you get it from FacetrackingManager - it doesn't, as far as I see. Hmm, I need to check this a bit more detaileded and fix it, if needed. Thank you for pointing out this issue!

Do you need help on how to transform the head's position, depending on sensor angle and height?

 

I think I have it figured out using KinectManager, though it's coming out to a weird value. With the sensor at about 1 meter off the ground the head position ends up lower than the sensor, so when I stand in front of it it shows my head as coming in at 0.7 meters when it should probably be at 1.7 meters.
I'm trying to get the info this way:
HeadPosition = KinectManager.Instance.GetJointPosition(player, 3);

I assume the head is int 3?

 

Love your packages man, they're awesome. Been using them for a year or so now.

I've used the regular ms kinect package in a couple exhibition/events/trade shows...even got paid more than I charged. Thanks so much!!

 

Yes, head is int 3, but I'd suggest that you use 'GetJointPosition(userId, (int)KinectWrapper.NuiSkeletonPositionIndex.Head)' instead.
Otherwise you're right. The height is 1m less as needed. The reason for this is this line in the Start() method of KinectManager: float heightAboveHips = SensorHeight - 1.0f;

I try to remember what was the reason for subtracting this 1m there, but can't. So, either comment out this subtraction or add 1 meter to the Y-component of the returned head (or other joint) position.

 

Thank you too, lmbarns!

 

Hi, roumenf, I got problem when trying to use KinectWrapper and the package at the same time:
I am using this to detect my hands states(grip or release) this is the codes

Code (csharp):

1. using UnityEngine;
2. using System.Collections;
3.  
4. public class GripRealeaseDetector : MonoBehaviour {
5.     public  InteractionManager manager;
6.     public UILabel tipLbl;
7.  
8.     void Awake()
9.     {
10.         manager = Camera.main.GetComponent<InteractionManager>();    
11.     }
12.  
13.     void Update()
14.     {
15.         if (manager == null) return ;
16.        
17.         // if there is a hand grip, try to select the underlying object and start dragging it.
18.         if (manager.IsLeftHandPrimary())
19.         {
20.             // if the left hand is primary, check for left hand grip
21.             if (manager.GetLeftHandEvent() == InteractionWrapper.InteractionHandEventType.Grip)
22.             {
23.                 tipLbl.text = "左拳握拳状态";
24.             }else if(manager.GetLeftHandEvent()==InteractionWrapper.InteractionHandEventType.Release)
25.             {
26.                 tipLbl.text = "左拳松开状态";
27.             }
28.         }
29.         else if (manager.IsRightHandPrimary())
30.         {
31.             // if the right hand is primary, check for right hand grip
32.             if (manager.GetRightHandEvent() == InteractionWrapper.InteractionHandEventType.Grip)
33.             {
34.                 tipLbl.text = "右拳握拳状态";
35.             }
36.             else if (manager.GetRightHandEvent() == InteractionWrapper.InteractionHandEventType.Release)
37.             {
38.                 tipLbl.text = "右拳松开状态";
39.             }
40.             else if (manager.GetRightHandEvent() == InteractionWrapper.InteractionHandEventType.None)
41.             {
42.                 //tipLbl.text = "右拳none状态";
43.             }
44.         }
45.         else
46.         {
47.             tipLbl.text = "No hand primary";
48.             print("right hand state"+InteractionWrapper.GetRightHandState());
49.         }      
50.        
51.     }  
52. }

this script works fine in your example scene, but when I using KinectWrapper to get the those data I need, I always got this output :
else
{
tipLbl.text = "No hand primary";
print("right hand state"+InteractionWrapper.GetRightHandState());
}
so I trace the code to find out where went wrong, and here is the result I got In InteractionManager.cs

Code (csharp):

1. void Update ()
2.  {
3.   if(!interactionInited)
4.    return;
5.  
6.        
7.   // update Kinect interaction
8.   if(InteractionWrapper.KinectInteractionUpdate() == 0)
9.   {
10.            
11.    int lastSkeletonTrackingID = skeletonTrackingID;
12.    skeletonTrackingID = (int)InteractionWrapper.GetSkeletonTrackingID();
13.    
14.    if(skeletonTrackingID != 0)
15.    {
16.                 print("UUUUUUUU");      

the skeletonTrackingID is always 0,
what 's the problem?InteractionWrapper and kinect wrapper doesn't get along ?
would you please tell me how to solve this problem? Thanks in advance

 

Hi Ribosome,

To integrate Kinect Interaction with Kinect Manager, you need to:
1. Open KinectScripts/InteractionWrapper.cs and at its beginning uncomment: #define USE_KINECT_MANAGER
2. Open KinectScripts/KinectWrapper.cs and at its beginning uncomment: #define USE_KINECT_INTERACTION_OR_FACETRACKING
3. Make sure that both KinectManager and InteractionManager are components of MainCamera or game object in the scene.

Have you done this?

Greetings!

 

thanks for answering , but can't find any thing like #define * in either InteractionWrapper or KinectWrapper

 

Is there any reason why when I rotate my avatar 180 on the y aixs the arm rotation on the x axis gets rotated?

To be explicit if I point up the avatar points down and vice versa. Pointing left and right work as expected.

 

Do not rotate the avatar. Parent it to an object (like UCharCtrlBack, UCharCtrlFront in the example) and rotate this control-object instead. Avatar under the ctrl-object should have 0,0,0 as rotation.

 

Is it possible to integrate KinectBackgroundRemoval to this package?

I could observe that there is a functionality in the package that simulates the background removal, using the color map and user map, however, the KinectBackgroundRemoval softens the edges and do not lose pixels.

Please, it would be very important to me your package can use the KinectBackgroundRemoval.

Thanks

 

Hello ikarusfr. First of all, as I see this, background removal has nothing to do with KinectExtras, but rather with 'Kinect with MS-SDK'-package. Secondly, as you have already noticed, you can get background removal automatically, if you check both 'Compute User Map'- and 'Compute Color Map'-parameters of KinectManager - a component of MainCamera in the example. KinectBackgroundRemoval-interface is not used at the moment in the wrapper, but if you need it explicitly, you can try to add and use its interface and function. The full source code of the wrapper is out there.

 

Hi, the background removal feature of your package is using the depth data from the Kinect, but this is too noisy to use for a smooth player mask.

I don't know how to add the KinectBackgroundRemoval-interface to the wrapper.

It would be a really good feature in your package "Kinect with MS-SDK" add the KinectBackgroundRemoval-interface.

Regards.

 

Ok, I'll put this feature into my todo-list.
Greetings!

 

Hi,

We recently bought the KinectExtras asset which is great so far.

We are using the InteractionWrapper to track each hand and treat them like cursors which can click, grip and drag etc. however the customer would like to have 2 users interacting simultaneously. I have noticed the InteractionWrapper only seems to track a single user. Is it possible for it to track 2, so that we could have 4 hands on screen performing grip/release etc?

I know this can be done if we use the KinectManager however it does not seem to have the same features as the InteractionManager.
Is it possible to adjust the interaction manager to allow for 2 users, perhaps have 2 InteractionManagers one per skeleton?

Thanks

Karl

 

Hi Karl,

At the moment only one skeleton can be tracked by the InteractionManager. I suppose this behaviour could be modified, but it will require some coding time and a lot of tests. It will require modification of the native wrapper too. If you want to have this done quickly and you have some C# and CPP knowledge, I can send you the sources of the dll and some instructions, so you could do the modifications it by yourself. Otherwise, I'll try do it, but at the moment can't say exactly how long it will take. Please send me an e-mail, if you want to get the sources.

Greetings,
Rumen

 

Thanks Rumen.

I will take you up on your offer for the source code, its something I need to have working in the next few days :sad:

I have sent you an email.

Regards

Karl

 

Hello,

I'm using the speech recognition demo, but I would like to change the language recognition (english by default - 1033 \ SpeechManager), for exmaple to spanish (1034), but I'm stuck, am I doing rigth changing that value?, what else do I have to change?.

Thanks

Cesar

 

Hi Cesar,

Apart of setting the language code (1034 is for Spanish-Spain), you need to install the respective 'Kinect for Windows Language Pack' - Spanish (es-ES) in your case. Have you done this? Here is the download link to the language packs: http://www.microsoft.com/en-us/download/details.aspx?id=34809

And, just in case, here is a list of the language codes: http://technet.microsoft.com/en-us/library/dd346950.aspx

Greetings!

 

Hello ,

I want to use two kinects at the same time but i don't know how to do it or even if it's possible . Can you tell me wether it's possible or not and maybe give me clues in order to do it?

thanks you by advance !

Seycho

 

Hello

I just bought this package, I want to know how to display the camera video as fullscreen, I observed that using facetrackmanager you put in partially in the screen, how can I change the position of the video to be full screen? the idea is to put avatars in a real world (camera video) and interact with the human detected by kinect

 

Here is a good way to display images on the background: http://answers.unity3d.com/questions/9729/how-can-i-display-a-flat-background-2d-image-not-a.html
As to the body parts' alignments, there is an Overlay-example in 'Kinect with MS-SDK'-package.

 

Does this SDK do live viseme recognition? I am interested in writing an app (for windows phones) that can turn a phone conversation live into an animated movie.

 

Hi Bill, KinectExtras is for desktops and laptops. Send me an e-mail, if you want to try it.
Please excuse my stupidity, but what does 'live viseme recognition' mean?

 

Is it possible to create solid interactive depth map with this? Something like this:

 

As to me, you need to convert the depth image from 2D to 3D and add a collider to the user's depth mesh. But I think I already answered this question by e-mail.

 

Thanks Rumen,
yes you have answered on email. I posted here before sending you an email.

 

heey everyone, Marry Xmas!

I would like to ask if this pack is for V1 or V2 of Kinect?
plus do you guys know how could I get the position where the kinect tracks from? What I mean is the relative space of the Kinect Sensor in relation with my built joints. For coordination issues it would be great if I could dynamically change in VR space where the Kinect track from.

greetings,
K

 

Merry Christmas and Happy 2015!
This package is for Kinect v1 only. Most of the KinectExtras managers and demos are already part of the v2-package "Kinect v2 with MS-SDK".
To measure the distance you can use the head's position from FacetrackingManager or integrate the KinectExtras with "Kinect with MS-SDK" (instructions are here: http://rfilkov.com/2013/12/17/kinectextras-with-mssdk/) and use the public functions of KinectManager.

 

Could i init the KinectInteraction and BackgroundRemoval at the same time.
now i have problem that when i execute the function InitBackgroundRemoval and InitKinectInteraction,the backgroundRemoval display with just a white plane,and do not work.
Thanks!

 

I tried it and got background removal working, but interaction - not any more. Anyway, there is an issue and I've put it in in my todo-list.

 

Maybe not the best topic to post this question, but is it possible to substract the background using Kinect V2 and replace it with a virtual scene, made in unity, while keeping the user not changed, as in exactly how he appears in reality ?

It's for a very simple app where "the player/avatar" is yourself, for example dancing on a virtual theatre scene, where near you there are little animated animals and toys, made in Unity, so the whole background (theatre scene in this case) is virtual, including the lighting, except yourself.

 

Hi, there is a separate thread for the K2-asset. The background removal is possible. There is a special background removal demo included in the asset. You can also combine it with the 3d objects in your scene. For examples, see the overlay-demo, depth-collider demo or this scene, created by Ricardo Salazar:

On the other hand, if you want to put the player's image on a 3d humanoid model, you can get the cut-out texture of the user, but as I'm not a designer, can't tell you how to apply it to the model without distortions. Also, keep in mind that background removal is a resource-hungry process. It will eat a lot the scene performance, especially in hi-res (color camera resolution). Hope this info helps.

 

Hi!For face tracking multiple users,how do i to make it?

 

Face tracking of multiple users is not supported for Kinect v1.

 

Thank u for requesting.
For tracking multiple users, determine the faces you want to track, e.g. by instancing IFTFaceTracker and calling DetectFaces. Create additional instances of IFTFaceTracker for each extra face you want to track.
i see it from msdn.i need use kinect v2 to achieve face tracking of multiple users?

 

Kinect v1 can track maximum two users. What I meant is that the native wrapper in KinectExtras tracks one face only. Kinect v2 can track up to six faces (and users).

 

Hi, I just purchased this version of the Kinect Plugins. and my current kinect SDK and runtime is 1.8, and followed your instruction on importing this version of kinect plugin.

After I imported this plugin, my Kinect unable to detect me.

At the beginning it does show DLL error, then it is gone.

(I am using Window 10 and Unity 5)

Please reply ASAP, thanks.

Edit: I kinda solved the problem, and hope that it won't appear again.

 

thank you.

 

It is possible to use facetracking manager to find the position of the eyes?

I look at the demo and can't figured out how it works.

Please teach me, thank you.

 

Hmm, maybe, but not as directly as in K2-facetracking, as far as I remember. I think you can use the model vertices (the FacetrackingManager.Instance.GetFaceModelVertex()-function), to locate the indices of the eyes in the face model.

 

I don't quite understand what happened and how exactly you solved it, but congratulations anyway. If the problem appears again, zip the project and send it to me via WeTransfer, so I could take a look. Also, please mind I don't work at weekends and holidays.

 

I have successfully used "Kinect with MS-SDK" from the Asset Store, to use a Kinect v1 with Unity 5.
However, I want to try face tracking, and if I'm understanding correctly, this requires the "KinectExtras with MS-SDK" package, also from RF Solutions.
I was able to get "Kinect with MS-SDK" from the Asset Store but "KinectExtras with MS-SDK" appears to be no longer available.
Is there still some way to get "KinectExtras with MS-SDK"? Or was it deprecated because there might be some better or alternative way of doing this now?
I realize most people have probably moved on to the Kinect v2 by now, and in fact I have one as well... but I have to use Windows 7 for what I'm doing, which means I need to use the Kinect v1.
Thank you!

 

I deprecated the KinectExtras, because it is fully integrated in the Kinect-v2 asset. The K2-package works with Kinect v1, too. But if you specifically need the KinectExtras-package, for instance to integrate it in your current project, you may contact me by e-mail, and I'll send it to you. No problem.

 

Please someone check my threat and help me if you could.
http://forum.unity3d.com/threads/camera-doesnt-destroy-after-scene.378587/