Kinect with MS-SDK

Hi roumenf, Thanks for your help,
(I have installed OpenNI2/NiTE2(Linux version) in Linux,was able to run sample Linux codes of openni (motion capture working) code is in c++,
but was not able to run c# code(this package) as the code for the OpenNI2/NiTE2 packages in Linux is in C++ (.h and .cpp files), which I think is not supported in Unity so would not be able to add UI features. WIll try to research more ... )

Thanks

 

Where I can find that mapping function for converting depth data to space points?

Thanks

 

Here is a link to the GitHub project, containing the source of UnityInterface2-library, which is the interface between the C# code and the native OpenNI2/NiTE2 libraries: https://github.com/rfilkov/OpenNi2UnityInterface It is for MacOS, but I suppose it won't be a problem to make it compile on Linux, too (by the way, the code is the same on Windows, only the IDE is different). After you compile it, put the library into the root-folder of the Unity project (this is the parent-directory of the Assets-directory). This should be enough, but I've never done it for Linux, hence cannot be 100% sure.

 

KinectManager.Instance.GetJointPosForDepthMapPos(), or
KinectWrapper.MapDepthPointToSkeletonPoint()

They can be used for any depth point, not just skeleton joints. The 2nd function is used by the first one, so they are generally the same.

 

Hi,

I have some questions about the joint position:

The first, and what I need the most: I need to know how to transform the position returned by kinect to a 3D world point. I think I have a good approximation, but I am having some problems about the way of getting the depth. So, please, if some knows a good way to make de transformation, I will be really happy.

On the other hand, but in the same way: I am not really sure about the difference between methods KinectManager.GetJointLocalPosition(), KinectManager.GetJointPosition(), and KinectManager.GetRawSkeletonJointPosition()

Thank you very much for your help

 

The position returned by the sensor is transformed to world point by multiplying the Kinect coordinates to kinect2world transformation matrix.

Regarding the functions, as far as I remember: GetRawSkeletonJointPosition() returns the raw joint position in Kinect coordinates, GetJointPosition() returns the transformed world position, GetJointLocalPosition() - not sure, but it should return the position relative to the parent joint. Hope this information helps.

 

Hi again,
Thank you for your reply, It really helped me

I have an other question now, do you know if I can mock the kinect ( or make some record ) to work with it instead of moving myself in front of the camera every time I need to test the application?

Thank you and if somebody knows how to do this I will be really thankful

 

I think you can use the Kinect studio (part of the Kinect SDK) to make recordings and then re-play them.

 

thanks roumenf, i found it very useful for my final project. is it possible to make the hand cursor click object / drag and drop object. looking forward for your response.

 

I think you need the interaction manager from KinectExtras-package. This package was deprecated some time ago, because it is now part of the K2-package. But you can e-mail me, and I'll send it to you.

 

thanks for your response roumenf !. i already email you.

my e-mail is jeffry.ant@gmail.com

 

I'm trying to setup the Cubeman using local transforms (cause I find them more useful in my case). But it seems that some parts of the skeleton are missing and I'm getting very weird poses due to that. Is there a working example of using local positions and rotations?



UPDATE: I just realized that all the bones should be there. They were just renamed differently in a Kinect example I was reading. Anyway, is that hierarchy correct in my scene?

UPDATE2: Got it fixed. My updated controller was still setting bone rotation instead of localRotation.

UPDATE3: Still having difficulties with the rotations. So an example skeleton with local transforms would be helpful. A short video of the problem http://rzq.pw/unity/unity-kinect-local-coordinates.webm (left Cubeman is in world coordinates, right one should be similar). I'm applying the rotation from GetJointLocalOrientation, position is not touched. I noticed the the flip parameter isn't used at all, that might be a problem.

UPDATE4: The fix was quite simple, I changed the GetJointLocalOrientation to be similar with GetJointOrientation:

//return Quaternion.LookRotation(localMat.GetColumn(2), localMat.GetColumn(1));
return ConvertMatrixToQuat(localMat, joint, flip);

 

Hi Roumenf,
I have one issue related to HandCursor.
I have added five game object in a row, used a raycasting to find the object then, fire an action, but the problem with hand cursor position, in playmode the cursor's position appears on the object but in the editor, it shows the different position of the cursor. so I am unable to get the proper raycasting. how do I resolve the issue

 

Hi, I am trying to make a local multiplayer boxing game using kinect. the problem is, I don't know how to introduce the second player. can anyone help me out?
Kinect MS- SDK is working fine just help out with this local multiplayer thing please

 

In Kinect Manager check "Two Users" and add avatars for player 1 and 2 to those lists. Otherwise it uses Player1 for every avatar (at least what I think it does after quickly reading the code).

 

well that only didn't work but I have something in my mind with this help..hope that works out thanks

 

nothing worked :/

 

finally it worked

 

What did you do to get it working then? I don't need 2 players myself but in case someone with the same problem reads this thread.

 

The easiest way would be to use the 'Maximize on play'-option in editor and full-screen mode of the player.

 

Hi roumenf.

Thank you for the free Kinect SDK v1!

I have an issue on the camera view.

I attached KinectManager on Main Camera and set a 3d animation with gesture script. Then, i set both compute user map and color map to true. The problem is that the user blocks the animation if they overlay. The models/ animation will always be behind the user. I want to make the animation go round the user when a gesture is used, or at least be in front of user. Setting layers and culling masks does not work. I have ideas to just display the user and color map on a plane, then just make the model go round the plane.

Any other method is highly appreciated.

The gestures work fine, btw. Gj.

 

Hi, and sorry for the delay. I don't really understand how the user blocks models/animation and why the layers and culling masks don't work. All this should work the same way as in KinectOverlayDemo-scene. Anyway, you can always get the color camera texture from KinectManager, like this: 'Texture texColor = KinectManager.Instance.GetUsersClrTex();'. Then set it as main-texture of the plane's shared material.

 

Hi roumenf,
I'm Developing a basketball game that allows the user to pick an object and throw
it at a target,
I have this working using a mouse, however I want to use kinect to perform these actions with MS-SDK V1 in unity.

Is there a way to get the force of arm throw? Is this the best way to do this?
I wonder how to create a gesture and upon gesture trigger I release the ball by arm joints random velocity or
force and random angle while jump is detected in first person ?
how to type in java script like that ,i 'm bad at programming but i can learn from your hint.

By the way, a little problem about Avatar jumping. when i is detected by kinect i do jump gesture in front of kinect,
the cubeman is jump as i jump but the avatar can't jump with me.
I've checked the checkbox called "vertical movement" but it still can't detect my movement either do the squat!
How to fix that bug?

Could you share a bit more detail about how it works to my e-mail or tell me where i should addon and well-manipulated the Kinect Manager?
My e-mail:robert458716@gmail.com

Greetings!

 

Hi roumenf, because before i want to create gesture to throw basketball,i used collide trigger to do that.
I was thinking was having a box collider on the user hand and the
object , when these objects collide I would attach the object to the
users hand, for throwing maybe I would a object above users head and when the
users hand collides with it I would throw the object.
Unfortunately, Ball still stick on hand when collider on hand collide the head's one
I've used OnTriggerEnter(script stick to hand) & OnTriggerExit(script stick to ball) to do this
How to fix that ? Is that possible to alternative the create gesture?

 

Hi,

Regarding the jumping and squatting issue: The avatar movement in Y-direction is controlled by the 'Vertical movement'-setting. If enabling it does not fix vertical movement, you could debug the issue. The avatar movement is done in the MoveAvatar()-function of AvatarController.cs-script. The full source code is there.

Regarding the hand throwing a ball: You can detect the hand positions during the throw, hence calculate the distance between them and hand velocity. Then apply it to the ball's rigid body and leave the rest to the physics. If you get stuck, you may e-mail me. I provided recently a similar case (throwing a ball to barrel) for the physics example in the K2-asset, so I could send you the script for reference.

 

Hi roumenf,really appreciate your help,
I have e-mailed you please check thank you !

 

Hi Roumenf,
could it possible to create the gesture of character turning around 360 degree while I turning my body ?

 

Hi,
I want to build a solution which detects only those users who hold an object in their hands and ignore the other users.
is it possible to build the solution? i am using your SDK with extras and Kinect v2

 

This will be difficult, because the sensor gets confused when the user turns around, but you could try.

 

For Kinect-v2 you need the other asset - 'Kinect-v2 Examples with MS-SDK', and some image analysis around the hand-joint positions in space. I think we discussed this on Skype recently, so I'll not go into details here.

 

Yes, Thanks

 

Like @evolvingtech, @megisto, @Whitt1985, @griffin2000 and others I too am encountering failure of the Kinect sensor after around a minute of any application, when using a Kinect v1 device on Windows (10 64bit) and the MS-SDK v1.8.

Just to confirm that removing all drivers (PrimeSense/NITE/OpenNI/MS Kinect), fresh installing the v1.8 MS-SDK, and then rewriting the project using Unity 4.6.3 did not resolve the issue for me - I still got the NuiInitialize Failed - Device is not genuine error.

If I'm honest, I don't really understand the exact purpose/difference between the MS-SDK and NiTE/OpenNI etc. - I'm assuming that if I can't get this version to work, it's not worth trying to use the Open NI2 version instead? (if this was the case, I'm guessing someone else would have suggested it already!) As far as I understand, they still require the Kinect SDK to be installed as well, so if it's that that's bugged, they're not going to avoid the problem, right?

Thankyou @roumenf for all the effort you have spent on this - it is much appreciated. I'm sure you're as frustrated as everyone else that we can't get to appreciate the work that's gone into this asset! If I were to purchase a Kinect for Windows (or, a Kinect for XBox One + a Windows adapter, as now seems to be sold) and used your paid asset instead, would the problem still exist?

 

Thank you for the feedback! Some time ago I used my paid Unity support time, and asked Unity guys to look what exactly went wrong with Unity 5 and Kinect-v1 asset. They said the Kinect driver crashed for some reason after these 1-2 minutes, and told me to turn to Microsoft. Unfortunately as you see, the MS support did not bother to respond so far.

The Kinect-v2 (aka Kinect for Xbox One) is a different sensor, with different SDK, hence utilized by different asset. Provided you have USB-3 port on your machine, it works quite good, without major hardware or SDK issues with Unity 5. To be honest, there is one (again MS-related) issue: they still have not provided Windows-10 libraries for the UWP Unity builds. The UWP builds should target the old Windows 8.1 SDK instead. Based on my experience with their support, this could take decades to solve

 

I am just getting back into an art project for the Kinect, which I developed initially on the mac years ago with the first generation Kinect.

I just bought your asset so that I could develop for both v1 and v2 and this time want to go with a plain vanilla MS-SDK development route, assuming I will have the least amount of hassles. It looks great and once I can test it look forward to working with it.

I am on Windows 10 and as this machine also develops for Oculus, I need to keep it constantly updated.

So far, the Kinect is not showing up. Am I correct in that developing on Windows 10 requires me to buy the "Xbox Kinect Adapter for Xbox One S and Windows 10 PC" for $39 to use the original Kinect? I thought Apple was the king of proliferating dongles!

I have no need for the new features and prefer the lower resolution of the old Kinect one for performance reasons. However, if there is a crashing issue in Unity builds, that is important to me as this installation runs non-stop 8 hours a day. Should I buy the dongle or upgrade to a newer Kinect?

 

Neither Kinect-v1 (aka for Xbox-360) nor Kinect-v2 (aka for Xbox-One) can be plugged in directly to the PC, without an external adapter. The adapters are different for the both sensors, but they both provide external power and USB-cable to connect the sensor to the PC.

What do you mean by 'Kinect is not showing up'? Do you mean the device manager, some of the data streams, developer samples or demo scenes in the K2-asset? For Kinect-v1 you need to install Kinect SDK 1.8, and (optionally) Developer toolkit 1.8. After that, please open the 'Developer toolkit browser', and check if Color-basics, Depth-basics & Skeleton-basics samples work. Check also the sensor connection and restart the PC if needed. For Kinect-v2 you need Kinect SDK 2.0. There are again color, depth and body samples + Kinect configuration verifier that may be checked first.

Do mind the both SDKs are different, but they can be installed, co-exist and be used on the same machine. If you have both SDKs installed on the same machine, but would like to use Kinect-v1 with the K2-asset, you need to make a little change in one of the interface scripts of the K2-asset. See this tip for more information: https://rfilkov.com/2015/01/25/kinect-v2-tips-tricks-examples/#t21 Regarding the freezing issue (the app is not crashing, but freezing), you need to experiment with the Kinect-v1 you already have and find out by yourself, if it works or not. On some machines it works without issues, on many others - freezes after 1-2 minutes. Microsoft did not care to provide any workaround to this issue so far. Unfortunately they're totally concentrated on HoloLens now.

I would personally recommend using Kinect-v2 over v1, because it is more precise in body tracking, the color and depth images are clearer and better, it can work in darkness, and doesn't have any major issues. Again, you would need to buy both the sensor and the adapter. Both could be bought on Amazon, for instance. One thing to have in mind in advance is that you must have at least one USB-3 port on your machine (it needs to be dedicated to the sensor), and the machine should have good performance.

 

Thanks! It's working perfectly now. I had installed the v2 SDK and was not seeing anything. MS is not really updating their documents, when I read the 1.8 SDK it appeared that it required Windows 8, so I thought I would use the SDK that mentioned Windows 10.

It's a really nice asset! I like how well the background subtraction works and all the other samples seem to be working great as well.

 

It works

But how do I expose joint values to the editor?

I know I need to use GetJointPosition but how and where?

 

No need I found out. Your Get position sample is amazing

 

hi roumen can i ask u something ? i want to add grip/grab hand gesture in your kinect sdk for unity v1 package. but i cant do it (i tried to add open/close fist detector but it doesnt work) can u give me an example code for grip/grab hand gesture ?

 

To get the hand grip/release functionality for Kinect-v1, you need the KinectExtras-package, too: https://rfilkov.com/2013/12/17/kinectextras-with-mssdk/ Please email me your request and I'll send you the package.

 

hi roumen i've just sent you an email, and thanks for your reply

 

Hello Sir, i just wana ask how to process it in GestureInProgress? I tried your code but i'm not clear about it..

 

GestureInProgress is a method of gesture listener in the scene (see SimpleGestureListener.cs in the avatar-demo for example). It has parameters like userId, gesture and progress. The 'progress' has a value between 0.0 and 1.0 (i.e. between 0% and 100%), so you can use it to determine if a continuous gesture is still in progress or not. Print its values in the console, run the scene, do the gesture, and you will see what I mean.

 

Hello,
I would like to use the principe of the script KinectOverlay.cs to control a mouse cursor with my right hand in my software.
It works well but I would like to modify the sensibility for the x and y axes of the position of the cursor.
I don't know what to modify, could somebody help me ?

 

1. Duplicate the KinectOverlay.cs-script and rename it for instance to CursorHandOverlay.cs (both file and class name).
2. Open the script, and in Update(), after the calculation of scaleX and scaleY-variables, invoke MouseControl.MouseMove() with 'new Vector3(scaleX, scaleY, 0)'.
3. Remove or comment out the rest of the Update()-method.

I'm only not sure if the scaleY should be calculated like it is now: (1.0f - posColor.y / ColorImageHeight), or (posColor.y - ColorImageHeight). This could be found out experimentally. Hope this info helps...

 

haii roumenf, robert
i have final project in my collage to create basketball throw too. i've been working step by step in here. i found script to throw ball you are ready shared in here and im stuck liked your stuck.
Could you share a bit more detail about how it works tome ?
My e-mail:ghufronhasanarrusdi@gmail.com
looking forward for your response.
Thanks

 

If you are a student, you are eligible to get the K2-asset (containing the mentioned scene and script), free of charge for academic use. Please send me your request from your university e-mail address, to prove you are really a student there.

 

i have university e-mail address ghufron12ti@mahasiswa.pcr.ac.id im student from indonesia. nice to meet you roumenf

 

Nice to meet you too. I meant that you send me an e-mail (with the same or similar message as above) from your university e-mail address. Otherwise I cannot be sure it is really yours.

 

may you send your e-mail to me ? i want send same message for you. thank you roumenf