Multiple Skeleton Tracking

[David Menard]

hello,
I want to get the skeletons from two different Kinect devices. I
followed this thread

http://groups.google.com/group/openni-dev/browse_thread/thread/689ede96786f15d3/ec60b8eb679ca644?lnk=gst&q=multiple+skeleton#ec60b8eb679ca644
And The node tree I get is indeed
User2 <-- Depth2 <-- Device2
User1 <-- Depth1 <-- Device1
The only problem is that when I get the SkeletonCapability for both
User1 and User2, the two are the exact same. The skeleton for User1 is
returned by User2 as well.
has anyone encountered this before? I tried creating a seperate
context for both userGenerators, as suggested in another thread, but
with no success.
here is my initialization code:


//Search and create all available depth nodes
NodeInfoList depthNodeInfoList;
ret = m_Context.EnumerateExistingNodes(depthNodeInfoList);
NodeInfoList::Iterator nodeinfoiter = depthNodeInfoList.Begin();
for(; nodeinfoiter != depthNodeInfoList.End(); nodeinfoiter++)
{
NodeInfo nodeInfo = *nodeinfoiter;
if(nodeInfo.GetDescription().Type == XN_NODE_TYPE_DEPTH)
{
Sensor* s = new Sensor();
s->Initialize(&m_Context,nodeInfo);
m_Sensors.push_back(s);
}
}

int Sensor::Initialize(Context* context, NodeInfo info)
{
try
{
//m_Context = context;
m_Context = new Context();
m_Context->Init();
m_Context->SetGlobalMirror(true);

XnStatus ret = m_Context-
>CreateProductionTree(info,m_DepthGenerator);
if(!CheckStatus(ret)) throw exception();
m_DepthGenerator.GetMirrorCap().SetMirror(true);

m_DepthMetaData = new DepthMetaData();

//Create a user generator for each depth node
// This is useful because there might be more than 1 userGenerator
per node
NodeInfoList userGenInfoList;
ret = context-
>EnumerateProductionTrees(XN_NODE_TYPE_USER,NULL,userGenInfoList);

NodeInfo userInfo =
FindUserForDepthGenerator(userGenInfoList,m_DepthGenerator);
if(userInfo == NULL)
{
throw exception();
}

ret = m_Context->CreateProductionTree(userInfo,m_UserGenerator);
if(!CheckStatus(ret)) throw exception();

//Register Callbacks
m_UserGenerator.RegisterUserCallbacks(Sensor::NewUser,
Sensor::LostUser, this, m_hUserCallbacks);

m_UserGenerator.GetPoseDetectionCap().RegisterToPoseCallbacks(Sensor::PoseDetected,
Sensor::PoseLost, this, m_hPoseCallbacks);
// Skeleton stuff
SkeletonCapability skelly = m_UserGenerator.GetSkeletonCap();
skelly.SetSkeletonProfile(XN_SKEL_PROFILE_ALL);
skelly.SetSmoothing(m_SmoothingFactor);
skelly.RegisterCalibrationCallbacks(Sensor::CalibrationStart,
Sensor::CalibrationEnd, this, m_hCalibrationCallbacks);

m_Context->StartGeneratingAll();

m_Initialized = true;
return 1;
}
catch(exception e)
{
//Cannot initialize, we can only playback
m_Initialized = false;
return -1;
}
}


Then when I update, I update both the global context and the context
for each userGen:


m_PlayBackContext.WaitAnyUpdateAll();
for(int i = 0; i < m_PlayBackSensors.size(); i++)
{
m_Sensors[i]->Update();
}

int Sensor::Update()
{
m_Context->WaitOneUpdateAll(m_UserGenerator);
m_DepthGenerator.GetMetaData(*m_DepthMetaData);
return 1;
}


Any help is really appreciated!
Thanks!
Dave

[dtr]

AFAIK 2 skeleton tracking kinects on one computer are still not
supported by NITE. You have to use a second computer. Fixing this is
my number one feature request...
OpenNI does support depth and image grabbing from 2 kinects though.

Best, Dieter

On May 22, 8:44 pm, David Menard <david.mena...@gmail.com> wrote:
> hello,
> I want to get the skeletons from two different Kinect devices. I
> followed this thread
>

> http://groups.google.com/group/openni-dev/browse_thread/thread/689ede...

[Frederic Steinke]

Hey,

thats only half the truth. As far as i know it is an openni issue (not a NITE one), that is caused by some not too well chosen design pattern. Therefore you can't get two different usergenerators work within the same process/programm instance, but it is of course possible to get two usergenerators running fine on the same computer. From there on you might work with some interprocess communication to merge the data from both kinects. A "bugfix" making the use of multiple usergenerators within the same instance possible would be very very desirable though!

Regards,
  Frederic

[dtr]

Hi Frederic,

That's an interesting piece of info. Have you (or someone else) tested
this? (2 reliable skeletons from 2 programs on the same machine)
I won't be around my development machine and Kinects before a week or
2.

Best, Dieter

On May 23, 10:12 am, Frederic Steinke

[David Menard]

Hey dtr,
I'll be testing this in the upcoming days, I will keep you updated!
Dave

[Frederic Steinke]

Hey dtr,

yep I was working with those multiple skeletons in different processes on the same machine for quite a while. You just have to keep on mind that you'll need two USB-Hubs (not ports) to be able to connect two kinects to the same machine. At the moment i'm not so much focussing on that work, because I'm too lazy to handle the upcoming syncronisation issues hoping the OpenNI "singleton bug" might be fixed "soon". My setup included two kinects, one in front of the person to be tracked and the other one watching the person from the side. The frontal kinect deliveres reliable skeleton informations while the side one (obviously) produces some more of wrong tracking results.

P.S. I haven't tested this with OpenNI 1.5 so far, my observations were based on using version 1.3.

Regards,
  Frederic

[dtr]

Hi Frederic,

Thanks for clarifying. That's really useful. I have a similar 2-kinect
setup but I'm using 2 computers because I understood that 2 skeleton
trackers don't work on one machine, no matter what. Feel stupid now :D

Lets see if I can get it to work with Processing and the SimpleOpenNI
wrapper library. I'm using that because compiling OpenNI apps myself
is too complicated for my very limited programming skills.


best, Dieter


On May 29, 10:50 am, Frederic Steinke

[Hala]

Just wondering
did you manage to make this work with processing and the SimpleOpenNI
library?

[David Menard]

I am still loosely working on it. I will try both OpenNI and Kinect
SDK.
I have been in contact with a person who claims to have had two
different skeletons in a single process.
His code is up in a google project (http://code.google.com/p/
snagasnaga/) but I have yet to reproduce it.
He generates a separate context for each device and only uses the
skeleton, completely ignoring the depth image.

I will keep updating.

[dtr]

I didn't succeed in Processing yet: http://groups.google.com/group/simple-openni-discuss/browse_thread/thread/7a3f433df32db2e2

Haven't had very much time to fiddle with it though. More in the coming 2 weeks.

best D.

[David Menard]

Here is a little update on the subject of Multiple Skeleton Tracking!

I have managed to get two separate skeletons on the same computer, in two separate processes. The code is fairly straightfoward, everything is the same, except when you look through userGenerators, skip to the one that matches your device index:

        //Look for depth generators

NodeInfoList depthNodeInfoList;

ret = m_Context.EnumerateProductionTrees(XN_NODE_TYPE_DEPTH,NULL,depthNodeInfoList);

if(!CheckStatus(ret)) throw exception();

if(depthNodeInfoList.IsEmpty()) throw exception();

NodeInfoList::Iterator nodeinfoiter = depthNodeInfoList.Begin();

for(int i = 0; nodeinfoiter != depthNodeInfoList.End(); nodeinfoiter++, i++)

{

NodeInfo nodeInfo = *nodeinfoiter;

// Make sure we are using the correct node in multi-Kinect situation

// OR use single node if playing back

// OR if we only have one device plugged in

if(i == m_DeviceIndex || m_NumDevices == 1)

{

XnStatus ret = m_Context.CreateProductionTree(nodeInfo,m_DepthGenerator);

if(!CheckStatus(ret)) throw exception();

m_DepthGenerator.GetMirrorCap().SetMirror(true);

m_DepthMetaData = new DepthMetaData();

//Look for user generators

ret = m_Context.EnumerateProductionTrees (XN_NODE_TYPE_USER, NULL, listUser, NULL);

// Check errors

if (listUser.Begin() == listUser.End()) {

printf("No devices found in context %d.\n",0);

exit(EXIT_FAILURE);

}

printf ("The following user generators were found \n");

int j = 0;

// Checking User type nodes

for (NodeInfoList::Iterator nodeIt = listUser.Begin(); (nodeIt != listUser.End()) &&  (j < m_NumDevices); j++) 

{

// If there are two devices plugged in, nodes 0 and 3 are valid.

// If there are 3 devices plugged in, nodes 0, 4 and 8 are valid.

// (Not tested) If 4 devices, 0, 5, 10, 15?

NodeInfo info = *nodeIt;

ret = m_Context.CreateProductionTree (info, m_UserGenerator);

CheckStatus(ret);

printf("Context %d User: vendor %s name %s, instance %s %d\n",

i,

description.strVendor, 

description.strName, 

info.GetInstanceName(),

description.Type);

if (m_UserGenerator.IsValid()) 

{

const XnProductionNodeDescription& description = info.GetDescription();

// Checking user generator capabilities

if(m_UserGenerator.IsCapabilitySupported (XN_CAPABILITY_SKELETON)) 

{

//Register Callbacks

m_UserGenerator.RegisterUserCallbacks(ProcessManager::NewUser, ProcessManager::LostUser, this, m_hUserCallbacks);

m_UserGenerator.GetPoseDetectionCap().RegisterToPoseCallbacks(ProcessManager::PoseDetected, ProcessManager::PoseLost, this, m_hPoseCallbacks);

m_UserGenerator.GetMirrorCap().SetMirror(true);

// Skeleton stuff

SkeletonCapability skelly = m_UserGenerator.GetSkeletonCap();

skelly.SetSkeletonProfile(XN_SKEL_PROFILE_ALL);

skelly.SetSmoothing(0.0);

skelly.RegisterCalibrationCallbacks(ProcessManager::CalibrationStart, ProcessManager::CalibrationEnd, this, m_hCalibrationCallbacks);

}

}

}

}

Sorry for the sloppy code, didn't get around to beautifying it yet!

I then run a server, which takes up a TCP connexion on 127.0.0.1 for each process and stream the skelly info, without the actual image data, and everything works great!

I have not made any further progress on multiple Skeletons in a single process, and anyways I think this is a better solution in any case because of the fact that you need 1 USB hub for every device. With separate processes, you can easily run a device on multiple computers and link them up on a server!

[Frederic Steinke]

Hey,

nice to hear that you succeded. I think one of the next steps might be to merge those multiple skeleton informations. Did you already create any ideas on that points? It should be a "simple" camera calibrationm but I wonder which method to choose in terms of the kinects and whether to use openni's real world space or not?

Regards,
  Frederic