Custom Hit Detection

[Casey Scalf]

What would be the possibility (or does it already exist) of making a
custom Hit Detection Patch. By this I mean being able to take any spot
in the XYZ grid and making a object to hit - definitely possible. Now
instead of the hand or the head, or the leg interacting with this what
about anything?

The values from the Kinect seem to give properties to the data
(attributing legs, hands, and such) but what about if anything could
satisfy a hit. Say a soccer ball, or any hand of any person, without
need for calibration?

I could see many possibilities where what you need to interact with is
not a Person, or better yet, is just a part (say a hand). So without
the need for full calibration wouldn't we get better performance?

What if we could make a patch for hit detection dealing with data
points instead of translated values a.k.a. a skeleton. Then be able to
use this with a skeleton possibly.

Has anybody else come to this?

[Sebastian Kox (gMail)]

Op 21 jun 2011, om 05:05 heeft Casey Scalf het volgende geschreven:

> What would be the possibility (or does it already exist) of making a
> custom Hit Detection Patch. By this I mean being able to take any spot
> in the XYZ grid and making a object to hit - definitely possible. Now
> instead of the hand or the head, or the leg interacting with this what
> about anything?

There is the multiple objects hit test patch in the latest version. With this you can generate an object hittable by one skeleton element. If you need it to react to multiple skeleton elements/users you'll need to copy the patch for each skeleton element (hand/foot etc.)/user, and connect the obect's coordinates time.

> The values from the Kinect seem to give properties to the data
> (attributing legs, hands, and such) but what about if anything could
> satisfy a hit. Say a soccer ball, or any hand of any person, without
> need for calibration?
>
> I could see many possibilities where what you need to interact with is
> not a Person, or better yet, is just a part (say a hand). So without
> the need for full calibration wouldn't we get better performance?
>
> What if we could make a patch for hit detection dealing with data
> points instead of translated values a.k.a. a skeleton. Then be able to
> use this with a skeleton possibly.
>
> Has anybody else come to this?

It could be possible, but without calibration you'll have to rely on some sort of blob detection.

This might help:
http://www.patriciogonzalezvivo.com/blog/?cat=118

[Casey Scalf]

Hello Sebastian,

I am downloading the Synapse project to take a look at it. It seems very intuitive and my Roommate DJ is going to help me with the Abelton side of things. Should be really cool. I totally see how this could save a lot of hassle so we will see where it goes. I think the hardest part of using the tryplex toolkit is the installation. After that it is easy sailing in my book ha. As for an app off of this, it would be interesting to see how to bundle it all together into a clickable app.

As far as the QC community goes, I think we should really try to pull together and collaborate to make this tool really great. I could imagine that the WWDC conference did not talk too much about it, but I feel it is really the way to go and should be progressed as well. I will start working on those letters and videos today, so that will help it along.

It will be interesting to see with Lion as well what updates were put in.

Also, I just got myself an Ipad to use with touchOSC and I think I saw your name on the forums. Have you been using that software? I just used it last night at a show and it rocked!!!

Until then,

Casey

[Sebastian Kox (gMail)]

Hi Casey,

> I am downloading the Synapse project to take a look at it. It seems very intuitive and my Roommate DJ is going to help me with the Abelton side of things. Should be really cool. I totally see how this could save a lot of hassle so we will see where it goes. I think the hardest part of using the tryplex toolkit is the installation. After that it is easy sailing in my book ha. As for an app off of this, it would be interesting to see how to bundle it all together into a clickable app.

True, it's definitely a great step forward in usability. I already contacted Ryan from Synapse, and he's willing to implement full skeleton tracking and maybe in the future some more stuff like multiple skeleton etc. so I'm hoping to put together a tryplex update soon.

> As far as the QC community goes, I think we should really try to pull together and collaborate to make this tool really great. I could imagine that the WWDC conference did not talk too much about it, but I feel it is really the way to go and should be progressed as well. I will start working on those letters and videos today, so that will help it along.

There are quite some discussions over at kineme right now about the position of quartz composer (you started one of those discussions), those are interesting reads. I think more exposure to QC, it's use for visualists and fast prototyping need to be propagated, and basic education is lacking. If the threshold will be less steep to start with QC it'll probably increase the user base. I've got time around the end of July, so it might be good to make some sort of plan together of what we're going to cover/how etc.

> It will be interesting to see with Lion as well what updates were put in.
>
> Also, I just got myself an Ipad to use with touchOSC and I think I saw your name on the forums. Have you been using that software? I just used it last night at a show and it rocked!!!

I did, and also used it for some live performances already. It's definitely really handy for a lot of stuff, although midi/homemade controllers with real buttons are still the best for slamming and rocking during live performances, but touchosc is a great companion!

Best,
Sebastian.

[Gabe Mott]

Kudos to the both of you for pushing this along. 

Synapse merged the depth of field with with the skeleton. That was a great step. In a VJ environment this is going to deepen the experience without losing the aesthetic tremendously. (I'm dealing with a harddrive crash so haven't tried implementing anything-- and then I'll be travelling). I am looking forward to finding out how well this functions in VDMX. It looks to me like he's (ryan at synapse) is going to run the visuals from the QC viewer. 

I was honestly considering running parallels and switching to Windows after the MS SDK came out for the kinect because it apparently does not rely on a calibration position. Then synapse happened giving me hope for staying on MacOSX. Incidentally, there is an interesting conversation on the OPENNI group about getting to no calibration. 

Here's a note i got re calibration position (or getting away from it):

I would suggest you to read the paper proposed by 
MS(http://research.microsoft.com/pubs/145347/BodyPartRecognition.pdf) 
about the method they used to "track"(if you read the paper you'll 
find that it's more like detection rather than tracking) full body 
parts and hence avoid pose calibration. 

Thanks again to the both of you!

[Casey Scalf]

I agree with Gabe, the Synapse take on using the Kinect with Quartz Composer is great. I plan on exploring that tonight using Abelton Live.

I am glad to hear you were able to contact Ryan, it seems like if he is doing this for Burning Man, and sharing the knowledge with others along the way, then it is bound for good. If this could be used in a Tryplex Toolkit then it would really bring everything together.

For getting a stronger QC base together I think we should go for it. I am still waiting on a copy of Lion but we need not wait just for that. I am out of school for the summer, and if you are going to have some free time soon then why not pool everyone's skills?

And yes, for the OSC app, awesome once again. I see using it with a Tryplex composition to to change what is happening on the fly, eliminating going back to the computer every time I want to change a slider. Also, I had a question about getting custom layouts to send the correct data types to QC. I posted it on the Hexler site, and I was wondering if/how you encountered this problem.

http://hexler.net/forum/viewthread/294/

Cheers,

Casey

[Casey Scalf]

It sounds like Blob Detection will be the way to go. I checked on the link you sent and I was impressed with what he had going on. Especially the bit with hand tracking, increased dexterity is always good.

Furthermore, do you know of a good blob tracker that could be loaded into Quartz Composer? I did not see a link or any further resources beyond the videos on his blog for QC related files.

Thanks,

Casey

[Casey Scalf]

I did remember using these by Dust before, as reminded by Gabe, but it
is a bit slow on the frame rate unfortunately.

http://kineme.net/composition/dust/clcentroidtracker

[Sebastian Kox (gMail)]

> It sounds like Blob Detection will be the way to go. I checked on the link you sent and I was impressed with what he had going on. Especially the bit with hand tracking, increased dexterity is always good.
>
> Furthermore, do you know of a good blob tracker that could be loaded into Quartz Composer? I did not see a link or any further resources beyond the videos on his blog for QC related files.

http://www.patriciogonzalezvivo.com/blog/?p=364
It can send out tuio messages that QC supports with this http://www.tuio.org/?software

Dust's sample gives a good representation of what is possible within the limits of QC, and there is also the Kineme CVtools. There are some basic samples in point tracking, and I think those are the basics that the openCCV also uses. However for speed I believe QC can't beat the speed that any C based external program can give you.

Also check out this thread:
http://kineme.net/forum/Discussion/DevelopingCompositions/OpenTSPSOSCdataintoQC

opentsps is something like openCCV, and here they actually got a workaround in getting blob outlines into QC with a processing reroute.

[Casey Scalf]

I checked out the links you gave and found them super useful.

On the KinectCoreVision with finger tracking, I got stuck when receiving TUIO/OSC messages into QC. I was able to get the qcOSC patch to recognize incoming data in port 3333 but was not sure how to interpret "tuio2Dcur" which is what appeared when I started tracking my fingers.

I tried using a structure key but wasn't sure which key to use, 0 and 1 didn't seem to get me too far.

Maybe there is a quick fix for this, but thanks for the info regardless.

Casey

[victor morales]

Hey sebastian

How compatible are triplex and synapse? triple-x-synapse ;)

Is it just a matter of linking the right joints to the right inputs in
qc to make the stickman and/or the puppet-sprite work?
I havent try it yet, i will look at it tomorrow, but if you have any
suggestions?

if i figure it out i will post the .qtz here.

thanks...!

ah regarding the new openNI drivers (from my other threat) i think i
will stick with synapse, seems so much more portable and hassle free,
specially if you and ryan are planning to collaborate, both are great
projects!

thanks!

victor

[Sebastian Kox (gMail)]

Hi Victor,

Synapse only outputs head&hands data, no full skeleton. I also ran into a problem with synapse, it freezes after running it for 30 minutes. Did you experience this?

Sebastian.

[moral...@gmail.com]

actually i ran it today for more than half an hour... and it was ok. I will do a proper consistent test tomorro

cheers

v

[victor morales]

Hey sebastian

I ran synapse today for hours without it crashing or giving errors...
also i was able to modify the max file so it outputs all joints/bones.
Are you interested on this? it may be a nice alternative to running
osceleton with the processing patch..
I could upload it here if you want.
Let me know!

cheers

victor

On Jul 4, 10:27 pm, moralvic...@gmail.com wrote:
> actually i ran it today for more than half an hour... and it was ok. I will  
> do a proper consistent test tomorro
>
> cheers
>
> v
>

[Sebastian Kox (gMail)]

Hi Victor!

Great! Maxmsp is expired on my system, so I couldn't edit the patches.
If you could mail or upload it would be very helpfull.
I also had the 1.1 version running for a full day without any problems.

Thanks!

Sebastian.

[Casey Scalf]

Hello Victor.

I would be interested in the updated Max file. Would you mind sharing it?

Casey

[victor morales]

Ok here are the files

Note that the joints are not the same as tryplex
the list of valit joints is

• righthand
• lefthand
• rightelbow
• leftelbow
• rightfoot
• leftfoot
• rightknee
• leftknee
• head
• torso

oh the standalone app i couldn't upload here
but you can download it here

http://dl.dropbox.com/u/6196104/quartz_passthrough_plus_plus.zip

hope it works, leit me know if you make any progress on the tryplex side. I don't know max very well, but i can hack at it, this is just a modification of the original file from Ryan, who made synapse, so all credits for him... you can find more info about other functionality here

http://synapsekinect.tumblr.com/post/6307752257/maxmsp-jitter

so it may need some tweaking to make it work with tryplex

the port is 12348

cheers

victor

[victor morales]

oh i forgot to upload the original max file

[victor morales]

Hey

anybody made any progress with synapse+tryplex?

On Jul 18, 11:32 pm, victor morales <moralvic...@gmail.com> wrote:
> oh i forgot to upload the original max file
>

>  quartz_passthrough_plus_plus.maxpat
> 16KViewDownload

[trypl...@gmail.com]

Expect an update soon, working on it!

[Emer Värk]

Hey Sebastian

I'm playing with Ni-mate to skeleton.

And I don't get kinect skeleton stickman patch to work when it's inside Render in Image macro patch.
Any workarounds for that?

Emer.

[Sebastian Kox (gMail)]

Hi,

It should work, could you share your composition (or the part that isn't working) so I can have a look?

Best,

Sebastian.