Quartz Composer-compatible OSC message format

[SteveElbows]

Hello,

Thanks again for the joy that is OSCeleton. I've not tried it on OS X
yet but it was me that was playing around with the windows version
some weeks back & requested a mode where OSCeleton would send messages
in a way that QC likes.

The issue is that Quartz Composer only accepts one type of value in a
message, and floats are the only thing it will accept more than one of
in a single OSC message.

The solution is to move the joint and user data to be part of the
message name/address instead of being sent as data values.

So instead of /joint s i f f f we would get /joint/s/i f f f (or /
joint/i/s f f f)

In other words a typical message may be something like /joint/head/1 f
f f when used in QC mode.

I see you put this in the wishlist file already, cheers. I thought I
would mention it here now that there is a google group, and also
because there could be a 'best way' to do this that I havent fully
considered. Specifically I wonder if we should think about a standard
for the address conventions that would not need to be redone in
future. When I asked about this previously, I had put an extra /pos at
the end of the address, which is not really necessary at all at this
stage, but if there are any plans to add other OpenNI data later then
it may make sense. eg Joint rotations, Im not sure if there is any
merit to sending these over OSC, given all the fun that comes with
Quaternions and other rotation mathemetic, it goes over my head so Im
not fit to judge, but I thought I would mention it anyway.

The other reason I am bringing it up here is because someone has
provided a Max/MSP patch which converts the current OSCeleton OSC
messages into something QC can use:

http://mansteri.com/2011/01/osceleton-with-quartz-composer/

I did try to modify OSCeleton to offer this message format but I am
not skilled at programming in C so my effort was quite a botch and is
not how this stuff should be done properly. It worked though, so I
think the theory is at least correct.

Thanks

[SteveElbows]

Just for reference, here is my basic hack which worked, but Im not
sure if its very efficient and it doesnt take account of multiple
users.

void genOscMsg(osc::OutboundPacketStream *p, char *name) {
char tempy[20];
strcpy (tempy,"/joint/");
strcat (tempy, name);
*p << osc::BeginMessage( tempy );
for (int i = 1; i < nDimensions+1; i++)
*p << jointCoords[i];
*p << osc::EndMessage;
}

(Have to put an include <string.h> in the c++ file to make above work)

Obviously a proper implementation of this would feature other code
changes so that this message format was available as an option via the
command line rather than destroying the default behaviour like mine
does.

Cheers

[George Toledo]

Steve,

Are you saying that QC could receive a structure over OSC that is:

0: 1.264646 1.282828 0.36363 -1.22727

etc., of actual "floats"... and that the structure actually receives properly? I was under the impression it could only work that way if it was a structure of strings with explicit "spaces" between each word (ala RSS patch).

From my standpoint, and this may sound odd, the "best" way to get the data would be as indices that are a mix of string (or indices) and vec4's, with string data (or index val) only indicating users, and all of the joint "string" data we get currently not being represented at all, but with us knowing that 0=right foot, 1=right knee, and so on (or however it lines up now). If a joint is lost, the w value of the vec4 should read 0. That will immediately reflect in OpenCL mesh not rendering anywhere w=0. Advantageously, we can configure another kernel to strip out w vals and replace with whatever we wish (if we want QC to ignore whenever joint data is lost).

This way, we can parse for user id via key, and output the resulting structure directly to QC's integrated geometry pipeline/OpenCL, which isn't a keyed system. It will be really advantageous to at some point put out non-keyed vec4's, because it can be plugged directly into the CL kernel to do skeletal weighting and skinning eventually without any prep other than using the structure key to break off the user.

I'm not saying a labelled/keyed struct isn't useful (or more useful for certain QC workflows), but that in QC, we only strictly need to have a string (or really, an index) that indicates the user ID, and that it's good to have ability strip out all of that joint string info before it hits QC, if possible, so that no prep has to be done to work with the Mesh patches.

So, something like.

0:0

  0:(x,y,z,w)

  1:(x,y,z,w)

  etc

1:1

  0:(x,y,z,w)

  etc....

-gt

--

George Toledo

gtol...@gmail.com

www.georgetoledo.com

The information contained in this E-mail and any attachments may be confidential.

If you have received this E-mail in error, please notify us immediately by telephone or return E-mail.

You should not use or disclose the contents of this E-mail or any of the attachments for any purpose or to any persons.

[SteveElbows]

Well George, I was looking at this from the point of view of the
average QC user, if there is such a thing. Your scenario may well
become popular once there are examples of using this data with OpenCL
mesh stuff etc, but right now I struggle to understand quite a bit of
the important info in your post, some of it seems unintuitive to me.

Lets start with the basics so we can understand where I am coming
from. Ive never done anything with structures of strings coming over
OSC in QC. Rather, just use the standard OSC receiver in QC. Go to the
settings for it in the inspector. Select 'Floats' from the dropdown.
Put in the message address. Press the plus sign. That QC patch will
now output a structure of floats if that OSC message is received and
its only values are floats. These can then be separated via structure
index stuff as normal. Repeat for each joint.

Certainly I can understand that there are scenarios where you may want
all the data to go to an OpenCL patch or whatever in QC, but without
QC examples it will be hard for me to fully get my head round your
ideas and I think whatever happens a nice simple approach that outputs
the different limbs as separate data ports in QC is going to be easier
for many QC users to handle. There is no reason we cant have both
these options, but I think you may have some more work to do to
explain exactly what format you would like OSCeleton to be outputting
the data in to meet your needs.

Cheers

Steve

> gtole...@gmail.comwww.georgetoledo.com

[SteveElbows]

And just to make completely sure that I am explaining myself properly,
here are examples of OSC messages that the standard QC OSC Receiver
can handle:

/joint/1/head 1.233 -0.756 0.887
/joint/1/neck 2.343 1.1 0.003

Each one of these will, when setup in QC, output a structure of 3
floats, and its then trivial to pull the individual x,y & z values out
of these structures by using the index 0, 1 or 2.

Perhaps if you also post an example OSC message that would fit your
requirements, I will better understand your previous post.

Cheers

Steve

[George Toledo]

On Wed, Jan 12, 2011 at 11:30 AM, SteveElbows <st...@dvmachine.com> wrote:

Well George, I was looking at this from the point of view of the
average QC user, if there is such a thing.

No such thing! I'm describing GL/CL centric structure, where there aren't keynames for verts.

 

Your scenario may well
become popular once there are examples of using this data with OpenCL
mesh stuff etc, but right now I struggle to understand quite a bit of
the important info in your post, some of it seems unintuitive to me.

I'm basing it around the actual way that QC works, and the data it takes in naturally, given it's actual geometry pipeline. In GL/CL there's no real reason or support for keyed info for the vec4's ( though we can parse a bunch of stuff and eventually build it into a proper vec4... it's just adding yet another layer of indirectness). 0 can just be foot, and 1 can be knee, etc., and there could be x/y/z/w value, and everything is "a-go" for working with OpenCL or GL Tools.

It's not so academic. If I can get that data in that format, I can write up a kernel that uses the skeletal struct to apply weights to vertices of a skin.

http://www.vimeo.com/18217241

That's just how QC4/SL/OpenCL works... what I'm laying out.

 

Lets start with the basics so we can understand where I am coming
from. Ive never done anything with structures of strings coming over
OSC in QC. Rather, just use the standard OSC receiver in QC. Go to the
settings for it in the inspector. Select 'Floats' from the dropdown.
Put in the message address. Press the plus sign. That QC patch will
now output a structure of floats if that OSC message is received and
its only values are floats. These can then be separated via structure
index stuff as normal. Repeat for each joint.

Ah, gotcha, thanks. I hadn't used that in that way. That's helpful to note.

 

Certainly I can understand that there are scenarios where you may want
all the data to go to an OpenCL patch or whatever in QC, but without
QC examples

Just look at any example where structure plugs into mesh, or mesh plugs into GL Tools via "get vertices".

 

it will be hard for me to fully get my head round your
ideas and I think whatever happens a nice simple approach that outputs
the different limbs as separate data ports in QC is going to be easier
for many QC users to handle.

I tend to think you'll see people taking all of those separate outs, and rebuilding into what is equivalent to the kind of struct I'm talking about, to interface with mesh or gl tools, with added overhead... but who knows!

 

There is no reason we cant have both
these options,

Exactly.

 

but I think you may have some more work to do to
explain exactly what format you would like OSCeleton to be outputting
the data in to meet your needs.

I don't know... I'm sorry if I've been unclear.

It needs to put out something like:

0:0          (note - this is the user1).

   0:0       (note - this is the user calibrated = 1, if true, 0, if false)

     0:(.2,.3,.4,1.)

     1:(.3.,.4,.4,1.)

     2:(.2,.5,.3,1.)   (note - these are the joints represented as array, vec4 data).

     3:etc...

This way, we can use structure indexes to divide the incoming structure up into different users. Then, we hook one more structure index up to derive wether or not user is calibrated or calibrating, to do "something with" if needed. After that second structure index, the subsequent structure would be immediately able to be hooked into GL Tools or QCMesh.

I don't know if it's possible to actually rejigger OSCeleton to do that or not, and the QC OSC implementation is notoriously bad, so it's possible it can't actually output a structure of vectors.

-gt

--

George Toledo

gtol...@gmail.com

www.georgetoledo.com

[SteveElbows]

OK well I really believe that the general QC user is used to receiving
OSC messages that have a pretty simple structure and are to a great
extent easily human-readable.

Like I said, I can appreciate cases where performance is the key, but
I think QC users are more likely to start with the very basics, such
as making a cubes x,y & z positions be controlled by a particular
joint. This is a stupidly basic example but its my starting point. Its
also the approach taken by the person who has written the Max/MSP
patch, which converts OSCeleton OSC messages into something close to
what Ive been discussing (although when I looked very briefly at their
example QC patches I dont think they are taking account of multiple
user skeletons either, just like I wasnt in my quick OSCeleton code
hack).

What I remain confused about is how exactly you are proposing your
desired structure of data be represented in OSC. If I look at the OSC
spec here http://opensoundcontrol.org/spec-1_0 then I can see several
different ways that your data format could be implemented. But as far
as I know the majority of OSC-enabled programs tend to keep things
pretty simple and people are used to just receivng a fairly small
number of integers, floats and strings via any one particular OSC
message. Multiple arrays and stuff like that can quickly go well
beyond this, and Im really not sure QC's built-in OSC receiver will
handle this stuff nicely at all. (although there are other OSC plugins
for QC which have more flexibility but many users are still likely to
end up with a mess of logic patches and wires everywhere as a result,
which has other perforamnce implications) So what are you proposing on
this front exactly, in terms of OSC message format? Sending your
structure as one long string or something far more elaborate? Bear in
mind that OSCeleton as it stands right now, in non-QC modes, sends
stuff in a way thats broadly similar to mine in terms of having a
simple OSC message structure, and separate OSC messages for each
joint.

By approaching this stuff from the opposite end, ie the vert data you
are used to working with, instead of the common norm for
straightforward OSC messages, you are making all this stuff seem way
more complicated than a lot of people are used to. At the very least
you should establish whether such structures will actually work in QC
in the way you need before asking others to support an untested format
that may end up having no practical use. Sorry if that sounds harsh,
but this stuff should be pretty simple and I dont like the thought
that a nice workable QC solution could be delayed due to you muddying
the waters in a strange way.

Steve

On Jan 12, 5:11 pm, George Toledo <gtole...@gmail.com> wrote:
> On Wed, Jan 12, 2011 at 11:30 AM, SteveElbows <st...@dvmachine.com> wrote:
> > Well George, I was looking at this from the point of view of the
> > average QC user, if there is such a thing.
>
> No such thing! I'm describing GL/CL centric structure, where there aren't
> keynames for verts.

[George Toledo]

On Wed, Jan 12, 2011 at 12:46 PM, fsk <fsk...@gmail.com> wrote:

i think this solution is not good. what steve is suggesting is the
correct way of doing it. its just how OSC is supposed to be done. even
if you can parse the data out in QC its just not the right way of
using OSC. the description of the data should be in the name and only
the data should be in the message. you can later build the nested
structure you want with something like structure maker. steves
suggestion would not only work for qc but for most apps that support
OSC. which is what OSC is about. sending nested structures over OSC is
poor practice. as is sending different data types mixed in the same
message.

fsk

Structure maker? That's 3rd party. It would be best done with OpenCL.

I'm just talking about how GL/CL/QC works. What you guys are suggesting, *is* actually sending mixed data (read what you're suggesting again ...mixes of string/float data), which then need to be parsed into floats, and then into an actual structure again, to do anything substantial. I'm just saying that if something is going to be modded to work w/ QC, it should bring in what QC *actually* is meant to use, in it's geometry pipeline (which is OpenCL.)

To be clear, I'm not saying that having keyed data is horrible or anything, but we essentially have that right now, so any more work in that route seems like adding salt or pepper.

-gt 

[George Toledo]

Sigh.

On Wed, Jan 12, 2011 at 12:51 PM, SteveElbows <st...@dvmachine.com> wrote:

OK well I really believe that the general QC user is used to receiving
OSC messages that have a pretty simple structure and are to a great
extent easily human-readable.

Ok, examples?

I'm laying out what QC uses.

There is nothing I'm suggesting that makes it not human readable. I'm suggesting array type data that's used in tons of scenarios besides QC. There's no reason why something has to say "ankle" when it can say "2". 

Like I said, I can appreciate cases where performance is the key, but
I think QC users are more likely to start with the very basics, such
as making a cubes x,y & z positions be controlled by a particular
joint.

That's an assumption I'm uncomfortable with.

It doesn't sound harsh at all. Structures definitely work how I'm conveying; I definitely don't need to test.

Similarly, I would suggest that you look at the OpenCL/Mesh/etc., and how geometry and structures get sent around in QC. *I* don't see that what I'm suggesting is complex, and honestly, I'm having a hard time understanding why it's being taken as "muddying the waters in a strange way", or "delaying" anything. I'm not suggesting that whatever you're suggesting not be implemented, or that it isn't more immediate.

It's not horrible if every thing is a keyed value and a vec3 or vec4, like mansteri's parser app puts out, just kind of cludgy (in my opinion). I'm just suggesting that a joint structure akin to what "get vertices" provides, nested inside of a separate index for each user, is the most eloquent way to get the data. It's fine if anyone disagrees with that, or doesn't see the point. I'm just sharing my thoughts, not trying to catch a beat-down.

-gt 

[SteveElbows]

I dont think you understand OSC very well, frankly. Theres nothing
more I can say, other than I expect your needs would be better met
with a OpenNI QC plugin rather than a weird use of OSC that seems to
throw away much of the point of OSC.

A more fruitful discussion for another day would be whether the world
needs some standards for OSC message formats that deal with skeletal
structures, in the same way that TUIO has done a lot with hand etc
tracking. I'll save this for another thread.

Cheers

Steve

On Jan 12, 5:55 pm, George Toledo <gtole...@gmail.com> wrote:

[SteveElbows]

And I am sorry if it comes across like a pointless beat-down, its not
supposed to be.

I've expressed frustration because you havent been able to explain the
OSC message structure you would like in a way that I can understand
from the OSC side (as opposed to once the data inside QC) ie in a way
that could be explained using the same terminology that the existing
OSCeleton readme does.

Right now OSceleton can output as /joint/ i s f f f, where i is the
user id integer, s is the joint name as a string, and the 3 f's are x
y and z floats.

Im asking for /joint/i/s f f f or /joint/s/i f f f (it doesnt matter
which).

In other words the joint name and the user ID become part of the OSC
messages address, rather than data values. I therefore find your
statement that I am sending 'mixed data' to be potentially misleading,
as the value data is all floats in my request, there are no strings.
The string is the OSC message address, and people using OSC are used
to receiving a variety of messages which have different addresses.

So you have yet to show me how an example OSC message would actually
look, you describe a structure for data but dont say how this will
actually look as an OSC message. Obviously there are too many joints
to give a full example if your desired OSC message contains data for
all joints inone go, so you can simplify by giving an example that
just has a few joints.

We are having this disagreement simply because I am coming at this
from the point of view of existing widespread and straightforward use
of multiple OSC messages. You are coming at it from a totally
different position because you are thinking of your particular usage
scenario and the structure of the data you are well used to
manipulating in this context, rather than thinking about the history
of OSC and its uses. This is not to say that OSC is utterly unsuitable
for your needs, there is so much flexibility with OSC, so my comment
about you being better off with a plugin may have been out of order,
but when considering a general audience it surely makes sense to start
with OSC conventions that have actually been used by people before.

[SteveElbows]

Oops I got something the wrong way round when talking about how
OSCeleton behaves right now, should be /joint s i f f f

Let me put this stuff another way.

The standard QC OSC patch does not support all that OSC has to offer.
It does not like to receive different sorts of data in a single
message, you can set any particular OSC address ( eg /joint/head/1 )
to receive one integer value, one string value, one float value, or
multiple floats.

Other people have written QC OSC plugins which are more sophisticated,
but the nature of QC does indeed cause things to get quite messy in a
hurry if you are having to parse strings and run different branches
depending on which string is present.

So if we look at something like TUIO, which sends stuff to several
different addresses, but where most of the info is contained in a lot
of values of different types per message, people ended up making a
TUIO plugin for QC rather than face a small nightmare dealing with the
standard TUIO OSc data in QC.

There are a variety of approaches to solving similar issues with
getting OpenNI/NITE data in OSC format from the likes of OSCeleton
into QC. Given that QC can accept multiple floats in a single message,
and has no problem dealing with multiple OSC addresses, I cam up with
a pretty trivial way of reformatting the messages so that OSCeleton
would work with QC in a simple way. And given that a main point of QC
is node based programming, exposing each joint as a separate patch
output has obvious merit, and will be what many QC users expect. I did
not take this to the complete extreme, by outputting the x, y and z of
a single joint as 3 different messages just so that QC users would
have no structure at all to deal with, and have a different patch
output for every single joint value, because this seems like a step
too far that is wasteful of network resources etc (3 times as many OSC
messages being sent & received). I saw at least one other person, who
made the Max/MSP OSc conversion patch, took a very similar approach,
so I was emboldened to talk about this stuff here.

I have had considerable trouble in combining your needs and
requirements into the above approach, there are some conflicts as we
can see, and for me this was made much worse by a failure on my part
to understand the exact OSC message you want, ie how the complete OSc
message would look, how the sctructure you want is to be represented
in OSC.

Cheers

Steve

[Tony Gonçalves]

Hi guys,

I think both aproaches are useful, and I'll implement both. However, since Steve's idea is closer to the usual conventions in OSC-based protocols, it seems to me better suited for general purpose development (not all kinect project involve 3d meshes and such) - it's also easier to implement based on current state of the code ;).

I'm a bit swamped with work this week, but I'll try to get it done together with some bugfixes during the weekend. If I see that I have some extra time, I'll do George's too.

Thanks to your discussion I have a pretty good idea of use cases for both approaches. I'd also like to thank you guys for the examples of specific OSC messages and the code samples, which will be of great help!


Cheers,
    Tony

--

-- 

Tony Gonçalves

Partner

to...@sensebloom.com

IPN Incubadora

Rua Pedro Nunes

3030-199 Coimbra, Portugal

Tel  : +351 927 474 665

Tel  : +351 239 700 358

Fax : +351 239 700 301

in...@sensebloom.com

www.sensebloom.com

[George Toledo]

Ah, I see what you're saying Steve. I guess my eventual intent is to plug in that list of floats into a CL kernel, and directly output a skinned model with skeletal hierarchy, rendering with mesh creator and mesh renderer, thereby ending up with way less patches, and having all of the QC processing be able to stay on the GPU wherever CL is supported, instead of having to use CPU to parse.

Alternately, one would plugin a model into input A of the CL kernel, the skeletal floats into input B, and a deform would happen where the skin mesh deforms based on the list of floats.

So, we're talking like, 3 or 4 patches on the QC side to render and control the model.

It's not out of line or taken badly when you suggest that a QC plugin might serve what I'm talking about better; it very well may. I think it would be nifty to bundle up the needed frameworks in a plugin so that people don't have to do all of the installer stuff, and use QC to send around info.

-gt

[SteveElbows]

Thanks Tony!

Also I am sorry to you and everyone for the excessive number of words
I have used in this thread! Im afraid I generally talk too much if I
get over-focussed on a particular subject, as we can see.

Despite all my comments about George's request, I would be more than
happy to see other formats added as time goes on, most of my
frustration was in trying to understand exactly what was desired, and
what the implications were.

I really am so very excited about all of the possibilities that come
from skeletal tracking. And even though several of the environments I
make stuff in are likely to have their own direct ways to harness
OpenNI (eg Unity wrapper, QC plugin), there are some advantages to
sticking with OSC. I am likely to want to control more than one
computer, and/or more than one application with the skeletal tracking
at once, and OSC offers some flexibility in this regard. So I am very
very happy that OSCeleton exists, and I am also keen to see if any
standards emerge for OSC skeletal joint data. None of this stuff is
really as complex as some posts in this thread probably make it sound,
and if it were not for variations in OSC support within different
client applications we would not really need this sort of discussion
at all.

Thanks again, maybe one day I will have a gesture controlled keyboard
that is so slow to operate there will no longer be a danger of me
posting too much here :p

Steve

> t...@sensebloom.com

>
> IPN Incubadora
>
> Rua Pedro Nunes
>
> 3030-199 Coimbra, Portugal
>
> Tel  : +351 927 474 665
>
> Tel  : +351 239 700 358
>
> Fax : +351 239 700 301
>

> i...@sensebloom.com
>
> www.sensebloom.com

[George Toledo]

If it would be possible to configure ports so that id is first:

i/f f f f f f f f f f f f f f f f f f f f f, etc....

... and those floats just came in a known order, it would be really useful. There doesn't have to be any kind of "structure nesting" as fsk was saying was bad form.

So, the first three floats would be "known" to be the x/y/z coordinates for a specific joint (head, if that's the first). Then the next three would be the neck, and so on.

This way, there's a hierarchy where there's an output for each user, and stuff actually work as a single lane of floats, 51 values, if we're talking about currently implemented joints, and no w value for if a joint has disappeared or not.

What would be *more* useful is:

i/f f f f f f f f f f f f , etc...

Where every fourth value was a W bit that shows joint lost as 0 or joint recognized as 1. 

Actually, a single lane of floats is really ideal for GPU based processing.

I'm sorry, I'm not trying to be really obscure or delay any positive movement. A single lane structure of floats, without key names (which is what i/f f f f f f f... would come across as, I believe), would be immediately usable with QC openCL "integrated geometry pipeline" (re:QC4 Read Me- OpenCL/Mesh= QC integrated geometry pipeline). Aurora.qtz is an "ok" example of taking a single lane structure and converting to vec4 via OpenCL, then rendering with meshcreator/meshrenderer. 

It's not a big deal - if no one sees use for it, I'm sure I'll do it myself sooner or later.

-gt

[SteveElbows]

OK thats a lot easier to understand, cheers!

The only bit thats open to misinterpretation is the start. I will
assume that you would like the OSC message address to simply be the ID
of the user. It would probably read more clearly, both interms of
discussing it here and how OSC messages should look, if you put
something before that in the message address, eg like OSCeleton
currently uses /joint

And there arent any implications to this for your use in QC as far as
I can figure at this moment in time.

So for example your message might be:

/joint/i f f f f f f f f f f f f f etc

In QC we would add the address /joint/1 with type floats, and that
would then show up as one output port from the standard QC OSC patch,
and it would be a straightforward indexed structure of floats. The
only question is whether there is a limit to the number of floats that
QC can receive for a given address, or indeed whether OSC has any
limitation in this regard (I dont know). Im also not sure how the
overheads compare between sending lots of shorter messages and sending
a few really long ones, or anything like that.

i is the user ID, and so if you wanted to receive data from a 2nd
skeleton you would need to add /joint/2 as an address for QC OSC to
monitor, and would then get a 2nd output port from the OSC patch that
contains a simple structure of floats just like the first.

If you think there would be an issue with adding /joint/ to the start
of the message address, or you dont want the user ID to be part of the
message address, you want to see it as a number and do something with
it other than have its contents appear as a separate port, then I have
misunderstood you somewhere again!

And sorry for ever suggesting that your discussion of this stuff could
slow down development, there is no evidence for that, I was just
afraid at that moment in time that this topic had become complicated
in a way I did not anticipate, and I wasnt sure how much it would mean
to people unfamiliar with QC.

Cheers

Steve

[matti]

Just throwing my ideas here.

I agree that the message format should be as Steve described (or that
should be an option in addition to the current one). Currently the Max/
MSP patch that I made to reformat the OSC has the user number as the
first member of the message. I also thought about putting it on the
address, but I was just too eager to get it going to actually bother
to do that. Especially since I knew Tony was planning on adding QC
compatible OSC support to OSCeleton.

I can also see a reason for the messages to be formatted in this way:

/joint/i/s/x f
/joint/i/s/y f
/joint/i/s/z f

i is the user, s is the joint name and f is the value.

This would allow for individual joints to be used as controllers in
VDMX (and perhaps some other sotware).

But George's approach would be useful in some cases as well. I don't
have an idea what is more efficient – sendig lots of smaller messages
or one really long message.

Matti

[George Toledo]

Cool.

I don't really see any problem with prefacing it with joint, I just wasn't sure if it was strictly necessary, or made sense. If the outcome is the same, it's sort of a moot point. 

I guess, organizationally, I think of joints belonging to an object, and those joints having points that belong to the joints... which belong to the object. So, user ID seems at the top of the hierarchy (in a general sense of dealing with objects and associated vertices). I know and understand that currently the organization of what OSCeleton puts out isn't really like this. I'm just trying to explain why I wrote that the way that I did. 

Basically, say that there's a way to get a single lane structure of floats of all of the joints from an output that only that user's info comes out of (which seems like it would be the result of what I suggested, and your schema as well), x y z w, repeating. 

One can pipe that right into a CL kernel where you take the total index count, and make every 1st,2nd,3rd,4th val in the sequence into a structure of float4's. This can hook *directly* to the mesh creator, and then the renderer, and now there's the stick guy / point man guy moving around from just osc->cl kernel->mesh creator->mesh renderer, instead of bunches of patches ( a renderer+ parsing patches for every joint), and it's all happening as a parallel process on the gpu.

(Using Mansteri's parser, which is basically what your method would replace, I'm thinking(?), I have 29 layers involved rendering points for each joint, and lines for "bones", each with a parser macro. 

To build that into structure to be ready for OpenCL or Kineme GLTools(3rd party add-on), you would need yet more patch/patches to build it into a usable struct.That becomes *really* indirect, and *many* extra nodes for the QC graph to evaluate. 

So, it just feel ineloquent, even if having big blazing names that say "r_shoulder" may make sense to many users instead of having to make the mental extrapolation that indices 0-4 represent a foot, 5-9 represent a knee, etc., for example, which is kind of the stripped down struct route that would be handy for me, and I truly believe a decent amount of other users to be able to use eventually. Maybe it's not mainstream(?), but it's laid out in Apple's own examples, and a suggested route for handling computation of verts (and pixels) in a gpu accelerated way using cuda and/or openCL, by nVidia, ATI, and intel, as applicable).

With some more maths, you can create normals for GL Lighting, which can hook to the mesh creator as well. With some more time, you can make a skinning setup. By getting in that struct without any of the key info, it can all happen on the GPU and be ultra speedy.

I don't know if I shed any more light on my reasoning, or added more questions! I've long admired your ability to tap into different areas Steve, and I absolutely think that the route you're suggesting will be preferable to many users... I'm just thinking of how I can possibly wrangle it into doing the most possible, in the most performant way, with the least patches.

-gt

[George Toledo]

On Wed, Jan 12, 2011 at 5:01 PM, matti <matti.n...@gmail.com> wrote:

Just throwing my ideas here.

I agree that the message format should be as Steve described (or that
should be an option in addition to the current one). Currently the Max/
MSP patch that I made to reformat the OSC has the user number as the
first member of the message. I also thought about putting it on the
address, but I was just too eager to get it going to actually bother
to do that. Especially since I knew Tony was planning on adding QC
compatible OSC support to OSCeleton.

(ah, ok, I see. Thanks for noting that.) 

I can also see a reason for the messages to be formatted in this way:

/joint/i/s/x f
/joint/i/s/y f
/joint/i/s/z f

i is the user, s is the joint name and f is the value.

I really like that suggestion. Though that requires many outputs in QC, I like that because there's an out for every value, and you're just getting a bunch of floats, without having to potentially parse out stuff. One can just combine. Strictly speaking, more nodes/more ports = slower, but combining the incoming floats in CL, and not having to do any parsing out of info, can be really speedy with the CL kernel (much less so w/javascript, mildly better w/ structure maker (3rd party add on).

 

This would allow for individual joints to be used as controllers in
VDMX (and perhaps some other sotware).

But George's approach would be useful in some cases as well. I don't
have an idea what is more efficient – sendig lots of smaller messages
or one really long message.

Yeah, I'm just throwing my thoughts out there.... based on what I know at the moment, and of course, based on my working methods and performance testing :) If all of a sudden it becomes apparent that sending a message of one user ID, and a bunch of floats (which we would be seeing as a  one lane (1-D) structure is really a performance nightmare in OSC for some reason I defer to the most performant route, obviously!

Throwing this out there for other QC readers using this - the tools we have available to use with OpenCL and writing custom deformers, tracking face to do spline based deformation, using kineme's value historian to record motion captures and embed them to file, or write them externally, can make it possible for us to immediately do a rich variety of things that other techs don't really have objects or modules for at the moment.

Repeat - using the kineme Value Historian for capturing data sequences for the joints is *awesome* in that it gives us our own simple mocap system right now. :)

-gt 

[SteveElbows]

Cheers for your thoughts, and sorry that I got the detail wrong about
what your Max/MSP patch does with the user ID. So are you sending the
UserID as a float rather than an integer so that there are no QC
problems with mixed data types?

And yeah, Im also only asking for the alternative QC-compatible OSC
messages to be available from OSCeleton if you put in a certain
command line parameter, Im not asking for a change to the default
behaviour of OSCeleton. My earlier code example was just a very crude
botch that I made to a slightly older version of OSCeleton, and it
does destroys the default OSC message behaviour as well as ignoring
User ID, so its not very close to being the proper solution.

If I get time in the coming days I will test out this approach you
mention of going even further and sending every float as a separate
message and see if this causes any overload problems. Frankly Ive been
quite surprised at how well QC worked with the OSC coming from my
modified OSCeleton, given that I have had all sorts of bad performance
problems with QC & OSC in the past with, for example, OSC coming from
a Jazzmutant Lemur.

Cheers

Steve

[SteveElbows]

Hello again,

So any ideas how much more computationally heavy or tricky to write
OpenCL code for it will be to combine floats that have been delivered
to QC in the three different ways we have been discussing here?

With my initial idea, you would end up with multiple structures
containing 3 floats each
With your request you would end up with one structure containing all
of the floats
With the latest suggestion you would end up with all the floats
individually, each on a separate port.

Its easy to judge how efficient these 3 options are in terms of the
number of QC wires needed to connect this data to the CL patch, your
request only needs 1 for each user, mine needs one for each joint, the
third way needs 3 for each joint. But what I have no idea about is how
easy it is to combine these differently organised sets of floats via
CL code into the format that you need to go on and do other clever
stuff with, or what the performance pro's and con's are for each
method. Is it harder to combine multiple structures of 3 indexed
floats each than it is to combine lots of individual floats, for
example. You know I love the potential of OpenCL but its been a while
since I played around with it at all, I lack the skills necessary to
use it to do very sophisticated stuff like you are planning to do, Ive
beenplanning on using Unity instead of QC for rigged character stuff.

Anyway Im sure we are going too far into QC territory for this group
now, but it would be good to ascertain how much difference these
issues can make so that it becomes clearer how many different
permutations of OSC message it will be really handy for OSCeleton to
support. Certainly for apps with extremely basic OSC 'only a single-
value per message' support such as VDMX, the 'every joint position
float sent as a separate OSC message with unique address' way of
sending this stuff seems necessary, so I really should test this to
see if it performs adequately.

As for your excitement about using the value historian to capture
motion data, yes theres a world of possibilities there. More broadly,
I think it would be nice if there were some polished OSC recording/
sequencing applications available. You can do it with value historian
in QC but something more general that served a non-QC audience and was
not subject to any of the foibles of QCs OSC implementation/general QC
timing issues would be real nice.

Cheers

Steve

On Jan 12, 10:24 pm, George Toledo <gtole...@gmail.com> wrote:

>
> I really like that suggestion. Though that requires many outputs in QC, I
> like that because there's an out for every value, and you're just getting a
> bunch of floats, without having to potentially parse out stuff. One can just
> combine. Strictly speaking, more nodes/more ports = slower, but combining
> the incoming floats in CL, and not having to do any parsing out of info, can
> be really speedy with the CL kernel (much less so w/javascript, mildly
> better w/ structure maker (3rd party add on).

[fsk]

i agree with you on the part that the nested structure would be the most practical format in QC. but as we are talking about QC compatible OSC messages not QC optimal OSC messages i thought its better to stick to a more standard OSC format. if i use MIDI as an analogy, what steve suggests is sending each joint on a separate control change message (not quite, since its still a structure of floats) and you are suggesting one big sysex message which includes everything. that is why i think steves solution is better. there are more programs out there which support control change messages then there are ones that support parsing of sysex messages. and with something as OSC, the more flies you kill with one slap, the better.

fsk

[Matti Niinimäki]

Just throwing my ideas here.

I agree that the message format should be as Steve described (or that should be an option in addition to the current one). Currently the Max/MSP patch that I made to reformat the OSC has the user number as the first member of the message. I also thought about putting it on the address, but I was just too eager to get it going to actually bother to do that. Especially since I knew Tony was planning on adding QC compatible OSC support to OSCeleton.

I can also see a reason for the messages to be formatted in this way:

/joint/i/s/x f
/joint/i/s/y f
/joint/i/s/z f

i is the user, s is the joint name and f is the value.

This would allow for individual joints to be used as controllers in VDMX (and perhaps some other sotware).

But George's approach would be useful in some cases as well. I don't have an idea what is more efficient – sendig lots of smaller messages or one really long message.

Matti

[George Toledo]

Chiming in, not sure how much use this is, as it's ultra QC specific.

Last night, I was realizing that it's pretty easy to trigger OSCeleton to open at the same time a Quartz Composition runs by using Command Line Tool.

Here's the basic rundown:

-Create an .sh file that reads:

#!/bin/bash

open /Users/georgetoledo/Desktop/osceleton-mac-8110/Sensebloom-OSCeleton-a9e5c55/osceleton-osx

Make the file path *exact*. (Obviously, insert your own file path in the manner that is described above.)

Now, take Command Line Tool, and place the *exact* file path of your .sh file in "path" input.

Now you'll find that when the qtz runs, OSCeleton triggers open automatically.

The only thing is, I haven't figured out a successful way to append arguments to the script, and I've tried a number of routes. 

For my own ease, I compiled a version of OSCeleton last night, changing the .cpp file to broadcast on 8110 so it would immediately work with Matti's app that parses the data. Then, again, since I couldn't get the arguments going successfully, just the open command, I did the QC offsets manually in QC by wrapping my stick figure man in a 3D transform (re: attached qtz).

So, I don't really know how useful it is in the long run. If anyone has any thoughts about how to successfully pass arguments using command line tool's argument function itself, or via the bash going on in the sh file, it's probably more useful than it is right now.

Multiple restarts are also problematic, since nothing is written to shut of OSCeleton; so it will likely result in segfault. *Also*, there's some bug with Command Line Tool that makes it so that the Viewer starts at double size after it's run one time (not much of a problem right now b/c of the fact you can't run the comp twice without segfault, b/c OSCeleton doesn't get shut off).

Interestingly, I'm noting that the latest build seems to keep the sensor active even when I hit control+c (forgetting about all of this Quartz Composer stuff for a moment). The IR looks like it keeps active, so when you start up OSCeleton again, unless you've unplugged/replugged the USB connection, a segfault usually occurs.

That was a bit to wade through I'm sure! 

So, I have very real caveats about the Command Line trigger via QC that I'm posting about, if anyone knows how to issue out the arguments correctly and/or remove the rendering glitch from Command Line Tool (that pops up after one run/stop/restart), it may eventually prove to be handy. 

As it is, if your only requirement is that you want to press play a QC file, and not touch terminal on open, this will do it. Matti's OSCeletonToQC app can be setup to open in the same way, with another sh file, so you can open up all of the stuff with one click, theoretically.

Again, this test demo won't work unless you recompile OSCeleton to transmit on 8110, and run Matti's OSCeletonToQC app, and one also needs to get the 10.5 Command Line Tool plugin developer example from the Apple Developer Archives (or your old dev folder), and compile to 32/64 bit universal, with i_386 and x86_64 for your valid architectures.

-gt

2011/1/12 Matti Niinimäki <matti.n...@gmail.com>

[SteveElbows]

I see that a new version of OSCeleton is now available that has the
Quartz Composer mode. Thanks so much for this!

I do wonder if the way the readme gives an example for user 0 may
cause some people confusion. It kind of implies that we may expect the
first user to have an ID of 0, but in fact, as far as I can tell, the
ID of the first user will always be 1.

Cheers

Steve

[Tony Gonçalves]

I pushed it but didn't have time to send an e-mail to the group at the time.
Oops, better fix the readme file so people don't go crazy trying to figure out what's wrong :P

--

-- 

Tony Gonçalves

Partner

to...@sensebloom.com

IPN Incubadora

Rua Pedro Nunes

3030-199 Coimbra, Portugal

Tel  : +351 927 474 665

Tel  : +351 239 700 358

Fax : +351 239 700 301

in...@sensebloom.com

www.sensebloom.com

[matti]

Hi,

Haven't been able to put the QC mode to the test until now. And I'm
running into a problem.

The joints from the right leg seem to get stuck all the time. This
does not happen to any other joints (except when they are off screen,
which is normal behavior). The right leg joints get stuck continuously
even when they are perfectly in the camera view.

I get this error repeatedly in the Console when this happens:
[0x0-0x69c69c].com.apple.QuartzComposer.editor[51852] *** ERROR: Bad
size count 36 in bundle (only 32 bytes left in entire bundle)

This does not happen when I use the normal OSC mode and route the
messages to QC via Max/MSP.

I was wondering if anyone else can confirm this? Or anyone have an
idea how to fix it?

I'm on Mac OS X 10.6.6, Macbook Pro 2.5 GHz (early 2008 model). I just
did a fresh git clone and compiled from that so I guess that is
OSCeleton 1.0

I also have some ideas about reformatting the QC OSC messages, but
I'll talk about that later when I have the chance to test the idea
properly.

-Matti
http://mansteri.com/

On Jan 18, 12:01 am, Tony Gonçalves <t...@sensebloom.com> wrote:
> I pushed it but didn't have time to send an e-mail to the group at the time.
> Oops, better fix the readme file so people don't go crazy trying to figure
> out what's wrong :P
>
> On Mon, Jan 17, 2011 at 4:57 PM, SteveElbows <st...@dvmachine.com> wrote:
> > I see that a new version of OSCeleton is now available that has the
> > Quartz Composer mode. Thanks so much for this!
>
> > I do wonder if the way the readme gives an example for user 0 may
> > cause some people confusion. It kind of implies that we may expect the
> > first user to have an ID of 0, but in fact, as far as I can tell, the
> > ID of the first user will always be 1.
>
> > Cheers
>
> > Steve
>
> --
>
> --
> Tony Gonçalves
> Partner
>

> t...@sensebloom.com

>
> IPN Incubadora
>
> Rua Pedro Nunes
>
> 3030-199 Coimbra, Portugal
>
> Tel  : +351 927 474 665
>
> Tel  : +351 239 700 358
>
> Fax : +351 239 700 301
>

> i...@sensebloom.com
>
> www.sensebloom.com

[Tony Gonçalves]

This is actually pretty interesting. The right leg is the last stuff to be updated on each snapshot, which makes me think this could be related to the crashes people have been complaining about (somehow the end of the structure or data buffer for the OSC might be getting corrupted / overwritten).

Can you tell me which specific joints have this behaviour (all the leg / just the foot / knee+foot)?

Thanks,

   Tony

[matti]

The entire leg. From hip to foot.

M

On Feb 7, 3:27 am, Tony Gonçalves <t...@sensebloom.com> wrote:
> This is actually pretty interesting. The right leg is the last stuff to be
> updated on each snapshot, which makes me think this could be related to the
> crashes people have been complaining about (somehow the end of the structure
> or data buffer for the OSC might be getting corrupted / overwritten).
>
> Can you tell me which specific joints have this behaviour (all the leg /
> just the foot / knee+foot)?
>
> Thanks,
>    Tony
>

[Tony Gonçalves]

Thanks, I'll try to investigate the issue as soon as i can get my hands on a friend's mac ;)

--

Tony Gonçalves

Partner

to...@sensebloom.com

IPN Incubadora

Rua Pedro Nunes

3030-199 Coimbra, Portugal

Tel  : +351 927 474 665

Tel  : +351 239 700 358

Fax : +351 239 700 301

in...@sensebloom.com

www.sensebloom.com

[SteveElbows]

I think I mentioned this *** ERROR: Bad size count 36 in bundle (only
32 bytes left in entire bundle) stuff here a few weeks ago.

It does not only happen with QC mode, it happens with normal mode too,
if I just happen to have QC running at the same time. Note that QC
cant actually read the normal messages, but its still listening to the
right port and notices the discrepancy between what the bundle size is
and what it expects it to be. I only found this out because during my
attempts to track down the cause of the bundle errors I ran OSCeleton
in normal mode just to see if the problem still existed, I was not
expecting QC to actually be able to do anything useful with the normal
QC messages, but this does rule out the problem only being limited to
the QC-specific parts of your code.

( You wont see these errors when using the Max patch to convert normal
mode to different OSC format because Max does not have this bug with
sending OSC bundles (or doesnt bandle messages at all?), and QC is not
getting any OSC directly from OSCeleton using this method.)

I was not actually sure if this was a bug with the way the OSC library
used by OSCeleton does bundles, or if there is a bug on the QC OSC
listener (ie the error message is itself could be in error). I have no
other tools to analyse the raw OSC data and see if the data is bad, I
guess this is more likely but I dont know how to be sure.

I have no idea if these errors are related to segfaults at all,
certainly I get these errors all the time in the console, especially
if I am moving most of my body, but OSCeleton doesnt crash every time
I see these errors, far from it.

I guess we could try modifying the OSCeleton code so that it does not
use bundles at all, but sends each message on its own. I would think
this is not good for network performance etc, but it would rule out
whether this stuff is actually causing any bigger problems, without
having to look at the underlying OSC library code which makes my head
hurt!

Cheers

Steve

> t...@sensebloom.com

>
> IPN Incubadora
>
> Rua Pedro Nunes
>
> 3030-199 Coimbra, Portugal
>
> Tel  : +351 927 474 665
>
> Tel  : +351 239 700 358
>
> Fax : +351 239 700 301
>

> i...@sensebloom.com
>
> www.sensebloom.com

[SteveElbows]

Just to clarify, in normal mode the error is very slightly different,
but still the same kind of thing:

ERROR: Bad size count 40 in bundle (only 36 bytes left in entire
bundle)

Like I said, I dont know if the OSC size count data is actually bad or
if this is an error with the code Apple have used for QC OSC receiver.
Anybody know of tools to analyse the raw OSC data?

Cheers

Steve

[Tony Gonçalves]

I can try to debug it with wireshark to see if there is a problem with the OSC packets, or if it's just QC's fault. Thanks for the extra info :)

--

Tony Gonçalves

Partner

to...@sensebloom.com

IPN Incubadora

Rua Pedro Nunes

3030-199 Coimbra, Portugal

Tel  : +351 927 474 665

Tel  : +351 239 700 358

Fax : +351 239 700 301

in...@sensebloom.com

www.sensebloom.com