Future of libfreenect / OpenNI

[Nicolas Burrus]

Hi all,

I was wondering, in what direction is going the opensource kinect
community? How are libfreenect/OpenNI going to interact ? Will
libfreenect keep including new features, taken from OpenNI source
code? Do they have a focus that is different enough to justify two
different projects, or will the two projects end up merging into a
single one ?

I must admit that right now it is unclear to me whether as developers
we should keep using libfreenect or start switching to OpenNI (letting
the skeleton tracking part aside).

Any insights?

Cheers,
Nicolas

[Florian Echtler]

Hello Nicolas,

I was wondering the same thing. In my opinion, there is a lot of value in
having both projects exist in parallel. OpenNI is significantly larger and
more complex than libfreenect, and depending on the application, that may
either be an advantage or not. I think that having libfreenect as a lightweight
HAL layer and OpenNI as a more full-featured alternative is fine.

Ideally, I would hope that the new features from OpenNI get merged into
libfreenect over time, making an integration at some point down the road
easier (i.e., libfreenect as the low-level access library and OpenNI as
the dataflow system on top).

Florian
--
SENT FROM MY DEC VT50 TERMINAL

[Nink]

Libfreenect is apache2 with an option of gpl2. Please do not take any code from openni and merge with libfreenect. You are welcome to fork the code and create a new version under gpl but to merge openNI into Libfreenect would not be possible.

Libfreenect is not far behind openNI and has support for motor etc. The project is still very valid especially if you want to develop a commercial product and not have to release your source code.


Sent from my BlackBerry

[Radu Bogdan Rusu]

Nink,

On 12/10/2010 08:29 AM, Nink wrote:
> Libfreenect is apache2 with an option of gpl2. Please do not take any code from openni and merge with libfreenect. You are welcome to fork the code and create a new version under gpl but to merge openNI into Libfreenect would not be possible.
>
> Libfreenect is not far behind openNI and has support for motor etc. The project is still very valid especially if you want to develop a commercial product and not have to release your source code.

OpenNI is not GPL :) It's LGPL on purpose, so that it can be used in commercial products.

Cheers,
Radu.
--
http://pointclouds.org

[Joshua Blake]

Nink is correct. If you make changes to Sensor or OpenNI and distribute it in binary form you would also need to distribute the source code per LGPL. This is not attractive to many companies and some have blanket policies against *GPL. If you don't make changes or don't distribute then most licenses are about the same (except GPL).

 

Anyone confused about incorporating other type of licenses should take a look at this page: http://www.dwheeler.com/essays/floss-license-slide.html Note that OpenKinect is GPLv2, not v2+, and any conversions of Apache 2 to LGPL 3 or GPL 3 might exclude version 4+ if they are not compatible with Apache 2. This gets very complex. The licensing doesn't really matter if you don't distribute the code, but if you do distribute it's easier to keep the projects separate.

 

Josh

 

[Mohamed Ikbel Boulabiar]

On Fri, Dec 10, 2010 at 7:47 PM, Joshua Blake <josh...@gmail.com> wrote:
> Nink is correct. If you make changes to Sensor or OpenNI and distribute it
> in binary form you would also need to distribute the source code per LGPL.
> This is not attractive to many companies and some have blanket policies
> against *GPL. If you don't make changes or don't distribute then most
> licenses are about the same (except GPL).

Complete the part companies like about LGPL !
But if you dynamically link to the library, you can have you code in
any license you want.
Even closed and proprietary ones.

So just modifications to the library itself should be published.

i

[likeBVH]

I agree: the community is making good progress towards milestones that
are beyond simply getting image data from the Kinect camera.
Opensource codes for gesture recognitions and soon the different ways
of extracting body skeleton will make their ways into our repositories
(e.g. beginning next year). The strength of Libfreenect libraries
lies on the facts that it has attracted three dedicated groups of
people:

The first one pioneering new creative ideas, new algorithms and
demonstrating new concepts of natural interaction.

The second one translates the opensource codes provided by the first
and merge into the library with different wrappers (ruby, python, c#,
actionscripts, and more along the way) using the "STANDARDS"
nomenclature.

With a community recognized "standards" comes the third group that
expands the applications of the library.

I am from the third group and I would like to thank the "midnight-oil-
burning" guys who are regular to the OpenKinect IRC and the wrappers
programmers.

The information released from the OpenNI has validated almost all of
their work but hopefully has not completely removed their attitude
towards challenges which they set up to do a few weeks back.

[likeBVH]

I am still trying to figure out OpenNI library. Am I right to say that
the skeletal extraction algorithm distributed with the OpenNI library
is currently available through the C++wrapper and hence a close
source? Any feedback?

[Joshua Blake]

likeBVH,

 

That's a good way to put things. Thanks.

The skeleton extraction and other hand tracking and gesture recognition code is in the NITE module, which is not open source. NITE plugs into the OpenNI framework as a "node."

 

Conceptually if we had an alternative open source solution for skeleton extraction and other things, we could expose that as OpenNI modules/nodes as well. (Patents may be a concern in that space, probably one reason why NITE is not OSS.)

 

Thanks,

Josh

---
Joshua Blake
Microsoft Surface MVP
OpenKinect Community Founder http://openkinect.org

(cell) 703-946-7176
Twitter: http://twitter.com/joshblake
Blog: http://nui.joshland.org
Multitouch on Windows book: http://manning.com/blake

[likeBVH]

Hi Josh, the IRC channels have been rather "un-exciting" lately. There
is no more fighting, no drama, no shouting. One or two characters kind
of MIA. This is making us nervous.

[Joshua Blake]

likeBVH,

 

Part of the natural flow of life, people getting busy. Brandyn is actually just busy with exams for example and the drama was just a bad coincidence.

 

Josh

[Mohamed Ikbel Boulabiar]

On Fri, Dec 10, 2010 at 8:29 PM, Joshua Blake <josh...@gmail.com> wrote:
> The skeleton extraction and other hand tracking and gesture recognition code
> is in the NITE module, which is not open source. NITE plugs into the OpenNI
> framework as a "node."

The skeleton extraction is the most important thing in all "OpenNI",
so with having it closed source, that library should change its name
to ClosedNI which reflects better the state.
(Freeware != GNU's Free Software)
Otherwise, what are the things that are really open source and not
available in libfreenect?
(I really wish here I am wrong...)

i

[Joshua Blake]

OpenNI is just a framework and API, and is open source (LGPL v3+).

 

NITE is the most important thing there, but it would be the same as if any other vendor produced a proprietary product (free or otherwise) that is OpenNI compatible. Hope that makes sense.

 

Josh

[Kai Ritterbusch]

I'm sure that the technology from skeleton extraction to the final
gesture recognition took years to research and refine. Its
understandable that this part is closed source. Still, I think OpenNI is
a good initiative for the community to enable a common framework that
could ensure interoperability between nodes, hardware etc.
(i think ROS does this for robotics) Perhaps universities join and open
their research to the public...

[Dougx]

The thing to remember is that libfreenect is basically a driver. It
has a finite development cycle; one it works, it's... done, basically.
I think the future for application development is an application level
framework (OpenNI or a similar alternative effort) that provides high
level skeleton, gesture, etc.

However, we're a long way from OpenNI being useful. At the moment:

- It needs more easily accessible documentation (Seriously! A chm
file? :P)
- It needs OS modules for gesture recognition, skeleton tacking, etc.
- It needs better drivers (eg. OSX support)
- It needs a better abstract hardware model for sensors that have
other features (eg. Kinect's motor, LEDs)

It's a good framework, but it's just that; a framework. The 'magic
smoke' that lives inside all the abstracted Driver / Hardware /
Recognition modules just doesn't exist.

Then we hit the other snag, which is that it's all entirely C++, and
doesn't have any bindings to other languages yet, and even the very
simple examples are so busy abstracting away behind interfaces they
really don't do anything.

Personally, I'm a little cynical. I can't really look at the OpenNI
initiative without thinking that Primesense is using this as a way of
protecting their own IP. If the efforts to produce open source
gesture, skeleton recognition, etc. are successful, their own IP
becomes vastly less valuable. By providing (closed source but free)
implementations to use through OpenNI they can direct people towards
producing application layer software that doesn't threaten their IP.

Really we have two ways to go:

1) Get behind OpenNI and start writing modules to do all the things
like need doing (Eg. a Sensor using the libfreenect driver), a new
hardware model, etc.

2) Make a new / add application level support to libfreenect.

The folks at WG seem keen to follow (1), but personally I'm not that
convinced a simple c library that does the same thing (abstracted
modules to implement feature recognition) that can easily support
bindings to other languages isn't a better choice. The only benefit
openni is really offering is access to the primesense feature
recognition software (great if you're making applications that need
feature recognition exactly like what's being provided), and hardware
abstraction, which personally I'm happy to handle at an application
layer.

In summary, people are free to do whatever they like, but I'm going to
be sticking to libfreenect.

~
Doug.

[Hector Martin]

On 12/10/2010 04:20 PM, Nicolas Burrus wrote:
> Hi all,
>
> I was wondering, in what direction is going the opensource kinect
> community? How are libfreenect/OpenNI going to interact ? Will
> libfreenect keep including new features, taken from OpenNI source
> code? Do they have a focus that is different enough to justify two
> different projects, or will the two projects end up merging into a
> single one ?

From a technical standpoint, OpenNI's driver doesn't have much to offer
over libfreenect. Although it supports many other features, it turns out
most of those either are useless (e.g. the compressed YUV format that
barely compresses and doesn't buy us anything) or aren't implemented on
the Kinect firmware (as far as I can tell neither frame sync nor
registration nor JPEG are implemented, etc.). I suspect we're at about
85% functionality for the video portion of the Kinect in libfreenect at
this stage. We'll figure out if we can sanely get some reasonable images
in the higher-res modes, interpret the calibration data and other
parameters, maybe make initialization more reliable... and that's about
it as far as things that OpenNI/Sensor does. The PrimeSense firmware
really has a very different feature set from the Kinect, even though
they both come from the same base.

The two projects are definitely not going to merge, as their focus is
entirely different: libfreenect aims to provide a simple,
straightforward C interface to the Kinect, while Sensor is a heavyweight
driver attempting to implement all features of the PrimeSense reference
design and plug into OpenNI, a complex C++ framework. However, we'll
probably see some places where they complement each other. Specifically,
libfreenect might end up taking advantage of the Windows kernel driver
portion of Sensor, and/or OpenNI might gain a libfreenect backend.

It's worth noting that the OpenNI/Sensor license is more restrictive
than the libfreenect license and as such no code will be taken directly.
We'll just reimplement whatever features we find useful using Sensor as
hardware documentation.

From a personal standpoint, I don't see much of a point to OpenNI. It's
(yet another) framework, but it does no actual processing that's useful
to us. The interesting bit (skeletal tracking) is closed source, so the
only real point to using OpenNI right now is to be able to plug into
that closed source blob. And heck, reading Sensor code just isn't fun.

--
Hector Martin (hec...@marcansoft.com)
Public Key: http://www.marcansoft.com/marcan.asc

[likeBVH]

My user name is likeBVH, and BVH stands for Biovision Hierarchy
character animation file format. My motivation of joining this
community is to hope that the effort will lead to a library that can
output motion capture or skeletal extraction data (preferably at
realtime and markerless ) at least in BVH format through kinect -
hopefully before Christmas.

With the availability of NITE samples and the recent Orge-based Nite's
driven avartar from PrimSense, it seems that I do get my Christmas
wish.

I am writing because, in my view, it is due to the community effort
that I get the wish. Since 14Nov, the founding date of the OpenKincet
IRC, we now have more than 1290 developers/users, a libfreenect
library that (according to Hector Martin, although it is only 85%
closed to OpenNI) has spawned far more interesting application areas
than what we can envisage from the combined promotion materials of
PrimSense and Microsoft. It is this achievement that has "encouraged"
the public availability of NITE.

I will use some Star Trak analogies here (sorry for the assumptions).
It took Vulcan 100 earth years to push from Warp 1 to 2 but the human
needs only two years. Both PrimSense and Microsoft have invested
heavily into making the technology (gesture and skeletal extraction)
sufficiently robust for commercial use. However, if you have ideas on
how to push from Warp 1 to 2 and beyond, taking a different route that
promise better performance and features, believing that having two
brains is better than one and working together is fun, than, like what
Hector has said, we may need to set the next bigger challenge,
defining a truly open framework library for human motion semantics and
understanding - based on the existing libfreenect library.

JIM

[Nicolas Burrus]

Thanks for your replies. I think (for the moment at least) that the
conclusion more or less comes from the names of the projects:

- if one wants to get access to raw sensor data from a kinect, then
use the lightweigth libfreenect.

- if you want to get access to skeleton tracking and "natural
interface features", then use Open_Natural_Interface.

Hopefully one day the skeleton tracking part will also be a simple
self-contained module on top of libfreenect, since I guess most people
don't actually want a full framework at that low-level stage, but just
pick up the features they need for their own app.

So I'll stick with libfreenect for the moment, hopefully with windows
support in the master branch soon :-)

Cheers,
Nicolas

[Nink]

Any time someone asks you for a license key tells us at some point we will be asked to pay for the key.

The good news is we can learn a lot from NITE while we are still given free access to the binary.


I would recommend we continue to focus on libfreenect and creating our own skeleton data.

Sent from my BlackBerry

-----Original Message-----
From: Nicolas Burrus <nicolas...@gmail.com>
Sender: openk...@googlegroups.com
Date: Sat, 11 Dec 2010 13:07:24
To: <openk...@googlegroups.com>
Reply-To: openk...@googlegroups.com

[bwh...@dappervision.com]

Josh is right, I'm taking exams now and you better believe when I'm
done I will be back causing drama and shouting : P I am fully
committed to contributing to libfreenect and whenever possible being
compatible with OpenNI as I believe that is the most helpful stance to
take. My sense is other people are doing the same. Before OpenNI we
had a major code push with thanksgiving break and I think we are all
surveying the situation and we will come back more informed and
directed than before.
-Brandyn

[bwh...@dappervision.com]

I am associated with a project to add similar functionality to
libfreenect, stay tuned.
-Brandyn

[originaltomato]

My first post on the group, I hope I'll have more :)

One thing no one mentioned is under what licence is NITE distributed? If I use OpenNI+NITE in binary form can I use it in commercial projects without having to release the code?

Regards

originaltomato

[Louis St-Amour]

I would assume given the mention, though lack of, NITE redistributable downloads, that we would need a license to redistribute NITE or we would need to direct people to download such from the NITE page. It's probably safe to say that if you're planning on commercial use of NITE, you should contact PrimeSense first, though if it's non-commercial or community-based, you can probably direct users to get the NITE SDK the same way you have, at least until the redistributable is released.

As to license, I'd suggest you can use it in all but GPL-ish projects, as NITE is closed source. That's where LGPL or Apache works better, but as licensed, closed source, it's ultimately up to PrimeSense as to whether you can even publish your code. Given the LGPL nature of OpenNI, it's safe to assume such linking in open projects to their closed SDK is allowed, though I'm not a lawyer. ;-)

Louis.

[Joshua Blake]

Commercial use of NITE is free, subject to the terms of the redistributable EULA.

 

Tamir sent this message on the OpenNI-dev mailing list:

 

"NITE is indeed PrimeSense's OpenNI compliant middleware product and as

for now it is not released as open source. The supported systems are

Windows and Linux and the R&D team at PrimeSense is now evaluating OSX

due to community requirements.

The license for NITE allows re-distribution of the redist version and

also free usage of the NITE SDK.

OpenNI compliant NITE can be downloaded from PrimeSense's website:

http://www.primesense.com/?p=515 and a free to use key for the

community can be found on www.openni.org.That key is:

0KOIk2JeIBYClPWVnMoRKn5cdY4=

hope this helps!

TamirB

OpenNI Forum Manager"

 

I also confirmed via email. I asked "Is NITE allowed to be used for commercial purposes or does that require a separate license?" and Tamir said:

 

"The current license for NITE allows distribution of the redist! (which will be available later on) the only requirement is that you make sure the redist’s EULA is signed by the end user (which happens automatically if you use our installer)"

 

Hope this clears things up!

 

Josh

 

[Vladimir Mitrovic]

well thank you that is a very clear answer. as far as comm.
contribution goes i will be posting a small kinect cable tutorial
tommorow..