Next generation API for OpenNI - we want your feedback!

[Ziv Hendel]

Dear OpenNI community,

We are now starting to design the next generation API for OpenNI and
we'd love to get some feedback from all of you to help us with the
process.

Basically, nothing is sacred and everything is open for a change so
please don't be shy to share your thoughts and insights with us :-)

Thanks,
Ziv.

[Roee Shenberg]

Off the top of my head:

Macro-level:

Too many layers of abstraction, resulting in writing way too much boilerplate for simple tasks (e.g. initializing and reading from the sensor) and a really complicated system that's hard to understand.

Also, OpenNI isn't really a spec - especially after having written generators, the spec is basically defined by how the Primesense-provided generators behave - you must match them. If I go to a random function, I'm unclear as to when it can fail, how the failure may affect the system as a whole (does that mean the node is in a bad state? when a node is in an error state, what does it mean? which functions will fail?) - my first random browsing brought me to xnStartGeneratingAll(). The entirety of the documentation is "Make sure all generators are generating data." and stating that the paramater XnContext * pContext is an OpenNI context...

To sum this up - the fact that there is a project called SimpleOpenNI is a somewhat damning indictment of the current API (yes, I know it's a processing wrapper, but it also takes out most of the abstraction layers so becomes much more concise and understandable). I understand that the API is trying to be more generic than say the MS Kinect SDK, but there has to be a simpler way of doing things, perhaps as a specialized wrapper on top of the uber-generic API.

Micro-level:

single-pixel depth->rgb registration - I want the ability to ask for a single depth pixel, what its coordinates are in the RGB image, for example to overlay the stick-man on the RGB image without ruining the depth by using SetAlternativeViewpoint.

Why is framesync default-off? It seems like I'd always want it active...

On reading this before send I realize that the tone is more than a bit harsh. Sorry for that, I really hope that OpenNI gets better - I think it's still too high a barrier for entry right now.

Roee

--
You received this message because you are subscribed to the Google Groups "OpenNI" group.
To post to this group, send email to openn...@googlegroups.com.
To unsubscribe from this group, send email to openni-dev+...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/openni-dev?hl=en.

[Sinan DİNÇ]

Nice to see someone talk about this. I was wondering when openni devs
will wake up and improve the API ?? With the version 1.5, Microsoft
SDK is now far more superior than Openni.
it can do better quality skeleton tracking, supports orientation (both
absolute and local), gestures, even face recognition! But openni
sensor can NOT EVEN
detect Kinect for Windows device, which is a shame for Openni! I
thought open source community would always be ahead of microsoft, but
now
they got too lazy I think??

[Roee Shenberg]

This is not the discussion, I think - Microsoft are investing a lot in their CV middleware, to the point that it has distinct advantages over other offerings. Changing the OpenNI API doesn't mean new capabilities, it means changing how you write the code - you can expect the same skeleton algorithm, etc, just a better way to use them from your code.

Roee

[Sinan DİNÇ]

Unfortunately our company decided to switch to ms sdk for our new
project. We were very loyal to openni until today but we have been
waiting for k4w support since February and only God knows when the
support will come. And whether there will be a near-mode support is
unclear. We cant use xbox 360 kinect for commercial projects and we
dont have xtion pro because it is not sold in my country. So the only
option we have is microsoft's original sdk. And personally I dont
think openni will be able to catch up any more.

And Roee, I agree with you about your suggestions, but I dont see any
better way in openni to use codes because ms sdk is much easier and
error free, no need to create tons of generators, you just type
KinectSensor.Start(), so simple. Like you said, too much abstraction
kills Openni.

[M@]

Personally I'd like to see the same API across all supported
languages. I think its a bit silly that the only java audio help I
can get (for a simple task) on this very forum is telling me to JNI
wrap the C++ API (which is in no way to say that I'm not appreciative
of that help!).

Further, could we use protocol buffers as the API data format? They
have tons of advantages (cross language, cross platform, forward and
backwards compatibility, speed, code generation, etc), but the one
that would be really handy on my project is that it would decouple the
data types that make the most sense for you from the data types that
make the most sense for me.

I'm not sure if this is already in the system, but if it isn't, it
would be handy to be able to change the hardware configuration at
runtime.

That's all that's coming to mind at the moment, I'll make sure to post
again as more comes to mind!

Thanks for involving us!

--M@

[Heresy]

I hope that the C++ wrapper could be easier...

First, in current OpenNI API, when need to get a data array, we need to:

1. get the size of array

2. allocate memory

3. and finally get the array.

4. after using the data, still need to free the array manually.

This coding style is not convenient to use.

It is very simple to been solved with STL container in the wrapper, such as this issue

https://github.com/OpenNI/OpenNI/issues/51 

Ziv Hendel於 2012年5月23日星期三UTC+8上午6時28分38秒寫道：

[Xavier Mendez]

First of all, I love OpenNI.

But, I agree with @Roee,

the API should be simpler.

See Connect, for example.

It provides a basic, yet powerful framework for making

and incorporating middleware (well, not that type of middleware)

in your application. Every middleware is just a function.

A mere function accepting a request, response, and callback

to invoke the next middleware in the chain.

2012/5/23 Roee Shenberg <shen...@gmail.com>

[Jeffrey Kesselman]

Layered APIs are a Very Good Thing (TM) and don't necessarily expose complexity as each layer of hgiher abstracton can wrap multiple lower layers and look exctly the same as a monolithic API would at that level.  What the layers give you however is a way to reach beneath the high level APIs when you have to.

So lets not confuse what might be an overly complex to use API with the fact that it has multiple defined layers of abstraction.  They are orthogonal issues.

Id like to see some additional higher level functionality, myself.  Soem help finding and tracking facial landmarks, for instance,

It's always darkest just before you are eaten by a grue.

[qoolelee]

Hi Ziv. ,

First, I want to than you guys for all those great works so far, (also
thanks to avin of course)

We are a small game developer for arcade machine, and had just
finished our first openni game recently, during the long struggle
development cycle. We found the SDK version you guys released is
getting more and more better. Including the improved skeleton
recognition speed, the tracking stability, which is very important for
us. And we can gain more control over incoming data than windows beta
SDK then. ( I havn't test windows SDK ver. 1 yet)

My only personal suggestion to the upcoming new openni API is

The better manual PLEASE, It's quite a pain in the ass to test every
Xn command to get clear what this command is really for. For example,
we had stock for 1 whole week only to find out how to reset skeleton
recognition after a game cycle, just by a stupid try and error
methods. And no one sample or hint in the manual mention about this.

Anyway, it's really a good news to hear you are planning to improve
the SDK, and thank you for your hard working again.

[qoolelee]

Oh, Sorry, forget to add one,

If the new API allows the developer to decide how many seconds a user
disappear from camera is to be a "Lost user" condition will be great.

Apologize for my bad English.

[Amir Hirsch]

here's a list of stuff i think should be enabled by OpenNI:

- Quaternions in the skeleton model

- UserPixels (userID) should return user pixel coordinates

- easier to access data in projective coordinates, perhaps as a second input to the function: getJointPosition(foo, WORLD_SPACE) getJointPosition(foo, IMAGE_SPACE)

- Face tracking

- gender / age recognition capabilities, possible combining face and gait analysis. (i'm talking more about exposing an API, and integrating with a few services)

- sensor-networking (transmit/receive datastreams via TCP/IP)

- better integration with OpenCV and PCL out-of-the-box (RGB generator and basic optical flow examples)

- hand tracking (finger skeleton, or simple open/closed)

- JavaScript spec

- Programmable gesture interface (an API to HMM or DTW system) both Omek or Flutter style gestures

- support for smart surfaces and virtual touch surface interfaces (like using phone as a controller, or assigning an area on a wall or desk to be touchable

- support for inertial sensors (accelerometer, gyro, wiimote etc)

- standard training interface (ONI files with frame tagging support)

Amir

On Tue, May 22, 2012 at 3:28 PM, Ziv Hendel <z...@openni.org> wrote:

[Felix]

I really like Amir's list, the only thing I would currently add is
local orientation for the joints in addition the current absolute one.

[Xavier Mendez]

2012/5/23 Sinan DİNÇ <sina...@gmail.com>

Nice to see someone talk about this.  I was wondering when openni devs
will wake up and improve the API ?? With the version 1.5, Microsoft
SDK is now far more superior than Openni.
it can do better quality skeleton tracking, supports orientation

OpenNI supports skeleton orientation.

PCL (combined with OpenNI) supports point cloud orientation.

gestures,

OpenNI also supports gestures (not custom gestures though).

even face recognition!

OpenCV (combined with OpenNI) supports face (or other objects)

recognition (not identification).

But openni sensor can NOT EVEN 

detect Kinect for Windows device, which is a shame for Openni!

You said it. Kinect for Windows. No open-source drivers.

Anyway, it hasn't taken more than a month to add support to it.

Now, the SensorKinect works with it.

I thought open source community would always be ahead of microsoft,

Well, OpenNI was released before Microsoft SDK.

Microsoft then made hi-quality middleware and

--as always-- published them as his own propietary,

closed-source, Windows-only SDK.

but now they got too lazy I think??

On May 23, 1:28 am, Ziv Hendel <z...@openni.org> wrote:
> Dear OpenNI community,
>
> We are now starting to design the next generation API for OpenNI and
> we'd love to get some feedback from all of you to help us with the
> process.
>
> Basically, nothing is sacred and everything is open for a change so
> please don't be shy to share your thoughts and insights with us :-)
>
> Thanks,
> Ziv.

[Len Zhong]

adding more:

- better naming: e.g., context, heuristic.

Choosing "nature" names against conceptual ones gives user an instinct sense of object's capability.

- more nodes: e.g., head(orientation, face capability), better hand(shape,fingers)

- provide callback method of data retrieving, besides existing polling one.

- attach timestamp/id to frame.

- better installer/doc/tutorial. 

(I think these are essential part of API - allowing user to understand easily.) e.g., some people still don't realize what OpenNI as a framework can do, what it can't do(usually that part belongs to NITE). And simple installer makes life easier. 

[Artium]

I suggest viewing the following lecture on API design:

http://www.infoq.com/presentations/effective-api-design

[Jeffrey Kesselman]

A big one for me...

if you could, please replace the .NET examples with Mono examples.  They would then be cross-platform.

[Xavier Mendez]

2012/6/7 Jeffrey Kesselman <jef...@gmail.com>

A big one for me... 

if you could, please replace the .NET examples with Mono examples.  They would then be cross-platform.

 I agree. Please replace them.

OpenNI = open-source

Mono = open-source

.NET = propietary

OpenNI = cross-platform

Mono = cross-platform

.NET = Windows only

[Roee Shenberg]

An executable which targets .NET 3.5 or lower will run using Mono - OpenNI.Net.dll can be used with Mono in linux and Mac (In fact, that's how we use OpenNI from the Unity3D wrapper, which works just fine in Mac too). I don't have a linux machine on hand, but SimpleViewer.net.exe should just run if you have Mono installed. You can also build it using only open-source tools - download Monodevelop, open SimpleViewer.net's .csproj, fix the reference to OpenNI.Net and build it (this I did do myself before sending this).

[eight]

Ability to load arbitrary 3d models and have openni match them (assign
labels to pixels according to the match) -- similar to Scene analysis
functionality.


--8

On May 22, 6:28 pm, Ziv Hendel <z...@openni.org> wrote:

[Jeffrey Kesselman]

On Fri, Jun 8, 2012 at 5:24 PM, Roee Shenberg <shen...@gmail.com> wrote:

An executable which targets .NET 3.5 or lower will run using Mono - OpenNI.Net.dll can be used with Mono in linux and Mac (In fact, that's how we use OpenNI from the Unity3D wrapper, which works just fine in Mac too). I don't have a linux machine on hand, but SimpleViewer.net.exe should just run if you have Mono installed.

Always beware statements. that start with "should"...

I re-iterate, please develop them IN Mono and then they will truly be cross platform. Rght now, not so much...

On OSX: 

Jeffrey-Kesselmans-iMac-3:x64-Debug jeffpk$ mono SimpleViewer.net.exe

Unhandled Exception: System.TypeInitializationException: An exception was thrown by the type initializer for System.Windows.Forms.ThemeEngine ---> System.TypeInitializationException: An exception was thrown by the type initializer for System.Windows.Forms.ThemeWin32Classic ---> System.TypeInitializationException: An exception was thrown by the type initializer for System.Drawing.KnownColors ---> System.TypeInitializationException: An exception was thrown by the type initializer for System.Drawing.GDIPlus ---> System.DllNotFoundException: /Library/Frameworks/Mono.framework/Versions/2.11/liblibgdiplus.dylib

  at (wrapper managed-to-native) System.Drawing.GDIPlus:GdiplusStartup (ulong&,System.Drawing.GdiplusStartupInput&,System.Drawing.GdiplusStartupOutput&)

  at System.Drawing.GDIPlus..cctor () [0x00000] in <filename unknown>:0 

  --- End of inner exception stack trace ---

  at System.Drawing.KnownColors..cctor () [0x00000] in <filename unknown>:0 

  --- End of inner exception stack trace ---

  at System.Drawing.Color.get_Black () [0x00000] in <filename unknown>:0 

  at System.Windows.Forms.ThemeWin32Classic..cctor () [0x00000] in <filename unknown>:0 

  --- End of inner exception stack trace ---

  at System.Windows.Forms.ThemeVisualStyles..ctor () [0x00000] in <filename unknown>:0 

  at System.Windows.Forms.ThemeEngine..cctor () [0x00000] in <filename unknown>:0 

  --- End of inner exception stack trace ---

  at System.Windows.Forms.SystemInformation.get_MenuAccessKeysUnderlined () [0x00000] in <filename unknown>:0 

  at System.Windows.Forms.Control..ctor () [0x00000] in <filename unknown>:0 

  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control:.ctor ()

  at System.Windows.Forms.WindowsFormsSynchronizationContext..cctor () [0x00000] in <filename unknown>:0 

[ERROR] FATAL UNHANDLED EXCEPTION: System.TypeInitializationException: An exception was thrown by the type initializer for System.Windows.Forms.ThemeEngine ---> System.TypeInitializationException: An exception was thrown by the type initializer for System.Windows.Forms.ThemeWin32Classic ---> System.TypeInitializationException: An exception was thrown by the type initializer for System.Drawing.KnownColors ---> System.TypeInitializationException: An exception was thrown by the type initializer for System.Drawing.GDIPlus ---> System.DllNotFoundException: /Library/Frameworks/Mono.framework/Versions/2.11/liblibgdiplus.dylib

  at (wrapper managed-to-native) System.Drawing.GDIPlus:GdiplusStartup (ulong&,System.Drawing.GdiplusStartupInput&,System.Drawing.GdiplusStartupOutput&)

  at System.Drawing.GDIPlus..cctor () [0x00000] in <filename unknown>:0 

  --- End of inner exception stack trace ---

  at System.Drawing.KnownColors..cctor () [0x00000] in <filename unknown>:0 

  --- End of inner exception stack trace ---

  at System.Drawing.Color.get_Black () [0x00000] in <filename unknown>:0 

  at System.Windows.Forms.ThemeWin32Classic..cctor () [0x00000] in <filename unknown>:0 

  --- End of inner exception stack trace ---

  at System.Windows.Forms.ThemeVisualStyles..ctor () [0x00000] in <filename unknown>:0 

  at System.Windows.Forms.ThemeEngine..cctor () [0x00000] in <filename unknown>:0 

  --- End of inner exception stack trace ---

  at System.Windows.Forms.SystemInformation.get_MenuAccessKeysUnderlined () [0x00000] in <filename unknown>:0 

  at System.Windows.Forms.Control..ctor () [0x00000] in <filename unknown>:0 

  at (wrapper remoting-invoke-with-check) System.Windows.Forms.Control:.ctor ()

  at System.Windows.Forms.WindowsFormsSynchronizationContext..cctor () [0x00000] in <filename unknown>:0 

[Joshua Marner]

The current framework is fine for the most part, though I find it a little confusing dealing with nodes that are dependent on other nodes, especially when I want multiple ones of the same type that are dependent on several different nodes.

For example:
I have 2 kinects connected to my computer, and I want each of them to have an independent skeleton tracker. So for each depth map that I find, I create a user generator. The problem is that the user generator is dependent on the scene generator and the gesture generator, which in turn are dependent on the depth map. However, in the list of user generators that pop up, most of them have the gesture generator dependent on one depth map and the scene generator dependent on the other depth map. This shows a somewhat convoluted model that looks at the raw data as "just another node" when it should be represented as the fundamental element.

Also, whoever programmed the skeletal trackers (aka "UserGenerators") built in some kind of memory conflict that refuses to allow it to work properly even when I get everything set up right with the two separate depth generators (thread: "About skeleton tracking with multiple kinects" in this Google Group) Right now I get around it by recording the two kinects to an *.oni file, they playing them back one at a time and extracting skeleton data (which works because I don't really need or want live data)

More intuitive support for accelerometer and motor data for devices that offer it (eg, Kinect) would be a major plus. Actually making sure that all the features on the Kinect are supported with the base shipment of OpenNI code would be a major plus, as the kinect most likely has the largest market share of sensors (and you do it with NITE). Most of us are using OpenNI because we're not working on Windows and we need access the power of the kinect without writing our own skeleton tracking algorithms (THANK YOU for that much).

So... yeah. Better integration with the kinect, which is the standard sensor right now, and better differentiation between the raw data provided and what is calculated from that raw data (ie, a cleaner hierarchy) Also, wouldn't OpenCV make a great middleware option? Y'all should work together more.

PS, please overload operator-> in the c++ wrapper for the iterators. That gets annoying after a while :P

[Krzysztof Abramczyk]

Thank you all for great work you do. OpenNI is great. I work in a
small comapny I based on our customers I would be nice if:
- more points on the skeleton. Fingers, more points on torso, head and
arms.
- simpler way to use it with kinnect. Asus sensors are more expensive
then kinnect but they work the same
- handgenerator could track multiple hands from multiple users the
same way that it tracks the first one
- multiple skeletontracking in one process.
- more gesturegenerators

[eight]

In case of multiple kinects, ability to create unified generators, say 2 kinects -- 1 depth map generator and other generators that follow.

Thanks.

--8

[alex_karate_do]

I quote eight... many sensors, one user generator please! It would solve many tracking problems...

[David Menard]

Many great ideas here. I would like to see some tools/framework to help with using multiple Kinects across multiple processes. 

Maybe add a InitializeDeviceProcess(int,DepthGenerator&,UserGenerator&, Callbacks&) that takes in the device index and callbacks, and return the depth and usergenerators.

Or, some kind of build-in IPC that uses sockets to span devices on multiple machines!

Anyways, I think this could give OpenNI a big big edge over KinectSDK! And it's already supported by the library, it's just not very easy to set up right now!

 

 

[Rafael]

The request may come a little late but here's one thing I would love to have in OpenNI:
Improved Android USB communication. At the moment all USB communication on Android is handled by libusb using Linux usbfs. However, this has many drawbacks in Android (usbfs is set back to read-only after every restart and reconnection, among other things) and it is not the supposed way of doing USB host communication in Android. In fact, usbfs is a completely unsupported way of doing USB when writing Android applications.

Android comes with an own Java package that handles USB host communication: android.hardware.usb. It would be very nice if OpenNI would use these classes to talk to the camera instead of libusb and usbfs.

Rafael