Kinect With Labview

[Kamarulzaman Kamarudin]

Hi,

Is there anybody working on Kinect with Labview? Which driver are you using? I found one driver developed by Ryan Gordon but

when I downloaded the source code, the wrapper wasn't there. Appreciate your time and help. Thanks.

Regards,

Arul

[Troy Boswell]

I had to use labview and kinect for an undergraduate project. I gave up on directly interfacing as I am not a fan of labview. I ultimately built a udp client-server. The server managed the kinect using an embedded linux development board and. The client received the stream and processed the data. It works pretty well and the kinect is very easy to control in linux with libfreenect.

[Kamarulzaman Kamarudin]

Thaks for the reply. What board (brand) are you using for Server and also for Client? I can do programming in Labview, C and C++. Do you use C for that? Anyway,  I would really appreciate if you could share your source code.

Thanks

On Thu, Sep 13, 2012 at 6:39 AM, Troy Boswell <troy.b...@gmail.com> wrote:

I had to use labview and kinect for an undergraduate project. I gave up on directly interfacing as I am not a fan of labview. I ultimately built a udp client-server. The server managed the kinect using an embedded linux development board and. The client received the stream and processed the data. It works pretty well and the kinect is very easy to control in linux with libfreenect.

--

Best Regards,

 

Arul Kamarudin

CEASTech Unimap

[Mike North]

I'd like to see more information on this as well. I'm currently part of a group looking into doing something very similar to use the Kinect as the primary sensor for a robot. I'd prefer not to go through trial and error finding a board that will work. Could a raspberry PI be sufficient if all it does is parse the data and pass it on to our NI controller? Or would we still need something beefier?

Thank's for any input.

[Γιάννης Γράβεζας]

I'm using a similar client server setup over TCP(HTTP). I've tried the Pi but it seems to drop all packets coming from the cameras so the kinect doesn't work with it. Some report success with the Xtion though, so I guess it's the hub internal to the kinect that's causing the problem. They Pi devs are trying to fix the driver(for a very crappy hardware usb impl) but I'm tired of waiting. I've used my setup with success on a pandaboard running both server and (graphical)client. If you're just interested on getting the data from the cameras and transmiting them over the network you'd need about 100Mhz of a modern ARM processor. A beaglebone should be more than sufficient. Maybe this http://www.acmesystems.it/aria as well but I haven't yet tested it

--
Yannis Gravezas

Founder@Intrael

Athens, Greece

+30 6947931815

[ErikH2000]

I'm not saying it can't be done, but I had problems with the Beagleboard and Beaglebone dropping frames. 

In general, to get this to work I would either look for someone saying "yes, I tried this exact config and it worked," or plan on wasting some time and money on trial and error. The USB hardware/software seems to be causing problems on different platforms despite ample CPU. So it's not really possible to look at specs and make a solid prediction a Kinect app will work on it.

Here is what worked for me for converting depth data to TCP: Intel Atom-based d2500hn board running Lubuntu. It's overkill and larger physical size than I wanted.

-Erik

[Troy Boswell]

I used a Pandaboard as my embedded Linux target platform. This platform ran Ubuntu 11.10 with libfreenect and opencv libraries installed. The application was written in C++ to create a UDP server that served detected objects to the client upon request.

The client was written in labview for a Crio, could easily be replaced by a desktop. A simple client sends a UDP packet to the server using Labview UDP vi's and waits for a reply, then does something meaningful with the reply. You could reply the entire Kinect images however you would need to investigate whether UDP or TCP would be appropriate for your application. For streaming Kinect data I would probably stick with UDP, however if only periodic access to the images is required, I would probably go for TCP.

Troy

[Troy Boswell]

I have had success on both Beagleboard and Pandaboad. I might buy a Pi and see what I can get going, however I am in the middle of uni exams so it might have to wait as it will be a tempting distraction.

Basically my implementation relies on Libfreenect and Opencv 2.1, thus I use Ubuntu 11.10 which provides OpenCv2.1 in the repository (I didn't get around to learning 2.3). Libfreenect installs as specified on the website, it was initially broken for ARM's cores however Drew Fisher was kindly able to repair a memory allocation problem. I initially ran LXDE during my development stages, however it does require considerable resources and a headless ubuntu is a much better option once functionality is achieved.

My client-server architecture is developed using sockets. The USB connection for the kinect is provided by the HS USB hub back the OMAP's ULPI port. I have found this to be a trouble free exercise with the Kinect and Ethernet both sharing the Hub back to the ULPI port. However I did not stream entire images as I allowed the application to perform all the required object detection and only return the detected objects. However I believe streaming is possible and again when time permits I'll see how it goes.

As for Pi, unless the USB interface is not a High Speed HUB or similar capability i cannot see why it will not work. As for dropped frames with Libfreenect, this is my biggest dislike about this library and my experience has found that this problem is not limited to ARM platforms. Libfreenect requires a separate thread to be spawned, this is a basic task with examples provided on OpenKinect, however if I block the UDP server waiting for a new UDP packet I get a nice stream of dropped packets. I have not being able to draw the link between the main thread impeding on the spawned thread. This problem is replicated on my x86 platform. Thus my only solution at current is to poll the UDP interface for a newly received packet, thus increasing CPU load.  To me this is the only draw back of Libfreenect, otherwise I have found the library to be fantastic and initializing the library in registered mode (Depth Image matched to the RGB image) provides images highly suited to problem of computer vision as a coloured three dimensional representation of the local environment can be achieved.

Unfortunately if I cannot solve the blocking problem I am going to have to try OpenNI to see if the problem is replicated their, however unlike Libfreenect the installation task of OpenNI has proved to be painful and time consuming.

Troy

[Γιάννης Γράβεζας]

On Sun, Oct 21, 2012 at 2:06 AM, Troy Boswell <troy.b...@gmail.com> wrote:

Basically my implementation relies on Libfreenect and Opencv 2.1, thus I use Ubuntu 11.10 which provides OpenCv2.1 in the repository (I didn't get around to learning 2.3). Libfreenect installs as specified on the website, it was initially broken for ARM's cores however Drew Fisher was kindly able to repair a memory allocation problem. I initially ran LXDE during my development stages, however it does require considerable resources and a headless ubuntu is a much better option once functionality is achieved.
 

My client-server architecture is developed using sockets. The USB connection for the kinect is provided by the HS USB hub back the OMAP's ULPI port. I have found this to be a trouble free exercise with the Kinect and Ethernet both sharing the Hub back to the ULPI port. However I did not stream entire images as I allowed the application to perform all the required object detection and only return the detected objects. However I believe streaming is possible and again when time permits I'll see how it goes.

I think we're on the same track here. Glad someone else thinks the client server approach has merit. Streaming the whole frames would require 100Mbit/s just for the depth frame though so I don't think it's feasible on arm boards(I haven't heard any that's got gigabit yet)

 

As for Pi, unless the USB interface is not a High Speed HUB or similar capability i cannot see why it will not work.

The Pi has a crappy hardware usb implementation. In my early experiments the kinect wouldn't even show up when probing the usb. I had to short circuit some polyfuses on the usb ports that where causing problems. Now I can see the kinect, tilt it, play with the leds but the camera, while appearing to initialize, don't yield any packets. Their usb is just bad. I was very dissapointed but it still makes a nice xbmc board so I could suggest it for that since it's cheap. I'm quoting from the most relevant thread on their forum

----- THREAD: http://www.raspberrypi.org/phpBB3/viewtopic.php?f=28&t=12097&start=575

USB is still an issue. Much of this stems from the (as I understand it) fact that the USB controller IP used in the SoC is pretty much undocumented beyond the (buggy) driver implementation provided by Synopsys. This is compounded by the fact that the Pi uses that IP in a way that most people don't, viz: as a controller rather than as a device.

This is *not* something that the Foundation have any control over, nor is it even the fault of Broadcom, who simply used a piece of silicon IP as part of the design of the 2385 SoC - in most use cases, the IP and driver work "well enough" not to cause any problems, so you can't even blame Broadcom. Yes, with hindsight, it's not the best, but it's too late now. I'm sure Gordon would be overjoyed if it suddenly, magically, got replaced with something else. Y'know, something that works out of the box.

Unfortunately, that's not gonna happen, and Gordon is heroically slaving away at actually fixing the issues. And he's doing a bloody good job of it; USB performance and stability has improved massively since the first release. It's still far from perfect, but we all know that.

 ------

As for dropped frames with Libfreenect, this is my biggest dislike about this library and my experience has found that this problem is not limited to ARM platforms. Libfreenect requires a separate thread to be spawned, this is a basic task with examples provided on OpenKinect, however if I block the UDP server waiting for a new UDP packet I get a nice stream of dropped packets. I have not being able to draw the link between the main thread impeding on the spawned thread. This problem is replicated on my x86 platform. Thus my only solution at current is to poll the UDP interface for a newly received packet, thus increasing CPU load.  To me this is the only draw back of Libfreenect, otherwise I have found the library to be fantastic and initializing the library in registered mode (Depth Image matched to the RGB image) provides images highly suited to problem of computer vision as a coloured three dimensional representation of the local environment can be achieved.

I'm not exactly certain if I understood your code setup, but I ended up triple buffering the camera data, processing them on a separate thread and serving them on the network through the main thread. I'm using select() and poll the socket list with a zero timeout whenever freenect_process_events() returns. This works nicely, I don't know if it's applicable to your case though.

Unfortunately if I cannot solve the blocking problem I am going to have to try OpenNI to see if the problem is replicated their, however unlike Libfreenect the installation task of OpenNI has proved to be painful and time consuming.

Nah, libfreenect is still the king :) BTW Are there any plans on merging the xtion and K4W branches? You know for completeness

 

[Troy Boswell]

Its quite amazing that USB issues are plaguing the Pi.
However the provided link does highlight the project is in the Beta version and I am sure Pi will overcome them and we will see libfreenect operational.

Additionally companies like TI and ST Micro are providing ULPI ports on M series cores to provide High Speed USB, the STM32F4 is a perfect example of a microcontroller powerful enough to operate the Kinect's High Speed USB Hub. How effectively 30Hz of both frames could be accessed may be a separate issue but this is a much lower end piece of silicon in comparison to an A series.

Back to Ethernet streaming

Yes 30Hz may not be achievable, but 30Hz may not be the desirable rate for all applications. Accessing the server at a decreased rate would allow streaming to occur, not applicable to all users but since human reaction time is in the order of milliseconds maybe 4 Hz could be achieved which would be about 50Mbit/s. But yes a bigger pipe would be nice.

As for my issue with dropping frames I have been advised previously to try triple buffer, now I've heard it twice I guess its time to make the modification.

Cheers Troy

[Γιάννης Γράβεζας]

On Sun, Oct 21, 2012 at 4:10 PM, Troy Boswell <troy.b...@gmail.com> wrote:

Its quite amazing that USB issues are plaguing the Pi.
However the provided link does highlight the project is in the Beta version and I am sure Pi will overcome them and we will see libfreenect operational.

Maybe on a Pi2 with a better processor. The usb is implemented in the SoC itself so there not much hope with the current incarnation. Personally, I'd like the Pi to go into the dustbin of history. It's essentially a different architecture(ARM11) compared to modern arm systems (ARM v7 aka Cortex Ax). This forces dual builds of binaries and denies devs of certain useful features that are considered normal these days, like NEON which is the ARM equivalent of SSE2. The good thing about the Pi though, is the GPU which is very impressive and very well supported. It has OpenGL ES2, OpenVG and even a solid OpenMAX implementation(An api for accelerated encoding/decoding of media). I'd love to see that GPU matched with a cortex. It should be feasible now at a marginal cost increase compared to their current offering. Fortunately lots of other decent boards are getting in the market, like the http://cubieboard.org/

 

Back to Ethernet streaming

Yes 30Hz may not be achievable, but 30Hz may not be the desirable rate for all applications. Accessing the server at a decreased rate would allow streaming to occur, not applicable to all users but since human reaction time is in the order of milliseconds maybe 4 Hz could be achieved which would be about 50Mbit/s. But yes a bigger pipe would be nice.

The problem with this approach is latency. An unpacked depth frame is around 3.3 Mbits. When transmitted over 100 Mbit/s ethernet(Which is around 70 Mbits real data transmission) would add almost 50 msecs of latency. Add to that the 10 or more msecs of latency for the usb bus itself to deliver the frames to the server and we can forget about any meaningful NUI application. It could be applicable to applications that do surface reconstruction though

[Kamarulzaman Kamarudin]

Do you guys think 60msec latency is good enough for dynamic application say for mobile robot use?

I am currently looking for something similar; streaming the kinect's depth data to a client running labview program.

Would netbook with Windows 7 Starter be compatible with Libfreenect and good enough to run this job?

Thanks for any input

Arul 

[Γιάννης Γράβεζας]

It depends on the function you want to accomplish with the depth input, you want to use it in order to avoid obstacles? That could be tough. Bear also in mind that these 60 msecs of latency are just for delivery of the frame. You have to also consider the latency of actually processing the frame and responding to stimuli. I haven't done any robotics but for user interfaces I believe the threshold is around 30-40 msecs from physical action to gui response if you want the user to feel that the interface is responsive  

On Mon, Oct 22, 2012 at 6:18 PM, Kamarulzaman Kamarudin <arul....@gmail.com> wrote:

Do you guys think 60msec latency is good enough for dynamic application say for mobile robot use?

I am currently looking for something similar; streaming the kinect's depth data to a client running labview program.

Would netbook with Windows 7 Starter be compatible with Libfreenect and good enough to run this job?

Thanks for any input

Arul 

[Troy Boswell]

Depends on how path planning and object detection are performed, and the speed of the robot. I built an occupancy grid so latency was an issue as the robot moved at 0.5m/s, thus I had to get the localization of the robot (its location in the map) to the object detection server and capture the kinect images prior to the images becoming invalid for the provided localisation data. This isn't the only approach available, due to the kinect having minimum depth range of 500mm you need close range sensor anyway. Use these for collision avoidance (direct control of drive frame) and stream the kinect at a reduced rate to look for distant obstacles, since there will be a large amount of error in the distance to the detected object you will need to determine how much error will exist and the decision that the robot will make.

[Γιάννης Γράβεζας]

A good option to reduce the min range is the nyko zoom. It needs some tuning but it picks up readings from 300mm. You can always reduce the latency by downsampling. A 40x30 map should be adequate for robotics(But I base that on absolutely no experience in the field). I agree with troy though that direct control of the drive frame is a must, you robot must have instincts to survive

On Wed, Oct 24, 2012 at 12:39 AM, Troy Boswell <troy.b...@gmail.com> wrote:

Depends on how path planning and object detection are performed, and the speed of the robot. I built an occupancy grid so latency was an issue as the robot moved at 0.5m/s, thus I had to get the localization of the robot (its location in the map) to the object detection server and capture the kinect images prior to the images becoming invalid for the provided localisation data. This isn't the only approach available, due to the kinect having minimum depth range of 500mm you need close range sensor anyway. Use these for collision avoidance (direct control of drive frame) and stream the kinect at a reduced rate to look for distant obstacles, since there will be a large amount of error in the distance to the detected object you will need to determine how much error will exist and the decision that the robot will make.

[Mike North]

Similar question regarding robotics obstacle-avoidance application. I'm working for a robotics team looking to use the kinect in this capacity as well, but I *think* that latency will not be an issue for us.

Our robot will only need to take kinect data, process it, and do route planning from rest. The robot can be completely stopped for this. Our arena is fairly small (about 1.5 times the length of the kinect's range) so we can analyze a series of stills (the kienct will be mounted on a servo), route plan, move as far as we could see with the kinect depth data, and then collect and analyze more data.

Given that, would any amount of latency matter...at all? Obstacle detection and route planning is problem I'm familiar with, it's this client-server data transfer that's an unknown to me.

Thanks again for the input folks!