Linux kernel driver for kinect v2

[Hiromasa YOSHIMOTO]

Hi everyone,

I wrote a linux kernel driver for kinect v2 device. The code is available at 

https://github.com/yoshimoto/gspca-kinect2

This driver uses video 4 linux 2 (v4l2) interface for both color and depth stream

so that you can use your kinect as a web-camera.

I also wrote a library, named libk4w2, for this kernel driver.

https://github.com/yoshimoto/libk4w2

It is basically rewrite version of libfreenect2. However, it also includes the following 

features;

- Optimized depth decoder by using OpenMP 

- CUDA-based color decoder, which may run faster than trubojpeg

I'd like to contribute to openkinect project with these codes.

Any comment is appreciated.

Thanks,
Hiromasa YOSHIMOTO

[Zafar Ansari]

Good job:) Thanks a lot.

--
You received this message because you are subscribed to the Google Groups "OpenKinect" group.
To unsubscribe from this group and stop receiving emails from it, send an email to openkinect+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Antonio Ospite]

On Mon, 8 Jun 2015 01:41:08 -0700 (PDT)

Hiromasa YOSHIMOTO <hiromasa....@gmail.com> wrote:

> Hi everyone,
>

Hi Hiromasa,

I am the author of the kinect1 gspca driver.

> I wrote a linux kernel driver for kinect v2 device. The code
> is available at
> https://github.com/yoshimoto/gspca-kinect2
>
> This driver uses video 4 linux 2 (v4l2) interface for both color and depth
> stream
> so that you can use your kinect as a web-camera.
>

I don't have a kinect2 so I don't know much about it and I cannot test
the code, bu I took a brief look at it anyway, it seems to be in
a decent shape, it only needs some touches before you cold propose it to
linux-media[1] and have it in the linux kernel.

[1] http://www.linuxtv.org/lists.php

Maybe start proposing the gspca.[ch] changes about no_clear_halt so
that you can remove these files from your repository.

And then send the patches to add support for the depth stream
pixelformat.

Is the V4L2_PIX_FMT_Y11BPACK the same exact packed format used by the
kinect 1? If so the name is OK, if not it would be better to come up
with another name.

When adding a new pixel format to videodev2.h it also needs to be
documented, follow the example of the Y10B format:
grep -r -i Y10B .../linux/Documentation/DocBook/

When submitting kinect2.[ch] to linux-media please describe briefly the
kinect2 hardware and explain what the pixel formats are; people in
linux-media don't necessarily know the details, for instance I don't.

Some comments on the kernel driver:

- run linux/script/checkpatch.pl -f kinect.[ch]

Check the style issues and also look at
linux/Documentation/CodingStyle and
linux/Documentation/SubmittingPatches

BTW most warnings about "line over 80 characters" can be ignored but
some actually make sense;

- avoid forward declarations of functions (sd_stopN,
sd_private_ioctl), just reorder the definitions;

- have separate functions and separate sd_desc for color and depth
stream, and just check for the interface in sd_probe() to register
the appropriate subdriver, look at kinect.c in the latest kernels to
get an idea.

AFAICS the kinect2 has color and depth stream on two separate
interfaces, right? So you are going to have two instances of the driver
for each device, each instance will have it's own subdriver.

Just for the records, the kinect 1 makes it harder to support color and
depth at the same time with the current gspca infrastructure because it
uses two different endpoints for them on the same interface.

> I also wrote a library, named libk4w2, for this kernel driver.
> https://github.com/yoshimoto/libk4w2
>

I didn't look at this, could you please add a README which describes
what it does?

From your brief comment below I take that libk4w2 can work either with
libusb or with your kernel driver, right? So, in a way it overlaps with
libfreenect2; can that cause fragmentation?

About using the kernel driver, have you looked at libv4l[2] from
v4l2-utils[3] to see if it's possible to use its plugin infrastructure
for the depth stream decoding?

[2] http://linuxtv.org/downloads/v4l-dvb-apis/libv4l.html
[3] http://git.linuxtv.org/v4l-utils.git

> It is basically rewrite version of libfreenect2. However, it also includes
> the following
> features;
> - Optimized depth decoder by using OpenMP
> - CUDA-based color decoder, which may run faster than trubojpeg
>
> I'd like to contribute to openkinect project with these codes.
> Any comment is appreciated.
>
>
> Thanks,
> Hiromasa YOSHIMOTO
>

Ciao ciao,
Antonio

--
Antonio Ospite
http://ao2.it

A: Because it messes up the order in which people normally read text.
See http://en.wikipedia.org/wiki/Posting_style
Q: Why is top-posting such a bad thing?

[Hiromasa YOSHIMOTO]

Hi Antonio,

Thank you for your valuable comments!

> 2015/06/08 23:06、Antonio Ospite <a...@ao2.it> のメール：

>
> On Mon, 8 Jun 2015 01:41:08 -0700 (PDT)
> Hiromasa YOSHIMOTO <hiromasa....@gmail.com> wrote:
>
>> Hi everyone,
>>
>
> Hi Hiromasa,
>
> I am the author of the kinect1 gspca driver.
>
>> I wrote a linux kernel driver for kinect v2 device. The code
>> is available at
>> https://github.com/yoshimoto/gspca-kinect2
>>
>> This driver uses video 4 linux 2 (v4l2) interface for both color and depth
>> stream
>> so that you can use your kinect as a web-camera.
>>
>
> I don't have a kinect2 so I don't know much about it and I cannot test
> the code, bu I took a brief look at it anyway, it seems to be in
> a decent shape, it only needs some touches before you cold propose it to
> linux-media[1] and have it in the linux kernel.
>
> [1] http://www.linuxtv.org/lists.php
>
> Maybe start proposing the gspca.[ch] changes about no_clear_halt so
> that you can remove these files from your repository.

From your comments below, I got a new idea to remove
no_clear_halt modification from gspca.[ch].
I’d like to test it first in this weekend.
(Unfortunately, I’m very busy now)

Anyway, I completely agree with you; changes should be proposed first.

> And then send the patches to add support for the depth stream
> pixelformat.
>
> Is the V4L2_PIX_FMT_Y11BPACK the same exact packed format used by the
> kinect 1? If so the name is OK, if not it would be better to come up
> with another name.
>
> When adding a new pixel format to videodev2.h it also needs to be
> documented, follow the example of the Y10B format:
> grep -r -i Y10B .../linux/Documentation/DocBook/

Thank you. I’ll check it later.

>
> When submitting kinect2.[ch] to linux-media please describe briefly the
> kinect2 hardware and explain what the pixel formats are; people in
> linux-media don't necessarily know the details, for instance I don't.
>
> Some comments on the kernel driver:
>
> - run linux/script/checkpatch.pl -f kinect.[ch]
>
> Check the style issues and also look at
> linux/Documentation/CodingStyle and
> linux/Documentation/SubmittingPatches
>
> BTW most warnings about "line over 80 characters" can be ignored but
> some actually make sense;
>
> - avoid forward declarations of functions (sd_stopN,
> sd_private_ioctl), just reorder the definitions;
>
> - have separate functions and separate sd_desc for color and depth
> stream, and just check for the interface in sd_probe() to register
> the appropriate subdriver, look at kinect.c in the latest kernels to
> get an idea.
>
> AFAICS the kinect2 has color and depth stream on two separate
> interfaces, right? So you are going to have two instances of the driver
> for each device, each instance will have it's own subdriver.
>
> Just for the records, the kinect 1 makes it harder to support color and
> depth at the same time with the current gspca infrastructure because it
> uses two different endpoints for them on the same interface.

OK, I think I can follow your advices. I will update kinect2.c in this weekend.

>> I also wrote a library, named libk4w2, for this kernel driver.
>> https://github.com/yoshimoto/libk4w2
>>
>
> I didn't look at this, could you please add a README which describes
> what it does?
>
> From your brief comment below I take that libk4w2 can work either with
> libusb or with your kernel driver, right? So, in a way it overlaps with
> libfreenect2; can that cause fragmentation?

You’re right. I want to merge with libfreenect2 as soon as possible.

>
> About using the kernel driver, have you looked at libv4l[2] from
> v4l2-utils[3] to see if it's possible to use its plugin infrastructure
> for the depth stream decoding?
>
> [2] http://linuxtv.org/downloads/v4l-dvb-apis/libv4l.html
> [3] http://git.linuxtv.org/v4l-utils.git
>

IMO, the code for stream decoding should be implemented as a cross-platform library.
For example, I've already defined decoder APIs in libk4w2/decoder.h.
If libfreenect2 provides official APIs like this, we can easily implement those plugins.


Regards,
Hiromasa YOSHIMOTO

[Daniel Johnson]

On Mon, Jun 8, 2015 at 7:06 AM, Antonio Ospite <a...@ao2.it> wrote:
> On Mon, 8 Jun 2015 01:41:08 -0700 (PDT)
> Hiromasa YOSHIMOTO <hiromasa....@gmail.com> wrote:
>
>> Hi everyone,
>>

>> I wrote a linux kernel driver for kinect v2 device. The code
>> is available at
>> https://github.com/yoshimoto/gspca-kinect2

Awesome!

>> I also wrote a library, named libk4w2, for this kernel driver.
>> https://github.com/yoshimoto/libk4w2

I noticed in your v4l2 code xioctl which makes me think you copied
some code from the v4l2 example app. From my understanding it's
licensed GPL2.0. i'm working on code to support the intel Realsense
camera which also shows up as two /dev/videoX devices. I wanted my own
code to be LGPL2.1 so I went out of my way to rewrite everything. I
think making my client code work with your kinect2 driver should be
pretty simple and allow for more code sharing between the various
depth cameras. I haven't shared my library code yet so I can release
it with an either or license if LGPL2.1 isn't compatible with your
project.

> From your brief comment below I take that libk4w2 can work either with
> libusb or with your kernel driver, right? So, in a way it overlaps with
> libfreenect2; can that cause fragmentation?

Again with the realsense camera there is a libusb driver I didn't
write that works, but I think supporting either driver is useful.

> About using the kernel driver, have you looked at libv4l[2] from
> v4l2-utils[3] to see if it's possible to use its plugin infrastructure
> for the depth stream decoding?

I think it's a GPL library and thus only GPL programs can use it.

Sorry that my feedback is mostly license related rather than
technical. I'd love to have a common camera library for the various
cameras. After I get the basic library, and calibration done for
realsense I want to work on more advanced library features like
gestures, and 3d scanning.

--
teknotus (Take Notice)