new Version of OpenKinectCamera class for managed c#/c++

[EisernSchild]

Due to the positive Echo to my first try to build a managed class from
the code here is the new version (v20101203)

note that You need the file "libusb0.dll" in the "C:\Windows\System32"
folder to run the dll.
It is Part of the libusb-win32 USB library.
http://sourceforge.net/projects/libusb-win32/

official OpenKinect drivers from
https://github.com/OpenKinect/libfreenect/tree/master/platform/windows/

Class KinectCamera
v2010-12-03

NEW
* as close to OpenKinect repo code as possible to make any further
updates easier
* provides managed Bitmap class images for both Video (ARGB) and Depth
(easy to use in AForge.NET C# Framework... http://www.aforgenet.com/)
* logging

TODO
* directX and XNA Texture direct Bayer->ARGB convert
* depth grayscale/color export
* set ARGB alpha channel
* motor/led
* camera does not turn off ??

C# Demo: (containing a built of the dll ("OpenKinectCamera.dll"))
http://dl.dropbox.com/u/13924535/KinectDemo.zip
(Visual C# Express project)

C++ Source:
http://dl.dropbox.com/u/13924535/OpenKinectCamera_v20101203.zip
(Visual C++ Express project)

cheers

P.S.: looking forward for the official C# wrapper from LostInCake ;-)

[likeBVH]

Hi EisernSchild,
Keep coding. Your codes are really helpful.

[likeBVH]

Hi EisernSchild,
Do consider interface with OpenCVSharp at code.google..
I find it the most efficient way to port OpenCV++ into .NET platform.

[GUNNM]

This is excellent!
The first version of C # wrapper that I tried was in xaml, that I do
not know,
Windows form is well know for those how use C#.
With you're code I will be able to use the KINECTS in C#.
But, please don't stop until you we'll be able to use the raw IR
image.

On 3 déc, 20:51, EisernSchild <eisernsch...@gmail.com> wrote:
> Due to the positive Echo to my first try to build a managed class from
> the code here is the new version (v20101203)
>
> note that You need the file "libusb0.dll" in the "C:\Windows\System32"
> folder to run the dll.
> It is Part of the libusb-win32 USB library.http://sourceforge.net/projects/libusb-win32/
>

> official OpenKinect drivers fromhttps://github.com/OpenKinect/libfreenect/tree/master/platform/windows/

>
> Class KinectCamera
> v2010-12-03
>
> NEW
> * as close to OpenKinect repo code as possible to make any further
>   updates easier
> * provides managed Bitmap class images for both Video (ARGB) and Depth

>   (easy to use in AForge.NET C# Framework...http://www.aforgenet.com/)

> * logging
>
> TODO
> * directX and XNA Texture direct Bayer->ARGB convert
> * depth grayscale/color export
> * set ARGB alpha channel
> * motor/led
> * camera does not turn off ??
>

> C# Demo: (containing a built of the dll ("OpenKinectCamera.dll"))http://dl.dropbox.com/u/13924535/KinectDemo.zip

[EisernSchild]

Here s the next version, all fixed.

Please post in the forum if You have any ideas for the class.

salut

Class KinectCamera
v2010-12-05

* as close to OpenKinect repo code as possible to make any further
updates easier
* provides managed Bitmap class images for both Video (ARGB) and Depth

(easy to use in AForge.NET C# Framework... http://www.aforgenet.com/)
* logging

* vertical Sync fixed
* depth grayscale/color gradient export
( use
"Camera.freenect_set_depth_format((freenect_depth_format)5);"(gradient)-

"Camera.freenect_set_depth_format((freenect_depth_format)4);"(grayscale) )
* motor/led
* ARGB alpha channel (for both depth and video in case of ARGB output)
* XNA Texture direct mapping
( done by IntPtr videoData & depthData like this :

Texture2D tx;

//create data buffer
byte[] bytes = new byte[640 * 480 * 4];

// copy bitmap data into buffer

System.Runtime.InteropServices.Marshal.Copy((IntPtr)KinectCamera.depthData,
bytes, 0, bytes.Length);

// set our buffer to the texture
tx.SetData(bytes);
)

(Release Built)
http://dl.dropbox.com/u/13924535/OpenKinectCamera_v20201205_Release.zip
(Code)
http://dl.dropbox.com/u/13924535/OpenKinectCamera_v20101205.zip
(Demo - Update)
http://dl.dropbox.com/u/13924535/KinectDemo.zip

[EisernSchild]

Hi likeBVH

I think that looks promising. Will definatly care about OpenCV !

[likeBVH]

I am trying to incorporate more freenect methods into your managed
class(CLI). This will simplify the process to update your managed
class to more recent updates in freenect.

e.g.
int freenect_process_events(); == your run() method.

By doing so, it also help to integrate with LostInCake C#wrapper (that
has many good features to ensure good frame rate)

Methods that are missing in zephod win32 v17 are:

int freenect_get_mks_accel(double %x, double %y, double %z);
freenect_raw_device_state* freenect_get_device_state();
int freenect_update_device_state();


int freenect_init();
int freenect_shutdown();

int freenect_num_devices();
int freenect_open_device( int index);
int freenect_process_events();

//void freenect_set_user()
//void *freenect_get_user();

int freenect_start_video();
int freenect_start_depth();
int freenect_stop_depth();
int freenect_stop_video();

void freenect_set_log_level(freenect_loglevel level);
int freenect_set_video_format(freenect_video_format fmt);
int freenect_set_depth_format(freenect_depth_format fmt);

int freenect_set_tilt_degs(double angle);
int freenect_set_led(freenect_led_options option);
double freenect_get_tilt_degs();

int freenect_get_mks_accel(double %x, double %y, double %z);
freenect_raw_device_state* freenect_get_device_state();
int freenect_update_device_state();

[likeBVH]

My previous post refers to the vesion prior to 6th Dec.

[likeBVH]

Hi EisernSchild
None of the IR mode is working. Hope you have time to figure out.

[EisernSchild]

Hi likeBVH


Know that the IR thing is not working meanwhile.. i try to figure out.

Your doing is really great, i also absolutely look forward to get
stuff together with LostInCake.


Goodnight, late here....;-)

[Jesse]

Great stuff. Lurking here, but will contribute when I have anything
worth adding. Please keep up the good work!

[EisernSchild]

Hi likeBVH !

IR now working ! Was quite a simple Bug (ARRG) : Did forget to clear
the line

video->pkts_per_frame = VIDEO_PKTS_PER_FRAME;

So it always did set it back to the Bayer Packets count...;-)

will include th convert_uyvy_to_rgb() these days !

now updated:
http://dl.dropbox.com/u/13924535/OpenKinectCamera_v20201205_Release.zip
http://dl.dropbox.com/u/13924535/OpenKinectCamera_v20101205.zip
http://dl.dropbox.com/u/13924535/KinectDemo.zip

On 6 Dez., 20:04, likeBVH <jim...@googlemail.com> wrote:

[Betti]

Hi Eisern,

Do you know why OpenKinectCamera DEmo c# program return to me the
message

send_cmd: Output control transfer failed (-5)

Kinect led it's fixed Red color .

I install Kinect with NUI Driver as usual . I can move motor and
change led color by using LibUsbDotNet dll and send command to Kinect
directly but i can't see any camera .

Any idea ?

Thank you
Ale

On 6 Dic, 14:03, EisernSchild <eisernsch...@gmail.com> wrote:
> Here s the next version, all fixed.
>
> Please post in the forum if You have any ideas for the class.
>
> salut
>
> Class KinectCamera
> v2010-12-05
>
> * as close to OpenKinect repo code as possible to make any further
>   updates easier
> * provides managed Bitmap class images for both Video (ARGB) and Depth

>   (easy to use in AForge.NET C# Framework...http://www.aforgenet.com/)

> * logging
> * vertical Sync fixed
> * depth grayscale/color gradient export
>   ( use
> "Camera.freenect_set_depth_format((freenect_depth_format)5);"(gradient)-
>
> "Camera.freenect_set_depth_format((freenect_depth_format)4);"(grayscale) )
> * motor/led
> * ARGB alpha channel (for both depth and video in case of ARGB output)
> * XNA Texture direct mapping
>   ( done by IntPtr videoData & depthData like this :
>
>             Texture2D tx;
>
>             //create data buffer
>             byte[] bytes = new byte[640 * 480 * 4];
>
>             // copy bitmap data into buffer
>
> System.Runtime.InteropServices.Marshal.Copy((IntPtr)KinectCamera.depthData,
> bytes, 0, bytes.Length);
>
>             // set our buffer to the texture
>             tx.SetData(bytes);
>   )
>
> (Release Built)http://dl.dropbox.com/u/13924535/OpenKinectCamera_v20201205_Release.zip
> (Code)http://dl.dropbox.com/u/13924535/OpenKinectCamera_v20101205.zip

> (Demo - Update)http://dl.dropbox.com/u/13924535/KinectDemo.zip

> > > P.S.: looking forward for the official C# wrapper from LostInCake ;-)- Nascondi testo citato
>
> - Mostra testo citato -

[likeBVH]

Hi Betti

OpenKinectCamera only works with the

window driver distributed here

https://github.com/OpenKinect/libfreenect/tree/master/platform/windows

Not the NUI Driver.

[likeBVH]

Tested, it works. Thanks.

[Betti]

Hi likeBVH

Yes !!! Thank you ..

There was a Xbox NUI driver installed not usable or old i don't know
--..

Now work well .

The strange thing was that i had the three drivers (Xbox NUI Camera ,
Audio , Motor) under Lib Usb 32 Hardware Installed Tree on window 7 .

Now are under Human Interface Device i think it's correct ..

I was doing a direct connect to kinect through LibUsbDotNet.dll by
send correct command i can drive motor and Led ... easy ... other
think it's make a camera interface ..

Ciao

On 8 Dic, 17:58, likeBVH <jim...@googlemail.com> wrote:
> Hi Betti
>
> OpenKinectCameraonly works with the

[Betti]

Hi likeBVH ,

I talk to fast ... after window7 reboot all work find ... i see the
camera exc... then if i stop the project KinectDemo and i make the
second cmpile .
the message : send_cmd: Output control transfer failed (-5) coming
again .
I think something are loked the camera .. led and motor work fine ..

Any Idea ..

Thank you


On 8 Dic, 18:06, likeBVH <jim...@googlemail.com> wrote:
> Tested, it works. Thanks.

[EisernSchild]

Update :

Class KinectCamera
v2010-12-10

* YUVU -> ARGB support
* vertical sync for all video modes
* raw modes provide IntPtr to raw data

http://dl.dropbox.com/u/13924535/OpenKinectCamera_v20101210.zip
http://dl.dropbox.com/u/13924535/OpenKinectCamera_v20101210_Release.zip

cheers

On 8 Dez., 13:46, EisernSchild <eisernsch...@gmail.com> wrote:
> Hi likeBVH !
>
> IR now working ! Was quite a simple Bug (ARRG) : Did forget to clear
> the line
>
> video->pkts_per_frame = VIDEO_PKTS_PER_FRAME;
>
> So it always did set it back to the Bayer Packets count...;-)
>
> will include th convert_uyvy_to_rgb() these days !
>

> now updated:http://dl.dropbox.com/u/13924535/OpenKinectCamera_v20201205_Release.ziphttp://dl.dropbox.com/u/13924535/OpenKinectCamera_v20101205.ziphttp://dl.dropbox.com/u/13924535/KinectDemo.zip

[EisernSchild]

Hi Betti !

Still seems to me a driver problem.

[dyadica]

Hi, Just want to say that I think your work is brill!

Can I ask if the current release supports/implements the get_depth()
function. Additionally do you have any suggestion on how I would
implement depth thresholding so that I can eradicate data over a
certain distance?

For your ref all releases have been working fine via Windows7 32 on a
bootcamp install.

Once again great stuff :)

[EisernSchild]

Hi dyadica !

I did throw out the get_depth() function, because per pixel scan was
very slow.

The (c++) code of this function was :

double depth = 0;
int raw_d = depth_frame[y*640+x];
if (raw_d < 2047)
{
depth = 1.0 / (raw_d * -0.0030711016 + 3.3309495161);
}
return depth;

Found this formular somewhere in this forum.

For depth thresholding i use the AForge.NET classes.

The "AForge.Imaging.Filters.Threshold" Filter just works with 8b/16b
grayscale bitmaps, so i use the
"AForge.Imaging.Filters.ChannelFiltering" Filter. (You could also use
the "ColorFiltering" Filter )

In this Filter you can set color channels' ranges to keep, so it shoud
be easy to use with the color gradient depth image ( set
"Camera.freenect_set_depth_format((freenect_depth_format)5);" )

Here is a demo i use it on grayscale depth image :

http://dl.dropbox.com/u/13924535/KinectAForgeDemo1.zip

cheers

[dyadica]

Hi EisernSchild,

Once again brill! I have been weighing up implementing a potential
solution in either AForge/EmguCV however your demo has convinced me to
look a bit further into AForge as you have filled in a few blanks.

Thanks for your help and a great starting point. Ill feed back some
info/code as things develop :)

Cheers

> > Once again great stuff :)- Hide quoted text -
>
> - Show quoted text -

[GUNNM]

Hello,

What the way to use get_depth(x, y); ? It always returned '0.0' in
your demo.

Another question, what the way to use it as a depth value? I mean, as
it's a 11 bit data, and we surely need a float. Is a simple cast
simply work? Something like that :
vertex.Z = (float) camera.get_depth(x, y);

On 3 déc, 20:51, EisernSchild <eisernsch...@gmail.com> wrote:

> Due to the positive Echo to my first try to build a managed class from
> the code here is the new version (v20101203)
>
> note that You need the file "libusb0.dll" in the "C:\Windows\System32"
> folder to run the dll.

> It is Part of the libusb-win32 USB library.http://sourceforge.net/projects/libusb-win32/
>

> official OpenKinect drivers fromhttps://github.com/OpenKinect/libfreenect/tree/master/platform/windows/

>
> Class KinectCamera
> v2010-12-03
>
> NEW
> * as close to OpenKinect repo code as possible to make any further
>   updates easier
> * provides managed Bitmap class images for both Video (ARGB) and Depth

>   (easy to use in AForge.NET C# Framework...http://www.aforgenet.com/)

> * logging
>
> TODO
> * directX and XNA Texture direct Bayer->ARGB convert
> * depth grayscale/color export
> * set ARGB alpha channel
> * motor/led
> * camera does not turn off ??
>

> C# Demo: (containing a built of the dll ("OpenKinectCamera.dll"))http://dl.dropbox.com/u/13924535/KinectDemo.zip

[EisernSchild]

Hi GUNNM !

Due to the problem that float maths is very slow i did throw the
get_depth() out of the class.

Instead, it now provides an IntPtr to the raw data in raw depth mode.

But, as you wish, i can also include the get_depth(x,y) again....

http://dl.dropbox.com/u/13924535/OpenKinectCamera_v20101210_Release.zip
http://dl.dropbox.com/u/13924535/OpenKinectCamera_v20101210.zip

Updated, but you need to be in FREENECT_DEPTH_11BIT mode to make it
work...
"Camera.freenect_set_depth_format((freenect_depth_format)0);"

salut

[EisernSchild]

Hi dyadica !

Great, looking forward to your code !

I m still at experimental state with AForge... the motion detector
seems very slow...=( But the image filters look good.

salut

[GUNNM]

Ok, thank you.
You're talking about : public static IntPtr depthData; I suppose ?
But how to use it? This way ( ? ) int z = depthData(x *640 + y)

On 12 déc, 14:43, EisernSchild <eisernsch...@gmail.com> wrote:
> Hi GUNNM !
>
> Due to the problem that float maths is very slow i did throw the
> get_depth() out of the class.
>
> Instead, it now provides an IntPtr to the raw data in raw depth mode.
>
> But, as you wish, i can also include the get_depth(x,y) again....
>

> http://dl.dropbox.com/u/13924535/OpenKinectCamera_v20101210_Release.ziphttp://dl.dropbox.com/u/13924535/OpenKinectCamera_v20101210.zip

[GUNNM]

This new version refuses to run with the runtime 3.5.
Even with the framework 4, I get the error : this mixed mode assembly
was created with 'v1.1.4322' version runtime and can not be loaded
into the runtime 4.0 '
Moreover I have no more depthData...
So I can't make it work.
Could you build it with the framework 3.5 or 4.0, please?

[Joshua Blake]

GUNNM:

 

Add an app.config to your project (if you don't have one) and make it look like this:

 

<configuration>
  <startup useLegacyV2RuntimeActivationPolicy="true">
    <supportedRuntime version="v4.0"/>
  </startup>
</configuration>

Josh

[GUNNM]

Thx Joshua. I changed the runtime in the project properties but this
did not appear in the app.config file. With your help it's fixed.

Who knows how to retreive the depth value in x and y from the IntPtr
depthData? I don't know how to use the IntPtr. (sorry)

[Mikkel Staunsholm]

Not a specific answer, but if you have the option, then use unsafe code (= use pointers in c#). Here is an example:

http://www.codeguru.com/Csharp/Csharp/cs_misc/graphicsandimages/article.php/c4259

Mkkl.

[Jeremiah Morrill]

Here's a bit of code I just wrote off the top of my head (didn't check).  I wouldn't expect to get great performance from this snippet either, but should be enough to get an understanding.  This also assumes 32bit process and 24bit RGB layout.

For speed of processing the entire image, I'd go with either native code, or pointers in C# as Mikkel suggested.

int x = 10;

int y = 10;

int bytesPerPixel = 3;

IntPtr pImageBuff = <imageBuffer_IntPtr>

stride = imageWidth * bitsPerPixel;

int r, g, b;

int position = new IntPtr(pImageBuff.ToInt32() + (y  * stride) + (x * bytesPerPixel));

r = Marshal.ReadByte(new IntPtr(position + 0);

g = Marshal.ReadByte(new IntPtr(position + 1);

b = Marshal.ReadByte(new IntPtr(position + 2);

Color c = new Color(r,g,b);

--
Microsoft MVP - Client Application Development

HJT, Inc Software Developer

[EisernSchild]

Hi Jeremiah !

Perfekt code snip ! Only thing to change is "bytesPerPixel" to 4 ( due
to ARGB format ).

And ARGB is in case BGRA, so A is position 3 and B is position 0.

On 12 Dez., 22:48, Jeremiah Morrill <jeremiah.morr...@gmail.com>
wrote:

> Here's a bit of code I just wrote off the top of my head (didn't check).  I
> wouldn't expect to get great performance from this snippet either, but
> should be enough to get an understanding.  This also assumes 32bit process
> and 24bit RGB layout.
>
> For speed of processing the entire image, I'd go with either native code, or
> pointers in C# as Mikkel suggested.
>
> int x = 10;
> int y = 10;
> int bytesPerPixel = 3;
>
> IntPtr pImageBuff = <imageBuffer_IntPtr>
>
> stride = imageWidth * bitsPerPixel;
>
> int r, g, b;
>
> int position = new IntPtr(pImageBuff.ToInt32() + (y  * stride) + (x *
> bytesPerPixel));
>
> r = Marshal.ReadByte(new IntPtr(position + 0);
> g = Marshal.ReadByte(new IntPtr(position + 1);
> b = Marshal.ReadByte(new IntPtr(position + 2);
>
> Color c = new Color(r,g,b);
>

> On Sun, Dec 12, 2010 at 1:14 PM, Mikkel Staunsholm <staunsh...@gmail.com>wrote:
>
>
>
>
>
>
>
>
>
> > Not a specific answer, but if you have the option, then use unsafe code (=
> > use pointers in c#). Here is an example:
>

> >http://www.codeguru.com/Csharp/Csharp/cs_misc/graphicsandimages/artic...
>
> > Mkkl.

[Jeremiah Morrill]

Thanks!  I knew there'd be some sort of type-o in there!  :)

[EisernSchild]

@GUNNM:

The get_depth now should work.

To get the depth value from IntPtr depthData use code similar to
Jeremiah´s here.

The pointer just points to the raw data if you are in depth 11bit
mode. You can also lock the depth Image, its the same data.

It is unsigned integer 16 bit, so you need to use Marshal.ReadInt16
Method instead of ReadByte.

The UInt you get is a depth information that you can use in the
formular

depth = 1.0 / (raw_d * -0.0030711016 + 3.3309495161);

to get a double depth out of it.

[EisernSchild]

Oh, and to get the correct pointer you also have to call
update_depth_image().

case FREENECT_DEPTH_11BIT_PACKED:
case FREENECT_DEPTH_10BIT_PACKED:
depthData = (IntPtr)depth->raw_buf;
break;

[GUNNM]

Ok, Thank for your help.
I can't test because I'm at work, but does this code sounds good?

//**************************************************************
int bytesPerPixel = 11;

IntPtr pdepth = <depthData>

stride = imageWidth * bitsPerPixel;

int position = new IntPtr(pdepth.ToInt32() + (y * stride) + (x *
bytesPerPixel));

double zz = Marshal.ReadByte(new IntPtr(position));

vertex.X = x;
vertex.Y = y;
vertex.z = zz; // or the next line ? (cause i don't understand your
explanation, EisernSchild)
// vertex.z = 1.0 / (zz * -0.0030711016 + 3.3309495161);

//**************************************************************

[GUNNM]

BitmapData bitmapData = KinectCamera.depthImage.LockBits(new
Rectangle(0, 0, 640, 480), ImageLockMode.ReadOnly,
PixelFormat.Format16bppArgb1555);
I've got an error : "bitmap is already locked "

And I first try this :

unsafe
{

int bytesPerPixel = 11;
IntPtr pdepth = KinectCamera.depthData ;
int stride = 640 * bytesPerPixel;

int id = 0;
for (int y = 0; y < 480 - 1; y++)
{
for (int x = 0; x < 640 - 1; x++)
{

IntPtr position = new IntPtr(pdepth.ToInt32() + (y *

stride) + (x * bytesPerPixel));

float zz = Marshal.ReadByte(new
IntPtr(position.ToInt32()));

Vector3 point1;
point1.X = (float)x;
point1.Y = (float)y;
point1.Z = zz;
}
}
}
But I've got the error : "try to attempt to a protected memory"

Any help will be appreciated. :D

On 12 déc, 23:35, EisernSchild <eisernsch...@gmail.com> wrote:

[EisernSchild]

I think there was a mistake...

int position = pdepth.ToInt32() + (y * stride) + (x * bytesPerPixel);

should be ok.

Then

short zz = Marshal.ReadInt16(new IntPtr(position));

should do it. At the end comes :

vertex.z = 1.0 / (float(zz) * -0.0030711016 + 3.3309495161);

And dont forget to use FREENECT_DEPTH_11BIT.
("Camera.freenect_set_depth_format((freenect_depth_format)0);")

[GUNNM]

Thank you,
I try with Camera.freenect_set_depth_format((freenect_depth_format)0);
And the same code as in the demo.
But It doesn't run better : "try to attempt to a protected memory"
unsafe
{
int bytesPerPixel = 11;

IntPtr pdepth = KinectCamera.depthData ;
int stride = 640 * bytesPerPixel;

int id = 0;
for (int y = 0; y < 480 - 1; y++)
{
for (int x = 0; x < 640 - 1; x++)
{

IntPtr position = new IntPtr(pdepth.ToInt32() + (y *

stride) + (x * bytesPerPixel));

short zz = Marshal.ReadInt16(position);

Vector3 point1;
point1.X = (float)x;
point1.Y = (float)y;

point1.Z = (float)(1.0 / ((float)zz * -0.0030711016 +
3.3309495161));

[GUNNM]

Here is my code : http://gunnm.is.free.fr/KinectD3D.zip
D3D is working but without depth for now.
Any idea?

On 13 déc, 18:09, EisernSchild <eisernsch...@gmail.com> wrote: