Render to Bitmap via C#

[dyadica]

Hi apologies for a basic level question but could someone point me in
the right direction for/provide a breakdown of the both the
VideoCamera.DataReceivedEventArgs and
DepthCamera.DataReceivedEventArgs

I am trying to use them to render two bitmaps within a windows form
using the C# wrapper.

To date I have tried to use the following to achieve this via using
both Image.Data byte array and the Image.DataPointer IntPtr in
numerous guises but to no avail? With a little more info I recon I’d
be able to track down the right procedure etc so far I have tried
things like:

http://msdn.microsoft.com/en-us/library/zy1a2d14.aspx
http://social.msdn.microsoft.com/forums/en-US/winforms/thread/e57f7731-c703-4c17-b1a2-32b155f9b745/
http://forums.create.msdn.com/forums/p/48040/287684.aspx

However without knowing what the point/data represents I am tree
barking at the moment. Additionally am I correct in assuming that
these events are the right place to trigger a frame update equivalent
etc.

Cheers d

[dyadica]

Ok I have had some success via the following:

private void HandleVideoDataReceived(object sender,
VideoCamera.DataReceivedEventArgs e)
{
IntPtr handle = e.Image.DataPointer;

int w = e.Image.Width;
int h = e.Image.Height;

System.Drawing.Imaging.PixelFormat format
= System.Drawing.Imaging.PixelFormat.Format24bppRgb;

Bitmap b = new Bitmap(w, h, (640 * 3 + 3) & ~3, format,
handle);
pictureBox1.Image = b;
}

My issue seems to be with both my stride and pixel format values/
combinations, the above outputs the image however with the wrong
colors. Can anybody provide me with a breakedown of stride and format
values or where I can track them down?

cheers d

[dyadica]

Me again from browsing some of the posts I have now the following
thought: could the color issue be due to the ARGB structure whilst
Windows requires ( B-G-R-A )?

[dyadica]

Ok whats wrong with this?

private void HandleVideoDataReceived(object sender,
VideoCamera.DataReceivedEventArgs e)
{

int w = e.Image.Width;
int h = e.Image.Height;

IntPtr handle = e.Image.DataPointer;

System.Drawing.Imaging.PixelFormat format
= System.Drawing.Imaging.PixelFormat.Format32bppArgb;

int bytesPerPixel = 0;

// calculate bytes per pixel
switch (format)
{
case PixelFormat.Format8bppIndexed:
bytesPerPixel = 1;
break;
case PixelFormat.Format16bppGrayScale:
bytesPerPixel = 2;
break;
case PixelFormat.Format24bppRgb:
bytesPerPixel = 3;
break;
case PixelFormat.Format32bppRgb:
case PixelFormat.Format32bppArgb:
bytesPerPixel = 4;
break;
case PixelFormat.Format48bppRgb:
bytesPerPixel = 6;
break;
case PixelFormat.Format64bppArgb:
bytesPerPixel = 8;
break;
}

// calculate stride
int stride = w * bytesPerPixel;

if (stride % 4 != 0)
{
stride += (4 - (stride % 4));
}

// int stride = 640 * 4; // (640 * 3 + 3) & ~3;

Bitmap b = new Bitmap(w, h, stride, format, handle);

pictureBox1.Image = b;
}

[GUNNM]

you helped me thank you, and you're right about the format, test
this :

int w = e.Image.Width;
int h = e.Image.Height;

byte[] pixels = new byte[3 * w * h];

System.Runtime.InteropServices.Marshal.Copy(e.Image.DataPointer,
pixels, 0, pixels.Length);
var bmp = new Bitmap(w, h,
System.Drawing.Imaging.PixelFormat.Format32bppArgb);

for (int i = 0; i < h; i++)
{
for (int j = 0; j < w; j++)
{
int p = 3 * (w * i + j);
bmp.SetPixel(j, i, Color.FromArgb(255, pixels[p +
0], pixels[p + 1], pixels[p + 2]));
}
}
pictureBox1.Image = bmp;

[GUNNM]

This format is really a problem, because it's much slower than:
pictureBox1.Image = new Bitmap(w, h,
w*3,System.Drawing.Imaging.PixelFormat.Format24bppRgb,
e.Image.DataPointer);
And it's too much slower for use as a texture.
Any idea ?

[Jeremiah Morrill]

GDI will render much faster if you use the same color space as your display settings.  Also creating a new "Bitmap" instance for each frame is a very heavy operation.  Try creating one bitmap, and just update the pixels for each frame.  This can be done by doing a:  bmpData = bmp.LockBits(...).  The bmpData will have a property called Scan0, which is a IntPtr.  Use something like this to copy your native pixels pointer from kinect to the bitmap:

[DllImport("kernel32.dll", EntryPoint = "RtlMoveMemory")]
private static extern void CopyMemory(IntPtr Destination, IntPtr Source, int Length);

You may have to Invalidate() your picturebox after the update to get it to show.

On a side note, you should give WPF a try and use the WriteableBitmap class and just copy your native bytes to WriteableBitmap.BackBuffer using the above p/invoke.  Take note on the Lock/unlock methods.

-Jer

--
Microsoft MVP - Client Application Development

HJT, Inc Software Developer

[dyadica]

Hi thanks for your replies however I still have loads of confusion due
to both a lack of basic knowledge and information on my part.

>And it's too much slower for use as a texture.
Any idea ?

Not a clue at the moment I'm afraid. I'm going round in circles! I
can’t help but feel that a bitmap/image addition to the wrapper/lib
(making it more novice/windows/.net friendly like OpenKinectCamera)
needs someone who knew what they were doing :( alas I’m obviously not
that man but I'll keep trying :)

Yup that was a blatant beg/plee on my part :) any takers lol.

>GDI will render much faster if you use the same colour space as your display settings.

Do you mean by this use PixelFormat.Format32bppArgb? Is it because the
example is using Format24bppRgb that I am getting the inverted colour
values? If so this is what I want to do however cannot calculate the
correct stride value in order to do so.

I assume that is should be as simple as 640*4 however in practice this
seems not to be the case? In order to try and gain a little insight/
verification I have been going through Eisern Schild’s
OpenKinectCamera code in an attempt to identify what settings/he does
in order to facilitate bitmap creation. As far as I can tell my
assumptions of Format32bppArgb with a stride of 640*4 are correct,
furthering my confusion?

What am I missing, besides a basic lack of any idea? That I know :)

I question as to whether the alpha value is not included within the
stream as I seem to get best picture clarity when using Rgb formats
with a stride of 640*3 however as indicated the stride is never right?
Also dotted around Eisern’s code I see mention of alphaVideo a few
times. If my assumption is correct do I need to perform some form of
packing/addition??

> Also creating a new "Bitmap" instance for each frame is a very heavy operation. Try creating one bitmap, and just update the pixels for each frame.

I agree and this is the overall plan however there is no point if I
cannot get the bitmap to render properly. I have already been down the
lockbits route however from what I can tell unless I am missing
something is that without the stride etc this will produce the same
results? Or do you mean perform a data sort here as means to rearrange
the order in ref to the 24Rgb example code?

> On a side note, you should give WPF a try and use the WriteableBitmap class and just copy your native bytes to WriteableBitmap.BackBuffer <http://msdn.microsoft.com/en-us/library/system.windows.media.imaging....> using the above p/invoke. Take note on the Lock/unlock methods.

Before I have a go could you verify that this works out the box as
suggested? I am begrudged to make the change to WPF as I would like to
use/integrate the hardware with a large back catalogue of works and
this could mean a massive upgrade :( however I am going to have to
take the plunge sometime :) and I guess this would be a good excuse.

[EisernSchild]

Hi dyadica !

Did not have the time to take a look at the new wrapper, will do so
soon.

In my class i used the Format32bppArgb with a stride of 640*4 bytes. I
mapped it directly to ARGB. This is the only difference to the
original freenect code that maps it to RGB. Note that the ARGB format
is in case BGRA ( due to 32 bit uint ). In case of using this format,
you can set the alpha channel by using alphaVideo variable. (for depth
image there is also a alphaDepth variable) This is useful for example
in XNA to get a seethrough image.

[dyadica]

Hi EisernSchild,

> Did not have the time to take a look at the new wrapper, will do so soon.

Please, please do I’m a lot balder than yesterday :)

>In my class I used the Format32bppArgb with a stride of 640*4 bytes. I mapped it directly to ARGB. This is the only difference to the original freenect code that maps it to RGB.

>Note that the ARGB format is in case BGRA ( due to 32 bit uint ). In case of using this format, you can set the alpha channel by using alphaVideo variable.

I thought this to be the case, and thus the cause of some of my
confusion. Is this why the RGB formats work better than the others?

Is the direct mapping done via the convert_bayer_to_rgb function? If
so this could be my missing link and I have something to work with :)

I’m thinking of reverting to your class as currently I’m not managing
to get anything done with the wrapper as is. I really think that the
current implementation would be greatly complimented with inclusion of
your conversion functionality.

Cheers d

[GUNNM]

Hi,
Here is two functions, one for kinect video, and one for the IR
video :
It is not too slow, because I followed the advices of Jeremiah, but it
can be better. Tell me if you improve them.

private Bitmap tempBitmap = new Bitmap(640,480);
private Bitmap IRBitmap = new Bitmap(640, 488);
private BitmapData BMPData;

private Bitmap
RGBptrToBitmap(VideoCamera.DataReceivedEventArgs e)

{
int w = e.Image.Width;
int h = e.Image.Height;

unsafe
{
int pixelSize = 3;

BMPData = tempBitmap.LockBits(new Rectangle(0, 0, w,
h), ImageLockMode.ReadWrite, PixelFormat.Format24bppRgb);

CopyMemory(BMPData.Scan0, e.Image.DataPointer,
pixelSize * h * w);

for (int y = 0; y < BMPData.Height; y++)
{
byte* row = (byte*)BMPData.Scan0 + (y *
BMPData.Stride);

for (int x = 0; x < BMPData.Width; x++)
{
byte tmp = row[x * pixelSize];
row[x * pixelSize] = row[x * pixelSize + 2];
row[x * pixelSize + 2] = tmp;
}
}
tempBitmap.UnlockBits(BMPData);
}
return tempBitmap;
}


Bitmap irbmp = new Bitmap(640, 488);
private BitmapData IRData;
private Bitmap IRptrToBitmap(VideoCamera.DataReceivedEventArgs

e)
{
int w = e.Image.Width;
int h = e.Image.Height;

unsafe
{
int pixelSize = 1;

BMPData = IRBitmap.LockBits(new Rectangle(0, 0, w, h),
ImageLockMode.ReadWrite, PixelFormat.Format8bppIndexed);

CopyMemory(BMPData.Scan0, e.Image.DataPointer,
pixelSize * h * w);

IRData = irbmp.LockBits(new Rectangle(0, 0, w, h),
ImageLockMode.ReadWrite, PixelFormat.Format24bppRgb);

for (int y = 0; y < BMPData.Height; y++)
{
byte* row = (byte*)BMPData.Scan0 + (y *
BMPData.Stride);
byte* rowOut = (byte*)IRData.Scan0 + (y *
IRData.Stride);

for (int x = 0; x < BMPData.Width; x++)
{
rowOut[x * 3] = row[x];
rowOut[x * 3 + 1] = row[x];
rowOut[x * 3 + 2] = row[x];
}
}

IRBitmap.UnlockBits(BMPData);
irbmp.UnlockBits(IRData);
}
return irbmp;

}


[DllImport("kernel32.dll", EntryPoint = "RtlMoveMemory")]
private static extern void CopyMemory(IntPtr Destination,
IntPtr Source, int Length);

[Jeremiah Morrill]

For WPF, here are some examples I have for efficient pixel buffer updates in WPF that are part of one of my open source projects:

For using InteropBitmap, which is the fastest use this:  http://silverlightviewport.codeplex.com/SourceControl/changeset/view/39341#1534506

Please note that its hardcoded for BGRA32 color space as WPF 3.51 only supported this color space with InteropBitmap.  WPF 4.0 supports other color spaces, but it will have to be modified accordingly.

For WriteableBitmap check out this sample:  http://silverlightviewport.codeplex.com/SourceControl/changeset/view/39341#1534512

WriteableBitmap supports more colors spaces, but in this example its hardcoded to BGRA32.  

Also take notice of "RasterRendererElement" superclass here:  http://silverlightviewport.codeplex.com/SourceControl/changeset/view/39341#1534508

Hope this helps,

-Jer

[EisernSchild]

Hi dyadica !

> Is this why the RGB formats work better than the others?

What do you mean exactly ?

> Is the direct mapping done via the convert_bayer_to_rgb function?

Yes, its this function. I took it from the official repo, modified it
from RGB to BGRA.

I will definately take a look at the wrapper on weekend and update my
class.

cheers

[GUNNM]

What is the best way to have a float from the depth pointer ?
Currently, I do like this:
Int16* ptrFront = (Int16*)this.depthHandleFront.AddrOfPinnedObject();
(...) // for i=0 to 640*480
float alt = Convert.ToSingle(ptrFront[i]);
But I feel that it lack of precision, and speed.

EisernSchild > Could you share your modified function (RGB to BGRA),
please?

[Kai Ritterbusch]

dont declare the variable inside the loop.

declare it outside the for loop and overwrite it:

float alt = 0;

Int16* ptrFront = (Int16*)this.depthHandleFront.AddrOfPinnedObject();
(...) // for i=0 to 640*480

alt = Convert.ToSingle(ptrFront[i]);

[GUNNM]

Thank you for your answer.
Yes, that's what I'm doing in my code, I filled an array of float.
I wrote this as an example.
It is therefore the best way to get a float? Isn't there a better
method, using a pointer in bytes, for a method with a logical &?

[GUNNM]

I made a mistake, I wanted to write "Isn't there a better method,
using a pointer in bytes, or a method with a logical & ? " (for a
correct bitwise value)

But I have now doubt about type of the pointed value, is it a float or
an int? Maybe I don't have to convert it to float?


On 3 jan, 12:48, GUNNM <steven.moy...@laposte.net> wrote:
> (...)

> It is therefore the best way to get a float? Isn't there a better
> method, using a pointer in bytes, for a method with a logical &?

> (...)

[Kai Ritterbusch]

Hi again,

I dont work with the c# wrapper, but the depth stream should contain
11bit values stored in a 16bit int buffer.

to calculate a distance in meters from it, you need a conversion
function like the one shown here:

http://nicolas.burrus.name/index.php/Research/KinectCalibration
"Transformation of raw depth values into meters"

what exactly do you want to do?

[Bartek P]

Kai,
To be clear - it doesn't matter if you declare the variable inside or
outside the loop.