How to marshal complex data structures

[James Whetstone]

Hi,

I'm using Pinvoke and C# interop to interface to a C DLL. The DLL function
signiture is as follows:

int fun(char *name, my_id_t *id, my_buffer_t *buffer);

The user defined types (in C) are as follows:

struct my_id_t
{
unsigned char guid[16];
}

struct my_subbuffer_t
{
int format;
}

struct my_buffer_t
{
int count;
my_subbuffer_t **bufferArray;
}

So I'm having trouble with the 3rd argument---my_buffer_t.

This is what I've got so far in C#:

[StructLayout(LayoutKind.Sequential)]
public class my_id_t
{
[MarshalAs(UnManagedType.ByValArray, SizeConst=16)]
public byte[] myGuid;
}

[StructLayout(LayoutKind.Sequential)]
public class my_subbuffer_t
{
public int format;
}

[StructLayout(LayoutKind.Sequential)]
public class my_buffer_t
{
public int count;
my_subbuffer_t [] buffers;
}

[James Whetstone]

Oops, forgot to add the pinvoke signiture:

[DllImport("mylib.dll")]

public static extern int fun(string name, my_id_t id, my_buffer_t buffer);

So when I run my code, the buffer is corrupted/invalid memory.

Can you guys help out with some code or point my in the right direction?

Thanks!
JW

"James Whetstone" <jameswh...@comcast.net> wrote in message
news:t7WdnSN0HqWVMlvY...@comcast.com...

[TDC]

I think the fact that you are using formatted classes instead of
structs in your code require you to flag the parameter with the
InAttribute and OutAttribute ( [In, Out] ). Otherwise it would
default to In only.

On Feb 4, 11:18 pm, "James Whetstone" <jameswhetst...@comcast.net>
wrote:

> Oops, forgot to add the pinvoke signiture:
>
> [DllImport("mylib.dll")]
>
> public static extern int fun(string name, my_id_t id, my_buffer_t buffer);
>
> So when I run my code, the buffer is corrupted/invalid memory.
>
> Can you guys help out with some code or point my in the right direction?
>
> Thanks!
> JW
>

> "James Whetstone" <jameswhetst...@comcast.net> wrote in message

> > }- Hide quoted text -
>
> - Show quoted text -

[James Whetstone]

Thanks for the suggestion. I just tried it by adding

[In, Out] to the parameter on the Pinvoke siginiture. So it looked like
this:

public static extern int fun(string name, my_id_t id, [In,Out]my_buffer_t
buffer);

I also changed the class to a struct and changed the pinvoke signiture to
this:

public static extern int fun(string name, my_id_t id, [In,Out] ref
my_buffer_t buffer);

But that didn't work either. Any other thoughts or comments?

Cheers,
JW

"TDC" <NOtcar...@lycos.com> wrote in message
news:1170685614.3...@h3g2000cwc.googlegroups.com...

[Michael Phillips, Jr.]

> public static extern int fun(string name, my_id_t id, [In,Out] ref
> my_buffer_t buffer);

The above should work. How did you allocate memory for buffer?

Did you allocate memory for my_buffer_t and the array my_subbuffer_t?

If so, you may want to try to lay out your structs explicitely rather than
sequentially to avoid implicit packing alignment issues of struct members.

[James Whetstone]

So I'm allocating memory for my_buffer_t like this:

my_buffer_t buf = new my_buffer_t();
buf.count = 1;

buf.buffers = new my_subbuffer_t[1];

buf.buffers[0].format = 100;

Does this look right? I don't need to allocate on the unmanaged heap,
right?

I'll try using an explicit layout. Thanks for the suggestions.

JW

"Michael Phillips, Jr." <mphil...@nospam.jun0.c0m> wrote in message
news:%23W65BDX...@TK2MSFTNGP03.phx.gbl...

[James Whetstone]

Okay,

So this is what I've got implemented now:

[StructLayout(LayoutKind.Sequential)]
public class my_subbuffer_t
{
public int format;
}

[StructLayout(LayoutKind.Sequential)]
public class my_buffer_t
{
public int count;

public my_subbuffer_t [] buffers;
}

[DllImport("mylib.dll")]

public static extern int fun(string name, my_id_t id, [In, Out] ref
my_buffer_t buffer);

my_buffer_t buf;

buf.buffers = new my_subbuffers_t[1];

buf.buffers[0].format = 100;

buf.count=1;

string name="Jimbo";
my_id_t id = new my_id_t();

fun(name,id,ref buf);

So now I get the following runtime error:
Invalid managed/unmanaged type combination (this value type must be paired
with Struct).

Best,
JW

"Michael Phillips, Jr." <mphil...@nospam.jun0.c0m> wrote in message
news:%23W65BDX...@TK2MSFTNGP03.phx.gbl...

[James Whetstone]

Sorry,

that wasn't quite right...I've changed the "classes" to "structs" like so:

[StructLayout(LayoutKind.Sequential)]
public struct my_subbuffer_t

{
public int format;
}
[StructLayout(LayoutKind.Sequential)]
public class my_buffer_t
{
public int count;
public my_subbuffer_t [] buffers;
}

...

Thanks for takin' the time,
JW

"James Whetstone" <jameswh...@comcast.net> wrote in message

news:TqudnZ9izd2ouVXY...@comcast.com...

[James Whetstone]

So to clarify, all my classes are now structs.
Thanks,
JW

"James Whetstone" <jameswh...@comcast.net> wrote in message

news:ZLSdnc1YYIQmuVXY...@comcast.com...

[Michael Phillips, Jr.]

Do you now get the expected result(i.e., no interop errors)?

[James Whetstone]

No. I'm getting the following runtime error:

Invalid managed/unmanaged type combination (this value type must be paired
with Struct).

"Michael Phillips, Jr." <mphil...@nospam.jun0.c0m> wrote in message

news:eNSpx5fS...@TK2MSFTNGP05.phx.gbl...

[Michael Phillips, Jr.]

Try using an explicit layout for the struct declarations, as follows:

[StructLayout(LayoutKind.Explicit)]
public struct my_subbuffer_t
{
[FieldOffset(0)] public int format;
}

[StructLayout(LayoutKind.Explicit)]
public struct my_buffer_t
{
[FieldOffset(0)] public int count;
[FieldOffset(4)] public my_subbuffer_t [] buffers;
}

public static extern int fun(string name, my_id_t id, [In,Out] ref
my_buffer_t buffer);

// allocating memory for my_buffer_t:

my_buffer_t buf = new my_buffer_t();
buf.count = 1;

buf.buffers = new my_subbuffer_t[1];

buf.buffers[0].format = 100;

// execute interop
fun( name, id, ref buf);

"James Whetstone" <jameswh...@comcast.net> wrote in message

news:uNednZmAj8sXLVXY...@comcast.com...

[James Whetstone]

Hi,

I tried this and I got a new exception as follows:

An unhandled exception of type 'System.ArgumentException' occurred in
Arm4InteropTest.exe

Additional information: The parameter is incorrect. (Exception from HRESULT:
0x80070057 (E_INVALIDARG))

Thanks :-) for your suggestions.

JW

"Michael Phillips, Jr." <mphil...@nospam.jun0.c0m> wrote in message

news:%23vfrEDh...@TK2MSFTNGP06.phx.gbl...

[James Whetstone]

So the "C" DLL signiture is slightly different than I first thought: a
couple function arguments are "const", including the one I'm having problems
with:

int fun(const char *name, my_id_t *id, const my_buffer_t *buffer);

Does this effect the results?

Cheers,
JW

"Michael Phillips, Jr." <mphil...@nospam.jun0.c0m> wrote in message

news:%23vfrEDh...@TK2MSFTNGP06.phx.gbl...

[James Whetstone]

So to help figure this out, I''m posting some test code for a "C" DLL. Here
the code:

//*** Start "C" DLL here

struct my_subbuffer_t
{
int format;
};

struct my_buffer_t
{
int count;
my_subbuffer_t **buffers;
};

extern "C" __declspec( dllexport ) int __stdcall fun(my_buffer_t *buf)
{
for(int i=0;i < buf->count; ++i)
{
int format = buf->buffers[i]->format;
}
return 0;
}

extern "C" __declspec( dllexport ) int __stdcall fun2(my_subbuffer_t **bufs,
int size)
{

for(int i=0;i < size; ++i)
{
int format = bufs[i]->format;
}

return 0;
}

//****** end C DLL HERE ****/

So I compile this into a DLL into "MyDLL.dll" and then run my interop
console app which looks like this:

//*** Start C# interop console test app

using System;
using System.Collections.Generic;
using System.Text;
using System.Runtime.InteropServices;
namespace InteropTest
{

[StructLayout(LayoutKind.Explicit)]
public struct my_subbuffer_t
{
[FieldOffset(0)]
public int format;
}

[StructLayout(LayoutKind.Explicit)]
public struct my_buffer_t
{
[FieldOffset(0)]
public int count;
[FieldOffset(4)]

public my_subbuffer_t[] bufs;
}

class Program
{
[DllImport("MyDLL.dll")]
public static extern int fun([In, Out] ref my_buffer_t buf);

[DllImport("MyDLL.dll")]
public static extern int fun2([In, Out] ref my_subbuffer_t[] bufs,
int size);

static void Main(string[] args)
{

my_buffer_t buf = new my_buffer_t();
buf.count = 1;

my_subbuffer_t []bufs = new my_subbuffer_t[1];
buf.bufs = bufs;
buf.bufs[0].format = 1;

fun2(ref bufs, 1);
fun(ref buf);

}
}
}

//*** End C# interop console app

So the result are that the "fun2" function executes fine, but the "fun"
function throws the following exception:

An unhandled exception of type 'System.ArgumentException' occurred in

InteropTest.exe
Additional information: The parameter is incorrect. (Exception from HRESULT:
0x80070057 (E_INVALIDARG))

So that's it. I'm pretty stumped by this.

Thanks in advance for any help/comments.

JW

"James Whetstone" <jameswh...@comcast.net> wrote in message

news:iNednSPhuu0SzlTY...@comcast.com...

[Michael Phillips, Jr.]

When I stepped through the original code, I saw that the memory for the
my_subbuffer_t struct member of my_buffer_t was not marshaled.
Rather than write a custom marshaler to solve your problem, I have come up
with a solution that works.

It is certainly not the only solution and not as elegant as I would like but
here it is:

**** Start of mydll.c *****

// mydll.c : Defines the entry point for the DLL application.
//

#include <windows.h>
#include <stdlib.h>
#include <stdio.h>

#ifdef _MANAGED
#pragma managed(push, off)
#endif

struct my_subbuffer_t
{
int format;
};

struct my_buffer_t
{
int count;
struct my_subbuffer_t *buffers;
};

BOOL APIENTRY DllMain( HMODULE hModule,
DWORD ul_reason_for_call,
LPVOID lpReserved
)
{
return TRUE;
}

__declspec( dllexport ) int __stdcall fun( HGLOBAL hMemory )
{
int i = 0;
char msg[32];

struct my_buffer_t *buf = (struct my_buffer_t *)GlobalLock(hMemory);

if ( NULL != buf )
{
for ( ;i < buf->count; ++i )
{
int format = buf->buffers[i].format;

sprintf(msg,TEXT("fun:format = %d\n"), format);
printf(msg);
}

GlobalUnlock(hMemory);
}
else
printf(TEXT("NULL argument passed to fun\n"));

return 0;
}

__declspec( dllexport ) int __stdcall fun2( struct my_subbuffer_t *bufs, int
size )
{
int i = 0;
char msg[32];

if ( NULL != bufs )
{
for ( ;i < size; ++i )
{
int format = bufs[i].format;
sprintf(msg, TEXT("fun2:format = %d\n"), format);
printf(msg);
}
}
else
printf(TEXT("NULL argument passed to fun2\n"));

return 0;
}

#ifdef _MANAGED
#pragma managed(pop)
#endif

**** End of mydll.c *****

**** Start of Program.cs ****

using System;
using System.Collections.Generic;
using System.Text;
using System.Runtime.InteropServices;
namespace InteropTest
{
[StructLayout(LayoutKind.Explicit)]
public struct my_subbuffer_t
{
[FieldOffset(0)] public int format;
}

[StructLayout(LayoutKind.Explicit)]
public struct my_buffer_t
{
[FieldOffset(0)] public int count;

[FieldOffset(4)] public IntPtr bufs;
}

class Program
{
[DllImport("MyDLL.dll", CharSet = CharSet.Ansi, CallingConvention =
CallingConvention.StdCall)]
public static extern int fun( [In] IntPtr buf);

[DllImport("MyDLL.dll", CharSet=CharSet.Ansi, CallingConvention =
CallingConvention.StdCall)]

public static extern int fun2([In, Out] ref my_subbuffer_t bufs, int
size);

static void Main(string[] args)
{
my_buffer_t buf = new my_buffer_t();
buf.count = 1;

my_subbuffer_t[] subbuf = new my_subbuffer_t[1];
subbuf[0].format = 1;

IntPtr pBuf =
Marshal.AllocHGlobal( Marshal.SizeOf(buf) +
Marshal.SizeOf(typeof(my_subbuffer_t)) );
buf.bufs = Marshal.AllocHGlobal( Marshal.SizeOf(
typeof(my_subbuffer_t) ) );

// Marshal managed to unmanaged
Marshal.StructureToPtr( subbuf[0], buf.bufs, false );
Marshal.StructureToPtr(buf, pBuf, false);

// Sanity check
my_buffer_t test= (my_buffer_t)Marshal.PtrToStructure( pBuf,
typeof(my_buffer_t) );
my_subbuffer_t testsubbuffer =
(my_subbuffer_t)Marshal.PtrToStructure(buf.bufs, typeof(my_subbuffer_t));

try
{
fun2(ref subbuf[0], 1);
fun(pBuf);
}
catch (SystemException ex)
{
Console.WriteLine(ex.Message);
}
finally
{
Marshal.FreeHGlobal(buf.bufs);
Marshal.FreeHGlobal(pBuf);
}
}
}
}

**** End of Program.cs ****

[James Whetstone]

Nicely done!

I've learned a lot about interop from your code---thanks. I have a couple
more questions for you if you have time for it...

So I've altered your C# routine so that the array of my_subbuffer_t objects
now holds 2 objects as follows:

*** Start c# Main routine ***

static void Main(string[] args)
{
my_buffer_t buf = new my_buffer_t();

buf.count = 2;

my_subbuffer_t[] subbuf = new my_subbuffer_t[2];

subbuf[0].format = 1;

subbuf[1].format = 2;

//Allocate enough unmanaged memory for the my_buffer_t object
and the first my_subbuffer_t object in the "bufs" array
IntPtr pBuf = Marshal.AllocHGlobal(Marshal.SizeOf(buf) +
Marshal.SizeOf(typeof(my_subbuffer_t)));

//Allocate enough unmanaged memory to hold 2 my_subbuffer_t
objects
buf.bufs =
Marshal.AllocHGlobal(Marshal.SizeOf(typeof(my_subbuffer_t))*2);

// Marshal managed to unmanaged - the first array element
Marshal.StructureToPtr(subbuf[0], buf.bufs, false);

//Marshal managed to unmanaged - the second array element
Marshal.StructureToPtr(subbuf[1], new IntPtr(buf.bufs.ToInt32()
+ Marshal.SizeOf(typeof(my_subbuffer_t))), false);

Marshal.StructureToPtr(buf, pBuf, false);

// Sanity check
my_buffer_t test =
(my_buffer_t)Marshal.PtrToStructure(pBuf,typeof(my_buffer_t));
my_subbuffer_t testsubbuffer =
(my_subbuffer_t)Marshal.PtrToStructure(buf.bufs, typeof(my_subbuffer_t));

try
{
fun2(ref subbuf[0], 2);

fun(pBuf);
}
catch (SystemException ex)
{
Console.WriteLine(ex.Message);
}
finally
{
Marshal.FreeHGlobal(buf.bufs);
Marshal.FreeHGlobal(pBuf);
}
}

*** End C# Main routine ***

So is this code correct? I'm a bit thrown off by the fact that we're
allocating memory for 3 my_subbuffer_t objects---1 in the my_buffer_t object
and 2 for the bufs array---when we're actually only using 2. In other
words, I would have guessed that for the my_buffer_t object, we'd only have
to allocate enough memory for a my_buffer_t object which is an int and an
array reference (the size of a pointer, right?), not a my_buffer_t object
AND a my_subbuffer_t object. So that's my first question.

My second question is---just for the sake of being complete---what would be
a more elegant way to do this? A custom marshaler?

Best,
JW

"Michael Phillips, Jr." <mphil...@nospam.jun0.c0m> wrote in message

news:e027YpuS...@TK2MSFTNGP03.phx.gbl...

[Michael Phillips, Jr.]

> So is this code correct? I'm a bit thrown off by the fact that we're
> allocating memory for 3 my_subbuffer_t objects---1 in the my_buffer_t
> object and 2 for the bufs array---when we're actually only using 2. In
> other words, I would have guessed that for the my_buffer_t object, we'd
> only have to allocate enough memory for a my_buffer_t object which is an
> int and an array reference (the size of a pointer, right?), not a
> my_buffer_t object AND a my_subbuffer_t object. So that's my first
> question.

My memory calculation was not correct. Allocating memory for m_buffer_t
already included 1 my_subbuffer_t.

Sorry about that. Thanks for catching that error.

Making the following minor changes yields the expected result for 2
my_subbuffer_t structs:

... // previous code not shown

my_buffer_t buf = new my_buffer_t();
buf.count = 2;

my_subbuffer_t[] subbuf = new my_subbuffer_t[2];
subbuf[0].format = 1;
subbuf[1].format = 2;

IntPtr pBuf =
Marshal.AllocHGlobal( Marshal.SizeOf(buf) +
Marshal.SizeOf(typeof(my_subbuffer_t)) ); // 2 my_subbuffer_t's

Marshal.StructureToPtr(buf, pBuf, false);

buf.bufs = Marshal.AllocHGlobal( Marshal.SizeOf( typeof(my_subbuffer_t) ) *
2);

// Marshal managed to unmanaged
Marshal.StructureToPtr( subbuf[0], buf.bufs, false );

Marshal.StructureToPtr( subbuf[1], new IntPtr(buf.bufs.ToInt32() +

Marshal.SizeOf(typeof(my_subbuffer_t))), false);
Marshal.StructureToPtr( buf, pBuf, false );

// Sanity check

my_buffer_t test= (my_buffer_t)Marshal.PtrToStructure( pBuf,
typeof(my_buffer_t) );
my_subbuffer_t testsubbuffer =
(my_subbuffer_t)Marshal.PtrToStructure(buf.bufs, typeof(my_subbuffer_t));

try
{
fun2(ref subbuf[0], 2);
fun(pBuf);
}

... // additional code not shown

> My second question is---just for the sake of being complete---what would
> be a more elegant way to do this? A custom marshaler?
>

Yes. But why bother. Microsoft has already coded Marshalers for all the
native types and other important structs (e.g., VARIANTS).

The .NET SDK includes sample code for marshaling different types and custom
marshaling, if you are curious to see how it is done.

[James Whetstone]

Right. Regarding the new code, if I'm not mistaken, you meant to change
this line of code:

IntPtr pBuf = Marshal.AllocHGlobal( Marshal.SizeOf(buf) +
Marshal.SizeOf(typeof(my_subbuffer_t)) ); // 2 my_subbuffer_t's

to this:

//We don't need to allocate for any my_subbuffer_t objects here. We only
need to allocate for the my_buffer_t because
//it already includes space for the pointer to the array of my_subbuffer_t
objects.

IntPtr pBuf = Marshal.AllocHGlobal( Marshal.SizeOf(buf));

Regarding a custom marshaler---I agree with you---no need to reinvent the
wheel. Thanks for the info though.

JW

"Michael Phillips, Jr." <mphil...@nospam.jun0.c0m> wrote in message

news:%23iz%23zS5SH...@TK2MSFTNGP05.phx.gbl...

[Michael Phillips, Jr.]

In case anyone is interested, here is a complete solution that demonstrates
the elegance of using custom marshaling versus non custom marshaling of
complex data structures:

****** Start of mydll.c ******

// mydll.c : Defines the entry point for the DLL application.
//

#include <windows.h>
#include <stdlib.h>
#include <stdio.h>

#ifdef _MANAGED
#pragma managed(push, off)
#endif

struct my_subbuffer_t
{
int format;
};

struct my_buffer_t
{
int count;
struct my_subbuffer_t *buffers;
};

struct my_buffer_custom_t
{
int count;
struct my_subbuffer_t buffers[1];
};

BOOL APIENTRY DllMain( HMODULE hModule,
DWORD ul_reason_for_call,
LPVOID lpReserved
)
{
return TRUE;
}

// The marshaler always allocates fixed global memory.
// GlobalLock is not necessary
__declspec( dllexport ) int __stdcall fun( struct my_buffer_t *buf )

{
int i = 0;
char msg[32];

if ( NULL != buf )

{
for ( ;i < buf->count; ++i )
{
int format = buf->buffers[i].format;

sprintf(msg,TEXT("fun:format = %d\n"), format);
printf(msg);
}
}

else
printf(TEXT("NULL argument passed to fun\n"));

return 0;
}

__declspec( dllexport ) int __stdcall fun2( struct my_subbuffer_t *bufs, int
size )
{
int i = 0;
char msg[32];

if ( NULL != bufs )
{
for ( ;i < size; ++i )
{
int format = bufs[i].format;
sprintf(msg, TEXT("fun2:format = %d\n"), format);
printf(msg);
}
}
else
printf(TEXT("NULL argument passed to fun2\n"));

return 0;
}

__declspec( dllexport ) int __stdcall fun3( struct my_buffer_custom_t *buf )

{
int i = 0;
char msg[32];

if ( NULL != buf )

{
for ( ;i < buf->count; ++i )
{
int format = buf->buffers[i].format;

sprintf(msg,TEXT("fun3:format = %d\n"), format);
printf(msg);
}
}
else

printf(TEXT("NULL argument passed to fun\n"));

return 0;
}

#ifdef _MANAGED
#pragma managed(pop)
#endif

****** End of mydll.c ******

****** Start of Program.cs ******

using System;
using System.Collections.Generic;
using System.Text;
using System.Runtime.InteropServices;
namespace InteropTest
{
[StructLayout(LayoutKind.Explicit)]
public struct my_subbuffer_t
{
[FieldOffset(0)] public int format;
}

[StructLayout(LayoutKind.Explicit)]
public struct my_buffer_t
{
[FieldOffset(0)] public int count;
[FieldOffset(4)] public IntPtr bufs;
}

[StructLayout(LayoutKind.Explicit)]
public struct my_buffer_custom_t
{
[FieldOffset(0)] public int count;
[MarshalAs(UnmanagedType.ByValArray, ArraySubType =
UnmanagedType.Struct, SizeConst = 2)]
[FieldOffset(4)] public my_subbuffer_t[] bufs;
}

class Program
{
[DllImport("MyDLL.dll", CharSet = CharSet.Ansi, CallingConvention =
CallingConvention.StdCall)]
public static extern int fun( [In] IntPtr buf);

[DllImport("MyDLL.dll", CharSet=CharSet.Ansi, CallingConvention =
CallingConvention.StdCall)]
public static extern int fun2([In, Out] ref my_subbuffer_t bufs, int
size);

[DllImport("MyDLL.dll", CharSet = CharSet.Ansi, CallingConvention =
CallingConvention.StdCall)]

public static extern int fun3(
[MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef =
typeof(SampleMarshaler),MarshalCookie = "2"),In,Out] object buf );

static void Main(string[] args)
{

my_buffer_t buf = new my_buffer_t();
buf.count = 2;

my_subbuffer_t[] subbuf = new my_subbuffer_t[2];
subbuf[0].format = 1;
subbuf[1].format = 2;

// allocate memory for the my_buffer_t struct and a pointer to
an array of my_subbuffer_t structs
IntPtr pBuf = Marshal.AllocHGlobal( Marshal.SizeOf(buf) );

Marshal.StructureToPtr(buf, pBuf, false);

buf.bufs = Marshal.AllocHGlobal( Marshal.SizeOf(
typeof(my_subbuffer_t) ) * 2);

// Marshal managed to unmanaged
Marshal.StructureToPtr( subbuf[0], buf.bufs, false );
Marshal.StructureToPtr( subbuf[1], new IntPtr(buf.bufs.ToInt32()
+ Marshal.SizeOf(typeof(my_subbuffer_t))), false);
Marshal.StructureToPtr( buf, pBuf, false );

// Test custom marshaler
my_buffer_custom_t bufCustom = new my_buffer_custom_t();
bufCustom.count = 2;
bufCustom.bufs = subbuf;
//box
object obj = (object)bufCustom;

try
{
fun2(ref subbuf[0], 2);

// Marshal the inelegant way
fun(pBuf);
// Marshal the elegant way with our custom marshaler
fun3(obj);

}
catch (SystemException ex)
{
Console.WriteLine(ex.Message);
}
finally
{
Marshal.FreeHGlobal(buf.bufs);
Marshal.FreeHGlobal(pBuf);
}
}
}

public class SampleMarshaler : ICustomMarshaler
{
static SampleMarshaler marshaler;
string cookie;

SampleMarshaler( string cookie )
{
this.cookie = cookie;
}

public object MarshalNativeToManaged( IntPtr pNativeData )
{
object obj = Marshal.PtrToStructure( pNativeData,
typeof(my_buffer_custom_t) );

return obj;
}

public IntPtr MarshalManagedToNative(object managedObj)
{
if ( managedObj == null )
return IntPtr.Zero;

if ( !(managedObj is my_buffer_custom_t) )
throw new MarshalDirectiveException("This custom marshaler
must be used with a my_buffer_custom_t type.");

my_buffer_custom_t cbuf = (my_buffer_custom_t)managedObj;

// allocate memory for the unmanaged my_buffer_custom_t struct
IntPtr pcBuf = Marshal.AllocCoTaskMem( Marshal.SizeOf(cbuf) );

// Marshal managed to unmanaged

Marshal.StructureToPtr( cbuf, pcBuf, false );

// Sanity check
my_buffer_custom_t test =
(my_buffer_custom_t)Marshal.PtrToStructure(pcBuf,
typeof(my_buffer_custom_t));

return pcBuf;
}

public void CleanUpNativeData( IntPtr pNativeData )
{
if ( IntPtr.Zero != pNativeData )
Marshal.FreeCoTaskMem(pNativeData);
}

public void CleanUpManagedData( object managedObj )
{

}

public int GetNativeDataSize()
{
return -1;
}

public static ICustomMarshaler GetInstance(string cookie)
{
if ( marshaler == null )
{
marshaler = new SampleMarshaler( cookie );
}

return marshaler;
}
}
}

****** End of Program.cs ******

[James Whetstone]

This is great. I've learned a lot from this code, so thanks for posting it.

"Michael Phillips, Jr." <mphil...@nospam.jun0.c0m> wrote in message

news:%23MyGkZI...@TK2MSFTNGP06.phx.gbl...

[James Whetstone]

Well this has been a good interop primer for me. However, I'm still having
issues with using interop to marshal the following struct:

struct my_buffer_t
{
int count;
struct my_subbuffer_t **buffers;
};

In the test code we've been working with, we've been using:

struct my_buffer_t
{
int count;
struct my_subbuffer_t *buffers;
};

So that's where I'm at.

"Michael Phillips, Jr." <mphil...@nospam.jun0.c0m> wrote in message

news:%23MyGkZI...@TK2MSFTNGP06.phx.gbl...

[James Whetstone]

My first attemp at marshalling the following structure:

struct my_buffer_t
{
int count;
my_subbuffer_t **buffers;
};

is the following C# program:

/**** Start C# Console app ***/

using System;
using System.Collections.Generic;
using System.Text;
using System.Runtime.InteropServices;
namespace InteropTest
{
[StructLayout(LayoutKind.Explicit)]
public struct my_subbuffer_t
{
[FieldOffset(0)]
public int format;
}

[StructLayout(LayoutKind.Explicit)]
public struct my_buffer_t
{
[FieldOffset(0)]
public int count;
[FieldOffset(4)]
public IntPtr bufs;
}

class Program

{
[DllImport("MyDLL.dll", CharSet = CharSet.Ansi, CallingConvention =
CallingConvention.StdCall)]
public static extern int fun([In] IntPtr buf);

[DllImport("MyDLL.dll", CharSet = CharSet.Ansi, CallingConvention =
CallingConvention.StdCall)]

public static extern int fun2([In, Out] ref my_subbuffer_t bufs, int
size);

static void Main(string[] args)
{

my_buffer_t buf = new my_buffer_t();
buf.count = 2;

my_subbuffer_t[] subbuf = new my_subbuffer_t[2];
subbuf[0].format = 1;
subbuf[1].format = 2;

IntPtr pBuf = Marshal.AllocHGlobal(Marshal.SizeOf(
typeof(my_buffer_t) ));

//Allocate memory for two pointers
buf.bufs = Marshal.AllocHGlobal(8);

//Allocate memory for 2 my_subbuffer_t objects
IntPtr p =
Marshal.AllocHGlobal(Marshal.SizeOf(typeof(my_subbuffer_t)) * 2);

// Marshal managed to unmanaged

Marshal.StructureToPtr(subbuf[0], p, false);

//Marshal the pointer to the first my_subbuffer_t
Marshal.StructureToPtr((int)p, buf.bufs, false);

//Marshal managed to unmanaged
Marshal.StructureToPtr(subbuf[1], new IntPtr(p.ToInt32() +
Marshal.SizeOf(typeof(my_subbuffer_t))), false);

//Marshal the pointer to the second my_subbuffer_t
Marshal.StructureToPtr((int)(p.ToInt32() +
Marshal.SizeOf(typeof(my_subbuffer_t))), new IntPtr(buf.bufs.ToInt32() +
Marshal.SizeOf(typeof(my_subbuffer_t))), false);

//Marshal managed to unmanaged
Marshal.StructureToPtr(buf, pBuf, false);

// Sanity check
my_buffer_t test = (my_buffer_t)Marshal.PtrToStructure(pBuf,
typeof(my_buffer_t));

my_subbuffer_t testsubbuffer =
(my_subbuffer_t)Marshal.PtrToStructure(buf.bufs, typeof(my_subbuffer_t));

try
{
//fun2(ref subbuf[0], 2);
fun(pBuf);
}

catch (SystemException ex)
{
Console.WriteLine(ex.Message);
}
finally
{
Marshal.FreeHGlobal(buf.bufs);
Marshal.FreeHGlobal(pBuf);
}
}
}
}

/*** End C# Console app ***/

Yeah, this looks pretty ugly, but it works. Any suggestions for a better
solution?

"James Whetstone" <jameswh...@comcast.net> wrote in message

news:O9GdnRX0AaxUwFPY...@comcast.com...

[Michael Phillips, Jr.]

Marshaling a struct member with double indirection is a problem considering
that the language does not support double indirection.

However, we can be creative and still get the job done!

****** Start of mydll.c ******

// mydll.c : Defines the entry point for the DLL application.
//

#include <windows.h>
#include <stdlib.h>
#include <stdio.h>

#ifdef _MANAGED
#pragma managed(push, off)
#endif

struct my_subbuffer_t
{
int format;
};

struct my_buffer_t
{
int count;
struct my_subbuffer_t *buffers;
};

struct my_buffer_double_indirection_t

{
int count;
struct my_subbuffer_t **buffers;
};

struct my_buffer_custom_t

{
int count;
struct my_subbuffer_t buffers[1];
};

BOOL APIENTRY DllMain( HMODULE hModule,
DWORD ul_reason_for_call,
LPVOID lpReserved
)
{
return TRUE;
}

__declspec( dllexport ) int __stdcall fun4( struct
my_buffer_double_indirection_t *buf )

{
int i = 0;
char msg[32];

if ( NULL != buf )
{
for ( ;i < buf->count; ++i )
{

int format = (*buf->buffers)[i].format;

sprintf(msg,TEXT("fun4:format = %d\n"), format);

return 0;
}

return 0;
}

return 0;
}

return 0;
}

[StructLayout(LayoutKind.Explicit)]
public struct PMY_SUBBUFFER
{
[FieldOffset(0)] public IntPtr bufs;
}

[StructLayout(LayoutKind.Explicit)]
public struct my_buffer_double_indirection_t
{
[FieldOffset(0)]public int count;
[FieldOffset(4)] public IntPtr pBufs;
}

[StructLayout(LayoutKind.Explicit)]
public struct my_buffer_custom_t
{
[FieldOffset(0)] public int count;
[MarshalAs(UnmanagedType.ByValArray, ArraySubType =
UnmanagedType.Struct, SizeConst = 2)]
[FieldOffset(4)] public my_subbuffer_t[] bufs;
}

class Program
{
[DllImport("MyDLL.dll", CharSet = CharSet.Ansi, CallingConvention =
CallingConvention.StdCall)]
public static extern int fun( [In] IntPtr buf);

[DllImport("MyDLL.dll", CharSet=CharSet.Ansi, CallingConvention =
CallingConvention.StdCall)]
public static extern int fun2([In, Out] ref my_subbuffer_t bufs, int
size);

[DllImport("MyDLL.dll", CharSet = CharSet.Ansi, CallingConvention =
CallingConvention.StdCall)]
public static extern int fun3(
[MarshalAs(UnmanagedType.CustomMarshaler, MarshalTypeRef =
typeof(SampleMarshaler),MarshalCookie = "2"),In,Out] object buf );

[DllImport("MyDLL.dll", CharSet = CharSet.Ansi, CallingConvention =
CallingConvention.StdCall)]
public static extern int fun4([In] IntPtr mypBuf);

static void Main(string[] args)
{
my_buffer_t buf = new my_buffer_t();
buf.count = 2;

my_subbuffer_t[] subbuf = new my_subbuffer_t[2];
subbuf[0].format = 1;
subbuf[1].format = 2;

// allocate memory for the my_buffer_t struct and a pointer to
an array of my_subbuffer_t structs
IntPtr pBuf = Marshal.AllocHGlobal( Marshal.SizeOf(buf) );

Marshal.StructureToPtr(buf, pBuf, false);

buf.bufs = Marshal.AllocHGlobal( Marshal.SizeOf(
typeof(my_subbuffer_t) ) * 2);

// Marshal managed to unmanaged
Marshal.StructureToPtr( subbuf[0], buf.bufs, false );
Marshal.StructureToPtr( subbuf[1], new IntPtr(buf.bufs.ToInt32()
+ Marshal.SizeOf(typeof(my_subbuffer_t))), false);
Marshal.StructureToPtr( buf, pBuf, false );

// Test custom marshaler
my_buffer_custom_t bufCustom = new my_buffer_custom_t();
bufCustom.count = 2;
bufCustom.bufs = subbuf;
//box
object obj = (object)bufCustom;

// double indirection
my_buffer_double_indirection_t dblIndirectbuf = new
my_buffer_double_indirection_t();
PMY_SUBBUFFER mybuf = new PMY_SUBBUFFER();
mybuf.bufs = buf.bufs;

IntPtr psubBuf = Marshal.AllocHGlobal( Marshal.SizeOf(mybuf) );
Marshal.StructureToPtr( mybuf, psubBuf, false );
dblIndirectbuf.pBufs = psubBuf;
dblIndirectbuf.count = 2;

IntPtr mypBuf =
Marshal.AllocHGlobal(Marshal.SizeOf(dblIndirectbuf));
Marshal.StructureToPtr(dblIndirectbuf, mypBuf, false);

try
{
fun2( ref subbuf[0], 2 );

// Marshal the inelegant way
fun( pBuf );
// Marshal the elegant way with our custom marshaler
fun3(obj);

// double indirection
fun4( mypBuf );

}
catch (SystemException ex)
{
Console.WriteLine(ex.Message);
}
finally
{
Marshal.FreeHGlobal(buf.bufs);
Marshal.FreeHGlobal(pBuf);

Marshal.FreeHGlobal(psubBuf);
Marshal.FreeHGlobal(mypBuf);

[James Whetstone]

The code looks good. I took a slightly different approach for the double
indirection and would like your feedback if you have the time.

One of the challenges I've run into with my current interop application is
the fact that the double indirection involves polymorphic objects. To
extend our example, imagine that the my_subbuffer_t object is a base class
where the "format" field indicates the type of object. My approach then was
to create a C# wrapper where the input is an ArrayList filled with objects
that are all subclasses of the my_subbuffer_t class.

Here's the code for the C# method:

public static int TheWrapper(ArrayList bufArray)
{

IntPtr pBuffers;
my_buffer_t buf;
buf.bufs = IntPtr.Zero;

if (null != bufArray)
{
buf = new my_buffer_t();
buf.count = bufArray.Count;

//allocate space for an array of pointers
buf.bufs =
Marshal.AllocHGlobal(Marshal.SizeOf(typeof(IntPtr)) * bufArray.Count);

//iterate through the subbuffer list
for (int i = 0; i < bufArray.Count; ++i)
{
if (bufArray[i] is my_subbuffer_t)
{
//allocate space for a my_subbuffer_t object
IntPtr p =
Marshal.AllocHGlobal(Marshal.SizeOf(typeof(my_subbuffer_t)));

//marshal the my_subbuffer_t object into the
unmanaged memory
Marshal.StructureToPtr((my_subbuffer_t)bufArray[i],
p, false);

//marshal the pointer to the newly created
my_subbuffer_t object to the array of pointers embedded in the my_buffer_t
object
Marshal.StructureToPtr((int)p, new
IntPtr(buf.bufs.ToInt32() + Marshal.SizeOf(typeof(my_subbuffer_t)) * i),
false);
}
}

//allocate unmanaged memory for the my_buffer_t object
pBuffers =
Marshal.AllocHGlobal(Marshal.SizeOf(typeof(my_buffer_t)));

//marshal the managed my_buffer_t object to unmanaged
Marshal.StructureToPtr(buf, pBuffers, false);

}

try
{
result = fun(pBuffers);
}
catch (SystemException ex)
{
System.Console.WriteLine(ex.Message);
}
finally
{
if (null != bufArray)
{
//iterate through the array of pointers embedded in the
my_buffer_t object
for(int i=0; i<bufArray.Count;++i)
{
//obtain the pointer
IntPtr p = Marshal.ReadIntPtr( new
IntPtr(buf.bufs.ToInt32() + i * Marshal.SizeOf(typeof(IntPtr))));

Marshal.FreeHGlobal(p);
}

Marshal.FreeHGlobal(buf.bufs);
Marshal.FreeHGlobal(pBuffers);
}

}

return result;
}

Thanks for all the excellent posts.

"Michael Phillips, Jr." <mphil...@nospam.jun0.c0m> wrote in message

news:uFhT36fT...@TK2MSFTNGP05.phx.gbl...

[Michael Phillips, Jr.]

Wrapping and unwrapping complex structures as in your method is another
solution that works.

You can take this a step forward by creating a custom marshaler that
seamlessly wraps and unwraps your polymorphic objects.

There are many other approaches to the problem. One of them is very old
school. For example, Microsoft uses the PROPVARIANT struct, that allows you
to package just about any conceivable type, including arrays of types,
references to types, etc. Using COM, you marshal the object and unpack it at
the other end.

"James Whetstone" <jameswh...@comcast.net> wrote in message

news:BKydne5LLImC9FLY...@comcast.com...

[James Whetstone]

Thanks for the validation---I appreciate it. I wanted to validate this
approach before moving the code into a custom marshaler. Also, thanks for
the info on the PROPVARIANT struct. I'd never heard of it and it might come
in handy.

"Michael Phillips, Jr." <mphil...@nospam.jun0.c0m> wrote in message

news:%23RUaueh...@TK2MSFTNGP02.phx.gbl...

[James Whetstone]

I'm using FreeHGlobal to free the memory used for my polymorphic objects.
This doens't look right to me though, because the runtime can't know what
the actual type is. So what is the proper way to free memory used by
polymorphic objects? Marshal.DestroyStructure?

For clarification, here's a code snippet where I'm walking down an array of
pointers, each of which points to an object that is a sub class of
my_subbuffer_t:

for (int i = 0; i < buf.count; ++i)
{
IntPtr p = Marshal.ReadIntPtr(new IntPtr(buf.bufs.ToInt32() + i *
Marshal.SizeOf(typeof(IntPtr))));
Marshal.FreeHGlobal(p);
}

My best guess is that it should be something like this:

my_buffer_t buf = Marsha.PtrToStructure(ptr, typeof(my_buffer_t));

for (int i = 0; i < buf.count; ++i)
{
IntPtr p = Marshal.ReadIntPtr(new IntPtr(buf.bufs.ToInt32() + i *
Marshal.SizeOf(typeof(IntPtr))));

my_subbuffer_t sbuf = Marshal.PtrToStructure(p, typeof(my_subbufer_t));

switch(sbuf.format)
{
case 1:
Marshal.DestroyStructure(p, typeof(my_subbuffer_t)); //the base
type
break;
default:
break;
}

}

"James Whetstone" <jameswh...@comcast.net> wrote in message

news:d8OdnYBf-a7r5lLY...@comcast.com...

[Michael Phillips, Jr.]

> This doens't look right to me though, because the runtime can't know what
> the actual type is. So what is the proper way to free memory used by
> polymorphic objects?

The runtime does not need to know. You pass a pointer to a chunk of memory
that represents an object. All that is necessary is that method that you
used to create the chunk has a corresponding method to deallocate the chunk.
That is what the marshaler does behind the scenes. The runtime assumes that
memory is allocated with AllocCoTaskMem. It deallocates with FreeCoTaskMem.

If you choose to use Marshal.AllocHGlobal, you must use Marshal.FreeHGlobal.

If you choose to use Marshal.AllocCoTaskMem, you must use
Marshal.FreeCoTaskMem.

Marshal.DestroyStructure calls the COM method SysFreeString. This would be
appropriate if the memory was allocated with SysAllocString.

AllocHGlobal is a nice general purpose allocator which works with all
versions of Windows.

AllocCoTaskMem is the modern day Window's replacement that is useful for COM
and general purposes.

The choice of allocator is up to you. Just make sure that you pair it up
with the appropriate deallocator.

"James Whetstone" <jameswh...@comcast.net> wrote in message

news:pcqdne2SYOQ8G1LY...@comcast.com...

[James Whetstone]

Okay, great. That goes against my intuition, but that's why I asked.

So to make sure I'm interpreting what your saying correctly, let me recap:

I can assume that if I allocate an object with AllocHGlobal and get a IntPtr
that points to an unmanaged memory block, when I deallocate the memory
using FreeHGlobal, the runtime will know the size of the memory block
without telling it the type of object that resides in the memory block or
otherwise indicate the size of the memory block. Right?

"Michael Phillips, Jr." <mphil...@nospam.jun0.c0m> wrote in message

news:uTHIHniT...@TK2MSFTNGP06.phx.gbl...

[Michael Phillips, Jr.]

> I can assume that if I allocate an object with AllocHGlobal and get a
> IntPtr that points to an unmanaged memory block, when I deallocate the
> memory using FreeHGlobal, the runtime will know the size of the memory
> block without telling it the type of object that resides in the memory
> block or otherwise indicate the size of the memory block. Right?

Take a look at the ICustomMarshaler interface. This is what the runtime
uses to marshal memory back and forth.

You will notice that it uses a System.Object type and an IntPtr. The
runtime asks the marshaler interface to create a chunk of unmanaged memory
that represents the Native Object. The InPtr returned represents that chunk
of memory. The size is not relevant since the unmanaged side knows what
that unmanaged memory represents and it's size. The same with the managed
side. The size of the Native Object and what type it represents in known
before and after marshaling.

When the marshaler returns from the interop call, it presents an IntPtr
which is a chunk of unmanaged memory to an unknown System.Object with
unknown size.
The marshaler creates a Native System.Object from that chunk of unmanaged
memory and requests that the custom marshaler deallocate the unmanaged
memory.

The Marshaler knows the type, size and methods used for allocating and
deallocating all objects because you code it.

[James Whetstone]

I'm not sure if get it yet...

If the runtime calls CleanUpNativeData(IntPtr pNativeData) and the
pNativeData is a structure the looks like this:

[StructLayout(LayoutKind.Explicit)]
public struct arm_buffer4_t

{
[FieldOffset(0)]
public int count;
[FieldOffset(4)]
public IntPtr bufs;
}

then, yes, I know the object type. But... I don't know what type of objects
are pointed to in the "bufs" array, which is an array of pointers to
polymorphic objects. When I traverse the array of bufs and call FreeHGlobal
on each IntPtr, I first need to determine the type of object so that I can
tell the runtime to deallocate the proper size of memory. This may be a
symptom of a flaw in my implementation, but when I create the "bufs", I
allocate a number of pointers equal to the number of polymorphic objects I'm
going to create. I call Marshal.AllocHGlobal( Marshal.SizeOf(IntPtr) * x)
where 'x' is the number of objects. Then I create each object with a
separete call to AllocHGlobal and marshal the managed object into the newly
created memory pointer. Then I marshal the objects address into the array
of pointers.

So that's why I'm confused.

"Michael Phillips, Jr." <mphil...@nospam.jun0.c0m> wrote in message

news:%234Vh0%23iTHH...@TK2MSFTNGP06.phx.gbl...

[Michael Phillips, Jr.]

> then, yes, I know the object type. But... I don't know what type of
> objects are pointed to in the "bufs" array, which is an array of pointers
> to polymorphic objects.

You do not need to know! When MarshalManagedToNative is called, you return
an IntPtr that represents a chunk of unmanaged memory. It points to the
beginning of the memory that you formatted.

You know the count of objects that this chunk represents because you record
it in a structure and you know how you created that chunk(e.g, linked list,
array, etc.).

If you never recorded the count of objects, you would still know it because
the marshaler is called with a cookie that represents the count!

When CleanUpNativeData is called, you unwind the process to deallocate the
unmanaged memory. You know how to deallocate the memory because you coded
what that chunk looks like(e.g, link list, array, etc.). The methods are
paired. You simply reverse the process used when allocationg the chunk of
memory to deallocate it. Again, the IntPtr points to the beginning of that
chunk of memory.

This process is no different than what is used by malloc and free or new and
delete. Your code knows exactly how that memory is formatted both before
and after interop marshaling.

The bufs' IntPtrs all point to the beginning of some chunk of memory that
you allocated based upon the original objects marshaled. The size was known
when that chunk was created and the layout of that memory was known.

When the time comes to deallocate it, the size is not the issue. The only
issue is that the IntPtr still points to the beginning of the original chunk
of memory allocated.
If it does not, the system will throw a memory corruption error.

"James Whetstone" <jameswh...@comcast.net> wrote in message

news:FP6dnZX0BeNAOlLY...@comcast.com...

[James Whetstone]

Thanks for the explanation---I get it now. Some pre-learned C++ concepts
about polymorhpism that just don't apply here were throwing me off.

Thanks again.

"Michael Phillips, Jr." <mphil...@nospam.jun0.c0m> wrote in message

news:uMX93mjT...@TK2MSFTNGP02.phx.gbl...

[James Whetstone]

I've written a custom marshaler and have begun debugging it. To make sure
I'm freeing all the unmanaged memory, I step through the code and note the
addresses of the IntPtrs for each of the memory blocks I allocate using
AllocHGlobal. This is all done inside the "MarshalManagedToNative" method.
Then, when the runtime calls "CleanUpNativeData", I match up the address of
each IntPtr that is being freed with FreeHGlobal. They all match up
perfectly. Then I test my code by running the pinvoke call in a loop
while I look at the process's memory consumption. What I see is what looks
like a memory leak. The process consumes several K of memory per second
while looping. Could there be another explanation other than a memory leak
for this kind of behavior?

"James Whetstone" <jameswh...@comcast.net> wrote in message

news:1NqdnWVUcdrbL1LY...@comcast.com...

[James Whetstone]

For my original memory observation, I used the Task Manager. But from some
of the stuff I've read online about the TM, it's not accurate. So anyways,
I ran a .NET memory profiler against the application, and it looks fine.

"James Whetstone" <jameswh...@comcast.net> wrote in message

news:AYidnZG117UmZ1LY...@comcast.com...