Handling allocation and serialization

[Neil T. Dantam]

Following Grey's request, I have a proposed change to better handle
allocation and serialization in Ach. Basically, this would add new
variations on ach_put() and ach_get() that take a function which would
transfer an object into or out of a channel buffer, possibly
serializing/deserializing or allocating the destination buffer along the
way. This would remove the intermediate copy when serializing data and
simplify data allocation/reallocation when receiving data.

Candidate prototypes of these functions are at:
https://github.com/golems/ach/commit/fe5db640364bfa2949e30ad5da80f83d4956ae9e

Cheers,
--
Neil Dantam
http://www.cc.gatech.edu/~ndantam3

[Michael Grey]

This sounds really cool and potentially tremendously useful, but I'm a little bit hazy right now on what the actual implementation is.

Correct me if I'm wrong, but it looks like the ach_put_fun() and ach_get_fun() are meant to be prototypes for user-defined functions. So I'd declare some function like

ach_put_fun serialize_greys_message;

and then give it a definition like

int serialize_greys_message( void *cx, void *chan_dst, const void *obj_src, size_t dst_size )

{
    /* do some serialization */

    return result; 

}

And then... how does it get used? Will there be a new version of ach_put which accepts an ach_put_fun function pointer so I can pass in whichever serialization function is relevant for that particular channel and message type? Or am I completely off base right now on how this is supposed to work?

Thanks!

--Grey

--
You received this message because you are subscribed to the Google Groups "ach-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ach-devel+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Neil T. Dantam]

On 04/29/2014 10:03 PM, Michael Grey wrote:
> Correct me if I'm wrong, but it looks like the ach_put_fun() and
> ach_get_fun() are meant to be prototypes for user-defined
> functions.

They are typedefs for user-defined functions.

> So I'd declare some function like
>

> *ach_put_fun serialize_greys_message;*
^ Don't need this part. ^

> and then give it a definition like
>
> int serialize_greys_message( void *cx, void *chan_dst, const void

> obj_src, size_t dst_size )
> {
> /* do some serialization */
>
> return result;
>
> }

Yes, you would define some kind of function like this.

> And then... how does it get used? Will there be a new version of ach_put
> which accepts an ach_put_fun function pointer so I can pass in whichever
> serialization function is relevant for that particular channel and message
> type?

Yes, exactly that.

Cheers,
-ntd

[Neil T. Dantam]

Ok, I did a quick copy-paste implementation of this. You can check
this test/demo program for a usage example:

https://github.com/golems/ach/blob/transfer-function/src/test/transfertest.c

Interface suggestions are welcome.

(and of course I'll clean up the now-duplicated code in the implementation)

[Michael Grey]

Cool. Just to make sure I understand how this is meant to work:

For ach_get_fun_memcpy the frame_size argument is the size of the incoming message as read by the ach channel, correct? So this gives us the opportunity to allocate more memory if our current buffer is not big enough.

I was originally thinking it would be useful to have an ach_peek() function which would let us see the next frame size or something to that effect, but I think the solution you're providing is much more elegant, and it doesn't suffer from the issue of a new message arriving between an attempt to ach_peek() and then the call to ach_get().

I like this a lot. If (read as "when") I do a major revision of the Hubo software that I'm currently writing, I'll probably take advantage of this to more elegantly handle variable-sized messages.

--Grey

[Neil T. Dantam]

On 04/30/2014 01:03 AM, Michael Grey wrote:
> For ach_get_fun_memcpy the frame_size argument is the size of the incoming
> message as read by the ach channel, correct?

Yes.

> So this gives us the opportunity to allocate more memory if our
> current buffer is not big enough.

Yes, and also error checking if doing de-serialization.

> I was originally thinking it would be useful to have an ach_peek()
> function which would let us see the next frame size or something to
> that effect,

You can actually already get this by passing a zero-size buffer to
ach_get(). It will still fill in the necessary frame_size.

> but I think the solution you're providing is much more elegant, and
> it doesn't suffer from the issue of a new message arriving between
> an attempt to ach_peek() and then the call to ach_get().

You're right about that race condition. I have lots of code where
ach_get() is called in a loop that reallocates a buffer if it's too
small.

> I like this a lot. If (read as "when") I do a major revision of the Hubo
> software that I'm currently writing, I'll probably take advantage of this
> to more elegantly handle variable-sized messages.

Cool, I'll probably put this on the shelf till you get there, since
that work may suggest some interface tweaks.

Thanks,