[beepwg] Asynchronous BEEP draft status

[Martin Thomson]

Hi all,

At the 73rd IETF meeting in Minneapolis, I talked to Lars Eggert, who
has agreed to sponsor my asynchronous channels draft. This means
that, pending review, this document will be published as a standards
track RFC.

<http://tools.ietf.org/html/draft-thomson-beep-async>

If you have any feedback on this document, I'd appreciate it if you
could send that feedback to this list, or to me directly.

Regards,
Martin

[Francis Brosnan Blazquez]

Hi Martin,

The draft looks good to me. Really interesting work. Congratulations!

--
Francis Brosnan Blazquez <fra...@aspl.es>
Advanced Software Production Line, S.L.

[Marshall Rose]

+1

[Sam Roberts]

-1

I am unconvinced of the value of this draft, for three reasons:

a - This draft describes how to build into BEEP something you can
already do with BEEP (several ways).

I can write a trivial example using vortex that implements the
motivating use-case for those unconvinced.

b - BEEP adoption suffers from perceived complexity.

I don't personally think it is more complex than it has to be, but it
would be better to document how to use what it offers than to extend
it.

c - BEEP adoption suffers from non-interoperability of existing toolkits.

If implementors are already not writing interoperable toolkits when
implementing the existing RFC3080/3081, optional extensions are going
to make this situation worse.

Cheers,
Sam

[Sam Roberts]

On Wed, Nov 26, 2008 at 10:17 AM, Sam Roberts <vieu...@gmail.com> wrote:
> a - This draft describes how to build into BEEP something you can
> already do with BEEP (several ways).
>
> I can write a trivial example using vortex that implements the
> motivating use-case for those unconvinced.

Sorry. Couldn't resist implementing this.

Draft states following use case:

"""
Asynchronous applications require a protocol that is able to support
a large number of concurrent outstanding requests. The analogy of a
channel as a thread does not scale to the large number of threads
used in modern systems. Modern applications regularly have large
numbers of concurrent processing threads. Thus, a better way of
multiplexing large numbers of concurrent requests is required.

This document describes an BEEP feature, an extension to BEEP, that
enables the creation of an asynchronous channel. An asynchronous
channel is a channel where response ordering is not fixed to the
order of the requests sent by the client peer. An asynchronous
channel is identical to other channels, using unmodified framing;
only requests may be processed in parallel and responses may be sent
in any order.
"""

Note that Vortex delivers each MSG on a seperate thread, exactly as
described above.

BEEP msgnos are ordered across a channel, of course, but application
msgs can be replied whenever you want. The profile designer specifies
the content and meaning of beep message payload.

Profile designers are not required to map BEEP internal msgno 1-to-1
to their profile's message identifiers. Often, this is convenient.
When it is not convenient, don't do it.

Server:

#include <vortex.h>

#define PROFILE "http://example.com/beep/delayed"
/*
MSG payload is msgid

server waits a random amount of time, and sends reply.

RPY payload is "msgid delay", where msgid is the msgid of the request being
responded to, and delay is how long the server delayed before sending the
reply.

Actual code might put msgid in a MIME header, and the payload would be the
request information. Or the request would be xml encoded, and include a msgid.

There are lots of variations of this, none of them requiring extending the BEEP
protocol.
*/

VortexQueue* msgno_queue;

void frame_received(VortexChannel* chan, VortexConnection* conn,
VortexFrame* frame, void*v)
{
int msgno = vortex_frame_get_msgno (frame);
int delay = (rand() % 30) + 1;
char* msgid = vortex_frame_get_payload(frame);
char str[128];

vortex_queue_push(msgno_queue, INT_TO_PTR(msgno));

printf("msgno %d msgid %s delay %d\n", msgno, msgid, delay);

sleep(delay);

msgno = PTR_TO_INT(vortex_queue_pop(msgno_queue));

sprintf(str, "%s %d", msgid, delay);

vortex_channel_send_rpy(chan, str, strlen(str), msgno);
}

const char USAGE[] = "usage: %s <port>";

int main(int argc, char ** argv)
{
if(!argv[1]) {
printf("%s\n", USAGE);
return 1;
}

msgno_queue = vortex_queue_new();

vortex_init();

vortex_log_enable(1);

vortex_profiles_register(PROFILE, NULL, NULL, NULL, NULL,
frame_received, NULL);

vortex_listener_new("0.0.0.0", argv[1], NULL, NULL);

vortex_listener_wait();

vortex_exit();

return 0;
}

Client:

#include <vortex.h>
#include <assert.h>

#define PROFILE "http://example.com/beep/delayed"

int msgid = 0;

void send_msg(VortexChannel* channel)
{
int ok = vortex_channel_send_msgv(channel, 0, "%d", ++msgid);
assert(ok);
}

void on_frame(VortexChannel* channel, VortexConnection* conn,
VortexFrame* frame, void* v)
{
int msgno = vortex_frame_get_msgno(frame);
const char* content = vortex_frame_get_payload(frame);

printf("(msgno %d) %s\n", msgno, content);

send_msg(channel);
}

int main (int argc, char ** argv)
{
VortexConnection * connection = NULL;
VortexChannel * channel = NULL;

vortex_init ();

connection = vortex_connection_new(argv[1], argv[2], NULL, NULL);

if (!vortex_connection_is_ok(connection, false)) {
fprintf(stderr, "Unable to connect remote server, error was: %s\n",
vortex_connection_get_message(connection));
return 1;
}

channel = vortex_channel_new(connection, 0,
PROFILE,
NULL, NULL, /* no close handling */
on_frame, NULL,
NULL, NULL /* no async channel creation */
);

if (channel == NULL) {
fprintf(stderr, "Unable to create the channel..\n");
return 1;
}

printf(".. send msg\n");

send_msg(channel);
send_msg(channel);
send_msg(channel);
send_msg(channel);
send_msg(channel);

{
char c;
read(0, &c, 1);
}
vortex_exit ();

return 0 ;
}

Client output:

% ./async-client localhost 3333
.. send msg
(msgno 0) 3 28
(msgno 1) 1 14
(msgno 2) 2 17
(msgno 3) 5 24
(msgno 4) 4 26
(msgno 5) 10 2
(msgno 6) 11 3
(msgno 7) 9 10
(msgno 8) 8 13
(msgno 9) 12 8

Not BEEP msgno order is maintained, application msg order is asynchronous.

Cheers,
Sam

[David Kramer]

On Nov 26, 2008, at 11:11 AM, Sam Roberts wrote:

> BEEP msgnos are ordered across a channel, of course, but application
> msgs can be replied whenever you want. The profile designer specifies
> the content and meaning of beep message payload.
>
> Profile designers are not required to map BEEP internal msgno 1-to-1
> to their profile's message identifiers. Often, this is convenient.
> When it is not convenient, don't do it.

I can provide some additional real-world evidence to back up Sam's
claims.

The way Xgrid uses BEEP is to send an immediate empty RPY for each MSG
that is received. The RPY in this case acts as nothing more than an
acknowledgement that the MSG was received. In Xgrid, all messages,
whether they are requests, responses, or notifications, are mapped to
BEEP MSGs. When Xgrid wants to reply to a request, it just sends it
in another MSG. The body of the MSGs indicate whether the message is
a request or a response, and each request includes a correlation ID
which is then also included in the corresponding response.

The bodies of requests Xgrid sends look something like this pseudo-xml:

<message>
<message-type>request</message-type>
<message-name>status</message-name>
<message-correlation-id>87</message-correlation-id>
<message-body>...</message-body>
</message>

It turns out that Xgrid actually responds to all messages in order, so
this technique isn't being used to allow for out-of-order responses,
even though it could be. In Xgrid's case, this technique is being
used to allow for notifications -- messages for which there is no
response. The use of the ANS-style replies is not suitable for Xgrid
because of the requirement that each ANS message have a unique ID,
which results in both the sender and receiver having to keep track of
every outstanding ANS ID. For long-running connections with millions
of notifications this puts pressure on memory and reduces performance.

So I have to agree with Sam that on the face of things this proposal
for a new asynchronous channel type seems unnecessary. Applications
do not need to use BEEP message-numbers, they can ignore that detail
of the BEEP implementation and just implement their own message
tracking through the message headers or bodies.

However, while we're on the topic of improving the BEEP protocol, if
it were up to me the one change I would consider making is to add a
new message type, the notification (NFN). This would be like a MSG,
except that the peer does not send a RPY. This might be a simpler
change that achieves the OPs goals. However, it really isn't very
important, because it is easy to send empty RPYs, and the advantage of
sending empty RPYs is that they act like an acknowledgement of
receipt. If you were using NFN style notification messages the only
acknowledgement you would get would be an eventual SEQ frame.

-David

[Thomson, Martin]

Hi Sam, David,

I'm sorry that you think that this is unnecessary. I acknowledge the fact that profiles are able to define and use other means of request-response correlation. The intent of this work is to save application protocols and clients from the additional complexity of managing disparate solutions to the problem.

As an intermediate layer, one of the goals of BEEP is to simplify the development of an application protocol. All the workarounds that I've seen have only pushed more complexity on the application protocol.

One other advantage of this work is that the method for addressing asynchrony becomes unified. There aren't multiple fragmented solutions, some using headers, others using protocol-specific parameters.

A point that I'd like to stress is that this is entirely optional. Firstly, both peers must support (and indicate willingness to use) the feature. Secondly, both peers must agree to use the mode before a channel can be created with the feature enabled. From this perspective, I cannot see why you are so resistant to this idea.

Sam:

> Note that Vortex delivers each MSG on a seperate thread, exactly as
> described above.

If this is the case, then this is directly against the stated requirement in RFC 3080, Section 2.6.1:

A BEEP peer acting in the server role must process all "MSG" messages
for a given channel in the same order as they are received.

It is possible, at the application to ignore this requirement if you so choose. That is entirely your prerogative, and as long the semantics of your profile are explicit, then this wont have any negative consequences.

Of course, to operate in this mode, a client will have to implement a demux layer that redirects responses to the entity that originated the request. Without this layer, the BEEP stack would redirect responses to the wrong entity. In your example, the entity that makes request 3 would receive response 5 instead. This is less of a problem in C, but much more of a problem for an OO language implementation, c.f. beep4j.

You've made a conscious decision to move the complexity to the application.

The goal with the asynchronous work is to provide application profiles the option to have correlation transparently handled by the stack.

Sam:

> BEEP msgnos are ordered across a channel, of course, but application
> msgs can be replied whenever you want.

This is in direct contravention of the quoted requirement, above.

Sam:

> c - BEEP adoption suffers from non-interoperability of existing toolkits.

Feature negotiation ensures that this option cannot be used without agreement. I can't see how this work affects interoperability.

David:

> The way Xgrid uses BEEP is to send an immediate empty RPY for each MSG
> that is received.

This is a valid choice for your application. This solution doesn't presume to prevent this sort of behaviour, only expand the choices available to application protocol designers. However I think that your use case is somewhat different.

I get the impression that the way that Xgrid uses BEEP doesn't quite fit with the use cases described in the document.

David:

> Applications
> do not need to use BEEP message-numbers, they can ignore that detail
> of the BEEP implementation and just implement their own message
> tracking through the message headers or bodies.

There is an interesting point here. With this work, the applications can ignore message tracking entirely. The BEEP stack ensures that responses get back to the correct component. No need for message tracking at the application at all.

David:

> if
> it were up to me the one change I would consider making is to add a
> new message type, the notification (NFN).

I too have a similar requirement. I am using empty RPYs in the same fashion. I'd be happy to work with you on defining a new message type if you think that is useful.

(Note that empty RPYs are a redundant receipt indication; you get receipt indications at the TCP layer as well.)

Regards,
Martin


------------------------------------------------------------------------------------------------
This message is for the designated recipient only and may
contain privileged, proprietary, or otherwise private information.
If you have received it in error, please notify the sender
immediately and delete the original. Any unauthorized use of
this email is prohibited.
------------------------------------------------------------------------------------------------
[mf2]

[David Kramer]

Hello Martin,

I have a couple of responses to your message, as well as some comments
on the draft.

===

>> c - BEEP adoption suffers from non-interoperability of existing
>> toolkits.
>
> Feature negotiation ensures that this option cannot be used without
> agreement. I can't see how this work affects interoperability.

If I create a profile that requires the use of a new feature, then
that profile can not be implemented on a BEEP stack that does not
provide that feature, so that profile cannot interoperate with that
BEEP stack. Perhaps this isn't the classic definition of
interoperability, but it is something to consider.

However, upon further consideration, I do not see how a profile could
actually _require_ the use of the async feature, so the
interoperability argument is moot.

>> The way Xgrid uses BEEP is to send an immediate empty RPY for each
>> MSG
>> that is received.
>
> This is a valid choice for your application. This solution doesn't
> presume to prevent this sort of behaviour, only expand the choices
> available to application protocol designers.

My point is that the protocol may already be flexible enough to solve
the problem without inventing new features. I do not think expanding
choices for the sake of choices is a good idea. If there is already a
good way of doing something, we shouldn't add an additional way.

> (Note that empty RPYs are a redundant receipt indication; you get
> receipt indications at the TCP layer as well.)

I do not think the TCP layer can indicate that that the remote
application has read all of the message's frames from the socket, can
it? That is what the empty RPY indicates, right?

===

Anyways, reading through the draft, I think one reason I am resistant
to this feature is that I do not fully understand when it would better
than the alternatives. As I see it, the two main alternatives to
async channels are empty replies and multiple channels.

The consequence of empty replies is that is pushes the burden of
message correlation onto the application layer. I can appreciate that
this is undesireable for applications that don't already have a reason
to correlate messages. But this could be hidden from the application
at the toolkit level. Rather than inventing a new type of channel, we
could define some standard MIME headers that applications that require
async messaging should use. And toolkits could add features to
automatically add these headers and correlate replies with messages
using them. Toolkits that didn't provide these features automatically
could still interoperate with those that did, as long as the profile
implementor handled the message correlation manually.

The other option is multiple channels. Sending messages on separate
channels has the asynchronous behavior you want, right? By using one
channel per message you can implement your profile on any BEEP stack.
The client can control the amount of parallel requests it makes by
limiting the number of channels it opens, and the server can control
the amount of parallel requests it has to handle by limiting the
number of channels it allows the client to open. Again, the toolkit
could provide features to automatically open and close channels for
each message that is sent, or keep channel pools open, so the
application still wouldn't have to deal with these details.

In fact, the one-channel-per-message strategy provides even greater
asynchrony than async channels, because the frames of both the MSGs
and the RPYs on independent channels can be interleaved over the
socket. The async channels feature specifically excludes this: "An
asynchronous channel must still observe the rules in [RFC3080]
regarding segmented messages. Each message must be completed before
any other message can be sent on that same channel."

However, I am concerned about this statement: "Different "ANS"
messages that are sent in a one-to-many exchange may be interleaved
with responses to other "MSG" messages." This seems to be a bad
idea... is this saying that ANS and RPY frames can be interleaved on
async channels? Why?

I am especially concerned about the note following that statement: "It
is recommended that BEEP peers do not generate interleaved ANS
segments." Why is it recommended that you do not interleave ANS
segments? Isn't this a basic feature of BEEP? Is this statement mean
only in the context of async channels, or all channels?

Based on this analysis it looks like async channels provide a
compromise between empty RPYs, which puts the burden of message
correlation onto the application, and multiple channels, which puts
the burden of channel management onto the application. Both of the
burdens could be lessened by toolkits or frameworks, without changing
the BEEP wire protocol. But if you just want your server to be able
to answer out of order, but you don't want to correlate the replies
yourself, and you don't want the answers interleaved, then I see how
the async channel is the most efficient solution.

-David

[Sam Roberts]

On Wed, Nov 26, 2008 at 3:21 PM, Thomson, Martin
<Martin....@andrew.com> wrote:
> Sam:
>> Note that Vortex delivers each MSG on a seperate thread, exactly as
>> described above.
>
> If this is the case, then this is directly against the stated requirement in RFC 3080, Section 2.6.1:
>
> A BEEP peer acting in the server role must process all "MSG" messages
> for a given channel in the same order as they are received.

This is far-fetched.

Vortex is acting correctly. It delivers the MSGs to the API serially
(to different threads).

BEEP places no requirements on the what "process" means for a profile.

[reordered]

> Sam:
>> BEEP msgnos are ordered across a channel, of course, but application
>> msgs can be replied whenever you want.
>
> This is in direct contravention of the quoted requirement, above.

BEEP RPYs are passing over the wire in the RFC defined order.

I could write the code in the other toolkits I use, too (beepy,
swirl/beepcore-c, and beep4j).

> Of course, to operate in this mode, a client will have to implement a demux layer that redirects responses to the entity that originated the request. Without this layer, the BEEP stack would redirect responses to the wrong entity.
> In your example, the entity that makes request 3 would receive response 5 instead. This is less of a problem in C, but much more of a problem for an OO language implementation, c.f. beep4j.
> You've made a conscious decision to move the complexity to the application.

Yes, where it belongs.

Demonstrably trivial complexity (implementable in a few lines of C
code in my example, I don't believe its harder with beep4j, we use it,
its a good toolkit).

The complexity of the code that implements application functionality
will dwarf these few lines.

This is what BEEP is designed for, to be a protocol to build
protocols, not a finished thing implementing everything you could
want.

> Sam:
>> c - BEEP adoption suffers from non-interoperability of existing toolkits.
>
> Feature negotiation ensures that this option cannot be used without agreement. I can't see how this work affects interoperability.

Some toolkits will have it, some won't. Profiles specified as needing
the extension will be implementable with some toolkits, not with
others.

Cheers,
Sam

[Thomson, Martin]

Hi David,

I went through exactly the same thought process in coming up with this solution. I think that only the conclusion is different.

(Actually, at first I wrongly assumed that msgno was used for request-response correlation. That lead to some interesting results.)

> However, upon further consideration, I do not see how a profile could
> actually _require_ the use of the async feature, so the
> interoperability argument is moot.

I can easily require the use of the feature by specifying in my definition of the profile that it is required. It really is that easy - the semantics associated with a profile are defined in specification, or some other form of mutual agreement. You can agree to anything. To block their use in practice, the serving peer can prevent the creation of a channel if the initiating peer doesn't indicate that it is asynchronous.

> My point is that the protocol may already be flexible enough to solve
> the problem without inventing new features. I do not think expanding
> choices for the sake of choices is a good idea. If there is already a
> good way of doing something, we shouldn't add an additional way.

The point is that this is something that people want to do. Standardizing a mechanism means that the solution can be of benefit to the entire community. There are many ways to skin this particular cat, I happen to prefer this one because of its simplicity. I also happen to think that this is the most logical and elegant way to achieve the goal.

> Anyways, reading through the draft, I think one reason I am resistant
> to this feature is that I do not fully understand when it would better
> than the alternatives. As I see it, the two main alternatives to
> async channels are empty replies and multiple channels.

I wouldn't characterise empty replies as solving the problem - they are part of the alternative solution. The entire solution requires correlation of request and response by some other means. As Sam demonstrated, empty replies don't have to be part of that (even though generating MSG frames in both directions is better than the approach Sam took for the reasons I shared in my previous mail).

> The consequence of empty replies is that is pushes the burden of
> message correlation onto the application layer. I can appreciate that
> this is undesireable for applications that don't already have a reason
> to correlate messages. But this could be hidden from the application
> at the toolkit level. Rather than inventing a new type of channel, we
> could define some standard MIME headers that applications that require
> async messaging should use. And toolkits could add features to
> automatically add these headers and correlate replies with messages
> using them. Toolkits that didn't provide these features automatically
> could still interoperate with those that did, as long as the profile
> implementor handled the message correlation manually.

Indeed we could, and I did consider this option. One advantage with using msgno is that you don't have any additional overhead. In effect, it's finally putting the msgno to use.

> The other option is multiple channels. Sending messages on separate
> channels has the asynchronous behavior you want, right? By using one
> channel per message you can implement your profile on any BEEP stack.
> The client can control the amount of parallel requests it makes by
> limiting the number of channels it opens, and the server can control
> the amount of parallel requests it has to handle by limiting the
> number of channels it allows the client to open. Again, the toolkit
> could provide features to automatically open and close channels for
> each message that is sent, or keep channel pools open, so the
> application still wouldn't have to deal with these details.

I know that Vortex implements channel pools to that end. However, you have a limit on the number of channels that can be opened and I have an application that would completely run out of channels based on this solution. Imagine a processing time of 45 seconds, with an incoming rate of 400 requests each second. To successfully use that approach you need 18000 channels. That approach also dilutes the effectiveness of the flow control mechanism by having 18000 different flow control windows.

> However, I am concerned about this statement: "Different "ANS"
> messages that are sent in a one-to-many exchange may be interleaved
> with responses to other "MSG" messages." This seems to be a bad
> idea... is this saying that ANS and RPY frames can be interleaved on
> async channels? Why?

If you consider that each MSG is processed in parallel, then this makes sense. If there is a process that generates 100 ANS responses, it would be unreasonable to hold up the channel until those 100 responses are completed. Therefore, the document stats that as long as complete frames are sent, other parallel processing components are able to send RPY or ANS frames at the same time.

> I am especially concerned about the note following that statement: "It
> is recommended that BEEP peers do not generate interleaved ANS
> segments." Why is it recommended that you do not interleave ANS
> segments? Isn't this a basic feature of BEEP? Is this statement mean
> only in the context of async channels, or all channels?

RFC 3080 allows for interleaving split ANS frames. It doesn't say why. I can't for the life of me work out how this would be a good idea.

My view is that there is no reason why a stack should treat ANS frames any differently to other frames once they are received from the application. Frame splitting is a flow control and throughput issue, which should be handled at a lower logical layer in the stack. This means that the stack needs to assemble a complete frame before it sends it off.

In an async channel the problem is compounded - if you allow interleaving, then you need to ensure that you are only interleaving with ANS replies to the same MSG. With replies (and even requests) potentially coming asynchronously at any time, this increases the chance of making a mistake.

> Based on this analysis it looks like async channels provide a
> compromise between empty RPYs, which puts the burden of message
> correlation onto the application, and multiple channels, which puts
> the burden of channel management onto the application. Both of the
> burdens could be lessened by toolkits or frameworks, without changing
> the BEEP wire protocol. But if you just want your server to be able
> to answer out of order, but you don't want to correlate the replies
> yourself, and you don't want the answers interleaved, then I see how
> the async channel is the most efficient solution.
>
> -David
>

Cheers,

[Sam Roberts]

On Wed, Nov 26, 2008 at 7:07 PM, Thomson, Martin
<Martin....@andrew.com> wrote:
> Hi David,
>
> I went through exactly the same thought process in coming up with this solution. I think that only the conclusion is different.

> RFC 3080 allows for interleaving split ANS frames. It doesn't say why. I can't for the life of me work out how this would be a good idea.
>
> My view is that there is no reason why a stack should treat ANS frames any
> differently to other frames once they are received from the application. Frame splitting is a flow control and throughput issue, which should be handled at a lower logical layer in the stack.

In below, "message" means a *complete* MSG, RPY, ERR, or ANS, not a frame.

Interleaving of ANS frames provides a useful feature not commonly
implemented by toolkits other than beepcore-c.

AFAICT, several toolkits (beep4j, for example) started off only
working in terms of messages, for both send and receive.

This doesn't allow a peer to process data as it arrives in frames, and
can cause deadlock if the window fills. So, eventually they allow
frames to be received by applications as they arrive, perhaps
optionally (vortex will accumulate all frames into a complete message,
if requested, for example, before presenting them to an application).

It is quite useful to expose frames to applications on the sending
side, too. For example, it allows a single RPY to have an arbitrarily
large size, so an entire file could be transferred in a single RPY
(ANS, MSG, etc.) in block-size frames.

All the toolkit APIs since beepcore-c (that I've seen) force the
application to present the entire payload of a MSG, RPY, ANS, etc. as
a contiguous chunk of data in a single API call when sending.

Anyhow, RFC3080 describes MSG/ANS as a one-to-many exchange, and each
one of the entities on the "many" side is allowed to generate its ANS
as a series of frames. Without this capability, the entities would be
forced to provide complete ANSs serially, instead of in parallel.

Sam

[Thomson, Martin]

Hi Sam,

> On Wed, Nov 26, 2008 at 3:21 PM, Thomson, Martin
> <Martin....@andrew.com> wrote:
> > Sam:
> >> Note that Vortex delivers each MSG on a seperate thread, exactly as
> >> described above.
> >
> > If this is the case, then this is directly against the stated
> requirement in RFC 3080, Section 2.6.1:
> >
> > A BEEP peer acting in the server role must process all "MSG"
> messages
> > for a given channel in the same order as they are received.
>
> This is far-fetched.
>
> Vortex is acting correctly. It delivers the MSGs to the API serially
> (to different threads).
>
> BEEP places no requirements on the what "process" means for a profile.

I can't present an argument against that if you insist on attributing a different meaning to this well-understood term. The full quote is:

A BEEP peer acting in the server role must process all "MSG" messages

for a given channel in the same order as they are received. As a
consequence, the BEEP peer must generate replies in the same order as
the corresponding "MSG" messages are received on a given channel.

Reading RFC 3117 and the BEEP book, which talks of an analogy of a channel as a thread, this is consistent with a definition of "process" that includes everything the serving peer does between receiving the request and sending the response.

The serial nature of a channel is important, particularly where a peer establishes state based on the results of previous requests. Being able to rely on serial processing is important.

> > Sam:
> >> BEEP msgnos are ordered across a channel, of course, but application
> >> msgs can be replied whenever you want.
> >
> > This is in direct contravention of the quoted requirement, above.
>
> BEEP RPYs are passing over the wire in the RFC defined order.
>
> I could write the code in the other toolkits I use, too (beepy,
> swirl/beepcore-c, and beep4j).

The contravention is in the processing order, not the on-wire representation. RFC 3080 establishes an expectation (and limitation) on the way that serving peers operate. You are, of course, free to ignore this, but therein lies interoperability problems.

> > You've made a conscious decision to move the complexity to the
> application.
>
> Yes, where it belongs.

That is entirely subjective.

> > Sam:
> >> c - BEEP adoption suffers from non-interoperability of existing
> toolkits.
> >
> > Feature negotiation ensures that this option cannot be used without
> agreement. I can't see how this work affects interoperability.
>
> Some toolkits will have it, some won't. Profiles specified as needing
> the extension will be implementable with some toolkits, not with
> others.

I don't see that as a valid argument against improving the protocol. Given that the implementation in application-space is demonstrably simple, adding to the toolkit can't be significantly more difficult.

Cheers,

[Sam Roberts]

> The contravention is in the processing order, not the on-wire representation.

RFC 3080 only defines wire representation. Peer behaviour is defined
by a profile, not RFC3080.

Cheers,
Sam

[Thomson, Martin]

Hi Sam,

You state that ANS interleaving is a useful feature of the protocol, and then point out that almost no-one bothers to implement it. Can you provide a use case?

The discussion on access to frames is interesting, but it doesn't contribute to the discussion.

> Anyhow, RFC3080 describes MSG/ANS as a one-to-many exchange, and each
> one of the entities on the "many" side is allowed to generate its ANS
> as a series of frames. Without this capability, the entities would be
> forced to provide complete ANSs serially, instead of in parallel.

This was a revelation of a sort for me. If we are talking about many serving entities then maybe there IS a use case. However, given that this is a one-to-many exchange of _messages_, I don't see how you make the leap from that to many entities.

[Thomson, Martin]

The text in question [1] talks about peer behaviour. This is often a point of contention in the IETF, but the discussion always comes back to the reality. Inevitably, some aspects of behaviour are visible on the wire.

The text in RFC 3080 covers both processing behaviour and on-wire behaviour, one sentence for each. If it were just about on-wire behaviour, the second sentence would have been sufficient.

Ta,
Martin

[1]

A BEEP peer acting in the server role must process all "MSG" messages
for a given channel in the same order as they are received. As a
consequence, the BEEP peer must generate replies in the same order as
the corresponding "MSG" messages are received on a given channel.

[Sam Roberts]

On Wed, Nov 26, 2008 at 8:09 PM, Thomson, Martin
<Martin....@andrew.com> wrote:
> Hi Sam,
>
> You state that ANS interleaving is a useful feature of the protocol, and then point out that almost no-one bothers to implement it.

Implementors are having enough trouble getting basic communication going!

Beepy and beep4j wouldn't communicate at all when we first tried,
beepy had broken SEQ handling, and beep4j had broken MIME header
format parsing, and beepcore-c wouldn't talk to vortex. I haven't
tried the first two against the latter two, yet. Hopefully it will go
well.

> Can you provide a use case?

But those gripes aside, I sure can. Both for an ANS, and just
generally for transferring files larger than should be manipulated
in-memory.

RFC3080: "in a one-to-many exchange, multiple answers may be
simultaneously in progress"

MSG: <get-all-movies actress="uma thurman">

ANS: a movie, type and name defined in MIME headers, with Uma Thurman in it
...

The MSG would kick into gear a heavy duty search of known movie
download sites, and each ANS would be a single movie, sent as it is
found in small chunks (movies are big!), possibly multiple ones would
be found at once and sent.

Anyhow, like async channels, its a feature that you can do without,
you can build it on top of smaller message exchanges tied together by
identifiers that are not BEEP message numbers.

For example, we transfer large dynamically generated PDFs over BEEP.
With beepcore-c, as data is ready we would write out a frame of a RPY
(which might get further subdivided as needed to fit window size), but
instead we RPY with a "its coming at you soon with this message-id",
and then send each chunk of data as it is available with a MSG (which
gets an empty RPY).

Mind you, we reply before we actually process the request to generate
and send the file.

Actually, we have a "thin" UI client (beep4j) that uses BEEP to talk
to its server (heavily modified beepy), and almost everything that
happens on the client involves a MSG being sent to the server,
followed by an immediate RPY <ok/>, and then the server starts doing
whatever processing is required for the requested action.

Would you really consider our use of BEEP a massive conformance breach
of an RFC3080 "process before sending RPY" restriction on the
implementation of profiles?

Cheers,
Sam

[Thomson, Martin]

Hi Sam,

> > Can you provide a use case?
>
> But those gripes aside, I sure can. Both for an ANS, and just
> generally for transferring files larger than should be manipulated
> in-memory.
>
> RFC3080: "in a one-to-many exchange, multiple answers may be
> simultaneously in progress"
>
> MSG: <get-all-movies actress="uma thurman">
>
> ANS: a movie, type and name defined in MIME headers, with Uma Thurman
> in it
> ...
>
> The MSG would kick into gear a heavy duty search of known movie
> download sites, and each ANS would be a single movie, sent as it is
> found in small chunks (movies are big!), possibly multiple ones would
> be found at once and sent.

Is there a real need to send those large files all at the same time?

> Actually, we have a "thin" UI client (beep4j) that uses BEEP to talk
> to its server (heavily modified beepy), and almost everything that
> happens on the client involves a MSG being sent to the server,
> followed by an immediate RPY <ok/>, and then the server starts doing
> whatever processing is required for the requested action.
>
> Would you really consider our use of BEEP a massive conformance breach
> of an RFC3080 "process before sending RPY" restriction on the
> implementation of profiles?

No - because the semantics of the request/response protocol only state that an <ok/> be generated. Processing (from the perspective of the protocol, and client) is completed by acknowledgement of the request. From a protocol perspective, the larger process that is triggered by the request is a side effect.

Ta,

[Rainer Gerhards]

Hi Martin,

I am not commenting on the root question, because I was too long non-involved with BEEP. But... (inline)

> -----Original Message-----
> From: beepwg-...@beepcore.org [mailto:beepwg-...@beepcore.org]

> On Behalf Of Thomson, Martin
> Sent: Thursday, November 27, 2008 5:09 AM
> To: Sam Roberts; bee...@beepcore.org
> Subject: RE: ANS interleaving (was: [beepwg] Asynchronous BEEP draft
> status)
>
> Hi Sam,
>
> You state that ANS interleaving is a useful feature of the protocol,
> and then point out that almost no-one bothers to implement it. Can you
> provide a use case?

RFC3195, a standards-track syslog RFC, heavily depends on interleaving ANS. Without them, the profile cannot be implemented.

Rainer Gerhards

[Thomson, Martin]

Hi Rainer,

> RFC3195, a standards-track syslog RFC, heavily depends on interleaving
> ANS. Without them, the profile cannot be implemented.

Actually, I don't see this requirement in the document. Keep in mind that we are talking about multiple ANS messages, each being split into frames and those frames interleaved.

From RFC 3195, this would be where you get something like:

I: ANS 1 0 * 0 42 0
I:
I: <29>Oct 27 13:21:08 ductwork imxpd[141]:END
I: ANS 1 0 * 42 42 1
I:
I: <29>Oct 27 13:22:15 ductwork imxpd[141]:END
I: ANS 1 0 . 84 19 0
I:
I: Heating emergency.END
I: ANS 1 0 . 103 16 1
I:
I: Contact Tuttle.END

I don't see a requirement for that in 3195, explicit or implied.

In either case, the discussion isn't relevant to the asynchronous work, I was only expressing my opinion that this behaviour is unnecessary.