Mixed Sync/Async communications pattern
dave.j...@googlemail.com
looking for a pointer
to example code or any reference to a standard solution.
Application A sends commands to B.
Some commands do not require a response from B, and some do, but I
know ahead of the time
which are which. Commands that require a response, should not return
to their caller in A,
because inevitably, the response is needed in the next line of code.
Application B sends responses back to A, but can also send async
events, as the user
interacts with B.
Async events from B can in turn cause one or more commands from A to
B, and B might
in turn fire off a sync reply, and one or more further async requests
as it executes A's commands.
This typically occurs when the user request an object to be created at
location X, and the
mouse is 'over' location X, and the object to be constructed fires off
'mouseOver' events,
which in turn.......
The combination of fileevent...readable and vwait varname cannot
handle the situation
100 % of the time
My thinking at the moment revolves around a FIFO queue of expected
responses, but
someone may already have solved the problem.
Any pointers ?
Dave
dave.j...@googlemail.com
<dave.joub...@googlemail.com> wrote:
> I have a problem that I am sure someone has solved before, and I am
> looking for a pointer to example code or any reference to a standard solution.
>
> My thinking at the moment revolves around a FIFO queue of expected
> responses, but someone may already have solved the problem.
>
> Any pointers ?
>
> Dave
Since there was no response, I have come up with a solution myself,
which revolves around two FIFO queues, and a loop around the vwait. I
would be happy to describe it if anyone wants any detail.
Dave
ddd
>
> Since there was no response, I have come up with a solution myself,
> which revolves around two FIFO queues, and a loop around the vwait. I
> would be happy to describe it if anyone wants any detail.
>
> Dave
>
I would be very interested. Can you put it on wiki.tcl.tk?
Fredderic
"dave.j...@googlemail.com" <dave.j...@googlemail.com> wrote:
> My thinking at the moment revolves around a FIFO queue of expected
> responses, but someone may already have solved the problem.
I think the reason you haven't got any replies, is that it's been
solved 101 different ways, and very few of them lend themselves to
anything particularly generic.
You seem to be on the right track to me... Another option is to give
each message a unique ID, and have that ID send back in the reply, and
use the ID to index an array of pending transactions. That way, you
don't even need to worry about the messages coming back in order, for
example, if you're requesting information which needs to be fetched
from a database.
Although in that case it is a good idea to be able to "expire" (or even
re-try) messages that for some reason weren't responded to at all.
I've had that happen with a Windoze "business server" that had a
too-small output buffer, and ended up just ditching messages when it
was under pressure. It's almost expected if you're communicating using
UDP to a remote server, and so forth.
But with reliable communications, a fifo queue aught to do just fine,
and is much less complicated, so if it'll do, then do it that way.
Although a transaction ID can still be useful to detecting if the whole
thing gets fowled up. ;)
I've also had to deal with one case where several modules within an
application would make similar (or even identical) requests, which
needed to be merged and batched. Usually the first one would set up a
timeout, after which the requests gathered so far would be sent off in a
single chunk. The response then needed to be divided up and fed back
to whomever had requested it, sometimes needing to take care to
respond in the same order that an individual requestee had asked for.
On that occasion the requests typically took a while to process, being
handed off to other servers to process at the remote end, so there were
usually several requests in the pipeline at any given time and the
responses would come back in an unpredictable order. So you'd send off
requests for all the information you know you'll be needing (and
occasionally even some you _might_ need), and then have to collect
together the responses until you have the ones you need to proceed,
every so often having to stop and wait for another piece of information
if it hasn't come back yet. It's all still the same sort of task.
Something that may be worth looking at (apart from continuations which
we don't have yet, but would be absolutely fantastic for this sort of
thing), are functions that package the current procedures variables
into a dict or similar. There's a few in the wiki that'll do the
job, and can be useful if you either just need to invoke a short script
when the request completes, or if you're invoking a follow-on procedure
which can simply use [dict with] to unpack the variables passed to it as
an argument, and continue on where the previous stage left off. That
leads to this sort of structure;
proc something:real {state env} {
switch -exact -- $part "start" {
dict with env {
... do some stuff ...
}
request-blah ... [list something:real part1 $env]
} "part1" {
dict with env {
... do some more stuff ...
}
request-blah ... [list something:real part2 $env]
} ...etc...
}
proc something {...whatever...} {
... check argument correctness ...
# grab initial environment
foreach _ [info locals] {dict set env $_ [set $_]}
# call the real thing
something:real start $env
}
Where [capture] performs that [foreach] to build the env dict within an
[uplevel], and uses [info level] to obtain the calling procs name, then
sandwiches those two around the supplied next-state value. Not going
to try write [capture] on the fly, there are good examples in the wiki
and it can be a little tricky to get right without trampling stuff
(the only "safe" way that I'm aware of off-hand, is to wrap [info
locals] in an uplevel, and then one-by-one [upvar] each variable and
add it to the dict).
I've never personally had need to go quite that far (usually
[namespace code] is all I've needed), although I did write something up
just to see how it'd look. I used a proc-like command that took
multiple body portions, and essentially did exactly what I described
there. It pretty much worked, but not quite to my liking, so I put it
aside somewhere (alas, I've got an awful lot of "somewhere" on my
system ;) ) to finish off if I ever had a real need for it.
Hope that gives you a few extra ideas? (Sorry for the late reply, I
must have marked the thread as read by accident. I've done this sort
of things a couple times myself, so I know how pesky it can be.)
Fredderic
dave.j...@googlemail.com
> I think the reason you haven't got any replies, is that it's been
> solved 101 different ways, and very few of them lend themselves to
> anything particularly generic.
>
> You seem to be on the right track to me... Another option is to give
> each message a unique ID, and have that ID send back in the reply, and
> use the ID to index an array of pending transactions. That way, you
> don't even need to worry about the messages coming back in order, for
> example, if you're requesting information which needs to be fetched
> from a database.
My solution was simple in the end, because I repartitioned the
problem.
As you say, maybe now the solution is no longer generic enough.
The first step was to add a unique id to each outgoing message. I
could see
no solution without doing this either explicitly or implicitly (by
examining
each message and trying to compute the context), and it was easier to
do
it explicitly.
After that it was purely a matter of refactoring the code to separate
the
message reception from the message decoding, to defer the decoding and
make
it possible to call it from two different places. Once that was in
place the
FIFOs were simple. (I have to use FIFOs rather than hash tables,
because
in-order execution was paramount; it may not be true for other
applications.)
Why not put it on the Wiki?
I am not sure whether the solution is generic enough/good enough to
merit it,
so I will add it to this post, and someone else can decide....
Aside:
The code forms part of a multi-layer project I am working on, and I
will be
releasing the standalone bits soon.
I have often looked at VRML/X3D and thought that one day I would like
to do
some work with it. Every time I think along those lines, I end up
looking at
the External Authoring Interface, and I run straight into Java, which
is not
my favorite topic. (In fact, working on this project has once more
reminded
me why I grew to dislike Java so much)
Finally, I started working on something which had to be 3D and
interactive, and
I had to bite the bullet.
What I will have for release soon (now that I have solved my last
major problems)
is a set of iTcl classes that replace the Java classes for EAI
communication
with the FreeWRL VRML/X3D browser. The lowest levels are FreeWRL
specific, because
the EAI spec does not mandate the comms pattern, only the high-level
API. FreeWRL
was the first browser I found that was cross-platform, opensource, and
still in active
development that actually encouraged the use of its EAI.
As a sub-sub-project, I have also done a OSC library for TCL, and an
OSC to FreeWRL
bridge. The use of OSC means that in turn you can connect the VRML
browser to something
like PD
So:
Hi-Level App(TCL) <---OSC---> whatever
|
|
Intermediate classes
|
|
EAI class library
|
|
OSC class library <---OSC---> OSC/FreeWRL bridge <---> FreeWRL Browser
<---> User
Some other APP <----OSC---->/
Dave
Notes: The bits of code do not all reside in the same file,
and the places that call the send routines are deep inside
some incr TCL class libraries; therefor upvar has been used
to maintain the critical variables...
set needSync unknown
set needSyncExpectQueue [list]
set needSyncResponseQueue [list]
set needSyncResponseQueueValues [list]
proc init {} {
........
fileevent $udpSock readable [list ::APPData::remoteMsgEvent
$udpSock]
}
proc sendMessage {senderObjName senderObjClass op args} {
upvar #0 needSync needSync
upvar #0 needSyncExpectQueue needSyncExpectQueue
upvar #0 needSyncResponseQueue needSyncResponseQueue
upvar #0 needSyncResponseQueueValues needSyncResponseQueueValues
if {$expectReply} {
lappend needSyncExpectQueue $outgoingUuid
}
.............
send the message
.............
if {$expectReply} {
while {[llength $needSyncExpectQueue] > 0} {
set needSync Pending
vwait needSync
if {[llength $needSyncExpectQueue] > 0} {
puts stdout "sendMessage : still waiting for = [join
$needSyncExpectQueue {,}]"
puts stdout "sendMessage : got [llength
$needSyncResponseQueue] responses queued up."
}
}
while {[llength $needSyncResponseQueue] > 0} {
set path [lindex $needSyncResponseQueue 0]
set values [lindex $needSyncResponseQueueValues 0]
decodeAPPmsg $path $values
set needSyncResponseQueue [lreplace $needSyncResponseQueue 0
0]
set needSyncResponseQueueValues [lreplace
$needSyncResponseQueueValues 0 0]
}
} else {
.............
}
.............
}
proc remoteMsgEvent {chan} {
.............
handleMsg $remoteMsg
set needSync Done
}
proc handleMsg {remoteMsg} {
.............
decodeAPPmsg $path $values
}
proc decodeAPPmsg {path values} {
upvar #0 needSync needSync
upvar #0 needSyncExpectQueue needSyncExpectQueue
upvar #0 needSyncResponseQueue needSyncResponseQueue
upvar #0 needSyncResponseQueueValues needSyncResponseQueueValues
#
# If the expect queue has entries in it, we may need to defer
# this message.
#
if {[llength $needSyncExpectQueue] > 0} {
set incomingUuid ........
if {$incomingUuid == [lindex $needSyncExpectQueue 0]} {
set needSyncExpectQueue [lreplace $needSyncExpectQueue 0 0]
} else {
#
# Defer the message by putting it at the back of the queue
#
lappend needSyncResponseQueue $path
lappend needSyncResponseQueueValues $values
return
}
}
.............
}
dave.j...@googlemail.com
Dave
Uwe Klein
and a scientific scale from a tcl script.
I had solved this for me using expect's exp_background,
some flags. and a queue of lines to send.
uwe