serving a file to a client --- background fcopy read fileevent event puts socket channel blocking nonblocking
vit...@gmail.com
large file with fcopy, I gave up on fcopy. I did come up with a
solution that is working nicely for me and simulates fcopy. Hopefully
my experience will help others since many have the same problems. May
be someone will suggest a better solution or improve on the idea.
First, some background on my experience with event driven programming
in TCL:
Event driven programming is not clearly documented. For example, if
you try to do something like this:
-----------------------------------------------------------------------------------------------------------------------------------------------------
#start of the code here
#more code
while {1} {#do something here; after 1000} #want to serve other
events, like sending a file to a socket
# since we have "vwait forever" (below) in our code
# you would think that this while loop will not block
#but it blocks anyway, not cool, that's not what documentation
#that I read implies.
vwait forever
-----------------------------------------------------------------------------------------------------------------------------------------------------
Here is how to make it work:
-----------------------------------------------------------------------------------------------------------------------------------------------------
#start of the code here
#more code
while {1} {#do something here; after 1000 [list set myvariable];
vwait myvariable}
vwait forever
-----------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------
Ok, now about transferring a large file without blocking all other
events:
The code:
-----------------------------------------------------------------------------------------------------------------------------------------------------
set fp [open /myfile]
#the socket $sock is already open by "socket -server"
set size 1024
set after_time 10
fconfigure $fp -translation binary -encoding binary -buffersize
$size -buffering full -blocking 0
fconfigure $sock -translation binary -encoding binary -buffersize
$size -buffering full -blocking 0
proc send_file {fp sock size after_time} {
while { ![eof $fp] && ![eof $sock] } {
puts -nonewline $sock [read $fp $size]
after $after_time [list set wait_var]
vwait wait_var
}
flush $sock #flush the remaining unsent data to the client
seek $fp 0 start # rewind the file pointer back to the beginning of
the file for the next transfer
}
-----------------------------------------------------------------------------------------------------------------------------------------------------
This assumes that you have a main "vwait forever" code in your script
to for all other events.
If you experiment with the values of the above variables "size and
after_time" you will have a very effective control of the speed of the
transfer (throttling) and the time set aside for other events.
NOTE: you cannot have more than one transfer of the same file using
the same file pointer at the same time. You need to open another file
pointer to the same file for each transfer request that happens while
another transfer is in progress.
----Victor
vit...@gmail.com
Sorry if the above code is not very readable. It seems that Google
reformats the messages for some reason.
----Victor
Uwe Klein
> On Mar 28, 10:32 am, vit...@gmail.com wrote:
>
>>After spending a lot of time trying to do a nonblocking transfer of a
>>large file with fcopy, I gave up on fcopy. I did come up with a
You never came across [fileevent] did you?
uwe
vit...@gmail.com
wrote:
:) nothing like a dose of sarcasm to start a working day :)
Of course I have seen/used fileevent. fcopy is supposed to use
fileevents automatically. My suggestion is for fcopy replacement code.
It is more flexible that fcopy (and a lot of people complain about
fcopy being buggy, not just me) and can do speed control and progress
notification.
Plus my code suggests how to make any code run in background, not just
code driven by events. That's not something everyone will just learn
from documentation. I know that a lot of people probably know this
already but there are many more who don't, who, like me, waste hours
of our time trying to figure out something that should be clearly
documented.
And, if you had a suggestion with a code using fileevents, why not
post it? I don't post here for entertainment value, I work on real
problems that need real solutions.
Uwe Klein
> On Mar 28, 10:58 am, Uwe Klein <uwe_klein_habertw...@t-online.de>
> wrote:
>
>>vit...@gmail.com wrote:
>>
>>>On Mar 28, 10:32 am, vit...@gmail.com wrote:
>>
>>>>After spending a lot of time trying to do a nonblocking transfer of a
>>>>large file with fcopy, I gave up on fcopy. I did come up with a
>>
>>You never came across [fileevent] did you?
>>
>>uwe
>
>
> :) nothing like a dose of sarcasm to start a working day :)
Well, there are a couple of issues:
vwait stacks, it does not work in parallel.
i.e any stacked vwait has to trigger to expose
the underlying vwaits.
This is an unpleasant situation at best.
Vwait works on variable change. Thus your
code sequence should never get further than the
entry to [vwait]
after $after_time [list set wait_var]
> vwait wait_var
>
next: you are vwaiting inside of a proc
on a variable change in local scope.
( no upvar, nor global statement )
but [after] expedites its payload in global scope
and your script only reads that var anyway.
( imho you should get a background error on this )
I am not clear how this works. But my guess is not
in the way you think it works.
On the other hand: I may be compleately wrong ;-)
uwe
Bruce Hartweg
> After spending a lot of time trying to do a nonblocking transfer of a
> large file with fcopy, I gave up on fcopy. I did come up with a
> solution that is working nicely for me and simulates fcopy. Hopefully
> my experience will help others since many have the same problems. May
> be someone will suggest a better solution or improve on the idea.
>
can I ask a few questions? I;m not sure of what problems you had with fcopy
and would like more info. were you using the -command version of it?
how was it being called? what was the negative effects?
> First, some background on my experience with event driven programming
> in TCL:
>
> Event driven programming is not clearly documented. For example, if
> you try to do something like this:
>
hmmm, i find it pretty well documented. could you explain what you
found lacking in detail
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> #start of the code here
> #more code
>
> while {1} {#do something here; after 1000} #want to serve other
> events, like sending a file to a socket
>
> # since we have "vwait forever" (below) in our code
>
> # you would think that this while loop will not block
>
> #but it blocks anyway, not cool, that's not what documentation
>
> #that I read implies.
what documentation did you read that made you think
while {1} {
# do stuff
after 1000
}
would have any way of processing events?
> vwait forever
>
> -----------------------------------------------------------------------------------------------------------------------------------------------------
> Here is how to make it work:
> -----------------------------------------------------------------------------------------------------------------------------------------------------
>
> #start of the code here
> #more code
>
> while {1} {#do something here; after 1000 [list set myvariable];
> vwait myvariable}
>
> vwait forever
>
very kludgy - why are you going this route instead of doing
your processing in an event, intead of an infinite loop?
this may solve your problem, but your problem may have been the structure of your
code in the first place. i don;t see anything here that couldn't be done
with fcopy and the -command and -size options
Bruce
vit...@gmail.com
Thanks Uwe,
You are right about stacking of vwaits. But they do seem to work on
local variables.
I was so excited about it. But it turns out that only ONE such code
instance will actually run alongside the regular file events. If I add
another one, one of them will suspend until the first one finishes.
Well, back to reality. At least I learned something new today. Thanks
again.
vit...@gmail.com
Well, it turns out that it doesn't work for more that one instance of
the loop code.
I was trying to do several things -- speed control of the transfer,
progress notification and backgrounding long running procs (the
infinite while loop was just an example).
Back to reality again.
Thanks
vit...@gmail.com
It looks like the only way to reliably do what I was trying to do is
to learn the Threads extension of TCL. I was postponing this step for
some time now because many binary extensions are not thread safe (at
least that's what I've read). The only binary extension I need to use
right now is Sqlite. Hopefully it is thread safe (don't know yet what
being thread safe really means).
Alan Anderson
<3c69be6e-ce92-4f23...@d4g2000prg.googlegroups.com>,
vit...@gmail.com wrote:
> It looks like the only way to reliably do what I was trying to do is
> to learn the Threads extension of TCL.
May I propose a better way? Learn how to work *with* the event loop
rather than fighting it. Start small. Try doing one thing, and post
here your solution. If feedback from some of the more experienced folk
here is positive, add one more thing, post here, repeat.
Based on the code you posted already, it's likely that feedback will
take the form of constructive suggestions on what to change in order to
make it work better. For example, I can think of no good reason *ever*
to write an infinite loop in an interactive Tcl application. I think
you might find that there is a teeny bit of misunderstanding on your
part regarding the "proper" way to do event-driven programming.
vit...@gmail.com
> In article
> <3c69be6e-ce92-4f23-9103-48da5f1fb...@d4g2000prg.googlegroups.com>,
Thanks Alan,
First of all, I had an infinite loop example to simulate a long
running procedure.
I have never so far written an actual infinite loop proc, I would at
least have
a break out code if I had written one.
But, nevertheless, lets attempt to go after a solution that does not
use threads.
How do you write a file to socket copy procedure that notifies me
about it's progress
every so many bytes and does not block while doing so. How about if I
want to control how fast
the copy progresses without blocking other events? Or, lets say,
theoretically, that I want
to stream a music or a video file where the user can fast forward and
go back, without
blocking and without sending the entire file over. And also, how about
a long running
procedure that doesn't block?
Thanks
Those are just theoretical questions that need generic answers.
Thanks
Alan Anderson
<7a3ef3f4-8af1-4078...@h11g2000prf.googlegroups.com>,
vit...@gmail.com wrote:
> How do you write a file to socket copy procedure that notifies me
> about it's progress
> every so many bytes and does not block while doing so.
Start with the [fcopy] manual page, and look at how the second example
works. The -command option along with a -size value does exactly what
you are asking. You can modify the CopyMore procedure to do additional
actions, such as setting the value of a progress bar widget.
> How about if I
> want to control how fast
> the copy progresses without blocking other events?
Change CopyMore to do smaller or larger chunks based on a control
widget's value, and/or to introduce a delay before rescheduling itself
to call the [fcopy] again.
Fredderic
vit...@gmail.com wrote:
> On Mar 28, 5:32 pm, Alan Anderson <arand...@insightbb.com> wrote:
>> May I propose a better way? Learn how to work *with* the event loop
>> rather than fighting it. Start small. Try doing one thing, and
>> post here your solution. If feedback from some of the more
>> experienced folk here is positive, add one more thing, post here,
>> repeat.
> But, nevertheless, lets attempt to go after a solution that does not
> use threads.
Last time I did this kind of thing manually (rather a while back), I
used a pair of [fileevent] handlers that represent "copy" and "wait"
modes of operation. (With, of course, non-blocking input and output.)
Start in "copy" mode; set up a writable [fileevent] on the outbound
stream, and when it fires, read a chunk of data from the source stream
and write it out. This will dump data as fast as we can to the output
stream, but would spin like crazy if there's no input to read. This is
optimised for the case where the source is faster then the destination,
for example, dumping a local file to a network socket.
When we get a zero-byte read from the input (not eof), you kill the
writable handler, and set up a readable [fileevent] on the source
stream instead. If you can tell how much data's being buffered on the
outgoing stream, then read and dump the data available and switch back
to the writable handler if you notice that a block or so of data is
just sitting in the outbound buffer (this supports the case where the
outbound stream is the faster one). If you can't, then just switch
right away to avoid buffering the whole source stream in memory. Either
way doing the usual copy-and-write will make sure there's as much data
available to be written as we've got.
It might have sub-optimal performance where small data chunks are
coming though erratically, where allowing data to collect in the output
buffers for a second or two can sometimes be useful. On the other
hand, if you're copying an interractive stream, then pushing the data
through regardless is a good idea. (This is a distinction that [fcopy]
doesn't seem to make.)
There are other algorithms, I'm sure, but this one is fairly simple,is
as generic as I have ever needed, and has served me well in the past.
Of course, if you know you'll always be reading from a file, or always
writing to a file, then the whole thing becomes a lot simpler because
you can get away with just one of the two handlers, and none of the
code to switch between them.
> How do you write a file to socket copy procedure that notifies me
> about it's progress every so many bytes and does not block while
> doing so.
If the [fileevent] handlers take an extra argument which is a notifier
function to call, similar to [fcopy]s -command, then just call it with
the current amount of data called, and it's up to that function to do
what it needs to do without blocking too long. Simply setting a
progress widget percentage shouldn't take too long.
> How about if I want to control how fast the copy progresses without
> blocking other events?
That's very difficult to do properly, in my experience... The easiest
way is to just keep a count of the current data transferred, and a
recurring timer that subtracts some fixed amount at fixed intervals
(optionally clipping if it goes negative). Then you stop the process
if the sent data exceeds some threshold, and restart it again from the
timer if it was stopped.
> Or, lets say, theoretically, that I want to stream a music or a video
> file where the user can fast forward and go back, without blocking
> and without sending the entire file over.
You're looking at something much more specialised there. For
starters, how will you know when the user has elected to rewind or
fast-forward the stream? Block-based protocols that request a specific
portion of the file to be sent at a time work well there, as can any
download protocol with a "resume at location" option. The client can
just cut the connection, and "resume" from a different arbitrary
point. It's still way beyond the realms of a generic [fcopy].
> And also, how about a long running procedure that doesn't block?
If it blocks, you're doing something wrong. That's what event driven
programming was invented for. There should only be one event loop, and
it runs from start to finish. Busy-loops are bad, use a timer.
Infinite loops are bad, use an idle timer. Polling is bad, use
[fileevent] instead.
Fredderic
Alexandre Ferrieux
> But, nevertheless, lets attempt to go after a solution that does not
> use threads.
Yes, it makes sense to try at least.
> How do you write a file to socket copy procedure that notifies me
> about it's progress every so many bytes and does not block while doing so.
Easy: use [fcopy -size ... -command ...]
The callback will of course have to repost the fcopy for the next
chunk until eof.
All this orchestrated by a single [vwait forever], no [while] loop.
If you have already tried this and encoutered problems, please explain
and possibly post them into the Tracker.
> How about if I want to control how fast
> the copy progresses without blocking other events?
Easy: instead or reposting the fcopy, the callback does [after
$somedelay repost].
> Thanks
> Those are just theoretical questions that need generic answers.
They do have very simple answers based on [fcopy] as you can see, at
least theoretically.
If in real life [fcopy] is not up to the promise, *that* is the single
most urgent thing to fix in the whole Tcl core, instead of reinventing
that wheel over and over again.
Of course, it is possible to reimplement [fcopy] in pure Tcl, and it
even is a good exercise for whoever wants to learn about the event
loop. But this will never have the performance of [fcopy], which is
documented as "leverag[ing] the buffering in the Tcl I/O system to
avoid extra copies and to avoid buffering too much data in main memory
when copying large files to slow destinations like network sockets".
-Alex
vit...@gmail.com
I read the manual for fcopy and duplicated the second example exactly
as it is on the page and it worked.
I was trying to run my previous code like this (simplified}:
set check 1
while {$check} {
fcopy $in $out -command [list CopyMore $in $out] -size 1024
}
proc CopyMore {in out args} {
if {[eof $in]} {set check 0; close $in}
}
I thought that it worked the way "read" command works, by reading from
where it left of but instead I was getting a "file busy" error.
I am still not sure why my code did not work.
I was able to simulate fcopy with TCL commands by using variable
trace. And it was a very good exercise. :)
vit...@gmail.com
Darren New
> I am still not sure why my code did not work.
You would need to show the whole of the code, including where you wait
for file events and where you open and close the files. At what line
were you getting "file busy errors"?
--
Darren New / San Diego, CA, USA (PST)
"That's pretty. Where's that?"
"It's the Age of Channelwood."
"We should go there on vacation some time."
vit...@gmail.com
I read the manual for fcopy and duplicated the second example exactly
as it is on the page and it worked.
I was trying to run my previous code like this (simplified}:
set check 1
while {$check} {
fcopy $in $out -command [list CopyMore $in $out] -size 1024
}
proc CopyMore {in out args} {
if {[eof $in]} {set check 0; close $in}
}
I thought that it worked the way "read" command works, by reading from
where it left of but instead I was getting a "file busy" error.
I am still not sure why my code did not work.
I was able to simulate fcopy with TCL commands by using variable
vit...@gmail.com
I deleted my original code, but this one does the same thing with the
same error:
set done 1
set in [open file1]
fconfigure $in -translation binary -encoding binary -blocking 0
set out [open file2 w]
fconfigure $out -translation binary -encoding binary -blocking 0
set check 1
set size 1024
while {$check} {
fcopy $in $out -command [list CopyMore $in $out $size] -size $size
}
proc CopyMore {in out size {error {}} } {
puts CallBack
if {[eof $in] || $error != "" } {set check 0; close $in}
}
vwait done
When I run it I get the following error: channel "file3" is busy
Alexandre Ferrieux
> while {$check} {
> fcopy $in $out -command [list CopyMore $in $out $size] -size $size
> }
>
> When I run it I get the following error: channel "file3" is busy
Well, maybe you could read that manpage once more :-)
When fcopy has a -command, it is *asynchronous*, which means:
- sets up internal fileevents and nonblocking modes
- (also prevents any further fileevents and IO on the two channels)
- returns immediately
Your [while] loop (which by the way at least two persons have asked
you to remove) then tries to repeatedly register that asynchronous
operation. Of course the [fcopy] here runs exactly twice: one which
works, and one which raises the "busy" error you reported.
What you want is rather:
proc StartOneChunk s {
fcopy -command CopyMore -size $s ...
}
# prime the pump, just once
StartOneChunk $size
proc CopyMore args {
if {finished} {set ::done 1;return}
StartOneChunk $::size
}
vwait done
-Alex
vit...@gmail.com
I deleted my original code, but this one does the same thing with the
same error:
set done 1
set in [open file1]
fconfigure $in -translation binary -encoding binary -blocking 0
set out [open file2 w]
fconfigure $out -translation binary -encoding binary -blocking 0
set check 1
set size 1024
while {$check} {
fcopy $in $out -command [list CopyMore $in $out $size] -size $size
}
proc CopyMore {in out size {error {}} } {
puts CallBack
if {[eof $in] || $error != "" } {set check 0; close $in}
Alexandre Ferrieux
Darren New
> if {[eof $in] || $error != "" } {set check 0; close $in}
Also, "set check 0" here doesn't do what you think it's doing.
vit...@gmail.com
wrote:
I was not trying to go back to the "while" loop :). It is the code
that started it all. I was just curious why it didn't work. The
process seemed to be very similar to the correct code (calling fcopy
repeatedly) and that's what made me confused. But your explanation
makes it clear.
Thanks,
----Victor
vit...@gmail.com
I was not trying to go back to the "while" loop :). It is the code
vit...@gmail.com
> Please don't repost this thrice, and see
>
> http://groups.google.com/group/comp.lang.tcl/tree/browse_frm/thread/5...
>
> -Alex
vit...@gmail.com
Hmmm, I was refreshing the page in order to read/post new messages.
May be that caused multiple postings. Sorry about that.
Fredderic
Alexandre Ferrieux <alexandre...@gmail.com> wrote:
> On Mar 29, 2:08 am, vit...@gmail.com wrote:
>> How do you write a file to socket copy procedure that notifies me
>> about it's progress every so many bytes and does not block while
>> doing so.
> Easy: use [fcopy -size ... -command ...]
> The callback will of course have to repost the fcopy for the next
> chunk until eof.
I've only ever used [fcopy] once, and now I remember why. Doesn't that
kind of usage strike anyone as being a seriously bodgy way to get
progress feedback, speed limiting, or, well, to do just about anything
other than pure automagical data movement...?
I tried using [ftell] on the inbound channel, from a periodic timer
that runs until the [fcopy -command] gets invoked, and it just reports
-1 all the time. That would at least make progress reporting possible.
I suppose, if we could tell how much data is waiting at the inbound
channel, we can cap it at some arbitrary value and use that as the -size
argument. But I don't know how to do that, and you'd still need your
own [fileevent] listener to run [fcopy].
Likewise for transfer rate limiting, there's no way to tell [fcopy] to
transfer one buffer full and finish so we can update our progress,
insert a pause, or whatever else. Perhaps a -once option would be
useful, but even then we still have to implement our own [fileevent]
handler.
And as soon as you start to dip into [fileevent] handling, you get the
problems I mentioned in my previous post. If the handler you're
listening to is on a channel that's significantly faster than the other
one, it'll spin like a mad man, so you need a means to listen to
whichever channel is slower. And just how big do you make that
transfer size? If your input stream is coming in sporadically, it'll
sit there waiting because the amount of data that came through in the
last burst was a single byte shorter than the size you specified. In
the meantime you could have posted the current progress, and/or done
your speed limiting calculations and associated adjustments.
I think personally, that being able to copy a single block (or as much
of one as is available), would make [fcopy] actually useful for
non-trivial functions. It's probably still not entirely optimal with
all the setup I suspect it's doing, but it's probably the best bet for
making it truly useful anytime soon.
> Of course, it is possible to reimplement [fcopy] in pure Tcl, and it
> even is a good exercise for whoever wants to learn about the event
> loop. But this will never have the performance of [fcopy], which is
> documented as "leverag[ing] the buffering in the Tcl I/O system to
> avoid extra copies and to avoid buffering too much data in main memory
> when copying large files to slow destinations like network sockets".
Alternatively, moving data from one channel to another is a fairly
common activity from what I've seen, both with and without processing
inbetwee. So I wonder if would be worth providing a means to create a
[fileevent] that only triggers when BOTH the input and output channels
are ready. (One-shot [fileevent]s would probably help there, we could
then post something lean and mean on the wiki to do the job, without
having to fiddle with continually setting and unsetting the [fileevent]
handlers.)
My choice, would be to split [fcopy] in half. An engine that invokes a
callback when there's both data waiting and somewhere to put it (output
buffers contain less than a block of data, for example), and a
mechanism for transferring up to a certain amount of data from one
channel to another with minimal memory copying:
fcopy infile outfile -onready command
fcopy infile outfile -single ?-size size?
The basic existing [fcopy] command could then be "expanded" as:
fcopy $in $out -onready [list fcopy $in $out -single]
The -onready option would be compatible with -command, though probably
not -size (although having -size pre-load the input buffer when used in
conjunction with -onready might offer some benefit, or at least
make [chan pending] more useful). In essence, the default -onready
command would simply invoke one execution of [fcopy]'s existing data
transfer code, which is exactly what the -single option does. I
haven't looked at the source to [fcopy], but I wouldn't imagine it to
be a particularly complex change.
Most importantly, rate limiting, progress reporting, and whatever else,
would grow quite readily out of that basic split in functionality.
Although for rate limiting you probably will want some means to "pause"
and/or cancel the handler (return value?), or a -continue option (or
something similar) that would normally be invoked internally by -single
(perhaps even just "-single -size 0" would be acceptible, albeit ugly).
Comments, anyone?
Fredderic
Alexandre Ferrieux
> On Sat, 29 Mar 2008 05:03:50 -0700 (PDT),
>
Hmmm... I think you've missed one tiny bit about [fcopy -size]'s
semantics :-)
It does exactly what you call "-once mode".
Thus, today's fcopy offers in a single "primitive" (in the sense that
it cannot be emulated at full efficiency in pure Tcl, though it can be
aproached):
- both synchronous and asynchronous transfers
- progress notification with configurable granularity
- bandwidth throttling with a simple [after] in the callback
- all of this with the guarantee of never accumulating data in
memory if the input is faster than the output (a mistake that is easy
to make when doing it by hand)
The above doesn't strike me as "seriously bodgy". I see it rather as a
well-designed compromise.
(Notice that this doesn't weaken the importance of fixing its current
implementation bugs, which we are actively analyzing right now. The
point is that the API is good, regardless of the quality of the
underlying code at any given time).
Now if there's one practical situation that it doesn't handle, or does
awkwardly, please describe it.
This may result in another attempt to push the envelope, and we both
enjoy that :-)
-Alex
Fredderic
Alexandre Ferrieux <alexandre...@gmail.com> wrote:
> On Mar 30, 5:00 am, Fredderic <my-name-h...@excite.com> wrote:
>> My choice, would be to split [fcopy] in half. An engine that
>> invokes a callback when there's both data waiting and somewhere to
>> put it (output buffers contain less than a block of data, for
>> example), and a mechanism for transferring up to a certain amount
>> of data from one channel to another with minimal memory copying:
>> fcopy infile outfile -onready command
>> fcopy infile outfile -single ?-size size?
>> The -onready option would be compatible with -command, though
>> probably not -size (although having -size pre-load the input buffer
>> when used in conjunction with -onready might offer some benefit, or
>> at least make [chan pending] more useful). In essence, the default
>> -onready command would simply invoke one execution of [fcopy]'s
>> existing data transfer code, which is exactly what the -single
>> option does. I haven't looked at the source to [fcopy], but I
>> wouldn't imagine it to be a particularly complex change.
> Hmmm... I think you've missed one tiny bit about [fcopy -size]'s
> semantics :-)
> It does exactly what you call "-once mode".
No, it doesn't. Let me explain, with an example (VERY quick-and-dirty):
# test1.tcl:
set fid [socket localhost 9988]
puts $fid 12345678912345
flush $fid
after 10000
puts $fid 12345678912345
flush $fid
after 10000
close $fid
# test2.tcl
set fid [socket -server accept 9988]
proc accept {fid addr port} {
puts "Accepting $fid from $addr at $port"
fcopy $fid stderr -size 20 -command [list lappend ::forever]
puts "Waiting ..."
}
vwait forever
And the output...
(waiting for connection)
Accepting sock6 from 127.0.0.1 at 57498
Waiting ...
12345678912345
(waiting for more input)
12345
(finishes as expected)
That's only a crude example, and there may well be some interplay with
blocking and buffering settings that I'm not aware of. But based on
that, my reading of the doc's, and a moderate look over the source
code, I still stand by my original message.
That "waiting for more input" is the problem that prevents accurate
rate and progress reporting. And as mentioned in my original post,
you can't even observe the progress of the [fcopy] from outside (ie.
some kind of periodic event).
> Thus, today's fcopy offers in a single "primitive" (in the sense that
> it cannot be emulated at full efficiency in pure Tcl,
NOTHING that is written in decent quality C (or just about any other
optimised compiled language) can be "emulated at full efficiency in pure
Tcl". You know better than to use marketing phrases like that.
Anyhow... Shame on you.
> - progress notification with configurable granularity
Yes, granularity is configurable. But it means trading off that
efficiency you mentioned for accuracy. That is what I'm getting at.
> - bandwidth throttling with a simple [after] in the callback
Yes, I know the way it works. It's quite good enough for basic
bandwidth throttling. Again [fcopy] is fine as long as it's being used
as a background type operation.
The thing I'm getting at, is if you can tell [fcopy] to move whatever's
waiting (optionally limited by the -size value) and then return, you get
all the efficiency, plus a measure of control simply unavailable.
Think of progress reporting. By default, [fcopy] seems to move data in
4096 byte chunks. When data is really rocking, that's a good thing;
trying to move a couple MB in 20-byte chunks is going to have a bit of
an impact. And setting the -size limit higher causes great long stalls
when the traffic isn't moving so fast. Progress reporting on a telnet
session in 4096 byte chunks isn't going to a whole lot better.
The division of functionality I mentioned should have negligible impact
on efficiency, have absolutely no impact on existing usage, but make
using [fcopy] under unpredictable circumstances simple.
> - all of this with the guarantee of never accumulating data in
> memory if the input is faster than the output (a mistake that is easy
> to make when doing it by hand)
Hence the reason I mentioned that issue several times in my original
post.
> The above doesn't strike me as "seriously bodgy". I see it rather as a
> well-designed compromise. (Notice that this doesn't weaken the
> importance of fixing its current implementation bugs, which we are
> actively analyzing right now. The point is that the API is good,
> regardless of the quality of the underlying code at any given time).
You'll fit in just fine... There's that lack of imagination again. ;)
Although I can't post over there from the address I signed up with, I
have been watching the progress with some degree of interest. The API
is good, as you say, within the limits of how it's presented. My
assertion is that the code is almost all there to do a whole lot more.
From what I can tell, most of what would be required to realise my
idea is to re-organse the CopyData() (and perhaps one of two of its
associated functions) to factor out the translate-and-transfer, and
minimise buffering outside of TCL's existing IO system. Handling of
the extra functions, and making sure you don't get excessive -onready
invocations, is all that would be needed to round it out.
Ans as I also pointed out, the functionality present in [fcopy],
without the actual copying of data, is itself rather useful. The
-onready functionality is where most of the code in my response was
aimed. Just look at the code that makes up [fcopy]. You've got a
chunk of argument shaping and validation, a chunk of asynchronous flow
control, and then this tiny little bit of task-specific code
responsible for the actual data transfer. Most people think of [fcopy]
as a function for moving data from A to B. So why, then, if that's all
it is, is that function such a small part of the code involved?
> Now if there's one practical situation that it doesn't handle, or does
> awkwardly, please describe it. This may result in another attempt to
> push the envelope, and we both enjoy that :-)
I believe I already did. Accurate timely progress reporting. [fcopy]
simply doesn't support it in a generically applicable fashion. Plus,
if you can do it in response to data being moved, it allows you to do
your calculations and what-not so that next time through the main loop
the data can be shipped out to its destination, rather than being
broken up into little chunks, fiddled across bit by bit, causing
excessive updates and what-not, re-accumulated, and then at some stage
sent off.
But besides what it doesn't do, is what it does. I've had to emulate
the asynchonous functionality already present in [fcopy] in several
projects (three that comes to mind), and have come across others that
have done it themselves.
Ultimately, I think [fcopy]s asynchronous functionality should be moved
into [fileevent] or something similar, and if it's really that
significant, its data transfer functionality highlights a place of
interest in optimising the existing IO handling; for example, perhaps
[read] with non-blocking mode could simply take the input channel's data
buffer (maybe after seeing if it can soak any more data out of the
system buffers) and return the buffer itself as a bytearray until
something decides to convert it into a "string", or hook it onto another
channel's output buffer, thus allowing a standard TCL script [read] and
[puts] to achieve the same data transfer efficiency as [fcopy]. (No
idea if that would actually work, through. ;) )
fileevent $in through $out puts $out -fromchan $in
Now, if that was all that was needed to achieve functionality as good
as the basic [fcopy] command, that would be impressive.
Fredderic
Alexandre Ferrieux
>
> - Show quoted text -
Sorry, you're overflowing my input buffer by a factor of several
thousands.
Anyhow one request I think I groked is something like
fcopy -update updateproc -command doneproc
which would call updateproc after each r/w roundtrip, hence be both
accurate for slow transfers and of minimal overhead for fast ones.
If *this* is one thing you'd want, then please say so (but if the
answer is "not exactly" and demands 3 pages of explanations, I'll be
lost again). Then you might post it as a feature request on the
tracker, and I might even send you a patch.
-Alex
Fredderic
That's the important part. Most of the rest is explaining that better.
Regaining control between the lines of output in that example, rather
than at arbitrary intervals, is really what started this. (Yes that
would be solved by the somewhat minimal -update suggestion.)
>> From what I can tell, most of what would be required to realise my
>> idea is to re-organse the CopyData() (and perhaps one of two of its
>> associated functions) to factor out the translate-and-transfer, and
>> minimise buffering outside of TCL's existing IO system. Handling of
>> the extra functions, and making sure you don't get excessive
>> -onready invocations, is all that would be needed to round it out.
Okay, in short... As short and direct as I can make it...
I'd have to look at it again, but it looks as though input data is read
into a private buffer, and translated from that private buffer to the
output channel (with the exception of binary-binary transfer, where the
data seems to be read directly into a buffer provided from somewhere
else). If, instead of using a private buffer, you pulled data into the
input channels buffer (as I think [gets] does if it can't find a whole
line), and then pulled out the translate and transfer code (which from
memory is mostly just an invocation of DoWrite() and a bit of buffer
cleanup, anyhow) to a distinct function, than it's almost done.
My [fcopy -single] suggestion invokes that function to move up to -size
bytes (defaulting to "all available") from A to B, and [fcopy -onready]
changes the default of invoking that function, to invoking something
user-supplied instead. That's all there is to it.
Two quite powerful distinct functions where there used to be just one
limited function with ugly workarounds. But none the less, the default
behaviour will still be exactly the same as it was.
The rest is some conversation, and further discussion on that;
>> And as I also pointed out, the functionality present in [fcopy],
>> without the actual copying of data, is itself rather useful. The
>> -onready functionality is where most of the code in my response was
>> aimed. Just look at the code that makes up [fcopy]. You've got a
>> chunk of argument shaping and validation, a chunk of asynchronous
>> flow control, and then this tiny little bit of task-specific code
>> responsible for the actual data transfer. Most people think of
>> [fcopy] as a function for moving data from A to B. So why, then,
>> if that's all it is, is that function such a small part of the code
>> involved?
> Sorry, you're overflowing my input buffer by a factor of several
> thousands.
Yeah, I get that trying to follow TCL's internals. I'm used to my own
coding style, and with my crappy memory it's often quicker for me to
re-write a piece of code than figure out how someone else's code
actually works (I was never any good at disassembly). That's why
people like you with the patience and memory to dig are so needed.
The gist is that this suggestion isn't just a pie in the sky idea being
put forth blindly in the hope of "pushing the envelope" (I do that a
lot also, just to try and get people thinking out of the quick-and-easy
box). It's actually, as far as I can tell, very much workable and
realistic, and very much better than the quick-and-easy suggestion you
mentioned (which is well and truly stuck within that box).
> Anyhow one request I think I groked is something like
> fcopy -update updateproc -command doneproc
> which would call updateproc after each r/w roundtrip, hence be both
> accurate for slow transfers and of minimal overhead for fast ones.
heh That'll do, if there's no interest in reworking the code to bring
out more complete functionality. I still won't be using it, but I'm
sure others will find it helpful.
But then, -update is probably a slightly better name for my suggestion,
anyhow. Another attempt to put it simply... Instead of -update simply
notifying you of what's been done, have it invoke a command to actually
do it, meaning you can do other stuff instead (some other script-based
heavy translation work) without having to fiddle with multiple
fileevent handlers and stopping the process if the output buffer is
filling up, or in addition to the copy (like include a block header
before the copied data).
> If *this* is one thing you'd want, then please say so (but if the
> answer is "not exactly" and demands 3 pages of explanations, I'll be
> lost again). Then you might post it as a feature request on the
> tracker, and I might even send you a patch.
Actually, it wasn't explained once in three pages, but more like three
times in three pages, with a bit of discussion in between.
The "pushing the envelope" tail to the message actually defines the
issue fairly clearly, I think...
>> Ultimately, I think [fcopy]s asynchronous functionality should be
>> moved into [fileevent] or something similar, and if it's really that
>> significant, its data transfer functionality highlights a place of
>> interest in optimising the existing IO handling;
There you go. In a nutshell, again. [fcopy] is not an operation that
exists on its own merits, as was suggested. It is a combination of two
distinct useful functionalities, which I personally would like to see
brought out either separately, or in the short term at least, through
new options to [fcopy] (since that name does still represent most uses
for either function reasonably well).
The rest was just my considered and re-considered thoughts for how best
to approach the issue.
Is one of those explanations simple enough to grok? If not I'll give
up and happily let a half-solution get passed off as sufficient. I'm
sure some people will think so, and many others won't notice the
difference either.
Fredderic
Alexandre Ferrieux
>
> Is one of those explanations simple enough to grok? If not I'll give
> up and happily let a half-solution get passed off as sufficient. I'm
> sure some people will think so, and many others won't notice the
> difference either.
I am facing a dilemma here. I love simplicity, and I fear that your
proposal drives us to unseen complexity levels (including the support
needed on this newsgroup when a newbie runs into your new API). At the
same time, I do feel that you're trying to draw attention on something
interesting outside the envelope; I also do understand individually
some of the refactoring efforts that you are putting forth, but I lack
the bug picture.
(See, I'm beginning to write very long sentences, it must be
contagious ;-)
In yet other words: can you concentrate on the problem rather than the
solution ?
(Disclaimer: again I'm not saying fcopy is flawless. A first example
is the [fcopy -command] bug 780533, which we solved a few hours ago.
Another one that's even scarier to me is the mix of sync and async
mode in fcopy's async mode: there is always a first immediate attempt
at transfer (if nonblocking) even with -command)
I'm merely asking for the "refactoring" of your request in the above
terms for my own understanding.
-Alex