data lost on uploading to a tcl network server

[Michael Niehren]

Hi together,

i'm running into an problem of uploading data to a tcl server. You see the code below.
The client takes a filename as input, read the data, encode it to base64 and send it
to the server. If i take small files (< 1 MB) it work's fine, but if the files are
greater than 1 MB (i did not check the real limit), some data is missing at the server
side. The greater the files are, the greater are the missing data.

Has someone an idea, where my fault is ?

best regards
Michael


Here is the simple server code:
-----------------------------
#!/usr/bin/tclsh

set clientdata(data) {}
proc serverOpen {sock ip port} {
global clientdata

set clientdata(data) {}
chan configure $sock -buffering line -encoding utf-8 -blocking 0 -translation crlf
chan event $sock readable [list readLine $sock]
}

proc readLine {sock} {
global clientdata

set n [gets $sock line]
lappend clientdata(data) $line
if {$n > 0} {
#puts "read: $line"
} else {
if {[chan eof $sock]} {
puts "Closed: [string length [join $clientdata(data) "\n"]] Bytes"
close $sock
}
}
}

set server [socket -server serverOpen 4711]
vwait forever
-----------------------------

and also the client code:
-----------------------------
#!/usr/bin/tclsh

package require Trf

proc encode_file_to_base64 {filename} {
set file [open $filename r]
fconfigure $file -translation binary
set data [read $file]
close $file
set erg [base64 -mode encode -- $data]
return $erg
}

set filename [lindex $argv 0]
set data [encode_file_to_base64 $filename]
puts "[string length $data]"

set sock [socket -async 192.168.69.1 4711]
fileevent $sock w {
set connected ok
}
vwait connected
fileevent $sock w {}
chan configure $sock -buffering line -encoding utf-8 -blocking 0 -translation crlf

puts -nonewline $sock $data
flush $sock
close $sock
-----------------------------

[Rich]

Michael Niehren <mic...@niehren.de> wrote:
> Hi together,
>
> i'm running into an problem of uploading data to a tcl server. You
> see the code below. The client takes a filename as input, read the
> data, encode it to base64 and send it to the server. If i take small
> files (< 1 MB) it work's fine, but if the files are greater than 1 MB
> (i did not check the real limit), some data is missing at the server
> side. The greater the files are, the greater are the missing data.
>
> Has someone an idea, where my fault is ?
>

> proc readLine {sock} {
> global clientdata

First problem I see. You are using a single global for handling data
being read in an event driven manner. As long as you always only ever
have a single sender, you'll be ok here. The moment you have two (or
more) senders at the same time, this will fail miserably.

> chan configure $sock -buffering line -encoding utf-8 -blocking 0
> -translation crlf
>
> puts -nonewline $sock $data
> flush $sock
> close $sock

The one item I see is you set the channel to non-blocking, the puts the
data (so puts returns immediately, then you flush and close, and then
you exit the sending script.

I suspect that the exiting of the sending script is why you are seeing
larger amounts of missing data the larger the file is that you try to
send. The flush (for sockets) is not "data has been transmitted to
server and acknowledged" but instead is "data has been turned over to
the I/O subsystem for transmission". When you then exit the script,
whatever data is not yet transmitted over the TCP channel never gets
transmitted (because the script ended, so all the OS I/O buffers
allocated to the process also get dropped).

Try adding a "vwait forever" at the end of the sender and see if the
issue with missing data goes away. If it does, then that was the
cause.

[jda...@gmail.com]

Based on the 8.6 manual page:

putting: fconfigure $sock -blocking 1
before the: close $sock

or

setting the environment variable TCL_FLUSH_NONBLOCKING_ON_EXIT
to anon-zero value


should cause TCL to wait for all the data to actually be sent before shutting down the socket.


Dave B

[Michael Niehren]

Hi Dave,

that was it. Many thank's for your help. I was not able to find it
out myself.

best regards
Michael


Am 27.08.19 um 20:42 schrieb jda...@gmail.com:

[Michael Niehren]

is there a way to see whether all data has been send on an non-blocking
channel ?
I do an puts, after that the flush and now i want to know, if all data
has been sended without blocking the channel. Is that possible ?

Michael

Am 27.08.19 um 20:42 schrieb jda...@gmail.com:

[Donal K. Fellows]

On 28/08/2019 07:28, Michael Niehren wrote:
> I do an puts, after that the flush and now i want to know, if all data
> has been sended without blocking the channel. Is that possible ?

The [chan pending] command is what you're after.

if {[chan pending output $sock] > 0} {
# There are still bytes waiting in Tcl's buffers
}

The OS may additionally buffer things, but that's less under control and
it's supposed to handle that correctly as part of the closing process.
It might be an idea before closing (when you suspect this is a problem
and yet think there's no data waiting) to put the socket in blocking
mode before doing the [close].

It's probably a good idea to try to structure the higher-level protocol
and its usage so that a socket closing early isn't fatal since there's
literally no way to stop it from happening. Such structuring might be
done by putting something in to allow the parties to validate whether a
transfer is right (and to resume it if it is short). That's all stuff
that isn't done at the ship-this-stream-of-bytes level, and requires
knowing the use cases a lot better.

Donal.
--
Donal Fellows — Tcl user, Tcl maintainer, TIP editor.