TCL C sockets and buffering

[Ralf Fassel]

I have a problem on a single computer with a TCL socket connection not
properly sending out data. OS is Windows 10, Tcl 8.5.19.

The socket is created in c++ code, and works on all computers except
this single one. The processing is as follows:

- open the socket to remote, synchronously
- configure socket "-blocking false -buffersize 1048576 -translation binary -buffering none"
- send a single byte via Tcl_Write()
- block for 100ms via usleep(), i.e. give the remote side a chance to
process the byte and send the response.
- loop for 2 seconds around Tcl_Read() and usleep(1000) waiting for the
response. On a working connection, the response is received within
5ms in this step, very often already after 1ms (i.e. with the first
Tcl_Read()).

The failure symptom on this single computer is that after establishing
the connection and sending the single byte, this single byte is not seen
on the wire (monitored by wireshark) until the connection is broken down
after waiting 2 seconds for a response from the remote side.


Stripped down code:

device = Tcl_OpenTcpClient(0, port, ip, 0, 0, 0);
// ensure device non-NULL omitted here
if (Tcl_SetChannelOption(0, device, "-blocking", "false") != TCL_OK
|| Tcl_SetChannelOption(0, device, "-buffersize", 1048576) != TCL_OK
|| Tcl_SetChannelOption(0, device, "-translation", "binary") != TCL_OK
|| Tcl_SetChannelOption(0, device, "-buffering", "none") != TCL_OK) {
Tcl_Close(0, device);
throw std::runtime_error("cant set channel to nonblocking mode");
}
// len = 1; buf = "O";
send_bytes(len = 1; buf = "O"; bool flush) {
while (len > 0) {
int count = Tcl_Write(device, (char*)(buf), len);
if (count < 0) {
// error handling omitted here
throw ...
break;
}
len -= count;
}
// makes no difference whether flush or !flush
if (flush) {
// xxx
Tcl_SetChannelOption(0, device, "-blocking", "true");
if (TCL_OK != Tcl_Flush(device)) {
warning("Tcl_Flush failed on channel %p", device);
}
Tcl_SetChannelOption(0, device, "-blocking", "false");
}
return len == 0;
}
// ensure send_bytes returned TRUE omitted here

// waitms = 100, usleep() waits in Microseconds
usleep(waitms*1000);

// wait at most 2 secs for an answer
do {
Tcl_DStringSetLength(line, 0);
num = Tcl_Gets(device, line);
if (num < 0) {
if (!Tcl_InputBlocked(device)) {
// error handling omitted
close/return -1
}
}
if (num < 0) {
// wait a bit
usleep(1000);
continue;
}
// handling of line omitted here
} while (elapsed_ms() <= 2000);

if (nothing_was_read) {
(1) send another byte "q"
raise an error which finally closes the conncetion
}

On this single computer somewhere around (1) do I see the O and the q on
the wire via wireshark. I can't say whether this is at the sending of
'q' or at the closing of the channel. Then the response to O comes
back, and then the connection is terminated.

I also temporarily changed the -blocking to true and called Tcl_Flush()
directly after the Tcl_Write(), but this did not change anything.

Note that I do not enter the TCL event loop here, but block in C code.
Does a nonblocking channel require the event loop to send out a single
byte of data? I would have thought a non-buffering channel would take
care of that, or at the latest the Tcl_Flush().

Question: how is it possible that the OS sends out the byte only when
a second byte is sent or the connection is finally broken down?
Can the network card be set to some weird buffering mode? How?

Thanks
R'

[two...@gmail.com]

On Wednesday, May 22, 2019 at 7:56:16 AM UTC-7, Ralf Fassel wrote:

> Question: how is it possible that the OS sends out the byte only when
> a second byte is sent or the connection is finally broken down?

Could this be the Nagle algorithm kicking in? There's the TCP_NODELAY
socket option to disable that, but 2 full seconds seems a bit long.

The Nagle algorithm was implemented to deal with small writes, such
as the single bytes you might see using telnet. Might be what your
seeing here.

What are the other computer OS's that don't see this?

[Ralf Fassel]

* two...@gmail.com

Probably... will have to read up on this.

| What are the other computer OS's that don't see this?

Same configuration: all Windows-10, tcl8.5.19. Type of network card is
yet unknown on the 'wrong' computer, will have to check that tomorrow.

TNX
R'

[two...@gmail.com]

On Wednesday, May 22, 2019 at 10:22:23 AM UTC-7, Ralf Fassel wrote:

> Probably... will have to read up on this.
>

You can apparently disable this on windows in the registry, see

https://www.optimizemswindows.com/disable-nagle-algorithm-to-increase-your-internet-speed-for-quick-response/

[Donal K. Fellows]

On 22/05/2019 15:56, Ralf Fassel wrote:
> - configure socket "-blocking false -buffersize 1048576 -translation binary -buffering none"

The relevant bit there is “-buffering none” as that says to Tcl to send
everything it has to the OS immediately. It's up to the OS to decide
what to do after that. I'm guessing you want TIP #344 to help out (which
I think is something that Jan Nijtmans is interested in sorting out
when his busy schedule of working on Tcl 9 permits). Also, bear in mind
that with a round trip, you can get problems from either side's
buffering; check the other side's code too. :-)

https://core.tcl-lang.org/tips/doc/trunk/tip/344.md

--
Donal Fellows — Tcl user, Tcl maintainer, TIP editor.

[Ralf Fassel]

* "Donal K. Fellows" <donal.k...@manchester.ac.uk>

Donal, thanks for the pointer.

I somehow doubt that Nagle is involved here, since:

- the network is exclusively used between the remote and the PC, no
other traffic (confirmed by wireshark listening on the network card)

- the one byte to be sent is the very first byte on that connection
after establishing the connection. There has been traffic of some
bytes (less than 100) on other sockets, and some more ports have been
opened with no traffic.
If I understand Nagle correctly, it should kick in only after the
network becomes saturated, i.e. new data is to be sent when the ACK
for previously sent data has not yet arrived. As this is the first
data to be sent, there can be no ACK for previous data.

- when the byte is finally sent, it is sent _single_, not with the other
byte following it. Both bytes go in their own TCP packets (confirmed
by wireshark). With Nagle kicking in, I would expect the two bytes go
in one TCP packet.

- when the byte is finally sent (according to wireshark), the answer
from the other side _always_ arrives within less than 2ms, so I don't
think there is a problem there.

In the meantime I have changed the software that whenever a single byte
is to be sent, I attach a newline to it, and this seems to work - the
data is sent out immediately (the remote can cope with that additional
\n). Yes, you think 'line-buffering', but other two-byte commands work
without \n and without explicit flush, and after every write I check
that Tcl_OutputBuffered() == 0. Always true.

I vaguely remember that in the past with a serial device, I had to
switch to full buffering with manual flush on every write on Windows to
make it reliably work (reason still unknown). Could not yet test that
approach on that machine with that network socket (though I had tested
flushing with buffering 'none', which did not make a difference).

TNX so far!
R'

[tombert]

If it's Nagles then try my distribution:
https://bitbucket.org/tombert/tcltk/downloads/

Thread topic:
https://groups.google.com/forum/#!topic/comp.lang.tcl/xtVodiKNr0I

use it:
chan configure <socket> -buffering none -nodelay 1

[Ralf Fassel]

* tombert <tomber...@live.at>

| If it's Nagles then try my distribution:
| https://bitbucket.org/tombert/tcltk/downloads/
>
| Thread topic:
| https://groups.google.com/forum/#!topic/comp.lang.tcl/xtVodiKNr0I

Interesting, especially the discussion about refactoring the socket
code. Maybe... well, first things first.

Next week I will hopefully get a chance to do more checks (the single
computer where this is a problem is at a remote location).

Since I'm currently seeing this on Windows only, I have BFI-hacked my
TCL and enabled the TCL_FEATURE_KEEPALIVE_NAGLE which Harald mentioned
in the other message, so I can check -nagle 0/1.

In addition, I'll try to see whether buffering-full-flush-after-each-write
will make a difference (had seen this on a serial device, though there
the problem was that the "\n" was sent separately from the content when
"buffering line" was enabled).

Stay tuned...
R'

[Gerald Lester]

8.5 is rather old -- any chance to upgrade and verify it still happens
with something modern?


--
+----------------------------------------------------------------------+
| Gerald W. Lester, President, KNG Consulting LLC |
| Email: Gerald...@kng-consulting.net |
+----------------------------------------------------------------------+

[Harald Oehlmann]

Ralf,

sounds weired. I can comment on this:

Am 22.05.2019 um 16:56 schrieb Ralf Fassel:
> Note that I do not enter the TCL event loop here, but block in C code.
> Does a nonblocking channel require the event loop to send out a single
> byte of data? I would have thought a non-buffering channel would take
> care of that, or at the latest the Tcl_Flush().

The event loop is only required for fileevent and server sockets.
If there is an issue with non-blocking I/O without event loop, it is a
bug. flush is always blocking, like close.

An idea is to run the test suite on this computer.

Harald

[Harald Oehlmann]

Am 25.05.2019 um 02:46 schrieb Gerald Lester:
> 8.5 is rather old -- any chance to upgrade and verify it still happens
> with something modern?

I can confirm, that all bug-fixes in 8.6 were back-ported to 8.5.
If you 8.5 is recent, there should be no difference on the bug level.

Nevertheless, the feature "IP6" is missing.

Enjoy,
Harald

[Ralf Fassel]

* Gerald Lester <Gerald...@KnG-Consulting.net>

| On 5/22/19 12:22 PM, Ralf Fassel wrote:
| > Same configuration: all Windows-10, tcl8.5.19. Type of network card is
| > yet unknown on the 'wrong' computer, will have to check that tomorrow.
>
| 8.5 is rather old -- any chance to upgrade and verify it still happens
| with something modern?

Not in the current timeline :-/
I'm still having issues with Tk-Events scheduled in a different fashion
in 8.6 compared to 8.5, and sorting this out in 20+ year old code from
the time when we started to use TCL 8.3 is somewhat tedious...
Altough... the code part relevant here should not be affected, so it
might be worth a try...
Thanks for the idea!
R'

[Ralf Fassel]

* two...@gmail.com
| 1. It seems from your post that your network topology is that
| only the one system is remote. Can we assume the others are
| all local, on the same lan? If so, if would be fascinating to
| see if moving that remote machine onto the same lan with the
| other systems that don't have the problem would then clear up
| this one issue. Probably too much work though :(

Should have been more specific in my original message...

'Remote' is not really 'remote' in the sense of WAN. It is a PC talking
to an embedded device running Busybox-Linux. The two are directly
connected via Ethernet via one cable, only these two share this
connection/cable. So 'remote' is only to denote "not on the same computer".

When I say "other machines", it means this embedded device directly
connected to another computer, Windows or Linux. The connection is
always "exclusive cable use" :-)

| 3. I misread the 2 second timeout data point. I now see that 2s
| is built into your test code with
>
| while (elapsed_ms() <= 2000)
>
| So, what if that were longer, say 5000, or
| even longer. Does it totally deadlock or would it eventually
| continue? That at least should be a simple test.

Easy to do, indeed, will provide a means to specify the timeout in the
tests scheduled later today. I do not expect to make that any difference.

| In poking around the internet for "slow" and "one byte" and "windows"
| one finds many discussions which make it sound like windows has some
| undocumented code along with the nagle algorithm. One thing keeps popping
| up is a mention of delayed acks. However, they are only supposed to be
| 200 ms. Probably just a red herring, but curious.

Yes, I've seen these 200ms articles, too. But I don't see a 200ms delay
on any other Windows machine. The 'O' is sent, I wait 100ms, and in the
next Tcl_Gets(), the answer is there.

set fd [socket 192.168.2.1 65422]
fconfigure $fd -buffering none -blocking 0 -encoding binary
fileevent $fd readable [list read_answer $fd]
proc read_answer {fd} {
set now [clock milliseconds]
set line [gets $fd]
if {$line ne ""} {
puts "LINE {$line} [expr {$now-$::start}]ms"
}
if {[eof $fd]} {
close $fd
}
}
puts -nonewline $fd "O"
set start [clock milliseconds]
vwait forever

Roundtrip time on Linux with no waiting at all after the 'O':
LINE {v3 ... } 2ms
LINE {v3 ... } 2ms
LINE {v3 ... } 2ms
LINE {v3 ... } 2ms
Windows:
LINE {v3 ... } 2ms
LINE {v3 ... } 3ms
LINE {v3 ... } 2ms
LINE {v3 ... } 2ms


| The bottom line for me is that it only occurs on 1 windows system, so that
| itself would seem to rule out the nagle algorithm, unless there's something
| happening on the hops between this system and your other systems. I think
| getting to what's different is probably the key.

As said before, Nagle should kick in only for the 2nd ff transmissions
on one socket. Since the single byte is the very first byte on that
connection, it should not be Nagle.

More tests follow this afternoon CEST, stay tuned...

R'

[two...@gmail.com]

On Monday, May 27, 2019 at 3:09:19 AM UTC-7, Ralf Fassel wrote:

>
> 'Remote' is not really 'remote' in the sense of WAN. It is a PC talking
> to an embedded device running Busybox-Linux. The two are directly
> connected via Ethernet via one cable, only these two share this
> connection/cable.

Still unsure of your setup. Maybe a diagram? Do you mean you are
moving the ethernet cable from embedded to the several windows
systems one at a time? And only on one of these do you see a problem?

I did a little tinkering myself at script level with python and tcl.

The client sends 6 single byte packets and each server echoes
them back, which the client then reads. I put a few delays (not shown)
here and there and still the same result.

Python was just to be able to rule out the tcl library in the server.
I tried this on the same machine and between 2 windows systems
with a switch in between them. And also a ubuntu linux in a vmware
vm to the same machine running win10.

Same "script level" results for all tests.

Did see differences in wireshark when running between the linux vm
and windows.

On win10 - win10 each single byte were in their own packet.
No delays or timeouts.
No evidence of any nagle action.
Wireshark nice and clean, see each 1 byte packet in both directions.

On the linux vm to windows wireshark results includes lots of tcp out-of-order,
dup ack, [PSH ACK] etc. I run this linux vm in bridge mode, probably a vmware
issue. Still, scripts don't see any of that, and produce same final results.


#-------------- python server
import socket

def server_program():
host = "192.168.1.75"
port = 5000

server_socket = socket.socket()
server_socket.bind((host, port))
print("host " + str(host) + " port " + str(port))
server_socket.listen(1)
while True:
conn, address = server_socket.accept()
print("Connection from: " + str(address))
while True:
data = conn.recv(1).decode()
if not data:
break
print("read /" + str(data) + "/")
conn.send(data.encode())
print("sent /" + str(data) + "/")
conn.close()
print("closed")

if __name__ == '__main__':
server_program()


#-------------- tcl server


proc accept {chan addr port} {
set n -1
fconfigure $chan -translation binary -buffering none -buffersize 1
puts "accept $addr:$port"
while { true } {
set c [read $chan 1]
puts "[incr n] read /$c/"
if { [eof $chan]} {
break
}
puts -nonewline $chan $c
}
puts "close"
close $chan
}
socket -server accept -myaddr 192.168.1.75 5000 ;# force ipv4 only
puts "started server on 5000"
vwait forever

#-------------- tcl client


set chan [socket 192.168.1.75 5000]
puts "chan= |$chan| "
fconfigure $chan -translation binary -buffering none -buffersize 1
for {set n 0} {$n <= 5} {incr n} {
set c [string index "abcdefghijklmnop" $n]
puts -nonewline $chan $c
flush $chan
puts "$n sent /$c/"

set in [read $chan 1]
puts "$n read /$in/ len=[string length $in]"
}
puts "close"
close $chan
after 5000 exit

[Ralf Fassel]

* two...@gmail.com

| On Monday, May 27, 2019 at 3:09:19 AM UTC-7, Ralf Fassel wrote:
| >
| > 'Remote' is not really 'remote' in the sense of WAN. It is a PC talking
| > to an embedded device running Busybox-Linux. The two are directly
| > connected via Ethernet via one cable, only these two share this
| > connection/cable.
>
| Still unsure of your setup. Maybe a diagram?

[PC - Ethernet-RJ45 100MBit/s] - Ethernet-Cable - [100MBit/s Ethernet-RJ45 - Embedded-Device]

Nothing magic here, the Embedded-Device is the DHCP server for that
Network. I checked in Network Manager that there is nothing fancy
enabled in the network settings on that Ethernet Connector (as far as I
can tell).

| Do you mean you are moving the ethernet cable from embedded to the
| several windows systems one at a time? And only on one of these do you
| see a problem?

Exactly. The Embedded can only be connected to one PC at a time, and
only this one PC has the problem.

Here are now the testing results:

- tcl 8.6: our program crashes on startup, reason unclear, probably some
system DLL missing, so unfortunately I can't test that.

- tcl 8.5.19:
Failures:
- patched for -nagle 0/1
(with stderr message added in the TCL library where setsockopt() is
called in tclWinSock.c)
=> made no difference whether TCP_NODELAY was enabled or not, the
one-byte message was not sent on Tcl_Write()
- waiting 5000ms instead 2000ms
=> made no difference, the one-byte messages was not sent on Tcl_Write()
- full-buffering with explicit Tcl_Flush() after Tcl_Write(), with
channel set to either blocking or non-blocking mode:
=> made no difference, the one-byte messages was not sent on Tcl_Write()

Success:
- always send at least two bytes, add \n if only one byte is to be sent
=> always works on several tries, response is there after a few milliseconds
It is not important that the second byte is a newline, since other
2-byte commands like "i1" are also sent immediately.

Identical behaviour on a built-in Ethernet-RJ45-Adapter and an USB-Ethernet
Adapter, so one can rule out the driver of a specific network adapter.

I have no idea about 'why' (and also no further idea how to find out),
but I know now 'how' to make our software work around that specific
problem.

Thanks for your interest and contributions :-)
R'