nonblocking tcltls example and question -- am I doing something wrong?

[currel...@gmail.com]

Hello,

I am trying to upgrade Richard Hipp's ex-mua email client which is downloadable from a link here:

https://www.sqlite.org/cvstrac/wiki?p=ExperimentalMailUserAgent

to support POP3 over TLS (currently it just uses raw sockets). But I can't seem to get the basic conversion from plain sockets to TLS sockets to work. I have extracted the basic code that does the initial POP3 request into a simple example, and it seems to me that the combination of fconfigure -blocking 0 and fileevent readable is not working once the socket is going over TLS.

To recap, for POP3, the server is supposed to send a message like

+OK POP3 server ready <1896.69...@dbc.mtview.ca.us>

once the connection has been established in order to kick off the session (as shown in the example session here http://www.freesoft.org/CIE/RFC/1725/9.htm ).

This works great as long as I am using blocking sockets over TLS, as in the following example:

% package require tls
1.7.16
% set msock [::tls::socket pop.gmx.com 995]
sock80e8d3a5c90
% gets $msock
+OK POP server ready H migmx101 1Mn1eZ-1ehnSB0lzb-00k6yh

However, when I try to transition to using nonblocking sockets, as is done in Dr. Hipp's email client, I can no longer seem to get any output from the pop3 server at all.

% package require tls
1.7.16
% set msock [::tls::socket pop.gmx.com 995]
sock1e198bd34c90
% fconfigure $msock -blocking 0
% fileevent $msock readable [subst -nocommand {puts [gets $msock]}]
% gets $msock
%

it hangs indefinitely and never shows any output

I have tested with self-compiled tcltls 1.7.16 on OpenBSD 6.3, self-compiled tcltls 1.7.16 on Ubuntu 14.04, and the tcltls version included in the latest ActiveTCL 8.6.7.0 download on Windows 10.

Does the above nonblocking example work on anyone else's computer? If not, do any of you know how to fix the nonblocking example to make it work? Probably I am doing something very simple that is wrong...

Thanks very much in advance for any help

-- Currell

[Ashok]

Are you using tclsh or wish? If the former, do you have the eventloop
running via vwait or similar? Remember non-blocking I/O requires the
event loop to be active.

/Ashok

[currel...@gmail.com]

On Saturday, April 14, 2018 at 10:22:55 PM UTC-5, Ashok wrote:
> Are you using tclsh or wish? If the former, do you have the eventloop
> running via vwait or similar? Remember non-blocking I/O requires the
> event loop to be active.
>
> /Ashok
>

Hello, thank you for your help. I think I had tried both tclsh and wish, but I wasn't enabling the event loop in tclsh as you said.

Even with the event loop running however, I still can't get the initial +OK response to come from either of the two POP3 servers I am testing against ( pop.gmx.com , pop.gmail.com ) when I am using nonblocking sockets. Blocking sockets work fine.

Would you mind running my nonblocking example script and checking whether you see the initial prompt from the server? I modified my test case a bit in response to your comment and posted it below.


package require tls

set msock [::tls::socket pop.gmx.com 995]

fconfigure $msock -blocking 0

fileevent $msock readable [subst -nocommand {puts [gets $msock]}]

vwait forever

I expect to see a +OK line printed out to the screen, as happens for my blocking example, but at least on my system I never do but instead get a blank line printed out. Any further ideas for things to change?

[currel...@gmail.com]

I figured out a way to get it working! (at least this simple example, still working on getting the email client to work with TLS).

I think the event loop is getting "stuck" here for some reason and it kind of needs a kick in the pants to get moving again. The following example works for me (including the tclsh output to help show what's going on).


$ tclsh8.6
% package require tls

set msock [::tls::socket pop.gmx.com 995]
fconfigure $msock -blocking 0

fileevent $msock readable [subst -nocommand {puts "fh: [gets $msock]"}]
after 1000 [subst -nocommand {flush $msock; puts "p2s:[gets $msock]:p2e"}]
vwait forever
1.7.16
% sock105564af5090
% % % after#0
% p2s::p2e
fh:
fh: +OK POP server ready H migmx029 0Lwegi-1eNW0K2tg0-016SZj

Thanks for your help Ashok,

-- Currell

[Keith Nash]

Hi Currell,

I found when using HTTPS that the TLS handshake does not occur until the
writable (puts) side of the channel is flushed (i.e. after the HTTPS
request). This is not a problem for HTTPS because the client writes before
reading. However, in POP3 it is the other way round, and so a flush will
probably have no effect, and you will wait for ever. (On reflection this
issue should be filed as a bug in TclTLS.)

You can tell when the handshake occurs by using the -command option to
tls::socket.

Can you send a NOOP and then [flush] the channel?

When reading back messages, you are likely to encounter another bug.

This second bug is somewhere in either TclTLS, Tcl stacked channels, the Tcl
[read] command, fileevent generation or buffering, and it causes stalling
when using [read] over TLS.

This bug is reported here:

https://sourceforge.net/p/tls/bugs/38/
http://core.tcl.tk/tcl/tktview/1945538fffffffffffff
https://core.tcl.tk/tcltls/tktview/581d50e6cdc97b0bb5f0e5516086ac469e077f04
https://core.tcl.tk/tcl/tktview/46b6edad51e645c70e88dd4f944a2d4afc63a96a

The first report includes an investigation of the problem in a PDF file.
The last report refers to a different bug, but includes a workaround for
this TLS issue. The workaround is included in a revised form of the http
library in the "bug-46b6edad51-concurrent-http" branch of Tcl, forked from
8.6.8.

In HTTP, the bug is manifested when fetching a resource of known length
using an unchunked transfer. The workaround is to ask for the exact number
of bytes that are known to remain (or more). Since POP3 tells you the exact
size of each message, you should be able to use a similar workaround.

This goes against the standard practice of [read chan numchars] where
numchars is a convenient size for processing. However, if numchars is ever
less than the number of bytes remaining, then TclTLS will stall, if not on
that [read] then on a later one. A non-blocking [read] does not wait for the
entire numchars to arrive: it will return with whatever input is available,
and may need to be called several times.

A third issue is empty reads when [fileevent] reports that input is
available. This problem is mentioned in the bug reports above.

I hope this helps.

Keith.

[currel...@gmail.com]

Hi Keith,

A belated thank you for your very helpful and informative reply. I was able to get the email client working adequately with TLS, at least for my purposes. A link to the patched sources is here: https://gitlab.com/vancan1ty/experimental-mua . (Note that this also includes a hack I did for my platform dealing with line-endings getting messed up on email download... I suspect but am not sure that the original was written for windows. I'm sure there's a cleaner way than what I did ;)).

Once I got the simple example above working with the "after 1000 [subst -nocommand {flush $msock; puts "p2s:[gets $msock]:p2e"}]" hack, I don't think I have ran into any more particular issues with TLS support for the email client. I have tested it with gmail and gmx mail, and it works fine for both. I'll keep the other pitfalls you listed in mind for the future.

-- Currell