HTTP/1.1 504 Gateway Timeout

[Alexandru]

Hi,

I'm using Twapi to initialize a TLS socket:

set tlssocket [http::register https 443 twapi::tls_socket]

See also https://groups.google.com/forum/#!msg/comp.lang.tcl/kUO34Yh5SZM

When I donwload large Files (>1GB) I often get this error:
HTTP 504 (Gateway Timeout)

Before using Twapi I never had this.

Google search relates this to Proxi time out problems. But I don't use a proxy.

Any help is apreciated. Thank you.

Regards
Alexandru

[Ashok]

On 6/22/2018 1:04 PM, Alexandru wrote:
> Hi,
>
> I'm using Twapi to initialize a TLS socket:
>
> set tlssocket [http::register https 443 twapi::tls_socket]
>
> See also https://groups.google.com/forum/#!msg/comp.lang.tcl/kUO34Yh5SZM
>
> When I donwload large Files (>1GB) I often get this error:
> HTTP 504 (Gateway Timeout)

You say "often". So it sometimes works and sometimes does not? When the
failure occurs, is it always after a specific amount of time or random?

>
> Before using Twapi I never had this.

What were you using before TWAPI? Check with curl a few times to see if
it always succeeds. Of course, anything is possible but I am a little
skeptical that it is a twapi tls issue. If it were, for example an error
in *re*negotiating TLS, you would get a socket error, not a properly
formed HTTP 504 response.

>
> Google search relates this to Proxi time out problems. But I don't use a proxy.
>

Even if you have not configured a proxy on the client, something like a
load balancer on the server side can still return that error if the back
end does not respond within a timeout.

> Any help is apreciated. Thank you.
>
> Regards
> Alexandru
>

Afraid not many ideas beyond this. If you have a URL I can try, perhaps
I might get further.

/Ashok

[Alexandru]

In the meanwhile I can reproduce the error with this file:
https://meshparts.de/download/test.bin

Can you test by downloading this file with Twapi and Tcl http?

Thank you!

[Ashok]

On 6/22/2018 4:19 PM, Alexandru wrote:> In the meanwhile I can reproduce

the error with this file:
> https://meshparts.de/download/test.bin
>
> Can you test by downloading this file with Twapi and Tcl http?
>
> Thank you!
>

That should help debug.

Will try over the weekend.

[Ashok]

Well, I have bad news and worse news.

First, I really doubt the 504 error you are seeing is caused by errors,
twapi or otherwise, from the client side. Looking at the code it is hard
to imagine how a client could trigger that error from the server
(assuming the server is coded properly to generate correct error codes!)

However, the worse news is that I saw different behaviour when I tried
it. At around 700MB memory footprint, tclsh silently exited with no
message or other error indication. To me this is often an indication of
memory corruption somewhere.

To remove twapi from the picture, I tried with the tls extension from
the BAWT distribution. This time I caught the memory exception in the
debugger. The stack showed the zlib library though of course with memory
corruption, it does not necessarily mean the bug is in that library.

So we are at the point where either both twapi and the tls extension
coincidentally have similar corruption bugs, or more likely imho Tcl
itself has an issue, likely with the zlib decompression.

I won't have time to look into this further right now but you might want
to see what happens if compression is disabled on the http connection.

/Ashok

[Alexandru]

Thanks for debugging. I tested again with zip conperssion turned off for the specific file type and see no effect. Firefoc can download the file correctly so it must be Tcl or Twapi.

[pal...@yahoo.com]

Alexandru,

On Sunday, June 24, 2018 at 7:17:18 PM UTC+5:30, Alexandru wrote:
> Thanks for debugging. I tested again with zip conperssion turned off for the specific file type and see no effect. Firefoc can download the file correctly so it must be Tcl or Twapi.

Again, I see no indication that twapi is involved in the issue you are having. But just to verify, I wrote the script below to do a raw http request (ie. without using the http package). The entire file was downloaded without errors with a 200 OK status code. Note that neither the http package, not Tcl's zip facilities are used.

The script (run in wish/tkcon):

proc httpreq {host relpath} {
global count

set count 0
destroy .l
pack [ttk::label .l -textvariable ::count]
set fd [file tempfile outfile]
puts "Writing to $outfile"
fconfigure $fd -translation binary


set so [twapi::tls_socket $host 443]
fconfigure $so -translation {binary crlf}
puts $so "GET $relpath HTTP/1.1"
puts $so "Host: $host"
puts $so "Connection: close"
puts $so ""
flush $so

set blksize 16384
while {1} {
set data [read $so $blksize]
set len [string length $data]
if {$len} {
puts -nonewline $fd $data
flush $fd
incr count $len
update ; # Update label display
}
if {$len < $blksize} break
}
close $fd
close $so
puts "Wrote $count bytes to $outfile."
}

Sample session:

% httpreq meshparts.de /download/test.bin
Writing to D:/temp/TCL8883.TMP
Wrote 1609433451 bytes to D:/temp/TCL8883.TMP.
% set fd [open D:/temp/TCL8883.TMP]
% gets $fd
HTTP/1.1 200 OK

At this point, I would say your 504 error comes either from a bug in the http package, or your not setting the request headers appropriately (maybe you can compare with what firefox is sending). Either way, unless there is some evidence twapi is at fault (noting again, it does not involved in the HTTP protocol or content at all other than transporting it), I won't pursue this matter further.

/Ashok

[Alexandru]

Many thanks for this code.
I trying to find out if it's the http package or the wrong header, I'm thinking, if the header was wrong, then I should also get a problem with other similar files, correct? The issue seems to be releated to the file size.