coroutine "::http::38EventCoroutine" is already running

[Alexandru]

Hi,

during a long download process in my Tcl/Tk app, I get the following error message without further information about the stack. So I don't know exactly where in the code the error happens:

coroutine "::http::38EventCoroutine" is already running
while executing
"::http::38EventCoroutine"

All I know, is that it happens if during the long download process the app executes other http requests.

Any idea what the reason could be? Is it an error in my code or an error in the http package?

Thanks!
Alexandru

[Harald Oehlmann]

I suppose, http package version would be helpful

[Alexandru]

Am Donnerstag, 14. November 2019 10:09:51 UTC+1 schrieb Harald Oehlmann:
> I suppose, http package version would be helpful

The Version of http is 2.9.0

[Ashok]

This error is in all likelihood due to the coroutine calling itself.

Some detail about coroutine use in your application would help. I don't
believe the http itself uses coroutines but it does do vwaits which may
cause reentrancy issues which is what this smells like to me.

Just hazarding a guess...

/Ashok

[Alexandru]

At least my code does not use coroutines. I don't even understand the concept, much less using it.

[Alexandru]

Am Donnerstag, 14. November 2019 12:45:11 UTC+1 schrieb Ashok:

Mi first impression was that it has to have something to do with the http package, because of the name of the routine:
::http::38EventCoroutine

[Brad Lanam]

Apparently coroutines have been added to the following http procedures:
http::ReceiveResponse http::BlockingRead http::BlockingGets
http::make-transformation-chunked
ReceiveResponse creates the ::http::<token>EventCoroutine.

An error in the http package apparently.

You could (a) revert to the older http package, or (b) rewrite your code
to use http callbacks and not use the blocking routines.

[Alexandru]

Hi Brad,

This error comes and comes lately.
It happens during a longer download if I send another http request.
Almost like parallel processing of multiple http request is not supported.
What do you mean by using http callbacks?

[Brad Lanam]

Multiple http requests are not supported, unfortunately.
I push subsequent requests on to a stack, and process them when the
first request is finished.

The http callbacks will not fix that, though they could make the stack
processing easier.

I guess another way to handle that would be to create a Tcl thread for
each http request. But I've never tried that.

http://www.tcl-lang.org/man/tcl8.6/TclCmd/http.htm#M21

Use the -command option of the http command.
I have in my http utilities script:

set htoken [::http::geturl $turl \
-keepalive true \
-command [list after idle \
[list ::httputils::_httpProcess $url $query $retryCount \
$callback $args]] \
{*}$getargs]

The [list after idle ...] works around the http's package error processing
so that you can actually process the errors.

$getargs is usually [list -get].
$retryCount is used by my _httpProcess() routine.
$callback is my script to process the returned data once _httpProcess()
has done all of the checks on the return code/etc.

proc _httpProcess { url query retryCount callback arglist htoken } {
set meta [::http::meta $htoken]
set ncode [::http::ncode $htoken]
...
{*}$callback [list $ncode $data]
}

[Alexandru]

Thanks, that helps alot.
I'll se if executing in different threads is an option for me.

[Keith Nash]

Hi Alexandru,

http 2.9 aims to support multiple concurrent requests.

Can you explain any more about your use case? If you have several downloads
including a long one, are they from different hosts or the same one? If
they are public URLs please can you list them so I can attempt to reproduce
the bug.

Best wishes,

Keith Nash.

[Alexandru]

Hi Keith,

It's always the same host.
Maybe you can reproduce the issue with those two links (first one much larger file size):

https://www.meshparts.de/download/software/MESHPARTS2_20200731_x64.exe
https://www.meshparts.de/download/20181205-Paper-Wittenstein-Meshparts.pdf

[Keith Nash]

Alexandru wrote:

> Hi Keith,
>
> It's always the same host.
> Maybe you can reproduce the issue with those two links (first one much
> larger file size):
>
> https://www.meshparts.de/download/software/MESHPARTS2_20200731_x64.exe
> https://www.meshparts.de/download/20181205-Paper-Wittenstein-Meshparts.pdf

Hi Alexandru,

It works for me. Please could you supply the http::geturl call that
you are using.

Best wishes,

Keith.

P.S. My code is:

package require http
package require tls
http::register https 443 [list ::tls::socket -ssl2 0 -ssl3 0 \
-tls1 1 -tls1.1 1 -tls1.2 1]

set urlA
https://www.meshparts.de/download/software/MESHPARTS2_20200731_x64.exe
set urlB https://www.meshparts.de/download/20181205-Paper-Wittenstein-Meshparts.pdf

proc fetchUrl {absUrl} {
::http::geturl $absUrl \
-blocksize 8192 \
-timeout 30000 \
-keepalive 1 \
-command httpCallback
}
proc httpCallback {token} {
upvar #0 $token state
puts "Completed fetch $token $state(status) $state(http) size
$state(currentsize) url $state(url)"
}

fetchUrl https://www.google.com/
fetchUrl $urlA
after 100 [list fetchUrl $urlB]
after 500 [list fetchUrl $urlB]
after 1000 [list fetchUrl $urlB]
after 2000 [list fetchUrl $urlB]
after 5000 [list fetchUrl $urlB]


(using http 2.9.1, and tls 1.7.21 built with libressl 3.1.3)

The results are:

Completed fetch ::http::1 ok HTTP/1.1 200 OK size 15594 url
https://www.google.com/
Completed fetch ::http::2 ok HTTP/1.1 200 OK size 27185257 url
https://www.meshparts.de/download/software/MESHPARTS2_20200731_x64.exe
Completed fetch ::http::3 ok HTTP/1.1 200 OK size 361495 url
https://www.meshparts.de/download/20181205-Paper-Wittenstein-Meshparts.pdf
Completed fetch ::http::4 ok HTTP/1.1 200 OK size 361495 url
https://www.meshparts.de/download/20181205-Paper-Wittenstein-Meshparts.pdf
Completed fetch ::http::5 ok HTTP/1.1 200 OK size 361495 url
https://www.meshparts.de/download/20181205-Paper-Wittenstein-Meshparts.pdf
Completed fetch ::http::6 ok HTTP/1.1 200 OK size 361495 url
https://www.meshparts.de/download/20181205-Paper-Wittenstein-Meshparts.pdf
Completed fetch ::http::7 ok HTTP/1.1 200 OK size 361495 url
https://www.meshparts.de/download/20181205-Paper-Wittenstein-Meshparts.pdf

If I modify the http::geturl call to use -keepalive 0, a separate connection
is used for each HTTP transaction, and the small files do not have to wait
for the large one to finish:

Completed fetch ::http::1 ok HTTP/1.1 200 OK size 15561 url
https://www.google.com/
Completed fetch ::http::3 ok HTTP/1.1 200 OK size 361495 url
https://www.meshparts.de/download/20181205-Paper-Wittenstein-Meshparts.pdf
Completed fetch ::http::5 ok HTTP/1.1 200 OK size 361495 url
https://www.meshparts.de/download/20181205-Paper-Wittenstein-Meshparts.pdf
Completed fetch ::http::4 ok HTTP/1.1 200 OK size 361495 url
https://www.meshparts.de/download/20181205-Paper-Wittenstein-Meshparts.pdf
Completed fetch ::http::6 ok HTTP/1.1 200 OK size 361495 url
https://www.meshparts.de/download/20181205-Paper-Wittenstein-Meshparts.pdf
Completed fetch ::http::7 ok HTTP/1.1 200 OK size 361495 url
https://www.meshparts.de/download/20181205-Paper-Wittenstein-Meshparts.pdf
Completed fetch ::http::2 ok HTTP/1.1 200 OK size 27185257 url
https://www.meshparts.de/download/software/MESHPARTS2_20200731_x64.exe

[Alexandru]

I tried to reproduce the issue in my app, by I couldn't.
As soon as I will get the issue again, I will try to reproduce it and send you an example.

[Keith Nash]

Hi Alexandru,

I've seen the bug myself now. I've filed a report at

https://core.tcl-lang.org/tcl/tktview/c2dc1da315f3df3b62cae1f80862ecaf3c2d7ca9

It happens when the http::geturl is used with the -handler or -progress
options, and the callback command enters the event loop. This allows the
next read from the socket to start before the current one has finished.

I'll post a fix shortly.

Best wishes,

Keith.

[Alexandru]

That's awesome!
Thank you very much for the hard work.
I imagine, it wasn't so easy to find the right situation, where the bug is happening.

Alexandru

[Alexandru]

Currently I experience the same error as describe in the OP but this time it's reproductible.
As Keith wrote, the bug should be fixed.
Now how to get the fix?
In the repository I see the content of the new http.tcl but this file seems to be completly different that hat I have in my library.
There is also a http-2.9.0.tm... Is this the actual package? Is there a newer version available?

Thanks
Alexandru