ConnectionClosedOK: received 1001 (going away); then sent 1001 (going away)

[Valery]

Hi all,

I am getting the ConnectionClosedOK errors when testing with browser bots. On the local machine, a simple page refresh works fine. On Heroku, refreshing the page brings the connection error again ... stuck.

I found one thread in the group about this error but no solution. It ended with Chris saying along the lines "need to investigate."

Any advice on what to try will be greatly appreciated.

--- runtime.txt ---

python-3.8.7

--- requirements.txt ---

otree>=5.4.6
psycopg2>=2.8.4
sentry-sdk>=0.7.9
requests>=2.26.0
SQLAlchemy==1.3.22

Regards

Valery

[Valery]

A brief update: partially good news - I was able to overcome getting stuck on this error page. Because I changed a few things I do not know the root cause.

I will post more later.

Regards,

Valery

[Chris @ oTree]

This issue happens intermittently. It appears to be caused by an unresolved bug in an underlying web socket library. I don’t have a solution for it yet though.

Sent from my phone

On Nov 25, 2021, at 4:37 PM, Valery <pavlo...@gmail.com> wrote:



--
You received this message because you are subscribed to the Google Groups "oTree help & discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to otree+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/otree/785225e1-0346-430b-8113-91395c4127b7n%40googlegroups.com.

[Valery]

Thanks, Chris.

Not sure if this gives or anyone any helpful information but here it is.

I have since changed  - runtime.txt - to python-3.9.7. I tested with 64 browser bots various (all?) combinations of dynos (free, hobby, standard 1X) & databases (sqlite, hobby dev, hobby basic, standard 0). After each try I reset the database and restarted dynos. The ConnectionClosedOK kept coming under all conditions but was easily fixable by a page refresh. That is, I was unable to replicate the issue that led to the original post.

At the same time, I noticed some groups could be a bit stuck at one of the waiting pages. Refreshing the waiting page of any members of the group seemed enough to get the group going.

Also wanted to try websockets 10.1 but heroku said oTree 5.4.7 needs 8.1. Noticed an hour ago 5.5.0 has been released - did not have time to try.

Regards,

Valery

[mika]

Hi there,

is there already an update here?

I also noticed that the issue is resolved after refresh, and I have the same problem with the waiting pages. 

However, I am not sure if I can to start my study like this (it happened in all test group session at least once in each group) as it could lead to one player being kicked out (timeout) and then the whole data would be gone, but of course we would still need to pay the participants as it is not their fault. 

Maybe there is some kind of workaround? 

Thank you! :)

Miriam

[Chris @ oTree]

I have published a new version with an upgrade to Starlette and related libraries like Uvicorn. It might fix this error, though I am not sure because it's not an error I have been able to consistently reproduce. Anyway, worth a try.

It's a beta, you can install with:

pip3 install -U otree --pre

(It also contains the enhancements to group_by_arrival_time described here: https://groups.google.com/g/otree/c/XsFMNoZR7PY/m/NcbU2L3yBQAJ)

[mika]

Hi Chris,

That sounds great! Thank you, I will try it out and let you know!

Cheers,

Miriam

[mika]

How can I make sure that this version gets also installed by Heroku?

On Saturday, February 19, 2022 at 11:39:03 AM UTC+1 Chris @ oTree wrote:

[Chris @ oTree]

If you’re using oTree Hub that will happen automatically. Otherwise you need to update your requirements file. 

Sent from my phone

On Feb 21, 2022, at 6:36 PM, mika <miriam.k...@gmail.com> wrote:

How can I make sure that this version gets also installed by Heroku?

You received this message because you are subscribed to a topic in the Google Groups "oTree help & discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/otree/xg1uMpZiTe0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to otree+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/otree/0f40d45b-0f27-46c0-9686-4fce428a9bd6n%40googlegroups.com.

[mika]

Hi, 

yes, I thought so, but how is this version called? 

Like this? 
otree[mturk]= pre

[Chris @ oTree]

Use “pip3 freeze” etc to see what version of oTree you have installed.

Sent from my phone

On Feb 21, 2022, at 6:48 PM, mika <miriam.k...@gmail.com> wrote:

Hi, 

To view this discussion on the web, visit https://groups.google.com/d/msgid/otree/f7af910d-bf66-4b60-8ae6-9c558842100cn%40googlegroups.com.

[Chris Crabbe]

Hi Chris - 

This is great news - I am eager to install this new version and test whether I still get these errors. 

I am a little concerned about the coupled change to the way group_by_arrival_time works - I don't want any ongoing experiments to change the way we're grouping midway through collecting data.  Is there any way to decouple these changes (or make the dropout-detection on the group_by_arrival_time be turned on and off in settings.py)?

Thanks,

--Chris

To view this discussion on the web, visit https://groups.google.com/d/msgid/otree/5f721e5e-6d41-4db4-95c6-b841280905dcn%40googlegroups.com.

[ccr...@gmail.com]

Hi Chris -

I've had a chance to install 5.8.0b2, and to test the app which was getting these ConnectionClosedOK errors with the new version.  Although the new version no longer gets the ConnectionClosedOK error, in its place I'm seeing a couple new errors:

(1) Many times per session (when running with browser bots) I get the following chain of errors in the server logs:

Exception in ASGI application
Traceback (most recent call last):
  File "d:\proj\ssel-otree5\venv\lib\site-packages\anyio\streams\memory.py", line 81, in
receive
    return self.receive_nowait()
  File "d:\proj\ssel-otree5\venv\lib\site-packages\anyio\streams\memory.py", line 76, in
receive_nowait
    raise WouldBlock
anyio.WouldBlock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\proj\ssel-otree5\venv\lib\site-packages\starlette\middleware\base.py", line 41
, in call_next
    message = await recv_stream.receive()
  File "d:\proj\ssel-otree5\venv\lib\site-packages\anyio\streams\memory.py", line 101, in
 receive
    raise EndOfStream
anyio.EndOfStream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "d:\proj\ssel-otree5\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", l
ine 377, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "d:\proj\ssel-otree5\venv\lib\site-packages\uvicorn\middleware\proxy_headers.py",
line 75, in __call__
    return await self.app(scope, receive, send)
  File "d:\proj\ssel-otree5\venv\lib\site-packages\starlette\applications.py", line 119,
in __call__
    await self.middleware_stack(scope, receive, send)
  File "d:\proj\ssel-otree5\venv\lib\site-packages\starlette\middleware\errors.py", line
181, in __call__
    raise exc
  File "d:\proj\ssel-otree5\venv\lib\site-packages\starlette\middleware\errors.py", line
159, in __call__
    await self.app(scope, receive, _send)
  File "d:\proj\ssel-otree5\venv\lib\site-packages\starlette\middleware\base.py", line 63
, in __call__
    response = await self.dispatch_func(request, call_next)
  File "d:\proj\ssel-otree5\venv\lib\site-packages\otree\middleware.py", line 23, in disp
atch
    response = await call_next(request)
  File "d:\proj\ssel-otree5\venv\lib\site-packages\starlette\middleware\base.py", line 45
, in call_next
    raise RuntimeError("No response returned.")
RuntimeError: No response returned.

(2) Occasionally (maybe once every two or three testing sessions), I get a client-side 500 error - this is the thing which worries me more, I think:

Application error (500)
RuntimeError: Unexpected ASGI message 'websocket.send', after sending 'websocket.close'.
Traceback

File d:\proj\ssel-otree5\venv\lib\site-packages\uvicorn\protocols\websockets\websockets_impl.py, line 275, in asgi_send+
File d:\proj\ssel-otree5\venv\lib\site-packages\starlette\middleware\sessions.py, line 75, in send_wrapper+
File d:\proj\ssel-otree5\venv\lib\site-packages\starlette\websockets.py, line 69, in send+
File d:\proj\ssel-otree5\venv\lib\site-packages\starlette\websockets.py, line 137, in send_text+
File d:\proj\ssel-otree5\venv\lib\site-packages\otree\channels\utils.py, line 31, in send+
File C:\Program Files\Python37\lib\asyncio\base_events.py, line 579, in run_until_complete+
File C:\Program Files\Python37\lib\asyncio\runners.py, line 43, in run+
File d:\proj\ssel-otree5\venv\lib\site-packages\otree\channels\utils.py, line 34, in sync_send+
File d:\proj\ssel-otree5\venv\lib\site-packages\otree\channels\utils.py, line 46, in sync_group_send+
File d:\proj\ssel-otree5\venv\lib\site-packages\otree\views\abstract.py, line 1080, in _mark_completed_and_notify+
File d:\proj\ssel-otree5\venv\lib\site-packages\otree\views\abstract.py, line 913, in _run_aapa_and_notify+
File d:\proj\ssel-otree5\venv\lib\site-packages\otree\views\abstract.py, line 980, in inner_dispatch_gbat+
File d:\proj\ssel-otree5\venv\lib\site-packages\otree\views\abstract.py, line 878, in get+
File d:\proj\ssel-otree5\venv\lib\site-packages\otree\views\abstract.py, line 870, in inner_dispatch+
File d:\proj\ssel-otree5\venv\lib\site-packages\anyio\_backends\_asyncio.py, line 743, in run+
File d:\proj\ssel-otree5\venv\lib\site-packages\anyio\_backends\_asyncio.py, line 805, in run_sync_in_worker_thread+

If you're interested, I can share my code (or set up a heroku project and add you as a collaborator).  Please let me know.

Thanks,

--Chris

[Chris @ oTree]

Thanks so much Chris! I will investigate these issues.

Sent from my phone

On Mar 1, 2022, at 7:58 AM, ccr...@gmail.com <ccr...@gmail.com> wrote:

Hi Chris -

You received this message because you are subscribed to a topic in the Google Groups "oTree help & discussion" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/otree/xg1uMpZiTe0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to otree+un...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/otree/070bff1c-b9c7-4263-ac21-2b8fb5856ba4n%40googlegroups.com.

[Chris @ oTree]

Hi, I have published a new beta update to fix these issues. I have run browser bots for various apps and it seems the errors (such as ConnectionClosedError, IncompleteReadError, ClientDisconnect) have stopped occurring on my end.

Please let me know if it helps.

[Chris Crabbe]

Hi Chris -

I have been running browser bots all day and this update has eliminated the exceptions, and I haven't been able to generate a 500 error either.  There is, however, that (increasingly long as the sessions proceed) default_dict being printed out to the console.

Thanks for looking into this - I will see if we can get back to running that set of experiments.

Thanks,

--Chris

To view this discussion on the web, visit https://groups.google.com/d/msgid/otree/ea7f5fec-1ade-4f68-9d3b-2406924a150cn%40googlegroups.com.

[Chris @ oTree]

That’s good to hear! And I just published an update getting rid of that default dict, it was just a debug thing. 

Sent from my phone

To view this discussion on the web, visit https://groups.google.com/d/msgid/otree/CAHj4W-1N7gbUCDKNCiWy%2BdmodUty1pvYTKjEXgcR7kgJAH9L6Q%40mail.gmail.com.

[mika]

Hi Chris,

is this the correct beta version oTree 5.8.0b5.? 

Thanks and best-

Miriam

[Chris @ oTree]

Yep

Sent from my phone

On Mar 8, 2022, at 8:33 PM, mika <miriam.k...@gmail.com> wrote:

Hi Chris,

To view this discussion on the web, visit https://groups.google.com/d/msgid/otree/f3becdf2-bf8e-45ae-9f7c-ceaedf4cbcfbn%40googlegroups.com.

[mika]

Hi Chris,

we tested in a couple of sessions and did not receive the error or any other error and it also felt much smoother than before!

Thank you very much! I think, I can start the data collection now! :)

Best-

Miriam