Programmatically creating and closing Waitress server: OSError: [Errno 9] Bad file descriptor

[Mikko Ohtamaa]

Hi,

I am trying to add webhook support to the otherwise command-line application using Waitress web server. Webhook endpoints are defined as Pyramid routes. I also wish to test this functionality using pytest.

The server gets created fine, but on pytest shutdown, I am getting an error: OSError: [Errno 9] Bad file descriptor. What would be the nice way to spin Waitress up and down on demand?

How do I create the server instance:

def create_webhook_server(host: str, port: int, username: str, password: str, queue: Queue) -> MultiSocketServer:
    app = create_pyramid_app(username, password, queue, production=False)
    server = create_server(app, host=host, port=port)
    logger.info("Webhook server will spawn at %s:%d", host, port)
    return server

How do I start and test the server in the unit test:

def test_auth_ok():
    """Username and password allow to access the webhook"""
    queue = Queue()
    server = create_webhook_server("127.0.0.1", 5000, "test", "test", queue)
    server_url = "http://test:te...@127.0.0.1:5000"
    webhook_thread = Thread(target=server.run)
    webhook_thread.start()
    # Test home view
    resp = requests.get(server_url)
    assert resp.status_code == 200
    server.close()

The exception I am getting:

tests/test_webhook_auth.py::test_auth_ok
  /Users/moo/Library/Caches/pypoetry/virtualenvs/tradeexecutor-13XfP0tc-py3.9/lib/python3.9/site-packages/_pytest/threadexception.py:75: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-1
 
  Traceback (most recent call last):
    File "/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner
      self.run()
    File "/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run
      self._target(*self._args, **self._kwargs)
    File "/Users/moo/Library/Caches/pypoetry/virtualenvs/tradeexecutor-13XfP0tc-py3.9/lib/python3.9/site-packages/waitress/server.py", line 322, in run
      self.asyncore.loop(
    File "/Users/moo/Library/Caches/pypoetry/virtualenvs/tradeexecutor-13XfP0tc-py3.9/lib/python3.9/site-packages/waitress/wasyncore.py", line 245, in loop
      poll_fun(timeout, map)
    File "/Users/moo/Library/Caches/pypoetry/virtualenvs/tradeexecutor-13XfP0tc-py3.9/lib/python3.9/site-packages/waitress/wasyncore.py", line 172, in poll
      r, w, e = select.select(r, w, e, timeout)
  OSError: [Errno 9] Bad file descriptor
 
    warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg))

tests/test_webhook_auth.py::test_auth_failed
  /Users/moo/Library/Caches/pypoetry/virtualenvs/tradeexecutor-13XfP0tc-py3.9/lib/python3.9/site-packages/_pytest/threadexception.py:75: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-2
 
  Traceback (most recent call last):
    File "/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner
      self.run()
    File "/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 910, in run
      self._target(*self._args, **self._kwargs)
    File "/Users/moo/Library/Caches/pypoetry/virtualenvs/tradeexecutor-13XfP0tc-py3.9/lib/python3.9/site-packages/waitress/server.py", line 322, in run
      self.asyncore.loop(
    File "/Users/moo/Library/Caches/pypoetry/virtualenvs/tradeexecutor-13XfP0tc-py3.9/lib/python3.9/site-packages/waitress/wasyncore.py", line 245, in loop
      poll_fun(timeout, map)
    File "/Users/moo/Library/Caches/pypoetry/virtualenvs/tradeexecutor-13XfP0tc-py3.9/lib/python3.9/site-packages/waitress/wasyncore.py", line 172, in poll
      r, w, e = select.select(r, w, e, timeout)
  OSError: [Errno 9] Bad file descriptor
 
    warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg))

[Michael Merickel]

Mikko,

Have you looked at the StopableWSGIServer in webtest? I wonder if it can help debug what you've been seeing.

https://docs.pylonsproject.org/projects/webtest/en/latest/api.html#webtest.http.StopableWSGIServer

- Michael

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAK8RCUtiHH3N00sUthcUKj6Ur-ZjNsAGkXByo5FEPvuKZY%3DgNg%40mail.gmail.com.

[Mikko Ohtamaa]

Hi Michael,

Have you looked at the StopableWSGIServer in webtest? I wonder if it can help debug what you've been seeing.

https://docs.pylonsproject.org/projects/webtest/en/latest/api.html#webtest.http.StopableWSGIServer

Thanks for the heads up. In this case, I hope to have a proper process and threading support for the server. FAIK StopableWSGIServer is single-threaded, but I could be wrong.

-Mikko

 

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/033C221D-0022-460C-8AC8-9FA05A784273%40gmail.com.

[Bert JW Regeer]

It’s using waitress under the hood, it’s not single threaded. 

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/CAK8RCUtU_e6unFeZMHxbBGRTQSOtj4a_kjMXyGhoXLYx9w_MJQ%40mail.gmail.com.

[Mikko Ohtamaa]

Hi Bert,

On Mon, 24 Jan 2022 at 19:11, Bert JW Regeer <xist...@0x58.com> wrote:

It’s using waitress under the hood, it’s not single threaded. 

Thank you so much. I had somehow misunderstood it is not the real thing. Then will proceed with StopableWSGIServer.

I assume I can pass the same construction arguments I could pass for a normal Waitress?

Br,

Mikko

 

To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/7589611F-9FE5-4E13-9429-175D902A637B%400x58.com.