Controlling the value of the ENABLE_ASYNCIO flag
136 views
Skip to first unread message
gordon.d...@gmail.com
Oct 22, 2021, 10:38:20 AM10/22/21
to sqlalchemy-devel
When I run
pytest --db=asyncpg test/dialect/test_suite.py
(i.e., with postgresql+asyncpg) the tests run fine because apparently the ENABLE_ASYNCIO flag as tested here
is True. However, when I try to do the same with cockroachdb+asyncpg that flag is False so the tests fail because
"greenlet_spawn has not been called; can't call await_() here. Was IO attempted in an unexpected place?"
I've looked through the code for postgresql+asyncpg and that flag is apparently not manipulated directly. It defaults to True here
and can be switched off here
so opt.disable_asyncio must be getting modified somewhere.
Just looking for the place where postgresql+asyncpg affects that flag so I can do the same thing with cockroachdb+asyncpg.
Mike Bayer
Oct 22, 2021, 12:48:29 PM10/22/21
to sqlalche...@googlegroups.com
I think ENABLE_ASYNCIO is true as long as you are in python 3 and you haven't passed --disable-asyncio.
However, on the next lines, if "config.any_async" is false, then you still get into that spot. you can put an assertion or breakpoint() in there to confirm this (I just did it on this end and it checks out, ENABLE_ASYNCIO stays true, config.any_async is false, if i test on plain sqlite).
config.any_async should be set to True if any of the dialects being tested have ".is_async = True". so the theory is, if your dialect has .is_async = True on it, it should use the async compatible runner.
--You received this message because you are subscribed to the Google Groups "sqlalchemy-devel" group.To unsubscribe from this group and stop receiving emails from it, send an email to sqlalchemy-dev...@googlegroups.com.To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy-devel/a3cc8922-8a42-4bbd-998c-271fb43d9a59n%40googlegroups.com.
gordon.d...@gmail.com
Oct 22, 2021, 2:46:18 PM10/22/21
to sqlalchemy-devel
Thanks, Mike. CockroachDBDialect_asyncpg subclasses PGDialect_asyncpg so I figured that it would inherit the is_async setting, and it appears that it does (see below).
class CockroachDBDialect_asyncpg(PGDialect_asyncpg, CockroachDBDialect):
I added a couple of breakpoints to verify:
(venv) gord@gord-HP-studio:~/git/sqlalchemy-cockroachdb$ pytest --db=async test/test_suite_sqlalchemy.py -k "BooleanTest and test_null"
============================================================================================== test session starts ===============================================================================================
platform linux -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /home/gord/git/sqlalchemy-cockroachdb/venv/bin/python
cachedir: .pytest_cache
rootdir: /home/gord/git/sqlalchemy-cockroachdb, configfile: setup.cfg
collecting ...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/gord/git/sqla-gerrit/lib/sqlalchemy/testing/suite/test_types.py(718)BooleanTest()
-> __backend__ = True
(Pdb) config.any_async
True
(Pdb) config.db.dialect
<sqlalchemy_cockroachdb.asyncpg.CockroachDBDialect_asyncpg object at 0x7f30750c6be0>
(Pdb) config.db.dialect.is_async
True
(Pdb) c
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB continue (IO-capturing resumed) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
collected 531 items / 530 deselected / 1 selected
test/test_suite_sqlalchemy.py::BooleanTest_cockroachdb+asyncpg_9_5_0::test_null
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/gord/git/sqla-gerrit/lib/sqlalchemy/testing/asyncio.py(85)_maybe_async()
-> if not ENABLE_ASYNCIO:
(Pdb) ENABLE_ASYNCIO
False
(Pdb) c
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB continue (IO-capturing resumed) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
ERROR [100%]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/gord/git/sqla-gerrit/lib/sqlalchemy/testing/asyncio.py(85)_maybe_async()
-> if not ENABLE_ASYNCIO:
(Pdb) ENABLE_ASYNCIO
False
(Pdb) c
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB continue (IO-capturing resumed) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
/home/gord/git/sqla-gerrit/lib/sqlalchemy/pool/base.py:249: RuntimeWarning: coroutine 'Connection.close' was never awaited
self.logger.error(
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
===================================================================================================== ERRORS =====================================================================================================
_______________________________________________________________________ ERROR at setup of BooleanTest_cockroachdb+asyncpg_9_5_0.test_null ________________________________________________________________________
Traceback (most recent call last):
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/testing/asyncio.py", line 119, in wrap_fixture
value = _maybe_async(call_next, gen)
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/testing/asyncio.py", line 85, in _maybe_async
if not ENABLE_ASYNCIO:
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/testing/asyncio.py", line 110, in call_next
return next(gen)
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/testing/fixtures.py", line 359, in _setup_tables_test_class
cls._setup_once_tables()
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/testing/fixtures.py", line 406, in _setup_once_tables
cls._tables_metadata.create_all(cls.bind)
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/sql/schema.py", line 4785, in create_all
bind._run_ddl_visitor(
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/engine/base.py", line 3107, in _run_ddl_visitor
conn._run_ddl_visitor(visitorcallable, element, **kwargs)
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/engine/base.py", line 2110, in _run_ddl_visitor
visitorcallable(self.dialect, self, **kwargs).traverse_single(element)
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/sql/visitors.py", line 524, in traverse_single
return meth(obj, **kw)
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/sql/ddl.py", line 823, in visit_metadata
[t for t in tables if self._can_create_table(t)]
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/sql/ddl.py", line 823, in <listcomp>
[t for t in tables if self._can_create_table(t)]
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/sql/ddl.py", line 788, in _can_create_table
return not self.checkfirst or not self.dialect.has_table(
File "/home/gord/git/sqlalchemy-cockroachdb/sqlalchemy_cockroachdb/base.py", line 189, in has_table
return any(t == table for t in self.get_table_names(conn, schema=schema))
File "/home/gord/git/sqlalchemy-cockroachdb/sqlalchemy_cockroachdb/base.py", line 181, in get_table_names
for row in conn.execute(
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/engine/base.py", line 1286, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
return connection._execute_clauseelement(
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/engine/base.py", line 1478, in _execute_clauseelement
ret = self._execute_context(
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/engine/base.py", line 1842, in _execute_context
self._handle_dbapi_exception(
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/engine/base.py", line 2027, in _handle_dbapi_exception
util.raise_(exc_info[1], with_traceback=exc_info[2])
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/engine/base.py", line 1799, in _execute_context
self.dialect.do_execute(
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/engine/default.py", line 717, in do_execute
cursor.execute(statement, parameters)
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/dialects/postgresql/asyncpg.py", line 449, in execute
self._adapt_connection.await_(
File "/home/gord/git/sqla-gerrit/lib/sqlalchemy/util/_concurrency_py3k.py", line 67, in await_only
raise exc.MissingGreenlet(
sqlalchemy.exc.MissingGreenlet: greenlet_spawn has not been called; can't call await_() here. Was IO attempted in an unexpected place? (Background on this error at: https://sqlalche.me/e/14/xd2s)
================================================================================== 530 deselected, 1 error in 89.10s (0:01:29) ===================================================================================
(venv) gord@gord-HP-studio:~/git/sqlalchemy-cockroachdb$
Mike Bayer
Oct 22, 2021, 4:52:02 PM10/22/21
to sqlalche...@googlegroups.com
that flag is not set to False unless you are typing the "--disable-asyncio" flag on your commandline, or you have it in setup.cfg in the pytest options.
Feel free to put "assert False" inside of plugin_base.py line 398 to make sure nothing is setting that option on the command line or in pytest.
To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy-devel/ab43ac1f-e54b-4133-b8fb-ca3f1acf8c7fn%40googlegroups.com.
gordon.d...@gmail.com
Oct 22, 2021, 5:50:24 PM10/22/21
to sqlalchemy-devel
Weird. I'm definitely not invoking it from the command line and the option is False in opt.disable_asyncio:
(venv) gord@gord-HP-studio:~/git/sqlalchemy-cockroachdb$ pytest --db=async test/test_suite_sqlalchemy.py -k "BooleanTest and test_null"
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/gord/git/sqla-gerrit/lib/sqlalchemy/testing/plugin/plugin_base.py(397)_set_disable_asyncio()
-> if opt.disable_asyncio or not py3k:
(Pdb) opt.disable_asyncio
False
(Pdb) opt.disable_asyncio or not py3k
False
(Pdb) from sqlalchemy.testing import asyncio
(Pdb) asyncio.ENABLE_ASYNCIO
(traceback as before)
gordon.d...@gmail.com
Oct 22, 2021, 6:00:31 PM10/22/21
to sqlalchemy-devel
and the if statement is behaving correctly and not executing the contents:
> /home/gord/git/sqla-gerrit/lib/sqlalchemy/testing/plugin/plugin_base.py(397)_set_disable_asyncio()
-> if opt.disable_asyncio or not py3k:
(Pdb) n
--Return--
> /home/gord/git/sqla-gerrit/lib/sqlalchemy/testing/plugin/plugin_base.py(397)_set_disable_asyncio()->None
-> if opt.disable_asyncio or not py3k:
(Pdb) n
> /home/gord/git/sqla-gerrit/lib/sqlalchemy/testing/plugin/plugin_base.py(300)post_begin()
-> for fn in post_configure:
Mike Bayer
Oct 22, 2021, 8:52:39 PM10/22/21
to sqlalche...@googlegroups.com
what happens if you just change plugin_base to not set the variable to False? with this kind of thing I just dig in there, add assertions etc, until the spot where it's going wrong pops out.
To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy-devel/95bb39c2-269a-4f2d-adcf-7fa1df842bfan%40googlegroups.com.
gordon.d...@gmail.com
Oct 22, 2021, 9:23:31 PM10/22/21
to sqlalchemy-devel
When I comment out
the failure still happens. So the ENABLE_ASYNCIO flag must be getting set to False somewhere else.
gordon.d...@gmail.com
Oct 23, 2021, 5:55:56 AM10/23/21
to sqlalchemy-devel
Aha! Alembic is changing it here:
If I change that to True then the error goes away.
Mike Bayer
Oct 23, 2021, 11:33:12 AM10/23/21
to sqlalche...@googlegroups.com
alembic. OK. is that because the dialect imports alembic? we should adjust that.
To view this discussion on the web visit https://groups.google.com/d/msgid/sqlalchemy-devel/0a358e6f-4005-4ca4-af81-3229b6c1f580n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages