Python 3.5 mysql-connector==2.1.6 SQLAlchemy==1.1.11 queue Empty

[josip povreslo]

Hello,

As mentioned in the subject, we have the following packages installed:

• Python 3.5 
• mysql-connector==2.1.6 
• SQLAlchemy==1.1.11

Our service is working until we get a bit more traffic (40 - 50 req/min), although I'm still not 100% if it's a pattern, however, then we get this error:

Traceback (most recent call last):

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line 1122, in _do_get

    return self._pool.get(wait, self._timeout)

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/queue.py", line 145, in get

    raise Empty

sqlalchemy.util.queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect

    return fn()

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line 328, in unique_connection

    return _ConnectionFairy._checkout(self)

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line 766, in _checkout

    fairy = _ConnectionRecord.checkout(pool)

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line 516, in checkout

    rec = pool._do_get()

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line 1138, in _do_get

    self._dec_overflow()

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__

    compat.reraise(exc_type, exc_value, exc_tb)

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/util/compat.py", line 187, in reraise

    raise value

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line 1135, in _do_get

    return self._create_connection()

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line 333, in _create_connection

    return _ConnectionRecord(self)

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line 461, in __init__

    self.__connect(first_connect_check=True)

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/pool.py", line 651, in __connect

    connection = pool._invoke_creator(self)

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect

    return dialect.connect(*cargs, **cparams)

  File "/usr/local/lib64/python3.5/site-packages/sqlalchemy/engine/default.py", line 393, in connect

    return self.dbapi.connect(*cargs, **cparams)

  File "/usr/local/lib64/python3.5/site-packages/MySQLdb/__init__.py", line 81, in Connect

    return Connection(*args, **kwargs)

  File "/usr/local/lib64/python3.5/site-packages/MySQLdb/connections.py", line 191, in __init__

    super(Connection, self).__init__(*args, **kwargs2)

_mysql_exceptions.OperationalError: (2003, "Can't connect to MySQL server on 'dbhost_obviously' (4)")

We haven't been seeing this error up until recently when we ported our service to Python 3.5. I've read most of the articles related to this queue empty error and most of those are from '12 or '13 and mentioning immature mysql drivers. Is this still the case, should we search for an alternative to mysql-connector or alternative to something else? Any help or guidance is appreciated!

Thanks!

Best,

Josip

[josip povreslo]

In addition to the previous email, what I do is the following:

When request comes in:

self.mysql_engine = create_engine('mysql+mysqldb://{}:{}@{}/{}'.format(self.db_params['dbuser'], self.db_params['dbpass'], self.db_params['db_hosts'][db_host_index], self.db_params['dbname']), pool_recycle=3600, connect_args={'connect_timeout': 2})

And then when need to interact with the dB:

connection = self.mysql_engine.connect()

connection.execute("raw query")

connection.close()

Best Regards,

Josip

[Mike Bayer]

that "mysqldb" URL is not mysql-connector - it's MySQL-Python (old and
unmaintained) or mysqlclient (much better, if you have it installed
and not conflicting w/ MySQL-Python). You'd need to specify
"mysql+mysqlconnector://" to use that DBAPI, but also I'd recommend
trying "mysql+pymysql://" to see if you get better results.

as far as "can't connect to server" you should make sure you are not
surpassing your max_connections limit, though it should be emitting
different error message for that.

The Queue.Empty error is not the "error", it has to do with how the
QueuePool works internally to test that the pool is empty, and Python
3 can't help but report every exception in the chain. The error is
the can't connect to server.

> --
> SQLAlchemy -
> The Python SQL Toolkit and Object Relational Mapper
>
> http://www.sqlalchemy.org/
>
> To post example code, please provide an MCVE: Minimal, Complete, and
> Verifiable Example. See http://stackoverflow.com/help/mcve for a full
> description.
> ---
> You received this message because you are subscribed to the Google Groups
> "sqlalchemy" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to sqlalchemy+...@googlegroups.com.
> To post to this group, send email to sqlal...@googlegroups.com.
> Visit this group at https://groups.google.com/group/sqlalchemy.
> For more options, visit https://groups.google.com/d/optout.

[Mike Bayer]

I've set up https://bitbucket.org/zzzeek/sqlalchemy/issues/4028/move-cant-connect-handling-outside-of
to deal with the misleading "pool.Empty" error for 1.2, though that is
not the problem you're having here.

[josip povreslo]

Thanks Mike!

Point valid, it's mysqlclient==1.3.9, we don't have MySQL-Python installed, inherited the system so I assumed it's mysql-connector from some reason, nevertheless. 

Hmmm, I was afraid that's the actual case (connection error). We were checking the connection limits and that's not the case, will try to switch everything over to mysql-connector then and get back with the results. Will do it on Monday (not pushing things live over the weekend :)). 

Best Regards,

Josip

[josip povreslo]

Thanks!

[josip povreslo]

Mike,

Just as an info, mysql-connector is a total NO for us, when I switched to mysql-connector we saw bigger problems and those happens much more often. PyMySQL is the last thing to switch to and try. For this processing piece I'm thinking to replace SQLAlchemy with a direct database driver usage, do you think that's the catch or this is a more general issue with the drivers itself?

Best,

Josip

On Friday, July 14, 2017 at 11:57:12 PM UTC+2, Mike Bayer wrote:

[josip povreslo]

To add more information to it:

We are running our dB on AWS RDS, this is what MySQL say in logs when we switched to mysql-connector:

[Note] Aborted connection 9191 to db: 'connstr' (Got an error reading communication packets)

Best Regards,
Josip

[Mike Bayer]

On Mon, Jul 17, 2017 at 4:23 AM, josip povreslo
<josip.p...@gmail.com> wrote:
> To add more information to it:
>
> We are running our dB on AWS RDS, this is what MySQL say in logs when we
> switched to mysql-connector:

Oh. That is an *extremely* different situation. Amazon RDS is a
MySQL variant that runs in an entirely Amazon-specific environment.
They are likely performing some kind of simple rate limiting on your
service, or otherwise having their own kinds of outages. It is not
normal that they should have basic problems like this with off the
shelf MySQL client libraries, and issues like "can't connect" are not
normal in the MySQL world unless there are network issues. I don't
have experience with RDS so you'd need to check with their support.

[josip povreslo]

Yep, that's where I've stopped with my research for now. Mysqlclient handles relatively ok for now although I think as well it's a problem with network there, wouldn't be the first time. Eithter way, thanks for your support, I'll get back with full details if/when I completely understand what's happening and how to configure the whole thing. 

Best,

Josip