Error with redis and web2py 2.24.1: Invalid input of type: 'NoneType'. Convert to a bytes, string, int or float first.

[Lisandro]

Hey there! 

I recently updated to Web2py Version 2.24.1-stable+timestamp.2023.03.23.05.07.17

It uses python 3.9.14, running in production serving around 60 requests per second, using resources efficiently and running really smoothly :D

Since the update, I'm seeing this error sporadically: redis.exceptions.DataError: Invalid input of type: 'NoneType'. Convert to a bytes, string, int or float first.

This is the traceback:

Traceback (most recent call last):
File "applications/eldia/compiled/models.db.py", line 113, in <module>
File "/var/www/medios/gluon/globals.py", line 979, in connect
row = table(record_id, unique_key=unique_key)
File "/var/www/medios/gluon/contrib/redis_session.py", line 134, in __call__
row = q.select()
File "/var/www/medios/gluon/contrib/redis_session.py", line 206, in select
rtn = {to_native(k): v for k, v in self.db.r_server.hgetall(key).items()}
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/commands/core.py", line 4776, in hgetall
return self.execute_command("HGETALL", name)
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/client.py", line 1238, in execute_command
return conn.retry.call_with_retry(
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/retry.py", line 49, in call_with_retry
fail(error)
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/client.py", line 1242, in <lambda>
lambda error: self._disconnect_raise(conn, error),
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/client.py", line 1228, in _disconnect_raise
raise error
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/client.py", line 1239, in <lambda>
lambda: self._send_command_parse_response(
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/client.py", line 1215, in _send_command_parse_response
return self.parse_response(conn, command_name, **options)
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/client.py", line 1254, in parse_response
response = connection.read_response()
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/connection.py", line 824, in read_response
response = self._parser.read_response(disable_decoding=disable_decoding)
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/connection.py", line 318, in read_response
raw = self._buffer.readline()
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/connection.py", line 249, in readline
self._read_from_socket()
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/connection.py", line 195, in _read_from_socket
raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/www/medios/gluon/main.py", line 439, in wsgibase
serve_controller(request, response, session)
File "/var/www/medios/gluon/main.py", line 173, in serve_controller
run_models_in(environment)
File "/var/www/medios/gluon/compileapp.py", line 563, in run_models_in
restricted(ccode, environment, layer=model)
File "/var/www/medios/gluon/restricted.py", line 219, in restricted
exec(ccode, environment)
File "applications/eldia/compiled/models.db.py", line 116, in <module>
File "applications/eldia/compiled/models.db.py", line 47, in raise_error
gluon.http.HTTP: 500 INTERNAL SERVER ERROR

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/www/medios/gluon/main.py", line 455, in wsgibase
session._try_store_in_db(request, response)
File "/var/www/medios/gluon/globals.py", line 1254, in _try_store_in_db
record_id = table.insert(**dd)
File "/var/www/medios/gluon/contrib/redis_session.py", line 167, in insert
pipe.execute()
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/client.py", line 2078, in execute
return conn.retry.call_with_retry(
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/client.py", line 2079, in <lambda>
lambda: execute(conn, stack, raise_on_error),
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/client.py", line 1922, in _execute_transaction
all_cmds = connection.pack_commands(
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/connection.py", line 895, in pack_commands
for chunk in self.pack_command(*cmd):
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/connection.py", line 858, in pack_command
for arg in map(self.encoder.encode, args):
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/connection.py", line 108, in encode
raise DataError(
redis.exceptions.DataError: Invalid input of type: 'NoneType'. Convert to a bytes, string, int or float first.

It's not clear to me where or how the error happens. From what I see, I understand that the problem is with redis session, but I can't figure out where. 

I run python 3.9.14 with virtualenv and Redis 4.3.5.

Any point or suggestion will be much appreciated.

Thanks!

[Dave S]

On Monday, May 8, 2023 at 6:36:13 AM UTC-7 Lisandro wrote:

Hey there! 

I recently updated to Web2py Version 2.24.1-stable+timestamp.2023.03.23.05.07.17

It uses python 3.9.14, running in production serving around 60 requests per second, using resources efficiently and running really smoothly :D

Since the update, I'm seeing this error sporadically: redis.exceptions.DataError: Invalid input of type: 'NoneType'. Convert to a bytes, string, int or float first.

This is the traceback:

Traceback (most recent call last):
File "applications/eldia/compiled/models.db.py", line 113, in <module>
File "/var/www/medios/gluon/globals.py", line 979, in connect
row = table(record_id, unique_key=unique_key)

 [...]

File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/connection.py", line 318, in read_response
raw = self._buffer.readline()
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/connection.py", line 249, in readline
self._read_from_socket()
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/connection.py", line 195, in _read_from_socket
raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/www/medios/gluon/main.py", line 439, in wsgibase
serve_controller(request, response, session)
File "/var/www/medios/gluon/main.py", line 173, in serve_controller
run_models_in(environment)
File "/var/www/medios/gluon/compileapp.py", line 563, in run_models_in
restricted(ccode, environment, layer=model)
File "/var/www/medios/gluon/restricted.py", line 219, in restricted
exec(ccode, environment)
File "applications/eldia/compiled/models.db.py", line 116, in <module>
File "applications/eldia/compiled/models.db.py", line 47, in raise_error
gluon.http.HTTP: 500 INTERNAL SERVER ERROR

tripped while already falling, methinks

 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/var/www/medios/gluon/main.py", line 455, in wsgibase
session._try_store_in_db(request, response)
File "/var/www/medios/gluon/globals.py", line 1254, in _try_store_in_db
record_id = table.insert(**dd)
File "/var/www/medios/gluon/contrib/redis_session.py", line 167, in insert
pipe.execute()
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/client.py", line 2078, in execute
return conn.retry.call_with_retry(
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/retry.py", line 46, in call_with_retry
return do()
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/client.py", line 2079, in <lambda>
lambda: execute(conn, stack, raise_on_error),
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/client.py", line 1922, in _execute_transaction
all_cmds = connection.pack_commands(
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/connection.py", line 895, in pack_commands
for chunk in self.pack_command(*cmd):
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/connection.py", line 858, in pack_command
for arg in map(self.encoder.encode, args):
File "/opt/virtualenvs/medios_py39/lib/python3.9/site-packages/redis/connection.py", line 108, in encode
raise DataError(
redis.exceptions.DataError: Invalid input of type: 'NoneType'. Convert to a bytes, string, int or float first.

This looks like it involves one of these being None:

        dd = dict(locked=0,
                  client_ip=response.session_client,
                  modified_datetime=request.now.isoformat(),
                  session_data=session_pickled,
                  unique_key=unique_key)

 but this seems to be in trying to clean up after the original issue.

It's not clear to me where or how the error happens. From what I see, I understand that the problem is with redis session, but I can't figure out where. 

What I would investigate is the exception at the top of the list, which means finding out why the socket was closed.

 

I run python 3.9.14 with virtualenv and Redis 4.3.5.

Any point or suggestion will be much appreciated.

Thanks!

Sorry, I'm not a redis expert, but in compound fractures, er, exceptions, the first one to occur is where the problem started.  Good luck!

/dps 

[Lisandro]

I see what you mean. I'll investigate why the socket was closed, I presume it could be a network problem, because the Redis server is separated from the one where the app resides, they communicate through the local network. Considering the servers have plenty of available resources, the app serves around 60 requests per second and the error only happens 3 or 4 times per day, I will look into the network.

Thank you for your time!

Best regards,

Lisandro

[Lisandro]

I've done some more investigation regarding this error. 

Apparently, the error "redis.exceptions.ConnectionError: Connection closed by server" is potentially related to the fact that the Python app doesn't exchange data with Redis for a certain time and the connection closes automatically. When the app tries to exchange again with Redis, the connection doesn't work anymore and you get the error "Connection closed by server". To resolve this, you can pass the health_check_interval.

More on this error:

https://github.com/redis/redis-py/issues/1186

https://github.com/redis/redis-py/issues/1232

https://stackoverflow.com/questions/70647697/handle-redis-connection-reset

In my web2py app I'm connecting to Redis using RConn provided by gluon.contrib.redis_utils. I've check the source code of RConn: it instantiates a RedisStrict connection with the provided parameters. As right now I'm not providing any other parameter besides host/port, I think the problem will solve providing health_check_interval and socket_timeout arguments. I will make those changes in production and see what happens, but I'm very confident :)