Redis and Python3?

[Joe Barnhart]

Has something changed in the operation of redis to store sessions in web2py when we use python3 instead of python2?  I'm hitting an error in the py-redis library that complains I can't store a boolean in redis (unsupported data type).  Since this seems to be a well known design feature of redis I'm wondering how it seemed to work before.  It's been awhile since I used it, but I swear it worked before.  Didn't it?  Maybe I dreamed it.

-- Joe

Message:

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28.

Traceback (most recent call last): File "/Users/jbarnhart/Work/w2p/w2p_ss_new/gluon/main.py", line 444, in wsgibase serve_controller(request, response, session) File "/Users/jbarnhart/Work/w2p/w2p_ss_new/gluon/main.py", line 206, in serve_controller raise HTTP(response.status, page, **response.headers) gluon.http.HTTP: 200 OK During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/jbarnhart/Work/w2p/w2p_ss_new/gluon/main.py", line 460, in wsgibase session._try_store_in_db(request, response) File "/Users/jbarnhart/Work/w2p/w2p_ss_new/gluon/globals.py", line 1239, in _try_store_in_db record_id = table.insert(**dd) File "/Users/jbarnhart/Work/w2p/w2p_ss_new/gluon/contrib/redis_session.py", line 150, in insert pipe.execute() File "/Users/jbarnhart/.virtualenv/py3/lib/python3.7/site-packages/redis/client.py", line 3520, in execute return execute(conn, stack, raise_on_error) File "/Users/jbarnhart/.virtualenv/py3/lib/python3.7/site-packages/redis/client.py", line 3385, in _execute_transaction all_cmds = connection.pack_commands([args for args, options in cmds File "/Users/jbarnhart/.virtualenv/py3/lib/python3.7/site-packages/redis/connection.py", line 687, in pack_commands for chunk in self.pack_command(*cmd): File "/Users/jbarnhart/.virtualenv/py3/lib/python3.7/site-packages/redis/connection.py", line 663, in pack_command for arg in imap(self.encoder.encode, args): File "/Users/jbarnhart/.virtualenv/py3/lib/python3.7/site-packages/redis/connection.py", line 114, in encode raise DataError("Invalid input of type: 'bool'. Convert to a " redis.exceptions.DataError: Invalid input of type: 'bool'. Convert to a byte, string or number first.

[Massimo Di Pierro]

Hello Joe,

based on this https://github.com/andymccurdy/redis-py/issues/1071#issuecomment-439199774

looks like Redis had a backward incompatible changes and keys in dictionaries cannot be bool (values can).

I reviewed the web2py code and I can confirm no change on our side.

Random thoughts: It was a design mistake on my side to allow pickle to store sessions. I should have used json. This would have restricted what can go in a session but would have made the serialized session more portable and the whole logic faster. In that case only str keys would have been allowed. Web3py will use json for sessions or move away from sessions completely.

Massimo

[Jim S]

Joe

I'm not sure if this is relevant as I don't remember the exact error I was getting, but I made the following change just last week when I updated to the latest web2py stable.

I went from:

rconn = RConn(ccfg.redis.server, ccfg.redis.port)

to:

rconn = RConn('redis_connection_name', ccfg.redis.server, ccfg.redis.port)

I'm using redis version 3.2.1.  Everything was working fine until I updated web2py and my redis version.  Then it didn't work without this additional parameter.

Please ignore if this is just noise.

-Jim

[Joe Barnhart]

Hi Massimo --

I think I understand your point.  But from where I sit it appears that Redis cannot handle even VALUES as "bool".  They now say it must be mapped to an integer or a string.  I'm pretty sure I'm not using a bool as a key in any dictionaries (they make lousy keys) but I AM using numerous bools as VALUES in dictionaries.

Do you think there is some kind of workaround to convert bools to strings for the purpose of storing sessions in Redis?

Warm regards,

Joe

[Leonel Câmara]

It's not your values Joe, pickle serializes them as a string so they would be fine. It's web2py session locked field which is True or False. It's probably easy to fix this in redis_session.py

[Joe Barnhart]

Hi Leonel --

My brain refuses to put together the words "session locked field".  I understand the part of pickle smashing the values and encoding them as one string.  The bool has something to do with session locking?  

I've only looked at the web2py redis code in the most scant way as I wasn't planning to become a redis expert.  Sessions in redis could be important to me, tho, so I may need to set aside some time to dig into it.  (Storing sessions in cookies has been problematic due to the limited space for cookies in browsers.)

-- Joe

[Jim S]

I changed the following in gluon/globals.py and mine is working now.  But, I'm unclear on how to test to see if it is handling the locking properly:

dd = dict(locked=False,
          client_ip=response.session_client,
          modified_datetime=request.now,
          session_data=session_pickled,
          unique_key=unique_key)

to 

dd = dict(locked='False',
          client_ip=response.session_client,
          modified_datetime=str(request.now),
          session_data=session_pickled,
          unique_key=unique_key)

This makes all my stuff work (I'm still on Python 2.7 but I think the problem has to do with the python redis client moving to version 3).

Leonel - I think this relates back to an issue you commented on late last year.   https://groups.google.com/forum/?pli=1#!searchin/web2py/redis%7Csort:date/web2py/PdquGF_9a2E/6VJpLqsnBgAJ

At that time I just continued using python redis 2.10.6.  But, can't do that forever.  Anyone able to test or improve upon the change I made above?  Like I said, it works for me, but I don't know how to see if I'm causing any other unforeseen damage.

-Jim

[Joe Barnhart]

Looking at the rest of globals.py I don't think this approach will work.  For one, it affects storing sessions in any database, not just redis.  Since the table is declared with a "boolean" field I don't think its gonna like the string version of "False".

I wonder why we don't have a simple minded dialect in DAL for redis.  Admittedly, it's not a RDBMS so a lot of DAL just won't ever work on redis.  But it would have provided a clean place to map "booleans" to strings instead of bools in this case, making it easy to special-case redis.

But maybe that would encourage people to complain that the implementation of the redis DAL dialect is too deficient.  Putting it in DAL would raise people's expectations of what it can handle.

-- Joe

[Jim Steil]

Thanks Joe

Not what I wanted to hear, but the right answer. I'm just going to have to keep redis back leveled for now.

Jim

--
Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/0iU2wqSik0k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jim S]

Ok, how about this for a fix:

I inserted the following at lines 147 and 225 in gluon/contrib/redis_session.py.

kwargs['locked'] = str(kwargs['locked'])
kwargs['modified_datetime'] = str(kwargs['modified_datetime'])

Any redis_session experts out there that can help with this?

My understanding is that when the python redis client went from v2 to v3 it now defaults to a StrictRedis client instead of a Redis client.  Details can be found here:  https://pypi.org/project/redis/

Any help would really be appreciated.

-Jim

To unsubscribe from this group and all its topics, send an email to web2py+unsubscribe@googlegroups.com.

[Massimo Di Pierro]

we could change True/False with 1/0 but a better approach would be to remove the value within the redis adapter. The value of locked does not do anything anyway on redis since it is not a relational database with transactions.

[Jim S]

Massimo - I've updated my change per your suggestion.  Also, I've made a pull request to get these changes merged.  This is my first time contributing (to any open source project) so please let me know if I messed something up.

-Jim

[rastafarien]

Hello

I tried my best but can't make  web2py 2.18.5 (latest) work with python3 and redis (2 or 3)

I tried to alter dd definition in globals.py as suggested by jim converting to string False, datetime and unique_key but i always end up with 

Traceback (most recent call last

):
  File "/usr/local/web2py/gluon/restricted.py", line 219, in restricted
    exec(ccode, environment)
  File "/usr/local/web2py/applications/welcome/models/db.py", line 42, in <module>
    session.connect(request, response, db = sessiondb)
  File "/usr/local/web2py/gluon/globals.py", line 966, in connect
    row = table(record_id, unique_key=unique_key)
  File "/usr/local/web2py/gluon/contrib/redis_session.py", line 117, in __call__
    row = q.select()
  File "/usr/local/web2py/gluon/contrib/redis_session.py", line 189, in select
    if rtn['unique_key'] == self.unique_key:
KeyError: 'unique_key'
Can anyone help me having web2py python3  and redis (whatever version) work ?
I don't want to go back to python2 - spent too much time making the changes so that may app works with python3 ...
Help would be greatly appreciated

Stéphane

[Jim Steil]

Have you tried python3 with web2py and python-redis 2?

My problem was going to python-redis 3 using python2 and web2py.

Would it help to change that line to:

if 'unique_key' in self and rtn['unique_key'] == self.unique_key:

-Jim

--

Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/0iU2wqSik0k/unsubscribe.

To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/367f279c-f329-47b2-830b-bc3c4f62e82a%40googlegroups.com.

[Stéphane Krikorian]

Thanks for your feedback Jim.

Since there are known problems with python-redis3 i force python-redis2 (2.10.6)

I tried your suggestion but it fails 

 if 'unique_key' in self and rtn['unique_key'] == self.unique_key:

TypeError: argument of type 'MockQuery' is not iterable

Did anyone succeed in running redis sessions in python3 ??

Thanks

stephane 

You received this message because you are subscribed to the Google Groups "web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to web2py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/CAERBpoB_kjwwc2Dm293AXHOxULcXkvOgS7vFp08R-7QGszGFcw%40mail.gmail.com.

[Joe Barnhart]

I sure wish somebody could fix this.  It's a real bottleneck to Python3 adoption for my site.  I tried using local storage for sessions but the limitation on data size makes it a no-go for my site.  If I can't use redis I'll have to drop back to storing sessions in files and that really sucks for a larger deployment.

-- Joe

[Jim Steil]

I agree

I submitted a patch but it got rejected for some reason.  I don't recall why.

-Jim

--

Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/0iU2wqSik0k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/a8c8cb9d-590a-43ff-9f1b-b757eabf8537%40googlegroups.com.

[rastafarien]

Hello

I have also proposed a patch to make redis work with my applications under wep2py latest ( 2.18.5-stable+timestamp.2019.04.08.04.22.03) and python3

it is attached to this post.

"onetwomany" also released a patch.

Basically

2 files are to be adjusted : redis_session.py and global.py

1/ Redis does not support bool so I replaced False by 0

2/ Redis does not support datetime so I replaced datetime by str(datetime)

Under linux and python3, I found out that it was the only way to have the "welcome" application ( and also my own applications) to work in a multinode environment accessing a distributed redis server for session caching. 

Let us know if it solves your problem.

Regards

Stephane

[Joe Barnhart]

YES.  This seems to work fine in my case.  It permits me to save both sessions and cache data to redis using Python3.  Maybe we can get it accepted and blessed for the next release.

Joe

[Jim Steil]

Is it backward compatible as well?

-Jim

--

Resources:
- http://web2py.com
- http://web2py.com/book (Documentation)
- http://github.com/web2py/web2py (Source code)
- https://code.google.com/p/web2py/issues/list (Report Issues)
---
You received this message because you are subscribed to a topic in the Google Groups "web2py-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/web2py/0iU2wqSik0k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/9ed9f575-1764-4dab-a93c-d7ec68e57638%40googlegroups.com.

[Joe Barnhart]

If you're asking if it runs unmodified under Python2 I do not know as of yet.  I would need to dust off a Python2 site, apply the patch, configure it for redis, and test.  

That might happen in a week or two as I don't have a Python2/redis site right now so I need to do some work to create a test bench for this. 

If someone else has a Python2/redis site already they may be able to test it quicker.

-- Joe

On Monday, September 9, 2019 at 3:00:50 PM UTC-7, Jim S wrote:

Is it backward compatible as well?

-Jim

To unsubscribe from this group and all its topics, send an email to web...@googlegroups.com.

[rastafarien]

It seems that redis3 introduced a couple of nasty breaking changes.

Yes it "used to" work with python2 as well, but more testing is welcomed !

Stephane

Le mardi 10 septembre 2019 00:04:53 UTC+2, Joe Barnhart a écrit :

If you're asking if it runs unmodified under Python2 I do not know as of yet.  I would need to e off a Python2 site, apply the patch, configure it for redis, and test.  

[Jim Steil]

I will confirm that it was redis 3 that broke this.

Jim

To unsubscribe from this group and all its topics, send an email to web2py+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/web2py/d6b522f1-66c2-4392-ac4d-11758d3ddc16%40googlegroups.com.

[Massimo Di Pierro]

py4web supports redis. Just saying. ;-)

[Massimo Di Pierro]

Thanks Rastafarian,

I included your patch with some modifications but did not test my changes. Please check it still woks for you.

massimo

[Joe Barnhart]

ROFL!  I don't think it's quite ready for my prime-time cash-producing website yet.  I'll definitely keep an eye on it and start playing around with it.  I need to keep you from making any decisions I don't like, after all!

-- Joe

[Massimo Di Pierro]

:-)

[Joe Barnhart]

Here is a patch that restores redis3 sessions for my system.  I'm using the latest web2py 2.21.1 and redis stable 6.0.9 (via brew)

diff --git a/gluon/contrib/redis_session.py b/gluon/contrib/redis_session.py

index 6413b52a..73c20422 100644

--- a/gluon/contrib/redis_session.py

+++ b/gluon/contrib/redis_session.py

@@ -13,7 +13,7 @@ from gluon import current

 from gluon.storage import Storage

 from gluon.contrib.redis_utils import acquire_lock, release_lock

 from gluon.contrib.redis_utils import register_release_lock

-from gluon._compat import to_native

+from gluon._compat import to_native,to_bytes

 from datetime import datetime

 

 logger = logging.getLogger("web2py.session.redis")

@@ -208,7 +208,7 @@ class MockQuery(object):

             if rtn:

                 if self.unique_key:

                     # make sure the id and unique_key are correct

-                    if rtn['unique_key'] == to_native(self.unique_key):

+                    if rtn['unique_key'] == to_bytes(self.unique_key):

                         rtn['update_record'] = self.update  # update record support

                     else:

                         rtn = None

Does anyone know how to make code blocks in this new stupid Google Groups window?  Sheesh.  One step sideways and three steps back.

-- Joe

[xgp.l...@gmail.com]

Joe, were you able to apply this patch?

Cheers,

[xgp.l...@gmail.com]

Joe, thanks for this.

I havent been able to port my app to py4web yet. So i have to step into python3 first.

Cheers,

El Friday, January 1, 2021 a la(s) 6:45:16 PM UTC-5, Joe Barnhart escribió: