python redis error

[Gopalakrishnan Subramani]

I am getting the following error while running redis along with celery
framework.

Is the python redis is a thread safe?

[2011-01-05 09:15:51,603: ERROR/PoolWorker-5]
[stacked.tasks.tasks.single_topic_downloader9(37e211b7-a237-4267-
acd7-21de6a63dba1)] error while extracting feeds
Traceback (most recent call last):
File "/home/krish/stacked/stacked/tasks/tasks.py", line 48, in
update
self.update_feeds(self.result.entries)
File "/home/krish/stacked/stacked/tasks/tasks.py", line 237, in
update_feeds
self.registry.save()
File "/home/krish/stacked/stacked/model/site.py", line 257, in save
Registry.meta.site_db.save_registry(self)
File "/home/krish/stacked/stacked/db/site.py", line 389, in
save_registry
self.save_keyed_object(self.registry_db, registry)
File "/home/krish/stacked/stacked/db/__init__.py", line 18, in
save_keyed_object
db.set(obj.get_key(), cPickle.dumps(obj,
cPickle.HIGHEST_PROTOCOL))
File "build/bdist.linux-i686/egg/redis/client.py", line 695, in set
return self.execute_command('SET', name, value)
File "build/bdist.linux-i686/egg/redis/client.py", line 339, in
execute_command
**options
File "build/bdist.linux-i686/egg/redis/client.py", line 324, in
_execute_command
return self.parse_response(command_name, **options)
File "build/bdist.linux-i686/egg/redis/client.py", line 399, in
parse_response
response = self._parse_response(command_name, catch_errors)
File "build/bdist.linux-i686/egg/redis/client.py", line 395, in
_parse_response
raise InvalidResponse("Unknown response type for: %s" %
command_name)
InvalidResponse: Unknown response type for: SET

not sure how to capture more details.

[Gopalakrishnan Subramani]

Another similar error in GET operation.

File "/env/stacked/lib/python2.6/site-packages/celery-2.1.4-
py2.6.egg/celery/execute/trace.py", line 34, in trace
return cls(states.SUCCESS, retval=fun(*args, **kwargs))
File "/env/stacked/lib/python2.6/site-packages/celery-2.1.4-
py2.6.egg/celery/task/base.py", line 248, in __call__
return self.run(*args, **kwargs)
File "/home/krish/stacked/stacked/tasks/tasks.py", line 355, in run
download_feed(single_topic_downloader7, logger)
File "/home/krish/stacked/stacked/tasks/tasks.py", line 304, in
download_feed
registry = Registry.get(registry_key)
File "/home/krish/stacked/stacked/model/site.py", line 272, in get
return Registry.meta.site_db.get_registry(key)
File "/home/krish/stacked/stacked/db/site.py", line 399, in
get_registry
return self.get_object(self.registry_db, name)
File "/home/krish/stacked/stacked/db/__init__.py", line 11, in
get_object
pickle_data = self.get_value(db, key)
File "/home/krish/stacked/stacked/db/__init__.py", line 8, in
get_value
return db.get(key)
File "build/bdist.linux-i686/egg/redis/client.py", line 587, in get
return self.execute_command('GET', name)

File "build/bdist.linux-i686/egg/redis/client.py", line 339, in
execute_command
**options
File "build/bdist.linux-i686/egg/redis/client.py", line 324, in
_execute_command
return self.parse_response(command_name, **options)
File "build/bdist.linux-i686/egg/redis/client.py", line 399, in
parse_response
response = self._parse_response(command_name, catch_errors)
File "build/bdist.linux-i686/egg/redis/client.py", line 395, in
_parse_response
raise InvalidResponse("Unknown response type for: %s" %
command_name)

InvalidResponse: Unknown response type for: GET


Should I lock the redis client?


On Jan 5, 9:21 am, Gopalakrishnan Subramani

<gopalakrishnan.subram...@gmail.com> wrote:
> I am getting the following error while running redis along with celery
> framework.
>

> I File "/env/stacked/lib/python2.6/site-packages/celery-2.1.4-py2.6.egg/celery/execute/trace.py", line 34, in trace
return cls(states.SUCCESS, retval=fun(*args, **kwargs))
File "/env/stacked/lib/python2.6/site-packages/celery-2.1.4-
py2.6.egg/celery/task/base.py", line 248, in __call__
return self.run(*args, **kwargs)
File "/home/krish/stacked/stacked/tasks/tasks.py", line 355, in run
download_feed(single_topic_downloader7, logger)
File "/home/krish/stacked/stacked/tasks/tasks.py", line 304, in
download_feed
registry = Registry.get(registry_key)
File "/home/krish/stacked/stacked/model/site.py", line 272, in get
return Registry.meta.site_db.get_registry(key)
File "/home/krish/stacked/stacked/db/site.py", line 399, in
get_registry
return self.get_object(self.registry_db, name)
File "/home/krish/stacked/stacked/db/__init__.py", line 11, in
get_object
pickle_data = self.get_value(db, key)
File "/home/krish/stacked/stacked/db/__init__.py", line 8, in
get_value
return db.get(key)
File "build/bdist.linux-i686/egg/redis/client.py", line 587, in get
return self.execute_command('GET', name)

File "build/bdist.linux-i686/egg/redis/client.py", line 339, in
execute_command
**options
File "build/bdist.linux-i686/egg/redis/client.py", line 324, in
_execute_command
return self.parse_response(command_name, **options)
File "build/bdist.linux-i686/egg/redis/client.py", line 399, in
parse_response
response = self._parse_response(command_name, catch_errors)
File "build/bdist.linux-i686/egg/redis/client.py", line 395, in
_parse_response
raise InvalidResponse("Unknown response type for: %s" %
command_name)

InvalidResponse: Unknown response type for: GET

[Josiah Carlson]

The standard redis-py available via github as https://github.com/andymccurdy/redis-py is thread safe. However, depending on how you create your processes and when, you may end up with underlying system issues with file handles (I had a fork after creating a connection result in the socket being shared by two processes, which caused some protocol errors; this is a fork() issue, not a Python issue).

If you are forking after connecting to Redis, you may want to consider forking before connecting to Redis.

Regards,

 - Josiah

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Andy McCurdy]

Hi Gopalakrishnan,

Can you open an issue over at http://github.com/andymccurdy/redis-py please? If there's any way you could produce a test or code snippet that causes this error, it would go a long way in helping debug it.

-andy

[Gopalakrishnan Subramani]

@Josiah,

Thank you so much for quickly spotting the issue. This issue caused my
aggregater randomly to stop from downloading RSS Data.

Hi Andy,

I am following up with celery group here

http://groups.google.com/group/celery-users/browse_thread/thread/f61270144533b97

Celery sole developer Ask mentioned that celery uses the fork and
redis connection must be reset/reconnected for each worker progress.

I will write back to you my finding. I will try to simulate the issue
with python multiprocessor module, which I have no experience. But I
am positive enough to simulate the issue. I think, even after
simulating, the issue is not with the redis, it is not a thread-safe
issue, but actually a fork issue (very common error in socket sharing)

Regards,

Krish

On Jan 5, 11:09 pm, Andy McCurdy <sed...@gmail.com> wrote:
> Hi Gopalakrishnan,
>
> Can you open an issue over athttp://github.com/andymccurdy/redis-pyplease?
> If there's any way you could produce a test or code snippet that causes this
> error, it would go a long way in helping debug it.
>
> -andy
>
> On Wed, Jan 5, 2011 at 9:21 AM, Gopalakrishnan Subramani <
>

> > redis-db+u...@googlegroups.com<redis-db%2Bunsu...@googlegroups.com>
> > .

[Andy McCurdy]

I'd suggest you create an instance of the Redis client in each of your workers. I have not tested sharing client instances with the multiprocessing library.

To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.

[Gopalakrishnan Subramani]

Hi Andy,

Solved my issues. Now in every worker process, I re-create redis
server connection.

Thank you,

Krish

On Jan 6, 9:56 pm, Andy McCurdy <sed...@gmail.com> wrote:
> I'd suggest you create an instance of the Redis client in each of your
> workers. I have not tested sharing client instances with the multiprocessing
> library.
>
> On Thu, Jan 6, 2011 at 5:23 AM, Gopalakrishnan Subramani <
>

> gopalakrishnan.subram...@gmail.com> wrote:
> > @Josiah,
>
> > Thank you so much for quickly spotting the issue. This issue caused my
> > aggregater randomly to stop from downloading RSS Data.
>
> > Hi Andy,
>
> > I am following up with celery group here
>

> >http://groups.google.com/group/celery-users/browse_thread/thread/f612...

> > <redis-db%2Bunsu...@googlegroups.com<redis-db%252Buns...@googlegroups.com>