Redis/Ruby gem exceptions

[Rob]

Hello list

I'm experiencing some difficult-to-solve and very frustrating problems
using Redis within my web application. Every few hours, my Rails
processes start spitting out exceptions, usually starting with one
like this:

Redis::ProtocolError: Protocol error, got '3' as initial reply
byte

or

Redis::ProtocolError: Protocol error, got '9' as initial reply
byte

or sometimes just

Errno::EAGAIN: Resource temporarily unavailable - Timeout reading from
the socket

After these errors occur, Redis starts to return incorrect and
unexpected results from commands, which naturally causes all sorts of
problems in my application. This continues until I kill and restart my
Rails processes (restarting Redis doesn't help).

I understand that this could well be a problem with the Redis gem,
rather than Redis, but I've asked elsewhere and not been able to find
any answers yet. So I thought I'd ask here instead, I hope somebody
can help me out.

I am using Redis 2.0.1, Redis gem 2.0.7, Rails 2.3.8 through Unicorn,
on EC2 (standard Fedora Core 8 AMI).

[Salvatore Sanfilippo]

Hello,

99.99% you are using the same Redis object across different threads.
But I bet the Redis-rb hackers listening in this list will be able to
provide some better help :)

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

--
Salvatore 'antirez' Sanfilippo
http://invece.org

"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele

[Pieter Noordhuis]

I've worked with Unicorn in the past, and my guess is that you access
Redis /before/ the worker processes are forked, which causes all
workers to share the same descriptor (exactly as Salvatore mentions).
You can fix this by making the Redis client explicitly reconnect in
Unicorn's after_fork hook.

Cheers,
Pieter

[Salvatore Sanfilippo]

Cool explanation, this could deserve a FAQ entry in the next site. So
I'm starting to tag this messages with
REDISFAQ so we can later search them in google group.

Cheers,
Salvatore

[Rob]

Thank-you for your quick and useful responses.

I added this in my after_fork block:

REDIS.quit

REDIS        # this forces the gem to reconnect to Redis, unless there's a better way?

And restarted Unicorn. I hope this has fixed the problem, I will have to leave it a few days to be sure though.

thanks again

Rob

[Damian Janowski]

On Fri, Sep 17, 2010 at 5:38 AM, Salvatore Sanfilippo <ant...@gmail.com> wrote:
> Hello,
>
> 99.99% you are using the same Redis object across different threads.
> But I bet the Redis-rb hackers listening in this list will be able to
> provide some better help :)

Exactly, as Salvatore and Pieter explained, you're sharing a file
descriptor between processes.

There's not much we can do about it except from better documentation.
So we included a very simple example of how to use the Ruby client
with Unicorn: http://github.com/ezmobius/redis-rb/commit/036f0eafcc67df3d4b31c489f42a6f386cc79a27

Also expanded the message for the exception so that it's more useful:
http://github.com/ezmobius/redis-rb/commit/70454014711581682f4235711bc8effe484af45b

Hope that helps.

[Luis Lavena]

The same as Unicorn applies to Passenger. You need reconnect after forking.

Details on usage under Passenger:

http://www.modrails.com/documentation/Users%20guide.html#_smart_spawning_gotcha_1_unintential_file_descriptor_sharing

--
Luis Lavena
AREA 17
-
Perfection in design is achieved not when there is nothing more to add,
but rather when there is nothing more to take away.
Antoine de Saint-Exupéry