ResponseError: LOADING Redis is loading the dataset in memory

[scott sturgeon]

I have a redis master running on a server, and a slave running on the
webserver. Sometimes when the python application tries to access the
slave, an error is returned that says "ResponseError: LOADING Redis is
loading the dataset in memory".

What does that mean??? and why would 'loading the dataset" be returned
as an error?

File "/usr/local/lib/python2.7/site-packages/redis/client.py", line
587, in get
return self.execute_command(\'GET\', name)'
File "/usr/local/lib/python2.7/site-packages/redis/client.py", line
339, in execute_command
**options\n', ' File "/usr/local/lib/python2.7/site-packages/redis/
client.py", line 321, in _execute_command
self.connection.send(command, self)',
File "/usr/local/lib/python2.7/site-packages/redis/client.py", line
82, in send
self.connect(redis_instance)',
File "/usr/local/lib/python2.7/site-packages/redis/client.py", line
67, in connect
redis_instance._setup_connection()
File "/usr/local/lib/python2.7/site-packages/redis/client.py", line
433, in _setup_connection
self.execute_command(\'SELECT\', self.connection.db)',
File "/usr/local/lib/python2.7/site-packages/redis/client.py", line
339, in execute_command **options',
File "/usr/local/lib/python2.7/site-packages/redis/client.py", line
324, in _execute_command
return self.parse_response(command_name, **options)',
File "/usr/local/lib/python2.7/site-packages/redis/client.py", line
399, in parse_response
response = self._parse_response(command_name, catch_errors)',
File "/usr/local/lib/python2.7/site-packages/redis/client.py", line
358, in _parse_response
raise ResponseError(response)\n', 'ResponseError: LOADING Redis is
loading the dataset in memory'

[Dvir Volk]

This probably happens because the slave is reconnecting with the master after a master restart or a network hick up.

See the previous thread I started about the dangers of doing this in a multi slave / single master environment.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--

Dvir Volk

System Architect, Do@, http://doat.com

[Salvatore Sanfilippo]

When you start Redis instead of not responding while loading data it
returns an error.
Your application should either wait in a busy loop when this error is
returned if it is not exposed to the public, or show a page or UI
element that communicates to the user that the application is not
ready.

Cheers,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

--
Salvatore 'antirez' Sanfilippo
open source developer - VMware

http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele

[scott sturgeon]

Thanks for your reply Salvatore.

Does this only happen on startup or could it happen during a SYNC as well?

[Salvatore Sanfilippo]

On Mon, Mar 21, 2011 at 11:17 PM, scott sturgeon <sc...@deqq.com> wrote:
> Does this only happen on startup or could it happen during a SYNC as well?

Only on startup. During sync there is a similar error if you configure
Redis to avoid serving data while the slave is not connected to the
master. Otherwise it will silently reconnect while replying to
queries. It depends on the application needs.

Salvatore

[Michael Hale]

I have a master-slave setup with "slave-serve-stale-data yes" on the
slave and 1 client polling the slave for a key in a hash every second.
If the master becomes unavailable and then comes back online a BGSAVE,
SYNC is initiated. At the end of that process the slave is effectively
unavailable for 20 seconds (for my dataset) while the dataset is
loaded from dump.rdb into memory. Here is the output from my client.
Also there are 6 seconds where the client is unable to connect to the
slave.

Fri Apr 15 20:50:40 +0000 2011: 123
Fri Apr 15 20:50:41 +0000 2011: Resource temporarily unavailable -
Timeout reading from the socket
Fri Apr 15 20:50:47 +0000 2011: LOADING Redis is loading the dataset
in memory
Fri Apr 15 20:50:48 +0000 2011: LOADING Redis is loading the dataset
in memory
Fri Apr 15 20:50:49 +0000 2011: LOADING Redis is loading the dataset
in memory
Fri Apr 15 20:50:50 +0000 2011: LOADING Redis is loading the dataset
in memory
Fri Apr 15 20:50:51 +0000 2011: LOADING Redis is loading the dataset
in memory
Fri Apr 15 20:50:52 +0000 2011: LOADING Redis is loading the dataset
in memory
Fri Apr 15 20:50:53 +0000 2011: LOADING Redis is loading the dataset
in memory
Fri Apr 15 20:50:54 +0000 2011: LOADING Redis is loading the dataset
in memory
Fri Apr 15 20:50:55 +0000 2011: LOADING Redis is loading the dataset
in memory
Fri Apr 15 20:50:56 +0000 2011: LOADING Redis is loading the dataset
in memory
Fri Apr 15 20:50:57 +0000 2011: LOADING Redis is loading the dataset
in memory
Fri Apr 15 20:50:58 +0000 2011: LOADING Redis is loading the dataset
in memory
Fri Apr 15 20:51:00 +0000 2011: LOADING Redis is loading the dataset
in memory
Fri Apr 15 20:51:01 +0000 2011: LOADING Redis is loading the dataset
in memory
Fri Apr 15 20:51:02 +0000 2011: 123

Is there anyway to avoid this situation? Can the time at which the
slave performs a sync be controlled?

[Salvatore Sanfilippo]

I think this is a bug, the slave should block the clients while
loading the dataset in memory...

I'll fix this ASAP and release a new version of Redis.

Thanks for submitting,
Salvatore

> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
>

--

[Michael Hale]

After reading back through this thread I realized I didn't phrase my
question very succinctly.

There are 2 points during the sync where my client is unable to
retrieve the value of 123 from redis.

1) It appears that redis disconnects the clients and is unavailable
for (6 seconds).
2) While redis is loading the dataset into memory (13 seconds).

Are you saying that my client should not be disconnected, but simply
see the message "LOADING Redis is loading the dataset in memory" for a
total of 19 seconds? What I would really like to do is have a
deterministic amount of time < 1 second where the client is unable to
retrieve the value of 123. If that is not possible the next best thing
would be to control when redis reloads the dataset. In other words,
for my usage it's preferable to serve stale data as opposed to no
data. If I can control when redis is unavailable that would allow me
to perform rolling syncs across the cluster to make sure I will always
be able to retrive data from redis.

- Michael

> > For more options, visit this group athttp://groups.google.com/group/redis-db?hl=en.

[Josiah Carlson]

On Mon, Apr 18, 2011 at 7:32 AM, Michael Hale <mike...@heroku.com> wrote:
> After reading back through this thread I realized I didn't phrase my
> question very succinctly.

It happens.

> There are 2 points during the sync where my client is unable to
> retrieve the value of 123 from redis.
>
> 1) It appears that redis disconnects the clients and is unavailable
> for (6 seconds).
> 2) While redis is loading the dataset into memory (13 seconds).
>
> Are you saying that my client should not be disconnected, but simply
> see the message "LOADING Redis is loading the dataset in memory" for a
> total of 19 seconds?

Yes.

> What I would really like to do is have a
> deterministic amount of time < 1 second where the client is unable to
> retrieve the value of 123. If that is not possible the next best thing
> would be to control when redis reloads the dataset. In other words,
> for my usage it's preferable to serve stale data as opposed to no
> data. If I can control when redis is unavailable that would allow me
> to perform rolling syncs across the cluster to make sure I will always
> be able to retrive data from redis.

When you know your master has gone down, tell your slaves "slaveof no
one". In a rolling fashion, on a schedule you determine, tell your
slaves "slaveof host port".

- Josiah

[Michael Hale]

On Apr 18, 12:00 pm, Josiah Carlson <josiah.carl...@gmail.com> wrote:

> On Mon, Apr 18, 2011 at 7:32 AM, Michael Hale <mikeh...@heroku.com> wrote:
> > After reading back through this thread I realized I didn't phrase my
> > question very succinctly.
>
> It happens.
>
> > There are 2 points during the sync where my client is unable to
> > retrieve the value of 123 from redis.
>
> > 1) It appears that redis disconnects the clients and is unavailable
> > for (6 seconds).
> > 2) While redis is loading the dataset into memory (13 seconds).
>
> > Are you saying that my client should not be disconnected, but simply
> > see the message "LOADING Redis is loading the dataset in memory" for a
> > total of 19 seconds?
>
> Yes.
>
> > What I would really like to do is have a
> > deterministic amount of time < 1 second where the client is unable to
> > retrieve the value of 123. If that is not possible the next best thing
> > would be to control when redis reloads the dataset. In other words,
> > for my usage it's preferable to serve stale data as opposed to no
> > data. If I can control when redis is unavailable that would allow me
> > to perform rolling syncs across the cluster to make sure I will always
> > be able to retrive data from redis.
>
> When you know your master has gone down, tell your slaves "slaveof no
> one". In a rolling fashion, on a schedule you determine, tell your
> slaves "slaveof host port".

So that works fine when I know that the master will go down or has
gone down, but if the master becomes unreachable for even a few
seconds and then becomes reachable again I will have to pay the 19
second penalty. If there were an option for the slave to not
automatically try to sync after a disconnect so I could control the
reconnect process I could perform rolling syncs on my own schedule. As
it stands now though I have no control over when a sync will happen so
I could effectively be DOSd by a series of momentary network issues.

[Josiah Carlson]

I don't know if there is a configuration setting to not resync on
reconnect (there definitely should be), but I think this should
work...

1. Give your master a password via requirepass in it's config file.
2. Add the proper SLAVEOF configuration to the .conf file on all of your slaves,
3. Add an improper password to the .conf file on all of your slaves.
4. Now, since each slave has the wrong password, it will fail to
(re)auth against the master, so won't (re)sync.
5. When you discover that a slave is (or some slaves are) out of sync
with the master (because they have disconnected, or were never
connected), you can in a rolling fashion tell them "slaveof no one",
"config set masterauth proper_password", "slaveof host port", and
finally "config set masterauth bad_password" to ensure that it doesn't
automatically resync in the future.

Sound good?

Regards,

[Sandro Tosi]

Hi Salvatore,

On Sat, Apr 16, 2011 at 13:20, Salvatore Sanfilippo <ant...@gmail.com> wrote:
> I think this is a bug, the slave should block the clients while
> loading the dataset in memory...
>
> I'll fix this ASAP and release a new version of Redis.

Has this been fixed and/or backported to 2.2 too? I can't see anything
referring to it on [1] and it's an interesting feature for our
platform :)

[1] https://raw.github.com/antirez/redis/2.2/00-RELEASENOTES

Cheers,
Sandro

ps: I'll follow-up with a feature request with an issue, related to
startup and slave sync with master.

--
Sandro Tosi (aka morph, morpheus, matrixhasu)
My website: http://matrixhasu.altervista.org/
Me at Debian: http://wiki.debian.org/SandroTosi