Redis slowing down

[Rajitha R]

Hi All,

I am using redis to insert key-value pairs into a single database.I
have understood that the redis database can take upto 2^32
keys(4,294,967,296.--close to 4 billion)

But after i insert around 4 million keys, the speed of my redis server
slows down.What can i do to increase the speed?

Kindly help.

Thanks in Advance,

Rajitha

[Dvir Volk]

Hi,

Some details are missing about your setup.

how much RAM is on the server? how much RAM is redis taking? is it a 32 or 64 machine?

did you set max memory on redis? did you enable VM in redis?

without knowing anything, it's probably a problem of redis extending the amount of physical memory on your machine.

Personally I keep tens of millions of keys on redis without any noticeable performance degradation.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

--

Dvir Volk

System Architect, DoAT, http://doat.com

[Rajitha R]

Hi,

Thanks for your reply.The RAM size is close to 4GB.Its a 64 bit
machine.I haven't set any max memory on redis and neither did i enable
VM.
I am not aware as to how do i find the RAM size that redis is
consuming..Would be kind of you if you could tell me how to find out
that?

One more related question to that,is that i am currently running the
whole setup in my local machine.Very soon i would be deploying redis
on a 32GB RAM machine.Then ,redis might very soon consume much of the
RAM .How do i allocate only some part say 10GB only for redis to use?


I am simply inserting key-value pairs into the db with appendonly mode
switched on.I want to get good performance just using simple redis
such that it inserts atleast 10million elements. Can i achieve it with
it or i have to go with redis sharding or multi-slave approaches?

Thanks in Advance,
Rajitha

[Didier Spezia]

You can use the redis-cli INFO command to know how much

memory is used by Redis.

Redis stores the data in RAM, so you need to be sure to have

enough RAM. It mostly depends on the number and average

size of your items. You could try to load 1 million of them,

check the memory consumption with INFO, and extrapolate.

To limit memory used by Redis, you may want to set

the maxmemory and maxmemory-policy parameters

in the redis configuration file (some documentation in the

file itself).

If you have enough RAM (depends on the size of your items),

Redis is certainly able to handle 10 millions items using one

instance without any sharding or replication mechanism.

Best regards,

Didier.

[Rajitha R]

That really helped me..Thanks a lot.

On Sep 15, 6:28 pm, Didier Spezia <didier...@gmail.com> wrote:
> You can use the redis-cli INFO command to know how much
> memory is used by Redis.
>
> Redis stores the data in RAM, so you need to be sure to have
> enough RAM. It mostly depends on the number and average
> size of your items. You could try to load 1 million of them,
> check the memory consumption with INFO, and extrapolate.
>
> To limit memory used by Redis, you may want to set

>.. the maxmemory and maxmemory-policy parameters

[Rajitha R]

Hi,

I have ran the redis-cli info command and see the following output
always:

redis_version:1.3.15
redis_git_sha1:00000000
redis_git_dirty:0
arch_bits:64
multiplexing_api:epoll
process_id:1149
uptime_in_seconds:2469
uptime_in_days:0
connected_clients:1
connected_slaves:0
blocked_clients:0
used_memory:789272
used_memory_human:770.77K
changes_since_last_save:0
bgsave_in_progress:0
last_save_time:1316147093
bgrewriteaof_in_progress:0
total_connections_received:8
total_commands_processed:7
expired_keys:0
hash_max_zipmap_entries:64
hash_max_zipmap_value:512
pubsub_channels:0
pubsub_patterns:0
vm_enabled:0
role:master
db0:keys=29,expires=0

The used-memory(i assume it is in bytes) and the db0 values always
remain to be the same irrespective of how many ever elements i insert
into the db.Can anyone please help me in understanding what exactly
above info means?

[Hampus Wessman]

It simply means that you have 29 distinct keys with values of varying
type (one may be a list and so on...) and Redis is using very little
memory. Perhaps you overwrite existing data when you put new elements in
there? It may be some old bug too. You're running version 1.3, which is
very old. Please consider upgrading to 2.2.

Regards,
Hampus

[Rajitha R]

But i am very new to redis and have installed redis very recently and
that too the latest version 2.2.12.When i run redis-server , i get the
following output too..

[4755] 16 Sep 10:56:18 - DB 0: 6557281 keys (6557281 volatile) in
8388608 slots HT.
[4755] 16 Sep 10:56:18 - 5 clients connected (0 slaves), 1446513024
bytes in use
[4755] 16 Sep 10:56:24 - DB 0: 6560110 keys (6560109 volatile) in
8388608 slots HT.
[4755] 16 Sep 10:56:24 - 5 clients connected (0 slaves), 1447078792
bytes in use
[4755] 16 Sep 10:56:35 - DB 0: 6561624 keys (6561623 volatile) in
8388608 slots HT.

How is this inconsistency happening..?..I dont understand..:(

[Rajitha R]

I even tried to reinstall the latest version and redo everything.The
elements seem to be adding to the db but the ./redis-cli info command
still gives the same output..

[Hampus Wessman]

Not sure. Double check that you have exactly one instance of Redis
running and that it is the latest version. Things like that can easily
create mysterious problems.

Also, note that all keys are volatile. Are you using expiration? Is that
the reason they disappear again? Try inserting a few keys with "SET
some-key-name some-value" (with different key names) from redis-cli and
make sure that works.

BR,
Hampus

[Rajitha R]

I am running only 1 instance of redis running ..the expiry period is
6hrs which is big enough before they could disappear..I have tried to
insert from redis-cli and it works there.The no of keys get
incremented.But my probelm remains the same..Even the version no. it
shows is incorrect..

> >>>> I have ran theredis-cliinfocommand and see the following output

> >>>>>> You can use theredis-cliINFOcommand to know how much

> >>>>>> memory is used by Redis.
> >>>>>> Redis stores the data in RAM, so you need to be sure to have
> >>>>>> enough RAM. It mostly depends on the number and average
> >>>>>> size of your items. You could try to load 1 million of them,

> >>>>>> check the memory consumption withINFO, and extrapolate.

[Rajitha R]

I have given just the ./redis-cli and get the following prompt:

rajitha@rajitha-ThinkPad-Edge:~/redis-2.2.13-slave/src$ ./redis-cli
redis 127.0.0.1:6379>

But i had set the port number in my conf file as 6380 since for 6379 i
was getting an error that bind address already in use.My script also
was listening to redis on the same port.Is it because of this reason
that i was getting the inconsistency? What do i do so that redis-cli
shows me the ryt port?

[Rajitha R]

I got it now..

./redis-cli -h 127.0.0.1 -p 6380
redis 127.0.0.1:6380> info

and then i get the correct output.

Thanks..

[Hampus Wessman]

Yes, that's the problem. Try './redic-cli -p 6380' instead. You probably
have an old redis package installed (or something similar) that
autostarts another redis on 6379. Check 'ps -A -o pid,user,command |
grep redis'.

BR,
Hampus

[Hampus Wessman]

Great :)