sharedobjects

[Aman Gupta]

Is there a way to tell how many objects are being shared? I'm working
a large dataset and not seeing very much memory savings with
sharedobjects, even though I know I have a lot of sets with the same
strings.

sharedobjects no:

08 Jul 08:17:42 . 0 clients connected (0 slaves), 1815084578 bytes in use
08 Jul 08:17:47 . DB 0: 60503 keys (0 volatile) in 65536 slots HT.
08 Jul 08:17:47 . DB 1: 2634133 keys (0 volatile) in 4194304 slots HT.
08 Jul 08:17:47 . DB 3: 97 keys (0 volatile) in 128 slots HT.
08 Jul 08:17:47 . DB 4: 6 keys (0 volatile) in 8 slots HT.
08 Jul 08:17:47 . DB 5: 97 keys (0 volatile) in 128 slots HT.
08 Jul 08:17:47 . DB 15: 21 keys (0 volatile) in 32 slots HT.

sharedobjects yes:

08 Jul 08:15:36 . 0 clients connected (0 slaves), 1804292361 bytes in use
08 Jul 08:15:41 . DB 0: 60503 keys (0 volatile) in 65536 slots HT.
08 Jul 08:15:41 . DB 1: 2634133 keys (0 volatile) in 4194304 slots HT.
08 Jul 08:15:41 . DB 3: 97 keys (0 volatile) in 128 slots HT.
08 Jul 08:15:41 . DB 4: 6 keys (0 volatile) in 8 slots HT.
08 Jul 08:15:41 . DB 5: 97 keys (0 volatile) in 128 slots HT.
08 Jul 08:15:41 . DB 15: 21 keys (0 volatile) in 32 slots HT.

[Salvatore Sanfilippo]

On Wed, Jul 8, 2009 at 10:22 AM, Aman Gupta<themast...@gmail.com> wrote:
>
> Is there a way to tell how many objects are being shared? I'm working
> a large dataset and not seeing very much memory savings with
> sharedobjects, even though I know I have a lot of sets with the same
> strings.

Hello Aman,

indeed shareobjects is saving too little RAM in your use case, you can
make a much larger shared objects pool using redis.conf:

shareobjectspoolsize 1024

this is the default, try with a very large value

shareobjectspoolsize 100000

and se what happens... of course the pool itself will use more memory
but should be able to reach a better saving, so the best thing to do
probably is to experiment with different values.

Cheers,
Salvatore

>
> sharedobjects no:
>
> 08 Jul 08:17:42 . 0 clients connected (0 slaves), 1815084578 bytes in use
> 08 Jul 08:17:47 . DB 0: 60503 keys (0 volatile) in 65536 slots HT.
> 08 Jul 08:17:47 . DB 1: 2634133 keys (0 volatile) in 4194304 slots HT.
> 08 Jul 08:17:47 . DB 3: 97 keys (0 volatile) in 128 slots HT.
> 08 Jul 08:17:47 . DB 4: 6 keys (0 volatile) in 8 slots HT.
> 08 Jul 08:17:47 . DB 5: 97 keys (0 volatile) in 128 slots HT.
> 08 Jul 08:17:47 . DB 15: 21 keys (0 volatile) in 32 slots HT.
>
> sharedobjects yes:
>
> 08 Jul 08:15:36 . 0 clients connected (0 slaves), 1804292361 bytes in use
> 08 Jul 08:15:41 . DB 0: 60503 keys (0 volatile) in 65536 slots HT.
> 08 Jul 08:15:41 . DB 1: 2634133 keys (0 volatile) in 4194304 slots HT.
> 08 Jul 08:15:41 . DB 3: 97 keys (0 volatile) in 128 slots HT.
> 08 Jul 08:15:41 . DB 4: 6 keys (0 volatile) in 8 slots HT.
> 08 Jul 08:15:41 . DB 5: 97 keys (0 volatile) in 128 slots HT.
> 08 Jul 08:15:41 . DB 15: 21 keys (0 volatile) in 32 slots HT.
>
> >
>

--
Salvatore 'antirez' Sanfilippo
http://invece.org

"Once you have something that grows faster than education grows,
you’re always going to get a pop culture.", Alan Kay

[Aman Gupta]

On Wed, Jul 8, 2009 at 3:15 AM, Salvatore Sanfilippo<ant...@gmail.com> wrote:
>
> On Wed, Jul 8, 2009 at 10:22 AM, Aman Gupta<themast...@gmail.com> wrote:
>>
>> Is there a way to tell how many objects are being shared? I'm working
>> a large dataset and not seeing very much memory savings with
>> sharedobjects, even though I know I have a lot of sets with the same
>> strings.
>
> Hello Aman,
>
> indeed shareobjects is saving too little RAM in your use case, you can
> make a much larger shared objects pool using redis.conf:
>
> shareobjectspoolsize 1024
>
> this is the default, try with a very large value
>
> shareobjectspoolsize 100000

awesome, thanks! I ran into a bug where it would not use the value
from the config, patch:

diff --git a/redis.c b/redis.c
index f4fa061..d44e900 100644
--- a/redis.c
+++ b/redis.c
@@ -751,7 +751,7 @@ static int serverCron(struct aeEventLoop
*eventLoop, long long id, void *clientD

/* Show information about connected clients */
if (!(loops % 5)) {
- redisLog(REDIS_DEBUG,"%d clients connected (%d slaves), %zu
bytes in use",
+ redisLog(REDIS_DEBUG,"%d clients connected (%d slaves), %zu
bytes in use, %d shared objects",
listLength(server.clients)-listLength(server.slaves),
listLength(server.slaves),
server.usedmemory,
@@ -900,6 +900,7 @@ static void initServerConfig() {
server.dbfilename = "dump.rdb";
server.requirepass = NULL;
server.shareobjects = 0;
+ server.sharingpoolsize = 1024;
server.maxclients = 0;
server.maxmemory = 0;
ResetServerSaveParams();
@@ -930,7 +931,6 @@ static void initServer() {
server.el = aeCreateEventLoop();
server.db = zmalloc(sizeof(redisDb)*server.dbnum);
server.sharingpool = dictCreate(&setDictType,NULL);
- server.sharingpoolsize = 1024;
if (!server.db || !server.clients || !server.slaves ||
!server.monitors || !server.el || !server.objfreelist)
oom("server initialization"); /* Fatal OOM */
server.fd = anetTcpServer(server.neterr, server.port, server.bindaddr);

with this patch, I am seeing some savings:

09 Jul 02:38:46 . 0 clients connected (0 slaves), 1539137609 bytes in
use, 5992320 shared objects

Aman

[Salvatore Sanfilippo]

Thanks Aman,
just merged.

On Thu, Jul 9, 2009 at 4:39 AM, Aman Gupta<themast...@gmail.com> wrote:

> with this patch, I am seeing some savings:
>
> 09 Jul 02:38:46 . 0 clients connected (0 slaves), 1539137609 bytes in
> use, 5992320 shared objects
>

--