Key to big to sit in memory, what happens in case of a rdb save?

[Simone Mosciatti]

Hi all,

I am wondering what happens if I try to save a very big key in the rdb file.

I am working with SQLite databases and it would be very interesting to have the possibility to embed those db into the RDB file.

However it seems that the API suppose that every and each key is compact enough to sit in memory for the whole process, which is a sound assumption in the case of redis.

In my case it may be different because I may have a big database (too big to sit in memory) backed by a standard disk file.
Are there any way to work around this?

Thanks,

Simone

[Dvir Volk]

Hi Simone. 

Redis comes with the assumption that the entire data-set is in RAM the whole time, so once you violate that, things get a little weird :)

However, there are ways to overcome this. Suppose you want to copy the SQLite database file to the RDB - it doesn't have to fully reside in memory while you are doing that. 

Simply get a read handle to the database dump or whatever you are persisting, then read chunks from it, and write those chunks to the RDB. You'll need to mark the end of the stream somehow, to know when to stop reading, but this doesn't mean you have to read everything into memory before writing one huge block.

I hope this makes sense...

--
You received this message because you are subscribed to the Google Groups "redis-module-devs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-module-d...@googlegroups.com.
To post to this group, send email to redis-mo...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/redis-module-devs/6fa7857b-1706-4a1f-bf25-f493656247f4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Simone Mosciatti]

Hi Dvir,

it always a pleasure receive your answers and comments are always so helpful.

Yes, it completely makes sense, thanks!

Best,

Simone