Extremely slow save/bgsave with vm enabled
a.
Hi,
I have a fairly small database:
- ~4.3M keys
- ~100k hashes (60-200 members)
- the rest are sorted sets with avg. 600 members
- max memory is set to 3gb
- vm is set to use 6gb swap file
- vm page size is 32 bytes (experimented with 128 and 2048, no changes)
I've bulk loaded the data from an sql database, so auto save is disabled. At ~3M keys i started a bgsave to test the performance and it's been an ahour ago and it's still running. I've seen in redis-tool that the server is swapping in all the items from disk (i reached the memory limit at ~2.5M keys), eventually reaching 5.3GB memory use.
The Activty Monitor shows 2mb/sec write peaks in every 5-10 seconds and my temp-PID.rdb is still only 200MB (should be around 8-900mb when finishes). The machnie is not IO nor CPU capped, so i have no idea what's going on.
macpro:~ a$ top
Processes: 89 total, 5 running, 1 stuck, 83 sleeping, 397 threads 01:06:45
Load Avg: 0.19, 0.22, 0.35 CPU usage: 2.52% user, 7.33% sys, 90.13% idle
SharedLibs: 6348K resident, 6880K data, 0B linkedit.
MemRegions: 23229 total, 6008M resident, 99M private, 2623M shared.
PhysMem: 717M wired, 7024M active, 3507M inactive, 11G used, 12M free.
VM: 218G vsize, 1042M framework vsize, 4002975(46) pageins, 702641(80) pageouts
The redis-server is the latest from github, currently compiled with -O3, but the default -O2 behaves the same. I'm on mac os x 10.6.5
Is the save supposed to be this slow? Any pointers on performance tunings?
a.
a.
unusable for real world usage.
- vm limit: 4gb
- memory in the server: 16gb
- redis memory usage is around 4-4.5 gigabytes
- total dataset size is ~6.5 gb
- bgsave takes _5_ hours on an SSD drive
- normal save takes ~15 minutes
with the change i mentioned in my other email i could take bgsave down
to ~4 hours but this is still unacceptable.
enabling AOF is not a solution, because in this case the startup takes
at least an hour (compared to the 10 minutes of the snapshot version)
any plans on making the VM more efficient?
a.
Salvatore Sanfilippo
> after some i can conclude that enabling VM makes redis totally
> unusable for real world usage.
Hello a,
yes with VM enabled, and many values swapped, loading / saving the DB
can be very slow since there is currently a lot of CPU used that could
be avoided, since values are loaded from swap to memory, then
converted again into the serialization format, while actually the VM
swapped objects and the .rdb objects have exactly the same format.
While it is indeed possible to improve this, we are currently working
much more on decreasing the memory usage of data in memory (see 2.2
release) and Redis Cluster. This are in my opinion both more
interesting long term solutions. We still plan to optimize the VM
loading/saving process in the future.
Since Redis master on github contains a modified version of the VM
compared to Redis 2.0, can you please re-try it against the new
version? It is possible that you'll see an improvement.
Btw I hope to find some time to profile this stuff in the next days to
check what is the main problem, but I think it's definitely the disk
-> memory -> disk business, while it should instead be a trivial disk
-> disk stuff. Only thing needed for this is prefixing all the VM swap
objects with the length of the encoded object. Not a lot of work and
may result in huge performance advantage.
Cheers,
Salvatore
--
Salvatore 'antirez' Sanfilippo
http://invece.org
"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele
a.
a.
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To post to this group, send email to redi...@googlegroups.com.
> To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.
>
a.
Decreasing the memory usage sounds good, but eventually there will be a point when the dataset cannot fit into the memory.
In my case the test data was only ~10% of the real data, which will be around 60 GB (and growing). Usually only 10-16% of the data is accessed, the rest will be laoded very rarely. (lots of historical data)
based on this i cannot warrant installing a 4-6 machine cluster, nor can i put 128 (and later 256 or more) GB memory into one machine.
also, memory saving optimizationsn should be carefully weighted. sometimes it's not worth saving 200mb of memory, because of the performance hits.
sorry for linking back again to the compareString stuff :), but for example if zsets would not compress the numeric keys then a lot of performance hit could be avoided.
if i have a key "12345678" then it will be stored as bc614e, saving 5 bytes. even if i have 10 million items it's only ~50 mb saved. and because of this compression redis must convert these 3 bytes back to the string representation several times. in a zset-only dataset the ll2string takes up 47% of the loading time.
i'm not afraid of making my hads dirty so if you have specific optimizations in mind for the VM, or just some pointers i'd be more than happy to spend some time on it.
i had to change back to mongodb (which im not happy about) after i got burned by the extreme loading times (i even wrote a client for redis), and i'd like to switch back to redis in the near future.
On Dec 1, 2010, at 3:31 PM, Salvatore Sanfilippo wrote:
Jak Sprats
if you can effectively partition your data, then VM will work for you.
10-16% of your data is "active". If you can separate the other 86-90%
into distinct key-spaces, then VM will do exactly what you want to do.
Example: the ZSET yearly_values, if you have values
[2010 ....][2009 ....][2008 ...]...[1999 ...][1998 ...].... data back
to the stone ages :)
And all you need is data from 2000+, then splitting up your data into
centuries would solve your problem
e.g.
yearly_values_2000 [2010 ....][2009 ....][2008 ...]...
yearly_values_1900 [1999 ...][1998 ...]....
yearly_values_1800 .....
In practice, only yearly_values_2000 will be in memory during usage.
On the loading time problem, if your use case does not need ll2string
and you have tested that everything works w/o it, then fork redis and
comment it out (this is open source code :)
- Jak