How to save keys fast

[Viettel Vas]

Hi there,
I'm newbie with Database and NoSQL and need some helps about how to use redis efficient.
I have to update, synchronize about 90 million records ( only id number of tables, not complex data) from MySQL to redis. What I do is fetching 100000 rows each time then putting them on redis.
So far my code is:
RedisTemplate template= new RedisTemplate();
//
.... set up connection for redis, mySQL
//
List<String> groupId = new ArrayList<String>();
groupId =fetchGoodId(db, field, table, startNumber, endNumber); // 100000 rows
                    for (int i = 0; i < groupId .size(); i++) {
                        String id= value+groupId .get(i);
                        template.opsForValue().set(id, "0");
                    }
However, the writing speed was unexpected, only 2000 keys/second. That took 1 hour for 7 miilion keys. My teacher was so disappointed about this, he asked me to improve the efficency because his Mongol DB is way faster, stores 5 millions keys around 10 minutes.
So I come here to ask for a solution for this. I think redis is very fast and even far more than Mongol. Please show me a better way to finish my task.
P/S: I'm using MySQL 5.4 and redis 3.0.3
My libs are: jedis 2.1.0 for RedisTemplate

Thank in advance.

[Matt Palmer]

On Fri, Jul 31, 2015 at 09:55:06PM -0700, Viettel Vas wrote:
> List<String> groupId = new ArrayList<String>();
> groupId =fetchGoodId(db, field, table, startNumber, endNumber); // 100000
> rows
> for (int i = 0; i < groupId .size(); i++) {
> String id= value+groupId .get(i);
> template.opsForValue().set(id, "0");
> }
> However, the writing speed was unexpected, only 2000 keys/second. That took
> 1 hour for 7 miilion keys. My teacher was so disappointed about this, he
> asked me to improve the efficency because his Mongol DB is way faster,
> stores 5 millions keys around 10 minutes.

Well, the first thing your teacher should have taught you is to profile your
code, to find out where the bottleneck is. Don't assume that it is Redis'
fault. Use the tools available to you (your teacher should be able to show
you how) to answer the question, "Is Redis the bottleneck?". It's entirely
possible that the problem is somewhere else -- perhaps the speed with which
MySQL can supply the values.

If it *is* Redis, you can get a significant speed increase by using the MSET
command, rather than SET. I don't have any experience with the client
library you're using, so I can't give you any more detailed advice than
that, but I'm sure you and your teacher can work it out from the
documentation.

- Matt

[Marc Gravell]

To add to what Matt said: if the bottleneck *is* Redis, my first thought would be: round trips. If the library / methods you are using forces a full round trip per call, it will introduce latency. This is not required for Redis, however: you can pipeline as many ops on the wire as you like. Check the library to see if there is a pipeline mode.

Alternatively: as already noted, MSET is a compromise to reduce round trips without  pipelining, by allowing multiple ops on a single round trip. 50 items in an MSET will reduce your round trips (and thus: latency cost) by a factor of 50.

Marc

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.