Mass Insertion from mysql to Redis DB

[Barun Kumar]

Hello All,

I am new to redis.

I am facing problem in mass Insertion from Mysql to Redis DB using java and spring-data redisTemplate.

Problem statement :

1.I have a mysql database with five column containing 3 crore rows 

   i have made key as composition of two column and each key have list of records.

2. I have used java program to access the jdbc ResultSet and  and iterating one by one and making list per key 

   and inserting into the redis using Spring-data redisRemplate.

   It is taking huge time to populate into the redis Databse.

It would be great if anybody help me or suggest me best way to do this or share some script to achieve this.

Thanks &Regards,

Barun 

 . 

[AlexanderB]

One major thing you can do to speed up the process would be to use pipelines to do some batching of the inserts into Redis. 
For reference there are some high level docs of the concept:
https://redis.io/topics/pipelining

In particular gathering enough entries from mysql into memory in java to fill up an entire packet before sending it off to Redis can be a massive performance improvement. Redis pays a fair bit of overhead per packet when parsing incoming commands, but you can often fit many small commands in each packet, reducing the overall number of packets Redis needs to process.

Here's another note on that subject from the benchmarking page: https://redis.io/topics/benchmarks
"When an ethernet network is used to access Redis, aggregating commands using pipelining is especially efficient when the size of the data is kept under the ethernet packet size (about 1500 bytes). Actually, processing 10 bytes, 100 bytes, or 1000 bytes queries almost result in the same throughput."  

There's lots of good performance tips in there, but I think using pipelining would be the biggest improvement for the effort required. I'm not familiar with the spring-data redis tech you are using though, but the docs seem to indicate this should work. http://docs.spring.io/spring-data/redis/docs/current/reference/html/#pipeline

[Barun Kumar]

Thank AlexanderB.

I have generated redisprotocal from mysql and imported to redis db.

It is working fine in the linux[through bash shell script] but

i am not able to execute this in window 7 the redis pipe command.

Any idea??

Regards,

Barun

--
You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/V6AuB8PhLMw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to redis-db+unsubscribe@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at https://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

[CharSyam]

What is your redis client?
maybe it supports redis pipeline, redis pipeline is not real pipe.

>> redis-db+u...@googlegroups.com.

>> To post to this group, send email to redi...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/redis-db.
>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to redis-db+u...@googlegroups.com.

[Barun Kumar]

StringRedisTemplate [This is provided by Spring ]

>> redis-db+unsubscribe@googlegroups.com.

>> To post to this group, send email to redi...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/redis-db.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --

> You received this message because you are subscribed to the Google Groups
> "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to redis-db+unsubscribe@googlegroups.com.

> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to a topic in the Google Groups "Redis DB" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/redis-db/V6AuB8PhLMw/unsubscribe.

To unsubscribe from this group and all its topics, send an email to redis-db+unsubscribe@googlegroups.com.

[CharSyam]

actually I dont know well string redis template.

I just found this example from internet

I hope this is helpful

 StringRedisTemplate redisTemplate = ctx.getBean("stringRedisTemplate", StringRedisTemplate.class);

SetOperations<String, String> ops = redisTemplate.opsForSet();

long start = System.currentTimeMillis();

for(int i=0; i < 100000; i++){

boolean rslt = ops.add("key:"+i, ""+i);

}

long end = System.currentTimeMillis();

System.out.println("Simple SET: " + ((end - start)/1000.0) + " seconds");

////////////////////////////////////////////////////////////////////////////////////////////////////////////

start = System.currentTimeMillis();

redisTemplate.execute(new RedisCallback<Object>(){

 

@Override

public Object doInRedis(RedisConnection connection)

throws DataAccessException {

 

connection.openPipeline();

for(int i=0; i < 100000; i++){

connection.set(("key."+i).getBytes(), (""+i).getBytes());

}

List<Object> rslts = connection.closePipeline();

System.out.println(">>>>>>>>>>> " + rslts.get(0));

return null;

}

});

end = System.currentTimeMillis();

System.out.println("Pipelined SET: " + ((end - start)/1000.0) + " seconds");

2017년 1월 25일 수요일, Barun Kumar<barun...@gmail.com>님이 작성한 메시지: