Redis high spedd requirement

[Prince Philip]

Hi,

I have a specific requirement where 50 clients will update some 10K keys.
Each client will need to INCR on all 10K keys.
Since the rate at which each key may get incremented is 1000/sec, I want to share the load across different Redis Servers.

My plan is that each of my Client App machine will have one Redis server running.

Client will INCR the key values in local Redis Server.

Whenever any other client in other machine try to GET or INCR, it should get the latest value of the key that is in the group of Redis servers.

How can I do this?

I'm not using key (hash) sharding because, each client will need all 10K keys and without any hardcoded configuration or redirection, I want to access data from local Redis server.

For this I tried one master and slave model with slave having right permission.

But if I do ICNR in slave, the value is not getting updated in master redis.

Is there a master<->master mode I can try?

Any suggestion is much appreciated.

thanks,

Prince

[Itamar Haber]

Hi Prince,

Inline.

On Nov 30, 2015 11:26 AM, "Prince Philip" <prin...@gmail.com> wrote:
>
> Hi,
>
> I have a specific requirement where 50 clients will update some 10K keys.
> Each client will need to INCR on all 10K keys.
> Since the rate at which each key may get incremented is 1000/sec, I want to share the load across different Redis Servers.
>
> My plan is that each of my Client App machine will have one Redis server running.
> Client will INCR the key values in local Redis Server.
>
> Whenever any other client in other machine try to GET or INCR, it should get the latest value of the key that is in the group of Redis servers.
> How can I do this?

Use a Redis cluster that can serve 10M ops and change your plan - Redis is a REmote (DIctionary) Server and you gain little by trying to collocate it with the app if scales are involved. The pattern you're describing is more suitable to the in-memory data grid family of technologies.

> I'm not using key (hash) sharding because, each client will need all 10K keys and without any hardcoded configuration or redirection, I want to access data from local Redis server.
>

The cluster doesn't require that you do the sharding - it does that for you. If you use a smart cluster-compatible Redis client, it could further optimize communication with the cluster by tracking the cluster's slots topology and performing best effort routing based on that.

>
> For this I tried one master and slave model with slave having right permission.
> But if I do ICNR in slave, the value is not getting updated in master redis.
>

That is perfectly normal and the defined behavior. Redis' replication is strictly one way, from the master to its slave(s). Any writes to slave may be written over by the downstream replication stream.

> Is there a master<->master mode I can try?
>

Redis Cluster (geq v3) is a multi master shared nothing "mode". A shared everything, master-master model is not supported by Redis.

> Any suggestion is much appreciated.
>

Decouple the app from the database.

> thanks,
> Prince
>
> --
> You received this message because you are subscribed to the Google Groups "Redis DB" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
> To post to this group, send email to redi...@googlegroups.com.
> Visit this group at http://groups.google.com/group/redis-db.
> For more options, visit https://groups.google.com/d/optout.

[Prince Philip]

Thanks Itamer.

Even in the case of cluster, all keys are not present in all servers, right?

Hence, my client sometimes may get (depending on location of data), redirection response from some redis-server. Will it affect my Application performance?

Because I need high rate data update.

Is it a good idea to shard my keys on multiple Redis server and use Twemproxy for routing?

So that I can avoid redirections.

rgds,

Prince

[Itamar Haber]

Inline

On Mon, Nov 30, 2015 at 1:35 PM, Prince Philip <prin...@gmail.com> wrote:

Thanks Itamer.

Even in the case of cluster, all keys are not present in all servers, right?

Correct - the keyspace is partitioned into (hash) slots, every slot can be owned by a single member in the cluster at any given time.

 

Hence, my client sometimes may get (depending on location of data), redirection response from some redis-server. Will it affect my Application performance?

If your Redis client is smart, the effect is minimal since during normal operation slots do not just move around the cluster hence redirections should be non-existent.

 

Because I need high rate data update.

Is it a good idea to shard my keys on multiple Redis server and use Twemproxy for routing?

A proxy-based solution could, indeed, save the need for redirection but has its own implications (i.e.g., requires resources, increases overall complexity, does additional processing, may add network hops, API coverage, ...).

 

So that I can avoid redirections.

rgds,

Prince

On Monday, November 30, 2015 at 2:56:49 PM UTC+5:30, Prince Philip wrote:

Hi,

I have a specific requirement where 50 clients will update some 10K keys.
Each client will need to INCR on all 10K keys.
Since the rate at which each key may get incremented is 1000/sec, I want to share the load across different Redis Servers.

My plan is that each of my Client App machine will have one Redis server running.

Client will INCR the key values in local Redis Server.

Whenever any other client in other machine try to GET or INCR, it should get the latest value of the key that is in the group of Redis servers.

How can I do this?

I'm not using key (hash) sharding because, each client will need all 10K keys and without any hardcoded configuration or redirection, I want to access data from local Redis server.

For this I tried one master and slave model with slave having right permission.

But if I do ICNR in slave, the value is not getting updated in master redis.

Is there a master<->master mode I can try?

Any suggestion is much appreciated.

thanks,

Prince

--

You received this message because you are subscribed to the Google Groups "Redis DB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to redis-db+u...@googlegroups.com.
To post to this group, send email to redi...@googlegroups.com.
Visit this group at http://groups.google.com/group/redis-db.
For more options, visit https://groups.google.com/d/optout.

--

Itamar Haber | Chief Developer Advocate
Redis Watch Newsletter | Curator and Janitor
Redis Labs | ~ of Redis

Mobile: +1 (415) 688 2443
Office: +1 (650) 461 4652
Mobile (IL): +972 (54) 567 9692
Office (IL): +972 (3) 720 8515 Ext. 123
Email: ita...@redislabs.com
Twitter: @itamarhaber
Skype: itamar.haber

Blog  |  Twitter  |  LinkedIn

[guo...@gmail.com]

maybe you need the xRedis https://github.com/0xsky/xredis

Redis C++ client, support the data slice storage, support the connection pool
Features:

• data slice storage
• support Redis master slave connection, Support read/write separation
• connection pool
• simultaneously connected multiple data slice clusters
  
• most REDIS commands have been implemented
• multi thread safety