Celluloid and highly concurrent writes

[Artem Yankov]

tl;dr is Celluloid good for doing high amount of concurrent writes to Redis?

I'm writing a rebalancing tool for Redis cluster (https://github.com/yankov/redis-migrator)

everything works, but I'm a bit stuck on making this tool fast. I need to do bunch of concurrent

write operations to Redis. Do you think celluloid could be a good solution for this kind of problem?

Redis cluster is hosted on separate machines. Say, I create a pool, and each actor populates a redis set with members. To do so I call future on a pool. Each time I go over 400 keys, (and pool size is also 400) actors begin to crash with ExitEvent or not enough memory (OS X, 8Gb RAM). But in real life I'll have to operate with millions and millions of keys and do it as fast as it possible. 

For example how I'm trying to populate sets in Reids https://github.com/yankov/redis-migrator/blob/master/migrator_benchmark.rb Am I doing anything wrong? Or maybe I picked the wrong tool for this job?

[Tony Arcieri]

So here's how Celluloid could help you:

• Celluloid provides non-blocking I/O with a synchronous interface via Celluloid::IO
• redis-rb provides a synchronous I/O interface
• With some work, redis-rb could support Celluloid::IO
• Celluloid::IO provides fast non-blocking I/O with a synchronous interface via nio4r

That said:

• Not a lot of people are using Celluloid::IO / nio4r and it's a bit buggy/greenfield/untested
• BUT I am 100% committed to making it stable
• BUT it's 100% guaranteed to be slower than EventMachine
• BUT the code you write on top of it is 100% guaranteed to be better than what you would write on top of EventMachine

Choose wisely ;)

--
Tony Arcieri

[Tony Arcieri]

On Wed, Jun 27, 2012 at 12:02 AM, Artem Yankov <artem....@gmail.com> wrote:

Redis cluster is hosted on separate machines. Say, I create a pool, and each actor populates a redis set with members. To do so I call future on a pool. Each time I go over 400 keys, (and pool size is also 400) actors begin to crash with ExitEvent or not enough memory (OS X, 8Gb RAM). But in real life I'll have to operate with millions and millions of keys and do it as fast as it possible. 

For example how I'm trying to populate sets in Reids https://github.com/yankov/redis-migrator/blob/master/migrator_benchmark.rb Am I doing anything wrong? Or maybe I picked the wrong tool for this job?

The problem is here:

https://github.com/yankov/redis-migrator/blob/master/migrator_benchmark.rb#L27

You are creating a MigratorBenchmark pool within a MigratorBenchmark actor. Creating the pool is in turn creating more MigratorBenchmark actors, so you are generating an unbounded number of actors to no useful end.

What you really want to do is change this line:

https://github.com/yankov/redis-migrator/blob/master/migrator_benchmark.rb#L37

To be:

bc = MigratorBenchmark.pool(["redis-host1.com:6379/1", "redis-host2.com:6379/1"],

                            ["redis-host1.com:6379/1", "redis-host2.com:6379/1", "redis-host3.com:6379/1"])

The code on line 29 should be moved elsewhere (e.g. into the toplevel code for the script below line 37, a class method, or a new class)

--
Tony Arcieri

[Artem Yankov]

Thanks a lot for your response! I split code into two classes and now creating a pool for MigratorBenchmark from a Populator class. I simplified an example also https://github.com/yankov/redis-migrator/blob/master/migrator_benchmark.rb. 

The same problem persists though - it starts failing if I'm trying to create more than 400 actors. But I think that might be that ulimit is not set properly on redis nodes, I'll try to play with it too.

Also, even with the pool size set to 400 and number of populating keys set to 400, I'm seeing that it spawns about 1.5k threads. Is it normal? I thought that number of threads should be equal to number of actors.

Also, does size of the pool means max amount of actors that can be running at the same time or total amount of actors that can be run?

[Tony Arcieri]

On Wed, Jun 27, 2012 at 11:33 PM, Artem Yankov <artem....@gmail.com> wrote:

Thanks a lot for your response! I split code into two classes and now creating a pool for MigratorBenchmark from a Populator class. I simplified an example also https://github.com/yankov/redis-migrator/blob/master/migrator_benchmark.rb. 

The same problem persists though - it starts failing if I'm trying to create more than 400 actors. But I think that might be that ulimit is not set properly on redis nodes, I'll try to play with it too.

You're making a new Redis::Distributed object each time, which is also making a bunch of TCP connections to all of those Redis hosts. This is extremely wasteful.

What you can do instead is define initialize for your MigratorBenchmark class and have it create a Redis::Distributed object

Also, even with the pool size set to 400 and number of populating keys set to 400, I'm seeing that it spawns about 1.5k threads. Is it normal?

It should only spawn 400 threads. Celluloid::Actor.all.count and Thread.list.count can give you more information

 

--
Tony Arcieri

[Artem Yankov]

Thanks, I'll try it out. The reason I create a separate redis connection for each actor is that I thought if I share the same connection between actors

they just gonna block each other. Am I wrong?

[Tony Arcieri]

If you create the connection in the initialize method, then each actor will have its own connection, e.g.

def initialize(hosts)

  @redis = Redis::Distributed.new(redis_hosts)
end

--
Tony Arcieri

[Artem Yankov]

Hm, but won't it create the same amount of Redis::Distributed objects?

[Tony Arcieri]

It will create a max of 400 of them, as opposed to one for each future

--
Tony Arcieri

[Artem Yankov]

Ah, right! Got it, thanks.