Need help (feedback) with Mojo::Redis2 - A new pure-perl non-blocking I/O Redis driver

[Jan Henning Thorsen]

Hi,

I'm trying to make a new Redis driver for Mojolicious that makes sense. Me and Marcus Ramberg has been maintaining Mojo::Redis, but I see some issues with the API:

* No blocking support

* Really poor error handling

* Trying to be clever with BLPOP and SUBSCRIBE, using an overloaded on() method

* Some really weird bugs that that I can't seem to figure out (Got one project at work which fail on random)

* Public methods that is confusing: connect(), connected(), disconnect() (?), protocol_redis() and timeout()

So now I want to make a new version that Makes Sense (tm). The project is currently just a branch under the old mojo-redis repo: https://github.com/marcusramberg/mojo-redis/tree/v2

# What I need help with

1. Look if the API makes sense.

2. Look at the code to see if it makes sense.

3. Tell me when something is awkward or plain wrong.

4. Figure out if ->execute() makes sense and/or if it could be replace with something "nicer"

I often do many commands at once, so what execute() fix is this:

  Mojo::IOLoop->delay(

    sub {

      my ($delay) = @_;

      $redis->get(foo => $delay->begin);

      $redis->get(bar => $delay->begin);

      $redis->get(baz => $delay->begin);

    },

    sub {

      my ($delay, $foo_err, $foo, $bar_err, $bar, $baz_err, $baz) = @_;

      # ^ ONE MILLION ARGUMENTS!!

    },

  );

5. Does it make sense for one $redis object to have multiple connections to the database?

This is one of the things I built in to the new module: blpop(), subscribe(), ... will make a new connection to the database instead of user of the module need to remember which method is blocking or take over the connection. But does it really make sense?

6. I need a method for UNSUBSCRIBE

The module inherit from Mojo::EventEmitter, so I can't really use the unsubscribe() method to UNSUBSCRIBE from channels. Got an idea for an alternative way, or some fancy hack to make it work with unsubscribe()? I've been considering $redis->unsubscribe(message => $channel_name); but it kind of breaks the signature.

7. Do Mojo::Redis2 really need "encoding" attribute?

Why isn't this always UTF-8? I don't get the encoding attribute, but I kept it because of legacy from the other redis modules.

8. What do you think about pipelined()?

Please have a look at https://github.com/marcusramberg/mojo-redis/blob/v2/t/pipelined.t

9. ???

# What is done (might be redesigned though)

* I got basic redis methods working blocking and non-blocking

* I got (p)subscribe methods working and (p)message events

* I got pipelining working.

[Daniel Mantovani]

  Hi Jan, I´ve using Mojo::Redis and find it very convenient as the glue that allows to comunicate non-blocking among different processes (toads) when running hypnotoad (using subscribe and publish, just as the "Websocket example" in the module's synopsis).

 Something I would suggest for you to take a look is related to the scalling of simultaneos redis clients needed for a realtime app. I understand that you would need a dedicated "subscribe" client plus a "publish" per connected websocket, if you follow the mencioned example.

 That could be a problem depending on the nature of your app. In my case I was working on an app that will have thousand of appliances with very low traffic but allways on, so I ended up using Mojo ::Redis with something like a psubscribe "app:*" client per toad, and just one publishing client (probably blocking) also per toad (Anyway  I'm not sure how this solution will scale in the field, we will see in the followings months probably).

  Hope you can add to Mojo::Redis2 some kind of support for a solution that allows to share psubscribe redis connections among many users on a single process or toad.

BR,

Daniel

[Jan Henning Thorsen]

Mojo::Redis2 can (or at least it will be able to) (p)subscribe multiple times, using just one connection. I will try to illustrate the difference to be sure I understand you: 

  my $old = Mojo::Redis->new;

  my $s1 = $old->subscribe("foo");

  my $s2 = $old->subscribe("foo");

  # now you two connections to the Redis database

  my $new = Mojo::Redis2->new;

  $new->subscribe("foo");

  $new->subscribe("foo");

  # still just one connection to the Redis database, since Mojo::Redis2 is re-using the same subscribe connection

Unfortunatly, issuing (P)SUBSCRIBE will only allow "subscribe like" commands, which renders any Redis module unable to do PUBLISH and SUBSCRIBE over the same database connection. 

[Jan Henning Thorsen]

I had some typos:

I meant subscribing to "foo" and then "bar". (not sure if that matters for the example though)

Also the $old comments should be "now there is two connections to the Redis database".

[Daniel Mantovani]

Yes it absolutly matters, I need a different channel per connected websocket

So if I understood correctly inside my websocket route I could write something like:

my $applianceid = ... (some unique identifier for each appliance, I get it from a custom header)

my $redis = Mojo::Redis2->new;

$redis->on(message => "myapp:$applianceid", sub {

  my ($rself, $msg) = @_;

  .....

  do something with the received redis msg

  .....

  $tx->send($something); # use my transaction object to send some data to my connected appliance

  etc

});

...

and still use just one redis client despite I may have thousands of websocket clients connected and subscribed to different channels ("myapp:$applianceid")?

 In this case I think the solution would be perfect, I don´t care using one extra redis client for publishing, that I guess I will be able to share among all websockets.

 So at the end of the day I will be using just two redis clients per toad when running on hypnotoad... is that right or I missunderstood?

[Jan Henning Thorsen]

The on(message => $channel => ...) API is not going to be part of Mojo::Redis2. It will just be on(message => sub { my ($self, $message, $channel) = @_ }); The reason for that is to have a consistent API. (on() is already defined in Mojo::EventEmitter)

And yes, it will only have one connection to the Redis server pr. $redis object, even though you do $redis->subscribe("channel:$_") for 1..10000;

I'm planning to have one connection for the pubsub commands, a new connection when you call blpop, brpop, brpoplpush, and one connection for everything else. The reason for creating a new connection on each blpop(), brpop(), brpoplpush() is that they are blocking on the server side.

[Daniel Mantovani]

Perfect then, thanks Jan

[Jan Henning Thorsen]

I pushed an experimental release to CPAN now:

https://metacpan.org/pod/Mojo::Redis2

-Any- feedback is more than welcome.

[Daniel Mantovani]

I was trying to handle several subscribe channels with only one redis client using Mojo::Redis2, but it seems that I'm doing something wrong, I end up using as many clients as open subscribed channels I have.

Maybe somebody can take a look at this test code and let me know :

https://github.com/dmanto/test-redis-connections.git

Thanks in advance,

[Jan Henning Thorsen]

This makes a new connection on each request:

helper redis => sub { shift->stash->{redis} ||= Mojo::Redis2->new(url => $redis_server)};