Replicated Space with Redis? Interest check

[Borna Novak]

Hello all :)

I did some benchmarking issues and got pretty bad results with concurrent connections on `SpaceUtil#nextLong` with the default persitent store and I'm also nervous about the DB files getting squashed by an ignorant sysadmin,  so I'm looking to move away from the default persistent store into something which is non-local, distributed, scalable and performant,

like many other people I'm already using Redis for other things in the infrastructure, it's well supported by cloud operators and it seems it would be pretty easy to make a good Linda interface implementation there that would be scalable to your hearts content,

I would base the adapter around https://github.com/mrniko/redisson probably, is there an interest for something like this to be merged into master?

Are there development guidelines I should follow?

Kind regards,

Borna

[Victor Salaman]

Hi:

I recently spent some time looking for the best way to achieve this as I needed something more robust than the filesystem approach or replicated space. After a lot of thought I decided that I was overthinking it, so I did it inside the database, which would be consistent among all replicas in a master/slave setup.

As for Redis, It'd be interesting. As long as you're aware that Redis follows a eventual consistency model and that in a distributed scenario without distributed locking, you might get collisions or unexpected behavior. If this is acceptable then I see it'd work perfectly as long as you have only a single master doing the INCR operation.  

/V

--
--
jPOS is licensed under AGPL - free for community usage for your open-source project. Licenses are also available for commercial usage. Please support jPOS, contact: sa...@jpos.org
 
Join us in IRC at http://webchat.freenode.net/?channels=jpos
 
You received this message because you are subscribed to the "jPOS Users" group.
Please see http://jpos.org/wiki/JPOS_Mailing_List_Readme_first
To post to this group, send email to jpos-...@googlegroups.com
To unsubscribe, send email to jpos-users+...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/jpos-users
---
You received this message because you are subscribed to the Google Groups "jPOS Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jpos-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/fd5be042-0627-4320-9819-6e2791b2cfd1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Alejandro Revilla]

Hi Borna,

First a little bit of history, a long time ago (very early 2000s) we had a Sequencer class in jPOS that had a dependency on a ‘reliableLogger’ that used to come with JINI (an awesome project that lost traction once it was handed to Apache and renamed to River). In order to get rid of that dependency, I created this SpaceUtil.nextLong method that is far from efficient, if you look at how it’s implemented, it requires two space operations in order to increment a value. That’s fast on a transient space, but JE space actually commits to disk (and syncs), so it’s not fast.

I made a quick benchmark and got between 0.6ms and 1ms penalty for each nextLong operations (using 100 simultaneous threads) using JESpace. It’s slower with JDBMSpace and super fast with TSpace.

We could add an optimised Space.inc operation, but we would be loosing the Linda spirit where everything can be done on top of the three basic operations (BTW, our ‘push’ operation is a violation of that spirit).

If what you need is just a sequencer, Victor is right, you can use the database, or if your business logic can tolerate a gap in a crash event, you could have a service issuing sequence numbers and getting them in chunks (i.e incrementing by 10 or 100).

But writing Space implementations is nice, I like it at lot. Some people play Sudoku, I like to write space implementations. I’ve written a few of them that were never published using different backends, like Voldemort for example. When I first saw Redis I dove into writing a Space implementation (using JEdis as the driver). Most operations map quite well from our Space API into Redis’, i.e:

    public void out(K k, V v) {
        jedis.rpush(k.toString(), v.toString());
    }
    public V in(Object o) {
        return (V) jedis.blpop(0, o.toString()).get(1);
    }
    public V in(Object k, long timeout) {
        List l = jedis.blpop((int) (timeout/1000), k.toString());
        return (V) (l != null ? l.get(1) : null);
    }
    public V inp(Object o) {
        return (V) jedis.lpop(o.toString());
    }
    public V rdp(Object o) {
        return (V) jedis.lindex(o.toString(), 0);
    }

​I remember I liked the fact that Redis has timeouts for the entries, that’s great, but it doesn’t have timeouts for the queue entries, so I found there was some friction to implement that.

When playing with Space implementations using different backends I always have mixed feelings, to give you an example, a few months ago I’ve started a ZooKeeper based implementation, and it was going well, but then I realised that ZK has a great API, with ephemeral versus persistent entries that I was missing by plugging the Space API as a front end, so instead of that I switched my attention to implement a ZK based MUX, not using the Space API but honouring the MUX API, taking advantage of ZK’s capabilities. I heard about Redisson but never got to use it, looks very interesting.

If you are keen to write a Space implementation and you’re happy to donate it to jPOS, we’ll be happy to merge it to jPOS-EE as a module initially, if it takes traction we may move it to jPOS core. I try to add stuff that requires dependencies as jPOS-EE modules, so they are optional for developers. We would just need a pull request with your CLA (https://github.com/jpos/jPOS/tree/master/legal). You can also host it in your own project and we’ll be happy to use it if the license allows.

If you choose to move forward, I suggest you join us in Slack (just sent you an invite - anyone willing to join just drop me an e-mail and I’ll send you invites right away).

​

​

--

@apr

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/CALK1SywSv38CamTpq4hUV-VvjhsCkvdTS0uwTv_%2BqKfToHUOKw%40mail.gmail.com.

[Borna Novak]

Hm, very good points Alejandro,

I've been having a bunch of ideas about possible Redis uses with jPOS, from publishing from the log log stream & queues to replicated space implementation to publishing background jobs on various Redis-backed workers such as Sidekiq (Ruby + Java), RQ (Python), Goworker (Go) etc

I'm in the middle of developing a project for which I'll be developing this as things stand now, I'll be very happy to contribute extractable components (such as a Jetty component that wraps around SMAdapter & Channel implementations, verry fast)

Borna

[Alejandro Revilla]

Awesome Borna. 

Please keep ZooKeeper (and Curator) in mind. That's what I'm currently playing with (started writing a ZK based Space implementation, but now I'm dreaming about a fully distributed Q2 ensemble, I'm going to call it Z2 - hope it gets to fly). MUX (will call it ZMUX) is particularly well suited to be implemented with ZK as a backend.

--

@apr

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/21d32bfb-275b-4e60-8846-671b5a9d71f0%40googlegroups.com.

[Victor Salaman]

Hmm... 

Nice idea... Consul based C2 and CMUX! :)

/V

To view this discussion on the web visit https://groups.google.com/d/msgid/jpos-users/CAAgSK%3DkMJTVDTa_nK6XV7xtquDwJ49_8CsMtVS4xKrVACz9ESA%40mail.gmail.com.