Re: [ANN] Scarlet 0.6.0 - An HTTP interface for Redis

[Michael Gorsuch]

Nick - thank you!  I love the idea of a Go frontend largely due to ease of deployment (don't need to make sure a supporting runtime is installed).

When you speak of replication setups, are you considering leveraging sentinel here?

Best,

Michael Gorsuch

On Thu, Nov 1, 2012 at 2:21 PM, Nick Saika <nick...@gmail.com> wrote:

In my spare time, I have been working on an HTTP frontend for Redis, written in Go: Scarlet. 

Version 0.6.0 of Scarlet is the first version to be "fully-functional"; you can create, read, update and delete keys. Some of my future plans for Scarlet include:

• Providing clustering support 
  
• Automatic failover for replication setups
• A nice, web-based monitoring interface for your clustering and replication setups
• Act as a load-balancer in front of a cluster, including forwarding all write operations to masters, and all read operations to slaves

If anyone is interested in trying it out, I would welcome any feedback!

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/8eyz24uB5OAJ.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Nick Saika]

To be perfectly honest, Michael, I am taking the approach of assuming there is no Sentinel; that being said though, I have to admit I haven't looked too much into Redis Sentinel, and I will definitely do that over my vacation, and see if I can leverage any of the feature(s) provided by Sentinel.

My initial plan for assisting with Redis' master-slave replication, was to build up a list of nodes in the chain via data gleaned from the INFO command. Should a master fail out, for any reason, Scarlet would elect one of that master's slaves to become the new master, and then update any of the old master's slaves to point to the new master. I haven't thought too-too far into this yet, but my initial thought would be to have the old master (once it comes alive again), to become a slave of the new master.

[CharSyam]

Good ^^ Thank you Nick.

2012/11/1 Nick Saika <nick...@gmail.com>

To view this discussion on the web visit https://groups.google.com/d/msg/redis-db/-/qVMMqzfprtwJ.

[Josiah Carlson]

Considering that you are describing essentially what Redis Sentinel
does, only Redis Sentinel does a lot more work to make sure it isn't
doing bad things, you really should work off of the assumption that
Redis Sentinel exists.

Regards,
- Josiah

> https://groups.google.com/d/msg/redis-db/-/qVMMqzfprtwJ.

[Andrea Campi]

On Fri, Nov 2, 2012 at 5:43 AM, Josiah Carlson <josiah....@gmail.com> wrote:

Considering that you are describing essentially what Redis Sentinel
does, only Redis Sentinel does a lot more work to make sure it isn't
doing bad things, you really should work off of the assumption that
Redis Sentinel exists.

I was thinking about that the other day on the bus…

Working for some time on GemFire left me with a healthy respect with what they've done…

And one of the thing that is quite normal customary in that context, is running a slave inside each client app.

In Redis—would it be useful for a smart client (and Scarlet qualifies) to essentially include most of the Sentinel code?

[Dvir Volk]

On Fri, Nov 2, 2012 at 6:43 AM, Josiah Carlson <josiah....@gmail.com> wrote:
> Considering that you are describing essentially what Redis Sentinel
> does, only Redis Sentinel does a lot more work to make sure it isn't
> doing bad things, you really should work off of the assumption that
> Redis Sentinel exists.

This. Sentinel is not finished yet, but in my company we're in the
process of integrating it into production, while hacking it a bit. So
my guess is that it will mature in the coming months.

What we're missing and added a patch for, was to enable some sort of
orchestration via sentinel -that is, a slave goes up, it doesn't need
to know who is his master, just the addresses of the sentinels, and
its "master name" or as I call it, "role". It connects to the
sentinel, asks for the master for that role, and connects to it. It
makes auto scaling redis master/slave clusters very very easy.

I hope our patches will be seen favorably by the powers that be, hint hint ;)

Dvir Volk
Chief Architect, Everything.me
http://everything.me

[Nick Saika]

To clarify my goals on my intentions for Scarlet, I was hoping to have Scarlet slide into the environment similarly to how Varnish slides in to your web stack - in front of stuff. If I can mimic Sentinel (after I get some time to read the document, and any code) I will, but I'm not looking to replace it. 

[Nick Saika]

Additionally, I was initially approaching this project as a simple web frontend to Redis, to make it a worthy contender with Riak (mind you, Scarlet + Redis is still two, moving parts versus one), because truth be told, I do enjoy Riak's HTTP interface, but I would much rather use Redis because of its wonderful types (sets, lists, sorted sets, and hashes).

As I started writing Scarlet, it seemed to have the potential to act as a clustering/replication management solution, should Redis Sentinel not be available to you (think of slow-moving enterprise-y setups, where everything must be standardized). That being said though, clustering is not my "next task". After I switch out the underlying driver (which is almost done), my next work item is going to be master-slave discovery insofar as sending all write-op commands to master(s), and load balancing all read-op commands between slaves.

[Josiah Carlson]

Have you benchmarked your http interface yet? No offense intended, but
I would be surprised if the parsing overhead of http, along with the
overhead of sending and receiving traffic (especially the data
copy/buffering) wouldn't significantly *reduce* the performance of the
stack.

In particular, HAProxy is the fastest known software HTTP proxy
(though it got beta SSL support in the most recent development release
:D ), and it peaks at 100k requests/second on a single machine,
without passing connections through. When using passthrough, it sees
40k requests/second. But Redis itself can handle 100k requests/second
easily across persistent connections, and with pipelining, I see
300k-400k/second without issue. With a bit of processor affinity
tweaking, I can see over 700k pipelined requests/second on a 2.4 ghz
core2 duo.

While it's awesome to want to build an http interface to Redis (thus
simplifying a variety of use-cases), note that you will likely be
slowing down Redis, and may never see a payoff in your load-balancing
proxy, unless each of your requests is expensive (large sort, union,
intersection, or difference operations).

Regards,
- Josiah

> https://groups.google.com/d/msg/redis-db/-/Hfgv0mMRDHMJ.

[Nick Saika]

I haven't done any benchmarking yet, but I figured there would be a bit of a slow-down; I mean, after all, Scarlet is a layer in front of Redis.

After the functionality goes in, I will work on optimization, and at an initial glance, this optimization may even be in the form of some very, short-lived, in-memory caching, Scarlet-side (maybe 50-100ms, for starters).