Introducing Redisql ...

[Jak Sprats]

Hi All,

I was planning on announcing Redisql (http://code.google.com/p/
redisql/) next week, but Alex Popescu beat me to it (http://
nosql.mypopescu.com/post/1353639583/low-latency-at-high-concurrency),
so here it goes.

I have been using redis for 6+ months and I am a gigantic fan of it on
so many levels. I am a C programmer and I needed additional
functionality, so I started tweaking the server code pretty much from
day one. The one thing I desperately needed was the richness of the
relational model @ redis' async non-blocking speed ... so I built a
SQL server on top of redis 2.0 and named it Redisql.

Redisql supports a large subset of SQL and ALL redis commands and it
supports "morph" commands (http://bit.ly/cNdb7L) that convert data
back and forth between redis data types (including sets of redis data
types) and relational tables.

Redisql has some great use cases for redis users (http://bit.ly/
diZqoW) which can be bullet pointed as
1.) Your SQL data and your Redis data under one roof - one server, one
client library, @ redis speed
2.) Import your data from mysql into redis simply and you can go
schemaless via redis-hashes :)
3.) Export your redis data to mysql if you need to back it up, archive
it, or do data analytics on it.
4.) Morph data in both directions between redis (sets-of-
string,lists,sets,zsets,hashes) and relational tables to fit your use-
case's optimal data structure

The clients for Redisql are (http://bit.ly/ccW5Ct):
1.) redisql-rb: Ruby client http://github.com/JakSprats/redis-rb (fork
of redis-rb)
2.) Redisqlska: PHP Client http://github.com/JakSprats/Rediska (a fork
of Rediska)
3.) Predisql: PHP Client: http://github.com/JakSprats/predis (a fork
of Predis)
4.) Node.js coming soon ... a prelim gist: http://gist.github.com/632740

Writing new clients should be trivial as Redisql is 100% compliant w/
redis' wire protocol.
Anyone interested in writing Redisql clients can read how Redisql uses
redis' protocol (http://bit.ly/9QpH8H) - I am more than happy to help
anyone w/ questions on writing clients.
The Ruby Redisql client is less than a 100 line patch to redis-rb.

The direction of redisql is to be a patch of redis, to remain being
built on top of redis and grow as redis grows. The code is tested and
stable but still young and the github code (http://github.com/
JakSprats/Redisql) is being heavily developed, w/ all sorts of hybrid
NOSQL/SQL commands in the pipeline.

I will be writing some blog entries soon on Redisql as many of the
commands and concepts are best explained w/ examples.

- Jak

[Dvir Volk]

Wow, this is awesome.

Are you using this in a production environment?

I am in the process of creating a front end oriented denormalized replica of a mysql database in redis, and this can save me tons of work.

--
You received this message because you are subscribed to the Google Groups "Redis DB" group.
To post to this group, send email to redi...@googlegroups.com.
To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/redis-db?hl=en.

[Paul Smith]

On Wed, Oct 20, 2010 at 6:48 AM, Jak Sprats <jaks...@gmail.com> wrote:

I was planning on announcing Redisql (http://code.google.com/p/
redisql/) next week, but Alex Popescu beat me to it (http://
nosql.mypopescu.com/post/1353639583/low-latency-at-high-concurrency),
so here it goes.

Interesting -- I was able to build it locally and play around with a few commands. I struggled a bit to get redisql-cli to parse my commands (looks like that part is a bit finicky with syntax), and I was able to crash redisql-server (though I can't reproduce it at the moment).

How do you see Redisql fitting in with Redis Cluster, or will it at all?

Cheers,

--
Paul Smith
@paulsmith
http://www.pauladamsmith.com/

[Xiangrong Fang]

Rather than fight with redisql, I would suggest Jak to collaborate
with antirez and the team to make redis better, for example:

- make redisql idea an "plugin" or "adapter" for redis
- make server side scripting of redis (e.g. define "macro" commands
which consists of a bunch of multi/exec wrapped commands)
- other enhancement in the area of improve redis persistence (aof etc)
and redis-cluster
- blah blah....

Shannon

2010/10/20 Paul Smith <paul...@gmail.com>:

[Ezra Zygmuntowicz]

I tend to agree, this would be cool as a redis plugin of some sort if there was a way to easily turn it on or off with a compilation flag or a configuration file option.

I played with it last night and thoug it was pretty cool. I would add Jak's additions to redis-rb as a plugin or option in the official redis/rb if there was a way to conditionally enable redisql on the server side.

Ezra

Sent from my iPhone

[Salvatore Sanfilippo]

On Wed, Oct 20, 2010 at 7:17 PM, Ezra Zygmuntowicz <ezmo...@gmail.com> wrote:
> I tend to agree, this would be cool as a redis plugin of some sort if there was a way to easily turn it on or off with a compilation flag or a configuration file option.
>
> I played with it last night and thoug it was pretty cool. I would add Jak's additions to redis-rb as a plugin or option in the official redis/rb if there was a way to conditionally enable redisql on the server side.

Hello!

replying to Ezra's email, but this is a general reply to Redisql thread.

Redisql in my opinion shows a few things. The first and most bold is
that Jak's is a remarkable hacker.
Another one is, and it's months that I think at it, that in-memory
btrees make sense. We should try to get good use in Redis of some code
in Redisql, starting from the btree code.

I've still to analyze it in details honestly, but assuming it is the
most brillant and stable thing ever, is this going to be merged with
Redis? I don't think so. It's a cool project but the Redis direction
is not to import an SQL implementation inside.

Implementing it as a plugin is hard since we don't have a plugin
system ;) And Redisql is a non trivial modification of Redis.
But anyway, it's not a matter of being able to include/exclude it. If
the right thing were to include it, why to optionally exclude it? This
is not going to make the binary huge or alike. It's just a matter of
what the Redis project is and what is the future directions we want to
take.

If it's possible to use the Redis code base to build interesting
things I'm happy, but this does not mean we should try to merge the to
projects.

Btw I was not able to compile it in my mac os x a few hours ago. I'll
try again in Linux tomorrow.

Cheers,
Salvatore

--
Salvatore 'antirez' Sanfilippo
http://invece.org

"We are what we repeatedly do. Excellence, therefore, is not an act,
but a habit." -- Aristotele

[Jak Sprats]

Dvir,
the core of the system was used in a production environment a few
months back. It had a very redis-centric syntax, not a SQL syntax, and
I showed it to a few people and they all were already familiar w/ SQL
and didnt want to learn my redis-btree syntax, so I built SQL on top
of it all, and then started adding some of the more advanced morph
commands.

Simple INSERT/UPDATE/DELETE/SELECT should be very solid ... NORM and
DENORM are bound to have some tiny bugs lurking somewhere.

Additionally, the redisql-benchmark has about ten new arguments to
stress test Redisql.
- Jak

On Oct 20, 4:34 am, Dvir Volk <dvir...@gmail.com> wrote:
> Wow, this is awesome.
> Are you using this in a production environment?
> I am in the process of creating a front end oriented denormalized replica of
> a mysql database in redis, and this can save me tons of work.
>

> > 2.) Redisqlska: PHP Clienthttp://github.com/JakSprats/Rediska(a fork
> > of Rediska)
> > 3.) Predisql: PHP Client:http://github.com/JakSprats/predis(a fork

> > of Predis)
> > 4.) Node.js coming soon ... a prelim gist:http://gist.github.com/632740
>
> > Writing new clients should be trivial as Redisql is 100% compliant w/
> > redis' wire protocol.
> > Anyone interested in writing Redisql clients can read how Redisql uses
> > redis' protocol (http://bit.ly/9QpH8H) - I am more than happy to help
> > anyone w/ questions on writing clients.
> > The Ruby Redisql client is less than a 100 line patch to redis-rb.
>
> > The direction of redisql is to be a patch of redis, to remain being
> > built on top of redis and grow as redis grows. The code is tested and
> > stable but still young and the github code (http://github.com/
> > JakSprats/Redisql) is being heavily developed, w/ all sorts of hybrid
> > NOSQL/SQL commands in the pipeline.
>
> > I will be writing some blog entries soon on Redisql as many of the
> > commands and concepts are best explained w/ examples.
>
> > - Jak
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Redis DB" group.
> > To post to this group, send email to redi...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > redis-db+u...@googlegroups.com<redis-db%2Bunsu...@googlegroups.com>
> > .

[Jak Sprats]

Hi Paul,

>> looks like that part (SQL parsing) is a bit finicky with syntax

There are some caveats to the Redisql syntax (http://code.google.com/p/
redisql/wiki/Protocol)

If you are using command line arguments, mimic the commands in this
file: http://github.com/JakSprats/Redisql/blob/master/bash_functions.sh
(which contains ALL Redisql commands)

Caveats: (remember ')','(','*', etc... need to be escaped command line
1.) INSERT's need the value list [INSERT INTO tbl VALUES
(value_list)] in a single commands w/ respect to the redis line
protocol, which makes sense as redis is space delimited and doing it
otherwise would generate a lot of overhead
Example: ./redisql-cli INSERT INTO employee VALUES
"(1,11,10000.99,jim)"
NOT ./redisql-cli INSERT INTO employee VALUES (1, 11, 10000.99,
jim)
2.) UPDATE's update_list also needs to be a single argument.
Example: ./redisql-cli UPDATE employee SET "salary=50000,
name=NEWNAME, division=66"
NOT: ./redisql-cli UPDATE employee SET salary= 50000, name =
NEWNAME, division = 66
This single argument philosophy was not used in CREATE TABLE or in any
WHERE-CLAUSE (which looks to be a bad choice .. meaning it will
eventually change - fortunately clients can change this w/o changing
their APIs)
e.g. ./redisql-cli CREATE TABLE employee \(id int primary key,
division int, salary TEXT, name TEXT\)
e.g. ./redisql-cli DELETE FROM customer WHERE id = 7

I hope that sheds some light ... running SQL from unix command line is
awkward in nature because control characters overlap. Any suggestions
on how to clean this up would be appreciated.

Any crashes or bugs, please report (Redisql has a mailing list) I will
fix em ASAP.

>> How do you see Redisql fitting in with Redis Cluster?
Redisql will fit w/ redis cluster just like they fit together now, but
Redisql will need additional client and server side logic to be able
to handle range-queries and joins that span multiple nodes, which is a
VERY difficult problem :)

- Jak

On Oct 20, 8:06 am, Paul Smith <paulsm...@gmail.com> wrote:

[Jak Sprats]

Hi Ezra and Xiangrong,

I think Redisql's clients (redis-rb, Rediska, predis, node_redis,
etc...) should become patches or their respective redis clients (e.g.
Monkey patch for Ruby, and node_redis will be an even smaller patch
when I get that finished w/ mranney).

As for merging the servers, I will do my best to keep Redisql a patch
of the most recent and stable redis. When redis 2.2 is VERY stable, I
will patch redis 2.2 to become the next Redisql. I will respond more
to this issue in my response to antirez.

- Jak

On Oct 20, 10:17 am, Ezra Zygmuntowicz <ezmob...@gmail.com> wrote:
> I tend to agree, this would be cool as a redis plugin of some sort if there was a way to easily turn it on or off with a compilation flag or a configuration file option.
>
> I played with it last night and thoug it was pretty cool. I would add Jak's additions to redis-rb as a plugin or option in the official redis/rb if there was a way to conditionally enable redisql on the server side.
>
> Ezra
>
> Sent from my iPhone
>
> On Oct 20, 2010, at 8:21 AM, Xiangrong Fang <xrf...@gmail.com> wrote:
>
> > Rather than fight with redisql, I would suggest Jak to collaborate
> > with antirez and the team to make redis better, for example:
>
> > - make redisql idea an "plugin" or "adapter" for redis
> > - make server side scripting of redis (e.g. define "macro" commands
> > which consists of a bunch of multi/exec wrapped commands)
> > - other enhancement in the area of improve redis persistence (aof etc)
> > and redis-cluster
> > - blah blah....
>
> > Shannon
>

> > 2010/10/20 Paul Smith <paulsm...@gmail.com>:

[Matt Ranney]

On Wed, Oct 20, 2010 at 12:37 PM, Jak Sprats <jaks...@gmail.com> wrote:

I think Redisql's clients (redis-rb, Rediska, predis, node_redis,
etc...) should become patches or their respective redis clients (e.g.
Monkey patch for Ruby, and node_redis will be an even smaller patch
when I get that finished w/ mranney).

My hope is that we can set up node_redis so that redisql plugs right in without any changes to node_redis.  

[Jak Sprats]

Hi Salvatore,

thanks for the kind words.

I am of like mind in terms of merging Redisql into redis. Redisql is
too young, and I think it will actually benefit both projects to be
seperate. Redisql will move in the direction of being a patch of redis
(just because Redisql NEEDs to upgrade w/ redis and the patching is in
and of itself very tedious boring work :) There are also some features
or Redisql: http://bit.ly/diK5sc that are very cool, but are
fundamentally different than the redis direction. So it makes sense
for both sides to not merge :)

> Another one is, and it's months that I think at it, that in-memory
btrees make sense. We should try to get good use in Redis of some code
in Redisql, starting from the btree code.

I did just this first. I came up w/ a redis like syntax for SQL
commands (e.g. "select" for simple selects, "iselect" for range-
queries/joins, etc....), meaning no "FROM", "WHERE", etc... and there
were 2 immediate problems
1.) everyone I showed it too said exactly this: "why do i have to
learn something new?" and my response was, because this is what SQL
actually does under the covers, but people were not interested in
learning my new "boiled down" SQL
2.) testing in SQL is quick and my mind quickly came up w/ about 400
different types of queries (because I have been using SQL for 10+
years) and these queries were not supported in my original syntax. SQL
actually does many many different types of operations (some are bad,
that is why I made the command "scanselect" for full table scans, and
some I dont support because they are a parsing nightmare or they are
bad in terms of performance (cascading foreigh key constraints ...
phooey)

So you are a natural at coming up w/ syntaxes for commands, but in the
SQL use-cases, and this goes against my initial assumption, people are
happier w/ SQL cause they dont need to learn anything. And most likely
you can find the best SQL use cases (i.e. range queries) but then
people will want a full-table-scan and then a full-table-scan with a
single non-indexed condition ... and after a while I kind of got why
SQL (as ugly as it is) is what it is.

>> Btw I was not able to compile it in my mac os x a few hours ago

The google code tarball was out of date, github compiles on OSX:
http://github.com/JakSprats/Redisql
and so does the google 0.0.12 tarball http://bit.ly/dg8Bh0
Cross OS compilation sucks, but better than the days before virtual
machines :)

- Jak
On Oct 20, 10:31 am, Salvatore Sanfilippo <anti...@gmail.com> wrote:

> >> 2010/10/20 Paul Smith <paulsm...@gmail.com>:

> >> For more options, visit this group athttp://groups.google.com/group/redis-db?hl=en.

>
> > --
> > You received this message because you are subscribed to the Google Groups "Redis DB" group.
> > To post to this group, send email to redi...@googlegroups.com.
> > To unsubscribe from this group, send email to redis-db+u...@googlegroups.com.
> > For more options, visit this group athttp://groups.google.com/group/redis-db?hl=en.
>
> --

> Salvatore 'antirez' Sanfilippohttp://invece.org

[Jak Sprats]

Hi Matt,

yeah node_redis and Redisql integration are going to define how
Redisql clients should be placed "on top" of redis clients.

I am predicting <50 lines of code for the plugin :)

- Jak

On Oct 20, 12:44 pm, Matt Ranney <m...@ranney.com> wrote:

[Jak Sprats]

One last note on Redisql, this is a blog post I wrote explaining why I
would do something as crazy as put a RDBMS on top of redis:
http://jaksprats.wordpress.com/2010/09/28/introducing-redisql-the-lightning-fast-polyglot/