new github code
11 views
Skip to first unread message
Jak Sprats
Jul 13, 2011, 7:57:25 PM7/13/11
to redisql-dev
Hi All,
just a heads up, after a long time, I was granted permission to open
source Alchemy Database's updates ... this is about 6 months worth of
work, so it is a very significant upgrade.
I am in the process of updating the google-code site, and adding a
bunch of functionality, so the github code is not yet stable.
When it is stable, I will post on the new features, there are a lot of
them, and the code (the stable parts) has been production tested, so
it is mature).
New github repo: https://github.com/JakSprats/Alchemy-Database
lots more to come
- Jak
just a heads up, after a long time, I was granted permission to open
source Alchemy Database's updates ... this is about 6 months worth of
work, so it is a very significant upgrade.
I am in the process of updating the google-code site, and adding a
bunch of functionality, so the github code is not yet stable.
When it is stable, I will post on the new features, there are a lot of
them, and the code (the stable parts) has been production tested, so
it is mature).
New github repo: https://github.com/JakSprats/Alchemy-Database
lots more to come
- Jak
David Yu
Jul 14, 2011, 1:36:32 AM7/14/11
to redis...@googlegroups.com
On Thu, Jul 14, 2011 at 7:57 AM, Jak Sprats <jaks...@gmail.com> wrote:
Hi All,
just a heads up, after a long time, I was granted permission to open
source Alchemy Database's updates ... this is about 6 months worth of
work, so it is a very significant upgrade.
I am in the process of updating the google-code site, and adding a
bunch of functionality, so the github code is not yet stable.
When it is stable, I will post on the new features, there are a lot of
them,
Cool. I sure am hoping better aof is one of those features. Is alchemy based on redis 2.4?
and the code (the stable parts) has been production tested, so
it is mature).
New github repo: https://github.com/JakSprats/Alchemy-Database
lots more to come
- Jak
--
When the cat is away, the mouse is alone.
- David Yu
David Yu
Jul 14, 2011, 4:42:40 AM7/14/11
to redis...@googlegroups.com
On Thu, Jul 14, 2011 at 1:36 PM, David Yu <david....@gmail.com> wrote:
On Thu, Jul 14, 2011 at 7:57 AM, Jak Sprats <jaks...@gmail.com> wrote:Hi All,
just a heads up, after a long time, I was granted permission to open
source Alchemy Database's updates ... this is about 6 months worth of
work, so it is a very significant upgrade.
I am in the process of updating the google-code site, and adding a
bunch of functionality, so the github code is not yet stable.
When it is stable, I will post on the new features, there are a lot of
them,Cool. I sure am hoping better aof is one of those features. Is alchemy based on redis 2.4?
From GPL to AGPL, I'm not getting the reason behind the change :/
and the code (the stable parts) has been production tested, so
it is mature).
New github repo: https://github.com/JakSprats/Alchemy-Database
lots more to come
- Jak
--
When the cat is away, the mouse is alone.
- David Yu
Jak Sprats
Jul 14, 2011, 9:43:26 AM7/14/11
to redisql-dev
Hi David,
Alchemy is based on redis unstable from about 2 weeks ago, so post
2.4. I refactored the code heavily and can now update the redis base
layer in about 4 hours, so the next time I do an Alchemy release I
will find the most recent stable redis release and use that as the
base. The base needs both scripting and cluster code, so it will stay
close to redis/unstable until redis 3.0. And on that note, the github
code is not yet stable (there are known bugs), I need until the end of
the month for that.
I did not tackle AOF improvements, I was hoping the redis team would
solve this and I could piggyback on them, but that did not happen. The
methodology to rewrite the AOF file in an external process using
deltas and what not has been discussed for about a year, but there has
not been significant progress on it, by me or the redis team. I almost
did the AOF improvements about 10 months ago, but figured it would
then just be duplicated by the redis team, so I didn't :) It is on my
TODO also, but not very high.
The license change from GP L to AGPL is complicated. Basically I have
been doing this project full-time for about 16 months, and after 9
months it became evident I needed sponsorship for it, otherwise I just
had no time, so I got sponsorship, in the form of working for
companies that used Alchemy, and that put bread on the table and
furthered the development of the product, but I also spent way too
much time on the customer's products and not on developing Alchemy,
but the strategy served its purpose of getting me to where I am now.
So I am following Neo4j's business model, they went AGPL and said: "if
you're open source, we're open source", and they also sold commercial
licenses, and eventually they came out w/ a GPL'ed community edition.
So that is exactly my plan, and purists are going to argue that I
should BSD the project, but if I do that, it will not get sponsored/
funded, and it will most likely not make it, so what purpose would
that serve. I have gotten interest in this project from very large
companies, many willing to pay for licenses, so the AGPL is necessary
to be able to charge them money. For people not working for HUGE
companies, I have no plans on charging them, because it doesn't make
any sense on any level ... so pretty much, shoot me an email, and
write: "I dont work for google or facebook" and I will give you a GPL
license and a lot of support happily :)
If someone has a better idea on how I can license Alchemy and make
enough money to continue developing it, I am all ears, please voice
your opinions. My real interests here are not to get rich quick w/
Alchemy, rather to build it up to something very large over time, but
w/o some form of revenue, it wont happen.
Here is a short list of new features (off the top of my head)
1.) improved query optimiser - non-complex joins now supported (this
is a huge win, any sane OLTP join is covered)
2.) REPLACE command
3.) INSERT bulk (i.e. many rows in a single command)
4.) INSERT ON DUPLICATE REPLACE
5.) ALTER TABLE ADD COLUMN
6.) partial row storage (since Alchemy uses streams for rows, sparse
data in SQL rows is possible -> NOSQL-esque)
7.) Multiple Column Indexes (compound indexes)
8.) Unique Indexes
9.) LONG datatype
10.) LRU Index - together w/ #12 LUATRIGGERs, its pretty easy to
create a LRU cache for SQL tables
11.) improved LUA, now calls "luafunc arg1 arg2 ...." - amazingly fast
and easier to use than redis' EVALSHA1
12.) LUATRIGGERs - call LUA routines automatically on the row you just
inserted - also very fast
13.) pure_redis output mode -> SQL rows are output in nested redis
protocol
14.) EXPLAIN query command -> for join commands and complicated range
queries, show the join plan
15.) webserver mode (this is insane), use Alchemy as a HTTP webserver
which calls whitelisted LUA functions to serve dynamic webpages (I
will blog on this, can we say dynamic webpages w/ database lookups @
65K HTTP REQ/s on a single 3.0 GHz core :)
16.) redis cluster integration for SQL tables (a work in progress, a
glimpse of the very large direction this is going in)
There are more, and I am in the process of updating the google code
page. I also took out a lot of code (morph commands) that I thought
was better server by doing it in LUA (and was not very popular/
useful). So I am in the process of cleaning up, fixing bugs, updating
documentation, extending the testing suite, etc...., release ready in
2-6 weeks (depending on if I am busy w/ other stuff).
- Jak
On Jul 14, 2:42 am, David Yu <david.yu....@gmail.com> wrote:
> On Thu, Jul 14, 2011 at 1:36 PM, David Yu <david.yu....@gmail.com> wrote:
Alchemy is based on redis unstable from about 2 weeks ago, so post
2.4. I refactored the code heavily and can now update the redis base
layer in about 4 hours, so the next time I do an Alchemy release I
will find the most recent stable redis release and use that as the
base. The base needs both scripting and cluster code, so it will stay
close to redis/unstable until redis 3.0. And on that note, the github
code is not yet stable (there are known bugs), I need until the end of
the month for that.
I did not tackle AOF improvements, I was hoping the redis team would
solve this and I could piggyback on them, but that did not happen. The
methodology to rewrite the AOF file in an external process using
deltas and what not has been discussed for about a year, but there has
not been significant progress on it, by me or the redis team. I almost
did the AOF improvements about 10 months ago, but figured it would
then just be duplicated by the redis team, so I didn't :) It is on my
TODO also, but not very high.
The license change from GP L to AGPL is complicated. Basically I have
been doing this project full-time for about 16 months, and after 9
months it became evident I needed sponsorship for it, otherwise I just
had no time, so I got sponsorship, in the form of working for
companies that used Alchemy, and that put bread on the table and
furthered the development of the product, but I also spent way too
much time on the customer's products and not on developing Alchemy,
but the strategy served its purpose of getting me to where I am now.
So I am following Neo4j's business model, they went AGPL and said: "if
you're open source, we're open source", and they also sold commercial
licenses, and eventually they came out w/ a GPL'ed community edition.
So that is exactly my plan, and purists are going to argue that I
should BSD the project, but if I do that, it will not get sponsored/
funded, and it will most likely not make it, so what purpose would
that serve. I have gotten interest in this project from very large
companies, many willing to pay for licenses, so the AGPL is necessary
to be able to charge them money. For people not working for HUGE
companies, I have no plans on charging them, because it doesn't make
any sense on any level ... so pretty much, shoot me an email, and
write: "I dont work for google or facebook" and I will give you a GPL
license and a lot of support happily :)
If someone has a better idea on how I can license Alchemy and make
enough money to continue developing it, I am all ears, please voice
your opinions. My real interests here are not to get rich quick w/
Alchemy, rather to build it up to something very large over time, but
w/o some form of revenue, it wont happen.
Here is a short list of new features (off the top of my head)
1.) improved query optimiser - non-complex joins now supported (this
is a huge win, any sane OLTP join is covered)
2.) REPLACE command
3.) INSERT bulk (i.e. many rows in a single command)
4.) INSERT ON DUPLICATE REPLACE
5.) ALTER TABLE ADD COLUMN
6.) partial row storage (since Alchemy uses streams for rows, sparse
data in SQL rows is possible -> NOSQL-esque)
7.) Multiple Column Indexes (compound indexes)
8.) Unique Indexes
9.) LONG datatype
10.) LRU Index - together w/ #12 LUATRIGGERs, its pretty easy to
create a LRU cache for SQL tables
11.) improved LUA, now calls "luafunc arg1 arg2 ...." - amazingly fast
and easier to use than redis' EVALSHA1
12.) LUATRIGGERs - call LUA routines automatically on the row you just
inserted - also very fast
13.) pure_redis output mode -> SQL rows are output in nested redis
protocol
14.) EXPLAIN query command -> for join commands and complicated range
queries, show the join plan
15.) webserver mode (this is insane), use Alchemy as a HTTP webserver
which calls whitelisted LUA functions to serve dynamic webpages (I
will blog on this, can we say dynamic webpages w/ database lookups @
65K HTTP REQ/s on a single 3.0 GHz core :)
16.) redis cluster integration for SQL tables (a work in progress, a
glimpse of the very large direction this is going in)
There are more, and I am in the process of updating the google code
page. I also took out a lot of code (morph commands) that I thought
was better server by doing it in LUA (and was not very popular/
useful). So I am in the process of cleaning up, fixing bugs, updating
documentation, extending the testing suite, etc...., release ready in
2-6 weeks (depending on if I am busy w/ other stuff).
- Jak
On Jul 14, 2:42 am, David Yu <david.yu....@gmail.com> wrote:
> On Thu, Jul 14, 2011 at 1:36 PM, David Yu <david.yu....@gmail.com> wrote:
David Yu
Jul 14, 2011, 10:50:33 AM7/14/11
to redis...@googlegroups.com
Hi Jak,
Thanks for the detailed response.
I do agree about your views about the license.
Feature #15 would certainly be interesting.
Making it talk to nginx(uwsgi) or mongrel2(zeromq) would definitely do it.
Cheers
Jak Sprats
Jul 14, 2011, 11:17:32 AM7/14/11
to redisql-dev
Hi David,
on #15, I actually built a HTTP server into Alchemy, it was easy, HTTP
is not that different from Redis' protocol.
I need a couple of weeks to clean it up, but it cuts out the middle-
man, which has lots of cons, but for certain use cases, it is near-
optimal
- Jak
on #15, I actually built a HTTP server into Alchemy, it was easy, HTTP
is not that different from Redis' protocol.
I need a couple of weeks to clean it up, but it cuts out the middle-
man, which has lots of cons, but for certain use cases, it is near-
optimal
- Jak
David Yu
Jul 14, 2011, 11:36:16 AM7/14/11
to redis...@googlegroups.com
On Thu, Jul 14, 2011 at 11:17 PM, Jak Sprats <jaks...@gmail.com> wrote:
Hi David,
on #15, I actually built a HTTP server into Alchemy, it was easy, HTTP
is not that different from Redis' protocol.
I need a couple of weeks to clean it up, but it cuts out the middle-
man, which has lots of cons, but for certain use cases, it is near-
optimal
I see your point. The frontends could still be useful for load-balancing and sticky sessions.
Btw, does alchemy have a concept of rollback? Say if you want to atomically update 2 tables.
Jak Sprats
Jul 14, 2011, 12:18:17 PM7/14/11
to redisql-dev
Hi David,
very good point on using the frontends for LBs & sticky-sessions ... I
recommend that only for advanced users, because you have to make sure
ALL the data for a web-request is on that node .. there is no
backend :) but it is doable, and then the performance is insanely fast
on a single machine basis.
Alchemy does not have rollbacks ... I have thought about implementing
them, but it requires a lot of engineering (e.g. MVCC) and is only
useful for a limited number of use-cases (e.g. database transactions
that can fail due to constraint violations - which Alchemy also does
not have) ... so I decided to not add the bloat, keep Alchemy as lean
and quick as possible for the bulk of OLTP use-cases. It is a trade-
off I am currently OK w/, but I have almost implemented rollbacks
several times ... so its a fence-sitter.
- Jak
On Jul 14, 9:36 am, David Yu <david.yu....@gmail.com> wrote:
very good point on using the frontends for LBs & sticky-sessions ... I
recommend that only for advanced users, because you have to make sure
ALL the data for a web-request is on that node .. there is no
backend :) but it is doable, and then the performance is insanely fast
on a single machine basis.
Alchemy does not have rollbacks ... I have thought about implementing
them, but it requires a lot of engineering (e.g. MVCC) and is only
useful for a limited number of use-cases (e.g. database transactions
that can fail due to constraint violations - which Alchemy also does
not have) ... so I decided to not add the bloat, keep Alchemy as lean
and quick as possible for the bulk of OLTP use-cases. It is a trade-
off I am currently OK w/, but I have almost implemented rollbacks
several times ... so its a fence-sitter.
- Jak
On Jul 14, 9:36 am, David Yu <david.yu....@gmail.com> wrote:
David Yu
Jul 14, 2011, 12:29:33 PM7/14/11
to redis...@googlegroups.com
On Fri, Jul 15, 2011 at 12:18 AM, Jak Sprats <jaks...@gmail.com> wrote:
Hi David,
very good point on using the frontends for LBs & sticky-sessions ... I
recommend that only for advanced users, because you have to make sure
ALL the data for a web-request is on that node .. there is no
backend :) but it is doable, and then the performance is insanely fast
on a single machine basis.
Alchemy does not have rollbacks ... I have thought about implementing
them, but it requires a lot of engineering (e.g. MVCC) and is only
useful for a limited number of use-cases (e.g. database transactions
that can fail due to constraint violations - which Alchemy also does
not have) ... so I decided to not add the bloat, keep Alchemy as lean
and quick as possible for the bulk of OLTP use-cases. It is a trade-
off I am currently OK w/, but I have almost implemented rollbacks
several times ... so its a fence-sitter.
I'm not sure if voltdb is using mvcc (no concurrent access to db like redis) but they sure are using rollback buffers.
Without rollbacks, I still think its doable to do 2 iterations (1st one verifies, 2nd one writes) if the cost of doing each iteration is O(1) (or near it). I'm not so sure how it'll perform with b+trees. An atomic operation on 2 tables would need 4 pk lookups.
Jak Sprats
Jul 14, 2011, 12:54:04 PM7/14/11
to redisql-dev
Hi David,
yeah VoltDB does rollback buffers. VoltDB defines everything in units
of stored procedures, so a stored procedure runs atomically, and if
operation 5 of the stored procedure fails, then operation 1-4 are
rolled back via the rollback buffers, it is a pretty slick/lean way to
support rollbacks.
Alchemy does not have any constraints that can fail, so operation 5
can not fail. My approach to rollbacks was to disallow any
functionality that could result in constraint-failure (e.g. unique
multiple column indexes can fail if someone tries to update an indexed
column, as the update could result in a unique constraint violation,
so I prohibit this exact type of update), this limits functionality
(in this case for an almost useless use-case) but gains a no-rollbacks-
necessary DB. For me, the code complexity that supporting rollback
buffers entails is not worth it.
To date (AFAIK), no use case that should be served by Alchemy Database
has required rollbacks. I also like the idea of a DB w/o rollbacks,
IMHO a lot of rollback logic is best dealt w/ at the application
level, and for the use-cases that absolutely need rollbacks in the DB
(usually deal w/ transferring money from accountA to accountB), use
another DB for now, or possibly have an atomic lua function that does
a first pass to check and a second pass to commit (like you said).
- Jak
On Jul 14, 10:29 am, David Yu <david.yu....@gmail.com> wrote:
yeah VoltDB does rollback buffers. VoltDB defines everything in units
of stored procedures, so a stored procedure runs atomically, and if
operation 5 of the stored procedure fails, then operation 1-4 are
rolled back via the rollback buffers, it is a pretty slick/lean way to
support rollbacks.
Alchemy does not have any constraints that can fail, so operation 5
can not fail. My approach to rollbacks was to disallow any
functionality that could result in constraint-failure (e.g. unique
multiple column indexes can fail if someone tries to update an indexed
column, as the update could result in a unique constraint violation,
so I prohibit this exact type of update), this limits functionality
(in this case for an almost useless use-case) but gains a no-rollbacks-
necessary DB. For me, the code complexity that supporting rollback
buffers entails is not worth it.
To date (AFAIK), no use case that should be served by Alchemy Database
has required rollbacks. I also like the idea of a DB w/o rollbacks,
IMHO a lot of rollback logic is best dealt w/ at the application
level, and for the use-cases that absolutely need rollbacks in the DB
(usually deal w/ transferring money from accountA to accountB), use
another DB for now, or possibly have an atomic lua function that does
a first pass to check and a second pass to commit (like you said).
- Jak
On Jul 14, 10:29 am, David Yu <david.yu....@gmail.com> wrote:
Reply all
Reply to author
Forward
0 new messages