more functionality, stable release looming

[Jak Sprats]

Hi All,

the following functionalities have been added to AlchemyDB as of late:

# Unlimited number of tables, indexes, & columns (allows for NOSQL
schema-less like data, combining w/ lua provides a very fast single-
node in-memory map-reduce building block)
#Sparse-Rows: tables w/ 1000s of columns, that are sparsely populated,
use a serialised hash table in the row's stream to store column
offsets. Sparse-Rows can be Orders-Of-Magnitude smaller than full rows
(NOTE: useful for unstructured data: just keep adding columns, and
Sparse-Rows limit memory usage)

The combination of these 2 functionalities is subtle:
1.) ALTER TABLE ADD COLUMN is instantaneous and adds only 16 bytes
2.) if a table has 50K columns, the metadata will be ~800KB, and then
the actual data in the row will determine the memory used by the table
3.) a table w/ 5K columns and a table w/ 50K columns, will both use up
2KB to store a row w/ 200 UINTs

This opens up the traditional NOSQL territory of unstructured data to
be in a relational model, as adding additional columns to a table has
very little overhead, filling those columns will (of course) use up
memory :)

An example of this would be if you had a users table, and you wanted
to keep adding attributes to the table as if it were a hash table. You
just need to keep adding columns ... and the bonus is, you can do
range-queries and joins on these attributes later.

# sqlappendonly writes ALL SQL CRUD to a file in MYSQL format

SQL commands in AlchemyDB will be reformatted to the MYSQL format and
written to the append-only file. If you put a pipe on this file that
goes to a mysql client, that pipes the commands to a mysql server, you
can have your data backed up in Mysql. Backing up data in Mysql has
the benefit of adding disk-persistence to this architecture.

A really cool to way Alchemy is as a relational cache. Using the
twitter example, you can store the last 100 tweets for each user in
Alchemy and ALL of the user's tweets, by
1.) setting up one of these sqlappendonly-file->mysql-server pipes and
2.) by putting a LUATRIGGER on the tweets tables, w/ a lua function
that deletes tweets 100+ (per user). Then your app needs the logic to
send get_tweets(<100) to AlchemyDB & get_tweets(100+) to Mysql.

Just an update, I need to test all of this more thoroughly and add a
few more things and then a stable release next month

- jak