NOSQL Resource Page

[edlich]

Dear NOSQL Community,

I am gathering all material about the nosql topic since a few month
for
a university lecture Databases Master. That's why I thought it could
be
useful for the community to give something back (what the community
gives
me with awesome tools ;-). So I copied it online:

http://nosql-databases.org

It is still in beta and there is soo much to do but I think it really
could
be a useful resource. Check out all the links under the point
"LINKS, ARTICLES, BLOGS [tons of]" because it includes nearly ALL
important links.
I also publish (the few available) nosql critical info to be fair and
complete.

If you have additional material, feedback, product descriptions (it
they
really do fit), improvements then please contact me.

Regards
Stefan Edlich
[Beuth University of Technology Berlin]
http://edlich.de

[rtweed]

Could I ask you to please add GT.M to your list? For a quick intro
see http://bit.ly/2fS2jy (this provides onward links for more info)

[Steve Harris]

Have you thought about adding RDF stores to the list?

Some are built ontop of SQL stores, but many share design elements
with the common key-value systems.

My company's is called 4store (http://4store.org/ GPLv3), but there's
also Jena (http://jena.sourceforge.net/ Apache licence), Sesame (http://www.openrdf.org/
GPL), Virtuoso (http://www.openlinksw.com/virtuoso/ GPL and
proprietary) and several others. They all speak a common data
language, query language and protocol.

- Steve

[Andy Seaborne]

Jena, http://openjena.org, has a BSD license (as does Sesame for the
majority of it's components)

Andy

On Nov 10, 10:52 am, Steve Harris <s.w.har...@gmail.com> wrote:
> Have you thought about adding RDF stores to the list?
>
> Some are built ontop of SQL stores, but many share design elements  
> with the common key-value systems.
>

> My company's is called 4store (http://4store.org/GPLv3), but there's  
> also Jena (http://jena.sourceforge.net/Apache licence), Sesame (http://www.openrdf.org/
>   GPL), Virtuoso (http://www.openlinksw.com/virtuoso/GPL and  

> proprietary) and several others. They all speak a common data  
> language, query language and protocol.
>
> - Steve
>
> On 10 Nov 2009, at 10:20, rtweed wrote:
>
>
>
> > Could I ask you to please add GT.M to your list?  For a quick intro

> > seehttp://bit.ly/2fS2jy(this provides onward links for more info)

[Steve Harris]

Oops, I misread http://jena.sourceforge.net/license.html, sorry Andy.

- Steve

[Gregory Burd]

Stefan,

Thanks for doing the work to compile this information. What about the
three Berkeley DB products (the original Berkeley DB, Berkeley DB Java
Edition, and Berkeley DB XML)? BDB is certainly one of the earliest
NoSQL key/value databases available. It also serves as the foundation
for many of the projects you list (memcachedb, Voldemort and more).
I'm curious, does it fit this new category? Why/why not? If so could
you please add them to your growing list?

Disclaimer: I worked for Sleepycat for years before it was acquired,
now I'm at Oracle as the Product Manager for Berkeley DB. Biased?
You bet. Proud? Damn right! Excited to see the key/value NoSQL
space develop into a market force? Heck yeah! ;-)

cheers, and thanks,

-greg

[Yang Zhang]

List members may also consider contributing to the Wikipedia page and
related pages.

http://en.wikipedia.org/wiki/NoSQL

--
Yang Zhang
http://www.mit.edu/~y_z/

[edlich]

Thanks for all the positive response.

I will integrate every comment. Please give me a while.

Meanwhile: If you have information about a tool I have not described
in details
(concerning the categories I have as API, Procotol, Query Method,
Scaling, Written in, Concurrency)
then please mail me so.

I would also be happy to register new links.

Thanks in advance
Stefan Edlich

FYI: Frankfurt might see a NoSQL Workshop in Summer 2010.

[Alex Cruise]

On 09-11-10 02:20 AM, rtweed wrote:
> Could I ask you to please add GT.M to your list? For a quick intro
> see http://bit.ly/2fS2jy (this provides onward links for more info)
>

Well, if you're going to bring MUMPS into it, why not Pick and its/his
multiple, multi-valued descendents? :)

Uni{Verse|Data}, JBase, Revelation, etc., etc...

@Record<-1> = "--"
@Record<-1> = "-0xe1a"
@Record<-1> = "Scarred for life."
@Record<-1> = ":)"

[Thomas Vial]

For the sake of completeness I would also mention IBM's IMS (http://en.wikipedia.org/wiki/Information_Management_System). Never worked with it but the "hierarchical database", "Full function database", "Fast path" and "High availability large databases" keywords on the Wikipedia page are a definitive eyecatcher.

Thomas

[K.S. Bhaskar]

I don't know much about Pick (I can spell it, but not a whole lot
more), but if it is a non-SQL database and if active implementations
exist for contemporary computer platforms, sure.

Cheers
-- Bhaskar

On Nov 13, 2:05 pm, Alex Cruise <a...@cluonflux.com> wrote:
> On 09-11-10 02:20 AM, rtweed wrote:> Could I ask you to please add GT.M to your list?  For a quick intro

> > seehttp://bit.ly/2fS2jy(this provides onward links for more info)

[Johannes Ernst]

Personally, I think now we are getting a bit too far.

What about morse-coded telegraph tapes? Comma-separated values? Not SQL either ...

I think the "NOSQL message" is "there are modern alternatives to standard SQL databases you might not have known of with specific advantages in specific circumstances". If the kitchen sink is included, any list of non-SQL-based technologies ceases to be useful.

Just my $0.02.

Cheers,

Johannes.

[peco]

A couple of attributes that are unique to the technologies discussed
in this group are being high-performance, distributed, and scalable
because of the volume of records that have to be processed. Back to
the original discussion of why NOSQL may be a confusing name for this
but ...

cheers

On Nov 13, 1:28 pm, Johannes Ernst <jernst+google....@netmesh.us>
wrote:

> Personally, I think now we are getting a bit too far.
>
> What about morse-coded telegraph tapes? Comma-separated values? Not SQL either ...
>

> I think the "NOSQLmessage" is "there are modern alternatives to standard SQL databases you might not have known of with specific advantages in specific circumstances". If the kitchen sink is included, any list of non-SQL-based technologies ceases to be useful.

[Jason Dusek]

2009/11/13 Johannes Ernst <jernst+g...@netmesh.us>:

> Personally, I think now we are getting a bit too far.
>
> What about morse-coded telegraph tapes? Comma-separated
> values? Not SQL either ...

The systems in question -- GT.M and IMS -- were both built
with goals that are very similar to the goals of modern NoSQL
databases: high availability and throughput, multi-site
operation, data retention and performant retrieval. It's true
that IMS has some features -- it's plethora of database types
and its multi-tenancy -- which aren't so relevant but it's
hardly as off-topic as Morse code would be.

>
> I think the "NOSQL message" is "there are modern alternatives
> to standard SQL databases you might not have known of with
> specific advantages in specific circumstances". If the kitchen
> sink is included, any list of non-SQL-based technologies
> ceases to be useful.

Maybe there should be a separate section or page that mentions
some of these systems that are definitely legacy (though GT.M
seems not to be).

Historical foundations are not, I think, adequately appreciated
by computer programmers. With regards to "graph databases", for
example, what's old is new again. Don't we want to learn from
the successes (and mistakes) of the past? How can we do that if
we fail to mention prior work?

--
Jason Dusek

[rtweed]

On 13 Nov, 19:50, Jason Dusek <jason.du...@gmail.com> wrote:

>   Maybe there should be a separate section or page that mentions
>   some of these systems that are definitely legacy (though GT.M
>   seems not to be).

One aspect that will almost certainly determine whether or not people
will check out these older NoSQL technologies and/or take them
seriously is their licensing. I would suggest that if it isn't Open
Source, it will probably be ignored.

>
>   Historical foundations are not, I think, adequately appreciated
>   by computer programmers. With regards to "graph databases", for
>   example, what's old is new again. Don't we want to learn from
>   the successes (and mistakes) of the past? How can we do that if
>   we fail to mention prior work?
>

Indeed so, well said! The need for speed is far from new, and old can
mean tried and tested, not old-fashioned.

[K.S. Bhaskar]

Just setting down some facts about GT.M (I manage it)...

1. GT.M (http://fis-gtm.com) is a key value (schema-free, multi-value)
data base.

2. The license for GT.M on GNU/Linux on x86 is FOSS - the license is
AGPL v3 (http://www.gnu.org/licenses/agpl.html). All user
documentation is on the Internet. It is distributed from Source Forge
(http://sourceforge.net/projects/fis-gtm). If you need or want it,
support is available on a commercial basis with assured service
levels.

3. You can access the database from C, Python and other languages,
including its native MUMPS. M/DB is built on GT.M and provides an API
compatible with Amazon SimpleDB that you can run outside the Amazon
cloud; or in it for that matter. EWD makes it easy to create Ajax/YUI
rich Internet applications on GT.M. [M/DB and EWD are also FOSS, also
AGPL v3 - you'll find them at http://mgateway.com ]

4. GT.M is both old (first went into production in 1986) and new (EWD
and M/DB are recent developments; in the core engine, database
encryption is an example of a recent enhancement). GT.M has been
continuously developed for the last 25 years. I think it is important
to distinguish between old, abandoned software (so called legacy
technology) and proven technology that has been around and which is
continuously evolving and improving (perhaps we should come up with
label such as "heritage").

5. Your bank balance may already be in the care of a GT.M database (it
is the legal system of record for tens of millions of bank accounts
around the world). Your health records may be on it soon. Databases
in the hundreds of GB are in routine production use at many sites. It
runs the largest single real time core processing system in the world
that I am aware of. The largest single database file I am aware of
was 2TB file (a logical database consists of a virtually unlimited
number of database files).

6. It is extremely fast. For online financial transactions, GT.M has
been benchmarked (depending on the transaction) at thousands of
transactions per second on an x86 GNU/Linux system with every
transaction having full ACID properties.

7. For all its performance and ability to handle massive databases,
the on-disk footprint of the engine itself is very small - 16MB for
the latest 32-bit version on x86 GNU/Linux). So, it not only scales
up to large applications, it scales down to virtual machines and cloud-
based applications.

I'd be glad to answer any questions.

Regards
-- Bhaskar

On Nov 13, 7:31 pm, rtweed <rob.tw...@gmail.com> wrote:
> On 13 Nov, 19:50, Jason Dusek <jason.du...@gmail.com> wrote:
>

> >   Maybe there should be a separate section or page that mentionstechnology

[Asaf Ary]

> 6. It is extremely fast.  For online financial transactions, GT.M has
> been benchmarked (depending on the transaction) at thousands of
> transactions per second on an x86 GNU/Linux system with every
> transaction having full ACID properties.

I'm guessing that you're using two-phase commit for ACID properties in
a distributed system.
Would you say that GT.M is a transaction optimized data store?
I mean if you are maintaining ACID properties than you must be
swapping Consistency for Availability.

I'll try to build the NoSQL profile of the GT.M user according to what
you said (correct me if I'm wrong):

1. I read and write all my data in a simple Key / Value form
2. I don't want schema-constraints for my values
3. My data needs to be strongly consistent (not eventually
consistent), clients should have a "read-your-writes" behavior.
4. I use the database for complex offline operations and not real time
online operations (such as logging or serving online requests)

On (4) what I mean is that since you maintain ACID it is probably
undesirable to use GT.M for cases like shopping-carts where failures
are best handled after the user checked-out rather than failing to add
stuff to his cart.

Please correct me if I'm wrong.
-- Asaf

[rtweed]

BTW Asaf, probably the easiest way to try out and play with GT.M is to
use the M/DB installer: This builds an "M/DB Appliance" complete with
a pre-configured fully-working GT.M implementation, so you should be
up and running in about 5-10 minutes and you can ignore all the M/DB
stuff if you like :-)

See http://gradvs1.mgateway.com/main/index.html?path=mdb/mdbDownload

You'll find the working preconfigured GT.M environment in /usr/local/
gtm/ewd (/mdb/ewd if you use an EBS volume on EC2)

[K.S. Bhaskar]

On Nov 14, 5:00 am, Asaf Ary <asaf.dap...@gmail.com> wrote:
> > 6. It is extremely fast.  For online financial transactions, GT.M has
> > been benchmarked (depending on the transaction) at thousands of
> > transactions per second on an x86 GNU/Linux system with every
> > transaction having full ACID properties.
>
> I'm guessing that you're using two-phase commit for ACID properties in
> a distributed system.

[KSB] GT.Muses optimistic concurrency control. Within code bracketed
by TStart / TCommit commands, a process keeps track of the transaction
number in each block that it has read (or any block that it wants to
update - a block has to be read to be updated). At TCommit, if none
of the blocks has changed, i.e., none of them has a larger transaction
number, it updates the journal file (including an fsync), and updates
the database. If even one of the blocks has changed, it restarts the
transaction at the TStart and repeats. If it is not able to commit on
the third attempt, it performs the entire transaction inside a
critical section on the fourth attempt. So, compared to pessimistic
algorithms, this works well in the typical case.

While GT.M transactions can span multiple database files on one node,
they do not span database files on multiple nodes. Two phase commit
is not compatible with high end transaction processing throughput.
For business continuity, GT.M supports asynchronous streaming to as
many as 16 secondary instances (each of which can stream to 16
tertiary instances, etc.).

> Would you say that GT.M is a transaction optimized data store?
> I mean if you are maintaining ACID properties than you must be
> swapping Consistency for Availability.

[KSB] No, it is not a transaction optimized data store. If you don't
put TStart / TCommit brackets around code, then each update is applied
to the database by itself. GT.M is a key-value data store that also
supports ACID transactions. It also has a locking commands for
applications that want them. In fact many applications built on GT.M
do not use transaction processing, but the FIS Profile real-time core
banking application built on GT.M (http://fis-profile.com) has
eschewed locking in favor of TP, and the resulting scalability has
given it a significant competitive edge (like a half to one order of
magnitude greater throughput).

With respect to the CAP conjecture, yes, GT.M trades Consistency. But
it does things slightly differently since in applications like
banking, eventual consistency does not suffice - we need eventual
Consistency. I can try to write up and post some examples if there is
interest.

> I'll try to build the NoSQL profile of the GT.M user according to what
> you said (correct me if I'm wrong):
>
> 1. I read and write all my data in a simple Key / Value form
> 2. I don't want schema-constraints for my values
> 3. My data needs to be strongly consistent (not eventually
> consistent), clients should have a "read-your-writes" behavior.

[KSB] These are true if the application uses TP.

> 4. I use the database for complex offline operations and not real time
> online operations (such as logging or serving online requests)
>
> On (4) what I mean is that since you maintain ACID it is probably
> undesirable to use GT.M for cases like shopping-carts where failures
> are best handled after the user checked-out rather than failing to add
> stuff to his cart.
>
> Please correct me if I'm wrong.

[KSB] Actually, I think it is OK for a shopping cart, but GT.M does
things a little differently for eventual Consistency. As I said, let
me know if you want me to write up some examples.

Regards
-- Bhaskar

[Asaf Ary]

Thanks Bhaskar, that was a very thorough reply. I now have a much
clearer sense of which types of applications suite GT.M.

I appreciate your openness and willingness to post information
regarding your project.
I would be very interested to learn how you employ eventual
consistency in GT.M, I would only suggest that we start a new post
since the information might get lost under this thread (due to the
very general topic)


Thanks Again
-- Asaf

[K.S. Bhaskar]

Thanks, Asaf. I'll write something up and post it in a new thread in
the next day or so (I'll be traveling, so timing is a little
unpredictable).

Regards
-- Bhaskar

[toby]

On Oct 30, 5:34 am, edlich <edl...@gmail.com> wrote:
> Dear NOSQL Community,
>
> I am gathering all material about the nosql topic since a few month
> for
> a university lecture Databases Master. That's why I thought it could
> be
> useful for the community to give something back (what the community
> gives
> me with awesome tools ;-). So I copied it online:
>
> http://nosql-databases.org

Very nice; it's just a little irritating how your anchors aren't the
highlighted headings, but rather the >> symbol only. That's
disconcerting as it runs against common practice. (Applies to main
listing, sidebars, etc.)

> ...