Some of us within the Mozilla Community have been pretty vocal about not
liking the idea of a SQLite-derived API [1][2][3], and, in addition to
proposing alternatives, have been energetic about actually giving us a
pretty substantial strawperson (BrowserCouch)[2].
Essentially, arguments made in favor of supporting the Web Storage API
[4] ("SQL API") include the fact that WebKit has already implemented it
[5], that developers are familiar with SQL, and that variations within
SQL (from the SQLite set) aren't substantial. Developers are already
familiar enough with SQL, and work with it *already*, or so the argument
goes.
But arguments against it point out that SQL may not be right for the web
(since most people write wrappers anyway), that this could possibly tie
us to SQLite (raising security concerns), and that SQL's evolution isn't
a desirable thing to be harnessed to.
Prominent alternatives so far are CouchDB (or BrowserCouch[2], which is
a JavaScript implementation of CouchDB). Arguments given in favor of
this so far are that the actual query semantics are delegated out to
JavaScript, making it easy to learn and standardize. Generally,
arguments in favor of JavaScript APIs for storage (including using HTTP
+ JSON mechanisms) make the case that these ways are more "web-like"
than raw SQL statements. Additionally, performance advantages using a
MapReduce paradigm (such as in CouchDB) are desirable to take advantage
of multiple processor cores. Arguments against suggest that yet another
API is a reinvention of the wheel -- why not standardize something many
developers are already familiar with?
Other alternatives include MongoDB [6], but I'm not sure we've
investigated that as thoroughly.
We've had some of this discussion on mozilla.dev.platform[7], but that
newsgroup is often used for much else. This listserv/newsgroup is for
discussing web standards, particularly emerging ones. We should try and
make concrete proposals to the W3C WebApps WG, but first we'll need some
discussion :-)
-- A*
[1] Vlad: http://blog.vlad1.com/2009/04/06/html5-web-storage-and-sql/
[2] Atul: http://www.toolness.com/wp/?p=580
[3] Dion:
http://almaer.com/blog/browser-storage-do-we-need-sql-or-would-a-json-approach-be-better
[4] http://dev.w3.org/html5/webstorage/
[5] Surfin' Safari:
http://webkit.org/blog/126/webkit-does-html5-client-side-database-storage/
[6] http://www.mongodb.org/display/DOCS/Philosophy
[7] m.d.p thread:
http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/49aa555219df43ae/d3a6e8049a6ce1cf?lnk=gst&q=SQL#d3a6e8049a6ce1cf
Do we worry any less about theora/ogg or cairo? I assume we'd have tests :)
I have two concerns.
First of all, is a SQL backend what you want if you're building an
library like BrowserCouch or MungoDB?
I'll note that many SQL databases are backed by table engines like
MyISAM, BerkleyDB, and InnoDB. So SQL is sometimes used as a frontend.
Perf-wise it's probably suboptimal to build a new frontend on top of
the SQL frontend.
So if all we're doing is to provide an API that we want people to
build libraries on top of, should we instead take a look at the
MYISAM/BerkleyDB/InnoDB APIs?
My second concern is that if we standardize a SQL dialect, how much
work will we have to do in order to exactly conform to that dialect.
The prospect of even writing an SQL parser is not that exiting to me,
much less if we have to implement parts of that dialect.
>> Not to mention what happens when SQLite changes, requires an upgrade, etc.
>
> I don't see that as any different than what we are required to handle today.
> SQLite, if chosen, wouldn't be the first 3rd party source we use to deliver
> web content features.
I know sometimes other databases makes upgrades that intentionally is
not compatible with previous versions. It's been a while since I dealt
with databases, but as I recall it every major upgrade came with some
incompatibilities that you had to check against before upgrading.
That's simply not an option on the web.
> If we're afraid to do anything around SQL for fear of creating an SQL
> monster, so be it. I still stand by my opposition to send a JS wrapper to
> the W3C for standardization.
This makes sense to me.
/ Jonas
Vladimir already mentioned that BrowserCouch isn't capable of working
on larger data sets than local storage can manage. I'd propose that we
consider a more complete strawman, actual CouchDB in the browser. For
the purposes of this discussion, this would mean a BrowserCouch
capable of working with larger data sets. But it would mean something
more important as well.
CouchDB is designed from the ground up for offline replication
scenarios and disconnected access. A CouchDB database is by its nature
shared by multiple hosts, and uses replication to transparently sync
changes.
Against this background, CouchDB's other features, like the REST API
and JavaScript Map Reduce, are merely nice to have. I think when we
talk about next-generation web storage, we should be thinking about
offline replication above all else. This is the feature that allows
users to treat cloud hosts as commodities, paving the way for the p2p
web.
I'd *love* to invite everyone in this list to my talk focussing on
standalone CouchDB applications and offline mode, but that's probably
not feasible. The most recent video of my offline-web talk is
available here for those who are interested:
http://blog.oskarsson.nu/2009/06/nosql-debrief.html
Cheers,
Chris
--
Chris Anderson
http://jchrisa.net
http://couch.io
On 6/23/2009 6:41 AM, Mark Finkle wrote:
> I still stand by my opposition to send a JS wrapper to the W3C for
> standardization.
Who's proposing that we standardize a JS wrapper? The W3C would
standardize an API, whether we implement that with JS or not, and
whether or not people can reach under it in our implementation to
another storage representation.
I like the Couch API better than SQL because it's easier to represent
non-relational data with it (esp. including hierarchies, which are
brutal to work with in SQL), and because it feels to me like its
semantics can be more clearly extended to cover both local and remote
data sources. (I say "feel" because I haven't tried to write to that
model yet.) I also think that the reuse-of-SQL-knowledge situation is
more of an uncanny valley than a significant gain for developers once
they get beyond using it for the simplest of things: sqlite's type
system and subset of SQL are going to require a lot of care.
Mike
I'll reiterate the importance of offline replication. Most existing
offline stores (I'm looking at you Gears) require application
developers to implement code to pull changes in from the server and
store them locally. This is a waste of developer time and results in
half-baked replication schemes.
Without arguing for CouchDB specifically, I'll ask you to consider the
benefits of a local storage which can be synced with multiple remote
hosts, without web developers having to do anything more complex than
triggering replication. This is the powerful bit. I'd go so far as to
say I'd rather have a SQL interface to a local store that had offline
replication, than a HTTP/JSON interface to a store that didn't. Of
course, CouchDB is the best of both these worlds. :)
>> My second concern is that if we standardize a SQL dialect, how much
>> work will we have to do in order to exactly conform to that dialect.
>> The prospect of even writing an SQL parser is not that exiting to me,
>> much less if we have to implement parts of that dialect.
>
> SQL dialect would be hard to specify. If we could get the benefits of
> indexing, queries and fast record access in large datasets without SQL as
> the primary dialect we'd probably be much better off.
>
> _______________________________________________
> community-web-standards mailing list
> community-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/community-web-standards
>
>
--
Is DB replication something that sites will want? One use case that
keeps being brought up is offline gmail. I would not expect gmail to
want to replicate the full mail database to the client simply because
it's really large.
Or is the idea to replicate only part of the database (sorry, i'm
revealing my lack of DB knowledge here)
However, if DB replication is something that we expect that people do
want, then I agree it's great if synchronization is part of the
specification and implementation.
As others have said, we really need to collect use cases and requirements.
/ Jonas
Cheers,
Shawn
It's not quite a dialect of SQL when it integrates that tightly with
language semantics (in this case C#). Something *that looks kind of
like that* as an API for JavaScript could be cool, but it takes a lot
more thought and work. Something like this could agnostically straddle
whatever kind of backend, but now maybe I've drunk too much LINQaid.
It's worth asking Microsoft about this (and I did).
-- A*
-- A*
Yes, there are variations on this approach (jsLINQ as a proof of
concept, osme others). All of them need an underlying storage mechanism
that they can be built on top of, though, and I think that's something
that we haven't really examined (the low level solution).
- Vlad
----- "Dion Almaer" <di...@mozilla.com> wrote:
> The reason that Gear's didn't implement syncing was because the team couldn't agree on a generic sync primitive that would work for the various Google products that were being worked on. Syncing is hard, so they punted. The team spent a LOT of time on this side of offline Gmail and I still find it to be fairly buggy. (I know Chris... maybe they should have used CouchDB ;).
Having done this for a Windows app in a previous life, I know that it's hard. I also wonder whether a "one-size-fits-all" sync mechanism will work in general.
Well, we should be focused on the right solution for the web -- doing
the low-level solution is one option. I think we're still examining all
approaches. A low-level API would be the right thing IMO, but I don't
know enough about the requirements to actually propose one, short of
something fairly simplistic.
- Vlad
> ----- "Dion Almaer" <di...@mozilla.com> wrote:
> > The reason that Gear's didn't implement syncing was because the team couldn't agree on a generic sync primitive that would work for the various Google products that were being worked on. Syncing is hard, so they punted. The team spent a LOT of time on this side of offline Gmail and I still find it to be fairly buggy. (I know Chris... maybe they should have used CouchDB ;).
> Having done this for a Windows app in a previous life, I know that it's hard. I also wonder whether a "one-size-fits-all" sync mechanism will work in general.
>
Mike
On 23-Jun-09, at 1:23 PM, Mark Finkle wrote:----- "Dion Almaer" <di...@mozilla.com> wrote:
> The reason that Gear's didn't implement syncing was because the team couldn't agree on a generic sync primitive that would work for the various Google products that were being worked on. Syncing is hard, so they punted. The team spent a LOT of time on this side of offline Gmail and I still find it to be fairly buggy. (I know Chris... maybe they should have used CouchDB ;).
Having done this for a Windows app in a previous life, I know that it's hard. I also wonder whether a "one-size-fits-all" sync mechanism will work in general.Are we conflating issues here? As I recall HTML5 has a specification for offline application notification which already indicates that dealing with sync, etc, is the responsibility of the application author.
Perhaps what you're saying is that whatever storage mechanism we provide should have easy-to-call methods for saying "syncUp()"?
cheers,mike
/sdwilsh
E4X all over again :)
I'm not convinced a relational store is necessarily the best answer. We all seem to spend a lot of time writing object/relational mappings and maybe it would be best to take a higher level approach for the web platform. I note also that there was a suggestion the other day on the WebApps mailing list for a more XML focused approach to structured storage using XPath or XQuery as the query language. That is one kind of alternative that I have been considering given that it may address many of the use cases for structured data and already has a bunch of ratified and implemented standards to support it.
I agree, although having a hierarchical store related to the XML
infoset might not involve actually seeing angle brackets and namespace
URIs. This is just one alternative, though, to a RDBMS.
> That said, JavaScripty versions of XPath and XQuery may not be unreasonable:
>
> http://www.sitepen.com/blog/2008/07/16/jsonquery-data-querying-beyond...
Absolutely.
> Essentially, arguments made in favor of supporting the Web Storage
> API [4] ("SQL API") include the fact that WebKit has already
> implemented it [5],
I can think of four futures regarding SQL:
1) Mozilla implements something else. Other browsers follow. SQL
becomes a WebKit-specific oddity and fades into irrelevance.
2) Mozilla implements SQL compatibly with WebKit. SQL becomes the
cross-browser thing.
3) Mozilla implements SQL early on but incompatibly with WebKit.
Developers have to write code for multiple SQL engines.
4) Mozilla implements something else. SQL wins anyway leaving
Mozilla at a competitive disadvantage. Mozilla gets dragged into
implementing SQL when existing WebKit-targeted content that assumes
SQLite peculiarities already abounds.
It seems to me that not getting heavily involved with speccing and
implementing a 'Web SQL' makes sense only if it's realistic to execute
future #1. If future #3 or #4 happened, it would have been better to
work towards future #2 instead and get the Web SQL dialect specced to
Mozilla's satisfaction to ASAP. (Web SQL probably should be
implementable on top of SQLite.)
I don't know if it is realistic to execute future #1.
--
Henri Sivonen
hsiv...@iki.fi
http://hsivonen.iki.fi/
#4 really is the same as #1. In any case #1 may not be that different
from #3 in the long term because basing things on SQLite and offering
a hard-to-standardize vocabulary means that there will be divergence
sooner rather than later.
Perhaps the WebKit team see reason in what Mozilla is trying to do.
But maybe it is futile to expect that.
All I can say is that my respect for Mozilla doing the right thing
just went up after going through this discussion.
> It seems to me that not getting heavily involved with speccing and
> implementing a 'Web SQL' makes sense only if it's realistic to execute
> future #1. If future #3 or #4 happened, it would have been better to
> work towards future #2 instead and get the Web SQL dialect specced to
> Mozilla's satisfaction to ASAP. (Web SQL probably should be
> implementable on top of SQLite.)
>
> I don't know if it is realistic to execute future #1.
>
It is entirely realistic to standardize something else and still find
a compromise on SQL support without necessarily creating a standard,
since it will be hard to create a public standard for SQL unless you
can silence all the criticism.
/sdwilsh