SQL Storage | What Should Be Done?

74 views
Skip to first unread message

Arun Ranganathan

unread,
Jun 22, 2009, 8:36:00 PM6/22/09
to community-w...@lists.mozilla.org
The dust is just about settling on our 1.9.1 branch release. For future
versions of Firefox following Fx 3.5, it's time to think about the SQL
Storage feature exposed to web content. We're not all reconciled on the
best course of action even within Mozilla.

Some of us within the Mozilla Community have been pretty vocal about not
liking the idea of a SQLite-derived API [1][2][3], and, in addition to
proposing alternatives, have been energetic about actually giving us a
pretty substantial strawperson (BrowserCouch)[2].

Essentially, arguments made in favor of supporting the Web Storage API
[4] ("SQL API") include the fact that WebKit has already implemented it
[5], that developers are familiar with SQL, and that variations within
SQL (from the SQLite set) aren't substantial. Developers are already
familiar enough with SQL, and work with it *already*, or so the argument
goes.

But arguments against it point out that SQL may not be right for the web
(since most people write wrappers anyway), that this could possibly tie
us to SQLite (raising security concerns), and that SQL's evolution isn't
a desirable thing to be harnessed to.

Prominent alternatives so far are CouchDB (or BrowserCouch[2], which is
a JavaScript implementation of CouchDB). Arguments given in favor of
this so far are that the actual query semantics are delegated out to
JavaScript, making it easy to learn and standardize. Generally,
arguments in favor of JavaScript APIs for storage (including using HTTP
+ JSON mechanisms) make the case that these ways are more "web-like"
than raw SQL statements. Additionally, performance advantages using a
MapReduce paradigm (such as in CouchDB) are desirable to take advantage
of multiple processor cores. Arguments against suggest that yet another
API is a reinvention of the wheel -- why not standardize something many
developers are already familiar with?

Other alternatives include MongoDB [6], but I'm not sure we've
investigated that as thoroughly.

We've had some of this discussion on mozilla.dev.platform[7], but that
newsgroup is often used for much else. This listserv/newsgroup is for
discussing web standards, particularly emerging ones. We should try and
make concrete proposals to the W3C WebApps WG, but first we'll need some
discussion :-)

-- A*

[1] Vlad: http://blog.vlad1.com/2009/04/06/html5-web-storage-and-sql/
[2] Atul: http://www.toolness.com/wp/?p=580
[3] Dion:
http://almaer.com/blog/browser-storage-do-we-need-sql-or-would-a-json-approach-be-better

[4] http://dev.w3.org/html5/webstorage/

[5] Surfin' Safari:
http://webkit.org/blog/126/webkit-does-html5-client-side-database-storage/
[6] http://www.mongodb.org/display/DOCS/Philosophy
[7] m.d.p thread:
http://groups.google.com/group/mozilla.dev.platform/browse_thread/thread/49aa555219df43ae/d3a6e8049a6ce1cf?lnk=gst&q=SQL#d3a6e8049a6ce1cf

Mark Finkle

unread,
Jun 22, 2009, 11:53:10 PM6/22/09
to ar...@mozilla.com, community-w...@lists.mozilla.org
IMO, talking about SQL Storage and CouchDB in the same context is missing the point. BrowserCouch is a wrapper around Local Storage. It's done. There is no need to standardize anything. In fact, we already have standardized the feature that makes BrowserCouch viable - Local Storage.

SQL storage is about creating a different type of storage _backend_. Of course people will create wrappers for it. Just like they create wrappers for Local Storage. Heck, people create wrappers for anything and everything and more power to them. That's another reason I really hope a JS wrapper (like BrowserCouch) does not get standardized by W3C. That's like saying jQuery should be the W3C DOM standard.

To me the question is: Do we want an alternative to Local Storage? My answer is "yes"

http://starkravingfinkle.org/blog/2009/04/day-dreaming-about-web-storage/
> _______________________________________________
> community-web-standards mailing list
> community-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/community-web-standards
>

Mike Beltzner

unread,
Jun 23, 2009, 12:20:52 AM6/23/09
to Mark Finkle, ar...@mozilla.com, community-w...@lists.mozilla.org
So what are you proposing, Mark? Just "SQLite, and hope that we're all talking about the same version"?

Your future sounds rosy and your points are interesting, but I think you're missing the middle part, which is the hard work about how we ensure that we don't fracture the space and end up with web authors who have to write multiple storage layers and libraries in order to work with the version of SQLite or MySQL (which is what IE is far, far, far more likely to ship) included with various browsers. Not to mention what happens when SQLite changes, requires an upgrade, etc.

Perhaps I'm missing a nuance in what you're writing, though.

cheers,
mike

Mark Finkle

unread,
Jun 23, 2009, 12:41:56 AM6/23/09
to Mike Beltzner, ar...@mozilla.com, community-w...@lists.mozilla.org
> So what are you proposing, Mark? Just "SQLite, and hope that we're all talking about the same version"?

I have no love for SQLite. I do think we should be considering an SQL backend.

>
Your future sounds rosy and your points are interesting, but I think you're missing the middle part, which is the hard work about how we ensure that we don't fracture the space and end up with web authors who have to write multiple storage layers and libraries in order to work with the version of SQLite or MySQL (which is what IE is far, far, far more likely to ship) included with various browsers.

There is plenty of work ahead to create a "web profile" for SQL, no doubt. In any case, I firmly believe that JS wrappers will be used over any form of SQL backend. Beyond convenience, those wrappers will also be used to mitigate incompatibilities - which will surely exist, even in my rosy future.


> Not to mention what happens when SQLite changes, requires an upgrade, etc.

I don't see that as any different than what we are required to handle today. SQLite, if chosen, wouldn't be the first 3rd party source we use to deliver web content features.

If we're afraid to do anything around SQL for fear of creating an SQL monster, so be it. I still stand by my opposition to send a JS wrapper to the W3C for standardization.

Robert O'Callahan

unread,
Jun 23, 2009, 1:13:57 AM6/23/09
to Mark Finkle, Shawn Wilsher, community-w...@lists.mozilla.org
On Tue, Jun 23, 2009 at 5:05 PM, Mark Finkle <mfi...@mozilla.com> wrote:
Do we worry any less about theora/ogg or cairo? I assume we'd have tests :)

cairo doesn't directly consume Web content. Ogg does, and that is a bit of a worry, but at least the Ogg developers know they're interpreting a frozen format which has an actual spec. I'm not sure the SQLite developers know this, or have a spec.

Rob
--
"He was pierced for our transgressions, he was crushed for our iniquities; the punishment that brought us peace was upon him, and by his wounds we are healed. We all, like sheep, have gone astray, each of us has turned to his own way; and the LORD has laid on him the iniquity of us all." [Isaiah 53:5-6]

Dion Almaer

unread,
Jun 23, 2009, 1:29:57 AM6/23/09
to rob...@ocallahan.org, Mark Finkle, Shawn Wilsher, community-w...@lists.mozilla.org
It personally feels too early to say either:

- We shouldn't do SQL at all as it isn't "Webby" enough
- We should only do SQL, as that is what the other chaps are doing

I am very conflicted. A lot of people ended up using wrappers on top of SQL back in the Gears days, but many other developers did like using SQL. They liked using their modeling tools, and reusing some of their skills. Standardizing on this stuff before we see what developers really want seems scary, and having Hixie talk about "just taking the behaviour of SQLite" .... yowser.

Maybe we can charge forward in a couple of dimensions? A SQL provider of some sort (openly talking about how "just doing SQLite" may seem easy, but long term is scary) but also putting energy into a RESTful JSON API like we have seen in experiments. I would also love to see a simple API that can sit on top of both ;)

There is a lot of room for interesting parties to get involved too. Sybase have a pretty cool replication service and they hacked Gears to use it as the provider. Ideally our SQL interface would allow new providers, just as Geolocation will allow. Much more work, but the right thing to do.

Cheers,

Dion



Jonas Sicking

unread,
Jun 23, 2009, 2:42:21 AM6/23/09
to Mark Finkle, ar...@mozilla.com, community-w...@lists.mozilla.org, Mike Beltzner
On Mon, Jun 22, 2009 at 9:41 PM, Mark Finkle<mfi...@mozilla.com> wrote:
>> So what are you proposing, Mark? Just "SQLite, and hope that we're all
>> talking about the same version"?
> I have no love for SQLite. I do think we should be considering an SQL
> backend.
>
>> Your future sounds rosy and your points are interesting, but I think you're
>> missing the middle part, which is the hard work about how we ensure that we
>> don't fracture the space and end up with web authors who have to write
>> multiple storage layers and libraries in order to work with the version of
>> SQLite or MySQL (which is what IE is far, far, far more likely to ship)
>> included with various browsers.
>
> There is plenty of work ahead to create a "web profile" for SQL, no doubt.
> In any case, I firmly believe that JS wrappers will be used over any form of
> SQL backend. Beyond convenience, those wrappers will also be used to
> mitigate incompatibilities - which will surely exist, even in my rosy
> future.

I have two concerns.

First of all, is a SQL backend what you want if you're building an
library like BrowserCouch or MungoDB?

I'll note that many SQL databases are backed by table engines like
MyISAM, BerkleyDB, and InnoDB. So SQL is sometimes used as a frontend.
Perf-wise it's probably suboptimal to build a new frontend on top of
the SQL frontend.

So if all we're doing is to provide an API that we want people to
build libraries on top of, should we instead take a look at the
MYISAM/BerkleyDB/InnoDB APIs?

My second concern is that if we standardize a SQL dialect, how much
work will we have to do in order to exactly conform to that dialect.
The prospect of even writing an SQL parser is not that exiting to me,
much less if we have to implement parts of that dialect.

>> Not to mention what happens when SQLite changes, requires an upgrade, etc.
>
> I don't see that as any different than what we are required to handle today.
> SQLite, if chosen, wouldn't be the first 3rd party source we use to deliver
> web content features.

I know sometimes other databases makes upgrades that intentionally is
not compatible with previous versions. It's been a while since I dealt
with databases, but as I recall it every major upgrade came with some
incompatibilities that you had to check against before upgrading.
That's simply not an option on the web.

> If we're afraid to do anything around SQL for fear of creating an SQL
> monster, so be it. I still stand by my opposition to send a JS wrapper to
> the W3C for standardization.

This makes sense to me.

/ Jonas

Chris Anderson

unread,
Jun 23, 2009, 2:48:33 AM6/23/09
to Mark Finkle, ar...@mozilla.com, community-w...@lists.mozilla.org
On Mon, Jun 22, 2009 at 8:53 PM, Mark Finkle<mfi...@mozilla.com> wrote:
> IMO, talking about SQL Storage and CouchDB in the same context is missing
> the point. BrowserCouch is a wrapper around Local Storage. It's done. There
> is no need to standardize anything. In fact, we already have standardized
> the feature that makes BrowserCouch viable - Local Storage.

Vladimir already mentioned that BrowserCouch isn't capable of working
on larger data sets than local storage can manage. I'd propose that we
consider a more complete strawman, actual CouchDB in the browser. For
the purposes of this discussion, this would mean a BrowserCouch
capable of working with larger data sets. But it would mean something
more important as well.

CouchDB is designed from the ground up for offline replication
scenarios and disconnected access. A CouchDB database is by its nature
shared by multiple hosts, and uses replication to transparently sync
changes.

Against this background, CouchDB's other features, like the REST API
and JavaScript Map Reduce, are merely nice to have. I think when we
talk about next-generation web storage, we should be thinking about
offline replication above all else. This is the feature that allows
users to treat cloud hosts as commodities, paving the way for the p2p
web.

I'd *love* to invite everyone in this list to my talk focussing on
standalone CouchDB applications and offline mode, but that's probably
not feasible. The most recent video of my offline-web talk is
available here for those who are interested:

http://blog.oskarsson.nu/2009/06/nosql-debrief.html

Cheers,
Chris

--
Chris Anderson
http://jchrisa.net
http://couch.io

Mike Shaver

unread,
Jun 23, 2009, 3:12:17 AM6/23/09
to community-w...@lists.mozilla.org
[resending to list, since the wrong address was subscribed for me
before, etc.]

On 6/23/2009 6:41 AM, Mark Finkle wrote:
> I still stand by my opposition to send a JS wrapper to the W3C for
> standardization.

Who's proposing that we standardize a JS wrapper? The W3C would
standardize an API, whether we implement that with JS or not, and
whether or not people can reach under it in our implementation to
another storage representation.

I like the Couch API better than SQL because it's easier to represent
non-relational data with it (esp. including hierarchies, which are
brutal to work with in SQL), and because it feels to me like its
semantics can be more clearly extended to cover both local and remote
data sources. (I say "feel" because I haven't tried to write to that
model yet.) I also think that the reuse-of-SQL-knowledge situation is
more of an uncanny valley than a significant gain for developers once
they get beyond using it for the simplest of things: sqlite's type
system and subset of SQL are going to require a lot of care.

Mike

Mark Finkle

unread,
Jun 23, 2009, 8:09:29 AM6/23/09
to Vladimir Vukicevic, ar...@mozilla.com, community-w...@lists.mozilla.org

----- "Vladimir Vukicevic" <vlad...@mozilla.com> wrote:
>
> ----- "Mark Finkle" <mfi...@mozilla.com> wrote:
>
> > IMO, talking about SQL Storage and CouchDB in the same context is
> > missing the point. BrowserCouch is a wrapper around Local Storage.
> > It's done. There is no need to standardize anything. In fact, we
> > already have standardized the feature that makes BrowserCouch viable -
> > Local Storage.
>
> The key difference here is that anything implemented on top of local storage can't really operate on large (out-of-core) data sets, nor is there any provision for efficient indexing/querying of that data.  Those are the component pieces that we're trying to figure out how to expose, without just going down the SQL (whether SQLite or something else) path because that path is almost completely undefined and underspecified.

Yes, I think this is key as well. It's the main reason to consider a storage option that is _not_ Local Storage.

>
> A good exercise would be to consider what the minimum set of missing capabilities from the web are, perhaps something like:
>
> 1) How do you provide access to out-of-core data in content javascript (that is, how do you expose the difference between "disk" and "memory");
>
> 2) Is it possible to implement efficient query execution purely in javascript, interpreting either a SQL-like syntax or a JS-like syntax.

Excellent points. If we could create a storage solution that meets these goals, I think we'd be on the right track. With such a solution, JS wrappers could manipulate the API in many different ways, but still be more performant than a Local Storage (single blob) solution.

> Would that be enough to implement efficient queries over large data using whatever wrapper the user wishes?  Deciding on some testcases/benchmarks here would be useful (e.g. 10,000 email messages, searching for those that contain some word)..

Yeah, coming up with some testcases would be very helpful.

Mark Finkle

unread,
Jun 23, 2009, 8:15:02 AM6/23/09
to Jonas Sicking, ar...@mozilla.com, community-w...@lists.mozilla.org, Mike Beltzner

----- "Jonas Sicking" <jo...@sicking.cc> wrote:

> So if all we're doing is to provide an API that we want people to
> build libraries on top of, should we instead take a look at the
> MYISAM/BerkleyDB/InnoDB APIs?

Agreed. I am less concerned with using SQL (or SQLite), and more concerned with creating a storage option that is different than Local Storage. I think we need a storage option that allows people to develop data acess systems that couldn't be efficiently handled by Local Storage.


> My second concern is that if we standardize a SQL dialect, how much
> work will we have to do in order to exactly conform to that dialect.
> The prospect of even writing an SQL parser is not that exiting to me,
> much less if we have to implement parts of that dialect.

SQL dialect would be hard to specify. If we could get the benefits of indexing, queries and fast record access in large datasets without SQL as the primary dialect we'd probably be much better off.

Chris Anderson

unread,
Jun 23, 2009, 9:27:47 AM6/23/09
to Mark Finkle, ar...@mozilla.com, community-w...@lists.mozilla.org, Jonas Sicking, Mike Beltzner
On Tue, Jun 23, 2009 at 5:15 AM, Mark Finkle<mfi...@mozilla.com> wrote:
>
> ----- "Jonas Sicking" <jo...@sicking.cc> wrote:
>
>> So if all we're doing is to provide an API that we want people to
>> build libraries on top of, should we instead take a look at the
>> MYISAM/BerkleyDB/InnoDB APIs?
>
> Agreed. I am less concerned with using SQL (or SQLite), and more concerned
> with creating a storage option that is different than Local Storage. I think
> we need a storage option that allows people to develop data acess systems
> that couldn't be efficiently handled by Local Storage.
>

I'll reiterate the importance of offline replication. Most existing
offline stores (I'm looking at you Gears) require application
developers to implement code to pull changes in from the server and
store them locally. This is a waste of developer time and results in
half-baked replication schemes.

Without arguing for CouchDB specifically, I'll ask you to consider the
benefits of a local storage which can be synced with multiple remote
hosts, without web developers having to do anything more complex than
triggering replication. This is the powerful bit. I'd go so far as to
say I'd rather have a SQL interface to a local store that had offline
replication, than a HTTP/JSON interface to a store that didn't. Of
course, CouchDB is the best of both these worlds. :)

>> My second concern is that if we standardize a SQL dialect, how much
>> work will we have to do in order to exactly conform to that dialect.
>> The prospect of even writing an SQL parser is not that exiting to me,
>> much less if we have to implement parts of that dialect.
>
> SQL dialect would be hard to specify. If we could get the benefits of
> indexing, queries and fast record access in large datasets without SQL as
> the primary dialect we'd probably be much better off.
>

> _______________________________________________
> community-web-standards mailing list
> community-w...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/community-web-standards
>
>

--

Jonas Sicking

unread,
Jun 23, 2009, 11:35:28 AM6/23/09
to Chris Anderson, Mark Finkle, ar...@mozilla.com, community-w...@lists.mozilla.org, Mike Beltzner
On Tue, Jun 23, 2009 at 6:27 AM, Chris Anderson<jch...@apache.org> wrote:
> On Tue, Jun 23, 2009 at 5:15 AM, Mark Finkle<mfi...@mozilla.com> wrote:
>>
>> ----- "Jonas Sicking" <jo...@sicking.cc> wrote:
>>
>>> So if all we're doing is to provide an API that we want people to
>>> build libraries on top of, should we instead take a look at the
>>> MYISAM/BerkleyDB/InnoDB APIs?
>>
>> Agreed. I am less concerned with using SQL (or SQLite), and more concerned
>> with creating a storage option that is different than Local Storage. I think
>> we need a storage option that allows people to develop data acess systems
>> that couldn't be efficiently handled by Local Storage.
>>
>
> I'll reiterate the importance of offline replication. Most existing
> offline stores (I'm looking at you Gears) require application
> developers to implement code to pull changes in from the server and
> store them locally. This is a waste of developer time and results in
> half-baked replication schemes.

Is DB replication something that sites will want? One use case that
keeps being brought up is offline gmail. I would not expect gmail to
want to replicate the full mail database to the client simply because
it's really large.

Or is the idea to replicate only part of the database (sorry, i'm
revealing my lack of DB knowledge here)

However, if DB replication is something that we expect that people do
want, then I agree it's great if synchronization is part of the
specification and implementation.

As others have said, we really need to collect use cases and requirements.

/ Jonas

Dion Almaer

unread,
Jun 23, 2009, 11:41:21 AM6/23/09
to Jonas Sicking, Mark Finkle, ar...@mozilla.com, Chris Anderson, community-w...@lists.mozilla.org, Mike Beltzner
The reason that Gear's didn't implement syncing was because the team couldn't agree on a generic sync primitive that would work for the various Google products that were being worked on. Syncing is hard, so they punted. The team spent a LOT of time on this side of offline Gmail and I still find it to be fairly buggy. (I know Chris... maybe they should have used CouchDB ;).

Developers started to roll their own which was a shame (and also rolled their own "isOffline" for browsers that couldn't use the navigator offline work) because even that wasn't offered....

Mark Finkle

unread,
Jun 23, 2009, 1:23:37 PM6/23/09
to Dion Almaer, ar...@mozilla.com, Chris Anderson, community-w...@lists.mozilla.org, Jonas Sicking, Mike Beltzner
----- "Dion Almaer" <di...@mozilla.com> wrote:
> The reason that Gear's didn't implement syncing was because the team couldn't agree on a generic sync primitive that would work for the various Google products that were being worked on. Syncing is hard, so they punted. The team spent a LOT of time on this side of offline Gmail and I still find it to be fairly buggy. (I know Chris... maybe they should have used CouchDB ;).

Having done this for a Windows app in a previous life, I know that it's hard. I also wonder whether a "one-size-fits-all" sync mechanism will work in general.

>
Developers started to roll their own which was a shame (and also rolled their own "isOffline" for browsers that couldn't use the navigator offline work) because even that wasn't offered....

I don't see this as a shame. There are a lot of moving parts to a sync system and it can be highly data format specific. As long as the storage system has hooks for bolting on a sync system, I think that vacuum will be filled.

Shawn Wilsher

unread,
Jun 23, 2009, 1:27:57 PM6/23/09
to ar...@mozilla.com, community-w...@lists.mozilla.org
On 6/22/09 11:18 PM, Arun Ranganathan wrote:
> Can anyone determine if an upgraded SQLite has *ever* broken previous
> SQLite installations, particularly with respect to SQL statements? I'm
> trying to determine whether this is a concrete worry or an abstract
> bogeyperson.
I can say, with certainty, that it has. Recently SQLite fixed a buggy
implementation of some of their functions that we were relying on (we
were, in fact wrong).

Cheers,

Shawn

sdwilsh.vcf

Arun Ranganathan

unread,
Jun 23, 2009, 2:27:44 PM6/23/09
to Rob Sayre, community-w...@lists.mozilla.org
Rob Sayre wrote:

> On 6/22/09 5:36 PM, Arun Ranganathan wrote:
>>
>> But arguments against it point out that SQL may not be right for the
>> web (since most people write wrappers anyway), that this could
>> possibly tie us to SQLite (raising security concerns), and that SQL's
>> evolution isn't a desirable thing to be harnessed to.
>
> I am concerned about taking a web-facing SQLite dependency until the
> end of time.
Me too (especially after sdwilsh's claim of annoying breakages).
> I'm curious about Microsoft's stance on this issue.
So I honestly don't think LINQ is terrible:
http://msdn.microsoft.com/en-us/magazine/cc163400.aspx

It's not quite a dialect of SQL when it integrates that tightly with
language semantics (in this case C#). Something *that looks kind of
like that* as an API for JavaScript could be cool, but it takes a lot
more thought and work. Something like this could agnostically straddle
whatever kind of backend, but now maybe I've drunk too much LINQaid.

It's worth asking Microsoft about this (and I did).

-- A*

Arun Ranganathan

unread,
Jun 23, 2009, 2:28:35 PM6/23/09
to Shawn Wilsher, community-w...@lists.mozilla.org
OK, that's pretty damning. Of course, a mitigation against breaking
stuff would be to have a spec. ("SQL for Web Apps") such that subsequent
iterations of SQLite don't break the spec. But the allure of spec'ing
out a subset of SQL is rapidly fading. I originally thought it wise to
give developers something they already knew, but there are better
proposals on the table. It would be good to lead with an implementation
here.

-- A*

Vladimir Vukicevic

unread,
Jun 23, 2009, 2:39:39 PM6/23/09
to ar...@mozilla.com, Rob Sayre, community-w...@lists.mozilla.org

Yes, there are variations on this approach (jsLINQ as a proof of
concept, osme others). All of them need an underlying storage mechanism
that they can be built on top of, though, and I think that's something
that we haven't really examined (the low level solution).

- Vlad

Mark Finkle

unread,
Jun 23, 2009, 2:57:51 PM6/23/09
to Vladimir Vukicevic, ar...@mozilla.com, Rob Sayre, community-w...@lists.mozilla.org
----- "Vladimir Vukicevic" <vlad...@mozilla.com> wrote:
> Yes, there are variations on this approach (jsLINQ as a proof of
> concept, osme others).  All of them need an underlying storage mechanism
> that they can be built on top of, though, and I think that's something
> that we haven't really examined (the low level solution).

Isn't that the part we should be focused on? The low-level solution.

Mike Beltzner

unread,
Jun 23, 2009, 3:01:36 PM6/23/09
to Mark Finkle, ar...@mozilla.com, Chris Anderson, community-w...@lists.mozilla.org, Jonas Sicking, Dion Almaer
On 23-Jun-09, at 1:23 PM, Mark Finkle wrote:

----- "Dion Almaer" <di...@mozilla.com> wrote: 
> The reason that Gear's didn't implement syncing was because the team couldn't agree on a generic sync primitive that would work for the various Google products that were being worked on. Syncing is hard, so they punted. The team spent a LOT of time on this side of offline Gmail and I still find it to be fairly buggy. (I know Chris... maybe they should have used CouchDB ;).

Having done this for a Windows app in a previous life, I know that it's hard. I also wonder whether a "one-size-fits-all" sync mechanism will work in general.

Are we conflating issues here? As I recall HTML5 has a specification for offline application notification which already indicates that dealing with sync, etc, is the responsibility of the application author.

Perhaps what you're saying is that whatever storage mechanism we provide should have easy-to-call methods for saying "syncUp()"?

cheers,
mike

Vladimir Vukicevic

unread,
Jun 23, 2009, 3:05:24 PM6/23/09
to Mark Finkle, ar...@mozilla.com, Rob Sayre, community-w...@lists.mozilla.org

Well, we should be focused on the right solution for the web -- doing
the low-level solution is one option. I think we're still examining all
approaches. A low-level API would be the right thing IMO, but I don't
know enough about the requirements to actually propose one, short of
something fairly simplistic.

- Vlad

Mark Finkle

unread,
Jun 23, 2009, 3:09:14 PM6/23/09
to Mike Beltzner, ar...@mozilla.com, Chris Anderson, community-w...@lists.mozilla.org, Jonas Sicking, Dion Almaer
----- "Mike Beltzner" <belt...@mozilla.com> wrote:
>
On 23-Jun-09, at 1:23 PM, Mark Finkle wrote:

> ----- "Dion Almaer" <di...@mozilla.com> wrote: 
> > The reason that Gear's didn't implement syncing was because the team couldn't agree on a generic sync primitive that would work for the various Google products that were being worked on. Syncing is hard, so they punted. The team spent a LOT of time on this side of offline Gmail and I still find it to be fairly buggy. (I know Chris... maybe they should have used CouchDB ;).

> Having done this for a Windows app in a previous life, I know that it's hard. I also wonder whether a "one-size-fits-all" sync mechanism will work in general.
>

>
Are we conflating issues here? As I recall HTML5 has a specification for offline application notification which already indicates that dealing with sync, etc, is the responsibility of the application author.

Exactly. I'm saying let's keep it that way. Conflating the issues would be to add a sync spec to the storage spec. The two do not need to be merged. Let's keep them apart. The storage spec should allow for the existence of a sync mechanism, without actually defining it. Events and callbacks have been enough in the past to allow for such things. Most web specs contain some form of events and callbacks. We should be fine.

>
Perhaps what you're saying is that whatever storage mechanism we provide should have easy-to-call methods for saying "syncUp()"?

No. That's too explicit. The storage mechanism doesn't need to be that aware of my data format and syncing requirements.

Mike Shaver

unread,
Jun 23, 2009, 3:11:04 PM6/23/09
to mfi...@mozilla.com, vlad...@mozilla.com, ar...@mozilla.com, rsa...@mozilla.com, community-w...@lists.mozilla.org
No, we should be focused on the API we want to see in all browsers. It
doesn't have to be low-level in a database sense.

Mike

Dion Almaer

unread,
Jun 23, 2009, 3:16:37 PM6/23/09
to Mike Beltzner, Mark Finkle, ar...@mozilla.com, Chris Anderson, community-w...@lists.mozilla.org, Jonas Sicking
Sorry, I am just riffing on the "sync" point that came up on the thread and warning that coming up with a generic sync solution should probably be a non-goal (Mark also talked about his experience with sync there too). Focusing on the low level storage solution makes sense as Vlad and others have said (with an eye on the overall developer experience). There are also practical issues such as: would we be better off having SQL in most modern browsers so developers have something they can work with now-ish, versus pushing back and re-thinking what the Web should really do for storage.... and doing these in parallel (and again, letting the market decide who wins in the long run).


On Tue, Jun 23, 2009 at 12:01 PM, Mike Beltzner <belt...@mozilla.com> wrote:
On 23-Jun-09, at 1:23 PM, Mark Finkle wrote:

----- "Dion Almaer" <di...@mozilla.com> wrote: 
> The reason that Gear's didn't implement syncing was because the team couldn't agree on a generic sync primitive that would work for the various Google products that were being worked on. Syncing is hard, so they punted. The team spent a LOT of time on this side of offline Gmail and I still find it to be fairly buggy. (I know Chris... maybe they should have used CouchDB ;).

Having done this for a Windows app in a previous life, I know that it's hard. I also wonder whether a "one-size-fits-all" sync mechanism will work in general.

Are we conflating issues here? As I recall HTML5 has a specification for offline application notification which already indicates that dealing with sync, etc, is the responsibility of the application author.

Perhaps what you're saying is that whatever storage mechanism we provide should have easy-to-call methods for saying "syncUp()"?

cheers,
mike

Shawn Wilsher

unread,
Jun 23, 2009, 3:18:31 PM6/23/09
to community-w...@lists.mozilla.org
On 6/23/09 12:16 PM, Dion Almaer wrote:
> Sorry, I am just riffing on the "sync" point that came up on the thread
> and warning that coming up with a generic sync solution should probably
> be a non-goal (Mark also talked about his experience with sync there
> too). Focusing on the low level storage solution makes sense as Vlad and others have said (with an eye on the overall developer experience). There are also practical
> issues such as: would we be better off having SQL in most modern
> browsers so developers have something they can work with now-ish, versus
> pushing back and re-thinking what the Web should really do for
> storage.... and doing these in parallel (and again, letting the market
> decide who wins in the long run).
The problem with just going with SQL for now is that we'll be stuck with it.

/sdwilsh

sdwilsh.vcf

Mark Finkle

unread,
Jun 23, 2009, 3:31:32 PM6/23/09
to Shawn Wilsher, community-w...@lists.mozilla.org
----- "Shawn Wilsher" <sdw...@mozilla.com> wrote:
> The problem with just going with SQL for now is that we'll be stuck with it. /sdwilsh

E4X all over again :)

Arun Ranganathan

unread,
Jun 23, 2009, 5:51:26 PM6/23/09
to community-w...@lists.mozilla.org
Some thoughts from the Microsoft guys. Note that this is now a
newsgroup
(http://groups.google.com/group/mozilla.community.web-standards/topics)
so those that wish to subscribe to it *only via newsgroup* may do so.

Kevin Dangoor

unread,
Jun 23, 2009, 6:02:15 PM6/23/09
to ar...@mozilla.com, community-w...@lists.mozilla.org
On Tue, Jun 23, 2009 at 2:51 PM, Adrian Bateman wrote:
I'm not convinced a relational store is necessarily the best answer. We all seem to spend a lot of time writing object/relational mappings and maybe it would be best to take a higher level approach for the web platform. I note also that there was a suggestion the other day on the WebApps mailing list for a more XML focused approach to structured storage using XPath or XQuery as the query language. That is one kind of alternative that I have been considering given that it may address many of the use cases for structured data and already has a bunch of ratified and implemented standards to support it.

I'd imagine I'm not alone in thinking that dealing with more XML is not high on most JavaScript programmers' lists of things they want to do.

That said, JavaScripty versions of XPath and XQuery may not be unreasonable:

http://www.sitepen.com/blog/2008/07/16/jsonquery-data-querying-beyond-jsonpath/

Kevin

--
Kevin Dangoor

work: http://labs.mozilla.com/
email: k...@blazingthings.com
blog: http://www.BlueSkyOnMars.com

Adrian Bateman [MSFT]

unread,
Jun 23, 2009, 6:20:00 PM6/23/09
to
On Jun 23, 3:02 pm, Kevin Dangoor <kdang...@mozilla.com> wrote:
> On Tue, Jun 23, 2009 at 2:51 PM, Adrian Bateman wrote:
> > I'm not convinced a relational store is necessarily the best answer. We all
> > seem to spend a lot of time writing object/relational mappings and maybe it
> > would be best to take a higher level approach for the web platform. I note
> > also that there was a suggestion the other day on the WebApps mailing list
> > for a more XML focused approach to structured storage using XPath or XQuery
> > as the query language. That is one kind of alternative that I have been
> > considering given that it may address many of the use cases for structured
> > data and already has a bunch of ratified and implemented standards to
> > support it.
>
> I'd imagine I'm not alone in thinking that dealing with more XML is not high
> on most JavaScript programmers' lists of things they want to do.

I agree, although having a hierarchical store related to the XML
infoset might not involve actually seeing angle brackets and namespace
URIs. This is just one alternative, though, to a RDBMS.

> That said, JavaScripty versions of XPath and XQuery may not be unreasonable:
>

> http://www.sitepen.com/blog/2008/07/16/jsonquery-data-querying-beyond...

Absolutely.

Henri Sivonen

unread,
Jun 24, 2009, 8:12:08 AM6/24/09
to community-w...@lists.mozilla.org
On Jun 23, 2009, at 03:36, Arun Ranganathan wrote:

> Essentially, arguments made in favor of supporting the Web Storage
> API [4] ("SQL API") include the fact that WebKit has already
> implemented it [5],

I can think of four futures regarding SQL:

1) Mozilla implements something else. Other browsers follow. SQL
becomes a WebKit-specific oddity and fades into irrelevance.

2) Mozilla implements SQL compatibly with WebKit. SQL becomes the
cross-browser thing.

3) Mozilla implements SQL early on but incompatibly with WebKit.
Developers have to write code for multiple SQL engines.

4) Mozilla implements something else. SQL wins anyway leaving
Mozilla at a competitive disadvantage. Mozilla gets dragged into
implementing SQL when existing WebKit-targeted content that assumes
SQLite peculiarities already abounds.

It seems to me that not getting heavily involved with speccing and
implementing a 'Web SQL' makes sense only if it's realistic to execute
future #1. If future #3 or #4 happened, it would have been better to
work towards future #2 instead and get the Web SQL dialect specced to
Mozilla's satisfaction to ASAP. (Web SQL probably should be
implementable on top of SQLite.)

I don't know if it is realistic to execute future #1.

--
Henri Sivonen
hsiv...@iki.fi
http://hsivonen.iki.fi/


Nikunj Mehta

unread,
Jun 24, 2009, 12:31:21 PM6/24/09
to
On Jun 24, 5:12 am, Henri Sivonen <hsivo...@iki.fi> wrote:
> On Jun 23, 2009, at 03:36, Arun Ranganathan wrote:
>
> > Essentially, arguments made in favor of supporting the Web Storage  
> > API [4] ("SQL API") include the fact that WebKit has already  
> > implemented it [5],
>
> I can think of four futures regarding SQL:
>
>   1) Mozilla implements something else. Other browsers follow. SQL  
> becomes a WebKit-specific oddity and fades into irrelevance.
>
>   2) Mozilla implements SQL compatibly with WebKit. SQL becomes the  
> cross-browser thing.
>
>   3) Mozilla implements SQL early on but incompatibly with WebKit.  
> Developers have to write code for multiple SQL engines.
>
>   4) Mozilla implements something else. SQL wins anyway leaving  
> Mozilla at a competitive disadvantage. Mozilla gets dragged into  
> implementing SQL when existing WebKit-targeted content that assumes  
> SQLite peculiarities already abounds.
>

#4 really is the same as #1. In any case #1 may not be that different
from #3 in the long term because basing things on SQLite and offering
a hard-to-standardize vocabulary means that there will be divergence
sooner rather than later.

Perhaps the WebKit team see reason in what Mozilla is trying to do.
But maybe it is futile to expect that.

All I can say is that my respect for Mozilla doing the right thing
just went up after going through this discussion.

> It seems to me that not getting heavily involved with speccing and  
> implementing a 'Web SQL' makes sense only if it's realistic to execute  
> future #1. If future #3 or #4 happened, it would have been better to  
> work towards future #2 instead and get the Web SQL dialect specced to  
> Mozilla's satisfaction to ASAP. (Web SQL probably should be  
> implementable on top of SQLite.)
>
> I don't know if it is realistic to execute future #1.
>

It is entirely realistic to standardize something else and still find
a compromise on SQL support without necessarily creating a standard,
since it will be hard to create a public standard for SQL unless you
can silence all the criticism.

Shawn Wilsher

unread,
Jun 24, 2009, 12:59:46 PM6/24/09
to community-w...@lists.mozilla.org
On 6/24/09 5:12 AM, Henri Sivonen wrote:
> I don't know if it is realistic to execute future #1.
I'm not sure how realistic any of your futures are since you seem to be
completely ignoring IE (except maybe future 1). I don't think it's
going away.

/sdwilsh

Reply all
Reply to author
Forward
0 new messages