New SQL API's for moz2
Shawn Wilsher
wrapper and create something to replace it for JS and C++. The
current API provides lots of ways for consumers to shoot themselves in
the foot if they aren't careful. While we've made some big strides in
1.9 to rectify this, remaining changes would break compatibility with
past versions of the API. We'd not only have to fix our own code, but
also inconvenience any add-on authors unfortunate enough to use the
API. While it's true that the interfaces are unfrozen, people use
them as if they are frozen. By making a new API, and leaving the old
one in for a release cycle, we give them an opportunity to change over
gradually (at least for our own code internally - I'm not sure if that
matters with add-on developers because I'm sure they'll complain
regardless).
Since we'd be breaking existing consumers pretty badly, I'm proposing
that we create a new interface for JS consumers written in a JS
friendly way that can be Components.utils.import'd into scripts that
want to use it (yes, this means it's not XPCOM). Ideally, I think if
we modeled it after SQLAlchemy (see also Bug 394732) that it'd be very
useful to developers. SQLAlchemy provides a good abstraction layer to
the database, making it easy to reflect database state/data into
objects. While using this abstraction, it makes it quite difficult to
shoot yourself in the foot. However, it also exposes basic
functionality for those consumers who do not want to use the
abstraction for perf or other reasons. In other words, this API gives
the best of both worlds.
In addition to the JS-friendly API, we'd need to make a C++-friendly
API for our C++ consumers. While I think we'll have much better code
when we move to garbage collection (I'm pretty sure this is happening,
so no more ADDREF and RELEASE), exceptions (this isn't so clear to me
if it's happening), and outparam removal (I think this is the least
likely thing to make it) when using XPCOM, it's still not very C+
+ish. Regardless, we'd still have the issue with consumers can shoot
themselves in the foot. We could similarly model this after
SQLAlchemy's API, but things would be quite different for many
situations because of language differences. I haven't put a whole lot
of thought into the API design of this, but I'm fairly certain we can
come up with something much better than what we currently have.
While learning the API for Intel's Threading Building Blocks (TBB),
I've come to the conclusion that a good API can make doing hard tasks
trivial (TBB makes concurrency easy in many situations). I don't know
if most people would consider working with databases a hard task, but
given an API that you can hurt yourself easily with, it can become
hard very quick. Making a good API can make working with databases
fun again!
I haven't really stated what the issues are with the API so far, so
I'm going to layout the ones that are big in my mind:
- dynamic parameters are bound with zero-based indices, while sqlite
uses one-based indices for it's queries.
our JS wrapper for mozIStorageConnections, while useful, is cumbersome
to create.
- we do not support the binding of named parameters (we do with the JS
wrapper, but see previous point). I'm aware that sqlite also does not
provide this, but I've seen bugs fly by where things were being bound
to the wrong parameter because it was being referenced with the wrong
number. It's hard to get the wrong name past a reviewer. Yes, this
makes the first point obsolete. (see also Bug XXX)
- we should only allow the binding of parameters by name (or
alternatively, with the SQLAlchemy API, pass in a array of the
parameters to bind). This also would make the first point obsolete.
-In both C++ and JS, the API just feels cumbersome to use, and it's
not really like any other DB abstraction layer out there. Developers
often like abstractions, and we really don't provide that.
These are my thoughts, and I'm happy to argue about them. At the very
least, I feel that we should do the JS stuff, but I think we could get
some big wins by doing the C++ end of things as well.
Shawn Wilsher
> wrapper, but see previous point). I'm aware that sqlite also does not
> provide this, but I've seen bugs fly by where things were being bound
> to the wrong parameter because it was being referenced with the wrong
> number. It's hard to get the wrong name past a reviewer. Yes, this
> makes the first point obsolete. (see also Bug XXX)
Whoops - that is referencing Bug 387945.
Cheers,
Shawn
Myk Melez
> I haven't really stated what the issues are with the API so far, so
> I'm going to layout the ones that are big in my mind:
Revisiting that API seems like a great idea, and this is a good list of
issues, although none cries out to me for wholesale redesign.
Nevertheless, I'm very interested in SQLAlchemy-like APIs, but perhaps a
dual effort would make sense here, with some work done to smooth the
ruffles of the current API (switching to one-based indices, making it
easier to create a connection wrapper, and perhaps wrapping the current
functionality in a simpler FUEL-like interface) while simultaneously
working on a next-gen SQLAlchemy-like ORM API.
-myk
Boris Zbarsky
> In addition to the JS-friendly API, we'd need to make a C++-friendly
> API for our C++ consumers.
Shawn, what's the plan for other languages (e.g. Python)?
I guess that's a general Moz2 question. Is there such a plan?
-Boris
Shawn Wilsher
That's actually a really good point that I hadn't though about. I
suppose that not providing some XPCOM interface would break those
languages, so not having one would be a bad idea. I suppose we could
do the JS API stuff with a new XPCOM interface with lots of class info
fun, but that still doesn't provide a nice C++ api.
Cheers,
Shawn
Shawn Wilsher
thread is that it is not clear that providing a C++ api would actually
benefit consumers. It's also been pointed out that code examples
would probably help to express why these new API's would be a good
thing.
At this point, (due to bz's comments about other languages), I don't
think we should obsolete the mozStorage API's. Because of that, I
think that there will be some major breakage with them for moz2. I
think we could get away with adding more C++ only helper methods (we
already have some) and make the experience better for consumers.
I'll put some code samples up some time soon to demonstrate some of
the benefits of these proposed changes.
Cheers,
Shawn