SQL Anywhere database backend

[Nathan Auch [Sybase]]

I'm a developer on the core engine team of the SQL Anywhere database
server (http://www.sybase.com/products/databasemanagement/sqlanywhere).
We're in the final stages of adding a SQL Anywhere database backend to
Django and would like to share our work with the community. What is the
process for getting our new database backend into the main Django
repository?

Thanks!

Nathan Auch
Sybase iAnywhere R&D

[Russell Keith-Magee]

On Tue, Apr 28, 2009 at 10:17 PM, Nathan Auch [Sybase] <na...@sybase.com> wrote:
>
> I'm a developer on the core engine team of the SQL Anywhere database
> server (http://www.sybase.com/products/databasemanagement/sqlanywhere).
> We're in the final stages of adding a SQL Anywhere database backend to
> Django and would like to share our work with the community. What is the
> process for getting our new database backend into the main Django
> repository?

Hi Nathan,

Thanks for going to the effort of supporting Django! The more backends
the merrier!

The process of getting your backend into Django trunk goes a little
something like this:

1) You open a new ticket describing the new feature (SQLAnywhere support)
2) You upload your backend as a patch against the Django trunk
3) You demonstrate over an extended period of time (6-12 months
minimum) that you are committed to keeping your backend up to date.
4) We accept the backend, and give the developers of that backend
limited commit access to keep the backend up to date.

Step 3 is the big hurdle here. It requires that you keep up to date
with any changes to the Django core - if we add a new function to the
backend interface, or a new feature to queries (e.g., aggregates), you
would be expected to port those changes in a timely fashion. It also
means that someone (and preferable multiple someones) from Sybase's
engineering team needs to get involved with the Django community,
engaging with design discussions that affect database backends, etc.
As a heads-up, there will be a few of these discussions in the near
future. Multiple DB support has been accepted as a Google Summer of
Code project - you can bet this will affect backends. Backend
modifications required to support column/key-value stores
(CouchDB/BigTable etc) may also crop up in the near future.

This is a deliberately lengthy process. When we add a database backend
to Django, we (the Django Core) are implicitly committing to keeping
that backend up to date. Our old MSSQL backend went stale simply
because the original contributor didn't maintain the code, and the
core team didn't have ready access (or, for that matter, the personal
inclination) to support the backend.

We don't want to see a repeat of this, so we will err on the side of
caution before adding a new backend. We're not going to add anything
until we know that we're going to get a couple of committers that are
both willing and able to maintain the backend for the long term. If
any licenses are going to be required to test your backend, we'll
probably hit you up for a few complimentary copies so the core team
can independently test your code.

We're also going to need to see some sort of demonstrated demand
before we add a new backend to trunk. Prior to the addition of the
Oracle backend, the Django mailing lists had regular requests for
Oracle support. We still have regular requests for MSSQL support.
Without wanting to squash egos, I can only find 2 requests in the
Django-users archive for SQL Anywhere support. I appreciate there is a
certain amount of "build it and they will come" involved here, but you
will need to convince us that they actually will come if we build it
:-)

The good news is that from a purely technical perspective, your code
doesn't need to be in the Django trunk in order for your userbase to
use Django. Django database backends can be referenced as an external
library. In your DATABASE_BACKEND setting, in addition to
'postgresql', 'sqlite3', and the other official backends, you can
provide a python module name, and Django will load and use that
backend.

Supporting your backend as an external module will be the preferred
approach for at least the short term. It's not a complete loss, though
- you can use this external support as part of the process of proving
you are committed to Django support. Download stats for that backend
could also be used to demonstrate the level of user demand.

As an interim measure, if you can show to us that you have a working
backend, we're happy to put a note in our documentation that directs
people to your external library and documentation.

Yours,
Russ Magee %-)

[Nathan Auch [Sybase]]

Hi Russ,

Thanks for taking the time to respond. We will finish off the SQL
Anywhere work in Django and then follow the procedure you outlined. It
sounds like starting with a externally maintained module and seeing
where that takes us is the way to go. I'll be reading this list going
forward so I'll make sure to keep my eyes peeled for anything that could
affect the SQL Anywhere db backend. Thanks for the heads up about
multi-database support and key-value stores.

We've done the port mainly to support the internal use of Review Board
(http://www.review-board.org/). What do you want to see to show that we
have a working back end? Should we just be trying to pass the included
unit tests or is there a specific set of Django applications with which
we should be testing?

Thanks!

Nathan Auch
Sybase iAnywhere R&D

[Alex Gaynor]

Passing the test suite is definitely a good place to start.  There's no other formal metric that exists, but testing your backend against large applications is definitely a good idea(you've said you're writing it to work with ReviewBoard so that's a good example).

Alex

--
"I disapprove of what you say, but I will defend to the death your right to say it." --Voltaire
"The people's good is the highest law."--Cicero

[Malcolm Tredinnick]

On Tue, 2009-04-28 at 14:38 -0400, Nathan Auch [Sybase] wrote:
> Hi Russ,
>
> Thanks for taking the time to respond. We will finish off the SQL
> Anywhere work in Django and then follow the procedure you outlined. It
> sounds like starting with a externally maintained module and seeing
> where that takes us is the way to go.

I would modify Russell's checklist very slightly... creating a patch in
Trac isn't going to be particularly helpful initially (I'm not quite
sure why he made that point #2 ahead of point #3 -- when the latter is
generally more important for an external feature -- but there may be
something I'm misunderstanding). This is only because an external module
very much is the best way to go in the initial phase. No modifications
to Django should be necessary for this and you don't need to live in the
django.db.backends.* namespace (in fact, avoid it as an external
module), as Russell noted.

A patch/ticket only really makes things a bit confusing for people
looking for this backend, as they won't know whether to use the patch or
some external project. By being purely an external project for a while,
there's no confusion. Anybody asking about an SQL Anywhere backend can
be pointed to the project website, download the latest version from
there, report bugs there and so on (it doesn't make sense for us to
handle bug reports for this backend, so, again, we want to make that
clear to potential users). There's also the short- to medium-term
problem of not really being any way to directly resolve such a ticket
and tickets should ideally have a way to be closed by somebody doing
some work on them ("waiting a while to see if it will be incorporated"
isn't an active strategy)

Obviously there are many ways to make this available and maybe you
already plan to place it under sybase.com/developer/opensource/
(although requiring membership before download is going to restrict the
number of people using this). One comparable case study here that might
be of interest, particularly for something that will hopefully be
updated regularly based on feedback, is IBM's db2 wrapper for Python.
They created a Google code project for it and then would transfer any
reported bugs back to their internal system for resolution. It's pretty
clunky, but partly required because they had a very restrictive policy
in place for that particular internal division regarding external
contributions (which appears to have been drastically relaxed and
improved lately, thank goodness). The advantage of what they did,
though, was it provided a subversion repository and publicly available
bug reporting path that was already familiar and available to a few tens
of thousands potential users and contributers.

Finally, one thing to consider before an initial release, given that
your ultimate goal might be inclusion in Django: the license. Django's
core and contrib applications are all BSD license, so, at a minimum
you'd have to be compatible with that before inclusion. This becomes a
consideration once you make an open source release because there's the
chance you'll be accepting contributions from the broader community. So
a later license change is very difficult (logistically) unless you go
for copyright assignment as part of contributing (which will restrict
the contributions). If I'm appearing to be teaching you to suck eggs
here, that's not the intention. Maybe this is all very familiar and well
planned on your side. I can't read any of the license information or
usage policies on existing stuff at the Sybase site without creating an
account and I'm not going to do that at the moment. A bit of forward
planning at this point (or working out if your corporate policies
conflict irrevocably with Django's ) might smooth things out a bit going
forwards. After all, a more restrictive license than the BSD license is
entirely possible, providing your end-game isn't inclusion in Django
core. We support external database modules as first-class citizens (with
the proviso that they're responsible for keeping up with any necessary
changes we have to make. The backend dataabaseAPI isn't part of our
forwards- or backwards-compatibility policy), so that's certainly a
supportable strategy as well.

Regards,
Malcolm

[Russell Keith-Magee]

On Wed, Apr 29, 2009 at 3:13 AM, Malcolm Tredinnick
<mal...@pointy-stick.com> wrote:
>
> On Tue, 2009-04-28 at 14:38 -0400, Nathan Auch [Sybase] wrote:
>> Hi Russ,
>>
>> Thanks for taking the time to respond. We will finish off the SQL
>> Anywhere work in Django and then follow the procedure you outlined. It
>> sounds like starting with a externally maintained module and seeing
>> where that takes us is the way to go.
>
> I would modify Russell's checklist very slightly... creating a patch in
> Trac isn't going to be particularly helpful initially (I'm not quite
> sure why he made that point #2 ahead of point #3 -- when the latter is
> generally more important for an external feature -- but there may be
> something I'm misunderstanding).

My reasoning here was that creating and maintaining a patch is one way
to demonstrate that someone is committed to Django support. You are
correct that an external (e.g., Google code) project would probably be
preferable, and is just as good at demonstrating long-term commitment.

Russ %-)