Moving database backends out of the core

[Michael Manfre]

Full disclosure, I maintain django-mssql and am biased toward having all database backends treated as if they were 3rd party backends.

The current discussion about "Switch to database-level autocommit" (http://bit.ly/ZlVERI) reminded me of a past discussion regarding moving the database backends out of the core. I don't remember exactly where I heard the discussion, but I'd like to revive it. My proposal is to move all database backends out of Django, except for an sqlite reference implementation.

Django has already done a similar transition with django.contrib.localflavor, which provides us with some of the basic answers about whether this sort of thing is possible and a rough guide about how to make it happen. 

Benefits

======

  - One of the biggest benefits of this change would be that database backends will no longer be bound to Django's release cycle and Django would not need to make a dot release for a database specific issue. 

  - This will result in Django having a more flexible/robust database API. Any of the existing complexity imposed on the core by a specific database backend could be abstracted out to the API. 

  - In some instances this will simplify the core's code by pushing the complexity in to the specific database backend.

Concerns

=======

These are the current issues/concerns faced by 3rd party database backends.

  - All backends would need to support multiple versions of Django, or at least properly document which versions of Django the backend works with

  - People will need to (pip) install the database backend that they want for each project

  - Less popular database backends may get left behind

  - Each project will have its own tests and will also need to run against the entire Django test suite

Migration Path:

===========

I believe this can be accomplished without needing a multi-version deprecation cycle because unlike localflavors, there should be very little, if any, user code that imports a specific database backend directly. The only instances I can think of are the DATABSE_ENGINE path and non standard features that I've added to django-mssql; custom field types, non-standard aggregates, etc. If a multi-version deprecation cycle is needed, it will be possible to fix the import paths as localflavors.

Django 1.6 or 1.7

  - Create separate github project for each database backend and move code

  - Abstract out any backend specific code in the ORM by extending the database API

  - Move backend specific tests to appropriate project

Regards,

Michael Manfre

[Florian Apolloner]

Hi,

On Tuesday, March 5, 2013 8:35:07 PM UTC+1, Michael Manfre wrote:

The current discussion about "Switch to database-level autocommit" (http://bit.ly/ZlVERI) reminded me of a past discussion regarding moving the database backends out of the core. I don't remember exactly where I heard the discussion, but I'd like to revive it. My proposal is to move all database backends out of Django, except for an sqlite reference implementation.

While I agree that moving database adapters out of core has some merit, I don't think that having sqlite as a reference implementation is a good idea: For one some features are somewhat hacky in sqlite (and people tend to copy from reference implementations, so it should be as clean as possible) and it's lack of data validation makes it imo a nogo. The reference implementation should imo also have strong support for GIS, which is somewhat okay on sqlite but quite hard to install. So if we were to do that I'd either vote for postgres or supporting postgres and sqlite inside of core (the later solely for fast tests).

Cheers,
Florian

[Shai Berger]

Hi,

On Tuesday 05 March 2013, Michael Manfre wrote:
> Full disclosure, I maintain django-mssql and am biased toward having all
> database backends treated as if they were 3rd party backends.
>

In recent years, I have been the main contributor to South's MSSQL and Oracle
backends. I am biased towards having MSSQL treated as an equal to the database
systems supported in core, but also towards support of backend-specific low-
level code in user apps. Also, I am a Linux user -- I use django-pyodbc for my
work on South, not django-mssql.

>
> Benefits
> ======
> - One of the biggest benefits of this change would be that database
> backends will no longer be bound to Django's release cycle and Django would
> not need to make a dot release for a database specific issue.
>

I think you are missing something crucial: Django depends on its commonly-used
backends. A critical problem with the Postgresql or MySql backend is a
critical problem for the Django project, regardless of separation to
repositories. And where the dependency is weaker, being in core was not a
guarantee for fixes -- Django carried significant Oracle problems for quite
long, and they were just not deemed crucial.

So, IMO, you can't escape the need to have a set of "blessed" backends, which
core is committed to their correct functioning, and thus need to be, at least
to some degree, in sync with the Django main project.

> - This will result in Django having a more flexible/robust database API.
> Any of the existing complexity imposed on the core by a specific database
> backend could be abstracted out to the API.
>
> - In some instances this will simplify the core's code by pushing the
> complexity in to the specific database backend.
>

These benefits are, IMO, orthogonal to the separation you suggest. I can't
speak for core, but I doubt that a proposed change to make the database API
more robust and flexible will be rejected just because it is originally
motivated by a 3rd-party, rather than core, backend.

> Concerns
> =======

>
> - Less popular database backends may get left behind
>

I am worried that the result of such change will not be that MSSQL is treated
as well as Postgres is today, but that Oracle is treated as bad as MSSQL is
today.

>
> Migration Path:
> ===========
>
> I believe this can be accomplished without needing a multi-version
> deprecation cycle because unlike localflavors, there should be very little,
> if any, user code that imports a specific database backend directly.

South certainly does, I'm sure it's not the only one.

Shai.

[Florian Apolloner]

Hi Shai,

On Tuesday, March 5, 2013 10:32:29 PM UTC+1, Shai Berger wrote:

In recent years, I have been the main contributor to South's MSSQL and Oracle
backends. I am biased towards having MSSQL treated as an equal to the database
systems supported in core, but also towards support of backend-specific low-
level code in user apps. Also, I am a Linux user -- I use django-pyodbc for my
work on South, not django-mssql.

I am obviously biased against postgres as my previous post indicated, but regardless of that I think that MSSQL should stay outside of core. No core-developer I know actually uses Windows as base OS and to my knowledge no one uses MSSQL. This would put the expected support for South below what we have for Oracle currently.

And where the dependency is weaker, being in core was not a
guarantee for fixes -- Django carried significant Oracle problems for quite
long, and they were just not deemed crucial.

Now that we test on Oracle once again I am committed to have a full passing testsuite with 1.6 (whatever that means, since I don't use Oracle on a daily base I can't say what a running testsuite actually means for a project).

I am worried that the result of such change will not be that MSSQL is treated
as well as Postgres is today, but that Oracle is treated as bad as MSSQL is
today.

That might as well be true, if Oracle is outside of core I personally wouldn't put much effort in fixing stuff there (this obviously might change depending on whether I use Oracle at work or not).

Cheers,
Florian

[Florian Apolloner]

On Tuesday, March 5, 2013 11:01:01 PM UTC+1, Florian Apolloner wrote:

I am obviously biased against postgres as my previous post indicated, but regardless of that I think that MSSQL should stay outside of core. No core-developer I know actually uses Windows as base OS and to my knowledge no one uses MSSQL. This would put the expected support for South below what we have for Oracle currently.

s/South/MSSQL sry

[Jonathan Slenders]

Le mardi 5 mars 2013 20:35:07 UTC+1, Michael Manfre a écrit :

Full disclosure, I maintain django-mssql and am biased toward having all database backends treated as if they were 3rd party backends.

The question is, how monolithic do we want to have Django?

If we would move database backends out of core. Why not move everything in django.db away from core? There are other ORMs for Django. And the same is true for the templating system.

For me, personally, it's really a big +1 for splitting Django up in several logical parts, and for removing the global state. (So that we could create a Django *instance*, and add a postgres *instance* to django ourself.)

I want to to see something like this:

d = Django(

        database_backend=Postgres(postgres_settings),

        urls=our_url_patters,

        templates=TemplateSystem(template_settings),

        midldeware=...,

       )

No disadvantages I think. But it's not an easy migration path.

Cheers,

Jonathan

[Andre Terra]

Hi Florian,

On Tue, Mar 5, 2013 at 7:01 PM, Florian Apolloner <f.apo...@gmail.com> wrote:

I am obviously biased against postgres as my previous post indicated, but regardless of that I think that MSSQL should stay outside of core. No core-developer I know actually uses Windows as base OS and to my knowledge no one uses MSSQL. This would put the expected support for South below what we have for Oracle currently.

If I may weigh in on the matter as an outsider, if we consider "The Django Project" as a "business", insofar as it aims to have as many users and be as ubiquitous as possible, there's considerable value in having MSSQL included in core. If it were up to me, I would develop on Linux using postgres and host everything on Heroku, but at work I'm restricted to corporate rules, MS SQL or Oracle, and Windows. 

I have a strong feeling that by embracing Windows as a first-class platform for developing and running Django, we're enabling current and future users to easily integrate the framework in existent business infrastructure, which can only be seen as a good thing for the project.

Cheers,

AT

[Aymeric Augustin]

On 5 mars 2013, at 20:35, Michael Manfre <mma...@gmail.com> wrote:

  - Less popular database backends may get left behind

To be honest, I wouldn't make any effort to keep the MySQL and Oracle backends at feature-parity with PostgreSQL and SQLite if they weren't in core.

-- 

Aymeric.

[Florian Apolloner]

Hi Andre,

On Tuesday, March 5, 2013 11:39:29 PM UTC+1, Andre Terra wrote:

but at work I'm restricted to corporate rules, MS SQL or Oracle, and Windows. 

Right, that's probably one of the reasons why Oracle is in core (aside from the fact that we were completely monolithic at that time).

I have a strong feeling that by embracing Windows as a first-class platform for developing and running Django, we're enabling current and future users to easily integrate the framework in existent business infrastructure, which can only be seen as a good thing for the project.

You are obviously right that it would help, but my point about not having any core developers in that area still stands and makes it hard to support something like that. FWIW, I think that supporting Windows would be far easier than supporting MSSQL. Looking at the bug tracker we have a few bugs for windows (some are false positives): https://code.djangoproject.com/query?status=assigned&status=new&description=~windows&col=id&col=summary&col=status&col=owner&col=type&col=component&order=priority

Cheers,
Florian

[Michael Manfre]

On Tuesday, March 5, 2013 3:31:59 PM UTC-5, Florian Apolloner wrote:

Hi,

While I agree that moving database adapters out of core has some merit, I don't think that having sqlite as a reference implementation is a good idea: For one some features are somewhat hacky in sqlite (and people tend to copy from reference implementations, so it should be as clean as possible)

I think I made a mistake in using the word "reference". "Included" would be a more accurate definition of its intended status. Ideally, no database backends would be in the core, but I realize that is not entirely practical because of the need to run tests. Sqlite also provides the lowest barrier for new users to get through the tutorial.

From my experience of maintaining django-mssql, I usually have to reference all of the backends depending on the issue I'm working on. I start by looking at whichever database provides the closest behavioral match and then compare it to the others. Every database is different, so it's not like copy & paste is going to happen. Database backends are also not something a large number of people will run out and create. I imagine that even with Postgres, mysql, and oracle in separate projects they will still be maintained by a collection of developers that will most likely include core devs.

 

and it's lack of data validation makes it imo a nogo.

The lack of data validation is definitely a nogo for production sites, but imo sqlite in production is also a nogo.

The reference implementation should imo also have strong support for GIS, which is somewhat okay on sqlite but quite hard to install. So if we were to do that I'd either vote for postgres or supporting postgres and sqlite inside of core (the later solely for fast tests).

. I've never tried to install GIS for sqlite, but is the difficulty due to lack of documentation or just sheer number of steps?

Regards,

Michael

[Michael Manfre]

On Tuesday, March 5, 2013 5:01:01 PM UTC-5, Florian Apolloner wrote:

Hi Shai,

On Tuesday, March 5, 2013 10:32:29 PM UTC+1, Shai Berger wrote:

In recent years, I have been the main contributor to South's MSSQL and Oracle
backends. I am biased towards having MSSQL treated as an equal to the database
systems supported in core, but also towards support of backend-specific low-
level code in user apps. Also, I am a Linux user -- I use django-pyodbc for my
work on South, not django-mssql.

I am obviously biased against postgres as my previous post indicated, but regardless of that I think that MSSQL should stay outside of core. No core-developer I know actually uses Windows as base OS and to my knowledge no one uses MSSQL. This would put the expected support for South below what we have for Oracle currently.

I agree that MSSQL should stay out of the core. I believe that adding any more database in to the core would be a large step in the wrong direction.

 

And where the dependency is weaker, being in core was not a
guarantee for fixes -- Django carried significant Oracle problems for quite
long, and they were just not deemed crucial.

Now that we test on Oracle once again I am committed to have a full passing testsuite with 1.6 (whatever that means, since I don't use Oracle on a daily base I can't say what a running testsuite actually means for a project).

Is it not possible to maintain the same standard with it as a separate project? Automated tests could still easily be accomplished and the only real difference about the backend working with Django is that the releases of the projects would not be a direct blocker for each other. 3rd party backends are only able to provide eventual support. Sometimes the new behaviors are in place ahead of the Django release, sometimes it lands shortly after. A huge benefit is that if there is a bug/change in the postgres backend, it doesn't require another Django release.

 

I am worried that the result of such change will not be that MSSQL is treated
as well as Postgres is today, but that Oracle is treated as bad as MSSQL is
today.

That might as well be true, if Oracle is outside of core I personally wouldn't put much effort in fixing stuff there (this obviously might change depending on whether I use Oracle at work or not). 

Having a backend outside of core might encourage others to step up and maintain it, especially if they depend on it. I'm sure many currently have the mentality of "the cores will take care of it, so I don't have to". 

It sounds as if Oracle is already not really a supported database and will only have a passing test suite. As you stated, "whatever that means, since I don't use Oracle on a daily base I can't say what a running testsuite actually means for a project".

Regards,

Michael

[Michael Manfre]

Hi Andre,

On Tuesday, March 5, 2013 5:39:29 PM UTC-5, Andre Terra wrote:

If I may weigh in on the matter as an outsider, if we consider "The Django Project" as a "business", insofar as it aims to have as many users and be as ubiquitous as possible, there's considerable value in having MSSQL included in core. If it were up to me, I would develop on Linux using postgres and host everything on Heroku, but at work I'm restricted to corporate rules, MS SQL or Oracle, and Windows. 

Business rules dictate Windows and MSSQL for me as well. Despite the fact that including MSSQL in the core would mean it has a better bare minimum amount of support, I'm strongly opposed to it being included. The core already has too many databases included.

 

I have a strong feeling that by embracing Windows as a first-class platform for developing and running Django, we're enabling current and future users to easily integrate the framework in existent business infrastructure, which can only be seen as a good thing for the project.

Agreed. Windows should be treated as a first-class OS, but I don't think any of the core devs use Windows as their primary OS. I only know of one core dev that even runs the Django testsuite on Windows from time to time.

Regards,

Michael

[Florian Apolloner]

Hi,

On Wednesday, March 6, 2013 3:32:45 PM UTC+1, Michael Manfre wrote: 

The lack of data validation is definitely a nogo for production sites, but imo sqlite in production is also a nogo.

Right, but shipping Django with a non production db might send interesting signals to endusers ;)

The reference implementation should imo also have strong support for GIS, which is somewhat okay on sqlite but quite hard to install. So if we were to do that I'd either vote for postgres or supporting postgres and sqlite inside of core (the later solely for fast tests).

. I've never tried to install GIS for sqlite, but is the difficulty due to lack of documentation or just sheer number of steps?

Well it's not to bad, we did document it after all, but it usually requires recompiling sqlite and pysqlite (most importantly you can't pull it from your distros repos).

Regards,
Florian

[Michael Manfre]

On Wed, Mar 6, 2013 at 12:47 PM, Florian Apolloner <f.apo...@gmail.com> wrote:

On Wednesday, March 6, 2013 3:32:45 PM UTC+1, Michael Manfre wrote: 

The lack of data validation is definitely a nogo for production sites, but imo sqlite in production is also a nogo.

Right, but shipping Django with a non production db might send interesting signals to endusers ;)

It might, but it's all about marketing. "Django, now 10% less monolithic!" 

Regards,

Michael

[Jacob Kaplan-Moss]

On Wed, Mar 6, 2013 at 3:22 AM, Marc Tamlyn <marc....@gmail.com> wrote:
> I don't know why Oracle is included and MSSQL is not [...]

It's essentially because a group of people made a concerted effort to
get the Oracle backend up to snuff (around the 1.0 timeline, I think?)
and then committed to maintaining it. Lately the original people who'd
made that commitment have dropped off a bit, but there's always been
enough attention to keep it up to date.

If someone -- preferably a group -- stepped up and committed to
keeping a MSSQL backend up-to-date in core, I'd be +1 on merging it
and giving those people commit access to keep it up to date.

[This is me speaking personally, not as a BDFL. It'd have to be a
discussion, not just my fiat.]

Jacob

[Alex Ogier]

It's worth mentioning that Django appears in the Python Tools for Visual Studio (it's one of the default project templates)[1]. There's recent introductory material on deploying Django applications to Windows Azure[2] and Windows Server 2008[3], though it seems the older material at least explicitly recommends MySQL and not SQL Server. I don't know if this is the kind of thing that leads to demand among Windows folks for Windows support for Django, and if any significant fraction of them want to use MSSQL with Django, but Django is certainly becoming mainstream for them.

[1]: http://pytools.codeplex.com/

[2]: http://www.windowsazure.com/en-us/develop/python/tutorials/django-with-visual-studio/

[3]: http://www.windowsazure.com/en-us/develop/python/tutorials/web-app-with-django-and-mysql/

Best,

Alex Ogier

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.

[Erik Romijn]

Hi all,

It seems to me we are mixing a few different discussions here:
1. Should Django core have as few database backends as possible?
1a. If yes, which ones should stay in Django core?
2. What should we do, if anything, with the current situation where
it seems difficult to guarantee the quality of the Oracle backend?
3. Should a MSSQL Django backend be included in Django core?

Regarding question 1, I feel we should keep a fair set of backends,
like the current one, included in Django core.

Yes, we could move them to separate projects. All the same arguments
apply here as why it's not necessarily good for a project to end up
in contrib. However, in theory any component of Django could be moved
out of core and be independently maintained. The whole ORM, the
templating system, the CSRF protection, and so on. I haven't seen any
reason why we should move the MySQL backend out, but not the templating
system.

Besides that, the close integration and development of all these parts
is exactly the reason I like Django. When I download a version of Django,
or upgrade, I know that all components in there will work well together.
I can run my site on any of the supported databases, and they will all
work together with the provided admin. I can build forms on the models
I build with the ORM. The forms will nicely fit in with the templates.

When I started using Django, I had looked at several alternatives which
were more modular. However, they required me to make tons of choices,
each involving numerous consequences. If I picked ORM A, I could have
databases X, Y and Z, but no admin. For the admin I had to use ORM B,
but that did not support database Z or many to many fields. And so on.
When I tried Django, it was a relief to see I could download a single
package, and everything would just work together.

If we move database backends out of core, my big concern is that
the same will happen there. This is fine for many components we use
in many Django projects - but not for something as fundamental as the
database backend. When the admin doesn't work on any of the officially
supported backends, that should be a release blocker for Django itself,
because I think it will be a blocker for the users.

Regarding question 2, Oracle support, I think a great step forward has
been made with the addition of Oracle to the Django continuous
integration setup. That alone should help us improve the consistent
quality of Oracle support.

cheers,
Erik

[Alex Ogier]

Here's something I've been thinking about:

As a rule, assuming that backends are not bug-riddled and do not have needlessly shoddy performance, nearly all of the value of an ORM is in the set of frontend features it exposes to application developers. Given a certain frontend API, then the only value in iterating the backend independently from the frontend is to fix uncaught bugs, or to improve performance, there's no value in adding features.

Also as a rule, adding features to a frontend independently from the backend has little value. Without an implementation in whatever backend you are on, it only serves as a guide for future backend development, so you might as well push for backends to catch up in the same release cycle. (Or the feature doesn't require backend support, but then it doesn't hurt to reversion the backend anyways).

So basically, there's not much value in independently versioning and maintaining frontends and backends to the ORM. This is in contrast to, say, localflavor, where an improvement in Mongolian localization can immediately help every Mongolian website in every Django version. So this primary motivation for independent maintenance is not a factor in the ORM. Hence I think the core team should include as many backends as it can handle in core (where "handle" means test that they function properly for each release). Oracle had been slipping, but from what I understand it's now in the CI server and back to passing most tests.

So I see no reason to split off backends unless the maintenance burden is such that they are chronically broken in every Django release. And every reason to add MSSQL unless the maintenance burden is too high.

Best,

Alex Ogier

[Russell Keith-Magee]

On Fri, Mar 8, 2013 at 7:57 AM, Alex Ogier <alex....@gmail.com> wrote:

Here's something I've been thinking about:

As a rule, assuming that backends are not bug-riddled and do not have needlessly shoddy performance, nearly all of the value of an ORM is in the set of frontend features it exposes to application developers. Given a certain frontend API, then the only value in iterating the backend independently from the frontend is to fix uncaught bugs, or to improve performance, there's no value in adding features.

Also as a rule, adding features to a frontend independently from the backend has little value. Without an implementation in whatever backend you are on, it only serves as a guide for future backend development, so you might as well push for backends to catch up in the same release cycle. (Or the feature doesn't require backend support, but then it doesn't hurt to reversion the backend anyways).

So basically, there's not much value in independently versioning and maintaining frontends and backends to the ORM. This is in contrast to, say, localflavor, where an improvement in Mongolian localization can immediately help every Mongolian website in every Django version. So this primary motivation for independent maintenance is not a factor in the ORM. Hence I think the core team should include as many backends as it can handle in core (where "handle" means test that they function properly for each release). Oracle had been slipping, but from what I understand it's now in the CI server and back to passing most tests.

So I see no reason to split off backends unless the maintenance burden is such that they are chronically broken in every Django release. And every reason to add MSSQL unless the maintenance burden is too high.

And that last clause is the catch.

I completely agree with Jacob's analysis of the status quo, and I agree largely with his position on having MSSQL in the core. 

I'd have no problem seeing MSSQL in the core - it's at least as high profile as Oracle, and although the demand for a MSSQL backend is restricted to a particular subset of the community, there is at least *some* demand for it to exist. Having MSSQL in core would allow us to hold our head high and say we support Microsoft platforms. Microsoft is even a member of the DSF at this point, so they're at least notionally interested in Django as a platform. 

The maintenance burden is the problem. Historically, we've already had to *remove* a MSSQL backend because of bit rot (see [1]) - this isn't a pattern I want to repeat in the future. I don't want to add a backend to core unless I'm *certain* that it will see long term maintenance.

[1] https://github.com/django/django/commit/c30a050e41

There is, however, a possible middle ground, following the example set by Flask: we introduce to Django a list of "officially recognised" extensions. These extensions are still maintained as external projects, but the core team promise to include these projects as part of the testing regimen for a release. If the MSSQL backend were recognised like this, we would run the full Django test suite against the MSSQL backend, and if that testing revealed a bug in the test suite which was identified as being due to a change in Django's codebase, that bug would be a release blocker (Bugs in the backend itself would still be the backend's responsibility, and not release blocking on Django)

Looking outside database backends, this could also apply to high-profile projects like haystack, django-registration, and so on. This would also serve to highlight projects in our community that are 'defacto standards', or good examples of reusability, without needing to expand the core team or the size of contrib, and show that the core project is explicitly interested in the broader ecosystem.

Yours,

Russ Magee %-)

[Michael Manfre]

On Thu, Mar 7, 2013 at 8:13 PM, Russell Keith-Magee <rus...@keith-magee.com> wrote:

I completely agree with Jacob's analysis of the status quo, and I agree largely with his position on having MSSQL in the core. 

I'd have no problem seeing MSSQL in the core - it's at least as high profile as Oracle, and although the demand for a MSSQL backend is restricted to a particular subset of the community, there is at least *some* demand for it to exist. Having MSSQL in core would allow us to hold our head high and say we support Microsoft platforms. Microsoft is even a member of the DSF at this point, so they're at least notionally interested in Django as a platform. 

/slightly off topic

I didn't realize they were a DSF member. I have been wanting to set up Jenkins to run the test suite on Windows with the ultimate goal of getting its results added to ci.djangoproject.com. Knowing that Microsoft is a DSF member will probably help gain access to a VM on Azure to make that a reality. I've so far managed to get an introduction to a PM on one of the Azure teams to help get a more direct answer to some Django, Windows, and SQL problems that have been raised to my attention.

/back on topic

The maintenance burden is the problem. Historically, we've already had to *remove* a MSSQL backend because of bit rot (see [1]) - this isn't a pattern I want to repeat in the future. I don't want to add a backend to core unless I'm *certain* that it will see long term maintenance.

[1] https://github.com/django/django/commit/c30a050e41

The demand for MSSQL would probably increase if it were included in core because companies could be more confident about its ongoing support.

There are three options for using MSSQL with Django; django-pyodbc, django-mssql, or write your own. The third option is not really a viable option for most users.

django-pyodbc is cross platform, but not updated very often and has limited stored procedure support. The official repo on Google code was last updated in 2011, but there is a github fork with its last commit 8 months ago.

django-mssql is actively maintained and will be for at least the next few years because it's used for my employer's production site that is critical to business operations. The backend also supports stored procedures almost to the same extent as a Windows business application. The only real downside to the backend is that it only works on Windows, reducing its usefulness to a much smaller, but growing, minority of Django users.

If Django wants to "hold [its] head high and support Microsoft platforms", then I'm willing and capable of helping out with MSSQL and Windows support. The only real caveat is that my employer needs me to maintain a certain level of stored procedure functionality, so if MSSQL were included in the core by using django-pyodbc instead of django-mssql, I would only be able to commit personal time toward maintaining it, instead of also being able to set aside business hours.

There is, however, a possible middle ground, following the example set by Flask: we introduce to Django a list of "officially recognised" extensions. These extensions are still maintained as external projects, but the core team promise to include these projects as part of the testing regimen for a release. If the MSSQL backend were recognised like this, we would run the full Django test suite against the MSSQL backend, and if that testing revealed a bug in the test suite which was identified as being due to a change in Django's codebase, that bug would be a release blocker (Bugs in the backend itself would still be the backend's responsibility, and not release blocking on Django)

This is sort of the idea I had in mind for backends while I was typing up this proposal. You articulated it better than I did. A main problem I now see with it is if the ownership of a bug cannot be mutually agreed upon between Django and the extension, then this could cause doubt about what it really means to be "officially recognized".

Regards,

Michael Manfre

Since DjangoCon, I now hear the email author's voice when reading some of these emails.

[VernonCole]

Huh??  I am running sqlite with GIS on my laptop right now.  It wasn't THAT hard to install.  Yes, the documentation could use some cleanup, but it got me through the tutorial okay and gives me a platform to learn GIS on.
  I really support the idea of sqlite and postgres in the core and moving everything else outside. 
(Not that my support counts for much.)

[Claude Paroz]

As for me, wearing my FOSS-only developper hat, I'd suggest we only include backends in core  for products having an OSI-approved license. And yes, the Oracle backend is already problematic at this regard. That would mean I'm -0 to -1 for including any other db backend (or any other component) based on a proprietary product.

Cheers,
Claude

[VernonCole]

I would love to volunteer for the project of MSSQL support.  It happens that django-mssql uses a fork of my adodbapi v2.1 code. I pulled those changes back into the main branch of adodbapi (and did a lot of useful refactoring thanks to the good work done on the fork) at version 2.3.  Apparently the announcement of that did not get to the right people, since the peek I just took shows the forked code still in use. 

The two major obstacles I can see to MSSQL support (either as part of the core or as a supported add-on) are:
1) lack of an MSSQL server and test bed which the maintainers can use on a regular basis.
2) django-mssql is not Python 3 ready.

The first of those obstacles has already been addressed.

The second is fairly easy.  I suspect (not having examined the code) that most of the Python3 problems will be found in the old fork of adodbapi.  If we simply change django-mssql to call the current version of adodbapi all of that goes away.  The current version is already present on their Python library, I am sure, because it ships as part of pywin32, which is also a prerequisite for the V2.1 code. Yes, all of pywin32, including adodbapi, runs on Python 3 -- has done for years.

Would there be a "market" for a supported version of django which runs MSSQL?  Does the bear live in the woods??? Take, for example, my present employer. (I escaped from the "we are a strictly Microsoft shop" employer where I used to work.) We here at eHealth Africa use Ubuntu and postgres as much as possible.  But it's not always possible.  For example, part of the project I am working on now will use ArcGIS -- which only runs on Windows.  Then one of our funding organizations (ever heard of the Gates Foundation?) required us to host a database on an MSSQL server. We bit the bullet, bought the licenses (it's their money, after all) and set up the SQL server.  Do you think we want to use django on it?  Huh?
 
If it will help kill polio, we will use anybody's software.

As to the question of moving MSSQL into the core, or moving the two Oracle dbs out of it (you do remember who owns MySQL?) -- I would suggest pulling all non-FOSS out.  Would you accept my patches to the core?  Of course not. and you should not: I don't know django well enough.  Would you accept my patches to an MSSQL back end?  I should hope so.

There has been talk. from time to time, about extending django to have more multi-database capability.  The changes that would be needed to run three different db engines from outside the core would easily extend to other engines.  People tend to think of ADO as only talking to Microsoft databases.  Nothing could be farther from the truth.  When maintaining adodbapi, I normally test against MS-SQL Server, Microsoft "Jet" (a.k.a. ACCESS), MySQL, and postgres. I have also personally used it to get data from IBM DB2, an Active Directory server, and a .csv file.  If some idiot has written an odbc driver for it, I will let you read it from Python.

Build the hooks.
--
Vernon Cole

[Florian Apolloner]

Hi,

On Friday, March 8, 2013 4:24:00 AM UTC+1, Michael Manfre wrote:

django-mssql is actively maintained and will be for at least the next few years because it's used for my employer's production site that is critical to business operations. The backend also supports stored procedures almost to the same extent as a Windows business application. The only real downside to the backend is that it only works on Windows, reducing its usefulness to a much smaller, but growing, minority of Django users.

What's the state of http://code.google.com/p/pymssql/ ? If we have MSSQL in core I'd really like to be able to talk with it from a Linux machine too, it would also make testing easier since we'd just need a VBox with MSSQL ;) Supporting a commercially available product but requiring to run even more commercial software seems like counter intuitive to me.

Regards,
Florian

[ry...@ryanhiebert.com]

from thread "Moving database backends out of the core"

On Mar 7, 2013, at 5:13 PM, Russell Keith-Magee <rus...@keith-magee.com> wrote:

> There is, however, a possible middle ground, following the example set by Flask: we introduce to Django a list of "officially recognised" extensions. These extensions are still maintained as external projects, but the core team promise to include these projects as part of the testing regimen for a release. If the MSSQL backend were recognised like this, we would run the full Django test suite against the MSSQL backend, and if that testing revealed a bug in the test suite which was identified as being due to a change in Django's codebase, that bug would be a release blocker (Bugs in the backend itself would still be the backend's responsibility, and not release blocking on Django)
>
> Looking outside database backends, this could also apply to high-profile projects like haystack, django-registration, and so on. This would also serve to highlight projects in our community that are 'defacto standards', or good examples of reusability, without needing to expand the core team or the size of contrib, and show that the core project is explicitly interested in the broader ecosystem.

I think this is a great idea. The backend testing makes sense to me (run Django's tests). How does testing sanctioned apps work, though? You run _their_ tests, and if they are caused by bugs in Django then they are release blockers?

I really like the idea, and would like to help if I can. I imagine the leg work will be mostly in setting up the testing infrastructure, although the difficult part will be setting guidelines for and choosing apps that should be in this group.

Ryan

[Javier Guerra Giraldez]

On Fri, Mar 8, 2013 at 4:25 AM, VernonCole <verno...@gmail.com> wrote:
> People tend to think of ADO as only talking to Microsoft databases. Nothing
> could be farther from the truth. When maintaining adodbapi, I normally test
> against MS-SQL Server, Microsoft "Jet" (a.k.a. ACCESS), MySQL, and postgres.
> I have also personally used it to get data from IBM DB2, an Active Directory
> server, and a .csv file. If some idiot has written an odbc driver for it, I
> will let you read it from Python.

nice.

does the client run on non-win32 plaforms? from a cursory read of the
website, adodbapi seems to require pywin32.

--
Javier

[Michael Manfre]

On Fri, Mar 8, 2013 at 6:35 AM, Florian Apolloner <f.apo...@gmail.com> wrote:

What's the state of http://code.google.com/p/pymssql/ ?

Their stable release relies upon a no longer supported Microsoft driver and has some known issues that make it unusable (lack of MARS support being the biggest issue encountered in production). They've been working toward switching to FreeTDS, but I have not actively followed their progress since last summer and don't know how close they are to the switch. I might be remembering incorrectly, but some of the issues with django-pyodbc's support of stored procedures might have been rooted in FreeTDS. I haven't reevaluated django-pyodbc for a few years because of its lack of support and the native client drivers available to django-mssql are faster.

 

If we have MSSQL in core I'd really like to be able to talk with it from a Linux machine too, it would also make testing easier since we'd just need a VBox with MSSQL ;) Supporting a commercially available product but requiring to run even more commercial software seems like counter intuitive to me.

Django needs a VBox running Windows for testing, regardless if MSSQL is in the core.

Regards,

Michael Manfre

[Michael Manfre]

adodbapi is included with pywin32 and requires it.

Regards,

Michael Manfre

[Florian Apolloner]

On Friday, March 8, 2013 3:35:53 PM UTC+1, Michael Manfre wrote:

If we have MSSQL in core I'd really like to be able to talk with it from a Linux machine too, it would also make testing easier since we'd just need a VBox with MSSQL ;) Supporting a commercially available product but requiring to run even more commercial software seems like counter intuitive to me.

Django needs a VBox running Windows for testing, regardless if MSSQL is in the core.

That doesn't mean we actually have to support Microsoft in making MSSQL only accessible from Windows… Also there is a large difference between running a vm with a database server and running a vm (read: jenkins slaves) to execute tests.

If Microsoft wants to use/support Django I guess they could easily publish a Python db-api 2.x driver. I'll happily code the Django part if the prerequisites would exist (assuming MSSQL actually does work ;))…

Regards,
Florian

[Russell Keith-Magee]

On Fri, Mar 8, 2013 at 11:23 AM, <ry...@ryanhiebert.com> wrote:

from thread "Moving database backends out of the core"

On Mar 7, 2013, at 5:13 PM, Russell Keith-Magee <rus...@keith-magee.com> wrote:

> There is, however, a possible middle ground, following the example set by Flask: we introduce to Django a list of "officially recognised" extensions. These extensions are still maintained as external projects, but the core team promise to include these projects as part of the testing regimen for a release. If the MSSQL backend were recognised like this, we would run the full Django test suite against the MSSQL backend, and if that testing revealed a bug in the test suite which was identified as being due to a change in Django's codebase, that bug would be a release blocker (Bugs in the backend itself would still be the backend's responsibility, and not release blocking on Django)
>
> Looking outside database backends, this could also apply to high-profile projects like haystack, django-registration, and so on. This would also serve to highlight projects in our community that are 'defacto standards', or good examples of reusability, without needing to expand the core team or the size of contrib, and show that the core project is explicitly interested in the broader ecosystem.

I think this is a great idea. The backend testing makes sense to me (run Django's tests). How does testing sanctioned apps work, though? You run _their_ tests, and if they are caused by bugs in Django then they are release blockers?

Caveat: I haven't given a huge amount of thought to this beyond the basic idea, so this is all my initial thoughts.

I can see three options:

1) We run the tests. This requires that we come up with a standard for describing how test environments work. This could be as simple as requiring that python setup.py test works, and produces output in a known format. Requiring the use of tox, or Jenkins/Travis configs would be another option.

2) We leave running the tests up to individual projects, but require them to expose test results in a standard way. No idea how this would be specified, other than possibly a HTTP API endpoint.

3) We make it completely manual. This is something that only needs to happen around release time, so as long as it's relatively easy to run the full suite of tests manually, it may be possible to manually maintain a database of "working/not working".

Option 3 might be the easiest way to get going - after all, you can always add automation infrastructure later. 

I really like the idea, and would like to help if I can. I imagine the leg work will be mostly in setting up the testing infrastructure, although the difficult part will be setting guidelines for and choosing apps that should be in this group. 

Agreed that this is more a policy/procedures issue than a technical issue. I'm not sure what the best approach is for this - I'm open to suggestions. 

As for how to get this ball rolling -- well, I suppose there's two tasks that could be addressed immediately:

 * A list of candidate packages -- even if it's not the final approved list, we need somewhere to start

 * A first cut at a site to host this information.

My local Django Users Group (in Perth, Western Australia) had some interesting discussions about this sort of packaging project at our last meeting; I'm hoping to start knocking together some code at our next meeting. As soon as we've got something to show off, I'll be sure to do so.

Yours,

Russ Magee %-)

[VernonCole]

[My reply is below, after the question...  scan down.  VDC]

Sorry about terse nature of the website information.  I need to repent and write better documentation.

ADO is a strictly Microsoft animal. It seems that Microsoft invented ODBC, which was quickly recognized as such a great idea that everyone adopted it.  So they had to invent something "better", and that was supposed to be ADO.  As it happens, the most redeeming feature of ADO is that in runs in ODBC mode by default. That having been said, it  is very powerful, and does have the ability to reach out to almost any form of tabular data. But it does so using COM calls, which are very proprietary.

I have been worrying over the problem of how to read an ADO dataset from Linux for years.  I was hoping that IronPython and Mono would provide the answer, but it turns out that ADO.NET also uses COM, and looses some of the power (like timely update of .rowcount) in the process.  It does not even work well in a pure Windows environment. I have abandoned that code fork.  The open source odbc products leave things to be desired, too, the largest being that they only work in ODBC mode and cannot switch to ADO mode when the connection string requires it. I have begun to think that eGenix is on to something with their "mxODBC connect" product: they use a connection server.

Here's how it might be done:  Somewhere, there would have to be a Windows box running an ADO connection server program.  Perhaps it could be on the actual database server, perhaps not.  A copy of adodbapi which notices that it is running where os.name != 'nt' would bring up a link to the connection server (using RPC?) which would do the actual data access and pass the results back.  A simple server would be trivial to write.  One which worked well under a heavy thread load might not be so easy, but my thinking is that a heavily used django server which made frequent use of an MSSQL database would already be running on a Windows box itself and would not need any connection server.

If there is interest, I will start coding a prototype.

[VernonCole]

Oh, MSSQL actually works (and works, and works, and works...) and finally comes up with your data. It may perform rather slowly, but it does perform.  And many of us are forced to use it by events beyond our control.  Most of us would probably prefer a Posix box for our django server, too, but we don't always have that choice, either.  Django servers are expected to run on Windows, and it would be a Really Good Idea (tm) if it were tested there during development, no matter which database it is using.

As for django servers reading MS SQL databases, I can see three use cases:
1) django server running Windows in heavy production.  This works fine, Michael has a pile of them.
2) django server running Posix in a test environment.  I think I can make this work using a connection server -- well enough to handle runsever at least.

3) django server running Posix in heavy production. There we run into problems with lightweight support for database drivers.  But will this situation ever exist in real life?  I don't think so.  I suspect that any heavy-use shops will also be using Windows to run their django servers.

As for Microsoft coding a db api 2.x driver -- why should they, when a perfectly good FOSS one already exists (If  I humbly say so myself)?
--
Vernon Cole

 

[Javier Guerra Giraldez]

On Sat, Mar 9, 2013 at 4:58 AM, VernonCole <verno...@gmail.com> wrote:
> But will this situation ever exist in real life? I don't think so. I
> suspect that any heavy-use shops will also be using Windows to run their
> django servers.

to me, that reads "heavy clients of MSSQL have to run windows".

what a pity.

--
Javier

[Rahul]

Hi,
 I maintain the IBM DB2 Django backend (ibm_db_django) project(http://code.google.com/p/ibm-db/). We have been maintaining this extension since Django-1.0 and have always kept it up-to-date.
  The http://code.google.com/p/ibm-db/ project also provides a Python DBI compliant driver, based on which the Django back-end is built. (For the record, in the past some of our customers have requested for Django to include the DB2 backend in core, which didnt happen). I really like Russell's idea to introduce officially recognized extensions. If that is the outcome of this thread, I would like to add the DB2 extension as officially recognized and will help in setting infrastructure for this if I can. I would prefer the automated test environment for recognized extensions so that any core Django commit will get expose if it cause any problem in some specific backend, In this way committer can raise a ticket against specific backend if any changes required in that.

Thanks,
Rahul Priyadarshi

[VernonCole]

My organization just hit a use case where we need MS-SQL support.

 I am jumping on board, so there are at least two of us who can do
maintenance.

I must say that I would prefer quasi-supported status (akin to admin
and geodjango) rather than actually being in the core.  I think it would
be a better fit for most situations.  We will always be a small minority of
django users.  I would just like some assurance that pull requests needed
to provide good hook support for external database backends got prompt
attention from the core developers.
--
Vernon Cole

[Johannes Hoppe]

I want to address a completely different point, and that innovation. I believe that 3rd party backends could lead to more innovation in the Django ORM.

Currently if you want to introduce a new feature for your database, you are faced with a lot of complications added by databases you might not be familiar with. Furthermore you might be requested to makes those features available for databases you haven't used before. This drastically increases the bar for contributing innovative new features. As an example, I wanted to add database defaults for Postgres and multiple insert return values. I finished the postgres feature in 2 sprints, but it took me another half year to implement the same for Oracle. Mainly because I never used Oracle before.

Beyond that I see that the sheer effort to remove the backends from core could lead to better design. We currently assume that all databases speak some flavor of Std SQL, which isn't even true for the currently supported databases and certainly not for the verify famous mongo-db backend.

In conclusion, more separation will lead to more diversity. But that is a good thing, something to embrace and can lead to great results. I would even go as far as to kick Postgres out too, contrary to what Florian suggested. I believe Postgres could benefit from a separate package.

I think Django has the best ORMs and us being able to make changes and innovate, can ensure that this is still true a decade from now.

[Tom Forbes]

I see where you are coming from, but I'm not sure it will have the intended effect. One of the great things about Django is that for the most part database features work everywhere. If we split them out there will be more innovation, sure, but we may loose the 'database transparency' we currently have leading to inconsistencies.

It may also be harder to coordinate changes across databases that require changes to core.

On your point about non-standard backends, maybe we should focus on making it easier to add third party backends and standardize some of the internals? We could treat the backend as external (i.e no special hacks for them) but keep them in the Django package. If we make the interfaces a bit more abstract it would be a lot easier for these backends to exist without hacks perhaps.

As with everything, it's a trade off. I'm not sure there is a clear answer or better solution for all cases than we currently have.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/978f91f8-709a-48dc-b2e2-b940e19a9f7f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Michael Manfre]

On Mon, Nov 26, 2018 at 10:49 AM Tom Forbes <t...@tomforb.es> wrote:

On your point about non-standard backends, maybe we should focus on making it easier to add third party backends and standardize some of the internals? We could treat the backend as external (i.e no special hacks for them) but keep them in the Django package. If we make the interfaces a bit more abstract it would be a lot easier for these backends to exist without hacks perhaps.

Over the past several years when I first started this thread, the API has improved and these types of improvements reduced the need to monkey patch or completely rewrite bits of the ORM in django-mssql. Continuing to move/keep any DB specific hacks out of the ORM and in to their respective backend makes Django friendlier for 3rd party database backends. The API is still deemed internal and if you push for a more official database backend API, I suggest clearly stating that it will still be exempt from the standard deprecation policy. There was a lot of strong opposition to changing that policy in the past.

Regards,

Michael Manfre

--

GPG Fingerprint: 74DE D158 BAD0 EDF8

keybase.io/manfre

[Dan Davis]

I think this would only work if most database specific backends were maintained by the djangoproject itself, allowing for integration tests that test compatibility.

To my mind, a strong ORM to backend API is a great thing, but we also need stronger backend integration tests.

[Johannes Hoppe]

How about we do pretend like we want to remove the backends from core and do the homework and refactor the backends to be less intertwined but keep them in the same repo/package. Maybe that’s how we can have the best of both worlds. It is going to be harder to motivate people to put in the work though :/

On 26. Nov 2018, 18:36 +0100, Dan Davis <dans...@gmail.com>, wrote:

I think this would only work if most database specific backends were maintained by the djangoproject itself, allowing for integration tests that test compatibility.

To my mind, a strong ORM to backend API is a great thing, but we also need stronger backend integration tests.

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/O-g06EM6XMM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/23759c70-4148-4ac7-8dfa-ca97f1dae97d%40googlegroups.com.

[Shai Berger]

Hi,

On Mon, 26 Nov 2018 00:57:04 -0800 (PST)
Johannes Hoppe <in...@johanneshoppe.com> wrote:

> I want to address a completely different point, and that

> *innovation*. I believe that 3rd party backends could lead to more

> innovation in the Django ORM.
> Currently if you want to introduce a new feature for your database,
> you are faced with a lot of complications added by databases you
> might not be familiar with. Furthermore you might be requested to
> makes those features available for databases you haven't used before.
> This drastically increases the bar for contributing innovative new
> features. As an example, I wanted to add database defaults for
> Postgres and multiple insert return values. I finished the postgres
> feature in 2 sprints, but it took me another half year to implement
> the same for Oracle. Mainly because I never used Oracle before.
>

I just wanted to point out that this road to innovation is already
open. There's nothing stopping you from writing a custom database
backend and releasing it to PyPI. Granted, then your new feature does
not become part of Django, but that would also be the case if the
standard backends were not part of Django.

I should point out that this point is not theoretical -- e.g. an
extended Oracle backend existed to support connection pooling[1], and
when I wanted to add database instrumentation[2], I first implemented
it (though I hadn't published that) in a custom backend, inheriting the
existing PG backend.

Shai

[1] https://github.com/JohnPapps/django-oracle-drcp
[2] https://docs.djangoproject.com/en/2.1/topics/db/instrumentation/

[Johannes Hoppe]

Hi Shai,

Thank you. You make a very good point. That is exactly what I meant. I have written small backends myself to add new features. The problem is that those features rarely make it into the core backend, because those are too static inside Django.

Best

-Joe

--
You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/O-g06EM6XMM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/20181128013339.2901e7a7.shai%40platonix.com.