Revisiting MSSQL and Azure SQL Support in Django

[Warren Chu]

Hi All,

There is increasing interest within Microsoft to have stronger ties between Microsoft SQL Server and Django. As you may be aware, Microsoft and their connectivity teams have been managing the 3rd party backend for "mssql-django" for over a year now at: https://github.com/microsoft/mssql-django

Inclusion of SQL Server as a 1st party backend is viewed as a potential big milestone in that regard.

@adamjohnson mentioned a year ago that ideally the community would like to see multiple years of ongoing Microsoft support before considering merging as a 1st party backend.

We'd love to hear thoughts and feedback around the possibility of moving forward with a DEP enhancement proposal, with a commitment from Microsoft to providing continued dedicated support for the 1st party backend through the Django project itself (rather than the 3rd party repo).

Cheers,
Team Microsoft

[Adam Johnson]

Hi again Warren,

Good work on maintaining the backend.

Merging the backend could be a good end goal, but I'd be concerned about merging it in the current state. The README lists many features that don't work: https://github.com/microsoft/mssql-django#limitations . This list includes some key features like timezones, constraints, and renaming models with foreign keys. The text also has very limited detail on the problems, for example: "Timezones, timedeltas not fully supported" - what doesn't work?

For comparison, the Cockroach DB backend lists its differences and deficiencies in much more detail: https://github.com/cockroachdb/django-cockroachdb#known-issues-and-limitations-in-cockroachdb-212x-and-earlier . Although the list there is physically longer, the issues are much more niche. This gives me the impression that the Cockroach DB backend has much better feature coverage.

Merging any code into core Django significantly limits the ability to iterate on it, given the strict release cycle. Users would have to wait for a new Django version to get improvements. I think that whilst the MS SQL backend is missing coverage for key features, effort would be better spent improving it than on any merge proposal.

Adam

P.S. I still don't see Microsoft on the sponsorship page: https://www.djangoproject.com/fundraising/ 😉

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/0c6ca059-d50e-4c84-bef6-ab0742fc4fa9n%40googlegroups.com.

[Warren Chu]

Thanks for the feedback Adam. Your suggestions are actionable and potential sponsorship has been raised for discussion as recently as this week (no promises or strings attached).

We'll reach out to you directly if we have any direct follow-up on filling in the feature gaps.

 -Warren

[Daryl]

My 2c;

The technical board has always done a stellar job of ensuring that good ideas end up in the code base, and unfinished, unsupported, incomplete, young, or non-reviewed code stays outside.

The quality of the core framework, and the ease of having 3rd party code exist as Extensions, Plug-ins and Related Libraries means that the technical board can be -1 on a request to add code to the core framework without negatively impacting that code's future potential. It might be a blow to the author(s) ego, but saying "No" isn't going to make the code any harder for end users to find, install, review, debug, and it gives the code more time to be improved. In the future, if the code demonstrates stability, performance and user support, maybe it gets another chance to be merged into core.

Django's quality is, in part, a direct result of saying "No" at the right times.

No personal disrespect intended the the authors of the MSSQL backend package (haven't seen it, haven't used it, probably never will given the quality of other available backends) but after spending nearly 3 decades developing with OSS DBs and MSSQL, at times teaching design and DBA, and having to support MS server and DB systems, Microsoft's code teams (in my opinion) haven't shown a commitment to high quality, peer reviewed code. Microsoft's marketing, telemetry and legal teams however have shown a very strong commitment to put their own interests above those of the users.

This may well be a reflection of (again, my opinion) the 'profit first, image second, quality where possible (security if we get time or negative PR requires it)' theme that permeates MS.

Code must stand on its own merits, and no code decision should be influenced by the reputation of the author, either negatively or positively, but having said that, if the technical board were to decide to include the MSSQL backend in core, there would have to be a commitment to review every single commit from that point on, specifically looking for the ways MS will put pressure on the codebase to bend towards the MS goals, not the Django goals. Where the goals are the same, that is fine. I remain skeptical that the goals are the same.

Finally, I congratulate Warren for getting the code this far.

D

  

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/0c6ca059-d50e-4c84-bef6-ab0742fc4fa9n%40googlegroups.com.

--

-- 
======================
Daryl Egarr,  Director

Kawhai Consultants Ltd
Cell       021 521 353
da...@kawhai.net
======================

[Florian Apolloner]

Hi Waren,

On Thursday, March 31, 2022 at 6:30:06 PM UTC+2 vwa...@gmail.com wrote:

We'd love to hear thoughts and feedback around the possibility of moving forward with a DEP enhancement proposal, with a commitment from Microsoft to providing continued dedicated support for the 1st party backend through the Django project itself (rather than the 3rd party repo).

As pointed out by Adam I do not think the backend is ready for core. At the minimum it would need to support most of what Django offers. Then there is also the question on how to ensure continued commitment from Microsoft -- what happens when Microsoft (or the current PM/PO or whatever) looses interest? The backend would be in Django then and we can't easily kick it out.

Given the backend APIs and the recent enhancements by Tim to allow backends to define which tests they can't pass etc I do not see any strong technical argument for including a new backend in core at all. Aside from a selling argument for Microsoft that Django includes the backend in core, what do we gain? It is not like one usually uses Django without installing any 3rd party dependencies, so installing one more certainly won't hurt.

Cheers,

Florian

[Carlton Gibson]

> Then there is also the question on how to ensure continued commitment from Microsoft -- what happens when Microsoft (or the current PM/PO or whatever) looses interest?

IMO. This is the biggest one. 

What I'd expect to see here is MS taking up a long-term corporate sponsorship of the DSF, demonstrating a commit, and giving the DSF resources to ensure the backend was covered. 

Every time that's been mentioned though it's like crickets — zero response. Certainly zero positive response. 

(I appreciate that's got to come from **someone's** budget, but no more than engineering time.) 

Kind Regards,

Carlton

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/826cc91f-1621-4255-8679-ef7e8da1cc09n%40googlegroups.com.

[Salman Mohammadi]

Hi,

IMO, Django is there to create value for *its* users.

I'm not aware how MS Team reached this conclusion that merging their incomplete package into Django core creates more value for Django users than when it is a third-party package. Would you please tell me how?

I have access to only two resources to calculate the popularity of mssql-django package, one of them is the last Django Survey,

https://www.djangoproject.com/weblog/2022/jan/03/django-developers-survey-2021-results/

and the other one is Github Stars. None of them show any sign that MS package is even semi-popular among Django users.

We have many popular packages that are residing outside of Django core and living happily their lives.

Probably MS wants to use this merge as a marketing point for their MSSQL, a proprietary software.

At the end I, as a total outsider with no direct connection of any kind with Django project, have nothing against merging MS package into core the same way that we have Oracle backend. But before that, there are some questions that need to addressed:

1. How can MS package create more value for Django users by getting merged into main branch?

2. According to your internal spying tools, how popular is your package?

3. How can you guarantee the long-term sustainability of your package?

4. What are the previous involvements of MS Package contributors in Django?

5. How does MS support Django Project for its long-term fundraising goals?

6. Is MS Team ready to follow Django Project deadlines? Especially release dates and critical bugs.

7. How does MS support Django Fellows in helping them triaging the tickets, related to MS package.

Cheers, Salman

[Tim Allen]

Full disclosure: I've helped maintain SQL Server Django engine packages in the past, and was involved in getting the project started for the official Microsoft Django engine package.

First, thanks to Warren and the folks from Microsoft for creating this backend. There are a fair amount of users that need a Microsoft SQL Server backend, and for years we've been having to shuttle between barely-maintained engines, forks, and pip installs to commit hashes while waiting for merges. It is less than ideal. Several big supporters of Django (The Wharton School, RedHat Ansible, to name two) have been among those looking for a solid backend solution for years. Microsoft has provided us a vastly improved option to what we'd been saddled with for the past decade.

The DB popularity index at db-engines.com has regularly listed the top four as Oracle, MySQL, Microsoft SQL Server, and PostgreSQL, in that order. I notice some comments in this thread about Microsoft being for-profit... well, what about Oracle? I don't see Oracle on the Support Django page either, yet two of their databases have support in core. MSSQL is the only one of the big-four RDBMS's without support in core Django. That seems to be a pretty big hole in Django's offering.

That said, I completely agree that the backend still needs work, and a solid commitment from Microsoft before being brought into core Django. I want to make sure Microsoft knows that many in our community appreciate their efforts in taking the lead on this backend: support for SQL Server in the Django ecosystem is far better than it ever has been in my eight years of being a Djangonaut. They have put a lot of time and effort into this project, and I think they're well on their way to where they need to be for the long-term goal of being in core Django.

A lot of the questions being asked of Microsoft in this thread just don't seem fair to me - we're not asking the same of Oracle, Redis Enterprise Software, or any of the other commercial products that Django has built-in support for. Why Microsoft and not the others?

Warm regards,

Tim

[Matthew Pava]

I agree with Tim’s sentiment here. Had a solid SQL Server backend been available 7 or 8 years ago, I would probably have used that due to the requirements from my supervisor. I had to convince him that PostgreSQL would be just as good.

 

Based on what I read on the Git page, though, the backend doesn’t pass some tests that it seems like it ought to. Once it is better, as Tim already stated, I would absolutely support and wish and hope that SQL Server would become a part of core. It is one of the most popular database backends in use.

 

One argument was made that we don’t need to add any other database backends to core because they are doing fine as third-party dependencies. To that I would ask, why have any database backend in core then? Why not have them all be third-party dependencies?

 

Just my two cents,

Matthew

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/6baed3f9-6cf5-459a-9937-f0c38d64f332n%40googlegroups.com.

[Warren Chu]

Hi All,

Thanks for the feedback thus far - this is going to help inform my ongoing discussions within Microsoft and I feel like we are getting to the heart of pros/cons/concerns around MSSQL + Django.

I appreciate there are diverse stances on this subject, and feel like this conversation is a helpful one. To be clear, my mission here is to ascertain if there might be a mutually beneficial solution.

Cheers,

Warren

[Carlton Gibson]

Hey Tim. 

FWIW I have directly asked people at Oracle if they would take up a corporate sponsorship of the DSF in order to help support the ongoing maintenance of the backend. 

Again, it was almost like I hadn't said anything. Rather than a Yes or a No, they did instead offer some cloud hosting, which we looked into but which wasn't appropriate. 

The Oracle backend would be in big trouble if Mariusz wasn't personally a fan. Oracle stepping up to support it would make it much more sustainable. If anyone knows how to get to have a sensible conversation with someone with the ability to make such a contribution to the DSF please do let us know. 

MySQL is slightly different, in that there are a number of active contributors for it.  Support from Oracle Corp would still be welcome, but it's not as pressing IMO. 

Personally, I think MSSQL support in core would be awesome, but we can't just take that on because it would be a Nice to Have; we have to be realistic about how it's going to be maintained. 

Kind Regards,

Carlton

--

[Dan Davis]

Warren,

We all know support for an RDBMS matters a lot in enterprise corporate environments. I think having this support in the https://github.com/microsoft organization is probably better for MSSQL users.

Coordinating support between a support organization, open source components, and Django is challenging, and even for projects hosted within an organization like that. It isn't always clea that support contracts for MSSQL include support for the Django-mssql extension and all layers in between. At least clarifying under Features that support fis included in a support contract will catch the eyes of Django devs who are also enterprise decision makers and influencers. You could also align things so that support requests that mention django-mssql always let you know (open a linked gitub issue, or something).

Clarifying that at DjangoCon will go so very far in making users in corporate environments happy.

I'm also going to suggest building more coordinated support for publishing to Power BI from Django applications.  Maybe you can include logic that understands how to publish to Dataverse based on Django models.  This will go over a lot of heads, but not mine :).

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAJwKpyQDqeiD8iLkk5Np_b8q2Vz2POqj710e3kLczahSFwLgeg%40mail.gmail.com.

[Florian Apolloner]

Hi Tim,

On Friday, April 1, 2022 at 5:02:00 PM UTC+2 Tim Allen wrote:

The DB popularity index at db-engines.com has regularly listed the top four as Oracle, MySQL, Microsoft SQL Server, and PostgreSQL, in that order. I notice some comments in this thread about Microsoft being for-profit... well, what about Oracle? I don't see Oracle on the Support Django page either, yet two of their databases have support in core. MSSQL is the only one of the big-four RDBMS's without support in core Django. That seems to be a pretty big hole in Django's offering.

I understand that comparing to existing databases seems like to make an argument for MSSQL but it is not that easy. First off, Django was released way earlier than Oracle acquired MySQL -- I think it is understandable that we do not simply drop support for a database just because Oracle buys it. Postgres and MySQL (or now MariaDB) are both easily installable via every Linux distribution and have been there since the beginning. Support for Oracle itself (iirc) was added because the team at that point in time thought it would enable Django to get access to areas where it hadn't access before. Oracle itself has (imo) been proven to be quite a burden over time and there had been discussion about removing it from core more than once.

Truth to be told, if the inclusion request for MSSQL gets serious we will have to start a discussion about whether or not we simply nuke all database backends (aside from sqlite maybe) from core. I'd be slightly in favor of simply setting a policy for core that it should only include backends of OSS databases. Why? Because it is way easier to install those on various systems than their commercial alternatives (even if there are test licenses and possibly free containers around there).

 

They have put a lot of time and effort into this project, and I think they're well on their way to where they need to be for the long-term goal of being in core Django.

I applaud to that, but I still do not understand why a well maintained database backend needs to be inside core? And I am not just talking about database backends here, we are saying no to pretty much every library inclusion.

 

A lot of the questions being asked of Microsoft in this thread just don't seem fair to me - we're not asking the same of Oracle, Redis Enterprise Software, or any of the other commercial products that Django has built-in support for. Why Microsoft and not the others?

We are asking Oracle but we are also not getting far, and simply kicking it out is not something we do easily. As for Redis, as far as I am concerned (and to the extend we support it) is open source.

Cheers,

Florian

[Tim Allen]

Thanks for the thoughtful replies, and the context - I'm mostly in agreement, and do hope that both Oracle and Microsoft step up as corporate sponsors (not to mention other companies that have been sold for eight or nine figures).

My hope is that we can give credit where credit is due, and acknowledge that Microsoft has made significant strides as an open-source partner. I want to encourage this behavior, as I've witnessed a change since Satya Nadella came on board. They have made life much easier for many of us in the Django community. My reply was primarily due to what I perceived as overly-harsh comments which don't seem to reflect the current-day Microsoft.

Take care,

Tim

[Gert Burger]

As a commercial user of all the builltin DB backends and the MSSQL backend (both the MS fork and its parent), I tend to agree more with Florian's arguments.

Our experience with the MS MSSQL and Oracle DB backends in supporting our enterprise customers have been difficult at best, in comparison to the other backends.

Due to the immaturity of these two backends we have resorted to maintaining our own set of patches for each, to handle the seemingly endless list of edge cases they currently do not support.

So I would argue that the MS MSSQL backend should not be included due to its current level of maturity and its rough history since the fork. On the surface it seems like Microsoft's current level of support is minimal, resulting in slow progress and varying levels of stability.

I do still hope that MS and Oracle will step up and provide a higher level of support so that their DB backends can match the popular community maintained ones.

Regards

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/efd08dee-e86f-4fef-a39a-5fb89448fc48n%40googlegroups.com.

[אורי‎]

Hi All,

After reading your comments, I think Django should keep supporting only open source databases, as well as other staff (operating systems, packages) and drop support for the closed source databases, operating systems etc. It seems to me that some of you think supporting Oracle was a mistake. I also think that it's better for an open source package (Django) to support other open source software and operating systems, and keep the closed source support for the providers. Nobody's stopping them from providing their own support for their closed source databases and packages, but I don't think Django should make an effort to support them in its core.

Thanks,

Uri Rodberg, Speedy Net.

אורי

u...@speedy.net

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/0c6ca059-d50e-4c84-bef6-ab0742fc4fa9n%40googlegroups.com.