Fellow Report - October 17, 2015

[Tim Graham]

Report for week ending October 17, 2015:

This week I spent three days at Microsoft HQ in Seattle at the first "open source data camp". As I understand it, the "data camps" are normally held to inform Microsoft partners about current and upcoming offerings in Microsoft cloud services and to get feedback about them. This session was the first time that they invited members of the open source community (3 for Django and 2 for Rails) as well as some startup/small business people who use open-source software (5 others).

Both groups were together on day one: an overview of Microsoft's cloud offerings and some technical discussion with Microsoft engineers about those products. This wasn't of much interest to me, but the idea seemed to be that we had to sit through this because the "data camp" budget was used to allow us to attend. On days two and three, the "users group" participated in more talks while the Django & Rails attendees worked with Microsoft engineers to give them some insight on what's needed to improve MSSQL/Azure database support as well as to actually write some some code.

Some of the things we accomplished:

* Completed an initial Jenkins integration on Windows: Microsoft engineers helped fill the gaps in my knowledge so that we could get our Jenkins CI infrastructure running the tests on Windows (SQLite, Python 2.7 + Python 3.5 only; however, I think this should be sufficient for now as I can’t remember a time when we received a bug report for a database-specific Windows issue). I’ll probably add pull request integration once things are running smoothly. We still have at least one time-dependent test failure on Windows (ticket #25562). Unfortunately, the setup process on the Windows slave is entirely manual (unlike the setup of the Linux machines which is automated with Ansible playbooks), however, it's not very difficult once you know the steps which I'll document when I get a chance. An idea from one of their engineers was to try Appveyor (think Travis CI for Windows) and got a test integration working there as well. It's an option if the Jenkins solution proves difficult to maintain, but I think there's no need to add a new service to our workflow for now. http://djangoci.com/job/django-windows/

* Improved teardown strategy in schema tests to fix MSSQL support: Michael had some work on the 1.7-era on this, but the patch was never committed and didn't entirely work, so I updated it and merged it to master, 1.9, and 1.8: https://github.com/django/django/pull/5433

* Investigated speeding up the django-mssql test suite: It takes something like 8 hours due to excessive constraint checking enabling/disabling which is very slow on MSSQL. This is no longer an issue on Django 1.9+ due the removal of deprecated initial data fixture loading, however, some solution for 1.8 seems important. I've proposed a pull request and am awaiting Michael's testing of it: https://github.com/django/django/pull/5438

* Discussed the long-term roadmap for MSSQL support in Django. I'll let Michael speak to the details of this.

On the last day, I wondered "what's next?" and voiced my skepticism about Microsoft following through with the resources to get support for Microsoft databases on par with our built-in database backends. During a conversation with Meet Bhagdev, the Microsoft program manager who invited us, I shared my idea of Microsoft hiring someone full-time to not only maintain the Microsoft database backends, but to participate in the Django community at large, a sort of "Microsoft Django fellow". Meet thought this wasn't out of the realm of possibility and encouraged me to bring it up at the executive Q&A that day with two of the SVPs. The execs seemed open to the idea and suggested it's mostly dependent on finding the right person. If you or someone you know is interested in this, let me know and I'll pass your name along.

There were also two Ruby on Rails community members in attendance. They learned about some difficulties of installing Rails on Windows and how that's really a diversity issues since most new programmers will be on Windows. That's something for the Django community to keep in mind, but I hope programs and tutorials like Django Girls will keep us honest.

Overall, I thought it was a worthwhile trip, especially if it results in a long-term partnership with Microsoft.

Other attendees for Django:

* Michiya Takahashi (maintainer of django-pyodbc-azure)

* Michael Manfre (maintainer of django-mssql)

Triaged

-------

https://code.djangoproject.com/ticket/25545 - ORM generates incorrect queries for m2m with disabled backward relations (duplicate/backported to 1.8)

https://code.djangoproject.com/ticket/25537 - Model objects not showing on admin changelist page after changing M2M to ForeignKey (worksforme)

https://code.djangoproject.com/ticket/25551 - makemigrations when adding a ForeignKey to a newly created model along with a unique_together constraint must create fields before the unique_together constraint (accepted)

https://code.djangoproject.com/ticket/25554 - Django permissions cause data migration depending on permissions to fail (duplicate)

https://code.djangoproject.com/ticket/25552 - Admin readonly field help text show 'None' (wontfix)

https://code.djangoproject.com/ticket/25557 - Fix the UserCreationForm for custom user model (duplicate)

https://code.djangoproject.com/ticket/21894 - Error in example code in django.forms.Form.clean() (accepted)

https://code.djangoproject.com/ticket/25541 - Improve error reporting when loading invalid JSON fixtures (accepted)

https://code.djangoproject.com/ticket/25547 - refresh_from_db leaves FieldFile with reference to db_instance (accepted)

https://code.djangoproject.com/ticket/25553 - Add lazy version of admin_static's static (wontfix)

https://code.djangoproject.com/ticket/25563 - Data migration causes incorrect tests behavior when custom manager is using .defer() (accepted)

https://code.djangoproject.com/ticket/25559 - Conditional admin inline causes ValidationError (wontfix)

Authored

--------

https://github.com/django/django/pull/5413 - Fixed #25550 -- Deprecated direct assignment to the reverse side of a related set.

https://github.com/django/django/pull/5433 - Used SchemaEditor.delete_model() for teardown in schema tests.

Reviewed/committed

------------------

https://github.com/django/django/pull/5419 - Fixed #25535 -- Made ForeignObject checks less strict.

https://github.com/django/django/pull/5396 - Fixed #25506 -- Allowed filtering over a RawSQL annotation.

https://github.com/django/django/pull/5434 - Fixed #25558 -- Fixed nondeterministic test failure on Windows: test_clearsessions_command.

https://github.com/django/django/pull/5425 - Fixed #25346 -- Allowed collectstatic to delete broken symlinks.

https://github.com/django/django/pull/5429 - Fixed #25517 -- Made Concat function idempotent on SQLite.

[Aymeric Augustin]

Thanks Tim!

Having CI on Windows is great, all the more considering your reminder that new programmers often use Windows.

I'm looking forward to Michael's proposal for the MSSQL roadmap.

-- 

Aymeric.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4e9ff664-a35c-43d4-9b60-35f1a32deac2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ola Sitarska]

Thanks Tim, great job.

If anyone is interested in installation of Python & Django on Windows, beginner friendly instructions are here. I heard from RailsGirls friends that Ruby/Rails situation on Windows is indeed a lot harder than ours.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/D0BC61D0-9640-40FD-A07E-31B0F9FB5F6D%40polytechnique.org.

[Tim Allen]

Thanks Tim, this is a promising update. While my department is slowly migrating to PostgreSQL, most of the other departments are still very much reliant on SQL Server. Since we're a Python/Django shop for our web stack, better support would be very welcome. Most of our developers are on Macs, but we do have some on PCs, so supporting both is very much of interest to us. I'm sorry I couldn't join via Skype, I was looking forward to it, but a nasty head cold got in the way.

Old, I've attempted to improve the Windows Installation instructions in the Django docs and issued a PR here:

https://github.com/django/django/pull/5376

I'd very much like more feedback. It uses the Windows Python 3.5 installer (so should wait until 1.9), which is greatly simplified. The addition of the checkbox to include in the Windows path removes a problem barrier I've seen many times, and virtualenvwrapper-win also helps remove many stumbling blocks with pathing that beginners really don't need to fully understand to get started, in my opinion.

Regards,

Tim

On Saturday, October 17, 2015 at 8:36:25 PM UTC-4, Tim Graham wrote:

Report for week ending October 17, 2015: 

  

 ........

[Meet Bhagdev]

Thanks for the thorough report Tim. We are glad you had a good experience here at Microsoft. I will send out a similar report by the end of the week on the SQL Server/Azure SQL Discussion on the mailing list: https://groups.google.com/forum/#!topic/django-developers/FbBcUCzrSZo. You bring out some great points especially about the long term partnership and I will update you as soon as I have more information and solid answer.

Thanks,

Meet

[Michael Manfre]

Tim gave a good overview of the week. I'll focus my response on the MSSQL roadmap.

On Sat, Oct 17, 2015 at 8:36 PM, Tim Graham <timog...@gmail.com> wrote:

* Discussed the long-term roadmap for MSSQL support in Django. I'll let Michael speak to the details of this.

I had the opportunity to speak with several engineers that had intimate knowledge of Microsoft's various SQL Server drivers; past, current, and future. The ADO based drivers used by django-mssql will not receive any future love from Microsoft. They don't have plans on removing them from Windows, but the advice was to use either ODBC or FreeTDS. The fastest and most feature rich drivers for connecting to SQL Server from Windows are the ODBC drivers. Microsoft is actively working on updating the ODBC drivers for Linux and expect them to be released this year. ODBC drivers for Mac are planned for sometime next year. The next best way of connecting to SQL Server is to use FreeTDS (without the ODBC interface), which has the benefit of having drivers available for all OSes right now.

The short-term plan for django-mssql is to get it to pass the Django 1.8 test suite, without making any substantial changes. This will allow any existing django-mssql users to upgrade to the Django 1.8 LTS. My long term plan is to switch out ADO for replaceable ODBC and FreeTDS. This may follow the current pattern Aymeric started when he created django-pymssql, or have them both in django-mssql. After 1.8 is fully supported, I need to think, discuss, and investigate more before making a decision.

ODBC seems to be the most performant and best option, but I have concerns about the current lack of good drivers for Linux and Mac. If Microsoft follows through with their commitment, then it should be a non-issue, but I'd rather hedge my bets and ensure an easier time with drivers in the long run. Supporting both also provides for a good way of comparing the performance of the underlying drivers.

I plan on evaluating django-mssql, django-pyodbc-azure, and starting from scratch to see which would be the best route for this new mssql database backend. There are pros and cons to each. Django-mssql has a lot of legacy cruft. Django-pyodbc-azure targets Azure SQL Server, which has some subtle differences with Microsoft's non-cloud version of the database server. Starting from scratch is potentially more work. Regardless of the decision, the repo will be hosted on github.

Regards,

Michael Manfre

--

GPG Fingerprint: 74DE D158 BAD0 EDF8

keybase.io/manfre

[Aymeric Augustin]

2015-10-20 5:04 GMT+02:00 Michael Manfre <mma...@gmail.com>:

My long term plan is to switch out ADO for replaceable ODBC and FreeTDS. This may follow the current pattern Aymeric started when he created django-pymssql, or have them both in django-mssql.

For the record, django-sqlserver did that as well. In django-pymssql, I chose to wrap django-mssql instead of forking it to avoid the having to maintain a fork until a better solution is available.

--

Aymeric.

[Michael Manfre]

Thanks for the reminder. I forgot about django-sqlserver. I'll also evaluate that backend.

Regards,

Michael

[Marc Tamlyn]

This is awesome news for Django's status in enterprise, great work everyone.

I hope something comes from the Microsoft Fellow concept, that would be brilliant.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAGdCwBu6rHaLXec2ZN-LHhCxb0w3HLqZHB1hWNNxYGsHGyQBZQ%40mail.gmail.com.