SSL support for Django-admin runserver‏

[Steven Berry]

While working with an OAuth library I was running into some issues signing requests on a local development server due to lack of HTTPS.

To solve this problem I was deploying web application code to a VM with a shared folder or SCP and running an TLS-enabled nginx server. Obviously the development/test latency here is unacceptable. I looked into what solutions were out there and found a Django plugin django-runsslserver (https://github.com/teddziuba/django-sslserver). I installed this app, but was disappointed by a few things:

• The included certificates didn't work due to an EOF openssl error
• I was required to install an additional app (even just doing a += on the INSTALLED_APPS tuple in a separate development settings module felt a bit wonky)
• I now had command bifurcation when running development servers -- runserver, runsslserver, and testserver (which in turn didn't support SSL)
• The code was necessarily repetitive because Django does not currently abstract the django.core.servers.basehttp module enough for this purpose

I felt like this was something Django should just optionally support out of the box for simple use cases such as my own. I forked Django and switched my installed version in my virtual environment to my fork with "pip -e". I used this myself for a while, but I thought my modest change would be objectively useful for others and would be relatively trivial to review and implement.

For reference I developed a fix here https://github.com/sjberry/django/commit/c8e808d70f4a0ac6ebf4634576a2538b4f35b83e and subsequently submitted a PR here https://github.com/django/django/pull/4633. Though the PR was admittedly out of order due to my inexperience with the contribution protocol.

Let me know if I'm off base here.

Regards,

Steven Berry

[Gert Van Gool]

Hi Steven,

It might be handier to use gunicorn with SSL support[1] and since there is already a section in the Django documentation on how to use gunicorn in a project[2].

[1]: http://gunicorn-docs.readthedocs.org/en/latest/settings.html#ssl

[2]: https://docs.djangoproject.com/en/1.8/howto/deployment/wsgi/gunicorn/

-- Gert

Twitter: @gvangool
Web: http://gertvangool.be

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/40207727-f555-4c1a-bdc4-dcc40e9be935%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Steven Berry]

From what I understand that's predicated on a *nix environment. So while it's a perfectly reasonable solution, and one which I may end up using myself, it doesn't cover all bases.

I can certainly understand if the goal is to keep runserver as slim as possible so as not to reproduce functionality found elsewhere. However, I'd be more comfortable with that demarcation erring on the inclusive side of rudimentary TLS support, which in today's internet is more or less a requirement for production development. Obviously runserver shouldn't start to reproduce more functionality from gunicorn such as UNIX sockets (I'm sure there's more, but I'm less familiar with it than I'm sure you are).

I defer to better and more experienced judgement, but as a developer and end-user those are my thoughts and persistent pain points.

Regards,

Steven

[Tim Graham]

Hi Steven,

I'm in favor of trying to switch runserver to use gunicorn instead of expanding the functionality of our own homegrown webserver (https://code.djangoproject.com/ticket/21978). Windows support still remains an unsolved issue (https://github.com/benoitc/gunicorn/issues/524), but I don't think the lack of Windows support should stall that effort or justify adding SSL support to the homegrown runserver.

Tim

[Steven Berry]

Tim,

Thanks for taking the time to respond to this and sending those links. I familiarized myself with the gunicorn issues list when Gert suggested using it, but I was unaware of #21978.

As the ticket suggests I don't think it's the current goal to deprecate the runserver command entirely (correct me if I'm wrong) as it offers validation, migration checking, switching settings, etc. by virtue of the base Django command functionality.

In perusing that ticket and its comments I noticed a few agreeable ideas:

1. https://code.djangoproject.com/ticket/21978#comment:1 suggests having a --gunicorn flag
   
2. https://code.djangoproject.com/ticket/21978#comment:2 suggests a perhaps better option of assuming you'd want to use gunicorn if it were installed
   
3. https://code.djangoproject.com/ticket/21978#comment:6 enhances the automatic gunicorn paradigm with the --no-gunicorn option
   

For reference the most recent PR associated with this ticket was https://github.com/django/django/pull/3461. Of particular note and relevance to this discussion was @cancan101's question on Nov 18th 2014 about whether or not SSL/TLS would be supported by the PR. It looks like the discussion was subsequently dropped and more recently the PR was closed due to inactivity.

Taking a look at gunicorn, Windows support is slated for R20.0 which has no stipulated release date. Furthermore, there is no active branch indicating any progress toward supporting Windows, nor any substantial discussion on the issue since Sep 22, 2014. This is despite the fact that it was originally suggested over two years ago.

With all that said I'm in favor of what you suggest -- rely on gunicorn where possible. However I don't think what I'm suggesting (and have already implemented) fundamentally interferes with #21978. As far as I can tell the django.core.servers.basehttp module exists solely for the runserver command. And the contents therein aren't so much of a homegrown webserver as they are convenient subclasses to Python's inherent wsgiref.simple_server. The onus of maintenance lies largely in the Python codebase.

The django.core.servers.basehttp module could be easily renamed django.core.servers.wsgi and serve as a compatibility module (much like six) that could make runserver function with or without gunicorn depending on whether or not it were installed.

Should this discussion otherwise be summarily abandoned, I maintain that --certfile and --keyfile arguments to enable SSL/TLS directly from runserver are a good idea, even if that simply means a CommandError is thrown if gunicorn isn't installed as per the goal of #21978.

Regards,

Steven

[Florian Apolloner]

On Monday, May 11, 2015 at 10:13:03 PM UTC+2, Steven Berry wrote:

With all that said I'm in favor of what you suggest -- rely on gunicorn where possible. However I don't think what I'm suggesting (and have already implemented) fundamentally interferes with #21978. As far as I can tell the django.core.servers.basehttp module exists solely for the runserver command. And the contents therein aren't so much of a homegrown webserver as they are convenient subclasses to Python's inherent wsgiref.simple_server. The onus of maintenance lies largely in the Python codebase.

It might not interfere,  but there are already perfectly fine alternatives like stunnel which provide you with TLS support. So unless there is a really compelling argument to TLS support to runserver itself (which I yet fail to see in this thread), I am -0 to -1. Keeping the code as simple as possible also makes migrations to a 3rd party server easier later on.

Cheers,
Florian

[Alex Gaynor]

I don't have a strongly held opinion on whether Django should include this, but if it's going to, it should include a *well configured* TLS server, utilizing modern TLS cipher suites, TLS versions, etc. We shouldn't be yet another part of the "well, it's not *my* job to configure OpenSSL correctly" problem.

Alex

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/d4e1a606-3af4-43d2-90e4-bed9bcf28ebb%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

"I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero

GPG Key fingerprint: 125F 5C67 DFE9 4084

[Russell Keith-Magee]

I'm completely on-board with the idea of switching to a "real" web server rather than continuously improving our own. 

However, there might be better options than gunicorn. Graham Dumpleton has released mod_wsgi-express:

http://blog.dscpl.com.au/2015/04/introducing-modwsgi-express.html

which is a fully fledged WSGI web server, but it looks like it may be a lot easier to get started and integrate.

Just throwing it out there...

Yours,

Russ Magee %-)

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/1428d3dc-d36b-4e1e-8b4a-c20b618d5f0a%40googlegroups.com.

[Carl Meyer]

Hi Russ,

On 05/11/2015 06:05 PM, Russell Keith-Magee wrote:
>
> I'm completely on-board with the idea of switching to a "real" web
> server rather than continuously improving our own.
>
> However, there might be better options than gunicorn. Graham Dumpleton
> has released mod_wsgi-express:
>
> http://blog.dscpl.com.au/2015/04/introducing-modwsgi-express.html
>
> which is a fully fledged WSGI web server, but it looks like it may be a
> lot easier to get started and integrate.

I'm all about evaluating alternatives to gunicorn. Waitress [1] is one
that already supports Windows and I think is worth looking at; all it's
missing is auto-reload capability (but I think we could easily layer
that on top of it if we want).

The CherryPy web server I think could be another great candidate, it's
just a little weird that it's not distributed independently of the
CherryPy framework. Practically I don't think that'd be an issue
(CherryPy is small), it's just a little odd to say "install this other
Python web framework to serve your Django project."

mod_wsgi-express looks clever, but AFAICT it still has a hard dependency
on Apache httpd and a compile tool-chain (which also makes me doubt that
it supports Windows or ever will), it just does some cleverness to
smuggle Apache in and compile it via "pip install". I definitely don't
think that for a built-in dev server we should choose something that
requires a compilation tool-chain.

Carl

[1] http://waitress.readthedocs.org/en/latest/

[Steven Berry]

All,

Thanks for the feedback and input here. I appreciate the experience you guys bring to the table.

On Monday, May 11, 2015 at 6:34:51 PM UTC-4, Alex_Gaynor wrote:

I don't have a strongly held opinion on whether Django should include this, but if it's going to, it should include a *well configured* TLS server, utilizing modern TLS cipher suites, TLS versions, etc. We shouldn't be yet another part of the "well, it's not *my* job to configure OpenSSL correctly" problem.

Alex, I think this touches on what Tim and Florian are fearful of. It isn't and shouldn't be Django's responsibility to configure OpenSSL correctly considering that runserver is explicitly recommended against in production. But on that note you bring up a good point, who am I to determine what the minimum runserver TLS support should be? Is it good enough to have simple TLS? Is it essential for developers to be able to specify the TLS version and ciphers?

On Monday, May 11, 2015 at 6:30:39 PM UTC-4, Florian Apolloner wrote:

On Monday, May 11, 2015 at 10:13:03 PM UTC+2, Steven Berry wrote:

With all that said I'm in favor of what you suggest -- rely on gunicorn where possible. However I don't think what I'm suggesting (and have already implemented) fundamentally interferes with #21978. As far as I can tell the django.core.servers.basehttp module exists solely for the runserver command. And the contents therein aren't so much of a homegrown webserver as they are convenient subclasses to Python's inherent wsgiref.simple_server. The onus of maintenance lies largely in the Python codebase.

It might not interfere,  but there are already perfectly fine alternatives like stunnel which provide you with TLS support. So unless there is a really compelling argument to TLS support to runserver itself (which I yet fail to see in this thread), I am -0 to -1. Keeping the code as simple as possible also makes migrations to a 3rd party server easier later on.

Florian, by that logic what's the point of having runserver in the first place? There are already perfectly fine alternatives that serve web content. What I was suggesting was moving something that is half-baked to something that's at least 60% baked while still taking advantage of what runserver does well. The trick, which I touched on in my second post, and which I think Tim addressed, is identifying what your intended scope is here. If the goal is to deprecate runserver entirely then I agree with you. But that intention hasn't been made clear, and what we're left with is a crippled webserver with a bit of an identity crisis.

Based on your feedback I'm convinced that deprecating runserver altogether is probably the best course of action (eventually). However, before that could even be considered I think the documentation would have to be prepared accordingly. I would suggest updating:

• https://docs.djangoproject.com/en/dev/howto/deployment/ to remove references to runserver. Perhaps the guide should also be retitled "Serving Django" which would be relevant to both production deployment and local development testing.
  
• Recommend running check instead of runserver.
• https://docs.djangoproject.com/en/dev/howto/static-files/ to remove references to runserver.
  
• Add support to the Django settings to automatically (and optionally) add static file URLs based on DEBUG = True and maybe another (new) setting SERVE_STATIC = True. I think this is the only feature I miss from the built-in runserver in switching to third-party servers. For reference there are currently workarounds like this http://stackoverflow.com/a/12801140, but integrating something like that under the hood may help ease the migration path.

This would make https://code.djangoproject.com/ticket/21978 as well as my current PR https://github.com/django/django/pull/4633 effectively moot.

In the interim I'm just going to pretend runserver doesn't exist and move on with my life (which is what I expect what most of you guys do anyway).

Best,

Steven

[Berker Peksağ]

On Mon, May 11, 2015 at 4:34 PM, Tim Graham <timog...@gmail.com> wrote:
> Hi Steven,
>
> I'm in favor of trying to switch runserver to use gunicorn instead of
> expanding the functionality of our own homegrown webserver
> (https://code.djangoproject.com/ticket/21978). Windows support still remains
> an unsolved issue (https://github.com/benoitc/gunicorn/issues/524), but I
> don't think the lack of Windows support should stall that effort or justify
> adding SSL support to the homegrown runserver.

https://github.com/benoitc/gunicorn/issues/935 is another smail
blocker for Gunicorn integration, but there is already a solution in
the issue.

--Berker

[Florian Apolloner]

Hi,

On Tuesday, May 12, 2015 at 4:43:41 AM UTC+2, Steven Berry wrote:

On Monday, May 11, 2015 at 6:30:39 PM UTC-4, Florian Apolloner wrote:

On Monday, May 11, 2015 at 10:13:03 PM UTC+2, Steven Berry wrote:

With all that said I'm in favor of what you suggest -- rely on gunicorn where possible. However I don't think what I'm suggesting (and have already implemented) fundamentally interferes with #21978. As far as I can tell the django.core.servers.basehttp module exists solely for the runserver command. And the contents therein aren't so much of a homegrown webserver as they are convenient subclasses to Python's inherent wsgiref.simple_server. The onus of maintenance lies largely in the Python codebase.

It might not interfere,  but there are already perfectly fine alternatives like stunnel which provide you with TLS support. So unless there is a really compelling argument to TLS support to runserver itself (which I yet fail to see in this thread), I am -0 to -1. Keeping the code as simple as possible also makes migrations to a 3rd party server easier later on.

Florian, by that logic what's the point of having runserver in the first place? There are already perfectly fine alternatives that serve web content. What I was suggesting was moving something that is half-baked to something that's at least 60% baked while still taking advantage of what runserver does well. The trick, which I touched on in my second post, and which I think Tim addressed, is identifying what your intended scope is here. If the goal is to deprecate runserver entirely then I agree with you. But that intention hasn't been made clear, and what we're left with is a crippled webserver with a bit of an identity crisis.

At the time Django started there where not many perfectly fine alternatives (if at all) to easily run a python webapp during development. And just because something is already there, does not mean it should be used as an argument for adding more stuff to it. And I am not really sure about the identity crisis, it does what it is supposed to easy and well during dev.

Cheers,
Florian

[Steven Berry]

Florian,

I was merely presenting a rhetorical question to illustrate a point. Perhaps there weren't other available web servers then, but there are now, as has been pointed out. Prevalence and expectation of TLS/SSL is another thing that's changed since 2005. We can choose to maintain runserver and keep it up to date with modern standards, abandon it, or leave it alone to effectively abandon it.

So far we've demonstrated that we have an interest in maintaining runserver supported by:

• The existence of https://code.djangoproject.com/ticket/21978
• The fact that runserver has not been deprecated (nor are there any warnings of future deprecation)
  

We also have demonstrated that we have no interest in maintaining runserver supported by the apparent direction of this discussion.

I'm fine with either case, but therein lies the so-called identity crises and the origination of my confusion.

As a developer using Django, I've long been directed toward runserver for development in the documentation yet it falls short of modern expectations. Which brings me back to my current, fundamental point.

We should update runserver or update the documentation to more clearly address modern, third-party alternatives and how to resolve them with desirable development features (e.g. serving static files from app directories).

I am willing to put in the work in any case, but I cannot do so without a decision and a ticket.

At any rate, Gert, thank you for the gunicorn suggestion from Saturday. This meets my needs after some adjustment to my wsgi module (I needed to include my requisite sys.path modifications so I moved them from my passenger_wsgi module).

Regards,

Steven

[Steven Berry]

Pull request https://github.com/django/django/pull/4633/ withdrawn.

Closing comments attached to the issue describing my interim (and likely permanent) fix to all of my Django sites after taking Gert's suggestion.

I'll likely create a few blog posts detailing deployment processes instead of pestering/campaigning for official documentation updates.

Regards,

Steven