Django App Hosted by Mod_wsgi express unreachable over 3G/4G

[Kyle Handy]

Hello,

I've been developing a Django REST API that supports an iPhone application's data needs and user tracking. We have been using mod_wsgi express to host the API. When working over HTTP the API works great, but we have recently configured the HTTPS portion of the server and it we cannot reach the server over 3G/4G internet connections. 

The server responds to the following HTTPS requests:

PC browser request on Wi-Fi

PC browser request on landline.

iPhone device request over Wi-Fi

The server DOES NOT even RECEIVE the following HTTPS requests:

iPhone device request over 3G

iPhone device request over 4G

Is there any special configuration or extra accommodation that is required by Django or the mod_wsgi module in order to support requests over 3G/4G via HTTPS?

Any help is appreciated.

[Graham Dumpleton]

What is the command line arguments you are giving to mod_wsgi-express?

It would need to be something like:

mod_wsgi-express start-server --https-port 8443 --server-name ssl.example.com --ssl-certificate-file server.crt --ssl-certificate-key-file server.key

You could also use the all in one combined '--ssl-certificate' option, or would have to if on older mod_wsgi-express version. That variant of the option would be specified as '--ssl-certificate server'. That is, the common base name of the SSL certificate and key file.

Very important is that you must use the '--server-name' option and it should match the host name allowed by the SSL certificate and must match the host name appearing in the URL used to access the site.

Also suggest adding the option '--access-log'. This will turn on request access logging and should show whether requests even reach the server, even if rejected. If they don't even reach the server, then would take it as being an external routing issue.

To validate there are no complaints from Apache on startup about the SSL certificate, check the error log. To be sure you can also add '--startup-log' and check the startup log for any extra error messages in case not being logged to the error log.

Graham

[Graham Dumpleton]

One more point. If it is necessary for the site to be accessible by multiple host names, then you need to use the '--server-alias other-hostname' option to say what the additional host names are. These would generally have to be sub domains under your same parent domain and you would need some sort of wildcard SSL certificate for your parent domain to avoid issues with complaints about SSL certificate not matching.

Graham

[Jason Garber]

nginx is a very easy to configure and very reliable web server that is perfectly suited for ssl termination, serving static assets, and proxing the dynamic requests to mod_wsgi.

We use it with a great deal of convenience and success.  While this isn't an answer to your question it is a way to solve the same issues and get a lot in return.

Thanks!

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[Graham Dumpleton]

Hi Jason

As far as using nginx as the front end, you may be interested in some new features in mod_wsgi that got added in the last few versions.

What has been added is directives for mod_wsgi which allow it to handle all the possible proxy headers that might get passed through to indicate real client IP, whether HTTPS used in front end etc.

The mechanism implemented in mod_wsgi is probably going to be a lot more robust than any WSGI middleware out there as you can specify trust relationships so that headers for requests coming via something other than your trusted proxy can be ignored.

So if using nginx and you have:

    proxy_set_header X-Real-IP  $remote_addr;

    proxy_set_header X-Forwarded-For $remote_addr;

    proxy_set_header Host $host;

In mod_wsgi you could then use either:

    WSGITrustedProxyHeaders X-Forwarded-For

or:

    WSGITrustedProxyHeaders X-Real-IP

The code in mod_wsgi is smart enough to know that since you are flagging these that your intent is that REMOTE_ADDR in WSGI environ be overridden.

Since two headers were supplied that can be used for the same thing, mod_wsgi will remove the header that wasn't used so that if the value were different, that a WSGI middleware in some framework wouldn't then override things with potentially wrong information.

For example, if instead you had said just:

    proxy_set_header X-Real-IP  $remote_addr;

    proxy_set_header Host $host;

and:

    WSGITrustedProxyHeaders X-Real-IP

any X-Forwarded-For header would be removed so that couldn't be forged by a client and be overridden by a WSGI framework.

The other thing you can do is:

    proxy_set_header X-Real-IP  $remote_addr;

    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_set_header Host $host;

In this case if use:

    WSGITrustedProxyHeaders X-Forwarded-For

with, the value of X-Forwarded-For being a potentially comma separated list of IPs, normally the first in that list would be used.

That value is technically completely un-trustable though.

To combat that and the whole issue of the request coming via a proxy or not, you can also give a list of IP or IP network masks:

    WSGITrustedProxies 127.0.0.1 10.0.0.0/24

for the trusted location of any proxies.

Any proxy type headers will only be trusted and removed otherwise if not coming from a trusted proxy.

In the case of getting a value like:

    1.2.3.4, 5.6.7.8, 10.0.0.255

where it can from proxy at 127.0.0.1, then the headers will be trusted, and when looking at the value of X-Forwarded-For in particular, it will use the last value appearing in the list prior to a listed trusted proxy.

Thus in this case for REMOTE_ADDR it would use 5.6.7.8 and not 1.2.3.4. So the IP for the immediate client before your trusted proxy is used and not the very first in the list which can be forged.

As well as headers used in setting up REMOTE_ADDR, there is also code for trusting values from headers corresponding to WSGI equivalents of:

static const char *wsgi_proxy_client_headers[] = {

    "HTTP_X_FORWARDED_FOR",

    "HTTP_X_REAL_IP",

    NULL,

};

static const char *wsgi_proxy_scheme_headers[] = {

    "HTTP_X_FORWARDED_HTTPS",

    "HTTP_X_FORWARDED_PROTO",

    "HTTP_X_FORWARDED_SCHEME",

    "HTTP_X_FORWARDED_SSL",

    "HTTP_X_HTTPS",

    "HTTP_X_SCHEME",

    NULL,

};

static const char *wsgi_proxy_host_headers[] = {

    "HTTP_X_FORWARDED_HOST",

    "HTTP_X_HOST",

    NULL,

};

static const char *wsgi_proxy_script_name_headers[] = {

    "HTTP_X_SCRIPT_NAME",

    "HTTP_X_FORWARDED_SCRIPT_NAME",

    NULL,

};

There are also some standalone one as well where usually only one header used for it:

            else if (!strcmp(name, "HTTP_X_FORWARDED_SERVER")) {

                if (value) {

                    /* Use the value as is. */

                    apr_table_setn(r->subprocess_env, "SERVER_NAME", value);

                }

            }

            else if (!strcmp(name, "HTTP_X_FORWARDED_PORT")) {

                if (value) {

                    /* Use the value as is. */

                    apr_table_setn(r->subprocess_env, "SERVER_PORT", value);

                }

            }

So if using nginx from SSL termination, can use:

    WSGITrustedProxyHeaders X-Fowarded-Scheme

if that is what you are using to let backend know that front end is actually accepting HTTPS and so wsgi.url_scheme needs to be updated.

The code in mod_wsgi knows the types of values to expect based on what header you said to use:

            else if (!strcmp(name, "HTTP_X_FORWARDED_PROTO") ||

                !strcmp(name, "HTTP_X_FORWARDED_SCHEME") ||

                !strcmp(name, "HTTP_X_SCHEME")) {

                match_scheme_header = 1;

                if (value) {

                    trusted_scheme_header = name;

                    /* Value can be either 'http' or 'https'. */

                    if (!strcasecmp(value, "https"))

                        apr_table_setn(r->subprocess_env, "HTTPS", "1");

                    else if (!strcasecmp(value, "http"))

                        apr_table_unset(r->subprocess_env, "HTTPS");

                }

            }

            else if (!strcmp(name, "HTTP_X_FORWARDED_HTTPS") ||

                     !strcmp(name, "HTTP_X_FORWARDED_SSL") ||

                     !strcmp(name, "HTTP_X_HTTPS")) {

                match_scheme_header = 1;

                if (value) {

                    trusted_scheme_header = name;

                    /*

                     * Value can be a boolean like flag such as 'On',

                     * 'Off', 'true', 'false', '1' or '0'.

                     */

                    if (!strcasecmp(value, "On") ||

                        !strcasecmp(value, "true") ||

                        !strcasecmp(value, "1")) {

                        apr_table_setn(r->subprocess_env, "HTTPS", "1");

                    }

                    else if (!strcasecmp(value, "Off") ||

                        !strcasecmp(value, "false") ||

                        !strcasecmp(value, "0")) {

                        apr_table_unset(r->subprocess_env, "HTTPS");

                    }

                }

            }

As before, any equivalent header which wasn't the one you said to trust is removed from WSGI environ so some WSGI middleware doesn't decide to use it and so pick up an untrusted header.

Anyway, thought this may interest you. You can find comments about it in:

    http://modwsgi.readthedocs.org/en/master/release-notes/version-4.4.9.html

    http://modwsgi.readthedocs.org/en/master/release-notes/version-4.4.10.html

There are options in mod_wsgi-express exposing the feature there as well.

Graham

[Kyle Handy]

Graham,

I've been using all of the command arguments you've suggested, with a couple additional ones

   mod_wsgi-express start-server --host 0.0.0.0 --port 8000 --https-port 8001 --ssl-certificate /path/to/cert/and/key --server-name FQDN --startup-log --access-log --log-to-terminal

When running I can access the server, again on the local network only, and their are no SSL warnings: the connection is secure. Thanks to the --access-log option I can see the request come in and the status code of the response checks out as it should be.

There is only one warning that catches my eye in the startup log:

[Mon Mar 23 21:37:47.275863 2015] [ssl:warn] [pid 24357:tid 139801017251648] AH01909: 0.0.0.0:8001:0 server certificate does NOT include an ID which matches the server name

I've followed this warning message to this serverfault post which instructs to check the CN of the cert to make sure it matches the server-name; it does.

I do not see this as a probable cause for our issue connecting from outside the local network.

More on the certificate....

Our cert was signed via InCommon CA, and the response from InCommon contained multiple links to different cert files. 

1. PKCS#7 Base64 encoded
2. PKCS#7 Bin encoded
3. X509 Base64 encoded
4. X509 Certificate only, Base64 encoded
5. X509 Intermediates/root only, Base64 encoded
   
6. X509 Intermediates/root only Reverse, Base64 encoded

Through multiple trial/error we found that the only successful certificate file for mod_wsgi-express is the X509 Certificate only, Base64 encoded.

Maybe the omitted component in choosing the certificate only could have led us to this issue?

Thanks for replying so quickly,

Kyle

[Graham Dumpleton]

On 25/03/2015, at 3:53 AM, Kyle Handy <kyleha...@gmail.com> wrote:

Graham,

I've been using all of the command arguments you've suggested, with a couple additional ones

   mod_wsgi-express start-server --host 0.0.0.0 --port 8000 --https-port 8001 --ssl-certificate /path/to/cert/and/key --server-name FQDN --startup-log --access-log --log-to-terminal

I don't think it will make an difference, but can you try without '--host 0.0.0.0'. The configuration should allow external clients to connect by default. The '--host' option should only be used if want to limit to listening on a specific device interface.

For --server-name, does the server name used in URL match that exactly?

When running I can access the server, again on the local network only, and their are no SSL warnings: the connection is secure. Thanks to the --access-log option I can see the request come in and the status code of the response checks out as it should be.

There is only one warning that catches my eye in the startup log:

[Mon Mar 23 21:37:47.275863 2015] [ssl:warn] [pid 24357:tid 139801017251648] AH01909: 0.0.0.0:8001:0 server certificate does NOT include an ID which matches the server name

You may see this warning because the generated configuration creates a dummy VirtualHost entry with no ServerName to act as catch all so that arbitrary name based virtual host access isn't handled.

In other words, configuration will only accept requests against site set by --server-name or --server-alias and all others should see Forbidden returned. At least for HTTP would get Forbidden. For HTTPS I will need to check again what happens if Host header doesn't match specified server name. I have seen cases where when I have stuffed things up for HTTPS that it just didn't connect or hung, which sounds sort of similar.

I will do some more checking on this.

I've followed this warning message to this serverfault post which instructs to check the CN of the cert to make sure it matches the server-name; it does.

I do not see this as a probable cause for our issue connecting from outside the local network.

More on the certificate....

Our cert was signed via InCommon CA, and the response from InCommon contained multiple links to different cert files. 

1. PKCS#7 Base64 encoded
2. PKCS#7 Bin encoded
3. X509 Base64 encoded
4. X509 Certificate only, Base64 encoded
5. X509 Intermediates/root only, Base64 encoded
   
6. X509 Intermediates/root only Reverse, Base64 encoded

Through multiple trial/error we found that the only successful certificate file for mod_wsgi-express is the X509 Certificate only, Base64 encoded.

Maybe the omitted component in choosing the certificate only could have led us to this issue?

So what you need are two files, the SSL certificate file, often using a .crt extension. And the SSL certificate key file, often using a .key extension.

Rather than use --ssl-certifcate to try and refer to both at same time, you can use separate options to be explicit if using recent mod_wsgi-express version.

--ssl-certificate-file some/path/server.crt --ssl-certificate-key-file some/path/server.key

If what you were given was a .pem file it from memory has both the certificate and key in the one file. I have generally found I have had to extract the .crt and .key files separately from that to get it to work.

I always have to Google for the steps to do that.

Anyway, let me do that checking I mentioned.

If you can at least confirm that the server name in URL used matches --server-name, and that the request isn't passing through any intermediary proxies, although since you are using HTTPS that is probably unlikely.

Graham

Thanks for replying so quickly,

Kyle

On Saturday, March 21, 2015 at 1:54:24 AM UTC-7, Graham Dumpleton wrote:

On 21/03/2015, at 10:13 AM, Kyle Handy <kyleha...@gmail.com> wrote:

> Hello,
>
> I've been developing a Django REST API that supports an iPhone application's data needs and user tracking. We have been using mod_wsgi express to host the API. When working over HTTP the API works great, but we have recently configured the HTTPS portion of the server and it we cannot reach the server over 3G/4G internet connections.
>
> The server responds to the following HTTPS requests:
>
> PC browser request on Wi-Fi
> PC browser request on landline.
> iPhone device request over Wi-Fi
>
> The server DOES NOT even RECEIVE the following HTTPS requests:
>
> iPhone device request over 3G
> iPhone device request over 4G
>
> Is there any special configuration or extra accommodation that is required by Django or the mod_wsgi module in order to support requests over 3G/4G via HTTPS?
>
> Any help is appreciated.

What is the command line arguments you are giving to mod_wsgi-express?

It would need to be something like:

    mod_wsgi-express start-server --https-port 8443 --server-name ssl.example.com --ssl-certificate-file server.crt --ssl-certificate-key-file server.key

You could also use the all in one combined '--ssl-certificate' option, or would have to if on older mod_wsgi-express version. That variant of the option would be specified as '--ssl-certificate server'. That is, the common base name of the SSL certificate and key file.

Very important is that you must use the '--server-name' option and it should match the host name allowed by the SSL certificate and must match the host name appearing in the URL used to access the site.

Also suggest adding the option '--access-log'. This will turn on request access logging and should show whether requests even reach the server, even if rejected. If they don't even reach the server, then would take it as being an external routing issue.

To validate there are no complaints from Apache on startup about the SSL certificate, check the error log. To be sure you can also add '--startup-log' and check the startup log for any extra error messages in case not being logged to the error log.

Graham