Best practices for running Pyramid (docker) and nginx (host) deployment

[Jens Troeger]

Hello,

I’ve seen some conversation here about running a Pyramid app server inside a Docker container, but none has really answered my questions.

My setup is that nginx runs on the host and currently uses proxy_pass to forward requests to the container’s external port, so that the requests are then processed and responded to by the Pyramid application running inside the container.

Question: Inside the container I’m running the Pyramid application using pserve which listens on the container’s mapped internal port. Should I switch to gunicorn instead? Does it matter in such a setup?

The proxy_pass URL is http://127.0.0.1:6543 which means that the external https gets lost. That, in turn, means that within the Pyramid app (inside of the container) calls to e.g. static_url() return a http route instead of the necessary & expected https.

Question: I currently use prefix WSGI middleware to rewrite responses (discussion) but that feels hacky. Unfortunately, I wasn’t able to make X-Forward-Proto HTTP header work quite yet so what’s the current recommendation here? Is the Using Behind a Reverse Proxy page current and working?

Question: Are there any benefits to using a UNIX socket for proxy_pass, instead of HTTP?

Much thanks in advance!

Jens

[Theron Luhn]

1)  pserve isn’t really comparable with gunicorn, its just a way to launch a server, such as gunicorn or waitress.  You’re probably using waitress, that’s what the Pyramid docs use.

I personally use gunicorn, but many on this mailing list are using waitress with success, so I think it’s a fine choice.

Since you’re using Docker, you might be interested in this containerized nginx proxy I built:  https://github.com/luhn/docker-gunicorn-proxy  (It’s called gunicorn-proxy, but should work with waitress too.)

2) I use X-Forwarded-Proto just fine; as far as I can tell that documentation is accurate.  You did type “X-Forward-Proto” in your email, so double check the spelling in your config :)

3) Unix sockets are supposedly more efficient than localhost because they don’t have to deal with that bothersome “internet” stuff.  It might shave a hair off your CPU usage.

— Theron

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/508f077e-ff7e-47c4-9e8f-ee5f018e9a7en%40googlegroups.com.

[Michael Merickel]

The linked waitress docs do work. I deploy waitress behind nginx and envoy and heroku regularly using the guides in there. Make sure to configure the trusted proxy settings and ensure the upstream is setting the right headers. 

- Michael

On Dec 13, 2021, at 18:22, Jens Troeger <jens.t...@gmail.com> wrote:

Hello,

--

[Andreas Kaiser]

If the other solutions don't work for you, you can also set url_scheme in your config, for example:

[server:main]
use = egg:waitress#main
port = 6543
url_scheme = https

HTH,
Andreas

[tonthon]

Hello,

I don't know about waitress.

I personnaly use gunicorn, gunicorn doesn't trust headers transmitted by a remote reverse proxy unless you tell him to through the --forwarded-allow-ips setting.

Maybe waitress has a similar setting ?

[Jens Troeger]

Thank you, everyone!

@Theron, yes I meant “forwardED”, just a typo. I’ll review the configuration and try again. Regarding the Docker link: my project has heaps of dependencies that I offloaded into another base image, so I think I won’t be able to build on top of yours.

@Michael, glad to know that waitress works in production. I’ll take a look at the docs once more to find out where things are hanging for me…

@Andreas, thank you for the url_scheme tip, if the X-Forwarded-Proto doesn’t work, I’ll try that one too.

Cheers,

Jens

[Jens Troeger]

Oh, and more question… Monitoring.

For example, to monitor Dramatiq’s asynchronous workers it provides a middleware for Prometheus to gather metrics. What’s does the community recommend for Pyramid to monitor its metrics? Didn’t see anything on Projects page or the Awesome Pyramid page.

Or is it better to hook up Prometheus to the nginx reverse proxy (link), which, however, in my case runs outside of the app container on the host 🤔

Much thanks,

Jens

[Steve Piercy]

Does Sentry monitor what you want?

https://docs.sentry.io/platforms/python/guides/pyramid/

--steve

On 12/19/21 4:22 PM, Jens Troeger wrote:
> Oh, and more question… Monitoring.
>

> For example, to monitor Dramatiq <https://dramatiq.io/>’s asynchronous workers it provides a middleware <https://dramatiq.io/reference.html#dramatiq.middleware.Prometheus> for Prometheus <https://prometheus.io/> to gather metrics. What’s does the community recommend for Pyramid to monitor its metrics? Didn’t see anything on Projects <https://pylonsproject.org/projects.html> page or the Awesome Pyramid <https://github.com/uralbash/awesome-pyramid> page.
>
> Or is it better to hook up Prometheus to the nginx reverse proxy (link <https://docs.nginx.com/nginx-ingress-controller/logging-and-monitoring/prometheus/>), which, however, in my case runs outside of the app container on the host 🤔
>
> Much thanks,
> Jens
>

> --
> You received this message because you are subscribed to the Google Groups "pylons-discuss" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com <mailto:pylons-discus...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/a3a27a0e-703f-4384-aec7-a89fc95137a1n%40googlegroups.com <https://groups.google.com/d/msgid/pylons-discuss/a3a27a0e-703f-4384-aec7-a89fc95137a1n%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Gerhard Schmidt]

Hi Jens,

you must edit setting at two places

in the nginx server definition you must add

proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host $host:$server_port;
proxy_set_header X-Forwarded-Port $server_port;

and in die pyramid ini file you must add to the server:main section

trusted_proxy = *
trusted_proxy_headers = x-forwarded-for x-forwarded-host
x-forwarded-proto x-forwarded-port

trusted_proxy = * should only be used in a container setting, as it
accepts these header from all host.

With these settings waitress knows everything about the original
connection and fills the values in the request object to fit the
original connection

Regards
Estartu

Am 14.12.21 um 01:21 schrieb Jens Troeger:

> Hello,
>
> I’ve seen some conversation here about running a Pyramid app server
> inside a Docker container, but none has really answered my questions.
>

> My setup is that nginx runs on the host and currently uses /proxy_pass/
> <https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass>

> to forward requests to the container’s external port, so that the
> requests are then processed and responded to by the Pyramid application
> running inside the container.
>

> *Question*: Inside the container I’m running the Pyramid application
> using pserve
> <https://docs.pylonsproject.org/projects/pyramid/en/latest/pscripts/pserve.html>

> which listens on the container’s mapped internal port. Should I switch

> to gunicorn <https://docs.gunicorn.org/en/stable/> instead? Does it

> matter in such a setup?
>

> The /proxy_pass/ URL is http://127.0.0.1:6543 which means that the

> external https gets lost. That, in turn, means that within the Pyramid
> app (inside of the container) calls to e.g. static_url()

> <https://docs.pylonsproject.org/projects/pyramid/en/latest/api/request.html#pyramid.request.Request.static_url>

> return a http route instead of the necessary & expected https.
>

> *Question*: I currently use prefix WSGI middleware to rewrite responses
> (discussion
> <https://github.com/Pylons/pyramid/issues/1435#issuecomment-61654089>)

> but that feels hacky. Unfortunately, I wasn’t able to make
> X-Forward-Proto

> <https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Forwarded-Proto> HTTP

> header work quite yet so what’s the current recommendation here? Is the
> Using Behind a Reverse Proxy

> <https://docs.pylonsproject.org/projects/waitress/en/latest/reverse-proxy.html>
> page current and working?
>
> *Question*: Are there any benefits to using a UNIX socket for
> /proxy_pass/, instead of HTTP?

>
> Much thanks in advance!
> Jens
>

> --
> You received this message because you are subscribed to the Google
> Groups "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to pylons-discus...@googlegroups.com
> <mailto:pylons-discus...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/pylons-discuss/508f077e-ff7e-47c4-9e8f-ee5f018e9a7en%40googlegroups.com
> <https://groups.google.com/d/msgid/pylons-discuss/508f077e-ff7e-47c4-9e8f-ee5f018e9a7en%40googlegroups.com?utm_medium=email&utm_source=footer>.

[silvi...@gmail.com]

Hi,

To monitor nginx from your host you can use telegraf with the nginx input (https://github.com/influxdata/telegraf/blob/release-1.20/plugins/inputs/nginx/README.md)

which will count as input and as output you have different options.

One is to expose a port on your hosts via the telegraf output (https://github.com/influxdata/telegraf/blob/release-1.20/plugins/outputs/prometheus_client/README.md) which means you

will have to configure your prometheus server to scrape it. You could as well remote write but you will need a prometheus configured to accept remote writes(not default)

Also you can use any other output plugins like influxdb etc.

To monitor your pyramid application you will need to instrument it yourself. Probably the best way to do it is via a tween

(https://docs.pylonsproject.org/projects/pyramid/en/2.0-branch/narr/hooks.html#creating-a-tween) and using 

https://github.com/prometheus/client_python

Not sure if you end up with gunicorn but this is an example how to expose metrics https://github.com/prometheus/client_python#multiprocess-mode-eg-gunicorn

Note there is also an example on how to use the push gateway from prometheus (you will need to deploy it)

cheers!

[Jonathan Vanasco]

On Monday, December 13, 2021 at 9:14:30 PM UTC-5 the...@luhn.com wrote:

1)  pserve isn’t really comparable with gunicorn, its just a way to launch a server, such as gunicorn or waitress.  You’re probably using waitress, that’s what the Pyramid docs use.

I personally use gunicorn, but many on this mailing list are using waitress with success, so I think it’s a fine choice.

I wish I saw this thread last month!

I personally use uWSGI.  waitress, gunicorn and uWSGI are **all** great application servers (as are some others) - however they each have their own sets of advantages/strengths and drawbacks/weaknesses across: concurrency, latency, cpu, ram, etc.

Depending on your exact application and traffic, you may get a significant performance boost by using one platform over the others. you may also see no discernible difference between the platform options. 

IMHO: as waitress is the default and production ready, you generally don't really need to consider other platforms until you need to scale into more than two nodes (redundancy is good), start to run into issues with capacity (concurrency, latency, cpu, ram, etc), or are using some sort of automatic scaling system to deploy more nodes. In those cases, waitress still might be the best option for you - but doing an audit and comparative benchmark of your applications' use of resources is warranted.