How to get route_url() and static_url() to work from a Celery process?

[Sean Hammond]

Hi,

When I call certain Pyramid methods like route_url() or static_url() from a Celery task they return the wrong URLs because they don't have the hostname and port that'd normally come from the HTTP request's Host header.

I've found that I can fix it by setting an HTTP_HOST envvar in my OS environment and then copying that envvar into the WSGI environment: `request.environ["HTTP_HOST"] = os.environ["HTTP_HOST"]`. This seems to work but I'm not sure if it's the right/best thing to do?

Details:

I run Celery like this:

celery -A lms.tasks.celery:app worker --loglevel=INFO

Link: https://github.com/hypothesis/lms/blob/002afc26750f83faa5b1a7d7781817da9a2ba260/conf/supervisord-dev.conf#L20

Then in my celery.py file I set up an artificial Pyramid request like this:

from contextlib import contextmanager
from pyramid.scripting import prepare
from lms.app import create_app

lms = create_app(None, **{})

@contextmanager
def request_context():
with prepare(registry=lms.registry) as env:
yield env["request"]

sender.app.request_context = request_context

Link: https://github.com/hypothesis/lms/blob/002afc26750f83faa5b1a7d7781817da9a2ba260/lms/tasks/celery.py#L75-L95

Each Celery task then does this to get an artificial Pyramid request:

with app.request_context() as request:
...

Example: https://github.com/hypothesis/lms/blob/002afc26750f83faa5b1a7d7781817da9a2ba260/lms/tasks/mailchimp.py#L20-L25

The problem is that if any code called by one of these Celery tasks uses request.route_url() or request.static_url() it's getting URLs without the right host and port.

One way I've found to fix this is to set an HTTP_HOST envvar in the OS environment and then inject it into the WSGI environment like this:

@contextmanager
def request_context():
with prepare(registry=lms.registry) as env:
request = env["request"]

# Make Pyramid things like route_url() and static_url() use the
# right hostname and port when called by Celery tasks.
request.environ["HTTP_HOST"] = os.environ["HTTP_HOST"]

yield request

See https://github.com/hypothesis/lms/pull/5657/. This works but I'm not sure whether it's the best/right thing to do?

Thanks!

[Mike Orr]

That's a perennial issue with Pyramid: getting things like URLs or
database connections outside a traditional request, like at startup or
in scripts or in a function where the request is several layers away.
I'm not an expert on Celery or specific environmental vars, but since
Pyramid's request is ultimately based on the WSGI environment, it
would make sense to fill things in from the corresponding environment
variables if it's not happening automatically. I'd just look at the
request object's scope to make sure it doesn't leak the data to
someplace undesired or where it's inaccurate.

> --
> You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pylons-discus...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/pylons-discuss/5b7c52a2-4cd0-4887-916d-87480feed124%40app.fastmail.com.



--
Mike Orr <slugg...@gmail.com>