Packaging django + celery + gunicorn with pex?

[Hari Kumar]

Hello folks,

I was wondering if any of you are using pex to package a full blown django + celery project? I'm not sure if this is even a proper use case, so I'll explain what I'm currently doing.

As we are not allowed to install any packages during deployment, we need an self-contained artifact built on Jenkins. For now, I create a virtualenv, pip install requirements and tar up the virtualenv, django and scripts directories. The scripts directory has scripts to configure, start and stop the app. As part of start logic, I start gunicorn and celery processes, and for this I rely on gunicorn and celery to be in virtualenv's bin directory. I'm trying to figure out if I could use pex to make this process cleaner.

~ Hari

[Kris Wilson]

definitely - this is a common use case for us. what we'd typically do in this situation would be to build a single pex that contained gunicorn, celery, django and the app itself and any of its bundled resources (single file deploy) and then run that on mesos/aurora.

from there, you'd make a small entrypoint handler `main.py` (the `source=` in a `python_binary`) that handles taking a command and inputs to pass to the appropriate 3rdparty entrypoint. this essentially just facilitates the idea of a 'multi-entrypoint pex'.

e.g. for celery, a small entrypoint shim to the normal `celery` cli script might look like:

from celery.bin.celery import main as celery_main

from twitter.common import app

@app.command

def celery(args, options):

  """Wrap the celery cli tool."""

  celery_args = ['celery'] + args

  return celery_main(celery_args)

app.main()

which would let you do:

./the.pex celery -arg -arg arg

just as if you were just calling `celery` in the virtualenv:

celery -arg -arg -arg

then you'd do something similar for the `gunicorn` bit. we use an internal WSGI library for this that has a number of pluggable backends and can run any WSGI-compatible application (e.g. any flask or django app) and wires up a number of options for controlling the server runtime from within the pex.

once your key entrypoints are plumbed, then your deploy would look something like:

./the.pex celery -arg -arg -arg

./the.pex gunicorn start -arg -arg -arg

you can also pass arbitrary entrypoints to any pex at runtime via `PEX_MODULE='my.lib:main' ./the.pex` to play around with the concept of a multi-entrypoint pex.

[Hari]

Thanks for the pointers, Kris. I have a main module with multiple commands are you suggested. However, I'm unable to make this main module as an entry point. The simple example described in https://www.ovni.io/post/pex-the-python-executable does not work for me due to module import issues. Any idea what am I doing wrong?

Here is what I have-

dev % tree hello
hello
├── MANIFEST.in
├── __init__.py
├── hello.py
└── setup.py

0 directories, 4 files

dev % cat hello/MANIFEST.in
include *.py

dev % cat hello/hello.py
def hello():
    print "hello!"

dev % cat hello/setup.py
from setuptools import setup, find_packages
setup( name='hello', version='0.0.1', packages=find_packages() )

dev % pex hello -e hello:hello -o hello.pex
dev % ./hello.pex
Traceback (most recent call last):
  File ".bootstrap/_pex/pex.py", line 328, in execute
  File ".bootstrap/_pex/pex.py", line 260, in _wrap_coverage
  File ".bootstrap/_pex/pex.py", line 292, in _wrap_profiling
  File ".bootstrap/_pex/pex.py", line 371, in _execute
  File ".bootstrap/_pex/pex.py", line 429, in execute_entry
  File ".bootstrap/_pex/pex.py", line 443, in execute_pkg_resources
  File ".bootstrap/pkg_resources/__init__.py", line 2208, in resolve
ImportError: No module named hello

-- Hari

[John Sirois]

On May 12, 2016 5:22 PM, "Hari" <harip...@gmail.com> wrote:
>
> Thanks for the pointers, Kris. I have a main module with multiple commands are you suggested. However, I'm unable to make this main module as an entry point. The simple example described in https://www.ovni.io/post/pex-the-python-executable does not work for me due to module import issues. Any idea what am I doing wrong?
>
> Here is what I have-
>
> dev % tree hello
> hello
> ├── MANIFEST.in
> ├── __init__.py
> ├── hello.py
> └── setup.py
>
> 0 directories, 4 files
>
> dev % cat hello/MANIFEST.in
> include *.py
>
> dev % cat hello/hello.py
> def hello():
>     print "hello!"
>
> dev % cat hello/setup.py
> from setuptools import setup, find_packages
> setup( name='hello', version='0.0.1', packages=find_packages() )
>
> dev % pex hello -e hello:hello -o hello.pex

Try `pex . -e hello:hello -o hello.pex`

IE: s/pex hello .../pex . .../

[Hari]

No luck. Thanks for the reply though.

dev % pex . -e hello:hello -o hello.pex

Traceback (most recent call last):

  File "/usr/local/bin/pex", line 11, in <module>
    sys.exit(main())
  File "/Library/Python/2.7/site-packages/pex/bin/pex.py", line 509, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/Library/Python/2.7/site-packages/pex/bin/pex.py", line 457, in build_pex
    resolvables = [Resolvable.get(arg, resolver_option_builder) for arg in args]
  File "/Library/Python/2.7/site-packages/pex/resolvable.py", line 61, in get
    raise cls.InvalidRequirement('Unknown requirement type: %s' % resolvable_string)
pex.resolvable.InvalidRequirement: Unknown requirement type: .


dev % cd hello

hello % pex . -e hello:hello -o hello.pex

hello % ./hello.pex

Traceback (most recent call last):
  File ".bootstrap/_pex/pex.py", line 328, in execute
  File ".bootstrap/_pex/pex.py", line 260, in _wrap_coverage
  File ".bootstrap/_pex/pex.py", line 292, in _wrap_profiling
  File ".bootstrap/_pex/pex.py", line 371, in _execute
  File ".bootstrap/_pex/pex.py", line 429, in execute_entry
  File ".bootstrap/_pex/pex.py", line 443, in execute_pkg_resources
  File ".bootstrap/pkg_resources/__init__.py", line 2208, in resolve
ImportError: No module named hello

-- Hari

[John Sirois]

Pex does not read `MANIFEST.in` iirc, can you update setup.py to include `packages=['.']` as an alternative?

[Hari]

Thanks, that worked.

Regarding the celery example which @Kris gave, it seems like twitter.common.python has moved to pex. Is there an equivalent of twitter.common.app in pex?

-- Hari

[Kris Wilson]

no, not in pex - but available on pypi: https://pypi.python.org/pypi/twitter.common.app

and for the record, you don't have to use twitter.common.app - things like argparse' sub-commands etc should accomplish the same thing - that's just what we primarily use in-house.

[Hari]

Got it. Thank you.

-- Hari