Compiling/packing Django to one binary

800 views
Skip to first unread message

Александр Христюхин

unread,
Oct 18, 2016, 4:29:11 PM10/18/16
to Django users

Hi!


I would like to ask community about methods of shipping Django in production.


I do have some limitations, one of which is I don't know what packages are installed on production server (for example, postgres dev libraries or specific version of python).


Right now my method is to create venv with specific version of Python, make venv relocatable, copy missing libraries into venv/lib, pack it into archive and ship to production servers.


On application launch server has to unpack archive and run it somewhat like that:

$ LD_LIBRARY_PATH=venv/lib venv/bin/python venv/bin/gunicorn -c gunicorn.conf myapp.wsgi:application


In my case LD_LIBRARY_PATH is required for libpython and libpq (PostgreSQL query library).


This does work, but I have to go through a lot of stuff.


What I want to do is pack my application with specific python and libraries into one binary (on build server) and only ship this binary to production server. So application launch will look somewhat like that:

$ my-awesome-python-bin gunicorn -c gunicorn.conf myapp.wsgi:application


Or if I could go even further and describe some startup logic:


def startup():
    ...
    args = argparser.parse()
    config = args.config
    gunicorn.server(config, myapp.wsgi, 'application').run()

...and then:

$ my-awesome-bin -c myapp.conf


Is there any way of doing that?

Or could you suggest any easier (well, more convenient) way of shipping Django?


I did ask practically the same question on StackOverflow couple weeks ago, but none of given answers provide simple solution for my problem. I've tried nuitka, pyinstalled and cx_freeze with no success.

Asad Jibran Ahmed

unread,
Oct 18, 2016, 4:40:24 PM10/18/16
to django...@googlegroups.com
Hi,
 While I haven't personally dealt with such a situation with my Django code, I have worked with people who had this issue (un-deterministic production servers). They used Docker to solve this issue. Essentially you're looking for deterministic deploys, and Docker is the first thing that jumps to my mind for something like this.

Is that a possibility that you can consider? If not, then I'm sure there will be better answers to your question from other people in this group.
Regards,


--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/51a3ef5f-800a-48db-9697-03d069238834%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Antonis Christofides

unread,
Oct 19, 2016, 2:22:11 AM10/19/16
to django...@googlegroups.com

I do have some limitations, one of which is I don't know what packages are installed on production server (for example, postgres dev libraries or specific version of python).

Hi,

What do you know about the production server? Do you know the operating system? Can you become a superuser? If not, how are you going to configure the web server? If yes, why can't you install the packages you want?

Regards,

Antonis Christofides
djangodeployment.com
--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

Jani Tiainen

unread,
Oct 19, 2016, 5:41:23 AM10/19/16
to django...@googlegroups.com

Hi,


Like someone else already pointed out, Docker containers are quite powerful feature to do deterministic deployments to unix-like machines. Only pre-requirement is to have Docker running on target machine but otherwise you're free to build your containers as you wish. This also eliminates quite nicely "works for me" problems.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/51a3ef5f-800a-48db-9697-03d069238834%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Jani Tiainen

GMail

unread,
Oct 19, 2016, 8:41:45 AM10/19/16
to django...@googlegroups.com
Thanks for replies!

I'm aware of Docker solution, but it's not quite what I'm looking for.
I don't have any control of a production server, I don't have sudo and I can't install any packages. It also concerns Docker, since it could be not installed.
And yes, OS is always Ubuntu Linux in my case, but version is not fixed, so two servers could have different Ubuntu versions.

I'm looking for something universal, like Go programs - you get one binary with all libraries included. It seems to me that utils like cx_freeze or py2exe do that, but they don't work with Django, since you need to expose WSGI application to WSGI server (gunicron/uWSGI/etc).

Like I said in first letter, virtualenv does provide a solution for this setup, but there's too much work to be done and you have to explicitly set LD_LIBRARY_PATH on production server.

Avraham Serour

unread,
Oct 19, 2016, 9:03:13 AM10/19/16
to django-users
using a virtualenv is common and suggested, but don't need to mess with LD_LIBRARY_PATH 

you can create the virtualenv in your home directory or the application directory and run the python executable from the virtualenv, no need to touch LD_LIBRARY_PATH 



To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

-- 
Jani Tiainen

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/35FCB771-1696-4C24-B5AB-ADB862EF701B%40gmail.com.

GMail

unread,
Oct 19, 2016, 9:18:49 AM10/19/16
to django...@googlegroups.com
In fact I do have to use LD_LIBRARY_PATH. Mostly because of libpq5.
And again, I have no control over production server, I don't even know in which directory my virtualenv will land.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-users.

Antonis Christofides

unread,
Oct 19, 2016, 10:37:17 AM10/19/16
to django-users
Sorry, you say you need to set LD_LIBRARY_PATH because of libpq5 (I guess you are compiling it yourself). So far so good. But what does this have to do with virtualenv? You need to set LD_LIBRARY_PATH irrespective of whether you use virtualenv or not. Have I misunderstood something?
> >>> I did ask practically the same question on StackOverflow <http://stackoverflow.com/questions/39913847/is-there-a-way-to-compile-python-application-into-static-binary/> couple weeks ago, but none of given answers provide simple solution for my problem. I've tried nuitka, pyinstalled and cx_freeze with no success.
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google Groups "Django users" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com <mailto:django-users...@googlegroups.com>.
> >>> To post to this group, send email to django...@googlegroups.com <mailto:django...@googlegroups.com>.
> >>> Visit this group at https://groups.google.com/group/django-users <https://groups.google.com/group/django-users>.
> >>> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/51a3ef5f-800a-48db-9697-03d069238834%40googlegroups.com <https://groups.google.com/d/msgid/django-users/51a3ef5f-800a-48db-9697-03d069238834%40googlegroups.com?utm_medium=email&utm_source=footer>.
> >>> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
> >>
> >> --
> >> Jani Tiainen
> >>
> >> --
> >> You received this message because you are subscribed to the Google Groups "Django users" group.
> >> To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com <mailto:django-users...@googlegroups.com>.
> >> To post to this group, send email to django...@googlegroups.com <mailto:django...@googlegroups.com>.
> >> Visit this group at https://groups.google.com/group/django-users <https://groups.google.com/group/django-users>.
> >> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/2d22ef8d-57ff-687b-d355-b4cdaf7c4f4d%40gmail.com <https://groups.google.com/d/msgid/django-users/2d22ef8d-57ff-687b-d355-b4cdaf7c4f4d%40gmail.com?utm_medium=email&utm_source=footer>.
> >> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups "Django users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com <mailto:django-users...@googlegroups.com>.
> > To post to this group, send email to django...@googlegroups.com <mailto:django...@googlegroups.com>.
> > Visit this group at https://groups.google.com/group/django-users <https://groups.google.com/group/django-users>.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/35FCB771-1696-4C24-B5AB-ADB862EF701B%40gmail.com <https://groups.google.com/d/msgid/django-users/35FCB771-1696-4C24-B5AB-ADB862EF701B%40gmail.com?utm_medium=email&utm_source=footer>.
> >
> > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups "Django users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com <mailto:django-users...@googlegroups.com>.
> > To post to this group, send email to django...@googlegroups.com <mailto:django...@googlegroups.com>.
> > Visit this group at https://groups.google.com/group/django-users <https://groups.google.com/group/django-users>.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CAFWa6tJk7u0k3%3De5Vww_FevZKCLhQFMmDc2fV2_jGov3613tHg%40mail.gmail.com <https://groups.google.com/d/msgid/django-users/CAFWa6tJk7u0k3%3De5Vww_FevZKCLhQFMmDc2fV2_jGov3613tHg%40mail.gmail.com?utm_medium=email&utm_source=footer>.
> > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
> To post to this group, send email to django...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-users.
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/FDF7452A-7204-48B2-9C41-F95C8AC9C387%40gmail.com.

GMail

unread,
Oct 19, 2016, 11:11:40 AM10/19/16
to django...@googlegroups.com
I'm not compiling libpq5, I'm just copying it to venv/lib so psycopg2 could work without dev postgres packages. And yes, it doesn't matter if I use virtualenv or not, this is just a workaround for this specific problem.

This is going a bit off topic though =/
> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/E1bwrzA-0005hR-Pj%40rmm6prod02.runbox.com.

Jani Tiainen

unread,
Oct 20, 2016, 2:15:46 AM10/20/16
to django...@googlegroups.com

Hi,

If that's truely the case that you don't control anything, deploying Django will be really a hard problem.

How do you can make sure that you can install all the libs your Django app, or any dependencies you may have are installed correctly?

How you do setup static file serving?

How you plan to run your Django app itself?

Also, how you can set LD_LIBRARY_PATH if you don't know if there is even libpq5 in the system, or is it even a correct version?

For more options, visit https://groups.google.com/d/optout.

-- 
Jani Tiainen

GMail

unread,
Oct 20, 2016, 6:41:35 AM10/20/16
to django...@googlegroups.com

How do you can make sure that you can install all the libs your Django app, or any dependencies you may have are installed correctly?

I'm installing dependencies in virtualenv.

How you do setup static file serving? 

I'm shipping Nginx binary and config along with app. Static files are collected and archived on build server, then shipped with app and Nginx.

How you plan to run your Django app itself?

I've posted this in my first message, currently I'm running Django with gunicorn with virtualenv's Python.

Also, how you can set LD_LIBRARY_PATH if you don't know if there is even libpq5 in the system, or is it even a correct version?

I'm not relying on system's libpq5, that's why I copy it to venv/lib and use LD_LIBRARY_PATH. So it would be loaded from venv/lib and not from the system.

To clarify, my current setup works in hostile environments where I could only rely on OS distro. What I'm looking for is alternative to virtualenv.


Jani Tiainen

unread,
Oct 20, 2016, 8:09:55 AM10/20/16
to django...@googlegroups.com



On 20.10.2016 13:39, GMail wrote:

How do you can make sure that you can install all the libs your Django app, or any dependencies you may have are installed correctly?

I'm installing dependencies in virtualenv.

How you do setup static file serving? 

I'm shipping Nginx binary and config along with app. Static files are collected and archived on build server, then shipped with app and Nginx.

What if system libraries are not the same as yours, you're not shipping all dependencies, including distribution libs as well?

How you plan to run your Django app itself?

I've posted this in my first message, currently I'm running Django with gunicorn with virtualenv's Python.

Also, how you can set LD_LIBRARY_PATH if you don't know if there is even libpq5 in the system, or is it even a correct version?

I'm not relying on system's libpq5, that's why I copy it to venv/lib and use LD_LIBRARY_PATH. So it would be loaded from venv/lib and not from the system.

To clarify, my current setup works in hostile environments where I could only rely on OS distro. What I'm looking for is alternative to virtualenv.


Well it may work or it may not, provided that all libraries that your system expects are there and are actually ABI compatible. So you make quite a lot of assumptions here.

There exist bunch of alternatives like LXC based solutions, LXD, snapd, and various virtual machine mechanism.

All of them has their pros and cons, but pretty much all of them require something on a target machine.


For more options, visit https://groups.google.com/d/optout.

-- 
Jani Tiainen

GMail

unread,
Oct 20, 2016, 8:46:21 AM10/20/16
to django...@googlegroups.com
What if system libraries are not the same as yours, you're not shipping all dependencies, including distribution libs as well?
When making virtualenv relocatable I pass --always-copy argument, so all libs are copied. Well, most of them, I manually copy some that aren't (like libpython).

As to VM - currently I'm using porto containers, but without images. Building images would be like building Docker containers.


Reply all
Reply to author
Forward
0 new messages