Compiling Python

[Wade Leftwich]

With the Pyramid docs in ebook form on my phone I tend to browse them
at odd times. Today, standing in line at the post office, I stopped at
this paragraph right near the beginning:
"""
It’s useful to use a Python interpreter that isn’t the “system” Python
interpreter to develop your software. The authors of Pyramid tend not
to use the system Python for development purposes; always a self-
compiled one. Compiling Python is usually easy, and often the “system”
Python is compiled with options that aren’t optimal for web
development.
"""
... and wondered what the best compilation options for web development
might be. I generally use whatever comes with Ubuntu, which seems to
work OK. Does anyone have config tips to share?

[Mike Orr]

Where does it say this? I've never heard of it. I always use the
system Python if it's the right version, but with a virtualenv.

However, it's correct that compiling Python is easy, (except on
Windows where a compiler doesn't come with the OS). For Ubuntu:

$ sudo apt-get install build-essential
$ cd Python-2.7
$ ./configure --prefix=/usr/local/bin
$ make
$ make install

But I don't know what nonstandard options would be "better for web development".

--
Mike Orr <slugg...@gmail.com>

[Mariano Mara]

Quoting Mike Orr (2012-06-12 22:49:36)

Just a little recommendation that I can _barely_ recall it gave me trouble in
the past: verify that you have the developer headers for sqlite3 and zlib,
otherwise you won't have access to sqlite and, regarding zlib, some third party
library might complain (such as PIL) while compiling.

[Charlie Meyer]

on ubuntu,

sudo apt-get build-dep python2.7

Charlie Meyer

Department of Computer Science
University of Illinois at Urbana-Champaign
MC/258
201 N. Goodwin Ave
Urbana, IL 61801



cha...@charliemeyer.net
http://www.charliemeyer.net

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.
To post to this group, send email to pylons-...@googlegroups.com.
To unsubscribe from this group, send email to pylons-discus...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.

[Marius Gedminas]

libreadline-dev is also something I would want to have if I ever had to
build my own Python. apt-get build-dep is a very good idea on Ubuntu.

Personally, I trust Ubuntu's Python more than a self-compiled one, but I
don't have good arguments to support that position, and I might even be
*gasp!* mistaken.

Marius Gedminas
--
"What's the name of the new OO COBOL -- an equivalent of C++?"
"ADD 1 TO COBOL GIVING COBOL"

[Benjamin Sims]

I've had to build my own before on servers which had old versions as default (*cough* Red Hat *cough*) and it comes in very handy as long as you combine with virtualenv to avoid confusion. You might find that certain libs are missing and you have to do it (as Mariano mentions) - I think I had to do this for MySQL; otherwise I would just stick with system until that day comes, then put in the options you require. 

[Chris McDonough]

On 06/12/2012 06:36 PM, Wade Leftwich wrote:
> With the Pyramid docs in ebook form on my phone I tend to browse them
> at odd times. Today, standing in line at the post office, I stopped at
> this paragraph right near the beginning:
> """

> It�s useful to use a Python interpreter that isn�t the �system� Python

> interpreter to develop your software. The authors of Pyramid tend not
> to use the system Python for development purposes; always a self-

> compiled one. Compiling Python is usually easy, and often the �system�
> Python is compiled with options that aren�t optimal for web

> development.
> """
> ... and wondered what the best compilation options for web development
> might be. I generally use whatever comes with Ubuntu, which seems to
> work OK. Does anyone have config tips to share?
>

That wording should probably be revisited because you're not the first
to ask. More accurate is the statement "it's a real pain in the ass
when someone walks into IRC and gives us a Python compilation scenario
that can only be replicated with their particular OS/platform combination".

This tends to happen all the time on Mac, in particular, and fairly
often on Ubuntu as well. This is usually because OS vendors tend to
actually use the system Python and consider it more "part of the OS"
than a development tool for third-party developers. OS vendors don't
compile Python for usage by you; they compile it and configure it for
usage by themselves. Their requirements may not be yours.

I don't use a Mac anymore, but when I did, and when I tried to help
people get things running on their system Python, I was continually
surprised by the number of things that went wrong and the amount of time
people would spend trying to fix them without just compiling and
installing another Python. If I sat down at a machine and tried to make
the system Python work with Pyramid, with enough time and effort, I
probably could. But someone new to Python probably could not, at least
within a reasonable amount of time, and in that case, it's literally
easier to just compile another one.

There's not one cause; each combination of Python version/OS
version/platform combination tends to have its own set of endearing
niggles that aren't much fun. In general, it's just a shortcut to
consider the "system" Python the domain of the OS vendor and compile
your own Python for your own work to shortcut all the potential landmines.

For example, Python on Ubuntu has the following default sys.path:

Python 2.7.3 (default, Apr 20 2012, 22:39:59)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2',
'/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old',
'/usr/lib/python2.7/lib-dynload',
'/usr/local/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages',
'/usr/lib/python2.7/dist-packages/PIL',
'/usr/lib/python2.7/dist-packages/gst-0.10',
'/usr/lib/python2.7/dist-packages/gtk-2.0',
'/usr/lib/pymodules/python2.7',
'/usr/lib/python2.7/dist-packages/ubuntu-sso-client',
'/usr/lib/python2.7/dist-packages/ubuntuone-client',
'/usr/lib/python2.7/dist-packages/ubuntuone-control-panel',
'/usr/lib/python2.7/dist-packages/ubuntuone-couch',
'/usr/lib/python2.7/dist-packages/ubuntuone-installer',
'/usr/lib/python2.7/dist-packages/ubuntuone-storage-protocol']

Note the inclusion of Ubuntu-specific locations by default. This isn't
a problem unless you don't use a virtualenv. But some (insanely) don't.
I've seen some weird things happen as the result of these nonstandard
dirs being included on sys.path by default in any case.

The system Python on Ubuntu (and many other Linux distros) is also
compiled with UCS4 Unicode, which effectively doubles the space required
in memory for each Unicode string without much benefit, as the 99% case
for webapps doesn't include characters outside the UCS2 range (at least
if those webapps don't use Asian scripts).

- C

[León Domingo]

Hi,

I'm an Ubuntu user (well, I'm a Linux Mint user actually) but I work with Ubuntu servers and at the time of Ubuntu 8 I wanted to have Python2.6 instead of the standard Python2.5 of the system. I used to compile it to get "my" version but this got me in problems with Apache+mod_python because I had to compiled them too. I did it.

I left behind that old technology and got into WSGI servers but I kept on working with Apache and mod_wsgi, this time. Similar problem. It's not easy (nor well resolved) to decide which Python will serve the application. This option didn't convince me.

But virtualenv+gunicorn (or uwsgi or whatever) came to save me (save us). Virtualenv is one of the best things of Python world. Now, I can have different environments, with different packages. I could even have different executables (compiled or not) in the same machine serving different apps, each of them with its own requirements. Django, Pyramid, Turbogears, (whatever) at the same time in the same machine.

Right now I use python-system+virtualenv and everything works fine, but I could make a compiled-python+virtualenv give me the same function. The key is virtualenv. In my case, I don't worry about details of python-system, just install as many environments as I need with the particular packages of the app which is served.

León

On Wed, Jun 13, 2012 at 3:21 PM, Chris McDonough <chr...@plope.com> wrote:

On 06/12/2012 06:36 PM, Wade Leftwich wrote:

With the Pyramid docs in ebook form on my phone I tend to browse them
at odd times. Today, standing in line at the post office, I stopped at
this paragraph right near the beginning:
"""

It’s useful to use a Python interpreter that isn’t the “system” Python

interpreter to develop your software. The authors of Pyramid tend not
to use the system Python for development purposes; always a self-

compiled one. Compiling Python is usually easy, and often the “system”
Python is compiled with options that aren’t optimal for web

development.
"""
... and wondered what the best compilation options for web development
might be. I generally use whatever comes with Ubuntu, which seems to
work OK. Does anyone have config tips to share?

That wording should probably be revisited because you're not the first to ask.  More accurate is the statement "it's a real pain in the ass when someone walks into IRC and gives us a Python compilation scenario that can only be replicated with their particular OS/platform combination".

This tends to happen all the time on Mac, in particular, and fairly often on Ubuntu as well.  This is usually because OS vendors tend to actually use the system Python and consider it more "part of the OS" than a development tool for third-party developers.  OS vendors don't compile Python for usage by you; they compile it and configure it for usage by themselves.  Their requirements may not be yours.

I don't use a Mac anymore, but when I did, and when I tried to help people get things running on their system Python, I was continually surprised by the number of things that went wrong and the amount of time people would spend trying to fix them without just compiling and installing another Python.  If I sat down at a machine and tried to make the system Python work with Pyramid, with enough time and effort, I probably could.  But someone new to Python probably could not, at least within a reasonable amount of time, and in that case, it's literally easier to just compile another one.

There's not one cause; each combination of Python version/OS version/platform combination tends to have its own set of endearing niggles that aren't much fun.  In general, it's just a shortcut to consider the "system" Python the domain of the OS vendor and compile your own Python for your own work to shortcut all the potential landmines.

For example, Python on Ubuntu has the following default sys.path:

Python 2.7.3 (default, Apr 20 2012, 22:39:59)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL', '/usr/lib/python2.7/dist-packages/gst-0.10', '/usr/lib/python2.7/dist-packages/gtk-2.0', '/usr/lib/pymodules/python2.7', '/usr/lib/python2.7/dist-packages/ubuntu-sso-client', '/usr/lib/python2.7/dist-packages/ubuntuone-client', '/usr/lib/python2.7/dist-packages/ubuntuone-control-panel', '/usr/lib/python2.7/dist-packages/ubuntuone-couch', '/usr/lib/python2.7/dist-packages/ubuntuone-installer', '/usr/lib/python2.7/dist-packages/ubuntuone-storage-protocol']

Note the inclusion of Ubuntu-specific locations by default.  This isn't a problem unless you don't use a virtualenv.  But some (insanely) don't.  I've seen some weird things happen as the result of these nonstandard dirs being included on sys.path by default in any case.

The system Python on Ubuntu (and many other Linux distros) is also compiled with UCS4 Unicode, which effectively doubles the space required in memory for each Unicode string without much benefit, as the 99% case for webapps doesn't include characters outside the UCS2 range (at least if those webapps don't use Asian scripts).

- C

--
You received this message because you are subscribed to the Google Groups "pylons-discuss" group.

To post to this group, send email to pylons-discuss@googlegroups.com.
To unsubscribe from this group, send email to pylons-discuss+unsubscribe@googlegroups.com.

[Jonathan Vanasco]

On Jun 13, 9:21 am, Chris McDonough <chr...@plope.com> wrote:

> That wording should probably be revisited because you're not the first
> to ask.  More accurate is the statement "it's a real pain in the ass
> when someone walks into IRC and gives us a Python compilation scenario
> that can only be replicated with their particular OS/platform combination".

I hear your pain. OSX can ship with an earlier version of Python
needed too, and replacing it could wipe out functionality on other
apps/the system -- so the easiest and safest scenario to manage is to
install a second, ancillary, version of Python and have that used for
all development via ENV vars.

[Vlad K.]

On 06/13/2012 03:21 PM, Chris McDonough wrote:
> On 06/12/2012 06:36 PM, Wade Leftwich wrote:
>> With the Pyramid docs in ebook form on my phone I tend to browse them
>> at odd times. Today, standing in line at the post office, I stopped at
>> this paragraph right near the beginning:
>> """

>> It’s useful to use a Python interpreter that isn’t the “system” Python

>> interpreter to develop your software. The authors of Pyramid tend not
>> to use the system Python for development purposes; always a self-

>> compiled one. Compiling Python is usually easy, and often the “system”
>> Python is compiled with options that aren’t optimal for web

>> development.
>> """
>> ... and wondered what the best compilation options for web development
>> might be. I generally use whatever comes with Ubuntu, which seems to
>> work OK. Does anyone have config tips to share?
>>
>
> That wording should probably be revisited because you're not the first
> to ask. More accurate is the statement "it's a real pain in the ass
> when someone walks into IRC and gives us a Python compilation scenario
> that can only be replicated with their particular OS/platform
> combination".
>

This is very interesting topic, I haven't considered this, always used
stock system Python (CentOS) but in virtualenv. So what would the
suggested configuration flags be (for a Pyramid app run under uwsgi,
with UTF-8)? I see some I'm not familiar with, as in I don't quite know
what they do, like --enable-profiling, --with-system-ffi, --with-pth,
and some others?


--

.oO V Oo.

[Mike Orr]

On Wed, Jun 13, 2012 at 12:32 AM, Marius Gedminas <mar...@gedmin.as> wrote:
>> Just a little recommendation that I can _barely_ recall it gave me trouble in
>> the past: verify that you have the developer headers for sqlite3 and zlib,
>> otherwise you won't have access to sqlite and, regarding zlib, some third party
>> library might complain (such as PIL) while compiling.
>
> libreadline-dev is also something I would want to have if I ever had to
> build my own Python.  apt-get build-dep is a very good idea on Ubuntu.

My Ubuntu 12.04 says there's no such package with 'build-dep' in its
name. Did you mean 'build-essential'. That gives you the C compiler
and all the configure/make/link tools. I can't find this other package
people are talking about.

You will need to install -dev packages for the optional subsystems you
want (readline, sqlite, zlib, jpeg, etc). The Python README lists what
those subsystems are, then you just have to do a bit of hunting to
find the corresponding -dev package. (Some programs' READMEs tell you
the Debian/Ubuntu package names directly, so you don't have to hunt.)

As i said, I only compile Python when the system version is
unacceptable. I rarely have trouble compiling Python itself. Sometimes
it's hard to compile other things like PIL or MySQLdb.

The Ubuntu Python is complete if you install all the pythonVERSION-*
packages the standard distribution is split into. In the past, Red Hat
and MacOS have shipped with old or incomplete Pythons, but I imagine
the current OS releases are respectable. The Mac Python does have a
lot of weird paths compiled in to comply with its
/Library/Frameworks/* directory system.

--
Mike Orr <slugg...@gmail.com>

[Przemyslaw Wegrzyn]

On 06/13/2012 10:52 PM, Mike Orr wrote:
> On Wed, Jun 13, 2012 at 12:32 AM, Marius Gedminas <mar...@gedmin.as> wrote:
>
>> libreadline-dev is also something I would want to have if I ever had to
>> build my own Python. apt-get build-dep is a very good idea on Ubuntu.
> My Ubuntu 12.04 says there's no such package with 'build-dep' in its
> name. Did you mean 'build-essential'. That gives you the C compiler
> and all the configure/make/link tools. I can't find this other package
> people are talking about.

It's not a package, it's apt-get's (very useful) command:

czajnik@czajnik:~$ sudo apt-get build-dep python
[sudo] password for czajnik:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Picking 'python-defaults' as source package instead of 'python'
The following NEW packages will be installed:
debiandoc-sgml libroman-perl libsgmls-perl libtext-format-perl sgmlspl
0 upgraded, 5 newly installed, 0 to remove and 1 not upgraded.
Need to get 180 kB of archives.
After this operation, 1,723 kB of additional disk space will be used.
Do you want to continue [Y/n]?

BR,
Przemek

[Eric Ongerth]

Seems unnecessarily scary to newcomers to mention compiling your own
Python early in the docs, even if we know it's no big deal.
Particularly when the main concern at hand (that many people's system
instance of python tends to be a poor choice for their Pyramid
development) is easily and almost trivially solved by using a
virtualenv. Would it not be enough to simply phrase the
recommendation of using a virtualenv in fairly strong terms? With a
reference to a deeper document location that discusses the compile-yer-
own-python approach for those interested.

- Eric

On Jun 13, 6:21 am, Chris McDonough <chr...@plope.com> wrote:
> On 06/12/2012 06:36 PM, Wade Leftwich wrote:
>
> > With the Pyramid docs in ebook form on my phone I tend to browse them
> > at odd times. Today, standing in line at the post office, I stopped at
> > this paragraph right near the beginning:
> > """

> > It s useful to use a Python interpreter that isn t the system Python

> > interpreter to develop your software. The authors of Pyramid tend not
> > to use the system Python for development purposes; always a self-

> > compiled one. Compiling Python is usually easy, and often the system
> > Python is compiled with options that aren t optimal for web

[Chris McDonough]

On 06/14/2012 09:24 PM, Eric Ongerth wrote:
> Seems unnecessarily scary to newcomers to mention compiling your own
> Python early in the docs, even if we know it's no big deal.
> Particularly when the main concern at hand (that many people's system
> instance of python tends to be a poor choice for their Pyramid
> development) is easily and almost trivially solved by using a
> virtualenv.

It would be unnecessarily scary if it was unncessary, but like I
mentioned in the text included in this email it is effectively necessary
on some platforms.

Would it not be enough to simply phrase the
> recommendation of using a virtualenv in fairly strong terms? With a
> reference to a deeper document location that discusses the compile-yer-
> own-python approach for those interested.

No, it's a separate issue.

- C

[Toni Mueller]

Hi,

On Wed, Jun 13, 2012 at 09:21:18AM -0400, Chris McDonough wrote:
> actually use the system Python and consider it more "part of the OS"
> than a development tool for third-party developers. OS vendors
> don't compile Python for usage by you; they compile it and configure
> it for usage by themselves. Their requirements may not be yours.

that should largely be a non-issue, imho. Please see below.

> effort, I probably could. But someone new to Python probably could
> not, at least within a reasonable amount of time, and in that case,
> it's literally easier to just compile another one.

One drawback with compiling your own is that it becomes harder to track
security updates in Python. I prefer to have the 'OS vendor' do that for
me, in which case I only need to keep my virtualenvs up to date.

> consider the "system" Python the domain of the OS vendor and compile
> your own Python for your own work to shortcut all the potential
> landmines.

It also works the other way 'round, if eg. the version from python.org
is maybe not very well suited to the platform - eg. I dimly recall
problems with stock Python needing some patches to run acceptably on
*bsd, where "compiling your own" may fail just as well.

> For example, Python on Ubuntu has the following default sys.path:
>
> Python 2.7.3 (default, Apr 20 2012, 22:39:59)
> [GCC 4.6.3] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sys
> >>> sys.path

> ...

> '/usr/lib/python2.7/dist-packages/ubuntu-sso-client',
> '/usr/lib/python2.7/dist-packages/ubuntuone-client',
> '/usr/lib/python2.7/dist-packages/ubuntuone-control-panel',
> '/usr/lib/python2.7/dist-packages/ubuntuone-couch',
> '/usr/lib/python2.7/dist-packages/ubuntuone-installer',
> '/usr/lib/python2.7/dist-packages/ubuntuone-storage-protocol']
>
> Note the inclusion of Ubuntu-specific locations by default. This
> isn't a problem unless you don't use a virtualenv. But some
> (insanely) don't.

Instead of suggesting that people should compile their own Python by
default, the docs should imho strongly emphasize that users create a
virtualenv (or maybe one per project) for their Python work. In case the
Python is at an acceptable version, that should really solve most of the
problems. Suggesting to instead

$ sudo apt-get install virtualenvwrapper

or what that would be on other platforms, and then maybe something like

$ mkvirtualenv -a . -i pyramid --no-site-packages virtualenvname

would go a long way to make development easier and encourage good
development practice.

> The system Python on Ubuntu (and many other Linux distros) is also
> compiled with UCS4 Unicode, which effectively doubles the space
> required in memory for each Unicode string without much benefit, as
> the 99% case for webapps doesn't include characters outside the UCS2
> range (at least if those webapps don't use Asian scripts).

Now this is an interesting corner case: Probably, "UCS4 Unicode" is off
by default, which would leave the one with a self-compiled Python - you
aimed at rather clueless people in the first place, right? - might run
into hard-to-track-down trouble somewhere down the road. This appears to
be just another trade-off between support effort and eg. machine effort
(memory).

I would be glad to not have to think about such an issue, and instead
simply assume(d) that my apps would run with little additional effort,
if any, using Asian scripts as well.


Just my 0.02 cents...


Kind regards,
--Toni++

[Marco Mariani]

On 19 June 2012 17:19, Toni Mueller <d...@tonimueller.org> wrote:

It also works the other way 'round, if eg. the version from python.org
is maybe not very well suited to the platform - eg. I dimly recall
problems with stock Python needing some patches to run acceptably on
*bsd, where "compiling your own" may fail just as well.

Actually, compiling one's own python is not even straightforward on the latest 2.7 on Ubuntu 12.04,
where Setup.dist has to be edited to enable SSL and look for libraries in /usr instead of /usr/local/ssl.

Python 2.6 on the Ubuntu 12.04 also needs an #ifndef patch to _ssl.c which is even harder to guess or look info for.
The docs might mention that if a user fails to build python on a given platform, he might try with a newer release.

[Wade Leftwich]

In case anyone is still interested in this topic, a caveat on compiling Python, at least on Ubuntu and related distros. Compiling with standard configuration creates /usr/local/bin/python, which can cause problems with system utilities.

On Linux Mint 13, after rolling my own Python,  I ran into a problem where Cinnamon desktop manager would not load its settings screens. Turned out that it was calling just plain old `python`, not `/usr/bin/python`, so it was getting /usr/local/bin/python which did not have some expected modules installed. Deleting /usr/local/bin/python, which is a symlink to python2.7, corrected the fault.

Also: if you have /usr/local/bin/python, then `sudo python`, `sudo easy_install`, etc will not call the same executables as `python` and `easy_install`.

[Chris McDonough]

On 07/11/2012 01:22 PM, Wade Leftwich wrote:
> In case anyone is still interested in this topic, a caveat on compiling
> Python, at least on Ubuntu and related distros. Compiling with standard
> configuration creates /usr/local/bin/python, which can cause problems
> with system utilities.

This is a problem with lots of stuff. E.g.

https://bugs.launchpad.net/ubuntu/+source/ubuntuone-client/+bug/984089

I've been trying to get Barry Warsaw to shame the various system utility
developers at Canonical to use an explicit #!/usr/bin/python rather than
#!/usr/bin/env python in their scripts. Hopefully the next major
release will have fewer problems like this.

- C

> --
> You received this message because you are subscribed to the Google
> Groups "pylons-discuss" group.

> To view this discussion on the web visit
> https://groups.google.com/d/msg/pylons-discuss/-/IFQelnUOVA8J.
> To post to this group, send email to pylons-...@googlegroups.com.

> To unsubscribe from this group, send email to

> pylons-discus...@googlegroups.com.