Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss

PyCon 2013 -- tentative title/abstract/outline -- feedback plz

15 views
Skip to first unread message

Yaroslav Halchenko

unread,
Sep 26, 2012, 11:50:02 AM9/26/12
to
To not be too ambitious and to not invest too much time I have decided to
submit only a talk. Here follows a perspective title, abstract and some
notes/outline which will not be a part of submission. I would really
appreciate (and of cause would acknowledge in the slides) any feedback, ideas,
comments, etc.

[originally in emacs org-mode]

* Title

Debian -- (rich) Python distribution for the bare metal

Alternatives:
The universal Python distribution or build your own stack
Debian & Python -- a happy couple with a character
Propelling Python to the masses with the universal OS

* Abstract

Through the years Python community strives to distill the ultimate
Python distribution utilities. Meanwhile, to overcome the problems of
the core Python and 3rd party FOSS Python projects distribution,
various free and commercial distribution bundles of Python appeared.
They made Python, as an environment with a pre-selected set of Python
modules, conveniently available (primarily) on proprietary systems.
What is rarely known is that for decades Python has been a part of the
largest in the world software distribution platform: Debian project
delivers a complete operating system with thousands of FOSS projects
making them available on 11 hardware architectures and 3 different
kernels (Linux, HURD, kFreeBSD). In the Linux world, Debian is known
as the most popular base distribution due its openness, ease of use,
versatility, and stability. By delivering a well integrated and
tested versatile OS, with a plethora of core libraries necessary for
nearly any field of endeavor, it became an ideal base for the
**complete** Python distribution. Majority of Python projects are
either already packaged for Debian or provide 1-2 lines instructions
on how to install necessary dependencies and build/install the product
on Debian-based systems. Recent advances in hardware virtualization
support followed in tandem with the explosion of cloud solutions, made
Debian systems popular not only among Linux "fan-boys" but for
various, especially scientific and community-driven, deployments. The
ease with which thousands of Python-based FOSS became installable and
maintainable made Debian the Python distribution with "**all**
batteries included".

In this talk I would like to briefly present the history of Python in
Debian (which can be traced to nineties with Python 1.4) and outline
benefits Debian provides for Python users and developers, keeping in
mind upcoming stable Debian release (wheezy). To familiarize
listeners with Python-in-Debian ecosystem I will then overview core
package naming, versioning, and modularization conventions in Debian,
and briefly present the "Debian packaging" helper tools, including
recent GSOC project aiming to provide automatic packaging of the
packages on PyPI. To facilitate the synergy between Python and Debian
communities, I will accent on common sense practices (following PEPs,
clean and exhaustive legal terms, CI, etc.) which would make any
Debian packaging and maintainership more efficient. I am planing to
conclude by presenting few easy ways on how to start using Debian.

As the outcome of the talk, I expect listeners to become more familiar
with the Debian project's standards and principles, become aware of
integration aspects involved in delivering such plethora of Python
FOSS solutions, and be intrigued enough to try Debian on their systems
or in the cloud.


Just NOTES:

* Python-in-Debian History
** Upstream: Python 1.0 - January 1994, Python 1.5 - December 31, 1997
** debian-python ML https://lists.debian.org/debian-python/1998/08/msg00000.html

To: debian...@lists.debian.org
Cc: hoff...@mathi.uni-heidelberg.DE, lor...@argon.roma2.infn.it
Subject: Welcome to debian-python
From: Hanno Wagner <wag...@fitug.de>
Date: Fri, 7 Aug 1998 09:27:05 +0200
Message-id: <1998080709...@beuel.rhein.de>
Reply-to: Hanno Wagner <wag...@fitug.de>

Good morning gentlemen,

this is the initial posting for debian-python, the
mailinglist is running now.

Here is the description for the mailinglist:

debian...@lists.debian.org

Description : Discussion of issues related to Python on Debian
systems with an stress on packaging standards.
Therefore relevant for maintainers of Python related
packages.
Moderated : no
Subscription: open



Have a nice start,

Ciao, Hanno, one of listm...@lists.debian.org
--
| Hanno Wagner | Member of the HTML Writers Guild | Rince@IRC |
| Eine gewerbliche Nutzung meiner Email-Adressen ist nicht gestattet! |
| 74 a3 53 cc 0b 19 - we did it! | Generation @ |

#Fachbegriffe der Informatik einfach erklaert, Teil 69:
#"It is essential that implementations by different vendors interoperate."
# == "Unsere proprietaeren Basteleien dokumentieren wir gar nicht erst."
# (Sven Tuerpe)
** python2 changelog (Python 2.0 was released on 16 October 2000)
python2 (2.0-1) unstable; urgency=low

* New upstream version. Initial release for python2.

-- Gregor Hoffleit <fli...@debian.org> Mon, 11 Dec 2000 22:39:46 +0100
* For the "users"
very convenient environment to install, update, remove.
** Statistics on the covered packages
XXX Python modules/extensions (tagcloud?)
XXX Python bindings for C/C++ libraries
*** By domains
web-frameworks: gluon, web2py, turbogeats, django, pyjamas...
scientific Python: numpy, scipy, ...
* For the "developers"
** Rich development environment
multiple-supported versions -- smooth-ed migration
IDEs:
python*-dbg
virtualenv

** Q: How do I keep my finger on the beat of my baby in Debian?
Subscribe to announcements on http://packages.qa.debian.org

** Q: How popular am I?
popcon.debian.org
Show numpy proliferation on Debian deployments
* (?) Python-in-core-Debian
Which core Debian tools are Python-based
- reportbug
- git-buildpackage
* Python-in-Debian internals
- In 99% it is not 'Bureaucracy' -- it is evolving open standards ;)
Python modules policy is not carved in stone (i.e. not part of the
official main Debian policy)

- Binary packages naming/modularization conventions or
"WTF they have done with my module..."
- python-* -- Python modules/extensions
- might be split into python-X arch:all and python-core/-lib/-bin
- might be complemented with
-dbg -- package (built against python-dbg)
-doc -- documentation
- python*-numpy-a[bp]i* packages
- How package is built
- source vs binary package
- dh + dh_python2 (python-central/python-support -- deprecated)
- pkg build time testing
TODO: stats -- how many, test in-place, test against
built/installed
- no build-time testing for arch:all packages (built ones/uploaded +
QA rebuilds)
- helpers:
- stdeb -- http://github.com/astraw/stdeb (python-stdeb package)
- GSOC 2012 -- pypi2deb -- PyPI to Debian converter
https://gitorious.org/pypi2deb
- Debian versioning
- Python2/3 Debian peculiarities
- TODO
* What do we ask developers about
- standard deployment schemes (setup.py install)
- follow PEPs:
- PEP XXX -- versioning
implemented only in python3, but at least rely on
distutils.version.LooseVersion
- is there PEP on testing, Barry?
- clean(er) separation of code and data
- clear and exhaustive LICENSE/COPYRIGHTs
- unittests
- tag ones requiring network access or better -- provide fixtures
- exercise against minimal supported versions
(of Python itself, and 3rd party libraries)
* How to get started with Debian
- install (dual-boot, ...)
- other Linuxes: chroot - lightweight virtualization (debootstrap, schroot)
- VM, e.g. http://neuro.debian.net/vm.html
- cloud




On Fri, 21 Sep 2012, Yaroslav Halchenko wrote:

> Hi everyone,

> Since the deadline for the submission of talks/tutorials for the PyCon
> 2013 is approaching (28th of Sep) I thought to check if anyone from the
> 'team' will be attending (Barry?) and may be someone already is
> planing to give a talk or might be even a tutorial?

> Debian-based systems become de-facto "the community Linux" in the
> Python world due to the simplicity of maintenance and deployment of
> Python software. But I think we still are far behind at promoting
> ourselves, so I thought it would be nice if "Debian" appears at PyCon
> (some corporate Linux-related entities are already among sponsors where
> Debian is unlikely to be listed). I am not sure yet if I would get any
> funds to attend but I am thinking about submitting two proposals for

> 1. talk on "The universal Python distribution or build your own stack"

> in many fields of endeavor people talk about stacks and python
> distributions which make easy to build/deploy suck stacks. So I
> thought that we should not be shy and present Debian as the best
> platform for anyone -- either ran on bare metal (preferable) or in a
> VM (for new adopters). But then we might have released wheezy which
> would be a good base for the talk -- present what fresh release has
> brought to the community in this stable environment.

> Previously I have done a similar talk with an accent on a scientific
> Python stack in Debian [1] which I thought was quite well accepted.

> 2. tutorial on "Debian packaging of Python modules/software"

> since tutorials are separate from the main registration (i.e. require
> separate payment if I got it right) I am not quite sure how many
> people would be interested to attend it. But I guess it should not
> hurt to submit one and for the committee to decide.

> Also it might be worth asking for a table/booth space (I think I saw
> that somewhere on pycon website) for the Debian project.

> I would be glad to get any feedback (i.e. "not worth the money/time
> spent", "you might like better to ...", ...) and recommendations on how
> to get funds for the trip (I will apply for the "financial aid" but more
> ideas e.g. "kickstarter project?", "I think my company might be
> interested to cover...", etc) ;)

> Cheers,
--
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20120926154...@onerussian.com

Yaroslav Halchenko

unread,
Sep 27, 2012, 9:00:01 AM9/27/12
to
not a single comment... bad... I guess I need to work on the text
more if even hardcore Debian people do not feel 'moved' ;-)
Archive: http://lists.debian.org/20120927125...@onerussian.com

Paul Tagliamonte

unread,
Sep 27, 2012, 11:50:02 AM9/27/12
to
On Thu, Sep 27, 2012 at 08:51:37AM -0400, Yaroslav Halchenko wrote:
> not a single comment... bad... I guess I need to work on the text
> more if even hardcore Debian people do not feel 'moved' ;-)

Well, i'll give my 2c as a pythonista and a Debian-folk
I can see this becoming a flamefest.

Most "hardcore" pythonistas (and the types to be at PyCon) refuse to
allow apt to install libs globally, and use virtualenv(wrapper) to
isolate deps for a few reasons -- the big ones being:

- more "up to date"
- isolates dependency hell

which (frankly) apt-get / Debian stable can't really address. Sometimes
Python packages in sid are out of date as well.

People don't care about API stability or anything like that, so I think
you might have to try to frame this in a way that doesn't provoke a
virtualenv-vs-apt battle -- because, frankly, neither side will win and
it'll just become a bit murky.

I'd be happy to help you prepare / do more interactive work with folks
at PyCon (I should likely be there) :)
--
.''`. Paul Tagliamonte <pau...@debian.org>
: :' : Proud Debian Developer
`. `'` 4096R / 8F04 9AD8 2C92 066C 7352 D28A 7B58 5B30 807C 2A87
`- http://people.debian.org/~paultag
signature.asc

Paul Boddie

unread,
Sep 27, 2012, 5:40:01 PM9/27/12
to
On Thursday 27 September 2012 17:50:04 Paul Tagliamonte wrote:
> On Thu, Sep 27, 2012 at 08:51:37AM -0400, Yaroslav Halchenko wrote:
> > not a single comment... bad... I guess I need to work on the text
> > more if even hardcore Debian people do not feel 'moved' ;-)
>
> Well, i'll give my 2c as a pythonista and a Debian-folk

I was going to give some feedback more as the kind of person who has gone to
Python conferences, and certainly, if you want a native speaker to give
feedback on the phrasing of your proposal, I'd be happy to make some
suggestions.

[...]

> I can see this becoming a flamefest.
>
> Most "hardcore" pythonistas (and the types to be at PyCon) refuse to
> allow apt to install libs globally, and use virtualenv(wrapper) to
> isolate deps for a few reasons -- the big ones being:
>
> - more "up to date"
> - isolates dependency hell
>
> which (frankly) apt-get / Debian stable can't really address. Sometimes
> Python packages in sid are out of date as well.

Python packaging has become somewhat insular over the years with
Python-centric solutions that work across different systems rather than
solutions that work well with the rest of the software on particular systems.
However, people appear to like things like virtualenv, especially the Web
crowd that makes up a lot of the audience at events like this, because it
lets them set up relatively cheap configurations for separate Web
applications or for experimenting.

I have advocated solutions based on fakechrooted debootstrapped installations
if only because you can manage the libraries below the Python modules and
extensions as well as the stuff that supports things like distutils and
setuptools. However, the people who can change this situation don't see the
need or the point: it's either "but I have root!" or "they can always build
from source!" No wonder people use stuff like virtualenv instead. It is in
this area where I feel that the Debian community could do more to meet others
half-way.

> People don't care about API stability or anything like that, so I think
> you might have to try to frame this in a way that doesn't provoke a
> virtualenv-vs-apt battle -- because, frankly, neither side will win and
> it'll just become a bit murky.
>
> I'd be happy to help you prepare / do more interactive work with folks
> at PyCon (I should likely be there) :)

The one case that many language-focused groups ignore, and where distributions
do well, is the case where a range of different technologies needs to be
managed and where administrators just wouldn't be able to keep up with Python
eggs, Ruby gems, CPAN, and the language-specific technology of the week.
Persuading the Python community to feed packages into Debian so that they
become a safer choice for people who routinely use or know other technologies
is definitely a worthwhile cause.

Paul


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20120927234...@boddie.org.uk

Yaroslav Halchenko

unread,
Sep 27, 2012, 6:30:01 PM9/27/12
to
Thank you Paul ;-)

Good comments -- once again, arguments seems to be oriented mostly
toward developers... I guess I should explicitly guide the
abstract more toward 'user-' and "sysadmin-" use cases: people
in need to have easy and uniform paths for software installation and
maintenance of the heterogeneous system. In scientific users domain it
becomes even more fun with heavier reliance on computational and I/O
libraries (blas/atlas, hdf5, etc) building and maintaining which might
be quite a bit of a hassle.

Few inline comments.

> I was going to give some feedback more as the kind of person who has gone to
> Python conferences, and certainly, if you want a native speaker to give
> feedback on the phrasing of your proposal, I'd be happy to make some
> suggestions.

I would appreciate "native speaker" feedback! since "accepting all
types of proposals through September 28, 2012", I guess I have the whole
tomorrow to revise and submit. I hope to find some time later today to
revise my abstract and will post it again for further phrasing
suggestions


That is true... Somewhat offtopic -- that is why with neuro.debian.net
we pretty much serve an unofficial backports repository for a good
portion of Python modules we maintain. Besides immediate benefit for
users, benefit from backporting for developers has been build-time
testing across various releases of Debians and Ubuntus, picking out
problems with specific versions of the core libraries... So, may be I
should add an accent that availability in Debian doesn't only guarantee
ease of installation (for users) but provides a good test bed for the
developers to preclude problems with future deployments on Debian-based
platforms... ?

> Python packaging has become somewhat insular over the years with
> Python-centric solutions that work across different systems rather than
> solutions that work well with the rest of the software on particular systems.
> However, people appear to like things like virtualenv, especially the Web
> crowd that makes up a lot of the audience at events like this, because it
> lets them set up relatively cheap configurations for separate Web
> applications or for experimenting.

virtualenv is indeed great for the reasons you guys point out AND
indeed, it is very Python-centric and maintenance of a configured
virtualenv might become cumbersome for projects with lots of 3rd party
dependencies and for regular users who would not want to care to switch
among different virtualenvs etc.

I guess I should revise abstract to aim a listener wondering "why should
I care about Debian if there is virtualenv" WITHOUT explicitly pointing
to its pros to not cause any flames. And not sure I would be able to
convince hard-core Python-ians, so I might not even try and orient
it more toward users/admins.

> I have advocated solutions based on fakechrooted debootstrapped installations

btw -- how is it working out for you? i.e. are you still pushing it
forward?

> if only because you can manage the libraries below the Python modules and
> extensions as well as the stuff that supports things like distutils and
> setuptools. However, the people who can change this situation don't see the
> need or the point: it's either "but I have root!" or "they can always build

many (users on managed boxes) -- don't, so I would have pushed these
approaches for them as well ;)

> from source!" No wonder people use stuff like virtualenv instead. It is in
> this area where I feel that the Debian community could do more to meet others
> half-way.

> > People don't care about API stability or anything like that, so I think
> > you might have to try to frame this in a way that doesn't provoke a
> > virtualenv-vs-apt battle -- because, frankly, neither side will win and
> > it'll just become a bit murky.

> > I'd be happy to help you prepare / do more interactive work with folks
> > at PyCon (I should likely be there) :)

> The one case that many language-focused groups ignore, and where distributions
> do well, is the case where a range of different technologies needs to be
> managed and where administrators just wouldn't be able to keep up with Python
> eggs, Ruby gems, CPAN, and the language-specific technology of the week.
> Persuading the Python community to feed packages into Debian so that they
> become a safer choice for people who routinely use or know other technologies
> is definitely a worthwhile cause.

indeed safer and more accessible choice.


--
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/2012092722...@onerussian.com

Yaroslav Halchenko

unread,
Sep 27, 2012, 6:30:01 PM9/27/12
to
Thank you Paul,

On Thu, 27 Sep 2012, Paul Tagliamonte wrote:
> I can see this becoming a flamefest.

oh no... I hoped to simply present our work and not cause
flamefests ;-)


> Most "hardcore" pythonistas (and the types to be at PyCon) refuse to
> allow apt to install libs globally, and use virtualenv(wrapper) to
> isolate deps for a few reasons -- the big ones being:

> - more "up to date"
> - isolates dependency hell

> which (frankly) apt-get / Debian stable can't really address. Sometimes
> Python packages in sid are out of date as well.

That is true... Somewhat offtopic -- that is why with neuro.debian.net
we pretty much serve an unofficial backports repository for a good
portion of Python modules we maintain. Besides immediate benefit for
users, benefit from backporting for developers has been build-time
testing across various releases of Debians and Ubuntus, picking out
problems with specific versions of the core libraries... So, may be I
should add an accent that availability in Debian doesn't only guarantee
ease of installation (for users) but provides a good test bed for the
developers to preclude problems with future deployments on Debian-based
platforms... ?


> People don't care about API stability or anything like that, so I think
> you might have to try to frame this in a way that doesn't provoke a
> virtualenv-vs-apt battle -- because, frankly, neither side will win and
> it'll just become a bit murky.

I think, as Paul pointed out (reply to his email will follow
shortly) that it might be worth to orient the talk more toward the
users who per se pretty much need to take care about the whole system --
regular mortals and sysadmins.


> I'd be happy to help you prepare / do more interactive work with folks
> at PyCon (I should likely be there) :)

Thanks in advance... let's see first if I would be going anywhere ;)

--
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20120927222...@onerussian.com

Paul Boddie

unread,
Sep 27, 2012, 7:00:02 PM9/27/12
to
On Friday 28 September 2012 00:23:10 Yaroslav Halchenko wrote:
> Thank you Paul ;-)
>
> Good comments -- once again, arguments seems to be oriented mostly
> toward developers... I guess I should explicitly guide the
> abstract more toward 'user-' and "sysadmin-" use cases: people
> in need to have easy and uniform paths for software installation and
> maintenance of the heterogeneous system. In scientific users domain it
> becomes even more fun with heavier reliance on computational and I/O
> libraries (blas/atlas, hdf5, etc) building and maintaining which might
> be quite a bit of a hassle.

Yes, I had cause to build NumPy from scratch recently, and it was quite
intimidating. It did happen to involve a fairly low-performance device with
fairly severe memory constraints, and the experience really pushed me towards
looking harder at Debian and the packaging work that I knew would have been
done to potentially solve some of the issues I was experiencing, one of which
being modularisation of the library, although I'm not sure how much this is
actually done with NumPy in Debian.

> Few inline comments.
>
> > I was going to give some feedback more as the kind of person who has gone
> > to Python conferences, and certainly, if you want a native speaker to
> > give feedback on the phrasing of your proposal, I'd be happy to make some
> > suggestions.
>
> I would appreciate "native speaker" feedback! since "accepting all
> types of proposals through September 28, 2012", I guess I have the whole
> tomorrow to revise and submit. I hope to find some time later today to
> revise my abstract and will post it again for further phrasing
> suggestions

I'll try and follow the list and get back to you.

> That is true... Somewhat offtopic -- that is why with neuro.debian.net
> we pretty much serve an unofficial backports repository for a good
> portion of Python modules we maintain. Besides immediate benefit for
> users, benefit from backporting for developers has been build-time
> testing across various releases of Debians and Ubuntus, picking out
> problems with specific versions of the core libraries... So, may be I
> should add an accent that availability in Debian doesn't only guarantee
> ease of installation (for users) but provides a good test bed for the
> developers to preclude problems with future deployments on Debian-based
> platforms... ?

Having gone through the packaging process, I know that a lot of the hurdles
actually do help packagers improve the reliability of the eventual package,
even though the process can be drawn out and frustrating. Even non-technical
stuff like auditing the licences is a valuable activity, however, that gives
users confidence that the end product is of high quality and not just zipped
up and uploaded to some site or other.

The Python conference scene seems to love testing, so if you can make a case
for Debian and quality assurance, and Debian has done things popular with
this crowd for years like automated builds and the use of very strict package
building tools that won't let you build without a precise specification of
the build dependencies, then that may appeal to some people.

> > Python packaging has become somewhat insular over the years with
> > Python-centric solutions that work across different systems rather than
> > solutions that work well with the rest of the software on particular
> > systems. However, people appear to like things like virtualenv,
> > especially the Web crowd that makes up a lot of the audience at events
> > like this, because it lets them set up relatively cheap configurations
> > for separate Web applications or for experimenting.
>
> virtualenv is indeed great for the reasons you guys point out AND
> indeed, it is very Python-centric and maintenance of a configured
> virtualenv might become cumbersome for projects with lots of 3rd party
> dependencies and for regular users who would not want to care to switch
> among different virtualenvs etc.

It's a Python tool for Python developers, primarily. If you're running a Web
operation with lots of Python developers and a commitment to Python then it
may make sense, but that sort of builds a wall that deters people from
adopting Python, too.

> I guess I should revise abstract to aim a listener wondering "why should
> I care about Debian if there is virtualenv" WITHOUT explicitly pointing
> to its pros to not cause any flames. And not sure I would be able to
> convince hard-core Python-ians, so I might not even try and orient
> it more toward users/admins.

I don't think you should worry too much about flames. My impression is that
the packaging people are trying to scale back their ambitions and just get
everyone to do the basic things like write decent metadata, mostly because I
think the process of delivering a comprehensive solution is deadlocked once
again, and I think that people do see the need to hear from distributions and
to try and get as much input from them as possible.

> > I have advocated solutions based on fakechrooted debootstrapped
> > installations
>
> btw -- how is it working out for you? i.e. are you still pushing it
> forward?

Not really, mostly because I was frequently using them for accessing newer
distributions, and then fakechroot wasn't quite capable enough to avoid
low-level library conflicts. So I now routinely use these installations as
genuine chroots instead, and I can convert them to run as User Mode Linux
installations, but I suppose the principle still stands. :-)

(For my packaging activities, I actually used squeeze under UML to launch a
sid chroot, since my host kernel is not modern enough for sid. I could
probably just launch sid under UML from an installer image, but that's
something for another time.)

> > if only because you can manage the libraries below the Python modules and
> > extensions as well as the stuff that supports things like distutils and
> > setuptools. However, the people who can change this situation don't see
> > the need or the point: it's either "but I have root!" or "they can always
> > build
>
> many (users on managed boxes) -- don't, so I would have pushed these
> approaches for them as well ;)

You can certainly make the case to anyone working in a large organisation.

[...]

> > The one case that many language-focused groups ignore, and where
> > distributions do well, is the case where a range of different
> > technologies needs to be managed and where administrators just wouldn't
> > be able to keep up with Python eggs, Ruby gems, CPAN, and the
> > language-specific technology of the week. Persuading the Python community
> > to feed packages into Debian so that they become a safer choice for
> > people who routinely use or know other technologies is definitely a
> > worthwhile cause.
>
> indeed safer and more accessible choice.

I actually have to do this with RHEL at work, but the point stands: if you
can't rely on a stream of packages maintained by someone else, your ability
to deploy and manage a hetergeneous suite of applications is impaired. And
the result of that may involve you deploying a bunch of not so great
applications instead. That latter part is arguably a difficult point to make
to an audience who may be convinced that all the best tools are written in
Python and work perfectly with virtualenv, pip, easy_install or whatever, but
it matters to potential users of Python software, and it may end up mattering
to them if their bosses actually prefer Perl, Ruby or PHP.

So you have something to really scare people with if you wanted to. ;-)

Paul


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20120928010...@boddie.org.uk

Yaroslav Halchenko

unread,
Sep 27, 2012, 10:30:01 PM9/27/12
to
ok -- here is my next take trying to make at least the title and introduction
more user oriented and mention those aspects which might be be of interest for
developers... As a result it probably became even less English and now
exercises working memory even harder ;)

Propelling Python to the masses with the universal OS

Python has found appreciation not only among professional developers
but also among students, scientists and programming novices due to its
scripting nature, "batteries included", good collection of 3rd party
libraries, and ability to interface to libraries written in other
languages and computing environments (e.g. R). To conveniently
deliver such a versatile Python platform to users (and their humble
system administrators), the community have been distilling the
ultimate Python distribution utilities and bundling pre-built Python
and core 3rd party libraries and modules for the distribution on
proprietary systems. Meanwhile nearly for two decades Python has been
a part of the largest community-driven software distribution platform
-- Debian. The Debian project delivers a complete operating system
with tens of thousands of FOSS projects available on 11 hardware
architectures and 3 different kernels (Linux, HURD, kFreeBSD). Being
a binary distribution Debian guarantees safer -- free of build-errors
-- installations and seamless upgrades. Coupled with the standardized
specification of build and run-time dependencies, it made it easy to
build, verify, or simply deploy projects of nearly arbitrary
complexity of inter-dependencies and varying implementation origins.
Such agnosticism to the origins of the software made Python-based
products a 1st citizen in this heterogeneous distribution ecosystem,
assuring that Python works well with the rest of it. Recent advances
in hardware virtualization support, followed in tandem with the
explosion of cloud solutions, made Debian systems popular not only
among Linux "fan-boys" but for various, especially scientific and
community-driven, deployments. The ease with which thousands of
Python-based FOSS became available and maintainable made Debian the
Python distribution with "**all** batteries (and virtualenv)
included".

In this talk I would like to briefly present the history of Python in
Debian (which can be traced to nineties with Python 1.4), outline
benefits Debian provides for Python users/developers and present what
to expect in upcoming stable (wheezy) release of Debian. To
familiarize listeners with Python-in-Debian ecosystem I will then
overview core package naming, versioning, and modularization
conventions in Debian and ongoing QA efforts (build-time testing,
full-archive rebuilds, etc). I will briefly present the "Debian
packaging" helper tools, including recent GSOC project aiming to
provide automatic packaging of the packages on PyPI. To facilitate
the synergy between Python and Debian communities, I will accent on
common sense practices (following PEPs, clean and exhaustive legal
terms, CI, etc.) which would make any Debian packaging and
maintainership more efficient and benefit upstream developers. I am
planing to conclude by presenting few easy ways on how to start using
Debian.

As the outcome of the talk, I expect listeners to become more familiar
with the Debian project goals, standards and principles, become aware
of integration aspects involved in delivering such plethora of Python
FOSS solutions, and become intrigued enough to try Debian on their
systems or in the cloud.


Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20120928022...@onerussian.com

Piotr Ożarowski

unread,
Sep 28, 2012, 8:50:02 AM9/28/12
to
[Yaroslav Halchenko, 2012-09-28]
> In this talk I would like to briefly present the history of Python in
> Debian (which can be traced to nineties with Python 1.4), outline
> benefits Debian provides for Python users/developers and present what
> to expect in upcoming stable (wheezy) release of Debian. To
> familiarize listeners with Python-in-Debian ecosystem I will then
> overview core package naming, versioning, and modularization
> conventions in Debian and ongoing QA efforts (build-time testing,

about conventions... please, please, please mention PEP386, PEP396,
PEP390 (, PEP384?, PEP8?) or documents like http://wiki.debian.org/UpstreamGuide
- maybe it's not directly related to your talk, but any occasion
is good to try to convince developers to follow conventions.

> full-archive rebuilds, etc). I will briefly present the "Debian
> packaging" helper tools, including recent GSOC project aiming to
> provide automatic packaging of the packages on PyPI. To facilitate

even today Natalia committed some changes related to exporting build/test
logs, statistics will follow soon, I hope :). We still need a server to
set this up, though - I'll try to arrange a webserver to host generated
repo/logs, but we still need access to a buildd (or a server where we
can set one up) to do the building part - Natalia's tool is prepared to
be invoked in cron and build (using f.e. sbuild) only new packages / for
new architectures or new distributions).

> the synergy between Python and Debian communities, I will accent on
> common sense practices (following PEPs, clean and exhaustive legal
> terms, CI, etc.) which would make any Debian packaging and
> maintainership more efficient and benefit upstream developers. I am
> planing to conclude by presenting few easy ways on how to start using
> Debian.

great!

> As the outcome of the talk, I expect listeners to become more familiar
> with the Debian project goals, standards and principles, become aware
> of integration aspects involved in delivering such plethora of Python
> FOSS solutions, and become intrigued enough to try Debian on their
> systems or in the cloud.

you can also advertise https://buildd.debian.org/status/package.php
(for those who want to see build logs from different architectures
of their libraries - we try to enable more and more tests during build)
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/2012092812...@piotro.eu

Yaroslav Halchenko

unread,
Sep 28, 2012, 9:10:02 AM9/28/12
to
Thanks Piotr!

On Fri, 28 Sep 2012, Piotr Ożarowski wrote:
> [Yaroslav Halchenko, 2012-09-28]
> > In this talk I would like to briefly present the history of Python in
> > Debian (which can be traced to nineties with Python 1.4), outline
> > benefits Debian provides for Python users/developers and present what
> > to expect in upcoming stable (wheezy) release of Debian. To
> > familiarize listeners with Python-in-Debian ecosystem I will then
> > overview core package naming, versioning, and modularization
> > conventions in Debian and ongoing QA efforts (build-time testing,

> about conventions... please, please, please mention

with pleasure ... but do you think it is worth listing (some of) them in
the abstract?

> PEP386

this one is my favorite (giving significant amount of suffer ;) )

> PEP396,

Status: Draft
Barry, was there any progress?

it would be cool if it was somewhat coupled with recipes for different
VCS. I know a few projects which go few extra steps to automate unique
version assignments for the GIT archive exports etc. e.g.
http://anonscm.debian.org/gitweb/?p=pkg-exppsy/pynifti.git;a=blob;f=setup.py;hb=HEAD


> PEP390

Status: Draft
seems needs to make friends with PEP396 regarding version information

dependencies specs are indeed would be valuable

> PEP384?

I guess I need to digest it more to explain how/if it is relevant for
Debian maintenance.

> PEP8?

I could start with this one of cause ;) but I hope they all know about
it by now. On a related note though: __file__ -- are we all friends
again ? ;)
it would of cause be worthwhile of at least mentioning

> - maybe it's not directly related to your talk, but any occasion
> is good to try to convince developers to follow conventions.

that is part of the aim indeed ;)

> > full-archive rebuilds, etc). I will briefly present the "Debian
> > packaging" helper tools, including recent GSOC project aiming to
> > provide automatic packaging of the packages on PyPI. To facilitate

> even today Natalia committed some changes related to exporting build/test
> logs, statistics will follow soon, I hope :). We still need a server to
> set this up, though - I'll try to arrange a webserver to host generated
> repo/logs, but we still need access to a buildd (or a server where we
> can set one up) to do the building part - Natalia's tool is prepared to
> be invoked in cron and build (using f.e. sbuild) only new packages / for
> new architectures or new distributions).

yeah -- without such at least a demo repository project felt
somewhat incomplete to me.

> > As the outcome of the talk, I expect listeners to become more familiar
> > with the Debian project goals, standards and principles, become aware
> > of integration aspects involved in delivering such plethora of Python
> > FOSS solutions, and become intrigued enough to try Debian on their
> > systems or in the cloud.

> you can also advertise https://buildd.debian.org/status/package.php
> (for those who want to see build logs from different architectures
> of their libraries - we try to enable more and more tests during build)

right!

I also had in mind pointing to http://packages.qa.debian.org as the
ultimate 'developer-oriented' page, where upstream could get to the
logs, subscribe to notifications (email, RSS), etc

--
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20120928130...@onerussian.com

Piotr Ożarowski

unread,
Sep 28, 2012, 9:30:02 AM9/28/12
to
[Yaroslav Halchenko, 2012-09-28]
> On Fri, 28 Sep 2012, Piotr Ożarowski wrote:
> > about conventions... please, please, please mention
>
> with pleasure ... but do you think it is worth listing (some of) them in
> the abstract?

no, but please make sure to bind a key that (when pressed) shows a
screen with these PEP numbers... and hit it every time you do a short
break to catch a breath or drink water ;)
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/2012092813...@piotro.eu

Yaroslav Halchenko

unread,
Sep 28, 2012, 9:50:01 AM9/28/12
to

On Fri, 28 Sep 2012, Piotr Ożarowski wrote:
> no, but please make sure to bind a key that (when pressed) shows a
> screen with these PEP numbers... and hit it every time you do a short
> break to catch a breath or drink water ;)

;) I do bind keys to important slides in impressive... now I will have 1
more -- what shortcut key would you recommend -- "Enter"? ;)

yeah -- I guess it would be nice to get some 'cheat slide' with the most
important links to condition people over and over again happen there is
a break ;)

--
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20120928134...@onerussian.com

Paul Wise

unread,
Sep 28, 2012, 9:50:02 AM9/28/12
to
On Fri, Sep 28, 2012 at 9:47 PM, Paul Tagliamonte wrote:

> ^^ this is a great idea. It'd be nice if we could prototype a flake8 /
> pyflakes run against the archive, and filter for serious errors

We did do that at one point with pyflakes:

http://qa.debian.org/daca/pyflakes/sid/

Unfortunately no-one has been working on DACA recently, hint hint ;)

--
bye,
pabs

http://wiki.debian.org/PaulWise


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/CAKTje6H-2qFzfogumqVke+G7=nbm2rVm6mNRr...@mail.gmail.com

Paul Tagliamonte

unread,
Sep 28, 2012, 9:50:01 AM9/28/12
to
^^ this is a great idea. It'd be nice if we could prototype a flake8 /
pyflakes run against the archive, and filter for serious errors

>
> > > Python packaging has become somewhat insular over the years with
> > > Python-centric solutions that work across different systems rather than
> > > solutions that work well with the rest of the software on particular
> > > systems. However, people appear to like things like virtualenv,
> > > especially the Web crowd that makes up a lot of the audience at events
> > > like this, because it lets them set up relatively cheap configurations
> > > for separate Web applications or for experimenting.
> >
> > virtualenv is indeed great for the reasons you guys point out AND
> > indeed, it is very Python-centric and maintenance of a configured
> > virtualenv might become cumbersome for projects with lots of 3rd party
> > dependencies and for regular users who would not want to care to switch
> > among different virtualenvs etc.
>
> It's a Python tool for Python developers, primarily. If you're running a Web
> operation with lots of Python developers and a commitment to Python then it
> may make sense, but that sort of builds a wall that deters people from
> adopting Python, too.

I've (personally) seen a use-case for it, for applications. (btw,
virtualenvwrapper is much nicer then doing all that voodoo by hand)

This *is* a bit crazy, but this is the sort of thing virtualenv is good
for --

We use the requirements.txt to work out a python application's
dependencies. Let's say app1 needs pyfoo1, and app2 needs pyfoo2 (they
broke api in pyfoo2)

When we install a lib, it gets shuffled into something out of the way,
and off the python path. /usr/share/debpy/libfoo/1/*,
/usr/share/debpy/libfoo/2/*. When we install an app, it creates a
virtualenv in /var/lib somewhere, and symlinks packages /usr/share into
the env.

This way, you can run app1 and app2 on the same system without conflicts
globally (ok, so we're back to what SONAME / symbol versioning can do)

Clearly, this idea is *insane*, but it's the sort of solution most
hardcore pythonistas I know would go for, which would make us
happy(ish), and them happy(ish).

btw -- let's not do this. it's insane.

>
> > I guess I should revise abstract to aim a listener wondering "why should
> > I care about Debian if there is virtualenv" WITHOUT explicitly pointing
> > to its pros to not cause any flames. And not sure I would be able to
> > convince hard-core Python-ians, so I might not even try and orient
> > it more toward users/admins.
>
> I don't think you should worry too much about flames. My impression is that
> the packaging people are trying to scale back their ambitions and just get
> everyone to do the basic things like write decent metadata, mostly because I
> think the process of delivering a comprehensive solution is deadlocked once
> again, and I think that people do see the need to hear from distributions and
> to try and get as much input from them as possible.

I just don't want a good discussion to get deraled by a few people
saying globally installing modules results in a bad dependency hell, and
we're tainting their whatever.
- Other Paul
signature.asc

Yaroslav Halchenko

unread,
Sep 28, 2012, 10:00:01 AM9/28/12
to

On Fri, 28 Sep 2012, Paul Tagliamonte wrote:

> > The Python conference scene seems to love testing, so if you can make a case
> > for Debian and quality assurance, and Debian has done things popular with
> > this crowd for years like automated builds and the use of very strict package
> > building tools that won't let you build without a precise specification of
> > the build dependencies, then that may appeal to some people.

> ^^ this is a great idea. It'd be nice if we could prototype a flake8 /
> pyflakes run against the archive, and filter for serious errors

I thought to provide stats on how many packages do have build-time
testing... then depending on the number of them may be do some analysis
to see if I could state that build-time tested packages have less bug
reports filed/opened (accounting at the same time against
popularity).

> Clearly, this idea is *insane*, but it's the sort of solution most
> hardcore pythonistas I know would go for, which would make us
> happy(ish), and them happy(ish).

> btw -- let's not do this. it's insane.

;)

> > I don't think you should worry too much about flames. My impression is that
> > the packaging people are trying to scale back their ambitions and just get
> > everyone to do the basic things like write decent metadata, mostly because I
> > think the process of delivering a comprehensive solution is deadlocked once
> > again, and I think that people do see the need to hear from distributions and
> > to try and get as much input from them as possible.
> I just don't want a good discussion to get deraled by a few people
> saying globally installing modules results in a bad dependency hell, and
> we're tainting their whatever.

yeah -- I would try to avoid discussion by simply stating that it works
in many cases and Debian is a clear example for that, so there is no
point to disguss it any further.

> > Paul
> - Other Paul
--
Yarik, just Yarik


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20120928135...@onerussian.com

Yaroslav Halchenko

unread,
Sep 28, 2012, 10:00:01 AM9/28/12
to

On Fri, 28 Sep 2012, Paul Wise wrote:
> > ^^ this is a great idea. It'd be nice if we could prototype a flake8 /
> > pyflakes run against the archive, and filter for serious errors
> We did do that at one point with pyflakes:
> http://qa.debian.org/daca/pyflakes/sid/
> Unfortunately no-one has been working on DACA recently, hint hint ;)

if only it was reincarnated... then we could also wrap it into a nice
matrix form with summary counts of warnings/errors etc

--
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20120928135...@onerussian.com

Scott Kitterman

unread,
Sep 28, 2012, 10:40:01 AM9/28/12
to
On Friday, September 28, 2012 09:53:42 AM Yaroslav Halchenko wrote:
> On Fri, 28 Sep 2012, Paul Wise wrote:
> > > ^^ this is a great idea. It'd be nice if we could prototype a flake8 /
> > > pyflakes run against the archive, and filter for serious errors
> >
> > We did do that at one point with pyflakes:
> > http://qa.debian.org/daca/pyflakes/sid/
> > Unfortunately no-one has been working on DACA recently, hint hint ;)
>
> if only it was reincarnated... then we could also wrap it into a nice
> matrix form with summary counts of warnings/errors etc

http://packages.qa.debian.org/l/lintian4python.html

Scott K


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/15158119.O9jAAX1m8D@scott-latitude-e6320

Barry Warsaw

unread,
Sep 28, 2012, 10:50:02 AM9/28/12
to
On Sep 28, 2012, at 09:04 AM, Yaroslav Halchenko wrote:

>> PEP396,
>
>Status: Draft
>Barry, was there any progress?

Nope. For whatever reason (maybe not enough controversy), there have been no
discussions on this in ages. I really should try to resurrect it.

>I could start with this one of cause ;) but I hope they all know about
>it by now. On a related note though: __file__ -- are we all friends
>again ? ;)

Not sure what you mean about enmity with __file__, but note that as of the
acceptance of PEP 420 (namespace packages), in Python 3.3, __file__ is
completely optional as an attribute of module objects.

Cheers,
-Barry


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20120928104...@limelight.wooz.org

Barry Warsaw

unread,
Sep 28, 2012, 10:50:03 AM9/28/12
to
On Sep 28, 2012, at 09:47 AM, Paul Tagliamonte wrote:

>^^ this is a great idea. It'd be nice if we could prototype a flake8 /
>pyflakes run against the archive, and filter for serious errors

First, we need to get tools like pyflakes ported to Python 3. It's rather
crazy that pyflakes will complain about print() functions unless you put the
appropriate __future__ import at the top.

(And actually, several of us are threatening to work on just this at the
upcoming UDS-R in Copenhagen.)

>I've (personally) seen a use-case for it, for applications. (btw,
>virtualenvwrapper is much nicer then doing all that voodoo by hand)

And don't forget that virtual environmentalism is built into Python 3.3.

-Barry
signature.asc

Yaroslav Halchenko

unread,
Sep 28, 2012, 1:00:02 PM9/28/12
to

> >I could start with this one of cause ;) but I hope they all know about
> >it by now. On a related note though: __file__ -- are we all friends
> >again ? ;)

> Not sure what you mean about enmity with __file__, but note that as of the
> acceptance of PEP 420 (namespace packages), in Python 3.3, __file__ is
> completely optional as an attribute of module objects.

I just vaguely remember that there were problems in some projects relying on
__file__ (some times opportunistically with os.path.realpath) to deduce the
path to other components of the project, which might had been symlinked
elsewhere ;-) and that is what dh_python has addressed:

Date: Fri, 15 Jan 2010 11:58:56 +0100
From: Piotr Ożarowski <pi...@debian.org>
To: debian...@lists.debian.org
Subject: Re: new dh_python proposal
...
* broken modules that use __file__ incorrectly will work without problems,


--
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20120928165...@onerussian.com

Barry Warsaw

unread,
Sep 28, 2012, 1:30:02 PM9/28/12
to
On Sep 28, 2012, at 12:53 PM, Yaroslav Halchenko wrote:

>I just vaguely remember that there were problems in some projects relying on
>__file__ (some times opportunistically with os.path.realpath) to deduce the
>path to other components of the project, which might had been symlinked
>elsewhere ;-) and that is what dh_python has addressed:

Ah, yes, don't do that. :) Use the pkg_resources API if you need to locate
files within your package.

Cheers,
-Barry


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20120928132...@resist.wooz.org

Yaroslav Halchenko

unread,
Sep 28, 2012, 5:20:04 PM9/28/12
to
Thanks everyone who helped with the submission!

Just for the reference -- here is the submitted version ( I believe it
will be possible to change it later on as well ):


Submitted by
Yaroslav Halchenko
Category
Packaging # I guess it was the closest to "distribution"
Audience Level
Intermediate
Extreme?
No
Duration
I prefer a 45 minute slot
Description

Debian delivers a complete operating system with a rich collection of
Python modules and extensions as its integral part. Such "*all* batteries
included" approach allows anyone to safely dive into using Python without being
exposed to possible internal complexity of build- and run-time requirements.
This talk will present the Python world of Debian with its offerings, standards
and QA efforts.

Abstract

Python has found appreciation not only among professional developers
but also among students, scientists and programming novices due to its
scripting nature, "batteries included", good collection of 3rd party libraries,
and ability to interface to libraries written in other languages and computing
environments (e.g. R). To conveniently deliver such a versatile Python platform
to users (and their humble system administrators), the Python community have
been developing the ultimate Python distribution utilities and bundling
pre-built Python and core 3rd party libraries and modules for distribution on
proprietary systems. Meanwhile nearly for two decades Python has been a part of
the largest community-driven software distribution platform -- Debian. The
Debian project delivers a complete operating system with tens of thousands of
FOSS projects available on 11 hardware architectures and 3 different kernels
(Linux, HURD, kFreeBSD). Being a binary distribution Debian guarantees safer --
free of build-errors -- installations and seamless upgrades. Coupled with the
standardized specification of build and run-time dependencies, it makes it easy
to build, verify, or simply deploy projects with complicated interdependencies
and using a variety of technologies. Since Debian attempts to support all
technologies equally well, Python-based products are first class citizens in
this heterogeneous distribution ecosystem, ensuring that Python works well with
the rest of it. Recent advances in hardware virtualization support, followed in
tandem with the explosion of cloud solutions, has made Debian systems popular
not only among Linux "fan-boys" but for various, especially scientific and
community-driven, deployments. The ease with which thousands of Python-based
FOSS have been made available and maintainable through Debian have made it the
Python distribution with all batteries included, even those like virtualenv
which also address package and dependency management.

In this talk I would like to briefly present the history of Python in
Debian (which can be traced back to the 1990s with Python 1.4), outline the
benefits Debian provides for Python users/developers and present what to expect
in the upcoming stable (wheezy) release of Debian. To familiarize listeners
with the Python-in-Debian ecosystem I will then give an overview of core
package naming, versioning, and modularization conventions in Debian and
ongoing QA efforts (build-time testing, full-archive rebuilds, etc). I will
briefly present the "Debian packaging" helper tools, including a recent GSOC
project aiming to provide automatic packaging of the packages on PyPI. To
facilitate the synergy between Python and Debian communities, I will emphasize
on common sense practices (following PEPs, thoroughly tracking contributions
and licensing, continuous integration testing, etc.) which would make any
Debian packaging and maintainership more efficient and benefit upstream
developers. I am planning to conclude by presenting few easy ways on how to
start using Debian.

As the outcome of the talk, I expect listeners to become more
familiar with the Debian project goals, standards and principles, become aware
of integration aspects involved in delivering such a plethora of Python FOSS
solutions, and become intrigued enough to try Debian on their systems or in the
cloud.

--
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20120928211...@onerussian.com

Éric Araujo

unread,
Sep 28, 2012, 5:30:04 PM9/28/12
to
> PEP390
That one is retired. setup.cfg is defined by documentation, not a PEP.

Cheers


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/5066161...@netwok.org

Nicolas Chauvat

unread,
Oct 2, 2012, 7:20:02 AM10/2/12
to
Hi Yaroslav,

On Wed, Sep 26, 2012 at 11:40:58AM -0400, Yaroslav Halchenko wrote:
> To not be too ambitious and to not invest too much time I have decided to
> submit only a talk. Here follows a perspective title, abstract and some
> notes/outline which will not be a part of submission. I would really
> appreciate (and of cause would acknowledge in the slides) any feedback, ideas,
> comments, etc.

I suggest you would also try to describe the differences between The
Complete Python Distribution On Debian and the others ways there are
to install Python packages.

When I say "I do not need all this easy_install, pip, virtualenv,
distribute/packaging, buildout, /etc/ for I have Debian!", I am
usually told:

- but we have to work on Windows
- but we are not root on the computer we are using and can't run apt-get
- but I want a newer version of X than the one included in Debian
- but I am not doing deployment/production and for development I need the
latest versions of these modules because this component I rely on
says so
- I am preparing things for production, so I need everything to be
reproducible independently of the underlying system
- etc.

I think being prepared to answer these questions and maybe address
some of these issues directly in your slides would help make clear
what Debian is a good solution for.

Possible answers are:

- windows: if it hurts, stop doing it and install virtualbox :p
- not root: try a virtual machine (or maybe a variant of chroot?)
- newer: are you ready to handle all the compatibility/dependency
problems on your own ?
- dev: packaging python modules is easier than getting a full
distribution to work right, take a look at the
GSoC project that packages PyPI/*, your new-and-shiny stuff is
probably there
- prod: you want a chroot or a virtual machine.
- etc.

Hope this helps,

PS: by the way, would anyone know of a way to use chroot or something
similar to allow any user to have any number of virtual environments
that use apt-get to install stuff and fall-back to the system if
something is not installed in the virtualenv ?

--
Nicolas Chauvat

logilab.fr - services en informatique scientifique et gestion de connaissances


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/2012100208...@volans.logilab.fr

Nicolas Chauvat

unread,
Oct 2, 2012, 8:50:02 AM10/2/12
to
Hi,

On Fri, Sep 28, 2012 at 10:40:27AM -0400, Barry Warsaw wrote:
> On Sep 28, 2012, at 09:47 AM, Paul Tagliamonte wrote:
>
> >^^ this is a great idea. It'd be nice if we could prototype a flake8 /
> >pyflakes run against the archive, and filter for serious errors
>
> First, we need to get tools like pyflakes ported to Python 3. It's rather
> crazy that pyflakes will complain about print() functions unless you put the
> appropriate __future__ import at the top.
>
> (And actually, several of us are threatening to work on just this at the
> upcoming UDS-R in Copenhagen.)

As far as I know, pylint already runs with Python3. Doesn't it?

--
Nicolas Chauvat

logilab.fr - services en informatique scientifique et gestion de connaissances


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20121002124...@volans.logilab.fr

Yaroslav Halchenko

unread,
Oct 2, 2012, 9:00:03 AM10/2/12
to
Hi Nicolas,

Thanks for the feedback -- valid concerns and besides first 3 points
indeed you give the answers I am usually give people: that is why we
provide NeuroDebian VM which is used by quite a few users who either
have admin access on their boxes or just pursuade IT personnel to do
just 1 custom installation for them -- VirtualBox ;-) As for
chroot'ing -- it is underused/under-marketed solution IMHO ATM although
I have been using it myself quite a bit and even at times advocating it
as a workaround for some deployment problems [1]. Also I haven't
played with fakechroot yet, which if possible to perfect, could serve as
an ultimate resolution for people without admin access.

As for the developers/production: first indeed VM still might be a
better choice, second -- we still provide all the Pythonic tools for
virtualization/isolation, and after all I decided to position the
talk more toward users so I hope not to fall into this debate ;)

[1] http://neuro.debian.net/blog/2011/2011-12-12_schroot_fslview.html

> PS: by the way, would anyone know of a way to use chroot or something
> similar to allow any user to have any number of virtual environments
> that use apt-get to install stuff and fall-back to the system if
> something is not installed in the virtualenv ?

what is your use case? chrootting does allow 'arbitrary' number of
virtual environments but I am not sure what kind of fall-back you need.
Usually I used chroots in the opposite way, but supplementing main
system with tools ran in chroots (see [1] above ;) )

Cheers
Yaroslav O. Halchenko
Postdoctoral Fellow, Department of Psychological and Brain Sciences
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20121002125...@onerussian.com

Thomas Kluyver

unread,
Oct 2, 2012, 9:10:02 AM10/2/12
to
On 2 October 2012 09:01, Nicolas Chauvat <nicolas...@logilab.fr> wrote:
> PS: by the way, would anyone know of a way to use chroot or something
> similar to allow any user to have any number of virtual environments
> that use apt-get to install stuff and fall-back to the system if
> something is not installed in the virtualenv ?

I'd be interested to see something like this, or something that would
allow installing Debian packages for a single user. Is something
possible with UnionFS/OverlayFS?

Thomas


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/CAOvn4qgPKgcgwEA+5a_o27ch4go4-sKLpqG9nFv7KzhO=Du...@mail.gmail.com

Barry Warsaw

unread,
Oct 2, 2012, 10:00:02 AM10/2/12
to
On Oct 02, 2012, at 02:42 PM, Nicolas Chauvat wrote:

>As far as I know, pylint already runs with Python3. Doesn't it?

pyflakes is the one we want to port.

Cheers,
-Barry
signature.asc

Nicolas Chauvat

unread,
Oct 2, 2012, 1:00:03 PM10/2/12
to
May I ask why ?

--
Nicolas Chauvat

logilab.fr - services en informatique scientifique et gestion de connaissances


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20121002165...@volans.logilab.fr

Dmitrijs Ledkovs

unread,
Oct 2, 2012, 1:10:01 PM10/2/12
to
On 02/10/12 17:57, Nicolas Chauvat wrote:
> On Tue, Oct 02, 2012 at 09:59:32AM -0400, Barry Warsaw wrote:
>> On Oct 02, 2012, at 02:42 PM, Nicolas Chauvat wrote:
>>
>>> As far as I know, pylint already runs with Python3. Doesn't it?
>>
>> pyflakes is the one we want to port.
>
> May I ask why ?
>

Because it was used with python2 code bases, now code is ported to
python3 and we'd like to continue using pyflakes with out python3 code
as before without leaving compatibility future imports around, just for
pyflakes to work.

ps. I'm with barry on this one, btw ;-)

--
Regards,
Dmitrijs.

signature.asc

Jakub Wilk

unread,
May 18, 2013, 7:00:01 PM5/18/13
to
* Barry Warsaw <ba...@python.org>, 2012-09-28, 10:40:
>we need to get tools like pyflakes ported to Python 3.

pyflakes >= 0.6 supports Python 3. There's no Debian package built for
Python 3 yet, but that's something that can be kluged up:

1) Install pyflakes_0.6.1-1~exp1 from experimental.
2) Put the attached script somewhere in your $PATH.
3) ...
4) PROFIT!^W^WGrumble that it doesn't work out of the box, and that you
need to resort to hacks like this instead.

--
Jakub Wilk
py3flakes

Barry Warsaw

unread,
May 24, 2013, 3:00:02 PM5/24/13
to
I'm working on updating svn to 0.7.2 and adding Python 3 support. I'll ask
for a review once I get something working locally.

Here's a question though, and it comes up again and again in different
contexts (e.g. nosetests). I guess we should have a /usr/bin/pyflakes and
/usr/bin/pyflakes3, yeah?

-Barry


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20130524145522.19fef5f2@anarchist

Barry Warsaw

unread,
May 24, 2013, 3:50:02 PM5/24/13
to
On May 24, 2013, at 02:55 PM, Barry Warsaw wrote:

>I'm working on updating svn to 0.7.2 and adding Python 3 support. I'll ask
>for a review once I get something working locally.

r9670

>Here's a question though, and it comes up again and again in different
>contexts (e.g. nosetests). I guess we should have a /usr/bin/pyflakes and
>/usr/bin/pyflakes3, yeah?

A few suggestions from IRC:

py3flakes - What I don't like about that is that it's harder to locate(1).

Replace the driver (Python) scripts with a shell script that takes a --py3
flag and invokes designated interpreter on the actual code. I rather like
this, although I can see there are some complications. You probably have to
sed the right interpreter(s) into a .in file and the actual entry point has to
live somewhere. You probably also want a --py2 flag for consistency, and for
when Python 3 is the default in Debian <0.5 wink>. You also have to strip out
that argument and pass in all the others to the real driver script. What if
those options collide with options the real script accepts?

I kind of like the idea of such a driver shell script. Thoughts?

-Barry


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20130524154030.1a6c8df2@anarchist

Jakub Wilk

unread,
May 24, 2013, 4:10:03 PM5/24/13
to
* Barry Warsaw <ba...@python.org>, 2013-05-24, 15:40:
>>Here's a question though, and it comes up again and again in different
>>contexts (e.g. nosetests). I guess we should have a /usr/bin/pyflakes
>>and /usr/bin/pyflakes3, yeah?

My answer normally would be "ask upstream what is their preference".
Although in this case the plan didn't work out:
https://bugs.launchpad.net/pyflakes/+bug/1132892
Either upstream doesn't care; or they do, but they're annoyed with the
discussion style.

>A few suggestions from IRC:
>
>py3flakes

+1

>- What I don't like about that is that it's harder to locate(1).

Who uses that anyway? ;)

>Replace the driver (Python) scripts with a shell script

Eww! :(

--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/2013052420...@jwilk.net

Barry Warsaw

unread,
May 24, 2013, 4:20:01 PM5/24/13
to
On May 24, 2013, at 10:02 PM, Jakub Wilk wrote:

>My answer normally would be "ask upstream what is their preference". Although
>in this case the plan didn't work out:
>https://bugs.launchpad.net/pyflakes/+bug/1132892 Either upstream doesn't
>care; or they do, but they're annoyed with the discussion style.

Thanks, I'd missed that bug. In any case, I started a conversation on
python-dev to see if we can get some general consensus upstream.

>>A few suggestions from IRC:
>>
>>py3flakes
>
>+1
>
>>- What I don't like about that is that it's harder to locate(1).
>
>Who uses that anyway? ;)

Maybe just dinosaurs? <wink>. But it also makes ls a bit more difficult. I
guess I'm a -0 on the pyfoo/py3foo naming convention.

>>Replace the driver (Python) scripts with a shell script
>
>Eww! :(

Can you elaborate on what you don't like about that? I'd like to at least
accurately understand your objections.

Cheers,
-Barry


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20130524161249.73709cb9@anarchist

Dmitrijs Ledkovs

unread,
May 24, 2013, 9:00:02 PM5/24/13
to
On 24 May 2013 20:40, Barry Warsaw <ba...@python.org> wrote:
> On May 24, 2013, at 02:55 PM, Barry Warsaw wrote:
>
> py3flakes - What I don't like about that is that it's harder to locate(1).
>

I like pyflakes3, but better yet I'd prefer:
python3 -m flakes

similar to how compileall / unittest / et al work.

Regards,

Dmitrijs.


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/CANBHLUh-EjCi6vrcxAFBGRJK...@mail.gmail.com

Jakub Wilk

unread,
May 25, 2013, 6:20:02 PM5/25/13
to
* Barry Warsaw <ba...@python.org>, 2013-05-24, 16:12:
>>>Replace the driver (Python) scripts with a shell script
>>Eww! :(
>Can you elaborate on what you don't like about that? I'd like to at
>least accurately understand your objections.

I want Python scripts to remain Python scripts, so that I can:
- run then with non-default Python interpreter easily;
- write a wrapper script that execfile()s the original script.

--
Jakub Wilk


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/2013052522...@jwilk.net

Barry Warsaw

unread,
May 28, 2013, 11:30:01 AM5/28/13
to
On May 25, 2013, at 01:52 AM, Dmitrijs Ledkovs wrote:

>On 24 May 2013 20:40, Barry Warsaw <ba...@python.org> wrote:
>> On May 24, 2013, at 02:55 PM, Barry Warsaw wrote:
>>
>> py3flakes - What I don't like about that is that it's harder to locate(1).
>>
>
>I like pyflakes3, but better yet I'd prefer:
>python3 -m flakes
>
>similar to how compileall / unittest / et al work.

I think we should *always* encourage upstream to provide a -m version for
invoking their script. The question then is whether and how to provide
/usr/bin scripts. We're having an interesting discussion on the topic in
python-dev.

-Barry


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20130528112234.1d841c44@anarchist

Barry Warsaw

unread,
May 28, 2013, 11:30:01 AM5/28/13
to
On May 26, 2013, at 12:13 AM, Jakub Wilk wrote:

>* Barry Warsaw <ba...@python.org>, 2013-05-24, 16:12:
>>>>Replace the driver (Python) scripts with a shell script
>>>Eww! :(
>>Can you elaborate on what you don't like about that? I'd like to at >least accurately understand your objections.
>
>I want Python scripts to remain Python scripts, so that I can:
>- run then with non-default Python interpreter easily;
>- write a wrapper script that execfile()s the original script.

Thanks, that's helpful. I definitely agree with your first point. I haven't
really ever used your second.

-Barry


--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20130528112319.58b907c5@anarchist
0 new messages