Is it worth back porting PEP 3147 to Python < 3.2?
Barry Warsaw
about Python on both Debian and Ubuntu gets a chance to weigh in.
On Friday, Guido approved and I landed the implementation of PEP 3147 on the
py3k trunk (what will be Python 3.2). This allows multiple versions of Python
to coexist on the same system without the need for symlinks to handle pyc file
incompatibility.
http://www.python.org/dev/peps/pep-3147/
This will be officially supported in Python starting with 3.2. It will not
however be available in upstream Python 2.7. The PEP does recognize however
that distros may want to back port the feature to get its advantages now.
Although I have not yet tried to do so, I think the essential elements of the
PEP should be fairly easy to back port to Python 2.6, 2.7 and 3.1. The
question is, should we do this?
As the PEP outlines, my preference would be to back port it but *not* enable
this by default in any Python < 3.2. Instead, the back port would add a
-Xenablecachedir switch, and an associated $PYTHONENABLECACHEDIR environment
variable that would have to be used to enable PEP 3147 pyc paths. Of course,
this would have to be supported in python-support and python-central too, but
I believe Piotr has started working on this.
With the back port it means that any Python modules installed by apt will not
need symlinks in order to share both the source and pyc locations. Normal
Python developers working on their own code however would still get
traditional pyc file locations by default (but of course, there would be
nothing stopping them from setting the above switch/envar).
It's possible that Debian and Ubuntu will come to different decisions about
what to do, but it would be nice if we could reach consensus. I would be
willing to do the work to back port the essential bits to earlier Pythons, or
at least spend some time to see whether it is feasible. I know that given the
release schedule of Python 2.7, it's possible that Ubuntu 10.10 could include
2.6, 2.7, and 3.1, though I don't think that's been decided yet. I'm not sure
how Python 2.7's scheduled release date of 2010-06-26 matches Debian's release
schedule. (Given that lenny was released on 2010-01-30 I'm guessing it will
be a while.)
Before I start working on this though, I'd like to get your opinion as to
whether it is worth it.
Thanks,
-Barry
Omer Zak
Yes, please backport PEP 3147 to at least Python 2.7.
The rationale: we'll need to support both Python 2.x and Python 3.x for
several years, and it will be nice if the same library package can be
made to support both 2.x and 3.x.
It would also be nice to define a way to specify that a tree of Python
2.7 scripts are to be compiled into Python 3.x bytecode, by specifying
that the compilation process has first to run 2to3
(http://docs.python.org/py3k/library/2to3.html#to3-reference) on the
Python scripts and actually compile the 2to3's output.
--- Omer
On Mon, 2010-04-19 at 17:53 -0400, Barry Warsaw wrote:
> On Friday, Guido approved and I landed the implementation of PEP 3147 on the
> py3k trunk (what will be Python 3.2). This allows multiple versions of Python
> to coexist on the same system without the need for symlinks to handle pyc file
> incompatibility.
>
> http://www.python.org/dev/peps/pep-3147/
>
> This will be officially supported in Python starting with 3.2. It will not
> however be available in upstream Python 2.7. The PEP does recognize however
> that distros may want to back port the feature to get its advantages now.
> Although I have not yet tried to do so, I think the essential elements of the
> PEP should be fairly easy to back port to Python 2.6, 2.7 and 3.1. The
> question is, should we do this?
--
$ python
>>> type(type(type))
<type 'type'> My own blog is at http://www.zak.co.il/tddpirate/
My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html
--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/1271735426.4...@c2.home.zak.co.il
Julian Andres Klode
> My take of the situation:
> Yes, please backport PEP 3147 to at least Python 2.7.
> The rationale: we'll need to support both Python 2.x and Python 3.x for
> several years, and it will be nice if the same library package can be
> made to support both 2.x and 3.x.
>
> It would also be nice to define a way to specify that a tree of Python
> 2.7 scripts are to be compiled into Python 3.x bytecode, by specifying
> that the compilation process has first to run 2to3
> (http://docs.python.org/py3k/library/2to3.html#to3-reference) on the
> Python scripts and actually compile the 2to3's output.
Use build_py_2to3 from distutils.command.build_py, for example:
if sys.version_info[0] == 3:
from distutils.command.build_py import build_py_2to3
cmdclass['build_py'] = build_py_2to3
--
Julian Andres Klode - Debian Developer, Ubuntu Member
See http://wiki.debian.org/JulianAndresKlode and http://jak-linux.org/.
Piotr Ożarowski
> My take of the situation:
> Yes, please backport PEP 3147 to at least Python 2.7.
> The rationale: we'll need to support both Python 2.x and Python 3.x for
> several years, and it will be nice if the same library package can be
> made to support both 2.x and 3.x.
you cannot (in most cases) share 2.X and 3.X Python code, so adding it
to 2.7 and not to 2.6 doesn't make sense
> It would also be nice to define a way to specify that a tree of Python
> 2.7 scripts are to be compiled into Python 3.x bytecode, by specifying
> that the compilation process has first to run 2to3
> (http://docs.python.org/py3k/library/2to3.html#to3-reference) on the
> Python scripts and actually compile the 2to3's output.
that has to be done at build time
> On Mon, 2010-04-19 at 17:53 -0400, Barry Warsaw wrote:
> > This will be officially supported in Python starting with 3.2. It will not
> > however be available in upstream Python 2.7. The PEP does recognize however
> > that distros may want to back port the feature to get its advantages now.
> > Although I have not yet tried to do so, I think the essential elements of the
> > PEP should be fairly easy to back port to Python 2.6, 2.7 and 3.1. The
> > question is, should we do this?
I doubt we'll touch 2.X packages in Debian Squeeze and in Squeeze+1
we'll most probably have only 1 Python 2.X version (if any) so I don't
see a point in backporting it in Debian.
If you want to ship two 2.X Python versions in Ubuntu, you could use my
new dh_python for that (I hope to finish it soon), but you'd have to
convert *all* packages to it and lets face it, python2.5 and python2.6
transitions in Ubuntu (at least in universe) were... well not even
close to Debian's quality and these transitions didn't require that much
work...
(IMHO, /me waits for other opinions)
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Barry Warsaw
>[Omer Zak, 2010-04-20]
>> My take of the situation:
>> Yes, please backport PEP 3147 to at least Python 2.7.
>> The rationale: we'll need to support both Python 2.x and Python 3.x for
>> several years, and it will be nice if the same library package can be
>> made to support both 2.x and 3.x.
>
>you cannot (in most cases) share 2.X and 3.X Python code, so adding it
>to 2.7 and not to 2.6 doesn't make sense
Just thinking out loud: How insane would it be to think of an import hook that
could do it at run time? Or, a compilall switch that would do it when the pyc
file was generated? You'd need some way to specify that a particular module
could/should not be auto-converted, or conversely if automatic support were
more the exception, some way to say it can be done. I wonder if something in
the debian/[control|rules] file could trigger that.
>I doubt we'll touch 2.X packages in Debian Squeeze and in Squeeze+1
>we'll most probably have only 1 Python 2.X version (if any) so I don't
>see a point in backporting it in Debian.
Good to know, thanks.
>If you want to ship two 2.X Python versions in Ubuntu, you could use my
>new dh_python for that (I hope to finish it soon), but you'd have to
>convert *all* packages to it and lets face it, python2.5 and python2.6
>transitions in Ubuntu (at least in universe) were... well not even
>close to Debian's quality and these transitions didn't require that much
>work...
Sorry, I don't really know the history of any of that so I can't comment. But
I would like to know more about your new dh_python, what changes it would
require, etc. Where can I find it, or information about it?
-Barry
Omer Zak
> On Apr 20, 2010, at 06:50 AM, Omer Zak wrote:
>
> >My take of the situation:
> >Yes, please backport PEP 3147 to at least Python 2.7.
> >The rationale: we'll need to support both Python 2.x and Python 3.x for
> >several years, and it will be nice if the same library package can be
> >made to support both 2.x and 3.x.
> >
> >It would also be nice to define a way to specify that a tree of Python
> >2.7 scripts are to be compiled into Python 3.x bytecode, by specifying
> >that the compilation process has first to run 2to3
> >(http://docs.python.org/py3k/library/2to3.html#to3-reference) on the
> >Python scripts and actually compile the 2to3's output.
>
> favor of back porting the PEP to Python 2.x.
Julian Andres Klode suggested a way to do so.
DISCLAIMER: I didn't verify that it actually works.
On Tue, 2010-04-20 at 07:47 +0200, Julian Andres Klode wrote:
>
> Use build_py_2to3 from distutils.command.build_py, for example:
>
> if sys.version_info[0] == 3:
> from distutils.command.build_py import build_py_2to3
> cmdclass['build_py'] = build_py_2to3
--
$ python
>>> type(type(type))
<type 'type'> My own blog is at http://www.zak.co.il/tddpirate/
My opinions, as expressed in this E-mail message, are mine alone.
They do not represent the official policy of any organization with which
I may be affiliated in any way.
WARNING TO SPAMMERS: at http://www.zak.co.il/spamwarning.html
--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/1271796600.5...@c2.home.zak.co.il
Barry Warsaw
>My take of the situation:
>Yes, please backport PEP 3147 to at least Python 2.7.
>The rationale: we'll need to support both Python 2.x and Python 3.x for
>several years, and it will be nice if the same library package can be
>made to support both 2.x and 3.x.
>
>It would also be nice to define a way to specify that a tree of Python
>2.7 scripts are to be compiled into Python 3.x bytecode, by specifying
>that the compilation process has first to run 2to3
>(http://docs.python.org/py3k/library/2to3.html#to3-reference) on the
>Python scripts and actually compile the 2to3's output.
If we can come up with a reasonable way to make that work, then I'd be in
favor of back porting the PEP to Python 2.x. Has anybody run a systematic
test to see which of the Python packages that are available in Debian work in
Python 3 with an automatic 2to3 conversion? Probably the trickiest part will
be working out the dependences (i.e. if a package you're dependent on isn't
2to3'able, then you aren't either).
I do think that if we back port to 2.7 we should also back port to 2.6. While
I don't yet know what the default Python version will be for Ubuntu 10.10, I
can certainly imagine it might be 2.6, with 2.7 available but not default. If
so, then it definitely makes sense to back port to both. If 10.10 includes
only Python 2.7, then sure, we'll only back port to that version.
I guess the same kind of systematic test for package compatibility with Python
3 would be useful to run for compatibility with Python 2.7. Does such a thing
exist?
-Barry
Scott Kitterman
I think it is difficult to know for sure what the future will hold. If the
backport is not technically complex, I think a backport with the default to off
would be a nice tool in the box is things go differently than we plan. Other
than the effort involved, I don't see a downside, but I have no idea how hard
this would be to do.
Scott K
--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/201004201657...@kitterman.com
Piotr Ożarowski
> On Apr 20, 2010, at 10:14 AM, Piotr Ożarowski wrote:
>
> >[Omer Zak, 2010-04-20]
> >> My take of the situation:
> >> Yes, please backport PEP 3147 to at least Python 2.7.
> >> The rationale: we'll need to support both Python 2.x and Python 3.x for
> >> several years, and it will be nice if the same library package can be
> >> made to support both 2.x and 3.x.
> >
> >you cannot (in most cases) share 2.X and 3.X Python code, so adding it
> >to 2.7 and not to 2.6 doesn't make sense
>
> Just thinking out loud: How insane would it be to think of an import hook that
> could do it at run time?
2to3 is not reliable, at least not for now. Example: even today I
backported Python 3 related patch from tip and upstream *did* test it
with 2to3 and python3.X before releasing Python 3.X compatible version.
The second python3-* package I maintain doesn't provide Python
extension for now, as it didn't even build for 3.X version; module (that
had to use custom 2to3 rules, BTW) uses it optionally, though.
If it doesn't always work with modules tested with python3, I'm pretty
sure it will not always work with modules written when python3.X wasn't
even released upstream so runtime conversion is not acceptable, IMHO.
Every maintainer has to check the module before providing python3-foo
package and that requires manpower Ubuntu doesn't have. That's
why Python transitions in Ubuntu didn't work that well in the past (I
don't recall a single one completed before Debian, and Ubuntu started
with 2.6 few years before us...)
> But
> I would like to know more about your new dh_python, what changes it would
> require, etc. Where can I find it, or information about it?
it will ship symlinks instead of creating them at runtime (and use
list of exceptions instead of shipping a list of files to compile) in
"old" mode and in PEP3147 mode, paths like:
/usr/share/python3/foo/__init__.py
/usr/share/python3/foo/bar.py
/usr/share/python3/foo/3.2-/bar.py
/usr/share/python3/3.1-3.5/baz.py
using __path__ to choose the right source file
(like in the example[1] I gave you the other day). Note that right now
it requires changing __init__.py file and that's where namespace PEP
comes in (and new .pth files).
See threads on debian-python mailing list with "dh_python" in Subject
for more info, some things changed since then (and I'm too sleepy right
now to list them :-P), though
What changes it will require? Well, I plan to provide wrappers that will
emulate dh_pycentral/dh_pysupport's API, but as recent
site-packages -> dist-packages transition showed - every package has to
be checked by its maintainer (there are lots of corner cases not easy to
predict like depending on helper tool's internal paths)
Right now only pycompile[2] and pyclean[3] are available in public.
[1] http://people.debian.org/~piotr/__path__.tar.gz
[2] http://people.debian.org/~piotr/pycompile
[3] http://people.debian.org/~piotr/pyclean
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Piotr Ożarowski
> If 10.10 includes
> only Python 2.7, then sure, we'll only back port to that version.
why do you want to backport it to 2.X for a single python2.x package?
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Barry Warsaw
>I think it is difficult to know for sure what the future will hold. If the
>backport is not technically complex, I think a backport with the default to
>off would be a nice tool in the box is things go differently than we plan.
>Other than the effort involved, I don't see a downside, but I have no idea
>how hard this would be to do.
It may not be worth porting the entire patch, but I think the core interesting
bits won't be too difficult. The one tricky extra thing to implement is
switching it on only when -Xenablecachedir is given. In Python 3.2, it's
always enabled.
We'll see how things go. If Ubuntu 10.10 is going to include multiple
versions of Python 2.x, then I think it makes sense. With one 2.x and one 3.x
it's not quiet as clear.
-Barry
Barry Warsaw
>Based on my experience with the Python 2.5 and 2.6 transitions, I'd really
>prefer to have Ubuntu follow Debian rather than lead.
Ideally, yes I'd like the same thing, but I'll need help because I'm so new to
Debian.
So if it makes sense for Ubuntu and doesn't harm Debian, then it would be
acceptable to patch Python in Debian even if it doesn't help much (because
Debian won't have multiple versions of Python 2.x)?
-Barry
Barry Warsaw
>[Barry Warsaw, 2010-04-20]
>> If 10.10 includes
>> only Python 2.7, then sure, we'll only back port to that version.
>
>why do you want to backport it to 2.X for a single python2.x package?
It would only make sense if we wanted to share source between multiple
versions of Python. So if 10.10 includes only Python 2.7 and we do not think
it's feasible or worth it to also share (many) packages with Python 3, then
the back port isn't worth it.
-Barry
Barry Warsaw
>2to3 is not reliable, at least not for now.
I agree that there's no way we can just enable it by default. Too many
upstream packages need modifications to work in Python 3. However, for those
that are Python 3-ready, it's a useful tool. The question then is whether we
want to enable it for those packages that it works with.
>Example: even today I backported Python 3 related patch from tip and upstream
>*did* test it with 2to3 and python3.X before releasing Python 3.X compatible
>version. The second python3-* package I maintain doesn't provide Python
>extension for now, as it didn't even build for 3.X version; module (that had
>to use custom 2to3 rules, BTW) uses it optionally, though. If it doesn't
>always work with modules tested with python3, I'm pretty sure it will not
>always work with modules written when python3.X wasn't even released upstream
>so runtime conversion is not acceptable, IMHO.
>
>Every maintainer has to check the module before providing python3-foo
>package and that requires manpower Ubuntu doesn't have. That's
>why Python transitions in Ubuntu didn't work that well in the past (I
>don't recall a single one completed before Debian, and Ubuntu started
>with 2.6 few years before us...)
How much of the transition testing is automated? It would be very interesting
for example, to have a test framework that could run any combination of Python
packages against various versions of Python, and get a report on the success
or failure of it. This may not be a project for the distros of course - I
think upstream Python would be very interested in something like this. For
example, a tool that grabbed packages from the Cheeseshop and tested them
against different versions would be cool. If snakebite.org ever gets off the
ground, that might be the best place to put something like this together
(though we'd care less about OSes that aren't Debian and Ubuntu).
>> But I would like to know more about your new dh_python, what changes it
>> would require, etc. Where can I find it, or information about it?
>
>it will ship symlinks instead of creating them at runtime (and use
>list of exceptions instead of shipping a list of files to compile) in
>"old" mode and in PEP3147 mode, paths like:
>
> /usr/share/python3/foo/__init__.py
> /usr/share/python3/foo/bar.py
> /usr/share/python3/foo/3.2-/bar.py
> /usr/share/python3/3.1-3.5/baz.py
Those are the symlinks, right? E.g. /usr/share/python3/foo/3.2-/bar.py would
symlink to /usr/shared/pyshared/bar.py?
I like the idea of shipping symlinks, but I thought they were created at
runtime to handle the case where a new version of Python was installed after
the package was installed. Is that the case, and if so, wouldn't all the
packages have to be updated when a new version of Python is supported?
>using __path__ to choose the right source file
>(like in the example[1] I gave you the other day). Note that right now
>it requires changing __init__.py file and that's where namespace PEP
>comes in (and new .pth files).
Cool.
>See threads on debian-python mailing list with "dh_python" in Subject
>for more info, some things changed since then (and I'm too sleepy right
>now to list them :-P), though
:). I'll look those up.
>What changes it will require? Well, I plan to provide wrappers that will
>emulate dh_pycentral/dh_pysupport's API, but as recent
>site-packages -> dist-packages transition showed - every package has to
>be checked by its maintainer (there are lots of corner cases not easy to
>predict like depending on helper tool's internal paths)
>
>Right now only pycompile[2] and pyclean[3] are available in public.
>
>[1] http://people.debian.org/~piotr/__path__.tar.gz
>[2] http://people.debian.org/~piotr/pycompile
>[3] http://people.debian.org/~piotr/pyclean
Thanks for the links. I'd like to help in any way I can.
-Barry
Scott Kitterman
"Barry Warsaw" <ba...@canonical.com> wrote:
Based on my experience with the Python 2.5 and 2.6 transitions, I'd really prefer to have Ubuntu follow Debian rather than lead.
Scott K
Piotr Ożarowski
> On Apr 21, 2010, at 12:01 AM, Piotr Ożarowski wrote:
> >2to3 is not reliable, at least not for now.
>
> I agree that there's no way we can just enable it by default. Too many
> upstream packages need modifications to work in Python 3. However, for those
> that are Python 3-ready, it's a useful tool. The question then is whether we
> want to enable it for those packages that it works with.
I was referring to the idea of using 2to3 at runtime. If 2to3 passes
tests at build time and upstream author/maintainer is satisfied with
results, we should use it, sure.
> How much of the transition testing is automated?
last time I was testing my cdbs / python-central changes, I followed
these steps:
* create cowbuilder env. with custom packages
* download all packages that are affected by the change
(build depend on python-central / python-support /
python{-all,}{,-dev}, etc. - grep-dctrl is very handy here)
* rebuilt all of them using slightly modified[1]
/usr/share/doc/pbuilder/examples/rebuild/buildall script
* read debc output, check if everything is ok, if not, find a pattern
and grep for it in other files, then use mass-bug[2] to file bugs
* read FTBFS logs, and do the same as with debc output files
[1] f.e. save `debc package.changes` output. It takes few days to
rebuilt all packages so make sure the script works correctly and
you have enough disk space :)
[2] I actually used my own Python script (/me maintains too many Python
template engines to not use them whenever he can ;)
> It would be very interesting
> for example, to have a test framework that could run any combination of Python
> packages against various versions of Python, and get a report on the success
> or failure of it.
IMHO we should encourage maintainers to enable tests and
build/bytecompile .py files for all supported Python versions, even in
Architecture: all packages (I found few bugs in my packages this way).
You can test if binary packages pass the byte compilation, but unit
tests are available usually only in source packages.
> > /usr/share/python3/foo/__init__.py
> > /usr/share/python3/foo/bar.py
> > /usr/share/python3/foo/3.2-/bar.py
> > /usr/share/python3/3.1-3.5/baz.py
this should be /usr/lib/python3/... actually
> Those are the symlinks, right? E.g. /usr/share/python3/foo/3.2-/bar.py would
> symlink to /usr/shared/pyshared/bar.py?
no, /usr/lib/python3/foo/3.2-/bar.py is the location in .deb and the
final location of foo.py on the disk (i.e. the final runtime location -
/usr/share/python3/ will be in all pytho3.X's sys.path), that's what
PEP3147 is about, no? No more symlinks. In above example bar.py has
different source for Python 3.1 and >= 3.2. It should actually be:
/usr/lib/python3/foo/bar.py
/usr/lib/python3/foo/3.1/bar.py
or f.e.
/usr/lib/python3/foo/bar.py
/usr/lib/python3/foo/-3.3/bar.py
if bar.py is the same in 3.1&3.2 and a little bit modified in >= 3.4
> I like the idea of shipping symlinks
symlinks will be shipped only in 2.X packages (if one will decide to
use new dh_python instead of pycentral/pysupport) and only if PEP3147
will not be backported (this approach requires rebuilding arch:all
packages which doesn't have many fans)
> , but I thought they were created at
> runtime to handle the case where a new version of Python was installed after
> the package was installed. Is that the case, and if so, wouldn't all the
> packages have to be updated when a new version of Python is supported?
pycompile (invoked[3] when new version is installed/removed) will take care
of creating appropriate __pycache__/bar.cpython-*.pyc files and pyclean
will remove them (including __pycache__ directory, if not needed anymore)
[3] see /usr/share/python/runtime.d/* for examples
Nicolas Chauvat
Nice to see someone of the core python team taking part in
distribution development.
On Thu, Apr 22, 2010 at 01:52:11PM -0400, Barry Warsaw wrote:
> How much of the transition testing is automated? It would be very interesting
> for example, to have a test framework that could run any combination of Python
> packages against various versions of Python, and get a report on the success
> or failure of it. This may not be a project for the distros of course - I
> think upstream Python would be very interested in something like this. For
> example, a tool that grabbed packages from the Cheeseshop and tested them
> against different versions would be cool. If snakebite.org ever gets off the
> ground, that might be the best place to put something like this together
> (though we'd care less about OSes that aren't Debian and Ubuntu).
Unfortunately, Logilab does not have a much man-power to offer to set
this up at the moment, but would something like
http://apycot.hg-scm.org/ fit your description of a test framework ?
We also have it running at logilab.org and cubicweb.org of course:
http://www.logilab.org/view?rql=testconfig&vid=summary
http://www.cubicweb.org/view?rql=testconfig&vid=summary
As you can see with these second and third links, tests include
lintian and piuparts checks.
Is it something like this that you had in mind?
--
Nicolas Chauvat
logilab.fr - services en informatique scientifique et gestion de connaissances
--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20100426163...@volans.logilab.fr
Barry Warsaw
>Nice to see someone of the core python team taking part in
>distribution development.
Well, it's official now - I've joined the Ubuntu platform team at Canonical,
so I'm very keen on helping to solve problems with Python on Debian and Ubuntu.
>On Thu, Apr 22, 2010 at 01:52:11PM -0400, Barry Warsaw wrote:
>> How much of the transition testing is automated? It would be very interesting
>> for example, to have a test framework that could run any combination of Python
>> packages against various versions of Python, and get a report on the success
>> or failure of it. This may not be a project for the distros of course - I
>> think upstream Python would be very interested in something like this. For
>> example, a tool that grabbed packages from the Cheeseshop and tested them
>> against different versions would be cool. If snakebite.org ever gets off the
>> ground, that might be the best place to put something like this together
>> (though we'd care less about OSes that aren't Debian and Ubuntu).
>
>Unfortunately, Logilab does not have a much man-power to offer to set
>this up at the moment, but would something like
>http://apycot.hg-scm.org/ fit your description of a test framework ?
That's for continuous integration of Mercurial, right?
>We also have it running at logilab.org and cubicweb.org of course:
>http://www.logilab.org/view?rql=testconfig&vid=summary
>http://www.cubicweb.org/view?rql=testconfig&vid=summary
>
>As you can see with these second and third links, tests include
>lintian and piuparts checks.
>
>Is it something like this that you had in mind?
Yes. What are you using to drive this? I'm not really up on CI tools, but
Hudson has been getting a lot of buzz.
What I like about your display is that a failure in one area does not
necessary mean a failure elsewhere. That way you can better see the overall
health of the package.
What I have in mind is defining a set of best practices, embodied as much as
possible in tools and libraries, that provide carrots to Python developers, so
that if they adhere to these best practices, the can get lots of benefits such
as nearly automatic and effortless packaging in Debian and Ubuntu. It's
things like 'python setup.py test' just working, and it has an impact on PyPI,
documentation, release management, etc. These best practices can be
opinionated and simple. If they cover only 80% of Python packages, that's
fine. Developers would never be forced to adhere to them, but it would be to
their advantage to do so.
-Barry
Nicolas Chauvat
[discussion started at
http://lists.debian.org/debian-python/2010/04/msg00046
should we continue or trim some of the cc'ed lists ?]
On Mon, Apr 26, 2010 at 06:41:16PM -0400, Barry Warsaw wrote:
> On Apr 26, 2010, at 06:35 PM, Nicolas Chauvat wrote:
> >On Thu, Apr 22, 2010 at 01:52:11PM -0400, Barry Warsaw wrote:
> >> How much of the transition testing is automated? It would be very interesting
> >> for example, to have a test framework that could run any combination of Python
> >> packages against various versions of Python, and get a report on the success
> >> or failure of it. This may not be a project for the distros of course - I
> >> think upstream Python would be very interested in something like this. For
> >> example, a tool that grabbed packages from the Cheeseshop and tested them
> >> against different versions would be cool. If snakebite.org ever gets off the
> >> ground, that might be the best place to put something like this together
> >> (though we'd care less about OSes that aren't Debian and Ubuntu).
> >
> >Unfortunately, Logilab does not have much man-power to offer to set
> >this up at the moment, but would something like
> >http://apycot.hg-scm.org/ fit your description of a test framework ?
>
> That's for continuous integration of Mercurial, right?
Yes.
> >We also have it running at logilab.org and cubicweb.org of course:
> >http://www.logilab.org/view?rql=testconfig&vid=summary
> >http://www.cubicweb.org/view?rql=testconfig&vid=summary
> >
> >As you can see with these second and third links, tests include
> >lintian and piuparts checks.
> >
> >Is it something like this that you had in mind?
>
> Yes. What are you using to drive this? I'm not really up on CI tools, but
> Hudson has been getting a lot of buzz.
>
> http://hudson-ci.org/
We are using http://www.logilab.org/project/apycot that is GPL
software mainly developed and maintained by Logilab, but slowly
reaching out to a larger audience.
It uses a web framework to store the information in a db and provide a
web user interface, plus slave testing bots running on one or more
hosts that get the next task from the queue, execute it and store the
results in the db.
> What I like about your display is that a failure in one area does not
> necessary mean a failure elsewhere. That way you can better see the overall
> health of the package.
You may find interesting the following blog posts about apycot and
ways to display its information http://bit.ly/9dZQQE
> as nearly automatic and effortless packaging in Debian and Ubuntu.
We tried fully automatic packaging of Python programs years (8?) ago
and did not succeed for distutils and setuptools were too far away
from Debian packaging concerns.
Introducing in mypackage/__pkginfo__.py and mypackage/setup.py all the
information needed to generate the debian/* files without the need to
modify them eventually meant more or less copying their whole content,
for their is actually not much to generate. It also meant using a less
efficient toolchain because of the added conversion step.
We moved to having tools that check the consistency of the information
provided by __pkginfo__ and debian/* files and make it easier to build
the Debian packages. These tools are
http://www.logilab.org/project/logilab-devtools
Packaging a piece of Python software now requires a bit of (easy) work
at first, but following releases only need one or two commands. And
all the dh_python* helper scripts reduced that work even further.
> What I have in mind is defining a set of best practices, embodied as much as
> possible in tools and libraries, that provide carrots to Python developers, so
> that if they adhere to these best practices, the can get lots of benefits such
> ...
> It's things like 'python setup.py test' just working, and it has an
> impact on PyPI, documentation, release management, etc. These best
> practices can be opinionated and simple. If they cover only 80% of
> Python packages, that's fine. Developers would never be forced to
> adhere to them, but it would be to their advantage to do so.
Sounds good to me :)
--
Nicolas Chauvat
logilab.fr - services en informatique scientifique et gestion de connaissances
Full link to blog posts:
http://www.cubicweb.org/view?rql=Any+X+WHERE+X+is+BlogEntry%2C+T+tags+X%2C+T+name+%22apycot%22
--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/2010042708...@volans.logilab.fr
Barry Warsaw
>[discussion started at
>http://lists.debian.org/debian-python/2010/04/msg00046
>should we continue or trim some of the cc'ed lists ?]
Is there a better place to discuss Python packaging issues that are of
interest to both Debian and Ubuntu? Are all Ubuntu developers who care about
shared Python concerns generally members of debian-python? I don't know these
things. :)
>We are using http://www.logilab.org/project/apycot that is GPL
>software mainly developed and maintained by Logilab, but slowly
>reaching out to a larger audience.
There's also buildbot of course, which I guess it the granddad of Python CI
tools, kind of.
>It uses a web framework to store the information in a db and provide a
>web user interface, plus slave testing bots running on one or more
>hosts that get the next task from the queue, execute it and store the
>results in the db.
Are there things like a API (REST or otherwise) for pulling data out of
apycot?
>> What I like about your display is that a failure in one area does not
>> necessary mean a failure elsewhere. That way you can better see the overall
>> health of the package.
>
>You may find interesting the following blog posts about apycot and
>ways to display its information http://bit.ly/9dZQQE
>
>> as nearly automatic and effortless packaging in Debian and Ubuntu.
>
>We tried fully automatic packaging of Python programs years (8?) ago
>and did not succeed for distutils and setuptools were too far away
>from Debian packaging concerns.
>
>Introducing in mypackage/__pkginfo__.py and mypackage/setup.py all the
>information needed to generate the debian/* files without the need to
>modify them eventually meant more or less copying their whole content,
>for their is actually not much to generate. It also meant using a less
>efficient toolchain because of the added conversion step.
I'm not familiar with __pkginfo__.py, but I do agree that setup.py makes
automated packaging difficult. We need a declarative syntax that can be
consumed by more tools, which is why I'm so excited about Tarek's work in
distutils-sig.
>We moved to having tools that check the consistency of the information
>provided by __pkginfo__ and debian/* files and make it easier to build
>the Debian packages. These tools are
>http://www.logilab.org/project/logilab-devtools
Is there a wiki or online documentation documenting these tools, or is it all
in the source?
>Packaging a piece of Python software now requires a bit of (easy) work
>at first, but following releases only need one or two commands. And
>all the dh_python* helper scripts reduced that work even further.
Is that easy work manual or automated? What does it take to Debianize
random-simple-pypi-package? (By that I mean "run a script" or "inspect
setup.py and write the debian/*" or "...?".
>> What I have in mind is defining a set of best practices, embodied as much as
>> possible in tools and libraries, that provide carrots to Python developers, so
>> that if they adhere to these best practices, the can get lots of benefits such
>> ...
>> It's things like 'python setup.py test' just working, and it has an
>> impact on PyPI, documentation, release management, etc. These best
>> practices can be opinionated and simple. If they cover only 80% of
>> Python packages, that's fine. Developers would never be forced to
>> adhere to them, but it would be to their advantage to do so.
>
>Sounds good to me :)
Right now, it's just an idea. There are a few existing attempts at this, so
it's worth looking at what exists first.
-Barry
Nicolas Chauvat
> On Apr 27, 2010, at 10:01 AM, Nicolas Chauvat wrote:
> There's also buildbot of course, which I guess it the granddad of Python CI
> tools, kind of.
Yes, buildbot is the old-timer and has loads of features. As far as I
know, extracting information out of buildbot is more difficult than it
is with a tool like Apycot that's built on top of CubicWeb.
> Are there things like a API (REST or otherwise) for pulling data out of
> apycot?
Sure, that's a basic functionnality of CubicWeb.
For example, just take any url and append vid=xml
http://apycot.hg-scm.org/projectenvironment/hg/full/108464
becomes
http://apycot.hg-scm.org/projectenvironment/hg/full/108464?vid=xml
If the data that's extracted isn't what you expect, file a ticket at
logilab.org/project/apycot and the 'xml' view will be enhanced.
As you could read from http://www.cubicweb.org/blogentry/779839 the
views you apply to data sets are not restricted to producing html or
xml, they can also directly produce json or pdf, graphs, etc.
Other examples of the view mechanism would be the ones that use a
specific vocabulary as in
http://www.logilab.org/project/apycot?vid=doap
or
http://www.cubicweb.org/blogentry/779839?vid=sioc
or
http://www.cubicweb.org/cwuser/nchauvat?vid=foaf
In short, CubicWeb was *designed* to publish its data under reusable
formats, html being just one way to present data.
Maybe we can restrict this part of the discussion to the cubicweb
mailing list ?
> I'm not familiar with __pkginfo__.py,
__pkginfo__ is a declarative format used at Logilab.
http://hg.logilab.org/logilab/devtools/file/tip/doc/pkginfo_variables.txt
> We need a declarative syntax that can be consumed by more tools,
> which is why I'm so excited about Tarek's work in distutils-sig.
I agree, Tarek's work is a great improvement over the current
situation.
> Is there a wiki or online documentation documenting these tools, or is it all
> in the source?
It is mainly in the source. You will find several people that will be
happy to help if you ask your questions on python-projects at
lists.logilab.org.
> Is that easy work manual or automated? What does it take to Debianize
> random-simple-pypi-package? (By that I mean "run a script" or "inspect
> setup.py and write the debian/*" or "...?".
Easy manual work. I'm cc'ing the people that do it often so that they
can provide details. Alexandre, Sylvain ?
--
Nicolas Chauvat
logilab.fr - services en informatique scientifique et gestion de connaissances
--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20100427200...@volans.logilab.fr
Sylvain Thénault
> On Tue, Apr 27, 2010 at 11:07:51AM -0400, Barry Warsaw wrote:
> > Is that easy work manual or automated? What does it take to Debianize
> > random-simple-pypi-package? (By that I mean "run a script" or "inspect
> > setup.py and write the debian/*" or "...?".
>
> Easy manual work. I'm cc'ing the people that do it often so that they
> can provide details. Alexandre, Sylvain ?
once you've metadata easily accessible it becomes possible to generate
a good part of the packaging. Though to get a proper debian packaging,
you'll need hand tuning in most cases. We used to have a tool generating
debian package from __pkginfo__ files (containing extra information for
that packaging), but that was painful to maintain, so we stopped using
it to do it by hand. That doesn't take much time, and you usually don't
create new project everyday...
With standardized metadata, and so a wider audience, such a tool
handling most of the repeating work would probably be worth it.
--
Sylvain Thénault LOGILAB, Paris (France)
Formations Python, Debian, Méth. Agiles: http://www.logilab.fr/formations
Développement logiciel sur mesure: http://www.logilab.fr/services
CubicWeb, the semantic web framework: http://www.cubicweb.org
--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/20100428112...@lupus.logilab.fr
Piotr Ożarowski
I think it's worth to backport it to python3.1 even if it will be the
only 3.X version supported in Squeeze as this should ease the transition
to python3.2 in Squeeze+1 and give some time to various tools to adopt
to PEP3147 changes.
There are not so many packages that support Python 3 for now and Lenny
doesn't have it at all so this transition shouldn't be too problematic.
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Barry Warsaw
>What do you think about backporting it to Python 3.1 in Squeeze?
>
>I think it's worth to backport it to python3.1 even if it will be the
>only 3.X version supported in Squeeze as this should ease the transition
>to python3.2 in Squeeze+1 and give some time to various tools to adopt
>to PEP3147 changes.
>
>There are not so many packages that support Python 3 for now and Lenny
>doesn't have it at all so this transition shouldn't be too problematic.
It sounds like a good idea. A few questions:
* Would you enable __pycache__ by default?
* If so, would you provide a command line option to disable it?
The easiest thing of course would be a high fidelity back port of exactly
what's in py3k: it's always on and you can't disable it. I bet that would
back port fairly easily. It's a bit more work to change that behavior but
users of Python 3.1 might be surprised by the difference from upstream (the
same questions apply to any back port to Python 2.x).
-Barry
Piotr Ożarowski
> On May 05, 2010, at 11:54 AM, Piotr Ożarowski wrote:
> >What do you think about backporting it to Python 3.1 in Squeeze?
> It sounds like a good idea. A few questions:
>
> * Would you enable __pycache__ by default?
yes
> * If so, would you provide a command line option to disable it?
I'd not provide one unless someone here will shout loudly ;-)
[...]
> users of Python 3.1 might be surprised by the difference from upstream (the
> same questions apply to any back port to Python 2.x).
I want the same behaviour in all 3.X versions so that Debian developers
and users get used to __pycache__ and will not start writing
tools (that deal with .pyc files) for 3.X that will not work in
Squeeze+1 (with 3.2).
--
Piotr Ożarowski Debian GNU/Linux Developer
www.ozarowski.pl www.griffith.cc www.debian.org
GPG Fingerprint: 1D2F A898 58DA AF62 1786 2DF7 AEF6 F1A2 A745 7645
--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Lino Mastrodomenico
> users of Python 3.1 might be surprised by the difference from upstream
It might be useful mentioning somewhere that the best way to detect if
the Python implementation used supports PEP 3147 is:
import imp
hasattr(imp, 'get_tag')
(or something similar) and not checking for an hard version number.
--
Lino Mastrodomenico
--
To UNSUBSCRIBE, email to debian-pyt...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Archive: http://lists.debian.org/o2xcc93256f1005060747zb...@mail.gmail.com
Barry Warsaw
>2010/5/5 Barry Warsaw <ba...@python.org>:
>> users of Python 3.1 might be surprised by the difference from upstream
>
>It might be useful mentioning somewhere that the best way to detect if
>the Python implementation used supports PEP 3147 is:
>
>import imp
>hasattr(imp, 'get_tag')
>
>(or something similar) and not checking for an hard version number.
That's a good point, thanks. I'll add that to the PEP.
-Barry