Migration to Python 3.x

[Stéfan van der Walt]

Hi everyone,

The previous migration thread stalled, and I would like to revisit the issue.

Here is my proposal:

1. All code written for the 0.13 release of scikit-image should still be Python 2.7 and 3.x compatible.

2. 0.13.x will be declared a "Long Term Support" release, with all point releases supporting Python 2.7 until 2020
3. Non-API-breaking features from 0.14 and later *may be backported* to 0.13.x as needed / requested.

4. Any new code written after 0.13 will be Python 3.x only (but see #3 above).

My motivations are to:

- Remove developer overhead associated with keeping both Python 2 and 3-isms in mind

- Reduce testing and maintenance overhead associated with multiple Python versions

- Utilize new language features such as the @-operator and required keyword arguments

Please let me know if you have any strong feelings to the contrary.

Best regards

Stéfan

[Emmanuelle Gouillart]

Hi Stéfan,

about backporting, what kind of backporting mechanism do you have in
mind? Merging commits by hand, or something more elaborate?

It may be a silly idea, but can't we do it the other way around, that is
have an experimental branch where developers who wish to write
Python3-only code can work, and in which commits to master are merged if
they are Python 3 compatible? It could give us some time to evaluate
whether most developers want to write code that is not compatible with
Python 2.

Best,
Emma

[Stéfan van der Walt]

Hi Emmanuelle

On 9 August 2016 at 14:44, Emmanuelle Gouillart <emmanuelle...@nsup.org> wrote:

about backporting, what kind of backporting mechanism do you have in
mind? Merging commits by hand, or something more elaborate?

Yes, I think backporting bug-fixes from time to time and making a point release should suffice.  The ports may sometimes require some 3-to-2'ing.

 

It may be a silly idea, but can't we do it the other way around, that is
have an experimental branch where developers who wish to write
Python3-only code can work, and in which commits to master are merged if
they are Python 3 compatible? It could give us some time to evaluate
whether most developers want to write code that is not compatible with
Python 2.

I don't think that's a silly idea, but I also don't think that addresses two of the points I raised:

- Remove developer overhead associated with keeping both Python 2 and 3-isms in mind

I'd like developers to start thinking in Python 3, not the other way around.  Barriers for Python 3 implementations should be removed, not raised, given that that's the direction we'll have to move eventually.

- Utilize new language features such as the @-operator and required keyword arguments

These are not available unless we make Python 3 the default.

As for whether most developers *want* to write code that is incompatible with Python 2, my guess is that they won't pay much attention.  I've moved to Python 3 more than a year ago already, and quite a few of my personal projects already do not run under Python 2---not because I actively pushed for Python 3, just because I didn't explicitly go back and fix it for the 2.7 case.  I also can't think of many reasons to justify that effort.

The only people who should be affected by the suggested change are those stuck on Python 2 but need the latest scikit-image features.  But perhaps they can also contribute some of the (fairly lightweight) backporting efforts to help out?

Best regards

Stéfan

[Ralf Gommers]

On Wed, Aug 10, 2016 at 9:58 AM, Stéfan van der Walt <ste...@berkeley.edu> wrote:

Hi Emmanuelle

On 9 August 2016 at 14:44, Emmanuelle Gouillart <emmanuelle...@nsup.org> wrote:

 

It may be a silly idea, but can't we do it the other way around, that is
have an experimental branch where developers who wish to write
Python3-only code can work, and in which commits to master are merged if
they are Python 3 compatible? It could give us some time to evaluate
whether most developers want to write code that is not compatible with
Python 2.

I don't think that's a silly idea, but I also don't think that addresses two of the points I raised:

- Remove developer overhead associated with keeping both Python 2 and 3-isms in mind

I'd like developers to start thinking in Python 3, not the other way around.  Barriers for Python 3 implementations should be removed, not raised, given that that's the direction we'll have to move eventually.

- Utilize new language features such as the @-operator and required keyword arguments

These are not available unless we make Python 3 the default.

Hi Stefan, do you propose then to drop Python 3.4 at the same time (@ is >= 3.5)?

Ralf

[Stéfan van der Walt]

Hi Ralf

On 9 August 2016 at 15:09, Ralf Gommers <ralf.g...@gmail.com> wrote:

Hi Stefan, do you propose then to drop Python 3.4 at the same time (@ is >= 3.5)?

Yes, I guess I am.  But if that's too much of a stretch, I will be open to >= 3.4.

Stéfan

[Juan Nunez-Iglesias]

@Emmanuelle I'm probably among the ones pushing hardest for this, and I can tell you, I can't wait for this change in policy, and would be sorely disappointed by having to work in an experimental branch of scikit-image. Both @-matmul and keyword-only arguments are, imho, compelling reasons to switch. (Imagine the amount of fiddling with the API that we could do with keyword-only arguments, without the annoying deprecation cycle!)

@Ralf I would argue in favour of 3.5, for the above reasons and because anyone who had the temerity to update to 3.4 is very likely to subsequently move to 3.5. (Self-selected group of early adopters, plus no backwards incompatibility issues between the two versions.)

Juan.

--
You received this message because you are subscribed to the Google Groups "scikit-image" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image+unsubscribe@googlegroups.com.
To post to this group, send email to scikit...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/scikit-image/CABDkGQmSC7Q7jZJDBuhutj2PXjQTDGDirxafVw-GFuqcN%3DK1_A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Nelle Varoquaux]

On 9 August 2016 at 17:28, Juan Nunez-Iglesias <jni....@gmail.com> wrote:
> @Emmanuelle I'm probably among the ones pushing hardest for this, and I can
> tell you, I can't wait for this change in policy, and would be sorely
> disappointed by having to work in an experimental branch of scikit-image.
> Both @-matmul and keyword-only arguments are, imho, compelling reasons to
> switch. (Imagine the amount of fiddling with the API that we could do with
> keyword-only arguments, without the annoying deprecation cycle!)
>
> @Ralf I would argue in favour of 3.5, for the above reasons and because
> anyone who had the temerity to update to 3.4 is very likely to subsequently
> move to 3.5. (Self-selected group of early adopters, plus no backwards
> incompatibility issues between the two versions.)

That implies that even ubuntu users will have to install python from
another source than the package manager. Do you really want this? That
means that only fairly advance python users will be able to use the
latest scikit-image release.

>
> Juan.
>
> On Wed, Aug 10, 2016 at 8:14 AM, Stéfan van der Walt <ste...@berkeley.edu>
> wrote:
>>
>> Hi Ralf
>>
>> On 9 August 2016 at 15:09, Ralf Gommers <ralf.g...@gmail.com> wrote:
>>>
>>> Hi Stefan, do you propose then to drop Python 3.4 at the same time (@ is
>>> >= 3.5)?
>>
>>
>> Yes, I guess I am. But if that's too much of a stretch, I will be open to
>> >= 3.4.
>>
>> Stéfan
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "scikit-image" group.
>> To unsubscribe from this group and stop receiving emails from it, send an

>> email to scikit-image...@googlegroups.com.

>> To post to this group, send email to scikit...@googlegroups.com.
>> To view this discussion on the web, visit
>> https://groups.google.com/d/msgid/scikit-image/CABDkGQmSC7Q7jZJDBuhutj2PXjQTDGDirxafVw-GFuqcN%3DK1_A%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google Groups
> "scikit-image" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to scikit-image...@googlegroups.com.

> To post to this group, send email to scikit...@googlegroups.com.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/scikit-image/CA%2BJHcKSM0sERcWwSMXc4dk6uhaAHbBwqtMcK70608b_D3-2H0g%40mail.gmail.com.

[Ralf Gommers]

On Wed, Aug 10, 2016 at 1:55 PM, Nelle Varoquaux <nelle.v...@gmail.com> wrote:

On 9 August 2016 at 17:28, Juan Nunez-Iglesias <jni....@gmail.com> wrote:
> @Emmanuelle I'm probably among the ones pushing hardest for this, and I can
> tell you, I can't wait for this change in policy, and would be sorely
> disappointed by having to work in an experimental branch of scikit-image.
> Both @-matmul and keyword-only arguments are, imho, compelling reasons to
> switch. (Imagine the amount of fiddling with the API that we could do with
> keyword-only arguments, without the annoying deprecation cycle!)
>
> @Ralf I would argue in favour of 3.5, for the above reasons and because
> anyone who had the temerity to update to 3.4 is very likely to subsequently
> move to 3.5. (Self-selected group of early adopters, plus no backwards
> incompatibility issues between the two versions.)

I agree with this argument, I was just asking to clarify. @ is at least something interesting that's specifically added for scientific users, so imho 3.5 is the first 3.x release where in some cases the benefits may start to be worth the costs.
 

That implies that even ubuntu users will have to install python from
another source than the package manager. Do you really want this? That
means that only fairly advance python users will be able to use the
latest scikit-image release.

In the grand scheme of things, does Ubuntu matter much for this decision? There are way more Windows and OS X users, so if it's OK for them (which is not a given) then it should also be OK for those fewer and on average more computer-literate Ubuntu users.

Ralf

[Nelle Varoquaux]

I don't have a good overview of what OS people are using, but in the
different research facilities I've worked or visited, it was always
linux based, and users where not really tech-savy. I checked on all of
the servers (UW's genome science, the Curie institute, UC Berkeley's
stats department, the Mines' machine learning computing facilities.) I
have access to, none have python3.5. Some of these servers are
"offline", thus conda is useless. All of these research institutes
have teams that use in some way image processing.

I'll just also mentionned that we had a keynote at scipy this year
mentionning she was using python because she just did not have the
time anymore to code in C++ and Java. These are the kind of people
that may be using scikit-image, that may be interested in the latest
version of the package, and yet just not have the time to compile
python3.5 from scratch on an outdated server.

N

>
> Ralf

>
> --
> You received this message because you are subscribed to the Google Groups
> "scikit-image" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scikit-image...@googlegroups.com.
> To post to this group, send email to scikit...@googlegroups.com.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/scikit-image/CABL7CQhEaD0uy0SyrcNO_j%2BDsE%3DX8ALKvsk4XF3v6_bzOocmEw%40mail.gmail.com.

[Ralf Gommers]

Fair enough, but there are likely much more outdated servers with Python 2.7 than with Python 3.4 on them .....

Ralf

[Nelle Varoquaux]

Well… I think it is better to keep python2.7 support :)
All the clusters I have access to but 1 has python3. The oldest one is
a red hat based cluster with python2.6 (which was python2.3 not so
long ago).

N

>
> Ralf
>
> --
> You received this message because you are subscribed to the Google Groups
> "scikit-image" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scikit-image...@googlegroups.com.
> To post to this group, send email to scikit...@googlegroups.com.
> To view this discussion on the web, visit

> https://groups.google.com/d/msgid/scikit-image/CABL7CQjye5g5jppRk1hej2g2P3VCKa0mrcHe4Z9u37nSrqBpsg%40mail.gmail.com.

[Johannes Schönberger]

As stated earlier, a big -1 from my side on moving to all Python 3.x for 0.14 already. Not because I personally don't think it provides some benefits, but let me explain:

I don't see enough of a benefit of replacing a few np.dot's with the @ syntax or pure keyword based arguments for the few functions we add/change in one version jump. It doesn't make the code faster, more correct or more feature complete. The API will look and work the same from a users perspective from 2.7-3.5. Required features (which are usually rare) from newer numpy/scipy versions can be handled easily inside the respective functions with an exception.

In my experience, many of the scikit-image users I know are actually less knowledgable about how to setup a new Python environment from scratch. And it seems the latest Ubuntu is still on 3.4 and Mac also doesn't ship with 3.5 - which leaves out the @ syntax anyway. Also, many of the cluster users are still stuck on 2.x for quite a while. I think, we will lose a significant amount of users without a real benefit.

I don't see a compelling reason to lock out X% of users at this point. I can only judge by the feedback in this thread, but 3 out of 5 people seem to have problems with the proposed way forward. It would be nice if there was some way of obtaining installation statistics from PyPI or Anaconda to get a better picture here.

The real benefit comes with 3.5, hence I propose to follow the current model to support 2.7-3.x and reevaluate the situation for 0.15. Eventually, 3.5 will be more widespread and in the meantime we do not really sacrifice anything on our side. Doing a find/replace on np.dot/__future__ will work in a year just as fine as it does now.

Best,
Johannes

> To post to this group, send an email to scikit...@googlegroups.com.
> To view this discussion on the web, visit https://groups.google.com/d/msgid/scikit-image/CAE-UAvT0C5z0jg84oDW0MtTiFkoWJgpdNrMCfGx82ZnwDzLeZg%40mail.gmail.com.

[Emmanuelle Gouillart]

Regarding an estimation of which OS people are using, let me go back to
one of my favorite hobbies :-): looking at the analytics of
http://scikit-image.org/. What it says in terms of OSs:

- 54% of Windows users
- 26% of Linux users
- 20% of OSx users

These statistics are pretty stable over the period where we have
analytics (since last fall). We have 20000 unique visitors on the website
every month, enough to make statistics!

So, the number of Linux users is certainly smaller than the number of
users for which the overwhelming solution will be Anaconda, but it's not
ridiculous either. Also, people working on a cluster with no Internet
access will not visit the website from the cluster, so it's rather an
underestimation of Linux users. And it's likely that a majority of Linux
users are running 2.7 (because of
https://www.python.org/dev/peps/pep-0394/).

I completely agree that it would be great to obtain statistics about
who uses Python 2.7 or 3.x but I can't see an easy way to do it. Could we
have a button on the website linking to a small form asking whether
people are using Python 2.7 or 3.x? Maybe it has already been mentioned
on the mailing-list.

I think the decision boils down to a developers vs. users dilemma --
let's face it, a very classical problem in open source :-). Personally, I
think that scikit-image is far from having reached its potential in terms
of number of users, and that multiplying the number of users by a
factor of 10 is perfectly feasible. Therefore, my personal goal, as a
developer, is rather to focus on the growth of the userbase, meaning
better usability, more documentation, more outreach, etc. I find it much
more satisfying to see the userbase grow than to be able to use the new
features of Python 3.5 -- in an ideal world, we'll have the two, of
course. Letting down 1/4 of users who won't benefit from new features
is no small decision.

That said,

1) if there is an important frustration from at least two of the most
active developers of scikit-image, we'll have to make this move soon
enough, much earlier than 2020 clearly.
2) so, to estimate when the change should happen (0.14, 0.15?), we need
to evaluate the "economics" of such a decision. How many users are going
to be impacted? How much development time are we losing by not switching
right now? How do we balance these two factors (or others)?

All the best,
Emma

On Wed, Aug 10, 2016 at 06:14:45AM +0200, Johannes Schönberger wrote:
> In my experience, many of the scikit-image users I know are actually less knowledgable about how to setup a new Python environment from scratch. And it seems the latest Ubuntu is still on 3.4 and Mac also doesn't ship with 3.5 - which leaves out the @ syntax anyway. Also, many of the cluster users are still stuck on 2.x for quite a while. I think, we will lose a significant amount of users without a real benefit.

> I don't see a compelling reason to lock out X% of users at this point. I can only judge by the feedback in this thread, but 3 out of 5 people seem to have problems with the proposed way forward. It would be nice if there was some way of obtaining installation statistics from PyPI or Anaconda to get a better picture here.

[Stéfan van der Walt]

Hi Nelle

On 9 August 2016 at 18:55, Nelle Varoquaux <nelle.v...@gmail.com> wrote:

That implies that even ubuntu users will have to install python from
another source than the package manager. Do you really want this? That
means that only fairly advance python users will be able to use the
latest scikit-image release.

Ubuntu users haven't had the latest scikit-image for quite a while, unfortunately.  We are working on fixing that, but not many people have noticed.

Stéfan

[Stéfan van der Walt]

On 10 August 2016 at 01:03, Emmanuelle Gouillart <emmanuelle...@nsup.org> wrote:

I completely agree that it would be great to obtain statistics about
who uses Python 2.7 or 3.x but I can't see an easy way to do it. Could we
have a button on the website linking to a small form asking whether
people are using Python 2.7 or 3.x? Maybe it has already been mentioned
on the mailing-list.

I'll work on getting hold of some statistics from PyPA.

 

2) so, to estimate when the change should happen (0.14, 0.15?), we need
to evaluate the "economics" of such a decision. How many users are going
to be impacted? How much development time are we losing by not switching
right now? How do we balance these two factors (or others)?

I don't think the developer cost is very high right now.  I wanted to test the waters a bit and see how the rest of the team felt, but I think there are enough compelling arguments to stick to 2.7-compatibility for now.  We can always revisit the issue again a year or so from now, when the landscape might have changed.

Thanks for all the feedback!

Stéfan

[Juan Nunez-Iglesias]

Hi everyone,

First of all, I excitedly jumped on the bandwagon because Stefan mentioned 0.13, but I actually even I considered that a bit early for dropping Py2 when I read it. =) I would say that the right time to drop Py2 is either 0.15 or 1.0, and that we should follow a standard deprecation cycle. What this means is some warning (on the home page, or on installation, for example) that as of version <foo>.<bar>, scikit-image will be Python 3-only, with the previous release being an LTS release with Python 2 support.

An important corollary to this thread is signing the Python 3 statement, which pledges dropping Python 2 support by **2020**. We can commit to a longer timeline than envisioned by Stéfan's initial proposal, but I would like to commit to a timeline, because I think having the scikit-image logo on that page would definitely add momentum to the transition. Which it (cough) clearly needs.

As Stéfan has noted, Ubuntu users haven't had 0.12 in the package manager in ages and the sky hasn't fallen. A long-term support release is *not* the same as abandoning our users. I would find it much less burdensome to backport certain bug fixes and critical features to 0.13.25 (or whatever) than to keep supporting 2.7 ad infinitum. The deprecation cycle in particular just bugs the hell out of me.

About the benefits of switching, Johannes, although you argue that Python3 doesn't make the code "more correct", this is not strictly the case: keyword-only arguments, better readability (with @), and typing, *all* make it easier to avoid bugs. You can hardly claim that skimage is bug-free. I'll note that in my own project, gala, a user discovered a monster bug that will result in me issuing a correction to the paper:

https://github.com/janelia-flyem/gala/issues/63

Both typing and keyword-only arguments would have prevented this bug.

Emma, thank you so much for those statistics. They are so valuable. =) I agree *a lot* about outreach + docs, and growing our user base. But I think all those efforts should focus on Python 3 anyway.

So anyway, I hope I've added some fuel to the fire with my bug disaster, above. =) And I hope we can agree on a timeline, even if the timeline is not quite as aggressive as Jupyter's or Stéfan's original proposal.

Thanks all!

Juan.

--

You received this message because you are subscribed to the Google Groups "scikit-image" group.

To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image+unsubscribe@googlegroups.com.

To post to this group, send email to scikit...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/scikit-image/CABDkGQ%3DEzzSY1j9qy%2BD3VkzVP4zeHknBrqkBZ%3D9m7Q_H4qSvCA%40mail.gmail.com.

[Thomas Kluyver]

A small factual correction:

On Wednesday, 10 August 2016 05:14:49 UTC+1, Johannes Schönberger wrote:

And it seems the latest Ubuntu is still on 3.4...

This is not accurate - Ubuntu 16.04 (the LTS release that came out in April this year), has Python 3.5:
http://packages.ubuntu.com/xenial/python3

Ubuntu 15.10 is no longer supported (non-LTS releases get 9 months support). Of course there will still be users on 14.04, and even 12.04, for some time.

I don't want to push an aggressive policy on the project, but I would encourage you to make plans for dropping Python 2 support at some time in the next few years, and sign http://python3statement.github.io/ . I know this may feel like 'users vs developers' at the moment, but I don't think it's that simple. The split between Python 3 and 2 has been causing users headaches for several years now. The only realistic way to end this is to complete the transition, so that Python 3 becomes the standard answer for everything but large legacy codebases. There's little impetus to make Python 3 available in institutional environments so long as people assume that all important Python projects will always support Python 2. So when we make this decision as a community, rather than individual projects, I think it ultimately helps both users and developers. These groups also have common interests - developers who spend less effort on compatibility work can pay more attention to more interesting improvements.

With the Python 3 statement, we believe we're setting a relaxed timetable: projects only commit to dropping Python 2 support by 2020, which is still four years away. We expect to be on Python 3.7 by that time. The aim is not to leave Python 2 users in the lurch, but to give everyone plenty of notice that support for Python 2 will come to an end, so that they can plan a transition in good time.

Thanks,
Thomas