Fixing the bash from activate script, again

[Sorin Sbarnea]

Hi!

It seems that the only way to get some attention on virtualenv is to use the plain old mailing list as other measures had zero effect (filing bug, opening PRs, irc messages).

I am trying to fix an old and recurring bug with virtualenv, the fact that it chokes when run using more restrictive bash:

https://github.com/pypa/virtualenv/issues/1029

There are two PRs linked to this bug: one fixes the current bug, the other is adding a test that should assure that the same kind of bug would not appear again.

Me and others are hit by this bug quite often and I think is time to have some safe code inside activate, so we are not forced to do dirty workarounds to make it work.

[Jakub Bocheński]

+1 for what it's worth...

I think it's really sad to see people go out of their way to help improve this project and are just ignored.

Remember this next time you hear a rant about "leeching open source"

[Paul Moore]

On 28 September 2017 at 18:11, Jakub Bocheński <kuba.bo...@gmail.com> wrote:
> +1 for what it's worth...
>
> I think it's really sad to see people go out of their way to help improve this project and are just ignored.

I've already responded in much more detail on the virtualenv-users
list, but I think this is massively over-simplifying the situation.
The proposed fix seems pretty innocuous, but it's for a relatively
limited use case (users running bash who have chosen to set a
particular setting about undeclared variables - sorry I don't recall
all the details, but I'm not a Linux user). It's easy to say "but it
couldn't possibly do any harm to merge it" - but we have users who use
many shells, not just bash, some of which are pretty obscure. The use
of the "${PS1:-}" construct (again, sorry if I got that wrong) may not
be supported on all of those - so we risk breaking the scripts for
some of our users in order to make them work for users who can easily
enough switch off the undeclared variable setting.

I'm not saying the PR shouldn't go in - the above argument could be
used to reject pretty much anything - but it's a judgement call that
needs to be made, and endless "me too" and "why are you ignoring this"
comments make it no easier to assess the risks.

The PyPA team is *very* small, and entirely made up of volunteers, and
there are other projects and issues that are much higher priority than
this issue (e.g. the rewrite of PyPI, the next version of pip, ...) So
sometimes things get left. And sadly, we can't easily get extra
resources that easily - particularly when potential contributors don't
seem to appreciate the responsibilities involved (of taking care when
potentially breaking stuff that our users rely on).

> Remember this next time you hear a rant about "leeching open source"

Next time you hear "project X never responds", remember that I spent a
substantial proportion of the free time I had today responding to
these emails, and as a result didn't get time to work on other PRs
that people have been asking for. Or would you have been happy if I'd
added a one-word "noted" comment to the issue and left it at that?
(Which would probably be about the same effort as many of the "me too"
comments cost their authors).

Paul

[Jakub Bocheński]

I think keeping the discussion in one place might save everybody's time.
It's cool if you prefer mailing list to PR/issue thread but at least linking the discussion there would go a long way.

> Or would you have been happy if I'd
added a one-word "noted" comment to the issue and left it at that?
(Which would probably be about the same effort as many of the "me too"
comments cost their authors).

No. But if you quickly explained why see the value/effort ratio is low here that would be fine.

Now: on the actual issue.
This is not some random bash setting. Setting it on is bash best practice -- don't take my word for it just google it.

> The use
of the "${PS1:-}" construct (again, sorry if I got that wrong) may not
be supported on all of those - so we risk breaking the scripts for
some of our users in order to make them work for users who can easily
enough switch off the undeclared variable setting.

I understand the concern, but it's not as bad as you think.
The ${PS1:-} construct is not a bash extension. It's in the POSIX standard http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02
Any POSIX-compatibile shell will accept this (which is basically any shell in use).

[Paul Moore]

On 28 September 2017 at 19:55, Jakub Bocheński <kuba.bo...@gmail.com> wrote:
> I think keeping the discussion in one place might save everybody's time.
> It's cool if you prefer mailing list to PR/issue thread but at least linking
> the discussion there would go a long way.

Feel free to do that. I've used up all my open source time for today, sorry.

>> Or would you have been happy if I'd
> added a one-word "noted" comment to the issue and left it at that?
> (Which would probably be about the same effort as many of the "me too"
> comments cost their authors).
>
> No. But if you quickly explained why see the value/effort ratio is low here
> that would be fine.

If I could have done that quickly, I would have.

> Now: on the actual issue.
> This is not some random bash setting. Setting it on is bash best practice --
> don't take my word for it just google it.

No need. See below.

>> The use
> of the "${PS1:-}" construct (again, sorry if I got that wrong) may not
> be supported on all of those - so we risk breaking the scripts for
> some of our users in order to make them work for users who can easily
> enough switch off the undeclared variable setting.
>
> I understand the concern, but it's not as bad as you think.
> The ${PS1:-} construct is not a bash extension. It's in the POSIX standard
> http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02
> Any POSIX-compatibile shell will accept this (which is basically any shell
> in use).

I can't comment on whether this is true. As I said, I'm not a Unix
user, and more specifically I have no feel for what's common Unix
practice (at my work, a major proportion of the servers I see use RHEL
5, which as I understand it is ancient, and some use Solaris and AIX,
with shells whose vintage I don't know, but they *certainly* aren't
bash). Luckily for me, I don't need to use Python on those servers...

Let me just be 100% clear here. I will not personally commit this
change. It is not in my area of expertise, and I'm not willing to be
browbeaten into doing so just because people keep telling me it's
fine. We have PyPA members who *are* Unix users, and whose judgement I
will trust on this. But they are very busy with other issues, so they
haven't had the time to look at this ticket. Sorry, but that's the
nature of volunteer-run open source. In the meantime, I've tried to
help give some perspective, by explaining the situation. I could have
just ignored the issue as not in my area of expertise (and indeed
that's what I did for some time). But I thought people might
appreciate at least getting a summary of the position. Maybe I was
wrong - you certainly don't seem pleased that I bothered.

Repeating your comment from above:

> No. But if you quickly explained why see the value/effort ratio is low here
> that would be fine.

Sure doesn't feel like you think it's "fine" that I've spent a number
of hours on this for you.

I'm done with this issue. Sorry you're not happy.
Paul

[Jakub Bocheński]

> Feel free to do that. I've used up all my open source time for today, sorry.

Would love to do that, but the list you are mentioning " virtualenv-users" doesn't seem to exist. At least google group only finds "virtualenv-api users" and it's invitation only

Jakub Bocheński

[Paul Moore]

On 28 September 2017 at 21:11, Jakub Bocheński <kuba.bo...@gmail.com> wrote:
>> Feel free to do that. I've used up all my open source time for today,
>> sorry.
> Would love to do that, but the list you are mentioning " virtualenv-users"
> doesn't seem to exist. At least google group only finds "virtualenv-api
> users" and it's invitation only

It's the one Sorin posted to. I just replied to his mail there.
python-v...@googlegroups.com
Paul

[Ionel Cristian Mărieș]

Just my two cents here. I was a bit angry

​ too​

with virtualenv some way back, up to a point that I've took dstufft's rewrite branch to a better state (supporting up to 3.4, integration tests and CI). But it was same deal, no one would really look at it - a cruel lack of enthusiasm. I can count on one hand how many people have actually tried it. So I gave up. But now looking back I realize this project is simply waiting for deprecation. I mean, what future it has? It's a huge pile of hacks for a problem that python 3 doesn't have. So what I mean: you cannot expect anything more than what you got so far, and that is: nothing.

What I suggest, if you really care about virtualenv, and know that you would have an audience for the fixes you're proposing: fork it, prove that you can maintain it, and people are using it, and then the maintainers will inevitably give you the project. I mean, they don't really care about it anyway, they just don't want to feel guilt/waste their own time on it.

Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro

[Paul Moore]

On 28 September 2017 at 23:41, 'Ionel Cristian Mărieș' via pypa-dev

<pypa...@googlegroups.com> wrote:
> Just my two cents here. I was a bit angry
> too
> with virtualenv some way back, up to a point that I've took dstufft's
> rewrite branch to a better state (supporting up to 3.4, integration tests
> and CI). But it was same deal, no one would really look at it - a cruel lack
> of enthusiasm. I can count on one hand how many people have actually tried
> it. So I gave up. But now looking back I realize this project is simply
> waiting for deprecation. I mean, what future it has? It's a huge pile of
> hacks for a problem that python 3 doesn't have. So what I mean: you cannot
> expect anything more than what you got so far, and that is: nothing.
>
> What I suggest, if you really care about virtualenv, and know that you would
> have an audience for the fixes you're proposing: fork it, prove that you can
> maintain it, and people are using it, and then the maintainers will
> inevitably give you the project. I mean, they don't really care about it
> anyway, they just don't want to feel guilt/waste their own time on it.

My suggestion - use the core venv module wherever possible. That
basically means everywhere except Python 2.

Submit PRs to projects that use virtualenv which allow them to use
venv where available, until we reach a point where virtualenv is only
needed by Python 2 users. And then the problem goes away in 2020...
;-)

Paul

[Sorin Sbarnea]

I see project forking as the least desirable alternative, like "war" to make an analogy. This should be used only when there is no way to find a common ground and we eliminated all other alternatives.

I did a special fork approach with Synergy project many years ago because it was unmaintained and there were patches floating around over the internet. I did an integration fork, did some CI work, with the only scope of reviving the project, I was successful and that become the only version, everyone joined forced but it was a lot of effort and it worked because that was a final product not a library.

You do know very well that it would be close to impossible to successfully fork virtualenv because nobody will use it, just because they would have to change their code to use the "alternative".

If <current-maintainer> does not have time or interest in spending more effort on the project they should gradually just handle more control to it others that do need it and that they are willing to give it few good more years of life. Lots of people are still using py27 in production and I am ready to bet that the EOL date may be updated.

I adopted quite a few Python orphan projects over the years and they all ended up on https://github.com/pycontribs because I believe that projects should not depend on a single person. I also assured that at least are at least two people with rights to make a new release on PyPI.

virtualenv is already under an organization so there is no need to move it anywhere, all it needs is some attention (looking at the open PRs, it clearly needs attention as people are raising valid contributions that are ignored for very long time).

I already said that I am willing to help with that and I could start by doing few reviews (no permissions to perform them now).

PS. I find but weird the fact that it was much easier to contribute the same fix to Python core than to virtualenv.

On 28 Sep 2017, 23:41 +0100, wrote

[Paul Moore]

On 29 September 2017 at 16:51, Sorin Sbarnea <ssba...@redhat.com> wrote:
> I already said that I am willing to help with that and I could start by
> doing few reviews (no permissions to perform them now).

I already suggested contributing improvements to the test suite, so we
get to a point where the risks of regressions aren't as high as they
are currently.

But feel free to review any of the existing open issues - you can
contribute comments, make suggestions, recommend closure of invalid or
out of date issues, lots of things, and you don't need formal reviewer
privileges to do so. You'll have to manage people's expectations when
they want an immediate fix but a committer isn't available to merge a
PR, or there's no planned date for a new release to offer them. I
guess you will understand where they are coming from, which should
help :-)

> PS. I find but weird the fact that it was much easier to contribute the same
> fix to Python core than to virtualenv.

Python core *explicitly* only supports a limited set of platforms (see
PEP 11 for an overview). Maybe virtualenv should do similar, but I'm
not aware that we do. Also, Python has many more core developers, and
a much more extensive test suite. And a buildbot fleet to run tests on
multiple platforms. Maybe some of those things help.

Paul

[Paul Moore]

On 28 September 2017 at 21:00, Paul Moore <p.f....@gmail.com> wrote:
> Let me just be 100% clear here. I will not personally commit this
> change. It is not in my area of expertise, and I'm not willing to be
> browbeaten into doing so just because people keep telling me it's
> fine.

I was very close to posting here to say that I didn't think battling
over principles was productive, and I was willing to put aside my
reservations and commit this change. I then realised that the change
is actually *wrong* - it alters the file in the virtualenv_embedded
subdirectory, but doesn't actually refresh the main script. I've added
a comment to the PR to that effect. All the endless debate here over
the failings of the virtualenv team caused me to nearly miss that
fact.

However, I am utterly drained and frustrated by the endless,
relentless pressure that has been put on me to defend the virtualenv
team over this honestly trivial issue. For better or worse, I find the
attitude of everyone involved to be totally unacceptable - I've
pointed out that I'm doing this in my free time, I've made it clear
that I'm only offering information, not a commitment to act, and yet
I've been subject of a flood of emails repeating over and over again
complaints about the virtualenv project that aren't new, that we know
about and are struggling to address, and that contribute nothing in
the way of constructive suggestions. I was sympathetic to the
frustration people seemed to have over the slow progress on the issue,
and I feel like I've been punished for that sympathy.

My personal life has suffered this week as a result of this debate -
which is something I honestly never thought I'd have to deal with as a
result of my open source work.

As a result, I will be taking an indefinite break from the virtualenv
issues list. I'll continue to work on areas of virtualenv that
interest me personally, and that are of benefit to me, but I will not
contribute to discussions on issues or PRs raised by others. I don't
know how long this will be for. Until I've calmed down, at a minimum.
I won't change my involvement in any of my other open source projects
- none of them are in my experience anywhere near as corrosive to work
on as virtualenv.

Paul

[Krzysztof Rosiński]

I'd like to follow up on this issue - are there any repo maintainers that could push this story to an end? The work is already done (code and reviews), and we're just waiting for an approval.