[Python-ideas] shutil.runret and shutil.runout
anatoly techtonik
subprocess is low level, cryptic, does too much, with poor usability,
i.e. "don't make me think" is not about it. I don't know about you,
but I can hardly write any subprocess call without spending at least
5-10 meditating over the documentation. So, I propose two high level
KISS functions for shell utils (shutil) module:
runret(command) - run command through shell, return ret code
runout(command) - run command through shell, return output
To avoid subprocess story (that makes Python too complicated) I
deliberately limit the scope to:
- executing from shell only
- return one thing at a time
I hope that this covers 80% of what _users_ need to execute commands
from Python. If somebody needs more - there is `subprocess`. But if
your own scripts are mostly outside these 80% - feel free to provide
your user story and arguments, why this should be done in shutil and
not in subprocess.
Open questions:
- security quoting for 'command'
--
anatoly t.
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas
Dirkjan Ochtman
> subprocess is low level, cryptic, does too much, with poor usability,
> i.e. "don't make me think" is not about it. I don't know about you,
> but I can hardly write any subprocess call without spending at least
> 5-10 meditating over the documentation. So, I propose two high level
> KISS functions for shell utils (shutil) module:
>
> runret(command) - run command through shell, return ret code
> runout(command) - run command through shell, return output
Have you seen subprocess.check_call() and subprocess.check_output()?
I don't think your proposed functions add much benefit over these two.
Cheers,
Dirkjan
Nick Coghlan
> Hello,
>
> subprocess is low level, cryptic, does too much, with poor usability,
> i.e. "don't make me think" is not about it. I don't know about you,
> but I can hardly write any subprocess call without spending at least
> 5-10 meditating over the documentation.
Hi Anatoly,
I believe you'll find the simple convenience methods you are
requesting already exist, in the form of subprocess.call(),
subprocess.check_call() and subprocess.check_output(). The
documentation has also been updated to emphasise these convenience
functions over the Popen swiss army knife.
If you do "pip install shell-command" you can also access the
shell_call(), shell_check_call() and shell_output() functions I
currently plan to include in subprocess for 3.3. (I'm not sure which
versions of Python that module currently supports though - 2.7 and
3.2, IIRC).
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Tarek Ziadé
Hello,
subprocess is low level, cryptic, does too much, with poor usability,
i.e. "don't make me think" is not about it. I don't know about you,
but I can hardly write any subprocess call without spending at least
5-10 meditating over the documentation. So, I propose two high level
KISS functions for shell utils (shutil) module:
runret(command) - run command through shell, return ret code
mmm you are describing subprocess.call() here... I don't see how this new command makes thing better, besides shell=True.
runout(command) - run command through shell, return output
what is 'output' ? the stderr ? the stdout ? a merge of both ?
what about subprocess.check_output() ?
To avoid subprocess story (that makes Python too complicated)
I seems to me that the only complication here is shell=True, which seems ok to me to have it at False for security reasons.
Cheers
--
Tarek Ziadé | http://ziade.org
anatoly techtonik
> On Fri, Feb 24, 2012 at 8:52 PM, anatoly techtonik <tech...@gmail.com> wrote:
>> Hello,
>>
>> subprocess is low level, cryptic, does too much, with poor usability,
>> i.e. "don't make me think" is not about it. I don't know about you,
>> but I can hardly write any subprocess call without spending at least
>> 5-10 meditating over the documentation.
>
> requesting already exist, in the form of subprocess.call(),
> subprocess.check_call() and subprocess.check_output(). The
> documentation has also been updated to emphasise these convenience
> functions over the Popen swiss army knife.
I don't find the names of these functions more intuitive than Popen().
I also think they far from being simple, because (in the order of appearance):
1. they require try/catch
2. docs still refer Popen, which IS complicated
3. contain shell FUD
4. completely confuse users with stdout=PIPE or stderr=PIPE stuff
http://docs.python.org/library/subprocess.html#subprocess.check_call
My verdict - these fail to be simple, and require the same low-level
system knowledge as Popen() for confident use.
> If you do "pip install shell-command" you can also access the
> shell_call(), shell_check_call() and shell_output() functions I
> currently plan to include in subprocess for 3.3. (I'm not sure which
> versions of Python that module currently supports though - 2.7 and
> 3.2, IIRC).
Don't you find strange that shell utils module don't have any
functions for the main shell function - command execution?
In game development current state of subprocess bloat is called
"featurecrepping" and the "scope definition" is a method to cope with
this disease.
--
anatoly t.
anatoly techtonik
> On Fri, Feb 24, 2012 at 11:52 AM, anatoly techtonik <tech...@gmail.com>
> wrote:
>>
>> subprocess is low level, cryptic, does too much, with poor usability,
>> i.e. "don't make me think" is not about it. I don't know about you,
>> but I can hardly write any subprocess call without spending at least
>> 5-10 meditating over the documentation. So, I propose two high level
>> KISS functions for shell utils (shutil) module:
>>
>> runret(command) - run command through shell, return ret code
>
>
> mmm you are describing subprocess.call() here... I don't see how this new
> command makes thing better, besides shell=True.
shutil.runret() - by definition has shell=True
>>
>> runout(command) - run command through shell, return output
>
>
> what is 'output' ? the stderr ? the stdout ? a merge of both ?
That's a high-level _user_ function. When user runs command in shell
he sees both. So, this 'shell util' is an analogue. If you have you
own user scripts that require stdout or stderr separately, I am free
to discuss the cases. The main purpose of this function is to be
useful from Python console, so the interface should be very simple to
remember from the first try. Like runout(command,
ret='stdout|stderr|both'). No universal PIPEs.
> what about subprocess.check_output() ?
See my reply above.
>> To avoid subprocess story (that makes Python too complicated)
>
>
> I seems to me that the only complication here is shell=True, which seems ok
> to me to have it at False for security reasons.
It won't be 'shell util' function anymore. If you're using shell
execution functions, you already realize that will happen if your
input parameters are not validated properly. Isolating calls that
require shell execution in shutil module will also simplify security
analysis for 3rd party libraries.
Mike Meyer
Tarek Ziadé <ziade...@gmail.com> wrote:
> On Fri, Feb 24, 2012 at 11:52 AM, anatoly techtonik <tech...@gmail.com>wrote:
> > Hello,
> > subprocess is low level, cryptic, does too much, with poor usability,
> > i.e. "don't make me think" is not about it. I don't know about you,
> > but I can hardly write any subprocess call without spending at least
> > 5-10 meditating over the documentation. So, I propose two high level
> > KISS functions for shell utils (shutil) module:
> > runret(command) - run command through shell, return ret code
> mmm you are describing subprocess.call() here... I don't see how this new
> command makes thing better, besides shell=True.
The stated purpose of the new functions is to allow people to run
shell commands without thinking about them. That's a bad idea (isn't
most programming without thinking about it?). The first problem is
that it's a great way to add data injection vulnerabilities to your
application. It's also a good way to introduce bugs in your
application when asked to (for instance) process user-provided file
names.
-1
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Nick Coghlan
> Don't you find strange that shell utils module don't have any
> functions for the main shell function - command execution?
> In game development current state of subprocess bloat is called
> "featurecrepping" and the "scope definition" is a method to cope with
> this disease.
They may still end up in shutil. I haven't really decided which
location I like better.
However, if you (or anyone else) wants to see Python's innate
capabilities improve in this area (and they really are subpar compared
to Perl 5, for example), your best bet is to download my Shell Command
module and give me feedback on any problems you find with it via the
BitBucket issue tracker.
http://shell-command.readthedocs.org
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Masklinn
>
> 1. they require try/catch
No.
> 2. docs still refer Popen, which IS complicated
True.
> 3. contain shell FUD
No, they contain warnings, against shell injection security
risks. Warnings are not FUD, it's not trying to sell some sort
of alternative it's just warning that `shell=True` is dangerous
on untrusted input.
> 4. completely confuse users with stdout=PIPE or stderr=PIPE stuff
>
> http://docs.python.org/library/subprocess.html#subprocess.check_call
On the one hand, these notes are a bit clumsy. On the other hand,
piping is a pretty fundamental concept of shell execution, I see
nothing wrong about saying that these functions *can't* be involved
in pipes. In fact stating it upfront looks sensible.
>> If you do "pip install shell-command" you can also access the
>> shell_call(), shell_check_call() and shell_output() functions I
>> currently plan to include in subprocess for 3.3. (I'm not sure which
>> versions of Python that module currently supports though - 2.7 and
>> 3.2, IIRC).
>
> Don't you find strange that shell utils module don't have any
> functions for the main shell function - command execution?
What "shell utils" module? Subprocess has exactly that in `call`
and its variants. And "shutil" does not bill itself as a
"shell utils" module right now, its description is
"High-level file operations".
> shutil.runret() - by definition has shell=True
Great, so your recommendation is to be completely insecure by default?
> That's a high-level _user_ function. When user runs command in shell
> he sees both. So, this 'shell util' is an analogue.
That makes no sense, when users invoke shell commands programmatically
(which is what these APIs are about), they expect two semantically
different reporting streams to be split, not to be merged,
indistinguishable and unusable as a default. Dropping stderr on the
ground may be an acceptable default but munging stdout and stderr is not.
> The main purpose of this function is to be useful from Python console
Then I'm not sure it belongs in subprocess or shutil, and users with that
need should probably be driven towards iPython which provides extensive
means of calling into the system shell in interactive sessions[0].
bpython may also provide such facilities.
It *may* belong in the interactive interpreter's own namespace.
> The main purpose of this function is to be
> useful from Python console, so the interface should be very simple to
> remember from the first try. Like runout(command,
> ret='stdout|stderr|both').
As opposed to `check_output(command)`?
> It won't be 'shell util' function anymore. If you're using shell
> execution functions, you already realize that will happen if your
> input parameters are not validated properly.
This assertion demonstrably does not match reality, shell injections
(the very reason for this warning) would not exist if this were the
case.
[0] http://ipython.org/ipython-doc/rel-0.12/interactive/reference.html#system-shell-access
anatoly techtonik
> On Fri, 24 Feb 2012 12:10:08 +0100
> Tarek Ziadé <ziade...@gmail.com> wrote:
>> On Fri, Feb 24, 2012 at 11:52 AM, anatoly techtonik <tech...@gmail.com>wrote:
>> > Hello,
>> > subprocess is low level, cryptic, does too much, with poor usability,
>> > i.e. "don't make me think" is not about it. I don't know about you,
>> > but I can hardly write any subprocess call without spending at least
>> > 5-10 meditating over the documentation. So, I propose two high level
>> > KISS functions for shell utils (shutil) module:
>> > runret(command) - run command through shell, return ret code
>> mmm you are describing subprocess.call() here... I don't see how this new
>> command makes thing better, besides shell=True.
>
> The stated purpose of the new functions is to allow people to run
> shell commands without thinking about them. That's a bad idea (isn't
> most programming without thinking about it?). The first problem is
> that it's a great way to add data injection vulnerabilities to your
> application. It's also a good way to introduce bugs in your
> application when asked to (for instance) process user-provided file
> names.
>
> -1
The proposal doesn't took into account security implications, so your
-1 is premature.
I agree with your point that users should think about *security* when
they run commands. But they should not think about how tons of
different ways to execute their command and different combinations on
different operating systems, *and* security implications about this.
This is *the main point* that make subprocess module a failure, and a
basis (main reason) of this proposal.
If users choose to trade security over simplicity, they should know
what the risks are, and what to do if they want to avoid them. So I
completely support the idea of shutil docs containing a user friendly
explanation of how to exploit and how to protect (i.e. use subprocess)
from the flaws provided by this method of execution - if they need to
protect. Python is not a Java - it should give users a choice of
simple API when they don't need security, and let this choice of
shooting themselves in the foot be explicit.. and simple.
--
anatoly t.
Mike Meyer
Failing to take into account security implications means the -1 isn't
premature, it's mandatory!
> I agree with your point that users should think about *security* when
> they run commands. But they should not think about how tons of
> different ways to execute their command and different combinations on
> different operating systems, *and* security implications about this.
This sounds like a documentation issue, not a code issue.
In fact, checking the shutil docs (via pydoc) turns up:
shutil - Utility functions for copying and archiving files and directory trees.
Clearly, running commands is *not* part of this functionality, so
these new functions don't belong there.
> If users choose to trade security over simplicity, they should know
> what the risks are, and what to do if they want to avoid them. So I
> completely support the idea of shutil docs containing a user friendly
> explanation of how to exploit and how to protect (i.e. use subprocess)
> from the flaws provided by this method of execution - if they need to
> protect. Python is not a Java - it should give users a choice of
> simple API when they don't need security, and let this choice of
> shooting themselves in the foot be explicit.. and simple.
So now look at use cases. The "simple" method you propose is *only*
safe to use on a very small set of constant strings. If any of the
values in the string are supplied by the user in any way, you can't
use it. If any of the arguments contain shell meta-characters, you
either have to quote them or not use your method. Since you're
explicitly proposing passing the command to the shell, the programmer
doesn't even know which characters are meta-characters when they write
the code. This means these functions - as proposed - are more
attractive nuisances than useful utilities.
Oddly enough, I read the Julia docs on external commands between my
first answer and your reply, and their solution is both as simple as
what you want, and safe. This inspired a counter proposal:
How about adding your new function to subprocess, except instead of
passing them to the shell, they use shlex to parse them, then call
Popen with the appropriate arguments? shlex might need some work for
this.
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Nick Coghlan
> This is *the main point* that make subprocess module a failure, and a
> basis (main reason) of this proposal.
Anatoly, this is the exact kind of blanket statement that pisses
people off and makes them stop listening to you. The subprocess module
is not a failure by any means. Safely invoking subprocesses is a *hard
problem*. Other languages make the choice "guarding against shell
injections is a problem for the user to deal with" and allow them by
default in their subprocess invocation interfaces. They also make the
choice that the risk of data leakage through user provided format
strings is something for the developer to worry about and allow
implicit string interpolation.
Python doesn't allow either of those as a *deliberate design choice*.
The current behaviour isn't an accident, or due to neglect, or because
we're stupid. Instead, we default to the more secure, less convenient
options, and allow people to explicitly request the insecure behaviour
if they either:
1. don't care; or
2. do care, but also know it isn't actually a problem for their use case.
This is a *good thing* if you're an application programmer - secure
defaults lets you conduct security audits by looking specifically for
cases where the safety checks have been bypassed. However, it mostly
sucks if you're wanting to use Python for system administration (or
similar) tasks where the shell is an essential tool rather than a
security risk and there's no untrusted data that comes anywhere near
your script.
I'll repeat my suggestion: if you want to do something *constructive*
about this, get Shell Command from PyPI and start using it, as it aims
to address both the shell invocation and the string interpolation
aspects of this issue. If you find problems, report them on the
module's issue tracker (although I'll point out in advance that STDERR
being separate from STDOUT by default is *deliberate*. If people want
them merged they can include a redirection in their shell command.
Otherwise STDERR needs to remain mapped to the same stream as it is in
the parent process so that tools like getpass() will still work in an
invoked shell command).
Regards,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Nick Coghlan
> How about adding your new function to subprocess, except instead of
> passing them to the shell, they use shlex to parse them, then call
> Popen with the appropriate arguments? shlex might need some work for
> this.
http://shell-command.readthedocs.org
>>> from shell_command import shell_call
>>> shell_call("ls *.py")
setup.py shell_command.py test_shell_command.py
0
>>> shell_call("ls {}", "*.py")
ls: cannot access *.py: No such file or directory
2
>>> shell_call("ls {!u}", "*.py")
setup.py shell_command.py test_shell_command.py
0
Unless someone uncovers a major design flaw in the next few months, at
least ShellCommand, shell_call, shell_check_call and shell_output are
likely to make an appearance in subprocess for 3.3.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Nick Coghlan
> Oddly enough, I read the Julia docs on external commands between my
> first answer and your reply, and their solution is both as simple as
> what you want, and safe.
That *is* rather nice, although they never get around to actually
explaining *how* to capture the output from the child processes
(http://julialang.org/manual/running-external-programs/, for anyone
else that's interested).
It should definitely be possible to implement something along those
lines as a third party library on top of subprocess (although it would
be a lot more complicated than Shell Command is).
Kenneth Reitz (author of "requests") has also spent some time
tinkering with subprocess invocation API design concepts:
https://github.com/kennethreitz/envoy
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Mike Meyer
Nick Coghlan <ncog...@gmail.com> wrote:
> On Fri, Feb 24, 2012 at 10:13 PM, Mike Meyer <m...@mired.org> wrote:
> > How about adding your new function to subprocess, except instead of
> > passing them to the shell, they use shlex to parse them, then call
> > Popen with the appropriate arguments? shlex might need some work for
> > this.
>
> http://shell-command.readthedocs.org
That says:
This module aims to take over where subprocess leaves off,
providing convenient, low-level access to the system shell, that
automatically handles filenames and paths containing whitespace,
as well as protecting naive code from shell injection
vulnerabilities.
That's a backwards approach to security. Rather than allowing anything
and turning off what you know isn't safe, you should disallow
everything and turn on what you know is safe. So rather than trying to
make the strings you pass to the shell safe, you should parse them
yourself and avoid calling the shell at all.
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
anatoly techtonik
> On 2012-02-24, at 12:12 , anatoly techtonik wrote:
>>
>> 1. they require try/catch
>
> No.
Quote from the docs:
"Run command with arguments. Wait for command to complete. If the
return code was zero then return, otherwise raise CalledProcessError."
http://docs.python.org/library/subprocess.html#subprocess.check_call
>> 2. docs still refer Popen, which IS complicated
>
> True.
>
>> 3. contain shell FUD
>
> No, they contain warnings, against shell injection security
> risks. Warnings are not FUD, it's not trying to sell some sort
> of alternative it's just warning that `shell=True` is dangerous
> on untrusted input.
Warnings would be o.k. if they provided at least some guidelines where
shell=True can be useful and where do you need to use Popen (or
escaping). Without positive examples, and a little research to show
attack vectors (so that users can analyse if they are applicable in
their specific case) it is FUD IMO.
>> 4. completely confuse users with stdout=PIPE or stderr=PIPE stuff
>>
>> http://docs.python.org/library/subprocess.html#subprocess.check_call
>
> On the one hand, these notes are a bit clumsy. On the other hand,
> piping is a pretty fundamental concept of shell execution, I see
> nothing wrong about saying that these functions *can't* be involved
> in pipes. In fact stating it upfront looks sensible.
The point is that it makes things more complicated than necessary. As
a system programmer I feel confident about all this stuff, but users
struggle to get it and they blame Python for complexity, and I have to
agree. We can change that with high level API. The API that will
automatically provide a rolling buffer for output if required to avoid
locks (for the missing info as a drawback), and remove headache about
"what to do about that?".
>>> If you do "pip install shell-command" you can also access the
>>> shell_call(), shell_check_call() and shell_output() functions I
>>> currently plan to include in subprocess for 3.3. (I'm not sure which
>>> versions of Python that module currently supports though - 2.7 and
>>> 3.2, IIRC).
>>
>> Don't you find strange that shell utils module don't have any
>> functions for the main shell function - command execution?
>
> What "shell utils" module? Subprocess has exactly that in `call`
> and its variants. And "shutil" does not bill itself as a
> "shell utils" module right now, its description is
> "High-level file operations".
>
>> shutil.runret() - by definition has shell=True
>
> Great, so your recommendation is to be completely insecure by default?
Not "by default" - only if it is impossible to make shutil.run*()
functions more secure. They only make sense with shell=True, so my
recommendation is to analyse security implications and *let* users
make their grounded choice. Not frighten them, but making them think
about security.
The difference. User friendly docs for shutil.run*() docs should be
structured as following:
1. you are free to use these functions
2. but know that they are insecure
3. in these cases:
3.1
3.2
3.3
4. if you think these cases won't apply to your project, then feel
free to use, otherwise look at subprocess
Of course, if some cases 3.1-3.3 have workarounds, they should be mentioned.
>> That's a high-level _user_ function. When user runs command in shell
>> he sees both. So, this 'shell util' is an analogue.
>
> That makes no sense, when users invoke shell commands programmatically
> (which is what these APIs are about), they expect two semantically
> different reporting streams to be split, not to be merged,
> indistinguishable and unusable as a default. Dropping stderr on the
> ground may be an acceptable default but munging stdout and stderr is not.
Conflict point:
Do users care about stdout/stderr when they invoke shell commands?
Do users care about stdout/stderr when they use Python syntax for
invoking shell commands?
These functions is no a syntax sugar for developers (as the
aforementioned "alternatives" from subprocess modules are). They are
helper for users. If you're a developer, who cares about pipes and
needs programmatic acces - there is already a low level subprocess
API with developer's defaults. If we speak about users:
The standard shell console behaviour is to output both streams to the
screen. That means that if I want to process this output, I don't know
if it comes from stderr or stdout. So, if I want to process the output
- I use Python to do this. If I know what I need the output from
stderr only, I specify this explicitly. That's my default user story.
>> The main purpose of this function is to be useful from Python console
>
> Then I'm not sure it belongs in subprocess or shutil, and users with that
> need should probably be driven towards iPython which provides extensive
> means of calling into the system shell in interactive sessions[0].
> bpython may also provide such facilities.
I think it is a good idea to unify interface across interactive mode
in Python. Hopefully shutil.copy and friends are already good enough
so that they don't have reasons to reimplement them (and users to
learn new commands).
>> The main purpose of this function is to be
>> useful from Python console, so the interface should be very simple to
>> remember from the first try. Like runout(command,
>> ret='stdout|stderr|both').
>
> As opposed to `check_output(command)`?
As opposed to check_output(command, *, stdin=None, stdout=None,
stderr=None, shell=True)
>> It won't be 'shell util' function anymore. If you're using shell
>> execution functions, you already realize that will happen if your
>> input parameters are not validated properly.
>
> This assertion demonstrably does not match reality, shell injections
> (the very reason for this warning) would not exist if this were the
> case.
It is not assertion, it is a wannabe for shutil documentation to
clarify shell injections problems to the level that allow users to
make a reasonable choice, so if the user is "using shell execution
functions he already realizes that will happen if his input parameters
are not validated properly".
--
anatoly t.
Mike Meyer
anatoly techtonik <tech...@gmail.com> wrote:
> On Fri, Feb 24, 2012 at 2:50 PM, Masklinn <mask...@masklinn.net> wrote:
> > On 2012-02-24, at 12:12 , anatoly techtonik wrote:
> >> 1. they require try/catch
> > No.
> Quote from the docs:
> "Run command with arguments. Wait for command to complete. If the
> return code was zero then return, otherwise raise CalledProcessError."
> http://docs.python.org/library/subprocess.html#subprocess.check_call
Quote from the docs:
subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False)
Run the command described by args. Wait for command to complete,
then return the returncode attribute.
No documented exceptions raised, so no need for try/catch.
> >> 2. docs still refer Popen, which IS complicated
> > True.
> >> 3. contain shell FUD
> > No, they contain warnings, against shell injection security
> > risks. Warnings are not FUD, it's not trying to sell some sort
> > of alternative it's just warning that `shell=True` is dangerous
> > on untrusted input.
> Warnings would be o.k. if they provided at least some guidelines where
> shell=True can be useful and where do you need to use Popen (or
> escaping). Without positive examples, and a little research to show
> attack vectors (so that users can analyse if they are applicable in
> their specific case) it is FUD IMO.
You mean something like (quoting from the docs):
Warning
Executing shell commands that incorporate unsanitized input from
an untrusted source makes a program vulnerable to shell injection,
a serious security flaw which can result in arbitrary command
execution. For this reason, the use of shell=True is strongly
discouraged in cases where the command string is constructed from
external input:
<example removed>
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
Tshepang Lekhonkhobe
> However, if you (or anyone else) wants to see Python's innate
> capabilities improve in this area (and they really are subpar compared
> to Perl 5, for example), your best bet is to download my Shell Command
> module and give me feedback on any problems you find with it via the
> BitBucket issue tracker.
Just curious: If put in the stdlib, will the above-mentioned module
bring CPython shell handling to Perl 5 level?
Nick Coghlan
> That's a backwards approach to security. Rather than allowing anything
> and turning off what you know isn't safe, you should disallow
> everything and turn on what you know is safe. So rather than trying to
> make the strings you pass to the shell safe, you should parse them
> yourself and avoid calling the shell at all.
Yes, that's why these are *separate functions* (each with "shell" in
the name to make the shell's involvement rather hard to miss). Any
application (rather than system administration script) that calls them
with user provided data should immediately fail a security audit.
The new APIs are intended specifically for system administrators that
want the *system shell*, not a language level "cross platform"
reinvention of it (and when it comes to shells, "cross platform"
generally means, "POSIX even if you're on Windows, because we're not
interesting in trying to reproduce Microsoft's idiosyncratic way of
doing things"). The automatic quoting feature is mainly there to
handle spaces in filenames - providing poorly structured programs with
some minimal defence against shell injections is really just a bonus
(although I admit I wasn't thinking about it that way when I wrote the
current docs).
As things stand, Python is a lousy language for system administration
tasks - the standard APIs are either *very* low level (os.system()) or
they're written almost entirely from the point of view of an
application programmer (subprocess). Even when I *am* the
administrator writing automation scripts for my own use, the
subprocess library still keeps getting in the way, telling me it isn't
safe to access my own shell.
Normally, Python is pretty good about striking a sensible balance
between "safe defaults" and "consenting adults", but it currently
fails badly on this particular point.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Masklinn
> On Fri, Feb 24, 2012 at 2:50 PM, Masklinn <mask...@masklinn.net> wrote:
>> On 2012-02-24, at 12:12 , anatoly techtonik wrote:
>>>
>>> 1. they require try/catch
>>
>> No.
>
> Quote from the docs:
> "Run command with arguments. Wait for command to complete. If the
> return code was zero then return, otherwise raise CalledProcessError."
> http://docs.python.org/library/subprocess.html#subprocess.check_call
Yes. If you want to run commands you just do. try/except are only needed
if you call commands which may fail and want to handle them without
quitting the whole interpreter. And for your stated use case of
interactively calling those functions, there is no need whatsoever for
try/catch.
And `subprocess.call` returns the status code, no exception ever thrown.
>>> 3. contain shell FUD
>>
>> No, they contain warnings, against shell injection security
>> risks. Warnings are not FUD, it's not trying to sell some sort
>> of alternative it's just warning that `shell=True` is dangerous
>> on untrusted input.
>
> Warnings would be o.k. if they provided at least some guidelines where
> shell=True can be useful and where do you need to use Popen (or
> escaping). Without positive examples, and a little research to show
> attack vectors (so that users can analyse if they are applicable in
> their specific case) it is FUD IMO.
http://docs.python.org/library/subprocess.html#frequently-used-arguments
>>> 4. completely confuse users with stdout=PIPE or stderr=PIPE stuff
>>>
>>> http://docs.python.org/library/subprocess.html#subprocess.check_call
>>
>> On the one hand, these notes are a bit clumsy. On the other hand,
>> piping is a pretty fundamental concept of shell execution, I see
>> nothing wrong about saying that these functions *can't* be involved
>> in pipes. In fact stating it upfront looks sensible.
>
> The point is that it makes things more complicated than necessary.
How?
> As
> a system programmer I feel confident about all this stuff
You feel confident about something which does not work, without warning?
>>>> If you do "pip install shell-command" you can also access the
>>>> shell_call(), shell_check_call() and shell_output() functions I
>>>> currently plan to include in subprocess for 3.3. (I'm not sure which
>>>> versions of Python that module currently supports though - 2.7 and
>>>> 3.2, IIRC).
>>>
>>> Don't you find strange that shell utils module don't have any
>>> functions for the main shell function - command execution?
>>
>> What "shell utils" module? Subprocess has exactly that in `call`
>> and its variants. And "shutil" does not bill itself as a
>> "shell utils" module right now, its description is
>> "High-level file operations".
>>
>>> shutil.runret() - by definition has shell=True
>>
>> Great, so your recommendation is to be completely insecure by default?
>
> Not "by default"
Oh? Because this:
> - only if it is impossible to make shutil.run*()
> functions more secure. They only make sense with shell=True, so my
> recommendation is to analyse security implications and *let* users
> make their grounded choice. Not frighten them, but making them think
> about security.
>
> The difference. User friendly docs for shutil.run*() docs should be
> structured as following:
> 1. you are free to use these functions
> 2. but know that they are insecure
> 3. in these cases:
> 3.1
> 3.2
> 3.3
> 4. if you think these cases won't apply to your project, then feel
> free to use, otherwise look at subprocess
>
> Of course, if some cases 3.1-3.3 have workarounds, they should be mentioned.
states precisely that the function would be insecure by default, and would
have caveat warnings in the docs. Which is the correct approach to
security… never as far as I know.
>>> The main purpose of this function is to be useful from Python console
>>
>> Then I'm not sure it belongs in subprocess or shutil, and users with that
>> need should probably be driven towards iPython which provides extensive
>> means of calling into the system shell in interactive sessions[0].
>> bpython may also provide such facilities.
>
> I think it is a good idea to unify interface across interactive mode
> in Python.
Considering IPython uses syntactic extentions (a "!" prefix) and does not
require any importing effort currently, I doubt that's going to happen.
>>> It won't be 'shell util' function anymore. If you're using shell
>>> execution functions, you already realize that will happen if your
>>> input parameters are not validated properly.
>>
>> This assertion demonstrably does not match reality, shell injections
>> (the very reason for this warning) would not exist if this were the
>> case.
>
> It is not assertion,
You may want to look up the definition of that word, I did not remove any
context, you asserted people using shell-exec functions are aware of the
risks. Which is, as, factually wrong.
> it is a wannabe for shutil documentation to
> clarify shell injections problems to the level that allow users to
> make a reasonable choice, so if the user is "using shell execution
> functions he already realizes that will happen if his input parameters
> are not validated properly".
Not sufficient when the default behavior is unsafe (and broken), as
numerous users *will* discover the function through third parties and
may never come close to the caveats they *should* know for the default
usage of the function.
Nick Coghlan
<tshe...@gmail.com> wrote:
> Just curious: If put in the stdlib, will the above-mentioned module
> bring CPython shell handling to Perl 5 level?
Closer, but it's hard to match backticks and implicit interpolation
for convenience (neither of which is going to happen in Python).
However, the trade-off is that you get things like the ability to
create pre-defined commands and easier invocation of shlex.quote when
appropriate, along with exceptions for some errors that would
otherwise pass silently.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Steven D'Aprano
> On Fri, Feb 24, 2012 at 11:11 PM, Tshepang Lekhonkhobe
> <tshe...@gmail.com> wrote:
>> Just curious: If put in the stdlib, will the above-mentioned module
>> bring CPython shell handling to Perl 5 level?
>
> Closer, but it's hard to match backticks and implicit interpolation
> for convenience (neither of which is going to happen in Python).
Anyone wanting to use Python as a system shell should look at IPython rather
than the standard Python interactive interpreter.
http://ipython.org/ipython-doc/dev/interactive/shell.html
--
Steven
Paul Moore
> Kenneth Reitz (author of "requests") has also spent some time
> tinkering with subprocess invocation API design concepts:
> https://github.com/kennethreitz/envoy
Vinay Sanjip extended this with "sarge" (available on PyPI, IIRC). One
key advantage of sarge for me is that it handles piping and
redirection in a cross-platfom manner, rather than just deferring to
the shell. (I think envoy does this too, but it's not very reliable on
WIndows from what I recall of my brief experiments).
Paul.
Nick Coghlan
> Anyone wanting to use Python as a system shell should look at IPython rather
> than the standard Python interactive interpreter.
>
> http://ipython.org/ipython-doc/dev/interactive/shell.html
Sure, but unless we add ! statements to Python itself, that doesn't
help with shell *scripting*.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Nick Coghlan
> On 24 February 2012 12:41, Nick Coghlan <ncog...@gmail.com> wrote:
>> Kenneth Reitz (author of "requests") has also spent some time
>> tinkering with subprocess invocation API design concepts:
>> https://github.com/kennethreitz/envoy
>
> Vinay Sanjip extended this with "sarge" (available on PyPI, IIRC). One
> key advantage of sarge for me is that it handles piping and
> redirection in a cross-platfom manner, rather than just deferring to
> the shell. (I think envoy does this too, but it's not very reliable on
> WIndows from what I recall of my brief experiments).
Ah, I knew I'd seen a more polished version of that somewhere - Vinay
posted about it a while back.
As I see it, the two complement each other fairly nicely:
shell_command is for direct access to the system shell. Appropriate
when you're writing platform specific administration scripts.
sarge is for cross platform scripting support. I'm actually not sure
what this is useful for (since the default Windows shell has different
spellings for so many basic commands and different syntax for
environment variable expansion, it seems easier to just use the
*actual* cross platform abstractions in the os module instead), but
apparently it's good for something (or Vinay wouldn't have taken the
time to write it).
Of course, since it's just a convenience wrapper around Popen,
ShellCommand does let you get pretty cute:
>>> import sys
>>> from functools import partial
>>> from shell_command import ShellCommand
>>> code = """
... def f():
... print("Python in a subprocess, easy as!")
... f()
... """
>>> PyCmd = partial(ShellCommand, executable=sys.executable)
>>> PyCmd(code).shell_call()
Python in a subprocess, easy as!
0
>>> x = PyCmd("print('Reporting for duty!')").shell_output()
>>> x
'Reporting for duty!'
(I didn't actually do a great deal in ShellCommand to enable that -
it's just a matter of passing all the keyword args through to
subprocess.Popen)
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Simon Sapin
> runret(command) - run command through shell, return ret code
> runout(command) - run command through shell, return output
Hi,
Brevity is nice, but I had no idea what either of these functions is
supposed to do before reading these descriptions. The names could be
more explicit.
(By the way, I agree with other issues raised in this thread. This was
only my first impression.)
Regards,
--
Simon Sapin
Andrew McNabb
>
> As things stand, Python is a lousy language for system administration
> tasks - the standard APIs are either *very* low level (os.system()) or
> they're written almost entirely from the point of view of an
> application programmer (subprocess). Even when I *am* the
> administrator writing automation scripts for my own use, the
> subprocess library still keeps getting in the way, telling me it isn't
> safe to access my own shell.
>
> Normally, Python is pretty good about striking a sensible balance
> between "safe defaults" and "consenting adults", but it currently
> fails badly on this particular point.
I disagree with this analysis. Python, with its fantastic subprocess
module, is the only language I really trust for system administration
tasks. Most languages provide "shell=True" as the default, making them
extremely frustrating for system administration. Every time I choose to
write a shell script instead of using Python, the lack of robustness
makes me eventually regret it (and then rewrite in Python with
subprocess).
Setting "shell=True" (or equivalent) seems really convenient in the
short term, but in the long term, scripts behave erratically and are
vulnerable to attacks. The subprocess module (with "shell=False") is a
wonderful balance between "safe defaults" and "consenting adults".
--
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868
Georg Brandl
> On Fri, Feb 24, 2012 at 1:59 PM, Nick Coghlan <ncog...@gmail.com> wrote:
>> On Fri, Feb 24, 2012 at 8:52 PM, anatoly techtonik <tech...@gmail.com> wrote:
>>> Hello,
>>>
>>> subprocess is low level, cryptic, does too much, with poor usability,
>>> i.e. "don't make me think" is not about it. I don't know about you,
>>> but I can hardly write any subprocess call without spending at least
>>> 5-10 meditating over the documentation.
>>
>> I believe you'll find the simple convenience methods you are
>> requesting already exist, in the form of subprocess.call(),
>> subprocess.check_call() and subprocess.check_output(). The
>> documentation has also been updated to emphasise these convenience
>> functions over the Popen swiss army knife.
>
> I don't find the names of these functions more intuitive than Popen().
> I also think they far from being simple, because (in the order of appearance):
>
> 1. they require try/catch
> 2. docs still refer Popen, which IS complicated
> 3. contain shell FUD
> 4. completely confuse users with stdout=PIPE or stderr=PIPE stuff
>
> http://docs.python.org/library/subprocess.html#subprocess.check_call
>
> My verdict - these fail to be simple, and require the same low-level
> system knowledge as Popen() for confident use.
And therefore they need to be completely replaced by something incompatible
and in another module? Sorry, Anatoly, this is not how Python development
happens. We usually work incrementally, improving on what we have rather
than throwing all out the door.
I think this is what rubs most people wrong about your posts: you invariably
propose radical changes that invalidate all previous work in the related
area. That's something apart from your style of expression, which was
discussed recently.
So here's some constructive advice: your point 1 was shown invalid. The
points 2-4 are "merely" documentation related: how about you think about
how to improve these docs to be less confusing?
Georg
Stephen J. Turnbull
> As things stand, Python is a lousy language for system administration
> tasks
Yeah, the worst possible sysadmin language except for all the others.
AFAICT it more than holds its own with distro maintainers, no?
Nick Coghlan
<ste...@xemacs.org> wrote:
> Nick Coghlan writes:
>
> > As things stand, Python is a lousy language for system administration
> > tasks
>
> Yeah, the worst possible sysadmin language except for all the others.
> AFAICT it more than holds its own with distro maintainers, no?
For applications where correctness in all circumstances is the
dominant criterion? Sure.
For throwaway scripts, though, most of the Linux sysadmins I know just
use shell scripts or Perl. For the devops (and deployment automation
in general) crowd, there's no real Python-based competitor to Chef and
Puppet (both Ruby based) (my understanding is that the Python-based
Fabric doesn't play in *quite* the same space as the other two).
As things currently stand, Python deliberately makes it hard to say "I
want my individual commands to be shell commands, but I also want
Python's superior flow control constructs to decide which shell
commands to run". For an application, that's a good thing. For
personal automation, it's not.
Cheers,
Nick.
--
Nick Coghlan | ncog...@gmail.com | Brisbane, Australia
Eli Bendersky
On Sun, Feb 26, 2012 at 12:03 AM, Stephen J. TurnbullFor applications where correctness in all circumstances is the
<ste...@xemacs.org> wrote:
> Nick Coghlan writes:
>
> > As things stand, Python is a lousy language for system administration
> > tasks
>
> Yeah, the worst possible sysadmin language except for all the others.
> AFAICT it more than holds its own with distro maintainers, no?
dominant criterion? Sure.
For throwaway scripts, though, most of the Linux sysadmins I know just
use shell scripts or Perl. For the devops (and deployment automation
in general) crowd, there's no real Python-based competitor to Chef and
Puppet (both Ruby based) (my understanding is that the Python-based
Fabric doesn't play in *quite* the same space as the other two).
As things currently stand, Python deliberately makes it hard to say "I
want my individual commands to be shell commands, but I also want
Python's superior flow control constructs to decide which shell
commands to run". For an application, that's a good thing. For
personal automation, it's not.
The Chef/Puppet/Fabric example is a good one to support this point - Ruby, like Python is also more a dev language than a sysadmin language, and yet Chef & Puppet are written in Ruby and not Perl.
Eli
Nick Coghlan
> The Chef/Puppet/Fabric example is a good one to support this point - Ruby,
> like Python is also more a dev language than a sysadmin language, and yet
> Chef & Puppet are written in Ruby and not Perl.
For the key operation I'm talking about here, though, Ruby works the
same way Perl does: it supports shell command execution via backtick
quoted strings with implicit string interpolation.
Is it really that hard to admit that there are some tasks that other
languages are currently just plain better for than Python, and perhaps
we can learn something useful from that? (And no, I'm not suggesting
we adopt backtick command execution or implicit string interpolation.
A convenience API that combines shell invocation, explicit string
interpolation and whitespace and shell metacharacter quoting, though,
*that* I support).
Serhiy Storchaka
> On Fri, Feb 24, 2012 at 10:13 PM, Mike Meyer<m...@mired.org> wrote:
>> Oddly enough, I read the Julia docs on external commands between my
>> first answer and your reply, and their solution is both as simple as
>> what you want, and safe.
Yes, I want this in Python:
readall(cmd('cut -d: -f3 $file', file='/etc/passwd') | cmd('sort -n') | cmd('tail -n5'))
or
cmd('cut', '-d:', '-f3', '/etc/passwd').pipe('sort', '-n').pipe('tail', '-n5').readlines()
or something similar.
> That *is* rather nice, although they never get around to actually
> explaining *how* to capture the output from the child processes
> (http://julialang.org/manual/running-external-programs/, for anyone
> else that's interested).
https://github.com/JuliaLang/julia/blob/10aabddc3834223568a87721149d05765e7e9997/j/process.j
See readall and each_line.
Matt Joiner
I strongly suspect such 3rd party library exists.
Stephen J. Turnbull
> For throwaway scripts, though, most of the Linux sysadmins I know just
> use shell scripts
Sure, but it's really hard to beat *sh plus GNU readline for brevity
in using recent history to create a script. At some point, we "just
don't want to go there." As for the Perl arm of your disjunction, do
those sysadmins use Python for anything? There's a lot of history in
the Linux sysadmin community favoring Perl. (Although the l33t
Perlmonger I know is a Ruby hacker now....)
> For the devops (and deployment automation in general) crowd,
> there's no real Python-based competitor to Chef and Puppet (both
> Ruby based) (my understanding is that the Python-based Fabric
> doesn't play in *quite* the same space as the other two).
No, there isn't, but creating one could be rather hard, as Puppet and
Chef both make heavy use of Ruby features conducive to writing DSLs.
Note that although Fabric plays in a distinct space, its
implementation looks like Chef, far more so than Puppet (ie, you write
Fabric configs in Python, and Chef configs in a (domain-specific
extension of) Ruby, while Puppet is a restricted DSL).
One of the Puppet rationales for using Puppet rather than Chef is
telling here:
3. Choice of configuration languages
The language which Puppet uses to configure servers is designed
specifically for the task: it is a domain language optimised for
the task of describing and linking resources such as users and
files.
Chef uses an extension of the Ruby language. Ruby is a good
general-purpose programming language, but it is not designed for
configuration management - and learning Ruby is a lot harder than
learning Puppet’s language.
Some people think that Chef’s lack of a special-purpose language
is an advantage. “You get the power of Ruby for free,” they
argue. Unfortunately, there are many things about Ruby which
aren’t so intuitive, especially for beginners, and there is a
large and complex syntax that has to be mastered.
-- http://bitfieldconsulting.com/puppet-vs-chef
That applies equally well to "DSL"s that are extensions of (function
calls in) Python.
Making it easier to write DSLs in Python has come up many times, and
so far the answer has always been "if you want to write a DSL in
Python, write a DSL in Python; but you can't, and won't soon be able
to, run it directly in the Python interpreter." DSLs have been done;
there's configparser for one, argparse and ancestors, and things like
gitosis. But it's hard to see Python beating Ruby at that game.
> As things currently stand, Python deliberately makes it hard to say
> "I want my individual commands to be shell commands, but I also
> want Python's superior flow control constructs to decide which
> shell commands to run".
I don't think that's ever been my motivation for writing a script in
Python. Really, is Python's for loop so much better than bash's? For
me, it's data structures: something where my sed fu isn't enough, or
the content has to persist longer than into the next pipe.
And quoting. Shell quoting is such a pain, especially if there's an
ssh remote command in there somewhere.
This is not to say I'm opposed to making it easier to use Python as a
command shell in principle, but I have to wonder whether it can be
done as easily as all that, and without sacrificing some of the things
we've insisted on in past discussions. On the other hand, for things
where avoiding shell makes sense, Python is one of my tools of choice
(the other being Emacs Lisp, where I want integration with my editor
and don't much care about performance).
Serhiy Storchaka
> I strongly suspect such 3rd party library exists.
I also hope this. And such library will be better candidate for
including in stdlib.
Stephen J. Turnbull
> Yes, I want this in Python:
>
> readall(cmd('cut -d: -f3 $file', file='/etc/passwd') | cmd('sort -n') | cmd('tail -n5'))
>
> or
>
> cmd('cut', '-d:', '-f3', '/etc/passwd').pipe('sort', '-n').pipe('tail', '-n5').readlines()
>
> or something similar.
But you can already do
sorted([l.split(":")[2] for l in open('/etc/passwd')])[-5:]
(and I don't really care whether you were being ironic or not; either
way that one-liner is an answer<wink/>).
Actually, I wrote that off the top of my head and it almost worked.
The problem I ran into is that I'm on a Mac, and there was a bunch of
cruft comments (which don't contain any colons) in the beginning of
the file. So I got a list index out of range when accessing the split
line. In this case, cut | sort | tail would produce the expected
output. But cut | sort | head would just produce garbage (the leading
comments in sorted order). So the failure modes differ. It might be
useful for people used to shell failure modes.
Matt Joiner
I did recently see "pyp" touted as a Python-like sed/awk.
I guess this stuff always comes down to what you're used to. To me it is insane to be still using Perl yet I prefer perl regex over posix anyday :)
Mike Meyer
Nick Coghlan <ncog...@gmail.com> wrote:
> On Sun, Feb 26, 2012 at 6:53 PM, Eli Bendersky <eli...@gmail.com> wrote:
> > The Chef/Puppet/Fabric example is a good one to support this point - Ruby,
> > like Python is also more a dev language than a sysadmin language, and yet
> > Chef & Puppet are written in Ruby and not Perl.
> For the key operation I'm talking about here, though, Ruby works the
> same way Perl does: it supports shell command execution via backtick
> quoted strings with implicit string interpolation.
Does Ruby also have something like Perl's -t/-T options and
supporting functions?
> Is it really that hard to admit that there are some tasks that other
> languages are currently just plain better for than Python, and perhaps
> we can learn something useful from that?
The key word is "perhaps". There are some things other languages are
better at than Python, and Python is the better off for it. I think
that "supporting code injection attacks" is one such feature.
> (And no, I'm not suggesting
> we adopt backtick command execution or implicit string interpolation.
> A convenience API that combines shell invocation, explicit string
> interpolation and whitespace and shell metacharacter quoting, though,
> *that* I support).
I'm only willing to support it if it's at least as safe as
Perl. Meaning that either 1) It doesn't really invoke the shell, but
handles provides those features explicitly, or 2) it throws errors if
passed tainted strings.
On the other hand, my support (or lack of it) isn't worth very much.
<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org