[Python-ideas] shutil.run (Was: shutil.runret and shutil.runout)

[anatoly techtonik]

Hello again,

I've finally found some time to partially process the replies and
came up with a better solution than subprocess.* and
shutil.runret/runout

Disclaimer: I don't say that suprocess suxx - it is powerful and very
awesome under the hood. What I want to say that its final user
interface is awful - for such complex thing as this it should have
been passed through several iteration cycles before settling down.


Therefore, inspired by Fabric API, I've finally found the solution -
shutil.run() function:
https://bitbucket.org/techtonik/shutil-run/src

run(command, combine_stderr=True):

Run command through a system shell, return output string with
additional properties:

output.succeeded - result of the operation True/False
output.return_code - specific return code
output.stderr - stderr contents if combine_stderr=False

`combine_stderr` if set, makes stderr merged into output string,
otherwise it will be available as `output.stderr` attribute.

Example:

from shellrun import run

output = run('ls -la')
if output.succeeded:
print(output)
else:
print("Error %s" % output.return_code)


That's the most intuitive way I found so far. Objective advantages:

1. Better than
subprocess.call(cmd, shell=true)
subprocess.check_call(cmd, shell=true)
subprocess.check_output(cmd, shell=True)
because it is just
shutil.run(cmd)
i.e. short, simple and _easy to remember_

2. With shutil.run() you don't need to rewrite your check_call() or
check_output() with Popen() if you need to get return_code in addition
to stderr contents on error

3. shutil.run() is predictable and consistent - its arguments are not
dependent on each other, their combination doesn't change the function
behavior over and over requiring you iterate over the documentation
and warnings again and again

4. shutil.run() is the correct next level API over subprocess base
level. subprocess executes external process - that is its role, but
automatic ability to execute external process inside another external
process (shell) looks like a hack to me. Practical, but still a hack.

5. No required exception catching, which doesn't work for shell=True anyway

6. No need to learn subprocess.PIPE routing magic (not an argument for
hackers, I know)


Subjective advantages:
1. More beautiful
2. More simple
3. More readable
4. Practical
5. Obvious
6. It easy to explain


Hopefully, it can find its way in stdlib instead of
http://shell-command.readthedocs.org/
--
anatoly t.
_______________________________________________
Python-ideas mailing list
Python...@python.org
http://mail.python.org/mailman/listinfo/python-ideas

[Mike Meyer]

On Tue, 22 May 2012 18:39:16 +0300
anatoly techtonik <tech...@gmail.com> wrote:

> Therefore, inspired by Fabric API, I've finally found the solution -
> shutil.run() function:
> https://bitbucket.org/techtonik/shutil-run/src
>
> run(command, combine_stderr=True):
>
> Run command through a system shell, return output string with
> additional properties:
>
> output.succeeded - result of the operation True/False
> output.return_code - specific return code
> output.stderr - stderr contents if combine_stderr=False
>
> `combine_stderr` if set, makes stderr merged into output string,
> otherwise it will be available as `output.stderr` attribute.

[...]

> That's the most intuitive way I found so far. Objective advantages:
>
> 1. Better than
> subprocess.call(cmd, shell=true)
> subprocess.check_call(cmd, shell=true)
> subprocess.check_output(cmd, shell=True)
> because it is just
> shutil.run(cmd)
> i.e. short, simple and _easy to remember_

-2

Unless there's some way to turn off shell processing (better yet, have
no shell processing be the default, and require that it be turned on),
it can't be used securely with tainted strings, so it should *not* be
used with tainted strings, which means it's pretty much useless in any
environment where security matters. With everything being networked,
there may no longer be any such environments.

> 3. shutil.run() is predictable and consistent - its arguments are not
> dependent on each other, their combination doesn't change the function
> behavior over and over requiring you iterate over the documentation
> and warnings again and again

As proposed, it certainly provides a predictable and consistent
vulnerability to code injection attacks.

> 4. shutil.run() is the correct next level API over subprocess base
> level. subprocess executes external process - that is its role, but
> automatic ability to execute external process inside another external
> process (shell) looks like a hack to me. Practical, but still a hack.

It's only correct if you are in an environment where you don't care
about security. If you care about security, you can't use it. If we're
going to add yet another system() replacement, let's at least try and
make it secure.

<mike
--
Mike Meyer <m...@mired.org> http://www.mired.org/
Independent Software developer/SCM consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

[Nick Coghlan]

Right, security implications are one of the reasons why I've held back from proposing Shell Command. The lack of cross platform support is also a pain. This suggestion shares both of those problems.

Having dealt with long running child processes lately, I can also say that producing output line-by-line would be on my personal list of requirements.

So, yeah, interesting idea, but this is still an area that needs a lot of exploration on PyPI before we select an answer for the stdlib.

--
Sent from my phone, thus the relative brevity :)

[Devin Jeanpierre]

On Tue, May 22, 2012 at 5:41 PM, Nick Coghlan <ncog...@gmail.com> wrote:
> Having dealt with long running child processes lately, I can also say that
> producing output line-by-line would be on my personal list of requirements.

You can do that with subprocess, right? Just have to be sure to close
stdin/stderr and read p.stdout with readline() repeatedly...

I think you might be able to even have the other file descriptors be
inputting/outputting if you use threads, but I'm scared of
experimenting with these things -- experiments don't tell you that it
doesn't work on an OS you don't have.

-- Devin

[Nick Coghlan]

On Wed, May 23, 2012 at 4:49 PM, Devin Jeanpierre
<jeanpi...@gmail.com> wrote:
> On Tue, May 22, 2012 at 5:41 PM, Nick Coghlan <ncog...@gmail.com> wrote:
>> Having dealt with long running child processes lately, I can also say that
>> producing output line-by-line would be on my personal list of requirements.
>
> You can do that with subprocess, right? Just have to be sure to close
> stdin/stderr and read p.stdout with readline() repeatedly...

Yep, subprocess is a swiss army knife - you can do pretty much
anything with it. That's the complaint, though - *because* it's so
configurable, even the existing convenience APIs aren't always that
convenient for simple operations.

Thus the current spate of efforts to provide a "friendlier" API for
performing shell operations from Python. The dust may settle well
enough in the 3.4 time frame for us to declare a "winner" and add
something to the standard library, but that's far from certain.

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia

[anatoly techtonik]

On Wed, May 23, 2012 at 12:41 AM, Nick Coghlan <ncog...@gmail.com> wrote:
> Right, security implications are one of the reasons why I've held back from
> proposing Shell Command. The lack of cross platform support is also a pain.
> This suggestion shares both of those problems.

Why shutil.run() is not cross-platform?
Is it technically feasible to make shutil.run() (or subprocess.* for
that purpose) cross-platform?

> Sent from my phone, thus the relative brevity :)

That's actually lowers a bounce rate for discussion. =)

[anatoly techtonik]

About security.

What does this "shell processing" involve to understand what to turn off?
Why there is no way to turn off "shell processing"?
What's the primary reason that it is impossible to be turned off?

>> 3. shutil.run() is predictable and consistent - its arguments are not
>> dependent on each other, their combination doesn't change the function
>> behavior over and over requiring you iterate over the documentation
>> and warnings again and again
>
> As proposed, it certainly provides a predictable and consistent
> vulnerability to code injection attacks.

suprocess.* with shell=True provides the same entrypoint for injection
attacks, and security through obscurity doesn't help here. People
still use shell=True, because that's sometimes the only way to execute
external utilities properly. Even my synapses were silent when I
reviewed and used shell=True for Rietveld upload script and Spyder
IDE.

What will help is a better simple explanation in a prominent place,
with an example that people can really remember instead of frightening
them with warnings. People will ignore warning eventually, and after
endless experiments will subprocess.* params mess will just leave
shell=True because it works (I did so).

No sane web developer will use subprocess calls on server side at all.
Regardless of shell=True or not. For example, how can I be sure that
Graphviz is save from exploit through malicious input? No sane
developer will run shell script on a web side either. For those who
still want - there will be this simple explanation right on the
shutil.run() page - with link to proper vulnerability analysis instead
of uncertainty inducting warning.

shutil.run() is aimed for local operations.

>> 4. shutil.run() is the correct next level API over subprocess base
>> level. subprocess executes external process - that is its role, but
>> automatic ability to execute external process inside another external
>> process (shell) looks like a hack to me. Practical, but still a hack.
>
> It's only correct if you are in an environment where you don't care
> about security. If you care about security, you can't use it. If we're
> going to add yet another system() replacement, let's at least try and
> make it secure.

I am all ears how to make shutil.run() more secure. Right now I must
confess that I don't even realize.how serious is this problems, so if
anyone can came up with a real-world example with explanation of
security concern that could be copied "as-is" into documentation, it
will surely be appreciated not only by me.

[Steven D'Aprano]

anatoly techtonik wrote:

> I am all ears how to make shutil.run() more secure. Right now I must
> confess that I don't even realize.how serious is this problems, so if
> anyone can came up with a real-world example with explanation of
> security concern that could be copied "as-is" into documentation, it
> will surely be appreciated not only by me.

Start here:

http://cwe.mitre.org/top25/index.html

Code injection attacks include two of the top three security vulnerabilities,
over even buffer overflows.

One sub-category of code injection:

OS Command Injection
http://cwe.mitre.org/data/definitions/78.html



--
Steven

[geremy condra]

On Wed, May 23, 2012 at 7:00 PM, Steven D'Aprano <st...@pearwood.info> wrote:

anatoly techtonik wrote:

I am all ears how to make shutil.run() more secure. Right now I must
confess that I don't even realize.how serious is this problems, so if
anyone can came up with a real-world example with explanation of
security concern that could be copied "as-is" into documentation, it
will surely be appreciated not only by me.

Start here:

http://cwe.mitre.org/top25/index.html

Code injection attacks include two of the top three security vulnerabilities, over even buffer overflows.

One sub-category of code injection:

OS Command Injection
http://cwe.mitre.org/data/definitions/78.html

I talked about this in my pycon talk this year. It's easy to avoid and disastrous to get wrong. Please don't do it this way.

Geremy Condra

[anatoly techtonik]

On Thu, May 24, 2012 at 6:24 AM, geremy condra <deba...@gmail.com> wrote:
> On Wed, May 23, 2012 at 7:00 PM, Steven D'Aprano <st...@pearwood.info>
> wrote:
>>
>> anatoly techtonik wrote:
>>
>>> I am all ears how to make shutil.run() more secure. Right now I must
>>> confess that I don't even realize.how serious is this problems, so if
>>> anyone can came up with a real-world example with explanation of
>>> security concern that could be copied "as-is" into documentation, it
>>> will surely be appreciated not only by me.
>>
>>
>> Start here:
>>
>> http://cwe.mitre.org/top25/index.html
>>
>> Code injection attacks include two of the top three security
>> vulnerabilities, over even buffer overflows.
>>
>> One sub-category of code injection:
>>
>> OS Command Injection
>> http://cwe.mitre.org/data/definitions/78.html

Great links. Thanks. Do they still too generic to be placed in docs?

>
> I talked about this in my pycon talk this year. It's easy to avoid and
> disastrous to get wrong. Please don't do it this way.

Sorry, don't have too much time to watch it right now. Any specific
slides, ideas or exceprts?
--
anatoly t.

[geremy condra]

On Mon, Jun 4, 2012 at 2:47 AM, anatoly techtonik <tech...@gmail.com> wrote:

On Thu, May 24, 2012 at 6:24 AM, geremy condra <deba...@gmail.com> wrote:
> On Wed, May 23, 2012 at 7:00 PM, Steven D'Aprano <st...@pearwood.info>
> wrote:
>>
>> anatoly techtonik wrote:
>>
>>> I am all ears how to make shutil.run() more secure. Right now I must
>>> confess that I don't even realize.how serious is this problems, so if
>>> anyone can came up with a real-world example with explanation of
>>> security concern that could be copied "as-is" into documentation, it
>>> will surely be appreciated not only by me.
>>
>>
>> Start here:
>>
>> http://cwe.mitre.org/top25/index.html
>>
>> Code injection attacks include two of the top three security
>> vulnerabilities, over even buffer overflows.
>>
>> One sub-category of code injection:
>>
>> OS Command Injection
>> http://cwe.mitre.org/data/definitions/78.html

Great links. Thanks. Do they still too generic to be placed in docs?

>
> I talked about this in my pycon talk this year. It's easy to avoid and
> disastrous to get wrong. Please don't do it this way.

Sorry, don't have too much time to watch it right now. Any specific
slides, ideas or exceprts?

The main idea was just that by combining a bit of awareness of common security anti-patterns (like this one) with a good test regimen and some script kiddie tools you can protect yourself from a lot of common vulnerabilities without being a security guru. I demonstrated how that process works on something fairly similar to this, but if you're interested in more details I'm happy to blather on or dredge up my slides.

Geremy Condra

--
anatoly t.

[Steven D'Aprano]

On Mon, Jun 04, 2012 at 11:00:34PM -0700, geremy condra wrote:

> The main idea was just that by combining a bit of awareness of common
> security anti-patterns (like this one) with a good test regimen and some
> script kiddie tools you can protect yourself from a lot of common
> vulnerabilities without being a security guru. I demonstrated how that
> process works on something fairly similar to this, but if you're interested
> in more details I'm happy to blather on or dredge up my slides.

I am interested in more details. Would this make a good How (Not) To for
the documentation?


--
Steven

[geremy condra]

On Mon, Jun 4, 2012 at 11:14 PM, Steven D'Aprano <st...@pearwood.info> wrote:

On Mon, Jun 04, 2012 at 11:00:34PM -0700, geremy condra wrote:

> The main idea was just that by combining a bit of awareness of common
> security anti-patterns (like this one) with a good test regimen and some
> script kiddie tools you can protect yourself from a lot of common
> vulnerabilities without being a security guru. I demonstrated how that
> process works on something fairly similar to this, but if you're interested
> in more details I'm happy to blather on or dredge up my slides.

I am interested in more details. Would this make a good How (Not) To for
the documentation?

Combined with some other material I have on hand it might. Only problem would be that I don't really know my way around Sphinx- if there are any doc wizards on hand to help with formatting we could probably make a pretty quick job of it.

Geremy Condra

[Nick Coghlan]

On Tue, Jun 5, 2012 at 4:45 PM, geremy condra <deba...@gmail.com> wrote:
> Combined with some other material I have on hand it might. Only problem
> would be that I don't really know my way around Sphinx- if there are any doc
> wizards on hand to help with formatting we could probably make a pretty
> quick job of it.

Yep, if you can provide a plain text version, we can take it from there.

I suggest attaching it to http://bugs.python.org/issue13515 (which is
about taking a more consistent and holistic approach to documenting
security considerations in the library reference without having
modules like subprocess stuck as a wall of red security warning
notices)

Cheers,
Nick.

--
Nick Coghlan   |   ncog...@gmail.com   |   Brisbane, Australia