Simulate `bash` behaviour using Python and named pipes.

[Luca Cerone]

Hi everybody,
I am trying to understand how to use named pipes in python to launch external processes (in a Linux environment).

As an example I am trying to "imitate" the behaviour of the following sets of commands is bash:

> mkfifo named_pipe
> ls -lah > named_pipe &
> cat < named_pipe

In Python I have tried the following commands:

import os
import subprocess as sp

os.mkfifo("named_pipe",0777) #equivalent to mkfifo in bash..
fw = open("named_pipe",'w')
#at this point the system hangs...

My idea it was to use subprocess.Popen and redirect stdout to fw...
next open named_pipe for reading and giving it as input to cat (still using Popen).

I know it is a simple (and rather stupid) example, but I can't manage to make it work..


How would you implement such simple scenario?

Thanks a lot in advance for the help!!!

Luca

[Paul Wiseman]

You can pipe using subprocess

p1 = subprocess.Popen(["ls", "-lah"], stdout=subprocess.PIPE)

p2 = subprocess.Popen(["cat"], stdin=p1.stdout)

p1.wait()

p2.wait()

You can also pass a file object to p1's stdout and p2's stdin if you want to pipe via a file.

with open("named_pipe", "rw") as named_pipe:

    p1 = subprocess.Popen(["ls", "-lah"], stdout=named_pipe)

    p2 = subprocess.Popen(["cat"], stdin=named_pipe)

    p1.wait()

    p2.wait()

 

Luca
--
http://mail.python.org/mailman/listinfo/python-list

[Luca Cerone]

Hi Paul, first of all thanks for the help.

I am aware of the first solutions, just now I would like to experiment a bit with using named pipes (I also know that the example is trivial, but it just to grasp the main concepts)

>
> You can also pass a file object to p1's stdout and p2's stdin if you want to >pipe via a file.

>
> with open("named_pipe", "rw") as named_pipe:
>     p1 = subprocess.Popen(["ls", "-lah"], stdout=named_pipe)
>     p2 = subprocess.Popen(["cat"], stdin=named_pipe)
>
>     p1.wait()
>     p2.wait()
>  

Your second example doesn't work for me.. if named_file is not a file in the folder I get an error saying that there is not such a file.

If I create named_pipe as a named pipe using os.mkfifo("named_file",0777) than the code hangs.. I think it is because there is not process that reads the
content of the pipe, so the system waits for the pipe to be emptied.

Thanks a lot in advance for the help in any case.
Luca

[MRAB]

Opening the pipe for reading will block until it's also opened for
writing, and vice versa.

In your bash code, 'ls' blocked until you ran 'cat', but because you
ran 'ls' in the background you didn't notice it!

In your Python code, the Python thread blocked on opening the pipe for
writing. It was waiting for another thread or process to open the pipe
for reading.

[Luca Cerone]

Hi MRAB, thanks for the reply!

>
> Opening the pipe for reading will block until it's also opened for
>
> writing, and vice versa.
>

OK.

>
>
> In your bash code, 'ls' blocked until you ran 'cat', but because you
>
> ran 'ls' in the background you didn't notice it!
>
>

Right.

>
> In your Python code, the Python thread blocked on opening the pipe for
>
> writing. It was waiting for another thread or process to open the pipe
>
> for reading.

OK. What you have written makes sense to me. So how do I overcome the block?
As you said in bash I run the ls process in background. How do I do that in Python?

Thanks again for the help,
Luca

[Alister]

Are you sure you are using the correct tool for the task?

I tend to find that in most cases if you are trying to execute bash
commands from Python you are doing it wrong.

certainly if you are trying to pipe one bash command into another you
would probably be better of with a bash script.




--
When I was crossing the border into Canada, they asked if
I had any firearms with me. I said, "Well, what do you need?"
-- Steven Wright

[MRAB]

You need to ensure that the pipe is already open at the other end.

Why are you using a named pipe anyway?

If you're talking to another program, then that needs to be running
already, waiting for the connection, at the point that you open the
named pipe from this end.

If you're using a pipe _within_ a program (a queue would be better),
then you should opening for writing in one thread and for reading in
another.

[Luca Cerone]

Hi Alister,

> Are you sure you are using the correct tool for the task?

Yes. For two reasons: 1. I want to learn how to do this in Python :) 2. for an application I have in mind I will need to run external tools (not developed by me) and process the output using some tools that I have written in Python.

For technical reasons I can't use the subprocess.communicate() (the output to process is very large) method, and due to a bug in the interactive shell I am using (https://github.com/ipython/ipython/issues/3884) I cannot pipe processes just using the standard subprocess.Popen() approach.

>
> I tend to find that in most cases if you are trying to execute bash
>
> commands from Python you are doing it wrong.

As I said, the example in my question is just for learning purposes, I don't want to reproduce ls and cat in Python...

I just would like to learn how to handle named pipes in Python, which I find it easier to do by using a simple example that I am comfortable to use :)

Thanks in any case for your answer,
Luca

[Luca Cerone]

Thanks MRAB,

>
> You need to ensure that the pipe is already open at the other end.

So I need to open the process that reads the pipe before writing in it?

>
>
>
> Why are you using a named pipe anyway?

For some bug in ipython (see my previous email) I can't use subprocess.Popen and pipe in the standard way.
One of Ipython developers has suggested me to use named pipes as a temporary workaround. So I am taking the occasion to learn :)

>
> If you're talking to another program, then that needs to be running
>
> already, waiting for the connection, at the point that you open the
>
> named pipe from this end.

I am not entirely sure I got this: ideally I would like to have a function that runs an external tool (the equivalent of ls in my example) redirecting its output in a named pipe.

A second function (the cat command in my example) would read the named_pipe, parse it and extract some features from the output.

I also would like that the named_pipe is deleted when the whole communication is ended.

>
> If you're using a pipe _within_ a program (a queue would be better),
>
> then you should opening for writing in one thread and for reading in
>
> another.

Let's stick with the pipe :) I will ask about the queue when I manage to use pipes ;)

I should have explained better that I have no idea how to run threads in Python :): how do I open a thread that executes "ls -lah" in background and writes into a named pipe? And how do I open a thread that reads from the named pipe?

Can you please post a small example, so that I have something to work on?

Thanks a lot in advance for your help!

Luca

[Neil Cerutti]

On 2013-08-05, Luca Cerone <luca....@gmail.com> wrote:
> I just would like to learn how to handle named pipes in Python,
> which I find it easier to do by using a simple example that I
> am comfortable to use :)

Names pipes are a unix concept that saves you the hassle and
limitations of writing to and reading from a temp file.

You'll have to create the temp file and manage attaching
processes to it yourself.

--
Neil Cerutti

[MRAB]

On 05/08/2013 16:27, Luca Cerone wrote:
> Thanks MRAB,
>>
>> You need to ensure that the pipe is already open at the other end.
>
> So I need to open the process that reads the pipe before writing in
> it?
>
>>
>> Why are you using a named pipe anyway?
>
> For some bug in ipython (see my previous email) I can't use
> subprocess.Popen and pipe in the standard way. One of Ipython
> developers has suggested me to use named pipes as a temporary
> workaround. So I am taking the occasion to learn :)
>

An alternative workaround is to use CPython. :-)

>> If you're talking to another program, then that needs to be
>> running already, waiting for the connection, at the point that you
>> open the named pipe from this end.
>
> I am not entirely sure I got this: ideally I would like to have a
> function that runs an external tool (the equivalent of ls in my
> example) redirecting its output in a named pipe.
>
> A second function (the cat command in my example) would read the
> named_pipe, parse it and extract some features from the output.
>
> I also would like that the named_pipe is deleted when the whole
> communication is ended.
>
>> If you're using a pipe _within_ a program (a queue would be
>> better), then you should opening for writing in one thread and for
>> reading in another.
>
> Let's stick with the pipe :) I will ask about the queue when I
> manage to use pipes ;)
>
> I should have explained better that I have no idea how to run
> threads in Python :): how do I open a thread that executes "ls -lah"
> in background and writes into a named pipe? And how do I open a
> thread that reads from the named pipe?
>
> Can you please post a small example, so that I have something to
> work on?
>

You could try something like this:

os.mkfifo("named_pipe", 0777)
ls_process = subprocess.Popen("ls -lah > named_pipe")
pipe = open("named_pipe", "r")
# Read the output of the subprocess from the pipe.

When the subprocess terminates (look at the docs for Popen objects),
close and delete the fifo.

[Luca Cerone]

Thanks this works (if you add shell=True in Popen).
If I don't want to use shell = True, how can I redirect the stdout to named_pipe? Popen accepts an open file handle for stdout, which I can't open for writing because that blocks the process...

[MRAB]

On 05/08/2013 17:54, Luca Cerone wrote:
> Thanks this works (if you add shell=True in Popen).
> If I don't want to use shell = True, how can I redirect the stdout to named_pipe? Popen accepts an open file handle for stdout, which I can't open for writing because that blocks the process...
>

You're back to using separate threads for the reader and the writer.
The one that opens the pipe first will block until the other one opens
the other end.

[Luca Cerone]

> You're back to using separate threads for the reader and the writer.
>

And how do I create separate threads in Python? I was trying to use the threading library without not too success..

[MRAB]

On 05/08/2013 22:47, Luca Cerone wrote:
>> You're back to using separate threads for the reader and the writer.
>>
> And how do I create separate threads in Python? I was trying to use the threading library without not too success..
>

To run a function in a separate thread:

import threading

def my_func(arg_1, arg_2):
...

my_thread = threading.Thread(target=my_func, args=(arg_1, arg_2))
my_thread.start()


Is the thread still running?

if my_thread.is_alive():
...


Wait for the thread to terminate:

my_thread.join()

[Luca Cerone]

> my_thread.join()

Thanks! I managed to make it work using the threading library :)

[Gregory Ewing]

Luca Cerone wrote:
> Thanks! I managed to make it work using the threading library :)

If at least one of the external programs can accept the source
or destination as a filename argument instead of redirecting its
stdin or stdout, you can also do something like this:

import subprocess

p2 = subprocess.Popen(["cat", "named_pipe"])
pipe = open("named_pipe", "w")
p1 = subprocess.Popen(["ls", "-lah"], stdout = pipe)
pipe.close()
p1.wait()
p2.wait()

That works because opening the reading end of the pipe is
done by the subprocess executing cat, leaving the main
process free to open the other end.

--
Greg