piping into a python script
Donn Ingle
(Gnu/Linux - Python 2.4/5)
Given these two examples:
1.
./fui.py *.py
2.
ls *.py | ./fui.py
How can I capture a list of the arguments?
I need to get all the strings (file or dir names) passed via the normal
command line and any that may come from a pipe.
There is a third case:
3.
ls *.jpg | ./fui.py *.png
Where I would be gathering strings from two places.
I am trying to write a command-line friendly tool that can be used in
traditional gnu/linux ways, otherwise I'd skip the pipe stuff totally.
I have tried:
1. pipedIn = sys.stdin.readlines()
Works fine for example 2, but example 1 goes into a 'wait for input' mode
and that's no good. Is there a way to tell when no input is coming from a
pipe at all?
2. import fileinput
for line in fileinput.input():
print (line)
But this opens each file and I don't want that.
I have seen a lot of search results that don't quite answer this angle of
the question, so I'm trying on the list.
\d
Marc 'BlackJack' Rintsch
> Given these two examples:
> 1.
> ./fui.py *.py
> 2.
> ls *.py | ./fui.py
>
> How can I capture a list of the arguments?
> I need to get all the strings (file or dir names) passed via the normal
> command line and any that may come from a pipe.
>
> There is a third case:
> 3.
> ls *.jpg | ./fui.py *.png
> Where I would be gathering strings from two places.
>
> I am trying to write a command-line friendly tool that can be used in
> traditional gnu/linux ways, otherwise I'd skip the pipe stuff totally.
>
> I have tried:
> 1. pipedIn = sys.stdin.readlines()
> Works fine for example 2, but example 1 goes into a 'wait for input' mode
> and that's no good. Is there a way to tell when no input is coming from a
> pipe at all?
Usually Linux tools that can get the data from command line or files treat
a single - as file name special with the meaning of: read from stdin.
So the interface if `fui.py` would be:
1. ./fui.py *.a
2. ls *.a | ./fui.py -
3. ls *.a | ./fui.py *.b -
Ciao,
Marc 'BlackJack' Rintsch
Paddy
Try the fileinput module.
What you describe above is pretty close to the unix 'standard' but not
quite.
if we substitute the lp command instead of ./fui, the command normally
takes a list of files to act on as its arguments, and anything piped
in goes to its stdin where it is processed if it has an argument of -
fileinput works that way but you may have problems with your:
ls *.jpg | ./fui.py *.png
Which might better be expressed as:
./fui.py `ls *.jpg` *.png
which would work for ls and a python program using the fileinput
module.
- Paddy.
Paddy
If X.a X.b Y.a Y.b are all files whose contents are to be processed
then
To process all files:
./fui.py *.a *.b
Or:
./fui.py `ls *.a *.b`
To process one file from a pipe unix usually does:
cat X.a | ./fui.py -
To get the filenames from stdin would usually need a command line
switch telling fui.py to read a file *list* from stdin. For verilog
simulators for example you have the -f switch that says insert further
command line arguments from the file name in the next argument, so you
could do:
ls *.a | ./fui.py -f - *.b
For equivalent functionality to my first example.
- Paddy.
Donn Ingle
I did give the fileinput module a go, but I can't find much info on it and
the help is ... well, it's python help ;)
> in goes to its stdin where it is processed if it has an argument of -
> fileinput works that way
Okay, I did think of the dash, but did not know how to handle it. Is it a
bash thing or will that dash get passed into the args? (I am using getopt
to parse the options and args)
> which would work for ls and a python program using the fileinput
> module.
Any examples of fileinput (that do not open each file) would be great!
(I'll go searching now anyway)
Thanks,
\d
Donn Ingle
> ls *.a | ./fui.py -f - *.b
where all the piped filenames will go, so *.b happens after *.a has been
expanded and they all get fed to -f, right?
I'm also guessing you mean that I should detect the single dash and then go
look for stdin at that point. How do I detect a lack of stdin?
Thanks,
\d
Paddy
> > Try the fileinput module.
>
> I did give the fileinput module a go, but I can't find much info on it and
> the help is ... well, it's python help ;)
Try http://effbot.org/librarybook/fileinput.htm
>
> > in goes to its stdin where it is processed if it has an argument of -
> > fileinput works that way
>
> Okay, I did think of the dash, but did not know how to handle it. Is it a
> bash thing or will that dash get passed into the args? (I am using getopt
> to parse the options and args)
- gets passed in and fileinput handles it.
>
> > which would work for ls and a python program using the fileinput
> > module.
>
> Any examples of fileinput (that do not open each file) would be great!
> (I'll go searching now anyway)
fileinput is set to process each file a line at a time unfortunately.
>
> Thanks,
Your welcome :-)
- Paddy.
Donn Ingle
> fileinput is set to process each file a line at a time unfortunately.
thought a simple list of strings, one from stdin and one from the args,
would be easy to get.
I *really* don't want to open each file, that would be insane.
Perhaps I shall have to forgo the stdin stuff then, after all.
\d
Donn
> it doesn't do anything, just waits for your input. Your applications
> probably should behave the same.
Okay, that works for me.
> Paddy wrote:
> > ls *.a | ./fui.py -f - *.b
> It doesn't seem to me that -f parameter is necessary for your
> application.
Yes and no, I have another option that needs to take a variable number of
args.
> It should treat all the arguments as the filenames,
> shouldn't it? And when one of the filenames is -, just try to read
> stdin.
I have tested getopt and it strips the lone '-' out. I can get it from
sys.argv, but then I am really doing more parsing than I want to. It's a
tricky job this. I think I will look in sys.argv, if I find a single dash the
I will replace that element in the list with whatever comes from stdin. Then
I'll pass all of it to getopt.
Thanks for the help.
\d
--
When you allow legends to rule your life, your world is based on fiction
-- Segio Aragones (Groo the Wanderer Number 99)
Fonty Python and other dev news at:
http://otherwiseingle.blogspot.com/
Hexamorph
> Paddy wrote:
>> fileinput is set to process each file a line at a time unfortunately.
> thought a simple list of strings, one from stdin and one from the args,
> would be easy to get.
>
> I *really* don't want to open each file, that would be insane.
>
> Perhaps I shall have to forgo the stdin stuff then, after all.
>
Hi!
I'm not sure if I completely get what you want, but what's about this:
#!/usr/bin/python
import sys
filelist = []
with_stdin=0
if len(sys.argv) > 1:
for file in sys.argv[1:]:
if file == "-":
with_stdin=1
continue
filelist.append(file)
else:
with_stdin=1
if with_stdin:
for file in sys.stdin:
filelist.append(file)
for file in filelist:
print "Processing file: %s" % file
It's a bit clumsy, but seems to do what I guess you want.
HTH
Reedick, Andrew
> From: python-list-bounces+jr9445=att...@python.org [mailto:python-
> list-bounces+jr9445=att...@python.org] On Behalf Of Donn
> Sent: Thursday, January 24, 2008 12:03 PM
> To: Michał Bentkowski
> Cc: pytho...@python.org
> Subject: Re: piping into a python script
>
> I have tested getopt and it strips the lone '-' out. I can get it from
Try 'foo.py -- -'. The '--' normally tells the parser to stop parsing args. Ex: date > -foo.txt; rm -foo.txt; rm -- -foo.txt
I think this will tell you if stdin is being piped in or not:
import sys
import os
print os.isatty(sys.stdin.fileno())
D:\>type a.txt | python a.py
False
D:\>python a.py
True
Also if you're lazy, look at the StringIO class:
if options.filelist is None and len(args) < 1: # read from stdin
f = sys.stdin
elif options.filelist is not None and len(args) < 1: # read filenames from file
f = open(options.filelist, 'r')
elif options.filelist is None and len(args) > 0: # filenames on command line
f = StringIO.StringIO('\n'.join(args))
else: ## Thanks for playing.
parser.print_help()
exit(1)
if f:
for filename in f:
> -- Segio Aragones (Groo the Wanderer Number 99)
Ah yes, Groo. Ever wonder who would win if Groo and Forrest Gump fought each other?
Donn
> Ah yes, Groo. Ever wonder who would win if Groo and Forrest Gump fought
> each other?
Heh ;) I reckon they'd both die laughing. Be fun to watch -- if anyone else
survived!
\d
--
"A computer without Windows is like chocolate cake without mustard."
-- Anonymous Coward /.
Nick Craig-Wood
Did anyone mention the (standard library) fileinput module? (I missed
the start of this thread.)
http://docs.python.org/lib/module-fileinput.html
11.2 fileinput -- Iterate over lines from multiple input streams
This module implements a helper class and functions to quickly write a
loop over standard input or a list of files.
The typical use is:
import fileinput
for line in fileinput.input():
process(line)
This iterates over the lines of all files listed in sys.argv[1:],
defaulting to sys.stdin if the list is empty. If a filename is '-', it
is also replaced by sys.stdin. To specify an alternative list of
filenames, pass it as the first argument to input(). A single file
name is also allowed.
--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick
Donn
Thanks for your tips. I managed to get a working script going. I am sure there
will be stdin 'issues' to come, but I hope not.
If anyone wants to have a look, it's on the cheese shop at:
http://pypi.python.org/pypi/fui
\d
--
"You know, I've gone to a lot of psychics, and they've told me a lot of
different things, but not one of them has ever told me 'You are an undercover
policewoman here to arrest me.'"
-- New York City undercover policewoman
Donn Ingle
> It's a bit clumsy, but seems to do what I guess you want.
Hey, thanks for that! I will have a go.
\d
Donn Ingle
> This iterates over the lines of all files listed in sys.argv[1:],
> defaulting to sys.stdin if the list is empty. If a filename is '-', it
> is also replaced by sys.stdin. To specify an alternative list of
> filenames, pass it as the first argument to input(). A single file
> name is also allowed.
Yeah it has been discussed. It seems the one problem with it is that it
opens each file. I only want the filenames.
Anyway, this has more-or-less been solved now.
Thanks,
\d
anonymous
I'm not sure I understane the question but my contribution is :
import sys
names = sys.argv[1:]
line = 'x'
while line:
line = sys.stdin.readline().strip()
if line: names.append (line)
print "names=", names
Called using:
ls | stdtest.py arg1 arg2 arg3
Does this help?
Andy