script files with python (instead of tcsh/bash)?
Esmail
I am wondering if anyone is using python to write script files?
Right now I have a bigg'ish bash/tcsh script that contain some grep/awk
command plus various files are processed and created, renamed and
moved to specific directories. I also write out some gnuplot scripts
that later get executed to generate .jpg images.
In any case, the scripts are starting to look pretty hairy and I was
wondering if it would make sense to re-write them in Python. I am not
sure how suitable it would be for this.
I've looked around the web w/o much luck for some examples but come
short. Any comments/suggestions?
Thanks,
Esmail
Aahz
These days, I always convert any even slightly complicated script to
Python.
>I've looked around the web w/o much luck for some examples but come
>short. Any comments/suggestions?
Not sure what you're looking for here -- many things you'd run an
external program for in scripting can be accomplished with Python library
calls, and for the rest, you can use the subprocess module (or os.system
if you have no acces to Python 2.4 or higher).
--
Aahz (aa...@pythoncraft.com) <*> http://www.pythoncraft.com/
"Programming language design is not a rational science. Most reasoning
about it is at best rationalization of gut feelings, and at worst plain
wrong." --GvR, python-ideas, 2009-3-1
Esmail
> In article <mailman.2374.1237641...@python.org>,
> Esmail <ebo...@hotmail.com> wrote:
>> I am wondering if anyone is using python to write script files?
>
> These days, I always convert any even slightly complicated script to
> Python.
well .. that sounds encouraging ...
>> I've looked around the web w/o much luck for some examples but come
>> short. Any comments/suggestions?
>
> Not sure what you're looking for here -- many things you'd run an
> external program for in scripting can be accomplished with Python library
> calls, and for the rest, you can use the subprocess module (or os.system
> if you have no acces to Python 2.4 or higher).
I have access to 2.5 or more recent. I guess I was looking for some
example scripts in Python and perhaps the equivalent in bash/tsch to
show some of the equivalences. I am being impatient, I guess I need to
dig into the language/library documentation a bit more on my own.
Esmail
Joe Riopel
> In any case, the scripts are starting to look pretty hairy and I was
> wondering if it would make sense to re-write them in Python. I am not
> sure how suitable it would be for this.
Are these scripts run on computers that are guaranteed to have Python
installed? If not, can you install Python on them?
I use Python when I can, but sometimes shell scripts just makes sense. T
Esmail
Joe Riopel wrote:
>
> Are these scripts run on computers that are guaranteed to have Python
> installed? If not, can you install Python on them?
>
> I use Python when I can, but sometimes shell scripts just makes sense. T
Yes. Currently I am running the bash/tcsh scripts under Ubuntu. The scripts
process the output of a set of big simulation runs using R. I can run R under
Windows too, but then I can't use the shell scripts to process them, so in
fact Python would make things more portable for me.
Shell scripting works for me for quick and relatively short tasks, as soon
as it gets a bit longer I miss having cleaner control structures and
functions, and Python having such a clean code would be great and make
maintenance easier too as I tweak the code.
Esmail
Esmail
D'Arcy J.M. Cain
Esmail <ebo...@hotmail.com> wrote:
> Hello all,
>
> I am wondering if anyone is using python to write script files?
All the time.
> Right now I have a bigg'ish bash/tcsh script that contain some grep/awk
> command plus various files are processed and created, renamed and
> moved to specific directories. I also write out some gnuplot scripts
> that later get executed to generate .jpg images.
>
> In any case, the scripts are starting to look pretty hairy and I was
> wondering if it would make sense to re-write them in Python. I am not
> sure how suitable it would be for this.
Your message is a little confused. In one paragraph you appear to be
talking about one specific script and the next you are talking about
more than one. Regardless, I will try to give some generic advice.
First, have a toolbox. No carpenter goes to work with just a hammer.
Sometimes he needs a screwdriver and more than one type at that. Try
to be comfortable with all of your tools. Once you reach that point
you should have no problem figuring out which tool you need for any
specific job.
Sometimes the answer is Perl, sed, awk or some other tool, perhaps a
specialized one that you wrote. OK, maybe it's never Perl. :-)
Naturally you can use popen and system to convert any shell script into
Python but if your Python script is overflowing with those calls then
you need to either refactor your code to use Python directly or stay
with the shell script.
There's nothing that says that you need to use one and only one tool
for any particular job. I have many times written shell scripts that
call Python (either by script or command line) to do things that are
easier in Python. Yesterday I did the opposite. I had a shell script
that generated some data and I used a Python script to popen it and
work with the information.
After all that, if you really do feel that you need to focus on one
tool I suppose you could do worse than choosing Python as your hammer.
> I've looked around the web w/o much luck for some examples but come
> short. Any comments/suggestions?
I think that the reason for that is that most people aren't so
concerned with conversion. If the shell script works, why bother?
They will just pick the best tool for their next project.
--
D'Arcy J.M. Cain <da...@druid.net> | Democracy is three wolves
http://www.druid.net/darcy/ | and a sheep voting on
+1 416 425 1212 (DoD#0082) (eNTP) | what's for dinner.
Aahz
If you post a sample script you're trying to convert, you may get some
responses that show how different people would write it in Python.
Peter Pearson
>
> I am wondering if anyone is using python to write script files?
If it can be done in a few simple lines of shell script,
fine: make it a shell script. But if it's more complex than
that, Python is clearer. Just my two cents.
--
To email me, substitute nowhere->spamcop, invalid->net.
Esmail
>
> If you post a sample script you're trying to convert, you may get some
> responses that show how different people would write it in Python.
That's a nice suggestion .. I may end up doing this after I do some
readings, just wanted to make sure this is not too outlandish of an
idea :-)
Esmail
Esmail
> On Sat, 21 Mar 2009 09:26:02 -0400, Esmail <ebo...@hotmail.com> wrote:
>> I am wondering if anyone is using python to write script files?
>
> If it can be done in a few simple lines of shell script,
> fine: make it a shell script. But if it's more complex than
> that, Python is clearer. Just my two cents.
Hi Peter,
Yes, I agree .. the complexity of the script(s) have evolved to the
stage that I was thinking moving to Python makes sense (if that's something
that Python is suitable for - which it appears it is :-)
Cheers,
Esmail
andrew cooke
> Peter Pearson wrote:
>> On Sat, 21 Mar 2009 09:26:02 -0400, Esmail <ebo...@hotmail.com> wrote:
>>> I am wondering if anyone is using python to write script files?
>> If it can be done in a few simple lines of shell script,
>> fine: make it a shell script. But if it's more complex than
>> that, Python is clearer. Just my two cents.
> stage that I was thinking moving to Python makes sense (if that's
> something
> that Python is suitable for - which it appears it is :-)
just a quick data point here - i once wrote a "blog engine" (that's a
rather grand name for some scripts that convert emails to web pages) in
bash. i later rewrote it in python, but it turned out to be a similar
size and complexity. now granted, i made the original design so that it
played to bash's strengths, but even so i was quite surprised. it brought
home to me that bash really is quite powerful.
so you might be better spending the time improving your bash skills than
doing what will be largely drudge work in a language you already know.
andrew
ps i just checked my notes - http://www.acooke.org/cute/Computeisb0.html -
and the one obvious advantage of python that was the improvement in speed.
Ned Deily
wrote:
> Aahz wrote:
> > In article <mailman.2374.1237641...@python.org>,
> > Esmail <ebo...@hotmail.com> wrote:
> >> I am wondering if anyone is using python to write script files?
> >
> > These days, I always convert any even slightly complicated script to
> > Python.
> >> short. Any comments/suggestions?
> > Not sure what you're looking for here -- many things you'd run an
> > external program for in scripting can be accomplished with Python library
> > calls, and for the rest, you can use the subprocess module (or os.system
> > if you have no acces to Python 2.4 or higher).
> example scripts in Python and perhaps the equivalent in bash/tsch to
> show some of the equivalences. I am being impatient, I guess I need to
> dig into the language/library documentation a bit more on my own.
Perhaps the recipe for Pyline might give you some ideas on how to write
python scripts that play well with other scripts.
<http://code.activestate.com/recipes/437932/>
--
Ned Deily,
n...@acm.org
Esmail
>
> Perhaps the recipe for Pyline might give you some ideas on how to write
> python scripts that play well with other scripts.
>
> <http://code.activestate.com/recipes/437932/>
ah .. very nice .. thanks!
Esmail
> Esmail wrote:
>
> just a quick data point here -
<..>
> so you might be better spending the time improving your bash skills than
> doing what will be largely drudge work in a language you already know.
I'll have to think about it .. at this point I know both languages about
equally "well" .. (not really), but Python has more universal applicability,
so I thought this might be a good reason to hone my Python skills and come
up with something useful at the same time.
Thanks for the post, the more data points,the better.
Cheers,
Esmail
Esmail
Nick Craig-Wood
> I am wondering if anyone is using python to write script files?
Yes!
> Right now I have a bigg'ish bash/tcsh script that contain some grep/awk
> command plus various files are processed and created, renamed and
> moved to specific directories. I also write out some gnuplot scripts
> that later get executed to generate .jpg images.
Almost any script that contains a loop I convert into python.
> In any case, the scripts are starting to look pretty hairy and I was
> wondering if it would make sense to re-write them in Python. I am not
> sure how suitable it would be for this.
With python you get the advantages of a language with a very clear
philosophy which is exceptionally easy to read and write.
Add a few classes and unit tests to your scripts and they will be
better than you can ever achieve with bash (IMHO).
> I've looked around the web w/o much luck for some examples but come
> short. Any comments/suggestions?
Here is a short script I wrote to recover my slrn spool when my disk
fills up. I could have written it in bash but I think it is much
clearer as python, and doesn't shell out to any external commands.
It searches through the spool directory looking for empty .minmax
files. It then finds the lowest and highest numeric file name (all
the files are numeric) and writes the minmax file with that.
#!/usr/bin/python
"""
rebuild all the .minmax files for slrnpull after the disk gets full
"""
import os
spool = "/var/spool/slrnpull"
def main():
for dirpath, dirnames, filenames in os.walk(spool):
if ".minmax" not in filenames:
continue
minmax_path = os.path.join(dirpath, ".minmax")
if os.path.getsize(minmax_path) != 0:
print "Skipping non empty %r" % minmax_path
continue
print dirpath
digits = [ int(f) for f in filenames if f.isdigit() ]
if not digits:
digits = [0]
digits.sort()
start = digits[0]
end = digits[-1]
f = open(minmax_path, "w")
f.write("%s %s" % (start, end))
f.close()
print "done"
if __name__ == "__main__": main()
--
Nick Craig-Wood <ni...@craig-wood.com> -- http://www.craig-wood.com/nick
Esmail
> Esmail <ebo...@hotmail.com> wrote:
>> I am wondering if anyone is using python to write script files?
>
> Yes!
>
>
> Almost any script that contains a loop I convert into python.
>
>> In any case, the scripts are starting to look pretty hairy and I was
>> wondering if it would make sense to re-write them in Python. I am not
>> sure how suitable it would be for this.
>
> With python you get the advantages of a language with a very clear
> philosophy which is exceptionally easy to read and write.
>
> Add a few classes and unit tests to your scripts and they will be
> better than you can ever achieve with bash (IMHO).
Hi Nick,
thanks for including the script, that really helps. Nice way of
finding files.
Two quick questions:
As a replacement for grep I would use the re module and its methods?
What about awk which I regularly use to extract fields based on position
but not column number, what should I be using in Python to do the same?
The other things I need to do consist of moving files, manipulating file
names and piping outputs of one command to the next, so I'm digging into
the documentation as much as I can.
So much to learn, so little time (but so much fun!)
Esmail
MRAB
> Nick Craig-Wood wrote:
>> Esmail <ebo...@hotmail.com> wrote:
>>> I am wondering if anyone is using python to write script files?
>>
>> Yes!
>>
>>
>> Almost any script that contains a loop I convert into python.
>>
>>> In any case, the scripts are starting to look pretty hairy and I was
>>> wondering if it would make sense to re-write them in Python. I am not
>>> sure how suitable it would be for this.
>>
>> With python you get the advantages of a language with a very clear
>> philosophy which is exceptionally easy to read and write.
>>
>> Add a few classes and unit tests to your scripts and they will be
>> better than you can ever achieve with bash (IMHO).
>
>
>
> thanks for including the script, that really helps. Nice way of
> finding files.
>
> Two quick questions:
>
> As a replacement for grep I would use the re module and its methods?
>
> What about awk which I regularly use to extract fields based on position
> but not column number, what should I be using in Python to do the same?
>
> The other things I need to do consist of moving files, manipulating file
> names and piping outputs of one command to the next, so I'm digging into
> the documentation as much as I can.
>
The 'os' and 'shutil' modules.
Gabriel Genellina
escribió:
> Esmail wrote:
>> Nick Craig-Wood wrote:
>>>> I am wondering if anyone is using python to write script files?
>> Two quick questions:
>> As a replacement for grep I would use the re module and its methods?
Perhaps; but strings have methods too (`"abc" in line` is easier to read,
and faster, than the corresponding r.e.)
>> The other things I need to do consist of moving files, manipulating file
>> names and piping outputs of one command to the next, so I'm digging into
>> the documentation as much as I can.
>>
> The 'os' and 'shutil' modules.
And for executing external commands with piping, I'd add the subprocess
module.
--
Gabriel Genellina
Nick Craig-Wood
> thanks for including the script, that really helps. Nice way of
> finding files.
Python has lots of useful stuff like that!
> Two quick questions:
>
> As a replacement for grep I would use the re module and its
> methods?
The re module works on strings not files, but basically yes.
Note that the re module uses a superset of the "grep -E" regexps, and
they are almost identical to those used in perl and php.
Here is a simple grep in python
#!/usr/bin/python
import re
import sys
import fileinput
pattern = sys.argv.pop(1)
for line in fileinput.input():
if re.search(pattern, line):
print line.rstrip()
Save in a file called grep.py then you can do
$ ./grep.py dizzy *.py
self.dizzy = 0
if self.dizzy:
$ ls | ./grep.py '\d{2}'
linux-image-2.6.24-21-eeepc_2.6.24-21.39eeepc1_i386.deb
linux-ubuntu-modules-2.6.24-21-eeepc_2.6.24-21.30eeepc6_i386.deb
> What about awk which I regularly use to extract fields based on position
> but not column number, what should I be using in Python to do the
> same?
I presume you mean something like this
... | awk '{print $2}'
In python, assuming you've got the line in the "line" variable, then
In python an equivalent of the above would be
import fileinput
for line in fileinput.input():
print line.split()[1]
Note that the fileinput module is really useful for making shell
command replacements!
> The other things I need to do consist of moving files, manipulating file
> names and piping outputs of one command to the next, so I'm digging into
> the documentation as much as I can.
Read up on the os module and the subprocess module. You'll find you
need to do much less piping with python as with shell because it has
almost everything you'll need built in.
Using built in functions is much quicker than fork()-ing an external
command too.
> So much to learn, so little time (but so much fun!)
;-)
Esmail
>
>> Two quick questions:
>>
>> As a replacement for grep I would use the re module and its methods?
>>
>> but not column number, what should I be using in Python to do the same?
>>
Would that be equivalent to splitting the string and then subscripting
on the result?
That seems to be the closest to awk .. wouldn't string slicing depend on
column numbers for the line (so to speak)i.e, specific know index values?
>> The other things I need to do consist of moving files, manipulating file
>> names and piping outputs of one command to the next, so I'm digging into
>> the documentation as much as I can.
>>
> The 'os' and 'shutil' modules.
Excellent, thanks for the pointers. I was aware of the os module, but
didn't know about the shutil module. The more I dig into Python, the
more I like it and the more I'm impressed. (I do miss bock comments,
when you are trying out new things that is useful .. I know I can use
""" as a substitute though).
Thanks again,
Esmail
Esmail
Gabriel Genellina wrote:
> En Sun, 22 Mar 2009 11:05:22 -0300, MRAB <goo...@mrabarnett.plus.com>
> escribió:
>> Esmail wrote:
>>> Nick Craig-Wood wrote:
>>>> Esmail <ebo...@hotmail.com> wrote:
<..>
>>> As a replacement for grep I would use the re module and its methods?
>
> Perhaps; but strings have methods too (`"abc" in line` is easier to
> read, and faster, than the corresponding r.e.)
I'm a big fan of readability (which is one reason I am attracted
to Python) .. thanks for pointing out the alternative, good to have
options.
>>> The other things I need to do consist of moving files, manipulating file
>>> names and piping outputs of one command to the next, so I'm digging into
>>> the documentation as much as I can.
>>>
>> The 'os' and 'shutil' modules.
>
> And for executing external commands with piping, I'd add the subprocess
> module.
I will take a look at the subprocess module, thanks!
Esmail
Esmail
thanks for the additional script example. I was able to put
something together where I read the whole file into a list
as a series of lines (via readlines()) and then loop through
the lines seeing if the target string was "in" the line .. seems
to have worked reasonably well.
I am sure over time I will pick up the more Python(ic?) ways of
doing things.
> I presume you mean something like this
>
> ... | awk '{print $2}'
>
> In python, assuming you've got the line in the "line" variable, then
>
> In python an equivalent of the above would be
>
> import fileinput
> for line in fileinput.input():
> print line.split()[1]
cool .. that is what I came up with too!
> Note that the fileinput module is really useful for making shell
> command replacements!
yes, I am going to have to process a number of files, so I'm also
looking at glob and os.walk(??)
>> The other things I need to do consist of moving files, manipulating file
>> names and piping outputs of one command to the next, so I'm digging into
>> the documentation as much as I can.
>
> Read up on the os module and the subprocess module. You'll find you
> need to do much less piping with python as with shell because it has
> almost everything you'll need built in.
>
> Using built in functions is much quicker than fork()-ing an external
> command too.
Thanks again for all the useful info and examples, this really has been
a great help.
Esmail
Alan G Isaac
> I also write out some gnuplot scripts
> that later get executed to generate .jpg images.
See Gnuplot.py
Alan Isaac
Esmail
Thanks Alan, I will!
Esmail
R. David Murray
> Hello again Nick,
>
> thanks for the additional script example. I was able to put
> something together where I read the whole file into a list
> as a series of lines (via readlines()) and then loop through
> the lines seeing if the target string was "in" the line .. seems
> to have worked reasonably well.
>
> I am sure over time I will pick up the more Python(ic?) ways of
> doing things.
Here's a more Pythonic way to do that:
with open('somefile') as f:
for line in f:
if 'somestring' in line:
#do something
In other words, you don't have to read the lines into a list first if all
you are going to do is iterate through them. (The 'with' clause closes
the file at block exit...which is overkill if this is all the program
is doing since the file will be closed at program termination anyway,
but is a good habit to get in to.)
--
R. David Murray http://www.bitdance.com
Esmail
R. David Murray wrote:
> Esmail <ebo...@hotmail.com> wrote:
>
> Here's a more Pythonic way to do that:
>
> with open('somefile') as f:
> for line in f:
> if 'somestring' in line:
> #do something
>
> In other words, you don't have to read the lines into a list first if all
> you are going to do is iterate through them.
Cool .. stored away for future reference.
In this case I believe I needed the contents in a list because the line
I was looking for was above one that I could easily identify. Ie, once
I had the index, I could go back/up one index to find the line I needed
to process.
> (The 'with' clause closes
> the file at block exit...which is overkill if this is all the program
> is doing since the file will be closed at program termination anyway,
> but is a good habit to get in to.)
thanks for reminding me about the 'with' clause. I agree with closing
files and in general being mindful about allocated resources.
Cheers,
Esmail
Esmail
Peter Otten
> Hello David,
>
> R. David Murray wrote:
>> Esmail <ebo...@hotmail.com> wrote:
>>
>> Here's a more Pythonic way to do that:
>>
>> with open('somefile') as f:
>> for line in f:
>> if 'somestring' in line:
>> #do something
>>
>> In other words, you don't have to read the lines into a list first if all
>> you are going to do is iterate through them.
>
> Cool .. stored away for future reference.
>
> In this case I believe I needed the contents in a list because the line
> I was looking for was above one that I could easily identify. Ie, once
> I had the index, I could go back/up one index to find the line I needed
> to process.
Here's one way to avoid the list:
last_line = None
for line in f:
if "somestring" in line and last_line is not None:
# do something with last_line
last_line = line
Peter
Esmail
>
>> In this case I believe I needed the contents in a list because the line
>> I was looking for was above one that I could easily identify. Ie, once
>> I had the index, I could go back/up one index to find the line I needed
>> to process.
>
> Here's one way to avoid the list:
>
> last_line = None
> for line in f:
> if "somestring" in line and last_line is not None:
> # do something with last_line
> last_line = line
>
> Peter
yup .. this will come in handy in case the file is too
big and or really needn't be all in memory. cool. thanks!
Esmail