Running 4000 processes
Jeannot
I have lots of processes to run (4000). They have to be submitted to a
server using the qsub program (qeuing system0. Usually I use the exec
command following the the command to execute but hte problem is that
qsub does not handle with arguments. So my solution was to use open "|
qsub"... which I am not used to!
this is inside a loop that will launch the 4000 processes:
set MUSCLE_cmd "muscle -in $Input"
set F [open "|qsub" r+]
puts $F $MUSCLE_cmd
puts "close $F"
flush $F
This seems to work fine unless the fact that after a certain number of
processes being launched I got this error message?
"Error in startup script: couldn't create pipe: too many open files"
Any idea of how I could fix this or enhance my script? I really want to
understand how this work which I am not sure of getting every
details... if there is so;ething to to with exec, please tell me also!!
May thanks in advance.
Jeannot
peace07
this message reflects the fact, that the underlaying OS can't open any
more files. On which OS are you working? Unix? If so, you are probably
fine, because you can simply change the number of max open files. To
see the limit, type
ulimit -a
on the command line.
If you reached the hard-limit allready, you have to change this in a
way, depending on your OS.
It has nothing to do with Tcl.
Regards,
Friedemann
Jeannot
improve the tcl scripts?
Jeannot
On 26 jan, 11:36, "peace07" <f.flo...@googlemail.com> wrote:
Andreas Leitgeb
> this is inside a loop that will launch the 4000 processes:
> set MUSCLE_cmd "muscle -in $Input"
> set F [open "|qsub" r+]
> puts $F $MUSCLE_cmd
> puts "close $F"
rather than execute it?
if you just wrote:
close $F
and then removed the following line:
> flush $F
Then you wouldn't be leaking file-descriptors, and then
you could open (, use and close!) 100s of millions of
filedescriptors during lifetime of that one script :-)
The problem is, you're not closing the open files,
and there is a limit on how many files can be open
at the *same time*.
There is no limit on how often you can open+close
files sequentially.
Jeannot
lot!!
Jeannot
On Jan 26, 12:41 pm, Andreas Leitgeb <a...@gamma.logic.tuwien.ac.at>
wrote:
> Jeannot <jean.mul...@igbmc.u-strasbg.fr> wrote:
> > this is inside a loop that will launch the 4000 processes:
> > set MUSCLE_cmd "muscle -in $Input"
> > set F [open "|qsub" r+]
> > puts $F $MUSCLE_cmd
> > puts "close $F"why do you just write the close command to your script's stdout,
> rather than execute it?
>
> if you just wrote: close $F
> and then removed the following line:
>
> > flush $FThen you wouldn't be leaking file-descriptors, and then
Cameron Laird
Jeannot <jean....@igbmc.u-strasbg.fr> wrote:
>Hi, yes this was my failure. I did it and now it works nice thanks a
>lot!!
>
>Jeannot
>
>On Jan 26, 12:41 pm, Andreas Leitgeb <a...@gamma.logic.tuwien.ac.at>
>wrote:
>> Jeannot <jean.mul...@igbmc.u-strasbg.fr> wrote:
>> > this is inside a loop that will launch the 4000 processes:
>> > set MUSCLE_cmd "muscle -in $Input"
>> > set F [open "|qsub" r+]
>> > puts $F $MUSCLE_cmd
>> > puts "close $F"why do you just write the close command to your
>script's stdout,
>> rather than execute it?
>>
>> if you just wrote: close $F
.
.
You've achieved what you're after. There's no need
to do more.
However, there's an aspect that might interest you
or others. Earlier you wrote about a difficulty with
exec. While I don't find enough details in this
thread to be certain of the facts, I suspect
exec qsub << $MUSCLE_cmd
is a single line which gives the same result as you
currently code with three.
Andreas Leitgeb
>>> > set MUSCLE_cmd "muscle -in $Input"
>>> > set F [open "|qsub" r+]
>>> > puts $F $MUSCLE_cmd
>
> However, there's an aspect that might interest you
> or others. Earlier you wrote about a difficulty with
> exec. While I don't find enough details in this
> thread to be certain of the facts, I suspect
> exec qsub << $MUSCLE_cmd
> is a single line which gives the same result as you
> currently code with three.
unless $MUSCLE_cmd could possibly contain \0-characters:
then, he'd be hit by a bug reported (on sf) long (almost 2 years)
ago, acknowedged, but still not yet fixed (bugId: 1189293) :-(
Donal K. Fellows
> unless $MUSCLE_cmd could possibly contain \0-characters:
What, qsub do something useful with NULs? Won't happen. While healthy
paranoia about bad characters is usually good, sometimes you can know a
lot about what's going on anyway and shortcut such things. And qsub is
one of those programs that just takes text anyway; a NUL would probably
break it.
Donal.