Help: An error in Java program
Sumedh
So, im trying to run shell commands through Java using the
Runtime.getRuntime().exec() facility.
Now, basically I need to run the same command with different arguments
several times over, like thousands of times. So I have the exec() call
in a for loop.
Now after about some time I get this error:
java.io.IOException: Too many open files
at java.lang.UNIXProcess.forkAndExec(Native Method)
at java.lang.UNIXProcess.<init>(UNIXProcess.java:54)
at java.lang.Runtime.execInternal(Native Method)
at java.lang.Runtime.exec(Runtime.java:566)
at java.lang.Runtime.exec(Runtime.java:491)
at java.lang.Runtime.exec(Runtime.java:457)
Now, basically, knowing some unix, I know that ports opened by files
etc show up as file descriptors, and there must be some limit to how
many file descriptors can be opened at one time.
However, my shell command does not open any ports, and is basically a
perl script doing some parsing.
Also, after each command exec(), I do a process.waitFor(), so I ensure
that the process ends before I execute the loop again. (Assuming Java
does this correctly)
So, basically, is the communication between the Java Runtime and the
exec'd process opening so many ports/pipes between them? Im sure
there will be the basic input/output/error between the two programs.
Or, is the process not ending, and waitFor() not working, that several
processes are being exec'd at the same time?
Any ideas on what could be the problem? I've cross-posted to
comp.lang.java.programmers as well as comp.unix.questions, since im
not sure where the problem lies.
Thanks a lot for your answers/thoughts.
sumedh
Gordon Beaton
> So, im trying to run shell commands through Java using the
> Runtime.getRuntime().exec() facility.
>
> Now, basically I need to run the same command with different
> arguments several times over, like thousands of times. So I have the
> exec() call in a for loop.
>
> Now after about some time I get this error:
>
> java.io.IOException: Too many open files
You need to close the streams connecting the Java application to the
child process.
Process p = Runtime.getRuntime().exec(cmd);
...
p.waitFor();
p.getInputStream().close();
p.getOutputStream().close();
p.getErrorStream().close();
/gordon
--
[ do not email me copies of your followups ]
g o r d o n + n e w s @ b a l d e r 1 3 . s e
Roedy Green
quoted :
>Also, after each command exec(), I do a process.waitFor(), so I ensure
>that the process ends before I execute the loop again. (Assuming Java
>does this correctly)
Did you open streams to communicate with your spawned processes? if
so, did you remember to close them?
For other hints see http://mindprod.com/jgloss/exec.html
--
Canadian Mind Products, Roedy Green.
Coaching, problem solving, economical contract programming.
See http://mindprod.com/jgloss/jgloss.html for The Java Glossary.
Roedy Green
quoted :
> p.getInputStream().close();
> p.getOutputStream().close();
> p.getErrorStream().close();
Is that necessary even if you don't use them?
Gordon Beaton
> On 10 Sep 2003 22:00:23 +0200, Gordon Beaton <n...@for.email> wrote or
> quoted :
>> p.getInputStream().close();
>> p.getOutputStream().close();
>> p.getErrorStream().close();
>
> Is that necessary even if you don't use them?
Using them has nothing to do with it - they're opened by exec() when
the child process is started.
j...@invalid.address
> So, im trying to run shell commands through Java using the
> Runtime.getRuntime().exec() facility.
>
> Now, basically I need to run the same command with different
> arguments several times over, like thousands of times. So I have the
> exec() call in a for loop.
>
> Now after about some time I get this error:
>
> java.io.IOException: Too many open files
> at java.lang.UNIXProcess.forkAndExec(Native Method)
> at java.lang.UNIXProcess.<init>(UNIXProcess.java:54)
> at java.lang.Runtime.execInternal(Native Method)
> at java.lang.Runtime.exec(Runtime.java:566)
> at java.lang.Runtime.exec(Runtime.java:491)
> at java.lang.Runtime.exec(Runtime.java:457)
>
> Now, basically, knowing some unix, I know that ports opened by files
> etc show up as file descriptors, and there must be some limit to how
> many file descriptors can be opened at one time.
>
> However, my shell command does not open any ports, and is basically a
> perl script doing some parsing.
>
> Also, after each command exec(), I do a process.waitFor(), so I
> ensure that the process ends before I execute the loop
> again. (Assuming Java does this correctly)
I don't know what process.waitfor() is supposed to do, but when you
spawn a child process, all its descriptors die with it. Unless Java is
doing something weird, it sounds like there's something else going on.
> So, basically, is the communication between the Java Runtime and the
> exec'd process opening so many ports/pipes between them? Im sure
> there will be the basic input/output/error between the two programs.
> Or, is the process not ending, and waitFor() not working, that
> several processes are being exec'd at the same time?
If java opens pipes between the parent and child process, it should
close them when the child exits.
Have you tried running this under truss, or whatever system call
tracer your OS has? You can probably see what's happening that
way. You probably want to put some bracketing output lines to find the
right places though, since you'll likely get a ton of output.
Joe
Roedy Green
quoted :
>
>Using them has nothing to do with it - they're opened by exec() when
>the child process is started.
I think you have discovered an other gotcha.
Sumedh
> On 10 Sep 2003 23:47:08 +0200, Gordon Beaton <n...@for.email> wrote or
> quoted :
>
> >
> >Using them has nothing to do with it - they're opened by exec() when
> >the child process is started.
>
> I think you have discovered an other gotcha.
Ok, so Gordon was right. I needed to close the stdin/stdout/stderr
pipes opened between Java and the exec'd process. Apparently Java
(even with all its garbage collection) does not close those fd's even
after the process ends. That was causing the runaway fd's.
I explicitly closed the streams, and it works!
Thanks all for those who replied!!
Sumedh
Steven Coco
> Ok, so Gordon was right. I needed to close the stdin/stdout/stderr
> pipes opened between Java and the exec'd process. Apparently Java
> (even with all its garbage collection) does not close those fd's even
> after the process ends. That was causing the runaway fd's.
>
> I explicitly closed the streams, and it works!
>
> Thanks all for those who replied!!
> Sumedh
Can you post the core lines of your code--including the added close
commands?
--
. Steven Coco .
........................................................................
When you're not sure; "Confess your heart" says the Lord, "and you'll be
freed."
Steven Coco
> I don't know what process.waitfor() is supposed to do, but when you
> spawn a child process, all its descriptors die with it. Unless Java is
> doing something weird, it sounds like there's something else going on.
That's what I was thinking.
Gordon Beaton
> Ok, so Gordon was right. I needed to close the stdin/stdout/stderr
> pipes opened between Java and the exec'd process. Apparently Java
> (even with all its garbage collection) does not close those fd's
> even after the process ends. That was causing the runaway fd's.
Garbage collection will *eventually* collect those streams, and the
descriptors will be closed when the appropriate finalizers run.
But gc is not triggered by lack of non-memory resources like file
descriptors, so when you invoke exec() in a loop you will use them up
long before gc gets involved.
The JVM can't close the descriptors automatically when the child
process terminates because there might be "grandchildren" that are
still active. The application itself needs to close them, and it
should probably wait for EOF before doing so.