Command failure: Too Many Files (rm, ls, tar)
Browder, Tom
I regularly copy the files to a transfer directory, cd to that
directory, and tar them and gzip them (tar cvzf xfer.tgz *) to take them
to another site.
Last week I started getting errors indicating there were too many files
for the command (rm *, ls *, tar cvzf xfer.tgz *) to continue.
I was able to get the commands to work by reducing the file count down
to about 1900. Is there a magic number limit on those commands? Or am
I doing something wrong. I'm currently running linux on an Intel PC
with a Fedora 7 distribution.
Thanks.
-Tom
Henrik Carlqvist
> Last week I started getting errors indicating there were too many files
> for the command (rm *, ls *, tar cvzf xfer.tgz *) to continue.
Some quick workarounds:
find . -type f -maxdepth 1 -name "*" -exec rm {} \;
The above will only remove files at your current directory, any files in
directories below will not be removed. If you have no subdirectorires it
will be more simple to type:
find . -exec rm {} \;
With that simple command you will get an error message about being unable
to remove . (current directory) and .. (directory above) and this command
will unlike "rm *" also remove files which start with a dot.
The command
find . -maxdepth 2 -name "*"
will give you approximately the same result as "ls *".
tar cvzf xfer.tgz .
might be more useful that using wildcard.
regards Henrik
--
The address in the header is only to prevent spam. My real address is:
hc3(at)poolhem.se Examples of addresses which go to spammers:
root@localhost postmaster@localhost
Bob Proulx
> Last week I started getting errors indicating there were too many files
> for the command (rm *, ls *, tar cvzf xfer.tgz *) to continue.
> Is there a magic number limit on those commands?
> Or am I doing something wrong.
You are running into the kernel's ARG_MAX limitation. Please see this
reference for more information.
http://www.gnu.org/software/coreutils/faq/#Argument-list-too-long
> I regularly copy the files to a transfer directory, cd to that
> directory, and tar them and gzip them (tar cvzf xfer.tgz *) to take
> them to another site.
For your 'tar' case you can avoid the argument expansion entirely
simply by giving '.' as the argument.
tar cvzf xfer.tgz .
For 'rm *' I would normally either simply remove the entire directory
with 'rm -rf ./dir' or use 'find' like this:
find . -exec rm {} +
For 'ls *' there is no need of the '*'. Just use ls. Actually using
'ls *' is inefficient because the shell expands the * to match every
filename in the directory and ls then lists it. But the entire
purpose of ls is to list directories. So 'ls *' does the same work
twice while 'ls' does it once. Better to simply use 'ls'.
ls
Bob
Browder, Tom
have an arbitrary limit.
And your examples and pointers are just what I need.
-Tom
Ralf Wildenhues
* Bob Proulx wrote on Tue, Apr 15, 2008 at 06:55:35PM CEST:
> You are running into the kernel's ARG_MAX limitation. Please see this
> reference for more information.
>
> http://www.gnu.org/software/coreutils/faq/#Argument-list-too-long
AFAIK this limitation has been lifted in Linux 2.6.23. That FAQ could
be updated for this. On my 2.6.24, the non-limit isn't shown:
$ getconf ARG_MAX
131072
but apparently it doesn't exist any more:
$ cd /usr/lib; /bin/echo * * * * * * * * * * * * * * * * * * * * * * * * * | wc
1 60950 1056000
Cheers,
Ralf
Karl Berry
In these specific cases Bob showed ways to avoid passing a zillion
arguments to the commands. But sometimes it is not possible to avoid.
In those cases, xargs is your friend, as in
find -name \*.c | xargs ls
or whatever.
As for tar, let me also mention that it has an option (T) to read the
file list from a file.
karl
Browder, Tom
to use it now.
-Tom
Browder, Tom
>From: Bob Proulx [mailto:b...@proulx.com]
>Sent: Tuesday, April 15, 2008 11:56
>To: Browder, Tom
>Cc: help-gn...@gnu.org
>Subject: Re: Command failure: Too Many Files (rm, ls, tar)
>Browder, Tom wrote:
>> Last week I started getting errors indicating there were too many
...
>You are running into the kernel's ARG_MAX limitation. Please
>see this reference for more information.
>For your 'tar' case you can avoid the argument expansion
>entirely simply by giving '.' as the argument.
>
> tar cvzf xfer.tgz .
Hm, when I try this the tgz archive gets sucked back into itself.
-Tom
Bob Proulx
> Bob Proulx wrote:
> >For your 'tar' case you can avoid the argument expansion
> >entirely simply by giving '.' as the argument.
> >
> > tar cvzf xfer.tgz .
>
> Hm, when I try this the tgz archive gets sucked back into itself.
Oops. You are right. I typed that in too quickly. Of course it is
possible to put the tar archive elsewhere.
tar cvzf ../xfer.tgz .
tar cvzf /tmp/xfer.tgz .
Also if you are simply sync'ing two directories then I recommend using
'rsync' for this. It is a great tool and designed for just such
activity.
rsync -av ./fromhere host1.example.com:/to/there/
rsync -avz --progress ./fromhere host1.example.com:/to/there/
Bob
Bob Proulx
Ralf Wildenhues wrote:
> AFAIK this limitation has been lifted in Linux 2.6.23. That FAQ could
> be updated for this.
Do you know which distributions have shipped with this or newer
kernel? I was holding off mostly because I think it is only
from-scratch builders that have that newest kernels that will see this
new behavior.
> On my 2.6.24, the non-limit isn't shown:
>
> $ getconf ARG_MAX
> 131072
I think this is a bug in glibc on kernels without a hard limit.
(However it may be a lessor of two evils it if avoids breakage of some
programs that always expect a limit.) I wonder if libc tries to
dynamically detect this or if there is a hard coded value that needs
to be updated. Hmm...
> but apparently it doesn't exist any more:
>
> $ cd /usr/lib; /bin/echo * * * * * * * * * * * * * * * * * * * * * * * * * | wc
> 1 60950 1056000
You say "apparently" but I am sure that it was you posted a note
elsewhere noting this change in the upstream kernel. I followed your
reference to where it was committed into Linus' kernel tree here:
It is only a matter of time before this propagates. But time is also
what keeps everything from happening all at once and so we will need
to wait a bit before this shows up on most machines in the wild. :-)
Bob
Ralf Wildenhues
> Ralf Wildenhues wrote:
> > AFAIK this limitation has been lifted in Linux 2.6.23. That FAQ could
> > be updated for this.
>
> Do you know which distributions have shipped with this or newer
> kernel? I was holding off mostly because I think it is only
> from-scratch builders that have that newest kernels that will see this
> new behavior.
Mine is from Debian testing or unstable.
> > $ getconf ARG_MAX
> > 131072
>
> I think this is a bug in glibc on kernels without a hard limit.
> (However it may be a lessor of two evils it if avoids breakage of some
> programs that always expect a limit.) I wonder if libc tries to
> dynamically detect this or if there is a hard coded value that needs
> to be updated. Hmm...
Oh, I might not be running the latest glibc. And I don't consider this
a serious bug, either. But it may cause libtool to run slower than
necessary (as it believes getconf).
> > but apparently it doesn't exist any more:
> You say "apparently" but I am sure that it was you posted a note
> elsewhere noting this change in the upstream kernel.
Yep, just checking whether you were paying attention. ;-)
> It is only a matter of time before this propagates. But time is also
> what keeps everything from happening all at once and so we will need
> to wait a bit before this shows up on most machines in the wild. :-)
Sure. But you can also spread the word now, maybe that will cause it to
propagate faster!
Cheers,
Ralf
Bob Proulx
> Mine is from Debian testing or unstable.
Of course Debian Testing at this time is in the queue for release as
the upcoming stable Lenny. But it hasn't released yet. I try to
avoid advocating normal users to use unreleased bits. Developers are
on their own and everyone has their own ideas of how to manage systems
anyway and it isn't productive trying to get developers to agree. And
of course Debian unstable Sid is always development and never
releases. (I know you know this but am just saying it for the archive
for readers who come later.)
> > It is only a matter of time before this propagates. But time is also
> > what keeps everything from happening all at once and so we will need
> > to wait a bit before this shows up on most machines in the wild. :-)
>
> Sure. But you can also spread the word now, maybe that will cause it to
> propagate faster!
You have prodded me sufficiently. I added a short blurb about it at
the end of this entry:
http://www.gnu.org/software/coreutils/faq/#Argument-list-too-long
If you would like to see more there feel free to suggest something! :-)
Bob