Argument list too long?

[Tuxedo]

Hello,

In trying to change permissions of a long list of sub-directories the
argument lits becomes too long for the operating system or for ls to handle:

ls -d /path/to/many/directories/*/ | xargs chmod 777

How can this be done in another way?

Many thanks for any tips.

Tuxedo

[Teemu Likonen]

tux...@mailinator.com [2017-06-09 08:36:47+02] wrote:

> In trying to change permissions of a long list of sub-directories the
> argument lits becomes too long for the operating system or for ls to
> handle:
>
> ls -d /path/to/many/directories/*/ | xargs chmod 777
>
> How can this be done in another way?

You can use "find -exec":

find /path/to/many/directories -mindepth 1 -maxdepth 1 -type d \
-exec chmod 777 -- {} +

Or use "find" with "xargs":

find /path/to/many/directories -mindepth 1 -maxdepth 1 -type d \
-print0 | xargs -0 -- chmod 777 --

--
/// Teemu Likonen - .-.. <https://keybase.io/tlikonen> //
// PGP: 4E10 55DC 84E9 DFF6 13D7 8557 719D 69D3 2453 9450 ///

[Tuxedo]

Teemu Likonen wrote:

> tux...@mailinator.com [2017-06-09 08:36:47+02] wrote:
>
>> In trying to change permissions of a long list of sub-directories the
>> argument lits becomes too long for the operating system or for ls to
>> handle:
>>
>> ls -d /path/to/many/directories/*/ | xargs chmod 777
>>
>> How can this be done in another way?
>
> You can use "find -exec":
>
> find /path/to/many/directories -mindepth 1 -maxdepth 1 -type d \
> -exec chmod 777 -- {} +
>
> Or use "find" with "xargs":
>
> find /path/to/many/directories -mindepth 1 -maxdepth 1 -type d \
> -print0 | xargs -0 -- chmod 777 --
>

Thanks, both work very well!

Tuxedo

[Kaz Kylheku]

On 2017-06-09, Tuxedo <tux...@mailinator.com> wrote:
> Hello,
>
> In trying to change permissions of a long list of sub-directories the
> argument lits becomes too long for the operating system or for ls to handle:
>
> ls -d /path/to/many/directories/*/ | xargs chmod 777
>
> How can this be done in another way?

for dir in /path/to/many/dirs/*/ ; do
[ -h "$dir" ] || chmod 777 "$dir"
done

I'm assuming your ls -d trick is to avoid treating symbolic links.

[Ben Bacarisse]

If the list is too long for 'ls', will 'for' be certain to work due to
some internal command trickery? (I tried to test this but gave up
trying to make an argument list that's too long.)

--
Ben.

[Janis Papanagnou]

Yes, for 'for' (or similar shell's own syntactical constructs) it should
work.

Janis

>

[Ben Bacarisse]

Thanks. Is there some standard wording that gives this assurance?

--
Ben.

[Chris F.A. Johnson]

chmod is an external command and its arguments are limited by the OS.

for is a shell keyword and the number of arguments is limited only by
the available memory.


--
Chris F.A. Johnson

[Janis Papanagnou]

Not sure whether it's stated explicitly in the standard. But if you consider
that the usual "line length limitation" is due to the limited fixed size of
the exec buffer when external programs are called, and that there's no such
buffer involved if the shell expands a wildcard there's probably no need for
an explicit statement in the standard.

Janis

[Casper H.S. Dik]

Yes, it should work because the shell will implement the globbing
internally and never passes the list of files as an list of arguments
to an external comment; that is the limit "Argument too long" you are
running in.

The "-h" is not actually needed; the final "/" makes sure that you only
match actual directories.

Casper

[Andy Walker]

On 09/06/17 07:36, Tuxedo wrote:
> In trying to change permissions of a long list of sub-directories the
> argument lits becomes too long for the operating system or for ls to handle:
> ls -d /path/to/many/directories/*/ | xargs chmod 777

I note the other replies, but wonder if we're overthinking this?
Does the simple

cd /path/to/many/directories
ls -d */ | xargs chmod 777

shorten the lists enough for the OP's needs?

[Or

for i in /path/to/directory_A /path/to/directory_B /path/to/directory_C
do cd $i; ls -d */ | xargs chmod 777; done

if there are several higher-level directories.]

--
Andy Walker,
Nottingham.

[Janis Papanagnou]

On 10.06.2017 13:18, Andy Walker wrote:
> On 09/06/17 07:36, Tuxedo wrote:
>> In trying to change permissions of a long list of sub-directories the
>> argument lits becomes too long for the operating system or for ls to handle:
>> ls -d /path/to/many/directories/*/ | xargs chmod 777
>
> I note the other replies, but wonder if we're overthinking this?
> Does the simple
>
> cd /path/to/many/directories
> ls -d */ | xargs chmod 777
>
> shorten the lists enough for the OP's needs?

Maybe, maybe not. The point is we don't know, and even if it works in the
current case it only postpones the inherent issue until the list exceeds
the possible length again. "ls *" is worse than "find", here.

Janis

[Kaz Kylheku]

On 2017-06-10, Casper H.S Dik <Caspe...@OrSPaMcle.COM> wrote:
> Ben Bacarisse <ben.u...@bsb.me.uk> writes:
>
>>Kaz Kylheku <686-67...@kylheku.com> writes:
>>> for dir in /path/to/many/dirs/*/ ; do
>>> [ -h "$dir" ] || chmod 777 "$dir"
>>> done

> The "-h" is not actually needed; the final "/" makes sure that you only
> match actual directories.

It doesn't. Before posting I tested on a symlink. Specifically,
a symlink with a directory target.

$ ln -sf /etc etc
$ echo e*/
empty/ etc/

Arguably, it *is* matching an actual directory; the behavior can
be justified as The Right Thing (tm).

It might be otherwise in your shell, and I would be irked to see it
occur with a non-directory symlink.

[Helmut Waitzmann]

Andy Walker <a...@cuboid.co.uk>:

> On 09/06/17 07:36, Tuxedo wrote:
>> In trying to change permissions of a long list of sub-directories the
>> argument lits becomes too long for the operating system or for ls to handle:
>> ls -d /path/to/many/directories/*/ | xargs chmod 777
>
> I note the other replies, but wonder if we're overthinking this?
> Does the simple
>
> cd /path/to/many/directories
> ls -d */ | xargs chmod 777
>
> shorten the lists enough for the OP's needs?

Yes, it does. “xargs” (quoting
<http://pubs.opengroup.org/onlinepubs/9699919799/utilities/xargs.html>)
takes care of the limits:

The xargs utility shall construct a command line consisting of
the utility and argument operands specified followed by as many
arguments read in sequence from standard input as fit in length
and number constraints specified by the options. The xargs
utility shall then invoke the constructed command line and wait
for its completion. This sequence shall be repeated until one of
the following occurs:


*

An end-of-file condition is detected on standard input.

*

An argument consisting of just the logical end-of-file
string (see the -E eofstr option) is found on standard
input after double-quote processing, <apostrophe>
processing, and <backslash>-escape processing (see next
paragraph). All arguments up to but not including the
argument consisting of just the logical end-of-file string
shall be used as arguments in constructed command lines.

*

An invocation of a constructed command line returns an
exit status of 255.

[Teemu Likonen]

Helmut Waitzmann [2017-06-10 22:15:47+02] wrote:

> Andy Walker <a...@cuboid.co.uk>:

>> Does the simple
>>
>> cd /path/to/many/directories
>> ls -d */ | xargs chmod 777
>>
>> shorten the lists enough for the OP's needs?
>
> Yes, it does. “xargs” (quoting
> <http://pubs.opengroup.org/onlinepubs/9699919799/utilities/xargs.html>)
> takes care of the limits:

xargs takes care of the argument list for chmod but not for ls. That is
why find command may be needed (or a shell construct without argument
list limitations).

[Stephane Chazelas]

2017-06-09 17:40:22 -0400, Chris F.A. Johnson:
[...]

> chmod is an external command

[...]

Though zsh (after zmodload zsh/files) and ksh93 (after
PATH=/opt/ast/bin:$PATH) also have a builtin version of it which
can be used to overcome that limitation of the execve() system
call.

See also zargs in zsh and command -x in ksh93 for alternatives
to xargs.

--
Stephane

[Casper H.S. Dik]

Kaz Kylheku <686-67...@kylheku.com> writes:

>It might be otherwise in your shell, and I would be irked to see it
>occur with a non-directory symlink.

It is a OS behaviour; there have been some issues with certain
versions of Solaris (a poor implementation of the "trailing /
elimination"). It was fixed in Solaris 10 (before it was released)

Casper

[Janis Papanagnou]

On 12.06.2017 09:36, Stephane Chazelas wrote:
> 2017-06-09 17:40:22 -0400, Chris F.A. Johnson:
> [...]
>> chmod is an external command
> [...]
>
> Though zsh (after zmodload zsh/files) and ksh93 (after
> PATH=/opt/ast/bin:$PATH) also have a builtin version of it which
> can be used to overcome that limitation of the execve() system
> call.

Is that /opt/ast/bin a logical component that triggers ksh to behave
so, or is it necessary that /opt/ast/bin exists physically and maybe
also have the respective (dynamic loadable?) files in there?

Janis

> [...]

[Stephane Chazelas]

2017-06-10 14:27:26 +0000, Kaz Kylheku:

While the expansion of */ will include all the files that can be
proven to be directories after symlink resolution (use *(/) in
zsh for actual directories only), [ -h link/ ] will always
return false (in that regard, link/ is like link/.)

There's also the problem of the case where */ doesn't match any
file. In that case many shell will leave the glob there
unexpanded. And if there happens to be a "*" file that is
neither a directory nor a symlink, then [ -h ] will also return
false on it (chmod is likely (but not guaranteed as I've seen
the / being ignored on some systems IIRC) to fail on it because
of the trailing / though).

The first problem can be worked around by doing

[ -h "${dir%/}" ]

But that would not address the second problem, so one might as
well do:

for file in *; do
[ -d "$file" ] || continue
[ -L "$file" ] && continue
chmod -- 775 "$file"
done

--
Stephane

[Stephane Chazelas]

2017-06-12 10:10:08 +0200, Janis Papanagnou:

[...]

No, it doesn't have to exist. You can also do "command
/opt/ast/bin/chmod...". The idea is that ksh would not shadow
non-AST commands with its own builtin version (as the builtin is
likely to behave differently from the one on the filesystem), so
you have to tell you want the ast variants for ksh to be willing
to optimize by using the built-in version (at least for those
commands like chmod that are not traditionally implemented as
built-ins).

Note the set of supported builtins depends on how ksh/ast-open
was configured at build time.

--
Stephane

[Casper H.S. Dik]

Stephane Chazelas <stephane...@gmail.com> writes:

>No, it doesn't have to exist. You can also do "command
>/opt/ast/bin/chmod...". The idea is that ksh would not shadow
>non-AST commands with its own builtin version (as the builtin is
>likely to behave differently from the one on the filesystem), so
>you have to tell you want the ast variants for ksh to be willing
>to optimize by using the built-in version (at least for those
>commands like chmod that are not traditionally implemented as
>built-ins).

>Note the set of supported builtins depends on how ksh/ast-open
>was configured at build time.

This is easily checked running truss/strace on ksh and set
the PATH properly.

Casper

[Kaz Kylheku]

On 2017-06-12, Stephane Chazelas <stephane...@gmail.com> wrote:
> 2017-06-10 14:27:26 +0000, Kaz Kylheku:

> The first problem can be worked around by doing
>
> [ -h "${dir%/}" ]

I'm looking at this on a GNU/Linux system.

It looks like this is actually a behavior of the lstat system call.

With the trailing slash, lstat doesn't see the symlink, but resolves
through it to the directory.

POSIX seems to require it, according to Pathname Resolution.

In the IEEE Std 1003.1-2008, 2016 Edition, it says in 4.13 Pathname
Resolution:

If a symbolic link is encountered during pathname resolution, the
behavior shall depend on whether the pathname component is at the end of
the pathname and on the function being performed. If all of the
following are true, then pathname resolution is complete:

1. This is the last pathname component of the pathname.

2. The pathname has no trailing <slash>.

3. The function is required to act on the symbolic link itself,
or certain arguments direct that the function act on the
symbolic link itself.

(3) makes it clear that this whole requirement is precisely for
funtions such as lstat which act on the symlink. For the sake of
those functions, pathname resolution leaves the last component
unresolved if it is a symlink --- but only if there is no trailing
slash.

[Janis Papanagnou]

Also typing the builtin command 'builtin' shows (besides some regular
builtins) also a couple of builtins with the /opt/ast/bin prefix, on
my system.

Janis

>
> Casper
>

[Helmut Waitzmann]

Teemu Likonen <tlik...@iki.fi>:

> Helmut Waitzmann [2017-06-10 22:15:47+02] wrote:
>
>> Andy Walker <a...@cuboid.co.uk>:
>>> Does the simple
>>>
>>> cd /path/to/many/directories
>>> ls -d */ | xargs chmod 777
>>>
>>> shorten the lists enough for the OP's needs?
>>
>> Yes, it does. “xargs” (quoting
>> <http://pubs.opengroup.org/onlinepubs/9699919799/utilities/xargs.html>)
>> takes care of the limits:
>
> xargs takes care of the argument list for chmod but not for ls. That is
> why find command may be needed (or a shell construct without argument
> list limitations).

Yes, of course. You are completely right. Thank you for the
correction.

[Hongyi Zhao]

On Sat, 10 Jun 2017 23:22:51 +0300, Teemu Likonen wrote:

> xargs takes care of the argument list for chmod but not for ls.

How do you know this?

In general, which tools/utilities will be cared by xargs, and which won't?

Regards

> That is
> why find command may be needed (or a shell construct without argument
> list limitations).





--

.: Hongyi Zhao [ hongyi.zhao AT gmail.com ] Free as in Freedom :.

[Helmut Waitzmann]

Hongyi Zhao <hongy...@gmail.com>:

> On Sat, 10 Jun 2017 23:22:51 +0300, Teemu Likonen wrote:
>
>> xargs takes care of the argument list for chmod but not for ls.
>
> How do you know this?
>
> In general, which tools/utilities will be cared by xargs, and which won't?

If you hadn't stripped Andy Walker's question and my wrong answer,
then the context of Teemu Likonen's followup would have been
clear.

“xargs” takes care of the argument lists for all processes it
starts. So, if you had a command fragment like

“| xargs -- ls -d --”

“xargs” would take care of the argument list for “ls”. But of
course, “xargs” can do nothing regarding the argument list of the
process at the left side of the pipe (“|”) in the following
command:

“ls -d -- */ | xargs chmod 777”.

And Teemu Likonen's suggestion, to use “find” or a shell construct
without argument list limitations to circumvent this limitation is
correct.

In the command

“find -L . \( ! -path './*' -o -prune \) \
\( ! \( ! -name '.*' -type d \) -o \
\( -exec chmod -- 777 '{}' + \) \)”,

“find” would take care of the argument list limit for the “chmod”
process.

In the command

“(
for dir in */
do printf '%s\n' "$dir"
done
) | xargs -E '' -- chmod -- 777”,

“xargs” would take care of the argument list limit for the “chmod”
process.