bash environmental variable size limits?
David Mathog
It seems that what happens is that once enough storage is consumed by a
single environmental variable bash starts to misbehave. I am curious
how one would determine this size, other than experimentally.
Example (experimental determination):
# export FOO=1234567890
# export FOO=$FOO$FOO ; echo $FOO | wc
repeat the previous line until it does:
bash: /usr/bin/wc: Argument list too long
bash: echo: write error: Broken pipe
# echo $FOO >killme
# ls -al killme
bash: /bin/ls: Argument list too long
note that bash is screwing up now even though that variable is
not involved. Fix bash by removing the variable.
# unset FOO
# ls -al killme
-rw-r--r-- 1 root root 163841 2009-08-07 09:34 killme
The size at which the "Argument list too long" appeared was 1310721 on
an old Solaris 8 system (bash 2.03.0(1))but only 163841 on two Mandriva
2008.1 systems(32 bit, bash 3.2.19(2)). "set" doesn't expose any
variables with obviously related numeric values, and "ulimit" doesn't
either, for instance on the linux system:
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 16382
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 16382
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I was able to set 4 different variables to size 81921 on the Mandriva
system without any issues, cumulatively much more storage than the
trigger point for a single variable, which suggests that the size of the
largest single variable is the issue.
Explanation?
Thanks,
David Mathog
André Gillibert
> What determines the size limit for environmental variables in bash?
>
For example, Linux had a MAX_ARG_PAGES limit, chosen at kernel compile time, on the total arg+env space, which has been removed in version 2.6.23 IIRC.
Practically, it means that older kernel have low limits such as 128K of arg+env space.
> It seems that what happens is that once enough storage is consumed by a
> single environmental variable bash starts to misbehave. I am curious
> how one would determine this size, other than experimentally.
>
This may be documented for your own system.
On Linux systems <=2.6.22, I guess you should look at MAX_ARG_PAGES in /usr/src/linux/include/linux/binfmts.h.
> Example (experimental determination):
>
> # export FOO=1234567890
> # export FOO=$FOO$FOO ; echo $FOO | wc
>
> repeat the previous line until it does:
>
> bash: /usr/bin/wc: Argument list too long
> bash: echo: write error: Broken pipe
> # echo $FOO >killme
> # ls -al killme
> bash: /bin/ls: Argument list too long
>
> note that bash is screwing up now even though that variable is
> not involved. Fix bash by removing the variable.
>
> # unset FOO
> # ls -al killme
> -rw-r--r-- 1 root root 163841 2009-08-07 09:34 killme
>
> The size at which the "Argument list too long" appeared was 1310721 on
> an old Solaris 8 system (bash 2.03.0(1))but only 163841 on two Mandriva
> 2008.1 systems(32 bit, bash 3.2.19(2)). "set" doesn't expose any
> variables with obviously related numeric values, and "ulimit" doesn't
> either, for instance on the linux system:
>
It looks like Mandriva 2008 has a 2.6.22 kernel, hence, it's limited.
> I was able to set 4 different variables to size 81921 on the Mandriva
> system without any issues, cumulatively much more storage than the
> trigger point for a single variable, which suggests that the size of the
> largest single variable is the issue.
I don't know about any such limitation. It might be due to the way bash allocate new env variables with, maybe, at some time, the old env var + the new env var allocated at once. I don't know.
You'd rather test with a simple C program, in order to eliminate bash-specific behavior.
--
André Gillibert.
Gordon Burditt
>>
>It depends on the operating system.
>For example, Linux had a MAX_ARG_PAGES limit, chosen at kernel compile
>time, on the total arg+env space, which has been removed in version
>2.6.23 IIRC.
>Practically, it means that older kernel have low limits such as 128K of
>arg+env space.
*REALLY* old kernels, such as UNIX v7, had some *really* low limits
on argument space, such as under 8K.
On FreeBSD, this limit is available with "sysctl kern.argmax". I
think it's in bytes (default 256*1024). You can't change it with
sysctl. Programs that need the value (like xargs(1)) are encouraged
to get it with sysconf(_SC_ARG_MAX) . This should be available on
POSIX systems.
I'm not sure how the usage to match against the limit is calculated:
I suspect it includes the strings, nul terminators for the strings,
and the argv[] and envp[] vectors.
Sven Mascheck
> Andr� Gillibert wrote:
>> David Mathog wrote
>>> What determines the size limit for environmental variables in bash?
>> It depends on the operating system.
...and it's only the exec() syscalls (not the shell itself),
which hit this limit.
>> For example, Linux had a MAX_ARG_PAGES limit,
On most systems, including Linux, the value in question is ARG_MAX,
or _SC_ARG_MAX especially for sysconf(). Often it's defined to
some other, more system specific values then.
>> [...] which has been removed in version 2.6.23 IIRC.
Yes, thus it's current Linux and Hurd who have no such limit, AFAIK.
However, you still might get a limit reported, either because the libc
has't caaught up, or to avoid confusing programs which wrongly expect
a finite value.
However, current Linux still has a maximum length for an argument,
MAX_ARG_STRLEN, which applies to both argv[] and envp[]. It's 128KB,
like the former ARG_MAX. This is also relevant for David's tests.
> *REALLY* old kernels, such as UNIX v7, had some *really* low limits
> on argument space, such as under 8K.
Yes, it was 5120 bytes and named NCARGS that time.
And yet older kernels, like V6 had 512 bytes as limit.
> On FreeBSD, this limit is available with "sysctl kern.argmax".
This method was introduced by the successor 4.4BSD. A widespread
user command nowadays (also adopted by the free BSDs) is
$ getconf ARG_MAX
> You can't change it with sysctl.
...although a very few systems allow to change this value.
But that's not widespread because increasing the limit usually
is not the solution.
> Programs that need the value (like xargs(1)) are encouraged
> to get it with sysconf(_SC_ARG_MAX) .
Apropos xargs: find is another tool which can make good use
of this ("-exec +"), and it obsoletes xargs then.
> I'm not sure how the usage to match against the limit is calculated:
> I suspect it includes the strings, nul terminators for the strings,
> and the argv[] and envp[] vectors.
...yes, so is the user command estimation I know:
$ expr `getconf ARG_MAX` - `env|wc -c` - `env|wc -l` \* 4
POSIX suggests to subtract another 2K for safety, for processes
modifying their own environment.
If you have to be certain, you could just exec() with increasing
length of arguments to find the actual limit.
--
http://www.in-ulm.de/~mascheck/various/argmax/
Sven Mascheck
> Andr� Gillibert wrote:
>> David Mathog wrote
>>> What determines the size limit for environmental variables in bash?
>> It depends on the operating system.
...and it's only the exec() syscalls (not the shell itself),
which hit this limit.
>> For example, Linux had a MAX_ARG_PAGES limit,
On most systems, including Linux, the value in question is ARG_MAX,
or _SC_ARG_MAX especially for sysconf(). Often it's defined to
some other, more system specific values then.
>> [...] which has been removed in version 2.6.23 IIRC.
Yes, thus it's current Linux and Hurd who have no such limit, AFAIK.
However, you still might get a limit reported, either because the libc
has't caaught up, or to avoid confusing programs which wrongly expect
a finite value.
However, current Linux still has a maximum length for an argument,
MAX_ARG_STRLEN, which applies to both argv[] and envp[]. It's 128KB,
like the former ARG_MAX. This is also relevant for David's tests.
> *REALLY* old kernels, such as UNIX v7, had some *really* low limits
> on argument space, such as under 8K.
Yes, it was 5120 bytes and named NCARGS that time.
And yet older kernels, like V6 had 512 bytes as limit.
> On FreeBSD, this limit is available with "sysctl kern.argmax".
This method was introduced by the ancestor 4.4BSD. A widespread
user command nowadays (also adopted by the free BSDs) is
$ getconf ARG_MAX
> You can't change it with sysctl.
...although a very few systems allow to change this value.
But that's not widespread because increasing the limit usually
is not the solution.
> Programs that need the value (like xargs(1)) are encouraged
> to get it with sysconf(_SC_ARG_MAX) .
Apropos xargs: find is another tool which can make good use
of this ("-exec +"), and it obsoletes xargs then.
> I'm not sure how the usage to match against the limit is calculated:
> I suspect it includes the strings, nul terminators for the strings,
> and the argv[] and envp[] vectors.
...yes, so is the user command estimation I know:
$ expr `getconf ARG_MAX` - `env|wc -c` - `env|wc -l` \* 4
POSIX suggests to subtract another 2K for safety, for processes
modifying their own environment.
If you have to be certain, you could just exec() with increasing
length of arguments to find the actual limit.
[typoe-supersede: successor->ancestor]
--
http://www.in-ulm.de/~mascheck/various/argmax/