Problems with buffer size in ksh after patches
Ilya
I have a number of HP-UX 11.0 systems that got patched with one
of the latest bundles, - all 30 days old or older.
The machine rebooted fine and all appeared normal, except that a
few days later one of the scripts stopped working when it was
executed. The script was in the format of
$variable = `
...
...
...
`
meaning, a long command probably containing more than 15 lines of
stuff. It is some kind of a DBA script and was brought to my attention
by DBAs. The short term fix was to remove some of the comments and
get it down to 1023 bytes in size, where it did function properly.
What I would like to know is:
What is the maximum POSIX compliant buffer size, i.e. command length?
I recall at one point hearing that 255 was the maximum size but
don't recall what operating system it was on.
Why was this triggered by patches and what particular patch could have
caused this? I can post the total list of patches installed on the
system.
What are the solutions? Even if the script incorrectly exceeded the
limit, I might not be in a position to state that and shift the blame
in that direction, because from the users perspective, the system
was running fine, then it got patched, and the scripts stoppped
working. Even if there is a bad programming practice involved,
which is what I suspect contributes to the problem. There are many
machines and possibly other scripts that exceed the limit, which
further complicates the problem.
Is there a kernel variable that can up the buffer space beyond 1024
bytes?
Is there a patch that addresses this?
All responses are greatly appreciated.
Sent via Deja.com http://www.deja.com/
Before you buy.
brian hiles
The compile-time parameter you are looking for is usually defined
in /usr/include/limits.h as "_POSIX_ARG_MAX", which at least on my
FreeBSD ISP host is 4096. On pre-POSIX machines, it is defined as
_SC_ARG_MAX. I believe 4096 bytes is the _minimum_ definable through
the POSIX specification; there is no maximum and frequently I see
systems with this value set to the 100Kb to 1Mb range.
grep _ARG_MAX /usr/include/*.h
_POSIX_ARG_MAX, as you are aware now, determines the maximum number
of characters of command lines and command substitutions, etc.
Increasing this parameter should obviate the limitation that is
being experienced with the aforementioned script.
I do not have specific experience with HPUX 11.x; re-making the
kernel is usually accomplished with a make(1) invocation or bourne
script; however, every Unix port is slightly different -- indeed
systems like Linux and Solaris have the capability of tuning kernel
parameters without shutdown or rebuilding binaries.
Hope this helps.
-Brian
Ilya
In article <suuolbq...@corp.supernews.com>,>
Thank you for the response. I set _POSIX_ARG_MAX to a much higher
value, 262144, which is 4096 * 64, ran mk_kernel, put the kernel
in /stand and rebooted the machine, tried the script again and I am
still getting the same error message:
/tmp/buffer.1[41]: /var/tmp/sh2051.1: Cannot find or open the file.
So it didn't have any effect. Any further suggestions? I am going
to put the original value of 4096 back.
BTW, I have identified the offending patch, which is PHCO_20816.
I am not going to install it on any further machines, when I remove
it and run my script, it runs without the above error.
> The compile-time parameter you are looking for is usually defined
> in /usr/include/limits.h as "_POSIX_ARG_MAX", which at least on my
> FreeBSD ISP host is 4096. On pre-POSIX machines, it is defined as
> _SC_ARG_MAX. I believe 4096 bytes is the _minimum_ definable through
> the POSIX specification; there is no maximum and frequently I see
> systems with this value set to the 100Kb to 1Mb range.
>
> grep _ARG_MAX /usr/include/*.h
>
> _POSIX_ARG_MAX, as you are aware now, determines the maximum number
> of characters of command lines and command substitutions, etc.
> Increasing this parameter should obviate the limitation that is
> being experienced with the aforementioned script.
>
> I do not have specific experience with HPUX 11.x; re-making the
> kernel is usually accomplished with a make(1) invocation or bourne
> script; however, every Unix port is slightly different -- indeed
> systems like Linux and Solaris have the capability of tuning kernel
> parameters without shutdown or rebuilding binaries.
>
> Hope this helps.
>
> -Brian
>
Geoff Clare
>The compile-time parameter you are looking for is usually defined
>in /usr/include/limits.h as "_POSIX_ARG_MAX", which at least on my
>FreeBSD ISP host is 4096. On pre-POSIX machines, it is defined as
>_SC_ARG_MAX. I believe 4096 bytes is the _minimum_ definable through
>the POSIX specification; there is no maximum and frequently I see
>systems with this value set to the 100Kb to 1Mb range.
Ugh. Lots of misinformation and confusion in the above.
POSIX.1 defines the following symbols:
ARG_MAX (limits.h)
Maximum total size of argument and environment strings that
can be passed to exec*(). Only defined in limits.h if
the limit has a fixed value on the implementation.
If the value can vary, then ARG_MAX is not defined in
limits.h, and applications must use sysconf() to query
the value at runtime instead.
_SC_ARG_MAX (unistd.h)
Used as an argument to sysconf() to find the "ARG_MAX" limit
at runtime. If sysconf(_SC_ARG_MAX) returns -1 without
setting errno, then it means there is no ARG_MAX limit
(i.e. it is only limited by available memory, etc.)
_POSIX_ARG_MAX (limits.h)
Minimum allowed value for ARG_MAX in limits.h and for
the return value of sysconf(_SC_ARG_MAX).
Always has the value 4096 (on compliant systems).
Before POSIX the various UNIX flavours used different symbol names
instead of ARG_MAX, but _SC_ARG_MAX definitely wasn't one of them.
(All of the _SC_* symbols, and the sysconf() function, were invented
by POSIX.)
--
Geoff Clare g...@unisoft.com
UniSoft Limited, London, England. g...@root.co.uk
brian hiles
> value, 262144, which is 4096 * 64, ran mk_kernel, put the kernel
> in /stand and rebooted the machine, tried the script again and I am
> still getting the same error message:
> /tmp/buffer.1[41]: /var/tmp/sh2051.1: Cannot find or open the file.
Files of the form $TMPDIR/shNNNN.N are created to echo here-files
defined in the running scripts. Error messages of the lack of same
I would think have a reason other than that of command substitution
or maximum argv lengths.
> So it didn't have any effect. Any further suggestions? I am going
> to put the original value of 4096 back.
Nope, I'm out. Time to talk to a HP/UX guru.
> BTW, I have identified the offending patch, which is PHCO_20816.
> I am not going to install it on any further machines, when I remove
> it and run my script, it runs without the above error.
This reinforces my above hypothesis.
To Geoff Clare:
Sorry for the misinformation and thanks for the clarification;
however, that information I did provide is composited verbatim from
one source, from memory on another, and from guess-work -- but
admitted as such.
As for POSIX itself, a lot of this confusion would be obviated by
these standards being Web accessible. I just don't understand what
a standards body could possibly be thinking by asking $100+ for
each 1003.x text -- shouldn't the goal of a standard intrinsically
be its dissemination?
-Brian
Ken Pizzini
brian hiles <b...@rainey.blueneptune.com> wrote:
>As for POSIX itself, a lot of this confusion would be obviated by
>these standards being Web accessible. I just don't understand what
>a standards body could possibly be thinking by asking $100+ for
>each 1003.x text -- shouldn't the goal of a standard intrinsically
>be its dissemination?
The standards bodies have a funding model based on the selling
of published copies of their product; this may make some sense
in industries such as power distribution systems, where the
players involved are medium-to-large corporations which can
afford the fees involved and where non-compliance with the
standard is not really a viable option.
But I completely agree that this model needs revision in the
information processing industry, where many important players
are individuals and compliance with the standards is a good
thing for the larger community but fairly optional for the
individual programmer.
--Ken Pizzini
Geoff Clare
>As for POSIX itself, a lot of this confusion would be obviated by
>these standards being Web accessible. I just don't understand what
>a standards body could possibly be thinking by asking $100+ for
>each 1003.x text -- shouldn't the goal of a standard intrinsically
>be its dissemination?
Ken has explained why the IEEE and ISO charge what they do for
printed standards. Some recent standards (e.g. C99) are also available
for download as PDF for a much reduced price over the printed version.
The current Single Unix Specification *is* available free on the web.
(See http://www.opengroup.org/pubs/catalog/web.htm).
Since this is a superset of POSIX, you can use it to find out
information about POSIX, as long as you don't assume that everything
in there is required by POSIX.
brian hiles
> brian hiles <b...@rainey.blueneptune.com> writes:
>>each 1003.x text -- shouldn't the goal of a standard intrinsically
>>be its dissemination?
> (See http://www.opengroup.org/pubs/catalog/web.htm).
(To Markus Gyger <mgy...@ite.gmu.edu> and Geoff Clare:)
I am aware of the XPG "Open Group" proposed standard, which I knew
to be either a superset or a subset of the POSIX 1003.x (and others?)
standards. The one that I have pored over many a time is, and have
known about since these documents were on digital.com's (?) web site is:
http://www.opengroup.org/onlinepubs/7908799/xcu/shellix.html
However, to answer a question and to edify myself a year plus ago I
determined to find out the web site of POSIX, which is surprisingly
difficult information to find!!
http://www.pasc.org/abstracts/posix.htm
(POSIX distributes their documents through IEEE, which requires IIRC
that one be a paying member to purchase documents, which I am not.)
-Brian