BPXI039I and SHRLIBRGNSIZE at 100%

[Lizette Koehler]

Tonight I got a message of BPXI039I for the SHRLIBRGNSIZE being 100% for the
value of 67108864

Now it tells me to raise this value with a SETOMVS or SET OMVS. However, is
there a way to see what is consuming this resource? And how to determine
how to safely raise the value if necessary?

I want to make sure we have sufficient resources to raise it up.


Thanks


Lizette

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to list...@bama.ua.edu with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

[Mike Schwab]

http://www-01.ibm.com/support/docview.wss?uid=swg21178068

If the shared library cache is not big enough, then z/OS will revert
to normal storage use and the load module will be loaded into the
broker address space.
[That would mean multiple copies of the programs.]

http://publib.boulder.ibm.com/infocenter/cicsts/v2r3/index.jsp?topic=/com.ibm.cics.ts23.doc/dfht3/dfht3gl.htm

Tuning the z/OS shared library region

The shared library region is a z/OS feature that enables address
spaces |to share dynamic link library (DLL) files. This feature
enables your CICS |regions to share the DLLs that are needed for JVMs,
rather than each region |having to load them individually. This can
greatly reduce the amount of real |storage used by MVS, and the time
it takes for the regions to load the files.

The storage that is reserved for the shared library region is
allocated |in each CICS region when the first JVM is started in the
region. (This might |be the master JVM that initializes the shared
class cache.) The amount of |storage that is allocated is controlled
by the SHRLIBRGNSIZE parameter in |z/OS, which is in the BPXPRMxx
member of SYS1.PARMLIB. The minimum is 16MB, |and the z/OS default is
64MB. You can tune the amount of storage that is allocated |for the
shared library region by investigating how much space you actually
|need, bearing in mind that other applications besides CICS might be
using |the shared library region, and adjusting the SHRLIBRGNSIZE
parameter accordingly.

[deleted] Bring up your normal workload on the z/OS system, then issue
|the command D OMVS,L to display the library statistics.
[deleted]

On Sat, Oct 29, 2011 at 10:46 PM, Lizette Koehler
<star...@mindspring.com> wrote:
> Tonight I got a message of BPXI039I for the SHRLIBRGNSIZE being 100% for the
> value of 67108864
>
> Now it tells me to raise this value with a SETOMVS or SET OMVS.  However, is
> there a way to see what is consuming this resource?  And how to determine
> how to safely raise the value if necessary?
>
> I want to make sure we have sufficient resources to raise it up.
>
> Thanks
>
> Lizette

--

Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

[John McKown]

I'm ignorant of this. But did a search in IBMLink.

It appears that there are "shared libraries" in z/OS UNIX which are
loaded somewhere? (not documented where). The size of this area is
specified in the SHRLIBRGNSIZE parameter. The load modules loaded into
this area are .so UNIX "dynamic subroutines" which are marked as
shareable in the filesystem via the "+l" extattr attribute. This area is
them mapped into every UNIX address space into "high private". I would
guess this would eventually involve IARVSERV.
http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/IEA2A9B0/5.0
and
http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/IEA2A6A1/20.0?SHELF=IEA2BKB2&DT=20101208204043

The larger the area, the more high private is lost in every UNIX address
space which attempts to load a shared library. And the more ECSA is used
for mapping tables.

It appears that if it is too small, it is not used at all and each
shared library is loaded into the user's address space as if it were not
shared. This increases I/O.

So I guess "shared library" region size is like an LLA for UNIX shared
libraries. But I don't read of any way for "obsolete" modules to be
purged and their storage reclaimed. And there doesn't appear to be any
way to actually determine what is loaded. I would guess into some
address space or data space controlled by the UNIX kernel.

You could find all the possibilities by searching all of your mounted
filesystems for files with the extended attribute of "l" specified.

find / -ext l >~/list.of.shared.library.files

The above will likely run a fair amount of time. I'd do it in batch.

In:
https://www-304.ibm.com/ibmlink/sis/viewAparDoc.wss?context=aparAndUsage&documentIds=OA18803&searchWords=SHRLIBRGNSIZE&libraryType=D&lc=en&cc=US
<quote>
For a BPX1LOD (loadhfs) syscall request, when the module
cannot be loaded as a user-shared library program, it should be
loaded into user private storage as non-shared. This is not
happening, and the loadhfs() syscall is failing with
rsn091E03B7 JRShrStgShortage due to MAXSHAREPAGES being
exceeded.
</quote>

https://www-304.ibm.com/ibmlink/sis/viewAparDoc.wss?context=aparAndUsage&documentIds=OA04183&searchWords=SHRLIBRGNSIZE&libraryType=D&lc=en&cc=US

<quote>
Change current statement as follows:
The storage used in the user address space to establish the
mapping to the shared library region is from the high end of
private storage;
| in most cases,
it does not interfere with the virtual storage
used by the application program.

Add the following paragraph:

The amount of storage that is carved out of the high end of
private storage of each address space that loads a system
shared library object is based on the SHRLIBRGNSIZE parmlib
parameter value. If this value is set too high, the storage
set aside for the shared library region mapping could
interfere with the private storage requirements of each of
these address spaces. For this reason, the value specified
for SHRLIBRGNSIZE should be the minimum size required to
contain all of the system shared library programs that are
to be used on the system.
</quote>

https://www-304.ibm.com/ibmlink/sis/viewEskbDoc.wss?context=aparAndUsage&documentIds=swg21178068&searchWords=SHRLIBRGNSIZE&libraryType=E&lc=en&cc=US
<quote>
Problem(Abstract)
What are the consequences (for example, are there abends) of specifying
a SHRLIBRGNSIZE that is not large enough for all the
shared-library-eligible (that is, have the "l" extended attribute set)
programs that are accessed?

Resolving the problem

If the shared library cache is not big enough, then z/OS will revert to
normal storage use and the load module will be loaded into the broker
address space.

</quote>



On Sat, 2011-10-29 at 23:46 -0400, Lizette Koehler wrote:
> SHRLIBRGNSIZE
--
John McKown
Maranatha! <><

[John Gilmore]

Lizette,

Your current value, 64MiB, is the very conservative z/OS default. If
you are 1) making significant use of Java and 2) have multiple CICS
regions in which it is in use you may safely double it and then work
back down by observing how much of it is in use over an interval of
one or two work weeks.

John Gilmore, Ashland, MA 01721 - USA

[Lizette Koehler]

John and Mike

Thanks. I guess this is going to be an ETR to IBM. I am not sure how to
monitor or identify when more is better or if I can use less. This happened
after an IPL, so I am sure that there are new "UNIX" functions and it tipped
it over this limit. However, with the lack of research tools (we do D
OMVS,L every hour), it is close to impossible for me to do any type of
capacity management on this area. I would hope that IBM has tools to list
the shared libraries, modules, usage type information rather than just a
blanket - so much storage available so much storage used. To me that is
just asking for an outage of a Unix process or system failure of some kind.

If I turn up anything with IBM I will let you know.

This is not something I just want to blindly increase as the documentation
is not clear on how my overall system performance is going to be impacted.

Thanks for the help

Lizette

[Mike Schwab]

On Sun, Oct 30, 2011 at 8:40 AM, Lizette Koehler
<star...@mindspring.com> wrote:
> John and Mike
>
> Thanks.  I guess this is going to be an ETR to IBM.  I am not sure how to
> monitor or identify when more is better or if I can use less.  This happened
> after an IPL, so I am sure that there are new "UNIX" functions and it tipped
> it over this limit.  However, with the lack of research tools (we do D
> OMVS,L every hour), it is close to impossible for me to do any type of
> capacity management on this area.  I would hope that IBM has tools to list
> the shared libraries, modules, usage type information rather than just a
> blanket - so much storage available so much storage used.  To me that is
> just asking for an outage of a Unix process or system failure of some kind.
>
> If I turn up anything with IBM I will let you know.
>
> This is not something I just want to blindly increase as the documentation
> is not clear on how my overall system performance is going to be impacted.
>
> Thanks for the help
>
> Lizette
>

Looks like the parameter value is only used once after an IPL.
Increase by an amount you feel comfortable with (10%, 20%, 30%, 50%,
100%) then see if the message occurs after the next IPL, and continue
monitoring actual usage. Asking support for a recommendation is fine,
but their system may have a different combination of JVM functions
from their machines.

--
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

[Shmuel Metz , Seymour J.]

In <039001cc9709$8dae4510$a90acf30$@mindspring.com>, on 10/30/2011

at 09:40 AM, Lizette Koehler <star...@MINDSPRING.COM> said:

>This is not something I just want to blindly increase as the
>documentation is not clear on how my overall system performance is
>going to be impacted.

I doubt that performance is an issue. What might be a problem is a
shortage of private virtual storage above the line.

BTW, am I the only one to think of Contract Bridge when I see a
reference to the line?

--
Shmuel (Seymour J.) Metz, SysProg and JOAT
ISO position; see <http://patriot.net/~shmuel/resume/brief.html>
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)

[Lizette Koehler]

>
>>This is not something I just want to blindly increase as the
>>documentation is not clear on how my overall system performance is
>>going to be impacted.
>
>I doubt that performance is an issue. What might be a problem is a
>shortage of private virtual storage above the line.
>
>BTW, am I the only one to think of Contract Bridge when I see a
>reference to the line?
>

IBM is resesarching as the D OMVS,L as not showing correct infomration based on what is actually using the SHRLIBRGNSIZE.

I will let you know what happens.


Lizette

[Shane]

On Mon, 31 Oct 2011 12:23:00 -0500 "Shmuel Metz (Seymour J.)" wrote:

...

> BTW, am I the only one to think of Contract Bridge when I see a
> reference to the line?

Can't say I've ever confused the two in all these years.
Of course, now that you've mentioned it ...

Shane ...

[John Gilmore]

Conservatism here--rejection of any reflexive, unthinking enlargement
of the [Unix] shared-library region size--is the right stuff.

I suspect, however, that my earlier post erred in not being
unambiguously directive. The performance penalties are, this time,
all on the down side, They are significant when this region size is
too small, and those of enlarging it in any reasonable way, say by
doubling the 64 Mib default, are exiguous.

Still, it is appropriate to think before jumping.

John Gilmore, Ashland, MA 01721 - USA

[Graham Harris]

Lizzette, I think what may help you out here a little in terms of seeing
what is 'inside' SHRLIBRGN is this nice little REXX:

ftp://public.dhe.ibm.com/s390/zos/tools/wjsigshl/wjsigshl.txt

Its accessible via the USS tools & toys site (although wasnt at the time we
were given it, a few years back), although it may not be terribly obvious
unless you are specifically looking for it.

We went through the loop you are going through right now, a few years back,
as we didnt fully understand the need nor the ramifications of increasing
SHRLIBRGNSIZE. After gaining some better understanding, we decided to
leave it at the 64Mb default, as changing it would have had a detrimental
impact on the amount of extended private available to OMVS address spaces,
such as 31bit webspheres, so that was a complete no-no for us. We decided
to just take the hit of the 'overspill' of SHRLIB-eligible objects getting
loaded into an address space's private once SHRLIB was full.

I setup some housekeeping using the above rexx to monitor the 'residency'
of what was in SHRLIBRGN over time, and it is quite interesting to see how
things change over an extended period of time.

I did have plans to 'manage' the attributes of certain objects such that
they would not be eligible for loading into SHRLIBRGN if just one task
was using them (for example), as there was clearly better justification for
objects that a large number of webspheres or CTGs all sharing the same
things, rather than just one outlier address space selfishly taking up
valuable SHRLIBRGN slots.
But they were only plans, and they didnt actually go anywhere..!!

Would be nice if IBM could allow this to be managed a little better than
the first-come-first-served principle, although I cant think right now how
it could easily be done any different, I must admit!
Moving SHRLIBRGN above the bar would make this a non-issue of
course.......that would definitely get my vote!

And I think someone mentioned that things dont always "tie up", and if I
remember correctly, I think this could be because of the "lazy flush" which
happens after the last user of an object closes down, such that an object
with '0 users' will get purged out of SHRLIBRGN a period of time afterwards
(cant remember the precise controls/timings around that though).

HTH.

Graham

[Mike Schwab]

On Sat, Nov 5, 2011 at 3:25 PM, Graham Harris <harr...@gmail.com> wrote:
<deleted>

> Would be nice if IBM could allow this to be managed a little better than
> the first-come-first-served principle, although I cant think right now how
> it could easily be done any different, I must admit!
> Moving SHRLIBRGN above the bar would make this a non-issue of
> course.......that would definitely get my vote!

<deleted>

Maintain a Peak User count for each module in a file on the system
(directory or ZZCOUNTS of that library?) Don't put into shared region
unless peak was >= 2 (or higher). Total of modules with peaks above
each value could be displayed.

--
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

[Graham Harris]

Some additional info:

Example output from the "SHRLIBRGN contents" rexx is given below:

Usage Meg Used-Unused-Pgs Pathname
44 1 40 216 /CTQ131/usr/lpp/java/J5.0/bin/libnet.so
44 1 142 114 /CTQ131/usr/lpp/java/J5.0/bin//libjclscar_23.so
44 1 63 193 /CTQ131/usr/lpp/java/J5.0/bin//libj9vrb23.so
44 1 81 175 /CTQ131/usr/lpp/java/J5.0/bin//libj9jvmti23.so
44 1 67 189 /CTQ131/usr/lpp/java/J5.0/bin//libj9dyn23.so
44 2 260 252 /CTQ131/usr/lpp/java/J5.0/bin//libj9gc23.so
44 7 1562 230 /CTQ131/usr/lpp/java/J5.0/bin//libj9jit23.so
44 1 73 183 /CTQ131/usr/lpp/java/J5.0/bin//libj9dmp23.so
44 1 89 167 /CTQ131/usr/lpp/java/J5.0/bin/libj9prt23.so
44 1 92 164 /CTQ131/usr/lpp/java/J5.0/bin/libjava.so
44 1 173 83 /CTQ131/usr/lpp/java/J5.0/bin/libj9vm23.so

Usage is the number of tasks that are making use of that particular entry.
Meg is the number of 1Mb slots in SHRLIBRGN that this entry is using (yes,
even a 4Kb object will use a 1Mb slot!)
Used & Unused pages, indicates how many 4Kb pages are used/unused within
that entry.
Pathname gives the last 64 characters of the pathname, thus may need a bit
of detective work to establish the exact pathname, if it comes from a
particularly deep directory nesting.

Having refreshed my memory a little, I think one of my 'plans of attack' to
maximise the effectiveness of SHRLIBRGN was to target small objects (in
terms of turning their shrlib 'bit' off), as little objects would use
little extra storage in an address space's private if it were to be loaded
there (where a 4Kb object would really only use 4Kb).

But as previously mentioned, my plans to try & micro-manage this
environment, all went out the window.

Something else to be aware of, is that even if EXACTLY the same shareable
object binaries are being used, if they are being loaded from different
paths (as can be the case for websphere installs for example), then they
are treated as separate SHRLIBRGN entries.