In our system for JES2 RBI1.PROCLIB and RBI2.PROCLIB were defined in
sequence resp. Later due to some reason RBI1.PROCLIB was deleted and
recreated.
But after that we are facing problem that PROCs from the RBI1.PROCLIB are
not getting invoked.
What could be the problem ?
JAcky
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to list...@bama.ua.edu with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html
Most likely due to the old proclib being in a different physical
location on the volume or even on a new volume than the newly allocated
dataset. If you have an alternate jes2 proclib concatenation defined you
can force a reopen of the proclib concatenation by submitting a job with
a jobparm requesting that alternate proclib concatenation which should
force a close of the primary concatenation and an open of the
concatenation that you requested.
Once another job gets converted using the default concatenation JES2
should reopen the proclib datasets which will then find the new location
of the dataset.
--
Mark Jacobs
Time Customer Service
Tampa, FL
----
In accordance to the principles of Doublethink, it
does not matter if the war is not real, or when it
is, that victory is not possible. The war is not
meant to be won. It is meant to be continuous.
The essential act of modern warfare is the destruction
of the produce of human labor. A hierarchical society
is only possible on the basis of poverty and ignorance.
In principle, the war effort is always planned to
keep society on the brink of starvation. The war is waged
by the ruling group against its own subjects. And its
object is not victory over Eurasia or Eastasia, but to
keep the very structure of society intact.
George Orwell - 1984
But after that we are facing problem that PROCs from the RBI1.PROCLIB are
not getting invoked. <<< snippage
Did the pointer get lost?
Daniel McLaughlin
Z-Series Systems Programmer
Information & Communications Technology
Crawford & Company
4680 N. Royal Atlanta
Tucker GA 30084
phone: 770-621-3256
fax: 770-621-3237
email: Daniel_M...@us.crawco.com
web: www.crawfordandcompany.com
Best Overall Third-Party Claims Administrator - 2007 "Business Insurance" Readers Choice Awards
Consider the environment before printing this message.
This transmission is intended exclusively for the individual or entity to which it is addressed. This communication may contain information that is confidential, proprietary, privileged or otherwise exempt from disclosure. If you are not the named addressee, you are NOT authorized to read, print, retain, copy or disseminate this communication, its attachments or any part of them. If you have received this communication in error, please notify the sender immediately and delete this communication from all computers. This communication does not form any contractual obligation on behalf of the sender, the sender's employer, or the employer's parent company, affiliates or subsidiaries.
Lizette
>>>
In our system for JES2 RBI1.PROCLIB and RBI2.PROCLIB were defined in
sequence resp. Later due to some reason RBI1.PROCLIB was deleted and
recreated.
But after that we are facing problem that PROCs from the RBI1.PROCLIB are
not getting invoked. <<< snippage
----------------------------------------------------------------------
IEC143I 213-04,IFG0194D,JES2,JES2,PROC13-0003,9800,RBI031,SCLM1.PROCLIB
JAcky
Is dataset SCLM1.PROCLIB actually on volume RBI031 after it was
recreated? If it moved was the catalog entry changed to match its new
location?
Alan Schwartz
Infrastructure Management Sr Analyst
-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@bama.ua.edu] On Behalf
Of Jacky Bright
Sent: Friday, March 28, 2008 8:43 AM
To: IBM-...@bama.ua.edu
Subject: Re: JES2 DD Concatenation issue
I am getting following erros in SYSLOG
IEC143I 213-04,IFG0194D,JES2,JES2,PROC13-0003,9800,RBI031,SCLM1.PROCLIB
JAcky
//JOB Card
/*JOBPARM PROC=PROC?? <-- Make the ?? whatever your Alternate PROC dd
statement is in JES2 PROC.
// EXEC PROC IEFBR14 (Do not need to exist)
This will close PROC00 and open PROC??. Then the next Proc expansion will
open PROC00 by default.
Lizette
I am getting following erros in SYSLOG
IEC143I 213-04,IFG0194D,JES2,JES2,PROC13-0003,9800,RBI031,SCLM1.PROCLIB
----------------------------------------------------------------------
How can I change the Catalog Entry ?
JAcky
>Hi,
>
>In our system for JES2 RBI1.PROCLIB and RBI2.PROCLIB were defined in
>sequence resp. Later due to some reason RBI1.PROCLIB was deleted and
>recreated.
>
>But after that we are facing problem that PROCs from the RBI1.PROCLIB are
>not getting invoked.
>
>What could be the problem ?
>
That RBI1.PROCLIB was deleted! ;-)
Even though the library is not ENQed by JES2 (due to the NODSI entry in
the PPT) the library is still in use. If the proclib is defined in the JES2 JCL
you'll have to $PJES2,ABEND and restart JES2. If you are using dynamic
proclibs, just issue $TPROCLIB(*) and that should fix it.
Mark
--
Mark Zelden
Sr. Software and Systems Architect - z/OS Team Lead
Zurich North America / Farmers Insurance Group - ZFUS G-ITO
mailto:mark....@zurichna.com
z/OS Systems Programming expert at http://expertanswercenter.techtarget.com/
Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html
>
>Most likely due to the old proclib being in a different physical
>location on the volume or even on a new volume than the newly allocated
>dataset.
Yes.
>If you have an alternate jes2 proclib concatenation defined you
>can force a reopen of the proclib concatenation by submitting a job with
>a jobparm requesting that alternate proclib concatenation which should
>force a close of the primary concatenation and an open of the
>concatenation that you requested.
>
That "trick" won't work for this situation. Since the extents are know already,
it would only works if the proclib was allocated in the exact same place
it originally existed (doubtful). That was what we would do when someone
needed to compress a JES2 JCL defined proclib and we weren't about to
IPL. See the archives for plenty more on that subject.
>Depending on your level of JES2 you may want to look at using PROCLIB
>statements in JES2 rather than coding DD statements. Or have the users
>start using JCLLIBs.
>
I hope they are running at least z/OS 1.2 by now! But I agree that it
should be done to help recover from a problem like this or prevent it
entirely by using JCLLIB.
RACF (or other) should be used to protect JES2 defined proclibs (either
dynamic or JCL defined) by only allowing ALTER authority to a select
few who understand that JES2 proclibs aren't ENQed. If shop standards
or politics won't allow that, then I have seen shops define the proclibs to
an arbitrary long running STC with DISP=SHR so they would get ENQed.
You cannot move a JES2 JCL defined PROCLIB while JES2 is up. If it is in an
SMS Managed pool, you probably need to look at coding PROCLIB statements in
JES2 rather than the JCL.
Mark makes a valid point.
When JES2 had JCL coded PROCLIBs they are not ENQUEUED so if your security
product does not have the general access of READ and only a few groups that
have ALTER, you can have these types of issues.
I would recommend that your security product specify a UACC of READ (if you
are a RACF Shop) and only the group responsible to support JES2 have ALTER.
When a JES2 JCL defined proclib is delete JES2 does not care. Once JES2 has
opened the PROCLIB data set it builds a TTR list and uses that rather than
going back to the dataset. It just reads the TTR pointers it built for that
data sets. I do not have the in depth details on how that works, but if you
are interested, I am sure there are several on this list (like Mark Z) who
can provide that.
Once the TTRs are built, if the data set is deleted, JES2 just goes on using
those TTRs to access the procs. Should you compress JES2 JCL defined
proclib you can run into similar issues, though I think you the a HASP
message that states an I/O error has occurred.
My preference has always been for the use of JCLLIB statements in
Application JCL, and PROCLIB statements to reduce these types of errors in
JES2.
Should the recommended actions not correct the problem, you will most likely
need to bounce JES2 so it can find the new home of the proclib data set and
rebuild its TTRs. If you do need to abend JES2 you do not need to stop
work. However, I think things that use the SAPI interface may abend with
JES2. This is like VPS or similar products. IIRC, I have abended JES2 and
not really seen any issues with currently running work.
Lizette
>Hi,
>
>In our system for JES2 RBI1.PROCLIB and RBI2.PROCLIB were defined in
>sequence resp. Later due to some reason RBI1.PROCLIB was deleted and
>recreated.
>
>But after that we are facing problem that PROCs from the RBI1.PROCLIB are
>not getting invoked.
>
>What could be the problem ?
>
That RBI1.PROCLIB was deleted! ;-)
Even though the library is not ENQed by JES2 (due to the NODSI entry in
the PPT) the library is still in use. If the proclib is defined in the JES2
JCL
you'll have to $PJES2,ABEND and restart JES2. If you are using dynamic
proclibs, just issue $TPROCLIB(*) and that should fix it.
----------------------------------------------------------------------
Error as :
IEC143I 213-04,IFG0194D,JES2,JES2,PROC13-0003,9800,RBI031,SCLM1.PROCLIB
Why the catalog is still expecting the dataset from RBI031 itself ?
JAcky
>My point is why CATALOG address space is not considering the new volume
>RBI035 why is it that still expecting RBI031 even though I have recreated
>the SCLM1.PROCLIB with catalog option and it got created in RBI035. If I see
>the LISTC for SCLM1.PROCLIB it shows the volume as RBI035
>
>Error as :
>
> IEC143I 213-04,IFG0194D,JES2,JES2,PROC13-0003,9800,RBI031,SCLM1.PROCLIB
>
>Why the catalog is still expecting the dataset from RBI031 itself ?
>
>JAcky
>
>
The catalog has nothing to do with it. Despite the fact that someone deleted
the data set, it is still ALLOCATED to JES2 and the original extents are in the
DEB. You can rename the new one, re-allocate it to its original volume and
copy it back or just leave it where it is. Either way you have to recycle JES2
to pick up the new extents.
Capiche?
Mark
--
Mark Zelden
Sr. Software and Systems Architect - z/OS Team Lead
Zurich North America / Farmers Insurance Group - ZFUS G-ITO
mailto:mark....@zurichna.com
z/OS Systems Programming expert at http://expertanswercenter.techtarget.com/
Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html
----------------------------------------------------------------------
Once JES2 has started the CATALOG is no longer part of JES2's process to
find its JCL defined Proclibs.
JES2 builds an internal table that probably holds information like the data
set name, the volser and the TTR locations in that dataset for the PROCs.
You can with the appropriate ALTER authority go out and delete - define -
catalog the JES2 PROCLIB anywhere. JES2 does not enqueue this data set.
Therefore the CATALOG and JES2 can disagree on where the proclib actually
lives.
The only way to get this correct is to reallocate the proclib back on the
original volume in the original location and pray that no one has already
overlaid the TTRs where the JES2 Proclib use to live.
Otherwise, if the proclib on the new volumes is good, you will need to
bounce JES2 (HOT START) This is done with a $PJES2,ABEND. You will need to
reply with the appropriate response. JES2 will then find the new home for
this JCL defined Proclib.
Lizette
My point is why CATALOG address space is not considering the new volume
RBI035 why is it that still expecting RBI031 even though I have recreated
the SCLM1.PROCLIB with catalog option and it got created in RBI035. If I see
the LISTC for SCLM1.PROCLIB it shows the volume as RBI035
Error as :
IEC143I 213-04,IFG0194D,JES2,JES2,PROC13-0003,9800,RBI031,SCLM1.PROCLIB
Why the catalog is still expecting the dataset from RBI031 itself ?
JAcky
----------------------------------------------------------------------
Thanks but this is like IPLing the system itself which involves outage which
many not be possible in our system So I was just looking for alternatives...
JAcky
On 3/28/08, Lizette Koehler <star...@mindspring.com> wrote:
>
>Liz,
>
>Thanks but this is like IPLing the system itself which involves outage which
>many not be possible in our system So I was just looking for alternatives...
>
>
There is really not much harm in doing $PJES2,ABEND and replying "END" to
not take a dump and restarting it. It will interrupt printing, NJE and remotes
if you have them. So other than draining printers you usually can just
restart your lines / NJE connections after a JES2 bounce. Much less
disruptive than an IPL.
Mark
--
Mark Zelden
Sr. Software and Systems Architect - z/OS Team Lead
Zurich North America / Farmers Insurance Group - ZFUS G-ITO
mailto:mark....@zurichna.com
z/OS Systems Programming expert at http://expertanswercenter.techtarget.com/
Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html
----------------------------------------------------------------------
In an earlier contribution, you mentioned that JES holds no ENQ
on the PROCLIBs. That sounds terribly dangerous. Why would they
design it that way?
Just curious: can the PROCLIB concatenation contain PDSEs?
-- gil
I would guess that the thought process was not to prevent JES2 from
starting if another system had an exclusive enqueue on a proclib dataset.
> Just curious: can the PROCLIB concatenation contain PDSEs?
>
>
It shouldn't be a problem since PDSE support is active well before JES2
starts up.
> -- gil
>
> ----------------------------------------------------------------------
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to list...@bama.ua.edu with the message: GET IBM-MAIN INFO
> Search the archives at http://bama.ua.edu/archives/ibm-main.html
>
>
--
Mark Jacobs
Time Customer Service
Tampa, FL
----
In accordance to the principles of Doublethink, it
does not matter if the war is not real, or when it
is, that victory is not possible. The war is not
meant to be won. It is meant to be continuous.
The essential act of modern warfare is the destruction
of the produce of human labor. A hierarchical society
is only possible on the basis of poverty and ignorance.
In principle, the war effort is always planned to
keep society on the brink of starvation. The war is waged
by the ruling group against its own subjects. And its
object is not victory over Eurasia or Eastasia, but to
keep the very structure of society intact.
George Orwell - 1984
----------------------------------------------------------------------
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
Thanks but this is like IPLing the system itself which involves outage
which many not be possible in our system So I was just looking for
alternatives...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
A dynamic PROCLIB concatenation can be repaired via a series of
$DEL/$ADD/$T PROCLIB JES2 commands.
A static //PROCnn concatenation can be replaced/ overridden via a new
dynamic one using /$ADD/$T PROCLIB JES2 commands.
...hth
Indeed.
> In an earlier contribution, you mentioned that JES holds no ENQ
> on the PROCLIBs. That sounds terribly dangerous. Why would they
> design it that way?
>
MVS programs honor the settings in the PPT. HASJES20 and IATINTK have
NODSI on their PPT entries. This can be changed if desired.
> Just curious: can the PROCLIB concatenation contain PDSEs?
>
Silly question. BPAM is BPAM.
--
Edward E Jaffe
Phoenix Software International, Inc
5200 W Century Blvd, Suite 800
Los Angeles, CA 90045
310-338-0400 x318
edj...@phoenixsoftware.com
http://www.phoenixsoftware.com/
>On Fri, 28 Mar 2008 09:44:20 -0500, Mark Zelden wrote:
>>
>>The catalog has nothing to do with it. Despite the fact that someone deleted
>>the data set, it is still ALLOCATED to JES2 and the original extents are
in the
>>DEB. You can rename the new one, re-allocate it to its original volume and
>>copy it back or just leave it where it is. Either way you have to recycle
JES2
>>to pick up the new extents.
>>
>Are DEBs created by ALLOCATE? I had imagined it was OPEN.
The DD associated with the proclib concatenation was previously opened.
And by multiple converter subtasks depending on PCEDEF CNVTNUM.
In my case CNVTNUM=10.
>
>In an earlier contribution, you mentioned that JES holds no ENQ
>on the PROCLIBs. That sounds terribly dangerous. Why would they
>design it that way?
>
How would you ever re-allocate a proclib if it was ENQed? At least
before TSO. Shutdown JES2, then run a batch job to do the rename.
Wait... you can't run a batch job, JES2 is down!
>Just curious: can the PROCLIB concatenation contain PDSEs?
Yes. That eliminates the COMPRESS issue or the problem you might
run into if the library takes an additional extent. The second problem
was fixed about 15 years ago in MVS/ESA V4. JES2 recognizes this
condition via I/O error and then will close and reopen the DD for
all converter tasks. Actually, knowing this works makes me think
that the "trick" mentioned an an earlier post (running jobs that point
to a nonexistent or another PROCxx DD) could work to fix the problem
without having to bounce JES2. But I don't know if there is some
extra code in the fix from 15 years ago that handles that situation
differently.
Mark
--
Mark Zelden
Sr. Software and Systems Architect - z/OS Team Lead
Zurich North America / Farmers Insurance Group - ZFUS G-ITO
mailto:mark....@zurichna.com
z/OS Systems Programming expert at http://expertanswercenter.techtarget.com/
Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html
----------------------------------------------------------------------
I've seen earlier items about Healthchecker and its ESQA checking, but I
have a general question to which I haven't been able to find the
information in the manuals.
Given that ESQA will just overflow into ECSA is there any real reason
not to allocate ESQA fairly low so that it is always at 100%, and then
size/monitor ECSA for the combined requirements? I am aware that that
would not be a good thing below the line, but I haven't read of any
clear reason to 'tune' ESQA rather than ESQA/ECSA as a combined area.
Best regards,
David Tidy
Tel:(31)115-67-1745
IS Technical Management/SAP-Mf
Dow Benelux B.V.
Mailto:dt...@dow.com
Inkoop Gbw (427)
H. H. Dowweg 5
4542NM Hoek
The Netherlands
Handelsregisternr. 24104547
I have always been of the opinion that a little SQA/ESQA conversion is okay, but not a lot.
IBM (I believe) used recommend as little as possible.
But, now with 64-bit addressing, and no expanded storage I think you cannot get away with it.
IIRC, IPL-processing needs more ESQA to build page tables, and that will not be allowed to overflow into CSA, since the entire system is not up yet.
There's a paper on the WTC site explaining this.
And, as my faded memory recalls, it might have been written by Kathy Welsh/Walsh (?).
-
Too busy driving to stop for gas!
>Yes. That eliminates the COMPRESS issue or the problem you might
>run into if the library takes an additional extent. The second problem
>was fixed about 15 years ago in MVS/ESA V4. JES2 recognizes this
>condition via I/O error and then will close and reopen the DD for
>all converter tasks. Actually, knowing this works makes me think
>that the "trick" mentioned an an earlier post (running jobs that point
>to a nonexistent or another PROCxx DD) could work to fix the problem
>without having to bounce JES2. But I don't know if there is some
>extra code in the fix from 15 years ago that handles that situation
>differently.
>
>Mark
>--
I was going to contribute to this before, to correct this statement you made,
Mark. Having had the actual hands-on experience of a JES2 proclib going
away "in anger", I know that the trick of running a job pointing to a non-
existing JES2 PROCxx DD actually works - IF THE PROCLIB DATA SETS ARE ON
THE SAME VOLUME AS BEFORE. JES2 will CLOSE and reOPEN the PROCnn
concatenation, and so long as the proclib data sets are on the exact same
VOLUME, new extent information is built and JES2 is happy.
Brian
>Yes. That eliminates the COMPRESS issue or the problem you might
>run into if the library takes an additional extent.
Yes, you can use PDSEs in the JES2 PROCLIB concatenation. LINKLIST too.
But, not such a great idea, IMHO.
Maybe I'm wrong here (and I could be), but I somehow remember that the
empty space in a PDSE is not reclaimed until the dataset is closed. Now, you
can't easily close a LINKLIST dataset; system shutdown doesn't quite do it.
Removing it from the active LINKLIST probably will. A JES2 PROCLIB is easier
to close; a temporary switch to a new concatenation usually works.
However, the caveat here is that it must be closed on ALL systems sharing it,
AT THE SAME TIME! Otherwise, the free space clean operation will not take
place.
I don't think that this has (yet) been addressed by IBM development.
Additional extents are not as large of a problem with PDSEs and they are with
PDSs. If a PDS (in a PROCLIB or a LINKLIST) takes a new extent, the only
address space to know about it is the initiator in which the copy job executed
(or the TSO user that took the new extent). Neither JES2 nor the LINKLIST
will be aware. With a PDSE, all of this activity takes place within the PDSE
address space. This address space can communicate this information to other
images within the same SYSPLEX. All JES2s and LINKLISTs become aware.
Corrections, if necessary, to the above are welcome!
>>
>> In an earlier contribution, you mentioned that JES holds no ENQ
>> on the PROCLIBs. That sounds terribly dangerous. Why would they
>> design it that way?
>>
>
> I would guess that the thought process was not to prevent JES2 from
> starting if another system had an exclusive enqueue on a proclib dataset.
>
I don't think this is it. If I remember correctly, if NODSI is
specified in the PPT the enq is still acquired at allocation,
and then freed. So JES would still wait if some task had an
exclusive enq on a proclib. I would guess the thinking might
have been to allow the use of DISP=OLD by jobs that were
updating or compressing a proclib (or other JES owned dataset).
--
Richard
>In an earlier contribution, you mentioned that JES holds no ENQ
>on the PROCLIBs. That sounds terribly dangerous. Why would they
>design it that way?
Yes, I believe it is very dangerous for JES2 to not hold an ENQ for its data
sets. Some months ago, I submitted the following requirement to JES2
through my marketing rep, and a similar one through the JES2 Project at
SHARE. To the best of my knowlege, there has been no formal response to
this requirement, but from conversations at SHARE on this issue, I believe the
requirement is "understood" (which is always a good first step).
Here's how I phrased my requirement to JES2 regarding this issue:
"Today by default, JES2 does not participate in standard z/OS data set
serialization (an ENQ on MAJOR SYSDSN, MINOR data.set.name). As a result,
by default, it is possible for a user to delete an in-use JES2 PROCLIB data set
without notification that the data set is in use. If the user codes JES2
PROCLIBs in JES2's JCL, the next start of JES2 will fail with a JCL error. Today,
JES2 depends only upon knowledgeable users to avoid inadvertent deletion of
data sets critical to the operation of JES2. To protect customers from an
inadvertent error causing a system outage, JES2 should use standard z/OS
facilities to protect via ENQ its PROCLIB data sets.
"Because of the value NODSI in the IBM default PPT entry for program
HASJES20, any DD statement coded in the SYS1.PROCLIB(JES2) JCL is
unprotected by ENQ. NODSI is defined as follows: "If NODSI is specified, the
system still issues an ENQ for all data sets requested by the program.
However, the ENQ is released before the problem program is started." So, the
protection exists as the system starts JES2, but is released after step
initiation and before control is passed to HASJES20.
"Further, JES2 extends its "no ENQ" philosophy by specifying S99NORES for
Dynamic PROCLIB data sets JES2 allocates via SVC 99. Because of the use of
S99NORES, even if the customer overrides the PPT entry for HASJES20 to
specify DSI, JES2 does not hold an ENQ for any of these data sets. The z/OS
Allocation component describes the responsibilities for the use of S99NORES as
follows: "Note: Data sets being allocated are normally serialized via ENQ with
MAJOR name SYSDSN, MINOR name -data set name-. When S99NORES is set,
there is NO data set serialization and multiple tasks may reference or update
the data set simultaneously, resulting in unpredictable effects. It is the
responsibility of the authorized program setting S99NORES to provide the
necessary serialization." It is the opinion of the author of this requirement that
JES2 provides no facility to provide the necessary serialization, as specified by
the documentation for S99NORES.
"Originally, OS/390 and MVS did not hold an ENQ for system link list data sets.
More than ten years ago, IBM added such ENQ protection for system link list
data sets. IBM also, in 1997, added the SETPROG LNKLST,UNALLOCATE
command to allow the user to release this ENQ protection in the event a same-
named data set required some sort of maintenance.
"Further, about seven years ago (in the OS/390 2.10 timeframe) IBM enhanced
DFP to add support for the authorized rename of apparently in-use data sets.
This authorization is based upon FACILITY class STGADMIN.DPDSRN.nnn and
allows a user who presumably knows what he is doing to rename a data set
otherwise protected by an ENQ. This capability is documented in manual 'z/OS
Using Data Sets'.
"More than a decade ago, IBM decided that it was important to protect the
system link list data sets with an ENQ. It is time for JES2 to do the same.
Doing so would help customers avoid outages which are caused by the
deletion of non serialized but critical and in-use data sets needed by JES2.
"Twenty years ago, JES2 systems programmers were "god" at most shops.
Today, in light of HIPPA, SOX, and other government regulations, the author of
this requirement has only READ authority to several JES2 PROCLIB data sets,
while other non-systems programmer staff have ALTER authority. It is no
longer sufficient to rely upon knowlegable sysprogs to protect JES2 - ENQ is
required.
"This exposure has resulted in a multi-system outage for the author of this
requirement. A JES2 PROCLIB data set, shared by every JES2 in the sysplex,
was deleted by an individual RACF-authorized to do so. Some hours later, after
the DASD extent containing the deleted PROCLIB's directory was reused by
another data set, every batch job and TSO Logon attempt within the entire
sysplex failed due to I/O Error in BLDL. A Hot Start of JES2 failed with a JCL
error due to the missing data set. It was only pure chance that the JES2
sysprog was still logged on to TSO prior to the I/O Errors starting which
allowed for a relatively non-disruptive recovery from this problem - the JES2
JCL was changed to remove the deleted PROCLIB, and JES2 hot-started on
every LPAR. Had the JES2 sysprog not been logged on, there may have been
no RACF authorized user still able to use the system, which would have made
recovery much more problematic.
"As a result, this customer has implemented DSI for the HASJES20 PPT entry,
and has had this value in effect for two years.
"Now, as this customer moves to take advantage of JES2 features to further
protect JES2 from damage by taking advantage of Dynamic PROCLIB within
JES2 startup parms, this user is losing the protection afforded by DSI in the
HASJES20 PPT entry, because of JES2's use of S99NORES."
Brian
Thanks.
I converted to 64-bit so long ago, that I had forgotten all the details.
But, I remember we had to increase our ESQA allocation in order to be able to IPL.
-
Too busy driving to stop for gas!
----------------------------------------------------------------------
In our case, z/Architecture was not the culprit. It was PAVs that blew
INITSQA out of the water!
--
Edward E Jaffe
Phoenix Software International, Inc
5200 W Century Blvd, Suite 800
Los Angeles, CA 90045
310-338-0400 x318
edj...@phoenixsoftware.com
http://www.phoenixsoftware.com/
----------------------------------------------------------------------
Jack Kelly
202-502-2390 (Office)
>
>A static //PROCnn concatenation can be replaced/ overridden via a new
>dynamic one using /$ADD/$T PROCLIB JES2 commands.
>
>
According to the manual: "Dynamic PROCLIB can override PROCxx DDs in
the JES2 start PROC but cannot alter nor delete them."
So I decided to try this on a sandbox. It does work and I guess is the
best solution since you don't have to bounce JES2 at all or worry about
the number of converter subtasks etc.
But in playing with it, I found something really interesting (at least to
me).
After I was done adding the dynamic proclib I then deleted the definition
to put me back in a "broken" state. This sandbox JES2 has 2 converter
subtasks and right now one is broke and the other one is still finding the
PROC I am referencing from the proclib I deleted on the original
volume (I allocated one with the same name on another volume).
The interesting thing is that JES2 alternates between the 2 subtasks, so
JES2's usage of them is round robin. Every other job I submit works or
gets a IEC143I 213-04 on the proclib and a IEFC612I PROCEDURE xxx
WAS NOT FOUND error message.
Mark
--
Mark Zelden
Sr. Software and Systems Architect - z/OS Team Lead
Zurich North America / Farmers Insurance Group - ZFUS G-ITO
mailto:mark....@zurichna.com
z/OS Systems Programming expert at http://expertanswercenter.techtarget.com/
Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html
----------------------------------------------------------------------
>On Fri, 28 Mar 2008 12:03:54 -0500, Mark Zelden wrote:
>
>>Yes. That eliminates the COMPRESS issue or the problem you might
>>run into if the library takes an additional extent. The second problem
>>was fixed about 15 years ago in MVS/ESA V4. JES2 recognizes this
>>condition via I/O error and then will close and reopen the DD for
>>all converter tasks. Actually, knowing this works makes me think
>>that the "trick" mentioned an an earlier post (running jobs that point
>>to a nonexistent or another PROCxx DD) could work to fix the problem
>>without having to bounce JES2. But I don't know if there is some
>>extra code in the fix from 15 years ago that handles that situation
>>differently.
>>
>>Mark
>>--
>
>I was going to contribute to this before, to correct this statement you made,
>Mark. Having had the actual hands-on experience of a JES2 proclib going
>away "in anger", I know that the trick of running a job pointing to a non-
>existing JES2 PROCxx DD actually works - IF THE PROCLIB DATA SETS ARE ON
>THE SAME VOLUME AS BEFORE. JES2 will CLOSE and reOPEN the PROCnn
>concatenation, and so long as the proclib data sets are on the exact same
>VOLUME, new extent information is built and JES2 is happy.
>
Makes sense.
I was going to write this:
"The problem is, just like with the COMPRESS issue, you
have to flood the system with enough jobs that every converter task
gets used at the same time so it is closed and re-opened in all of them.
In my case, that would probably be difficult given the speed of todays
machines and the fact that I have 10 of them."
The "flooding" part is always how I thought it worked (and maybe it
did but has changed). See my last post:
http://bama.ua.edu/cgi-bin/wa?A2=ind0803&L=ibm-main&D=1&O=D&T=0&P=214254
Maybe it would be fixed after 10 jobs where submitted in a row with the
non-existing JES2 PROCxx DD.
Anyway... any time this has ever happened in a shop I was at (which has
probably been no more than 5 times in the past 15-20 years) I have always
just bounced JES2 (after making sure the PROCLIB existed and / or was moved
back to its proper location). It's quick enough and always fixes the problem.
BTW, I like your requirement. 5 times was 5 times too many. No reason the
ENQ can't be there and removed with an operator command when needed.
If JES2 is up, then:
o Submit a job with:
- STEP1 IDCAMS ALTER NAME
- STEP2 IEFBR14 with DISP=OLD on the PROCLIB catenands.
o The job waits on PROCLIB ENQ
o Stop JES2
o Start JES2
o JES2 waits on PROCLIB ENQ
o The RENAME job completes freeing the ENQs
o JES2 allocates PROCLIB and continues.
Hmmm. This requires JES2 stopped and started on all systems
in the GRSplex. But the timing isn't entirely critical;
ENQ helps.
It would help if there were a JES2 command to close, free,
allocate, and open PROCLIB; even better if it did some
thorough verification of PROCLIB validity and required
operator confirmation before proceeding if it detected
any anomaly ( CANCEL | RETRY | CONTINUE :-)
-- gil
I honesty don't remember the specifics after so many years (we upgraded
to ESS (2105-800) from RVA in 2002). But, the phenomenon is documented
in APAR II12396:
OS/390 V2 R6 and above:
SQA Space Shortage During NIP: During NIP processing,
it is possible that the system's minimum allocation for
SQA and extended SQA might be depleted before NIP processes
the SQA parameter. To avoid this situation, you can increase
the minimum SQA and/or extended SQA allocations, using the
INITSQA parameter in LOADxx. (See OS/390 MVS Initialization
Tuning Reference for information on the INITSQA parameter.)
---------------------------------------------------
This type of storage exhaustion has been seen when
adding ESS d/t2105 subsystems, since this subsystem
typically requires the addition of many devices.
FWIW, our LOADxx contains the following:
INITSQA 0000K 1536K
--
Edward E Jaffe
Phoenix Software International, Inc
5200 W Century Blvd, Suite 800
Los Angeles, CA 90045
310-338-0400 x318
edj...@phoenixsoftware.com
http://www.phoenixsoftware.com/
----------------------------------------------------------------------
In the above scenario, you would never get JES2 to shut down cleanly while a
job is still executing. (It would need to be an STC running SUBSYS=MASTER.)
And, even if this did work, you better hope that the job is successful; else,
you are left without the PROCLIB and JES2 won't start!
We run TCAS under MSTR and allow TSO/E users to logon under MSTR. This
can come in handy if JES is down. We also found that being able to issue
arbitrary TSO/E commands from MCS consoles using System REXX provides
some nice capabilities under similar circumstances. (One of my
contributions to the "Bit Bucket" @ SHARE in Orlando.)
--
Edward E Jaffe
Phoenix Software International, Inc
5200 W Century Blvd, Suite 800
Los Angeles, CA 90045
310-338-0400 x318
edj...@phoenixsoftware.com
http://www.phoenixsoftware.com/
----------------------------------------------------------------------
><snip>
>In our case, z/Architecture was not the culprit. It was PAVs that blew
>INITSQA out of the water!
><snip>
>Ed,
> Since we're finally getting to PAV, would you please expand on the
>problem/issue, e.g. how many PAVs and how much of an increase to the
>initial SQA?
>
More UCBs = more ESQA.
--
Mark Zelden
Sr. Software and Systems Architect - z/OS Team Lead
Zurich North America / Farmers Insurance Group - ZFUS G-ITO
mailto:mark....@zurichna.com
z/OS Systems Programming expert at http://expertanswercenter.techtarget.com/
Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html
----------------------------------------------------------------------
>On Fri, 28 Mar 2008 12:03:54 -0500, Mark Zelden wrote:
>>
>>>In an earlier contribution, you mentioned that JES holds no ENQ
>>>on the PROCLIBs. That sounds terribly dangerous. Why would they
>>>design it that way?
>>>
>>
>>How would you ever re-allocate a proclib if it was ENQed? At least
>>before TSO. Shutdown JES2, then run a batch job to do the rename.
>>Wait... you can't run a batch job, JES2 is down!
>>
>To my meager understanding, if JES2 is down you're SOL, whether
>or not it ENQs (but is there a special case in which JES2 might
>crash but fail to free the ENQS?) By my ancient experience,
>if JES2 is down, TSO is likewise SOL (or was it Roscoe, then?)
Exactly. So with no ENQ the procedure would usually be to
reallocate the PROCLIB just before a planned IPL. Allocate under
a new name, rename the existing one to a backup name, rename the
new one to the original name, then IPL. Note the rename as opposed
to delete. Even though the PROCLIB is renamed while open to JES2,
it is still accessible and usable. The same thing was done with LNKLST libs
and still can be done if you release the ENQ.
>
>If JES2 is up, then:
>
>o Submit a job with:
>
> - STEP1 IDCAMS ALTER NAME
>
> - STEP2 IEFBR14 with DISP=OLD on the PROCLIB catenands.
>
>o The job waits on PROCLIB ENQ
>
>o Stop JES2
>
>o Start JES2
>
>o JES2 waits on PROCLIB ENQ
>
>o The RENAME job completes freeing the ENQs
>
>o JES2 allocates PROCLIB and continues.
>
>Hmmm. This requires JES2 stopped and started on all systems
>in the GRSplex. But the timing isn't entirely critical;
>ENQ helps.
>
And all that doesn't sound terribly dangerous too? :-) Think back to the
days when you had a single system. I'll take the old way with RACF protection
to keep someone who didn't know what they were doing from deleting or
moving the proclib. As I mentioned... if that wasn't good enough you could
easily add the PROCLIBs with DISP=SHR to some arbitrary long running
STC to get the ENQ.
Mark
>The "flooding" part is always how I thought it worked (and maybe it
>did but has changed). See my last post:
>http://bama.ua.edu/cgi-bin/wa?A2=ind0803&L=ibm-main&D=1&amp;O=D&T=0&P=214254
>
>Maybe it would be fixed after 10 jobs where submitted in a row with the
>non-existing JES2 PROCxx DD.
>
>Anyway... any time this has ever happened in a shop I was at (which has
>probably been no more than 5 times in the past 15-20 years) I have always
>just bounced JES2 (after making sure the PROCLIB existed and / or was moved
>back to its proper location). It's quick enough and always fixes the problem.
>
Lots more of this in a few IBM APARs. OY45900, OW51308 and II07133.
You need IBMLINK to look at OY45900 and OW51308 as I can't find
a docview link to them. OY45900 doesn't mention anything
about "round robin" usage of the converter tasks:
"Thus, if there are multiple Converter PCEs /
tasks it will be necessary to abend JES2 and restart it to
guarantee that all proclibs are freshly opened ($P JES2,ABEND
.. HOTSTART)."
And here in II07133 (from 1993) there is more...
http://www-1.ibm.com/support/docview.wss?uid=isg1II07133
However, II07133 says:
Note: It is NOT necessary to force a close and open of a
Proclib after it's been compressed. Any I/O errors
resulting from a compress would have occurred while the
compress was in progress, and THAT would have forced a
close and open of the proclib. If no I/O errors occurred
while the compress was in progress, then no I/O errors
would be expected after the compress had completed, and
thus it is NOT necessary to perform the close and open.
In my experience, the statement above is not true. I have seen
problems caused by compress when no I/O errors occurred.
OW51308 is more recent and mentions using the dynamic proclib
facility in z/OS 1.2 instead of bouncing JES2. But it doesn't mention
anything about a COMPRESS.
I'm a bit surprised that no one appears to be using the JES2 backup proc.
S JES2BKUP,JOBNAME=JES2,SUB=MSTR
(Where the name of the proc is whatever you'd like it to be).
A stripped down version that allows you to bring up JES2 regardless of the
state of PROCLIBs would seem a reasonable action.
Adam
Thanks for the responses and pointer to Kathy's White Paper. It is true
that we have INITSQA coded, and that was for IPL failures which we
thought were related to additions/changes to the storage configuration.
However it did seem that INITSQA did not actually help much - just the
action of a second IPL was enough to avoid the shortage then, and anyway
that is due to requirements before the IEASYS parameter is handled by
the system. I say that because we have had failures even with INITSQA
coded, and reIPLing without changes appears to resolve things. These
were new systems - we haven't actually had the failures that recently,
so I hope my memory isn't tricking me.
As far as I understand from the White Paper, as long as you can
comfortably support an IPL, there is no real reason to worry about ESQA
filling and overflowing during the life of the system. She does mention
that some installations do follow a policy of minimal allocation (though
she doesn't recommend this).
I think my likely conclusion is to check our systems needs after they
are fully IPLed, and aim to cover that comfortably, but not to tune for
the Healthchecker 80% recommendation. Ideally we would not have any
spare ESQA by the next IPL, allowing us to manage ECSA and ESQA as a
combined pool.
Best regards,
David Tidy Tel:(31)115-67-1745
IS Technical Management/SAP-Mf Fax:(31)115-67-1762
Dow Benelux B.V. Mailto:dt...@dow.com
>That "trick" won't work for this situation.
It should, as long as he allows for multiple converter tasks. Or were you
assuming that he had PROCxx DD statements in his JES2 PROC?
>Since the extents are know already,
They aren't; JES2 closes the old proclib concatenation before it opens the
alternate proclib concatenation. Where you run into trouble is if you have
multiple Converter tasks and don't force all of them to close the bad
proclib concatenation.
--
Shmuel (Seymour J.) Metz, SysProg and JOAT
ISO position; see <http://patriot.net/~shmuel/resume/brief.html>
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)
>The catalog has nothing to do with it. Despite the fact that someone
>deleted the data set, it is still ALLOCATED to JES2 and the original
>extents are in the DEB. You can rename the new one, re-allocate it to
>its original volume and copy it back or just leave it where it is.
>Either way you have to recycle JES2 to pick up the new extents.
You only need to recycle JES2 to change the volume serial numbers of
static allocations, not to pick up new extents.
>The only way to get this correct is to reallocate the proclib back on the
>original volume in the original location
No. You must close the concatenation before you delete the data set and
there is no requirement that the new one go into the same location, just
that it go on the same volume with the same name.
Of course, if he switched to dynamic then it would be a lot easier. Unless
he's already using dynamic.
WRITELOG START?
V SYSLOG,HARDCPY?
--
Edward E Jaffe
Phoenix Software International, Inc
5200 W Century Blvd, Suite 800
Los Angeles, CA 90045
310-338-0400 x318
edj...@phoenixsoftware.com
http://www.phoenixsoftware.com/
----------------------------------------------------------------------
Have you tried the command "V SYSLOG,HARDCPY" ?
--
Richard
See
http://publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/IEA2G133/4.58.1?S
HELF=EZ2CMZ34&DT=20031016134357
What command was issued to cause the problem in the first place? What
console messages did you get?
Regards,
Ulrich Krueger
-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@BAMA.UA.EDU] On Behalf
Of Yukus, Mary J CIV USMEPCOM
Sent: Monday, March 31, 2008 09:21
To: IBM-...@BAMA.UA.EDU
Subject: SYSLOG problem
Look at the WRITELOG command in MVS to force the syslog to spool and start another one.
To have the hardcopy message set recorded on the system log, enter:
V SYSLOG,HARDCPY
This should be all you need.
Lizette
Try command 'W START'
Jerry
"Yukus, Mary J CIV USMEPCOM" <mary....@MEPCOM.ARMY.MIL>
Sent by: IBM Mainframe Discussion List <IBM-...@BAMA.UA.EDU>
03/31/2008 01:04 PM
Please respond to
IBM Mainframe Discussion List <IBM-...@BAMA.UA.EDU>
Subject
SYSLOG problem
>Hi Everyone,
>We have an issue with our syslog. The command was issued to stop the syslog
>to write it out. Normally a new syslog is created. Unfortunately this time
>that didn't happen. We are now using the logger data set on this partition,
>just the hard copy. Does anyone know how to get a new syslog out there?
>This is on z/OS 1.4.
>Thanks,
>Mary
V SYSLOG,HARDCPY
--
Mark Zelden
Sr. Software and Systems Architect - z/OS Team Lead
Zurich North America / Farmers Insurance Group - ZFUS G-ITO
mailto:mark....@zurichna.com
z/OS Systems Programming expert at http://expertanswercenter.techtarget.com/
Mark's MVS Utilities: http://home.flash.net/~mzelden/mvsutil.html
----------------------------------------------------------------------
-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@BAMA.UA.EDU] On Behalf
Of Yukus, Mary J CIV USMEPCOM
Sent: Monday, March 31, 2008 11:21 AM
To: IBM-...@BAMA.UA.EDU
Subject: SYSLOG problem
Since CSA/ECSA does not overflow to SQA/ESQA, a sufficiently
sized SQA/ESQA prevents CSA/ECSA exhaustion from causing SQA/ESQA
exhaustion.
So a storage leak in CSA/ECSA or runaway allocator of CSA/ECSA wouldn't
cause SQA/ESQA exhaustion. Whether or not this provides any signicant
benefit to system reliability is difficult to say. My guess would be that
if you exhaust CSA/ESCA your system is likely to be doomed anyway.
Also, overflow of SQA/ESQA to CSA/ECSA may result in slightly longer
pathlengths for some request to allocate and free SQA/ESQA storage.
Whether this would result in any measurable performance degradation
with your workload would be difficult to say. My guess would be that
the degradation would be insignificant. It might be possible to
construct an artificial testcase workload where the degradation is
measurable.
Jim Mulder z/OS System Test IBM Corp. Poughkeepsie, NY
The WRITELOG START did the trick.
Thanks, all you help is greatly appreciated.
Mary :-)
--
Peter Hunkeler
CREDIT SUISSE
We ended up doing a W START followed by a VARY SYSLOG,HARDCPY,CMDS to
finally get it working.
-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@BAMA.UA.EDU] On Behalf
Of Hunkeler Peter (KIUK 3)
Sent: Tuesday, April 01, 2008 6:14 AM
To: IBM-...@BAMA.UA.EDU
Subject: Re: SYSLOG problem
If you haven't automated this fairly basic and universal process, you may
find other opportunities as well for simple 'headache relief'. ;-)
.
.
JO.Skip Robinson
Southern California Edison Company
Electric Dragon Team Paddler
SHARE MVS Program Co-Manager
626-302-7535 Office
323-715-0595 Mobile
JO.Skip....@sce.com
"Yukus, Mary J
CIV USMEPCOM"
<mary.yukus@MEPCO To
M.ARMY.MIL> IBM-...@BAMA.UA.EDU
Sent by: IBM cc
Mainframe
Discussion List Subject
<IBM-...@BAMA.UA Re: SYSLOG problem
.EDU>
04/01/2008 07:00
AM
Please respond to
IBM Mainframe
Discussion List
<IBM-...@BAMA.UA
.EDU>