Delete part of spool for active job

[Hilary Hurwitz]

If a job is running and putting out many lines of spool - maybe there was an over active programmer that printed all details of every customer in a multimillion line file - is there a way of deleting the lines that have been written to the spool ?

Or do I have to purge the job ?

Hilary Hurwitz
NIOI

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to list...@listserv.ua.edu with the message: INFO IBM-MAIN

[Lizette Koehler]

There are a couple of options,

If the JCL has coded on the SYSOUT FREE=CLOSE,SPIN=UNALLOC then you can purge
just that portion of the job without impacting anything else.

You can use SDSF (if you have that) and change the one DD statement that has the
huge spool output to a different class (There are JES2 $TO commands also) and
then route the output from the rest of the job to the destinations they need to
go. Once all the good parts have been handled, you can then purge the offending
piece of output

You can use the JES2 XMIT/RECEIVE process to offload the whole job or any part
to a tape/dasd dataset
You can use SDSF (if you have it) to print (XDC commands) to a dataset to save
the good parts and then purge the offending section.

There may be some other options I am missing. IMO - the FREE=CLOSE,SPIN=UNALLOC
is a good standard for SYSOUT datasets.

So it will depend on how much effort that is needed on doing this. To save all
of the job with the exception of this one SYSOUT will require a little effort.

Do you use SDSF? Or have you researched the $TO commands?

Lizette

[Hilary Hurwitz]

We have SDSF

I am not familiar with $TO

(I am actually a DBA, not a SYSPROG)

<p> <u1:p></u1:p></p><p class="MsoNormal"><b><span style="color: rgb(0, 102, 0);"><font face="Arial, Helvetica, sans-serif" size="2">This message was scanned by the National Insurance of Israel mail protection system</font></span></b><o:p></o:p></p> <u1:p></u1:p>

[Lizette Koehler]

So, you may wish to contact your z/OS Sysprog for assistance. They will
probably be able to help.

So in SDSF you go to the ST display for the job and do a ? next to it. It will
show you the Jes2 Dataset Numbers. You can then use that for the $TO commands
as needed.

But talking to your Sysprog would be a good thing.

[Shmuel Metz , Seymour J.]

In
<6943F159376140438E93A...@NioiMbx1.mr.nioi.gov.il>,
on 12/17/2015

at 01:13 PM, Hilary Hurwitz <hil...@NIOI.GOV.IL> said:

>is there a way of deleting the lines that have been written to the
>spool ?

That depends. Does your DD statement specify SEGMENT=foo?
SPIN=UNALLOC? How is your output class defined?

--
Shmuel (Seymour J.) Metz, SysProg and JOAT
ISO position; see <http://patriot.net/~shmuel/resume/brief.html>
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)

[Skip Robinson]

There are two distinct conditions that must be considered.

1. A large output needs to be managed but is not causing a spool problem.
2. An ginormous output--together with benign output--has pushed spool
utilization to 100%.

In the case of #1, others have suggested ways to preserve at least some of
the output. Basically, assuming that the JCL was not coded to anticipate
this situation, you need to copy some portion of the output to another class
or destination. You could also save output to a data set. All spool
management tools I know of have this capability. Once desired output has
been preserved outside the jumbo job, El Gordo can be purged.

In the case of #2, your options are limited. If spool shortage cannot be
relieved, you may not even be able to logon to the MAS because even a TSO
address space needs some spool space. I don't believe that you can purge an
individual segment while a job is still writing to it. When spool-full
condition has hit us, the offending job is usually still running, so that
space recovered by purging some disposable output is immediately filled up
again by the same job. If this situation occurs, you have little choice but
to cancel *and purge* the offending job, which unfortunately means losing
all the output.

.
.
.
J.O.Skip Robinson
Southern California Edison Company
Electric Dragon Team Paddler
SHARE MVS Program Co-Manager
323-715-0595 Mobile
JO.Skip....@att.net
JO.Skip....@gmail.com

> > - is there a way of deleting the lines that have been written to the
spool ?
> >

> > Or do I have to purge the job ?
> >
> > Hilary Hurwitz
> > NIOI

[Gibney, David Allen,Jr]

In the case of #2, and if you are an E(JES) customer, and you run their standalone VTAM interface, then logging on to TSO with a full spool is not an issue.

> -----Original Message-----
> From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU]
> On Behalf Of Skip Robinson
> Sent: Thursday, December 17, 2015 11:04 AM
> To: IBM-...@LISTSERV.UA.EDU
> Subject: Re: Delete part of spool for active job
>

[Elardus Engelbrecht]

Skip Robinson wrote:

>There are two distinct conditions that must be considered.

>1. A large output needs to be managed but is not causing a spool problem.
>2. An ginormous output--together with benign output--has pushed spool utilization to 100%.

>In the case of #1, others have suggested ways to preserve at least some of the output. Basically, assuming that the JCL was not coded to anticipate this situation, you need to copy some portion of the output to another class or destination. You could also save output to a data set. All spool management tools I know of have this capability. Once desired output has been preserved outside the jumbo job, El Gordo can be purged.

True. That is if the output is deallocated or freed by the offending program in the first place while that program is running.

>In the case of #2, your options are limited. If spool shortage cannot be relieved, you may not even be able to logon to the MAS because even a TSO address space needs some spool space.

True. I still have the scars of that, luckily I have a spare TSO session where I could fix those guzzling jobs properly. (and also sorted out an El Gordo XMIT data on the JES2 SPOOL, which guzzled up the whole spool. I deleted that and showed that grumbling XMITter how to do FTP instead of XMIT.)

And trained that submitter of those jobs to insert a SYSOUT FREE=CLOSE,SPIN=UNALLOC before re-submitting that offending job.

Or teach them to redirect the output to a dataset.

Hard work to train those dummies... ;-D

>I don't believe that you can purge an individual segment while a job is still writing to it.

True. Also while that step is still writing to it. Wait for the step to free or deallocate the output and then you can purge it.

>When spool-full condition has hit us, the offending job is usually still running, so that space recovered by purging some disposable output is immediately filled up again by the same job. If this situation occurs, you have little choice but to cancel *and purge* the offending job, which unfortunately means losing all the output.

Sometimes the only option. Just preserve the 'x millions lines written' in the SYSLOG to pacify the angry submitter...

I sometimes threatened to write that JES2 exit to limit/crush those spool eating jobs outputs, but my users cancelled my obsession to keep the JES2 spool empty... ;-)

Groete / Greetings
Elardus Engelbrecht

[Peter Hunkeler]

In a case # 2 situation, you could force the job to be swapped out unconditionally by quiescing it: E jobname,QUIESCE. This might give you some time to purge enough unneeded output to be able to login to TSO and perform further actions. Actions might be to save the output to a data set then cancel the job or allocate new spool space and let the job continue (E jobname,RESET), or whatever suits your needs.


--
Peter Hunkeler

[Skip Robinson]

(I could start a new thread if needed.)

We are reviewing our automated response to 'full spool' condition because it's very disruptive and not always recognized right away. I'd like to purge an offending job. $DSPL,JOBS=nn will report jobs using more than nn per cent of spool. Problem is that the response to a REXX CONSOLE command does not include job number. If more than one job of the same name is found, $PJ will not work without the number. Any advice?

.
.
.
J.O.Skip Robinson
Southern California Edison Company
Electric Dragon Team Paddler
SHARE MVS Program Co-Manager
323-715-0595 Mobile
JO.Skip....@att.net
JO.Skip....@gmail.com

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On Behalf Of Elardus Engelbrecht
Sent: Thursday, December 17, 2015 11:52 AM
To: IBM-...@LISTSERV.UA.EDU
Subject: [Bulk] Re: Delete part of spool for active job

[Elardus Engelbrecht]

Skip Robinson wrote:

>We are reviewing our automated response to 'full spool' condition because it's very disruptive and not always recognized right away. I'd like to purge an offending job. $DSPL,JOBS=nn will report jobs using more than nn per cent of spool. Problem is that the response to a REXX CONSOLE command does not include job number. If more than one job of the same name is found, $PJ will not work without the number. Any advice?

What about checking on SYSLOG this line? (Filter for jobname and jobnr as needed)

J<job nr> 00000281 $HASP375 <jobname> ESTIMATED LINES EXCEEDED BY 850,000 LINES

And then you could use this command to fix that spool-eater:

-$DJ '???',SPOOL
J0093725 $HASP890 JOB(???)
$HASP890 JOB(???) SPOOL=(VOLUMES=(?????,03),TGS=2,
$HASP890 PERCENT=0.0001)

Just wondering...

[nitz-ibm]

> In a case # 2 situation, you could force the job to be swapped out unconditionally by quiescing it: E jobname,QUIESCE. This might give you some time to purge enough unneeded output to be able to login to TSO and perform further actions. Actions might be to save the output to a data set then cancel the job or allocate new spool space and let the job continue (E jobname,RESET), or whatever suits your needs.

Good idea, but have you had only a z/OS console in a 100% spool shortage recently? Problems I encountered in that situation:
- Logon not possible
- The system's console had an American keyboard layout (on a really stupid 3270 emulation that doesn't know the difference between ctrl and enter, remote login) and my actual keyboard is German: Just issuing the JES2 commands with all its special characters to determine *which* was the offending job became a nightmare.
- Cancel command of offending job got accepted, but the job did not terminate (because it needs spool space to terminate - TGSs were at 100%)
- Job had to get forced (force arm also didn't work).
- I am used to using SDSF, so using the JES2 commands (aside from the keyboard issue) to purge the offending job was a real challenge.

The first time I hit these problems, I learned my lesson:
1. Always keep a spare spool volume (or two) around and have written notes on how to activate that. Also, keep in mind that some JES shortages (I forgot which) prevent the addition of new volumes, IIRC because that also needs *some* spool space. (An ADCD system comes with a minuscule spool, and I went through some iterations for increasing check point sizes, too, to accomodate the larger spool.)
2. Always keep some older output around that can be purged so that the TSO logon can succeed (even if I understand the zeal to have the JES2 spool as empty as possible). Always attempt to logon before purging anything. If you're able to logon, first save the offending output (even if it gets a B37) so you have some chance of problem determination.
3. All else failing, have an established procedure for a JES2 cold start, *especially* if you run a monoplex.
4. Have automation procedures in place to cancel a job consuming more than 50% of spool space (I had to change the JES init parms to accomplish that).
5. Have automation procedures in place to report shortages before they hit 100% (jobs are not executed anymore or new address spaces started once you're in 100%).

The primary goal in this situation is always to logon to TSO again. Some installations have a local non-SNA TSO user that is always running and never logs off. At least that allows for easier problem analysis (aside from the keyboard issue), but I was unlucky enough having to purge a job that the logged-on userid didn't have RACF access to, and the userid didn't have RACF SPECIAL, either.

I firmly believe that Murphy's law applies in any JES2 shortage! :-)


Barbara

[Jousma, David]

Skip,

Here is what I devised, and has worked pretty well. We use netview/SA390 for our automation. The following rules get tripped at 80% or 85% utilization.

SPOOLSHORT :
CMD=(,,'MVS $D JOBQ,SPOOL=(%>01)')
CMD=(PASS1,,'MVS $CJ(E*,T*),SPOOL=(%>10),PURGE') << in our shop E* and T* are non-prod jobs, and this usually kills the culprit
CMD=(PASS2,,'MVS $PJ(E*,T*),SPOOL=(%>5)')
CMD=(PASS3,,'MVS $PJ(E*,T*),SPOOL=(%>2)')
CMD=(PASS4,,'MVS $PO JOBQ,READY,A>14')
CMD=(PASS5,,'MVS $PO TSU(*),ALL,A>1')
CMD=(PASS6,,'MVS $PO STC(*),ALL,A>10')
CMD=(PASS7,,'MVS $PO JOBQ,Q=D,A>1')
CMD=(PASS8,,'MVS $PO JOBQ,ALL,A>10')

_________________________________________________________________
Dave Jousma
Assistant Vice President, Mainframe Engineering
david....@53.com
1830 East Paris, Grand Rapids, MI  49546 MD RSCB2H
p 616.653.8429
f 616.653.2717

This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated.

[Elardus Engelbrecht]

Jousma, David wrote:

>Here is what I devised, and has worked pretty well. We use netview/SA390 for our automation. The following rules get tripped at 80% or 85% utilization.

Good solution, but Skip still wants the job number. Can your Netview solution also handle jobnumber?

CMD=(,,'MVS $D JOBQ,SPOOL=(%>01)')

Hmmm. It differs from Skip's command $DSPL,JOBS=nn, but job numbers are also shown on the SYSLOG together with the other details. It should help Skip very much.

I prefer this one $D JOBQ,SPOOL=(TGS>nnnn) because some of our jobs with say TGS = large are only occupying TGS% < 1. (I know you can do this $D JOBQ,SPOOL=(%>0.5) for % smaller than 1.)

Just a little sample on my SDSF showing my spool eaters:

TGNum TGPct
4925 0.43
6673 0.58
7294 0.63

> CMD=(PASS1,,'MVS $CJ(E*,T*),SPOOL=(%>10),PURGE') << in our shop E* and T* are non-prod jobs, and this usually kills the culprit

Excellent! I will tell my z/OS team about this killer! ;-D

Granted, also no job numbers, but they want a solution based on job name.

Thanks!

[Jousma, David]

Yea, really don’t care what the job numbers is in my case. I care that a job is eating more spool than it should, and want to kill it. Not sure why getting the job number is a requirement I guess.

_________________________________________________________________
Dave Jousma
Assistant Vice President, Mainframe Engineering
david....@53.com
1830 East Paris, Grand Rapids, MI  49546 MD RSCB2H
p 616.653.8429
f 616.653.2717


-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On Behalf Of Elardus Engelbrecht
Sent: Friday, December 18, 2015 7:50 AM
To: IBM-...@LISTSERV.UA.EDU
Subject: Re: Delete part of spool for active job

This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated.

[Shmuel Metz , Seymour J.]

In
<4EE2851A2279B94CB70CD...@S1FLOKYDCE2KX01.dm0001.info53.com>,
on 12/18/2015

at 06:41 PM, "Jousma, David" <David....@53.COM> said:

>Yea, really don t care what the job numbers is in my case.

You don't care if you purge the wrong job?

--
Shmuel (Seymour J.) Metz, SysProg and JOAT
ISO position; see <http://patriot.net/~shmuel/resume/brief.html>
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)

[Skip Robinson]

Purging the wrong job would be sad, but alas, if you try to purge a job by
name only, and there is more than one match in spool, JES2 will do nothing
other than tell you what a doofus you are for being so vague. I think I have
the answer thanks to IBM-Main advice on and off list. $D SPOOL is the wrong
command. What's needed is $D JOBQ, which returns job number(s) that can then
be acted on properly. Thanks to all who contributed to this denouement.

.
.
.
J.O.Skip Robinson
Southern California Edison Company
Electric Dragon Team Paddler
SHARE MVS Program Co-Manager

626-302-7535 Office
323-715-0595 Mobile
JO.Skip....@att.net
JO.Skip....@gmail.com

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On

Behalf Of Shmuel Metz (Seymour J.)
Sent: Friday, December 18, 2015 12:44 PM
To: IBM-...@LISTSERV.UA.EDU
Subject: [Bulk] Re: Delete part of spool for active job

In
<4EE2851A2279B94CB70CD...@S1FLOKYDCE2KX01.dm0001.info53.co

m>,
on 12/18/2015
at 06:41 PM, "Jousma, David" <David....@53.COM> said:

>Yea, really don t care what the job numbers is in my case.

You don't care if you purge the wrong job?

--
Shmuel (Seymour J.) Metz, SysProg and JOAT

[Jousma, David]

When I am in a situation, where spool is short, and any of these commands below are executed, then no, I don't care about the job number, because that is not how the candidate jobs are getting selected and cancelled/purged.

CMD=(PASS1,,'MVS $CJ(E*,T*),SPOOL=(%>10),PURGE')

CMD=(PASS2,,'MVS $PJ(E*,T*),SPOOL=(%>5)')
CMD=(PASS3,,'MVS $PJ(E*,T*),SPOOL=(%>2)')

_________________________________________________________________
Dave Jousma
Assistant Vice President, Mainframe Engineering
david....@53.com
1830 East Paris, Grand Rapids, MI  49546 MD RSCB2H
p 616.653.8429
f 616.653.2717

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On Behalf Of Shmuel Metz (Seymour J.)
Sent: Friday, December 18, 2015 3:44 PM
To: IBM-...@LISTSERV.UA.EDU
Subject: Re: Delete part of spool for active job

This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated.

[wjip...@gmail.com]

@Skip, if you are using a REXX anyway, then the SDSF API 'DA' call can give you the jobnr of the running job.