Dataset enqueue, how to find the culprit.

[Toni Cecil]

Hello,
yesterday I got the following dsn enqueue:
IKJ56225I DATA SET TLF.ZPT.JCL.CNTL ALREADY IN USE, TRY LATER+
IKJ56225I DATA SET IS ALLOCATED TO ANOTHER JOB OR USER
SDAA004I - RETURN=(DD),PERM=YES DSN=TLF.ZPT.JCL.CNTL(ZZZTXXXB),
DISP=SHR
SDAB005I - ERR=0210, INFO=0000, REQUESTED DATA SET NOT AVAILABLE.
SDAB005I ALLOCATED TO ANOTHER JOB.

Is there a way to find today, who was "locking" the pds library ?? I run
DAF tool against TLF.ZPT.JCL.CNTL to see if something was shown about the
enqueue, but I didn't find anything, am I doing something wrong ?? Or is
there any other utility that could be more appropriate to check this
"problem" ??

Many thx, A.Cecilio.

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to list...@listserv.ua.edu with the message: INFO IBM-MAIN

[Jousma, David]

One of the quickest ways, is to go to ISPF option 3.4, and pull up that dataset, and do a "R" to rename it(don’t really rename it though). When it says 'dataset in use", hit PF1 and you get a list of jobs with enqueues on it.

_________________________________________________________________
Dave Jousma
Assistant Vice President, Mainframe Engineering
david....@53.com
1830 East Paris, Grand Rapids, MI  49546 MD RSCB2H
p 616.653.8429
f 616.653.2717

This e-mail transmission contains information that is confidential and may be privileged. It is intended only for the addressee(s) named above. If you receive this e-mail in error, please do not read, copy or disseminate it in any manner. If you are not the intended recipient, any disclosure, copying, distribution or use of the contents of this information is prohibited. Please reply to the message immediately by informing the sender that the message was misdirected. After replying, please erase it from your computer system. Your assistance in correcting this error is appreciated.

[Jousma, David]

Sorry, you have to hit PF1 twice to get the display I mentioned.

[Lizette Koehler]

If you have RMF setup for ISPF, and the RMF VSAM files are available, you can use the ISPF RMF tool to go back in time for the enqueue information.

Did you look in syslog for any GRS information?
Did you go and look and see what other jobs were running when this job had this issue?


Lizette

> -----Original Message-----
> From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On
> Behalf Of Toni Cecil
> Sent: Friday, September 18, 2015 9:21 AM
> To: IBM-...@LISTSERV.UA.EDU
> Subject: Dataset enqueue, how to find the culprit.
>

[Lizette Koehler]

That only works if the file is allocated now.

I think the OP wants to go back in time to find the enqueue. The enqueue happened yesterday and today I want to know who had the file.


Lizette

[Richard Pinion]

The old RMF TSO command RMFMON suboption SENQR will also tell you
this information.


--- star...@MINDSPRING.COM wrote:

From: Lizette Koehler <star...@MINDSPRING.COM>
To: IBM-...@LISTSERV.UA.EDU
Subject: Re: Dataset enqueue, how to find the culprit.

Date: Fri, 18 Sep 2015 09:24:56 -0700

If you have RMF setup for ISPF, and the RMF VSAM files are available, you can use the ISPF RMF tool to go back in time for the enqueue information.

Did you look in syslog for any GRS information?
Did you go and look and see what other jobs were running when this job had this issue?

Lizette


> -----Original Message-----
> From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On
> Behalf Of Toni Cecil
> Sent: Friday, September 18, 2015 9:21 AM
> To: IBM-...@LISTSERV.UA.EDU
> Subject: Dataset enqueue, how to find the culprit.
>
> Hello,
> yesterday I got the following dsn enqueue:
> IKJ56225I DATA SET TLF.ZPT.JCL.CNTL ALREADY IN USE, TRY LATER+ IKJ56225I DATA
> SET IS ALLOCATED TO ANOTHER JOB OR USER SDAA004I - RETURN=(DD),PERM=YES
> DSN=TLF.ZPT.JCL.CNTL(ZZZTXXXB), DISP=SHR SDAB005I - ERR=0210, INFO=0000,
> REQUESTED DATA SET NOT AVAILABLE.
> SDAB005I ALLOCATED TO ANOTHER JOB.
>
> Is there a way to find today, who was "locking" the pds library ?? I run DAF tool
> against TLF.ZPT.JCL.CNTL to see if something was shown about the enqueue, but I
> didn't find anything, am I doing something wrong ?? Or is there any other utility that
> could be more appropriate to check this "problem" ??
>
> Many thx, A.Cecilio.
>

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to list...@listserv.ua.edu with the message: INFO IBM-MAIN

_____________________________________________________________
Netscape. Just the Net You Need.

[Steely, Mark]

I created a REXX called WHI (who has it). Here is the code:

PROC 1 DSN
/* CLRSCRN */
ISRDDN E &DSN

Then in 3.4 enter WHI and it will display any enqueue.

Thanks

[J O Skip Robinson]

As others have pointed out, enqueue is often a fleeting problem. By the time you get around to looking into it, it's long gone. After experiencing this problem a while back, we instructed our automation product to issue a D GRS command at the time of the conflict based on msgid IKJ56225I.

IKJ56225I DATA SET already-in-use-dataset ALREADY IN USE, TRY LATER+

IKJ56225I DATA SET IS ALLOCATED TO ANOTHER JOB OR USER

D GRS,RES=(*, already-in-use-dataset)

If you decide to implement a similar process, be aware of some gotchas. (1) The msgid appears twice. If you're not careful, you'll be issuing D GRS for dataset 'IS'. Not a big deal, but it looks squirrelly. (2) The command itself might be issued too late to capture the contention. It can happen, for instance, that a task is actually enqueuing on itself (!). If the task terminates upon the enqueue, D GRS will find no contention. As odd as this sounds, we have seen cases like this.

.
.
.
J.O.Skip Robinson
Southern California Edison Company
Electric Dragon Team Paddler
SHARE MVS Program Co-Manager
626-302-7535 Office
323-715-0595 Mobile
JO.Skip....@sce.com

[Lizette Koehler]

So, it will depend on how often the TSO message for enqueue is produced. If it is not frequent, you may have to set slips or live with rerunning the job.

Or if it is frequent, you might try adding a first step with DISP=OLD on your file. then you would have a better idea of what is doing it or have the job wait until the file is freed.

You did not show what type of process is being used when this occurs. So if this is REXX/CLIST that does LM functions, or an IEBCOPY, or other????

Perhaps if you could describe the process and functions being used that this condition occurred, we might provide better diags or alternatives.



Lizette

[Richard Pinion]

I think this setting for RMF will cut an SMF record for enqueue conditions.
RMF must be started pointing to the member which specifies ENQ(DETAIL).

EDIT USER.PARMLIB(ERBRMF00) - 01.05 Columns 00001 00080
Command ===> Scroll ===> CSR
000014 ENQ(DETAIL)

--- star...@MINDSPRING.COM wrote:

From: Lizette Koehler <star...@MINDSPRING.COM>
To: IBM-...@LISTSERV.UA.EDU
Subject: Re: Dataset enqueue, how to find the culprit.

_____________________________________________________________
Netscape. Just the Net You Need.

[J O Skip Robinson]

Well I'll be darned. I had not looked carefully at OP's message. That is *exactly* the context of the enqueue case I posted generically. SDAA004I is NDM or ConnectDirect depending on your perspective. We seem to have a lot of these self-clashes in our environment. I have not talked with the affected users, but it appears from the NDM control statements echoed in syslog that they are trying to send a data set from SystemX to SystemX with the same name. I suspect that they are requesting exclusive control of the 'target', i.e. OLD. But NDM is already using the data set as 'source', so it cannot obtain an exclusive enqueue for the same name. When the enqueue fails, the NDM process terminates, so a D GRS command a split second later shows no enqueue. That's how our (ancient) version of NDM works.

OP's situation may not be so simple, but I suggest looking the NDM process to see if there's a logic problem.

.
.
.
J.O.Skip Robinson
Southern California Edison Company
Electric Dragon Team Paddler
SHARE MVS Program Co-Manager
626-302-7535 Office
323-715-0595 Mobile
JO.Skip....@sce.com

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On Behalf Of Lizette Koehler
Sent: Friday, September 18, 2015 11:17 AM
To: IBM-...@LISTSERV.UA.EDU

[Barry Merrill]

You can enable RMF Monitor I ENQ(DETAIL) option to create
SMF 77 ENQUE records, which reports all conflicts,
including SYSDSN.

Barry


Herbert W. “Barry” Merrill, PhD
President-Programmer
MXG Software
Merrill Consultants
10717 Cromwell Drive
Dallas, TX 75229-5112
Ba...@MXG.COM
Fax: 214 350 3694 – Still works, received as email
Tel: 214 351 1966 – Unreliable, please use email

www.mxg.com HomePage: FAQ answers most questions
ad...@mxg.com License Forms, Invoice, Payment, ftp information
sup...@mxg.com Technical Issues
MXG-L FREE ListServer http://www.mxg.com/mxg-l_listserver/

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On Behalf Of Toni Cecil
Sent: Friday, September 18, 2015 11:21 AM
To: IBM-...@LISTSERV.UA.EDU
Subject: Dataset enqueue, how to find the culprit.

[Ted MacNEIL]

There is NO SUCH THINGS as a culprit!
They are just doing their job and so are you. They just happened to collide.

-
-teD
-
  Original Message  
From: Richard Pinion
Sent: Friday, September 18, 2015 14:31
To: IBM-...@LISTSERV.UA.EDU
Reply To: IBM Mainframe Discussion List

[Greg Shirey]

According to Merriam-Webster online dictionary, one definition of "culprit" is "the source or cause of a problem."

I suppose now there needs to be a discussion about whether the OP's situation could be defined as a problem...

Regards,
Greg Shirey
Ben E. Keith Company

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On Behalf Of Ted MacNEIL
Sent: Friday, September 18, 2015 1:56 PM
To: IBM-...@LISTSERV.UA.EDU
Subject: Re: Dataset enqueue, how to find the culprit.

There is NO SUCH THINGS as a culprit!
They are just doing their job and so are you. They just happened to collide.

[Toni Cecil]

Hello everyone,
I wnat to know how to debug this situation after it occurred, in this case,
the next day. In this particular case Connect Direct was unable to pick
this library for a transmission task becuase it was in use by someone (user
ou job).
I did run RMFMONIII DSND "Data set delays" with 100sec sampling moving
forward and rewinding on time and I coundn't find anything.
But basically, what tool do you normally use for a situation like this ???
to find the enqueue after the problem take place ??

Many thx for all your sugestions, A.Cecilio.

[Richard Pinion]

Run RMF, I think monitor I, with the following option. This create
an RMF SMF enqueue record.

VIEW USER.PARMLIB(ERBRMF00) - 01.05
Command ===>
000014 ENQ(DETAIL)

Then run the RMF postprocessor as below.

//STEP010 EXEC PGM=ERBRMFPP
//MFPINPUT DD DISP=SHR,DSN=your.smf.data (Could be the currently active MANx, or an SMF offload data set.
//MFPMSGDS DD SYSOUT=*
//SYSIN DD *
SYSOUT (X)
DATE(mmddyyyy,mmddyyyy)
NOSUMMARY
REPORTS(ENQ)



--- acb...@GMAIL.COM wrote:

From: Toni Cecil <acb...@GMAIL.COM>
To: IBM-...@LISTSERV.UA.EDU
Subject: Re: Dataset enqueue, how to find the culprit.

_____________________________________________________________
Netscape. Just the Net You Need.

[Roland Kinsman]

Very nice, Mark Steely. Consider WHI stolen!

[Joseph W Gentile]

Hi, you could also run ISGECJES from SAMPLIB (z/OS 2.1). It is a simple
dataset enqueue contention monitor written in assembler.
It won't go back in time. It will alert you to dataset contention going
forward. We wrote an article about it in z/OS Hot Topics Issue 27, page
54. http://publibfp.dhe.ibm.com/epubs/pdf/e0z3n100.pdf

-Joe

Joe Gentile
z/OS GRS Lead
(845)435-2184 (T/L 295-2184)
jwge...@us.ibm.com

[Dave Barry]

The problem is that the SVC99 dynamic allocation is unconditional. If the resource is already owned, the subsequent allocation request fails. At that point there is no enqueue conflict to show in any monitor because no exclusive or shared waiter was added for that resource in the GRS list.

Static allocation is an option because allocation services causes the job to wait (indefinitely) for enqueued datasets named in your JCL.

This is a perennial issue that begs to be resolved.

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On Behalf Of Joseph W Gentile
Sent: Tuesday, September 22, 2015 11:04 AM
To: IBM-...@LISTSERV.UA.EDU
Subject: Re: Dataset enqueue, how to find the culprit.

[Mike Schwab]

How about checking who has that dataset if the allocation fails?

--
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

[Paul Gilmartin]

On 2015-09-23 11:54, Dave Barry wrote:
> The problem is that the SVC99 dynamic allocation is unconditional. If the resource is already owned, the subsequent allocation request fails. ...
>
Not if the requesting program is entitled to use S99WTDSN.

> Static allocation is an option because allocation services causes the job to wait (indefinitely) for enqueued datasets named in your JCL.
>

True.

> This is a perennial issue that begs to be resolved.
>

At least RACF should prevent a program's obtaining an ENQ when it
has no authority to access the associated resource.

I know; two different countries, and the gulf between them is
narrow but deep.

-- gil

[Joseph W Gentile]

Good point. Some SVC99's specify not to wait for the ENQ (S99CNENQ I
think). GRS considers this "waitless" contention. GRS also issues ENF 51
signals for contention (including waitless contention). RMF uses these
signals to report on contention, so you may be able to track waitless
contention through RMF.

-Joe

Joe Gentile
z/OS GRS Lead
(845)435-2184 (T/L 295-2184)
jwge...@us.ibm.com

[Paul Gilmartin]

On 2015-09-18, at 10:47, Steely, Mark wrote:

> I created a REXX called WHI (who has it). Here is the code:
>
> PROC 1 DSN
> /* CLRSCRN */
> ISRDDN E &DSN
>
> Then in 3.4 enter WHI and it will display any enqueue.
>

And it's better than the RENAME approach in that it identifies
the culprit LPAR.

But it brings up a GUI. Is there a command line interface;
something I might invoke remotely via ssh?

The ISPF doc I find for ISRDDN is meager; it doesn't describe
the arguments (such as "E" and "&DSN"). Is there a way to
specify QNAME, RNAME and OWNER on that command line?

Thanks,
gil

[Pommier, Rex]

Hi Gil,

I don't have time to play with it right now, but short answer to your question is "yes". I got this from the help screen within ISRDDN. Unfortunately it doesn't give syntax on how to actually use the parms.

Commands for viewing ENQs and ENQ contention:

o Enq
Show existing ENQs on the system or systems. You can specify RNAME,
QNAME, system name and job or user name to narrow down the search.

o Con
Show existing ENQ contention.


Rex

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On Behalf Of Paul Gilmartin
Sent: Thursday, September 24, 2015 8:18 AM
To: IBM-...@LISTSERV.UA.EDU
Subject: Re: Dataset enqueue, how to find the culprit.

The information contained in this message is confidential, protected from disclosure and may be legally privileged. If the reader of this message is not the intended recipient or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any disclosure, distribution, copying, or any action taken or action omitted in reliance on it, is strictly prohibited and may be unlawful. If you have received this communication in error, please notify us immediately by replying to this message and destroy the material in its entirety, whether in electronic or hard copy format. Thank you.

[Paul Gilmartin]

On Thu, 24 Sep 2015 14:33:03 +0000, Pommier, Rex wrote:
>
>Commands for viewing ENQs and ENQ contention:
>
>o Enq
> Show existing ENQs on the system or systems. You can specify RNAME,
> QNAME, system name and job or user name to narrow down the search.

> ...
Thanks. I fear that "You can specify ..." means that when ISRDDN displays
a panel that panel will contain fields in which to enter those options, not
that they may be entered (from Rexx) on the command line.

Oh, I forgot to mention that I'd like to be able to OUTTRAP (or similar)
the result for further processing. ZSCREEN? Probably not; my process
runs headless. GUIs are often overrated.

Thanks again,

[Anthony Thompson]

For trapping the results in a REXX you could use the SDSF ISFSLASH service to capture the results of a D GRS, RES= or D GRS,C command. I imagine you could use the TSO CONSOLE command too. Assuming you have the authority for either...

Ant.

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On Behalf Of Paul Gilmartin
Sent: Friday, 25 September 2015 1:44 AM
To: IBM-...@LISTSERV.UA.EDU
Subject: Re: Dataset enqueue, how to find the culprit.

[Alfred Singlestone]

On Friday, September 18, 2015 at 12:47:56 PM UTC-4, Steely, Mark wrote:
> I created a REXX called WHI (who has it). Here is the code:
>
> PROC 1 DSN
> /* CLRSCRN */
> ISRDDN E &DSN
>
> Then in 3.4 enter WHI and it will display any enqueue.

Or a command table entry:

WHOHAS 0 SELECT CMD(ISRDDN E &ZPARM)

[Graham Harris]

MXI has comprehensive ENQ query capabilities, and a very usable REXX
interface as well.

On 25 September 2015 at 01:44, Anthony Thompson <Anthony....@nt.gov.au>
wrote:

[Gereldy]

I like the variety of ways folks have offered for finding the holder(s) of the resource for which the OP's enqueue failed.

I would just like to see one of the holders included in the message. Quite often, there is just one. If multiple, I don't know how best to decide which one would be displayed (most recent, oldest, best looking, ...) but give me one please!

[Dave Barry]

I seem to recall a recent enhancement to z/FTP dynamic allocation that reports contention and retries some number of times at user-defined intervals. TSO ALLOC and IDCAMS might take that as an example for future development.

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On Behalf Of Paul Gilmartin
Sent: Wednesday, September 23, 2015 3:58 PM
To: IBM-...@LISTSERV.UA.EDU
Subject: Re: Dataset enqueue, how to find the culprit.

[Paul Gilmartin]

On Mon, 28 Sep 2015 18:35:07 +0000, Dave Barry wrote:

>I seem to recall a recent enhancement to z/FTP dynamic allocation that reports contention and retries some number of times at user-defined intervals. TSO ALLOC and IDCAMS might take that as an example for future development.
>

So instead of deadlocks you can have livelocks? You can code that loop yourself.

[John McKown]

On Mon, Sep 28, 2015 at 3:54 PM, Paul Gilmartin <
0000000433f0781...@listserv.ua.edu> wrote:

> On Mon, 28 Sep 2015 18:35:07 +0000, Dave Barry wrote:
>
> >I seem to recall a recent enhancement to z/FTP dynamic allocation that
> reports contention and retries some number of times at user-defined
> intervals. TSO ALLOC and IDCAMS might take that as an example for future
> development.
> >
> So instead of deadlocks you can have livelocks? You can code that loop
> yourself.
>

​Perhaps what is needed is a new option on the ENQ. Normally, when you do
an ALLOCATE and the DSN is unavailable, you get an immediate failure. OTOH,
there is a parameter in the SVC 99 control block which allows an APF
authorized program to WAIT "forever". Which basically leads to the thought
that it would be nice to have an ENQ option with a "timeout" value. I.e.
give me this and, if not available, wait for at most ???? <time intervals>
in the hopes it will soon will be available. This could be emulated by
having the ENQ logic in a subtask. The main task invokes this subtask, the
subtask does the ENQ followed by a POST of an ECB. The main task then sets
up a STIMER for the "max wait" interval. It then waits on one of two ECBs
to be posted. The ENQ subtask ECB or the STIMER expiration ECB. If the
STIMER expires, just force DETACH the subtask to terminate it.​

>
> -- gil
>
> ----------------------------------------------------------------------
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to list...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--

Schrodinger's backup: The condition of any backup is unknown until a
restore is attempted.

Yoda of Borg, we are. Futile, resistance is, yes. Assimilated, you will be.

He's about as useful as a wax frying pan.

10 to the 12th power microphones = 1 Megaphone

Maranatha! <><
John McKown

[Paul Gilmartin]

On Mon, 28 Sep 2015 16:04:55 -0500, John McKown wrote:
>>
>> >I seem to recall a recent enhancement to z/FTP dynamic allocation that
>> reports contention and retries some number of times at user-defined
>> intervals. TSO ALLOC and IDCAMS might take that as an example for future
>> development.
>> >
>> So instead of deadlocks you can have livelocks? You can code that loop
>> yourself.
>
>​Perhaps what is needed is a new option on the ENQ. Normally, when you do
>an ALLOCATE and the DSN is unavailable, you get an immediate failure. OTOH,
>there is a parameter in the SVC 99 control block which allows an APF
>authorized program to WAIT "forever". Which basically leads to the thought
>

Yes. I believe that's TU S99WTDSN.

>that it would be nice to have an ENQ option with a "timeout" value. I.e.
>give me this and, if not available, wait for at most ???? <time intervals>
>

With an installation defined max?

>in the hopes it will soon will be available. This could be emulated by
>having the ENQ logic in a subtask. The main task invokes this subtask, the
>subtask does the ENQ followed by a POST of an ECB. The main task then sets
>up a STIMER for the "max wait" interval. It then waits on one of two ECBs
>to be posted. The ENQ subtask ECB or the STIMER expiration ECB. If the
>STIMER expires, just force DETACH the subtask to terminate it.​
>

"subtask" or parent task?

SMP/E appears to do something of this sort. After ? interval wait it
gives up and terminates.

I'd imagine a more sophisticated approach: For an unauthorized caller
specifying S99WTDSN, Allocation should traverse the GRS graph and
if the caller is creating a deadlock, return failure immediately.

[Joseph W Gentile]

Hi John/Gil, if you would like to open a request for enhancement (RFE) to
that effect, you can do that here https://www.ibm.com/developerworks/rfe/

-Joe

Joe Gentile
z/OS GRS Lead
(845)435-2184 (T/L 295-2184)
jwge...@us.ibm.com

[Jo Clark]

This is an old thread, but thank you all so much!! Just an FYI, the command RENAME works as well as R, UNLESS you're using it to see what has the DSN enqueued!!! :-) Only R will let you do the F1 command twice and return the list of offenders!!