SYSDUMP locking

[Juha Vuori]

VSEers,

When accessing SYSDUMP simultaneously in two VSE guests, the other process waits for a lock with

0T04I RESOURCE ?DÎË;4úíè' B IS IN USE BY SYSTEM FF01010128288000

I'd understand that if SYSDUMP was a shared resource, but it is not - each VSE has its own local
SYSDUMP, a SAM file in CKD dasd SYSWK1, and SYSWK1 is not a shared DASD in this environment.

E.g. DTSFILE is a similar file resource: local to every VSE guest, but it can be updated
simultaneously in every VSE..

What might be the special resource that is locked when SYSDUMP is accessed in a VSE?
I don't see any reason why a multi-VSE level lock is set when a local resource is accessed, and I
have never intentionally set any additional locking to SYSDUMP..

This is an one-LPAR system, z/VSE 5.2 guests running under z/VM 6.3, and VSE lock file resides on VM
VDISK, and only one CKD dasd is intentionally shared between all guests.
All the VSE images are originally cloned from one Grand Mother VSE Image - I wonder if "resource"
?DÎË;4úíè' B is some unique key generated for SYSDUMP in VSE installation process but its holy
uniqueness has been ruined in the cloning process (seen that kind of uniqueness problems with other
OSes and keys many times before after cloning)?

--
Juha Vuori
Fujitsu Finland

_______________________________________________
VSE-L mailing list
VS...@lists.lehigh.edu
https://lists.lehigh.edu/mailman/listinfo/vse-l

[Stuart, David]

Juha,

And you've verified that no z/VSE guest anywhere has any of their volumes, except the Lock File, added as shared? (been bitten that way a couple times. Thought there weren't any shared disks. But I was wrong)

And if you don't share any disks, why the Lock File?


Dave


Dave Stuart
Principal Information Systems Support Analyst
Information Technology Services
County of Ventura, CA
805-662-6731
David....@ventura.org

[Kevin Corkery]

I recently had this situation with Vollie. I was trying to set-up a shared VSAM catalog to be the only shared resource (besides the lock file) across two identical systems. The member locking mechanism in Vollie, would determine that a lock file existed and then would lock a member based on the OLL file name, the member's directory, and member name. This could result in a cross system lock even though the Vollie OLL did not reside on a shared volume. Looks like someone decided to take a short cut or two in figuring out the proper enque requirement. Of course, expecting a change in the code is not reasonable so I abandoned the "shared" aspect of the catalog; it's really only there for non-production use in moving data to/from LPAR images.

-----Original Message-----
From: VSE-L [mailto:vse-l-bounces+kcorkery=live...@lists.lehigh.edu] On Behalf Of Stuart, David
Sent: Wednesday, December 28, 2016 1:14 PM
To: VSE Discussion List
Subject: RE: SYSDUMP locking

Juha,

And you've verified that no z/VSE guest anywhere has any of their volumes, except the Lock File, added as shared? (been bitten that way a couple times. Thought there weren't any shared disks. But I was wrong)

And if you don't share any disks, why the Lock File?


Dave


Dave Stuart
Principal Information Systems Support Analyst Information Technology Services County of Ventura, CA
805-662-6731
David....@ventura.org

[Stuart, David]

MSHP does it, too.

If MSHP is running anywhere, on any System that is sharing a Lock File, you'll receive an MSHP message that 'MSHP is already active... ' (don't remember the exact message text).


Dave

-----Original Message-----
From: VSE-L [mailto:vse-l-bounces+david.stuart=ventu...@lists.lehigh.edu] On Behalf Of Kevin Corkery
Sent: Wednesday, December 28, 2016 10:42 AM
To: 'VSE Discussion List' <vs...@lists.lehigh.edu>

[Juha Vuori]

Dave,

I rechecked, and the only devices added with SHR in all ipl procedures are

ADD 400,FBA,SHR VM VDISK SHARED BY VSES
ADD C3F,ECKD,SHR 1X DS8K 3390-3 SHARED FC DISK, LCU 00

lock file being defined in the shared vdisk vdev=400 with

DLF VOLID=VDSK01,BLK=18,NBLK=6144,TYPE=N,NCPU=16

Vdev c3f volid DYNCAT does contain shared files:

--- Data Set Name --- sorted by EXTENT ----- Ext Begin-end Reltrk,
1...5...10...15...20...25...30...35...40.... seq Cyl-hd Cyl-hd numtrks
*** FREE EXTENT *** 0 0 1 774 14 1,11624
CAI.DYNAM.CATALOG.FILE 0 775 0 814 14 11625,600
*** FREE EXTENT *** 0 815 0 944 14 12225,1950
CAI.DYNAM.AUDIT.FILE 0 945 0 948 14 14175,60
*** FREE EXTENT *** 0 949 0 1023 14 14235,1125
VSE.SHRLIB1.LIBRARY 0 1024 0 1123 14 15360,1500
*** FREE EXTENT *** 0 1124 0 3336 14 16860,33195
DOS.LOCK.FILE 0 3337 0 3337 14 50055,15
*** VTOC EXTENT *** 0 3338 0 3338 14 50070,15
*** This volume is currently 4 percent full with 47894 tracks available

so actually CA Dynam/T catalog and one librarian are the only resources shared (DOS.LOCK.FILE is not
used anymore).

Why SYSDUMP seems to be logically shared even that it is not physically - that's the mystery.

Regards,
Juha

[Stuart, David]

Juha,

Something I noticed. You said that the VM MDISK Device 400 contains the Lock File. But in the VTOC you sent, it shows a DOS.LOCK.FILE on C3F. Is that an old Lock File? Are all the VSE guests using the same Lock File? It almost looks like you might have two.


Dave

[Juha Vuori]

Interesting, very interesting..

I can somehow understand those 1-2 shortcuts by 3rd party designers, but for SYSDUMP and MSHP - it's
by Böblingen, wow.

I would expect a change in the software of those VSE core components, though :)
Or a logical explanation....

Regards,
Juha

[Juha Vuori]

Dave,

Yes, we originally had DOS.LOCK.FILE in the physical dasd C3F but migrated it to VM VDISK 400 for
performance reasons.

I now scratched DOS.LOCK.FILE from C3F, and apparently it was not used because no VSE crashed.. :)

(It's been there unused for around 5 years, as a rollback backup in case VM VDISK would not be
stable - it was good time for it to go forever..)

I retried using SYSDUMP simultaneously in two VSE guests, and 0T04I did reappear. Expected, as DLF
defines which lock file is used.

Regards,
Juha

[Mick Poil]

Not that it may help much, but VSE Lock resource names are normally at least partly character. Librarian locks start with 'C' as I remember. I expect VSE would ask for a LOCK trace

Mike

[Mick Poil]

VSE keeps the Lock File information that it gets during IPL in storage as I remember, so saying no crash when scratched and you have not IPLed is not conclusive to my mind.

[Juha Vuori]

Mike,

You almost got me little worried, so I reIPLed 4/6 VSE guests, and they just select their lock file
during the IPL like before:

BG 0000 DLF VOLID=VDSK01,BLK=18,NBLK=6144,TYPE=N,NCPU=16
BG 0000 0J20I DEVICE RECOGNITION IN PROGRESS
BG 0000 0I52I LOCK FILE ON 400: LOW HIGH
BG 0000 BLOCK: 18 6161

and 0T04I events keep on appearing when SYSDUMP is accessed simultaneously.

Hm, even though this is just a nuisance for this system, I'll open a low-prio PMR for this.
I just can't let resources that start with ?DIE keep on locking in my systems :)

Regards, Juha

On 28.12.2016 21:47, Mick Poil wrote:
> VSE keeps the Lock File information that it gets during IPL in storage as I remember, so saying no
> crash when scratched and you have not IPLed is not conclusive to my mind.
>
>

[Martin Truebner]

Juha,

how is this: the lock is supposed to be in char form and someone used
the wrong method for the argument while coding the stuff- this is how
the garbage ended up in the name of the lock.

Along with the wrong method for the name goes the operand for the
scope of the lock-

voila-

here you have a bug which will not be discovered during regular testing.

Let us know what the outcome is.

Martin

[Mick Poil]

Juha,

The post was done to make sure that you didn't get bitten.

Mike

[Juha Vuori]

Mystery resolved by VSE lab:

"I looked into INFOANA code and found that LOCK control blocks were
defined with SCOPE=EXT and not modified.
Customer may open a requirement for improvement in a future release.
https://www-03.ibm.com/systems/z/os/zvse/contact/requirement.html "

Now I started to wonder what is the standard way in code to find out, whether a resource, let's say
SYSDUMP, is shared or local.
A: If it is on a shared disk or not.
But is it easy to find out when you're programming a VSE module?
Just wondering..

Note that my observations on SYSDUMP locking appear to be specific to INFOANA tool.
I actually observe the symptoms when deleting dumps with INFOANA..

Regards, Juha

[Stuart, David]

I would guess, then, that MSHP is coded to always perform an External lock, as well.

Dave


Dave Stuart
Principal Information Systems Support Analyst
Information Technology Services
County of Ventura, CA
805-662-6731
David....@ventura.org


-----Original Message-----
From: VSE-L [mailto:vse-l-bounces+david.stuart=ventu...@lists.lehigh.edu] On Behalf Of Juha Vuori
Sent: Thursday, December 29, 2016 10:39 AM
To: VSE Discussion List <vs...@lists.lehigh.edu>
Subject: Re: SYSDUMP locking

[Kevin Corkery]

If you look at the SIR output on a non-shared VSE system you see a good many
external locks being issued. This would indicate that many programs just
code for external in all cases. I suppose lacking a lock file, VSE locks
the resource using the internal mechanism. With a lock file, the resource
is locked on physical medium. I'll suspect that VSAM is very aware of the
state of the shared environment and uses the minimally invasive lock type;
high volume, high payback. INFOANAL and MSHP might be less worried
regarding this as a performance issue.

-----Original Message-----
From: VSE-L [mailto:vse-l-bounces+kcorkery=live...@lists.lehigh.edu] On
Behalf Of Juha Vuori

Sent: Thursday, December 29, 2016 1:39 PM
To: VSE Discussion List

[Mick Poil]

Juha,

LOCK is designed to check for Internal/External only if the VOLID= parameter is used to govern the scope. That way the LOCK code can check if the dasd device is shared or not and set SCOPE= appropriately. Otherwise it is completely down to SCOPE=. See the DTL macro definition.

Can you give me the PMR number? I wonder why such a cryptic resource name, that is unusual for IBM code.

Mike

[juha....@pp2.inet.fi]

RFE id 99132 "Enhancement in resource locking by some VSE tools" raised, in case anybody would be willing to vote for it.

http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=99132

--
Regards,
Juha

[Avihu Gershoni]

Kevin Corkery  wrote: "I'll suspect that VSAM is very aware of the state of the shared environment and uses the minimally invasive lock type; high volume, high payback.".

Well, it seems that your suspicion is quite incorrect.

 

I tried to do a massive delete of files in one shared catalog that controlled 48 volumes. Deleting a file in such situation caused a huge number of EXCPs and that delete job run more than 15 minutes.

During this time VSAM locked any VSAM open/close processing across the board on any VSAM file, specifically those that are defined in totally different catalogs.

This caused eventually a stall in all CICSs that run in another shared VSE images under z/VM, none of them had any file defined in the above mentioned catalog, with the message that VSYSOPEN is locked in another system. I had to cancel the delete job.

 

I opened a PMR about that.

 

Avihu Gershoni

Mainframe System Department Manager

Hilan Limited

8 Meitav Street

Tel Aviv

Israel

 

[Kevin Corkery]

Interesting.  Could be the same problem with IDCAMS delete processing.  I can’t believe that normal VSAM access locking would have this much overhead on an unshared file and go unnoticed.