CPU Balancing via the SYSDEF TD

[Steve Mondy]

I am testing the use of CPU Balancing. I have read the posts on the list, Hints & Tips and Ingolf’s blog. I am running on a z9, 3-way, with z/VM 5.4, z/VSE 4.2 & 5.2 guests. The VSE guests will have two virtual processors defined. The VSE guests I am considering for CPU Balancing all run a combination of CICS and batch.

 

On one of the guests the NP/TOT runs between 0.528 – 0.539, so I believe that a second CPU may only provide minimal benefit. I set the balancing at SYSDEF TD,INT=60,THR=95,STOPQ.

 

On another guest the NP/TOT is between 0.365 – 0.395, so it should benefit more than the other guest. My first setting will be SYSDEF TD,INT=60,THR=95,STOPQ.

I have additional VSE guests that run in this environment but just want to start with these two.

 

 

Of those with experience with TD CPU Balancing, what balancing values have you used that were most beneficial?  Longer or shorter intervals? I know YMMV.

 

And are there any messages produced on the console when CPU Balancing activates or quiesce a CPU? The only way I have found is to issue the QUERY TD command and catch the second CPU active or it shows quiesced values on the second CPU.

 

Thanks,

Steve

 

Steve Mondy
Sr. Systems Administrator

Open Solutions Division

Fiserv
Office:    713-965-8457
Mobile:   281-409-2870

Fax:        281-334-2642
www.fiserv.com

 

FORTUNE World's Most Admired Companies^® 2014 | 2015 | 2016

Facebook: Like Fiserv  ·  Twitter: Follow @Fiserv  ·  LinkedIn: Connect Fiserv

Flipboard: Follow Fiserv Forward  ·  Careers: Join Fiserv

 

---

NOTICE:
This e-mail is intended solely for the use of the individual to whom it is addressed and may contain information that is privileged, confidential or otherwise exempt from disclosure. If the reader of this e-mail is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify us by replying to the original message at the listed email address. Thank You.

[Tony Thigpen]

Steve,

This question raised another question I don't know the answer to:

When comparing a VSE guest with only one vCPU against a guest which is
defined has having multiple vCPUs, yet has only one vCPU turned on via
the TD command, does the fact that there are multiple CPUs available
cause additional overhead that is not seen in a single vCPU VSE guest?

Maybe somebody has the answer?

Tony Thigpen

Steve Mondy wrote on 09/27/2016 10:39 AM:
> I am testing the use of CPU Balancing. I have read the posts on the
> list, Hints & Tips and Ingolf’s blog. I am running on a z9, 3-way, with
> z/VM 5.4, z/VSE 4.2 & 5.2 guests. The VSE guests will have two virtual
> processors defined. The VSE guests I am considering for CPU Balancing
> all run a combination of CICS and batch.
>
> On one of the guests the NP/TOT runs between 0.528 – 0.539, so I believe
> that a second CPU may only provide minimal benefit. I set the balancing
> at SYSDEF TD,INT=60,THR=95,STOPQ.
>
> On another guest the NP/TOT is between 0.365 – 0.395, so it should
> benefit more than the other guest. My first setting will be SYSDEF
> TD,INT=60,THR=95,STOPQ.
>
> I have additional VSE guests that run in this environment but just want
> to start with these two.
>
> Of those with experience with TD CPU Balancing, what balancing values
> have you used that were most beneficial? Longer or shorter intervals? I
> know YMMV.
>
> And are there any messages produced on the console when CPU Balancing
> activates or quiesce a CPU? The only way I have found is to issue the
> QUERY TD command and catch the second CPU active or it shows quiesced
> values on the second CPU.
>
> Thanks,
>
> Steve
>

> *Steve Mondy**
> *Sr. Systems Administrator
>
> Open Solutions Division**
>
> *Fiserv*

> Office: 713-965-8457
> Mobile: 281-409-2870
>
> Fax: 281-334-2642

> www.fiserv.com <http://www.fiserv.com/>
>
> FORTUNE World's Most Admired Companies^® 2014|2015|2016**
>
> *Facebook:*Like Fiserv <http://www.facebook.com/fiserv>* ·**Twitter:*
> Follow @Fiserv <http://www.twitter.com/fiserv>* ·**LinkedIn:* Connect
> Fiserv <http://www.linkedin.com/company/fiserv>__
>
> *Flipboard:*Follow Fiserv Forward <http://fisv.co/fiserv-forward>*·*
> *Careers*: Join Fiserv <http://fisv.co/1orAdkH>
>
>
> ------------------------------------------------------------------------

>
> NOTICE:
> This e-mail is intended solely for the use of the individual to whom it
> is addressed and may contain information that is privileged,
> confidential or otherwise exempt from disclosure. If the reader of this
> e-mail is not the intended recipient or the employee or agent
> responsible for delivering the message to the intended recipient, you
> are hereby notified that any dissemination, distribution, or copying of
> this communication is strictly prohibited. If you have received this
> communication in error, please immediately notify us by replying to the
> original message at the listed email address. Thank You.
>
>

> _______________________________________________
> VSE-L mailing list
> VS...@lists.lehigh.edu
> https://lists.lehigh.edu/mailman/listinfo/vse-l
>

_______________________________________________
VSE-L mailing list
VS...@lists.lehigh.edu
https://lists.lehigh.edu/mailman/listinfo/vse-l

[Steve Mondy]

Tony,
That is one of my concerns as well.

And having a maximum of on 99 seconds interval would seem to add some unnecessary overhead during long periods when you know you are not going to reach the threshold. I would think then a minute value would be better, i.e., INT=99S (seconds) | 15M (minutes). Just a thought.

I could schedule with FAQS a SYSDEF TD,INT=0 to turn CPU Balancing off during those time periods. But if I am going to do that I just might as well schedule SYSDEF TD,START=ALL & SYSDEF TD,STOPQ=ALL.

Steve

-----Original Message-----
From: VSE-L [mailto:vse-l-bounces+steve.mondy=opensolu...@lists.lehigh.edu] On Behalf Of Tony Thigpen
Sent: Tuesday, September 27, 2016 10:15 AM
To: VSE Discussion List
Subject: Re: CPU Balancing via the SYSDEF TD

Steve,

This question raised another question I don't know the answer to:

When comparing a VSE guest with only one vCPU against a guest which is defined has having multiple vCPUs, yet has only one vCPU turned on via the TD command, does the fact that there are multiple CPUs available cause additional overhead that is not seen in a single vCPU VSE guest?

Maybe somebody has the answer?

Tony Thigpen

Steve Mondy wrote on 09/27/2016 10:39 AM:
> I am testing the use of CPU Balancing. I have read the posts on the
> list, Hints & Tips and Ingolf's blog. I am running on a z9, 3-way,
> with z/VM 5.4, z/VSE 4.2 & 5.2 guests. The VSE guests will have two
> virtual processors defined. The VSE guests I am considering for CPU
> Balancing all run a combination of CICS and batch.
>

> On one of the guests the NP/TOT runs between 0.528 - 0.539, so I

> believe that a second CPU may only provide minimal benefit. I set the
> balancing at SYSDEF TD,INT=60,THR=95,STOPQ.
>

> On another guest the NP/TOT is between 0.365 - 0.395, so it should

> benefit more than the other guest. My first setting will be SYSDEF
> TD,INT=60,THR=95,STOPQ.
>
> I have additional VSE guests that run in this environment but just
> want to start with these two.
>
> Of those with experience with TD CPU Balancing, what balancing values
> have you used that were most beneficial? Longer or shorter intervals?
> I know YMMV.
>
> And are there any messages produced on the console when CPU Balancing
> activates or quiesce a CPU? The only way I have found is to issue the
> QUERY TD command and catch the second CPU active or it shows quiesced
> values on the second CPU.
>
> Thanks,
>
> Steve
>
> *Steve Mondy**
> *Sr. Systems Administrator
>
> Open Solutions Division**
>
> *Fiserv*
> Office: 713-965-8457
> Mobile: 281-409-2870
>
> Fax: 281-334-2642
> www.fiserv.com <http://www.fiserv.com/>
>

> FORTUNE World's Most Admired Companies^(r) 2014|2015|2016**
>
> *Facebook:*Like Fiserv <http://www.facebook.com/fiserv>* ***Twitter:*
> Follow @Fiserv <http://www.twitter.com/fiserv>* ***LinkedIn:* Connect
> Fiserv <http://www.linkedin.com/company/fiserv>__
>
> *Flipboard:*Follow Fiserv Forward <http://fisv.co/fiserv-forward>***

> *Careers*: Join Fiserv <http://fisv.co/1orAdkH>
>
>
> ----------------------------------------------------------------------
> --
>
> NOTICE:
> This e-mail is intended solely for the use of the individual to whom
> it is addressed and may contain information that is privileged,
> confidential or otherwise exempt from disclosure. If the reader of
> this e-mail is not the intended recipient or the employee or agent
> responsible for delivering the message to the intended recipient, you
> are hereby notified that any dissemination, distribution, or copying
> of this communication is strictly prohibited. If you have received
> this communication in error, please immediately notify us by replying
> to the original message at the listed email address. Thank You.
>
>
> _______________________________________________
> VSE-L mailing list
> VS...@lists.lehigh.edu
> https://lists.lehigh.edu/mailman/listinfo/vse-l
>

_______________________________________________
VSE-L mailing list
VS...@lists.lehigh.edu
https://lists.lehigh.edu/mailman/listinfo/vse-l

________________________________

[Mick Poil]

Steve,

From my experience and testing, a job runs a bit slower when you use multiple cpus compared to when you run the same job on a single cpu, so the idea of only using one cpu when you don't need two is a good idea. My first question would be what is driving you to need more than 1 cpu? Do you have e.g. CPUMON data to show the need? An awful  lot can happen in 60 seconds and with a threshold of 95% I wonder how effective it would be. I think that using CPUMON with a short interval will show the switching on/off. You can simulate the operation with Rexx  The lower the NP/TOT, the better. The algorithm that VSE uses to decide how many cpus can be exploited based on NP/TOT may be out a bit as the relative intensity of cpu vs SVC and key zero will vary, and it is that activity that affects cpu usage. Batch SVC activity on one cpu can affect CICS to some degree under certain circumstances.If you would like to discuss it in more detail, feel free to email me privately on gmail or poil...@uk.ibm.com.

I had to deal with a real nasty on VM where I found it stopped dispatching one Virtual cpu and CICS was going to sleep even though VSE said it was being dispatched. I left the customer to work with the VM folks, so I can't remember the fix. Not sure if it was V5 or V6 now.

To answer Tony's question, I found that the multi-cpu overhead went away when I did a STOPQ. All my VSE systems at Hursley run under VM, so I only ever use Virtual cpus.

HTH

Mike

[Ken and Mary Meyer]

The overhead of a vCPU not active is negligible. Multiple CPUs were mostly
for slower systems to improve performance, but it never really helped much in
my testing, unless you have a partition that will run mostly parallel and can be
independent of other partitions. If your NP is 40% or more, it is not likely to
help much at all.

Ken


On Tue, Sep 27, 2016 at 10:42 AM, Steve Mondy
snip..

[Mick Poil]

Just an FYI, I have worked with actual customers where not having 2 cpus is not an option.

[Mick Poil]

And an EC12-702 is not a slow processor.

[Mick Poil]

Steve,

If you want this during the a batch window, you could enable two cpus while the hog job is running and then see what happens to elapsed times overall. In one very extreme case I have seen an extremely SVC intensive batch hog cause CICS to slow down when 2 cpus are used.

Using equal priority in an appropriate way should allow a hog to coexist with other jobs as that should promote I/O-intensive over cpu-intensive work.

Mike

[Tony Thigpen]

Reminds me of a monthly job I once was told to 'fix' because it took
over 24 hours so had to be run on the weekend after month-end.

By adding 64K buffers to the JCL, the job only took 30 minutes. So, we
moved it to run on actual EOM. (I don't want to get into VSAM tuning
issues, but it was a *LOT* of random reads of a KSDS files. On was keyed
zip-code and the report needed the city names.)

Big mistake. While it took only 30 minutes to run, *nothing* else would
run during that time. It even affected jobs with priorities above it.

It sounds like you may want to reduce the buffers for this hog and slow
it down a bit.

Tony Thigpen

Steve Mondy wrote on 09/27/2016 02:39 PM:
> Mike,
>
> I am not familiar with CPUMON. We have CA-EXPLORE/VSE& EXPLORE/VM. We
> used them in the past to analyze this one application job that pushes
> the VSE to 100% for something close to an hour. And has extremely heavy
> IO to one of the VSAM datasets it uses. I don’t have the number right
> now. Anything below the job gets little or no CPU during this time. So
> I was thinking this might be somewhere we could utilize the second CPU.
>
> We have ask the application development staff for some time to try to
> rework the program. They just don’t see a way to improve it. We also
> have one or two other CPU intensive programs that have always pushed the
> VSEs to 100% no matter what the physical process size.
>
> I know 60 seconds is a long time. That is the reason for my original
> post was to see what others were using. And to see if anyone had any
> data on the overhead caused using a shorter interval. The threshold of
> 95% is my understanding that for VSE one CPU is better for performance
> than two CPUs then I only wanted to activate the second one just when it
> was needed. Am I looking at this correctly?
>
> Steve
>
> *From:*VSE-L
> [mailto:vse-l-bounces+steve.mondy=opensolu...@lists.lehigh.edu]
> *On Behalf Of *Mick Poil
> *Sent:* Tuesday, September 27, 2016 1:01 PM
> *To:* VSE Discussion List
> *Subject:* Re: CPU Balancing via the SYSDEF TD

>
> Steve,
>
> From my experience and testing, a job runs a bit slower when you use
> multiple cpus compared to when you run the same job on a single cpu, so
> the idea of only using one cpu when you don't need two is a good idea.
> My first question would be what is driving you to need more than 1 cpu?
> Do you have e.g. CPUMON data to show the need? An awful lot can happen
> in 60 seconds and with a threshold of 95% I wonder how effective it
> would be. I think that using CPUMON with a short interval will show the
> switching on/off. You can simulate the operation with Rexx The lower
> the NP/TOT, the better. The algorithm that VSE uses to decide how many
> cpus can be exploited based on NP/TOT may be out a bit as the relative
> intensity of cpu vs SVC and key zero will vary, and it is that activity
> that affects cpu usage. Batch SVC activity on one cpu can affect CICS to
> some degree under certain circumstances.If you would like to discuss it
> in more detail, feel free to email me privately on gmail or

> poil...@uk.ibm.com <mailto:poil...@uk.ibm.com>.

>
> I had to deal with a real nasty on VM where I found it stopped
> dispatching one Virtual cpu and CICS was going to sleep even though VSE
> said it was being dispatched. I left the customer to work with the VM
> folks, so I can't remember the fix. Not sure if it was V5 or V6 now.
>
> To answer Tony's question, I found that the multi-cpu overhead went away
> when I did a STOPQ. All my VSE systems at Hursley run under VM, so I
> only ever use Virtual cpus.
>
> HTH
>
> Mike
>
>

> ------------------------------------------------------------------------

[Mick Poil]

Somewhere I I have a "degrader" program that was used to slow a cpu hog down without making any changes to it.

[Jeffrey Barnard]

On 09/27/2016 11:14 AM, Tony Thigpen wrote:
> Steve,
>
> This question raised another question I don't know the answer to:
>
> When comparing a VSE guest with only one vCPU against a guest which is
> defined has having multiple vCPUs, yet has only one vCPU turned on via the
> TD command, does the fact that there are multiple CPUs available cause
> additional overhead that is not seen in a single vCPU VSE guest?
>
> Maybe somebody has the answer?
>

The answer is Yes.

Ingolf and I had a discussion about this topic once.

The short answer is ... Within the z/VSE dispatcher (previously known as
the Turbo Dispatcher) a single CPU environment with no other CPU defined
has fast paths within the code. If more than one CPU is defined, even if
disabled/inactive, the fast paths can not be used.

When I asked about how much overhead it was suggested that is it almost the
same as having the 2nd CPU active. Almost, actually using the 2nd CPU is
bit more since you would actually be loading control regs, lowcore fields
setup, etc., etc.

Jeff

[Ken and Mary Meyer]

Perhaps some clarification is required. Are you defining additional CPUs to
z/VSE or vCPUs to z/VM? You also mentioned balancing, but that does not
require additional CPUs. If you define another CEC (CPU) to z/VSE, it will
cost some performance if it is active or not. If under z/VM, if possible it may
help to run jobs like this on a separate z/VSE instance. If your NP is high, you
probably will not do much better with an extra CEC. A situation like this could
be very good for balancing between partitions, which could possibly slow down
the I/O and CPU intense jobs. I would suggest you balance your partitions
running batch and online applications, and then adjust their individual weights
until you find a compromise you can live with. An additional CEC will
only allow
multiple partitions to run if partitions can run in parallel enough to
overcome the
loss of performance that is inherent with running multiple CECs.

Ken


snip..

[Steve Mondy]

Ken,
The z9 is a 3-way and that is all we define to VM. We run just one LPAR to give all the resources to VM. My questions relate to defining additional CPUs to some of the VSEs that we run under VM. All our production work is on the VSEs.

Q PROCESSOR
PROCESSOR 00 MASTER CP
PROCESSOR 01 ALTERNATE CP
PROCESSOR 02 ALTERNATE CP

IND USER ESADEV-D
USERID=ESADEV-D MACH=ESA STOR=300M VIRT=V XSTORE=NONE
IPLSYS=DEV 02A3 DEVNUM=00171
PAGES: RES=00075915 WS=00075638 LOCKEDREAL=00000275 RESVD=00000000
NPREF=00000000 PREF=00000000 READS=00004206 WRITES=00164550
XSTORE=000000 READS=000000 WRITES=000000 MIGRATES=000000
CPU 00: CTIME=87:44 VTIME=641:12 TTIME=698:15 IO=645854
RDR=034696 PRT=999999 PCH=006961 TYPE=CP CPUAFFIN=ON

USERID=ESADEV-D MACH=ESA STOR=300M VIRT=V XSTORE=NONE
IPLSYS=DEV NONE DEVNUM=00171
PAGES: RES=00075915 WS=00075638 LOCKEDREAL=00000275 RESVD=00000000
NPREF=00000000 PREF=00000000 READS=00004206 WRITES=00164550
XSTORE=000000 READS=000000 WRITES=000000 MIGRATES=000000
CPU 01: CTIME=87:44 VTIME=002:43 TTIME=003:52 IO=567301
RDR=000000 PRT=000406 PCH=000000 TYPE=CP CPUAFFIN=ON
Ready; T=0.01/0.01 16:58:05

USER ESADEV-D XXXXXXXX 300M 300M BCEFG 28 |
ACCOUNT ESADEV-D VSEGUEST
CPU 00 CPUID 000015
CPU 01 CPUID 010015
IPL 190 PARM AUTOCR NOSPROF INSTSEG NO
IUCV ANY PRIORITY MSGLIMIT 64
IUCV ALLOW
MACHINE ESA
OPTION CPUID 000015 MAXCONN 64 MIH MAINTCCW SVMSTAT STGEXEMPT
OPTION RMCHINFO TODENABLE
SHARE RELATIVE 175
...

Steve

-----Original Message-----
From: VSE-L [mailto:vse-l-bounces+steve.mondy=opensolu...@lists.lehigh.edu] On Behalf Of Ken and Mary Meyer
Sent: Tuesday, September 27, 2016 3:44 PM
To: VSE Discussion List

________________________________

NOTICE:
This e-mail is intended solely for the use of the individual to whom it is addressed and may contain information that is privileged, confidential or otherwise exempt from disclosure. If the reader of this e-mail is not the intended recipient or the employee or agent responsible for delivering the message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please immediately notify us by replying to the original message at the listed email address. Thank You.

[Ken and Mary Meyer]

I assume you are running a virtual disk lock file?

When you refer to posting transactions, are you talking about CICS
transactions or pieces of business data?

Since you have EXPLORE (and I assume ASO), have you investigated where
the CPU cycles are being mostly used?

Ordinarily CICS is a great choice for balancing with a lot of short CPU accesses
per balance cycle because of its tendency to be interrupted by devices very
often. For batch it depends on ratio of processing to SVCs. I used to have an
IMOD (may still exist) that would monitor use of CPU for a given
partition. If the
value exceeded a threshold, the PRTY command would be issued to lower its
priority. The threshold could be in either direction.

Ken

[Mick Poil]

With regard to what Jeff said, my comments are based on  measurements on my own system and a customer's. I have to bear in mind that there is software working out the cpu usage data, and I have seen some "unusual" cpu usage values at times (and that includes CICS-reported TCB cpu times, which can affect Vendor CICS monitoring), and usage might not align with the LPAR-level data.

VSE themselves are unequivocally clear - using only one cpu gives you the best performance.

At the end of the day, if you need two cpus, you need two cpus. The customer I work with a lot could not exist on one cpu for VSE, and they have the EC12-702 which is almost the fastest cpu you can buy now from IBM. The top z13 cpu is only about 10% faster.

There was probably a good reason why Cpu Balancing was developed, in which case, use it if you can. There is probably no "best" value, you need to play.

Speculation won't tell you how well or not VSE/CICS/whatever will perform, you need to try and and measure the results.

I will comment on Steve's responses later.

Mike

[Mick Poil]

Steve,

Yep, SHR(4) is a killer for a number of reasons, which includes the cost of cpu for VSAM and VSE I/O processing. Assuming it is a KSDS, put it in its own LSRPOOL with as few data buffers as you can get away with based on STRNO. SHR(4) Index CIs do seem to get a good amount of Lookaside, Data CIs very little. LSR is better than NSR in terms of cpu time from what I see.

I used the trick of changing SHR(4) to SHR(2) with good effect for the batch window, but I don't think you can ALTER a file that is open in CICS.

CICS itself can never exploit more than one cpu, and that is a VSE restriction.

I find emails frustrating because it takes so much time compared to a phone call. If you would like to chat about it at a time that works for both of us, email me at IBM and I can set up a free conference call.

Mike

[Mick Poil]

Forgot one thing. CPUMON reports active and available cpus at the interval in seconds that you ask for, so you can track how Cpu Balancing is actually working. Don't forget that a lot can happen in CICS in even one second. Btach is arguably less sensitive.

[Mick Poil]

Steve,

FYI, I am on holiday from 1st October to 17th. No horrid PMRs to look at for more than 2 weeks :-)

Mike

[Ken and Mary Meyer]

There is a very popular BIM product and a CA product that will funnel all VSAM
batch I/O for this file through CICS (when available), allowing you to
do away with
shareoptions(4) and still protect your file(s). The names evade me at
the moment,
but I think one of those would help.

Ken

[Mick Poil]

Steve,

If performance of this transaction is an issue to you, and you would like to know more about what is going on, and you can run it in test with no other task activity with a large CICS internal trace table and force a dump, I can analyze the trace and tell you what is going on and how long is spent in VSAM I/O etc. It would need an SR (PMR) opened unless you can the DFHPD410 DATA TR=2 formatted trace is less than 30MB for emailing to me.

CICS Auxtrace skews the performance of a task due to slow and synchronous auxtrace I/O.

CICS monitoring data (which is also used by TMON/CICS for part of the reported task performance data) can give a lot of detail, but FCIOWTT does not include things like CICS suspend time in FCXCWAIT even though you can see the effect of he FCXCWAIT activity elsewhere, although I can provide a test fix to include that. The reported task cpu usage is also less than actual.

I like the idea of a product that allows SHR(2), but I would want to know a bit about how that works. EXCI is not a performance option.

Mike

[Eugene S Hudders CTREK Corp. via VSE-L]

Hi:

 

Doesn't BSI  (Jeff B) have a VSAM buffering product that supports SHROPT 4?

 

Regards,

Gene

 

[Mike Moore]

Yep, BSI has OPTICACHE and takes care of the SHR 4 problems. We have used for several years and it works well for us.

 

 

 

 

Mike Moore

IT Manager

Alabama Judicial Datacenter

300 Dexter Avenue

Montgomery, Alabama 36104

334-954-5025

 

“This is the final test of a gentleman: his respect for those who can be of no possible service to him.” – William Lyon Phelps

 

From: VSE-L [mailto:vse-l-bounces+mike.moore=alacou...@lists.lehigh.edu] On Behalf Of Eugene S Hudders CTREK Corp. via VSE-L
Sent: Wednesday, September 28, 2016 9:47 AM
To: vs...@lists.lehigh.edu
Cc: Eshu...@aol.com
Subject: Re: CPU Balancing via the SYSDEF TD

 

Hi:

[Mick Poil]

I am not the main person on that one. We have a developer looking at it as well, and we are trying to work out if CICS registers have been corrupted somewhere along the way. The code is pretty fundamental to CICS starting and it has worked since 1999 without any hint of a problem, and that is if you ignore the fact that the same code was working on OS/390 before that. It is a real puzzle.