Why are highly busy zIIPs worse than highly busy CPs?

[Peter Hunkeler]

There are some statements around zIIP utilization which I read here and there. Statements like:

- "You should not utilize one zIIP more than 30%, two zIIPs more than 60%..."
- "A task may become delayed for up to 3.2 ms (actually ZIIPAWMT) before the busy zIIP asks for help from a CP".



For this discussion, lets assume equal speed CPs and zIIPs, and a reasonable CP to zIIP ratio, and more than on processor of each kind.


It has been a long time strength of IBM Z (and all the predecessors) that the CPs in an LPAR can be utilized way above 90% without major problems arising. I seem to understand that this has changed lately, but still some 85% (?) should be fine.


Now, all work running on zIIPs was once work running on CPs (and still is if there are no zIIPs). So the work is no different (apart from much being run under an SRB instead of a TCB), and the response time requirement is no different. Right?


If so, how comes that busy zIIPs are said to be more of a problem than busy CPs? If the work can accept some queueing when run on CPs, why not when run on zIIPs. Queueing theory should apply equally to both.



When a processor is busy 50%, then 50% of the time there is at least one ready task, the one executing. Maybe there are some more waiting on the work queue. But these 50% say nothing about the delay of the tasks on the work queue.


In a simplified case, assume 5 tasks with equal priority, each one quickly, say after 0.5 ms, coming to the point where it has to give up the processor for a very short period of time before being requeued on the work queue. They all constantly work that way for 30 seconds in row, then become undispatchable for the remaining 30 seconds of that 50% busy minute. During the first 30 seconds, the zIIP is 100% busy, and after 3.2ms (ZIIPAWMT), the zIIP will ask a CP for help.


None of the tasks has been delayed by 3.2ms, although the ZIIP recognized its work queue has not become empty for 3.2ms and asked for help. To the contrary, the work has gotten better service because two processors are now serving the single work queue. (Again for simplicity, not currently taking priorities into account).


Same case but the task are working 1ms each time. Now it always takes more than 3.2ms for the last task on the work queue before it is being redispatched as long as the zIIP has not asked for help. But the zIIP will ask for help after 3.2ms, and the delay for the tasks will shrink.


Isn't this a better situation for zIIP work than for non-zIIP work? Same scenario on CPs. There is no-one to help.


Any thoughts?


--
Peter Hunkeler

----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to list...@listserv.ua.edu with the message: INFO IBM-MAIN

[Elardus Engelbrecht]

Peter Hunkeler wrote:

>- "You should not utilize one zIIP more than 30%, two zIIPs more than 60%..."

Who said it? And why 30%? Just curious.

My one zIIP (for about 8 LPARs on one machine) is usually anything from 5% to 40-50% in workdays, while the zIIP management overhead is usually from 0.5% to 1%.

No complaints from my users or colleagues.

Groete / Greetings
Elardus Engelbrecht

[Mike Schwab]

Once the delay is long enough, the CP does the work. They cost about
10X the price of zIIPs.

--
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

[Phil Smith]

Peter Hunkeler wrote, in part:

>There are some statements around zIIP utilization which I read here and there. Statements like:

>- "You should not utilize one zIIP more than 30%, two zIIPs more than 60%..."

>- "A task may become delayed for up to 3.2 ms (actually ZIIPAWMT) before the busy zIIP asks for help from a CP".

There are some knobs that can control this. And it's surely YMMV. If you had, for example, one heavyweight thread that was zIIP-eligible, one might assume that it could saturate that zIIP without issues. But that's not going to be a common scenario.

Scheduling an SRB isn't cheap. I don't know whether that's why, but for whatever reason, IBM has built this "fall back to the CP" mechanism. Kathy Walsh of IBM has a pretty good session that talks about this; I saw it at the IBM Systems Technical University in Orlando last month. Session is z101268.pdf; I can't find it on the web, either under that number or by searching some key strings in it. Sorry...
--
...phsiii

Phil Smith III
Senior Architect & Product Manager, Mainframe & Enterprise
Distinguished Technologist
Micro Focus (Voltage)

phs...@microfocus.com
T 703-476-4511
M 703-568-6662
Herndon, VA

[Charles Mills]

Isn't "fall back to the CP" because one would typically want one's work to
run *somewhere* even if a zIIP were not available but perhaps a CP was?

zIIP's are all about software costs, right? So scheduling work is like
buying a seat on Southwest. I need to get there, so if none of the "wanna
get away" seats are available, I will buy a full-fare seat.

Charles

[Peter Hunkeler]

>Once the delay is long enough, the CP does the work. They cost about 10X the price of zIIPs.

I understand the potential impact on the software bill zIIP-on-CP might have. That is not the point I want to get a better understanding. I'm interested in the technical aspects, only.


IMHO, from a technical point of view the invention of zIIPs and zAAPs is pure bullshit. From a financial point of view they are helping lowering costs -- and that is their only reason to be.


I predict (and no I don't have any insight into IBM's plans) zIIPs will be gone in the not so distant future. Container Pricing makes the zIIP/zAAP concept superfluous.


--
Peter Hunkeler

[Peter Hunkeler]

>Scheduling an SRB isn't cheap. I don't know whether that's why, but for whatever reason, IBM has built this "fall back to the CP" mechanism.

What falls back is still SRBs. The scheduling overhead has already been done before.


--
Peter Hunkeler

[Peter Hunkeler]

>Isn't "fall back to the CP" because one would typically want one's work to
run *somewhere* even if a zIIP were not available but perhaps a CP was?

If you meant no zIIPs are available to the LPAR (not configured or the CEC does not have some), then there is no fall-back. Work units get queue on CP work queues initially; the zIIP work queue is not being used. At least this is how I understand it.


If you meant no free zIIP capacity is available, then part of the decision might have been that initially you could have only half as many zIIPs as you had CPs.


--
Peter Hunkeler

[Jesse 1 Robinson]

We've had zIIPs for years. We monitored usage but were never terribly concerned as long as numbers didn't skyrocket. When we moved to DB2 V10, however, we were warned that 'some customers' were experiencing serious performance problems when zIIP eligible work spilled over to general CPs. The reason, we were told, was that while general CP management had evolved over the decades with a huge boatload of OS code to handle contention, zIIPs were newcomers that were more or less on their own in playground competition. The wiry little guys with little bully protection.

The message we got was that an overloaded zIIP could lead to performance problems--and rolling average spikes--that could be worse (!) than having no zIIP at all. We were sufficiently alarmed that we were moved to buy yet another zIIP to guard against calamity in production. Because we acquired the extra zIIP early on, I can't say what the consequence would have been if we had done nothing, but this is certainly the tale I'd rather be telling.

.
.
J.O.Skip Robinson
Southern California Edison Company
Electric Dragon Team Paddler
SHARE MVS Program Co-Manager
323-715-0595 Mobile
626-543-6132 Office ⇐=== NEW
robi...@sce.com

[Martin Packer]

As you'll see from my slides you really wouldn't want to delay the
Prefetch and Deferred Write engines in any DB2 you care about.

That's one example, perhaps the main one.

Martin Packer

zChampion, Systems Investigator & Performance Troubleshooter, IBM

+44-7802-245-584

email: martin...@uk.ibm.com

Twitter / Facebook IDs: MartinPacker

Blog:
https://www.ibm.com/developerworks/mydeveloperworks/blogs/MartinPacker

Podcast Series (With Marna Walle): https://developer.ibm.com/tv/mpt/ or

https://itunes.apple.com/gb/podcast/mainframe-performance-topics/id1127943573?mt=2


Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA

From: Peter Hunkeler <ph...@GMX.CH>
To: IBM-...@LISTSERV.UA.EDU
Date: 07/06/2018 17:14
Subject: AW: Re: Why are highly busy zIIPs worse than highly busy
CPs?

Unless stated otherwise above:
IBM United Kingdom Limited - Registered in England and Wales with number
741598.
Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

[Martin Packer]

Right. Though you've never been in my audience :-( you would've got the

same message about DB2 V10, a fortiori 11, 12 from me.



Cheers, Martin

Martin Packer



zChampion, Systems Investigator & Performance Troubleshooter, IBM



+44-7802-245-584



email: martin...@uk.ibm.com



Twitter / Facebook IDs: MartinPacker



Blog:

https://www.ibm.com/developerworks/mydeveloperworks/blogs/MartinPacker



Podcast Series (With Marna Walle): https://developer.ibm.com/tv/mpt/ or



https://itunes.apple.com/gb/podcast/mainframe-performance-topics/id1127943573?mt=2





Youtube channel: https://www.youtube.com/channel/UCu_65HaYgksbF6Q8SQ4oOvA







From: Jesse 1 Robinson <Jesse1....@SCE.COM>

To: IBM-...@LISTSERV.UA.EDU

Date: 07/06/2018 17:54

Subject: Re: Why are highly busy zIIPs worse than highly busy CPs?

Sent by: IBM Mainframe Discussion List <IBM-...@LISTSERV.UA.EDU>

Unless stated otherwise above:

IBM United Kingdom Limited - Registered in England and Wales with number

741598.

Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

[Peter Hunkeler]

>... however, we were warned that 'some customers' were experiencing serious performance problems when zIIP eligible work spilled over to general CPs.

Yeah, that is what I read and hear also, and I have no reason not to believe it. In fact, I suspect we've just been bitten by zIIP overload. However, this is not at the level of technical detail I'm seeking to understand.

>The reason, we were told, was that while general CP management had evolved over the decades with a huge boatload of OS code to handle contention, zIIPs were newcomers that were more or less on their own in playground competition. The wiry little guys with little bully protection.

Huh? Do you want to say you've been told that zIIP management (dispatcher and WLM/SRM) is new code that is distinct from the CP management (dispatcher and WLM/SRM) code? I must be misunderstanding you.

zIIPs are general purpose processors exactly the same as CPs are, aren't they. They will run whatever work unit happens to be found on their dispatcher queue, don't they? The decision which work queue to chain a newly ready work unit on is made at queueing time. It is just an (not really) arbitrary decision that IBM chose only enclave SRBs are eligible to be queued on the zIIP WUQ.

The only new thing is that the dispatcher, when running on a zIIP, may ask, i.e. flag a CP to also look for work on the zIIP WUQ. And it is the dispatcher, when running on that CP, which looks at both WUQ to select the next work unit to dispatch. This is how I think it works. Tell me, anyone, if this is not correct.

[Joel C. Ewing]

On 06/07/2018 02:26 AM, Peter Hunkeler wrote:
> There are some statements around zIIP utilization which I read here and there. Statements like:
>
> - "You should not utilize one zIIP more than 30%, two zIIPs more than 60%..."
> - "A task may become delayed for up to 3.2 ms (actually ZIIPAWMT) before the busy zIIP asks for help from a CP".
>
>
>

> For this discussion, lets assume equal speed CPs and zIIPs, and a reasonable CP to zIIP ratio, and more than on processor of each kind.
>
>
> It has been a long time strength of IBM Z (and all the predecessors) that the CPs in an LPAR can be utilized way above 90% without major problems arising. I seem to understand that this has changed lately, but still some 85% (?) should be fine.
>
>
> Now, all work running on zIIPs was once work running on CPs (and still is if there are no zIIPs). So the work is no different (apart from much being run under an SRB instead of a TCB), and the response time requirement is no different. Right?
>
>
> If so, how comes that busy zIIPs are said to be more of a problem than busy CPs? If the work can accept some queueing when run on CPs, why not when run on zIIPs. Queueing theory should apply equally to both.
>
>
>
> When a processor is busy 50%, then 50% of the time there is at least one ready task, the one executing. Maybe there are some more waiting on the work queue. But these 50% say nothing about the delay of the tasks on the work queue.
>
>
> In a simplified case, assume 5 tasks with equal priority, each one quickly, say after 0.5 ms, coming to the point where it has to give up the processor for a very short period of time before being requeued on the work queue. They all constantly work that way for 30 seconds in row, then become undispatchable for the remaining 30 seconds of that 50% busy minute. During the first 30 seconds, the zIIP is 100% busy, and after 3.2ms (ZIIPAWMT), the zIIP will ask a CP for help.
>
>
> None of the tasks has been delayed by 3.2ms, although the ZIIP recognized its work queue has not become empty for 3.2ms and asked for help. To the contrary, the work has gotten better service because two processors are now serving the single work queue. (Again for simplicity, not currently taking priorities into account).
>
>
> Same case but the task are working 1ms each time. Now it always takes more than 3.2ms for the last task on the work queue before it is being redispatched as long as the zIIP has not asked for help. But the zIIP will ask for help after 3.2ms, and the delay for the tasks will shrink.
>
>
> Isn't this a better situation for zIIP work than for non-zIIP work? Same scenario on CPs. There is no-one to help.
>
>
> Any thoughts?
>
>

I suspect the point of the guidelines was primarily to insure that work
intended for zIIPs didn't get redirected to general CPs and raise z/OS
software charges.  But, if those guidelines have been observed, zIIP
workloads would have been experiencing minimal CPU queueing delays and
any increase in delay from higher zIIP utilization and offloading part
of the workload to CPs may be perceived by users as a significant
performance hit.

How prevalent are installations today where the CPs run at top speed, in
other words at the same speed as zIIP engines? In other words, Is it
that valid to assume equal speed processors?  Clearly guidelines for
lower zIIP utilization matter more when there is a difference, as
offloading any zIIP work to a slower CP would elongate processing and
response time, even if there is no delay waiting for a processor.

Another possible issue is that application work that is zIIP-eligible at
least historically tended to be things like java that were more
CPU-intensive.  These applications quite possibly are in a service class
of lesser importance.  As long as they can run on a zIIP engine, they
only compete with similar workloads.  But, if the zIIP utilization
reaches the point that work is offloaded to a CP, it would seem logical
to me to expect other "more important" workloads competing for the CP to
get served first, and if that is indeed the case the queueing time to
actually get service for lower importance zIIP workload from an equally
busy CP could be much greater.  It doesn't seem valid to me to assume
that just because another CP is allowed to to zIIP work that a CP will
immediately be free to actually do that work, or that a CP will have
queueing delays similar to the zIIP engines -- the workload on the CP is
totally different.

    Joel C. Ewing

--
Joel C. Ewing, Bentonville, AR jce...@acm.org

[Parwez Hamid]

Re the comment:

How prevalent are installations today where the CPs run at top speed, in other words at the same speed as zIIP engines? In other words, Is it that valid to assume equal speed processors? Clearly guidelines for lower zIIP utilization matter more when there is a difference, as offloading any zIIP work to a slower CP would elongate processing and response time, even if there is no delay waiting for a processor.

For any given Z system, the 'speed' (GHz) is always the same for all types to (PUs) processing units (CPs, IFLs, zIIPs, ICFs, IFP and SAP). In case of CPs, these can be different 'sub capacity settings' All the rest are always full capacity setting. Speed is always the same.

[Scott Chapman]

This seems to come up a lot.

I'm going to start by taking the opposite tack: you probably shouldn't run your GCPs at 90-100% busy either. Busier CPUs are generally going to have more cache contention which means the work is generally going to run "somewhat" less efficiently (i.e. more CPU time / unit of work) than if the CPUs were running less busy. Apparently some IBMer at some point measured that as a 4% increase in CPU time / unit of work per 10% increase in overall utilization. Alas I have no further details about the source of that number. I believe that it's at least directionally correct though.

Now if you have a mix of different importances / priorities for the work that is driving the machine to 100% busy, then likely the most (but not only) impacted work is the lower importance work. So maybe that's ok. But, in all likelihood, if the machine had more capacity and was running at only say 70% busy, then likely the same work would consume fewer MSUs. Which may be a good thing.

From purely a performance perspective, running less busy is always better as there's less chance for queueing for a processor. But rarely is "as fast as possible" the required and most cost effective answer.

But this question is about zIIPs. But zIIPs are the same processors as GCPs and the aforementioned discussion is mostly the same: you can run the zIIPs busy, but things may not be as efficient. Of course "less efficient" matters a little less on the zIIPs given that the hardware is cheaper and they don't increase software costs.

The primary issue as I see it comes in where the zIIPs are running busier than the GCPs and so work is more delayed trying to get through the zIIP queue than if they had been just dispatched directly to the GCP. If that work is very important work (such as perhaps DB2 system tasks) then that could have relatively widespread negative impacts. Are those impacts greater than if there were no zIIPs in the configuration at all and the GCPs were running a similarly busy level? Maybe a bit due to the extra overhead of (unsuccessfully) scheduling work onto the zIIP.

But I believe the potential for harm is very situation dependent. In particular with the way the rules are today, and with the mix of the different workloads that are zIIP eligible, if you have zIIP capacity (both in terms of MIPS and engines) greater or equal to your GCP capacity, I'm hard pressed to believe that there's a significant risk to running the zIIPs as busy as you're comfortable running your GCPs. But my belief is also that many are comfortable running their GCPs hotter than is really ideal.

In Kathy Walsh's presentation from Orlando one of her slides has the statement: "Can run zIIPs very busy IF there are multiple classes of work with different response time objectives, but watch IIPCP time". I think that's a very reasonable statement. If you're starting to see significant crossover from the zIIPs to the GCPs, you're probably running the zIIPs too busy for that particular workload. Note that the crossover amounts are not necessarily well-correlated with zIIP busy, although generally when zIIPs are busy they are more likely to see significant crossover.

A potential issue is that some systems only have a single zIIP. When you have a single CP, your threshold for where you'll start feeling pain is significantly lower vs. having evern 2 CPs. The situation of having a 710 with a single zIIP is a significantly different situation than having a say a 505 with 4 zIIPs. I'm going to be a lot more concerned about zIIP utilization in the former vs. the latter. In the former, that zIIP very well might become a bottleneck that could be more problematic than not having a zIIP at all. That would seem to be much less likely in the second case.

My personal belief is that given the amount of work that's zIIP eligible today, most systems should have at least 2 zIIPs, and ideally the system should have zIIP capacity at least similar to the GCP capacity. Yes, there's a hardware cost to that, but in the grand scheme of things, the costs are not nearly as significant as GCP capacity, so err on the side of having too much zIIP capacity. That would be an interesting study: what's common ratio of zIIP capacity to GCP capacity? I suspect that that ratio has been creeping up over the years.

Scott Chapman

[Peter Hunkeler]

>.... the workload on the CP is totally different.



Is it? If you think about Java, maybe. But when it comes to workload such as DB2, Sort, Monitors, that have shifted more and more of its task towards zIIPs, isn't this still the same workload?


--
Peter Hunkeler

[Peter Hunkeler]

>How prevalent are installations today where the CPs run at top speed, in other words at the same speed as zIIP engines?

I haven't got the faintest idea. We do, but that doesn't matter for this discussion. I thought this is complex enough, so I take one part of complexity out first: Different speed engines.


--
Peter Hunkeler

[Joel C. Ewing]

On 06/08/2018 02:50 AM, Parwez Hamid wrote:
> Re the comment:
>
> How prevalent are installations today where the CPs run at top speed, in other words at the same speed as zIIP engines? In other words, Is it that valid to assume equal speed processors? Clearly guidelines for lower zIIP utilization matter more when there is a difference, as offloading any zIIP work to a slower CP would elongate processing and response time, even if there is no delay waiting for a processor.
>
> For any given Z system, the 'speed' (GHz) is always the same for all types to (PUs) processing units (CPs, IFLs, zIIPs, ICFs, IFP and SAP). In case of CPs, these can be different 'sub capacity settings' All the rest are always full capacity setting. Speed is always the same.

I can't find a decent explanation of how the sub capacity enforcement on
a CP is managed beyond "the clock frequency... remains unchanged ...
adjustment is achieved through other means".  That statement doesn't
actually say that instruction execution speed in unaffected, only that
clock frequency is constant:  one way to enforce 50% capacity at the
same clock frequency would be the old IBM 407 approach of running at
full speed but doing  no productive work on half the clock cycles;  but,
considering all the parallelism in today's processors, I suspect it
would be much simpler to just force the entire CP to be idle or
non-dispatchable for an interval of time when the utilization exceeds
allowable levels. 

Would be curious if anyone has seen a more complete description about
how the sub-capacity CP enforcement works. 

On 06/08/2018 02:36 PM, Peter Hunkeler wrote:
>
>> .... the workload on the CP is totally different.
>
>
> Is it? If you think about Java, maybe. But when it comes to workload such as DB2, Sort, Monitors, that have shifted more and more of its task towards zIIPs, isn't this still the same workload?
> --
> Peter Hunkeler

The zIIP-eligible criteria for choosing a subset of  tasks to run on
zIIP engines, as I understand it, has nothing to do with installation
defined service classes but is totally  based on IBM marketing
strategy.  There is no reason to expect the mix of tasks eligible for
zIIP resources to have the same service-class mix and CPU/IO usage
patterns as those restricted at that same time to CP resources --the
zIIP utilization may even peak at a totally different time of day.  The
total system workload may be the same as before things were made zIIP 
eligible, but with artificial separation into those that prefer a zIIP
and those that must run on a CP, that workload is now artificially
subdivided into two distinct and different workload subsets when
competing for CPU resources.  If zIIP utilization forces something that
would normally run on zIIP onto a CP, it is now competing for CPU
resources with a different subset of that total workload, and it would
be surprising if that shift didn't affect response time.

    Joel C. Ewing

--
Joel C. Ewing, Bentonville, AR jce...@acm.org

[Ed Jaffe]

On 6/8/2018 2:06 PM, Joel C. Ewing wrote:
>
> Would be curious if anyone has seen a more complete description about
> how the sub-capacity CP enforcement works.

Every so often (I believe 4 usec on my z13s), there is something akin to
a timer interrupt in the chip that runs a millicode housekeeping
routine. At the end of that routine, before returning to running normal
instructions, a tight CPU loop is executed to burn up any excess
capacity you didn't pay for.

--
Phoenix Software International
Edward E. Jaffe
831 Parkview Drive North
El Segundo, CA 90245
http://www.phoenixsoftware.com/
--------------------------------------------------------------------------------
This e-mail message, including any attachments, appended messages and the
information contained therein, is for the sole use of the intended
recipient(s). If you are not an intended recipient or have otherwise
received this email message in error, any use, dissemination, distribution,
review, storage or copying of this e-mail message and the information
contained therein is strictly prohibited. If you are not an intended
recipient, please contact the sender by reply e-mail and destroy all copies
of this email message and do not otherwise utilize or retain this email
message or any or all of the information contained therein. Although this
email message and any attachments or appended messages are believed to be
free of any virus or other defect that might affect any computer system into
which it is received and opened, it is the responsibility of the recipient
to ensure that it is virus free and no responsibility is accepted by the
sender for any loss or damage arising in any way from its opening or use.

[Christopher Y. Blaicher]

I wish Peter Relson would comment on this so we all can get the straight answer, but he may not be able to.

From what little I know and most that I summarize and guess at, it seems the following to be what is happening.

First of all, the dispatcher code for ZIIP processing is not the same as the GP dispatcher.

I think the queuing process is basically FIFO and once unit of work is dispatched, it runs to the end or a program break point happens. A program break point could be some form of PAUSE or a IEAVXTSW wait.

Some of this dispatcher design by IBM could be based on the assumption that all the work is SRB and will be high priority work and of short duration.

There is the ZIIPAWMT parameter in IEAOPTxx which affects how often idle zIIPs get woken up to see if there is work. This has two effects. A unit of work put on the zIIP queue has to wait up to that long to get dispatched on a zIIP, or if at the end of that interval all zIIPs are busy, only then will the unit of work be dispatched on a GP. To be fair, when a zIIP completes a unit of work, it checks for more work to process, so not everything waits a dispatch cycle.

So what does this all mean? In a low activity environment, your mean time to dispatch is around half the ZIIPAWMT time. In a high activity environment it means the mean time to dispatch on a GP because all he zIIPs are busy is ZIIPAWMT.

This starts to get into queuing theory, but if a single zIIP processor is over 30% busy, about 30% or more of the work to run on a zIIP will wind up waiting the ZIIPAWMT time and then get dispatched on a GP, thus adding the insult of delaying the processing of the work to the injury of running on a GP that can cost real money.

If you add more zIIP engines, there is a greater chance of a unit of work ending on one of the engines and thus the work getting dispatched on a zIIP without waiting the full ZIIPAWMT.

A reasonable set of indicators can be gotten from the SMF type 30 record. The SMF30_TIME_ON_ZIIP and SMF30_TIME_ZIIP_ON_CP are valuable indicators.

If SMF30_TIME_ZIIP_ON_CP is 30% or more of SMF30_TIME_ON_ZIIP, then you probably need another zIIP engine. Because type 30 records are job centric, I would suggest using the SMF 30 SUBTYPE 2 and 3 records for all the jobs in a time interval and totaling the data. Using data from a single job or group of jobs may not provide a complete picture. Also, the 30% value mentioned maybe more or less than your environment can tolerate. YMMV, so only use it as a starting point.

As I said, most of this is just a guess at what is happening with zIIP dispatching. None of this information has been validated and neither Syncsort or I are implying or stating any course of action a user should pursue. Each user must evaluate their own needs and requirements and make decisions based solely on their own research and evaluation of their circumstances.

Chris Blaicher
Technical Architect
Mainframe Development
P: 201-930-8234 | M: 512-627-3803
E: cbla...@syncsort.com

Syncsort Incorporated
2 Blue Hill Plaza #1563
Pearl River, NY 10965
www.syncsort.com

Data quality leader Trillium Software is now a part of Syncsort.

-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On Behalf Of Peter Hunkeler

________________________________



ATTENTION: -----

The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.

[Mike Schwab]

I have seen press stories where Full speed CPs see a lot of wait time
(waiting for data, interrupt, etc.), where slower CPs credit the
delays to the reduced capacity so they will see little to no wait
time.

--
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

[Peter Hunkeler]

>> Is it? If you think about Java, maybe. But when it comes to workload such as DB2, Sort, Monitors, that have shifted more and more of its task towards zIIPs, isn't this still the same workload?
>> --
>> Peter Hunkeler
>
>The zIIP-eligible criteria for choosing a subset of tasks to run on
zIIP engines, as I understand it, has nothing to do with installation
defined service classes but is totally based on IBM marketing
strategy. There is no reason to expect the mix of tasks eligible for
zIIP resources to have the same service-class mix and CPU/IO usage
patterns as those restricted at that same time to CP resources --the
zIIP utilization may even peak at a totally different time of day. The
total system workload may be the same as before things were made zIIP
eligible, but with artificial separation into those that prefer a zIIP
and those that must run on a CP, that workload is now artificially
subdivided into two distinct and different workload subsets when
competing for CPU resources. If zIIP utilization forces something that
would normally run on zIIP onto a CP, it is now competing for CPU
resources with a different subset of that total workload, and it would
be surprising if that shift didn't affect response time.

Didn't think that way when I read your previous statement. I now see what you meant and I agree. Very good point. Thanks.


--
Peter Hunkeler

[Martin Packer]

To correct one thing: Not all the work is high priority. Three examples:



1) DDF threads selected to be zIIP-eligible.



2) Java - via zAAP on zIIP.



3) System XML



Cheers, Martin



Sent from my iPad



> On 8 Jun 2018, at 22:49, Christopher Y. Blaicher <cbla...@SYNCSORT.COM>

wrote:

>Unless stated otherwise above:

IBM United Kingdom Limited - Registered in England and Wales with number 741598.

Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU

[Peter Hunkeler]

>Some of this dispatcher design by IBM could be based on the assumption that all the work is SRB and will be high priority work and of short duration.

This doesn't sound correct to me. Client SRBs (preemptive SRBs) were invented to have some work done in another address space at client priority. Enclave SRBs were invented to let those client SRBs be run on zIIPs. So, there is a mix of SRBs with different priorities (WLM goals, WLM service classes) competing for zIIP capacity.


And it's all preemptive kind of work.

[Peter Hunkeler]

>First of all, the dispatcher code for ZIIP processing is not the same as the GP dispatcher.

Do you know this, or is it just an assumption on your side? After all I read, it still would't make sense to me.


If you think of the "need help" process for the zIIP to be special, isn't there a similar process for CPs? With hiperdispatch, the system tries to redispatch work to the same (L)CP as much as possible. In support of this, the CPs are grouped into nodes (I believe that is the term). Each node mainly serving its own, separate work init queue. But CPs from other nodes will help if one node becomes overloaded.
The above is how I remember it from a discussion I had with Robert Vaupel (IBM).

[Joel C. Ewing]

That is consistent with Ed Jaffe's explanation of there being a
short-period (4µsec), periodic millicode CPU loop to eat up the excess,
un-bought CP capacity.  That would have the effect of making a program
running on a subcapacity CP run slower and longer with a lower average
instructions/sec rate than on a full speed CP, even though individual
instructions when doing productive work would run at full clock speed.  

With a reduced effective execution rate on a subcapacity CP, the balance
between CPU and I/O speeds  is shifted, and one would expect reduced I/O
wait time when running on the effectively-slower CP.
    Joel C. Ewing

On 06/08/2018 05:26 PM, Mike Schwab wrote:
> I have seen press stories where Full speed CPs see a lot of wait time
> (waiting for data, interrupt, etc.), where slower CPs credit the
> delays to the reduced capacity so they will see little to no wait
> time.
> On Fri, Jun 8, 2018 at 4:48 PM Christopher Y. Blaicher
> <cbla...@syncsort.com> wrote:

--
Joel C. Ewing, Bentonville, AR jce...@acm.org

[Ed Jaffe]

On 6/9/2018 3:48 AM, Peter Hunkeler wrote:
>> First of all, the dispatcher code for ZIIP processing is not the same as the GP dispatcher.
> Do you know this, or is it just an assumption on your side? After all I read, it still would't make sense to me.

The dispatcher is the dispatcher -- and it works with different WUQs. No
need to make things overly complex...

--
Phoenix Software International
Edward E. Jaffe
831 Parkview Drive North
El Segundo, CA 90245
http://www.phoenixsoftware.com/
--------------------------------------------------------------------------------
This e-mail message, including any attachments, appended messages and the
information contained therein, is for the sole use of the intended
recipient(s). If you are not an intended recipient or have otherwise
received this email message in error, any use, dissemination, distribution,
review, storage or copying of this e-mail message and the information
contained therein is strictly prohibited. If you are not an intended
recipient, please contact the sender by reply e-mail and destroy all copies
of this email message and do not otherwise utilize or retain this email
message or any or all of the information contained therein. Although this
email message and any attachments or appended messages are believed to be
free of any virus or other defect that might affect any computer system into
which it is received and opened, it is the responsibility of the recipient
to ensure that it is virus free and no responsibility is accepted by the
sender for any loss or damage arising in any way from its opening or use.

[Christopher Y. Blaicher]

Peter,
I started off as saying, a lot of the descriptions are based on assumptions, as IBM has let out little on how the zIIP dispatcher works.

Also, I was only talking about SRBs on zIIPs, so non-enclave SRBs were not part of the discussion.

I believe hyper dispatch is very different from zIIP dispatch. I stand by my assumption that GP dispatch is very different from zIIP dispatch, or why would there be the ZIIPAWMT parameter and have the comment about waking up after that interval to see if there is work. When non-zIIP work comes ready and a processor is free, the work is dispatched. No waiting. The comments in the parameter description for ZIIPAWMT is describing a polling environment, at least to me.

I have looked through the internet, OK not the be-all and end-all but a reasonable place to start, and there is a lot on what can get dispatched on a zIIP, but no detail on how.

As I said, I wish someone from IBM would at least chime in with some level of description as to how zIIP dispatch works and why high zIIP utilization rates are, shall we say, not good. Then maybe we can stop guessing.

Chris Blaicher
Technical Architect
Mainframe Development
P: 201-930-8234 | M: 512-627-3803
E: cbla...@syncsort.com

Syncsort Incorporated
2 Blue Hill Plaza #1563
Pearl River, NY 10965
www.syncsort.com

Data quality leader Trillium Software is now a part of Syncsort.


-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@LISTSERV.UA.EDU] On Behalf Of Peter Hunkeler
Sent: Saturday, June 9, 2018 6:49 AM
To: IBM-...@LISTSERV.UA.EDU
Subject: AW: Re: Why are highly busy zIIPs worse than highly busy CPs?

________________________________



ATTENTION: -----

The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.

[Jim Mulder]

zIIP dispatching is the same as GP dispatching. ZIIPAWMT
has analogous parameters for GP (CCCAWMT) and zAAP (ZAAPAWMT).
Alternate wait management was created long before there were
specialty engines.

Jim Mulder z/OS Diagnosis, Design, Development, Test IBM Corp.
Poughkeepsie NY

"IBM Mainframe Discussion List" <IBM-...@LISTSERV.UA.EDU> wrote on
06/09/2018 10:58:25 AM:

> From: "Christopher Y. Blaicher" <cbla...@SYNCSORT.COM>
> To: IBM-...@LISTSERV.UA.EDU
> Date: 06/09/2018 12:46 PM
> Subject: Re: Why are highly busy zIIPs worse than highly busy CPs?
> Sent by: "IBM Mainframe Discussion List" <IBM-...@LISTSERV.UA.EDU>
>

> Peter,
> I started off as saying, a lot of the descriptions are based on
> assumptions, as IBM has let out little on how the zIIP dispatcher works.
>
> Also, I was only talking about SRBs on zIIPs, so non-enclave SRBs
> were not part of the discussion.
>
> I believe hyper dispatch is very different from zIIP dispatch. I
> stand by my assumption that GP dispatch is very different from zIIP
> dispatch, or why would there be the ZIIPAWMT parameter and have the
> comment about waking up after that interval to see if there is work.
> When non-zIIP work comes ready and a processor is free, the work is
> dispatched. No waiting. The comments in the parameter description
> for ZIIPAWMT is describing a polling environment, at least to me.
>
> I have looked through the internet, OK not the be-all and end-all
> but a reasonable place to start, and there is a lot on what can get
> dispatched on a zIIP, but no detail on how.
>
> As I said, I wish someone from IBM would at least chime in with some
> level of description as to how zIIP dispatch works and why high zIIP
> utilization rates are, shall we say, not good. Then maybe we can
> stop guessing.

[Christopher Y. Blaicher]

Jim,
Thank you for the clarification, however, that still leaves open the question of WHY more than 30% on a single zIIP or 60% on two is not a good idea.

That is what triggered this whole chain of emails. Why can GPs run reasonably well at over 95% and zIIPs struggle, or so people have reported. We are a development shop and so rarely, if ever, push them that hard. We have, but that was doing things you would not do in the real world, like having a loop in your code you didn't know about. :)

If you can't tell us the why, can you tell us what reasonable utilization numbers are for a range of numbers of zIIPs? I think the user community as a whole is interested.

Chris Blaicher
Technical Architect
Mainframe Development
P: 201-930-8234 | M: 512-627-3803
E: cbla...@syncsort.com

Syncsort Incorporated
2 Blue Hill Plaza #1563
Pearl River, NY 10965
www.syncsort.com

Data quality leader Trillium Software is now a part of Syncsort.


-----Original Message-----

________________________________



ATTENTION: -----

The information contained in this message (including any files transmitted with this message) may contain proprietary, trade secret or other confidential and/or legally privileged information. Any pricing information contained in this message or in any files transmitted with this message is always confidential and cannot be shared with any third parties without prior written approval from Syncsort. This message is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any use, disclosure, copying or distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and/or Syncsort and destroy all copies of this message in your possession, custody or control.

[Peter Relson]

>the 'speed' (GHz) is always the same for all types

The 'speed' is not generally thought to be the cycle time. It is true that
the cycle time is the same across the models, but the effective speed is
not the same (such as MIPS or the MSU rating).

>the dispatcher code for ZIIP processing is not the same as the GP
dispatcher.

Well, it is "the same" but there are swizzles (such as the fact that a
zIIP processor will not run CP work, but a CP might run zIIP work).

>the queuing process is basically FIFO

The queueing process is the same. For all processors it can be thought of
as FIFO for a specific priority (where the priority has many sub-factors)

>It runs to the end or a program break point happens. A program break

point could be some form of PAUSE or a IEAVXTSW wait.

zIIP work is timed just like CP work. There are dispatch durations
established and when the duration is exceeded, the timeslice ends and the
work will be undispatched if there is equal or higher priority work to
run. That applies regardless of the type of processor. It might be, but is
not necessarily, the case that the dispatch durations are the same for the
different processor types.

You really want Gary King or Dan Rosa to chime in, not me. I can rarely
keep the details straight. If they provide me with information to relay on
their behalf, I will be glad to.

Much of the discussion has been on target. If zIIPs are "overloaded" then
the work queues for zIIP will get "too long" and the zIIPs will ask for
"help". The percentages mentioned to some extent translate into how likely
it is that something will not find the resources available and then must
"wait" (or get someone else to do it). "Help" is a very complex process
which can include helping from other zIIPs (in part related to
hiperdispatch trying to run work as close as possible to the original work
to limit negative cache effects) and possibly CPs.

If you're running CPs at 100% and zIIPs need "help", then something is
going to suffer. You just don't have enough resources for the work you're
trying to accomplish.
Then there are also the questions of whether or not you allow CPs to help
or insist that zIIP-eligible work run only on zIIPs.

Peter Relson
z/OS Core Technology Design

[Peter Hunkeler]

>I believe hyper dispatch is very different from zIIP dispatch. I stand by my assumption that GP dispatch is very different from zIIP dispatch, or why would there be the ZIIPAWMT parameter and have the comment about waking up after that interval to see if there is work. When non-zIIP work comes ready and a processor is free, the work is dispatched. No waiting. The comments in the parameter description for ZIIPAWMT is describing a polling environment, at least to me.

There is CCAWTM which does the same thing as ZAAPAWMT and ZIIPAWMT but for CPs. I think your statement about CP work being dispatched immediately when a CP is free, does not hold true.


It was before "alternate wait management" was introduced (I don't remember when that was), when newly ready, higher priority work had interrupted lower priority work on a CP, or woken up a waiting CP.

>I have looked through the internet, OK not the be-all and end-all but a reasonable place to start, and there is a lot on what can get dispatched on a zIIP, but no detail on how.

The last five words are the very reason I started this thread. When it comes to zIIPs, there is much only fuzzy information out there. I did google and read every presentation I could get a hand on.


So, coming back to one of my questions: The fuzzy information say that zIIP eligible work will suffer from a ZIIPAWMT delay when the zIIPs are overloaded and are asking for help from CPs.


You mentioned the ZIIPAWMT delay *before* zIIPs idle zIIPs will start to work on newly arrived work. Ok, this is part of the delay. But as soon as there is a steady load of zIIP work, that start up delay will not occur again (think of my simplified case I initially described). Similarly the "ask for help ZIIPAWMT delay" will occur only once when there is work constantly seen on the the zIIP WUQ.


I still think that, theoretically, zIIP eligible work will be served somewhat better than non zIIP eligible work *when* the zIIPs is being helped (depending on the works priority, of course). There are two processors looking after the single work queue. Oversimplified, you think? Maybe but its my starting point anyway.

[Peter Hunkeler]

>zIIP dispatching is the same as GP dispatching. ZIIPAWMT
has analogous parameters for GP (CCCAWMT) and zAAP (ZAAPAWMT).
Alternate wait management was created long before there were
specialty engines.

Thank, Jim, much appreciated.
Sorry, guys, for not reading the latest posts before writing mine talking about CCCAWMT....


--
Peter Hunkeler

[Peter Hunkeler]

>You really want Gary King or Dan Rosa to chime in, not me. I can rarely
keep the details straight. If they provide me with information to relay on
their behalf, I will be glad to.

Thanks, for the offer. How about forwarding my initial post. I had written the points of interest to me, and I think to many here.


--
Peter Hunkeler

[Timothy Sipples]

Scott Chapman wrote:
>A potential issue is that some systems only have a single

>zIIP.... My personal belief is that given the amount of work

>that's zIIP eligible today, most systems should have at least

>2 zIIPs....

I would add an important footnote. The current and prior model machine
generations have zIIP engines that are capable of simultaneous
multithreading (SMT). To a first order approximation anyway, one zIIP now
behaves like two when SMT is enabled. Even if you have one zIIP, enabling
SMT2 could be a reasonable way to get the "rule of thumb pair" you
describe.

SMT2 support for zIIPs is available in z/OS 2.1 and higher on IBM z13
machines. Please refer to APARs OA43622 and OA43366.

--------------------------------------------------------------------------------------------------------
Timothy Sipples
IT Architect Executive, Industry Solutions, IBM Z & LinuxONE,
Multi-Geography
E-Mail: sip...@sg.ibm.com

[srigan...@gmail.com]

Going with an SMT will be an good option. Just incase if SMT is enabled in 2.1, It is recommended to go with PTF UA83506 to avoid the system/SRB request to be in a stalled state and the same goes with UA83505 for 2.2

More details at: http://www-01.ibm.com/support/docview.wss?uid=isg1OA51419

.. In addition to SMT, IIPHonorPriority is a good option too. Even if the zIIP is busy, if we have an IIPHONORPRIORITY set to YES, ideally the SRB workloads will be routed to GCP (Setting IIPHONORPRIORTY to NO will park the SRB queues for zIIP processing rather than routing to GCP which will end-up in having a delay or worse situation)