i5 performance problems

[tom]

Hi.

My client works on 9771-MMA machine, i5OS 5.4.
From time to time they notice performance problems.

I check on WRKACTJOB scree that CPU usage is about 40%,
on WRKSYSSTS that DB util. is about 15%.
So everything looks fine.

They claims that their queries runs very slow (usually runs much faster).

What other thing shall I check. Maybe LAN utlization? How?

Regards,
Tomasz

[iseriesflorida]

I would probably install performance tools and gather some statistics
on your machine, you can install it and run it for up to 70 days.

5722PT1

You could also do a wrkactjob seq(*cpu)

[Kirk]

I would also look at WRKDSKSTS and look at the far right col to see how
busy the disks are. They should be 40%

[Brad]

As well as PT1, there are some good performance monitoring tools in
Management Central section of System i Access, you may want to look
into these as they are generally easier to follow if you are new to
this.

Also, when using WRKACTJOB to check the performance of the machine you
need to be very aware of the elasped time field at the top, as the
figures you are getting are an average percentage over that period.
So, if you have an elasped time of 60mins and a problem that only
started 30seconds ago then the CPU % associated with the problem job
will still be low, try pressing F10 to reset stats.

If all else fails then you can try WRKSYSACT

Good luck, Brad

[WDS]

I'd also try WRKSYSSTS and see if anything looks odd.

[Grzegorz Goryszewski]

tom pisze:

Hi,
first of all run 'Collection Services'.
Then CRTPFRDTA and play with PRT* commands like PRTSYSRPT,PRTCPTRPT, and
so on go cmdpfr .
You can also turn on 'expert cache' if the problem is batch processing
related. If You know which job is all about try PEX (
http://www.redbooks.ibm.com/abstracts/sg247457.html?Open ).
There is a good instrumentation in i5 nowadays (better then in Oracle ;)) .
Regards.
Grzegorz

[tom]

Hi all.

1) 5722PT1 - application expierd few months ago :( so WRKSYSACT won't work
2) WRKDSKSTS - % Busy column shows values between 12-20% - isn't it normal?
3) MGTC - unfortunatelly QYPSJSVR job is END status for long time and I
can't run Management Central. IBM want's me to IPL machine to fix it - I
can't do it right now :( It's production system.
4) WRKSYSSTS look quite normal:

% CPU used . . . . . . . : 61,8
% DB capability . . . . : 27,6
Elapsed time . . . . . . : 00:00:01
Jobs in system . . . . . : 24096
% perm addresses . . . . : 0,106
% temp addresses . . . . : 3,712
------
System ASP . . . . . . . : 2328 G
% system ASP used . . . : 28,0533
Total aux stg . . . . . : 2328 G
Current unprotect used . : 38123 M
Maximum unprotect . . . : 60474 M

5) I can't run Collection Services because of problem with job QYPSJSVR

So, is IPL the only way to find out what's wrong?

Tomasz

tom pisze:

[Grzegorz Goryszewski]

tom pisze:
> Hi all.

> 2) WRKDSKSTS - % Busy column shows values between 12-20% - isn't it normal?

quite normal

> 4) WRKSYSSTS look quite normal:
>
> % CPU used . . . . . . . : 61,8
> % DB capability . . . . : 27,6
> Elapsed time . . . . . . : 00:00:01
> Jobs in system . . . . . : 24096
> % perm addresses . . . . : 0,106
> % temp addresses . . . . : 3,712
> ------
> System ASP . . . . . . . : 2328 G
> % system ASP used . . . : 28,0533
> Total aux stg . . . . . : 2328 G
> Current unprotect used . : 38123 M

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^36 GB in temp objects is
suspicious. Is this value increasing with time ?

> Maximum unprotect . . . : 60474 M

> So, is IPL the only way to find out what's wrong?

Yep, as always :).

IF You know which jobs got performance problems try to monitor them with
STRTDBMON
http://www-1.ibm.com/servers/eserver/iseries/perfmgmt/pdf/dbmonitor.pdf
looks like temporary indexes may have been created for 'optimal access
path'.

GG

[Brad]

Hi Tom

Assuming your PT1 license grace period was just the 70 day one from
loading, you should still be able to get one free temp free key from
IBM Key Centre (phone number here in the UK is 0800965441, not sure
where you are but you should be able to track the equivalent down in
your region) which will give you between 2 and 4 weeks to use the PT1
tool.

I am suspicious about the amount of Temp Space your system is and has
been using; these figures aren't in themselves proof a problem but a
red flag.

Good luck! Brad

[tom]

Hi
I'll talk with IBM about PT1.

What shall I do about temp space?
Where shall I look for the reasons of that behaviour?
Is there any possibility that MIMIX causes it? There is lot of DB
operations that mimix need to propagete on second machine.

Regards,
Tomasz

Brad pisze:

[walker.l2]

> What shall I do about temp space?

Buy more RAM, or run fewer programs / applications /queries
simultaneously, or re-write your programs / applications / queries to
use less memory.

> Where shall I look for the reasons of that behaviour?

See which memory pools are showing high paging and faulting values. If
you are running everything in *BASE and *INTERACT, separate different
applications into their own memory pools first.

> Is there any possibility that MIMIX causes it? There is lot of DB
> operations that mimix need to propagete on second machine.

Doubtful. We run MIMIX, and have never seen it use excessive amounts
of memory.

[WDS]

On Mar 20, 6:49 am, "walker.l2" <walker...@ukonline.co.uk> wrote:
> > What shall I do about temp space?
>
> Buy more RAM, or run fewer programs / applications /queries
> simultaneously, or re-write your programs / applications / queries to
> use less memory.

See if you can track it down to one program. I've seen some that run
for long, long times that don't completely free up memory they are
using until they exit.

[Brad]

It could be Mimix and if so then getting up to the latest version will
help but not remove it, although I liked the install more RAM idea, on
power5 servers IBM used to say budget 8GB per full processor core
active on power6 it's now 16GB.

Before you get too concerned about that get your perf tools running
and see what's going on.

Brad

[tom]

MIMX is latest version.
It's 9117-MMA machine, this partition has 1,7 CPU and 23GB RAM assigned.

tomasz

Brad pisze:

[tom]

Hi

My memory pools looks like:

Defined Max Allocated
Pool Size (M) Active Size (M)
*MACHINE 1559,01 +++++ 1559,01
*BASE 9083,07 716 9083,07
*INTERACT 12594,20 587 12594,20
*SPOOL 234,70 15 234,70

I also see that high paging and faulting is for pool *BASE and *INTERACT.

Pool Reserved Max ----DB----- --Non-DB---
Size M Size M Act Fault Pages Fault Pages
1527,83 553,45 +++++ 0,0 0,0 12,1 24,1
9098,66 67,23 716 51,8 336,2 367,2 899,0
12609,79 0,00 587 13,9 82,9 60,5 110,4
234,70 0,00 15 0,0 2,7 0,2 0,5

There is few programs on that system that runs in QBATCH for many hours
and do lot of queries for big number of physical files. I suppose that
programs can cause my problems. Before thay run I don't have any
performance problems.

Can you advice me if I should create another memory pool just for
programs I mentioned above? As far as I know then I should add pool
definition into QBATCH SBS definition and somhow make these programs to
use that pool.

tomasz

walker.l2 pisze:

[walker.l2]

On Mar 20, 1:35 pm, tom <t...@nospam.com> wrote:
> Hi
>
> My memory pools looks like:
>
>                 Defined    Max   Allocated
>   Pool         Size (M)  Active  Size (M)
>   *MACHINE      1559,01   +++++     1559,01
>   *BASE         9083,07     716     9083,07
>   *INTERACT    12594,20     587    12594,20
>   *SPOOL         234,70      15      234,70
>
> I also see that high paging and faulting is for pool *BASE and *INTERACT.
>
>      Pool   Reserved    Max  ----DB-----  --Non-DB---
>     Size M   Size M     Act  Fault Pages  Fault Pages
>    1527,83    553,45  +++++    0,0   0,0   12,1  24,1
>    9098,66     67,23    716   51,8 336,2  367,2 899,0
>   12609,79      0,00    587   13,9  82,9   60,5 110,4
>     234,70      0,00     15    0,0   2,7    0,2   0,5
>
> There is few programs on that system that runs in QBATCH for many hours
> and do lot of queries for big number of physical files. I suppose that
> programs can cause my problems. Before thay run I don't have any
> performance problems.
>

Your QBATCH stuff will be running in the 9G assigned to *BASE, what
5250 stuff are you running that burns through 16G? That seem
suspiciously high to me, I suspect you have a poorly performing query
running somewhere behind a 5250 session.

If everything is left to auto-tune, OS400 will move RAM from *BASE to
*INTERACT to preserve low user response times, in your system this
seems to be causing a lot of paging and faulting in *BASE (meaning
your batch programs would prefer more RAM).

[Mark S Waterbury]

Check QSYSOPR message queue (DSPMSG QSYSOPR) around the time the problem
occurs, and see if you are getting messages like:

Interactive usage approaching installed capacity

Certain SQL queries over some files or tables that might not have all
the necessary indexes could cause the system to build temporary indexes
and this can be quite expensive in terms of CPU and I/O, and could help
to push your system "over the limit" if someone is running SQL queries
interactively (or embedded SQL in an interactive program).

You can also issue DSPLOG for a range of dates/times shortly prior to
until somewhat after the last time the problem occurred, to see if any
interesting messages show up there.

HTH

[tomasz]

HI.
I'm wondering why interactive jobs takes 16GB.
There are only about 70-80 interactive sessions.
Can I somehow measure/check what interactive jobs consumes such a big
amount of RAM?

Most of work on DB is via ODBC connections from unix system with PHP
application.
Perhaps some queries are written in wrong way (indexes etc.) but how
can I find out who/what program
does it? If I get an information who/what I can prevent them to do it
this way.

Regards,
Tomasz