UniData performance tuning
Dave
Are your processes globally catalogued ?
Are you using System Builder ?
Barry Burtenshaw wrote:
> I'm looking for information on UniData performance tuning. We are in the
> process of centralizing about 30 UniData databases from remote, smaller, older
> RS/6000 systems onto 1 new IBM RS/6000 S7A with 4 262MHz cpus and 4GB of RAM.
> We are using EMC for storage. To date we have migrated about 24 of these
> database.
>
> So far, average cpu utilization on the new system is less than 20%. Doesn't
> appear that we have any memory or io bottlenecks. Yet the users say the
> performance isn't that great. Admittedly, some of their reports run
> significantly faster, maybe 4 times. However this system is probably 10-50
> times faster than any of the old boxes. I would have expected stuff to run
> even faster. Is UniData the limiting factor here?
>
> Anyone have any suggestions on how to tune UniData to take advantage of the
> horsepower of the new system? Any advice on udtconfig parameters?
>
> Thanks in advance,
> Barry
Barry Burtenshaw
Barry Burtenshaw
>
>Are your processes globally catalogued ?
>
>Are you using System Builder ?
We are running an application called PRIMAC from Vercom.
http://www.vercom.com/vercom.htm
Unfortunately, I know very little about UniData. So I've no idea if the
processes are "globally cataloged".
Ken Wallis
>I'm looking for information on UniData performance tuning. We are in the
>process of centralizing about 30 UniData databases ... to date we have
migrated about 24 of these databases.
>
>So far, average cpu utilization on the new system is less than 20%.
Doesn't
>appear that we have any memory or io bottlenecks. Yet the users say the
>performance isn't that great.
[snip]
>Anyone have any suggestions on how to tune UniData to take advantage of the
>horsepower of the new system? Any advice on udtconfig parameters?
and also , in follow up, wrote:
>We are running an application called PRIMAC from Viacom.
>... I've no idea if the processes are "globally cataloged".
PRIMAC is a print business vertical app. It doesn't use SB+, but IIRC more
recent versions use a 4GL which is known as POSH in Australia, but which may
simply be engineered into Vercom's products without a 'brand name'.
I presume the concern was that 4GL's can introduce strange overheads and odd
bottlenecks on parameter files. Certainly it's worth bearing in mind.
Are you centralising into a single larger database with 30 small print shops
data now consolidated, or are you running 30 distinct databases but with a
single faster box? If the former, then there may be all sorts of
interesting file sizing factors coming into play as more and more data gets
squeezed into your files. If simply the latter then how has your connection
model changed? Were the individual print shops running terminals and PC's
off serial lines direct into async ports on their smaller RS/6000s and now
are they connecting in over a wide area network? If so, then response may
seem a bit poor simply because direct asynchronous ports tend to buffer up
characters much more snappily than telnet over TCP/IP does.
How did you determine the current set of udtconfig parameters? Are they
essentially the same as one of the original sites, but with the NUSERS
parameter higher? Or did you just install udt and let it configure itself?
By all means email me the udtconfig and the output from the UNIX level
command "kp" (run it as root - it needs to read /dev/kmem) offline and I'll
let you know if there's anything glaringly obvious wrong.
Do you have UniData's monitoring tool licensed? Try running "udtmon" and
seeing if you get a license problem. This will let you see what Unidata is
spending its time on - reads/ writes/ overflowed files/ cataloged program
calls - from there you can work out if anything needs attention.
Hope that helps you progress it a little further, come back to me here or
offline when you want to go further.
Best Regards
Ken Wallis
Empower Data Solutions Pty Limited, ACN 079 955 196
Envision, enable, enhance... empower
Tom Packert
modified routines inhouse and made them globally catalogued.
We are release 10B of Primac. The best thing you can do is put some indices
on key large files like SQV, PO, MISC.PO, JOB, JOB.TIME most of the
processes create WORK files then roll up to end of month files then roll up
to archive files. So the only real spead gain we have found is to index the
large files that relatively frequent queries are run on.
Tom
>>What applications are you runnning ?
>>
>>Are your processes globally catalogued ?
>>
>>Are you using System Builder ?
>
>We are running an application called PRIMAC from Vercom.
>http://www.vercom.com/vercom.htm
>
>Unfortunately, I know very little about UniData. So I've no idea if the
Barry Burtenshaw
>data now consolidated, or are you running 30 distinct databases but with a
>single faster box? If the former, then there may be all sorts of
>interesting file sizing factors coming into play as more and more data gets
>squeezed into your files. If simply the latter then how has your connection
>model changed? Were the individual print shops running terminals and PC's
>off serial lines direct into async ports on their smaller RS/6000s and now
>are they connecting in over a wide area network? If so, then response may
>seem a bit poor simply because direct asynchronous ports tend to buffer up
>characters much more snappily than telnet over TCP/IP does.
>
>How did you determine the current set of udtconfig parameters? Are they
>essentially the same as one of the original sites, but with the NUSERS
>parameter higher? Or did you just install udt and let it configure itself?
>By all means email me the udtconfig and the output from the UNIX level
>command "kp" (run it as root - it needs to read /dev/kmem) offline and I'll
>let you know if there's anything glaringly obvious wrong.
>
>Do you have UniData's monitoring tool licensed? Try running "udtmon" and
>seeing if you get a license problem. This will let you see what Unidata is
>spending its time on - reads/ writes/ overflowed files/ cataloged program
>calls - from there you can work out if anything needs attention.
>
>Hope that helps you progress it a little further, come back to me here or
>offline when you want to go further.
30 separate databases.
Vercom setup the current udtconfig with (I assume) Ardent's help. I ran
systest to a file, compared it to our current file and it showed a number of
differences.
All of the plants now access the central system over our WAN. So I'm sure some
of the lack of interactive response is due to network delay. Link speeds range
anywhere from 128Kb to 512Kb. We are using Digi PortServers to connect some
remote terminals, printers, shopfloor PCs and Janus scanners. Users are also
using PC with a variety of telnet programs. Most of the printers are network
connected (Jetdirect).
A 45 minute run of udtmon this afternoon showed i/o like this:
Current Average Minimum Maximum Total
File Open ł 28 103 0 1641 163244
File Close ł 12.67 50.35 0.00 824.67 80058
TempFile Close ł 0.00 0.00 0.00 0.00 0
Record Read ł 174 1337 0 15884 2126484
Record Write ł 14 128 0 2361 203543
Record Delete ł 2.00 7.56 0.00 115.33 12020
Level 1 Overflow ł 41 187 0 9146 297905
Level 2 Overflow ł 10 106 0 8698 167760
I've no idea how good or bad this is, but it looks like a lot of overflows to
me. I think our PRIMAC team currently resizes quarterly. Not sure if they use
guide or checkover to decide what to resize.
A vmstat 1200 during this period looked like this:
0 2 204334 334 0 0 0 5 7 0 704 10151 743 5 3 83 9
0 2 204872 602 0 0 0 12 18 0 748 23908 856 9 4 72 15
0 2 207932 843 0 0 10 102 246 0 789 9978 914 9 5 65 22
0 2 204809 1234 0 0 0 26 131 0 773 15013 900 12 4 68 15
0 2 206234 62530 0 0 0 2 2 0 787 8942 995 9 5 63 23
0 2 204665 39976 0 0 0 0 0 0 743 6321 933 10 4 71 16
0 2 204220 314 0 0 2 43 104 0 730 12437 826 11 4 69 16
0 2 203364 288 0 0 0 47 165 0 750 8398 1429 9 4 64 22
Thanks,
Barry
Ken Wallis
>30 separate databases.
I wonder if that means they've had to introduce some local cataloging or if
the application software was identical for all 30 sites. Local cataloging
would show itself as a problem by using CPU and memory. One of the other
pages of udtmon shows you how many local and how many global calls you are
making over time.
>Vercom setup the current udtconfig with (I assume) Ardent's help. I ran
>systest to a file, compared it to our current file and it showed a number
of
>differences. [emailed offline]
> root@haprimac:/usr/ud41/include # diff udtconfig udtconfig.blb1
> 8c8
> < LOCKFIFO=0
> ---
> > LOCKFIFO=1
Bit of a worry that LOCKFIFO differs I'd ask Vercom about that.
> < SPLIT_LOAD=80
> < MERGE_LOAD=20
> ---
> > SPLIT_LOAD=60
> > MERGE_LOAD=40
I presume Vercom must make use of dynamic files if they've changed this.
I'd also expect they must want to use KEYDATA type DFs instead of the
default KEYONLY if they've pushed SPLIT_LOAD up. I'm not sure if that will
acheive the desired goal. Certainly if you have KEYONLY dynamic files
anywhere, having a default SPLIT_LOAD of 80% is BAD. I'd never set
SPLIT_LOAD above 50% (try 45-48), but then I prefer KEYONLY.
I also noticed 2 lines in the udtconfig which weren't highlighted:
>TMP=/tmp/
and
>SHM_LPAGESZ=8
Check the setting for TMP with Vercom. /tmp is intended by most versions of
UNIX to contain small, files which don't live very long. UniData uses its
TMP space for all sorts of things including damn big select list files and
printer output. I usually like to set TMP to somewhere else.
As far as SHM_LPAGESZ is concerned you really don't want to divide such big
global pages [32MB (I'd make them bigger still on AIX)] into such small
local pages [4K]. It gives smm and udt so much work to do shuffling them
around and keeping track of them all. I'd push SHM_LPAGESZ up to 32 or so.
>All of the plants now access the central system over our WAN. So I'm sure
some
>of the lack of interactive response is due to network delay.
I sort of suspected that. Remember perception is reality.
>A 45 minute run of udtmon this afternoon showed i/o like this:
> Current Average Minimum Maximum Total
>
> File Open ł 28 103 0 1641 163244
> File Close ł 12.67 50.35 0.00 824.67 80058
> TempFile Close ł 0.00 0.00 0.00 0.00 0
> Record Read ł 174 1337 0 15884 2126484
> Record Write ł 14 128 0 2361 203543
> Record Delete ł 2.00 7.56 0.00 115.33 12020
> Level 1 Overflow ł 41 187 0 9146 297905
> Level 2 Overflow ł 10 106 0 8698 167760
>
>I've no idea how good or bad this is, but it looks like a lot of overflows
to
>me. I think our PRIMAC team currently resizes quarterly. Not sure if they
use
>guide or checkover to decide what to resize.
More than 10% of your I/Os are going into oveflow. The L1 overflow isn't a
real problem, but any time you see any L2 overflow you want to eliminate it.
checkover just tells you what is in L2 overflow so I'd run that and have a
look at what it says. If any of the VOC files are in L2 overflow fix them
quick (they are read and written all the time) if any other files are in L2
consider making them dynamic.
>A vmstat 1200 during this period looked like this:
> 0 2 204334 334 0 0 0 5 7 0 704 10151 743 5 3 83 9
> 0 2 204872 602 0 0 0 12 18 0 748 23908 856 9 4 72 15
> 0 2 207932 843 0 0 10 102 246 0 789 9978 914 9 5 65 22
> 0 2 204809 1234 0 0 0 26 131 0 773 15013 900 12 4 68 15
> 0 2 206234 62530 0 0 0 2 2 0 787 8942 995 9 5 63 23
> 0 2 204665 39976 0 0 0 0 0 0 743 6321 933 10 4 71 16
> 0 2 204220 314 0 0 2 43 104 0 730 12437 826 11 4 69 16
> 0 2 203364 288 0 0 0 47 165 0 750 8398 1429 9 4 64 22
I have little recollection of what the columns are on AIX's vmstat ;^)
HTH,
Anthony Patterson
The vmstat output shows that file sizing may be an issue.
The level 2 overflow needs to be minimised. Run 'guide' from the unix
directory of each account and look at the three GUIDE*LIS files that
are output. Anything in the ERR file is a corruption. In the ADVISE file
look for the suggested optimum sizes that differ greatly from the
current sizes, particularly where the Separation is smaller than the average
record size. This situation would result in at least two physical reads for
every logical read.
Compare response times for PC applications over the WAN to determine
whether it is isolated to the IBM. Use something like "Etherboy" to
check what is happening on the network.
When you resize files run 'fuser -u <filename> from AIX first to make sure
that the file is not in use. Always resize the VOC file from AIX to ensure
that you don't have it open.
(After a backup, of course!)
regards,
Anthony Patterson