I've spent much time looking at database stats and tweaking different
things at the database level but nothing has helped.
One thing I noticed between the two servers is that the test box has
all memory 16-way interleaved while the production box has a mix of
8-way and 16-way. I looked at memory configuration after I found memory
intensive queries ran faster on test.
How much impact does the interleave factor has on performance?
Here's the prtdiag output for both servers.
TEST SERVER:
/usr/platform/sun4u/sbin/prtdiag -v
System Configuration: Sun Microsystems sun4u Sun Fire 6800
System clock frequency: 150 MHz
Memory size: 32768 Megabytes
========================= CPUs
===============================================
CPU Run E$ CPU CPU
FRU Name ID MHz MB Impl. Mask
---------- ------- ---- ---- ------- ----
/N0/SB1/P0 4 1200 8.0 US-III+ 11.0
/N0/SB1/P1 5 1200 8.0 US-III+ 11.0
/N0/SB1/P2 6 1200 8.0 US-III+ 11.0
/N0/SB1/P3 7 1200 8.0 US-III+ 11.0
/N0/SB3/P0 12 1200 8.0 US-III+ 11.0
/N0/SB3/P1 13 1200 8.0 US-III+ 11.0
/N0/SB3/P2 14 1200 8.0 US-III+ 11.0
/N0/SB3/P3 15 1200 8.0 US-III+ 11.0
/N0/SB5/P0 20 1200 8.0 US-III+ 11.0
/N0/SB5/P1 21 1200 8.0 US-III+ 11.0
/N0/SB5/P2 22 1200 8.0 US-III+ 11.0
/N0/SB5/P3 23 1200 8.0 US-III+ 11.0
========================= Memory Configuration
===============================
Logical Logical Logical
Port Bank Bank Bank DIMM Interleave
Interleave
FRU Name ID Num Size Status Size Factor
Segment
------------- ---- ---- ------ ----------- ------ ----------
----------
/N0/SB1/P0/B0 4 0 512MB pass 256MB 16-way
0
/N0/SB1/P0/B1 4 1 512MB pass 256MB 16-way
0
/N0/SB1/P0/B0 4 2 512MB pass 256MB 16-way
0
/N0/SB1/P0/B1 4 3 512MB pass 256MB 16-way
0
/N0/SB1/P1/B0 5 0 512MB pass 256MB 16-way
0
/N0/SB1/P1/B1 5 1 512MB pass 256MB 16-way
0
/N0/SB1/P1/B0 5 2 512MB pass 256MB 16-way
0
/N0/SB1/P1/B1 5 3 512MB pass 256MB 16-way
0
/N0/SB1/P2/B0 6 0 512MB pass 256MB 16-way
0
/N0/SB1/P2/B1 6 1 512MB pass 256MB 16-way
0
/N0/SB1/P2/B0 6 2 512MB pass 256MB 16-way
0
/N0/SB1/P2/B1 6 3 512MB pass 256MB 16-way
0
/N0/SB1/P3/B0 7 0 512MB pass 256MB 16-way
0
/N0/SB1/P3/B1 7 1 512MB pass 256MB 16-way
0
/N0/SB1/P3/B0 7 2 512MB pass 256MB 16-way
0
/N0/SB1/P3/B1 7 3 512MB pass 256MB 16-way
0
/N0/SB3/P0/B0 12 0 512MB pass 256MB 16-way
1
/N0/SB3/P0/B1 12 1 512MB pass 256MB 16-way
1
/N0/SB3/P0/B0 12 2 512MB pass 256MB 16-way
1
/N0/SB3/P0/B1 12 3 512MB pass 256MB 16-way
1
/N0/SB3/P1/B0 13 0 512MB pass 256MB 16-way
1
/N0/SB3/P1/B1 13 1 512MB pass 256MB 16-way
1
/N0/SB3/P1/B0 13 2 512MB pass 256MB 16-way
1
/N0/SB3/P1/B1 13 3 512MB pass 256MB 16-way
1
/N0/SB3/P2/B0 14 0 512MB pass 256MB 16-way
1
/N0/SB3/P2/B1 14 1 512MB pass 256MB 16-way
1
/N0/SB3/P2/B0 14 2 512MB pass 256MB 16-way
1
/N0/SB3/P2/B1 14 3 512MB pass 256MB 16-way
1
/N0/SB3/P3/B0 15 0 512MB pass 256MB 16-way
1
/N0/SB3/P3/B1 15 1 512MB pass 256MB 16-way
1
/N0/SB3/P3/B0 15 2 512MB pass 256MB 16-way
1
/N0/SB3/P3/B1 15 3 512MB pass 256MB 16-way
1
/N0/SB5/P0/B0 20 0 1024MB pass 512MB 16-way
2
/N0/SB5/P0/B1 20 1 1024MB pass 512MB 16-way
2
/N0/SB5/P0/B0 20 2 1024MB pass 512MB 16-way
2
/N0/SB5/P0/B1 20 3 1024MB pass 512MB 16-way
2
/N0/SB5/P1/B0 21 0 1024MB pass 512MB 16-way
2
/N0/SB5/P1/B1 21 1 1024MB pass 512MB 16-way
2
/N0/SB5/P1/B0 21 2 1024MB pass 512MB 16-way
2
/N0/SB5/P1/B1 21 3 1024MB pass 512MB 16-way
2
/N0/SB5/P2/B0 22 0 1024MB pass 512MB 16-way
2
/N0/SB5/P2/B1 22 1 1024MB pass 512MB 16-way
2
/N0/SB5/P2/B0 22 2 1024MB pass 512MB 16-way
2
/N0/SB5/P2/B1 22 3 1024MB pass 512MB 16-way
2
/N0/SB5/P3/B0 23 0 1024MB pass 512MB 16-way
2
/N0/SB5/P3/B1 23 1 1024MB pass 512MB 16-way
2
/N0/SB5/P3/B0 23 2 1024MB pass 512MB 16-way
2
/N0/SB5/P3/B1 23 3 1024MB pass 512MB 16-way
2
PRODUCTION SERVER:
/usr/platform/sun4u/sbin/prtdiag -v | more
System Configuration: Sun Microsystems sun4u Sun Fire 15K
System clock frequency: 150 MHz
Memory size: 57344 Megabytes
========================= CPUs =========================
CPU Run E$ CPU CPU
Slot ID ID MHz MB Impl. Mask
-------- ------- ---- ---- ------- ----
/SB00/P0 0 1200 8.0 US-III+ 11.0
/SB00/P1 1 1200 8.0 US-III+ 11.0
/SB00/P2 2 1200 8.0 US-III+ 11.0
/SB00/P3 3 1200 8.0 US-III+ 11.0
/SB01/P0 32 1200 8.0 US-III+ 11.0
/SB01/P1 33 1200 8.0 US-III+ 11.0
/SB01/P2 34 1200 8.0 US-III+ 11.0
/SB01/P3 35 1200 8.0 US-III+ 11.0
/SB02/P0 64 1200 8.0 US-III+ 11.0
/SB02/P1 65 1200 8.0 US-III+ 11.0
/SB02/P2 66 1200 8.0 US-III+ 11.0
/SB02/P3 67 1200 8.0 US-III+ 11.0
/SB14/P0 448 1200 8.0 US-III+ 11.0
/SB14/P1 449 1200 8.0 US-III+ 11.0
/SB14/P2 450 1200 8.0 US-III+ 11.0
/SB14/P3 451 1200 8.0 US-III+ 11.0
========================= Memory Configuration
=========================
Logical Logical Logical
Port Bank Bank Bank DIMM Interleave
Interleave
Slot ID ID Number Size Status Size Factor
Segment
----------- ---- ------- ------- -------- ------ ----------
----------
/SB00/P0/B0 0 0 1024MB okay 512MB 8-way
0
/SB00/P0/B0 0 2 1024MB okay 512MB 8-way
0
/SB00/P1/B0 1 0 1024MB okay 512MB 8-way
0
/SB00/P1/B0 1 2 1024MB okay 512MB 8-way
0
/SB00/P2/B0 2 0 1024MB okay 512MB 8-way
0
/SB00/P2/B0 2 2 1024MB okay 512MB 8-way
0
/SB00/P3/B0 3 0 1024MB okay 512MB 8-way
0
/SB00/P3/B0 3 2 1024MB okay 512MB 8-way
0
/SB01/P0/B0 32 0 1024MB okay 512MB 8-way
1
/SB01/P0/B0 32 2 1024MB okay 512MB 8-way
1
/SB01/P1/B0 33 0 1024MB okay 512MB 8-way
1
/SB01/P1/B0 33 2 1024MB okay 512MB 8-way
1
/SB01/P2/B0 34 0 1024MB okay 512MB 8-way
1
/SB01/P2/B0 34 2 1024MB okay 512MB 8-way
1
/SB01/P3/B0 35 0 1024MB okay 512MB 8-way
1
/SB01/P3/B0 35 2 1024MB okay 512MB 8-way
1
/SB02/P0/B0 64 0 2048MB okay 1024MB 16-way
2
/SB02/P0/B1 64 1 2048MB okay 1024MB 16-way
2
/SB02/P0/B0 64 2 2048MB okay 1024MB 16-way
2
/SB02/P0/B1 64 3 2048MB okay 1024MB 16-way
2
/SB02/P1/B0 65 0 2048MB okay 1024MB 16-way
2
/SB02/P1/B1 65 1 2048MB okay 1024MB 16-way
2
/SB02/P1/B0 65 2 2048MB okay 1024MB 16-way
2
/SB02/P1/B1 65 3 2048MB okay 1024MB 16-way
2
/SB02/P2/B0 66 0 2048MB okay 1024MB 16-way
2
/SB02/P2/B1 66 1 2048MB okay 1024MB 16-way
2
/SB02/P2/B0 66 2 2048MB okay 1024MB 16-way
2
/SB02/P2/B1 66 3 2048MB okay 1024MB 16-way
2
/SB02/P3/B0 67 0 2048MB okay 1024MB 16-way
2
/SB02/P3/B1 67 1 2048MB okay 1024MB 16-way
2
/SB02/P3/B0 67 2 2048MB okay 1024MB 16-way
2
/SB02/P3/B1 67 3 2048MB okay 1024MB 16-way
2
/SB14/P0/B0 448 0 1024MB okay 512MB 8-way
3
/SB14/P0/B0 448 2 1024MB okay 512MB 8-way
3
/SB14/P1/B0 449 0 1024MB okay 512MB 8-way
3
/SB14/P1/B0 449 2 1024MB okay 512MB 8-way
3
/SB14/P2/B0 450 0 1024MB okay 512MB 8-way
3
/SB14/P2/B0 450 2 1024MB okay 512MB 8-way
3
/SB14/P3/B0 451 0 1024MB okay 512MB 8-way
3
/SB14/P3/B0 451 2 1024MB okay 512MB 8-way
3
<eavi...@yahoo.com> wrote in message
news:1138918380.2...@g47g2000cwa.googlegroups.com...
The same batch process runs about 50% faster on the test server.
Cary Millsap's book "Optimizing Oracle Performance" contains a proven
methodology for solving problems like yours. I recommend highly taking
the time to read this book as the primary way of determining how to
attack your problem.
You can get 10046 traces for a representative time of the batch run on
the production server and see what "is slowing it down". Compare and
contrast this with what you see from the batch run on the test server.
Still, my original question remains unanswered. Given the interleave
factor difference, how much impact would that have on memory intensive
database queries? Is it negligible or is it a big deal?
Thanks.
-Seongbae, usually as seongb...@sun.com
It can be a big deal if your queries are CPU limited.
It can be nothing if your bottleneck is instead IO.
Hence the request to go through Cary's approach with
10046: it will tell you exactly what's going on and where
to look for problems.
Interleave can be a big factor for cpu intensive work,
be it database or anything else. Try to get a memory
speed test program and do some comparative tests.
One thing I noticed is that one is a 6800 and the other is a 15K. These
are _very_ different beasts that work in very different ways.
(6800 is snoopy coherency, F15K is ccNUMA)
There's far too many variables in the platform difference alone to start
with memory...
Scott.
Others have suggested the best approach for tuning Oracle. Since I have
never used Oracle, I can't comment on that.
This may be (probably is) too simplistic, but there is a program for
testing memory bandwidth:
http://icl.cs.utk.edu/projects/llcbench/
This is the part of output I get from my Ultra 80. As you can see,
figures in Mb/s drop quite dramatically at 4 MB, which I assume is
because the cache size of the X1195A 450 MHz CPUs (there are 4) is 4 MB.
sparrow /export/home/drkirkby/llcbench/cachebench % ./cachebench
Memory Set Library Cache Test
C Size Nanosec MB/sec % Chnge
------- ------- ------- -------
256 0.75 1279.72 1.00
256 0.72 1316.86 0.97
384 0.61 1567.58 0.84
384 0.60 1585.15 0.99
512 0.54 1764.63 0.90
512 0.54 1752.30 1.01
768 0.48 2001.51 0.88
768 0.48 1990.85 1.01
1024 0.45 2136.23 0.93
1024 0.45 2121.47 1.01
1536 0.41 2304.18 0.92
1536 0.41 2307.56 1.00
2048 0.40 2391.77 0.96
2048 0.40 2393.02 1.00
3072 0.38 2482.46 0.96
3072 0.38 2493.88 1.00
4096 2.53 377.16 6.61
4096 2.58 370.25 1.02
6144 2.57 371.07 1.00
Of course you have a very different ssort of machine. Here's data from a
similar is machine to the above. This time a Sun Ultra 60 with a single
360 MHz CPU. Again the cache size is 4 MB on this.
main-webserver # ./cachebench
Memory Set Library Cache Test
C Size Nanosec MB/sec % Chnge
------- ------- ------- -------
256 0.91 1052.24 1.00
256 0.91 1045.29 1.01
384 0.76 1248.05 0.84
384 0.76 1261.58 0.99
512 0.68 1404.07 0.90
512 0.68 1403.34 1.00
768 0.60 1590.35 0.88
768 0.61 1564.57 1.02
1024 0.56 1693.51 0.92
1024 0.56 1701.08 1.00
1536 0.52 1827.60 0.93
1536 0.53 1813.95 1.01
2048 0.50 1901.40 0.95
2048 0.50 1902.84 1.00
3072 0.48 1972.52 0.96
3072 0.48 1977.79 1.00
4096 1.43 666.12 2.97
4096 2.43 392.98 1.70
6144 2.41 396.41 0.99
6144 2.48 384.96 1.03
Anyway, it might confuse matters more, but the option is there of
running that if you want. It was pretty easy to build.
--
Dave K
http://www.southminster-branch-line.org.uk/
Please note my email address changes periodically to avoid spam.
It is always of the form: month-year@domain. Hitting reply will work
for a couple of months only. Later set it manually. The month is
always written in 3 letters (e.g. Jan, not January etc)