Reduced TMF latency
nico...@yahoo.com
I could not attend last NonStop conference call. I am interested in
some data regarding the new TMF version.
We have some problems with TMF latency (elapsed time between a begin
and commit transaction).
In the current TMF version, the minimal latency stays over the
centisecond. This fact, prevents TMF usage for very low latency
transactions.
The old commit timer Snoop parameter (now TMP timer?), that controls
TMP "polling pace" is specified in centiseconds.
I wonder if someone knows if the new TMF version allows under the
centisecond latencies.
Thanks in advance,
Niki.
wbreidbach
Hi,
In former times you could completely switch off the latency using
SNOOP. We did this, because we had a mix of online transactions and
RDF-initiated transactions which caused some problems with the online
performance.
The new parameter for TMFCOM is TmpWaitTimer. With INFO TMF you get
the actual value. On our system the value is TmpWaitTimer Off and that
switches all the latency.
regards
Wolfgang
nico...@yahoo.com
>
> - Show quoted text -
Thanks Wolfrang,
We have also tried that (commit timer set to -1 in older TMF
versions). Doing that, latencies still remain over the centisecond,
even with transactions involving one single IO. I do not know if the
TMF audit trail write is responsible for that, and if the new TMF
version, and the new disk drives can get the latency down. I would
appreciate some production statistical information if possible.
People tend to compare average latencies provided by other operating
systems, and this limitation keeps us a little behind.
Regards.
Niki
MicroTech
>
> In the current TMF version, the minimal latency stays over the
> centisecond. This fact, prevents TMF usage for very low latency
> transactions.
>
suboptimal "low level" TS/MP parameter configuration. You can grab a
zip archive with two documents, that may be of interest, from my
Internet file folder, by stuffing this URL into your browser's address
slot:
http://www.box.net/shared/ol9y4jjjbx
The archive contains two documents: (1) A TS/MP capacity report
(prepared by myself some time ago), and (2) SNOOP (the "inofficial" TS/
MP poking and peeking utility) documentation. Both in Microsoft Word
format.
Where TS/MP "minimal latency" is concerned, the SNOOP commands of
particular interest are PIOBUFFER and SETCOMMITTIMER.
Caveat Emptor: (1) I have tested nothing discussed above on Itanium,
and (2) The SNOOP document is (was?) not really supported, and the
version I offer here dates back to 2004: Things may have changed. I
strongly recommend that you discuss any SNOOP experimentation with
your friendly HP Support Analyst!
Good luck, and please let us know how things pan out for you! This is
interesting stuff!
Cheers,
Henry Norman
MicroTech Consulting
www.microtechnonstop.com
http://groups.google.com/group/MicroTech_Software
MicroTech
> http://www.box.net/shared/ol9y4jjjbx
Unfortunately (and I do apologize if you tried to download from there
and failed) that URL is not the correct one.
However, this one is:
http://www.box.net/shared/bzp4gmuh7j
Again, sorry about this mishap -- mea culpa! The rest of my previous
post does apply.
nico...@yahoo.com
Thanks Henry,
Very interesting information.
I have one question about the statistics provided. What does it mean
"Milliseconds
per TM/MP Transid" in your document tables? It seems to me that it
means, millisenconds per record queued. When you queue 50 records
under the same TMF transaction, of course you get the best throughput
figures, and best average latencies per record, but not per TMF
transaction.
So I cannot see the figures of average latencies of each TMF
transaction (except in the case of Box Car = 1, where each records
and transactions mean the same thing).
So if I am right, latency of each individual TMF transaction stays far
over the centisecond.
The snoop commit timer parameter is the one we set to -1 in some of
our tests, but we where never able to get below the centisecond.
We do group IO´s under the same transaction when traffic allows us to
do so. The problem arrises when traffic is very low, and we get one
message at a time. Our client still wants to have average under the
centisecond latencies for each individual record.
Regards,
Niki
wbreidbach
> On May 22, 12:28 pm, MicroTech <Henry.KO.Nor...@gmail.com> wrote:
>
>
>
>
>
> > On May 22, 5:15 pm, MicroTech <Henry.KO.Nor...@gmail.com> wrote:
>
> > > http://www.box.net/shared/ol9y4jjjbx
>
> > Unfortunately (and I do apologize if you tried to download from there
> > and failed) that URL is not the correct one.
> > However, this one is:
>
> > http://www.box.net/shared/bzp4gmuh7j
>
> > Again, sorry about this mishap -- mea culpa! The rest of my previous
> > post does apply.
>
> > Cheers,
>
> > Henry Norman
>
> Thanks Henry,
>
> Very interesting information.
>
> I have one question about the statistics provided. What does it mean
> "Milliseconds
> per TM/MP Transid" in your document tables? It seems to me that it
> means, millisenconds per record queued. When you queue 50 records
> under the same TMF transaction, of course you get the best throughput
> figures, and best average latencies per record, but not per TMF
> transaction.
>
> So I cannot see the figures of average latencies of each TMF
> transaction (except in the case of Box Car = 1, where each records
> and transactions mean the same thing).
>
> So if I am right, latency of each individual TMF transaction stays far
> over the centisecond.
>
> The snoop commit timer parameter is the one we set to -1 in some of
> our tests, but we where never able to get below the centisecond.
>
> We do group IO´s under the same transaction when traffic allows us to
> do so. The problem arrises when traffic is very low, and we get one
> message at a time. Our client still wants to have average under the
> centisecond latencies for each individual record.
>
> Regards,
>
>
> - Zitierten Text anzeigen -
Are you sure the problem is related to TMF? Does the centisecond refer
to the commit or to the total transaction duration? We set the
committimer to -1 in the previously described environment and the
throughput for the online transactions could be dramatically
increased. And that happened on a S72000. I think if wood be a good
idea to do a measurement on the complete transaction. Perhaps that
will give more information.
wbreidbach
>
>
>
>
>
> > On May 22, 5:15 pm, MicroTech <Henry.KO.Nor...@gmail.com> wrote:
>
> > > http://www.box.net/shared/ol9y4jjjbx
>
> > Unfortunately (and I do apologize if you tried to download from there
> > and failed) that URL is not the correct one.
> > However, this one is:
>
> > http://www.box.net/shared/bzp4gmuh7j
>
> > Again, sorry about this mishap -- mea culpa! The rest of my previous
> > post does apply.
>
> > Cheers,
>
> > Henry Norman
>
> Thanks Henry,
>
> Very interesting information.
>
> I have one question about the statistics provided. What does it mean
> "Milliseconds
> per TM/MP Transid" in your document tables? It seems to me that it
> means, millisenconds per record queued. When you queue 50 records
> under the same TMF transaction, of course you get the best throughput
> figures, and best average latencies per record, but not per TMF
> transaction.
>
> So I cannot see the figures of average latencies of each TMF
> transaction (except in the case of Box Car = 1, where each records
> and transactions mean the same thing).
>
> So if I am right, latency of each individual TMF transaction stays far
> over the centisecond.
>
> The snoop commit timer parameter is the one we set to -1 in some of
> our tests, but we where never able to get below the centisecond.
>
> We do group IO´s under the same transaction when traffic allows us to
> do so. The problem arrises when traffic is very low, and we get one
> message at a time. Our client still wants to have average under the
> centisecond latencies for each individual record.
>
> Regards,
>
MicroTech
You are correct, in the particular case of our queue file capacity
testing, "per TM/MP transid" refers to the number of queue file
records written per second. In that particular application (decoupling
two systems via a DP2 queue file), to the Producer of queue records,
one write to the queue file was considered one transaction. The
objective at the time was to tune the system for an optimum number of
"Producer transactions" per second.
Thanks for pointing this out--it is indeed misleading (nobody spotted
this back then!)!
To never see latencies below centisecond(s) would (to me, anyway)
indicate that something, somewhere, is amiss... It would be very
interesting to receive whatever follow-up information you could
provide!
Which software releases are you dealing with here? G or H series?
Cheers,
Henry Norman
nico...@yahoo.com
Hi all,
I have tested in G and H. In H (Integrity) I had no rights to reset
the TMP timer, so I cannot tell which the results would be.
By TMF lacency, I mean elapsed time between the begin transaction and
the completion of the end transaction. So the whole transaction is
taken into account. Anyway, our tests where done with transactions
involving one single write to a queue type enscribe file. Under such
conditions, average latency remained over the centisecond.
Reproducing the test is trivial, provided you may alter the commit
timer (or TMP timer) at your will.
Any volunteer for integrity systems?
Regards,
Niki
JShepherd
nico...@yahoo.com says...
NS16000
TmpWaitTimer Off
Transactions 20000
Elapsed time 00:00:07.611945
Microsec Per OP 380
OPs per MilliSec 2
TmpWaitTimer 10000 Microseconds
Transactions 20000
Elapsed time 00:03:45.274802
Microsec Per OP 11263
OPs per MilliSec 0
--------------------------------
S7600
TmpWaitTimer Off
Transactions 20000
Elapsed time 00:00:14.345102
Microsec Per OP 717
OPs per MilliSec 1
--------------------------------
#include <stdio.h> nolist
#include <stdlib.h> nolist
#include <string.h> nolist
#include <cextdecs> nolist
long i;
long long startts;
long long endts;
long long interval;
short systime[8];
long maxops = 20000;
int main(void)
{
startts = JULIANTIMESTAMP();
for (i = 0; i < maxops; i++)
{
BEGINTRANSACTION();
ENDTRANSACTION();
}
endts = JULIANTIMESTAMP();
interval = endts - startts;
INTERPRETINTERVAL(interval ,
&systime[3],&systime[4],
&systime[5],&systime[6], &systime[7] );
printf("Transactions %ld\n", maxops);
printf("Elapsed time %02hd:%02hd:%02hd.%03hd%03hd\n",
systime[3], systime[4],
systime[5], systime[6], systime[7] );
printf("Microsec Per OP %Ld\n",
(interval / (long long)maxops));
interval = interval / 1000;
if (interval < 1)
interval = 1;
printf("OPs per MilliSec %Ld\n",
((long long)maxops / interval) );
return(0);
}
nico...@yahoo.com
This is the kind of thing I was looking for. The only change I would
do to make the exercise more realistic, is doing one IO within the
transaction. It is very possible that the begin-end with no IO in
between prevents writes to the audit trail, that may be the main
responsibles of TMF latencies.
I suggest issuing a write to an audited enscribe queue type file, and
then running the test again.
Regretfully I do no have acces to the new TMF version.
The file may be created with this fup obey file:
SET TYPE K
SET EXT ( 400 PAGES, 200 PAGES )
SET REC 80
SET BLOCK 4096
SET MAXEXTENTS 16
SET AUDIT
SET QUEUEFILE
SET KEYLEN 8
CREATE QUEFILE
Code changed to do the Io is: (no error handling)
#include <stdio.h> nolist
#include <stdlib.h> nolist
#include <string.h> nolist
#include <cextdecs> nolist
long i;
long long startts;
long long endts;
long long interval;
short systime[8];
char buffer [256];
char filename [40];
short filenum;
long maxops = 1000; /* should be enough */
int main(void)
{
strcpy (filename, "QUEFILE"); /* file has to be created in default
subvolume */
FILE_OPEN_ (filename, (short) strlen (filename), &filenum);
startts = JULIANTIMESTAMP();
for (i = 0; i < maxops; i++)
{
BEGINTRANSACTION();
sprintf (buffer, " MESSAGE NUMBER [%d]", i);
WRITEX (filenum, buffer, (short)strlen(buffer));
ENDTRANSACTION();
}
endts = JULIANTIMESTAMP();
FILE_CLOSE_ (filenum);
interval = endts - startts;
INTERPRETINTERVAL(interval ,
&systime[3],&systime[4],
&systime[5],&systime[6], &systime[7] );
printf("Transactions %ld\n", maxops);
printf("Elapsed time %02hd:%02hd:%02hd.%03hd%03hd\n",
systime[3], systime[4],
systime[5], systime[6], systime[7] );
printf("Microsec Per OP %Ld\n",
(interval / (long long)maxops));
interval = interval / 1000;
if (interval < 1)
interval = 1;
printf("OPs per MilliSec %Ld\n",
((long long)maxops / interval) );
return(0);
}
Can someone run it and return the results?
Thanks in advance,
Niki
JShepherd
and added a PURGEDATA so it's easily runnable multiple times.
The BEGINTRANSACTION() and ENDTRANSACTION() calls were commented
out for the last NON-TMF test.
--------------------
1 process 1 file.
NS16000
TmpWaitTimer Off
Transactions 20000
Elapsed time 00:02:14.013763
Microsec Per OP 6700
OPs per MilliSec 0
TmpWaitTimer 10000 Microseconds
Transactions 20000
Elapsed time 00:03:52.334347
Microsec Per OP 11616
OPs per MilliSec 0
-------------------
4 processes in cpus (0 thru 3) to 4 files (on the same vol)
TmpWaitTimer Off
Transactions 20000
Elapsed time 00:04:28.732222
Microsec Per OP 13436
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:04:29.356056
Microsec Per OP 13467
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:04:29.166579
Microsec Per OP 13458
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:04:31.996357
Microsec Per OP 13599
OPs per MilliSec 0
TmpWaitTimer 10000 Microseconds
Transactions 20000
Elapsed time 00:03:53.647250
Microsec Per OP 11682
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:03:53.246521
Microsec Per OP 11662
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:03:53.335311
Microsec Per OP 11666
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:03:53.358547
Microsec Per OP 11667
OPs per MilliSec 0
-------------------
4 processes in cpus (0 thru 3) to 4 files (on $data01 thru $data04)
TmpWaitTimer Off
Transactions 20000
Elapsed time 00:04:30.623685
Microsec Per OP 13531
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:04:29.639083
Microsec Per OP 13481
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:04:30.843622
Microsec Per OP 13542
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:04:30.037063
Microsec Per OP 13501
OPs per MilliSec 0
TmpWaitTimer 10000 Microseconds
Transactions 20000
Elapsed time 00:03:54.261808
Microsec Per OP 11713
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:03:54.378138
Microsec Per OP 11718
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:03:53.848812
Microsec Per OP 11692
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:03:53.868663
Microsec Per OP 11693
OPs per MilliSec 0
NON-TMF BUFFERED QUEUE FILEs (app still does begin/end trans )
4 processes in cpus (0 thru 3) to 4 files (on the same vol)
TmpWaitTimer Off
Transactions 20000
Elapsed time 00:00:15.908986
Microsec Per OP 795
OPs per MilliSec 1
Transactions 20000
Elapsed time 00:00:15.902899
Microsec Per OP 795
OPs per MilliSec 1
Transactions 20000
Elapsed time 00:00:15.798803
Microsec Per OP 789
OPs per MilliSec 1
Transactions 20000
Elapsed time 00:00:15.930984
Microsec Per OP 796
OPs per MilliSec 1
TmpWaitTimer 10000 Microseconds
Transactions 20000
Elapsed time 00:03:48.754510
Microsec Per OP 11437
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:03:48.788168
Microsec Per OP 11439
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:03:48.813614
Microsec Per OP 11440
OPs per MilliSec 0
Transactions 20000
Elapsed time 00:03:48.709908
Microsec Per OP 11435
OPs per MilliSec 0
NON-TMF BUFFERED QUEUE FILEs (NO begin/end trans )
4 processes in cpus (0 thru 3) to 4 files (on the same vol)
Transactions 20000
Elapsed time 00:00:09.227383
Microsec Per OP 461
OPs per MilliSec 2
Transactions 20000
Elapsed time 00:00:08.158469
Microsec Per OP 407
OPs per MilliSec 2
Transactions 20000
Elapsed time 00:00:09.672177
Microsec Per OP 483
OPs per MilliSec 2
Transactions 20000
Elapsed time 00:00:08.916796
Microsec Per OP 445
OPs per MilliSec 2
========================================
Scripts used
== Same vol
delete define =quefile
add define =quefile, class map, file $data01.zzzjunk.que00
run tmftest3 /cpu 0, pri 150, name, nowait /
delete define =quefile
add define =quefile, class map, file $data01.zzzjunk.que01
run tmftest3 /cpu 1, pri 150, name, nowait /
delete define =quefile
add define =quefile, class map, file $data01.zzzjunk.que02
run tmftest3 /cpu 2, pri 150, name, nowait /
delete define =quefile
add define =quefile, class map, file $data01.zzzjunk.que03
run tmftest3 /cpu 3, pri 150, name, nowait /
pause
== 4 VOLS
delete define =quefile
add define =quefile, class map, file $data01.zzzjunk.que00
run tmftest3 /cpu 0, pri 150, name, nowait /
delete define =quefile
add define =quefile, class map, file $data02.zzzjunk.que01
run tmftest3 /cpu 1, pri 150, name, nowait /
delete define =quefile
add define =quefile, class map, file $data03.zzzjunk.que02
run tmftest3 /cpu 2, pri 150, name, nowait /
delete define =quefile
add define =quefile, class map, file $data04.zzzjunk.que03
run tmftest3 /cpu 3, pri 150, name, nowait /
pause
== NON_TMF (BUFFERED)
delete define =quefile
add define =quefile, class map, file $data01.zzzjunk.nque00
run tmftest3 /cpu 0, pri 150, name, nowait /
delete define =quefile
add define =quefile, class map, file $data01.zzzjunk.nque01
run tmftest3 /cpu 1, pri 150, name, nowait /
delete define =quefile
add define =quefile, class map, file $data01.zzzjunk.nque02
run tmftest3 /cpu 2, pri 150, name, nowait /
delete define =quefile
add define =quefile, class map, file $data01.zzzjunk.nque03
run tmftest3 /cpu 3, pri 150, name, nowait /
pause
===================
/*
SET TYPE K
SET EXT ( 400 PAGES, 200 PAGES )
SET REC 80
SET BLOCK 4096
SET MAXEXTENTS 16
SET AUDIT
SET QUEUEFILE
SET KEYLEN 8
CREATE QUEFILE
*/
#include <stdio.h> nolist
#include <stdlib.h> nolist
#include <string.h> nolist
#include <cextdecs> nolist
short erc;
long i;
long long startts;
long long endts;
long long interval;
short systime[8];
char buffer [256];
char filename [40];
short filenum;
long maxops = 20000;
int main(void)
{
strcpy (filename, "=QUEFILE");
erc = FILE_OPEN_ (filename, (short)strlen (filename), &filenum);
if (erc)
{
PROCESS_DEBUG_();
}
CONTROL(filenum, 20);
FILE_GETINFO_(filenum, &erc);
if (erc)
{
PROCESS_DEBUG_();
}
startts = JULIANTIMESTAMP();
for (i = 0; i < maxops; i++)
{
BEGINTRANSACTION();
sprintf (buffer, " MESSAGE NUMBER [%d]", i);
WRITEX (filenum, buffer, (short)strlen(buffer));
FILE_GETINFO_(filenum, &erc);
if (erc)
{
PROCESS_DEBUG_();
nico...@yahoo.com
>
> - Show quoted text -
Thanks again.
Very complete test indeed.
A quick summary of the results could be:
1. TMF "realistic" minimal latency stays over (very close to, 1,16)
the centisecond.
2. Scalability is really linear. Latency remains the same with four
processes working in parallel (although TMF has a centralized
component)
3. Removing TMF and working with buffered queue files, divides latency
by a factor of more than 3.
4. There is not a big difference between swithcing off the TMP timer
or setting it to one centisecond (still lowest configurable value in
the new TMF version?)
5. In fact, with higher load, setting it to 1 centisecond gives better
results than swithing it off (perhaps, due to the fact that TMP
optimizes bulk IO´s to the audit trails).
If write to the audit trail is an oportant factor (just an hipotesis),
using more high speed disks could improve this results.
Please, let me know any comments about these conclusions.
Thanks again for this wonderful work.
Niki
wbreidbach
>
> - Zitierten Text anzeigen -
Hi,
please do not forget one very important thing:
If you have a mix of short and long transaction, as we had with RDF
and OLTP, and you use the default configuration, TMF will calculate a
wait-timer from both types and this timer can be rather long! In this
case switching off the timer will be of great help. And that timer is
not related to system load. As we used the default timing, without RDF
we were processing 100 OLTP-transactions/sec with medium system load.
With RDF started we could only process about 40 OLTP-transactions/sec
with decreased system load. After switching of the latency we could
again process 100 OLTP/sec with RDF started.So the procedure how TMF
calculates the latency, may be of great impact.
nico...@yahoo.com
>
> read more »- Hide quoted text -
>
> - Show quoted text -
Hi again,
Just a typo,
In the third summary line, it should say 30 intead of 3.
3. Removing TMF and working with buffered queue files, divides
latency
by a factor of more than 30.
So there seems to be some room from improvementes (high speed
disks?).
I agree with the last comment. I have seen the timer set by the system
to 100 miliseconds under some circunstances.
Regards,
Niki.
nico...@yahoo.com
>
> read more »- Hide quoted text -
>
> - Show quoted text -
Typo:
Hi again,
Just to correct a typo. 3 in the third summarypoint should be 30.
3. Removing TMF and working with buffered queue files, divides latency
by a factor of more than 30.
So there seem to be some room for improvements (higher speed disks?)
Do you think that a centisecond is the proper lowest TMP timer figure
in Integrity systems? Should TMP allow setting a shorter timer?
Does H series support timers under the centisecond?
Wolgang, I know about that probblem. In fact, in some of our client
environments, TMF decided to set the timer to 100 miliseconds, so
every transaction on the system took at least 100 miliseconds to
complete.
The interesting thing, is that setiting the parameter to 2 o 3
centiseconds, drastically improved the latency, and did not seem to
affect the heavier transactions.
Regards,
Niki
nico...@yahoo.com
taking out the quoted text.
1. TMF "realistic" minimal latency stays over (very close to, 1,16)
the centisecond.
2. Scalability is really linear. Latency remains the same with four
processes working in parallel (although TMF has a centralized
component)
3. Removing TMF and working with buffered queue files, divides
latency
by a factor of more than 30.
4. There is not a big difference between swithcing off the TMP timer
or setting it to one centisecond (still lowest configurable value in
the new TMF version?)
5. In fact, with higher load, setting it to 1 centisecond gives
better
results than swithing it off (perhaps, due to the fact that TMP
So there seem to be some room for improvements (higher speed disks?)