SMF timestamps
McKown, John
Do I just "give up"? Or am I missing something simple?
John McKown
Systems Engineer IV
IT
Administrative Services Group
HealthMarkets(r)
9151 Boulevard 26 * N. Richland Hills * TX 76010
(817) 255-3225 phone *
john....@healthmarkets.com * www.HealthMarkets.com
Confidentiality Notice: This e-mail message may contain confidential or proprietary information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. HealthMarkets(r) is the brand name for products underwritten and issued by the insurance subsidiaries of HealthMarkets, Inc. -The Chesapeake Life Insurance Company(r), Mid-West National Life Insurance Company of TennesseeSM and The MEGA Life and Health Insurance Company.SM
----------------------------------------------------------------------
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to list...@bama.ua.edu with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html
Barry Merrill
The 8-byte "SMFSTAMP" format with time to 2 decimals and Julian date
is used not only in the standard timestamp in the SMF header bytes 3-10,
which is ALWAYS on the LOCAL time zone, but SMFSTAMP8 is used for many
other datetime values, especially in the job-related records, like the
READTIME. However, some of the job event records contain only the
four-byte time so their datetime values must be inferred from the
Date part of READTIME or SMFTIME (notable, INITTIME and ALOCTIME in
the 30s are only provided as LOCAL times without a date).
There are 634 datetime fields that are input with SMFSTAMP8 in MXG Software).
I believe all of these SMFSTAMP instances in SMF records are on the LOCAL
time zone, although any vendor can write any SMF record with any value in that
format.
However, there are many other instances of datetime values that are
written in 8-byte "TODSTAMP" format, (1145 instances in MXG) and I
believe they are all on the GMT/UTC Time Zone, although there are
instances of two TODSTAMP values in some records, with one on LOCAL
and the other on GMT.
And there are some really strange-looking character date and time
and datetime values scattered throughout SMF data, especially for
newer records that may come from opensystems sources.
In many SMF records, the GMT Offset value (CVTTZ) is provided, but
by no means is it always present; in some records, there may be two
datetimestamps for the same event that can be compared to determine
the GMT offset, but that can require fuzzy logic, for example in
the SMF 30s, since SMF30IST is in SMFSTAMP8 format, rounded, and
with 10 millisecond resolution, while it's GMT event counterpart,
SMF30ISS is in TODSTAMP format with microsecond resolution.
And, SMF30ISS only exists in the Subtype 2 and 3 Interval Records.
And, SMF30IST does NOT exist in the "MULTI-DD" records, which contain
only the ID and EXCP segments (for steps/intervals with more DDs
that will fit in a single 32K SMF record).
And, some SMF vendor-created records have only GMT fields, and
in the long past, there were user-written records with the
header SMFTIME in GMT, but I've not seen that in the current
millennium.
So, it "ain't simple", and each record can only be examined for
its specific contents.
Merrilly yours,
Barry Merrill
Herbert W. Barry Merrill, PhD
President-Programmer
Merrill Consultants
MXG Software
10717 Cromwell Drive
Dallas TX 75229
214 351 1966 tel
214 350 3695 fax
www.mxg.com
P.S. Long ago, I had discussions with IBM SMF developer Bill Richardson
for a proposed extension to every SMF record precisely to contain
the GMT OFFSET, since expanding the SMF header would have been
a VERY incompatible change to the many programs that read SMF with
expected fixed field locations, but because many records do not
contain a version number, causing the length of the record to be
required to determine version (e.g., JES 26 records), that extension
was never implemented.
P.P.S But there is currently a SHARE Requirement in development that is
looking at a suite of fields that are needed for all JOB-related
records, for cost-recovery, including JCTJOBID, ACCOUNTn, and
SYSPLEX, so the addition of the GMT Offset might be a future
consideration, if IBM reviews that requirement in a favorable light.
Elardus Engelbrecht
>The SMF timestamps in SMF records, such as reader start date & time, where the date is PL4'0cyydddF' and the time is a fullword binary number of 1/100ths of a second past midnight. But what is confounding me is that this seems to be the LOCAL time, not GMT/UTC (yes, I know GMT != UTC).
This is indeed LOCAL. SMF doc is not that useful describing LOCAL against GMT/UTC times.
> The problem that I am having is that I really want to convert this to GMT/UTC and then format it to RFC3339 encoding (yyyy-mm-ddThh:mm:ss.thZ).
Try this (from one of my live SMF type 30 extract jobs)
MVC CONTIME,SMF30TME
MVC CONDATE,SMF30DTE
--------------------------
CONVTOD CONVVAL=CONVVAL,
TIMETYPE=DEC,
DATETYPE=YYYYMMDD,
ETODVAL=ETODVAL
CONVVAL DS 0F
CONTIME DC 2F'0'
CONDATE DS F
DC F'0'
ETODVAL DS 2F
and try the OFFSET= keyword for your time. Of course you need to do some editing to insert '-', ':' and '.'.
>Do I just "give up"? Or am I missing something simple?
You may NOT 'give up' :-D
Groete / Greetings
Elardus Engelbrecht
McKown, John
--
John McKown
Systems Engineer IV
IT
Administrative Services Group
HealthMarkets(r)
9151 Boulevard 26 * N. Richland Hills * TX 76010
(817) 255-3225 phone *
john....@healthmarkets.com * www.HealthMarkets.com
Confidentiality Notice: This e-mail message may contain confidential or proprietary information. If you are not the intended recipient, please contact the sender by reply e-mail and destroy all copies of the original message. HealthMarkets(r) is the brand name for products underwritten and issued by the insurance subsidiaries of HealthMarkets, Inc. -The Chesapeake Life Insurance Company(r), Mid-West National Life Insurance Company of TennesseeSM and The MEGA Life and Health Insurance Company.SM
Walt Farrell
>... That is, being in the US Central Timezone, when we go to Daylight Saving time (from TIMEZONE=W.06 to TIMEZONE=W.05) at 02:00, the time range from 02:00-02:59 repeats. Is this correct? If so, I don't know if 02:10, for example. is local time for 08:10 or 07:10 GMT.
>
>Do I just "give up"? Or am I missing something simple?
For reader start date/time you would probably need to keep track of when you first saw that date/time combination, and remember whether you were in daylight savings time at that point or not. And then hope that you didn't have a job start during the first 02:00-02:59 interval, and another one start at the exact same time during the second 02:00-02:59 interval.
There's another timestamp, though, for the date/time the record was provided to SMF. That one is a bit easier, I think. If you're processing the records from that day sequentially, if you're lucky then when you start getting records from the second 02:00-02:59 interval you'll notice the timestamp move backward, and you would then know that you've crossed the boundary. Of course, you might get unlucky and have no records during the first 02:00-02:59 interval, or have some from that interval but have the ones from the second interval occur late enough that you can't observe the timestamp moving backward.
So, there will be some cases you can't determine, I think.
--
Walt
Richard L Peurifoy
> I should have said that I know how to convert the SMFSTAMP8 to RCF3339 format. The problem is determining the timezone offset for a specific date& time rendered in LOCAL time. Now, for my own shop, there is only one problem: the 1 hour overlap when converting from standard time to daylight saving time. I think I'm "dead in the water" on this point. I simply don't have any way to confirm which offset to use, W.06 or W.05, just based on the individual SMF record.<sigh>
Perhaps we need a requirement for an option to use GMT in the SMF
header, or maybe to just use TOD timestamps. It would be nice to
have the GMT offset in the records too, but this would require
changes in the record structure which will be harder to do.
I think there are still some unused bits in the header flag byte,
perhaps one could be used to indicate a new version of the header.
If this is done, much thought should be given to what is needed.
--
Richard
Ed Gould
Barry, did a great job of explaining the issue(s). what we did with users is to say all times are where the system (CPU) is. This simplified issues immensely. In our case there were 2 locations one set and in Europe(and. Chicago) Both were fine with the agreement. They were not concerned with start stop issues so much as totals.
Ed
Gerhard Postpischil
> Perhaps we need a requirement for an option to use GMT in the SMF
> header, or maybe to just use TOD timestamps. It would be nice to
> have the GMT offset in the records too, but this would require
> changes in the record structure which will be harder to do.
The easiest would be to keep existing formats, but add a new SMF
record type that would record clock information whenever that is
changed (including the daylight savings transition), and provide
the IPL settings.
While it would be lovely to have only TOD timestamps, there is
no business case for a complete SMF rewrite.
Gerhard Postpischil
Bradford, VT
Paul Gilmartin
>
>For reader start date/time you would probably need to keep track of when you first saw that date/time combination, and remember whether you were in daylight savings time at that point or not. And then hope that you didn't have a job start during the first 02:00-02:59 interval, and another one start at the exact same time during the second 02:00-02:59 interval.
>
in local time, as opposed to UTC?
What century is this, anyway? I thought it had long been generally
believed that critical timestamps should be recorded in UTC.
On Tue, 6 Sep 2011 10:04:25 -0500, Richard L Peurifoy wrote:
>
>Perhaps we need a requirement for an option to use GMT in the SMF
>header, ...
Yup. But I suspect that many shops would fear to DTRT.
-- gil
Ted MacNEIL
in local time, as opposed to UTC?
It depends on the record/vendor, as Dr. Barry stated.
>What century is this, anyway? I thought it had long been generally believed that critical timestamps should be recorded in UTC.
Records/programmes have to be re-engineered to accomplish that.
Also, users depending on the existing formats wll be affected.
Again, it's the old compatibility issues.
I remember, back in the early 1980's, when IBM changed the format of the RMF Type74's (Device Activity) to a collapsed linked list, with the introduction of the 3380.
MICS & MXG weren't available yet; I had to completely re-write the extract programme, from scratch -- the only thing that had not changed was the standard 18-byte header.
And, NO, they didn't convert it to UTC.
-
Ted MacNEIL
eama...@yahoo.ca
Twitter: @TedMacNEIL
Paul Gilmartin
>
>The easiest would be to keep existing formats, but add a new SMF
>record type that would record clock information whenever that is
>changed (including the daylight savings transition), and provide
>the IPL settings.
>
whenever CVTLSO or CVTLDTO changes. And whenever a new SMF
recording data set is opened (that should cover IPL settings?)
There's no help for shops that tweak CVTLDTO to compensate
for TOD clock drift.
-- gil
Elardus Engelbrecht
>I should have said that I know how to convert the SMFSTAMP8 to RCF3339 format. The problem is determining the timezone offset for a specific date & time rendered in LOCAL time.
Ok. You want to determine what the GMT/UTC is at the time of LOCAL time?
If so, sorry that I'm not being able to help you out on this one... :(
>Now, for my own shop, there is only one problem: the 1 hour overlap when converting from standard time to daylight saving time. I think I'm "dead in the water" on this point. I simply don't have any way to confirm which offset to use, W.06 or W.05, just based on the individual SMF record.
Damn! While we are not using those daylight saving time thing, I see your problem...
What about a table driven solution? Say for hours 1-hours 2 you use W.06, hours 2 - hours 3 you use W.05? I admit this is *ugly* and may not work every year.
Hopefully you can get some good solutions to your problem. Please tell us if you get a good solution.
Groete / Greetings
Elardus Engelbrecht
----------------------------------------------------------------------
Gord Tomlin
John felt it necessary to state that he knows GMT and UTC are not the
same thing.
--
Regards, Gord Tomlin
Action Software International
(a division of Mazda Computer Corporation)
Tel: (905) 470-7113, Fax: (905) 470-6507
On 2011-09-06 09:23, McKown, John wrote:
> The SMF timestamps in SMF records, such as reader start date& time, where the date is PL4'0cyydddF' and the time is a fullword binary number of 1/100ths of a second past midnight. But what is confounding me is that this seems to be the LOCAL time, not GMT/UTC (yes, I know GMT != UTC). The problem that I am having is that I really want to convert this to GMT/UTC and then format it to RFC3339 encoding (yyyy-mm-ddThh:mm:ss.thZ). I just don't see a way to arbitrarily do this. I have always assumed that the timestamp is basically the local time as you might see from a D T comand. That is, being in the US Central Timezone, when we go to Daylight Saving time (from TIMEZONE=W.06 to TIMEZONE=W.05) at 02:00, the time range from 02:00-02:59 repeats. Is this correct? If so, I don't know if 02:10, for example. is local time for 08:10 or 07:10 GMT.
Walt Farrell
>
>Damn! While we are not using those daylight saving time thing, I see your problem...
>
>What about a table driven solution? Say for hours 1-hours 2 you use W.06, hours 2 - hours 3 you use W.05? I admit this is *ugly* and may not work every year.
The problem with the transition from daylight-saving time to standard time, Elardus, is that the hour from 1 a.m. to 2 a.m. (0100 to 0200) happens twice: once while you're on daylight saving time, and once when you've switched back to standard time. (Assuming I got the times, and the direction correct. I think I was wrong about that in my prior note.) So I don't think a table-driven approach can work.
You can infer whether you've made the transition if you examine the records carefully, and note the times you see in them, and detect the change by noticing overlapping records (one record that says, e.g., 0159 followed by a record that says 0101. But if you don't see that kind of change you can't be sure whether the first set of 0100-0200 was daylight saving time or not. And that only works for the SMF record date/time, not for the reader start date/time fields.
--
Walt
Shmuel Metz , Seymour J.
at 11:31 AM, Gerhard Postpischil <ger...@VALLEY.NET> said:
>The easiest would be to keep existing formats, but add a new SMF
>record type
Ot new segments in the type 30.
--
Shmuel (Seymour J.) Metz, SysProg and JOAT
ISO position; see <http://patriot.net/~shmuel/resume/brief.html>
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)
Shane
> Am I supposed to infer from this discussion that SMF timestamps are
> recorded in local time, as opposed to UTC?
>
> What century is this, anyway? I thought it had long been generally
> believed that critical timestamps should be recorded in UTC.
LOL - I wondered how long it'd take for gil to arch up ... yet again.
Some systems make (sensible) use of tzdata.
Shane ...
Elardus Engelbrecht
>The problem with the transition from daylight-saving time to standard time, Elardus, is that the hour from 1 a.m. to 2 a.m. (0100 to 0200) happens twice: once while you're on daylight saving time, and once when you've switched back to standard time. (Assuming I got the times, and the direction correct. I think I was wrong about that in my prior note.) So I don't think a table-driven approach can work.
Thanks for kindly reply to my bad ( ???? :-D ) suggestion. I forgot about those two incidents of switching.
>You can infer whether you've made the transition if you examine the records carefully, and note the times you see in them, and detect the change by noticing overlapping records (one record that says, e.g., 0159 followed by a record that says 0101. But if you don't see that kind of change you can't be sure whether the first set of 0100-0200 was daylight saving time or not. And that only works for the SMF record date/time, not for the reader start date/time fields.
Very interesting about that overlapping thing. Thanks very much! :-D
I wonder if John McKown could consider your good reply?
Groete / Greetings
Elardus Engelbrecht
----------------------------------------------------------------------
Anthony Thompson
Ant.
Northern Territory Government of Australia
-----Original Message-----
From: IBM Mainframe Discussion List [mailto:IBM-...@bama.ua.edu] On Behalf Of Elardus Engelbrecht
Sent: Wednesday, 7 September 2011 4:46 PM
To: IBM-...@bama.ua.edu
Subject: Re: SMF timestamps
John McKown
relative physical sequence of individual records never change. Which
implies not having been run through a sort process. Of course, we have
multiple z/OS images and we merge all the SMF data together using
DFSORT. This is done every Sunday because Sundays are usually not very
busy.
I hate local time. DST is the work of the IT devil. Long live UTC! Of
course most people, at least in the shops I've worked in, are not too
smart in this area. They can't handle a 24 hour clock at all. And if it
isn't in local time, they freak out. Just say "19:30 hours Zulu" to them
and you may as well be speaking ancient Sumerian. And that's the
professional programmers.
On Wed, 2011-09-07 at 17:47 +0930, Anthony Thompson wrote:
> I suppose the trick is to ensure SMF records are processed in the
> order they are written. Look for Type 0 records to determine the time
> zone offset at IPL, start processing records of interest with that
> offset while also looking for Type 90 records with a SET
> CLOCK/DATE/RESET sub-type for any variations to the time zone offset.
> I'm wondering if a SET TIMEZONE command gets written as a SET CLOCK
> record...
>
> Ant.
> Northern Territory Government of Australia
>
--
John McKown
Maranatha! <><
Al Sherkow
One issue I have run into with my product is that this record only has the 4-char SMF System, not the 8-character SYSNAME or the SYSPLEX. (I have clients that have multiple 'SYSA's in their environment.)
--
Al Sherkow, I/S Management Strategies, Ltd.
Consulting Expertise on Capacity Planning, Performance Tuning,
WLC, LPARs, and LCS Software
Seminars on IBM SW Pricing, and LPARs
Voice: +1 414 332-3062
Web: www.sherkow.com
Ed Gould
SMF timestamps are a little bit tricky as they are not when the record is written but when they are created.
So a record could be in storage and not written for as long as a week maybe more. We had one job that lasted a week. We had to change code to allow for this and it wasn't easy to change the code especially across months and years. We also wrestled with the DLS time issue a lot.
There is also a issue with DB2 with changing date which I have heard was really bad when the time is changed backwards.
Ed
Robert A. Rosenberg
>I should have said that I know how to convert the SMFSTAMP8 to
>RCF3339 format. The problem is determining the timezone offset for a
>specific date & time rendered in LOCAL time. Now, for my own shop,
>there is only one problem: the 1 hour overlap when converting from
>standard time to daylight saving time. I think I'm "dead in the
>water" on this point. I simply don't have any way to confirm which
>offset to use, W.06 or W.05, just based on the individual SMF
>record. <sigh>
Since each record has a time stamp in its header showing (in local
time) when it was written, I would think that if record X+1's time
stamp is an hour earlier than record X's time stamp that would be a
red flag that the time shift just occurred (so long is the date is
the correct day for the switch). All you need to use is a sanity
check routine to keep track of if the switch has occurred. Once you
get to 3AM on the switch date you are past the ambiguous hour.
Vernooij, CP - SPLXM
news:<1315436928.60134...@web161421.mail.bf1.yahoo.com>...
> Interesting topic.
> SMF timestamps are a little bit tricky as they are not when the record
is written but when they are created.
> So a record could be in storage and not written for as long as a week
maybe more.
You mean by 'created' when it was given to SMF? You can control the time
it can wait in SMF buffers with the MAXDORM(nnnn) parameter.
Kees.
We had one job that lasted a week. We had to change code to allow for
this and it wasn't easy to change the code especially across months
and years. We also wrestled with the DLS time issue a lot.
> There is also a issue with DB2 with changing date which I have heard
was really bad when the time is changed backwards.
>
> Ed
>
> ----------------------------------------------------------------------
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to list...@bama.ua.edu with the message: GET IBM-MAIN INFO
> Search the archives at http://bama.ua.edu/archives/ibm-main.html
********************************************************
For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message.
Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The Netherlands, with registered number 33014286
********************************************************
Walt Farrell
>
>Since each record has a time stamp in its header showing (in local
>time) when it was written, I would think that if record X+1's time
>stamp is an hour earlier than record X's time stamp that would be a
>red flag that the time shift just occurred (so long is the date is
>the correct day for the switch). All you need to use is a sanity
>check routine to keep track of if the switch has occurred. Once you
>get to 3AM on the switch date you are past the ambiguous hour.
--
Walt