A search to find misconfigured timezones and clocks on forwarders (or TZ values in conf files)

[Gregg Woodcock]

Here is the actual search I have been using to find TZ/clock problems:

index=* | eval delay_secs=abs(_time - _indextime) | where delay_secs > 1800 | eval delay_mins=(delay_secs / 60) | eval delay_hrs=(delay_mins/60) | stats count avg(delay_mins) avg(delay_hrs) by host,source

[Todd Gow]

Hi Gregg, 

Great stuff and thanks for sharing. 

You mentioned a search command or flag that you put on the end of all of your searches. What was that again? 

NOT notify or something like that?

Thanks in advance, 

-todd

splunk> I am speed

Todd "shamgow" Gow

Sr. Inside Sales Engineer

Splunk, Inc.

Office:  +1.972.244.8753

Mobile: +1.972.955.2552

Email: tg...@splunk.com

Plano > San Francisco Cupertino  London  Hong Kong  Washington D.C.  Seattle

 

Are you a member of the Dallas Splunk User's Group yet? 

If not, you might want to join today. 

http://www.meetup.com/Splunk/Plano-TX/34952/

 

 

This message is intended only for the personal, confidential, and authorized use of the recipient(s) named above. If you are not that person, you are not authorized to review, use, copy, forward, distribute or otherwise disclose the information contained in the message.

[Rick Curry]

On Wednesday, June 13, 2012 8:47:03 AM UTC-5, Gregg Woodcock wrote:

Here is the actual search I have been using to find TZ/clock problems:

index=* | eval delay_secs=abs(_time - _indextime) | where delay_secs > 1800 | eval delay_mins=(delay_secs / 60) | eval delay_hrs=(delay_mins/60) | stats count avg(delay_mins) avg(delay_hrs) by host,source

 

Gregg,

Thank you for this. I ran this on my test system and am finding very "interesting" results. I am not totally clear what I do with this info. I now have the hosts with potentially misconfigured TZ or date/time settings and the source from which that information is coming. 99.9% are Forwarders (I have a few hits from one of my Splunk Servers!) so I am certain as all of my Forwarders have been built from a common install image that there are no misconfigured TZ settings on them (they are also either on "locked down" servers and/or the Forwarder path of the file system has restricted access defined to it), the mal-adjusted date/time must be in a system setting. Safe assumption? Not so sure as some of our app groups do testing for "future event" processing, adjusting the date/time either in their application via special properties settings, special processing parameters passed another way or by adjusting the system settings and this must be accepted as is for those servers. Naturally, these situation create for a large number of 'hits' with the mal-adjusted time. (8-$)

[Gregg Woodcock]

See other post on "unknownify" macro.

[Gregg Woodcock]

On Friday, June 15, 2012 8:52:18 AM UTC-5, Rick Curry wrote:

I ran this on my test system and am finding very "interesting" results. I am not totally clear what I do with this info.

The search is showing you how many inputs have events that are significantly non-realtime.  It will only consider events that were indexed more than 30 minutes different than the *interpreted* timestamp of the event.

Somewhere in Splunk, probably either in inputs.conf or props.conf, you have a "TZ = <something>" line for your input (if not, you should do this).  This is what tells Splunk how to normalize/interpret the timestamp when analyzing or presenting data to the user.

If you find events from a source are showing to be an average of 1 hour off, I would suspect that maybe you have TZ = CST6CDT when the data is actually not daylight savings (should be TZ=US/Central).  Or perhaps you are relying on the fact that Splunk will ALWAYS use a TZ value inside an event timestamp (overrides all other settings) and most of your devices are outputting timestamps like that but some of them are not so Splunk is using whatever TZ value it uses for default (probably UTC) and the solution is to configure that device to include the TZ value like it is supposed to.  It all depends on what your policy is for TZ and timestamping your events.