Hi all,
I've got an event source (happens to be nxlog) that's sending in JSON to a fluentd listener
JSON:
{"EventTime":"2015-09-03 08:45:39","Hostname":"ahmwhswnpr01","Keywords":-9223372036854775808,"EventType":"INFO","SeverityValue":2....
Fluentd input:
<source>
type tcp
tag tcp.events # required
format json
port 24223
# time_key EventTime
# time_format "%Y-%m-%d %H:%M:%S"
</source>
This seems to work OK, however the times that are coming in are actually in local timezone (without TZ suffix), and fluentd/ruby appears to be assuming that they're in UTC, resulting in wrong timestamps downstream. I tried to make the event source output a "+1000" (etc) suffix so the default parser would work OK, but y'know Windows only wants to output something like "AUS Eastern Standard Time" (ugh!)
So... I thought just filter them on fluentd, and reparse the time from string. I tested with a static string, worked fine - but I can't figure out how to access fields from the input message. So, fluentd config like:
<filter tcp.events>
<record>
# Input EventTime is a time in local timezone without TZ so we need to use TZ from local machine
newtime ${Time.parse("2015-09-03 09:40:27", Time.now).to_i}
fixed1 ${EventTime} # null?!?
fixed2 ${tag_parts[1]} # works OK
fixed3 ${eventtime} # empty
fixed4 ${no_such_field} # empty
# fixed2 ${Time.parse(@EventTime, Time.now).to_i}
</record>
type record_transformer
# renew_time_key newtime
enable_ruby
log_level debug
</filter>
I tried EventTime, @EventTime etc; but could never access this values:
2015-09-03 12:57:37 +1000 [warn]: plugin/filter_record_transformer.rb:216:rescue in expand: failed to expand `${EventTime}` error_class=NameError error="uninitialized constant Fluent::RecordTransformerFilter::RubyPlaceholderExpander::EventTime"
2015-09-03 12:57:37 +1000 [warn]: plugin/filter_record_transformer.rb:123:expand_placeholders: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/filter_record_transformer.rb:214:in `block in expand'
2015-09-03 12:57:37 +1000 [warn]: plugin/filter_record_transformer.rb:123:expand_placeholders: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.12/lib/fluent/plugin/filter_record_transformer.rb:214:in `eval'
...
2015-09-03 12:57:37 +1000 tcp.events: {"EventTime":"2015-09-03 09:45:39 AUS Eastern Standard Time","Hostname":"ahmwhswnpr01","Keywords":-9223372036854775808,"EventType":"INFO","SeverityValue":2,"Severity":"INFO","EventID":37,"SourceName":"Microsoft-Windows-Time-Service","ProviderGuid":"{06EDCFEB-0FD0-4E53-ACCA-A6F8BBF81BCB}","Version":0,"Task":0,"OpcodeValue":0,"RecordNumber":17159,"ProcessID":972,"ThreadID":4124,"Channel":"System","Domain":"NT AUTHORITY","AccountName":"LOCAL SERVICE","UserID":"LOCAL SERVICE","AccountType":"Well Known Group","Message":"The time provider NtpClient is currently receiving valid time data from ntp2.aws.medibank.local,0x8 (ntp.m|0x8|0.0.0.0:123->10.168.11.248:123).","Opcode":"Info","TimeSource":"ntp2.aws.medibank.local,0x8 (ntp.m|0x8|0.0.0.0:123->
10.168.11.248:123)","EventReceivedTime":"2015-09-02 15:06:05","SourceModuleName":"in","SourceModuleType":"im_msvistalog","newtime":"1441237227","fixed1":null,"fixed2":"events","fixed3":"","fixed4":""}
So what dumb thing am I doing wrong? How can I view all the placeholders? Are the values nested inside some other dict/map?