Apache logs in milliseconds format to elasticsearch

Riz vaughan

unread,

Dec 3, 2021, 12:24:15 PM12/3/21

to Fluentd Google Group

Hello all,

I am trying to transport apache logs with time in milliseconds to elasticsearch but having troubles doing it.

When I change logformat of apache to milliseconds since epoch, the logs arrive in elasticsearch but elasticseasrch create index for each log.

We have hybris logs in (e.g)"2021-12-03T16:30:03.834+0100" format and with the following config

type tail

format json

path /opt/hybris_logs/tomcat/console.json

pos_file /var/log/td-agent/buffer/hybris.console

tag x.x.x.x.hybris.console

</source>

and the logs arrive in elasticsearch without any issue and with subsecond precision.

I changed apache log format and apache emits logs in the same format as the above e.g "2021-12-03T15:32:47.588+0000" but td-agent doesn't forward the logs.

I get "[warn]: #0 pattern not matched".

Config for apache logs is identical to the hybris one(above).

Could someone help me out with this?

Thank you

José Lecaros

unread,

Dec 9, 2021, 6:21:05 AM12/9/21

to Fluentd Google Group

Hi Riz,

I'd like to invite you to make your questions either on StackOverflow or in Fluentd Slack channel (fluent-all.slack.com). This group will be closed for new requests soon.

Have a great day

Riz vaughan

unread,

Dec 20, 2021, 5:52:23 PM12/20/21

to Fluentd Google Group

Thank you for your suggestion. I think I know how to do it now(for futher references). "time_key use_nil" in fluentd with apache logs in "%{msec}t" helped me.

Have a great day to you too!

Sloik Konfitur

unread,

Dec 21, 2021, 6:29:53 AM12/21/21

to flu...@googlegroups.com

Hi Jose,

How to join Fluentd Slack channel? It looks it is accepting only users from 3 following domains

Thanks
-Sloik

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fluentd/8ced905b-2ee8-490b-97cc-6aa2a9f95408n%40googlegroups.com.

Sloik Konfitur

unread,

Dec 21, 2021, 6:31:35 AM12/21/21

to flu...@googlegroups.com

Hi Jose,

How to join Fluentd Slack channel? It looks it is accepting only users from 3 following domains

You can use any account with the domain:

Thanks
-Sloik

On 09-Dec-21 12:21 PM, José Lecaros wrote:

--

Anurag Gupta

unread,

Dec 22, 2021, 12:53:01 AM12/22/21

to Fluentd Google Group

Please use slack.Fluentd.org to sign up. The previous link is mainly for accessing the slack channel once registered.