collecting data from cloudwatch

[aruna pavan]

Hi,,

can someone please help me in collecting logs from cloudwatch and send them to mysql using fluentd?

im completely new to fluentd.Please help me

[Mr. Fiber]

Did you try https://github.com/yunomu/fluent-plugin-cloudwatch ?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[aruna pavan]

Hi,

Thanks for responding. I installed the input plug-in for cloudwatch and also configured the td-agent.conf file.

Attached is my config file. Please let me know where i missed.

TIA

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/4wtkZ8HZPmc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

[Mr. Fiber]

> I installed the input plug-in for cloudwatch and also configured the td-agent.conf file.

It means cloudwatch plugin doesn't work on your environment?

What the error did you see in the log?

[aruna pavan]

I did not see any error.But the file where i'm trying to put my cloudwatch logs is blank

[Mr. Fiber]

Your configuration refers cloudwatch logs service, not cloudwatch.

AFAIK, cloudwatch logs and cloudwatch have different APIs.

If you want to use cloudwatch logs, not cloudwatch, then

cloudwatch-logs plugin is better.

BTW, you should not upload API keys information in the public

even though it is test user.

[aruna pavan]

Hi Fiber,

 

Thank you so much for your suggestion. i installed cloudwatch-logs plugin and modified the config file as you said.

<source>
  type cloudwatch_logs
  tag cloudwatch.in
  log_group_name xxxxxxx
  log_stream_name xxxxxxxxx
  use_log_stream_name_prefix true
  state_file /var/lib/fluent/group_stream.in.state
</source>
<match cloudwatch>
  @type copy
 <store>
  @type file
  path xxxxx
 </store>
</match>

 

and restarted the td-agent. but  the path file is still blank.

 

kindly guide me.

[Mr. Fiber]

"<match cloudwatch>" doesn't match "cloudwatch.in" tag.

See http://docs.fluentd.org/articles/config-file#match-pattern-how-you-control-the-event-flow-inside-fluentd

[aruna pavan]

<source>

  type cloudwatch_logs

  tag test.cloudwatch_logs.in

  log_group_name /var/log/messages

  log_stream_name xxxxxxx

  use_log_stream_name_prefix true

  state_file /var/lib/fluent/group_stream.in.state

</source>

<match test.cloudwatch_logs.in>

  type stdout

 </match>

when i checked the output from /var/log/td-agent/td-agent.log, it is giving me error as no patterns matched tag="cloudwatch"

cloudwatch:watcher thread is not working after ............  restarting..

[Mr. Fiber]

Please paste your entire logs without secret parameters.

[aruna pavan]

Attache dis my output. Kindly guide me in resolving this issue.

[Mr. Fiber]

Did you use above configuration?

This log is generated by cloudwatch plugin, not cloudwatch-logs plugin.

But your pasted configuration doesn't include cloudwatch plugin.

https://github.com/yunomu/fluent-plugin-cloudwatch/blob/master/lib/fluent/plugin/in_cloudwatch.rb#L95

Please re-check your environemnt.

Currently, I can't help you because problems are mixed.

[aruna pavan]

yes..i used the above configuration only.checked my config file again.

the aws cli is also working fine.