data is not going in continuous manner on server side mongodb

[Vinayak Choubey]

Dear all,

    I am a fresher in software field and i want to use fluentd to store my data on server side.as per my understanding i have written a right configuration file for client as well as server.But sometimes data goes to the database and sometimes it dosenot.So therefore kindly request to all that please help me for this as soon as possible.

First i am copying my client side configuration code here:

Clinet side config. code

.....................................................................

<source>

    type tail

   read_from_head true

     path /home/sourav/Downloads/aaaa.json

      pos_file /home/sourav/Downloads/aaaa.json.pos

    tag mongo.**

    format json

</source>

<match mongo.**>

  type forward

  send_timeout 60s

  recover_wait 10s

  heartbeat_interval 1s

  phi_threshold 16

  hard_timeout 60s 

    

        <server>

       host 192.168.1.43

          port 22576

       </server>

   

</match>

Now i am attaching a server side code:

............................................................................................

<source>

 type forward 

port 22576

bind 0.0.0.0

</source>

<match mongo.**>

type copy

     <store **>

     type mongo

    database mydbabse

    collection testdata

    host localhost

    port 27017

    flush_interval 10s

  </store>

</match>

These two configuration file i am using.Please help me

[Mr. Fiber]

Hi,

> But sometimes data goes to the database and sometimes it dosenot.

I want to know more detail of your situation.

Are there no errors or warnings in client and server fluentd logs?

And what does 'sometimes it dosenot' means?

Logs are not shown in MongoDB or delayed?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vinayak Choubey]

In client side after running td agent, i am getting this:

2015-04-24 12:53:05 +0530 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2015-04-24 12:53:05 +0530 [info]: starting fluentd-0.12.7
2015-04-24 12:53:05 +0530 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-04-24 12:53:05 +0530 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-04-24 12:53:05 +0530 [info]: gem 'fluent-plugin-mongo' version '0.7.8'
2015-04-24 12:53:05 +0530 [info]: gem 'fluent-plugin-multiprocess' version '0.1.2'
2015-04-24 12:53:05 +0530 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2015-04-24 12:53:05 +0530 [info]: gem 'fluent-plugin-s3' version '0.5.7'
2015-04-24 12:53:05 +0530 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2015-04-24 12:53:05 +0530 [info]: gem 'fluent-plugin-secure-forward' version '0.2.6'
2015-04-24 12:53:05 +0530 [info]: gem 'fluent-plugin-td' version '0.10.26'
2015-04-24 12:53:05 +0530 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.0'
2015-04-24 12:53:05 +0530 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2015-04-24 12:53:05 +0530 [info]: gem 'fluentd' version '0.12.7'
2015-04-24 12:53:05 +0530 [info]: using configuration file: <ROOT>

  <source>
    type tail
    read_from_head true
    path /home/sourav/Downloads/aaaa.json
    pos_file /home/sourav/Downloads/aaaa.json.pos
    tag mongo.**

    format none

  </source>
  <match mongo.**>
    type copy
    <store>

      type stdout
    </store>
    <store>

      type forward
      send_timeout 60s
      recover_wait 10s
      heartbeat_interval 1s
      phi_threshold 16
      hard_timeout 60s

      buffer_type file
      buffer_path /var/log/buffer
      flush_interval 1s
      flush_at_shutdown true
      <server>
        host 54.201.191.10
        port 22577
      </server>
    </store>
    <secondary>
      type file
      path /var/log/failed
    </secondary>
  </match>
</ROOT>
2015-04-24 12:53:05 +0530 [info]: adding match pattern="mongo.**" type="copy"
2015-04-24 12:53:05 +0530 [info]: adding forwarding server '54.201.191.10:22577' host="54.201.191.10" port=22577 weight=60 plugin_id="object:1716c80"
2015-04-24 12:53:05 +0530 [info]: adding source type="tail"
2015-04-24 12:53:05 +0530 [warn]: parameter 'type' in <secondary>
  type file
  path /var/log/failed
</secondary> is not used.
2015-04-24 12:53:05 +0530 [warn]: parameter 'path' in <secondary>
  type file
  path /var/log/failed
</secondary> is not used.
2015-04-24 12:53:05 +0530 [info]: following tail of /home/sourav/Downloads/aaaa.json
2015-04-24 12:53:42 +0530 [warn]: detached forwarding server '54.201.191.10:22577' host="54.201.191.10" port=22577 phi=16.238953285134595
2015-04-24 12:53:54 +0530 [info]: detected rotation of /home/sourav/Downloads/aaaa.json; waiting 5 seconds
2015-04-24 12:53:54 +0530 [info]: following tail of /home/sourav/Downloads/aaaa.json

2015-04-24 12:53:54 +0530 [warn]: emit transaction failed: error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/buffer.mongo.home.sourav.Downloads.aaaa.json.b51473475d23d39bf.log" tag="mongo.home.sourav.Downloads.aaaa.json"
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_file.rb:23:in `initialize'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_file.rb:23:in `open'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_file.rb:23:in `initialize'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_file.rb:125:in `new'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_file.rb:125:in `new_chunk'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/buffer.rb:181:in `block in emit'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/2.1.0/monitor.rb:211:in `mon_synchronize'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/buffer.rb:179:in `emit'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:440:in `emit'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:32:in `next'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_stdout.rb:48:in `emit'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:32:in `next'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:73:in `emit'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:88:in `emit_stream'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:229:in `receive_lines'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:320:in `call'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:320:in `wrap_receive_lines'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:513:in `call'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:513:in `on_notify'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:345:in `on_notify'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:326:in `attach'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:135:in `setup_watcher'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:174:in `update_watcher'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:402:in `call'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:402:in `on_rotate'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:567:in `call'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:567:in `on_notify'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:343:in `on_notify'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:446:in `call'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:446:in `on_change'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run_once'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run'
  2015-04-24 12:53:54 +0530 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_tail.rb:214:in `run'

[Mr. Fiber]

> 2015-04-24 12:53:42 +0530 [warn]: detached forwarding server '54.201.191.10:22577' host="54.201.191.10" port=22577 phi=16.238953285134595

From this line, fluentd on client side can't connect to fluentd on server side.

> emit transaction failed: error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/buffer.mongo.home.sourav.Downloads.aaaa.json.b51473475d23d39bf.log" tag="mongo.home.sourav.Downloads.aaaa.json"

And this line shows your /var/log doesn't provide a permission for td-agent.

Maybe, your setting has some problems.

Could you resolve above problems?

[Vinayak Choubey]

tail is not working .anybody have idea then please help me out.i am using this configuration:

<source>
    type tail
    read_from_head false
    path /media/MWUR_6_1_7600/JSON.json
    pos_file /home/sourav/Desktop/logs/JSON.json.pos
    tag mongo.**
     rotate_wait 20

    format json
</source>


<match mongo.**>

type copy

<store>
    type stdout
  </store>
<store>

type mongo
database mydb
collection testData
port 27017
host localhost
</store>
<store>
  type forward
  phi_failure_detector false

  hard_timeout 60s 
  buffer_type file
  buffer_path /var/log/buffer
  flush_interval 1s
  flush_at_shutdown true

</store>
</match>

whenever i tried to add new data it reads all the data present in the file.

[Mr. Fiber]

> whenever i tried to add new data it reads all the data present in the file.

It means stdout plugin prints all data at each call?

I tested your configuration without mongo and it works.

- fluentd log

% fluentd -c test.conf
2015-04-30 08:55:54 +0900 [info]: reading config file path="test.conf"
2015-04-30 08:55:54 +0900 [info]: starting fluentd-0.12.8
[snip]
2015-04-30 08:55:54 +0900 [info]: gem 'fluentd' version '0.12.8'
2015-04-30 08:55:54 +0900 [info]: using configuration file: <ROOT>

  <source>
    type tail
    read_from_head false

    path /Users/repeatedly/tmp/fluentd/JSON.json
    pos_file /Users/repeatedly/tmp/fluentd/JSON.json.pos

    tag mongo.**
    rotate_wait 20
    format json
  </source>

  <match mongo.**>
    type copy
    <store>
      type stdout
    </store>

  </match>
</ROOT>
2015-04-30 08:55:54 +0900 [info]: adding match pattern="mongo.**" type="copy"
2015-04-30 08:55:54 +0900 [info]: adding source type="tail"
2015-04-30 08:55:54 +0900 [info]: following tail of /Users/repeatedly/tmp/fluentd/JSON.json
2015-04-30 08:56:36 +0900 mongo.Users.repeatedly.tmp.fluentd.JSON.json: {"level":"WARN","id":1,"message":"Hi!"}

- Append json log

/Users/repeatedly/tmp/fluentd% wc -l JSON.json
       3 JSON.json

/Users/repeatedly/tmp/fluentd% echo '{"level":"WARN", "id":1, "message":"Hi!"}' >> JSON.json
/Users/repeatedly/tmp/fluentd% wc -l JSON.json
       4 JSON.json

[Vinayak Choubey]

Hello Sir,

    Hope you are fine.By your sincere guidance and help i have completely resolved my all the issues and now its running smoothly.Now i want to know that how can we install td-agent in windows operating system.I am using windows 8 with 64 bit.Because i am in dilema that is there any td-agent version which we can install in our windows operating system.

Thanking you in advance.

--
You received this message because you are subscribed to a topic in the Google Groups "Fluentd Google Group" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/fluentd/FXVYwJcpPaA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to fluentd+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks & Regards

Vinayak Choubey

(9893702438)

[Mr. Fiber]

> i have completely resolved my all the issues and now its running smoothly

Good to hear!

> is there any td-agent version which we can install in our windows operating system.

Curretly, td-agent doesn't provide windows support.

Fluentd itself woks on Windows using α version of windows branch.

https://github.com/fluent/fluentd/tree/windows

Conclusion, there is no production ready windows support for now.