Installing fluentd-geoip-plugin whileoffline onto centos7 minimal

[Jack Borg-Cardona]

Hi, I'm struggling to get this installed and working i was hoping someone has gone through this before or is able to point me in the right direction.

As suggested in the title i cannot use the prescribed method from the github repository README.md

I have to do it manually without using the internet during the install process. I can however dl files and copy them over.  So far i have got the dependencies sorted. and tried a whole number of things.

First following the plugin-man guide i have triedputting the folder in the right place and restarting fluentd with the plugin in its folder. On restarting the td-agent i get the following error on the log file:
#[error]: config error file="/etc/td-agent/td-agent.conf" error="Unknown output plugin 'geoip'. Run 'gem search -rd fluent-plugin' to find plugins
so i tried:
#td-agent-gem search -rld fluent-plugin

and sure enough it does not list the geoip plugin so i could not have installed this properly.

So next i have tried and got back:
#td-agent-gem install fluentd-plugin-geoip
#Error: Could not find a valid gem 'fluent-plugin-geoip/' (>= 0), here is why: Unable to download data from https://rubygems.org/ - no such name (https://rubygem.org/latest_specs.4.8.gz

SO i figured let me add that .gz file as a local source but:
#td-agent-gem sources --add file://127.0.0.1/etc/td-agent/latest_spec.4.8.gz
#error fetching file://127.0.0.1/etc/td-agent/latest_specs.4.8.gz
Errrno:ENOENT: No such file or dirctory @ rb_sysopen - /etc/td-agent/specs.4.8.gz (file://127.0.0.1/etc/td-agent/specs.4.8.gz)
I've tried dozens of other variations of this and check permssions and file ownership. SO no idea why i can't add this sources. I assume i've done something wrong???

In addition i have tried and received:

#td-agent-gem install Gemfile
#Error Could not find valid 'Gemfile' (>=0), hereos why:Unable to download data from https://rubygems.org - no such name (https://rubygems.org/latest_specs.4.8.gz

I'm kind stumped at this point, as a ruby newb having never used it before and this being the first time i'm installing a fluentd plugin i'm assuming I am missing something blatant... Any help would be appreciated! I have alot more offline plugin installs coming!

Thanks
Jack

[Jack Borg-Cardona]

UPDATE

After some head scratching and more googling i tried the following:

#td-agent-gem install -f --local fluent-plugin-geoip-0.4.0.gem
#Successfully installed
luent-plugin-geoip-0.4.0.gem

#Installing ri documentation for luent-plugin-geoip-0.4.0.gem

#Done installing documentation for luent-plugin-geoip-0.4.0.gem
#1 gem installed


and then checked it with

#~td-agent-gem search -rld fluent-plugin

and the local gem is listed there
'Installed at /opt/td-agent/embedded/lib/ruby/gems/2.1.0'

So i would have thought from this that was success. But if i restart the td-agent service to reload the conf file i get an error in the conf file:

#[error]: config error file="/etx/td-agent/td-agent.conf" error="Unknown output plugin 'geoip'. Run gem search -rd fluent-plugin' to find plugins.

But as demonstrated above that search for installed gems shows geoip-plugin to be installed?
Any ideas what's missing at this stage...

[Mr. Fiber]

That's weird.

Could you paste entire following result?

1. sudo /usr/sbin/td-agent-gem list

2. cat /var/log/td-agent/td-agent.log

3. cat /etc/td-agent/td-agent.conf

4. How to run td-agent? sudo service td-agent start?

Masahiro

--
You received this message because you are subscribed to the Google Groups "Fluentd Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fluentd+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Jack Borg-Cardona]

Hi Masahiro,

Ok so I've collected the data, here it is...
1. sudo /usr/sbin/td-agent-gem list

actionmailer (4.1.9)
actionpack (4.1.9)
actionview (4.1.9)
activemodel (4.1.9)
activerecord (4.1.9)
activesupport (4.1.9)
addressable (2.3.6)
arel (5.0.1.20140414130214)
aws-sdk (1.60.1)
aws-sdk-v1 (1.60.1)
bigdecimal (1.2.4)
bson (1.10.2)
bson_ext (1.10.2)
builder (3.2.2)
bundler (1.7.4)
celluloid (0.15.2)
cool.io (1.2.4)
diff-lcs (1.2.5)
draper (1.4.0)
erubis (2.7.0)
fluent-logger (0.4.10)
fluent-mixin-config-placeholders (0.3.0)
fluent-mixin-plaintextformatter (0.2.6)
fluent-plugin-geoip (0.4.0)
fluent-plugin-mongo (0.7.5)
fluent-plugin-rewrite-tag-filter (1.4.1)
fluent-plugin-s3 (0.5.3)
fluent-plugin-scribe (0.10.14)
fluent-plugin-td (0.10.25)
fluent-plugin-td-monitoring (0.2.0)
fluent-plugin-webhdfs (0.4.1)
fluentd (0.10.60)
fluentd-ui (0.3.14)
font-awesome-rails (4.2.0.0)
haml (4.0.6)
haml-rails (0.5.3)
hike (1.2.3)
hirb (0.7.3)
http_parser.rb (0.6.0)
httpclient (2.4.0)
i18n (0.6.11)
io-console (0.4.2)
ipaddress (0.8.0)
jbuilder (2.2.6)
jquery-rails (3.1.2)
json (1.8.1)
kramdown (1.5.0)
kramdown-haml (0.0.3)
ltsv (0.1.0)
mail (2.6.3)
mime-types (2.4.3)
mini_portile (0.6.0)
minitest (5.4.2, 4.7.5)
mixlib-cli (1.5.0)
mixlib-config (2.1.0)
mixlib-log (1.6.0)
mixlib-shellout (2.0.1)
mongo (1.10.2)
msgpack (0.5.11)
multi_json (1.10.1)
nokogiri (1.6.3.1)
ohai (6.20.0)
parallel (0.6.5)
psych (2.0.5)
puma (2.11.0)
rack (1.5.2)
rack-test (0.6.2)
rails (4.1.9)
railties (4.1.9)
rake (10.1.0)
rdoc (4.1.0)
request_store (1.1.0)
rubyzip (1.1.6, 0.9.9)
sass (3.2.19)
sass-rails (4.0.4)
settingslogic (2.0.9)
sigdump (0.2.2)
sprockets (2.11.3)
sprockets-rails (2.2.0)
string-scrub (0.0.5)
sucker_punch (1.0.5)
systemu (2.5.2)
td (0.11.6)
td-client (0.8.67)
td-logger (0.3.24)
test-unit (2.1.3.0)
thor (0.19.1)
thread_safe (0.3.4)
thrift (0.8.0)
tilt (1.4.1)
timers (1.1.0)
tzinfo (1.2.2)
tzinfo-data (1.2015.1)
uuidtools (2.1.5)
webhdfs (0.6.0)
yajl-ruby (1.2.1)

2. cat /var/log/td-agent/td-agent.log
2015-04-23 08:52:55 +0100 [info]: reading config file path="/etc/td-agent/td-agent.conf"
2015-04-23 08:52:55 +0100 [info]: starting fluentd-0.10.60
2015-04-23 08:52:55 +0100 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-04-23 08:52:55 +0100 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-04-23 08:52:55 +0100 [info]: gem 'fluent-plugin-geoip' version '0.4.0'
2015-04-23 08:52:55 +0100 [info]: gem 'fluent-plugin-mongo' version '0.7.5'
2015-04-23 08:52:55 +0100 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2015-04-23 08:52:55 +0100 [info]: gem 'fluent-plugin-s3' version '0.5.3'
2015-04-23 08:52:55 +0100 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2015-04-23 08:52:55 +0100 [info]: gem 'fluent-plugin-td' version '0.10.25'
2015-04-23 08:52:55 +0100 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.0'
2015-04-23 08:52:55 +0100 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2015-04-23 08:52:55 +0100 [info]: gem 'fluentd' version '0.10.60'
2015-04-23 08:52:55 +0100 [info]: using configuration file: <ROOT>
  <source>
    type tail
    read_from_head true
    path /mnt/juniperextended.txt
    pos_file /mnt/junipergeo.pos
    tag geoip.
    format /^.*?"Juniper: id=(?<id>\S*) time=\\"(?<time>.*?)\\" pri=(?<priority>\S*) fw=(?<firewall>\S*) vpn=(?<nodename>\S*) user=(?<user>\S*) realm=\\"(?<realm>[^\\]*)\\" roles=\\"(?<roles>[^\\]*)\\" proto=(?<proto>.*?) src=(?<sourceip>\S*) dst=(?<destip>\S*) dstname=(?<dstname>\S*) type=(?<type>\S*) op=(?<op>\S*) arg=\\"(?<arg>.*?)\\" result=(?<result>\S*) sent=(?<bytes_sent>\S*) rcvd=(?<bytes_received>\S*) agent=\\"(?<agent>.*?)\\" duration=(?<duration>\S*) msg=\\"(?<eventcode>.*?):(?<eventmessage>.*?)""}/
  </source>
  <match geoip.**>
    type geoip
    geoip_lookup_key sourceip
    enable_key_country_code geoip_country
    enable_key_city geoip_city
    enable_key_latitude geoip_lat
    enable_key_longitude geoip_lon
    flush_interval 10s
    remove_tag_prefix geoip.
    add_tag_prefix squeeze.
    buffer_type file
    buffer_queue_limit 1000000
    buffer_path /mnt/buffer/geo/squeeze-isogeo.*.buffer
    buffer_chunk_limit 50m
  </match>
  <match squeeze.**>
    type file
    path /mnt/junipergeoout.txt
    append true
  </match>
</ROOT>
2015-04-23 08:52:55 +0100 [info]: adding source type="tail"
2015-04-23 08:52:55 +0100 [info]: adding match pattern="geoip.**" type="geoip"
2015-04-23 08:52:55 +0100 [error]: config error file="/etc/td-agent/td-agent.conf" error="Unknown output plugin 'geoip'. Run 'gem search -rd fluent-plugin' to find plugins"
2015-04-23 08:52:55 +0100 [info]: process finished code=256
2015-04-23 08:52:55 +0100 [warn]: process died within 1 second. exit.

cat /etc/td-agent/td-agent.conf
<source>
    type         tail
    read_from_head     true
    path         /mnt/juniperextended.txt
    pos_file     /mnt/junipergeo.pos
    tag        geoip.
    format        /^.*?"Juniper: id=(?<id>\S*) time=\\"(?<time>.*?)\\" pri=(?<priority>\S*) fw=(?<firewall>\S*) vpn=(?<nodename>\S*) user=(?<user>\S*) realm=\\"(?<realm>[^\\]*)\\" roles=\\"(?<roles>[^\\]*)\\" proto=(?<proto>.*?) src=(?<sourceip>\S*) dst=(?<destip>\S*) dstname=(?<dstname>\S*) type=(?<type>\S*) op=(?<op>\S*) arg=\\"(?<arg>.*?)\\" result=(?<result>\S*) sent=(?<bytes_sent>\S*) rcvd=(?<bytes_received>\S*) agent=\\"(?<agent>.*?)\\" duration=(?<duration>\S*) msg=\\"(?<eventcode>.*?):(?<eventmessage>.*?)""}/

</source>
<match geoip.**>
                type geoip
                geoip_lookup_key                sourceip
                enable_key_country_code         geoip_country
                enable_key_city                 geoip_city
                enable_key_latitude             geoip_lat
                enable_key_longitude            geoip_lon
                flush_interval                  10s
                remove_tag_prefix               geoip.
                add_tag_prefix                  squeeze.

                buffer_type                     file
                buffer_queue_limit              1000000
                buffer_path                     /mnt/buffer/geo/squeeze-isogeo.*.buffer
                buffer_chunk_limit              50m

</match>
<match squeeze.**>
    type     file
    path     /mnt/junipergeoout.txt
    append    true
</match>

4. How to run td-agent? sudo service td-agent start?

I have systemd set to control td-agent; so yeh 'service td-agent start' is what is use.

i ran a journalctl -xf asi did a service td-agent start/stop  this is the output messages that came:

-- Unit td-agent.service has begun starting up.
 localhost.localdomain runuser[24119]: pam_unix(runuser:session): session opened for user td-agent by (uid=0)
 localhost.localdomain td-agent[24116]: Starting td-agent: 2015-04-29 09:56:35 +0100 [error]: fluent/supervisor.rb:275:rescue in main_process: config error file="/etc/td-agent/td-agent.conf" error="Unknown output plugin 'geoip'. Run 'gem search -rd fluent-plugin' to find plugins"
 localhost.localdomain runuser[24119]: pam_unix(runuser:session): session closed for user td-agent
 localhost.localdomain td-agent[24116]: [FAILED]
 localhost.localdomain systemd[1]: td-agent.service: control process exited, code=exited status=1
localhost.localdomain systemd[1]: Failed to start LSB: td-agent's init script.
-- Subject: Unit td-agent.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit td-agent.service has failed.
--
-- The result is failed.
 localhost.localdomain systemd[1]: Unit td-agent.service entered failed state.

Thanks

[Mr. Fiber]

I think your fluent-plugin-geoip installation failed because

your gem list doesn't have geoip-c.

fluent-plugin-geoip depends on geoip-c gem so it should be included.

You specified -f option so some dependency errors are ignored.

I don't know why you can't connect to internet on deploy phase.

If you don't want to waste the time for tricky deployment,

building td-agent like package with your used plugins is better.

[Li Ming Tsai]

Hi,

I'm trying to figure out gemfile.lock to use together with bundle so that I can get a consistent known set of gems to use in a deployment without Internet access.

Can you describe how to achieve a known set of plugins/gems for using td-agent rpm? Or should I go fluentd using gems? Or build my own rpm? 

Thanks!

[Mr. Fiber]

> Or build my own rpm? 

Some companies use this way.

Other way is downloading all dependent gems before and install it via deployment tool like chef/ansible/etc.