Issue installing Stackdriver Logging Agent - google-fluentd service won't start

[Thomas Johnston]

Hello Stackdriver community!

Our team recently developed a need to monitor a website that we have running on Google's compute engine and we'd very much like to use stackdriver to do so. We are, however, running into an issue trying to install the logging agent (monitoring agent installed fine).

After running sudo bash install-logging-agent.sh, we encountered an issue with the google-fluentd service starting:

==============================================================================Starting installation of google-fluentd==============================================================================Installing agents for Debian or Ubuntu.OKSelecting previously unselected package google-fluentd.(Reading database ... 40316 files and directories currently installed.)Preparing to unpack .../google-fluentd_1.5.36-1_amd64.deb ...Unpacking google-fluentd (1.5.36-1) ...Selecting previously unselected package google-fluentd-catch-all-config.Preparing to unpack .../google-fluentd-catch-all-config_0.7_all.deb ...Unpacking google-fluentd-catch-all-config (0.7) ...Setting up google-fluentd (1.5.36-1) ...Adding system user `google-fluentd' (UID 109) ...Adding new group `google-fluentd' (GID 114) ...Adding new user `google-fluentd' (UID 109) with group `google-fluentd' ...Not creating home directory `/home/google-fluentd'.Installing default conffile /etc/google-fluentd/google-fluentd.conf ...Job for google-fluentd.service failed because the control process exited with error code.See "systemctl status google-fluentd.service" and "journalctl -xe" for details.invoke-rc.d: initscript google-fluentd, action "start" failed.● google-fluentd.service - LSB: data collector for Treasure Data Loaded: loaded (/etc/init.d/google-fluentd; generated; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2018-09-27 01:05:28 UTC; 129ms ago Docs: man:systemd-sysv-generator(8) Process: 2625 ExecStart=/etc/init.d/google-fluentd start (code=exited, status=1/FAILURE)Sep 27 01:05:27 wordpress-2-vm systemd[1]: Starting LSB: data collector for Treasure Data...Sep 27 01:05:28 wordpress-2-vm google-fluentd[2625]: Starting google-fluentd 1.5.36: Disabled via metadata .…ning).Sep 27 01:05:28 wordpress-2-vm google-fluentd[2625]: google-fluentd ... failed!Sep 27 01:05:28 wordpress-2-vm systemd[1]: google-fluentd.service: Control process exited, code=exited status=1Sep 27 01:05:28 wordpress-2-vm systemd[1]: Failed to start LSB: data collector for Treasure Data.Sep 27 01:05:28 wordpress-2-vm systemd[1]: google-fluentd.service: Unit entered failed state.Sep 27 01:05:28 wordpress-2-vm systemd[1]: google-fluentd.service: Failed with result 'exit-code'.Hint: Some lines were ellipsized, use -l to show in full.dpkg: error processing package google-fluentd (--configure): subprocess installed post-installation script returned error exit status 1dpkg: dependency problems prevent configuration of google-fluentd-catch-all-config: google-fluentd-catch-all-config depends on google-fluentd (>= 1.3.0); however: Package google-fluentd is not configured yet.dpkg: error processing package google-fluentd-catch-all-config (--configure): dependency problems - leaving unconfiguredErrors were encountered while processing: google-fluentd google-fluentd-catch-all-configE: Sub-process /usr/bin/dpkg returned an error code (1)Job for google-fluentd.service failed because the control process exited with error code.See "systemctl status google-fluentd.service" and "journalctl -xe" for details.==============================================================================Errors occurred while installing google-fluentd-1.5.36-1. See the log snippetabove or run: sudo service google-fluentd status==============================================================================

I've attempted the followup steps mentioned in https://groups.google.com/forum/#!topic/google-stackdriver-discussion/DI0YlKZK7Nw but they don't provide additional information:

1. Running status on the service, again, reports it as failed and repeats output from /etc/init.d/google-fluentd start indicating the issue with LSB failing to start.
   
2. /var/log/google-fluentd/google-fluent.log doesn't exist so I'm not really sure where to pull more information on this.
   

This is attempting to install the agent on a f1-micro VM in compute engine running Debian GNU/Linux 9.

Any help in getting us setup with the stackdriver logging agent would be greatly appreciated and thanks in advance!

[Thomas Johnston]

Looks like some of the message got cut out. Here's the important piece from the install that indicates the service failing to start:

● google-fluentd.service - LSB: data collector for Treasure Data Loaded: loaded (/etc/init.d/google-fluentd; generated; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2018-09-27 01:12:26 UTC; 26s ago Docs: man:systemd-sysv-generator(8) Process: 2826 ExecStart=/etc/init.d/google-fluentd start (code=exited, status=1/FAILURE)Sep 27 01:12:26 wordpress-2-vm systemd[1]: Starting LSB: data collector for Treasure Data...Sep 27 01:12:26 wordpress-2-vm google-fluentd[2826]: Starting google-fluentd 1.5.36: Disabled via metadata ... (warSep 27 01:12:26 wordpress-2-vm google-fluentd[2826]: google-fluentd ... failed!Sep 27 01:12:26 wordpress-2-vm systemd[1]: google-fluentd.service: Control process exited, code=exited status=1Sep 27 01:12:26 wordpress-2-vm systemd[1]: Failed to start LSB: data collector for Treasure Data.Sep 27 01:12:26 wordpress-2-vm systemd[1]: google-fluentd.service: Unit entered failed state.Sep 27 01:12:26 wordpress-2-vm systemd[1]: google-fluentd.service: Failed with result 'exit-code'.

[Igor Peshansky]

Thomas,

Sorry you're having trouble. We'll need some additional details.

What environment was this in (GCP, AWS)? What Linux distro/version? Did you follow the troubleshooting steps at http://cloud.google.com/logging/docs/agent/troubleshooting?

Right after the error it says:

See "systemctl status google-fluentd.service" and "journalctl -xe" for details.

Did you run those two commands? What was the output?

        Igor
-- sent from a mobile device, please excuse tyops and omissns

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To post to this group, send email to google-stackdr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/9d1b1c7e-a9f0-43de-ab55-a894db5a46f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thomas Johnston]

Thanks for helping me out Igor!

I am on GCP on a f1-micro VM in compute engine running Debian GNU/Linux 9.

I looked at http://cloud.google.com/logging/docs/agent/troubleshooting but logger "Some test message" never populated the log viewer with the test message and I couldn't find a link on troubleshooting that.

I don't know much about linux distros but systemctl status google-fluentd.service did not run. I think it's meant for a different distro maybe. Running sudo service google-fluentd status showed the lines indicated above in the reply to my original post.

Running journalctl -xe yields the following:

-- Unit google-fluentd.service has begun starting up.
Sep 27 06:40:01 wordpress-2-vm google-fluentd[1567]: Starting google-fluentd 1.5.36: Disabled via metadata ... (war
Sep 27 06:40:01 wordpress-2-vm google-fluentd[1567]: google-fluentd ... failed!
Sep 27 06:40:01 wordpress-2-vm systemd[1]: google-fluentd.service: Control process exited, code=exited status=1
Sep 27 06:40:01 wordpress-2-vm systemd[1]: Failed to start LSB: data collector for Treasure Data.
-- Subject: Unit google-fluentd.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit google-fluentd.service has failed.
-- 
-- The result is failed.
Sep 27 06:40:01 wordpress-2-vm systemd[1]: google-fluentd.service: Unit entered failed state.
Sep 27 06:40:01 wordpress-2-vm systemd[1]: google-fluentd.service: Failed with result 'exit-code'.
Sep 27 06:40:01 wordpress-2-vm sudo[1545]: pam_unix(sudo:session): session closed for user root

[Igor Peshansky]

Aha! You're using one of the Cloud Launcher images, aren't you? Here's your reason:

Starting google-fluentd 1.5.36: Disabled via metadata

You will find the google-logging-enable instance metadata key set to 0. Somehow, when you created the VM, that key got set. Either delete that key or set it to 1 and the service should start.

Sorry I missed both "GCE" and "Debian 9" in your original message...

        Igor
-- sent from a mobile device, please excuse tyops and omissns

--

© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To post to this group, send email to google-stackdr...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/80baa116-5ad7-4dc8-9563-74e41f508022%40googlegroups.com.

[steve...@narvar.com]

I am trying to install google-fluentd in an AWS EC2 instance.  I am getting simular error "Failed to start LSB":

[root@ip-10-101-24-133 google-fluentd]# systemctl status google-fluentd.service

● google-fluentd.service - LSB: data collector for Treasure Data

   Loaded: loaded (/etc/rc.d/init.d/google-fluentd; bad; vendor preset: disabled)

   Active: failed (Result: exit-code) since Wed 2018-10-03 23:21:49 UTC; 16min ago

     Docs: man:systemd-sysv-generator(8)

  Process: 26690 ExecStart=/etc/rc.d/init.d/google-fluentd start (code=exited, status=127)

    Tasks: 109

   Memory: 344.4M

   CGroup: /system.slice/google-fluentd.service

           ├─15997 /opt/google-fluentd/embedded/bin/ruby /usr/sbin/google-fluentd --log /var/log/google-fluentd/google-fluentd.log --use-v1-config --suppress-repeated-stacktrace --group...

           └─16002 /opt/google-fluentd/embedded/bin/ruby -Eascii-8bit:ascii-8bit /usr/sbin/google-fluentd --log /var/log/google-fluentd/google-fluentd.log --use-v1-config --suppress-rep...

Oct 03 23:21:49 ip-10-101-24-133.us-west-2.compute.internal systemd[1]: Starting LSB: data collector for Treasure Data...

Oct 03 23:21:49 ip-10-101-24-133.us-west-2.compute.internal google-fluentd[26690]: Starting google-fluentd 1.5.36: /lib/lsb/init-functions: line 22: /etc/redhat-lsb/lsb_log_mess...irectory

Oct 03 23:21:49 ip-10-101-24-133.us-west-2.compute.internal systemd[1]: google-fluentd.service: control process exited, code=exited status=127

Oct 03 23:21:49 ip-10-101-24-133.us-west-2.compute.internal systemd[1]: Failed to start LSB: data collector for Treasure Data.

Oct 03 23:21:49 ip-10-101-24-133.us-west-2.compute.internal systemd[1]: Unit google-fluentd.service entered failed state.

Oct 03 23:21:49 ip-10-101-24-133.us-west-2.compute.internal systemd[1]: google-fluentd.service failed.

 

Even though the service does not seem to be running, I can see the my logs are pushed to StackDriver fine.  The ps -aux command output shows that google-fluentd is running but it is not running as a service as the above output shows.

[root@ip-10-101-24-133 google-fluentd]# !ps

ps ax | grep fluentd

15997 ?        Sl     0:01 /opt/google-fluentd/embedded/bin/ruby /usr/sbin/google-fluentd --log /var/log/google-fluentd/google-fluentd.log --use-v1-config --suppress-repeated-stacktrace --group root --daemon /var/run/google-fluentd/google-fluentd.pid

16002 ?        Sl     4:32 /opt/google-fluentd/embedded/bin/ruby -Eascii-8bit:ascii-8bit /usr/sbin/google-fluentd --log /var/log/google-fluentd/google-fluentd.log --use-v1-config --suppress-repeated-stacktrace --group root --daemon /var/run/google-fluentd/google-fluentd.pid --under-supervisor

17885 pts/0    S+     0:00 grep --color=auto fluentd

[root@ip-10-101-24-133 google-fluentd]# 

When I installed google-fluentd, I set repo_gpgcheck=0 in the install shell script.  I also did "yum install deltarpm".  These are the only deviations from instruction provided here https://cloud.google.com/logging/docs/agent/installation

Because google-fluentd is not running as a service, I can not easily restart the agent using this sort of command:

sudo service google-fluentd restart

Is there a solution to this?

Thanks

Steve

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

[steve...@narvar.com]

It turned out Amazon Linux 2 AMI don't have /etc/redhat-lsb/

I will like to try deploying stackdriver agent with a docker container instead: https://hub.docker.com/r/fluent/fluentd-kubernetes-daemonset/

[Igor Peshansky]

Sorry, that is not the Stackdriver agent. We do have a docker image used for Kubernetes, but it's not set up to ingest logs from raw instances.

Does installing the redhat-lsb package fix the original startup issue? If so, we are about to make this exact change in our install scripts for other reasons, so you should see a fix soon.

        Igor
-- sent from a mobile device, please excuse tyops and omissns

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.

To post to this group, send email to google-stackdr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/80baa116-5ad7-4dc8-9563-74e41f508022%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.

To post to this group, send email to google-stackdr...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/29c31a7e-575e-4d69-8563-bc4faef260cb%40googlegroups.com.

[Thomas Johnston]

No worries and thanks for the help Igor.

I'm rather new to this level of VM management.

Can you help me understand where the key is?

[Igor Peshansky]

Sure. You should follow the instructions at https://cloud.google.com/compute/docs/storing-retrieving-metadata#updatinginstancemetadata. When you go to the "Custom metadata" section on the "Edit instance" page, you should see the "google-logging-enable" key. Clicking the "X" symbol to the right of the key/value pair will delete it.

        Igor

[aung ye]

[Rishab Kapadia]

Thank you so much brother. Spent days to resolve this. 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdriver-discu...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-discussion+unsub...@googlegroups.com.

[suba mohan]

 google-fluentd.service - LSB: data collector for Treasure Data

   Loaded: loaded (/etc/init.d/google-fluentd; generated)
   Active: failed (Result: exit-code) since Mon 2019-06-24 03:46:28 UTC; 1h 49min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 14415 ExecStart=/etc/init.d/google-fluentd start (code=exited, status=1/FAILURE)

Jun 24 03:46:28 ip-172-31-19-83 google-fluentd[14415]:         from /opt/google-fluentd/embedded/lib/ruby/site_ruby/2.4.0/rubygems/core_ext/kernel_require.rb:54:in `require'
Jun 24 03:46:28 ip-172-31-19-83 google-fluentd[14415]:         from /opt/google-fluentd/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.4.2/bin/fluentd:8:in `<top (required)>'
Jun 24 03:46:28 ip-172-31-19-83 google-fluentd[14415]:         from /opt/google-fluentd/embedded/bin/fluentd:23:in `load'
Jun 24 03:46:28 ip-172-31-19-83 google-fluentd[14415]:         from /opt/google-fluentd/embedded/bin/fluentd:23:in `<top (required)>'
Jun 24 03:46:28 ip-172-31-19-83 google-fluentd[14415]:         from /usr/sbin/google-fluentd:7:in `load'
Jun 24 03:46:28 ip-172-31-19-83 google-fluentd[14415]:         from /usr/sbin/google-fluentd:7:in `<main>'
Jun 24 03:46:28 ip-172-31-19-83 google-fluentd[14415]:  * google-fluentd
Jun 24 03:46:28 ip-172-31-19-83 systemd[1]: google-fluentd.service: Control process exited, code=exited status=1
Jun 24 03:46:28 ip-172-31-19-83 systemd[1]: google-fluentd.service: Failed with result 'exit-code'.

Jun 24 03:46:28 ip-172-31-19-83 systemd[1]: Failed to start LSB: data collector for Treasure Data.

I have this sort of problem .Could you please suggest me the solution

[Igor Peshansky]

Hi. This seems to be a different issue than previously mentioned in this thread — in your case, the fluentd gem won't load properly.

Can you please provide some details on the OS/distro you're using, and whether you're running in GCE or AWS EC2?

Just to set expectations, help offered on this mailing list is best-effort only. If you have a Cloud Support contact, it's best to open a case so that it's properly tracked and followed by our support folks, and so you get dedicated resources for handling your issues.

        Igor
-- sent from a mobile device, please excuse tyops and omissns

--
© 2016 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 

Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.

---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.

To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/87b5a0f9-8410-4dfe-b871-69f7d45342eb%40googlegroups.com.