Safe to install both monitoring agents on instance? How?

[Frank Shimizu]

Hello,

According to the documentation there are currently two agents one can install on a Linux instance to collect metrics and logs:

* Ops Agent

* Legacy Monitoring agent

We want to collect custom logs from files from our own applications on the instances. It seems that the Ops Agent does not support this, so we installed and configured the Legacy Monitoring agent to do that.

However, it seems that the Legacy agent does not collect all system metrics. For example, we can't find Disk/Volume usage in the metrics. After some research it seems that the Ops Agent does collect those.

Now, one idea is to install both agents and configure the Legacy agent to collect our custom logs. Then disable log collection in the Ops Agent to have it only collect metrics. However, the Ops Agent documentation suggests that this is not a safe thing to do:

> Ensure your VM doesn't have the legacy Cloud Logging agent or Cloud Monitoring agent installed on it. This can cause ingestion of duplicate logs or a conflict in metrics ingestion. In addition, this agent uses new configuration files that are not compatible with the old agents.

https://cloud.google.com/monitoring/agent/ops-agent/installation

For our use case, would it be safe to install both agents on the same instances if we disabled log collection in the Ops Agent?

If this is not possible, how can we achieve our goal, i.e. collection of all metrics and also collection of non-standard logs in files from our own applications?

Regards

Frank

[Igor Peshansky]

Hi, Frank,

Before we proceed, a standard disclaimer: this list provides answers on a best-effort basis — for real support, please open a Cloud Support case if you're in a position to do so.

Running the Ops agent alongside the standalone monitoring and/or logging agent is currently not supported.

However, the Ops agent does support ingesting certain kinds of custom logs. Can you please clarify what form your custom logs take? It seems that they're written as files on-disk. What format are the logs being written in? Are they structured or plaintext? If the latter, and they need to be parsed, would a regular expression parser suffice? Any additional information you can provide here can help us make a better recommendation.

I'm also a bit confused by your statement that the standalone monitoring agent does not collect disk usage metrics — can you please elaborate on that and give some examples? The set of disk usage metrics collected by both agents should be identical.

        Igor

--
© 2021 Google Inc. 1600 Amphitheatre Parkway, Mountain View, CA 94043
 
Email preferences: You received this email because you signed up for the Google Stackdriver Discussion Google Group (google-stackdr...@googlegroups.com) to participate in discussions with other members of the GoogleStackdriver community.
---
You received this message because you are subscribed to the Google Groups "Google Stackdriver Discussion Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-stackdriver-d...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/a12ed732-24e0-4cf4-bd3a-df724a203895n%40googlegroups.com.

[Frank Shimizu]

Hi Igor,

Thank you for taking the time and for your elaborate reply.

I should have been more precise about the logs. Our logs are plain text files. Unfortunately they are very unstructured and a mix of output of different utilities and the like. Because of that it doesn't make sense to try and create a parser for that and reworking the logs is also not economically feasible. It would be enough for us to simply ingest them line by line and push them to Google Cloud Logging. This is why we went to the Legacy agent for that, because it can do that with minimal setup.

Also apologies if I came to the wrong conclusion about the Legacy agent not collecting disk usage metrics. To be more precise, when I look through the Metrics Explorer, I do see various metrics for disk read/write operations and such. What I'm missing is disk space usage. We would like to monitor and alert if a file system or disk is about to become full, but with the Legacy agent installed, I can't seem to find a metric for that. Could you please give me a hint, is there a disk space usage metric collected by the Legacy agent?

Regards

Frank

[Igor Peshansky]

The Ops agent has the "files" receiver type, which allows scraping arbitrary on-disk files. Configuring such receivers is fairly straightforward, with some examples in the docs [1]. You can even set up multiple receiver pipelines with different parsers, if you can enumerate (e.g., with a glob) log files with specific formats.

The standalone monitoring agent includes a number of disk metrics [2]. You are very likely looking for agent.googleapis.com/disk/bytes_used and/or agent.googleapis.com/disk/percent_used — is there a reason why those wouldn't work for you? In general, it may be easier to look through the organized lists of metrics in our documentation, rather than trying to find what you need in Metrics Explorer.

Hope this helps,

        Igor

[1] https://cloud.google.com/stackdriver/docs/solutions/agents/ops-agent/configuration#logging-receiver-examples

[2] https://cloud.google.com/monitoring/api/metrics_agent#agent-disk

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/94a2c2d0-3c2b-4c55-9eaf-96953060eb84n%40googlegroups.com.

[Frank Shimizu]

Hi Igor,

Thanks for your pointers. Indeed, it seems that I misunderstood the documentation. The "files" receiver type is exactly what I need. I thought a processor is also mandatory, but it's not.

With the Ops agent we now receive the logs we need and the disk usage metrics you mentioned are now also coming in. Again, thanks a lot for your help.

Regards

Frank

[Igor Peshansky]

Glad to hear it worked for you!

        Igor

To view this discussion on the web visit https://groups.google.com/d/msgid/google-stackdriver-discussion/0340f5ab-5cbe-496d-bb7a-7c58958a7052n%40googlegroups.com.