Need help figuring out where(or how) to find logs

[Midhun Nair]

Hi Community,

I'm running Prometheus in Kubernetes and now I have to forward the Prometheus logs to Splunk. In order to forward I need log file where Prometheus writes its logs. So doing a bit of research I found out that Prometheus only writes to stdout and stderr. So I tried appending the stdout and stderr to a file but could only find level=error logs in the file and not info level. I tried giving the --log-level=info but still appends only error level logs to the file.But doing kubectl logs <prometheus pod name> gives info level logs. Is there any better way to get logs(including info level) for Prometheus? 

[Matthias Rampke]

Kubernetes collects and handles the logs already. Can Splunk read the logs directly from Kubernetes, e.g. via a daemonset?

With the redirect, something may be going wrong with redirecting stdout *and* stderr? I cannot coherently say why, but I found that

command > logfile 2>&1

works in that case – I think of it as "send stdout into the file, then bend stderr into wherever stdout is going". Alternatively, if you want to preserve the Kubernetes logs as well,

command 2>&1 | tee -a logfile

/MR

On Mon, Nov 16, 2020 at 12:09 PM Midhun Nair <midhun....@gmail.com> wrote:

Hi Community,

I'm running Prometheus in Kubernetes and now I have to forward the Prometheus logs to Splunk. In order to forward I need log file where Prometheus writes its logs. So doing a bit of research I found out that Prometheus only writes to stdout and stderr. So I tried appending the stdout and stderr to a file but could only find level=error logs in the file and not info level. I tried giving the --log-level=info but still appends only error level logs to the file.But doing kubectl logs <prometheus pod name> gives info level logs. Is there any better way to get logs(including info level) for Prometheus? 

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/48427e6e-459c-4f4b-8292-0fe319cb4989n%40googlegroups.com.