Promtail - fatal error: runtime: out of memory

[Dilshan Nanayakkara]

Hi,

I'm trying to start the Promtail service but every time it failed with "fatal error: runtime: out of memory"
Herewith attached the same error information
promtail-error-info.txt

server -: AWS linux ec2

Memory information
[root@ip-]# free -m
total used free shared buff/cache available
Mem: 3884 885 2756 0 242 2783
Swap: 0 0 0

[root@ip-]# ./promtail --version
promtail, version 1.6.1 (branch: HEAD, revision: 623858d)
build user: root@dd6ba3f86fac
build date: 2020-08-24T19:10:42Z
go version: go1.14.2

Promtail config file

server:
http_listen_port: 9080
grpc_listen_port: 0

positions:
filename: /tmp/positions.yaml

clients:

• url: http://x.x.x.x:3100/loki/api/v1/push

scrape_configs:

• job_name: Test-Job
  pipeline_stages:
  static_configs:
  • targets:
    • localhost
      labels:
      job: Test-Job-Log
      host: x.x.x.x
      path: /var/log/app/application.log

Can you please help to sort out this issue.

--
Thank you
Dilshan

[Danny Kopping]

Hi Dilshan

From the information you've provided it seems that Promtail should run just fine within the 800MB free RAM, although this is not sufficient info to debug this issue.

Your server is quite low on memory though and you have no swap configured.

Please run `/usr/bin/time -v <your promtail command>` and send the output.

[Dilshan Nanayakkara]

Hi Danny,

Please find the promtail output file.

--

Thank you

Dilshan Nanayakkara

[Danny Kopping]

Thank you Dilshan.

Do you have Prometheus setup on that server or some other monitoring system to show how much RAM is free over time?
Also, how large are the files that Promtail is attempting to read?

[Danny Kopping]

User responded privately again. For posterity, no server monitoring is in place and the file size is 77MB.

Dilshan,

I was asking if you have anything that tracks memory on an ongoing basis, since a point-in-time snapshot of how much memory you're using is not enough to diagnose this issue.

In the absense of server monitoring, a very low-tech thing you could try is running this in another window before you start up Promtail:
while true; do date >> /tmp/memory; free -k >> /tmp/memory; sleep 0.2; done

When Promtail crashes, run tail -n200 /tmp/memory and send the results.

In any case, I don't think this is an issue relating to Promtail; try setting up some swap space on your server, or increasing the memory.