Prometheus failing after few hours of restart.

[Pooja Chauhan]

Hi ,

For the past 2 days after restart of prometheus ,it stops in few hours.

output of journalctl command is :

-- Logs begin at Wed 2020-03-11 05:15:30 UTC, end at Wed 2020-04-15 15:40:24 UTC. --

Apr 15 14:23:38 ip-172-28-48-25 systemd[1]: prometheus.service: Failed with result 'exit-code'.

Apr 15 14:23:38 ip-172-28-48-25 systemd[1]: prometheus.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:         /app/scrape/scrape.go:854 +0x822

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85e80, 0x14f46b0400, 0x6fc23ac00, 0x0)

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: goroutine 573 [select]:

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:         /app/scrape/scrape.go:336 +0x45d

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:         /app/scrape/scrape.go:774 +0x113

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85d80, 0x14f46b0400, 0x6fc23ac00, 0x0)

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: goroutine 572 [select, 1 minutes]:

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:         /app/scrape/scrape.go:336 +0x45d

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:         /app/scrape/scrape.go:774 +0x113

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85c80, 0x14f46b0400, 0x6fc23ac00, 0x0)

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: goroutine 571 [select, 1 minutes]:

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:         /app/scrape/scrape.go:336 +0x45d

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:         /app/scrape/scrape.go:813 +0x487

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85b80, 0x14f46b0400, 0x6fc23ac00, 0x0)

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:         /app/scrape/scrape.go:471 +0x111

and ubuntu@ip-172-28-48-25:/etc/systemd/system$ sudo service prometheus status

● prometheus.service - Prometheus Server

   Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: enabled)

   Active: failed (Result: exit-code) since Wed 2020-04-15 14:23:38 UTC; 1h 25min ago

     Docs: https://prometheus.io/docs/introduction/overview/

  Process: 6138 ExecStart=/etc/prometheus/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --storage.tsdb.retention=7d (c

 Main PID: 6138 (code=exited, status=2)

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85d80, 0x14f46b0400, 0x6fc23ac00, 0x0)

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:         /app/scrape/scrape.go:774 +0x113

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:         /app/scrape/scrape.go:336 +0x45d

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: goroutine 573 [select]:

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: github.com/prometheus/prometheus/scrape.(*scrapeLoop).run(0xc051b85e80, 0x14f46b0400, 0x6fc23ac00, 0x0)

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]:         /app/scrape/scrape.go:854 +0x822

Apr 15 14:23:38 ip-172-28-48-25 prometheus[6138]: created by github.com/prometheus/prometheus/scrape.(*scrapePool).sync

Apr 15 14:23:38 ip-172-28-48-25 systemd[1]: prometheus.service: Main process exited, code=exited, status=2/INVALIDA

Please suggest how to resolve this

[Ben Kochie]

That looks like an incomplete stack trace. Can you get the full stack trace from your syslog?

There's not enough information to go on here. Please mention what version of Prometheus this is, and include your prometheus.yml.

--
You received this message because you are subscribed to the Google Groups "Prometheus Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to prometheus-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/d6d0c130-0bf6-4fde-b4a2-117fce042f6e%40googlegroups.com.

[Murali Krishna Kanagala]

Looks like a memory issue. How much RAM you have on the server?

https://github.com/prometheus/prometheus/issues/4945

To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmo2vxZpVy2xzgkHpgCU151Q4Jjd4Z_fh17fTE8PrwEwHw%40mail.gmail.com.