Prometheus remote write is not sending all metrics to remote server
Egzon Brajshori
I'm trying to setup a remote write from a Prometheus Server inside AWS EKS to Amazon Managed Prometheus.
I've setup the remote write like this :
serviceAccounts:
server:
name: amp-iamproxy-ingest-service-account
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::accountID:role/amp-iamproxy-ingest-role server:
remoteWrite: -
url: https://aps-workspaces.region.amazonaws.com/workspaces/ws-id/api/v1/remote_write
sigv4:
region: $region
queue_config:
max_samples_per_send: 1500
max_shards: 200
capacity: 6000
And I can confirm by checking the Prometheus server logs on EKS that it connects just fine, there are no errors connected to the remote write operation. When I try to check the data on the remote prometheus server(Amazon Managed Prometheus), I am not getting every metric that is being scraped from the local Prometheus server. e.g.
The container_cpu_usage_seconds_total metric, when I query it on the local prometheus server, I get back results with scraped data just fine. When I do the same on Amazon Managed Prometheus, I get no data at all, it's blank
But when I query data for the kube_pod_container_status_running, I get scraped data back from both prometheus servers, the local one and the one that is set as the remote write destination(Amazon Managed Prometheus).
Has anyone had any issue like this before where, Prometheus only remote writes some metrics to the destination Prometheus server?