Can 2 prometheus agent scraping same targets have difference in prometheus_agent_active_series metric

[Vikas Budhvant]

We have 2 replicas of prometheus agent setup in kubernetes cluster.

We noticed that both these prometheus agents are not scraping same number of metrics. this we verified using prometheus_agent_active_series metrics.

However when checked for prometheus agent targets its exactly same for both agents. the difference in prometheus_agent_active_series is consistent (attached is prometheus_agent_active_series graph)

We were expecting prometheus_agent_active_series should match for both agents. not sure if the prometheus_agent_active_series is not reliable or something else.

please let me know what else can I check to drill this down?

[Brian Candler]

That looks fine to me. The "head block" contains all timeseries which were active in the last two hours; if you have some series churn (i.e. new series being created, old ones which stop) then the head block will grow as new series are added, but you'll only see the old ones drop out when it makes a new block.  You can clearly see those dips in the green graph; I'd expect the same in the yellow one at 12:45 and 16:45 but they're just out of frame.

Since the two prometheus instances don't change head block at the same time, the number of timeseries in each are different.

- green swapped at about 14:08, then starts to grow

- yellow swapped at about 14:45, dropping to the level where green was at 14:08, and then starts to grow from there