Does prometheus instances need to be the same version for federation?

[Cvijan Uros]

Hi all, 

Have a question, we have several really old prometheus instances, version 2.2 and want to do some upgrades.. We also have one central prometheus instance in which federation is set up to these two instances.. 

I first upgraded this central instance to 2.19 and now federation metrics are not present. It says 404 server not found, but from the server where my new prometheus instance is installed i can run CURL to my federated instance and i get the metrics. 

So i can run:

 curl http://federated_instance/federate?match%5B%5D=%7Bjob%21%3D%22%22%7D

ANd I get the output, but from prometheus itself I get 404. 

Setup for this federation is pretty simple:

- job_name: federate

  honor_labels: true

  honor_timestamps: true

  params:

    match[]:

    - '{job!=""}'

  scrape_interval: 15s

  scrape_timeout: 10s

  metrics_path: /federate

  scheme: http

  static_configs:

  - targets:

    - federated_instance1

    - federated_instance2

Federation works with the prometheus version which is the same as federated instances. So current stuff works when all is 2.2, but not for some reason when I moved central prometheus to 2.19. 

Do I need to have all the versions to be the same or nearly the same in versions? 

Regards, 

Uros

[Cvijan Uros]

NVM, its problem with the other setup that is infront of those federated instances.