federation prometheus can't scrape certain target

[haowells]

Hi, 

I have a federation Prometheus instance to scape multiple targets of child Prometheus instance in different clusters.

I can't scrape some certain metrics from some target, for example, I can not get metric "prometheus.kube-system.svc.target2"  from "target2" even I  have configured it in targets list, but I can scrape the same metric from other targets.

I have checked endpoint like "http://prometheus.kube-system.svc.target2/federate?match[]={__name__=~"cap:remain_cpu_cores:dedicated"}" , it can return result.

  scrape_configs:

    - job_name: 'federate'

      scheme: http

      metrics_path: '/federate'

      params:

        'match[]':

          - '{k8s_app="kube-state-metrics"}'

          - '{app="alertmanager"}'

          - '{__name__=~"ALERTS"}'

          - '{__name__=~"cap:remain_cpu_cores:dedicated"}'

          - '{__name__=~"cap:remain_memory_gb:dedicated"}'

          - '{__name__=~"kubelet_volume_stats_used_bytes"}'

          - '{job="prometheus-io-auto-scrape"}'

      static_configs:

        - targets:

          - prometheus.kube-system.svc.target1

          - prometheus.kube-system.svc.target2

          - prometheus.kube-system.svc.target3

          - prometheus.kube-system.svc.target4

want to know why the same metric can't be collected in a certain configured target. anyone have the idea to debug this issue?