Some timeseries getting dropped

[Venkata Bhagavatula]

Hi All,

I am seeing some time series from the target are getting dropped and not  written to tsdb. Initially i thought it might be related to wrong metric relabel configs, but i have removed all the relabel configs, even then the time series is not being written.

Target is giving the time series. 

Following the is curl output from the target:

istio_requests_total{response_code="200",reporter="destination",source_workload="ind-cpro",source_workload_namespace="csd-ns",source_principal="unknown",source_app="cpro",source_version="unknown",destination_workload="ind-nrfp",destination_workload_namespace="csd-ns",destination_principal="unknown",destination_app="nrfp",destination_version="unknown",destination_service="csd-nnrfp.csd-ns.svc.cluster.local",destination_service_name="csd-nnrfp",destination_service_namespace="csd-ns",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="none",source_canonical_service="cpro",destination_canonical_service="nrfp",source_canonical_revision="2.15.1",destination_canonical_revision="latest"} 3017
istio_requests_total{response_code="200",reporter="source",source_workload="ind-nrfp",source_workload_namespace="csd-ns",source_principal="unknown",source_app="nrfp",source_version="unknown",destination_workload="unknown",destination_workload_namespace="unknown",destination_principal="unknown",destination_app="unknown",destination_version="unknown",destination_service=" abc.default.svc.cluster.local",destination_service_name="abc.default.svc.cluster.local",destination_service_namespace="unknown",request_protocol="grpc",response_flags="-",grpc_response_status="0",connection_security_policy="unknown",source_canonical_service="nrfp",destination_canonical_service="unknown",source_canonical_revision="latest",destination_canonical_revision="latest"} 276

Prometheus job configuration is as follows(ip address is masked):

    - honor_labels: true
      job_name: chal
      metrics_path: /stats/prometheus
      static_configs:
        - targets:
          - abc.xyz.def.pqr:15090

In the above time series from target, the time series with source_workload="ind-nrfp" is not present in prometheus.  There are no warnings/errors seen in the debug logs 

Can you let me know how to debug this issue?

Thanks n Regards,

Chalapathi.

[Bjoern Rabenstein]

On 16.08.21 11:49, Venkata Bhagavatula wrote:
>
> Following the is curl output from the target:
> istio_requests_total{response_code="200",reporter="destination",

> *source_workload="ind-cpro"*,source_workload_namespace="csd-ns",source_principal="unknown",source_app="cpro",source_version="unknown",destination_workload="ind-nrfp",destination_workload_namespace="csd-ns",destination_principal="unknown",destination_app="nrfp",destination_version="unknown",destination_service="csd-nnrfp.csd-ns.svc.cluster.local",destination_service_name="csd-nnrfp",destination_service_namespace="csd-ns",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="none",source_canonical_service="cpro",destination_canonical_service="nrfp",source_canonical_revision="2.15.1",destination_canonical_revision="latest"}

> 3017
> istio_requests_total{response_code="200",reporter="source",

> *source_workload="ind-nrfp"*,source_workload_namespace="csd-ns",source_principal="unknown",source_app="nrfp",source_version="unknown",destination_workload="unknown",destination_workload_namespace="unknown",destination_principal="unknown",destination_app="unknown",destination_version="unknown",destination_service="

> abc.default.svc.cluster.local",destination_service_name="abc.default.svc.cluster.local",destination_service_namespace="unknown",request_protocol="grpc",response_flags="-",grpc_response_status="0",connection_security_policy="unknown",source_canonical_service="nrfp",destination_canonical_service="unknown",source_canonical_revision="latest",destination_canonical_revision="latest"}
> 276

> [...]

> In the above time series from target, the time series with

> *source_workload="ind-nrfp"
> *is not present in prometheus. There are no warnings/errors seen in the
> debug logs

Wild guess: Perhaps your 2nd line doesn't end with a '\n' character?

(This should never happen if your exposition happens via the usual
instrumentation libraries, but perhaps in your case, you have some
custom processing? Only having two lines in the output makes me
suspect that.)

--
Björn Rabenstein
[PGP-ID] 0x851C3DA17D748D03
[email] bjo...@rabenste.in

[Bjørn Rustad]

2 years have passed, did you manage to resolve this Chalapathi?

I'm curious because I seem to have the exact same problem, with the interaction between the istio proxy metrics and a Prometheus operator installation. I'm seeing the metrics when curl'ing the stats/prometheus endpoints, but they are not collected by prometheus.