Opentelemetry GRPC connection fails in v2.6.3

[Natalio Gatti]

Hi. I have krakend running and reporting OTLP metrics to a datadog agent. 

It worked correctly in version 2.6.1. I upgraded to v2.6.3 and now krakend seems unable to connect to agent via GRPC anymore with the following error:

{"@timestamp":"2024-06-18T02:34:56.148183+00:00","@version":1,"host":"myapp-krakend-78b674f994-8pkjf","level":"ERROR","message":"[SERVICE: OpenTelemetry] failed to upload metrics: context deadline exceeded: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 10.0.120.65:4317: connect: connection refused\"","module":"[KRAKEND]"}

I tried to connect from krakend pod (it's running in k8s) manually to 10.0.120.65:4317  via nc and connection is established correctly. 

Should I report this as an issue on github?

Thanks,

Natalio.

[Natalio Gatti]

btw, this is the otlp configuration:

 "extra_config": {
    "telemetry/logging": {
      "level": "DEBUG",
      "syslog": false,
      "stdout": true,
      "format": "logstash",
      "prefix": "[KRAKEND]"
    },
    "telemetry/logstash":{
      "enabled": true
    },
    "telemetry/opentelemetry": {
      "service_name": "myapp-krakend",
      "metric_reporting_period": 1,
      "@comment": "Report 30% of traces",
      "trace_sample_rate": 0.3,
      "exporters": {
        "otlp": [
              {
                  "name": "k8s_datadog_agent",
                  "host": "$DD_AGENT_HOST",
                  "port": 4317,
                  "use_http": false
              }
          ]
      }
    },

[Albert Lombarte]

Hello Natalio,

Make sure you have not changed something else other than upgrading the container that is preventing the KrakenD container reach the datadog agent. Here there is a fully reproducible example using v2.6.3 that you can test locally and confirm the configuration/software is not the problem:

#docker-compose.yml
version: '3'
services:
  krakend:
    image: devopsfaith/krakend:2.6.3
    volumes:
      - "./:/etc/krakend"
    command: ["run", "-c", "krakend.json"]
    ports:
      - "8080:8080"
  datadog:
    image: gcr.io/datadoghq/agent:7
    pid: host
    environment:
     - DD_API_KEY=WRITE-YOUR-KEY-HERE
     - DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT=0.0.0.0:4317
     - DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT=0.0.0.0:4318
     - DD_SITE=datadoghq.com
    volumes:
     - /var/run/docker.sock:/var/run/docker.sock
     - /proc/:/host/proc/:ro
     - /sys/fs/cgroup:/host/sys/fs/cgroup:r

A simple KrakenD config:

{
    "version": 3,
    "$schema": "https://www.krakend.io/schema/krakend.json",
    "host": [
        "http://localhost:8080"
    ],
    "debug_endpoint": true,
    "echo_endpoint": true,
    "extra_config": {
        "telemetry/opentelemetry": {
            "exporters": {
                "otlp": [
                    {
                        "use_http": false,
                        "port": 4317,
                        "host": "datadog",
                        "name": "dd_agent",
                        "disable_metrics": false,
                        "disable_traces": false
                    }
                ]
            },
            "trace_sample_rate": 1,
            "service_name": "krakend_dd_telemetry",
            "metric_reporting_period": 1
        }
    },
    "endpoints": [
        {
            "endpoint": "/github/{user}",
            "backend": [
                {
                    "host": ["https://api.github.com"],
                    "url_pattern": "/users/{user}"
                }
            ]
        }
    ]
}

El dia dimarts, 18 de juny del 2024 a les 21:15:10 UTC+2, nat...@tmm.ventures va escriure:

[Natalio Gatti]

Hi Albert, thanks for your information. As a test, I tried downgrading to v2.6.1 and connection to datadog agent started to work again. No other changes have been made besides the version. 

Your sample configs are similar to mine. I will try v2.6.2 tomorrow.

[Natalio Gatti]

Version 2.6.2 also works ok. 

[Albert Lombarte]

Thanks for sharing the outcome of your test Natalio,

Can you share your full setup, writing an XXXXX in any sensitive places of the configuration to try to reproduce it?

This is the full list of changes between KrakenD 2.6.2 and 2.6.3: https://github.com/krakend/krakend-otel/compare/v0.4.0...v0.5.0

El dia dijous, 20 de juny del 2024 a les 12:44:37 UTC+2, nat...@tmm.ventures va escriure:

[Natalio Gatti]

This is my service.json file:

{
  "port": 8080,
  "default_host": "client",
  "timeout": "30s",
  "cache_ttl": "60s",
  "output_encoding": "no-op",
  "debug_endpoint": false,
  "echo_endpoint": false,
  "apiVersions": ["v1", "v2"],
  "hosts": {
    "app1": "http://service-myproject-app1:8080",
    "app2": "http://service-myproject-app2:8080",
    "app3": "http://service-myproject-app3:8080"
  },

  "extra_config": {
    "telemetry/logging": {
      "level": "DEBUG",
      "syslog": false,
      "stdout": true,
      "format": "logstash",
      "prefix": "[KRAKEND]"
    },
    "telemetry/logstash":{
      "enabled": true
    },
    "telemetry/opentelemetry": {

      "service_name": "myproject-krakend",

      "metric_reporting_period": 1,
      "@comment": "Report 30% of traces",
      "trace_sample_rate": 0.3,
      "exporters": {
        "otlp": [
              {
                  "name": "k8s_datadog_agent",
                  "host": "$DD_AGENT_HOST",
                  "port": 4317,
                  "use_http": false
              }
          ]
      }
    },

    "plugin/http-server":{
      "name":["myproject-static"],
      "myproject-static":{
        "path": "./static/",
        "map": {
          "home": {"path": "/", "file": "index.html"},
          "favicon": {"path": "/favicon.ico", "file": "assets/favicon.ico"}
        }
      }
    },
    "router": {
      "hide_version_header": true,
      "return_error_msg": true,
      "auto_options": true,
      "logger_skip_paths":[
        "/__health"
      ]
    }
  }
}

Is there anyway to enable debug on otel connection? Maybe it's connecting but trying to send data that is rejected by datadog.

[David Hontecillas]

Hello, 

I've been testing this, and I could not reproduce the issue. I am actually sending the metrics as expected. Perhaps you could check the logs of the agent. How do you run the agent ? I've been using it this way:

docker run --name dd-agent --rm \
    -e DD_SITE="datadoghq.com" \
    -e DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_GRPC_ENDPOINT="0.0.0.0:4317" \
    -e DD_OTLP_CONFIG_RECEIVER_PROTOCOLS_HTTP_ENDPOINT="0.0.0.0:4318" \
    -e DD_API_KEY=8c9a43a9157cd2715e920158f2f74764 \
    -e DD_APM_ENABLED=true \
    -p 14317:4317 \
    -p 14318:4318 \
    -v /var/run/docker.sock:/var/run/docker.sock:ro \
    -v /proc/:/host/proc/:ro \
    -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
    -v /var/lib/docker/containers:/var/lib/docker/containers:ro \
    gcr.io/datadoghq/agent:7

In my case, since I run it that way, I see the "logs" directly in the console. But as I told you, I could not reproduce the issue (using devopsfaith/krakend:2.6.3-watch docker image in the community playgorund). 

Also, reviewing the changes from 2.6.2, to 2.6.3 I could not see anything that could be making an impact in how metrics / traces are reported. 

I don't know, you might want to leave the `metric_reporting_period` and the `trace_sample_rate` to their default values ?  I really don't know what could be the issue.