Feed Envoy Observability Data to New Relic

dane....@gmail.com

unread,

Jun 28, 2018, 2:43:05 PM6/28/18

to envoy-users

Hi Matt/Others,

Here at Ancestry.com we are evaluating Envoy and Nginx for our API Gateway. I have heard from Matt's presentation that Envoy is so powerful in the area of Observability and it should be very fast as it is written in a native language.

I want to prove that to our team. Therefore, I am working on Visualizing the metric data of Envoy's statistic and performance. Thus, could you please help with following:

Is there any way I can feed Envoy Observability Data to New Relic for Visualization?
Amount the Observability Data, is there any metric data indicates how much time Envoy spent on routing the requests to upstream servers? Look like I couldn't seem to find any in the following:

cluster.cms.bind_errors: 0
cluster.cms.external.upstream_rq_200: 32
cluster.cms.external.upstream_rq_2xx: 32
cluster.cms.external.upstream_rq_404: 2
cluster.cms.external.upstream_rq_4xx: 2
cluster.cms.lb_healthy_panic: 0
cluster.cms.lb_local_cluster_not_ok: 0
cluster.cms.lb_recalculate_zone_structures: 0
cluster.cms.lb_subsets_active: 0
cluster.cms.lb_subsets_created: 0
cluster.cms.lb_subsets_fallback: 0
cluster.cms.lb_subsets_removed: 0
cluster.cms.lb_subsets_selected: 0
cluster.cms.lb_zone_cluster_too_small: 0
cluster.cms.lb_zone_no_capacity_left: 0
cluster.cms.lb_zone_number_differs: 0
cluster.cms.lb_zone_routing_all_directly: 0
cluster.cms.lb_zone_routing_cross_zone: 0
cluster.cms.lb_zone_routing_sampled: 0
cluster.cms.max_host_weight: 1
cluster.cms.membership_change: 1
cluster.cms.membership_healthy: 4
cluster.cms.membership_total: 4
cluster.cms.outlier_detection.ejections_active: 0
cluster.cms.outlier_detection.ejections_consecutive_5xx: 0
cluster.cms.outlier_detection.ejections_detected_consecutive_5xx: 0
cluster.cms.outlier_detection.ejections_detected_consecutive_gateway_failure: 0
cluster.cms.outlier_detection.ejections_detected_success_rate: 0
cluster.cms.outlier_detection.ejections_enforced_consecutive_5xx: 0
cluster.cms.outlier_detection.ejections_enforced_consecutive_gateway_failure: 0
cluster.cms.outlier_detection.ejections_enforced_success_rate: 0
cluster.cms.outlier_detection.ejections_enforced_total: 0
cluster.cms.outlier_detection.ejections_overflow: 0
cluster.cms.outlier_detection.ejections_success_rate: 0
cluster.cms.outlier_detection.ejections_total: 0
cluster.cms.retry_or_shadow_abandoned: 0
cluster.cms.update_attempt: 17285
cluster.cms.update_empty: 0
cluster.cms.update_failure: 0
cluster.cms.update_success: 17285
cluster.cms.upstream_cx_active: 0
cluster.cms.upstream_cx_close_notify: 0
cluster.cms.upstream_cx_connect_attempts_exceeded: 0
cluster.cms.upstream_cx_connect_fail: 0
cluster.cms.upstream_cx_connect_timeout: 0
cluster.cms.upstream_cx_destroy: 0
cluster.cms.upstream_cx_destroy_local: 0
cluster.cms.upstream_cx_destroy_local_with_active_rq: 0
cluster.cms.upstream_cx_destroy_remote: 0
cluster.cms.upstream_cx_destroy_remote_with_active_rq: 0
cluster.cms.upstream_cx_destroy_with_active_rq: 0
cluster.cms.upstream_cx_http1_total: 34
cluster.cms.upstream_cx_http2_total: 0
cluster.cms.upstream_cx_idle_timeout: 0
cluster.cms.upstream_cx_max_requests: 0
cluster.cms.upstream_cx_none_healthy: 0
cluster.cms.upstream_cx_overflow: 0
cluster.cms.upstream_cx_protocol_error: 0
cluster.cms.upstream_cx_rx_bytes_buffered: 0
cluster.cms.upstream_cx_rx_bytes_total: 71338
cluster.cms.upstream_cx_total: 34
cluster.cms.upstream_cx_tx_bytes_buffered: 0
cluster.cms.upstream_cx_tx_bytes_total: 30952
cluster.cms.upstream_flow_control_backed_up_total: 0
cluster.cms.upstream_flow_control_drained_total: 0
cluster.cms.upstream_flow_control_paused_reading_total: 0
cluster.cms.upstream_flow_control_resumed_reading_total: 0
cluster.cms.upstream_rq_200: 32
cluster.cms.upstream_rq_2xx: 32
cluster.cms.upstream_rq_404: 2
cluster.cms.upstream_rq_4xx: 2
cluster.cms.upstream_rq_active: 0
cluster.cms.upstream_rq_cancelled: 0
cluster.cms.upstream_rq_maintenance_mode: 0
cluster.cms.upstream_rq_pending_active: 0
cluster.cms.upstream_rq_pending_failure_eject: 0
cluster.cms.upstream_rq_pending_overflow: 0
cluster.cms.upstream_rq_pending_total: 34
cluster.cms.upstream_rq_per_try_timeout: 0
cluster.cms.upstream_rq_retry: 0
cluster.cms.upstream_rq_retry_overflow: 0
cluster.cms.upstream_rq_retry_success: 0
cluster.cms.upstream_rq_rx_reset: 0
cluster.cms.upstream_rq_timeout: 0
cluster.cms.upstream_rq_total: 34
cluster.cms.upstream_rq_tx_reset: 0
cluster.cms.version: 0

Thank in advance,
Dane

Michael P.

unread,

Jun 28, 2018, 2:51:48 PM6/28/18

to envoy-users

I think you will want to leverage statsd statistics vs. the /stats output. New Relic supports - https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/statsd-monitoring-integration. More details are at https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/statistics#arch-overview-statistics

dane....@gmail.com

unread,

Jul 5, 2018, 9:21:51 PM7/5/18

to envoy-users

Thanks for the clue, Michael! Appreciate that!

I've got my statsd server setup and run on a container available on local: 127.0.0.1, port: 8125. I tested using "echo "envoy.test:34|c" | nc -u -w 1 127.0.0.1 8125" and I could see the log coming.

However, when I configure envoy.yaml to enable the statsd based on the above address and port, I didn't see any metrics coming at all. Any ideas?

My envoy.yml:

...

stats_sinks:

- name: envoy.statsd

config:

address:

socket_address:

address: 127.0.0.1

port_value: 8125

protocol: UDP

stats_flush_interval:

seconds: 5

...

Michael Payne

unread,

Jul 6, 2018, 10:32:56 AM7/6/18

to dane....@gmail.com, envoy...@googlegroups.com

My JSON config converted to YAML is:

stats_sinks:
- config:
address:
socket_address:
address: "::1"
port_value: 9125
name: envoy.statsd

So your config looks OK. I assume your statsd server and your Envoy
are on the same machine? What are you using as a statsd server?

> --
> You received this message because you are subscribed to the Google Groups "envoy-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.
> To post to this group, send email to envoy...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/d740acbf-0c44-4abd-89bc-6a1f789d74da%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

Dane Savot

unread,

Jul 6, 2018, 11:48:45 AM7/6/18

to Michael Payne, envoy...@googlegroups.com

Right Michael! There are running on the same machine except they are running on separate containers in different port.

Specifically for the Statsd server, I did expose the port with UDP and I could communicate to the server just fine with echo command from the host machine. I'm using Statsd server to further publish the metrics to new relic infrastructure Agent, and it is working fine too.

My assumption: look like Enovy is not able to publish the metrics to statsd server with the given IP and port and nothing I could do now as I didn't see any error from Enovy log except it says statsd config is loaded.

Any other thoughts?

Reply all

Reply to author

Forward