K8S Envoy Edge proxy - upstream_reset_before_response_started

[Michal Basl]

hi there...

I'm creating envoy edge proxy at GCP K8S cluster and I'm getting "upstream_reset_before_response_started{local_reset}" status with 503 HTTP code. Seems CPU & Memory is ok and I have no idea what could cause that...

K8S Config:

image: docker.io/envoyproxy/envoy-alpine:v1.17.1

resources: CPU 750m, Memory: 512Mi

Envoy Config:

overload_manager:
refresh_interval: 0.25s
resource_monitors:
- name: envoy.resource_monitors.fixed_heap
typed_config:
"@type": type.googleapis.com/envoy.config.resource_monitor.fixed_heap.v2alpha.FixedHeapConfig
max_heap_size_bytes: 402653184 # 384MiB
actions:
- name: envoy.overload_actions.shrink_heap
triggers:
- name: envoy.resource_monitors.fixed_heap
threshold:
value: 0.95
- name: envoy.overload_actions.stop_accepting_requests
triggers:
- name: envoy.resource_monitors.fixed_heap
threshold:
value: 0.98

admin:
access_log_path: /var/log/admaccess.log
address:
socket_address:
protocol: TCP
address: 0.0.0.0
port_value: 9901

static_resources:
listeners:
- address:
socket_address:
address: 0.0.0.0
port_value: 10080
per_connection_buffer_limit_bytes: 65536 # 64KiB
filter_chains:
filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: connection_manager
use_remote_address: true
common_http_protocol_options:
idle_timeout: 3600s
headers_with_underscores_action: REJECT_REQUEST
stream_idle_timeout: 300s
request_timeout: 300s
codec_type: AUTO
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /var/log/gwaccesslog.log
log_format:
omit_empty_values: false
json_format:
# https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage#config-access-log-format-dictionaries
startTime: "%START_TIME%"
requestMethod: "%REQ(:METHOD)%"
requestPath: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
fullPath: "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%"
protocol: "%PROTOCOL%"
responseCode: "%RESPONSE_CODE%"
responseCodeDetails: "%RESPONSE_CODE_DETAILS%"
duration: "%DURATION%"
upstreamServiceTime: "%RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%"
bytesSent: "%BYTES_SENT%"
bytesReceived: "%BYTES_RECEIVED%"
responseFlags: "%RESPONSE_FLAGS%"
correlationId: "%REQ(Correlation-Id?:correlation-id)%"
soapaction: "%REQ(soapaction)%"
route_config:
name: local_route
virtual_hosts:
- name: service
domains: [ "*" ]
routes:
- match: { prefix: "/" }
route:
cluster: service-https
host_rewrite_literal: some.uri
timeout: 4s
idle_timeout: 15s
http_filters:
- name: envoy.filters.http.local_ratelimit
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
stat_prefix: rate_limiter
token_bucket:
max_tokens: 150
tokens_per_fill: 150
fill_interval: 1s
filter_enabled:
runtime_key: local_rate_limit_enabled
default_value:
numerator: 100
denominator: HUNDRED
filter_enforced:
runtime_key: local_rate_limit_enforced
default_value:
numerator: 100
denominator: HUNDRED
response_headers_to_add:
- append: false
header:
key: x-local-rate-limit
value: 'true'
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

clusters:
- name: service-https
connect_timeout: 4s
per_connection_buffer_limit_bytes: 65536 # 64KiB
type: STRICT_DNS
lb_policy: ROUND_ROBIN
dns_refresh_rate: 300s
dns_lookup_family: V4_ONLY
load_assignment:
cluster_name: service-https
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: some.uri
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext

layered_runtime:
layers:
- name: static_layer
static_layer:
envoy:
resource_limits:
listener:
example_listener_name:
connection_limit: 10000
overload:
global_downstream_max_connections: 50000

Do you have an idea what could cause this issue, pls?

[Zizon Qiu]

rejected by containing underscore header?

--
You received this message because you are subscribed to the Google Groups "envoy-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-dev+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-dev/fdeb9604-cfe1-4dc4-bee8-7ba16ccf9069n%40googlegroups.com.

[Nguyen Anh Tu]

I got something similar, slight difference in detail code. Michael have you figured out the root cause ? Thanks 

```

Sending local reply with details upstream_reset_before_response_started{connection termination}

```