Envoy can't connect to local upstream server

[Chris Odd]

Hi,

I'm trying to set up my development environment so that I can easily use my Eclipse based debugger to step through/observe a running gRPC Server.

The infrastructure is running entirely on my local OSX machine:

Local REACT gRPC client -> Envoy running inside Docker, listening on Port 8000 -> Java based gRPC Server running in Eclipse, listening on Port 8069.

I have been unable to successfully configure my Envoy/Docker session so that it connects to local port 8069.

I can see from the Envoy console that the request from the gRPC Client arrives, but when Envoy tries to connect to local port 8069, it cannot.

Running curl -v 127.0.0.1:8069 from an OSX terminal session is able to connect to the gRPC Server just fine (though of course it throws an Exception as the format of the request is HTTP/1.x and it's expecting HTTP/2)

When envoy tries to connect to the gRPC Server, the gRPC Server does not get the connection request at all (ie. Envoy appears not to be even attempting to connect to it).

I've included below: my Docker launch command, the envoy.demo.yaml file, and the envoy console.   If there's anything else I should provide, please let me know.

Hopefully someone can point out what's wrong with my configuration.   Thanks in advance!

LAUNCHING THE DOCKER INSTANCE:

==============================

docker run -p 8000:8000 -p 8081:8081 -v $(pwd)/envoy.yaml:/etc/envoy/envoy.demo.yaml --rm envoyproxy/envoy-dev -c /etc/envoy/envoy.demo.yaml --log-level debug

ENVOY.DEMO.YAML:

================

admin:

  access_log_path: /tmp/admin_access.log

  address:

    socket_address: { address: 0.0.0.0, port_value: 8081 }

static_resources:

  listeners:

  - name: listener_0

    address:

      socket_address: { address: 0.0.0.0, port_value: 8000 }

    filter_chains:

    - filters:

      - name: envoy.filters.network.http_connection_manager

        typed_config:

          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager

          codec_type: auto

          stat_prefix: ingress_http

          route_config:

            name: local_route

            virtual_hosts:

            - name: local_service

              domains: ["*"]

              routes:

              - match: { prefix: "/" }

                route:

                  cluster: gateway

                  timeout: 0s

                  max_stream_duration:

                    grpc_timeout_header_max: 0s

              cors:

                allow_origin_string_match:

                - prefix: "*"

                allow_methods: GET, PUT, DELETE, POST, OPTIONS

                allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout

                max_age: "1728000"

                expose_headers: custom-header-1,grpc-status,grpc-message

          http_filters:

          - name: envoy.filters.http.grpc_web

          - name: envoy.filters.http.cors

          - name: envoy.filters.http.router

  clusters:

  - name: gateway

    connect_timeout: 0.25s

    type: static

    http2_protocol_options: {}

    lb_policy: round_robin

    load_assignment:

      cluster_name: gateway

      endpoints:

        - lb_endpoints:

            - endpoint:

                address:

                  socket_address:

                    address: host.docker.internal

                    port_value: 8069

ENVOY CONSOLE SHOWING INBOUND REQUEST AND FAILED CONNECTION TO UPSTREAM SERVER

==============================================================================

[2021-05-17 18:07:24.189][18][debug][http] [source/common/http/conn_manager_impl.cc:261] [C0] new stream

[2021-05-17 18:07:24.189][18][debug][http] [source/common/http/conn_manager_impl.cc:882] [C0][S5067595768646632115] request headers complete (end_stream=true):

':authority', '127.0.0.1:8000'

':path', '/protoPAD.Login/StartSession'

':method', 'OPTIONS'

'connection', 'keep-alive'

'accept', '*/*'

'access-control-request-method', 'POST'

'access-control-request-headers', 'content-type,x-grpc-web'

'origin', 'http://localhost:3000'

'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'

'sec-fetch-mode', 'cors'

'sec-fetch-site', 'cross-site'

'sec-fetch-dest', 'empty'

'referer', 'http://localhost:3000/'

'accept-encoding', 'gzip, deflate, br'

'accept-language', 'en-US,en;q=0.9,fr;q=0.8'

[2021-05-17 18:07:24.190][18][debug][http] [source/common/http/filter_manager.cc:781] [C0][S5067595768646632115] request end stream

[2021-05-17 18:07:24.192][18][debug][http] [source/common/http/conn_manager_impl.cc:1453] [C0][S5067595768646632115] encoding headers via codec (end_stream=true):

':status', '200'

'access-control-allow-origin', 'http://localhost:3000'

'access-control-allow-methods', 'GET, PUT, DELETE, POST, OPTIONS'

'access-control-allow-headers', 'keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout'

'access-control-max-age', '1728000'

'access-control-expose-headers', 'custom-header-1,grpc-status,grpc-message'

'date', 'Mon, 17 May 2021 18:07:24 GMT'

'server', 'envoy'

[2021-05-17 18:07:24.199][18][debug][http] [source/common/http/conn_manager_impl.cc:261] [C0] new stream

[2021-05-17 18:07:24.199][18][debug][http] [source/common/http/conn_manager_impl.cc:882] [C0][S7721288019291608610] request headers complete (end_stream=false):

':authority', '127.0.0.1:8000'

':path', '/protoPAD.Login/StartSession'

':method', 'POST'

'connection', 'keep-alive'

'content-length', '5'

'sec-ch-ua', '" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"'

'x-grpc-web', '1'

'sec-ch-ua-mobile', '?0'

'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'

'content-type', 'application/grpc-web+proto'

'accept', '*/*'

'origin', 'http://localhost:3000'

'sec-fetch-site', 'cross-site'

'sec-fetch-mode', 'cors'

'sec-fetch-dest', 'empty'

'referer', 'http://localhost:3000/'

'accept-encoding', 'gzip, deflate, br'

'accept-language', 'en-US,en;q=0.9,fr;q=0.8'

[2021-05-17 18:07:24.199][18][debug][router] [source/common/router/router.cc:445] [C0][S7721288019291608610] cluster 'gateway' match for URL '/protoPAD.Login/StartSession'

[2021-05-17 18:07:24.200][18][debug][router] [source/common/router/router.cc:631] [C0][S7721288019291608610] router decoding headers:

':authority', '127.0.0.1:8000'

':path', '/protoPAD.Login/StartSession'

':method', 'POST'

':scheme', 'http'

'sec-ch-ua', '" Not A;Brand";v="99", "Chromium";v="90", "Google Chrome";v="90"'

'x-grpc-web', '1'

'sec-ch-ua-mobile', '?0'

'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'

'content-type', 'application/grpc'

'accept', '*/*'

'origin', 'http://localhost:3000'

'sec-fetch-site', 'cross-site'

'sec-fetch-mode', 'cors'

'sec-fetch-dest', 'empty'

'referer', 'http://localhost:3000/'

'accept-encoding', 'gzip, deflate, br'

'accept-language', 'en-US,en;q=0.9,fr;q=0.8'

'x-forwarded-proto', 'http'

'x-request-id', 'e7edc90f-cd9c-425c-8ff6-b116f9241d66'

'te', 'trailers'

'grpc-accept-encoding', 'identity'

[2021-05-17 18:07:24.200][18][debug][pool] [source/common/http/conn_pool_base.cc:74] queueing stream due to no available connections

[2021-05-17 18:07:24.200][18][debug][pool] [source/common/conn_pool/conn_pool_base.cc:241] trying to create new connection

[2021-05-17 18:07:24.200][18][debug][pool] [source/common/conn_pool/conn_pool_base.cc:143] creating a new connection

[2021-05-17 18:07:24.200][18][debug][http2] [source/common/http/http2/codec_impl.cc:1222] [C1] updating connection-level initial window size to 268435456

[2021-05-17 18:07:24.201][18][debug][client] [source/common/http/codec_client.cc:60] [C1] connecting

[2021-05-17 18:07:24.201][18][debug][connection] [source/common/network/connection_impl.cc:860] [C1] connecting to 127.0.0.1:8069

[2021-05-17 18:07:24.201][18][debug][connection] [source/common/network/connection_impl.cc:876] [C1] connection in progress

[2021-05-17 18:07:24.201][18][debug][http] [source/common/http/filter_manager.cc:781] [C0][S7721288019291608610] request end stream

[2021-05-17 18:07:24.201][18][debug][connection] [source/common/network/connection_impl.cc:674] [C1] delayed connection error: 111

[2021-05-17 18:07:24.201][18][debug][connection] [source/common/network/connection_impl.cc:243] [C1] closing socket: 0

[2021-05-17 18:07:24.201][18][debug][client] [source/common/http/codec_client.cc:107] [C1] disconnect. resetting 0 pending requests

[2021-05-17 18:07:24.201][18][debug][pool] [source/common/conn_pool/conn_pool_base.cc:393] [C1] client disconnected, failure reason: 

[2021-05-17 18:07:24.201][18][debug][router] [source/common/router/router.cc:1076] [C0][S7721288019291608610] upstream reset: reset reason: connection failure, transport failure reason: 

[2021-05-17 18:07:24.202][18][debug][http] [source/common/http/filter_manager.cc:883] [C0][S7721288019291608610] Sending local reply with details upstream_reset_before_response_started{connection failure}

[2021-05-17 18:07:24.202][18][debug][http] [source/common/http/conn_manager_impl.cc:1453] [C0][S7721288019291608610] encoding headers via codec (end_stream=false):

':status', '503'

'content-length', '0'

'content-type', 'application/grpc-web+proto'

'access-control-allow-origin', 'http://localhost:3000'

'access-control-expose-headers', 'custom-header-1,grpc-status,grpc-message'

'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection failure'

'grpc-status', '14'

'date', 'Mon, 17 May 2021 18:07:24 GMT'

'server', 'envoy'

[Chris Odd]

So it turns out the yaml file I included worked just fine, my docker run command was just specifying the wrong yaml file as input.

the command:

docker run -p 8000:8000 -p 8081:8081 -v $(pwd)/envoy.yaml:/etc/envoy/envoy.demo.yaml --rm envoyproxy/envoy-dev -c /etc/envoy/envoy.demo.yaml --log-level debug

should have been:

docker run -p 8000:8000 -p 8081:8081 -v $(pwd)/envoy.demo.yaml:/etc/envoy/envoy.demo.yaml --rm envoyproxy/envoy-dev -c /etc/envoy/envoy.demo.yaml --log-level debug