HPE_INVALID_METHOD when using http_connection_manager

[Stephen Murphy]

Hello envoy-users,

I am using envoy proxies to turn on TLS between pods in my kubernetes cluster.
I am currently trying to do this to the opensource party cube-js codebase (see sample deployment here ), which contains multiple deployments/statefulsets.

I have used HttpConnectionManager to encrypt traffic between several of cube-js pods.
However, 2 of the cube-js pods - the router and the worker - are problematic (The worker statefulset deployment sits behind a headless service and each replica has a unique FQDN. The router is configured with all these FQDNs and 'routes' to each one based on its own internal hashing algorithm).

I can only use TcpProxy to successfully encrypt traffic between these 2 pods. 

When I try to use HttpConnectionManager I see the error in the envoy logs shown below (I need to use http_connection_manager so that I can add appropriate routing based on the FQDN being used):

[2023-10-23 18:38:35.137][16][debug][http] [source/common/http/conn_manager_impl.cc:1729] [Tags: "ConnectionId":"28","StreamId":"64508131548947556"] encoding headers via codec (end_stream=false):
':status', '400'
'content-length', '11'
'content-type', 'text/plain'
'date', 'Mon, 23 Oct 2023 18:38:34 GMT'
'server', 'envoy'
'connection', 'close'

[2023-10-23 18:38:35.191][16][debug][http] [source/common/http/conn_manager_impl.cc:1834] [Tags: "ConnectionId":"28","StreamId":"64508131548947556"] Codec completed encoding stream.
[2023-10-23 18:38:35.199][16][debug][http] [source/common/http/conn_manager_impl.cc:221] [Tags: "ConnectionId":"28","StreamId":"64508131548947556"] doEndStream() resetting stream
[2023-10-23 18:38:35.135][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:358] dns resolution for dbg-cubestore-worker-0.dbg-cubestore-worker-headless.acceptance-tests.svc.cluster.local started
[2023-10-23 18:38:35.235][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:155] dns resolution without records for dbg-cubestore-worker-0.dbg-cubestore-worker-headless.acceptance-tests.svc.cluster.local
[2023-10-23 18:38:35.255][16][debug][http] [source/common/http/conn_manager_impl.cc:1798] [Tags: "ConnectionId":"28","StreamId":"64508131548947556"] stream reset: reset reason: local reset, response details: http1.codec_error
[2023-10-23 18:38:35.255][16][debug][connection] [source/common/network/connection_impl.cc:139] [C28] closing data_to_write=156 type=2
[2023-10-23 18:38:35.255][16][debug][connection] [source/common/network/connection_impl_base.cc:47] [Tags: "ConnectionId":"28"] setting delayed close timer with timeout 1000 ms
[2023-10-23 18:38:35.255][16][debug][http] [source/common/http/conn_manager_impl.cc:403] [Tags: "ConnectionId":"28"] dispatch error: http/1.1 protocol error: HPE_INVALID_METHOD
[2023-10-23 18:38:35.256][16][debug][connection] [source/common/network/connection_impl.cc:139] [C28] closing data_to_write=156 type=2
[2023-10-23 18:38:35.256][1][debug][dns] [source/extensions/network/dns_resolver/cares/dns_impl.cc:278] dns resolution for

The router config:

    listeners:
        - name: cubestore_worker_listener # Outbound traffic to cubestore worker
          address:
            socket_address:
              address: 0.0.0.0
              port_value: 9001
          filter_chains:
            - filters:
                - name: envoy.filters.network.http_connection_manager
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                    codec_type: AUTO
                    stat_prefix: cubestore_router_to_worker
                    upgrade_configs:
                      - upgrade_type: websocket
                    http_filters:
                      - name: envoy.filters.http.router
                        typed_config:
                          "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                    route_config:
                      name: local_route
                      virtual_hosts:
                        - name: app
                          domains:
                            - "*"
                          routes:
                            - match:
                                prefix: "/"
                              route:
                                cluster: cubestore_worker_cluster
                                timeout: 0s
      clusters:
        - name: cubestore_worker_cluster
          type: STRICT_DNS
          lb_policy: ROUND_ROBIN
          load_assignment:
            cluster_name: cubestore_worker_cluster
            endpoints:
              - lb_endpoints:
                  - endpoint:
                      address:
                        socket_address:
                          address: dbg-cubestore-worker-0.dbg-cubestore-worker-headless.acceptance-tests.svc.cluster.local
                          port_value: 9011
  

The worker config:

    listeners:
        - name: cubestore_worker_listener # Inbound traffic to cubestore worker
          address:
            socket_address:
              address: 0.0.0.0
              port_value: 9011
          filter_chains:
            - filters:
                - name: envoy.filters.network.http_connection_manager
                  typed_config:
                    "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
                    stat_prefix: inbound_https_to_http_port
                    upgrade_configs:
                      - upgrade_type: websocket
                    http_filters:
                      - name: envoy.filters.http.router
                        typed_config:
                          "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                    route_config:
                      name: local_route
                      virtual_hosts:
                        - name: local_service
                          domains: ["*"]
                          routes:
                            - match:
                                prefix: "/"
                              route:
                                cluster: cubestore_worker_cluster
      clusters:
        - name: cubestore_worker_cluster
          type: STRICT_DNS
          load_assignment:
            cluster_name: cubestore_worker_cluster
            endpoints:
              - lb_endpoints:
                  - endpoint:
                      address:
                        socket_address:
                          address: 127.0.0.1
                          port_value: 9001            

[Stephen Murphy]

FYI - I mistakenly omitted the TLS certificate transport sockets from the above configs, but rest assured that they are there. 

HttpConnectionManager fails with or without TLS turned on with the same "protocol error: HPE_INVALID_METHOD" message in the logs..

[Stephen Murphy]

I am attaching an tcpdump excerpt, which I assume is capturing the 400 Bad Request shown in the above logs, which can be read in wireshark. Here is also a screenshot of this file..

--
You received this message because you are subscribed to a topic in the Google Groups "envoy-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/envoy-users/hvsRQ-w8JMk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to envoy-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/27e7c60c-b01d-45b6-bab5-c1fcc25d3e0fn%40googlegroups.com.

[Beatriz Baldaia]

Hello Stephen
I had the same problem with envoy not being able to encode the headers and returning HPE_INVALID_METHOD: https://github.com/cilium/proxy/issues/370

However, my setup was different. My problem was with having AWS LB using Proxy protocol v2, which adds headers with the source and destination IP addresses and ports in a binary format, and not having envoy configured to support it.