Seeking guidance about AWS request signing filter

[Palash Karmore]

Hi there,
I am trying to intercept HTTP traffic between s3 client (aws cli) and aws s3 and execute some conditional path re-write. I have tried following configuration and it works when I hit the proxy directly like this:

curl http://127.0.0.1:10000 <?xml version="1.0" encoding="UTF-8"?> <ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>0dab8005a5603754ea4b292348d935d9a0fb11399d3eb8ac83cdc476fe5b8ca9</ID></Owner><Buckets><Bucket><Name>bucket-ho-to-aisi</Name><CreationDate>2024-03-07T10:15:36.000Z</CreationDate></Bucket></Buckets></ListAllMyBucketsResult>%

But when I try to use it with aws cli it gives me this error:

aws s3 --endpoint-url http://127.0.0.1:10000 ls An error occurred (AuthorizationHeaderMalformed) when calling the ListBuckets operation: The authorization header is malformed; the authorization header requires three components: Credential, SignedHeaders, and Signature.

Here's the configuration:

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          use_remote_address: true
          codec_type: AUTO
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: app
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: s3_cluster
          http_filters:
          - name: envoy.filters.http.aws_request_signing
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.aws_request_signing.v3.AwsRequestSigning
              host_rewrite: "s3.ap-south-1.amazonaws.com"
              service_name: s3
              region: ap-south-1
              use_unsigned_payload: true
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
  - name: s3_cluster
    type: STRICT_DNS
    dns_lookup_family: V4_ONLY
    load_assignment:
      cluster_name: s3_cluster
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: s3.ap-south-1.amazonaws.com
                port_value: 443
    transport_socket:
      name: envoy.transport_sockets.tls
      typed_config:
        "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
        common_tls_context:
          validation_context:
            trusted_ca: {filename: /etc/ssl/certs/ca-certificates.crt}

Any help is deeply appreciated. Thank you.

Envoy version: 10e1e425fd84cc2cde985179655da08f68c9cb30/1.29.3/Clean/RELEASE/BoringSSL
Docker image used: envoyproxy/envoy:debug-v1.29-latest

Here's the truncated log:

. . envoy-test-envoy-1 | [2024-04-09 05:27:21.414][1][debug][config] [source/common/listener_manager/listener_manager_impl.cc:89] filter #0: envoy-test-envoy-1 | [2024-04-09 05:27:21.414][1][debug][config] [source/common/listener_manager/listener_manager_impl.cc:102] name: envoy.filters.network.http_connection_manager envoy-test-envoy-1 | [2024-04-09 05:27:21.414][1][debug][config] [source/common/listener_manager/listener_manager_impl.cc:105] config: {"@type":"type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager","stat_prefix":"ingress_http","route_config":{"name":"local_route","virtual_hosts":[{"name":"app","domains":["*"],"routes":[{"match":{"prefix":"/"},"route":{"cluster":"s3_cluster"}}]}]},"http_filters":[{"name":"envoy.filters.http.aws_request_signing","typed_config":{"@type":"type.googleapis.com/envoy.extensions.filters.http.aws_request_signing.v3.AwsRequestSigning","service_name":"s3","region":"ap-south-1","host_rewrite":"s3.ap-south-1.amazonaws.com","use_unsigned_payload":true}},{"name":"envoy.filters.http.router","typed_config":{"@type":"type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"}}],"use_remote_address":true} envoy-test-envoy-1 | [2024-04-09 05:27:21.414][1][debug][config] [./source/common/http/filter_chain_helper.h:110] http filter #0 envoy-test-envoy-1 | [2024-04-09 05:27:21.414][1][debug][aws] [source/extensions/common/aws/credentials_provider_impl.cc:757] Using environment credentials provider envoy-test-envoy-1 | [2024-04-09 05:27:21.415][1][debug][aws] [source/extensions/common/aws/credentials_provider_impl.cc:761] Using credentials file credentials provider envoy-test-envoy-1 | [2024-04-09 05:27:21.415][1][debug][aws] [source/extensions/common/aws/credentials_provider_impl.cc:827] Using instance profile credentials provider envoy-test-envoy-1 | [2024-04-09 05:27:21.415][1][debug][filter] [source/extensions/filters/http/aws_request_signing/config.cc:39] Signing Algorithm is SigV4 envoy-test-envoy-1 | [2024-04-09 05:27:21.415][1][debug][config] [./source/common/http/filter_chain_helper.h:157] name: envoy.filters.http.aws_request_signing envoy-test-envoy-1 | [2024-04-09 05:27:21.415][1][debug][config] [./source/common/http/filter_chain_helper.h:160] config: {"@type":"type.googleapis.com/envoy.extensions.filters.http.aws_request_signing.v3.AwsRequestSigning","service_name":"s3","region":"ap-south-1","host_rewrite":"s3.ap-south-1.amazonaws.com","use_unsigned_payload":true} . . [source/common/grpc/google_async_client_impl.cc:54] completionThread running envoy-test-envoy-1 | [2024-04-09 05:27:21.437][15][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster ec2_instance_metadata_server_internal added 1 removed 0 envoy-test-envoy-1 | [2024-04-09 05:27:21.437][15][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1284] adding TLS cluster s3_cluster envoy-test-envoy-1 | [2024-04-09 05:27:21.437][15][debug][upstream] [source/common/upstream/cluster_manager_impl.cc:1542] membership update for TLS cluster s3_cluster added 8 removed 0 envoy-test-envoy-1 | [2024-04-09 05:27:21.437][18][debug][grpc] [source/common/grpc/google_async_client_impl.cc:54] completionThread running envoy-test-envoy-1 | [2024-04-09 05:27:25.134][16][debug][conn_handler] [source/common/listener_manager/active_tcp_listener.cc:160] [Tags: "ConnectionId":"0"] new connection from 172.18.0.1:60846 envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][http] [source/common/http/conn_manager_impl.cc:393] [Tags: "ConnectionId":"0"] new stream envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][http] [source/common/http/conn_manager_impl.cc:1192] [Tags: "ConnectionId":"0","StreamId":"4039138991358760308"] request headers complete (end_stream=true): envoy-test-envoy-1 | ':authority', '127.0.0.1:10000' envoy-test-envoy-1 | ':path', '/' envoy-test-envoy-1 | ':method', 'GET' envoy-test-envoy-1 | 'accept-encoding', 'identity' envoy-test-envoy-1 | 'user-agent', 'aws-cli/2.13.38 Python/3.11.6 Darwin/23.2.0 exe/x86_64 prompt/off command/s3.ls' envoy-test-envoy-1 | 'x-amz-date', '20240409T052725Z' envoy-test-envoy-1 | 'x-amz-content-sha256', 'e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855' envoy-test-envoy-1 | 'authorization', 'AWS4-HMAC-SHA256 Credential=AKIAQA3HN2RMOJ6EZLHL/20240409/ap-south-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2d3e757c0ff9fe5b73d44806c3f1dc3ecac4f9625031a53ef4d498821242fb91' envoy-test-envoy-1 | envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][http] [source/common/http/conn_manager_impl.cc:1175] [Tags: "ConnectionId":"0","StreamId":"4039138991358760308"] request end stream envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"0"] current connecting state: false envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][filter] [source/extensions/filters/http/aws_request_signing/aws_request_signing_filter.cc:51] aws request signing from decodeHeaders use_unsigned_payload: true envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][aws] [source/extensions/common/aws/credentials_provider_impl.cc:70] Getting AWS credentials from the environment envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][aws] [source/extensions/common/aws/credentials_provider_impl.cc:83] Found following AWS credentials in the environment: AWS_ACCESS_KEY_ID=******, AWS_SECRET_ACCESS_KEY=*****, AWS_SESSION_TOKEN= envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][aws] [source/extensions/common/aws/signer_base_impl.cc:85] Canonical request: envoy-test-envoy-1 | GET envoy-test-envoy-1 | / envoy-test-envoy-1 | envoy-test-envoy-1 | accept-encoding:identity envoy-test-envoy-1 | authorization:AWS4-HMAC-SHA256 Credential=AKIAQA3HN2RMOJ6EZLHL/20240409/ap-south-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2d3e757c0ff9fe5b73d44806c3f1dc3ecac4f9625031a53ef4d498821242fb91 envoy-test-envoy-1 | host:s3.ap-south-1.amazonaws.com envoy-test-envoy-1 | user-agent:aws-cli/2.13.38 Python/3.11.6 Darwin/23.2.0 exe/x86_64 prompt/off command/s3.ls envoy-test-envoy-1 | x-amz-content-sha256:UNSIGNED-PAYLOAD envoy-test-envoy-1 | x-amz-date:20240409T052725Z,20240409T052700Z envoy-test-envoy-1 | x-envoy-internal:true envoy-test-envoy-1 | x-request-id:3a78ccd8-ee6a-49ea-9e6b-6949ea3f9bf2 envoy-test-envoy-1 | envoy-test-envoy-1 | accept-encoding;authorization;host;user-agent;x-amz-content-sha256;x-amz-date;x-envoy-internal;x-request-id envoy-test-envoy-1 | UNSIGNED-PAYLOAD envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][aws] [source/extensions/common/aws/signer_base_impl.cc:90] String to sign: envoy-test-envoy-1 | AWS4-HMAC-SHA256 envoy-test-envoy-1 | 20240409T052700Z envoy-test-envoy-1 | 20240409/ap-south-1/s3/aws4_request envoy-test-envoy-1 | f9ad491ddf5a8e1692a14a2cba4106d6d18331c78f35e746cc34c064b3a33388 envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][aws] [source/extensions/common/aws/signer_base_impl.cc:99] Signing request with: AWS4-HMAC-SHA256 Credential=AKIAQA3HN2RMOJ6EZLHL/20240409/ap-south-1/s3/aws4_request, SignedHeaders=accept-encoding;authorization;host;user-agent;x-amz-content-sha256;x-amz-date;x-envoy-internal;x-request-id, Signature=2d7d551add1d35fc53116269b27cb16dea3b55a6c2d3b022c9c973cf1dbc2f3e envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][router] [source/common/router/router.cc:514] [Tags: "ConnectionId":"0","StreamId":"4039138991358760308"] cluster 's3_cluster' match for URL '/' envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][router] [source/common/router/router.cc:731] [Tags: "ConnectionId":"0","StreamId":"4039138991358760308"] router decoding headers: envoy-test-envoy-1 | ':authority', 's3.ap-south-1.amazonaws.com' envoy-test-envoy-1 | ':path', '/' envoy-test-envoy-1 | ':method', 'GET' envoy-test-envoy-1 | ':scheme', 'http' envoy-test-envoy-1 | 'accept-encoding', 'identity' envoy-test-envoy-1 | 'user-agent', 'aws-cli/2.13.38 Python/3.11.6 Darwin/23.2.0 exe/x86_64 prompt/off command/s3.ls' envoy-test-envoy-1 | 'x-amz-date', '20240409T052725Z' envoy-test-envoy-1 | 'authorization', 'AWS4-HMAC-SHA256 Credential=AKIAQA3HN2RMOJ6EZLHL/20240409/ap-south-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=2d3e757c0ff9fe5b73d44806c3f1dc3ecac4f9625031a53ef4d498821242fb91,AWS4-HMAC-SHA256 Credential=AKIAQA3HN2RMOJ6EZLHL/20240409/ap-south-1/s3/aws4_request, SignedHeaders=accept-encoding;authorization;host;user-agent;x-amz-content-sha256;x-amz-date;x-envoy-internal;x-request-id, Signature=2d7d551add1d35fc53116269b27cb16dea3b55a6c2d3b022c9c973cf1dbc2f3e' envoy-test-envoy-1 | 'x-forwarded-for', '172.18.0.1' envoy-test-envoy-1 | 'x-forwarded-proto', 'http' envoy-test-envoy-1 | 'x-envoy-internal', 'true' envoy-test-envoy-1 | 'x-request-id', '3a78ccd8-ee6a-49ea-9e6b-6949ea3f9bf2' envoy-test-envoy-1 | 'x-amz-content-sha256', 'UNSIGNED-PAYLOAD' envoy-test-envoy-1 | 'x-amz-date', '20240409T052700Z' envoy-test-envoy-1 | 'x-envoy-expected-rq-timeout-ms', '15000' envoy-test-envoy-1 | envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][pool] [source/common/http/conn_pool_base.cc:78] queueing stream due to no available connections (ready=0 busy=0 connecting=0) envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][pool] [source/common/conn_pool/conn_pool_base.cc:291] trying to create new connection envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][pool] [source/common/conn_pool/conn_pool_base.cc:145] creating a new connection (connecting=0) envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][connection] [./source/common/network/connection_impl.h:98] [Tags: "ConnectionId":"1"] current connecting state: true envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][client] [source/common/http/codec_client.cc:57] [Tags: "ConnectionId":"1"] connecting envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][connection] [source/common/network/connection_impl.cc:1009] [Tags: "ConnectionId":"1"] connecting to 52.219.158.161:443 envoy-test-envoy-1 | [2024-04-09 05:27:25.135][16][debug][connection] [source/common/network/connection_impl.cc:1028] [Tags: "ConnectionId":"1"] connection in progress envoy-test-envoy-1 | [2024-04-09 05:27:25.154][16][debug][connection] [source/common/network/connection_impl.cc:746] [Tags: "ConnectionId":"1"] connected envoy-test-envoy-1 | [2024-04-09 05:27:25.182][16][debug][client] [source/common/http/codec_client.cc:88] [Tags: "ConnectionId":"1"] connected envoy-test-envoy-1 | [2024-04-09 05:27:25.182][16][debug][pool] [source/common/conn_pool/conn_pool_base.cc:328] [Tags: "ConnectionId":"1"] attaching to next stream envoy-test-envoy-1 | [2024-04-09 05:27:25.182][16][debug][pool] [source/common/conn_pool/conn_pool_base.cc:182] [Tags: "ConnectionId":"1"] creating stream envoy-test-envoy-1 | [2024-04-09 05:27:25.182][16][debug][router] [source/common/router/upstream_request.cc:578] [Tags: "ConnectionId":"0","StreamId":"4039138991358760308"] pool ready envoy-test-envoy-1 | [2024-04-09 05:27:25.182][16][debug][client] [source/common/http/codec_client.cc:141] [Tags: "ConnectionId":"1"] encode complete envoy-test-envoy-1 | [2024-04-09 05:27:25.197][16][debug][router] [source/common/router/router.cc:1506] [Tags: "ConnectionId":"0","StreamId":"4039138991358760308"] upstream headers complete: end_stream=false envoy-test-envoy-1 | [2024-04-09 05:27:25.197][16][debug][http] [source/common/http/conn_manager_impl.cc:1869] [Tags: "ConnectionId":"0","StreamId":"4039138991358760308"] encoding headers via codec (end_stream=false): envoy-test-envoy-1 | ':status', '400' envoy-test-envoy-1 | 'x-amz-request-id', 'V9F1NQHC3RM25H46' envoy-test-envoy-1 | 'x-amz-id-2', 'mhZblFa9D6MzK6e5cfzcJNTjhZG8m2xCtrtvQAeQ3Dsw7S5t/hVFu6MxLcqUx0oQxpNcax9M9sM=' envoy-test-envoy-1 | 'content-type', 'application/xml' envoy-test-envoy-1 | 'date', 'Tue, 09 Apr 2024 05:27:24 GMT' envoy-test-envoy-1 | 'server', 'envoy' envoy-test-envoy-1 | 'x-envoy-upstream-service-time', '62' envoy-test-envoy-1 | envoy-test-envoy-1 | [2024-04-09 05:27:25.197][16][debug][client] [source/common/http/codec_client.cc:128] [Tags: "ConnectionId":"1"] response complete envoy-test-envoy-1 | [2024-04-09 05:27:25.198][16][debug][http] [source/common/http/conn_manager_impl.cc:1974] [Tags: "ConnectionId":"0","StreamId":"4039138991358760308"] Codec completed encoding stream. envoy-test-envoy-1 | [2024-04-09 05:27:25.198][16][debug][pool] [source/common/http/http1/conn_pool.cc:53] [Tags: "ConnectionId":"1"] response complete envoy-test-envoy-1 | [2024-04-09 05:27:25.198][16][debug][pool] [source/common/http/http1/conn_pool.cc:59] [Tags: "ConnectionId":"1"] saw upstream close connection envoy-test-envoy-1 | [2024-04-09 05:27:25.198][16][debug][connection] [source/common/network/connection_impl.cc:146] [Tags: "ConnectionId":"1"] closing data_to_write=0 type=1 envoy-test-envoy-1 | [2024-04-09 05:27:25.198][16][debug][connection] [source/common/network/connection_impl.cc:278] [Tags: "ConnectionId":"1"] closing socket: 1 envoy-test-envoy-1 | [2024-04-09 05:27:25.198][16][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:329] [Tags: "ConnectionId":"1"] SSL shutdown: rc=1 envoy-test-envoy-1 | [2024-04-09 05:27:25.198][16][debug][client] [source/common/http/codec_client.cc:107] [Tags: "ConnectionId":"1"] disconnect. resetting 0 pending requests envoy-test-envoy-1 | [2024-04-09 05:27:25.198][16][debug][pool] [source/common/conn_pool/conn_pool_base.cc:495] [Tags: "ConnectionId":"1"] client disconnected, failure reason: envoy-test-envoy-1 | [2024-04-09 05:27:25.198][16][debug][pool] [source/common/conn_pool/conn_pool_base.cc:463] invoking 1 idle callback(s) - is_draining_for_deletion_=false envoy-test-envoy-1 | [2024-04-09 05:27:25.198][16][debug][connection] [source/common/network/connection_impl.cc:714] [Tags: "ConnectionId":"1"] remote close . . .

[Wanpanman]

Did you ever manage to solve this issue? I am facing the same issue when I use the awscliv2 or if I use one of their sdk clients to call s3. It works fine with curl.