Struggling to make gRPC-web and HTTPS work
1,309 views
Skip to first unread message
Nicholas Palmer
Apr 13, 2021, 9:53:37 PM4/13/21
to envoy-users
I've got a SPA web app that uses gRPC web and envoy to proxy back to a server that speaks gRPC. This all works great, no problems.
I'm trying to make this secure using HTTPS/TLS and just keep running into issues and can't make it work.
Our setup is this:
Web Client SPA app (accessed from web node.js server also running on the lahinch server. URL is https://lahinch.mycorp.com ). Web app connects to the envoy proxy using this address "https://coxos.mycorp.COM:8090"
\
\
Envoy Proxy (coxos - 172.16.0.116) - listens on port 8090 and proxies to port 50251
\
\
Backend gRPC server (lahinch - 172.16.0.109) - listens on port 50251
From reading the envoy docs, the web client is downstream and the backend server is upstream.
Here is my envoy.yaml file
admin:
access_log_path: /tmp/admin_access.log
address:
socket_address:
address: 0.0.0.0
port_value: 9901
static_resources:
listeners:
- name: listener_0
address:
socket_address:
address: 0.0.0.0
port_value: 8090
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
'@type': >-
stat_prefix: ingress_http
access_log:
- name: envoy.access_loggers.file
typed_config:
'@type': >-
path: /dev/stdout
- name: envoy.access_loggers.http_grpc
typed_config:
'@type': >-
common_config:
log_name: envoygrpclog
grpc_service:
envoy_grpc:
cluster_name: controlweb_backendservice
transport_api_version: V3
route_config:
name: local_route
virtual_hosts:
- name: local_service
domains:
- '*'
routes:
- match:
prefix: /
route:
cluster: controlweb_backendservice
hash_policy:
- header:
header_name: x-session-hash
max_stream_duration:
grpc_timeout_header_max: 300s
cors:
allow_origin_string_match:
- safe_regex:
google_re2: {}
regex: .*
allow_methods: 'GET, PUT, DELETE, POST, OPTIONS'
allow_headers: >-
keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,grpc-status-details-bin,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout,access-token,x-session-hash
expose_headers: >-
grpc-status-details-bin,grpc-status,grpc-message,access-token
max_age: '1728000'
http_filters:
- name: envoy.filters.http.grpc_web
typed_config:
'@type': >-
- name: envoy.filters.http.cors
typed_config:
'@type': >-
- name: envoy.filters.http.router
typed_config:
'@type': >-
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
common_tls_context:
tls_certificates:
- certificate_chain:
# Certificate must be PEM-encoded
filename: /etc/lahinch.pem
private_key:
filename: /etc/lahinch.key.pem
validation_context:
trusted_ca:
filename: /etc/ssl/certs/ZZZ-CA256.pem
clusters:
- name: controlweb_backendservice
type: LOGICAL_DNS
connect_timeout: 0.25s
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
load_assignment:
cluster_name: cluster_controlweb_backendservice
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: lahinch.mycorp.com
port_value: 50251
http2_protocol_options: {}
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
'@type': >-
common_tls_context:
tls_certificates:
- certificate_chain:
filename: /etc/lahinch.pem
private_key:
filename: /etc/lahinch.key.pem
validation_context:
trusted_ca:
filename: /etc/ssl/certs/ZZZ-CA256.pem
Using this, I'm getting the following in the envoy log when I try and run my web app:
[2021-04-09 22:08:33.939][17][debug][conn_handler] [source/server/connection_handler_impl.cc:501] [C2] new connection
[2021-04-09 22:08:33.945][17][debug][http] [source/common/http/conn_manager_impl.cc:254] [C2] new stream
[2021-04-09 22:08:33.945][17][debug][http] [source/common/http/conn_manager_impl.cc:886] [C2][S3055347406573314092] request headers complete (end_stream=false):
':authority', 'coxos.mycorp.com:8090'
':path', '/WanderAuth.HostService/LogIn'
':method', 'POST'
'connection', 'keep-alive'
'content-length', '124'
'accept', 'application/grpc-web-text'
'x-user-agent', 'grpc-web-javascript/0.1'
'access-token', ''
'x-grpc-web', '1'
'user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.68'
'grpc-timeout', '90000m'
'content-type', 'application/grpc-web-text'
'origin', 'https://lahinch.mycorp.com'
'sec-fetch-site', 'same-site'
'sec-fetch-mode', 'cors'
'sec-fetch-dest', 'empty'
'referer', 'https://lahinch.mycorp.com/'
'accept-encoding', 'gzip, deflate, br'
'accept-language', 'en-US,en;q=0.9'
[2021-04-09 22:08:33.946][17][debug][router] [source/common/router/router.cc:425] [C2][S3055347406573314092] cluster 'controlweb_backendservice' match for URL '/WanderAuth.HostService/LogIn'
[2021-04-09 22:08:33.946][17][debug][router] [source/common/router/router.cc:582] [C2][S3055347406573314092] router decoding headers:
':authority', 'coxos.mycorp.com:8090'
':path', '/WanderAuth.HostService/LogIn'
':method', 'POST'
':scheme', 'https'
'accept', 'application/grpc-web-text'
'x-user-agent', 'grpc-web-javascript/0.1'
'access-token', ''
'x-grpc-web', '1'
'user-agent', 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36 Edg/89.0.774.68'
'grpc-timeout', '90000m'
'content-type', 'application/grpc'
'origin', 'https://lahinch.mycorp.com'
'sec-fetch-site', 'same-site'
'sec-fetch-mode', 'cors'
'sec-fetch-dest', 'empty'
'referer', 'https://lahinch.mycorp.com/'
'accept-encoding', 'gzip, deflate, br'
'accept-language', 'en-US,en;q=0.9'
'x-forwarded-proto', 'https'
'x-request-id', 'a4a041ab-dc29-4ed7-a342-90ac03b3be3c'
'te', 'trailers'
'grpc-accept-encoding', 'identity'
'x-envoy-expected-rq-timeout-ms', '15000'
[2021-04-09 22:08:33.946][17][debug][pool] [source/common/http/conn_pool_base.cc:79] queueing stream due to no available connections
[2021-04-09 22:08:33.946][17][debug][pool] [source/common/conn_pool/conn_pool_base.cc:106] creating a new connection
[2021-04-09 22:08:33.946][17][debug][client] [source/common/http/codec_client.cc:41] [C3] connecting
[2021-04-09 22:08:33.946][17][debug][connection] [source/common/network/connection_impl.cc:860] [C3] connecting to 172.16.0.109:50251
[2021-04-09 22:08:33.946][17][debug][connection] [source/common/network/connection_impl.cc:876] [C3] connection in progress
[2021-04-09 22:08:33.946][17][debug][http2] [source/common/http/http2/codec_impl.cc:1184] [C3] updating connection-level initial window size to 268435456
[2021-04-09 22:08:33.946][17][debug][http] [source/common/http/filter_manager.cc:755] [C2][S3055347406573314092] request end stream
[2021-04-09 22:08:33.947][17][debug][connection] [source/common/network/connection_impl.cc:666] [C3] connected
[2021-04-09 22:08:33.947][17][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:224] [C3] TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
[2021-04-09 22:08:33.948][17][debug][connection] [source/common/network/connection_impl.cc:241] [C3] closing socket: 0
[2021-04-09 22:08:33.948][17][debug][connection] [source/extensions/transport_sockets/tls/ssl_socket.cc:224] [C3] TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
[2021-04-09 22:08:33.948][17][debug][client] [source/common/http/codec_client.cc:99] [C3] disconnect. resetting 0 pending requests
[2021-04-09 22:08:33.948][17][debug][pool] [source/common/conn_pool/conn_pool_base.cc:343] [C3] client disconnected, failure reason: TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
[2021-04-09 22:08:33.948][17][debug][router] [source/common/router/router.cc:1026] [C2][S3055347406573314092] upstream reset: reset reason: connection failure, transport failure reason: TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
[2021-04-09 22:08:33.948][17][debug][http] [source/common/http/filter_manager.cc:839] [C2][S3055347406573314092] Sending local reply with details upstream_reset_before_response_started{connection failure,TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER}
[2021-04-09 22:08:33.948][17][debug][http] [source/common/http/conn_manager_impl.cc:1484] [C2][S3055347406573314092] encoding headers via codec (end_stream=false):
':status', '503'
'content-length', '190'
'content-type', 'application/grpc-web-text+proto'
'access-control-allow-origin', 'https://lahinch.mycorp.com'
'access-control-expose-headers', 'grpc-status-details-bin,grpc-status,grpc-message,access-token'
'date', 'Fri, 09 Apr 2021 22:08:33 GMT'
'server', 'envoy'
[2021-04-09 22:08:36.139][9][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:101] starting async DNS resolution for lahinch.mycorp.com
[2021-04-09 22:08:36.139][9][debug][main] [source/server/server.cc:199] flushing stats
[2021-04-09 22:08:36.141][9][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:109] async DNS resolution complete for lahinch.mycorp.com
[2021-04-09 22:08:36.141][9][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:155] DNS refresh rate reset for lahinch.mycorp.com, refresh rate 5000 ms
So the error appears to be this: TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
I've looked up this error and it appears to be be related to security and certificates. But I haven't been able to find a good answer as to what I'm doing wrong.
When it comes to the required certs, should the certs be the same that is used by client(downstream), the proxy or the backend(upstream server) or both? I've tried using different certs for the different servers and the same certs for the servers and I still get the same error.
Nyah Check
May 7, 2022, 7:28:11 PM5/7/22
to envoy-users
Hey, I just encountered this today, How did you resolve this
Bin Patel
May 9, 2022, 4:56:19 PM5/9/22
to envoy-users
same
Vamsee Lakamsani
Jul 20, 2022, 11:12:18 AM7/20/22
to envoy-users
Hi,
I got this error today with a similar setup (envoy proxying to a backend golang grpc service). On my backend grpc service I have a startup flag that tells it to run with TLS or not. When I got this error I was running the backend service without TLS. Once I re-ran it with TLS the error went away. Also, unless you are doing mTLS, the minimal config you need to make an upstream TLS connection is just this:
==
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
==
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
==
hope that helps.
On Saturday, May 7, 2022 at 4:28:11 PM UTC-7 Nyah Check wrote:
Reply all
Reply to author
Forward
0 new messages