Upstream cluster with client cert auth - TLS issue OPENSSL_internal:NO_RENEGOTIATION

[s.abr...@gmail.com]

Using simple configuration extended from google_com_proxy.v2.yaml  (See configuration) where I configured TLS context.  Unfortunately can't establish connection with mutual TLS (client cert  auth)  

- envoyproxy/envoy:v1.6.0 

- Server .Net self hosted,   requires mutual TLS 

- curl with same cert from same box works fine, OpenSSL also can connect to server . 

- Client (Envoy) sends encrypted  Alert and close connections. 

I would appreciate any help or ideas how debug this  issue.  

Upstream cluster: 

  clusters:

  - name: service_google

    connect_timeout: 0.25s

    type: LOGICAL_DNS

    # Comment out the following line to test on v6 networks

    dns_lookup_family: V4_ONLY

    lb_policy: ROUND_ROBIN

    hosts: [{ socket_address: { address: serverhostname, port_value: 443 }}]

    tls_context: {

          common_tls_context: {

          tls_certificates: {

            certificate_chain: { filename: "/etc/_client.cer" },          

            private_key: { filename: "/etc/_client.key" }

          },

            validation_context: {

              trusted_ca: { filename: "/etc/ssl/certs/serverca.cer" }

            }          

        }

      }

WireShark

Logs:  

[2018-06-06 04:04:45.677][25][debug][client] source/common/http/codec_client.cc:25] [C15] connecting

[2018-06-06 04:04:45.677][25][debug][connection] source/common/network/connection_impl.cc:568] [C15] connecting to 104.42.70.149:443

[2018-06-06 04:04:45.677][25][debug][connection] source/common/network/connection_impl.cc:577] [C15] connection in progress

[2018-06-06 04:04:45.677][25][debug][pool] source/common/http/http1/conn_pool.cc:99] queueing request due to no available connections

[2018-06-06 04:04:45.677][25][trace][http] source/common/http/conn_manager_impl.cc:672] [C8][S13804855024196549486] decode headers called: filter=0x2dab7c0 status=1

[2018-06-06 04:04:45.677][25][trace][http] source/common/http/http1/codec_impl.cc:322] [C8] parsed 424 bytes

[2018-06-06 04:04:45.677][25][trace][connection] source/common/network/connection_impl.cc:229] [C8] readDisable: enabled=true disable=true

[2018-06-06 04:04:45.677][25][trace][connection] source/common/network/connection_impl.cc:386] [C8] socket event: 2

[2018-06-06 04:04:45.677][25][trace][connection] source/common/network/connection_impl.cc:454] [C8] write ready

[2018-06-06 04:04:45.711][25][trace][connection] source/common/network/connection_impl.cc:386] [C15] socket event: 2

[2018-06-06 04:04:45.711][25][trace][connection] source/common/network/connection_impl.cc:454] [C15] write ready

[2018-06-06 04:04:45.711][25][debug][connection] source/common/network/connection_impl.cc:464] [C15] connected

[2018-06-06 04:04:45.712][25][debug][connection] source/common/ssl/ssl_socket.cc:110] [C15] handshake error: 2

[2018-06-06 04:04:45.751][25][trace][connection] source/common/network/connection_impl.cc:386] [C15] socket event: 3

[2018-06-06 04:04:45.751][25][trace][connection] source/common/network/connection_impl.cc:454] [C15] write ready

[2018-06-06 04:04:45.751][25][debug][connection] source/common/ssl/ssl_socket.cc:110] [C15] handshake error: 2

[2018-06-06 04:04:45.751][25][trace][connection] source/common/network/connection_impl.cc:424] [C15] read ready

[2018-06-06 04:04:45.751][25][debug][connection] source/common/ssl/ssl_socket.cc:110] [C15] handshake error: 2

[2018-06-06 04:04:45.751][25][trace][connection] source/common/network/connection_impl.cc:386] [C15] socket event: 2

[2018-06-06 04:04:45.751][25][trace][connection] source/common/network/connection_impl.cc:454] [C15] write ready

[2018-06-06 04:04:45.751][25][debug][connection] source/common/ssl/ssl_socket.cc:110] [C15] handshake error: 2

[2018-06-06 04:04:45.751][25][trace][connection] source/common/network/connection_impl.cc:386] [C15] socket event: 3

[2018-06-06 04:04:45.751][25][trace][connection] source/common/network/connection_impl.cc:454] [C15] write ready

[2018-06-06 04:04:45.753][25][debug][connection] source/common/ssl/ssl_socket.cc:110] [C15] handshake error: 2

[2018-06-06 04:04:45.753][25][trace][connection] source/common/network/connection_impl.cc:424] [C15] read ready

[2018-06-06 04:04:45.753][25][debug][connection] source/common/ssl/ssl_socket.cc:110] [C15] handshake error: 2

[2018-06-06 04:04:45.786][25][trace][connection] source/common/network/connection_impl.cc:386] [C15] socket event: 3

[2018-06-06 04:04:45.786][25][trace][connection] source/common/network/connection_impl.cc:454] [C15] write ready

[2018-06-06 04:04:45.786][25][debug][connection] source/common/ssl/ssl_socket.cc:99] [C15] handshake complete

[2018-06-06 04:04:45.786][25][debug][client] source/common/http/codec_client.cc:63] [C15] connected

[2018-06-06 04:04:45.786][25][debug][pool] source/common/http/http1/conn_pool.cc:225] [C15] attaching to next request

[2018-06-06 04:04:45.786][25][debug][router] source/common/router/router.cc:966] [C8][S13804855024196549486] pool ready

[2018-06-06 04:04:45.786][25][trace][connection] source/common/network/connection_impl.cc:323] [C15] writing 548 bytes, end_stream false

[2018-06-06 04:04:45.786][25][trace][connection] source/common/network/connection_impl.cc:454] [C15] write ready

[2018-06-06 04:04:45.786][25][trace][connection] source/common/ssl/ssl_socket.cc:173] [C15] ssl write returns: 548

[2018-06-06 04:04:45.786][25][trace][connection] source/common/network/connection_impl.cc:424] [C15] read ready

[2018-06-06 04:04:45.786][25][trace][connection] source/common/ssl/ssl_socket.cc:57] [C15] ssl read returns: -1

[2018-06-06 04:04:45.787][25][trace][connection] source/common/network/connection_impl.cc:386] [C15] socket event: 2

[2018-06-06 04:04:45.787][25][trace][connection] source/common/network/connection_impl.cc:454] [C15] write ready

[2018-06-06 04:04:45.817][25][trace][connection] source/common/network/connection_impl.cc:386] [C15] socket event: 3

[2018-06-06 04:04:45.817][25][trace][connection] source/common/network/connection_impl.cc:454] [C15] write ready

[2018-06-06 04:04:45.817][25][trace][connection] source/common/network/connection_impl.cc:424] [C15] read ready

[2018-06-06 04:04:45.817][25][trace][connection] source/common/ssl/ssl_socket.cc:57] [C15] ssl read returns: -1

[2018-06-06 04:04:45.817][25][debug][connection] source/common/ssl/ssl_socket.cc:138] [C15] SSL error: 268435638:SSL routines:OPENSSL_internal:NO_RENEGOTIATION

[2018-06-06 04:04:45.817][25][debug][connection] source/common/network/connection_impl.cc:448] [C15] remote close

[2018-06-06 04:04:45.817][25][debug][connection] source/common/network/connection_impl.cc:134] [C15] closing socket: 0

[2018-06-06 04:04:45.817][25][debug][connection] source/common/ssl/ssl_socket.cc:209] [C15] SSL shutdown: rc=-1

[2018-06-06 04:04:45.817][25][debug][connection] source/common/ssl/ssl_socket.cc:138] [C15] SSL error: 268435650:SSL routines:OPENSSL_internal:PROTOCOL_IS_SHUTDOWN

[2018-06-06 04:04:45.818][25][trace][http] source/common/http/http1/codec_impl.cc:305] [C15] parsing 0 bytes

[2018-06-06 04:04:45.818][25][trace][http] source/common/http/http1/codec_impl.cc:322] [C15] parsed 0 bytes

[2018-06-06 04:04:45.818][25][debug][client] source/common/http/codec_client.cc:81] [C15] disconnect. resetting 1 pending requests

[2018-06-06 04:04:45.818][25][debug][client] source/common/http/codec_client.cc:104] [C15] request reset

[Matt Klein]

+Piotr who might have some ideas.

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/34bef8a6-51e4-4f58-81c1-59a76f6530ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Matt Klein

Software Engineer

mkl...@lyft.com

https://calendly.com/mattklein123

[Sergey Abrosimov]

thank you, looks like renegotiates started when server asks for client cert and client (envoy) sends  

Found article in Boring ssl that it requires special configuration. didn't find that in envoy source 

https://boringssl.googlesource.com/boringssl/+/HEAD/PORTING.md#TLS-renegotiation

any recommendations ?  am I missing something simple ?  

To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.

To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/34bef8a6-51e4-4f58-81c1-59a76f6530ee%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Piotr Sikora]

Coincidentally, I sent a PR that should fix that a few hours before your initial email:
https://github.com/envoyproxy/envoy/pull/3551

Let me know if it solves the problem for you.

Best regards,

Piotr Sikora

[Sergey Abrosimov]

Great! Thanks a lot, 

Is it expected to be merged to latest only on in 1,6 as well ? 

I'll try as soon as envoy container Latest is available with this changes .  

[Piotr Sikora]

Envoy doesn't maintain stable releases and/or backport patches

(and this patch wouldn't be backport-worthy anyway), so only latests.

Best regards,

Piotr Sikora

[Sergey Abrosimov]

Thank you. Latest works just fine.  Saw your comment in pull request. Server is .Net self-hosted app

[Sergey Abrosimov]

Hi, tried latest build version: 49b122d85aa8553798fc5817cdb764c217da2809/1.7.0-dev/Clean/RELEASE

option "allow_renegotiation" in tls context  fail config validation. 

[2018-06-13 04:58:07.661][10][critical][main] source/server/server.cc:77] error initializing configuration '/etc/envoy.yaml': Unable to parse JSON as proto (INVALID_ARGUMENT:static_resources.clusters[0].tls_context: Cannot find field.):

Could you please add this option to config 

as 

"tls_context": {

"allow_renegotiation": true,

"sni": "serverurl",

"common_tls_context": {

"validation_context": {

"trusted_ca": {

"filename": "/etc/ssl/certs/serverca.cer"

}

},

"tls_certificates": {

"private_key": {

"filename": "/etc/_client.key"

},

"certificate_chain": {

"filename": "/etc/_client.cer"

}

}

}

},

[Piotr Sikora]

Hi Sergey,

49b122d85aa8553798fc5817cdb764c217da2809 is from ~3 weeks ago,

you need a more recent version for "allow_renegotiation".

Best regards,

Piotr Sikora

To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/CAAqbL3KD0ZS7A4N9fg_PD8-t1T8EmZYUp6h_c6mi7OsqhPb_vQ%40mail.gmail.com.

[Sergey Abrosimov]

Thanks a lot !

Connected to server without any problem.

[msn...@gmail.com]

Hi Sergey,

I have exact same use case as yours, apigw -> envoy on MASSL.

I have the config as 

tls_context:

common_tls_context:

validation_context:

filename: /etc/apig-cert.crt

tls_certificates:

- certificate_chain:

filename: /etc/localhost.crt

private_key:

filename: /etc/localhost.key

do you know how I can test this locally to verify that envoy validates the api-cert ? Also have you tried envoy to force the cert using option require_client_certificate: true ?

cheers

Senthil