Upstream Closing Connection Causes Next Request to Fail

2,603 views
Skip to first unread message

Bob

unread,
Jan 12, 2018, 8:51:01 AM1/12/18
to envoy-users
I am just beginning to experiment with Envoy.  As an initial experiment, I have an application that connects to our identity server to retrieve information about a user.  The application works correctly if I do not use Envoy, my goal is to just point the application to use Envoy by changing the base URL the application uses from "https://idm-dev.my.domain/" to "http://envoy.local:9001/" (with envoy.local being in my /etc/hosts with the proxy's IP address).

At this point, I have the application connecting to Envoy.  I am finding that every other downstream connection through Envoy returns a status of 400 without ever connecting to the upstream server.  When I turn on debug logging, the only message I see produced for the failing downstream connection is "[debug][http] source/common/http/conn_manager_impl.cc:225] [C2] dispatch error: http/1.1 protocol error: HPE_CLOSED_CONNECTION" which I am assuming is an indication that the upstream server closed the connection when the previous request completed.

How do I configure things so Envoy doesn't fail every other downstream request?

The route configuration for the 9001 endpoint is:
route_config:
  # Routing to IDM dev
  name: idm_dev
  virtual_hosts:
  - name: idm_dev
    domains:
      # List of "name:port" in the Host header this route matches
      "envoy.local:9001"
    routes:
    - match: { prefix: "/" }
      route: { cluster: idm_dev, retry_policy: { retry_on: "5xx,connect-failure,refused-stream", num_retries: 10 } }

The upstream cluster configuration is:
clusters:
- name: idm_dev
  # Type is STATIC, STRICT_DNS, LOGICAL_DNS, or EDS
  type: LOGICAL_DNS
  dns_refresh_rate: 300s
  dns_lookup_family: V4_ONLY
  #eds_cluster_config for dynamic member configuration
  connect_timeout: 0.25s
  lb_policy: ROUND_ROBIN
  # List of addresses, in this case just the ELB address
  hosts:
  - socket_address: { address: "idm-dev.my.domain", port_value: 443 }
  # Use HTTPS
  tls_context: { common_tls_context: { tls_params: { tls_minimum_protocol_version: TLS_AUTO } } }

Side note:  I had to add the V4_ONLY for proper DNS lookups.  If left at AUTO, Envoy queries for a V6 address and receives a response that includes the CNAMEs but no V6 address; Envoy then assumes the upstream service is down rather than falling back to requesting a V4 address.

Matt Klein

unread,
Jan 12, 2018, 8:57:11 AM1/12/18
to Bob, envoy-users
Is your upstream server speaking HTTP/1.0? If HTTP/1.1, is it correctly setting the "connection: close" header?

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/2afd39b4-525f-4d38-9fde-03b734f674a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--

bob_a...@mcgraw-hill.com

unread,
Jan 12, 2018, 9:36:46 AM1/12/18
to envoy-users
It's HTTP/1.1.  When I was running non-TLS (port 80) it was sending that correctly based on examining the packets.  Now that I switched to TLS, my only way to see the conversation is the logging within Envoy.  I'm not sure if the Connection header I'm seeing in the log is from the downstream or upstream conversation.  Is there a reliable way to figure out which is which?
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.

To post to this group, send email to envoy...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/envoy-users/2afd39b4-525f-4d38-9fde-03b734f674a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Matt Klein

unread,
Jan 12, 2018, 11:07:08 AM1/12/18
to Bob, envoy-users
Sorry I misread that Envoy is returning a 400, not a 503. I just took a quick look at the http_parser code and AFAICT the only way this can happen is if the client is sending another pipelined request after sending the "connection: close" header. Can you provide the full debug level logs from Envoy of the exchange?

To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.

To post to this group, send email to envoy...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

bob_a...@mcgraw-hill.com

unread,
Jan 12, 2018, 2:56:32 PM1/12/18
to envoy-users
Here is everything pushed to stdout/stderr.  I added a few notes (lines starting with "--") of when particular actions occurred.

[2018-01-12 19:31:45.267][1][info][main] source/server/server.cc:158] initializing epoch 0 (hot restart version=9.200.16384.127)

[2018-01-12 19:31:45.361][1][debug][main] source/server/server.cc:202] admin address: 0.0.0.0:9901

[2018-01-12 19:31:45.368][1][debug][upstream] source/common/upstream/cluster_manager_impl.cc:176] No ADS clusters defined, ADS will not be initialized.

[2018-01-12 19:31:45.465][1][debug][upstream] source/common/upstream/logical_dns_cluster.cc:71] starting async DNS resolution for idm-dev.my.domain

[2018-01-12 19:31:45.465][1][debug][upstream] source/common/network/dns_impl.cc:129] Setting DNS resolution timer for 5000 milliseconds

[2018-01-12 19:31:45.465][1][debug][upstream] source/common/upstream/cluster_manager_impl.cc:62] cm init: adding: cluster=idm_dev primary=1 secondary=0

[2018-01-12 19:31:45.465][1][debug][upstream] source/common/upstream/cluster_manager_impl.cc:508] adding TLS initial cluster idm_dev

[2018-01-12 19:31:45.466][1][info][config] source/server/configuration_impl.cc:45] loading 1 listener(s)

[2018-01-12 19:31:45.466][1][debug][config] source/server/configuration_impl.cc:47] listener #0:

[2018-01-12 19:31:45.466][1][debug][config] source/server/listener_manager_impl.cc:215] begin add/update listener: name=port_9001 hash=6574090823113024789

[2018-01-12 19:31:45.466][1][debug][config] source/server/listener_manager_impl.cc:29]   filter #0:

[2018-01-12 19:31:45.466][1][debug][config] source/server/listener_manager_impl.cc:30]     name: envoy.http_connection_manager

[2018-01-12 19:31:45.562][1][debug][config] source/server/config/network/http_connection_manager.cc:226]     filter #0

[2018-01-12 19:31:45.562][1][debug][config] source/server/config/network/http_connection_manager.cc:227]       name: envoy.router

[2018-01-12 19:31:45.562][1][debug][config] source/server/listener_manager_impl.cc:162] add active listener: name=port_9001, hash=6574090823113024789, address=0.0.0.0:9001

[2018-01-12 19:31:45.562][1][info][config] source/server/configuration_impl.cc:83] loading tracing configuration

[2018-01-12 19:31:45.562][1][info][config] source/server/configuration_impl.cc:109] loading stats sink configuration

[2018-01-12 19:31:45.563][1][info][main] source/server/server.cc:351] starting main dispatch loop

[2018-01-12 19:31:45.563][1][debug][upstream] source/common/network/dns_impl.cc:129] Setting DNS resolution timer for 4686 milliseconds

[2018-01-12 19:31:45.564][1][debug][upstream] source/common/network/dns_impl.cc:129] Setting DNS resolution timer for 3124 milliseconds

[2018-01-12 19:31:45.564][1][debug][upstream] source/common/network/dns_impl.cc:129] Setting DNS resolution timer for 3749 milliseconds

[2018-01-12 19:31:45.680][1][debug][upstream] source/common/upstream/logical_dns_cluster.cc:79] async DNS resolution complete for idm-dev.my.domain

[2018-01-12 19:31:45.680][1][debug][upstream] source/common/upstream/cluster_manager_impl.cc:84] cm init: init complete: cluster=idm_dev primary=0 secondary=0

[2018-01-12 19:31:45.680][1][info][upstream] source/common/upstream/cluster_manager_impl.cc:131] cm init: all clusters initialized

[2018-01-12 19:31:45.680][1][info][main] source/server/server.cc:335] all clusters initialized. initializing init manager

[2018-01-12 19:31:45.680][1][info][config] source/server/listener_manager_impl.cc:482] all dependencies initialized. starting workers

[2018-01-12 19:31:45.680][7][debug][main] source/server/worker_impl.cc:107] worker entering dispatch loop

[2018-01-12 19:31:45.680][6][debug][main] source/server/worker_impl.cc:107] worker entering dispatch loop

[2018-01-12 19:31:45.680][7][debug][upstream] source/common/upstream/cluster_manager_impl.cc:508] adding TLS initial cluster idm_dev

[2018-01-12 19:31:45.680][6][debug][upstream] source/common/upstream/cluster_manager_impl.cc:508] adding TLS initial cluster idm_dev

[2018-01-12 19:31:50.562][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:31:55.562][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:32:00.562][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:32:05.632][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:32:10.632][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:32:15.632][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:32:20.632][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:32:25.633][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:32:30.633][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:32:35.633][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:32:40.633][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:32:45.633][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:32:50.633][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:32:55.633][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:33:00.633][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:33:05.633][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:33:10.633][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:33:15.633][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:33:20.633][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:33:25.660][1][debug][main] source/server/server.cc:131] flushing stats

--

-- Start application, makes one request

--

[2018-01-12 19:33:30.660][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:33:35.662][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:33:40.664][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:33:45.665][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:33:48.074][7][debug][main] source/server/connection_handler_impl.cc:129] [C1] new connection

[2018-01-12 19:33:48.080][7][debug][http] source/common/http/conn_manager_impl.cc:181] [C1] new stream

[2018-01-12 19:33:48.080][7][debug][http] source/common/http/conn_manager_impl.cc:439] [C1][S5009107559932340921] request headers complete (end_stream=false):

[2018-01-12 19:33:48.080][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C1][S5009107559932340921]   'content-length':'135'

[2018-01-12 19:33:48.080][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C1][S5009107559932340921]   'content-type':'application/x-www-form-urlencoded; charset=UTF-8'

[2018-01-12 19:33:48.080][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C1][S5009107559932340921]   ':authority':'envoy.local:9001'

[2018-01-12 19:33:48.080][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C1][S5009107559932340921]   'connection':'Keep-Alive'

[2018-01-12 19:33:48.080][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C1][S5009107559932340921]   'user-agent':'ShibbolethIdp/3.3.2 OpenSAML/3.3.1'

[2018-01-12 19:33:48.080][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C1][S5009107559932340921]   'accept-encoding':'gzip,deflate'

[2018-01-12 19:33:48.080][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C1][S5009107559932340921]   ':path':'/login'

[2018-01-12 19:33:48.080][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C1][S5009107559932340921]   ':method':'POST'

[2018-01-12 19:33:48.080][7][debug][router] source/common/router/router.cc:239] [C1][S5009107559932340921] cluster 'idm_dev' match for URL '/login'

[2018-01-12 19:33:48.080][7][debug][router] source/common/router/router.cc:284] [C1][S5009107559932340921]   'content-length':'135'

[2018-01-12 19:33:48.080][7][debug][router] source/common/router/router.cc:284] [C1][S5009107559932340921]   'content-type':'application/x-www-form-urlencoded; charset=UTF-8'

[2018-01-12 19:33:48.080][7][debug][router] source/common/router/router.cc:284] [C1][S5009107559932340921]   ':authority':'envoy.local:9001'

[2018-01-12 19:33:48.080][7][debug][router] source/common/router/router.cc:284] [C1][S5009107559932340921]   'user-agent':'ShibbolethIdp/3.3.2 OpenSAML/3.3.1'

[2018-01-12 19:33:48.080][7][debug][router] source/common/router/router.cc:284] [C1][S5009107559932340921]   'accept-encoding':'gzip,deflate'

[2018-01-12 19:33:48.080][7][debug][router] source/common/router/router.cc:284] [C1][S5009107559932340921]   ':path':'/login'

[2018-01-12 19:33:48.080][7][debug][router] source/common/router/router.cc:284] [C1][S5009107559932340921]   ':method':'POST'

[2018-01-12 19:33:48.080][7][debug][router] source/common/router/router.cc:284] [C1][S5009107559932340921]   'x-forwarded-proto':'http'

[2018-01-12 19:33:48.080][7][debug][router] source/common/router/router.cc:284] [C1][S5009107559932340921]   'x-request-id':'8ed04e4d-2e7c-4ec6-8de9-1e539892f9ca'

[2018-01-12 19:33:48.080][7][debug][router] source/common/router/router.cc:284] [C1][S5009107559932340921]   'x-envoy-expected-rq-timeout-ms':'15000'

[2018-01-12 19:33:48.080][7][debug][router] source/common/router/router.cc:284] [C1][S5009107559932340921]   ':scheme':'http'

[2018-01-12 19:33:48.080][7][debug][pool] source/common/http/http1/conn_pool.cc:65] creating a new connection

[2018-01-12 19:33:48.080][7][debug][client] source/common/http/codec_client.cc:23] [C2] connecting

[2018-01-12 19:33:48.080][7][debug][connection] source/common/network/connection_impl.cc:526] [C2] connecting to 52.71.191.138:80

[2018-01-12 19:33:48.080][7][debug][connection] source/common/network/connection_impl.cc:535] [C2] connection in progress

[2018-01-12 19:33:48.080][7][debug][pool] source/common/http/http1/conn_pool.cc:91] queueing request due to no available connections

[2018-01-12 19:33:48.080][7][debug][http] source/common/http/conn_manager_impl.cc:652] [C1][S5009107559932340921] request end stream

[2018-01-12 19:33:48.160][7][debug][connection] source/common/network/connection_impl.cc:495] [C2] connected

[2018-01-12 19:33:48.160][7][debug][client] source/common/http/codec_client.cc:52] [C2] connected

[2018-01-12 19:33:48.160][7][debug][pool] source/common/http/http1/conn_pool.cc:217] [C2] attaching to next request

[2018-01-12 19:33:48.160][7][debug][router] source/common/router/router.cc:902] [C1][S5009107559932340921] pool ready

[2018-01-12 19:33:48.280][7][debug][router] source/common/router/router.cc:553] [C1][S5009107559932340921] upstream headers complete: end_stream=false

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:859] [C1][S5009107559932340921] encoding headers via codec (end_stream=false):

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C1][S5009107559932340921]   'date':'Fri, 12 Jan 2018 19:33:48 GMT'

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C1][S5009107559932340921]   'content-type':'application/json;charset=UTF-8'

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C1][S5009107559932340921]   'x-content-type-options':'nosniff'

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C1][S5009107559932340921]   'x-xss-protection':'1; mode=block'

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C1][S5009107559932340921]   'cache-control':'no-cache, no-store, max-age=0, must-revalidate'

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C1][S5009107559932340921]   'pragma':'no-cache'

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C1][S5009107559932340921]   'expires':'0'

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C1][S5009107559932340921]   'x-frame-options':'DENY'

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C1][S5009107559932340921]   'x-application-context':'enterprise.identity-dev:default:7028'

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C1][S5009107559932340921]   'x-content-type-options':'nosniff'

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C1][S5009107559932340921]   'content-language':'en-US'

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C1][S5009107559932340921]   ':status':'200'

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C1][S5009107559932340921]   'x-envoy-upstream-service-time':'199'

[2018-01-12 19:33:48.281][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C1][S5009107559932340921]   'server':'envoy'

[2018-01-12 19:33:48.281][7][debug][client] source/common/http/codec_client.cc:81] [C2] response complete

[2018-01-12 19:33:48.281][7][debug][pool] source/common/http/http1/conn_pool.cc:192] [C2] response complete

[2018-01-12 19:33:48.281][7][debug][pool] source/common/http/http1/conn_pool.cc:212] [C2] moving to ready

[2018-01-12T19:33:48.080Z] "POST /login HTTP/1.1" 200 - 135 1142 201 199 "-" "ShibbolethIdp/3.3.2 OpenSAML/3.3.1" "8ed04e4d-2e7c-4ec6-8de9-1e539892f9ca" "envoy.local:9001" "52.71.191.138:80"

[2018-01-12 19:33:50.667][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:33:55.667][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:34:00.668][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:34:05.669][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:34:10.669][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:34:15.668][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:34:20.669][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:34:25.670][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:34:30.671][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:34:35.672][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:34:40.672][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:34:45.674][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:34:48.286][7][debug][connection] source/common/network/connection_impl.cc:450] [C2] remote close

[2018-01-12 19:34:48.286][7][debug][connection] source/common/network/connection_impl.cc:160] [C2] closing socket: 0

[2018-01-12 19:34:48.287][7][debug][client] source/common/http/codec_client.cc:70] [C2] disconnect. resetting 0 pending requests

[2018-01-12 19:34:48.287][7][debug][pool] source/common/http/http1/conn_pool.cc:107] [C2] client disconnected

[2018-01-12 19:34:50.673][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:34:55.674][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:35:00.675][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:35:05.674][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:35:10.674][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:35:15.675][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:35:20.676][1][debug][main] source/server/server.cc:131] flushing stats

--

-- Attempt to log in, makes one request that fails

--

[2018-01-12 19:35:25.676][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:35:30.677][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:35:32.498][7][debug][http] source/common/http/conn_manager_impl.cc:225] [C1] dispatch error: http/1.1 protocol error: HPE_CLOSED_CONNECTION

[2018-01-12 19:35:32.498][7][debug][connection] source/common/network/connection_impl.cc:127] [C1] closing data_to_write=66 type=0

[2018-01-12 19:35:32.498][7][debug][connection] source/common/network/connection_impl.cc:520] [C1] write flush complete

[2018-01-12 19:35:32.498][7][debug][connection] source/common/network/connection_impl.cc:160] [C1] closing socket: 1

[2018-01-12 19:35:32.498][7][debug][main] source/server/connection_handler_impl.cc:59] [C1] adding to cleanup list

[2018-01-12 19:35:35.678][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:35:40.678][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:35:45.678][1][debug][main] source/server/server.cc:131] flushing stats

--

-- Repeat login attempt, first call OK so makes second call as well

--

[2018-01-12 19:35:50.679][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:35:55.681][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:36:00.680][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:36:01.269][7][debug][main] source/server/connection_handler_impl.cc:129] [C3] new connection

[2018-01-12 19:36:01.269][7][debug][http] source/common/http/conn_manager_impl.cc:181] [C3] new stream

[2018-01-12 19:36:01.269][7][debug][http] source/common/http/conn_manager_impl.cc:439] [C3][S4747446213670924955] request headers complete (end_stream=false):

[2018-01-12 19:36:01.269][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C3][S4747446213670924955]   'content-length':'154'

[2018-01-12 19:36:01.269][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C3][S4747446213670924955]   'content-type':'application/x-www-form-urlencoded; charset=UTF-8'

[2018-01-12 19:36:01.269][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C3][S4747446213670924955]   ':authority':'envoy.local:9001'

[2018-01-12 19:36:01.269][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C3][S4747446213670924955]   'connection':'Keep-Alive'

[2018-01-12 19:36:01.269][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C3][S4747446213670924955]   'user-agent':'ShibbolethIdp/3.3.2 OpenSAML/3.3.1'

[2018-01-12 19:36:01.269][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C3][S4747446213670924955]   'accept-encoding':'gzip,deflate'

[2018-01-12 19:36:01.269][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C3][S4747446213670924955]   ':path':'/login'

[2018-01-12 19:36:01.269][7][debug][http] source/common/http/conn_manager_impl.cc:444] [C3][S4747446213670924955]   ':method':'POST'

[2018-01-12 19:36:01.269][7][debug][router] source/common/router/router.cc:239] [C3][S4747446213670924955] cluster 'idm_dev' match for URL '/login'

[2018-01-12 19:36:01.269][7][debug][router] source/common/router/router.cc:284] [C3][S4747446213670924955]   'content-length':'154'

[2018-01-12 19:36:01.269][7][debug][router] source/common/router/router.cc:284] [C3][S4747446213670924955]   'content-type':'application/x-www-form-urlencoded; charset=UTF-8'

[2018-01-12 19:36:01.269][7][debug][router] source/common/router/router.cc:284] [C3][S4747446213670924955]   ':authority':'envoy.local:9001'

[2018-01-12 19:36:01.269][7][debug][router] source/common/router/router.cc:284] [C3][S4747446213670924955]   'user-agent':'ShibbolethIdp/3.3.2 OpenSAML/3.3.1'

[2018-01-12 19:36:01.269][7][debug][router] source/common/router/router.cc:284] [C3][S4747446213670924955]   'accept-encoding':'gzip,deflate'

[2018-01-12 19:36:01.269][7][debug][router] source/common/router/router.cc:284] [C3][S4747446213670924955]   ':path':'/login'

[2018-01-12 19:36:01.269][7][debug][router] source/common/router/router.cc:284] [C3][S4747446213670924955]   ':method':'POST'

[2018-01-12 19:36:01.269][7][debug][router] source/common/router/router.cc:284] [C3][S4747446213670924955]   'x-forwarded-proto':'http'

[2018-01-12 19:36:01.269][7][debug][router] source/common/router/router.cc:284] [C3][S4747446213670924955]   'x-request-id':'df2cd47c-5abf-4079-9cf9-aa81a66209fc'

[2018-01-12 19:36:01.270][7][debug][router] source/common/router/router.cc:284] [C3][S4747446213670924955]   'x-envoy-expected-rq-timeout-ms':'15000'

[2018-01-12 19:36:01.270][7][debug][router] source/common/router/router.cc:284] [C3][S4747446213670924955]   ':scheme':'http'

[2018-01-12 19:36:01.270][7][debug][pool] source/common/http/http1/conn_pool.cc:65] creating a new connection

[2018-01-12 19:36:01.270][7][debug][client] source/common/http/codec_client.cc:23] [C4] connecting

[2018-01-12 19:36:01.270][7][debug][connection] source/common/network/connection_impl.cc:526] [C4] connecting to 52.71.191.138:80

[2018-01-12 19:36:01.270][7][debug][connection] source/common/network/connection_impl.cc:535] [C4] connection in progress

[2018-01-12 19:36:01.271][7][debug][pool] source/common/http/http1/conn_pool.cc:91] queueing request due to no available connections

[2018-01-12 19:36:01.271][7][debug][http] source/common/http/conn_manager_impl.cc:652] [C3][S4747446213670924955] request end stream

[2018-01-12 19:36:01.316][7][debug][connection] source/common/network/connection_impl.cc:495] [C4] connected

[2018-01-12 19:36:01.316][7][debug][client] source/common/http/codec_client.cc:52] [C4] connected

[2018-01-12 19:36:01.316][7][debug][pool] source/common/http/http1/conn_pool.cc:217] [C4] attaching to next request

[2018-01-12 19:36:01.316][7][debug][router] source/common/router/router.cc:902] [C3][S4747446213670924955] pool ready

[2018-01-12 19:36:01.453][7][debug][router] source/common/router/router.cc:553] [C3][S4747446213670924955] upstream headers complete: end_stream=false

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:859] [C3][S4747446213670924955] encoding headers via codec (end_stream=false):

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C3][S4747446213670924955]   'date':'Fri, 12 Jan 2018 19:36:01 GMT'

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C3][S4747446213670924955]   'content-type':'application/json;charset=UTF-8'

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C3][S4747446213670924955]   'x-content-type-options':'nosniff'

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C3][S4747446213670924955]   'x-xss-protection':'1; mode=block'

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C3][S4747446213670924955]   'cache-control':'no-cache, no-store, max-age=0, must-revalidate'

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C3][S4747446213670924955]   'pragma':'no-cache'

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C3][S4747446213670924955]   'expires':'0'

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C3][S4747446213670924955]   'x-frame-options':'DENY'

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C3][S4747446213670924955]   'x-application-context':'enterprise.identity-dev:default:7028'

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C3][S4747446213670924955]   'x-content-type-options':'nosniff'

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C3][S4747446213670924955]   'content-language':'en-US'

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C3][S4747446213670924955]   ':status':'200'

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C3][S4747446213670924955]   'x-envoy-upstream-service-time':'182'

[2018-01-12 19:36:01.453][7][debug][http] source/common/http/conn_manager_impl.cc:864] [C3][S4747446213670924955]   'server':'envoy'

[2018-01-12 19:36:01.455][7][debug][client] source/common/http/codec_client.cc:81] [C4] response complete

[2018-01-12 19:36:01.455][7][debug][pool] source/common/http/http1/conn_pool.cc:192] [C4] response complete

[2018-01-12 19:36:01.455][7][debug][pool] source/common/http/http1/conn_pool.cc:212] [C4] moving to ready

[2018-01-12 19:36:02.181][7][debug][http] source/common/http/conn_manager_impl.cc:225] [C3] dispatch error: http/1.1 protocol error: HPE_CLOSED_CONNECTION

[2018-01-12 19:36:02.181][7][debug][connection] source/common/network/connection_impl.cc:127] [C3] closing data_to_write=66 type=0

[2018-01-12 19:36:02.181][7][debug][connection] source/common/network/connection_impl.cc:520] [C3] write flush complete

[2018-01-12 19:36:02.181][7][debug][connection] source/common/network/connection_impl.cc:160] [C3] closing socket: 1

[2018-01-12 19:36:02.182][7][debug][main] source/server/connection_handler_impl.cc:59] [C3] adding to cleanup list

[2018-01-12 19:36:05.680][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12T19:36:01.269Z] "POST /login HTTP/1.1" 200 - 154 3574 185 182 "-" "ShibbolethIdp/3.3.2 OpenSAML/3.3.1" "df2cd47c-5abf-4079-9cf9-aa81a66209fc" "envoy.local:9001" "52.71.191.138:80"

[2018-01-12 19:36:10.680][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:36:15.680][1][debug][main] source/server/server.cc:131] flushing stats

[2018-01-12 19:36:20.680][1][debug][main] source/server/server.cc:131] flushing stats





On Friday, January 12, 2018 at 11:07:08 AM UTC-5, Matt Klein wrote:
Sorry I misread that Envoy is returning a 400, not a 503. I just took a quick look at the http_parser code and AFAICT the only way this can happen is if the client is sending another pipelined request after sending the "connection: close" header. Can you provide the full debug level logs from Envoy of the exchange?

... Removed previous messages for brevity 

Matt Klein

unread,
Jan 12, 2018, 4:07:00 PM1/12/18
to Bob, envoy-users
The issue is on the downstream side w/ your client, but I'm not really sure what the issue is. If possible, can you use plain text and dump the actual HTTP plain text (including accurate white space) using tcpdump/wireshark, and paste it?

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

bob_a...@mcgraw-hill.com

unread,
Jan 12, 2018, 9:25:46 PM1/12/18
to envoy-users
Here is the dump from tcpdump.  I have redacted some information that might be sensitive but that only obscures URL or form data, I believe.

01:49:43.799631 IP (tos 0x0, ttl 64, id 8941, offset 0, flags [DF], proto TCP (6), length 60)


    172.17.0.6.38570 > 172.17.0.5.9001: Flags [S], cksum 0x585c (incorrect -> 0xf4c6), seq 2119481083, win 29200, options [mss 1460,sackOK,TS val 96572438 ecr 0,nop,wscale 7], length 0


        0x0000:  4500 003c 22ed 4000 4006 bfa1 ac11 0006  E..<".@.@.......


        0x0010:  ac11 0005 96aa 2329 7e54 b6fb 0000 0000  ......#)~T......


        0x0020:  a002 7210 585c 0000 0204 05b4 0402 080a  ..r.X\..........


        0x0030:  05c1 9416 0000 0000 0103 0307            ............


01:49:43.799681 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)


    172.17.0.5.9001 > 172.17.0.6.38570: Flags [S.], cksum 0x585c (incorrect -> 0x8798), seq 706325020, ack 2119481084, win 28960, options [mss 1460,sackOK,TS val 96572438 ecr 96572438,nop,wscale 7], length 0


        0x0000:  4500 003c 0000 4000 4006 e28e ac11 0005  E..<..@.@.......


        0x0010:  ac11 0006 2329 96aa 2a19 aa1c 7e54 b6fc  ....#)..*...~T..


        0x0020:  a012 7120 585c 0000 0204 05b4 0402 080a  ..q.X\..........


        0x0030:  05c1 9416 05c1 9416 0103 0307            ............


01:49:43.799698 IP (tos 0x0, ttl 64, id 8942, offset 0, flags [DF], proto TCP (6), length 52)


    172.17.0.6.38570 > 172.17.0.5.9001: Flags [.], cksum 0x5854 (incorrect -> 0x26a0), seq 1, ack 1, win 229, options [nop,nop,TS val 96572438 ecr 96572438], length 0


        0x0000:  4500 0034 22ee 4000 4006 bfa8 ac11 0006  E..4".@.@.......


        0x0010:  ac11 0005 96aa 2329 7e54 b6fc 2a19 aa1d  ......#)~T..*...


        0x0020:  8010 00e5 5854 0000 0101 080a 05c1 9416  ....XT..........


        0x0030:  05c1 9416                                ....


01:49:43.803926 IP (tos 0x0, ttl 64, id 8943, offset 0, flags [DF], proto TCP (6), length 445)


    172.17.0.6.38570 > 172.17.0.5.9001: Flags [P.], cksum 0x59dd (incorrect -> 0xc541), seq 1:394, ack 1, win 229, options [nop,nop,TS val 96572443 ecr 96572438], length 393


        0x0000:  4500 01bd 22ef 4000 4006 be1e ac11 0006  E...".@.@.......


        0x0010:  ac11 0005 96aa 2329 7e54 b6fc 2a19 aa1d  ......#)~T..*...


        0x0020:  8018 00e5 59dd 0000 0101 080a 05c1 941b  ....Y...........


        0x0030:  05c1 9416 504f 5354 202f 7631 2f74 6f6b  ....POST./v1/tok


        0x0040:  656e 2048 5454 502f 312e 310d 0a43 6f6e  en.HTTP/1.1..Con


        0x0050:  7465 6e74 2d4c 656e 6774 683a 2031 3335  tent-Length:.135


        0x0060:  0d0a 436f 6e74 656e 742d 5479 7065 3a20  ..Content-Type:.


        0x0070:  6170 706c 6963 6174 696f 6e2f 782d 7777  application/x-ww


        0x0080:  772d 666f 726d 2d75 726c 656e 636f 6465  w-form-urlencode


        0x0090:  643b 2063 6861 7273 6574 3d55 5446 2d38  d;.charset=UTF-8


        0x00a0:  0d0a 486f 7374 3a20 6c6f 6361 6c2e 656e  ..Host:.local.en


        0x00b0:  766f 793a 3930 3031 0d0a 436f 6e6e 6563  voy:9001..Connec


        0x00c0:  7469 6f6e 3a20 4b65 6570 2d41 6c69 7665  tion:.Keep-Alive


        0x00d0:  0d0a 5573 6572 2d41 6765 6e74 3a20 5368  ..User-Agent:.Sh


        0x00e0:  6962 626f 6c65 7468 4964 702f 332e 332e  ibbolethIdp/3.3.


        0x00f0:  3220 4f70 656e 5341 4d4c 2f33 2e33 2e31  2.OpenSAML/3.3.1


        0x0100:  0d0a 4163 6365 7074 2d45 6e63 6f64 696e  ..Accept-Encodin


        0x0110:  673a 2067 7a69 702c 6465 666c 6174 650d  g:.gzip,deflate.


        0x0120:  0a43 6f6e 6e65 6374 696f 6e3a 2063 6c6f  .Connection:.clo


        0x0130:  7365 0d0a 0d0a 636c 6965 6e74 5f69 643d  se....client_id=


        0x0140:  5072 6f64 7563 7450 6f72 7461 6c26 636c  ProductPortal&cl


        0x0150:  6965 6e74 5f73 6563 7265 743d xxxx xxxx  ient_secret=xxxx


        0x0160:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0170:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0180:  xxxx xxxx xxxx xxxx xxxx xxxx 2667 7261  xxxxxxxxxxxx&gra


        0x0190:  6e74 5f74 7970 653d 636c 6965 6e74 5f63  nt_type=client_c


        0x01a0:  7265 6465 6e74 6961 6c73 2673 636f 7065  redentials&scope


        0x01b0:  3d70 726f 7669 7369 6f6e 696e 67         =provisioning


01:49:43.803941 IP (tos 0x0, ttl 64, id 15883, offset 0, flags [DF], proto TCP (6), length 52)


    172.17.0.5.9001 > 172.17.0.6.38570: Flags [.], cksum 0x5854 (incorrect -> 0x2507), seq 1, ack 394, win 235, options [nop,nop,TS val 96572443 ecr 96572443], length 0


        0x0000:  4500 0034 3e0b 4000 4006 a48b ac11 0005  E..4>.@.@.......


        0x0010:  ac11 0006 2329 96aa 2a19 aa1d 7e54 b885  ....#)..*...~T..


        0x0020:  8010 00eb 5854 0000 0101 080a 05c1 941b  ....XT..........


        0x0030:  05c1 941b                                ....


01:49:44.229096 IP (tos 0x0, ttl 64, id 15884, offset 0, flags [DF], proto TCP (6), length 1694)


    172.17.0.5.9001 > 172.17.0.6.38570: Flags [P.], cksum 0x5ebe (incorrect -> 0x9201), seq 1:1643, ack 394, win 235, options [nop,nop,TS val 96572868 ecr 96572443], length 1642


        0x0000:  4500 069e 3e0c 4000 4006 9e20 ac11 0005  E...>.@.@.......


        0x0010:  ac11 0006 2329 96aa 2a19 aa1d 7e54 b885  ....#)..*...~T..


        0x0020:  8018 00eb 5ebe 0000 0101 080a 05c1 95c4  ....^...........


        0x0030:  05c1 941b 4854 5450 2f31 2e31 2032 3030  ....HTTP/1.1.200


        0x0040:  204f 4b0d 0a64 6174 653a 2053 6174 2c20  .OK..date:.Sat,.


        0x0050:  3133 204a 616e 2032 3031 3820 3031 3a34  13.Jan.2018.01:4


        0x0060:  393a 3434 2047 4d54 0d0a 636f 6e74 656e  9:44.GMT..conten


        0x0070:  742d 7479 7065 3a20 6170 706c 6963 6174  t-type:.applicat


        0x0080:  696f 6e2f 6a73 6f6e 3b63 6861 7273 6574  ion/json;charset


        0x0090:  3d55 5446 2d38 0d0a 782d 636f 6e74 656e  =UTF-8..x-conten


        0x00a0:  742d 7479 7065 2d6f 7074 696f 6e73 3a20  t-type-options:.


        0x00b0:  6e6f 736e 6966 660d 0a78 2d78 7373 2d70  nosniff..x-xss-p


        0x00c0:  726f 7465 6374 696f 6e3a 2031 3b20 6d6f  rotection:.1;.mo


        0x00d0:  6465 3d62 6c6f 636b 0d0a 6361 6368 652d  de=block..cache-


        0x00e0:  636f 6e74 726f 6c3a 206e 6f2d 6361 6368  control:.no-cach


        0x00f0:  652c 206e 6f2d 7374 6f72 652c 206d 6178  e,.no-store,.max


        0x0100:  2d61 6765 3d30 2c20 6d75 7374 2d72 6576  -age=0,.must-rev


        0x0110:  616c 6964 6174 650d 0a70 7261 676d 613a  alidate..pragma:


        0x0120:  206e 6f2d 6361 6368 650d 0a65 7870 6972  .no-cache..expir


        0x0130:  6573 3a20 300d 0a78 2d66 7261 6d65 2d6f  es:.0..x-frame-o


        0x0140:  7074 696f 6e73 3a20 4445 4e59 0d0a 782d  ptions:.DENY..x-


        0x0150:  6170 706c 6963 6174 696f 6e2d 636f 6e74  application-cont


        0x0160:  6578 743a 2065 6e74 6572 7072 6973 652e  ext:.enterprise.


        0x0170:  6964 656e 7469 7479 2d64 6576 3a64 6566  identity-dev:def


        0x0180:  6175 6c74 3a37 3032 380d 0a78 2d63 6f6e  ault:7028..x-con


        0x0190:  7465 6e74 2d74 7970 652d 6f70 7469 6f6e  tent-type-option


        0x01a0:  733a 206e 6f73 6e69 6666 0d0a 636f 6e74  s:.nosniff..cont


        0x01b0:  656e 742d 6c61 6e67 7561 6765 3a20 656e  ent-language:.en


        0x01c0:  2d55 530d 0a78 2d65 6e76 6f79 2d75 7073  -US..x-envoy-ups


        0x01d0:  7472 6561 6d2d 7365 7276 6963 652d 7469  tream-service-ti


        0x01e0:  6d65 3a20 3336 330d 0a73 6572 7665 723a  me:.363..server:


        0x01f0:  2065 6e76 6f79 0d0a 7472 616e 7366 6572  .envoy..transfer


        0x0200:  2d65 6e63 6f64 696e 673a 2063 6875 6e6b  -encoding:.chunk


        0x0210:  6564 0d0a 0d0a 3362 300d 0a7b 2261 6363  ed....3b0..{"acc


        0x0220:  6573 735f 746f 6b65 6e22 3a22 6579 4a68  ess_token":"eyJh


        0x0230:  6247 6369 4f69 4a53 557a 4931 4e69 4973  bGciOiJSUzI1NiIs


        0x0240:  496e 5235 6343 4936 496b 7058 5643 4973  InR5cCI6IkpXVCIs


        0x0250:  496d 7470 5a43 4936 496a 6330 4e44 5978  ImtpZCI6Ijc0NDYx


        0x0260:  4d7a 5535 496e 302e 6579 4a7a 6457 4969  MzU5In0.eyJzdWIi


        0x0270:  4f69 4a51 636d 396b 6457 4e30 5547 3979  OiJQcm9kdWN0UG9y


        0x0280:  6447 4673 4969 7769 5a57 6c6b 496a 7075  dGFsIiwiZWlkIjpu


        0x0290:  6457 7873 4c43 4a6a 6247 6c6c 626e 5266  dWxsLCJjbGllbnRf


        0x02a0:  6547 6c6b 496a 7075 6457 7873 4c43 4a70  eGlkIjpudWxsLCJp


        0x02b0:  6333 4d69 4f69 4a6f 6448 5277 637a 6f76  c3MiOiJodHRwczov


        0x02c0:  4c32 6c6b 6253 316b 5a58 5975 6257 686c  L2lkbS1kZXYubWhl


        0x02d0:  5a48 566a 5958 5270 6232 3475 5932 3974  ZHVjYXRpb24uY29t


        0x02e0:  4969 7769 5932 7870 5a57 3530 5832 6c6b  IiwiY2xpZW50X2lk


        0x02f0:  496a 6f69 5548 4a76 5a48 566a 6446 4276  IjoiUHJvZHVjdFBv


        0x0300:  636e 5268 6243 4973 496d 4631 5a43 4936  cnRhbCIsImF1ZCI6


        0x0310:  496c 4279 6232 5231 5933 5251 6233 4a30  IlByb2R1Y3RQb3J0


        0x0320:  5957 7769 4c43 4a34 6157 5169 4f6d 3531  YWwiLCJ4aWQiOm51


        0x0330:  6247 7773 496e 4e6a 6233 426c 496a 7062  bGwsInNjb3BlIjpb


        0x0340:  496e 4279 6233 5a70 6332 6c76 626d 6c75  InByb3Zpc2lvbmlu


        0x0350:  5a79 4a64 4c43 4a70 5a43 4936 496d 5131  ZyJdLCJpZCI6ImQ1


        0x0360:  5957 5531 5a6a 4269 4c54 4533 5954 4974  YWU1ZjBiLTE3YTIt


        0x0370:  4e47 4530 5979 3034 4e7a 5130 4c54 4178  NGE0Yy04NzQ0LTAx


        0x0380:  5957 4a6c 4d44 5933 5a54 426c 4d69 4973  YWJlMDY3ZTBlMiIs


        0x0390:  496d 5634 6343 4936 4d54 5578 4e54 6777  ImV4cCI6MTUxNTgw


        0x03a0:  4f54 6b34 4d79 7769 6157 4630 496a 6f78  OTk4MywiaWF0Ijox


        0x03b0:  4e54 4531 4f44 4134 4d54 677a 4c43 4a71  NTE1ODA4MTgzLCJq


        0x03c0:  6447 6b69 4f69 4a6b 4e57 466c 4e57 5977  dGkiOiJkNWFlNWYw


        0x03d0:  5969 3078 4e32 4579 4c54 5268 4e47 4d74  Yi0xN2EyLTRhNGMt


        0x03e0:  4f44 6330 4e43 3077 4d57 4669 5a54 4132  ODc0NC0wMWFiZTA2


        0x03f0:  4e32 5577 5a54 4969 4c43 4a6a 6157 5169  N2UwZTIiLCJjaWQi


        0x0400:  4f6d 3531 6247 7839 2e64 7856 7064 786f  Om51bGx9.dxVpdxo


        0x0410:  7a4c 6146 4a34 3449 7265 4e6d 4e4c 324c  zLaFJ44IreNmNL2L


        0x0420:  636b 3639 2d43 7933 7933 3569 4937 4768  ck69-Cy3y35iI7Gh


        0x0430:  5066 545a 5973 4759 4f48 4755 5535 524a  PfTZYsGYOHGUU5RJ


        0x0440:  4f5f 7a41 5a32 486e 4d6d 4e4d 5742 4772  O_zAZ2HnMmNMWBGr


        0x0450:  6b65 6747 6d71 2d36 3373 5933 7454 5141  kegGmq-63sY3tTQA


        0x0460:  3144 4f38 7073 6b70 386b 6677 514c 6966  1DO8pskp8kfwQLif


        0x0470:  4468 6c72 6336 7452 4c61 7969 5054 5268  Dhlrc6tRLayiPTRh


        0x0480:  514c 4e58 5f4d 6772 4b57 4452 644a 5346  QLNX_MgrKWDRdJSF


        0x0490:  4a42 5f52 2d46 615f 7861 7955 4539 4230  JB_R-Fa_xayUE9B0


        0x04a0:  6e78 7059 367a 4e42 6c69 7733 5754 4a46  nxpY6zNBliw3WTJF


        0x04b0:  5648 4166 6664 6142 6f48 426d 676c 3161  VHAffdaBoHBmgl1a


        0x04c0:  4342 4c51 4b58 6f76 5871 356c 554f 496b  CBLQKXovXq5lUOIk


        0x04d0:  7274 3878 4249 6c34 4464 4246 5a54 5050  rt8xBIl4DdBFZTPP


        0x04e0:  506e 4337 575f 6b67 4a6b 6672 4164 6b65  PnC7W_kgJkfrAdke


        0x04f0:  466d 6145 7532 7a6d 7861 6d5f 7472 5472  FmaEu2zmxam_trTr


        0x0500:  7550 4561 5069 4853 3571 7243 4959 3659  uPEaPiHS5qrCIY6Y


        0x0510:  5968 6170 4b34 6375 7468 6832 4674 3877  YhapK4cuthh2Ft8w


        0x0520:  766e 7a74 316b 3167 3872 746e 7246 3832  vnzt1k1g8rtnrF82


        0x0530:  5976 4354 6f32 7774 6c48 3643 4b71 6356  YvCTo2wtlH6CKqcV


        0x0540:  6262 6478 5655 7358 5372 6a64 4a70 6d50  bbdxVUsXSrjdJpmP


        0x0550:  6c56 616a 2d69 535a 3648 566c 6641 4122  lVaj-iSZ6HVlfAA"


        0x0560:  2c22 746f 6b65 6e5f 7479 7065 223a 2262  ,"token_type":"b


        0x0570:  6561 7265 7222 2c22 6578 7069 7265 735f  earer","expires_


        0x0580:  696e 223a 3137 3939 2c22 7363 6f70 6522  in":1799,"scope"


        0x0590:  3a22 7072 6f76 6973 696f 6e69 6e67 222c  :"provisioning",


        0x05a0:  2261 7564 223a 2250 726f 6475 6374 506f  "aud":"ProductPo


        0x05b0:  7274 616c 222c 2273 7562 223a 2250 726f  rtal","sub":"Pro


        0x05c0:  6475 6374 506f 7274 616c 220d 0a63 360d  ductPortal"..c6.


        0x05d0:  0a2c 2265 6964 223a 6e75 6c6c 2c22 7869  .,"eid":null,"xi


        0x05e0:  6422 3a6e 756c 6c2c 2263 6c69 656e 745f  d":null,"client_


        0x05f0:  7869 6422 3a6e 756c 6c2c 2269 7373 223a  xid":null,"iss":


        0x0600:  2268 7474 7073 3a2f 2f69 646d 2d64 6576  "https://idm-dev


        0x0610:  2exx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  .xxxxxxxxxxxxxxx


        0x0620:  222c 2269 6422 3a22 6435 6165 3566 3062  ","id":"d5ae5f0b


        0x0630:  2d31 3761 322d 3461 3463 2d38 3734 342d  -17a2-4a4c-8744-


        0x0640:  3031 6162 6530 3637 6530 6532 222c 2269  01abe067e0e2","i


        0x0650:  6174 223a 3135 3135 3830 3831 3833 2c22  at":1515808183,"


        0x0660:  6369 6422 3a6e 756c 6c2c 226a 7469 223a  cid":null,"jti":


        0x0670:  2264 3561 6535 6630 622d 3137 6132 2d34  "d5ae5f0b-17a2-4


        0x0680:  6134 632d 3837 3434 2d30 3161 6265 3036  a4c-8744-01abe06


        0x0690:  3765 3065 3222 7d0d 0a30 0d0a 0d0a       7e0e2"}..0....


01:49:44.229187 IP (tos 0x0, ttl 64, id 8944, offset 0, flags [DF], proto TCP (6), length 52)


    172.17.0.6.38570 > 172.17.0.5.9001: Flags [.], cksum 0x5854 (incorrect -> 0x1b37), seq 394, ack 1643, win 254, options [nop,nop,TS val 96572869 ecr 96572868], length 0


        0x0000:  4500 0034 22f0 4000 4006 bfa6 ac11 0006  E..4".@.@.......


        0x0010:  ac11 0005 96aa 2329 7e54 b885 2a19 b087  ......#)~T..*...


        0x0020:  8010 00fe 5854 0000 0101 080a 05c1 95c5  ....XT..........


        0x0030:  05c1 95c4                                ....


01:56:15.784293 IP (tos 0x0, ttl 64, id 8945, offset 0, flags [DF], proto TCP (6), length 464)


    172.17.0.6.38570 > 172.17.0.5.9001: Flags [P.], cksum 0x59f0 (incorrect -> 0x66df), seq 394:806, ack 1643, win 254, options [nop,nop,TS val 96964424 ecr 96572868], length 412


        0x0000:  4500 01d0 22f1 4000 4006 be09 ac11 0006  E...".@.@.......


        0x0010:  ac11 0005 96aa 2329 7e54 b885 2a19 b087  ......#)~T..*...


        0x0020:  8018 00fe 59f0 0000 0101 080a 05c7 8f48  ....Y..........H


        0x0030:  05c1 95c4 504f 5354 202f 7631 2f74 6f6b  ....POST./v1/tok


        0x0040:  656e 2048 5454 502f 312e 310d 0a43 6f6e  en.HTTP/1.1..Con


        0x0050:  7465 6e74 2d4c 656e 6774 683a 2031 3534  tent-Length:.154


        0x0060:  0d0a 436f 6e74 656e 742d 5479 7065 3a20  ..Content-Type:.


        0x0070:  6170 706c 6963 6174 696f 6e2f 782d 7777  application/x-ww


        0x0080:  772d 666f 726d 2d75 726c 656e 636f 6465  w-form-urlencode


        0x0090:  643b 2063 6861 7273 6574 3d55 5446 2d38  d;.charset=UTF-8


        0x00a0:  0d0a 486f 7374 3a20 6c6f 6361 6c2e 656e  ..Host:.local.en


        0x00b0:  766f 793a 3930 3031 0d0a 436f 6e6e 6563  voy:9001..Connec


        0x00c0:  7469 6f6e 3a20 4b65 6570 2d41 6c69 7665  tion:.Keep-Alive


        0x00d0:  0d0a 5573 6572 2d41 6765 6e74 3a20 5368  ..User-Agent:.Sh


        0x00e0:  6962 626f 6c65 7468 4964 702f 332e 332e  ibbolethIdp/3.3.


        0x00f0:  3220 4f70 656e 5341 4d4c 2f33 2e33 2e31  2.OpenSAML/3.3.1


        0x0100:  0d0a 4163 6365 7074 2d45 6e63 6f64 696e  ..Accept-Encodin


        0x0110:  673a 2067 7a69 702c 6465 666c 6174 650d  g:.gzip,deflate.


        0x0120:  0a43 6f6e 6e65 6374 696f 6e3a 2063 6c6f  .Connection:.clo


        0x0130:  7365 0d0a 0d0a 636c 6965 6e74 5f69 643d  se....client_id=


        0x0140:  5072 6f64 7563 7450 6f72 7461 6c26 636c  ProductPortal&cl


        0x0150:  6965 6e74 5f73 6563 7265 743d xxxx xxxx  ient_secret=xxxx


        0x0160:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0170:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0180:  xxxx xxxx xxxx xxxx xxxx xxxx 2667 7261  xxxxxxxxxxxx&gra


        0x0190:  6e74 5f74 7970 653d 7061 7373 776f 7264  nt_type=password


        0x01a0:  2673 636f 7065 3d61 7574 6826 7573 6572  &scope=auth&user


        0x01b0:  6e61 6d65 3dxx xxxx xxxx xxxx 2670 6173  name=xxxxxxx&pas


        0x01c0:  7377 6f72 643d xxxx xxxx xxxx xxxx xxxx  sword=xxxxxxxxxx


01:56:15.784331 IP (tos 0x0, ttl 64, id 15886, offset 0, flags [DF], proto TCP (6), length 52)


    172.17.0.5.9001 > 172.17.0.6.38570: Flags [.], cksum 0x5854 (incorrect -> 0x2693), seq 1643, ack 806, win 243, options [nop,nop,TS val 96964424 ecr 96964424], length 0


        0x0000:  4500 0034 3e0e 4000 4006 a488 ac11 0005  E..4>.@.@.......


        0x0010:  ac11 0006 2329 96aa 2a19 b087 7e54 ba21  ....#)..*...~T.!


        0x0020:  8010 00f3 5854 0000 0101 080a 05c7 8f48  ....XT.........H


        0x0030:  05c7 8f48                                ...H


01:56:15.791745 IP (tos 0x0, ttl 64, id 15887, offset 0, flags [DF], proto TCP (6), length 118)


    172.17.0.5.9001 > 172.17.0.6.38570: Flags [P.], cksum 0x5896 (incorrect -> 0x3c05), seq 1643:1709, ack 806, win 243, options [nop,nop,TS val 96964431 ecr 96964424], length 66


        0x0000:  4500 0076 3e0f 4000 4006 a445 ac11 0005  E..v>.@.@..E....


        0x0010:  ac11 0006 2329 96aa 2a19 b087 7e54 ba21  ....#)..*...~T.!


        0x0020:  8018 00f3 5896 0000 0101 080a 05c7 8f4f  ....X..........O


        0x0030:  05c7 8f48 4854 5450 2f31 2e31 2034 3030  ...HHTTP/1.1.400


        0x0040:  2042 6164 2052 6571 7565 7374 0d0a 636f  .Bad.Request..co


        0x0050:  6e74 656e 742d 6c65 6e67 7468 3a20 300d  ntent-length:.0.


        0x0060:  0a63 6f6e 6e65 6374 696f 6e3a 2063 6c6f  .connection:.clo


        0x0070:  7365 0d0a 0d0a                           se....


01:56:15.791841 IP (tos 0x0, ttl 64, id 8946, offset 0, flags [DF], proto TCP (6), length 52)


    172.17.0.6.38570 > 172.17.0.5.9001: Flags [.], cksum 0x5854 (incorrect -> 0x2638), seq 806, ack 1709, win 254, options [nop,nop,TS val 96964431 ecr 96964431], length 0


        0x0000:  4500 0034 22f2 4000 4006 bfa4 ac11 0006  E..4".@.@.......


        0x0010:  ac11 0005 96aa 2329 7e54 ba21 2a19 b0c9  ......#)~T.!*...


        0x0020:  8010 00fe 5854 0000 0101 080a 05c7 8f4f  ....XT.........O


        0x0030:  05c7 8f4f                                ...O


01:56:15.791885 IP (tos 0x0, ttl 64, id 15888, offset 0, flags [DF], proto TCP (6), length 52)


    172.17.0.5.9001 > 172.17.0.6.38570: Flags [F.], cksum 0x5854 (incorrect -> 0x2642), seq 1709, ack 806, win 243, options [nop,nop,TS val 96964431 ecr 96964431], length 0


        0x0000:  4500 0034 3e10 4000 4006 a486 ac11 0005  E..4>.@.@.......


        0x0010:  ac11 0006 2329 96aa 2a19 b0c9 7e54 ba21  ....#)..*...~T.!


        0x0020:  8011 00f3 5854 0000 0101 080a 05c7 8f4f  ....XT.........O


        0x0030:  05c7 8f4f                                ...O


01:56:15.832257 IP (tos 0x0, ttl 64, id 8947, offset 0, flags [DF], proto TCP (6), length 52)


    172.17.0.6.38570 > 172.17.0.5.9001: Flags [.], cksum 0x5854 (incorrect -> 0x260e), seq 806, ack 1710, win 254, options [nop,nop,TS val 96964472 ecr 96964431], length 0


        0x0000:  4500 0034 22f3 4000 4006 bfa3 ac11 0006  E..4".@.@.......


        0x0010:  ac11 0005 96aa 2329 7e54 ba21 2a19 b0ca  ......#)~T.!*...


        0x0020:  8010 00fe 5854 0000 0101 080a 05c7 8f78  ....XT.........x


        0x0030:  05c7 8f4f                                ...O


01:56:20.470448 IP (tos 0x0, ttl 64, id 17279, offset 0, flags [DF], proto TCP (6), length 60)


    172.17.0.6.39010 > 172.17.0.5.9001: Flags [S], cksum 0x585c (incorrect -> 0x4cdb), seq 3943949033, win 29200, options [mss 1460,sackOK,TS val 96969110 ecr 0,nop,wscale 7], length 0


        0x0000:  4500 003c 437f 4000 4006 9f0f ac11 0006  E..<C.@.@.......


        0x0010:  ac11 0005 9862 2329 eb13 e2e9 0000 0000  .....b#)........


        0x0020:  a002 7210 585c 0000 0204 05b4 0402 080a  ..r.X\..........


        0x0030:  05c7 a196 0000 0000 0103 0307            ............


01:56:20.470479 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)


    172.17.0.5.9001 > 172.17.0.6.39010: Flags [S.], cksum 0x585c (incorrect -> 0xd16a), seq 4241217573, ack 3943949034, win 28960, options [mss 1460,sackOK,TS val 96969110 ecr 96969110,nop,wscale 7], length 0


        0x0000:  4500 003c 0000 4000 4006 e28e ac11 0005  E..<..@.@.......


        0x0010:  ac11 0006 2329 9862 fccb d825 eb13 e2ea  ....#).b...%....


        0x0020:  a012 7120 585c 0000 0204 05b4 0402 080a  ..q.X\..........


        0x0030:  05c7 a196 05c7 a196 0103 0307            ............


01:56:20.470494 IP (tos 0x0, ttl 64, id 17280, offset 0, flags [DF], proto TCP (6), length 52)


    172.17.0.6.39010 > 172.17.0.5.9001: Flags [.], cksum 0x5854 (incorrect -> 0x7072), seq 1, ack 1, win 229, options [nop,nop,TS val 96969110 ecr 96969110], length 0


        0x0000:  4500 0034 4380 4000 4006 9f16 ac11 0006  E..4C.@.@.......


        0x0010:  ac11 0005 9862 2329 eb13 e2ea fccb d826  .....b#).......&


        0x0020:  8010 00e5 5854 0000 0101 080a 05c7 a196  ....XT..........


        0x0030:  05c7 a196                                ....


01:56:20.470845 IP (tos 0x0, ttl 64, id 17281, offset 0, flags [DF], proto TCP (6), length 464)


    172.17.0.6.39010 > 172.17.0.5.9001: Flags [P.], cksum 0x59f0 (incorrect -> 0xb5a3), seq 1:413, ack 1, win 229, options [nop,nop,TS val 96969110 ecr 96969110], length 412


        0x0000:  4500 01d0 4381 4000 4006 9d79 ac11 0006  E...C.@.@..y....


        0x0010:  ac11 0005 9862 2329 eb13 e2ea fccb d826  .....b#).......&


        0x0020:  8018 00e5 59f0 0000 0101 080a 05c7 a196  ....Y...........


        0x0030:  05c7 a196 504f 5354 202f 7631 2f74 6f6b  ....POST./v1/tok


        0x0040:  656e 2048 5454 502f 312e 310d 0a43 6f6e  en.HTTP/1.1..Con


        0x0050:  7465 6e74 2d4c 656e 6774 683a 2031 3534  tent-Length:.154


        0x0060:  0d0a 436f 6e74 656e 742d 5479 7065 3a20  ..Content-Type:.


        0x0070:  6170 706c 6963 6174 696f 6e2f 782d 7777  application/x-ww


        0x0080:  772d 666f 726d 2d75 726c 656e 636f 6465  w-form-urlencode


        0x0090:  643b 2063 6861 7273 6574 3d55 5446 2d38  d;.charset=UTF-8


        0x00a0:  0d0a 486f 7374 3a20 6c6f 6361 6c2e 656e  ..Host:.local.en


        0x00b0:  766f 793a 3930 3031 0d0a 436f 6e6e 6563  voy:9001..Connec


        0x00c0:  7469 6f6e 3a20 4b65 6570 2d41 6c69 7665  tion:.Keep-Alive


        0x00d0:  0d0a 5573 6572 2d41 6765 6e74 3a20 5368  ..User-Agent:.Sh


        0x00e0:  6962 626f 6c65 7468 4964 702f 332e 332e  ibbolethIdp/3.3.


        0x00f0:  3220 4f70 656e 5341 4d4c 2f33 2e33 2e31  2.OpenSAML/3.3.1


        0x0100:  0d0a 4163 6365 7074 2d45 6e63 6f64 696e  ..Accept-Encodin


        0x0110:  673a 2067 7a69 702c 6465 666c 6174 650d  g:.gzip,deflate.


        0x0120:  0a43 6f6e 6e65 6374 696f 6e3a 2063 6c6f  .Connection:.clo


        0x0130:  7365 0d0a 0d0a 636c 6965 6e74 5f69 643d  se....client_id=


        0x0140:  5072 6f64 7563 7450 6f72 7461 6c26 636c  ProductPortal&cl


        0x0150:  6965 6e74 5f73 6563 7265 743d xxxx xxxx  ient_secret=xxxx


        0x0160:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0170:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0180:  xxxx xxxx xxxx xxxx xxxx xxxx 2667 7261  xxxxxxxxxxxx&gra


        0x0190:  6e74 5f74 7970 653d 7061 7373 776f 7264  nt_type=password


        0x01a0:  2673 636f 7065 3d61 7574 6826 7573 6572  &scope=auth&user


        0x01b0:  6e61 6d65 3dxx xxxx xxxx xxxx 2670 6173  name=xxxxxxx&pas


        0x01c0:  7377 6f72 643d xxxx xxxx xxxx xxxx xxxx  sword=xxxxxxxxxx


01:56:20.470851 IP (tos 0x0, ttl 64, id 31152, offset 0, flags [DF], proto TCP (6), length 52)


    172.17.0.5.9001 > 172.17.0.6.39010: Flags [.], cksum 0x5854 (incorrect -> 0x6ed0), seq 1, ack 413, win 235, options [nop,nop,TS val 96969110 ecr 96969110], length 0


        0x0000:  4500 0034 79b0 4000 4006 68e6 ac11 0005  E..4y.@.@.h.....


        0x0010:  ac11 0006 2329 9862 fccb d826 eb13 e486  ....#).b...&....


        0x0020:  8010 00eb 5854 0000 0101 080a 05c7 a196  ....XT..........


        0x0030:  05c7 a196                                ....


01:56:20.899995 IP (tos 0x0, ttl 64, id 31153, offset 0, flags [DF], proto TCP (6), length 4120)


    172.17.0.5.9001 > 172.17.0.6.39010: Flags [P.], cksum 0x6838 (incorrect -> 0xd77d), seq 1:4069, ack 413, win 235, options [nop,nop,TS val 96969539 ecr 96969110], length 4068


        0x0000:  4500 1018 79b1 4000 4006 5901 ac11 0005  E...y.@.@.Y.....


        0x0010:  ac11 0006 2329 9862 fccb d826 eb13 e486  ....#).b...&....


        0x0020:  8018 00eb 6838 0000 0101 080a 05c7 a343  ....h8.........C


        0x0030:  05c7 a196 4854 5450 2f31 2e31 2032 3030  ....HTTP/1.1.200


        0x0040:  204f 4b0d 0a64 6174 653a 2053 6174 2c20  .OK..date:.Sat,.


        0x0050:  3133 204a 616e 2032 3031 3820 3031 3a35  13.Jan.2018.01:5


        0x0060:  363a 3230 2047 4d54 0d0a 636f 6e74 656e  6:20.GMT..conten


        0x0070:  742d 7479 7065 3a20 6170 706c 6963 6174  t-type:.applicat


        0x0080:  696f 6e2f 6a73 6f6e 3b63 6861 7273 6574  ion/json;charset


        0x0090:  3d55 5446 2d38 0d0a 782d 636f 6e74 656e  =UTF-8..x-conten


        0x00a0:  742d 7479 7065 2d6f 7074 696f 6e73 3a20  t-type-options:.


        0x00b0:  6e6f 736e 6966 660d 0a78 2d78 7373 2d70  nosniff..x-xss-p


        0x00c0:  726f 7465 6374 696f 6e3a 2031 3b20 6d6f  rotection:.1;.mo


        0x00d0:  6465 3d62 6c6f 636b 0d0a 6361 6368 652d  de=block..cache-


        0x00e0:  636f 6e74 726f 6c3a 206e 6f2d 6361 6368  control:.no-cach


        0x00f0:  652c 206e 6f2d 7374 6f72 652c 206d 6178  e,.no-store,.max


        0x0100:  2d61 6765 3d30 2c20 6d75 7374 2d72 6576  -age=0,.must-rev


        0x0110:  616c 6964 6174 650d 0a70 7261 676d 613a  alidate..pragma:


        0x0120:  206e 6f2d 6361 6368 650d 0a65 7870 6972  .no-cache..expir


        0x0130:  6573 3a20 300d 0a78 2d66 7261 6d65 2d6f  es:.0..x-frame-o


        0x0140:  7074 696f 6e73 3a20 4445 4e59 0d0a 782d  ptions:.DENY..x-


        0x0150:  6170 706c 6963 6174 696f 6e2d 636f 6e74  application-cont


        0x0160:  6578 743a 2065 6e74 6572 7072 6973 652e  ext:.enterprise.


        0x0170:  6964 656e 7469 7479 2d64 6576 3a64 6566  identity-dev:def


        0x0180:  6175 6c74 3a37 3032 380d 0a78 2d63 6f6e  ault:7028..x-con


        0x0190:  7465 6e74 2d74 7970 652d 6f70 7469 6f6e  tent-type-option


        0x01a0:  733a 206e 6f73 6e69 6666 0d0a 636f 6e74  s:.nosniff..cont


        0x01b0:  656e 742d 6c61 6e67 7561 6765 3a20 656e  ent-language:.en


        0x01c0:  2d55 530d 0a78 2d65 6e76 6f79 2d75 7073  -US..x-envoy-ups


        0x01d0:  7472 6561 6d2d 7365 7276 6963 652d 7469  tream-service-ti


        0x01e0:  6d65 3a20 3432 380d 0a73 6572 7665 723a  me:.428..server:


        0x01f0:  2065 6e76 6f79 0d0a 7472 616e 7366 6572  .envoy..transfer


        0x0200:  2d65 6e63 6f64 696e 673a 2063 6875 6e6b  -encoding:.chunk


        0x0210:  6564 0d0a 0d0a 6466 360d 0a7b 2261 6363  ed....df6..{"acc


        0x0220:  6573 735f 746f 6b65 6e22 3a22 6579 4a68  ess_token":"eyJh


        0x0230:  6247 6369 4f69 4a53 557a 4931 4e69 4973  bGciOiJSUzI1NiIs


        0x0240:  496e 5235 6343 4936 496b 7058 5643 4973  InR5cCI6IkpXVCIs


        0x0250:  496d 7470 5a43 4936 496a 6330 4e44 5978  ImtpZCI6Ijc0NDYx


        0x0260:  4d7a 5535 496e 302e 6579 4a7a 6457 4969  MzU5In0.eyJzdWIi


        0x0270:  4f69 4930 5a44 637a 4d7a 6731 4f43 316a  OiI0ZDczMzg1OC1j


        0x0280:  4d6a 6332 4c54 5268 5a57 4d74 596a 6b35  Mjc2LTRhZWMtYjk5


        0x0290:  4d53 3077 5a6a 6778 5932 5179 4e6a 466c  MS0wZjgxY2QyNjFl


        0x02a0:  4e6d 4969 4c43 4a7a 5a58 4e7a 6157 3975  NmIiLCJzZXNzaW9u


        0x02b0:  5833 6870 5a43 4936 496e 5679 626a 706a  X3hpZCI6InVybjpj


        0x02c0:  6232 3075 6257 686c 5a48 566a 5958 5270  b20ubWhlZHVjYXRp


        0x02d0:  6232 3475 6233 426c 626d 786c 5958 4a75  b24ub3BlbmxlYXJu


        0x02e0:  6157 356e 4f6d 5675 6447 5679 6348 4a70  aW5nOmVudGVycHJp


        0x02f0:  6332 5575 6157 526c 626e 5270 6448 6b36  c2UuaWRlbnRpdHk6


        0x0300:  5a47 5632 4c6d 6473 6232 4a68 6244 707a  ZGV2Lmdsb2JhbDpz


        0x0310:  5a58 4e7a 6157 3975 4f6d 4979 4e44 6c6b  ZXNzaW9uOmIyNDlk


        0x0320:  4e7a 5a6b 4c57 5977 4f44 4d74 4e44 5534  NzZkLWYwODMtNDU4


        0x0330:  5a53 3034 4e6a 5135 4c54 4930 4d6a 4135  ZS04NjQ5LTI0MjA5


        0x0340:  4f54 5577 4d6a 4a68 4d53 4973 496e 567a  OTUwMjJhMSIsInVz


        0x0350:  5a58 4a66 626d 4674 5a53 4936 496c 4250  ZXJfbmFtZSI6IlBP


        0x0360:  556c 5242 5445 4d69 4c43 4a70 6333 4d69  UlRBTEMiLCJpc3Mi


        0x0370:  4f69 4a6f 6448 5277 637a 6f76 4c32 6c6b  OiJodHRwczovL2lk


        0x0380:  6253 316b 5a58 5975 6257 686c 5a48 566a  bS1kZXYubWhlZHVj


        0x0390:  5958 5270 6232 3475 5932 3974 4969 7769  YXRpb24uY29tIiwi


        0x03a0:  6332 567a 6332 6c76 626c 3970 5a43 4936  c2Vzc2lvbl9pZCI6


        0x03b0:  496d 4979 4e44 6c6b 4e7a 5a6b 4c57 5977  ImIyNDlkNzZkLWYw


        0x03c0:  4f44 4d74 4e44 5534 5a53 3034 4e6a 5135  ODMtNDU4ZS04NjQ5


        0x03d0:  4c54 4930 4d6a 4135 4f54 5577 4d6a 4a68  LTI0MjA5OTUwMjJh


        0x03e0:  4d53 4973 496d 4e73 6157 5675 6446 3970  MSIsImNsaWVudF9p


        0x03f0:  5a43 4936 496c 4279 6232 5231 5933 5251  ZCI6IlByb2R1Y3RQ


        0x0400:  6233 4a30 5957 7769 4c43 4a68 6457 5169  b3J0YWwiLCJhdWQi


        0x0410:  4f69 4a51 636d 396b 6457 4e30 5547 3979  OiJQcm9kdWN0UG9y


        0x0420:  6447 4673 4969 7769 6347 5679 6332 3975  dGFsIiwicGVyc29u


        0x0430:  5833 6870 5a43 4936 496e 5679 626a 706a  X3hpZCI6InVybjpj


        0x0440:  6232 3075 6257 686c 5a48 566a 5958 5270  b20ubWhlZHVjYXRp


        0x0450:  6232 3475 6233 426c 626d 786c 5958 4a75  b24ub3BlbmxlYXJu


        0x0460:  6157 356e 4f6d 5675 6447 5679 6348 4a70  aW5nOmVudGVycHJp


        0x0470:  6332 5575 6157 526c 626e 5270 6448 6b36  c2UuaWRlbnRpdHk6


        0x0480:  5a47 5632 4c6d 6473 6232 4a68 6244 7077  ZGV2Lmdsb2JhbDpw


        0x0490:  5a58 4a7a 6232 3436 4e47 5133 4d7a 4d34  ZXJzb246NGQ3MzM4


        0x04a0:  4e54 6774 597a 4933 4e69 3030 5957 566a  NTgtYzI3Ni00YWVj


        0x04b0:  4c57 4935 4f54 4574 4d47 5934 4d57 4e6b  LWI5OTEtMGY4MWNk


        0x04c0:  4d6a 5978 5a54 5a69 4969 7769 6547 6c6b  MjYxZTZiIiwieGlk


        0x04d0:  496a 6f69 6458 4a75 4f6d 4e76 6253 3574  IjoidXJuOmNvbS5t


        0x04e0:  6147 566b 6457 4e68 6447 6c76 6269 3576  aGVkdWNhdGlvbi5v


        0x04f0:  6347 5675 6247 5668 636d 3570 626d 6336  cGVubGVhcm5pbmc6


        0x0500:  5a57 3530 5a58 4a77 636d 6c7a 5a53 3570  ZW50ZXJwcmlzZS5p


        0x0510:  5a47 5675 6447 6c30 6554 706b 5a58 5975  ZGVudGl0eTpkZXYu


        0x0520:  5a32 7876 596d 4673 4f6e 426c 636e 4e76  Z2xvYmFsOnBlcnNv


        0x0530:  626a 6f30 5a44 637a 4d7a 6731 4f43 316a  bjo0ZDczMzg1OC1j


        0x0540:  4d6a 6332 4c54 5268 5a57 4d74 596a 6b35  Mjc2LTRhZWMtYjk5


        0x0550:  4d53 3077 5a6a 6778 5932 5179 4e6a 466c  MS0wZjgxY2QyNjFl


        0x0560:  4e6d 4969 4c43 4a7a 5932 3977 5a53 4936  NmIiLCJzY29wZSI6


        0x0570:  5779 4a68 6458 526f 496c 3073 496d 6c6b  WyJhdXRoIl0sImlk


        0x0580:  496a 6f69 5a44 6330 4f44 5133 5a47 5174  IjoiZDc0ODQ3ZGQt


        0x0590:  596a 5534 4e69 3030 4e54 6333 4c57 4932  YjU4Ni00NTc3LWI2


        0x05a0:  4e6a 4974 4f44 6735 4f54 426b 4e7a 5a6c  NjItODg5OTBkNzZl


        0x05b0:  5a57 5531 4969 7769 5a58 6877 496a 6f78  ZWU1IiwiZXhwIjox


        0x05c0:  4e54 4531 4f44 4577 4d7a 6777 4c43 4a70  NTE1ODEwMzgwLCJp


        0x05d0:  5958 5169 4f6a 4531 4d54 5534 4d44 6731  YXQiOjE1MTU4MDg1


        0x05e0:  4f44 4173 496d 7030 6153 4936 496d 5133  ODAsImp0aSI6ImQ3


        0x05f0:  4e44 6730 4e32 526b 4c57 4931 4f44 5974  NDg0N2RkLWI1ODYt


        0x0600:  4e44 5533 4e79 3169 4e6a 5979 4c54 6734  NDU3Ny1iNjYyLTg4


        0x0610:  4f54 6b77 5a44 6332 5a57 566c 4e53 4973  OTkwZDc2ZWVlNSIs


        0x0620:  496d 4e70 5a43 4936 496c 4279 6232 5231  ImNpZCI6IlByb2R1


        0x0630:  5933 5251 6233 4a30 5957 7769 6651 2e4e  Y3RQb3J0YWwifQ.N


        0x0640:  7849 5648 626f 7566 744f 5665 4f4e 504c  xIVHbouftOVeONPL


        0x0650:  3679 3270 4a69 3579 5659 4853 5a49 5975  6y2pJi5yVYHSZIYu


        0x0660:  5f78 3772 616f 666f 3778 5241 3068 5f65  _x7raofo7xRA0h_e


        0x0670:  6a7a 6e37 3730 6134 3855 5445 5f4a 6756  jzn770a48UTE_JgV


        0x0680:  5078 7348 3147 3651 7769 4d57 5968 6d43  PxsH1G6QwiMWYhmC


        0x0690:  6258 5354 6f69 5873 7855 4a45 6430 6951  bXSToiXsxUJEd0iQ


        0x06a0:  5468 7573 736d 6859 6b62 3670 3850 6165  ThussmhYkb6p8Pae


        0x06b0:  4733 5f64 7536 475a 6a57 4271 6956 4c4a  G3_du6GZjWBqiVLJ


        0x06c0:  774c 5249 4645 4c4d 4b33 7263 5a56 4a42  wLRIFELMK3rcZVJB


        0x06d0:  525a 744c 5647 5644 5473 6639 5669 4251  RZtLVGVDTsf9ViBQ


        0x06e0:  6338 7435 3153 6c52 656e 644f 4945 4c5f  c8t51SlRendOIEL_


        0x06f0:  6347 4335 4a50 584a 6746 7154 6678 4946  cGC5JPXJgFqTfxIF


        0x0700:  615f 3734 6864 5674 5844 6661 6d5f 4374  a_74hdVtXDfam_Ct


        0x0710:  4273 4e6b 6866 7577 464f 394d 6555 564c  BsNkhfuwFO9MeUVL


        0x0720:  6952 4666 6d6f 3051 6b44 7731 756a 6b7a  iRFfmo0QkDw1ujkz


        0x0730:  7279 4e57 517a 6c57 6664 5a4d 6d6b 2d6a  ryNWQzlWfdZMmk-j


        0x0740:  4c39 5949 5572 4945 4836 5f66 6954 6c4c  L9YIUrIEH6_fiTlL


        0x0750:  6d6e 6b50 527a 4569 6569 4568 5f42 6479  mnkPRzEieiEh_Bdy


        0x0760:  6d67 4633 4978 4950 6a59 3051 2d7a 5251  mgF3IxIPjY0Q-zRQ


        0x0770:  5249 6574 7342 6d6b 714e 4841 3544 5276  RIetsBmkqNHA5DRv


        0x0780:  4732 6749 7367 7578 786f 7436 6138 5f42  G2gIsguxxot6a8_B


        0x0790:  714c 7578 7722 2c22 746f 6b65 6e5f 7479  qLuxw","token_ty


        0x07a0:  7065 223a 2262 6561 7265 7222 2c22 7265  pe":"bearer","re


        0x07b0:  6672 6573 685f 746f 6b65 6e22 3a22 6579  fresh_token":"ey


        0x07c0:  4a68 6247 6369 4f69 4a53 557a 4931 4e69  JhbGciOiJSUzI1Ni


        0x07d0:  4973 496e 5235 6343 4936 496b 7058 5643  IsInR5cCI6IkpXVC


        0x07e0:  4973 496d 7470 5a43 4936 496a 6330 4e44  IsImtpZCI6Ijc0ND


        0x07f0:  5978 4d7a 5535 496e 302e 6579 4a7a 6457  YxMzU5In0.eyJzdW


        0x0800:  4969 4f69 4930 5a44 637a 4d7a 6731 4f43  IiOiI0ZDczMzg1OC


        0x0810:  316a 4d6a 6332 4c54 5268 5a57 4d74 596a  1jMjc2LTRhZWMtYj


        0x0820:  6b35 4d53 3077 5a6a 6778 5932 5179 4e6a  k5MS0wZjgxY2QyNj


        0x0830:  466c 4e6d 4969 4c43 4a7a 5a58 4e7a 6157  FlNmIiLCJzZXNzaW


        0x0840:  3975 5833 6870 5a43 4936 496e 5679 626a  9uX3hpZCI6InVybj


        0x0850:  706a 6232 3075 6257 686c 5a48 566a 5958  pjb20ubWhlZHVjYX


        0x0860:  5270 6232 3475 6233 426c 626d 786c 5958  Rpb24ub3BlbmxlYX


        0x0870:  4a75 6157 356e 4f6d 5675 6447 5679 6348  JuaW5nOmVudGVycH


        0x0880:  4a70 6332 5575 6157 526c 626e 5270 6448  Jpc2UuaWRlbnRpdH


        0x0890:  6b36 5a47 5632 4c6d 6473 6232 4a68 6244  k6ZGV2Lmdsb2JhbD


        0x08a0:  707a 5a58 4e7a 6157 3975 4f6d 4979 4e44  pzZXNzaW9uOmIyND


        0x08b0:  6c6b 4e7a 5a6b 4c57 5977 4f44 4d74 4e44  lkNzZkLWYwODMtND


        0x08c0:  5534 5a53 3034 4e6a 5135 4c54 4930 4d6a  U4ZS04NjQ5LTI0Mj


        0x08d0:  4135 4f54 5577 4d6a 4a68 4d53 4973 496e  A5OTUwMjJhMSIsIn


        0x08e0:  567a 5a58 4a66 626d 4674 5a53 4936 496c  VzZXJfbmFtZSI6Il


        0x08f0:  4250 556c 5242 5445 4d69 4c43 4a70 6333  BPUlRBTEMiLCJpc3


        0x0900:  4d69 4f69 4a6f 6448 5277 637a 6f76 4c32  MiOiJodHRwczovL2


        0x0910:  6c6b 6253 316b 5a58 5975 6257 686c 5a48  lkbS1kZXYubWhlZH


        0x0920:  566a 5958 5270 6232 3475 5932 3974 4969  VjYXRpb24uY29tIi


        0x0930:  7769 6332 567a 6332 6c76 626c 3970 5a43  wic2Vzc2lvbl9pZC


        0x0940:  4936 496d 4979 4e44 6c6b 4e7a 5a6b 4c57  I6ImIyNDlkNzZkLW


        0x0950:  5977 4f44 4d74 4e44 5534 5a53 3034 4e6a  YwODMtNDU4ZS04Nj


        0x0960:  5135 4c54 4930 4d6a 4135 4f54 5577 4d6a  Q5LTI0MjA5OTUwMj


        0x0970:  4a68 4d53 4973 496d 4e73 6157 5675 6446  JhMSIsImNsaWVudF


        0x0980:  3970 5a43 4936 496c 4279 6232 5231 5933  9pZCI6IlByb2R1Y3


        0x0990:  5251 6233 4a30 5957 7769 4c43 4a68 6457  RQb3J0YWwiLCJhdW


        0x09a0:  5169 4f69 4a51 636d 396b 6457 4e30 5547  QiOiJQcm9kdWN0UG


        0x09b0:  3979 6447 4673 4969 7769 6347 5679 6332  9ydGFsIiwicGVyc2


        0x09c0:  3975 5833 6870 5a43 4936 496e 5679 626a  9uX3hpZCI6InVybj


        0x09d0:  706a 6232 3075 6257 686c 5a48 566a 5958  pjb20ubWhlZHVjYX


        0x09e0:  5270 6232 3475 6233 426c 626d 786c 5958  Rpb24ub3BlbmxlYX


        0x09f0:  4a75 6157 356e 4f6d 5675 6447 5679 6348  JuaW5nOmVudGVycH


        0x0a00:  4a70 6332 5575 6157 526c 626e 5270 6448  Jpc2UuaWRlbnRpdH


        0x0a10:  6b36 5a47 5632 4c6d 6473 6232 4a68 6244  k6ZGV2Lmdsb2JhbD


        0x0a20:  7077 5a58 4a7a 6232 3436 4e47 5133 4d7a  pwZXJzb246NGQ3Mz


        0x0a30:  4d34 4e54 6774 597a 4933 4e69 3030 5957  M4NTgtYzI3Ni00YW


        0x0a40:  566a 4c57 4935 4f54 4574 4d47 5934 4d57  VjLWI5OTEtMGY4MW


        0x0a50:  4e6b 4d6a 5978 5a54 5a69 4969 7769 6547  NkMjYxZTZiIiwieG


        0x0a60:  6c6b 496a 6f69 6458 4a75 4f6d 4e76 6253  lkIjoidXJuOmNvbS


        0x0a70:  3574 6147 566b 6457 4e68 6447 6c76 6269  5taGVkdWNhdGlvbi


        0x0a80:  3576 6347 5675 6247 5668 636d 3570 626d  5vcGVubGVhcm5pbm


        0x0a90:  6336 5a57 3530 5a58 4a77 636d 6c7a 5a53  c6ZW50ZXJwcmlzZS


        0x0aa0:  3570 5a47 5675 6447 6c30 6554 706b 5a58  5pZGVudGl0eTpkZX


        0x0ab0:  5975 5a32 7876 596d 4673 4f6e 426c 636e  YuZ2xvYmFsOnBlcn


        0x0ac0:  4e76 626a 6f30 5a44 637a 4d7a 6731 4f43  Nvbjo0ZDczMzg1OC


        0x0ad0:  316a 4d6a 6332 4c54 5268 5a57 4d74 596a  1jMjc2LTRhZWMtYj


        0x0ae0:  6b35 4d53 3077 5a6a 6778 5932 5179 4e6a  k5MS0wZjgxY2QyNj


        0x0af0:  466c 4e6d 4969 4c43 4a7a 5932 3977 5a53  FlNmIiLCJzY29wZS


        0x0b00:  4936 5779 4a68 6458 526f 496c 3073 496d  I6WyJhdXRoIl0sIm


        0x0b10:  4630 6153 4936 496d 5133 4e44 6730 4e32  F0aSI6ImQ3NDg0N2


        0x0b20:  526b 4c57 4931 4f44 5974 4e44 5533 4e79  RkLWI1ODYtNDU3Ny


        0x0b30:  3169 4e6a 5979 4c54 6734 4f54 6b77 5a44  1iNjYyLTg4OTkwZD


        0x0b40:  6332 5a57 566c 4e53 4973 496d 6c6b 496a  c2ZWVlNSIsImlkIj


        0x0b50:  6f69 5a44 6330 4f44 5133 5a47 5174 596a  oiZDc0ODQ3ZGQtYj


        0x0b60:  5534 4e69 3030 4e54 6333 4c57 4932 4e6a  U4Ni00NTc3LWI2Nj


        0x0b70:  4974 4f44 6735 4f54 426b 4e7a 5a6c 5a57  ItODg5OTBkNzZlZW


        0x0b80:  5531 4969 7769 5a58 6877 496a 6f78 4e54  U1IiwiZXhwIjoxNT


        0x0b90:  4534 4e44 4177 4e54 6777 4c43 4a70 5958  E4NDAwNTgwLCJpYX


        0x0ba0:  5169 4f6a 4531 4d54 5534 4d44 6731 4f44  QiOjE1MTU4MDg1OD


        0x0bb0:  4173 496d 7030 6153 4936 496d 5131 4d7a  AsImp0aSI6ImQ1Mz


        0x0bc0:  5532 4d7a 6734 4c54 6c6a 4d47 4d74 4e47  U2Mzg4LTljMGMtNG


        0x0bd0:  5531 4d79 3034 597a 6730 4c54 6379 4e32  U1My04Yzg0LTcyN2


        0x0be0:  4977 4d32 5134 4d44 566b 4e43 4973 496d  IwM2Q4MDVkNCIsIm


        0x0bf0:  4e70 5a43 4936 496c 4279 6232 5231 5933  NpZCI6IlByb2R1Y3


        0x0c00:  5251 6233 4a30 5957 7769 6651 2e47 6765  RQb3J0YWwifQ.Gge


        0x0c10:  5155 5949 6d56 7842 4249 4e50 335f 6571  QUYImVxBBINP3_eq


        0x0c20:  6572 666f 3859 4d33 6744 4f59 3444 4b52  erfo8YM3gDOY4DKR


        0x0c30:  3132 4c70 6b6b 6d6d 6b62 7864 7870 3469  12Lpkkmmkbxdxp4i


        0x0c40:  7730 4f63 5a47 7951 5939 3953 3368 6156  w0OcZGyQY99S3haV


        0x0c50:  6265 4957 7a77 5769 6f44 4461 665a 4e6c  beIWzwWioDDafZNl


        0x0c60:  4c36 6c6f 6343 4a30 4274 7a2d 524d 687a  L6locCJ0Btz-RMhz


        0x0c70:  4949 756c 4b31 6742 7671 466f 7257 2d6c  IIulK1gBvqForW-l


        0x0c80:  6545 4d69 4179 7439 3563 3065 3645 7571  eEMiAyt95c0e6Euq


        0x0c90:  4634 3132 4b36 344d 5177 356e 6357 4759  F412K64MQw5ncWGY


        0x0ca0:  365a 456b 3242 316e 4850 6844 4a79 6248  6ZEk2B1nHPhDJybH


        0x0cb0:  5237 664e 6a48 6873 6e36 4734 7436 5635  R7fNjHhsn6G4t6V5


        0x0cc0:  7938 716c 7455 3252 3342 336a 3659 3433  y8qltU2R3B3j6Y43


        0x0cd0:  3552 756b 416c 6377 6752 4831 7364 774f  5RukAlcwgRH1sdwO


        0x0ce0:  4c6a 5057 5754 664a 4167 3957 7159 7142  LjPWWTfJAg9WqYqB


        0x0cf0:  4f5a 7a35 314c 7a74 3454 6c33 7759 485a  OZz51Lzt4Tl3wYHZ


        0x0d00:  4d47 5437 6268 6e79 6651 7154 4238 5750  MGT7bhnyfQqTB8WP


        0x0d10:  7533 734a 746e 6456 3845 3058 6554 6563  u3sJtndV8E0XeTec


        0x0d20:  754f 3768 3156 6e4b 5763 5369 6841 4732  uO7h1VnKWcSihAG2


        0x0d30:  7032 7159 3778 4d66 4849 5370 4b44 5939  p2qY7xMfHISpKDY9


        0x0d40:  3549 6746 6f6c 4341 4334 4e75 4439 5377  5IgFolCAC4NuD9Sw


        0x0d50:  3039 7248 5875 6335 4e66 6d7a 3673 6243  09rHXuc5Nfmz6sbC


        0x0d60:  4d73 6722 2c22 6578 7069 7265 735f 696e  Msg","expires_in


        0x0d70:  223a 3137 3939 2c22 7363 6f70 6522 3a22  ":1799,"scope":"


        0x0d80:  6175 7468 222c 2261 7564 223a 2250 726f  auth","aud":"Pro


        0x0d90:  6475 6374 506f 7274 616c 222c 2273 7562  ductPortal","sub


        0x0da0:  223a 2234 6437 3333 3835 382d 6332 3736  ":"4d733858-c276


        0x0db0:  2d34 6165 632d 6239 3931 2d30 6638 3163  -4aec-b991-0f81c


        0x0dc0:  6432 3631 6536 6222 2c22 7065 7273 6f6e  d261e6b","person


        0x0dd0:  5f78 6964 223a 2275 726e 3axx xxxx xxxx  _xid":"urn:xxxxx


        0x0de0:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0df0:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0e00:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0e10:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0e20:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0e30:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0e40:  xxxx 222c 2278 6964 223a 22xx xxxx xxxx  xx","xid":"xxxxx


        0x0e50:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0e60:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0e70:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0e80:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0e90:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0ea0:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0eb0:  xxxx xxxx xxxx 222c 2273 6573 7369 6f6e  xxxxxx","session


        0x0ec0:  5f78 6964 223a 2275 726e 3axx xxxx xxxx  _xid":"urn:xxxxx


        0x0ed0:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0ee0:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0ef0:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0f00:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0f10:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0f20:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0f30:  xxxx xx22 2c22 6973 7322 3a22 6874 7470  xxx","iss":"http


        0x0f40:  733a 2f2f 6964 6d2d 6465 762e xxxx xxxx  s://idm-dev.xxxx


        0x0f50:  xxxx xxxx xxxx xxxx xxxx xx22 2c22 7365  xxxxxxxxxxx","se


        0x0f60:  7373 696f 6e5f 6964 223a 2262 3234 3964  ssion_id":"b249d


        0x0f70:  3736 642d 6630 3833 2d34 3538 652d 3836  76d-f083-458e-86


        0x0f80:  3439 2d32 3432 3039 3935 3032 3261 3122  49-2420995022a1"


        0x0f90:  2c22 6964 223a 2264 3734 3834 3764 642d  ,"id":"d74847dd-


        0x0fa0:  6235 3836 2d34 3537 372d 6236 3632 2d38  b586-4577-b662-8


        0x0fb0:  3839 3930 6437 3665 6565 3522 2c22 6961  8990d76eee5","ia


        0x0fc0:  7422 3a31 3531 3538 3038 3538 302c 2263  t":1515808580,"c


        0x0fd0:  6964 223a 2250 726f 6475 6374 506f 7274  id":"ProductPort


        0x0fe0:  616c 222c 226a 7469 223a 2264 3734 3834  al","jti":"d7484


        0x0ff0:  3764 642d 6235 3836 2d34 3537 372d 6236  7dd-b586-4577-b6


        0x1000:  3632 2d38 3839 3930 6437 3665 6565 3522  62-88990d76eee5"


        0x1010:  7d0d 0a30 0d0a 0d0a                      }..0....


01:56:20.900055 IP (tos 0x0, ttl 64, id 17282, offset 0, flags [DF], proto TCP (6), length 52)


    172.17.0.6.39010 > 172.17.0.5.9001: Flags [.], cksum 0x5854 (incorrect -> 0x5b59), seq 413, ack 4069, win 292, options [nop,nop,TS val 96969539 ecr 96969539], length 0


        0x0000:  4500 0034 4382 4000 4006 9f14 ac11 0006  E..4C.@.@.......


        0x0010:  ac11 0005 9862 2329 eb13 e486 fccb e80a  .....b#)........


        0x0020:  8010 0124 5854 0000 0101 080a 05c7 a343  ...$XT.........C


        0x0030:  05c7 a343                                ...C


01:56:21.674174 IP (tos 0x0, ttl 64, id 17283, offset 0, flags [DF], proto TCP (6), length 1175)


    172.17.0.6.39010 > 172.17.0.5.9001: Flags [P.], cksum 0x5cb7 (incorrect -> 0x6048), seq 413:1536, ack 4069, win 292, options [nop,nop,TS val 96970313 ecr 96969539], length 1123


        0x0000:  4500 0497 4383 4000 4006 9ab0 ac11 0006  E...C.@.@.......


        0x0010:  ac11 0005 9862 2329 eb13 e486 fccb e80a  .....b#)........


        0x0020:  8018 0124 5cb7 0000 0101 080a 05c7 a649  ...$\..........I


        0x0030:  05c7 a343 4745 5420 2f76 312f 5573 6572  ...CGET./v1/User


        0x0040:  732f 7572 6e3a xxxx xxxx xxxx xxxx xxxx  s/urn:xxxxxxxxxx


        0x0050:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0060:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0070:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0080:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x0090:  xxxx xxxx xxxx xxxx xxxx xxxx xxxx xxxx  xxxxxxxxxxxxxxxx


        0x00a0:  xxxx xxxx xxxx xxxx xxxx xxxx xx20 4854  xxxxxxxxxxxxx.HT


        0x00b0:  5450 2f31 2e31 0d0a 4175 7468 6f72 697a  TP/1.1..Authoriz


        0x00c0:  6174 696f 6e3a 2042 6561 7265 7220 6579  ation:.Bearer.ey


        0x00d0:  4a68 6247 6369 4f69 4a53 557a 4931 4e69  JhbGciOiJSUzI1Ni


        0x00e0:  4973 496e 5235 6343 4936 496b 7058 5643  IsInR5cCI6IkpXVC


        0x00f0:  4973 496d 7470 5a43 4936 496a 6330 4e44  IsImtpZCI6Ijc0ND


        0x0100:  5978 4d7a 5535 496e 302e 6579 4a7a 6457  YxMzU5In0.eyJzdW


        0x0110:  4969 4f69 4a51 636d 396b 6457 4e30 5547  IiOiJQcm9kdWN0UG


        0x0120:  3979 6447 4673 4969 7769 5a57 6c6b 496a  9ydGFsIiwiZWlkIj


        0x0130:  7075 6457 7873 4c43 4a6a 6247 6c6c 626e  pudWxsLCJjbGllbn


        0x0140:  5266 6547 6c6b 496a 7075 6457 7873 4c43  RfeGlkIjpudWxsLC


        0x0150:  4a70 6333 4d69 4f69 4a6f 6448 5277 637a  Jpc3MiOiJodHRwcz


        0x0160:  6f76 4c32 6c6b 6253 316b 5a58 5975 6257  ovL2lkbS1kZXYubW


        0x0170:  686c 5a48 566a 5958 5270 6232 3475 5932  hlZHVjYXRpb24uY2


        0x0180:  3974 4969 7769 5932 7870 5a57 3530 5832  9tIiwiY2xpZW50X2


        0x0190:  6c6b 496a 6f69 5548 4a76 5a48 566a 6446  lkIjoiUHJvZHVjdF


        0x01a0:  4276 636e 5268 6243 4973 496d 4631 5a43  BvcnRhbCIsImF1ZC


        0x01b0:  4936 496c 4279 6232 5231 5933 5251 6233  I6IlByb2R1Y3RQb3


        0x01c0:  4a30 5957 7769 4c43 4a34 6157 5169 4f6d  J0YWwiLCJ4aWQiOm


        0x01d0:  3531 6247 7773 496e 4e6a 6233 426c 496a  51bGwsInNjb3BlIj


        0x01e0:  7062 496e 4279 6233 5a70 6332 6c76 626d  pbInByb3Zpc2lvbm


        0x01f0:  6c75 5a79 4a64 4c43 4a70 5a43 4936 496d  luZyJdLCJpZCI6Im


        0x0200:  5131 5957 5531 5a6a 4269 4c54 4533 5954  Q1YWU1ZjBiLTE3YT


        0x0210:  4974 4e47 4530 5979 3034 4e7a 5130 4c54  ItNGE0Yy04NzQ0LT


        0x0220:  4178 5957 4a6c 4d44 5933 5a54 426c 4d69  AxYWJlMDY3ZTBlMi


        0x0230:  4973 496d 5634 6343 4936 4d54 5578 4e54  IsImV4cCI6MTUxNT


        0x0240:  6777 4f54 6b34 4d79 7769 6157 4630 496a  gwOTk4MywiaWF0Ij


        0x0250:  6f78 4e54 4531 4f44 4134 4d54 677a 4c43  oxNTE1ODA4MTgzLC


        0x0260:  4a71 6447 6b69 4f69 4a6b 4e57 466c 4e57  JqdGkiOiJkNWFlNW


        0x0270:  5977 5969 3078 4e32 4579 4c54 5268 4e47  YwYi0xN2EyLTRhNG


        0x0280:  4d74 4f44 6330 4e43 3077 4d57 4669 5a54  MtODc0NC0wMWFiZT


        0x0290:  4132 4e32 5577 5a54 4969 4c43 4a6a 6157  A2N2UwZTIiLCJjaW


        0x02a0:  5169 4f6d 3531 6247 7839 2e64 7856 7064  QiOm51bGx9.dxVpd


        0x02b0:  786f 7a4c 6146 4a34 3449 7265 4e6d 4e4c  xozLaFJ44IreNmNL


        0x02c0:  324c 636b 3639 2d43 7933 7933 3569 4937  2Lck69-Cy3y35iI7


        0x02d0:  4768 5066 545a 5973 4759 4f48 4755 5535  GhPfTZYsGYOHGUU5


        0x02e0:  524a 4f5f 7a41 5a32 486e 4d6d 4e4d 5742  RJO_zAZ2HnMmNMWB


        0x02f0:  4772 6b65 6747 6d71 2d36 3373 5933 7454  GrkegGmq-63sY3tT


        0x0300:  5141 3144 4f38 7073 6b70 386b 6677 514c  QA1DO8pskp8kfwQL


        0x0310:  6966 4468 6c72 6336 7452 4c61 7969 5054  ifDhlrc6tRLayiPT


        0x0320:  5268 514c 4e58 5f4d 6772 4b57 4452 644a  RhQLNX_MgrKWDRdJ


        0x0330:  5346 4a42 5f52 2d46 615f 7861 7955 4539  SFJB_R-Fa_xayUE9


        0x0340:  4230 6e78 7059 367a 4e42 6c69 7733 5754  B0nxpY6zNBliw3WT


        0x0350:  4a46 5648 4166 6664 6142 6f48 426d 676c  JFVHAffdaBoHBmgl


        0x0360:  3161 4342 4c51 4b58 6f76 5871 356c 554f  1aCBLQKXovXq5lUO


        0x0370:  496b 7274 3878 4249 6c34 4464 4246 5a54  Ikrt8xBIl4DdBFZT


        0x0380:  5050 506e 4337 575f 6b67 4a6b 6672 4164  PPPnC7W_kgJkfrAd


        0x0390:  6b65 466d 6145 7532 7a6d 7861 6d5f 7472  keFmaEu2zmxam_tr


        0x03a0:  5472 7550 4561 5069 4853 3571 7243 4959  TruPEaPiHS5qrCIY


        0x03b0:  3659 5968 6170 4b34 6375 7468 6832 4674  6YYhapK4cuthh2Ft


        0x03c0:  3877 766e 7a74 316b 3167 3872 746e 7246  8wvnzt1k1g8rtnrF


        0x03d0:  3832 5976 4354 6f32 7774 6c48 3643 4b71  82YvCTo2wtlH6CKq


        0x03e0:  6356 6262 6478 5655 7358 5372 6a64 4a70  cVbbdxVUsXSrjdJp


        0x03f0:  6d50 6c56 616a 2d69 535a 3648 566c 6641  mPlVaj-iSZ6HVlfA


        0x0400:  410d 0a48 6f73 743a 206c 6f63 616c 2e65  A..Host:.local.e


        0x0410:  6e76 6f79 3a39 3030 310d 0a43 6f6e 6e65  nvoy:9001..Conne


        0x0420:  6374 696f 6e3a 204b 6565 702d 416c 6976  ction:.Keep-Aliv


        0x0430:  650d 0a55 7365 722d 4167 656e 743a 2053  e..User-Agent:.S


        0x0440:  6869 6262 6f6c 6574 6849 6470 2f33 2e33  hibbolethIdp/3.3


        0x0450:  2e32 204f 7065 6e53 414d 4c2f 332e 332e  .2.OpenSAML/3.3.


        0x0460:  310d 0a41 6363 6570 742d 456e 636f 6469  1..Accept-Encodi


        0x0470:  6e67 3a20 677a 6970 2c64 6566 6c61 7465  ng:.gzip,deflate


        0x0480:  0d0a 436f 6e6e 6563 7469 6f6e 3a20 636c  ..Connection:.cl


        0x0490:  6f73 650d 0a0d 0a                        ose....


01:56:21.674208 IP (tos 0x0, ttl 64, id 31156, offset 0, flags [DF], proto TCP (6), length 52)


    172.17.0.5.9001 > 172.17.0.6.39010: Flags [.], cksum 0x5854 (incorrect -> 0x5110), seq 4069, ack 1536, win 253, options [nop,nop,TS val 96970314 ecr 96970313], length 0


        0x0000:  4500 0034 79b4 4000 4006 68e2 ac11 0005  E..4y.@.@.h.....


        0x0010:  ac11 0006 2329 9862 fccb e80a eb13 e8e9  ....#).b........


        0x0020:  8010 00fd 5854 0000 0101 080a 05c7 a64a  ....XT.........J


        0x0030:  05c7 a649                                ...I


01:56:21.674513 IP (tos 0x0, ttl 64, id 31157, offset 0, flags [DF], proto TCP (6), length 118)


    172.17.0.5.9001 > 172.17.0.6.39010: Flags [P.], cksum 0x5896 (incorrect -> 0x6689), seq 4069:4135, ack 1536, win 253, options [nop,nop,TS val 96970314 ecr 96970313], length 66


        0x0000:  4500 0076 79b5 4000 4006 689f ac11 0005  E..vy.@.@.h.....


        0x0010:  ac11 0006 2329 9862 fccb e80a eb13 e8e9  ....#).b........


        0x0020:  8018 00fd 5896 0000 0101 080a 05c7 a64a  ....X..........J


        0x0030:  05c7 a649 4854 5450 2f31 2e31 2034 3030  ...IHTTP/1.1.400


        0x0040:  2042 6164 2052 6571 7565 7374 0d0a 636f  .Bad.Request..co


        0x0050:  6e74 656e 742d 6c65 6e67 7468 3a20 300d  ntent-length:.0.


        0x0060:  0a63 6f6e 6e65 6374 696f 6e3a 2063 6c6f  .connection:.clo


        0x0070:  7365 0d0a 0d0a                           se....


01:56:21.674546 IP (tos 0x0, ttl 64, id 17284, offset 0, flags [DF], proto TCP (6), length 52)


    172.17.0.6.39010 > 172.17.0.5.9001: Flags [.], cksum 0x5854 (incorrect -> 0x50a6), seq 1536, ack 4135, win 292, options [nop,nop,TS val 96970314 ecr 96970314], length 0


        0x0000:  4500 0034 4384 4000 4006 9f12 ac11 0006  E..4C.@.@.......


        0x0010:  ac11 0005 9862 2329 eb13 e8e9 fccb e84c  .....b#).......L


        0x0020:  8010 0124 5854 0000 0101 080a 05c7 a64a  ...$XT.........J


        0x0030:  05c7 a64a                                ...J


01:56:21.674801 IP (tos 0x0, ttl 64, id 31158, offset 0, flags [DF], proto TCP (6), length 52)


    172.17.0.5.9001 > 172.17.0.6.39010: Flags [F.], cksum 0x5854 (incorrect -> 0x50cc), seq 4135, ack 1536, win 253, options [nop,nop,TS val 96970314 ecr 96970314], length 0


        0x0000:  4500 0034 79b6 4000 4006 68e0 ac11 0005  E..4y.@.@.h.....


        0x0010:  ac11 0006 2329 9862 fccb e84c eb13 e8e9  ....#).b...L....


        0x0020:  8011 00fd 5854 0000 0101 080a 05c7 a64a  ....XT.........J


        0x0030:  05c7 a64a                                ...J


01:56:21.715920 IP (tos 0x0, ttl 64, id 17285, offset 0, flags [DF], proto TCP (6), length 52)


    172.17.0.6.39010 > 172.17.0.5.9001: Flags [.], cksum 0x5854 (incorrect -> 0x507c), seq 1536, ack 4136, win 292, options [nop,nop,TS val 96970355 ecr 96970314], length 0


        0x0000:  4500 0034 4385 4000 4006 9f11 ac11 0006  E..4C.@.@.......


        0x0010:  ac11 0005 9862 2329 eb13 e8e9 fccb e84d  .....b#).......M


        0x0020:  8010 0124 5854 0000 0101 080a 05c7 a673  ...$XT.........s


        0x0030:  05c7 a64a                                ...J




On Friday, January 12, 2018 at 4:07:00 PM UTC-5, Matt Klein wrote:
The issue is on the downstream side w/ your client, but I'm not really sure what the issue is. If possible, can you use plain text and dump the actual HTTP plain text (including accurate white space) using tcpdump/wireshark, and paste it?

Matt Klein

unread,
Jan 12, 2018, 10:33:30 PM1/12/18
to Bob, envoy-users
Your client is sending both "connection: keep-alive" as well as "connection: close". This is causing Envoy's close detection logic to not work correctly for reasons I won't get into.

Technically, this is a broken client. Can you fix the client? If not, please open an issue and I will try to figure out some way to deal with this sanely.

Thanks,
Matt

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

bob_a...@mcgraw-hill.com

unread,
Jan 19, 2018, 10:43:51 AM1/19/18
to envoy-users
I'm not completely convinced the client is broken:
  • This code works correctly (for me and thousands of other Shibboleth installations) if I do not use Envoy
  • RFC 2616 section 14.10 indicates that the Connection header is a comma-separated list of the names of headers that a proxy must strip before forwarding and that the option "close" may be used to indicate that the connection should be closed, but the RFC does not specify that the close option has to be alone
  • RFC 2616 section 4.2 indicates that headers may be repeated if the value is a comma-separated list with the final list built by concatenating the lists together

Matt Klein

unread,
Jan 19, 2018, 12:23:41 PM1/19/18
to Bob, envoy-users
I agree with your reading of how headers are parsed, but your client is doing two strange things (1 thing that is arguably wrong and 1 thing that is definitely wrong):

1) Sending "connection: keep-alive,close" (either comma delimited or split into two headers) doesn't make any sense. a) keep-alive only applied to HTTP/1.0, but b) they are contradictory. 

2) The client is reusing the connection after it has set "connection: close". This is definitely incorrect as specified here: https://tools.ietf.org/html/rfc7230#section-6.1. Envoy detecting "close" and immediately closing the connection is a convenience, the client should not be depending on it (this is actually where the 400 is coming from).

The only workaround I can think of right now is for you to set a very low idle timeout on the HTTP connection manager such that Envoy will idle close incoming connections, but I really would recommend fixing the client. 

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users+unsubscribe@googlegroups.com.
To post to this group, send email to envoy...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Irvi Aini

unread,
Aug 24, 2020, 11:47:45 AM8/24/20
to envoy-users
+Matt

I find the same condition which causing 503 in envoy v.114.2 with sni tls context. Any idea why?

On Friday, January 19, 2018 at 6:23:41 PM UTC+1 Matt Klein wrote:
I agree with your reading of how headers are parsed, but your client is doing two strange things (1 thing that is arguably wrong and 1 thing that is definitely wrong):

1) Sending "connection: keep-alive,close" (either comma delimited or split into two headers) doesn't make any sense. a) keep-alive only applied to HTTP/1.0, but b) they are contradictory. 

2) The client is reusing the connection after it has set "connection: close". This is definitely incorrect as specified here: https://tools.ietf.org/html/rfc7230#section-6.1. Envoy detecting "close" and immediately closing the connection is a convenience, the client should not be depending on it (this is actually where the 400 is coming from).

The only workaround I can think of right now is for you to set a very low idle timeout on the HTTP connection manager such that Envoy will idle close incoming connections, but I really would recommend fixing the client. 

On Fri, Jan 19, 2018 at 7:43 AM, <bob_a...@mcgraw-hill.com> wrote:
I'm not completely convinced the client is broken:
  • This code works correctly (for me and thousands of other Shibboleth installations) if I do not use Envoy
  • RFC 2616 section 14.10 indicates that the Connection header is a comma-separated list of the names of headers that a proxy must strip before forwarding and that the option "close" may be used to indicate that the connection should be closed, but the RFC does not specify that the close option has to be alone
  • RFC 2616 section 4.2 indicates that headers may be repeated if the value is a comma-separated list with the final list built by concatenating the lists together

On Friday, January 12, 2018 at 10:33:30 PM UTC-5, Matt Klein wrote:
Your client is sending both "connection: keep-alive" as well as "connection: close". This is causing Envoy's close detection logic to not work correctly for reasons I won't get into.

Technically, this is a broken client. Can you fix the client? If not, please open an issue and I will try to figure out some way to deal with this sanely.

Thanks,
Matt

--
You received this message because you are subscribed to the Google Groups "envoy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to envoy-users...@googlegroups.com.

To post to this group, send email to envoy...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages