Websocket of Jupyter Notebook terminal disconnecting after 300 seconds

[Abe Sharp]

Hi, Kubeflow community. Has anyone else seen the issue of the websocket used by the Jupyter Notebook Terminal function disconnecting at exactly the 5 minute mark (see attached screen shot)? This is with K8s 1.20.2, KF 1.3 and Istio 1.9, and is totally reproducible.

If I bypass Kubeflow (by creating a service directly to the notebook pod itself), or deploy the same notebook images (tensorflow-1.15.2-notebook-cpu:latestC, or any other JNB image) _without_ Kubeflow (but still using Istio 1.9 and accessing via Istio Ingress gateway) the problem does not happen.

In the Istio Ingressgateway logs I see messages like this:
[2021-09-06T19:03:07.335Z] "GET /notebook/alice/aa333/terminals/websocket/1 HTTP/1.1" 101 - max_duration_timeout - "-" 372 8763 299999 - "..." "ec2-3-128-72-234.us-east-2.compute.amazonaws.com:10012" "10.192.1.147:8888" outbound|80||aa333.alice.svc.cluster.local 10.192.1.136:50838 10.192.1.136:8080 10.192.0.0:14445 - -

If the '299999' represents milliseconds, as applied to 'max_duration_timeout' of the websocket connection, that could be the issue. I can't find where this is configured or how to change it. Could be that an envoy filter is needed.

Thanks

Abe