jupiterhub tljh kernel not connecting

236 views

Skip to first unread message

Antonio Rago

unread,

Feb 15, 2021, 4:44:20 PM2/15/21

to Project Jupyter

Dear group,

I am writing to seek help on the behaviour a recent installation of Jupiterhub is showing.

I hope someone can help me to single out where the problem is.

I have followed the how-to (https://tljh.jupyter.org/en/latest/ ) for the server installation which is hosted on a virtual server with a public ip.

The installation worked fine and I have done just minor modifications as suggested in the document.

The hub seems to work fine, I've enabled ssl and open the firewall ports.

I can connect and start the lab (the app I've set as default)

However from time to time when I try to run any notebook/console/terminal everything freeze and the kernel keeps on trying to connect for a while, switching to disconnected after some time.

I have tried to dig into the log (journalctl -u jupyterhub, journalctl -u traefik, journalctl -u jupyter-rago) but none of the above showed any clear error.

My tljh config is as follow:

root@AItesting:/home# tljh-config show
users:
admin:
- rago
https:
enabled: true
letsencrypt:
email: antoni...@xxxx.com
domains:
- xxxxxxx(fully resolved website)
user_environment:
default_app: jupyterlab

Can anyone suggest a way to diagnose the problem?

Thank you in advance.

Antonio Rago

Below are the final part of my logs :

root@AItesting:/home# journalctl -u jupyterhub

Feb 15 22:28:23 AItesting python3[2015]: [I 2021-02-15 22:28:23.251 JupyterHub log:181] 200 GET /hub/api/users (cull...@127.0.0.1) 8.34ms

Feb 15 22:29:23 AItesting python3[2015]: [I 2021-02-15 22:29:23.255 JupyterHub log:181] 200 GET /hub/api/users (cull...@127.0.0.1) 11.77ms

Feb 15 22:29:26 AItesting python3[2015]: /opt/tljh/hub/lib/python3.8/site-packages/jupyterhub_traefik_proxy/toml.py:104: RuntimeWarning: Escape character '_' cannot be a safe character. Set allow_co>

Feb 15 22:29:26 AItesting python3[2015]: escaped_routespec = escapism.escape(traefik_routespec, safe=safe)

Feb 15 22:29:26 AItesting python3[2015]: [I 2021-02-15 22:29:26.843 JupyterHub proxy:320] Checking routes

Feb 15 22:29:35 AItesting python3[2015]: [I 2021-02-15 22:29:35.046 JupyterHub pages:148] User is running: rago

Feb 15 22:29:35 AItesting python3[2015]: [I 2021-02-15 22:29:35.047 JupyterHub log:181] 302 GET /hub/spawn?next=/hub/home -> /hub/home (ra...@87.74.180.148) 9.69ms

Feb 15 22:29:35 AItesting python3[2015]: [I 2021-02-15 22:29:35.085 JupyterHub log:181] 200 GET /hub/home (ra...@87.74.180.148) 15.55ms

Feb 15 22:29:36 AItesting python3[2015]: [I 2021-02-15 22:29:36.562 JupyterHub log:181] 200 GET /hub/api/authorizations/token/[secret] (ra...@127.0.0.1) 15.78ms

Feb 15 22:30:23 AItesting python3[2015]: [I 2021-02-15 22:30:23.255 JupyterHub log:181] 200 GET /hub/api/users (cull...@127.0.0.1) 11.19ms

Feb 15 22:31:23 AItesting python3[2015]: [I 2021-02-15 22:31:23.254 JupyterHub log:181] 200 GET /hub/api/users (cull...@127.0.0.1) 10.63ms

Feb 15 22:32:23 AItesting python3[2015]: [I 2021-02-15 22:32:23.258 JupyterHub log:181] 200 GET /hub/api/users (cull...@127.0.0.1) 14.82ms

Feb 15 22:33:06 AItesting python3[2015]: [I 2021-02-15 22:33:06.488 JupyterHub log:181] 200 POST /hub/api/users/rago/activity (ra...@127.0.0.1) 15.77ms

Feb 15 22:33:23 AItesting python3[2015]: [I 2021-02-15 22:33:23.251 JupyterHub log:181] 200 GET /hub/api/users (cull...@127.0.0.1) 8.08ms

Feb 15 22:34:23 AItesting python3[2015]: [I 2021-02-15 22:34:23.254 JupyterHub log:181] 200 GET /hub/api/users (cull...@127.0.0.1) 10.77ms

Feb 15 22:34:26 AItesting python3[2015]: /opt/tljh/hub/lib/python3.8/site-packages/jupyterhub_traefik_proxy/toml.py:104: RuntimeWarning: Escape character '_' cannot be a safe character. Set allow_co>

Feb 15 22:34:26 AItesting python3[2015]: escaped_routespec = escapism.escape(traefik_routespec, safe=safe)

Feb 15 22:34:26 AItesting python3[2015]: [I 2021-02-15 22:34:26.843 JupyterHub proxy:320] Checking routes

Feb 15 22:35:23 AItesting python3[2015]: [I 2021-02-15 22:35:23.255 JupyterHub log:181] 200 GET /hub/api/users (cull...@127.0.0.1) 11.72ms

root@AItesting:/home# journalctl -u traefik

Feb 15 21:25:30 AItesting traefik[2012]: time="2021-02-15T21:25:30+01:00" level=info msg="Skipping same configuration for provider file"

Feb 15 21:25:31 AItesting traefik[2012]: time="2021-02-15T21:25:31+01:00" level=info msg="Server configuration reloaded on :80"

Feb 15 21:25:31 AItesting traefik[2012]: time="2021-02-15T21:25:31+01:00" level=info msg="Server configuration reloaded on :443"

Feb 15 21:25:31 AItesting traefik[2012]: time="2021-02-15T21:25:31+01:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099"

Feb 15 21:25:33 AItesting traefik[2012]: time="2021-02-15T21:25:33+01:00" level=info msg="Server configuration reloaded on :80"

Feb 15 21:25:33 AItesting traefik[2012]: time="2021-02-15T21:25:33+01:00" level=info msg="Server configuration reloaded on :443"

Feb 15 21:25:33 AItesting traefik[2012]: time="2021-02-15T21:25:33+01:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099"

Feb 15 21:25:40 AItesting traefik[2012]: {"BackendAddr":"127.0.0.1:33115","BackendName":"backend__2Fuser_2Fblucini","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"127.0.0.1:33115","Pa>

Feb 15 21:51:23 AItesting traefik[2012]: time="2021-02-15T21:51:23+01:00" level=info msg="Skipping same configuration for provider file"

Feb 15 21:51:23 AItesting traefik[2012]: time="2021-02-15T21:51:23+01:00" level=info msg="Server configuration reloaded on :80"

Feb 15 21:51:23 AItesting traefik[2012]: time="2021-02-15T21:51:23+01:00" level=info msg="Server configuration reloaded on :443"

Feb 15 21:51:23 AItesting traefik[2012]: time="2021-02-15T21:51:23+01:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099"

Feb 15 21:51:25 AItesting traefik[2012]: time="2021-02-15T21:51:25+01:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099"

Feb 15 21:51:25 AItesting traefik[2012]: time="2021-02-15T21:51:25+01:00" level=info msg="Server configuration reloaded on :80"

Feb 15 21:51:25 AItesting traefik[2012]: time="2021-02-15T21:51:25+01:00" level=info msg="Server configuration reloaded on :443"

Feb 15 22:23:02 AItesting traefik[2012]: time="2021-02-15T22:23:02+01:00" level=info msg="Skipping same configuration for provider file"

Feb 15 22:23:02 AItesting traefik[2012]: time="2021-02-15T22:23:02+01:00" level=info msg="Server configuration reloaded on :80"

Feb 15 22:23:02 AItesting traefik[2012]: time="2021-02-15T22:23:02+01:00" level=info msg="Server configuration reloaded on :443"

Feb 15 22:23:02 AItesting traefik[2012]: time="2021-02-15T22:23:02+01:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099"

Feb 15 22:23:05 AItesting traefik[2012]: time="2021-02-15T22:23:05+01:00" level=info msg="Server configuration reloaded on :80"

Feb 15 22:23:05 AItesting traefik[2012]: time="2021-02-15T22:23:05+01:00" level=info msg="Server configuration reloaded on :443"

Feb 15 22:23:05 AItesting traefik[2012]: time="2021-02-15T22:23:05+01:00" level=info msg="Server configuration reloaded on 127.0.0.1:8099"

Feb 15 22:23:09 AItesting traefik[2012]: {"BackendAddr":"127.0.0.1:43581","BackendName":"backend__2Fuser_2Frago","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"127.0.0.1:43581","Path">

Feb 15 22:29:39 AItesting traefik[2012]: {"BackendAddr":"127.0.0.1:43581","BackendName":"backend__2Fuser_2Frago","BackendURL":{"Scheme":"http","Opaque":"","User":null,"Host":"127.0.0.1:43581","Path">

root@AItesting:/home# journalctl -u jupyter-rago.service

Feb 15 22:41:06 AItesting bash[23379]: [I 2021-02-15 22:41:06.199 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425266090 (ra...@87.74.180.148) 1.86ms

Feb 15 22:41:06 AItesting bash[23379]: [I 2021-02-15 22:41:06.290 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernelspecs?1613425266182 (ra...@87.74.180.148) 2.32ms

Feb 15 22:41:07 AItesting bash[23379]: [I 2021-02-15 22:41:07.570 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425267462 (ra...@87.74.180.148) 1.62ms

Feb 15 22:41:07 AItesting bash[23379]: [I 2021-02-15 22:41:07.753 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425267642 (ra...@87.74.180.148) 6.97ms

Feb 15 22:41:12 AItesting bash[23379]: [I 2021-02-15 22:41:12.788 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425272674 (ra...@87.74.180.148) 7.92ms

Feb 15 22:41:15 AItesting bash[23379]: [I 2021-02-15 22:41:15.104 SingleUserNotebookApp log:181] 200 GET /user/rago/api/contents/aitrading-repo?content=1&1613425274994 (ra...@87.74.180.148) 3.37ms

Feb 15 22:41:16 AItesting bash[23379]: [I 2021-02-15 22:41:16.201 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernels?1613425276091 (ra...@87.74.180.148) 1.85ms

Feb 15 22:41:16 AItesting bash[23379]: [I 2021-02-15 22:41:16.290 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425276184 (ra...@87.74.180.148) 2.09ms

Feb 15 22:41:17 AItesting bash[23379]: [I 2021-02-15 22:41:17.596 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425277490 (ra...@87.74.180.148) 1.50ms

Feb 15 22:41:17 AItesting bash[23379]: [I 2021-02-15 22:41:17.821 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425277708 (ra...@87.74.180.148) 8.63ms

Feb 15 22:41:22 AItesting bash[23379]: [I 2021-02-15 22:41:22.854 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425282740 (ra...@87.74.180.148) 7.80ms

Feb 15 22:41:25 AItesting bash[23379]: [I 2021-02-15 22:41:25.134 SingleUserNotebookApp log:181] 200 GET /user/rago/api/contents/aitrading-repo?content=1&1613425285024 (ra...@87.74.180.148) 3.26ms

Feb 15 22:41:26 AItesting bash[23379]: [I 2021-02-15 22:41:26.227 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernels?1613425286121 (ra...@87.74.180.148) 1.47ms

Feb 15 22:41:26 AItesting bash[23379]: [I 2021-02-15 22:41:26.314 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425286208 (ra...@87.74.180.148) 1.50ms

Feb 15 22:41:27 AItesting bash[23379]: [I 2021-02-15 22:41:27.622 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425287515 (ra...@87.74.180.148) 1.42ms

Feb 15 22:41:27 AItesting bash[23379]: [I 2021-02-15 22:41:27.884 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425287773 (ra...@87.74.180.148) 6.97ms

Feb 15 22:41:28 AItesting bash[23379]: [I 2021-02-15 22:41:28.489 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernelspecs?1613425288381 (ra...@87.74.180.148) 3.30ms

Feb 15 22:41:32 AItesting bash[23379]: [I 2021-02-15 22:41:32.916 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425292804 (ra...@87.74.180.148) 7.15ms

Feb 15 22:41:35 AItesting bash[23379]: [I 2021-02-15 22:41:35.162 SingleUserNotebookApp log:181] 200 GET /user/rago/api/contents/aitrading-repo?content=1&1613425295054 (ra...@87.74.180.148) 2.69ms

Feb 15 22:41:36 AItesting bash[23379]: [I 2021-02-15 22:41:36.252 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernels?1613425296146 (ra...@87.74.180.148) 1.45ms

Feb 15 22:41:36 AItesting bash[23379]: [I 2021-02-15 22:41:36.340 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425296235 (ra...@87.74.180.148) 1.49ms

Feb 15 22:41:37 AItesting bash[23379]: [I 2021-02-15 22:41:37.649 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425297543 (ra...@87.74.180.148) 1.59ms

Feb 15 22:41:37 AItesting bash[23379]: [I 2021-02-15 22:41:37.949 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425297836 (ra...@87.74.180.148) 7.20ms

Feb 15 22:41:42 AItesting bash[23379]: [I 2021-02-15 22:41:42.982 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425302868 (ra...@87.74.180.148) 6.77ms

Feb 15 22:41:45 AItesting bash[23379]: [I 2021-02-15 22:41:45.194 SingleUserNotebookApp log:181] 200 GET /user/rago/api/contents/aitrading-repo?content=1&1613425305084 (ra...@87.74.180.148) 2.79ms

Feb 15 22:41:46 AItesting bash[23379]: [I 2021-02-15 22:41:46.284 SingleUserNotebookApp log:181] 200 GET /user/rago/api/kernels?1613425306173 (ra...@87.74.180.148) 1.75ms

Feb 15 22:41:46 AItesting bash[23379]: [I 2021-02-15 22:41:46.366 SingleUserNotebookApp log:181] 200 GET /user/rago/api/sessions?1613425306261 (ra...@87.74.180.148) 1.69ms

Feb 15 22:41:47 AItesting bash[23379]: [I 2021-02-15 22:41:47.674 SingleUserNotebookApp log:181] 200 GET /user/rago/api/terminals?1613425307569 (ra...@87.74.180.148) 1.15ms

Feb 15 22:41:48 AItesting bash[23379]: [I 2021-02-15 22:41:48.019 SingleUserNotebookApp log:181] 200 GET /user/rago/metrics?1613425307903 (ra...@87.74.180.148) 11.07ms

Reply all

Reply to author

Forward

0 new messages