jupyter-notebook and bullseye

[D. R. Evans]

I don't use jupyter-notebook often, so I only just discovered that I am
encountering a problem with it following my upgrade from buster to bullseye a
couple of months ago. It worked fine on buster, and I have changed nothing
related to jupyter since the upgrade.

When jupyter-notebook starts the browser, I see a big red "kernel error"
button, and when I press it, the following pops up:

----

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/tornado/web.py", line 1704, in _execute
result = await result
File "/usr/lib/python3/dist-packages/tornado/gen.py", line 769, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File
"/usr/lib/python3/dist-packages/notebook/services/sessions/handlers.py", line
69, in post
model = yield maybe_future(
File "/usr/lib/python3/dist-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/lib/python3/dist-packages/tornado/gen.py", line 769, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File
"/usr/lib/python3/dist-packages/notebook/services/sessions/sessionmanager.py",
line 88, in create_session
kernel_id = yield self.start_kernel_for_session(session_id, path, name,
type, kernel_name)
File "/usr/lib/python3/dist-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/usr/lib/python3/dist-packages/tornado/gen.py", line 769, in run
yielded = self.gen.throw(*exc_info) # type: ignore
File
"/usr/lib/python3/dist-packages/notebook/services/sessions/sessionmanager.py",
line 100, in start_kernel_for_session
kernel_id = yield maybe_future(
File "/usr/lib/python3/dist-packages/tornado/gen.py", line 762, in run
value = future.result()
File
"/usr/lib/python3/dist-packages/notebook/services/kernels/kernelmanager.py",
line 176, in start_kernel
kernel_id = await maybe_future(self.pinned_superclass.start_kernel(self,
**kwargs))
File
"/usr/lib/python3/dist-packages/jupyter_client/multikernelmanager.py", line
185, in start_kernel
km.start_kernel(**kwargs)
File "/usr/lib/python3/dist-packages/jupyter_client/manager.py", line 313,
in start_kernel
self.kernel = self._launch_kernel(kernel_cmd, **kw)
File "/usr/lib/python3/dist-packages/jupyter_client/manager.py", line 222,
in _launch_kernel
return launch_kernel(kernel_cmd, **kw)
File "/usr/lib/python3/dist-packages/jupyter_client/launcher.py", line
134, in launch_kernel
proc = Popen(cmd, **kwargs)
File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.9/subprocess.py", line 1823, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/python'

-----

It is certainly true that that file does not exist, but something is obviously
telling jupyter to look for it. So something about the jupyter configuration
seems to have been hosed by the upgrade.

The command "jupyter kernelspec list" produces:

----

[ZB:~] jupyter kernelspec list
-bash: /home/n7dr/.local/bin/jupyter: /usr/bin/python: bad interpreter: No
such file or directory
[ZB:~]

----

In buster I had perhaps half a dozen kernels available (and they all worked).

Can someone suggest how I might get back to the fully-working set of kernels
that I had in buster?

Doc

--
Web: http://enginehousebooks.com/drevans

[Reco]

Hi.

On Thu, Dec 16, 2021 at 12:43:51PM -0700, D. R. Evans wrote:
> FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/python'

...

> Can someone suggest how I might get back to the fully-working set of kernels that I had in buster?

Try this:

apt install python-is-python3

Reco

[D. R. Evans]

Reco wrote on 12/17/21 6:10 AM:

Thank you very much.

That was certainly a help (although I wonder why it was necessary for me to do
that manually), but ultimately I am still unable to do anything.

"jupyter kernelspec list" now looks better:

----

[ZB:jupyter] jupyter kernelspec list
Available kernels:
ir /home/n7dr/.local/share/jupyter/kernels/ir
markdown /home/n7dr/.local/share/jupyter/kernels/markdown
bash /usr/local/share/jupyter/kernels/bash
gnuplot /usr/local/share/jupyter/kernels/gnuplot
sos /usr/local/share/jupyter/kernels/sos
python3 /usr/share/jupyter/kernels/python3
[ZB:jupyter]

----

But if I actually run jupyter-notebook on a known-good .ipynb file I get the
following:

----

[ZB:jupyter] jn CQ*
[I 13:22:37.809 NotebookApp] Loading IPython parallel extension
[I 13:22:37.824 NotebookApp] Serving notebooks from local directory:
/home/n7dr/notebooks/jupyter
[I 13:22:37.824 NotebookApp] Jupyter Notebook 6.2.0 is running at:
[I 13:22:37.824 NotebookApp]
http://localhost:8888/?token=5e06127359465bc598e53eb5b48ef202592e96e3a1fe4ba7
[I 13:22:37.824 NotebookApp] or
http://127.0.0.1:8888/?token=5e06127359465bc598e53eb5b48ef202592e96e3a1fe4ba7
[I 13:22:37.825 NotebookApp] Use Control-C to stop this server and shut down
all kernels (twice to skip confirmation).
[C 13:22:42.319 NotebookApp]

To access the notebook, open this file in a browser:
file:///home/n7dr/.local/share/jupyter/runtime/nbserver-2497406-open.html
Or copy and paste one of these URLs:

http://localhost:8888/?token=5e06127359465bc598e53eb5b48ef202592e96e3a1fe4ba7
or
http://127.0.0.1:8888/?token=5e06127359465bc598e53eb5b48ef202592e96e3a1fe4ba7
[W 13:22:50.477 NotebookApp] 404 GET
/nbextensions/widgets/notebook/js/extension.js?v=20211231132234 (127.0.0.1)
96.380000ms referer=http://localhost:8888/notebooks/CQ%20WW.ipynb
[I 13:22:51.078 NotebookApp] 302 GET /notebooks/activity-160.png (127.0.0.1)
0.670000ms
[I 13:22:51.171 NotebookApp] 302 GET /notebooks/2017-ALL.png (127.0.0.1)
0.610000ms
[I 13:22:51.394 NotebookApp] 302 GET /notebooks/cq-ww-qso-nlogs-a-u.png
(127.0.0.1) 0.620000ms
[I 13:22:51.456 NotebookApp] 302 GET /notebooks/cq-ww-qso-percentiles.png
(127.0.0.1) 0.640000ms
[I 13:22:51.576 NotebookApp] Kernel started:
9636f23d-6e5c-4ce6-931a-f0844f8876e5, name: bash
/usr/bin/python: No module named bash_kernel
[I 13:22:54.576 NotebookApp] KernelRestarter: restarting kernel (1/5), new
random ports
/usr/bin/python: No module named bash_kernel
[I 13:22:57.590 NotebookApp] KernelRestarter: restarting kernel (2/5), new
random ports
/usr/bin/python: No module named bash_kernel
[I 13:23:00.598 NotebookApp] KernelRestarter: restarting kernel (3/5), new
random ports
/usr/bin/python: No module named bash_kernel
[I 13:23:03.605 NotebookApp] KernelRestarter: restarting kernel (4/5), new
random ports
/usr/bin/python: No module named bash_kernel
[W 13:23:06.621 NotebookApp] KernelRestarter: restart failed
[W 13:23:06.621 NotebookApp] Kernel 9636f23d-6e5c-4ce6-931a-f0844f8876e5 died,
removing from map.
[W 13:23:12.691 NotebookApp] Replacing stale connection:
9636f23d-6e5c-4ce6-931a-f0844f8876e5:1820effccd3749579998b085be46704b
[W 13:23:34.746 NotebookApp] Replacing stale connection:
9636f23d-6e5c-4ce6-931a-f0844f8876e5:1820effccd3749579998b085be46704b
[W 13:23:51.683 NotebookApp] Timeout waiting for kernel_info reply from
9636f23d-6e5c-4ce6-931a-f0844f8876e5
[E 13:23:51.686 NotebookApp] Error opening stream: HTTP 404: Not Found (Kernel
does not exist: 9636f23d-6e5c-4ce6-931a-f0844f8876e5)
[W 13:23:51.689 NotebookApp] 404 GET
/api/kernels/9636f23d-6e5c-4ce6-931a-f0844f8876e5/channels?session_id=1820effccd3749579998b085be46704b
(127.0.0.1): Kernel does not exist: 9636f23d-6e5c-4ce6-931a-f0844f8876e5
[W 13:23:51.690 NotebookApp] 404 GET
/api/kernels/9636f23d-6e5c-4ce6-931a-f0844f8876e5/channels?session_id=1820effccd3749579998b085be46704b
(127.0.0.1) 39005.310000ms referer=None
[W 13:23:51.690 NotebookApp] 404 GET
/api/kernels/9636f23d-6e5c-4ce6-931a-f0844f8876e5/channels?session_id=1820effccd3749579998b085be46704b
(127.0.0.1): Kernel does not exist: 9636f23d-6e5c-4ce6-931a-f0844f8876e5
[W 13:23:51.691 NotebookApp] 404 GET
/api/kernels/9636f23d-6e5c-4ce6-931a-f0844f8876e5/channels?session_id=1820effccd3749579998b085be46704b
(127.0.0.1) 16950.430000ms referer=None
[W 13:23:55.701 NotebookApp] Replacing stale connection:
9636f23d-6e5c-4ce6-931a-f0844f8876e5:1820effccd3749579998b085be46704b
[W 13:24:23.733 NotebookApp] Replacing stale connection:
9636f23d-6e5c-4ce6-931a-f0844f8876e5:1820effccd3749579998b085be46704b

----

And in the browser window that opens, the message: "kernel starting, please
wait" is shown for perhaps twenty seconds, and then it says: "Connecting to
kernel", but never actually completes the connection.

Presumably this is a result of the message:
No module named bash_kernel
that appears above.

If I look for /usr/local/share/jupyter/kernels/bash, I see:

----

[ZB:jupyter] ls -al /usr/local/share/jupyter/kernels/bash
total 7
drwxr-xr-x 2 root root 3 Mar 1 2020 .
drwxr-xr-x 5 root root 5 Mar 1 2020 ..
-rw-r--r-- 1 root root 170 Mar 1 2020 kernel.json
[ZB:jupyter]

----

So I don't know what the problem is. As I said in my prior post, all this
worked fine before the upgrade to bullseye.

[Reco]

Hi.

On Fri, Dec 31, 2021 at 01:35:47PM -0700, D. R. Evans wrote:
> Reco wrote on 12/17/21 6:10 AM:
> > Hi.
> >
> > On Thu, Dec 16, 2021 at 12:43:51PM -0700, D. R. Evans wrote:
> > > FileNotFoundError: [Errno 2] No such file or directory: '/usr/bin/python'
> > ...
> > > Can someone suggest how I might get back to the fully-working set of kernels that I had in buster?
> >
> > Try this:
> >
> > apt install python-is-python3
>
> Thank you very much.
>
> That was certainly a help (although I wonder why it was necessary for me to do that manually),

It's official Debian policy now, believe it or not.
python 2.x is /usr/bin/python2.
python 3.x is /usr/bin/python3.

If the user really wants /usr/bin/python the user should install
python-is-python2 or python-is-python3. And these two packages conflict
with each other.

> but ultimately I am still unable to do anything.

I'm not familiar with jupyter and I'm not using it.

What I do know is:
a) /usr/bin/python was python 2.x in Debian 10.
b) Python 2.x and python 3.x modules are not compatible nor they are
interchangeable.

Judging from [1], you're required to reinstall all these "jupyter
kernels", because what you have was installed for python2, but what you
need is to install them for python3.

But then again, I could be wrong. Sorry, cannot help you further.

Reco

[1] https://github.com/takluyver/bash_kernel

[D. R. Evans]

Reco wrote on 12/31/21 1:47 PM:

>>
>> That was certainly a help (although I wonder why it was necessary for me to do that manually),
>
> It's official Debian policy now, believe it or not.
> python 2.x is /usr/bin/python2.
> python 3.x is /usr/bin/python3.
>
> If the user really wants /usr/bin/python the user should install
> python-is-python2 or python-is-python3. And these two packages conflict
> with each other.
>

Once upon a time, not really that long ago, Debian seemed to make very
sensible decisions to keep everything stable and working across upgrades. In
the past few years, however, I find myself shaking my head and wondering "what
were they thinking?" It's not that some of the things they've done are
necessarily *wrong* per se, but they have certainly been a lot more
experimental than one wants in an environment that one expects to keep working
properly across upgrades; it seems that somehow the importance of keeping the
users' systems functioning as one hopes they will is now a much lower priority
than it used to be.

>> but ultimately I am still unable to do anything.
>
> I'm not familiar with jupyter and I'm not using it.
>

Pretty wise; I think. I was sucked in a bit about the hype that surrounds it
and put in quite a bit of effort to build some useful notebooks a few years
ago. But now I find that it's pretty much like the majority of experiments
I've tried over the years: it looks nifty, and doubtless some people find it
useful, but for me it's too fragile and ultimately the cost in time isn't
worth the possible benefit.

But it certainly would be nice to at least be able to use my old jupyter
notebooks, even if it's unlikely that I'll create any new ones.

>
> Judging from [1], you're required to reinstall all these "jupyter
> kernels", because what you have was installed for python2, but what you
> need is to install them for python3.
>
> But then again, I could be wrong. Sorry, cannot help you further.

That's probably a good bet. I don't remember how any of those kernels got
installed [I thought that all except the sos kernel were from debian
repositories, but my memory might be faulty], so I'll have to search around
and see what I can dig up. The evidence to hand does seem to suggest that they
don't auto-upgrade and therefore need to be upgraded manually somehow.

Thank you for taking the time to share your thoughts, especially as you don't
use jupyter yourself.

[Dan Ritter]

D. R. Evans wrote:
> > If the user really wants /usr/bin/python the user should install
> > python-is-python2 or python-is-python3. And these two packages conflict
> > with each other.
>
> Once upon a time, not really that long ago, Debian seemed to make very
> sensible decisions to keep everything stable and working across upgrades. In
> the past few years, however, I find myself shaking my head and wondering
> "what were they thinking?"

95% of the time, making the best available choices in a world
where nobody else cares.

4% of the time, shrugging and saying "everybody else is doing
it".

1% of the time, inexplicable.

I think that these ratios are better than other distributions,
which is why I keep using and recommending Debian.

> It's not that some of the things they've done are
> necessarily *wrong* per se, but they have certainly been a lot more
> experimental than one wants in an environment that one expects to keep
> working properly across upgrades; it seems that somehow the importance of
> keeping the users' systems functioning as one hopes they will is now a much
> lower priority than it used to be.

You can't have a bug-free system, you can't have a stable system, and
you can't have an up-to-date system. You can lean closer to any two of
those by getting further from the third.

In general, each Debian stable=>stable upgrade has been less disruptive
than the previous one. A distro can go to a rolling release, but that
means that either something is broken all the time or there is only one
canonical way to do things: a notional Debian "Luxo" rolling release that
tried to produce a stabilized stream from testing would require twice
as many Debian Developers, or an 85% cut in the number of packages,
or perhaps both.

-dsr-

[Nicholas Geovanis]

On Fri, Dec 31, 2021, 4:08 PM D. R. Evans <doc....@gmail.com> wrote:

......

Pretty wise; I think. I was sucked in a bit about the hype that surrounds it
and put in quite a bit of effort to build some useful notebooks a few years
ago. But now I find that it's pretty much like the majority of experiments
I've tried over the years: it looks nifty, and doubtless some people find it
useful, but for me it's too fragile and ultimately the cost in time isn't
worth the possible benefit.

For better and for worse, it has become the communication medium for machine learning and knowledge engineering. Lots of open-source notebooks are distributed.

.....

[David Wright]

On Fri 31 Dec 2021 at 15:07:39 (-0700), D. R. Evans wrote:
> Reco wrote on 12/31/21 1:47 PM:
>
> > > That was certainly a help (although I wonder why it was necessary for me to do that manually),
> >
> > It's official Debian policy now, believe it or not.
> > python 2.x is /usr/bin/python2.
> > python 3.x is /usr/bin/python3.
> >
> > If the user really wants /usr/bin/python the user should install
> > python-is-python2 or python-is-python3. And these two packages conflict
> > with each other.
>
> Once upon a time, not really that long ago, Debian seemed to make very
> sensible decisions to keep everything stable and working across
> upgrades.

I don't understand: the exchange above seems to indicate that they've
done what you asked for, to provide an upgrade path for your Python2
software that is now one year beyond EOL.

> In the past few years, however, I find myself shaking my
> head and wondering "what were they thinking?" It's not that some of
> the things they've done are necessarily *wrong* per se, but they have
> certainly been a lot more experimental than one wants in an
> environment that one expects to keep working properly across upgrades;
> it seems that somehow the importance of keeping the users' systems
> functioning as one hopes they will is now a much lower priority than
> it used to be.

I see no evidence of that. The Release Notes for buster gave
notice of the end of Python2 support, and so the provision of
python-is-python2 might be seen as something of a plus.

There's not much point in blaming Debian for the demise of Python2.
Were Debian to ignore the lack of security support and just observe
the consequences, that's what I'd call "being experimental".

> But it certainly would be nice to at least be able to use my old
> jupyter notebooks, even if it's unlikely that I'll create any new
> ones.

> > Judging from [1], you're required to reinstall all these "jupyter
> > kernels", because what you have was installed for python2, but what you
> > need is to install them for python3.

> That's probably a good bet. I don't remember how any of those kernels
> got installed [I thought that all except the sos kernel were from
> debian repositories, but my memory might be faulty], so I'll have to
> search around and see what I can dig up. The evidence to hand does
> seem to suggest that they don't auto-upgrade and therefore need to be
> upgraded manually somehow.

One might assume that others have tackled this conversion process,
probably a few years ago. I certainly get google hits that look as if
that's what people have done, though I can't be certain as I don't use
Jupyter. (I typed convert python2 python3 jupyter.) Perhaps it's
worth backing up your data and then trying out some of these
processes. I can't see Python2 sticking around for ever.

Alternatively, you could just install an older Debian version onto
a junk machine, and keep it safely away from the connected world.

Cheers,
David.