Jupyter notebook RAM memory problems with long sessions

[Robin Nemeth]

Hi all,
I am running jupyter notebook (installe via conda) on a server at work and have some memory problems. The server is accesbile from the internet only via VPN if that makes difference.
I run it as daemon under supervisord with following config (python is the username)

[program:notebook]
command=/home/python/anaconda3/envs/virtualbox_env_conda/bin/jupyter-notebook --ip='*' --no-browser
directory=/home/python/notebook/
process_name=python
user=python
group=python
stdout_logfile=/home/python/logs/stdout.log
stdout_logfile_maxbytes=2MB
stdout_logfile_backups=5
stdout_capture_maxbytes=2MB
stdout_events_enabled=false
stderr_logfile=/home/python/logs/stderr.log
stderr_logfile_maxbytes=2MB
stderr_logfile_backups=5
stderr_capture_maxbytes=2MB
stderr_events_enabled=false

start it with

$ supervisorctl start notebook

It is working when it first started, but after lot of notebooks were opened (and closed), the server eventually (after few days of nonstop uptime) runs out of memory (12GB)  (even if only few notebooks are running and they definately dont use whole 12gigs)
I know this happens by command $ free , or $ htop, or when  I try to open previously closed notebook, the kernel is dead and displays this message:

OSError: [Errno 12] Cannot allocate memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/notebook/base/handlers.py", line 458, in wrapper
    result = yield gen.maybe_future(method(self, *args, **kwargs))
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/tornado/gen.py", line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/notebook/services/sessions/handlers.py", line 58, in post
    sm.create_session(path=path, kernel_name=kernel_name))
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/tornado/gen.py", line 1014, in run
    yielded = self.gen.throw(*exc_info)
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/notebook/services/sessions/sessionmanager.py", line 73, in create_session
    self.kernel_manager.start_kernel(path=kernel_path, kernel_name=kernel_name)
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/tornado/gen.py", line 1008, in run
    value = future.result()
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/tornado/concurrent.py", line 232, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 3, in raise_exc_info
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/tornado/gen.py", line 282, in wrapper
    yielded = next(result)
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/notebook/services/kernels/kernelmanager.py", line 87, in start_kernel
    super(MappingKernelManager, self).start_kernel(**kwargs)
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/jupyter_client/multikernelmanager.py", line 109, in start_kernel
    km.start_kernel(**kwargs)
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/jupyter_client/manager.py", line 244, in start_kernel
    **kw)
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/jupyter_client/manager.py", line 190, in _launch_kernel
    return launch_kernel(kernel_cmd, **kw)
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/site-packages/jupyter_client/launcher.py", line 123, in launch_kernel
    proc = Popen(cmd, **kwargs)
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/subprocess.py", line 950, in __init__
    restore_signals, start_new_session)
  File "/home/python/anaconda3/envs/virtualbox_env_conda/lib/python3.5/subprocess.py", line 1483, in _execute_child
    restore_signals, start_new_session, preexec_fn)
SystemError: <built-in function enable> returned a result with an error set


and $ htop shows the notebook has spawned lots of threads despite only one opened notebook.

$ conda list throws theese packages at me (shortened)
ipython                   4.1.1                    py35_0    defaults
ipykernel                 4.2.2                    py35_0    defaults
ipython                   4.1.1                    py35_0    defaults
ipython-genutils          0.1.0                     <pip>
ipython_genutils          0.1.0                    py35_0    defaults
jupyter                   1.0.0                    py35_1    defaults
jupyter-client            4.1.1                     <pip>
jupyter-console           4.1.0                     <pip>
jupyter-core              4.0.6                     <pip>
jupyter-themes            0.1                       <pip>
jupyter_client            4.1.1                    py35_0    defaults
jupyter_console           4.1.0                    py35_0    defaults
jupyter_core              4.0.6                    py35_0    defaults



Any idea what might cause these memory problems? Anyone got idea for a solution?
Is the server suitable for these long non-restart sessions, or it should be restarted fairly often?
Is there any other way to daemonize the notebook?

[Thomas Kluyver]

Most of the threads in that screenshot are in the kernel, and that's mostly dependent on what you are doing in the kernel. You may be using a library that starts multiple threads internally, for instance.

The top four threads are in the notebook server itself. I don't think four threads there is any cause for concern.

It looks like both your kernel and your notebook server are using multiple gigabytes of memory. That certainly looks like a bug in the notebook server, unless it's trying to save a really large notebook and the document itself takes up that much space. The kernel's memory use is more likely to be down to what you're doing in the kernel again.

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/7c77fb85-9769-48b7-b128-0e7a54626b5f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.