The Kernel appears to have died. It will restart automatically ? Please help !

[Karthik Ram]

Hi All,

I have been using the following Jupyter notebook configuration for less than couple of months, so I'm relatively new. While my script seems to work on smaller tables data, when I try to use the same to access a larger table (with ~10 billion rows), the kernel dies on me. I have searched and implemented some suggestions mentioned in stack overflow with similar issues but it didn't help me over the last couple of days. Attaching error message. I have the following questions. Also Any other suggestions on how to further debug/fix this issue will be really helpful.

https://stackoverflow.com/questions/47022997/jupyter-the-kernel-appears-to-have-died-it-will-restart-automatically 

1. I have updated MKL using "conda update mkl" but it seems to update my Anaconda2 while I noticed jupyter installation uses python3. When I checked the jupyter kernel environment , it seems like it is using python2.

Before MKL update: $ jupyter-kernelspec list

 

Available kernels:  python2    /home/rpatel/miniconda2/lib/python2.7/site-packages/ipykernel/resources

After MKL update: $ jupyter-kernelspec list

 

Available kernels:   python2    /home/rpatel/.local/share/jupyter/kernels/python2

      

      2. Any details on allocating extra RAM to Jupyter will be really helpful. (Most of the suggestions point to increasing RAM but doesn't provide details on how exactly it can be achieved).

Server Information:

You are using Jupyter notebook.

The version of the notebook server is 5.0.0 and is running on:

Python 3.6.1 |Anaconda custom (64-bit)| (default, May 11 2017, 13:09:58) 

[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]

Current Kernel Information:

Python 3.6.1 |Anaconda custom (64-bit)| (default, May 11 2017, 13:09:58) 

Type 'copyright', 'credits' or 'license' for more information

IPython 6.1.0 -- An enhanced Interactive Python. Type '?' for help.

[Karthik Ram]

I also changed the following in jupyterhub_config.py file. But still seeing the issue.

## Minimum number of bytes a single-user notebook server is guaranteed to have

#  available.

#

#  Allows the following suffixes:

#    - K -> Kilobytes

#    - M -> Megabytes

#    - G -> Gigabytes

#    - T -> Terabytes

#

#  This needs to be supported by your spawner for it to work.

#c.Spawner.mem_guarantee = 8G

## Maximum number of bytes a single-user notebook server is allowed to use.

#

#  Allows the following suffixes:

#    - K -> Kilobytes

#    - M -> Megabytes

#    - G -> Gigabytes

#    - T -> Terabytes

#

#  If the single user server tries to allocate more memory than this, it will

#  fail. There is no guarantee that the single-user notebook server will be able

#  to allocate this much memory - only that it can not allocate more than this.

#

#  This needs to be supported by your spawner for it to work.

#c.Spawner.mem_limit = 10G

[Thomas Kluyver]

On 19 November 2017 at 22:28, Karthik Ram <just...@gmail.com> wrote:

I also changed the following in jupyterhub_config.py file. But still seeing the issue.

...

#c.Spawner.mem_guarantee = 8G

You'll need to uncomment these lines for them to affect anything, i.e. remove the # from the beginning. It also probably needs to be a string - put "8G" inside quotation marks.

Thomas

[Karthik Ram]

Argg. Thank you Thomas. It did run longer(17 min as opposed to 10 min) this time after I un-commented those lines, but still saw the same issue. Is there any limitation in Jupyter that it cannot handle more than certain GB data or query more than certain million or billion rows from post gre SQL DB ?

Its strange because when I do run the SQL script from SQL work bench locally on my machine(which is less powerful than the server on which jupyter is running), I do get the resulting rows.

--
You received this message because you are subscribed to a topic in the Google Groups "Project Jupyter" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jupyter/hY0Jq-E5Ki0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jupyter+unsubscribe@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAOvn4qiR9cRz4ydTya2nwPivkbjsS_yuT4RwwqXO3_T6oSti5g%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

"In the depth of winter, I finally learned that within me there lay an invincible summer"

[Thomas Kluyver]

On 20 November 2017 at 18:58, Karthik Ram <just...@gmail.com> wrote:

Argg. Thank you Thomas. It did run longer(17 min as opposed to 10 min) this time after I un-commented those lines, but still saw the same issue. Is there any limitation in Jupyter that it cannot handle more than certain GB data or query more than certain million or billion rows from post gre SQL DB ?

Jupyterhub is the only bit that could impose such a limit, and you've already found the config options for it.

 

Its strange because when I do run the SQL script from SQL work bench locally on my machine(which is less powerful than the server on which jupyter is running), I do get the resulting rows.

I would guess that the SQL work bench doesn't load all of the selected rows into memory at once. Databases are designed to work with data larger than will fit into memory, so database software is usually written to load a bit, process it, discard that, and load the next bit. But if you try to make a pandas DataFrame from an SQL query, pandas does try to load all the rows at once.

Thomas

[Brian Granger]

I haven't looked into the details of your configuration, but in
general, Jupyter and the python kernels have no problem using whatever
RAM the system makes available to it. I have created single NumPY
arrays with 1TB of RAM with no problem. If you are running out of RAM,
it is a system level thing related to the configuration of the system,
physical RAM of the system and your deployment of Jupyter.

>> jupyter+u...@googlegroups.com.

>> To post to this group, send email to jup...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/jupyter/CAOvn4qiR9cRz4ydTya2nwPivkbjsS_yuT4RwwqXO3_T6oSti5g%40mail.gmail.com.
>>
>> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
> --
> "In the depth of winter, I finally learned that within me there lay an
> invincible summer"
>

> --
> You received this message because you are subscribed to the Google Groups
> "Project Jupyter" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to jupyter+u...@googlegroups.com.

> To post to this group, send email to jup...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/jupyter/CAFZay%2Bh5L-6fMTE8dX8DHhS2ZRByaojFbTa-HeNS01SiYyOwaA%40mail.gmail.com.

>
> For more options, visit https://groups.google.com/d/optout.



--

Brian E. Granger
Associate Professor of Physics and Data Science
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgra...@calpoly.edu and elli...@gmail.com

[Brian Granger]

How much RAM do you expect to be available and how much RAM do you
expect to be using?