Issue when re-opening a notebook [run all cells]

[Luca Marconi]

Hi everybody,

I am working with Jupyter Notebook for big data analysis and machine learning.

I always do "save and checkpoint" before closing the file. 

Unfortunately, I always have to re-run all the cells (all the realized code) when I re-open my file.

Is there a quick way to avoid this necessity?

I would need to open the notebook and start working from the last cell onwards, since the time needed to re-run everything is a lot (I have to work with very large datasets).

Thanks for you support.

Best regards,

Luca Marconi

[Jason Moore]

I don't think this is something that jupyter can solve. You should likely investigate how to cache intermediate results to disk in your analysis pipelines if you want to skip repeating some computations.

Jason

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/188c2702-3e52-4bd8-876d-4ca26242061en%40googlegroups.com.

[Luca Marconi]

Hi Jason,

thank you for your quick answer.

Are there any simple ways to do what you suggest? 

Otherwise, could I open the Jupyter Notebook on Google cloud to prevent the re-run everytime?

Or, are there any other possibilities?

Unfortunately, I have been working with Jupyter from few weeks, so I am not expert in how to optimize this issue.

Thanks for any help. 

Best regards,

Luca

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAP7f1Ai9X7oetuzEQcRDDs57EUk%2BA%3DoGpiaBZhOb15i3A3tkXA%40mail.gmail.com.

--

Cordiali saluti,

Luca Marconi

[Jason Moore]

The basic idea is to save the results of a long computation to a file and then skip the computation if that file exists. You can then delete the file if you want the computation to run. Many people use the pickle module to save Python objects to disk but there are thousands of other ways to manage caching information to disk.

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CANUWwiTP%2BpVqXXJ743rZsBLdQfFkQY6kO%3DCWtbVUKRHFkcu0Rg%40mail.gmail.com.

[Michał Krassowski]

If you are interested in a generic solution, hibernating kernels could be proposed in a Jupyter Enhancement Proposal[1]. However, supporting this across operating systems may be a challenge (CryoPID or BLCR could work on Linux, not sure about feasibility on Windows) and would require significant investment and resources to maintain.

For now, you could either:

- use a kernel-specific solution such as dill's dump_session and load_session for Python kernels (as discussed on SO [2]), or

- host your jupyter-server (or kernel) on a remote machine which would stay on even when you turn your client/workstation off (of note JupyterLab Desktop [3] exposes a friendly GUI for connecting to remote jupyter servers)

[1] https://jupyter.org/enhancement-proposals/README.html

[2] https://stackoverflow.com/questions/34342155/how-to-pickle-or-store-jupyter-ipython-notebook-session-for-later

[3] https://github.com/jupyterlab/jupyterlab-desktop

Best wishes,

Michał Krassowski

https://github.com/krassowski

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAP7f1Aiy68jrJCkXd_T1bJoFrW6M13istvwDYrX3yXHtPUCKbA%40mail.gmail.com.

--

Best regards,

Michał Krassowski

[Pete[r] Landwehr]

There are a few magics people have built to help with this, though I’m not sure how actively maintained they are - see the store extension (

https://ipython.readthedocs.io/en/stable/config/extensions/storemagic.html) or ipycache (

https://github.com/rossant/ipycache). Your best approach is still -alas- to come up with your own pattern and use that. Me, I like to designate a folder for interim notebook outputs and add a little setting for whether or not to overwrite the intermediate output if it already exists. (For extra fanciness, consider checking whether the inputs have changed as well.) if you need to run from the command line, consider using papermill to execute the notebook with parameter settings. (Papermill is a moderately heavy install, IMO, and I wish the simplest bits of it were rolled into Jupiter proper. Some day!)

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAPfyr1K8RoPRW8OREjc2p6o8-YUsGWGX_ATBmQaHAwhq_pegAw%40mail.gmail.com.