File explorer VERY slow to show upon start (sometimes)

[oscar6echo]

Hi,

On my work PC, when I run 'jupyter notebook --debug' in a directory that contains about 100 folders/files, it takes up to 10-12s for the directory to show.

I see this delay whether the directory is local or in a network disk.

Not all colleagues see the same delay.

I do not observe this delay on may home Mac.

This very long delay corresponds to the following line (obtained with --debug)

200 GET /api/contents?type=directory&_=1477423247957 (::1) 8.34ms

(Here it's fast as it's my home Mac)

Has somebody had the same problem ?

Could anybody point to possible causes / corrections ?

Thx

[Chris Colbert]

Do you have a virus scanner (like Symantec) running on your work machine. I've seen those sorts of programs really slow down filesystem operations depending on how IT has it configured.

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+unsubscribe@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/d2a975d6-5c7c-41b2-9967-a1906aba8a3d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Thomas Kluyver]

On 25 October 2016 at 20:34, oscar6echo <olivier....@gmail.com> wrote:

This very long delay corresponds to the following line (obtained with --debug)

200 GET /api/contents?type=directory&_=1477423247957 (::1) 8.34ms

This may be a point where carefully using the new os.scandir() function included in Python 3.5 could make a significant difference - that request is listing all the files in a directory and then stat-ing each one multiple times. That's exactly the kind of case that scandir is meant to speed up.

If we're willing to take on a new compiled dependency, we could use the scandir backport module to do this before bumping the Python requirement:
https://pypi.python.org/pypi/scandir

Thomas

[oscar6echo]

Yes, there is a Symantec on my work PC. But it's difficult to know what it does exactly..

More info:

The folder contains 174 files / folders

The server log is:

[I 20:20:11.329 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/

[I 20:20:11.329 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).

[D 20:20:11.495 NotebookApp] Using contents: services/contents

[D 20:20:11.725 NotebookApp] 200 GET /tree (::1) 245.02ms

[D 20:20:11.996 NotebookApp] 304 GET /custom/custom.css (::1) 266.03ms

[D 20:20:12.120 NotebookApp] 200 GET /static/services/contents.js?v=20161026202010 (::1) 7.00ms

[D 20:20:12.125 NotebookApp] 200 GET /custom/custom.js?v=20161026202010 (::1) 4.00ms

[D 20:20:12.135 NotebookApp] 200 GET /api/config/tree?_=1477506012044 (::1) 2.00ms

[D 20:20:12.136 NotebookApp] 200 GET /api/config/common?_=1477506012045 (::1) 0.00ms

[D 20:20:12.146 NotebookApp] Native kernel (python3) available from C:\HOMEWARE\anaconda-3-x86_64\lib\site-packages\ipykernel\resources

[D 20:20:12.147 NotebookApp] Native kernel (python3) available from C:\HOMEWARE\anaconda-3-x86_64\lib\site-packages\ipykernel\resources

[D 20:20:12.149 NotebookApp] 200 GET /api/kernelspecs (::1) 10.00ms

[D 20:20:12.150 NotebookApp] 200 GET /api/sessions?_=1477506012046 (::1) 1.00ms

[D 20:20:27.150 NotebookApp] 200 GET /api/contents?type=directory&_=1477506012047 (::1) 14864.17ms

 

In this case, 14s to display the list of files/folders.

In a notebook run from this folder, I ran

 

1/

for e in os.scandir('.'):

    print(e)

which took 10ms

2/

for e in os.listdir('.'):

    print(e)

which took 12ms

About the same and very fast.

So maybe it's just some parasite on this PC...

[Thomas Kluyver]

On 26 October 2016 at 20:37, oscar6echo <olivier....@gmail.com> wrote:

2/

for e in os.listdir('.'):

    print(e)

Can you try something like this:

for e in os.listdir('.'):

    os.path.isdir(e)

    print(e)

listdir() itself is probably not making it slow, it's listdir and then doing things with each file that's the issue. scandir returns results with a bit more information, so it doesn't necessarily have to go back to the operating system to check 'is this a directory'.

Thomas

[oscar6echo]

Thanks for the clarification.

I see:

1/

%%time

for e in os.scandit('.'):

    print(e)

--> 14ms

2/

%%time

for e in os.listdir('.'):

    print(e)

--> 16ms

3/

%%time

for e in os.listdir('.'):

    os.path.isdir(e)

    print(e)

--> 1.03s

Much longer !

Maybe this, combined with a parasite antivirus, makes it dog slow...?

[Thomas Kluyver]

On 27 October 2016 at 07:11, oscar6echo <olivier....@gmail.com> wrote:

3/

%%time

for e in os.listdir('.'):

    os.path.isdir(e)

    print(e)

--> 1.03s

Much longer !

I've had a go at improving our directory listing code without scandir, just writing it more carefully to avoid unnecessary operations. Can you have a go with this PR:
https://github.com/jupyter/notebook/pull/1854

It won't make it fast, but I hope it will bring your case under 5 seconds.

Thomas

[oscar6echo]

New test:

1/

notebook - std

[D 08:10:05.172 NotebookApp] Using contents: services/contents

[D 08:10:05.442 NotebookApp] 200 GET /tree (::1) 330.00ms

[D 08:10:05.773 NotebookApp] 304 GET /custom/custom.css (::1) 324.00ms

[D 08:10:05.930 NotebookApp] 200 GET /static/services/contents.js?v=20161031081001 (::1) 7.00ms

[D 08:10:05.935 NotebookApp] 200 GET /custom/custom.js?v=20161031081001 (::1) 4.00ms

[D 08:10:05.938 NotebookApp] 200 GET /static/base/images/favicon.ico?v=30780f272ab4aac64aa073a841546240 (::1) 2.00ms

[D 08:10:05.948 NotebookApp] 200 GET /api/config/tree?_=1477897805807 (::1) 2.00ms

[D 08:10:05.950 NotebookApp] 200 GET /api/config/common?_=1477897805808 (::1) 1.00ms

[D 08:10:05.961 NotebookApp] Native kernel (python3) available from C:\HOMEWARE\anaconda-3-x86_64\lib\site-packages\ipykernel\resources

[D 08:10:05.962 NotebookApp] Native kernel (python3) available from C:\HOMEWARE\anaconda-3-x86_64\lib\site-packages\ipykernel\resources

[D 08:10:05.965 NotebookApp] 200 GET /api/kernelspecs (::1) 13.00ms

[D 08:10:05.972 NotebookApp] 200 GET /api/sessions?_=1477897805809 (::1) 7.00ms

[D 08:10:30.290 NotebookApp] 200 GET /api/contents?type=directory&_=1477897805810 (::1) 24021.60ms

[D 08:10:32.375 NotebookApp] 200 GET /api/sessions?_=1477897805811 (::1) 0.00ms

[D 08:10:46.468 NotebookApp] 200 GET /api/contents?type=directory&_=1477897805812 (::1) 14091.60ms

2/

notebook - PR#1854

[D 11:01:18.471 NotebookApp] Using contents: services/contents

[D 11:01:18.515 NotebookApp] 200 GET /tree (::1) 50.00ms

[D 11:01:18.793 NotebookApp] 200 GET /static/components/jquery-ui/themes/smoothness/jquery-ui.min.css?v=9b2c8d3489227115310662a343fce11c (::1) 274.00ms

[D 11:01:18.795 NotebookApp] 200 GET /static/components/jquery-typeahead/dist/jquery.typeahead.min.css?v=7afb461de36accb1aa133a1710f5bc56 (::1) 2.00ms

[D 11:01:18.801 NotebookApp] 200 GET /custom/custom.css (::1) 3.00ms

[D 11:01:18.803 NotebookApp] 200 GET /static/components/es6-promise/promise.min.js?v=f004a16cb856e0ff11781d01ec5ca8fe (::1) 2.00ms

[D 11:01:18.805 NotebookApp] 200 GET /static/components/requirejs/require.js?v=6da8be361b9ee26c5e721e76c6d4afce (::1) 2.00ms

[D 11:01:18.806 NotebookApp] 200 GET /static/style/style.min.css?v=807166c81ded9672f91872a576572dcd (::1) 11.00ms

[D 11:01:18.835 NotebookApp] 200 GET /static/base/images/logo.png?v=7c4597ba713d804995e8f8dad448a397 (::1) 1.00ms

[D 11:01:18.859 NotebookApp] 200 GET /static/tree/js/main.min.js?v=cee9d5ded70fc8733bb888581c22f633 (::1) 50.00ms

[D 11:01:18.979 NotebookApp] 200 GET /static/services/contents.js?v=20161031110117 (::1) 3.00ms

[D 11:01:18.983 NotebookApp] 200 GET /custom/custom.js?v=20161031110117 (::1) 1.00ms

[D 11:01:18.985 NotebookApp] 200 GET /static/base/images/favicon.ico?v=30780f272ab4aac64aa073a841546240 (::1) 1.00ms

[D 11:01:18.994 NotebookApp] 200 GET /api/config/tree?_=1477908078912 (::1) 2.00ms

[D 11:01:18.995 NotebookApp] 200 GET /api/config/common?_=1477908078913 (::1) 0.00ms

[D 11:01:19.003 NotebookApp] Native kernel (python3) available from C:\HOMEWARE\anaconda-3-x86_64\lib\site-packages\ipykernel\resources

[D 11:01:19.004 NotebookApp] Native kernel (python3) available from C:\HOMEWARE\anaconda-3-x86_64\lib\site-packages\ipykernel\resources

[D 11:01:19.006 NotebookApp] 200 GET /api/kernelspecs (::1) 8.00ms

[D 11:01:19.009 NotebookApp] 200 GET /static/components/font-awesome/fonts/fontawesome-webfont.woff?v=4.2.0 (::1) 3.00ms

[D 11:01:19.010 NotebookApp] 200 GET /api/sessions?_=1477908078914 (::1) 1.00ms

[D 11:01:28.362 NotebookApp] 200 GET /api/contents?type=directory&_=1477908078915 (::1) 9223.00ms

It does improve - significantly.

Still slow, but this is not the notebook. It's probably an antivirus, or some sort of corporate network scanning.

I've tried lauching Jupyter from my local disk and it's notably faster (<1s).

We should soon have deployed JupyterHub in the corporate LAN by the end of the year. We'll see then how this delay evolves.

Thank you very much for looking into this issue !

[Doug Blank]

FYI, I had a student last week whose entire jupyterhub account became unresponsive. Everyone else's account was fine. It turned out that a rogue kernel (not IPython) had begun dropping log files into her home directory (she had 18,000 log files there).

Perhaps a "Loading Directory... [Cancel]" message and method of cancelling would be worthwhile here for those two functions: providing information, and to allow notebook to continue.