Notebook websocket change

[MinRK]

Alternative notebook frontend authors:

There is an upcoming change to websocket connections to the notebook server. Instead of three websockets per notebook, one for each zmq channel, there is a single websocket connection between the server and client. The channel associated with each message is identified by a channel key in the message dict itself.

Relevant changes:

• url is /channels instead of /shell, etc.
• when sending a message, add a channel key with the channel name ('shell', 'stdin')
• when receiving a message, check the channel key for routing

-MinRK

​

[Brian Granger]

So far we have felt that it isn't worth the complexity cost for us to support people that deliberately choose to break the internet. It is not like WebSockets is some crazy, insecure, unsupported hack over a weird port. It is just different bytes going over the same HTTP/HTTPS socket connection and is essentially a formal standard (I know you know this - this is our messaging to companies who complain).

Our situation is a bit different though than SMC because most of our current users run the notebook on their own computers. This may be something that we eventually want to rethink as more people run this in the cloud. However, I would like to push back on the companies choosing to break the internet really hard before giving in. I am guessing that in most cases, WebSockets are broken in enterprise setting not because of some important deliberate choice, but rather because people don't understand it and are using default settings that disable them (I could be wrong though)....

Cheers,

Brian

On Thu, Jan 8, 2015 at 10:10 AM, William Stein <wst...@gmail.com> wrote:

Thanks.

Of related interest, what the situation involving providing a fallback
to websockets?

I tried to only support websockets connections with SageMathCloud for
a few weeks, and immediately ran into many complaints from users who
couldn't connect as a result.   This is even with https and so on --
it's just a sad fact that in some corporate or otherwise constrained
environments that websockets don't work.   By far the best approach I
found was to use Primus + Engine.io, which uses websockets if
possible, but will fallback to polling.   So this is what SMC does
now, and everything works, even for people that don't have websockets
as an option... except for IPython-in-SMC of course, which doesn't
work.

Another unrelated issue is that I'm now hardcoding making this change
around line 171 of IPython/html/notebookapp.py, since I want to serve
the purely static html of IPython from a completely different web
server:

            #static_url_prefix = url_path_join(base_url,'/static/'),
            static_url_prefix = '/static/ipython/',

(Sorry for derailing the thread.)

 -- William

>
>
> _______________________________________________
> IPython-dev mailing list
> IPyth...@scipy.org
> http://mail.scipy.org/mailman/listinfo/ipython-dev
>



--
William Stein
Professor of Mathematics
University of Washington
http://wstein.org
_______________________________________________
IPython-dev mailing list
IPyth...@scipy.org
http://mail.scipy.org/mailman/listinfo/ipython-dev

--

Brian E. Granger
Cal Poly State University, San Luis Obispo
@ellisonbg on Twitter and GitHub
bgra...@calpoly.edu and elli...@gmail.com

[MinRK]

Some time ago, I made a PR to IPython switching to socket.io, which required the same basic changes since it only allowed a single connection per page, so I know it's not too hard. We may start to see more pressure as hosted instances, and reconsider the fallback in the future, but at this point, we don't feel that pressure. Wind proper websockets to be sufficiently preferable that we will wait as long as we can before appeasing old, broken, or misconfigured network environments that prevent websockets from working.

The majority of notebook users seem to be running as a desktop app via localhost, where websockets are rarely a problem (The nightmare of Sophos AV/firewall on Windows excluded).

-MinRK

On Thu, Jan 8, 2015 at 10:55 AM, William Stein <wst...@gmail.com> wrote:

On Thu, Jan 8, 2015 at 10:19 AM, Brian Granger <elli...@gmail.com> wrote:
> So far we have felt that it isn't worth the complexity cost for us to
> support people that deliberately choose to break the internet. It is not
> like WebSockets is some crazy, insecure, unsupported hack over a weird port.
> It is just different bytes going over the same HTTP/HTTPS socket connection
> and is essentially a formal standard (I know you know this - this is our
> messaging to companies who complain).

I know -- that's why I switched SMC to pure websockets for a while.
Then I received complaints at a rate of about 2 very desperate users
per week, and these were the people who cared enough to complain
repeatedly.

> Our situation is a bit different though than SMC because most of our current
> users run the notebook on their own computers. This may be something that we
> eventually want to rethink as more people run this in the cloud. However, I
> would like to push back on the companies choosing to break the internet
> really hard before giving in. I am guessing that in most cases, WebSockets
> are broken in enterprise setting not because of some important deliberate
> choice, but rather because people don't understand it and are using default
> settings that disable them (I could be wrong though)....

In my experience, for some people that's exactly the problem.  For one
person I helped it turned out their specific problem was the Windows
install they had access to had websockets explicitly disabled --
perhaps that was an IT policy.  For other people the problem is
further along the stack (e.g., possibly involving caching).

Anyway, this change so that "there is a single websocket connection
between the server and client. " makes implementing an alternative to
WebSocket as a fallback dramatically easier, which I greatly
appreciate.

> Our situation is a bit different though than SMC because most of our current
> users run the notebook on their own computers. This may be something that we

Just curious -- how do you know that most people using IPython aren't
already using it through SMC?

William

[Kyle Kelley]

You're right Brian, it's about them buying some $FIREWALL_IPS_SUPER_PRICEY_DEVICE that doesn't proxy websockets appropriately and the person purchasing having no clue other than the fact that $VENDOR gave them a neat trinket at a conference.

Firewalls and proxies that do handle websockets appropriately do exist. I'd be tempted to make a list of ones that do it well and display them prominently, if only for the benefit of our own users.

I have witnessed these flavors of IPython in the wild (not including our own recent stunts):

* Corporate entities using an Anaconda server (websockets not a problem internally but were externally)

* People of all backgrounds using Anaconda's IPython notebook bundling

* Vanilla installations from package managers or pip

* Hosted instance of their own flavor

* DIT4C

Most users that we hear from are using a local installation and there's probably a large amount of confirmation bias. It would be great if we had actual shared metrics between us but we only know:

* Unique viewers/users of the Nature demo (> 14,000 total, according to GA)

* Viewers on the notebook viewer (> 200k per month, also according to GA)

* Downloads on PyPI (skewed, not unique)

* Clones from github (also skewed)

We could actually be tracking who has websocket support of those hitting the notebook viewer for a bigger sense of potential users (since viewers are not necessarily IPython/Jupyter users).

On Thu, Jan 8, 2015 at 12:55 PM, William Stein <wst...@gmail.com> wrote:

On Thu, Jan 8, 2015 at 10:19 AM, Brian Granger <elli...@gmail.com> wrote:

> So far we have felt that it isn't worth the complexity cost for us to
> support people that deliberately choose to break the internet. It is not
> like WebSockets is some crazy, insecure, unsupported hack over a weird port.
> It is just different bytes going over the same HTTP/HTTPS socket connection
> and is essentially a formal standard (I know you know this - this is our
> messaging to companies who complain).

I know -- that's why I switched SMC to pure websockets for a while.
Then I received complaints at a rate of about 2 very desperate users
per week, and these were the people who cared enough to complain
repeatedly.

> Our situation is a bit different though than SMC because most of our current
> users run the notebook on their own computers. This may be something that we
> eventually want to rethink as more people run this in the cloud. However, I
> would like to push back on the companies choosing to break the internet
> really hard before giving in. I am guessing that in most cases, WebSockets
> are broken in enterprise setting not because of some important deliberate
> choice, but rather because people don't understand it and are using default
> settings that disable them (I could be wrong though)....

In my experience, for some people that's exactly the problem.  For one
person I helped it turned out their specific problem was the Windows
install they had access to had websockets explicitly disabled --
perhaps that was an IT policy.  For other people the problem is
further along the stack (e.g., possibly involving caching).

Anyway, this change so that "there is a single websocket connection
between the server and client. " makes implementing an alternative to
WebSocket as a fallback dramatically easier, which I greatly
appreciate.

> Our situation is a bit different though than SMC because most of our current
> users run the notebook on their own computers. This may be something that we

Just curious -- how do you know that most people using IPython aren't
already using it through SMC?

William

>

--

Kyle Kelley (@rgbkrk; http://lambdaops.com)

[Brian Granger]

Just curious -- how do you know that most people using IPython aren't
already using it through SMC?

Great question!

As with all such things, if you are asking for a precise data driven answers, then we don't know.

But here is the constellation of evidence that would lead me to this conclusion (that SMC represents a minority fraction of the overall number of ipython users):

* We estimate that the total ipython user base is in the 0.5-1.5 million range. However, this is likely a very conservative estimate. Fernando has the best summary of where we get these estimates (I don't remember where the most recent information is - probably in a private email to someone). If SMC has this number of users, then my assertion is clearly to be questioned.

* We spend a significant amount of time out meeting with ipython users an conferences, universities, research labs, companies. Over this sample, I know of very few people using SMC (less than 1%). I will be the first to admit that this is both anecdotal and subject my the sampling bias of my particular travel schedule...

* Users get and use ipython through a large number of avenues: anaconda, pip, github, SMC, Canopy, apt-get, yum, etc. Our overall usage numbers come from trying to estimate the install base of each of these things and adding them up - we then double check with things like nbviewer usage, etc. Anecdotally, it appears that anaconda is the most common way that people get and use ipython, but the most reliable numbers we have come from apt-get and yum. 

* Commercial usage of the notebook (in private with private data) is quickly overtaking the more open academic and educational uses of the notebook. The difficulty here is that companies tend to keep their usage private. However, each time we get a glimpse at commercial usage (which is happening more and more) we are blown away by the scale of it. My assumption is that industry usage with private code+data is not the main audience for SMC.

I want to emphasize that I don't in any way mean to minimize the impact or importance of SMC. We all think it is fantastic what you are doing and I personally recommend it to people all the time.

ps - are you willing to share SMC usage stats with us? It would really help us to reduce the error bars on our usage numbers, which is very helpful in talking to funding agencies...

Cheers, Brian

[Fernando Perez]

On Thu, Jan 8, 2015 at 2:08 PM, Brian Granger <elli...@gmail.com> wrote:

* We estimate that the total ipython user base is in the 0.5-1.5 million range. However, this is likely a very conservative estimate. Fernando has the best summary of where we get these estimates (I don't remember where the most recent information is - probably in a private email to someone).

This is how I most recently answered the question "what is the impact and user base of IPython", indeed in the context of a private email as part of the EU Sage-math grant proposal:

Getting metrics for usage is very tough, but we have some information. You can read the report we wrote early this year for Sloan, that provides a bunch of info (notebook and PDF format):

http://nbviewer.ipython.org/github/ipython/sloan-2013-reports/blob/master/IPython%202013%20Progress%20Report%20-%20Sloan%20Foundation.ipynb

http://archive.ipython.org/sloan-2013-reports/IPython%202013%20Progress%20Report%20-%20Sloan%20Foundation.pdf

As you can see, rather than a single blurb of download stats, we paint a more complex (but I think richer) picture of broad usage and impact.  A few more things since:

- Recently highlighted in the cover of Nature:

http://www.nature.com/news/interactive-notebooks-sharing-the-code-1.16261

and that article links to a live, interactive demo here:

http://www.nature.com/news/ipython-interactive-demo-7.21492

This demo broke readership records for Nature, to the best of our understanding (we blew past the estimates the editors gave us in terms of highest numbers of simultaneous readers to expect).

- The NBviewer site has very significant usage, these are the Google Analytics numbers for the period Nov 8, 2014-Dec 8, 2014:

Sessions

423,334

Users

215,546

Pageviews

844,523

That's almost 1 million page views for people sharing content as IPython notebooks (that's the only purpose of that site).

- Our Gallery of notebooks is a good showcase of the breadth of usage and impact of the notebook across domains:

https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks

- This list shows the adoption of our kernel architecture across languages: 23 independent implementations and counting, meaning this system is becoming the lingua franca of interactive computing:

https://github.com/ipython/ipython/wiki/IPython-kernels-for-other-languages

- Here is a list of other projects (in academia and industry) using IPython:

https://github.com/ipython/ipython/wiki/Projects-using-IPython

- It's very hard to get accurate user counts, but we *estimate* at least ~2 Million users.  This is a rough estimate, but I think realistic (and if anything, a conservative undercount). The estimate goes as follows:

a) Estimates of linux users (not datacenter servers) range from 20M for Ubuntu to ~70M across distros:

http://en.wikipedia.org/wiki/Ubuntu_(operating_system)#Installed_base

http://en.wikipedia.org/wiki/Linux_adoption#Measuring_desktop_adoption

b) The Debian 'popcon' statistics:

https://qa.debian.org/popcon.php?package=ipython

show IPython to be installed on ~ 5% of Debian installs. 

If we use that as a baseline, and estimate total linux user counts at say ~50M (rough average of the above two numbers), we get about 2.5M installs of IPython on linux. This doesn't count source installs, pip installs, nor any of the Windows and Mac OSX users who get it via Anaconda, Enthought Canopy, etc.  Nor does it count the increasing number of server-side hosted deployments we see more and more of.  

But at least it gives a credible way to justify that IPython probably has *at least* 2M users.

--

Fernando Perez (@fperez_org; http://fperez.org)
fperez.net-at-gmail: mailing lists only (I ignore this when swamped!)
fernando.perez-at-berkeley: contact me here for any direct mail

[Brian Granger]

Thanks Fernando, that was it exactly!

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAHAreOrQ%2BWvU2V2uxKYrho%3DXGYqSS6F1BrGmZth88kKU0s4Uaw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.