Issues with sagenb and ipython

[Enrique Artal]

I have experienced severe troubles in a server running sagenb with sage versions greater than 7.3. I followed the advice to track which ticket was responsible for the troubles and it seems it is the upgrade to Ipython 5. Basically after a certain amount of activity the nb server was gone; after changing some parameters in ulimit (in the server), the nb server remain alive but at some point it disconnected from sage (no actual message indicating why) and it was possible to run some active worksheets but impossible to open any new one. We use server_pool for security and I can run any proposed test to discover the issue. Enrique Artal.

[kcrisman]

Thanks for the post.  It sounds like the discussion has already been started at https://groups.google.com/forum/#!msg/sage-devel/WymRLQtiR4Y/GsSpHywLBQAJ so I will recommend anyone looking to check there too.

I have experienced severe troubles in a server running sagenb with sage versions greater than 7.3. I followed the advice to track which ticket was responsible for the troubles and it seems it is the upgrade to Ipython 5. Basically after a certain amount of activity the nb server was gone; after changing some parameters in ulimit (in the server), the nb server remain alive but at some point it disconnected from sage (no actual message indicating why) and it was possible to run some active worksheets but impossible to open any new one. We use server_pool for security and I can run any proposed test to discover the issue. Enrique Artal.

It seems very likely that IPython upgrades might cause this.  The sagenb standalone still works fine even with Sage 8.0 betas, so it must have something to do with server alone.

Note that http://sagenb.mc.edu runs Sage 7.5 ; I've cc:ed the admin, who may have some comments.

[William Stein]

On Thu, Apr 27, 2017 at 11:33 AM, kcrisman <kcri...@gmail.com> wrote:

Thanks for the post.  It sounds like the discussion has already been started at https://groups.google.com/forum/#!msg/sage-devel/WymRLQtiR4Y/GsSpHywLBQAJ so I will recommend anyone looking to check there too.

I have experienced severe troubles in a server running sagenb with sage versions greater than 7.3. I followed the advice to track which ticket was responsible for the troubles and it seems it is the upgrade to Ipython 5. Basically after a certain amount of activity the nb server was gone; after changing some parameters in ulimit (in the server), the nb server remain alive but at some point it disconnected from sage (no actual message indicating why) and it was possible to run some active worksheets but impossible to open any new one. We use server_pool for security and I can run any proposed test to discover the issue. Enrique Artal.

It seems very likely that IPython upgrades might cause this.  

By "very likely" do you actually mean "very **unlikely**"?    

I ask because -- at least when I wrote it -- there was no use of any of IPython in sagenb at all, and your sentence below doesn't seem to back up your assertion either.

 

The sagenb standalone still works fine even with Sage 8.0 betas, so it must have something to do with server alone.

Note that http://sagenb.mc.edu runs Sage 7.5 ; I've cc:ed the admin, who may have some comments.

--
You received this message because you are subscribed to the Google Groups "sage-notebook" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-notebook+unsubscribe@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-notebook.
For more options, visit https://groups.google.com/d/optout.

--

William (http://wstein.org)

[Enrique Artal]

Thanks both for the feedback. The reason to blame ipython is that I launched services from sage 7.3 at each stage in git (following the info for each beta-release) and it worked OK just before the upgrade of ipython. I start to suspect about some garbage in the directory of the notebook. I started a notebook in a different (and new directory) and i started several worksheets (as admin if is important) with long and nonsense computations. Around 15 worksheets worked ok for hours, and it was possible to open new ones. I started one of the old notebooks with the same worksheets; 15 minutes later these worksheets still worked but if I created a new one, any evaluation failed (1+1 runs forever, or %sh ls, or whatever). No log info is given neither in the terminal where I launched the server (with screen) or in syslog.

El jueves, 27 de abril de 2017, 21:04:09 (UTC+2), William Stein escribió:

On Thu, Apr 27, 2017 at 11:33 AM, kcrisman <kcri...@gmail.com> wrote:

Thanks for the post.  It sounds like the discussion has already been started at https://groups.google.com/forum/#!msg/sage-devel/WymRLQtiR4Y/GsSpHywLBQAJ so I will recommend anyone looking to check there too.

I have experienced severe troubles in a server running sagenb with sage versions greater than 7.3. I followed the advice to track which ticket was responsible for the troubles and it seems it is the upgrade to Ipython 5. Basically after a certain amount of activity the nb server was gone; after changing some parameters in ulimit (in the server), the nb server remain alive but at some point it disconnected from sage (no actual message indicating why) and it was possible to run some active worksheets but impossible to open any new one. We use server_pool for security and I can run any proposed test to discover the issue. Enrique Artal.

It seems very likely that IPython upgrades might cause this.  

By "very likely" do you actually mean "very **unlikely**"?    

I ask because -- at least when I wrote it -- there was no use of any of IPython in sagenb at all, and your sentence below doesn't seem to back up your assertion either.

 

The sagenb standalone still works fine even with Sage 8.0 betas, so it must have something to do with server alone.

Note that http://sagenb.mc.edu runs Sage 7.5 ; I've cc:ed the admin, who may have some comments.

--
You received this message because you are subscribed to the Google Groups "sage-notebook" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-noteboo...@googlegroups.com.

Visit this group at https://groups.google.com/group/sage-notebook.
For more options, visit https://groups.google.com/d/optout.

--

William (http://wstein.org)

[Enrique Artal]

I do not know if it has to do something with the issue. In a term, I have tried to list the users accessing to the notebook with 'import sagenb.notebook.notebook', 'nb = sagenb.notebook.notebook.Notebook(name_of_the_directory), 'ad= = nb.user_manager()' and 'ad.users()'.

 With 7.3, the list is printed. With 7.6, after the list, one gets:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-5-482b3d979d40> in <module>()
----> 1 ad.users()

/usr/local/sage-7.6/local/lib/python2.7/site-packages/IPython/core/displayhook.pyc in __call__(self, result)
   244             self.start_displayhook()
   245             self.write_output_prompt()
--> 246             format_dict, md_dict = self.compute_format_data(result)
   247             self.update_user_ns(result)
   248             self.fill_exec_result(result)

/usr/local/sage-7.6/local/lib/python2.7/site-packages/IPython/core/displayhook.pyc in compute_format_data(self, result)
   148  
   149         """
--> 150         return self.shell.display_formatter.format(result)
   151  
   152     # This can be set to True by the write_output_prompt method in a subclass

/usr/local/sage-7.6/local/lib/python2.7/site-packages/sage/repl/display/formatter.pyc in format(self, obj, include, exclude)
   158         # First, use Sage rich output if there is any
   159         PLAIN_TEXT = u'text/plain'
--> 160         sage_format, sage_metadata = self.dm.displayhook(obj)
   161         assert PLAIN_TEXT in sage_format, 'plain text is always present'
   162         if sage_format.keys() != [PLAIN_TEXT]:

TypeError: 'NoneType' object is not iterable

[Enrique Artal]

Finally I thinkf is that nginx configuration is to blame. The same notebook works if it is launched directly but it stops if one starts around 20 "heavy compuations" with nginx proxy. I do not know what should be modified to make it work. It seems that ipython5 opens more processes or files or whatever. Some ideas?

[kcrisman]

On Saturday, May 13, 2017 at 9:09:44 AM UTC-4, Enrique Artal wrote:

Finally I thinkf is that nginx configuration is to blame. The same notebook works if it is launched directly but it stops if one starts around 20 "heavy compuations" with nginx proxy. I do not know what should be modified to make it work. It seems that ipython5 opens more processes or files or whatever. Some ideas?

Hmm, yeah that is weird because ipython  shouldn't be involved.  But proxies do weird things and I know people have had to work for making them work.  You might possibly find https://github.com/sagemath/sagenb/pull/328 useful here (no guarantees because I know nothing about proxies).

[Enrique Artal]

I still think that something changed ipython probably not directly in sagenb but somewhere in sage affecting to number of files open or active processes, since the error does not happen before the upgrading of ipython (in 7.4beta0 I think). For 7.3 the proxy configuration works (with nginx, I am not able for the moment to do it in apache2) but for 7.6 it stops after some computations (not the active worksheets but any new worksheet); no problem if no proxy is used by the way.

I tried to use site_name but I got an error.

Thanks for your kind help, Enrique.

[Jonathan Gutow]

Enrique,

To the best of my knowledge the pull I requested and wrote that includes site_name and proper hostmonster internal link translation is NOT included in sagenb at present. You would have to install it yourself to get the functionality. I have not tried it with the latest sage releases so do not know if it would need some additional fixes to work.

Jonathan

--

You received this message because you are subscribed to the Google Groups "sage-notebook" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-noteboo...@googlegroups.com.
Visit this group at https://groups.google.com/group/sage-notebook.
For more options, visit https://groups.google.com/d/optout.

-- 
                        Dr. Jonathan H. Gutow
Chemistry Department                                 gu...@uwosh.edu
UW-Oshkosh                                           Office:920-424-1326
800 Algoma Boulevard                                 FAX:920-424-2042
Oshkosh, WI 54901
                http://www.uwosh.edu/facstaff/gutow/

[Enrique Artal]

I have a test installation (including last changes from Dima, I will try to get it installed there.

[Enrique Artal]

I forgot a question. Is it possible to use the apache redirect without the site_name? It is possible with nginx, but I am not able to identify which parameter stops the communication with sage.

[Jonathan Gutow]

If I understand correctly you want your site to reside at the root of the domain. You can do that in apache2. I recommend against that because it reduces flexibility for your site. For example, what happens when you need to bring up a test updated site? If I recall hosting at the root even works OK behind a proxy without the patches I made. The issue is mostly one of security so that your site cannot be spoofed. Ideally all links in a site should be absolute paths, but to do that and run something behind a protective proxy like nginx or apache2 all the links have to be properly rewritten. That is why I implemented the hostmonster support.

Jonathan

[Enrique Artal]

I try to summarize our situation. We have two computers with sagemath installed to be used in math labs for more or less the whole University. In these servers, there are six instances of notebook running on distinct ports, and each worksheet is managed by one of the servers since we are using server_pool (for balance and security). The main server has different dns names and via nginx accessing throug one of these names reroute to the corresponding server.

This scheme has been working with success until 7.3; in 7.4 we started to have problems, each instance stopped when having several worksheets running and the logs didn't help. We discovered that the problem started after switching from ipython 4 to 5, but we did not know what was the exact cause in this huge change.

We made more tests to discover the problem and launching an instance of the notebook at a different port and not using nginx (accessing through any site name but adding the port in the address bar of the browser) the problem disappeared. I tried to change (quite blindly) some parameters in the conf of nginx, no change.

Accessing to the servers without explicitely writing the port is useful for several reasons; it is easier, but moreover, some libraries or residences block any address with a port inside. I was trying to use apache2 to avoid nginx issues but I was not able to make the right configurations. I think we will try to do this next month, when teaching stops, using your updates.

Do you know some experiences of using jupyterhub and sage with jupyter? I Enrique.

[Jonathan Gutow]

Based on your description, I do not think the the work I did will solve your problems. It will make it possible to run properly behind an apache2 proxy. However, my patches did not touch the server pool stuff. I have not looked at this stuff in at least a year, but my recollection is that the server pool was used for backend computations. All user interaction was still done through the main server. Based on this I am going to take a wild guess that the problem is something about how communications is being controlled on your network between the participants in the pool. Have you upgraded any OS stuff recently (OpenSSH or something like that)?

Remember this is a wild guess and my expertise is far afield from this area. I do theoretical and experimental chemistry. I only got involved with this stuff because I was using Sagemath in my classes.

Jonathan

[Enrique Artal]

I do not think server_pool is the key point (I put in the description for completness), since the servers up to 7.3 work fine and also a server with 7.6 without nginx (this is why I wanted to use apache2 or any other proxy method)