ulimit issues and IPython 5.0

[Enrique Artal]

I start a new thread in this list as a continuation of "How to limit heavy computations" in sage-support. I try to summarize the problem. In my University, we have two servers to run math labs; it is done in several instances of sagenb using server_pool to enhance (at least a little bit) security. The sagenb instances are launched with ulimit options but it seems they have no effect.

In the first term of this academic year, we had several issues affecting the use of this server. Some students started long loops or heavy computations that stop the server (basically their computations blocked one instance and the error messages of the nginx processes grew so much that filled the hard disk). After restarting the machines, the problems came back again; at this point, when we detected such a process it was immediately stopped, but for some reason its instance of sagenb became unusable, causing problems to other students, since it needed restarting (and someone to do it). It was possible (not easy, thanks to Jori Mäntysalo) to identify the actual owner of the process, since stopping the computation by himself caused less harm.

Using ulimit in the OS, namely with the restrictions hard  "as   6000000" and "hard   cpu   10", it worked. If a computation was too heavy, the process stopped with some error message, but the instance was usable by any other user of the sagenb instance (including the person with the long computation). This was true for Sage 7.3, but no more in Sage 7.4 (and 7.5) where the process was stopped by the instance became unusable in few minutes.

Dima Pasechnik proposed to git-bisect and with the help of Miguel Marco, I think I found the patch causing this behavior. I started checking with 7.4 beta and rc and I saw that beta0 had the issue. Looking at the history of changes, it worked after 

2b3eb14 Trac #20621: Simpler code and better error messages in Sequence()

and it didn't work after

9831d4b Trac #21006: Upgrade to IPython 5.0

Is there any chance to look into this change in order to localize and solve the problem?

Regards, Enrique.

[Enrique Artal]

The following may be related, I encountered it doing some administration in our notebooks (sagenb). The following happens in both Sage 7.4 and 7.5 (it executes the orders but the warning may suggest some issue in the relationship of sagenb and ipython). No warning in 7.3

sage: import sys
sage: 1+1
2
sage: import sagenb.notebook.notebook
sage: 1+1
2
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-4cc2ceee564b> in <module>()
----> 1 Integer(1)+Integer(1)

/usr/local/sage-7.4/local/lib/python2.7/site-packages/IPython/core/displayhook.pyc in __call__(self, result)
    244             self.start_displayhook()
    245             self.write_output_prompt()
--> 246             format_dict, md_dict = self.compute_format_data(result)
    247             self.update_user_ns(result)
    248             self.fill_exec_result(result)

/usr/local/sage-7.4/local/lib/python2.7/site-packages/IPython/core/displayhook.pyc in compute_format_data(self, result)
    148
    149         """
--> 150         return self.shell.display_formatter.format(result)
    151
    152     # This can be set to True by the write_output_prompt method in a subclass

/usr/local/sage-7.4/local/lib/python2.7/site-packages/sage/repl/display/formatter.pyc in format(self, obj, include, exclude)
    158         # First, use Sage rich output if there is any
    159         PLAIN_TEXT = u'text/plain'
--> 160         sage_format, sage_metadata = self.dm.displayhook(obj)
    161         assert PLAIN_TEXT in sage_format, 'plain text is always present'
    162         if sage_format.keys() != [PLAIN_TEXT]:

TypeError: 'NoneType' object is not iterable

[Enrique Artal]

As expected, no change in 7.6. I thought about creating a ticket but besides the fact that the upgrade of ipython seems to be responsible I have no idea how.

[Enrique Artal]

I tried again to launch sagenb server (7.6) again avoiding the ulimit restrictions, and it seems ulimit is not responsible for it. After some hours the server does not respond, syslog register tons of these messages:

Apr 19 12:41:47 sage-mtm saged.mtm[15141]: 2017-04-19T12:41:47+0200 [twisted.protocols.tls.TLSMemoryBIOFactory] Could not accept new connection (EMFILE)
Apr 19 12:41:47 sage-mtm saged.mtm[15141]: message repeated 12402 times: [ 2017-04-19T12:41:47+0200 [twisted.protocols.tls.TLSMemoryBIOFactory] Could not accept new con

And after the failure in the command line one gets this:

sage: 1+1
2
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-4cc2ceee564b> in <module>()
----> 1 Integer(1)+Integer(1)

/usr/local/sage-7.6/local/lib/python2.7/site-packages/IPython/core/displayhook.pyc in __call__(self, result)

   244             self.start_displayhook()
   245             self.write_output_prompt()
--> 246             format_dict, md_dict = self.compute_format_data(result)
   247             self.update_user_ns(result)
   248             self.fill_exec_result(result)

/usr/local/sage-7.6/local/lib/python2.7/site-packages/IPython/core/displayhook.pyc in compute_format_data(self, result)

   148
   149         """
--> 150         return self.shell.display_formatter.format(result)
   151
   152     # This can be set to True by the write_output_prompt method in a subclass

/usr/local/sage-7.6/local/lib/python2.7/site-packages/sage/repl/display/formatter.pyc in format(self, obj, include, exclude)

   158         # First, use Sage rich output if there is any
   159         PLAIN_TEXT = u'text/plain'
--> 160         sage_format, sage_metadata = self.dm.displayhook(obj)
   161         assert PLAIN_TEXT in sage_format, 'plain text is always present'
   162         if sage_format.keys() != [PLAIN_TEXT]:

TypeError: 'NoneType' object is not iterable

I checked that before the upgrade of IPython everything works OK. Thanks, Enrique.

[Samuel Lelievre]

Not sure if this is related, but one thing that goes wrong with

the upgrade to IPython 5.0 is that multiline output gets an

extraneous blank line, for example instead of

    sage: identity_matrix(2)

    [1 0]

    [0 1]

we get

    sage: identity_matrix(2)

    [1 0]

    [0 1]

This is solved in IPython 5.3 but Sage has not upgraded to

that version yet. You can fix this on your installation by editing

the "prompts.py" file located at (from your SAGE_ROOT):

    local/lib/python/site-packages/IPython/terminal/prompts.py

and replacing the definition of the RichPromptDisplayHook class

by the version at

https://github.com/takluyver/ipython/blob/83d273e24b28b9c06f0fa4f5bf90581db7c29be8/IPython/terminal/prompts.py

ie

#####

class RichPromptDisplayHook(DisplayHook):

    """Subclass of base display hook using coloured prompt"""

    def write_output_prompt(self):

        sys.stdout.write(self.shell.separate_out)

        # If we're not displaying a prompt, it effectively ends with a newline,

        # because the output will be left-aligned.

        self.prompt_end_newline = True

        if self.do_full_cache:

            tokens = self.shell.prompts.out_prompt_tokens()

            prompt_txt = ''.join(s for t, s in tokens)

            if prompt_txt and not prompt_txt.endswith('\n'):

                # Ask for a newline before multiline output

                self.prompt_end_newline = False

            if self.shell.pt_cli:

                self.shell.pt_cli.print_tokens(tokens)

            else:

                sys.stdout.write(prompt_txt)

#####

In case this extra blank line is interpreted somehow

as something returning None, this might explain why

you get the error message you quote:

    TypeError: 'NoneType' object is not iterable

Just a wild guess, maybe worth a try...

[Enrique Artal]

Not sure either, but I changed the file. Anyway, probably the issue had to do with the number of open files. I added the following lines in limits.conf:

root soft nofile 655360
sage soft nofile 655360
root hard nofile 655360
sage hard nofile 655360

(sage is the user owning sage notebooks). After a reboot the 7.6 notebooks work OK.

[Enrique Artal]

Sorry for the noise. With this modification, the notebook server runs but at some unexpected moment it stops. I saw these error messages in log, maybe related:

Apr 24 14:10:08 sage-mtm saged.mtm[14139]: 2017-04-24T14:10:08+0200 [-] WSGI application error
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011Traceback (most recent call last):
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "/usr/local/sage-7.6/local/lib/python2.7/site-packages/twisted/python/threadpool.py", line 262, in <lambda>
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "/usr/local/sage-7.6/local/lib/python2.7/site-packages/twisted/python/context.py", line 118, in callWithContext
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011    return self.currentContext().callWithContext(ctx, func, *args, **kw)
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "/usr/local/sage-7.6/local/lib/python2.7/site-packages/twisted/python/context.py", line 83, in callWithContext
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011    self.contexts.pop()
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "/usr/local/sage-7.6/local/lib/python2.7/site-packages/twisted/web/wsgi.py", line 521, in run
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011    self.started = True
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011--- <exception caught here> ---
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "/usr/local/sage-7.6/local/lib/python2.7/site-packages/twisted/web/wsgi.py", line 499, in run
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011    self.write(elem)
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "/usr/local/sage-7.6/local/lib/python2.7/site-packages/twisted/web/wsgi.py", line 454, in write
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011    self.reactor, wsgiWrite, self.started)
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "/usr/local/sage-7.6/local/lib/python2.7/site-packages/twisted/internet/threads.py", line 122, in blockingCallFromThread
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011    result.raiseException()
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "<string>", line 2, in raiseException
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011     
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011exceptions.AttributeError: 'NoneType' object has no attribute 'write'
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: 2017-04-24T14:10:08+0200 [-] Unhandled Error
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011Traceback (most recent call last):
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "/usr/local/sage-7.6/local/lib/python2.7/site-packages/twisted/application/app.py", line 390, in startReactor
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011    self.config, oldstdout, oldstderr, self.profiler, reactor)
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "/usr/local/sage-7.6/local/lib/python2.7/site-packages/twisted/application/app.py", line 311, in runReactorWithLogging
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011    reactor.run()
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "/usr/local/sage-7.6/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1194, in run
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011    self.mainLoop()
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "/usr/local/sage-7.6/local/lib/python2.7/site-packages/twisted/internet/base.py", line 1203, in mainLoop
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011    self.runUntilCurrent()
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011--- <exception caught here> ---
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "/usr/local/sage-7.6/local/lib/python2.7/site-packages/twisted/internet/base.py", line 798, in runUntilCurrent
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011    f(*a, **kw)
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "/usr/local/sage-7.6/local/lib/python2.7/site-packages/twisted/web/wsgi.py", line 509, in wsgiError
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011    self.request.loseConnection()
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011  File "/usr/local/sage-7.6/local/lib/python2.7/site-packages/twisted/web/http.py", line 1345, in loseConnection
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011    self.channel.loseConnection()
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011exceptions.AttributeError: 'NoneType' object has no attribute 'loseConnection'
Apr 24 14:10:08 sage-mtm saged.mtm[14139]: #011