Computing a Groebner Basis (singular algorithm) raises an error

[Fabian Weise]

Hello there,

as you can see below I am defining a polynomial ring, an ideal and computing its Groebner Basis in a for loop (iterating threw different .xml files).

During the computation I set an alarm for 5s (in the example below still 60s) and let the program print a message in the error case. But since that happened the next computation raises an error by the groebner_basis() function.

In the output below you can see that the first computation is successful, the seconds aborts and the third raises an error due to some kind of "basering" definition. Googling I came up with this (http://www.gap-system.org/Manuals/pkg/singular/doc/chap1.html#X82260C8E82090E87, cp 1-7-5), because I am using the Singular algorithm to compute the Groebner Basis.

But since I do not know how to fix this in Sage, I please for your help.

Thanks for any answers.

With best regards

Fabian

1 Cassou_1.xml

Multivariate Polynomial Ring in b, c, d, e over Rational Field

Degree reverse lexicographic term order

1 loops, best of 1: 219.62 ms per loop

1 Cohn_2.xml

Multivariate Polynomial Ring in t, x, y, z over Rational Field

Degree reverse lexicographic term order

abort after 60s

1 Curves.curve10_20.xml

Multivariate Polynomial Ring in x0, x1, x2, x3, x4, x5, x6, x7, x8, x9, x10, u, v over Rational Field

Degree reverse lexicographic term order

// ** redefining i_par **

// ** redefining # **

// ** redefining P **

Exception RuntimeError: RuntimeError('Error raised calling singular function',) in 'sage.libs.singular.function.LibraryCallHandler.handle_call' ignored

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

/Applications/sage/local/lib/python2.7/site-packages/sage/all_cmdline.pyc in <module>()

----> 1 load('/Users/kenneth/Desktop/example.sage')

/Applications/sage/local/lib/python2.7/site-packages/sage/structure/sage_object.so in sage.structure.sage_object.load (sage/structure/sage_object.c:9540)()

/Applications/sage/local/lib/python2.7/site-packages/sage/misc/preparser.pyc in load(filename, globals, attach)

   1771             # Preparse in memory only for speed.

   1772             exec_file_is(fpath)

-> 1773             exec preparse_file(open(fpath).read()) + "\n" in globals

   1774     elif fpath.endswith('.spyx') or fpath.endswith('.pyx'):

   1775         exec_file_is(fpath)

/Applications/sage/local/lib/python2.7/site-packages/sage/all_cmdline.pyc in <module>()

/Applications/sage/local/lib/python2.7/site-packages/sage/misc/sage_timeit_class.so in sage.misc.sage_timeit_class.SageTimeit.__call__ (sage/misc/sage_timeit_class.c:1177)()

/Applications/sage/local/lib/python2.7/site-packages/sage/misc/sage_timeit_class.so in sage.misc.sage_timeit_class.SageTimeit.eval (sage/misc/sage_timeit_class.c:965)()

/Applications/sage/local/lib/python2.7/site-packages/sage/misc/sage_timeit.pyc in sage_timeit(stmt, globals_dict, preparse, number, repeat, precision, seconds)

    241                     break

    242 

--> 243         series = [s/number for s in timer.repeat(repeat, number)]

    244         best = min(series)

    245 

/Applications/sage/local/lib/python/timeit.pyc in repeat(self, repeat, number)

    221         r = []

    222         for i in range(repeat):

--> 223             t = self.timeit(number)

    224             r.append(t)

    225         return r

/Applications/sage/local/lib/python/timeit.pyc in timeit(self, number)

    193         gc.disable()

    194         try:

--> 195             timing = self.inner(it, self.timer)

    196         finally:

    197             if gcold:

/Applications/sage/local/lib/python2.7/site-packages/sage/all_cmdline.pyc in inner(_it, _timer)

/Applications/sage/local/lib/python2.7/site-packages/sage/misc/cachefunc.so in sage.misc.cachefunc.CachedMethodCaller.__call__ (sage/misc/cachefunc.c:9194)()

/Applications/sage/local/lib/python2.7/site-packages/sage/misc/cachefunc.so in sage.misc.cachefunc.CachedMethod._instance_call (sage/misc/cachefunc.c:11887)()

/Applications/sage/local/lib/python2.7/site-packages/sage/rings/polynomial/multi_polynomial_ideal.pyc in groebner_basis(self, algorithm, deg_bound, mult_bound, prot, *args, **kwds)

   3825         if algorithm is '':

   3826             try:

-> 3827                 gb = self._groebner_basis_libsingular("groebner", deg_bound=deg_bound, mult_bound=mult_bound, *args, **kwds)

   3828             except (TypeError,NameError) as msg: # conversion to Singular not supported

   3829                 try:

/Applications/sage/local/lib/python2.7/site-packages/sage/rings/polynomial/multi_polynomial_ideal.pyc in wrapper(*args, **kwds)

    503         """

    504         with LibSingularDefaultContext():

--> 505             return func(*args, **kwds)

    506     return wrapper

    507 

/Applications/sage/local/lib/python2.7/site-packages/sage/rings/polynomial/multi_polynomial_ideal.pyc in _groebner_basis_libsingular(self, algorithm, *args, **kwds)

    836             S = slimgb_libsingular(self)

    837         elif algorithm == "groebner":

--> 838             S = groebner(self)

    839         else:

    840             try:

/Applications/sage/local/lib/python2.7/site-packages/sage/libs/singular/function.so in sage.libs.singular.function.SingularFunction.__call__ (sage/libs/singular/function.cpp:13039)()

/Applications/sage/local/lib/python2.7/site-packages/sage/libs/singular/function.so in sage.libs.singular.function.call_function (sage/libs/singular/function.cpp:14576)()

RuntimeError: Error in Singular function call 'groebner':

 `basering` is undefined

 error occurred in or before standard.lib::groebner line 887: `  def P=basering;`

[Tracy Hall]

Hello,

Has this ever been addressed? Five years later this is still happening. Something appears to be fundamentally broken in the way that Sage is using the Singular C library; in particular it seems to have undefined behavior in subsequent calls after it has been interrupted by alarm().

Things I have observed:

1. Memory leaks. (In a long series of calculations, the ones that run too long are interrupted. Even though all variables are redefined, garbage collection appears to miss cleaning up the calculations that are interrupted, because the machine runs out of memory unless the kernel is restarted occasionally.)

2. A string of redefinition messages to standard error such as

  // ** redefining # (parameter def i_par; parameter list #; )

that occur, not when a calculation is interrupted, but only after a different calculation is subsequently started, when the same messages never appear in the case that all calculations complete before their alarms.

3. Unrelated Python-based Sage code hanging (using the CPU but making zero progress) after Groebner basis calculations have been interrupted by an alarm, different Groebner basis calculations have subsequently been executed during some fraction of a second, then control has been passed back to pure Python code.

4. The exact error reported by Fabian Weise below.

It makes me worry that the database of calculation results I have been building, many of them having been performed after a different calculation timed out, may not be reliable.

Should a warning be added to the alarm() documentation that it is incompatible with computations involving multivariable polynomials? That would be a pity, because having access to alarm() is the difference between my being able to complete my current research project using Sage or not. The thousands of Groebner basis calculations I want to do can be stated in many different ways, and there appears to be a variation of several orders of magnitude in how long they can take, so that the way to complete them all is to interrupt the slow ones and try them again phrased differently.

—Tracy Hall

[Jeroen Demeyer]

On 2019-05-27 16:11, Tracy Hall wrote:
> in particular it seems to have undefined
> behavior in subsequent calls after it has been interrupted by alarm().

This is simply a fact of life and not really considered to be a bug. You
should not rely on the fact that interruptions (using either CTRL-C or
alarm) work perfectly. Don't use alarm() for serious computations.

If you need proper timeouts, one thing you can do is spawn multiple Sage
processes or fork one Sage process. Then you can kill the whole process
after a certain amount of time.

[Tracy Hall]

Thanks for the reply. That's disappointing—I'm not crazy about the idea of spawning a hundred million Sage processes, most of them to do about 1/50th of a second's worth of calculation, just because a few of them will need to be timed out before they take hours or days or run out of memory. I'm finding ways to work around it and still work within a Jupyter worksheet, but I guess that means I'll have to verify any results I really care about in their own standalone session, if every interruption is considered to transform Sage into unreliable software as a fact of life.

Is there any sort of a command that I can run after interrupting a Groebner basis calculation that will purge the Singular state, that will clear whatever intermediate results Singular seems to be keeping in memory after an interruption?

If anyone is curious to see an example of the sort of behavior I was talking about, the following sequence seems to trip the error in a fairly reproducible way (for example, I got it to work on the CoCalc server). Notice that at the end the same calculation repeated three times results in different behavior each time, with the first one giving a runtime error.

----------------------------------------------------------------

sage: P = PolynomialRing(QQ, ['a' + str(ii) for ii in range(8)])

sage: P.inject_variables()

Defining a0, a1, a2, a3, a4, a5, a6, a7

sage: anideal = ideal(-a0*a2*a4 + a0*a4*a5,

....:                 -a2*a4 + a4*a5,

....:                 a0*a2*a3*a7 - a0*a1*a4,

....:                 a0*a2*a3 + a0*a2*a7 + a2*a3*a7 - a0*a4 - a1*a4 - a3*a6,

....:                 a0*a1*a3*a7 + a0*a2 + a2*a3 + a2*a7 - a4 - a6,

....:                 a0*a1*a3 + a0*a1*a7 + a0*a3*a7 + a1*a3*a7 + a2,

....:                 a0*a1 + a0*a3 + a1*a3 + a0*a7 + a1*a7 + a3*a7,

....:                 a0 + a1 + a3 + a7 - 1)

sage: for ii in range(1):

....:     alarm(10)

....:     try:

....:         if prod(P.gens())^2 in anideal:

....:             cancel_alarm()

....:             print('Success: True.')

....:             break

....:         else:

....:             cancel_alarm()

....:             print('Success: False.')

....:             break

....:     except AlarmInterrupt:

....:         print('Timeout.')

....:         break

....:

Timeout.

sage: P = PolynomialRing(QQ, ['w', 'y', 'z'])

sage: P.inject_variables()

Defining w, y, z

sage: anideal = ideal(w^20*y^10, z^5)

sage: anideal.radical()

---------------------------------------------------------------------------

RuntimeError                              Traceback (most recent call last)

<ipython-input-9-1b5ff2b0d4f2> in <module>()

----> 1 anideal.radical()

/opt/sagemath-8.6/local/lib/python2.7/site-packages/sage/rings/polynomial/multi_polynomial_ideal.pyc in __call__(self, *args, **kwds)

    295         if not R.base_ring().is_field():

    296             raise ValueError("Coefficient ring must be a field for function '%s'."%(self.f.__name__))

--> 297         return self.f(self._instance, *args, **kwds)

    298

    299 require_field = RequireField

/opt/sagemath-8.6/local/lib/python2.7/site-packages/sage/libs/singular/standard_options.pyc in wrapper(*args, **kwds)

    138         """

    139         with LibSingularGBDefaultContext():

--> 140             return func(*args, **kwds)

    141     return wrapper

/opt/sagemath-8.6/local/lib/python2.7/site-packages/sage/rings/polynomial/multi_polynomial_ideal.pyc in radical(self)

   1636         import sage.libs.singular.function_factory

   1637         radical = sage.libs.singular.function_factory.ff.primdec__lib.radical

-> 1638         r = radical(self)

   1639

   1640         S = self.ring()

/opt/sagemath-8.6/local/lib/python2.7/site-packages/sage/libs/singular/function.pyx in sage.libs.singular.function.SingularFunction.__call__ (build/cythonized/sage/libs/singular/function.cpp:15056)()

   1328         if not (isinstance(ring, MPolynomialRing_libsingular) or isinstance(ring, NCPolynomialRing_plural)):

   1329             raise TypeError("Cannot call Singular function '%s' with ring parameter of type '%s'"%(self._name,type(ring)))

-> 1330         return call_function(self, args, ring, interruptible, attributes)

   1331

   1332     def _instancedoc_(self):

/opt/sagemath-8.6/local/lib/python2.7/site-packages/sage/libs/singular/function.pyx in sage.libs.singular.function.call_function (build/cythonized/sage/libs/singular/function.cpp:17032)()

   1526     if errorreported:

   1527         errorreported = 0

-> 1528         raise RuntimeError("error in Singular function call %r:\n%s" %

   1529             (self._name, "\n".join(error_messages)))

   1530

RuntimeError: error in Singular function call 'radical':

no ring active

sage: anideal.radical()

// ** redefining i (parameter ideal i; parameter list #;  )

// ** redefining Pl (  list Pl=ringlist(P0);)

Ideal (z, w*y) of Multivariate Polynomial Ring in w, y, z over Rational Field

sage: anideal.radical()

Ideal (z, w*y) of Multivariate Polynomial Ring in w, y, z over Rational Field

[Daniel Krenn]

On 27.05.19 18:08, Jeroen Demeyer wrote:
> On 2019-05-27 16:11, Tracy Hall wrote:
>> in particular it seems to have undefined
>> behavior in subsequent calls after it has been interrupted by alarm().
>
> This is simply a fact of life and not really considered to be a bug. You
> should not rely on the fact that interruptions (using either CTRL-C or
> alarm) work perfectly. Don't use alarm() for serious computations.

Still, this is kind of annoying (I agree with the original poster), and
it would be great to find an easy applicable solution.

@Jeron (or whoever): Isn't there anything we can do to make it work
somehow? (Unfortunately, I do not know enough about this part of SageMath.)

Daniel

[Daniel Krenn]

On 27.05.19 20:40, Tracy Hall wrote:
> If anyone is curious to see an example

> [...]

> sage: for ii in range(1):
> ....:     alarm(10)
> ....:     try:
> ....:         if prod(P.gens())^2 in anideal:
> ....:             cancel_alarm()

> [...]

> ....:     except AlarmInterrupt:
> ....:         print('Timeout.')
> ....:         break

+1 for trying to get something like this to work in SageMath; it would
be very convenient.

[Nils Bruin]

On Monday, May 27, 2019 at 11:40:00 AM UTC-7, Tracy Hall wrote:

Thanks for the reply. That's disappointing—I'm not crazy about the idea of spawning a hundred million Sage processes, most of them to do about 1/50th of a second's worth of calculation, just because a few of them will need to be timed out before they take hours or days or run out of memory.

That shouldn't be quite necessary. A normal way of setting up longer-running computations is to write "check points" at regular intervals. In your case, that could consist of writing the results of each individual case to a file (and be sure to flush buffers when you do, so that the file is actually updated -- open file, append line, close file is the simple way of ensuring that happens). When your long-running process crashes, you know which things were completed, so you can restart at the point where things were left off. An advantage of such a setup is that it also lends itself to parallelization.

If there is a chance that some jobs won't finish, you could run the process with a CPU time quota, so that it gets killed when it has used up a lot of time. The case it was working on at that time either got killed because it was unlucky to be scheduled when almost no quota was left or because it took a lot of time itself. That's easily figured out by rerunning this one job by itself.

You'd not be spawning a hundred million processes, just perhaps a few hundred over the course of the computation. Setting the quota gives you a trade-off between how many restarts and how much time you want to waste on impossible jobs.

It's only after an unexpected interruption that we can't fully trust the state of various libraries (and quite frankly, I wouldn't trust any computer algebra system under those conditions. It requires a lot of work and overhead to ensure that interruptions leave the system in an entirely consistent state (and even then -- which one?). I don't trust programmers of high-performance computational software to not cut corners on that somewhere (if they even try).

The reason why they probably don't try is because for long-running computations you need to checkpoint anyway, which means the cost of restarting isn't huge, so it's probably a price worth paying for the increased performance of not caring about interrupt clean-up too much.

It's worth learning the general tools that unix provides for setting quota and for scripting batch jobs in a restartable way, because they are applicable to all software. It's much more efficient than equipping each individual program with such tools and requiring people to learn the particulars of that one program.