SEGV caused by CTL-C in C/C++ code probably related to signals

[Georgi Guninski]

I don't have reproducible testcase, but my pain is that sometimes if I
press CTL-C e.g. when solve_mod() is called in a loop, I get SEGV and
abort.
I suspect besides signals, it is hitting race condition.
Is this a known issue?

Very long ago there was vulnerability in sendmail, where it did
printf() or similar in signal handler and this was documented in the
literature.

[Vincent Delecroix]

I do not remember anything specific about solve_mod. Though, there are
many places in Cython source code where sig_on/sig_off is not handled
carefully enough (and many that were fixed).

The fact that it is not reproducible is not necessarily a huge problem. However,
- Could you share the code (as minimal as possible) that provokes this
on your machine?
- Do you have the complete (C) trace of the crash? You might need to
install gdb depending on your setup.

Vincent

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/CAGUWgD_7t9OTNdo6Ovrt%3DJdNqmGJLPGsZHVoJK0mNPQhuz1CKQ%40mail.gmail.com.

[Nils Bruin]

Also not that there is plenty of code that, in the course of computation, writes into attributes of objects. If you're doing multiple of those it's very hard in python to make such a modification atomic. So if an interrupt ends up splitting an update, you may leave the system in an inconsistent state that could affect future computations. It's a nice aspirational goal to have Ctrl-C work decently, but if you want your results with a reasonably degree of confidence in correctness, you should probably avoid interruptions anyway.

Most code is hopefully written defensively: work on a local copy and only in the last bit modify the attribute by a single assignment. That minimizes the chance for getting inconsistent states from interrupts, but I don't think you'll get it to zero, because of the non-atomicity of multiple attribute assignments.

[Georgi Guninski]

Giving short testcase, can someone else reproduce it?
Also attaching backtrace.
On second thought, if I don't crash, there could be some memory
corruption and sage runs with screwed memory.
```
x,y=var('x,y');n=10**6
while True: so=solve_mod(x*y-1,n)
#press CTL-C, crashes with probability about 1/3
```

[dim...@gmail.com]

On Thu, Apr 11, 2024 at 09:27:15AM +0300, Georgi Guninski wrote:
> Giving short testcase, can someone else reproduce it?
> Also attaching backtrace.

yes, I can reproduce this on a recent 10.4.beta - the backtrace I get it
more or less the same as the one attached - and it's in an unexpected to
me place, in src/sage/rings/finite_rings/integer_mod_ring.py.
If I read the backtrace right,
sa2si_ZZmod() - in Cython interface to Singular -
is called from Python code, then sa2si_ZZmod() calls characteristic() in the line

nr2mModul = d.parent().characteristic()

As sa2si_ZZmod() is declared noexcept(), we see

Exception ignored in: 'sage.libs.singular.singular.sa2si_ZZmod'

But then in characteristic() an attempt to handle the exception is made,
and it leads to a crash. Should

nr2mModul = d.parent().characteristic()

be wrapped in some cysignals functions, or we currently just have to
means to deal with it?

--------------------------------------------------------------------------------------------------------

KeyboardInterrupt:
sage: x,y=var('x,y');n=10**6
....: while True: so=solve_mod(x*y-1,n)
^C---------------------------------------------------------------------------
KeyboardInterrupt Traceback (most recent call last)
File /mnt/opt/Sage/sage-dev/src/sage/rings/finite_rings/integer_mod_ring.py:1029, in IntegerModRing_generic.characteristic(self)
1025 ans.append(Factorization([(p, e - 1)]) *
1026 factor(p - 1, int_=(self.__order < 2**31)))
1027 return ans
-> 1029 def characteristic(self):
1030 """
1031 EXAMPLES::
1032
(...)
1038 18
1039 """
1040 return self.__order

File src/cysignals/signals.pyx:341, in cysignals.signals.python_check_interrupt()

KeyboardInterrupt:
Exception ignored in: 'sage.libs.singular.singular.sa2si_ZZmod'
Traceback (most recent call last):
File "/mnt/opt/Sage/sage-dev/src/sage/rings/finite_rings/integer_mod_ring.py", line 1029, in characteristic
def characteristic(self):

File "src/cysignals/signals.pyx", line 341, in cysignals.signals.python_check_interrupt
KeyboardInterrupt:
------------------------------------------------------------------------
/usr/lib/python3.11/site-packages/cysignals/signals.cpython-311-x86_64-linux-gnu.so(+0x9beb)[0x7f0fea9a6beb]
/usr/lib/python3.11/site-packages/cysignals/signals.cpython-311-x86_64-linux-gnu.so(+0x9caf)[0x7f0fea9a6caf]
/usr/lib/python3.11/site-packages/cysignals/signals.cpython-311-x86_64-linux-gnu.so(+0xb7d0)[0x7f0fea9a87d0]
/lib64/libc.so.6(+0x39660)[0x7f0ff85a0660]
/usr/lib64/libgmp.so.10(__gmpz_cmp_ui+0x0)[0x7f0fe9b96a20]
/usr/lib64/libpolys-4.3.2.so(+0xf9dce)[0x7f0e9ceb1dce]
/mnt/opt/Sage/sage-dev/src/sage/libs/singular/polynomial.cpython-311-x86_64-linux-gnu.so(+0x7427)[0x7f0e9d22d427]
/mnt/opt/Sage/sage-dev/src/sage/rings/polynomial/multi_polynomial_libsingular.cpython-311-x86_64-linux-gnu.so(+0x4c072)[0x7f0e9d284072]
/mnt/opt/Sage/sage-dev/src/sage/rings/polynomial/multi_polynomial_libsingular.cpython-311-x86_64-linux-gnu.so(+0x4c908)[0x7f0e9d284908]
/usr/lib64/libpython3.11.so.1.0(+0x1acf22)[0x7f0ff88d1f22]
/mnt/opt/Sage/sage-dev/src/sage/rings/polynomial/polynomial_element.cpython-311-x86_64-linux-gnu.so(+0x2d6ea)[0x7f0fa1bb16ea]
/mnt/opt/Sage/sage-dev/src/sage/rings/polynomial/polynomial_element.cpython-311-x86_64-linux-gnu.so(+0x2dc8d)[0x7f0fa1bb1c8d]
/usr/lib64/libpython3.11.so.1.0(+0x1acf22)[0x7f0ff88d1f22]
/mnt/opt/Sage/sage-dev/src/sage/categories/map.cpython-311-x86_64-linux-gnu.so(+0x13db0)[0x7f0fe93e8db0]
/mnt/opt/Sage/sage-dev/src/sage/structure/parent.cpython-311-x86_64-linux-gnu.so(+0x2dbb9)[0x7f0fe9445bb9]
/mnt/opt/Sage/sage-dev/src/sage/rings/polynomial/multi_polynomial_libsingular.cpython-311-x86_64-linux-gnu.so(+0x6fe67)[0x7f0e9d2a7e67]
/usr/lib64/libpython3.11.so.1.0(_PyObject_Call+0x68)[0x7f0ff887fa28]
/usr/lib64/libpython3.11.so.1.0(_PyEval_EvalFrameDefault+0x57e3)[0x7f0ff8825613]
/usr/lib64/libpython3.11.so.1.0(+0x1741f5)[0x7f0ff88991f5]
/usr/lib64/libpython3.11.so.1.0(+0x252600)[0x7f0ff8977600]
/usr/lib64/libpython3.11.so.1.0(_PyEval_EvalFrameDefault+0x6daf)[0x7f0ff8826bdf]
/usr/lib64/libpython3.11.so.1.0(+0x1741f5)[0x7f0ff88991f5]
/usr/lib64/libpython3.11.so.1.0(+0x186381)[0x7f0ff88ab381]
/usr/lib64/libpython3.11.so.1.0(+0x1867cc)[0x7f0ff88ab7cc]
/usr/lib64/libpython3.11.so.1.0(_PyEval_EvalFrameDefault+0x7034)[0x7f0ff8826e64]
/usr/lib64/libpython3.11.so.1.0(PyEval_EvalCode+0x22c)[0x7f0ff897fecc]
/usr/lib64/libpython3.11.so.1.0(+0x255b50)[0x7f0ff897ab50]
/usr/lib64/libpython3.11.so.1.0(_PyEval_EvalFrameDefault+0x699d)[0x7f0ff88267cd]
/usr/lib64/libpython3.11.so.1.0(+0x1738dd)[0x7f0ff88988dd]
/usr/lib64/libpython3.11.so.1.0(_PyEval_EvalFrameDefault+0xa61d)[0x7f0ff882a44d]
/usr/lib64/libpython3.11.so.1.0(+0x1738dd)[0x7f0ff88988dd]
/usr/lib64/libpython3.11.so.1.0(_PyEval_EvalFrameDefault+0xa61d)[0x7f0ff882a44d]
/usr/lib64/libpython3.11.so.1.0(+0x174a95)[0x7f0ff8899a95]
/usr/lib64/libpython3.11.so.1.0(+0x1681c4)[0x7f0ff888d1c4]
/usr/lib64/libpython3.11.so.1.0(PyObject_Vectorcall+0x33)[0x7f0ff8880313]
/usr/lib64/libpython3.11.so.1.0(_PyEval_EvalFrameDefault+0x42f8)[0x7f0ff8824128]
/usr/lib64/libpython3.11.so.1.0(PyEval_EvalCode+0x22c)[0x7f0ff897fecc]
/usr/lib64/libpython3.11.so.1.0(+0x2a5779)[0x7f0ff89ca779]
/usr/lib64/libpython3.11.so.1.0(_PyRun_SimpleFileObject+0x164)[0x7f0ff89cc074]
/usr/lib64/libpython3.11.so.1.0(_PyRun_AnyFileObject+0x3c)[0x7f0ff89cc6cc]
/usr/lib64/libpython3.11.so.1.0(Py_RunMain+0x880)[0x7f0ff89ecd50]
/usr/lib64/libpython3.11.so.1.0(Py_BytesMain+0x57)[0x7f0ff89ed2c7]
/lib64/libc.so.6(+0x23eea)[0x7f0ff858aeea]
/lib64/libc.so.6(__libc_start_main+0x85)[0x7f0ff858afa5]
python3(_start+0x21)[0x55b5f92ee081]
------------------------------------------------------------------------
Attaching gdb to process id 19968.
Traceback (most recent call last):
File "/usr/lib/python-exec/python3.11/cysignals-CSI", line 225, in <module>
main(args)
File "/usr/lib/python-exec/python3.11/cysignals-CSI", line 174, in main
trace = run_gdb(args.pid, not args.nocolor)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python-exec/python3.11/cysignals-CSI", line 98, in run_gdb
stdout, stderr = cmd.communicate(gdb_commands(pid, color))
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python-exec/python3.11/cysignals-CSI", line 71, in gdb_commands
with open(script, 'rb') as f:
^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/usr/lib/python-exec/python3.11/../share/cysignals/cysignals-CSI-helper.py'
------------------------------------------------------------------------
Unhandled SIGSEGV: A segmentation fault occurred.
This probably occurred because a *compiled* module has a bug
in it and is not properly wrapped with sig_on(), sig_off().
Python will now terminate.
------------------------------------------------------------------------
/mnt/opt/Sage/sage-dev/src/bin/sage-python: line 2: 19968 Segmentation fault sage -python "$@"
dima@hilbert /mnt/opt/Sage/sage-dev $

> On second thought, if I don't crash, there could be some memory
> corruption and sage runs with screwed memory.
> ```
> x,y=var('x,y');n=10**6
> while True: so=solve_mod(x*y-1,n)
> #press CTL-C, crashes with probability about 1/3
> ```
>

[...]

Dima

[Georgi Guninski]

Are the non-crashing codepaths in consistent state?
SEGV is in general sign of memory corruption and it may lead to wrong
math results or possibly have security implications.

[dim...@gmail.com]

Here is an attempt to fix it; it appears that the call to
characteristic() which causes a crash is a leftover which can simply be
removed.
https://github.com/dimpase/sage/pull/6
(not yet a PR to the Sage repo, just want to see results of tests)

Dima

[Dima Pasechnik]

This should be fixed by https://github.com/sagemath/sage/pull/37792

please review

[Georgi Guninski]

On Fri, Apr 12, 2024 at 11:35 AM Dima Pasechnik <dim...@gmail.com> wrote:
>
> This should be fixed by https://github.com/sagemath/sage/pull/37792
> please review
>

I suspect reproducing is hard, since it depends on the time of pressing CTL-C.
Also, lack a crash may be silent memory corruption.

It is weird that the crash is not always.

Do you fix the attached new testcase (no interaction required, may
need changing some constants)?

[Dima Pasechnik]

On 12 April 2024 12:42:39 CEST, Georgi Guninski <ggun...@gmail.com> wrote:
>On Fri, Apr 12, 2024 at 11:35 AM Dima Pasechnik <dim...@gmail.com> wrote:
>>
>> This should be fixed by https://github.com/sagemath/sage/pull/37792
>> please review
>>
>
>I suspect reproducing is hard, since it depends on the time of pressing CTL-C.

Well, in all my tries the crash happened in the same place, the place that now is no longer reachable.

And the reason for the crash was, AFAIU, due to calling a plain Python code (where Ctrl-C was caught) from a "noexcept" Cython code. That is, it's a crash due to inability to properly process Ctrl-C.

How many more similar places are left there in Cython code in Sage, I don't know.

Dima

[Georgi Guninski]

My results differ. If you want more systematic approach for QA and security
(the recent paranoia induced by the xz backdoor) you will need moderate
budget possibly with zero trust to me.

Have a nice weekend.

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/C701D76E-A116-4D83-B427-24C305F51201%40gmail.com.

[julian...@fsfe.org]

Thanks a lot for this really quick fix Dima. I gave it positive review already.

On Friday, April 12, 2024 at 4:27:31 PM UTC+3 Georgi Guninski wrote:

My results differ. If you want more systematic approach for QA and security
(the recent paranoia induced by the xz backdoor) you will need moderate
budget possibly with zero trust to me.

Georgi, I don't understand what you are trying to say here exactly. I guess with "my results differ" you mean that the crash happens in different places? But why should we have "zero trust" in you?

julian