Python 3 and flushing output from external libraries

[John H Palmieri]

I am writing to ask for help fixing a Python 3 problem. On some platforms, there are Python 3 doctest failures in

- libs/eclib/interface.py (#28454)

- rings/polynomial/polynomial_rational_flint.pyx (#28334)

and both occur for the same reason: warning messages printed by C or C++ libraries are not appearing when they are supposed to, but rather some time later. So for example, in

 FLINT memory errors do not crash Sage (:trac:`17629`)::

     sage: t^(sys.maxsize//2)
     Traceback (most recent call last):
     ...
     RuntimeError: FLINT exception

the "..." should capture a message "Exception (FLINT memory_manager). Unable to allocate memory", and when you run it from the command line, this message appears. When running doctests, though, with Python 3 only and on some platforms only (debian, ubuntu — don't know why), this message only pops up at some later time, leading to this:

Failed example:
   G = f.galois_group(); G
Expected:
   Transitive group number 5 of degree 4
Got:
   Exception (FLINT memory_manager). Unable to allocate memory.
   Transitive group number 5 of degree 4

The `f.galois_group()` command has nothing to do with this error, but something about it triggers the old output to be flushed.

How do we fix this? Note that patching FLINT is not ideal, since it won't help users who are using the system's version of the library. FLINT's most recent release was in 2015, so I don't know how much we can expect from upstream regarding new releases. So maybe the ideal solution would be to get Python to flush the output from these external libraries. I don't know how to do this; can anyone else help?

I'm also curious about what's causing the problem. Is it a bug in Python 3? If not, is this change in behavior documented anywhere? Why does it only appear on some platforms?

-- 
John

[Jeroen Demeyer]

On Mon, Sep 9, 2019 at 7:51 PM John H Palmieri <jhpalm...@gmail.com> wrote:
> I am writing to ask for help fixing a Python 3 problem. On some platforms, there are Python 3 doctest failures in
>
> - libs/eclib/interface.py (#28454)

I believe that this has been fixed upstream, it has certainly been
reported by me. Are we using the latest eclib version?

> How do we fix this? Note that patching FLINT is not ideal, since it won't help users who are using the system's version of the library. FLINT's most recent release was in 2015, so I don't know how much we can expect from upstream regarding new releases. So maybe the ideal solution would be to get Python to flush the output from these external libraries. I don't know how to do this; can anyone else help?

Ideally, the upstream library should properly flush when producing
output. They should use fflush (in C) or std::endl or std::flush (in
C++). I guess that Sage could also do the flushing whenever we return
from a library API call.

> I'm also curious about what's causing the problem. Is it a bug in Python 3?

It's just a completely different implementation of I/O, independent of
the C standard I/O library (Python 2 uses the <stdio.h> functionality)

> Why does it only appear on some platforms?

I don't think it should be platform-dependent. Let's not focus on this too much.

PS: apologies for the quick and short email, I need to sleep ;-)

[John H Palmieri]

On Monday, September 9, 2019 at 2:06:21 PM UTC-7, Jeroen Demeyer wrote:

On Mon, Sep 9, 2019 at 7:51 PM John H Palmieri <jhpalm...@gmail.com> wrote:
> I am writing to ask for help fixing a Python 3 problem. On some platforms, there are Python 3 doctest failures in
>
> - libs/eclib/interface.py (#28454)

I believe that this has been fixed upstream, it has certainly been
reported by me. Are we using the latest eclib version?

John Cremona is part of the discussion at #28454, and I assume he knows the current state of eclib. I'll let him speak to this.

> How do we fix this? Note that patching FLINT is not ideal, since it won't help users who are using the system's version of the library. FLINT's most recent release was in 2015, so I don't know how much we can expect from upstream regarding new releases. So maybe the ideal solution would be to get Python to flush the output from these external libraries. I don't know how to do this; can anyone else help?

Ideally, the upstream library should properly flush when producing
output. They should use fflush (in C) or std::endl or std::flush (in
C++).

I can confirm that patching flint to use fflush fixes the problem, and similarly for eclib. Patching is not ideal, though, since we are trying to move to better support libraries installed system-wide.

 

I guess that Sage could also do the flushing whenever we return
from a library API call.

Sounds okay to me, but I don't know how to do it. My impression is that `sys.stdout.flush()` just flushes Python output, not output coming from external libraries.

[John Cremona]

On Mon, 9 Sep 2019 at 22:52, John H Palmieri <jhpalm...@gmail.com> wrote:

On Monday, September 9, 2019 at 2:06:21 PM UTC-7, Jeroen Demeyer wrote:

On Mon, Sep 9, 2019 at 7:51 PM John H Palmieri <jhpalm...@gmail.com> wrote:
> I am writing to ask for help fixing a Python 3 problem. On some platforms, there are Python 3 doctest failures in
>
> - libs/eclib/interface.py (#28454)

I believe that this has been fixed upstream, it has certainly been
reported by me. Are we using the latest eclib version?

John Cremona is part of the discussion at #28454, and I assume he knows the current state of eclib. I'll let him speak to this.

Until yesterday it was not the case that eclib flushed all output in error situations such as the ones relevant here.  I spent time yesterday dealing with that, and the next eclib version will have this desirable property.  I hope to have the new verson Sage-ready very soon.

I think that Jeroen is remembering a similar situation which happened before, where we managed to avoid the problem by doing the flushing from within the Sage-eclib interface (e.g. see line 203 of sage.libs.eclib.wrap.cpp).  Such methods could probably be used again here, as a shorter fix for (the eclib part of) this issue.  But JHP's question was a broader one.

Since we cannot trust these C/C++ libraries to flush their output, the wrapping code in Sage must do it.  I think this is better than patching the upstream sources, and will go off to #28454 to suggest this.

John

 

> How do we fix this? Note that patching FLINT is not ideal, since it won't help users who are using the system's version of the library. FLINT's most recent release was in 2015, so I don't know how much we can expect from upstream regarding new releases. So maybe the ideal solution would be to get Python to flush the output from these external libraries. I don't know how to do this; can anyone else help?

Ideally, the upstream library should properly flush when producing
output. They should use fflush (in C) or std::endl or std::flush (in
C++).

I can confirm that patching flint to use fflush fixes the problem, and similarly for eclib. Patching is not ideal, though, since we are trying to move to better support libraries installed system-wide.

 

I guess that Sage could also do the flushing whenever we return
from a library API call.

Sounds okay to me, but I don't know how to do it. My impression is that `sys.stdout.flush()` just flushes Python output, not output coming from external libraries.

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/4df27a03-a625-47bb-ad4c-f2580f4b1298%40googlegroups.com.

[John H Palmieri]

On Monday, September 9, 2019 at 2:06:21 PM UTC-7, Jeroen Demeyer wrote:

On Mon, Sep 9, 2019 at 7:51 PM John H Palmieri <jhpalm...@gmail.com> wrote:
> I am writing to ask for help fixing a Python 3 problem. On some platforms, there are Python 3 doctest failures in
>
> - libs/eclib/interface.py (#28454)

I believe that this has been fixed upstream, it has certainly been
reported by me. Are we using the latest eclib version?

> How do we fix this? Note that patching FLINT is not ideal, since it won't help users who are using the system's version of the library. FLINT's most recent release was in 2015, so I don't know how much we can expect from upstream regarding new releases. So maybe the ideal solution would be to get Python to flush the output from these external libraries. I don't know how to do this; can anyone else help?

Ideally, the upstream library should properly flush when producing
output. They should use fflush (in C) or std::endl or std::flush (in
C++). I guess that Sage could also do the flushing whenever we return
from a library API call.

This problem arises with polynomial_rational_flint.pyx (on Ubuntu and Debian but not OS X, for whatever reason). See https://trac.sagemath.org/ticket/28334. How can we do this flushing? The only other solution I can see is to patch FLINT, but this won't work if users are using a system-wide installation of FLINT.

Other flushing problems appear at #28454, now since fixed with an upgrade to eclib, and #28622: pure Python code, so adding some `sys.stdout.flush()` commands cleared up that problem. But the FLINT problem remains.

--

John

[Dima Pasechnik]

an internet search finds

https://stackoverflow.com/questions/4178614/suppressing-output-of-module-calling-outside-library

is it what would work for us?

--
You received this message because you are subscribed to the Google Groups "sage-devel" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/bc6e7a4a-c8a3-4511-94ec-718e0126b19e%40googlegroups.com.

[John H Palmieri]

I found that and https://stackoverflow.com/questions/5081657/how-do-i-prevent-a-c-shared-library-to-print-on-stdout-in-python and maybe other related items, but I couldn't figure out how to use them to fix this particular problem. Others will know what they're doing more than I do regarding a Python/Cython interface to an external library, so others might have more luck.

To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.

[John Cremona]

On Wed, 23 Oct 2019 at 21:12, John H Palmieri <jhpalm...@gmail.com> wrote:

I found that and https://stackoverflow.com/questions/5081657/how-do-i-prevent-a-c-shared-library-to-print-on-stdout-in-python and maybe other related items, but I couldn't figure out how to use them to fix this particular problem. Others will know what they're doing more than I do regarding a Python/Cython interface to an external library, so others might have more luck.

That all looks very complicated.  Before I fixed the eclib source code the previous method did work, namely to flush stdout and stderr from the c/python wrapper code after calling any library function (in some cases this is certainly redundant, but in the case of eclib it was very hard for anyone other than me to know which library functions might spit out some error or warning message).  Somewhere in src/sage/libs/flint are the wrapper functions which need this.

Otherwise we might hope that a FLINT developer is reading this (Hello Bill!) and do what I did with eclib.

 

To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/73733782-6706-4839-a1ec-1477287dd112%40googlegroups.com.

[Nils Bruin]

On Wednesday, October 23, 2019 at 1:26:46 PM UTC-7, John Cremona wrote:

That all looks very complicated.  Before I fixed the eclib source code the previous method did work, namely to flush stdout and stderr from the c/python wrapper code after calling any library function (in some cases this is certainly redundant, but in the case of eclib it was very hard for anyone other than me to know which library functions might spit out some error or warning message).  Somewhere in src/sage/libs/flint are the wrapper functions which need this.

 Given that fflush is a system call, it probably has quite a serious overhead. When the routine is meant to do I/O anyway, it's a fair price to pay, but just flushing on the odd chance that a warning was generated could very well be quite costly. For a low-level library like flint, that might be too high a price. I think you want to test for performance regression before working around lacking output flushes by flushing always.

[John H Palmieri]

Is there a way to just flush output immediately after the relevant doctest? For the record, the doctest is

FLINT memory errors do not crash Sage (:trac:`17629`)::

    sage: t^(sys.maxsize//2)
    Traceback (most recent call last):
    ...
    RuntimeError: FLINT exception

and the "..." is supposed to capture "Exception (FLINT memory_manager). Unable to allocate memory." This text is what needs to be flushed, because otherwise it may not appear here, instead appearing in the output of some later doctest. So if we can add something at the end of the doctest, an analogue of

    sage: sys.stdout.flush()
    ...

that could work and would not impact performance whenever FLINT is used.

-- 
John

[Nils Bruin]

On Wednesday, October 23, 2019 at 2:47:54 PM UTC-7, John H Palmieri wrote:

On Wednesday, October 23, 2019 at 1:41:35 PM UTC-7, Nils Bruin wrote:

On Wednesday, October 23, 2019 at 1:26:46 PM UTC-7, John Cremona wrote:

That all looks very complicated.  Before I fixed the eclib source code the previous method did work, namely to flush stdout and stderr from the c/python wrapper code after calling any library function (in some cases this is certainly redundant, but in the case of eclib it was very hard for anyone other than me to know which library functions might spit out some error or warning message).  Somewhere in src/sage/libs/flint are the wrapper functions which need this.

 Given that fflush is a system call, it probably has quite a serious overhead. When the routine is meant to do I/O anyway, it's a fair price to pay, but just flushing on the odd chance that a warning was generated could very well be quite costly. For a low-level library like flint, that might be too high a price. I think you want to test for performance regression before working around lacking output flushes by flushing always.

Is there a way to just flush output immediately after the relevant doctest? For the record, the doctest is

FLINT memory errors do not crash Sage (:trac:`17629`)::

    sage: t^(sys.maxsize//2)
    Traceback (most recent call last):
    ...
    RuntimeError: FLINT exception

Wouldn't something like

try:

    t^(sys.maxsize//2)

except:

    sys.stdout.flush()

    raise

do the trick, to ensure that the message gets flushed before the error gets raised?

shouldn't a message like that occur on sys.stderr, though?

[John H Palmieri]

On Wednesday, October 23, 2019 at 9:49:56 PM UTC-7, Nils Bruin wrote:

On Wednesday, October 23, 2019 at 2:47:54 PM UTC-7, John H Palmieri wrote:

On Wednesday, October 23, 2019 at 1:41:35 PM UTC-7, Nils Bruin wrote:

On Wednesday, October 23, 2019 at 1:26:46 PM UTC-7, John Cremona wrote:

That all looks very complicated.  Before I fixed the eclib source code the previous method did work, namely to flush stdout and stderr from the c/python wrapper code after calling any library function (in some cases this is certainly redundant, but in the case of eclib it was very hard for anyone other than me to know which library functions might spit out some error or warning message).  Somewhere in src/sage/libs/flint are the wrapper functions which need this.

 Given that fflush is a system call, it probably has quite a serious overhead. When the routine is meant to do I/O anyway, it's a fair price to pay, but just flushing on the odd chance that a warning was generated could very well be quite costly. For a low-level library like flint, that might be too high a price. I think you want to test for performance regression before working around lacking output flushes by flushing always.

Is there a way to just flush output immediately after the relevant doctest? For the record, the doctest is

FLINT memory errors do not crash Sage (:trac:`17629`)::

    sage: t^(sys.maxsize//2)
    Traceback (most recent call last):
    ...
    RuntimeError: FLINT exception

Wouldn't something like

try:

    t^(sys.maxsize//2)

except:

    sys.stdout.flush()

    raise

do the trick, to ensure that the message gets flushed before the error gets raised?

shouldn't a message like that occur on sys.stderr, though?

The relevant print command in FLINT is

    flint_printf("Exception (FLINT memory_manager). Unable to allocate memory.\n");

and the problem is fixed by adding

    fflush(stdout);

So maybe it should be on stderr, but it's not. Regarding sys.stdout.flush(), my understanding, as confirmed by my experience with this particular problem, is that this only flushes output coming from Python, not from external library calls.

[Nils Bruin]

On Thursday, October 24, 2019 at 8:39:00 AM UTC-7, John H Palmieri wrote:

So maybe it should be on stderr, but it's not. Regarding sys.stdout.flush(), my understanding, as confirmed by my experience with this particular problem, is that this only flushes output coming from Python, not from external library calls.

Ah right, because of python3 ditching clib output for some reason. you could try sys.stdout.buffer.flush() but I suspect that's not going to work either then (I am not familiar with how this stuff is implemented).

So you really want to interface with clib's fflush.

with cython it's pretty clear how to do the interfacing. I guess via ctypes it would be possible too. I don't know if clib is already wrapped somewhere.

sage: cython("""from stdio cimport fflush
....: def cyflush():
....:     fflush(NULL)
....: """)

and then call cyflush() rather than stdout.flush()

(documentation  shows that fflush with NULL should flush all output)

This is getting a little heavy-handed for a doctest, but perhaps you like it better than patching FLINT

[Nils Bruin]

On Thursday, October 24, 2019 at 10:29:48 AM UTC-7, Nils Bruin wrote:

I guess via ctypes it would be possible too.

Browsing the documentation, something like:

libc=ctypes.cdll.LoadLibrary("libc.so.6")

libc.fflush(0r)

should work. And this should cause way less overhead than calling cython (because we don't have to call a c compiler). Problem here of course is how to make the "libc.so.6" cross-platform. So perhaps the cython solution is better ...

[Dima Pasechnik]

this is something that can be easily done at configure time, figuring
out the correct value for this string and putting it into
an environment variable.

> So perhaps the cython solution is better ...
>

> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/3d6e24b1-4ace-4b12-98ea-aa2b4ac70c99%40googlegroups.com.

[John H Palmieri]

On Thursday, October 24, 2019 at 10:57:09 AM UTC-7, Dima Pasechnik wrote:

On Thu, Oct 24, 2019 at 6:48 PM Nils Bruin <nbr...@sfu.ca> wrote:
>
> On Thursday, October 24, 2019 at 10:29:48 AM UTC-7, Nils Bruin wrote:
>>
>>
>> I guess via ctypes it would be possible too.
>
>
> Browsing the documentation, something like:
>
> libc=ctypes.cdll.LoadLibrary("libc.so.6")
> libc.fflush(0r)
>
> should work. And this should cause way less overhead than calling cython (because we don't have to call a c compiler). Problem here of course is how to make the "libc.so.6" cross-platform.

this is something that can be easily done at configure time, figuring
out the correct value for this string and putting it into
an environment variable.

For one doctest, I don't think we want to bother with configure time and an environment variable. Both seem to work, but the cython solution looks better to me.

 

> So perhaps the cython solution is better ...
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.

[Dima Pasechnik]

On Thu, Oct 24, 2019 at 7:15 PM John H Palmieri <jhpalm...@gmail.com> wrote:
>
>
>
> On Thursday, October 24, 2019 at 10:57:09 AM UTC-7, Dima Pasechnik wrote:
>>
>> On Thu, Oct 24, 2019 at 6:48 PM Nils Bruin <nbr...@sfu.ca> wrote:
>> >
>> > On Thursday, October 24, 2019 at 10:29:48 AM UTC-7, Nils Bruin wrote:
>> >>
>> >>
>> >> I guess via ctypes it would be possible too.
>> >
>> >
>> > Browsing the documentation, something like:
>> >
>> > libc=ctypes.cdll.LoadLibrary("libc.so.6")
>> > libc.fflush(0r)
>> >
>> > should work. And this should cause way less overhead than calling cython (because we don't have to call a c compiler). Problem here of course is how to make the "libc.so.6" cross-platform.
>>
>> this is something that can be easily done at configure time, figuring
>> out the correct value for this string and putting it into
>> an environment variable.
>
>
> For one doctest, I don't think we want to bother with configure time and an environment variable. Both seem to work, but the cython solution looks better to me.

I am thinking that having a cross-platform

sage_ctypes=ctypes.cdll.LoadLibrary("...")

mihgt be useful to have beyond 1 test...

>
>>
>>
>>
>> > So perhaps the cython solution is better ...
>> >
>> > --
>> > You received this message because you are subscribed to the Google Groups "sage-devel" group.
>> > To unsubscribe from this group and stop receiving emails from it, send an email to sage-...@googlegroups.com.
>> > To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/3d6e24b1-4ace-4b12-98ea-aa2b4ac70c99%40googlegroups.com.
>
> --
> You received this message because you are subscribed to the Google Groups "sage-devel" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/sage-devel/eddcc521-5dbc-4336-a63d-4419d3614ca2%40googlegroups.com.

[Nils Bruin]

On Thursday, October 24, 2019 at 11:23:02 AM UTC-7, Dima Pasechnik wrote:

I am thinking that having a cross-platform

sage_ctypes=ctypes.cdll.LoadLibrary("...")

mihgt be useful to have beyond 1 test...

I think so too, but if you google a bit you see there are various wrappers (including numpy) to the LoadLibrary machinery, but they don't seem to address the path problem. That makes me a bit concerned it's not something that is so easily solved in a sufficiently robust way (otherwise it would have been done already). It's also a "YAGNI" feature at this point for sage ... Which c libraries are so important that you want to interface with them and yet have such a simple interface that you can call them through ctypes, without the need of wrapping the API and/or packing/unpacking of data structures? ctypes is always rather hacky.