How to flush <stdio.h>'s stdout stream?

[Grant Edwards]

In Python 3.7+, how does one flush the stdout FILE stream? I mean the
FILE *declared as 'stdio' in <stdio.h>. I'm _not_ asking how to flush the
Python file object sys.stdio.

The problem is that some Python modules write to FILE *stdio, but
don't flush the stream so that the data gets written to the standard
output file descriptor (fd 1). If you need to interleave the output
from those modules with writes from other Python code to fd 1 (via
sys.stdout or whatever), you need a way to flush FILE *stdio.

I know how to use ctypels to call fflush(), but I can't figure out how
to get the FILE *stdio value to pass to fflush().

Is there some os.??? or sys.??? function that I've missed?

[Eryk Sun]

On 1/15/21, Grant Edwards <grant.b...@gmail.com> wrote:
> In Python 3.7+, how does one flush the stdout FILE stream? I mean the
> FILE *declared as 'stdio' in <stdio.h>. I'm _not_ asking how to flush the
> Python file object sys.stdio.

You can flush all output streams via C fflush(NULL). If it has to be
just stdout, the code will vary by platform. As far as I know, there
is no standard way at the ABI level to reference C stdin, stdout, and
stderr. Typically these are macros at the API level. The macro might
directly reference the FILE record, or maybe it calls an inline
function that returns the reference. CPython doesn't have a common
wrapper function for this. With ctypes, you'll have to special case
common platform CRTs such as glibc in Linux and ucrt in Windows
(possible, but undocumented). For example:

import sys
import time
import ctypes

if sys.platform == 'win32' and sys.version_info >= (3, 5):
libc = ucrt = ctypes.CDLL('ucrtbase', use_errno=True)
IOLBF = 0x0040
libc.__acrt_iob_func.restype = ctypes.c_void_p
stdout = ctypes.c_void_p(libc.__acrt_iob_func(1))
elif sys.platform == 'linux':
libc = cglobal = ctypes.CDLL(None, use_errno=True)
IOLBF = 1
stdout = ctypes.c_void_p.in_dll(libc, 'stdout')

# ensure that stdout is buffered for this test
# has to be called before stdout is used
libc.setvbuf(stdout, None, IOLBF, 4096)

def test(flush=False):
print(f'test with flush={flush}')
libc.fwrite(b'spam', 1, 4, stdout)
if flush:
libc.fflush(stdout)
time.sleep(5)
libc.fflush(None)
print()

test(flush=False)
test(flush=True)

With flush=False, the test should wait 5 seconds before 'spam' is
flushed to the console/terminal. With flush=True, there should be no
delay.

[Grant Edwards]

On 2021-01-15, Eryk Sun <ery...@gmail.com> wrote:
> On 1/15/21, Grant Edwards <grant.b...@gmail.com> wrote:
>> In Python 3.7+, how does one flush the stdout FILE stream? I mean the
>> FILE *declared as 'stdio' in <stdio.h>. I'm _not_ asking how to flush the
>> Python file object sys.stdio.
>
> You can flush all output streams via C fflush(NULL).

Brilliant! I read through the fflush() man page earlier but missed that.

fflush(NULL) should work.

> If it has to be just stdout, the code will vary by platform. As far
> as I know, there is no standard way at the ABI level to reference C
> stdin, stdout, and stderr. Typically these are macros at the API
> level. The macro might directly reference the FILE record, or maybe
> it calls an inline function that returns the reference.

That's what I was trying to figure out: whether 'stdio' was actually a
global symbol. On Linux it appears to be. I was afraid it was a macro
that referred indirectly to some per-thread storage or something...

> CPython doesn't have a common wrapper function for this. With
> ctypes, you'll have to special case common platform CRTs such as
> glibc in Linux and ucrt in Windows (possible, but undocumented). For
> example:
>
> import sys
> import time
> import ctypes
>
> if sys.platform == 'win32' and sys.version_info >= (3, 5):
> libc = ucrt = ctypes.CDLL('ucrtbase', use_errno=True)
> IOLBF = 0x0040
> libc.__acrt_iob_func.restype = ctypes.c_void_p
> stdout = ctypes.c_void_p(libc.__acrt_iob_func(1))
> elif sys.platform == 'linux':
> libc = cglobal = ctypes.CDLL(None, use_errno=True)
> IOLBF = 1
> stdout = ctypes.c_void_p.in_dll(libc, 'stdout')

That part I hadn't figured out. I've always used ctypes to call
functions in .so libraries that were explicitly loaded by the Python
app.