[E] POSIX and errno

[Nala Ginrut]

On Sat, Mar 19, 2022 at 6:52 AM John Cowan <co...@ccil.org> wrote:

It's important to realize that SRFI 18 (as opposed to its sample implementation) can be implemented using OS threads, green threads, or both (M:N threads), so you have to be very careful about what is supposed to be per-thread.  errno in particular is only per-thread if SRFI 18 uses OS threads.

Speaking to errno, it's necessary to open a new thread of E to discuss.

For a modern design, I think we shouldn't let users handle errno by themselves.

I can list some designs here:

1. Throw errno as the second returned value.

2. Hide errno into the exception

It's not exhausted, further discussion is required.

Best regards.

[Marc Nieper-Wißkirchen]

Am Sa., 19. März 2022 um 02:37 Uhr schrieb Nala Ginrut <nalag...@gmail.com>:
>
>
> On Sat, Mar 19, 2022 at 6:52 AM John Cowan <co...@ccil.org> wrote:
>>
>>
>> It's important to realize that SRFI 18 (as opposed to its sample implementation) can be implemented using OS threads, green threads, or both (M:N threads), so you have to be very careful about what is supposed to be per-thread. errno in particular is only per-thread if SRFI 18 uses OS threads.

Even without a multithreaded environment, errno (or any other global
variable of the C library) cannot be accessed directly from the Scheme
side because any Scheme program can be interrupted at any time by the
GC, for example. So something like this

(begin
(foreign-procedure)
(unless (zero? (errno))
..))

cannot work unless errno above is a Scheme parameter object and the
native function called by (foreign-procedure) stores the C value
(before returning) in that parameter.

That said, I don't think that using a parameter object to mimic C is a
good design. C neither has a simple method to unpack multiple return
values nor does it have exceptions, limiting what is possible in C
APIs.

[Nala Ginrut]

First, we can't make all POSIX syscall available in RnRS since it's huge work and unnecessary.

To my experience, we need to consider two situations:

1. Some important POSIX equivalent API will be in RnRS, for these APIs, the standard spec should handle errno with exceptions properly.

2. We must provide errno properly in FFI for any other C functions.

For this situation, FFI must provide errno to users if they need it.

In Guile, I had a patch (which was modified and applied by Mark Weaver) to provide errno with an optional keyword:

---------------------------cut----------------------------------

pointer->procedure return_type func_ptr arg_types [#:return-errno?=#f]

--------------------------end-------------------------------

The low-level implementation was to store errno after a C call immediately, which is necessary since the errno is volatile if another C call happened.

I think we need to do similar for FFI spec.

I guess these features are related to B, E, F.

Comments?

Best regards.

--
You received this message because you are subscribed to the Google Groups "scheme-reports-wg2" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scheme-reports-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scheme-reports-wg2/CAEYrNrT1-SDTmVe_w7PE4HGY%2Bx1gv%3DrzVuNaHR1-_J_gTjJGPQ%40mail.gmail.com.

[Nala Ginrut]

Sorry, I mean POSIX or syscall, not POSIX syscall.

[Marc Nieper-Wißkirchen]

Am Sa., 19. März 2022 um 10:08 Uhr schrieb Nala Ginrut <nalag...@gmail.com>:
>
> First, we can't make all POSIX syscall available in RnRS since it's huge work and unnecessary.
> To my experience, we need to consider two situations:
> 1. Some important POSIX equivalent API will be in RnRS, for these APIs, the standard spec should handle errno with exceptions properly.
>
> 2. We must provide errno properly in FFI for any other C functions.
> For this situation, FFI must provide errno to users if they need it.
> In Guile, I had a patch (which was modified and applied by Mark Weaver) to provide errno with an optional keyword:
> ---------------------------cut----------------------------------
> pointer->procedure return_type func_ptr arg_types [#:return-errno?=#f]
> --------------------------end-------------------------------
> The low-level implementation was to store errno after a C call immediately, which is necessary since the errno is volatile if another C call happened.
> I think we need to do similar for FFI spec.
>
> I guess these features are related to B, E, F.
> Comments?

I'm not sure whether an FFI API that allows calling (mostly) arbitrary
C procedures in, say, dynamic objects will be compatible with a wide
range of implementation strategies of Scheme.

For an FFI suitable for RnRS (which will necessarily be more limited
than native FFIs), we may have to restrict ourselves to calling C
procedures that follow some kind of restricted ABI so that the general
C library function will have to be wrapped (on the C side) anyway. If
this is the case, saving errno would be done by the individual wrapper
procedure. One advantage of it is that this would not handle errno
specifically but would also work for any other volatile global
variable on the C side.

The point is that we have no control over the C standard, which may
introduce something like errno2 in the future with analogous semantics
to errno. The Guile approach would break down then (for future
procedures wishing to use the errno2 value).

In any case, all these ideas/comments about what an FFI has to
provide/where its pitfalls are great. If at the end of these
considerations, an FFI still looks like a viable idea, F will provide
one so that the E libraries can be implemented at least portably
(although a native implementation may still be more efficient). If an
FFI is not an option, the E libraries will have to ship with native
implementations. A third option would be to standardize a minimal FFI,
which would be optional in language F. Scheme systems not supporting
it would then have to provide native implementations for the E
libraries.

Marc

[John Cowan]

On Fri, Mar 18, 2022 at 9:37 PM Nala Ginrut <nalag...@gmail.com> wrote:

Speaking to errno, it's necessary to open a new thread of E to discuss.

For a modern design, I think we shouldn't let users handle errno by themselves.

I can list some designs here:

1. Throw errno as the second returned value.

2. Hide errno into the exception

SRFI 170 chose a variant of the second option:  if the Posix function detects an error condition, the *name* corresponding to errno is packaged into the posix-error object.  This is done because the actual *value* of errno is system-dependent.  See section 3.1 of SRFI 170.

[John Cowan]

On Sat, Mar 19, 2022 at 4:32 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

errno (or any other global
variable of the C library) cannot be accessed directly from the Scheme
side because any Scheme program can be interrupted at any time by the
GC, for example.

I don't understand the causal relationship here, unless the GC invokes an errno-modifying Posix function (or macro).

(begin
  (foreign-procedure)
  (unless (zero? (errno))
    ..))

This is not in fact the correct protocol for the use of errno.  Each Posix function has its own protocol for indicating success or failure (for example, open() fails by returning -1), and only if it fails is errno set.  On success, errno is unspecified, and in any case is never set to 0 by any Posix function.  In the case where there is no function-specific protocol for failure, e.g. the getpriority() function, the caller must set errno to 0 and then check it after the call returns.

[Marc Nieper-Wißkirchen]

Am Sa., 19. März 2022 um 21:26 Uhr schrieb John Cowan <co...@ccil.org>:
>
>
>
> On Sat, Mar 19, 2022 at 4:32 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:
>
>> errno (or any other global
>> variable of the C library) cannot be accessed directly from the Scheme
>> side because any Scheme program can be interrupted at any time by the
>> GC, for example.
>
>
> I don't understand the causal relationship here, unless the GC invokes an errno-modifying Posix function (or macro).

The point is that the Scheme programmer has no control over what C
library functions are run between two evaluation steps of the Scheme
code. The GC is one obvious example of code that can be run in between
two Scheme evaluation steps. And the GC may invoke an errno-modifying
C function, e.g. mmap.

>>
>> (begin
>> (foreign-procedure)
>> (unless (zero? (errno))
>> ..))
>
>
> This is not in fact the correct protocol for the use of errno. Each Posix function has its own protocol for indicating success or failure (for example, open() fails by returning -1), and only if it fails is errno set. On success, errno is unspecified, and in any case is never set to 0 by any Posix function. In the case where there is no function-specific protocol for failure, e.g. the getpriority() function, the caller must set errno to 0 and then check it after the call returns.

You are right, of course! I should have been a bit more careful with
writing down my example code. The idea of the example is not affected
but promoting a wrong idea of a protocol is never good.

[Amirouche Boubekki]

On Sat, Mar 19, 2022 at 10:08 AM Nala Ginrut <nalag...@gmail.com> wrote:
>
> First, we can't make all POSIX syscall available in RnRS since it's huge work and unnecessary.
>

> Based on my experience, we need to consider two situations:

>
> 1. Some important POSIX equivalent API will be in RnRS, for these APIs, the standard spec should handle errno with exceptions properly.
>
> 2. We must provide errno properly in FFI for any other C functions.
>
> For this situation, FFI must provide errno to users if they need it.
> In Guile, I had a patch (which was modified and applied by Mark Weaver) to provide errno with an optional keyword:
>
> ---------------------------cut----------------------------------
> pointer->procedure return_type func_ptr arg_types [#:return-errno?=#f]
> --------------------------end-------------------------------
>
> The low-level implementation was to store errno after a C call immediately, which is necessary since the errno is volatile if another C call happened.

> I think we need to do something similar for the FFI spec.

Something similar was suggested in Cisco's Chez:

https://github.com/cisco/ChezScheme/issues/550#issuecomment-812032331

[Nala Ginrut]

On Sat, Mar 19, 2022 at 5:33 PM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

For an FFI suitable for RnRS (which will necessarily be more limited
than native FFIs), we may have to restrict ourselves to calling C
procedures that follow some kind of restricted ABI so that the general
C library function will have to be wrapped (on the C side) anyway. If
this is the case, saving errno would be done by the individual wrapper
procedure. One advantage of it is that this would not handle errno
specifically but would also work for any other volatile global
variable on the C side.

 

The point is that we have no control over the C standard, which may
introduce something like errno2 in the future with analogous semantics
to errno. The Guile approach would break down then (for future
procedures wishing to use the errno2 value).

Frankly, I don't think we have to be afraid of the changes in the future C. 

IMHO, C is old and the committee seems to prefer to be conservative to provide the compatible things, rather than radically improvement just like what C++ does. 

So we don't have to consider errno2 unless it becomes popular (I confess I don't see any).

 

In any case, all these ideas/comments about what an FFI has to
provide/where its pitfalls are great. If at the end of these
considerations, an FFI still looks like a viable idea, F will provide
one so that the E libraries can be implemented at least portably
(although a native implementation may still be more efficient). If an
FFI is not an option, the E libraries will have to ship with native
implementations. A third option would be to standardize a minimal FFI,
which would be optional in language F. Scheme systems not supporting
it would then have to provide native implementations for the E
libraries.

Can we elaborate on the specific work under E?

I mean the FFI case. It sounds like E will take most of the implementation guide work.

Best regards.

[Marc Nieper-Wißkirchen]

Am Mi., 23. März 2022 um 17:19 Uhr schrieb Nala Ginrut <nalag...@gmail.com>:

>> The point is that we have no control over the C standard, which may
>> introduce something like errno2 in the future with analogous semantics
>> to errno. The Guile approach would break down then (for future
>> procedures wishing to use the errno2 value).
>>
>
> Frankly, I don't think we have to be afraid of the changes in the future C.
> IMHO, C is old and the committee seems to prefer to be conservative to provide the compatible things, rather than radically improvement just like what C++ does.
> So we don't have to consider errno2 unless it becomes popular (I confess I don't see any).

You miss the point here, I think. (And it is my fault for not having
made it clear enough.)

If it's not the C library, it may be any other library (system or not)
that uses another thread-local besides errno and that a Scheme library
using FFI would like to access and that is volatile (because arbitrary
C code can run in the thread between the evaluation of two Scheme
primitives).

That said, even the current C standard (and C does move on, albeit
much slower than C++) already contains more thread-local state than
just errno, for example, the floating-point environment.

While the Guile approach may work (now), it is not totally
well-thought-out in my opinion. At least, it only covers only some
part of the general problem.

>> In any case, all these ideas/comments about what an FFI has to
>> provide/where its pitfalls are great. If at the end of these
>> considerations, an FFI still looks like a viable idea, F will provide
>> one so that the E libraries can be implemented at least portably
>> (although a native implementation may still be more efficient). If an
>> FFI is not an option, the E libraries will have to ship with native
>> implementations. A third option would be to standardize a minimal FFI,
>> which would be optional in language F. Scheme systems not supporting
>> it would then have to provide native implementations for the E
>> libraries.
>
>
>
> Can we elaborate on the specific work under E?
> I mean the FFI case. It sounds like E will take most of the implementation guide work.

If a standardized FFI is feasible, it will be part of the Foundations.
So it makes sense for E to say to F what has to be made possible by
such an FFI. For example, one requirement is that errno has to be
accessible somehow. The exact conventions of how the FFI will finally
be used are not important at this stage, I'd say.

Should an FFI become too hard to standardize in a reasonable way, E
won't be able to ship a portable sample implementation. But this is
not worse than the state we currently have.

Marc

[Nala Ginrut]

On Thu, Mar 24, 2022 at 12:36 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

That said, even the current C standard (and C does move on, albeit
much slower than C++) already contains more thread-local state than
just errno, for example, the floating-point environment.

While the Guile approach may work (now), it is not totally
well-thought-out in my opinion. At least, it only covers only some
part of the general problem.

I agree with your points.

However, my point is parallel with your consideration.

I think there're 2 reasons that we should think about the simple FFI situation:

1. We can't wrap every POSIX API since it's huge work.

The reasonable way is to let users do the simple FFI for them.

Of course, RnRS may want to define a core group of POSIX APIs in the standard, this is out of the topic here.

2. There're many POSIX APIs that are simple without a C wrapper, errno is the only thing the runtime needs to take care of.

The situation is similar outside POSIX, any libraries can share this convenience.

If we provide simple FFI with errno, the RnRS can help users to save some work, rather than wrap too many things in C.

This is meaningful work. 

 

> Can we elaborate on the specific work under E?
> I mean the FFI case. It sounds like E will take most of the implementation guide work.

If a standardized FFI is feasible, it will be part of the Foundations.
So it makes sense for E to say to F what has to be made possible by
such an FFI. For example, one requirement is that errno has to be
accessible somehow. The exact conventions of how the FFI will finally
be used are not important at this stage, I'd say.

Should an FFI become too hard to standardize in a reasonable way, E
won't be able to ship a portable sample implementation. But this is
not worse than the state we currently have.

I think  I'm able to share your concern about the portability around non-POSIX, or even among POSIX for the implementation-defined errno value.

So I think F may not be the right place to consider the errno, but E does.

However, I think it's still confusing for me about the cooperation between F and E.

Are we going to establish a base-spec in F, and let E provide various interfaces according to the environment?

It sounds like E will not provide a unified API for the different implementations, which is different from the general standard cases.

 Best regards.

[Marc Nieper-Wißkirchen]

Am Do., 24. März 2022 um 04:33 Uhr schrieb Nala Ginrut <nalag...@gmail.com>:

> 1. We can't wrap every POSIX API since it's huge work.
> The reasonable way is to let users do the simple FFI for them.
> Of course, RnRS may want to define a core group of POSIX APIs in the standard, this is out of the topic here.

I think it is part of Committee's E work to define what POSIX
functionality and in which form will be offered to the user of the
large language.

> 2. There're many POSIX APIs that are simple without a C wrapper, errno is the only thing the runtime needs to take care of.
> The situation is similar outside POSIX, any libraries can share this convenience.
> If we provide simple FFI with errno, the RnRS can help users to save some work, rather than wrap too many things in C.
> This is meaningful work.

I understand; it would be a reasonable convenience procedure on top of
a more fundamental FFI. I don't see why it shouldn't make sense to add
it on top of a minimal FFI. Whether such a convenience procedure would
belong to F, B, or E could be determined later. For the majority of
the API defined by E, it would be just an implementation detail.

> However, I think it's still confusing for me about the cooperation between F and E.
> Are we going to establish a base-spec in F, and let E provide various interfaces according to the environment?
> It sounds like E will not provide a unified API for the different implementations, which is different from the general standard cases.

I think we have to distinguish between APIs and their implementations
here. On the level of APIs (for which SRFI 170 is one example), it is
up to Committee E to select which functionality to offer and in which
form. It will also be Committee E's work to suitably abstract over
POSIX/Windows/Android/... to increase portability. The only overlap
between Committee E and F in this regard would be in the area where
R6RS and R7RS already specify procedures accessing the environment. As
the set of procedures of this kind is rather small in R6RS and R7RS, I
don't see a big problem here. What may have to be agreed upon is the
general representation of filenames and strings coming from the
environment. In R6RS and R7RS they are usually modeled as Scheme
strings (that said, R6RS allows filenames of a more general type),
which is not a fully correct abstraction as OS filenames and strings
do not have to be valid Unicode strings and vice versa. A bytevector
interface or the specification of an abstract pathname type (see
C++17's std::filesystem::path for one attempt) that extends a suitable
subset of Scheme strings sounds like something that should be pondered
about by Committee E and F together.

As far as the implementation level is concerned, the current situation
(e.g. as in SRFI 170) is such that the concrete implementation is
outside of the standard. In the beginning, Committee E wouldn't need
to change this approach, I think. After a while, they will have a list
of what they would need from a standardized FFI to be able to
implement their APIs so far portably. Then, Committee F will have to
see whether it is reasonable to add an FFI with such capabilities into
the core. Using a standardized FFI will likely not squeeze out maximal
performance (because of a lot of compromises that will have to be made
to accommodate many different types of Scheme implementation
techniques), but being able to ship portable implementations of the
standard libraries would be a great boon.

Just my thoughts from my understanding, of course.

Marc

[John Cowan]

On Thu, Mar 24, 2022 at 3:46 AM Marc Nieper-Wißkirchen <marc....@gmail.com> wrote:

 

I think it is part of Committee's E work to define what POSIX
functionality and in which form will be offered to the user of the
large language.

+1.  Providing all of Posix (1911 functions and macros, 51 data types) is unreasonable.

 

As far as the implementation level is concerned, the current situation
(e.g. as in SRFI 170) is such that the concrete implementation is
outside of the standard. In the beginning, Committee E wouldn't need
to change this approach, I think. After a while, they will have a list
of what they would need from a standardized FFI to be able to
implement their APIs so far portably.

I do not know what "portably" means in this context.  A definition of a particular procedure in terms of an FFI would not be portable across operating systems, nor would it necessarily be portable across R7RS-Large implementations.

[Marc Nieper-Wißkirchen]

Portable in the sense of being independent of a particular Scheme
implementation, not necessarily independent of the environment, of
course.