O_TEXT for PDOS/386

[Paul Edwards]

Hi.

Cygwin has an O_TEXT on the open() call to let
"the OS layer" (can be quibbled) that the file
is being opened in text mode. And as such, it
gives that layer an opportunity to insert CRs.

PDOS/386 (available at http://pdos.org) is able
to run a hello world Linux ELF binary, and recent
developments mean that I should be able to run
more than just a hello world soon.

But even with those recent developments, I am
still missing the same thing that Cygwin was
missing, and added O_TEXT to solve.

I'd like to add an O_TEXT flag to Linux. It would
be good if Linux officially reserved a flag for
this use, but in the absence of that, I'm happy
to create my own.

I think Linux adds flags going up, so I would add
any flags I want going down.

So I'm thinking of making O_TEXT 0x80000000

Another possibility would be 0x40000000 to avoid
being seen as a negative number.

This is different from Cygwin, which uses 0x20000.

I would be happy to use the Cygwin value, but I
think Linux may have already encroached that.

And it is not important to maintain the same value
as Cygwin, because Cygwin creates PE executables,
while mine is for ELF executables.

Any suggestions?

Thanks. Paul.

[Richard Kettlewell]

Paul Edwards <muta...@gmail.com> writes:
> Cygwin has an O_TEXT on the open() call to let "the OS layer" (can be
> quibbled) that the file is being opened in text mode. And as such, it
> gives that layer an opportunity to insert CRs.
>
> PDOS/386 (available at http://pdos.org) is able to run a hello world
> Linux ELF binary, and recent developments mean that I should be able
> to run more than just a hello world soon.
>
> But even with those recent developments, I am still missing the same
> thing that Cygwin was missing, and added O_TEXT to solve.

IMO Cygwin got this one wrong. The text/binary file distinction should
be managed in the C library or higher. Applications should either use
the appropriate arguments to fopen(), or manage any translation required
themselves.

> I'd like to add an O_TEXT flag to Linux. It would be good if Linux
> officially reserved a flag for this use, but in the absence of that,
> I'm happy to create my own.

You’d need to ask on the linux-kernel mailing list, I think. But I
wouldn’t predict a very positive response.

--
https://www.greenend.org.uk/rjk/

[Paul Edwards]

On 20/02/24 17:53, Richard Kettlewell wrote:

> Paul Edwards <muta...@gmail.com> writes:
>> Cygwin has an O_TEXT on the open() call to let "the OS layer" (can be
>> quibbled) that the file is being opened in text mode. And as such, it
>> gives that layer an opportunity to insert CRs.
>>
>> PDOS/386 (available at http://pdos.org) is able to run a hello world
>> Linux ELF binary, and recent developments mean that I should be able
>> to run more than just a hello world soon.
>>
>> But even with those recent developments, I am still missing the same
>> thing that Cygwin was missing, and added O_TEXT to solve.
>
> IMO Cygwin got this one wrong. The text/binary file distinction should
> be managed in the C library or higher. Applications should either use
> the appropriate arguments to fopen(), or manage any translation required
> themselves.

Hi Richard. Thanks for your reply.

The C library that I use for Linux is a Linux port
of PDPCLIB, and is designed for Linux, and thus, no
translation will be done.

It is only when I run that Linux ELF binary on
PDOS/386 (mini Win32 clone) that I want translation
done. And even that would be a PDOS/386 configuration
thing. Maybe I would like to run a NL-only system.

So it can't easily (the non-Linux environment would
need to be detected, and assumptions made) be done in
the (Linux) C library, and it wouldn't be appropriate
either.

>> I'd like to add an O_TEXT flag to Linux. It would be good if Linux
>> officially reserved a flag for this use, but in the absence of that,
>> I'm happy to create my own.
>
> You’d need to ask on the linux-kernel mailing list, I think. But I
> wouldn’t predict a very positive response.

That's what I'm predicting too. So can you suggest
a flag? I think Linux is consuming flags going up,
so I could go down for anything I need, and have
0x80000000 mean "O_TEXT".

But that makes it a negative number, which may not
be nice, so I could make it 0x40000000.

Another possibility would be to commandeer some
existing unimportant flag. I think there might be
one for "access time"? Since I don't care whether
or not access time is updated, if access time is
requested, I could make it mean "text".

Or some combination of flags.

Any suggestions within the constraints of the Linux
people having different goals to me?

Thanks. Paul.

[Lew Pitcher]

On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:

> Hi.
>
> Cygwin has an O_TEXT on the open() call to let
> "the OS layer" (can be quibbled) that the file
> is being opened in text mode. And as such, it
> gives that layer an opportunity to insert CRs.

[snip]
> I'd like to add an O_TEXT flag to Linux.[snip]
> Any suggestions?

Yes, one. Dont.

Microsoft Windows (on top of which Cygwin must run) makes a low-level
distinction between "binary" files and "text" files. I believe it is
because Windows guarantees (or must guarantee) certain data
transformations on text that do not apply to other forms of data.
(I.e. translating <cr><lf> sequences into C newline characters, etc)

HOWEVER, Linux (or, /any/ Unix) makes NO distinction between text
and binary data, neither in the standard I/O library, nor in the OS
itself.

You are effectively asking that Linux /NOW/ make such a distinction,
even if only to ignore the O_TEXT flag that you ask for.

It ain't gonna fly. Such a flag would invalidate /decades/ worth of
Unix/Linux code, and /not/ provide any functionality either to
the OS layer, the stdio library layer, or the application code layer.

So, dont even try to "add an O_TEXT flag" to Linux. Just dont.
--
Lew Pitcher
"In Skills We Trust"

[Paul Edwards]

On 20/02/24 22:14, Lew Pitcher wrote:
> On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:
>
>> Hi.
>>
>> Cygwin has an O_TEXT on the open() call to let
>> "the OS layer" (can be quibbled) that the file
>> is being opened in text mode. And as such, it
>> gives that layer an opportunity to insert CRs.
> [snip]
>> I'd like to add an O_TEXT flag to Linux.[snip]
>> Any suggestions?
>
> Yes, one. Dont.
>
> Microsoft Windows (on top of which Cygwin must run) makes a low-level
> distinction between "binary" files and "text" files. I believe it is
> because Windows guarantees (or must guarantee) certain data
> transformations on text that do not apply to other forms of data.
> (I.e. translating <cr><lf> sequences into C newline characters, etc)
>
> HOWEVER, Linux (or, /any/ Unix) makes NO distinction between text
> and binary data, neither in the standard I/O library, nor in the OS
> itself.
>
> You are effectively asking that Linux /NOW/ make such a distinction,
> even if only to ignore the O_TEXT flag that you ask for.

The C library that most people use already has
that distinction - and it is indeed ignored.
Works perfectly fine.

> It ain't gonna fly. Such a flag would invalidate /decades/ worth of
> Unix/Linux code, and /not/ provide any functionality either to
> the OS layer, the stdio library layer, or the application code layer.

Sorry - what is being invalidated? Can you give an
example?

BFN. Paul.

[Richard Kettlewell]

Lew Pitcher <lew.p...@digitalfreehold.ca> writes:
> On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:
>> Cygwin has an O_TEXT on the open() call to let
>> "the OS layer" (can be quibbled) that the file
>> is being opened in text mode. And as such, it
>> gives that layer an opportunity to insert CRs.
> [snip]
>> I'd like to add an O_TEXT flag to Linux.[snip]
>> Any suggestions?
>
> Yes, one. Dont.
>
> Microsoft Windows (on top of which Cygwin must run) makes a low-level
> distinction between "binary" files and "text" files. I believe it is
> because Windows guarantees (or must guarantee) certain data
> transformations on text that do not apply to other forms of data.
> (I.e. translating <cr><lf> sequences into C newline characters, etc)

AFAIK it’s not that low level. I can see no sign of it in CreateFile,
ReadFile, etc, which are approximately equivalent to open(), read() etc
in the Windows API. I think the implementation is in the C runtime
(effectively the Libc).

Cygwin does not appear rely on Windows for the implementation of its
O_TEXT and O_BINARY - you can see the translation code in
fhandler_base::read.

--
https://www.greenend.org.uk/rjk/

[Lew Pitcher]

Lets take an accounting...

1) POSIX open() does not make any distinction between "text" and "binary"
files

2) Standard C / POSIX fopen() /does/ make a distinction between "text"
and "binary" files. POSIX issues a caveat regarding the mode
parameter that "The character 'b' shall have no effect, but is allowed
for ISO C standard conformance.", referring to the mode character "b"
specified by the ISO C standard as indicating that the file should be
opened for "binary" I/O, rather than the default "text" I/O.

3) You specifically reference the open() call, not the fopen() call.

So, you ask that the open() call, a call that, under Linux and other
POSIX-compatible operating systems, makes no distinction between binary
and text files, now recognize a flag indicating that the open() should
act apon a text file.

That leaves the Linux kernel developers with a choice to make: either
a) accept your request and add an O_TEXT flag that conditions the I/O
for text data, or
b) accept your request and add an O_TEXT flag that does nothing, or
c) ignore or deny your request

Choice (a) would drastically change the behaviour of the open()
call, and any existing code that uses open() to access a text file would
have to change to specify the O_TEXT flag. I believe that this would
be a no-starter, even if the default for O_TEXT would leave behaviour
as it currently is.

Choice (b) is ... useless. It recognizes a non-issue, and does nothing.
It would be a header change that reserves a flag that would be ignored
by everyone. I believe that the maintainers would simply veto this
as being a change that makes no change.

So, choice (c) is my bet. The maintainers simply ignore (if they want
to be polite) or deny your proposal.

>
>> It ain't gonna fly. Such a flag would invalidate /decades/ worth of
>> Unix/Linux code, and /not/ provide any functionality either to
>> the OS layer, the stdio library layer, or the application code layer.
>
> Sorry - what is being invalidated? Can you give an
> example?

Today, I can
open("/etc/passwd",O_RDONLY)
and read text from the text file /etc/passwd. I can
also
open("/sbin/init",O_RDONLY)
and read binary data from the binary file /sbin/init

If I understand correctly, you propose that code now
must
open("/etc/passwd",O_RDONLY | O_TEXT)
to read text from the text file /etc/passwd.

What would happen if, under your proposal, code still
executed
open("/etc/passwd",O_RDONLY)
? Could the code still read() text data, or would the
open() fail on something like EACCESS or EINVAL or
EOPNOTSUPP? What /would/ the consequences be of setting
O_TEXT?

Of course, if you answer that, under a POSIX-like system,
there would be /no/ consequences to O_TEXT, then you
have presented choice (b), a change that changes nothing.

[Richard Kettlewell]

Paul Edwards <muta...@gmail.com> writes:
> On 20/02/24 17:53, Richard Kettlewell wrote:
>> Paul Edwards <muta...@gmail.com> writes:
>>> I'd like to add an O_TEXT flag to Linux. It would be good if Linux
>>> officially reserved a flag for this use, but in the absence of that,
>>> I'm happy to create my own.
>>
>> You’d need to ask on the linux-kernel mailing list, I think. But I
>> wouldn’t predict a very positive response.
>
> That's what I'm predicting too. So can you suggest
> a flag? I think Linux is consuming flags going up,
> so I could go down for anything I need, and have
> 0x80000000 mean "O_TEXT".

No, I can’t suggest a flag. I have no more control over the Linux ABI
than you do. That’s why I pointed you to linux-kernel.

> Any suggestions within the constraints of the Linux people having
> different goals to me?

I think you should rethink your goals and/or your approach to meeting
them. open/read/write is the wrong place to translate between line
ending conventions.

--
https://www.greenend.org.uk/rjk/

[Lew Pitcher]

On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:

> Hi.
>
> Cygwin has an O_TEXT on the open() call to let
> "the OS layer" (can be quibbled) that the file
> is being opened in text mode. And as such, it
> gives that layer an opportunity to insert CRs.

[snip]

> I am
> still missing the same thing that Cygwin was
> missing, and added O_TEXT to solve.

And, what would that be? What does the Cygwin
O_TEXT flag on open(2) do?

[snip]

> And it is not important to maintain the same value
> as Cygwin, because Cygwin creates PE executables,
> while mine is for ELF executables.

Are you just trying to keep your Linux source
source-compatible with your (Cygwin-based) Windows
source? Or are you actually looking for the functionality
behind the O_TEXT flag?

If you just want to keep source compatibility, I don't see
any problem with using a system-specific predefined macro
to implement some conditional compilation. Something like

#ifndef O_TEXT
#define O_TEXT 0
#endif

open("/etc/passwd",O_RDONLY | O_TEXT);

HTH

[Paul Edwards]

Almost everyone.

PDOS/386 would use it.

> I believe that the maintainers would simply veto this
> as being a change that makes no change.
>
> So, choice (c) is my bet. The maintainers simply ignore (if they want
> to be polite) or deny your proposal.

Sure. Hence my question.

I'll take a random flag myself. Any suggestions?

Correct. Nothing is invalidated.

It is only when run on a Windows-like system that
something happens.

BFN. Paul.

[Paul Edwards]

On 21/02/24 03:34, Lew Pitcher wrote:
> On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:
>
>> Hi.
>>
>> Cygwin has an O_TEXT on the open() call to let
>> "the OS layer" (can be quibbled) that the file
>> is being opened in text mode. And as such, it
>> gives that layer an opportunity to insert CRs.
> [snip]
>> I am
>> still missing the same thing that Cygwin was
>> missing, and added O_TEXT to solve.
>
> And, what would that be? What does the Cygwin
> O_TEXT flag on open(2) do?

It will add CRs to such files on a write().

> [snip]
>> And it is not important to maintain the same value
>> as Cygwin, because Cygwin creates PE executables,
>> while mine is for ELF executables.
>
> Are you just trying to keep your Linux source
> source-compatible with your (Cygwin-based) Windows
> source? Or are you actually looking for the functionality
> behind the O_TEXT flag?

The latter. Functionality on PDOS/386, not Linux.

And not Linux source. Linux binaries.

> If you just want to keep source compatibility, I don't see
> any problem with using a system-specific predefined macro
> to implement some conditional compilation. Something like
>
> #ifndef O_TEXT
> #define O_TEXT 0
> #endif
>
> open("/etc/passwd",O_RDONLY | O_TEXT);

Not what I'm after.

BFN. Paul.

[Lew Pitcher]

If PDOS/386 needs it, then PDOS/386 probably should use it.

However, the needs of PDOS/386 aren't the needs of Linux or POSIX, and
neither Linux in particular, nor POSIX in general, need bother with
it.

If you are concerned with PDOS/386 being able to use source code written
to the POSIX standard, then I suggest that you select the value and
treatment of your O_TEXT flag to work in absence, as POSIX code won't
have it or use it.

The alternative is to specify that such a flag /is/ required for PDOS/386,
which restricts you to using code written specifically for PDOS/386, and
not the more general-purpose and widely available POSIX standard.

[snip]


Best of luck.

[Paul Edwards]

On 21/02/24 10:51, Lew Pitcher wrote:
> On Wed, 21 Feb 2024 05:37:45 +0800, Paul Edwards wrote:

>>> Choice (b) is ... useless. It recognizes a non-issue, and does nothing.
>>> It would be a header change that reserves a flag that would be ignored
>>> by everyone.
>>
>> Almost everyone.
>>
>> PDOS/386 would use it.
>
> If PDOS/386 needs it, then PDOS/386 probably should use it.

Not so much "needs" as "would be desirable".

> However, the needs of PDOS/386 aren't the needs of Linux or POSIX, and
> neither Linux in particular, nor POSIX in general, need bother with
> it.

Sure. As I said in a previous message - "different goals".

> If you are concerned with PDOS/386 being able to use source code written
> to the POSIX standard,

Not just source - that is easy to deal with.
The executable.

> then I suggest that you select the value and
> treatment of your O_TEXT flag to work in absence, as POSIX code won't
> have it or use it.

That's exactly what I'm doing. I'm just asking
which bit would be good to use. I showed which
bit that Cygwin was using, and I made a suggestion
that I could work downwards on the assumption
that Linux is working upwards.

> The alternative is to specify that such a flag /is/ required for PDOS/386,

That's the alternative I am after.

> which restricts you to using code written specifically for PDOS/386, and

It is only code that uses the new flag that will
work nicely on PDOS/386, yes.

The other code would still run, it just won't
automatically put the CR before the NL, so it
won't look good on (this) Windows-like environment.

> not the more general-purpose

I consider adding O_TEXT to be more general-purpose
than one that assumes the code will only be used on
a Unix-like environment, never a Windows-like
environment.

> and widely available POSIX standard.

It still follows the rest of that, so it will still
work on any POSIX environment.

The *application* code will need to have:

#ifndef O_TEXT
/* uh oh, we're on a standard POSIX environment,
missing the PDOS/386-inspired Linux enhancement
that hasn't YET (as of 2024-02-21) made it to a
formal POSIX standard */
#define O_TEXT 0
#endif

... open(... | O_TEXT ...)

BFN. Paul.

[J.O. Aho]

On 20/02/2024 22.40, Paul Edwards wrote:
> On 21/02/24 03:34, Lew Pitcher wrote:
>> On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:

>> [snip]
>>> And it is not important to maintain the same value
>>> as Cygwin, because Cygwin creates PE executables,
>>> while mine is for ELF executables.
>>
>> Are you just trying to keep your Linux source
>> source-compatible with your (Cygwin-based) Windows
>> source? Or are you actually looking for the functionality
>> behind the O_TEXT flag?
>
> The latter. Functionality on PDOS/386, not Linux.

Think you need to look into the windows source code, as open() already
exists in windows and those cygwin don't need to reimplement that, it
only provides things that do not exists in windows but is part of POSIX.

> And not Linux source. Linux binaries.

Cygwin is not:

- a way to run native Linux apps on Windows. You must rebuild your
application from source if you want it to run on Windows.


If you want to make changes to POSIX API, you need to start contacting
Austin Group, with time changes will appear in Linux. I doubt you would
have more luck with the Austin Group than convincing Linus to add a
patch to add something that Linux don't need.

--
//Aho

[Richard Kettlewell]

Paul Edwards <muta...@gmail.com> writes:
> On 21/02/24 10:51, Lew Pitcher wrote:
>> If you are concerned with PDOS/386 being able to use source code written
>> to the POSIX standard,
>
> Not just source - that is easy to deal with.
> The executable.

Executables won’t have your extra flag unless you’ve personally modified
and recompiled them.

>> The alternative is to specify that such a flag /is/ required for
>> PDOS/386,
>
> That's the alternative I am after.
>
>> which restricts you to using code written specifically for PDOS/386,
>> and
>
> It is only code that uses the new flag that will work nicely on
> PDOS/386, yes.

How will you modify other syscalls (for example fstat and lseek) to be
play nicely with the newline translation?

--
https://www.greenend.org.uk/rjk/

[Paul Edwards]

On 21/02/24 17:48, Richard Kettlewell wrote:
> Paul Edwards <muta...@gmail.com> writes:
>> On 21/02/24 10:51, Lew Pitcher wrote:
>>> If you are concerned with PDOS/386 being able to use source code written
>>> to the POSIX standard,
>>
>> Not just source - that is easy to deal with.
>> The executable.
>
> Executables won’t have your extra flag unless you’ve personally modified
> and recompiled them.

Yes, that's exactly what I will do. I will start
producing executables that have that flag.

At least MY executables will live up to MY
requirements.

Which is all I'm really after.

If someone else finds that useful, for the same
reasons I find that useful, great.

If I am alone in the world, that's fine too.

The latter state could change at a later date
though. You never know what will happen a
millenia from now.

>>> which restricts you to using code written specifically for PDOS/386,
>>> and
>>
>> It is only code that uses the new flag that will work nicely on
>> PDOS/386, yes.
>
> How will you modify other syscalls (for example fstat and lseek) to be
> play nicely with the newline translation?

I only intend to run C90-compliant programs
(in this case, with a statically-linked C
runtime library).

fstat is not used by PDPCLIB, nor part of the
C90 standard.

lseek is used by PDPCLIB - that's how fseek()
is implemented.

According to C90, fseek() is not expected to
behave sensibly on text files.

And indeed - it won't (on PDOS/386 anyway; when
run on Linux it will be fine).

BFN. Paul.

[Paul Edwards]

On 21/02/24 15:06, J.O. Aho wrote:

>>>> And it is not important to maintain the same value
>>>> as Cygwin, because Cygwin creates PE executables,
>>>> while mine is for ELF executables.
>>>
>>> Are you just trying to keep your Linux source
>>> source-compatible with your (Cygwin-based) Windows
>>> source? Or are you actually looking for the functionality
>>> behind the O_TEXT flag?
>>
>> The latter. Functionality on PDOS/386, not Linux.
>
> Think you need to look into the windows source code, as open() already
> exists in windows

There is no such syscall. kernel32 exports CreateFile.

Regardless - Windows won't run the Linux ELF binary -
not even the one I produce myself - regardless. Although
if you install WSL it would, but that's installing
Linux - and of course Linux programs run under Linux.

It is PDOS/386 that will run it. And receive the
INT 80H. And check for the flag.

Because PDOS/386 is sort of a DOS/Windows clone,
all files are expected to use CRLF as line terminators.
That's the user requirements, basically.

So I'm trying to get my Linux ELF binaries (ones that
I build myself) to fit into that existing user
requirement.

> and those cygwin don't need to reimplement that, it
> only provides things that do not exists in windows but is part of POSIX.

It is Cygwin that invented O_TEXT, not Windows.
To satisfy that same user requirement above.

>> And not Linux source. Linux binaries.
>
> Cygwin is not:
>
> - a way to run native Linux apps on Windows. You must rebuild your
> application from source if you want it to run on Windows.

I know.

That's why I'm not using Cygwin, I'm using PDOS/386.

Which ALREADY runs a hello world Linux ELF binary.

(unlike Cygwin - which never did)

> If you want to make changes to POSIX API, you need to start contacting
> Austin Group, with time changes will appear in Linux.

https://en.wikipedia.org/wiki/Austin_Group

Ok, thanks.

> I doubt you would
> have more luck with the Austin Group than convincing Linus to add a
> patch to add something that Linux don't need.

Ok. I'll try the Austin Group first though. As you
noted, this isn't for Linux. It's for (quibbling
aside) a competitor of Linux, so the Austin Group
would be more appropriate.

BFN. Paul.

[Richard Kettlewell]

Paul Edwards <muta...@gmail.com> writes:
> On 21/02/24 17:48, Richard Kettlewell wrote:
>> How will you modify other syscalls (for example fstat and lseek) to be
>> play nicely with the newline translation?
>
> I only intend to run C90-compliant programs (in this case, with a
> statically-linked C runtime library).
>
> fstat is not used by PDPCLIB, nor part of the C90 standard.

Standard C doesn’t have open(), read() or write() either, but you seem
to want to modify the behaviour of those.

--
https://www.greenend.org.uk/rjk/

[Paul Edwards]

On 21/02/24 20:25, Richard Kettlewell wrote:
> Paul Edwards <muta...@gmail.com> writes:
>> On 21/02/24 17:48, Richard Kettlewell wrote:
>>> How will you modify other syscalls (for example fstat and lseek) to be
>>> play nicely with the newline translation?
>>
>> I only intend to run C90-compliant programs (in this case, with a
>> statically-linked C runtime library).
>>
>> fstat is not used by PDPCLIB, nor part of the C90 standard.
>
> Standard C doesn’t have open(), read() or write() either,

Indeed - because PDPCLIB's implementation of fopen()
needs to do the open syscall (INT 80H EAX = 5H).

fopen already knows whether the file is being
opened as text or binary, and I don't want that
information lost when I do that interrupt above.
Even though Linux (currently, and very likely
forever) won't use it, PDOS/386 has a use for it
(in the short term).

> but you seem to want to modify the behaviour of those.

I don't think that is an accurate statement.

What behavior, where?

Adding an extra flag doesn't change the behavior
of anything anywhere, immediately.

PDOS/386 would then be in a position to have a
particular behavior, but it's not really a
change, is it? It's new development/enhancement.

BFN. Paul.

[J.O. Aho]

On 21/02/2024 12.57, Paul Edwards wrote:
> On 21/02/24 15:06, J.O. Aho wrote:
>
>>>>> And it is not important to maintain the same value
>>>>> as Cygwin, because Cygwin creates PE executables,
>>>>> while mine is for ELF executables.
>>>>
>>>> Are you just trying to keep your Linux source
>>>> source-compatible with your (Cygwin-based) Windows
>>>> source? Or are you actually looking for the functionality
>>>> behind the O_TEXT flag?
>>>
>>> The latter. Functionality on PDOS/386, not Linux.
>>
>> Think you need to look into the windows source code, as open() already
>> exists in windows
>
> There is no such syscall. kernel32 exports CreateFile.

_sopen_s() (secure version of _open()) and _O_TEXT/_O_BINARY do exists
and I think this is what Cygwin wraps to open() and O_TEXT/O_BINARY.

> Because PDOS/386 is sort of a DOS/Windows clone,
> all files are expected to use CRLF as line terminators.
> That's the user requirements, basically.

Why don't you ask Microsoft to make ms-windows POSIX compatible, then
this issue would disappear at once ;)

> So I'm trying to get my Linux ELF binaries (ones that
> I build myself) to fit into that existing user
> requirement.

That's why other projects has solved it as define O_TEXT/O_BINARY to 0x0
if they are missing when they want to support multiple OS:es. They don't
expect that all OS has AmigaOS dos library implemented to open files
IDOS->Open("c:dir", MODE_OLDFILE)

--
//Aho

[Peter 'Shaggy' Haywood]

Groovy hepcat Lew Pitcher was jivin' in alt.os.linux on Wed, 21 Feb 2024
04:55 am. It's a cool scene! Dig it.

> On Tue, 20 Feb 2024 22:25:05 +0800, Paul Edwards wrote:
>> On 20/02/24 22:14, Lew Pitcher wrote:
>>> On Tue, 20 Feb 2024 14:51:15 +0800, Paul Edwards wrote:
>>>
>>>> Cygwin has an O_TEXT on the open() call to let
>>>> "the OS layer" (can be quibbled) that the file
>>>> is being opened in text mode. And as such, it
>>>> gives that layer an opportunity to insert CRs.
>>> [snip]
>>>> I'd like to add an O_TEXT flag to Linux.[snip]
>>>> Any suggestions?
>>>
>>> Yes, one. Dont.

[Snip.]

This would be useless, true. But it would be harmless. It could be
defined simply as

#define O_TEXT 0

which would mean a mode argument of, say, O_RDONLY | O_TEXT would
ammount to O_RDONLY | 0, which would in turn ammount to simply
O_RDONLY.
I can't see the kernel devs going for it, though.

I think what the OP is proposing is option (b) above: allow an O_TEXT
identifier having no effect.

> Of course, if you answer that, under a POSIX-like system,
> there would be /no/ consequences to O_TEXT, then you
> have presented choice (b), a change that changes nothing.

Right. This is trivially doable. But, as I said, I can't see this
happening.
To the OP: a solution to your problem is simple. Put the following
lines in your source code before any use of O_TEXT:

#ifndef O_TEXT
#define O_TEXT 0
#endif

--


----- Dig the NEW and IMPROVED news sig!! -----


-------------- Shaggy was here! ---------------
Ain't I'm a dawg!!

[Paul Edwards]

On 21/02/24 11:36, Peter 'Shaggy' Haywood wrote:

>> b) accept your request and add an O_TEXT flag that does nothing, or
>>

>> Choice (b) is ... useless. It recognizes a non-issue, and does
>> nothing. It would be a header change that reserves a flag that would
>> be ignored by everyone. I believe that the maintainers would simply
>> veto this as being a change that makes no change.
>
> This would be useless, true. But it would be harmless. It could be
> defined simply as
>
> #define O_TEXT 0

No. It can't be that. It needs to be non-zero so
that I can detect it in a non-Linux environment.

> I think what the OP is proposing is option (b) above: allow an O_TEXT
> identifier having no effect.

No effect ... on Linux.

An effect ... on PDOS/386.

> To the OP: a solution to your problem is simple. Put the following
> lines in your source code before any use of O_TEXT:
>
> #ifndef O_TEXT
> #define O_TEXT 0
> #endif

No, that doesn't solve my problem.

When the ELF executable is run on PDOS/386, I will
have no way of knowing that this open is text, and
thus PDOS/386 (the OS) should add CRs whenever it
sees a LF.

Because we're beyond the point when anyone else
would add the CR.

BFN. Paul.

[Richard Kettlewell]

Paul Edwards <muta...@gmail.com> writes:
> On 21/02/24 20:25, Richard Kettlewell wrote:
>> Standard C doesn’t have open(), read() or write() either,
>
> Indeed - because PDPCLIB's implementation of fopen()
> needs to do the open syscall (INT 80H EAX = 5H).
>
> fopen already knows whether the file is being opened as text or
> binary, and I don't want that information lost when I do that
> interrupt above. Even though Linux (currently, and very likely
> forever) won't use it, PDOS/386 has a use for it (in the short term).
>
>> but you seem to want to modify the behaviour of those.
>
> I don't think that is an accurate statement.
>
> What behavior, where?

You’ve been talking about doing translation _somewhere_, and cited
Cygwin as an example in your first posting, and have been consistently
using the flag name from Cygwin. Cygwin’s O_TEXT causes read() and
write() to translate between newline conventions, so if you meant
something else then it was a confusing decision to use Cygwin as an
example.

If, in fact, you don’t want to add translation to read() and write()
then you don’t need to add any new flags to open(). As an existing
example, Windows manages to do translation in its <stdio.h>
implementation without any assistance from CreateFile and its
friends. Presumably it records the text/binary distinction in its
implementation of the FILE type.

--
https://www.greenend.org.uk/rjk/

[Richard Kettlewell]

"J.O. Aho" <us...@example.net> writes:
> _sopen_s() (secure version of _open()) and _O_TEXT/_O_BINARY do exists
> and I think this is what Cygwin wraps to open() and O_TEXT/O_BINARY.

Cygwin it has its own independent implementation of newline translation.

--
https://www.greenend.org.uk/rjk/

[Paul Edwards]

On 21/02/24 20:52, J.O. Aho wrote:

>>> Think you need to look into the windows source code, as open() already
>>> exists in windows
>>
>> There is no such syscall. kernel32 exports CreateFile.
>
> _sopen_s() (secure version of _open()) and _O_TEXT/_O_BINARY do exists
> and I think this is what Cygwin wraps to open() and O_TEXT/O_BINARY.

Thanks for the pointer.

That allowed me to find this:

https://learn.microsoft.com/en-us/cpp/c-runtime-library/reference/open-wopen?view=msvc-170

https://learn.microsoft.com/en-us/cpp/c-runtime-library/text-and-binary-mode-file-i-o?view=msvc-170

And those functions exist here:

C:\WINNT\system32>odwin -x msvcrt.dll | grep -i open
[ 218] _fdopen
[ 242] _fsopen
[ 403] _open
[ 404] _open_osfhandle
[ 414] _popen
[ 445] _sopen
[ 527] _wfdopen
[ 534] _wfopen
[ 535] _wfreopen
[ 536] _wfsopen
[ 547] _wopen
[ 550] _wpopen
[ 558] _wsopen
[ 620] fopen
[ 628] freopen

C:\WINNT\system32>

(on my Windows 2000 system)

However, do note that this msvcrt.dll is undocumented
and nominally not meant to be used directly, and only
as part of Visual Studio (and even then, it's no longer
called that - it gets a version number added).

So this is very different from a kernel32.dll equivalent
of a syscall.

However, it does at least give prior art - on top of
Cygwin.

So it is something that I can potentially go to the
Austin Group with.

>> Because PDOS/386 is sort of a DOS/Windows clone,
>> all files are expected to use CRLF as line terminators.
>> That's the user requirements, basically.
>
> Why don't you ask Microsoft to make ms-windows POSIX compatible, then
> this issue would disappear at once ;)

Well that's what everyone has been saying for decades
now, and Microsoft wins every time.

So you can continue fighting that war - no problem.

However, I do want to approach it from the other
direction - a Windows clone.

And I've been working on mine for decades too.

>> So I'm trying to get my Linux ELF binaries (ones that
>> I build myself) to fit into that existing user
>> requirement.
>
> That's why other projects has solved it as define O_TEXT/O_BINARY to 0x0
> if they are missing when they want to support multiple OS:es.

That solves a different problem a different way.

> They don't
> expect that all OS has AmigaOS dos library implemented to open files
> IDOS->Open("c:dir", MODE_OLDFILE)

Well that's an interesting analogy.

One that I have already visited.

So, let's start with PCs with the same processor,
in this case 68000. Atari is a competitor to the
Amiga. Can the Atari run Amiga software?

The equivalent of "everyone should be POSIX" is
"everyone should be Atari". And that's what Atari
thinks. And the Amiga will win every time.

So, my approach - can the Atari run Amiga software?
At least C90-compliant software.

And the answer is - no. At their heart, the Amiga
executables work (without DLLs) by hardcoding the
number 4. All the Amigados functions hang off that.

And on the Atari, address 4 cannot be used.

So should we give up?

No.

First of all, we change the Amiga applications - in
a way that doesn't affect them running on the Amiga.

See here:

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/pdpclib/amistart.asm

I carry an alternate sysbase in d7. It won't be set
when running under genuine AmigaOS, but when run on
a competitor - perhaps not Atari's OS, but some
other OS running on Atari hardware - like, PDOS/Atari
or something like that - then d7 is set to a register
that points to memory that the Atari can actually
address properly.

And then those - select - AmigaOS executables - basically
ones compiled against PDPCLIB which is the only C library
I know of that implements the d7 protocol - which I only
created a couple of years ago - will run on alternate
hardware/OS.

And now I'm doing the same with Linux executables.

First I change the Linux executables with a view to
getting them to run on competing vendor platforms,
build those executables, and then later bring the
competing platform up to speed.

The d7 Amiga "standard" has been invented years (so
far) in advance of a competitor actually arising.

But that's for a different reason - I'm not that
interested in the Amiga API.

But I am interested in the OS/2 2.0 API.

And indeed - I have already prepared - or in the
midst of preparing - modifications there too.
Instead of depending on the 16-bit thunk libraries
to exist, I'm getting raw keyboard input by using
DosDevIOCtl. You can see that here:

https://sourceforge.net/p/pdos/gitcode/ci/master/tree/pdpclib/stdio.c#l2184

So now I am getting into position for PDOS/386 to
support the 32-bit MSDOS API (which I sort of
created myself because Microsoft didn't do that),
Win32, OS/2 2.0 and Linux ELF.

All in the same OS, all with very little code to
support the different OSes, and the OS hopefully
fits on a 360k floppy even with that extra
functionality (or should come very close, anyway -
noting that it isn't a design goal to fit on a
360k floppy - 720k would be period accurate I think).

BFN. Paul.

[Lew Pitcher]

On Wed, 21 Feb 2024 21:54:02 +0800, Paul Edwards wrote:
[snip]

> When the ELF executable is run on PDOS/386, I will
> have no way of knowing that this open is text, and
> thus PDOS/386 (the OS) should add CRs whenever it
> sees a LF.

ISTM that you are looking at this from the wrong end.

POSIX open() does not make a distinction between a "text"
and a "binary" file; all files are "binary". Programs
(including Linux programs) that are coded to the POSIX
standard already accommodate this. So, any code you port
from a Linux environment (either source code or ELF
executable) is already in a state to read and write
a binary file. Given this, then there is no need for
an O_TEXT flag for a Linux program, as no Linux program
expects open() to provide special conditions for text
data.

Given this, there only one problem to resolve: how to
allow for text data to be interchanged transparently
between the MSDOS/Windows side of your PDOS/386 and
the Linux side.

And, that's not the problem you are chasing ATM.

[snip]

[Paul Edwards]

On 22/02/24 04:28, Lew Pitcher wrote:
> On Wed, 21 Feb 2024 21:54:02 +0800, Paul Edwards wrote:
> [snip]
>> When the ELF executable is run on PDOS/386, I will
>> have no way of knowing that this open is text, and
>> thus PDOS/386 (the OS) should add CRs whenever it
>> sees a LF.
>
> ISTM that you are looking at this from the wrong end.
>
> POSIX open() does not make a distinction between a "text"
> and a "binary" file; all files are "binary". Programs

Sure. And I'm talking about changing POSIX to
add the distinction, for anyone who has a use
for such a thing.

> (including Linux programs) that are coded to the POSIX
> standard already accommodate this. So, any code you port
> from a Linux environment (either source code or ELF
> executable) is already in a state to read and write
> a binary file.

This would be new code obviously, and include
an O_TEXT.

> Given this, then there is no need for
> an O_TEXT flag for a Linux program, as no Linux program
> expects open() to provide special conditions for text
> data.

I want to be able to WRITE such a Linux program.

> Given this, there only one problem to resolve: how to
> allow for text data to be interchanged transparently
> between the MSDOS/Windows side of your PDOS/386 and
> the Linux side.

PDOS/386 *will be Linux*. A mini-clone. Also a
mini-clone of OS/2. It's already a mini-clone
of Windows, and sort of a clone of MSDOS too.

All on a 360k floppy.

The only question is - on this all-encompassing OS -
are text files going to follow DOS (CP/M) convention
or Unix convention?

And currently I am going for CP/M, which means my
Linux executables need to provide the appropriate
information so that they run seamlessly in this
CP/M-standard environment.

One day I'll probably do the reverse. And since it
is the C library that is adding CRs for Windows,
it will require a similar enhancement for the
Windows CreateFile() interface to pass a "text"
flag so that PDOS/386 can STRIP the CRs that will
be flowing through.

But that looks even more difficult:

https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilea

BFN. Paul.

[Paul Edwards]

On 21/02/24 22:20, Richard Kettlewell wrote:
> Paul Edwards <muta...@gmail.com> writes:
>> On 21/02/24 20:25, Richard Kettlewell wrote:
>>> Standard C doesn’t have open(), read() or write() either,
>>
>> Indeed - because PDPCLIB's implementation of fopen()
>> needs to do the open syscall (INT 80H EAX = 5H).
>>
>> fopen already knows whether the file is being opened as text or
>> binary, and I don't want that information lost when I do that
>> interrupt above. Even though Linux (currently, and very likely
>> forever) won't use it, PDOS/386 has a use for it (in the short term).
>>
>>> but you seem to want to modify the behaviour of those.
>>
>> I don't think that is an accurate statement.
>>
>> What behavior, where?
>
> You’ve been talking about doing translation _somewhere_,

Yes - when run under PDOS/386.

Currently the only Linux ELF program that runs
under PDOS/386 is a "hello world".

And I don't think anyone cares if I change the
behavior of that.

And that program doesn't even use open(), so is
not subject to my proposed change.

I guess I am talking about changing FUTURE
behavior of a FUTURE program.

But I thought you were implying I was going
to break something.

> and cited
> Cygwin as an example in your first posting, and have been consistently
> using the flag name from Cygwin. Cygwin’s O_TEXT causes read() and
> write() to translate between newline conventions, so if you meant
> something else then it was a confusing decision to use Cygwin as an
> example.

Cygwin is a correct example, I believe.

And Cygwin doesn't change the behavior of Linux ELF
executables either.

Nor does it change the behavior of existing POSIX
source code.

> If, in fact, you don’t want to add translation to read() and write()

I do.

I guess it's just a semantic debate as to whether
this constitutes a change in behavior.

All existing Linux ELF programs are unchanged.

Even new Linux ELF programs won't change when
run under Linux.

The flag only takes effect when the ELF is run
on a competing system.

BFN. Paul.

[Richard Kettlewell]

Paul Edwards <muta...@gmail.com> writes:
> Cygwin is a correct example, I believe.
>
> And Cygwin doesn't change the behavior of Linux ELF
> executables either.

AFAIK it can’t run them at all, so I don’t know why that would be
relevant.

> Nor does it change the behavior of existing POSIX source code.

Your use case was C90 programs, not POSIX programs, last time you
mentioned it. Has that changed now?

>> If, in fact, you don’t want to add translation to read() and write()
> I do.

OK, so an ELF executable (that had been modified to use your
hypothetical O_TEXT flag) would get newline translation from read() and
write() when run on your toy OS, but not when run on a real Linux
kernel. There’s the change in behavior.

At any rate we’re back to the issue that you say want to add
Cygwin-style translation to read() and write(), but you also say that
you will only be running programs written in C90, which doesn’t have
open(), so nothing will ever pass O_TEXT and the translation will never
be activated.

> I guess it's just a semantic debate as to whether this constitutes a
> change in behavior.

This would go a lot quicker if you didn’t engage in “semantic debates”.

--
https://www.greenend.org.uk/rjk/

[Paul Edwards]

On 22/02/24 17:57, Richard Kettlewell wrote:
> Paul Edwards <muta...@gmail.com> writes:

>> Cygwin is a correct example, I believe.
>>
>> And Cygwin doesn't change the behavior of Linux ELF
>> executables either.
>
> AFAIK it can’t run them at all, so I don’t know why that would be
> relevant.

That's exactly the point - it doesn't run them.
No behavior is being changed by Cygwin either.

>> Nor does it change the behavior of existing POSIX source code.
>
> Your use case was C90 programs, not POSIX programs, last time you
> mentioned it. Has that changed now?

I am the vendor of a C90 runtime library for Linux (PDPCLIB).

fopen() necessarily does an open syscall.

>>> If, in fact, you don’t want to add translation to read() and write()
>> I do.
>
> OK, so an ELF executable (that had been modified to use your
> hypothetical O_TEXT flag) would get newline translation from read() and
> write() when run on your toy OS, but not when run on a real Linux
> kernel. There’s the change in behavior.

Well you can call it that if you want. But as I said,
I think it is odd to call something a change in behavior
when the software that would be "changing" hasn't even
been written yet. Or at least, the INT 80H open() processing
of PDOS/386 hasn't been written yet. Only write to stdout
has been written so far. And exit.

Also, when you say "the ELF executable that has been
modified", I would use the word "built" rather than
"modified". I am building new ELF executables according
to a hypothetical version of POSIX/Linux that includes
a new O_TEXT definition.

> At any rate we’re back to the issue that you say want to add
> Cygwin-style translation to read() and write(), but you also say that
> you will only be running programs written in C90, which doesn’t have
> open(), so nothing will ever pass O_TEXT and the translation will never
> be activated.

PDPCLIB will do that.

I clearly haven't done a very good job of explaining
what I want.

I am organizing contact with the "Austin Group" (I've
become a member and downloaded the latest POSIX draft).

Is there any wording I should use to avoid the several
days of confusion we've had here?

When I wrote my original message I thought it was a very
simple suggestion, using the widely-known Cygwin prior art.

Instead, I'm not even sure my proposal is understood.

>> I guess it's just a semantic debate as to whether this constitutes a
>> change in behavior.
>
> This would go a lot quicker if you didn’t engage in “semantic debates”.

I wasn't attempting to. I was just trying to identify
the point of contention as semantics, not a technical
issue.

It's still not clear to me that it really is semantics,
because you are using language that I wouldn't use.

BFN. Paul.

[J.O. Aho]

On 22/02/2024 10.57, Richard Kettlewell wrote:
> Paul Edwards <muta...@gmail.com> writes:

>> I guess it's just a semantic debate as to whether this constitutes a
>> change in behavior.
>
> This would go a lot quicker if you didn’t engage in “semantic debates”.

For me it whole feels like a repeat of systemd when they stupidly picked
a namespace already used by the kernel and wanted the kernel to change
nem of their namespace.

The whole can be solved in the header to define missing macro.

--
//Aho

[Paul Edwards]

Define it to what? That's my original question.

And by who?

BFN. Paul.

[Lew Pitcher]

That's up to whomever defines it.

> And by who?

That would be you. And /not/ by Linux.