How to get the next line character(s)

[w...@totalbb.net.tw]

How to get the OS dependent new line character(s) and store in a variable?

const char newline[]=std::endl; // this line shows intention but don't work

[Melzzzzz]

On 2018-02-15, w...@totalbb.net.tw <w...@totalbb.net.tw> wrote:
> How to get the OS dependent new line character(s) and store in a variable?
>
> const char newline[]=std::endl; // this line shows intention but don't work

What dou you want? '\n' is for new line , but if you want to follow
specific OS then look what character(s) are needed.

--
press any key to continue or any other to quit...

[Paavo Helde]

On 15.02.2018 4:53, w...@totalbb.net.tw wrote:
> How to get the OS dependent new line character(s) and store in a variable?
>
> const char newline[]=std::endl; // this line shows intention but don't work
>

In practice (assuming the old classic Mac OS is derelict):

#ifdef _WIN32
const char newline[]="\r\n";
#else
const char newline[]="\n";
#endif

However, there is little to do with that information because there is no
guarantee that any text files you read have been actually written on the
same platform, and for any text files you write you can easily produce
platform-specific newlines by opening the file in the (default) text mode.

hth
Paavo

[Robert Wessel]

On Wed, 14 Feb 2018 18:53:50 -0800 (PST), w...@totalbb.net.tw wrote:

>How to get the OS dependent new line character(s) and store in a variable?
>
>const char newline[]=std::endl; // this line shows intention but don't work

It's quite a hack, but you could open a file in text mode, write a
"\n" to it close it, reopen in binary and read the result. Of course
that's not actually portable, there's no guarantee that there are
*any* newline characters on a particular platform (on some the file
will be a sequence of records).

Unfortunately it's implementation dependent whether a stringstream pay
any attention to binary mode, so that's not an option.

[Chris M. Thomasson]

On 2/14/2018 9:09 PM, Robert Wessel wrote:
> On Wed, 14 Feb 2018 18:53:50 -0800 (PST), w...@totalbb.net.tw wrote:
>
>> How to get the OS dependent new line character(s) and store in a variable?
>>
>> const char newline[]=std::endl; // this line shows intention but don't work
>
>
> It's quite a hack, but you could open a file in text mode, write a
> "\n" to it close it, reopen in binary and read the result.

Well, afaict, that should give you access to any extra injected, or
ejected bytes...

[Alf P. Steinbach]

On 15.02.2018 03:53, w...@totalbb.net.tw wrote:
> How to get the OS dependent new line character(s) and store in a variable?
>
> const char newline[]=std::endl; // this line shows intention but don't work
>

In practice, if it's Windows then it's "\r\n", and otherwise it's "\n",
even on the Mac (in the old days the Mac had "\r").

I.e.

#ifdef _WIN32
char const[] newline = "\r\n";
#else
char const[] newline = "\n";
#endif

Where `_WIN32` is the common convention on both 32-bit and 64-bit Windows.

So much is different in Windows, e.g. the native encoding of
std::filesystem, that one should maybe define a reusable boolean constant:

// <url: https://github.com/alf-p-steinbach/stdlib>
// Relevant macro symbols (may be needed with any particular compiler):
// STDLIB_USE_EXPERIMENTAL_CPP17, STDLIB_HOIST_UP_STRINGVIEW_TEMPLATE
#include <stdlib/string_view.hpp>

namespace os {
using std::string_view;

#ifdef _WIN32
constexpr bool is_windows = true;
#else
constexpr bool is_windows = false;
#endif

constexpr string_view newline = (is_windows? "\r\n" : "\n");
} // namespace os

auto main()
int
{ return os::newline.length(); }

However, both g++ 7.1 and Visual C++ 2017 fail to implement a proper
`constexpr` `std::string_view` constructor taking `char const*`.

Workaround:

#include <stdlib/string_view.hpp>

namespace os {
using std::string_view;

#ifdef _WIN32
constexpr bool is_windows = true;
#else
constexpr bool is_windows = false;
#endif

constexpr auto newline =
is_windows? string_view{ "\r\n", 2 } : string_view{ "\n", 1 };
} // namespace os

auto main()
-> int
{ return os::newline.length(); }

Cheers & hth.,

- Alf

[James Kuyper]

On 02/15/2018 12:09 AM, Robert Wessel wrote:
> On Wed, 14 Feb 2018 18:53:50 -0800 (PST), w...@totalbb.net.tw wrote:
>
>> How to get the OS dependent new line character(s) and store in a variable?
>>
>> const char newline[]=std::endl; // this line shows intention but don't work
>
>
> It's quite a hack, but you could open a file in text mode, write a
> "\n" to it close it, reopen in binary and read the result. Of course
> that's not actually portable,

The job to be performed (getting the OS dependent method of indicating
line endings) is inherently unportable. The standard provides strong
guarantees when a file is written in binary mode and read back in binary
mode (though they are slightly weaker than you might expect: an
arbitrary number of null characters can be added to the end of your file
without violating those guarantees). It provides weaker guarantees for
writing and reading in text mode, and no guarantees at all for writing
in one mode and reading in the other. However, using that approach is no
more unportable than the task it's intended to perform, and should work
on any implementation I can imagine.

> ... there's no guarantee that there are

> *any* newline characters on a particular platform (on some the file
> will be a sequence of records).

Correct. On many systems, files are allocated in fixed-size blocks, and
some such systems require that each line in a text file start in a
different block. There's various strategies used to indicate where the
end of a line is inside the block: among other possibilities, the block
could be padded with blanks or null characters, or there might be a
character count at the beginning of the block, in which case the rest of
the block could be padded with garbage, instead. You can discover all
these things by the method you describe above, but it's much better to
read an implementation's documentation to discover how end of line is
represented in text files.

> Unfortunately it's implementation dependent whether a stringstream pay
> any attention to binary mode, so that's not an option.

I couldn't find anything in the standard to support that claim. I'm not
saying you're wrong, I might have failed to look in the right location.
But could you provide a citation to back up that claim?
There are implementations (especially ones targeting Unix-like systems)
where there's no difference between text and binary mode - but that's
because text mode works just like binary mode - not because binary mode
is ignored.

[Jorgen Grahn]

On Thu, 2018-02-15, Melzzzzz wrote:
> On 2018-02-15, w...@totalbb.net.tw <w...@totalbb.net.tw> wrote:
>> How to get the OS dependent new line character(s) and store in a variable?
>>
>> const char newline[]=std::endl; // this line shows intention but don't work
>
> What dou you want? '\n' is for new line , but if you want to follow
> specific OS then look what character(s) are needed.

And /why/ does he want it? In my experience, you usually have a
design problem if you need to know. (E.g. Python has os.linesep,
but I suspect it's not used much.)

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[jak]

Il 15/02/2018 03:53, w...@totalbb.net.tw ha scritto:
> How to get the OS dependent new line character(s) and store in a variable?
>
> const char newline[]=std::endl; // this line shows intention but don't work
>

a way but in C:

int main(int ac, char *av[])
{
int fdp[2], i, l;
char str[3];
FILE *fpw, *fpr;

if(pipe(fdp) != -1)
if((fpw = fdopen(fdp[1], "wt")) != NULL)
if((fpr = fdopen(fdp[0], "rb")) != NULL)
{
fputc('\n', fpw);
fflush(fpw);
fgets(str, sizeof str, fpr);
fclose(fpw);
fclose(fpr);
printf("%d:", strlen(str));
for(i = 0, l = strlen(str); i < l; i++)
printf(" 0x%02X", str[i]);
return 0;
}
perror("something wrong");
return 1;
}

[James Kuyper]

On 02/15/2018 11:26 AM, jak wrote:
> Il 15/02/2018 03:53, w...@totalbb.net.tw ha scritto:
>> How to get the OS dependent new line character(s) and store in a variable?
>>
>> const char newline[]=std::endl; // this line shows intention but don't work
>>
> a way but in C:
>
> int main(int ac, char *av[])
> {
> int fdp[2], i, l;
> char str[3];
> FILE *fpw, *fpr;
>
> if(pipe(fdp) != -1)
> if((fpw = fdopen(fdp[1], "wt")) != NULL)
> if((fpr = fdopen(fdp[0], "rb")) != NULL)
> {
> fputc('\n', fpw);
> fflush(fpw);
> fgets(str, sizeof str, fpr);

"A text stream is ... Characters may have to be added, altered, or
deleted on input and output to conform to differing conventions for
representing text in the host environment. Thus, there need not be a
one-to-one correspondence between the characters in a stream and those
in the external representation. ..." (7.21.2p2). This implies that for
files opened in text mode, there maybe a conversion process between the
characters in the stream and the external representation, and it is the
responsibility of C's I/O routines to implement that conversion. You can
think of those conversions as being done by fputc() and fgetc(), since
all of the other functions that write to or read from a file have their
behavior ultimately defined in terms of calls those functions (though
they need not actually call them).

fgets() stops after reading in a new-line character. But if the external
representation of a new-line in a text file is anything other than a
single byte with a value of '\n', then for a file opened in binary mode,
no such conversion is performed, and fgets() might not even read such a
character. If it does, it might do so in some location other than
precisely at the end of a line.

To keep things clear and simple, I've removed all error handling from
the following code. Since every single function called here can fail,
the proper version of this code is 60 lines long.

#include <stdio.h>
int main(void)
{
// Write a new line in text mode.
FILE *file = fopen("line_ending.txt", "w");
putc('\n', file);
fclose(file);

// Read and hexdump the file in binary mode.
file = fopen("line_ending.txt", "rb");
fputs("Line ending:", stdout);
for(int c; (c = getc(file)) != EOF; )
printf(" %#x", c);
putchar('\n');

return 0;
}

[Chris Vine]

On Thu, 15 Feb 2018 09:15:58 -0500
James Kuyper <james...@verizon.net> wrote:
> On 02/15/2018 12:09 AM, Robert Wessel wrote:

[snip]

> > Unfortunately it's implementation dependent whether a stringstream
> > pay any attention to binary mode, so that's not an option.
>
> I couldn't find anything in the standard to support that claim. I'm
> not saying you're wrong, I might have failed to look in the right
> location. But could you provide a citation to back up that claim?
> There are implementations (especially ones targeting Unix-like
> systems) where there's no difference between text and binary mode -
> but that's because text mode works just like binary mode - not
> because binary mode is ignored.

Out of interest, since stringstreams do not write out their contents to
an external representation, what would the difference between binary and
text mode actually mean? To a string, and therefore to a stringstream,
a '\n' character is a '\n' character.

I am not saying that either of you is particularly wrong (I have no
idea): I am puzzling about the consequences of the issue.

Chris

[Paavo Helde]

On 15.02.2018 6:31, Paavo Helde wrote:
> On 15.02.2018 4:53, w...@totalbb.net.tw wrote:
>> How to get the OS dependent new line character(s) and store in a
>> variable?
>>
>> const char newline[]=std::endl; // this line shows intention but
>> don't work
>>
>
> In practice (assuming the old classic Mac OS is derelict):
>
> #ifdef _WIN32
> const char newline[]="\r\n";
> #else
> const char newline[]="\n";
> #endif

Appears this is not so easy, one should use at least

#if defined(_WIN32) || defined(__CYGWIN__)

and probably there are more gotchas.

[James Moe]

On 02/14/2018 07:53 PM, w...@totalbb.net.tw wrote:

> How to get the OS dependent new line character(s) and store in a variable?
>

Why?

--
James Moe
jmm-list at sohnen-moe dot com
Think.

[James Kuyper]

On 02/15/2018 12:34 PM, Stefan Ram wrote:
> Supersedes: <new-line-20...@ram.dialup.fu-berlin.de>

>
> James Kuyper <james...@verizon.net> writes:
>> The job to be performed (getting the OS dependent method of indicating
>> line endings) is inherently unportable.
>

> For unknown reasons I don't see the OP here, so I reply
> to your post instead.
>
> I was curious what strftime( "%n" ) would write into a
> buffer, so I wrote this program.
>
> #include <ctime>
> #include <iostream>
> #include <ostream>
>
> int main()
> { char buff[ 256 ];
>
> ::std::time_t now { ::std::time( nullptr ) };
>
> size_t const size = ::std::strftime
> ( buff,
> sizeof( buff ),
> "%n",
> ::std::localtime( &now ));
>
> ::std::cout << "size = " << size << '\n';
> ::std::cout << "buff = \"" << buff << "\"\n";
>
> { int i = 0; for( size_t s = size; s; --s )
> ::std::cout << static_cast< long >( buff[ i++ ])<< '\n'; }}
>
> Most probably, the »%n« will just write »\n«, so it will bring
> no gain in knowledge about the external representation of »\n«.

Correct - that's what's specified by the C++ standard.

> But what I observed here under a port of GCC under Windows
> is instead that »strftime« will return an empty buffer as
> soon as »%n« is used, ...

As I understand it, that's non-conforming behavior.

> ... while under a variant of Linux, the
> program behaves fine (and »%n« inserts static_cast< char >( 10 )).

[Egor]

w...@totalbb.net.tw wrote:
> How to get the OS dependent new line character(s) and store in a variable?
>
> const char newline[]=std::endl; // this line shows intention but don't work

In text mode '\n' will get transformed to the platform specific newline
sequence. If you need to know a newline sequence for something other
than outputting to standard output or a file, you'll need to create a
temp file, write a '\n' into it and then read it back in binary mode.
Something like this should work:

#include <stdio.h>

char tempfn[L_tmpnam];
tmpnam(tempfn);
FILE* f = fopen(tempfn, "w");
putc('\n', f);
f = freopen(tempfn, "rb", f);
/* this assumes that new line sequence is no longer than 2 characters.
* if for some reason you need more, use fseek and ftell to get file
* size, and then allocate buffer of appropriate size */
char newlineseq[3] = {0};
fread((void*)newlineseq,sizeof(char),2,f);
fclose(f);

Idealy your build system should figure it out for you and then put it in
a header file.

[James Kuyper]

On 02/15/2018 12:35 PM, Chris Vine wrote:
> On Thu, 15 Feb 2018 09:15:58 -0500
> James Kuyper <james...@verizon.net> wrote:
>> On 02/15/2018 12:09 AM, Robert Wessel wrote:
> [snip]
>>> Unfortunately it's implementation dependent whether a stringstream
>>> pay any attention to binary mode, so that's not an option.
>>
>> I couldn't find anything in the standard to support that claim. I'm
>> not saying you're wrong, I might have failed to look in the right
>> location. But could you provide a citation to back up that claim?
>> There are implementations (especially ones targeting Unix-like
>> systems) where there's no difference between text and binary mode -
>> but that's because text mode works just like binary mode - not
>> because binary mode is ignored.
>
> Out of interest, since stringstreams do not write out their contents to
> an external representation,

I didn't catch that point when reading Robert's message. Since the
process he described in the previous paragraph [which was snipped
above) involved writing to and reading from a file, the behavior of
stringstream should be irrelevant - it's the behavior of fstream that's
relevant, which is why I didn't notice that he was referring to
stringstream.

[James Kuyper]

On 02/15/2018 12:34 PM, James Kuyper wrote:
> On 02/15/2018 11:26 AM, jak wrote:
>> Il 15/02/2018 03:53, w...@totalbb.net.tw ha scritto:
>>> How to get the OS dependent new line character(s) and store in a variable?
>>>
>>> const char newline[]=std::endl; // this line shows intention but don't work
>>>
>> a way but in C:
>>
>> int main(int ac, char *av[])
>> {
>> int fdp[2], i, l;
>> char str[3];
>> FILE *fpw, *fpr;
>>
>> if(pipe(fdp) != -1)
>> if((fpw = fdopen(fdp[1], "wt")) != NULL)
>> if((fpr = fdopen(fdp[0], "rb")) != NULL)
>> {
>> fputc('\n', fpw);
>> fflush(fpw);
>> fgets(str, sizeof str, fpr);
>
> "A text stream is ... Characters may have to be added, altered, or
> deleted on input and output to conform to differing conventions for
> representing text in the host environment. Thus, there need not be a
> one-to-one correspondence between the characters in a stream and those
> in the external representation. ..." (7.21.2p2).

I forgot that I was posting to a C++ group. That was a citation from the
C standard, not the C++ standard. It's still appropriate, because C++
doesn't address such issues itself, cross-referencing the C standard for
that purpose.

[Richard]

[Please do not mail me a copy of your followup]

"Alf P. Steinbach" <alf.p.stein...@gmail.com> spake the secret code
<p63dp4$8ea$1...@dont-email.me> thusly:

>However, both g++ 7.1 and Visual C++ 2017 fail to implement a proper
>`constexpr` `std::string_view` constructor taking `char const*`.

I bumped into this when I made the string view koan. I'm surprised
that g++ got it wrong; I thought I tested it on g++ and got it
working.

>Workaround:
>
> #include <stdlib/string_view.hpp>
>
> namespace os {
> using std::string_view;
>
> #ifdef _WIN32
> constexpr bool is_windows = true;
> #else
> constexpr bool is_windows = false;
> #endif
>
> constexpr auto newline =
> is_windows? string_view{ "\r\n", 2 } : string_view{ "\n", 1 };
> } // namespace os

Here's an alternative workaround (untested):

namespace os
{
using std::string_view;

using namespace std::string_view_literals;

// ...

constexpr auto newline = is_windows ? "\r\n"sv : "\n"sv;
}
--
"The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
The Terminals Wiki <http://terminals-wiki.org>
The Computer Graphics Museum <http://computergraphicsmuseum.org>
Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>

[jak]

I'm sorry because, probably, my bad English prevents me from
understanding about you do not agree. Now I take your sample code and it
does not work differently from mine: when I open the file in text mode
and write in a '\n' I get a file which in windows is 2 bytes long while
in linux only 1. Respectively the two files contain 0A 0D and 0A.
Reopening the files, this time in binary mode, I read from them exactly
what 'they contain either with fread, fgetc or fgets and that allows me
to understand which end line adopts the operating system. Maybe you want
to tell me that I'm using the fgets function inappropriately because the
file was opened in binary mode? in this case you're right but I thought
it would work anyway and I allowed the imprecision. If I misunderstood,
then I apologize because I did not understand.
this program gives the same result as yours:

#include <stdio.h>
int main(void)
{
char str[3];

// Write a new line in text mode.

FILE *file = fopen("line_ending.txt", "wb");

putc('\n', file);
fclose(file);

// Read and hexdump the file in binary mode.
file = fopen("line_ending.txt", "rb");

fgets(str, 3, file);

fputs("Line ending:", stdout);

for(int c = 0; c < 3; c++)
printf(" %#x", str[c]);

[Richard]

[Please do not mail me a copy of your followup]

Jorgen Grahn <grahn...@snipabacken.se> spake the secret code
<slrnp8b822.e...@frailea.sa.invalid> thusly:

> [...] (E.g. Python has os.linesep,

>but I suspect it's not used much.)

Sometimes you need to preserve the existing line endings. I fought
with Python at one point over this and lost; Python simply insisted on
mangling the data in the file to match the EOL convention of the OS
instead of preserving the EOL convention of the file.

[Richard]

[Please do not mail me a copy of your followup]

Paavo Helde <myfir...@osa.pri.ee> spake the secret code
<p64h0d$rsv$1...@dont-email.me> thusly:

>Appears this is not so easy, one should use at least
>
>#if defined(_WIN32) || defined(__CYGWIN__)

Well, there's you're problem. Don't use cygwin, it's an abomination :)

[Paavo Helde]

On 15.02.2018 20:56, Richard wrote:
> [Please do not mail me a copy of your followup]
>
> Paavo Helde <myfir...@osa.pri.ee> spake the secret code
> <p64h0d$rsv$1...@dont-email.me> thusly:
>
>> Appears this is not so easy, one should use at least
>>
>> #if defined(_WIN32) || defined(__CYGWIN__)
>
> Well, there's you're problem. Don't use cygwin, it's an abomination :)

Well, I don't, I just use it for testing random snippets for c.l.c++ ;-)

It just was that the pipe example from jak did not compile in Visual
C++, but compiled with CygWin g++ and produced 0x0D 0x0A, despite not
having _WIN32 defined.

OTOH, writing "\n" into a c++ text ofstream and rereading it with a
binary ifstream produced just 0x0A with the CygWin g++. Not sure why a
pipe and a file behave differently. If anything, it's just another
reason for avoiding the "text mode".

Cheers
Paavo

[James Kuyper]

On 02/15/2018 01:55 PM, jak wrote:
> Il 15/02/2018 18:34, James Kuyper ha scritto:

...

Part of the problem is that you probably don't have access to any system
where line endings are indicated in a manner that would cause serious
problems for fgets(). Neither do I, so I can't demonstrate the problem
directly, but I can tell you what I would expect your program to do on
systems using some of the more popular alternatives:

> #include <stdio.h>
> int main(void)
> {
> char str[3];

This array is uninitialized.

> // Write a new line in text mode.
> FILE *file = fopen("line_ending.txt", "wb");
> putc('\n', file);
> fclose(file);
>
> // Read and hexdump the file in binary mode.
> file = fopen("line_ending.txt", "rb");
> fgets(str, 3, file);
> fputs("Line ending:", stdout);
> for(int c = 0; c < 3; c++)
> printf(" %#x", str[c]);

Unless your code filled up the entire array (which is possible), you're
printing at least one uninitialized value. For character types, that's
safe, but in general that's something to be avoided. I'll ignore the
uninitialized values in my discussion below.

> putchar('\n');
>
> return 0;
> }

Let's consider what your code would do in systems with a variety of
different ways of representing the end of a line.

1. End of line is indicated by a newline character
Your program would cause str[0] to end up containing '\n', while str[1]
would contain a terminating '\0'.

2. End of line is indicated by "\n\r".
In text mode, fgets() would read both the '\n' and the '\r', but would
convert them into a single '\n' before storing it in str[0], the same as
in case 1.
However, since you're using binary mode, fgets would stop after reading
in '\n', storing it in str[0], leaving the '\r' unread, and would place
a terminating '\0' in str[1], exactly the same as in text mode.
Therefore, it would NOT have read the entire line.

3. End of line is indicated by "\r\n".
In text mode, fgets would read both the '\r' and the '\n', but would
convert them into a single '\n' before storing it in str[0].
However, in binary mode, fgets() would simply read in the '\r' directly
into str[0], then reading the '\n' into str[1], at which point it would
stop and place a terminating '\0' in str[2]. In this case, it would read
the entire line.

4. End of line is indicated by '\r'.
In text mode, fgets() would read in the '\r' and convert it to '\n'
before storing it in str[0].
However, in binary mode, it would read the '\r' and then reach the end
of the file without having read in a newline character, so the contents
of str[] would be unchanged, and it would return a null pointer,
indicating failure. Since your code doesn't bother checking whether
fgets() failed, it wouldn't notice. To be fair, I suppressed all error
checking in my code, too - but my code wouldn't result in this error
condition.

5. Text files are stored in fix sized blocks of length 256 bytes, with
the first byte of each block indicating how many bytes of that block are
used. The blocks are padded with blanks. The result of writing a '\n'
character to a file is a block containing a use-count of 0 followed by
254 blank characters.
In text mode, fgets() would read in the entire block, notice that the
use count is 0, and write only a single '\n' to str[0], and a
terminating '\0' to str[1].
However, in binary mode, it would read the use count of 0 into str[0],
and the first padding blank character into str[1]. It would NEVER read
in a new-line character from a text file. It would, however, reach the
limit of 3 characters that you gave fgets(), so it would insert a
terminating '\0' in str[2], without having read in the entire line.

Exercise for the student: If text files are stored in fix-sized blocks
of length 256, with the end of a line indicated by padding the rest of
the block with null characters, what would jak's program do?

[Vir Campestris]

It's the 1980s since I last used a system with files that are sequences
of records - and I didn't run 'C' on them.

I suspect that writing a newline in text, then reading back in binary,
would give you nothing at all. But I'm not sure.

It's kind of implicit in the design of C I/O that files are _not_
sequences of records (let alone clever things like ISAM and HRAM files!).

And it's implicit in the design of record-based filesystems that you
don't read a byte, or a char, or any such thing - you read a record.

Then process that.

Andy
--
Yes, I do mean the 1980s. Early 80s, so I won't be surprised to be out
of date. And curiously when I search for ISAM and HRAM Google tells me
about Islam...

[Richard]

[Please do not mail me a copy of your followup]

Vir Campestris <vir.cam...@invalid.invalid> spake the secret code
<p64tq6$t4j$2...@dont-email.me> thusly:

>Yes, I do mean the 1980s. Early 80s, so I won't be surprised to be out
>of date. And curiously when I search for ISAM and HRAM Google tells me
>about Islam...

Yeah, you have to go to wikipedia otherwise search engines helpfully
"correct" your spelling into something you didn't search for.

<https://en.wikipedia.org/wiki/ISAM>

Around the same time as you, I was using RSTS/E on a PDP-11 and they
had "Record Management Services (RMS)" for record-based file I/O.
<https://en.wikipedia.org/wiki/Record_Management_Services>

[Robert Wessel]

On Thu, 15 Feb 2018 13:12:11 -0500, James Kuyper

I was (unclearly) commenting that the (obvious?) alternative to
writing an actual file and reading it back in to achieve the desired
effect, namely writing in text mode to a stringstream, and then
reading from the resulting buffer with a stringstream in binary mode,
was not something that is (I think) specified to work.

[Robert Wessel]

On Thu, 15 Feb 2018 21:24:22 +0000, Vir Campestris
<vir.cam...@invalid.invalid> wrote:

>On 15/02/2018 05:09, Robert Wessel wrote:
>> On Wed, 14 Feb 2018 18:53:50 -0800 (PST), w...@totalbb.net.tw wrote:
>>
>>> How to get the OS dependent new line character(s) and store in a variable?
>>>
>>> const char newline[]=std::endl; // this line shows intention but don't work
>>
>>
>> It's quite a hack, but you could open a file in text mode, write a
>> "\n" to it close it, reopen in binary and read the result. Of course
>> that's not actually portable, there's no guarantee that there are
>> *any* newline characters on a particular platform (on some the file
>> will be a sequence of records).
>>
>> Unfortunately it's implementation dependent whether a stringstream pay
>> any attention to binary mode, so that's not an option.
>>
>It's the 1980s since I last used a system with files that are sequences
>of records - and I didn't run 'C' on them.

I did it a couple of hours ago - I had a C program read a stream
attached to a record oriented file (it happened to be a member in a
PDS, but it could have been an ordinary sequential file) containing
text (under zOS).

>I suspect that writing a newline in text, then reading back in binary,
>would give you nothing at all. But I'm not sure.

Typically that's correct on zOS. "Straight" binary mode on a record
oriented file just treats the records as a stream of bytes without
delimiters. Text files, especially on files with variable length
records, usually don't store the \n in the record. There are, of
course, extensions for seeing the higher level structures.

>It's kind of implicit in the design of C I/O that files are _not_
>sequences of records (let alone clever things like ISAM and HRAM files!).
>
>And it's implicit in the design of record-based filesystems that you
>don't read a byte, or a char, or any such thing - you read a record.
>
>Then process that.

Yes, but it's often useful to have a mapping, even if it doesn't cover
all cases. For example, being able to read a text file with some
configuration data that was prepared by the "normal" text editor on
the system. But you can't get too carried away in expecting that
it'll behave exactly like a *nix stream-of-bytes.

[Robert Wessel]

Doing this with a *nix style pipe has even more portability issues
than the other solutions. The above, for example, won't work for a
normal Windows application (it should in the Posix subsystem). There
is a _pipe() which is similar, but not identical, though.

[red floyd]

On 2/15/2018 1:24 PM, Vir Campestris wrote:

> It's the 1980s since I last used a system with files that are sequences
> of records - and I didn't run 'C' on them.

Today. HPE NonStop (the old Tandem computers)

[James Kuyper]

On 02/15/2018 04:24 PM, Vir Campestris wrote:
...

> It's the 1980s since I last used a system with files that are sequences
> of records - and I didn't run 'C' on them.
>
> I suspect that writing a newline in text, then reading back in binary,
> would give you nothing at all. But I'm not sure.
>
> It's kind of implicit in the design of C I/O that files are _not_
> sequences of records (let alone clever things like ISAM and HRAM files!).

Actually, it's not. There's several clauses in the C standard that exist
for the sole purpose of allowing such things. For instance, binary
streams are allowed to have an implementation-defined number of null
characters appended to the end of the stream, which can be used to pad
the file to a multiple of the relevant record length.

[James Kuyper]

The meaning for std::binary is defined by table 132 in 27.9.1.4p2 for
basic_filebuf::open(). std::stringstream uses basis_stringbuf(), which
has no such member function, and correspondingly, nothing comparable to
table 132. I don't think it's correct to say that std::binary won't work
for std::stringstream - but rather that it has no effect. Since
std::stringstream never converts between the in-memory and external
representations, it doesn't matter whether or not it's in binary mode.

[Alf P. Steinbach]

On 15.02.2018 19:53, Richard wrote:
> namespace os
> {
> using std::string_view;
> using namespace std::string_view_literals;
>
> // ...
>
> constexpr auto newline = is_windows ? "\r\n"sv : "\n"sv;
> }

That worked with both g++ and MSVC.

Strange.


Cheers!, & thanks

- Alf

[jak]

HI,
I tried the program in both systems just adding this line of code under win:

#define pipe(fdp) _pipe(fdp, 4096, O_BINARY)

[jak]

Il 15/02/2018 21:07, James Kuyper ha scritto:
> On 02/15/2018 01:55 PM, jak wrote:
>> Il 15/02/2018 18:34, James Kuyper ha scritto:
> ...

...

>> #include <stdio.h>
>> int main(void)
>> {
>> char str[3];
>
> This array is uninitialized.
>
>> // Write a new line in text mode.
>> FILE *file = fopen("line_ending.txt", "wb");
>> putc('\n', file);
>> fclose(file);
>>
>> // Read and hexdump the file in binary mode.
>> file = fopen("line_ending.txt", "rb");
>> fgets(str, 3, file);
>> fputs("Line ending:", stdout);
>> for(int c = 0; c < 3; c++)
>> printf(" %#x", str[c]);
>
> Unless your code filled up the entire array (which is possible), you're
> printing at least one uninitialized value. For character types, that's
> safe, but in general that's something to be avoided. I'll ignore the
> uninitialized values in my discussion below.
>

This does not matter to me when I do tests. I do not think it's bad to
see all 3 bytes I'm not confused by these little things :)

>
> 2. End of line is indicated by "\n\r".
> In text mode, fgets() would read both the '\n' and the '\r', but would

...> 3. End of line is indicated by "\r\n".
...

You look a little confused. Is the end of line "\r\n" or "\n\r"? On my
systems it is "\r\n", for that the program works. You insist but I also
said I should not use the fgets.

> 5. Text files are stored in fix sized blocks of length 256 bytes, with
> the first byte of each block indicating how many bytes of that block are
> used. The blocks are padded with blanks. The result of writing a '\n'
> character to a file is a block containing a use-count of 0 followed by
> 254 blank characters.
> In text mode, fgets() would read in the entire block, notice that the
> use count is 0, and write only a single '\n' to str[0], and a
> terminating '\0' to str[1].
> However, in binary mode, it would read the use count of 0 into str[0],
> and the first padding blank character into str[1]. It would NEVER read
> in a new-line character from a text file. It would, however, reach the
> limit of 3 characters that you gave fgets(), so it would insert a
> terminating '\0' in str[2], without having read in the entire line.
>

A file like this can not be called a TEXT file.

> Exercise for the student: If text files are stored in fix-sized blocks
> of length 256, with the end of a line indicated by padding the rest of
> the block with null characters, what would jak's program do?

> scan resume...

[Paavo Helde]

You say you tried, but what was the result? In my test it worked
incorrectly in Windows and produced "1: 0x0A" regardless of whether I
used _O_BINARY or _O_TEXT in the #define (note that O_BINARY is a
"non-standard" alias, the official name is _O_BINARY).

[jak]

Sorry, you're right. I did not copy the line of code from the source but
I wrote it on the fly and forgot the initial underscore. What you tell
me is interesting. Can I ask you in which environment did you try?
(operating system, compiler, msys / msys2, other ...)

[James R. Kuyper]

In the general case, using an uninitialized value can have undefined
behavior (as a practical matter, this is mainly an issue with pointers
and floating point values).

>> 2. End of line is indicated by "\n\r".
>> In text mode, fgets() would read both the '\n' and the '\r', but would
> ...> 3. End of line is indicated by "\r\n".
> ...
>
> You look a little confused. Is the end of line "\r\n" or "\n\r"?

I thought I was quite clear that each case describes a system with a
different method for marking the end of a line. Each of these methods is
in use by actual systems.

It's "\n" in case 1, "\r\n" in case 2, and "\n\r" in case 3, "\r" in
case 4, and is represented in a more complicated fashion in case 5.
These are among the most common ways of marking the end of a line, but
they are far from being the only ones.

> ... On my

> systems it is "\r\n", for that the program works. You insist but I also
> said I should not use the fgets.
>
>> 5. Text files are stored in fix sized blocks of length 256 bytes, with
>> the first byte of each block indicating how many bytes of that block are
>> used. The blocks are padded with blanks. The result of writing a '\n'
>> character to a file is a block containing a use-count of 0 followed by
>> 254 blank characters.
>> In text mode, fgets() would read in the entire block, notice that the
>> use count is 0, and write only a single '\n' to str[0], and a
>> terminating '\0' to str[1].
>> However, in binary mode, it would read the use count of 0 into str[0],
>> and the first padding blank character into str[1]. It would NEVER read
>> in a new-line character from a text file. It would, however, reach the
>> limit of 3 characters that you gave fgets(), so it would insert a
>> terminating '\0' in str[2], without having read in the entire line.
>>
> A file like this can not be called a TEXT file.

Perhaps you would not call it a text file, but the users of such systems
have that format as a permitted format for text files (and on some of
them, it's the only permitted format). They do refer to files in that
format as "text files". Code written for such systems to work with text
data will allow (and may require) such data to be in that format.

More to the point, the C standard's wording about how end of line is
represented is deliberately vague enough to permit an implementation of
C to use that format when a file is opened in text mode. C++ inherits
that feature, since it incorporates the relevant wording from the C
standard by reference.

[Paavo Helde]

Windows 7 Enterprise, SP1, Visual C++ 2017, x64.

[jak]

this in my area is called "climbing on the mirrors"

[James R. Kuyper]

On 02/16/2018 11:06 AM, jak wrote:
> Il 16/02/2018 15:24, James R. Kuyper ha scritto:
>> On 02/16/2018 05:30 AM, jak wrote:
>>> Il 15/02/2018 21:07, James Kuyper ha scritto:

...

>>> ... On my
>>> systems it is "\r\n", for that the program works. You insist but I also
>>> said I should not use the fgets.

Sorry - I should have responded to that paragraph in my previous message.

w...@totalbb.net.tw asked for a way to determine the method by which a
line ending is indicated. That goal implies a desire to have code that
works on systems with a wide variety of ways of indicating a line
ending. If you only care about whether it gives the right result on
systems where the line ending is "\r\n", you have no need to write such
complicated code. The following minor modification to his original
(unworkable) code would be sufficient:

const char newline[] = "\r\n";

I've never heard that phrase before. A google search located a
discussion of how to translate the Italian idiom "arrampicarsi sugli
specchi", which literally means "climbing on mirrors", but whose actual
meaning was quite different. Several of the other hits I got were
English text, poorly written by people with Italian names, supporting
the idea that it's an Italian idiom poorly translated into English.

The discussion I located involved one person who normally translated
that phrase as "clutching at straws", and another who said that "Since
the Italian has a double meaning here which is lost in English I think
you need more than just climbing on mirrors. The other meaning of
arrampicarsi sugli specchi is, trying to justify your existence, save
your skin."

I don't see any particular connection between that idiom and this issue.
It's a very real fact that there have been (and may still be) systems
which used such methods to indicate the end of a line, and that there
were C implementations for such systems which recognized those methods
as such, and that there is wording in the C standard which was
deliberately inserted there to allow such implementations to qualify as
fully conforming. If you want to dismiss such facts as "climbing on the
mirrors", feel free to do so. It's unfortunately not the weirdest reason
I've seen people give for ignoring facts ("no reason at all" is one of
the most popular).

[jak]

Il 16/02/2018 17:57, James R. Kuyper ha scritto:

> I've never heard that phrase before. A google search located a
> discussion of how to translate the Italian idiom "arrampicarsi sugli
> specchi", which literally means "climbing on mirrors", but whose actual
> meaning was quite different. Several of the other hits I got were
> English text, poorly written by people with Italian names, supporting
> the idea that it's an Italian idiom poorly translated into English.

Oh, so now you will understand how difficult it is for me to understand
without misunderstanding, that river of words that you answer to me
every time.
When I read the OP question, I found it interesting because the tendency
to write multi-platform programs is more present every day. Among all
the problems that can be encountered, the two most common are:
identifying the separator character of the paths and how the system
encodes the end of the text line. For the latter problem, most of the
proposed solutions suggest to open a file in text mode, write in a \ n
', reopen it in binary mode and read how it has been encoded. Looking
for alternative solutions, I found someone who thought of using a pipe
channel instead of a file. this seemed interesting and I proposed it by
writing a very short piece of code with the intention of presenting the
idea.

At this point you have questioned me that using the fgets on a file open
in binary mode is a bad idea.

I replied: I know, sorry, it worked and I left the code unchanged.

Then you added: the array is not initialized.

I do not care if the variable is used initially to the left of a = or
passed to a function that intends to write in it. If the function does
not work or works but writes bad data to my array, then it's a badly
written function.

Then again you said: there can be multiple types of end of line ("\n",
"\r", "\n\r", "\r\n").

From my point of view it is not important. The compiler will be
suitable for that system and that code should tell me how it encoded the
"\n".

Add again: there may be text files where the first byte indicates the
length from the line.

First of all it should not be called a text file because this should be
readable by any text editor / viewer and would not be the case with this
file. Furthermore the aim was to understand how the system encodes the
end line in text mode, not to read a file.

Honestly I do not understand if you answer me for your pleasure to
contradict me or if you are trying to explain something that, in your
opinion, I do not know or do not understand. If you fear about my
problems by manipulating files, please do not. When the CLIPPER was
decommissioned I had to write a library in C to access the db produced
by it and the only help came from hexdump of the index files (no
internet, no google, no newsgroups, all these did not yet exist).

regards

[Paavo Helde]

Feel safe, nobody here cares if you can manipulate files or not. Also,
nobody cares what you find important or do not. You have not posted in
this forum long enough to convince anyone that the things you find
important are actually important for other people, or vice versa.

All of this thread is nitpicking about a problem which in practice does
not exist or can be solved trivially by an "#ifdef _WIN32", but which in
theory is very non-trivial and the question itself is probably
ill-formed. All this nitpicking is done by patrons finding some interest
in this strange (non-)problem, there is nothing personal (at least not
yet), so you should not feel offended!

So let the nit-picking continue! Instead of trying to neglect the
questions by labeling them as not important, try to explain why your
pipe/_pipe program produces a wrong result in Visual C++, but not in
Cygwin. If there is an actual reason for that we all might get a bit wiser.

Cheers
Paavo

[jak]

Il 17/02/2018 19:56, Paavo Helde ha scritto:

> So let the nit-picking continue! Instead of trying to neglect the
> questions by labeling them as not important, try to explain why your
> pipe/_pipe program produces a wrong result in Visual C++, but not in
> Cygwin. If there is an actual reason for that we all might get a bit wiser.
>
> Cheers
> Paavo
>
>

I'm doing it but because I'm interested in it and not because you're
saying it.

[jak]

Hi,
I found, on the microsoft site, a document that talks about _pipe
function that says 2 things. the first of these is that to enable the
end-line coding it is necessary to use a different flag with respect to
the one used _O_TEXT. I tried it but works in the same way. The second
info, talks about the need not to use the reading descriptor and the
writing descriptor in the same process/thread if you do not want to
encounter problems. This thing happens in my program. I also found an
example program where, to use both descriptors in the same process, they
are duplicated and used their duplications. As soon as I find some time
I will try if using the descriptors in separate processes the mechanism
will work correctly. Unfortunately, even if it will work it will result
an excessive work only to know how the system encodes the end of line.

[Louis Krupp]

Here's yet another program that might help. My apologies if it matches
something that's already been suggested:

#include <iomanip>
#include <iostream>
#include <sstream>
#include <string>

int main()
{
std::ostringstream oss;

oss << std::endl;

const char *p = oss.str().data();

std::cout << std::setfill('0') << std::setw(2) << std::hex;
while (char c = *p++)
std::cout << int(c);

std::cout << "\n";

return 0;
}

Louis

[Paavo Helde]

On 18.02.2018 6:30, Louis Krupp wrote:
> Here's yet another program that might help. My apologies if it matches
> something that's already been suggested:
>
> #include <iomanip>
> #include <iostream>
> #include <sstream>
> #include <string>
>
> int main()
> {
> std::ostringstream oss;
>
> oss << std::endl;
>
> const char *p = oss.str().data();

This creates a dangling pointer.

>
> std::cout << std::setfill('0') << std::setw(2) << std::hex;
> while (char c = *p++)
> std::cout << int(c);
>
> std::cout << "\n";
>
> return 0;
> }

Yes, this has been proposed and it would not work even if the dangling
pointer bug is fixed. The linefeed translation only appears with an
"external representation" and there is nothing external in a stringstream.

Cheers
Paavo

[Louis Krupp]

On Sun, 18 Feb 2018 10:14:50 +0200, Paavo Helde
<myfir...@osa.pri.ee> wrote:

>On 18.02.2018 6:30, Louis Krupp wrote:
>> Here's yet another program that might help. My apologies if it matches
>> something that's already been suggested:
>>
>> #include <iomanip>
>> #include <iostream>
>> #include <sstream>
>> #include <string>
>>
>> int main()
>> {
>> std::ostringstream oss;
>>
>> oss << std::endl;
>>
>> const char *p = oss.str().data();
>
>This creates a dangling pointer.

OK. It's off-topic, but how would you fix it?

>
>>
>> std::cout << std::setfill('0') << std::setw(2) << std::hex;
>> while (char c = *p++)
>> std::cout << int(c);
>>
>> std::cout << "\n";
>>
>> return 0;
>> }
>
>Yes, this has been proposed and it would not work even if the dangling
>pointer bug is fixed. The linefeed translation only appears with an
>"external representation" and there is nothing external in a stringstream.
>
>Cheers
>Paavo
>

OK. So even if it were correct, it would be useless.

Louis

[Öö Tiib]

On Sunday, 18 February 2018 12:43:43 UTC+2, Louis Krupp wrote:
> On Sun, 18 Feb 2018 10:14:50 +0200, Paavo Helde
> <myfir...@osa.pri.ee> wrote:
>
> >On 18.02.2018 6:30, Louis Krupp wrote:
> >> Here's yet another program that might help. My apologies if it matches
> >> something that's already been suggested:
> >>
> >> #include <iomanip>
> >> #include <iostream>
> >> #include <sstream>
> >> #include <string>
> >>
> >> int main()
> >> {
> >> std::ostringstream oss;
> >>
> >> oss << std::endl;
> >>
> >> const char *p = oss.str().data();
> >
> >This creates a dangling pointer.
>
> OK. It's off-topic, but how would you fix it?

By writing it simply:

std::string s = oss.str();
// ...
for (char c : s) { std::cout << int(c); }

Avoid raw pointers then those can not dangle.

[Christian Gollwitzer]

Am 18.02.18 um 13:04 schrieb Öö Tiib:

It also demonstrates nicely how modern C++ is a much cleaner language.
It could even be

for (char c : oss.str()) { ... }

-> "give me every char from the stream" without worrying about
NULL-terminators and such.

Christian

[James R. Kuyper]

On 02/17/2018 07:54 AM, jak wrote:
...

> encodes the end of the text line. For the latter problem, most of the
> proposed solutions suggest to open a file in text mode, write in a \ n
> ', reopen it in binary mode and read how it has been encoded. Looking
> for alternative solutions, I found someone who thought of using a pipe
> channel instead of a file. this seemed interesting and I proposed it by
> writing a very short piece of code with the intention of presenting the
> idea.
>
> At this point you have questioned me that using the fgets on a file open
> in binary mode is a bad idea.
>
> I replied: I know, sorry, it worked and I left the code unchanged.

The purpose of the code was to determine how line endings are
represented on the current platform. There's only a limited number of
ways that line endings can be represented, where your program would give
the correct result. Those ways are among the most common ways, which is
why it's not implausible that your program worked on the systems where
you tested it. However, did you test it on any system where a line
ending is represented in one of the ways that I said your program would
handle incorrectly? You might not care about such systems, but the OP
said nothing to suggest that he wasn't interested in them, so your code
might not be a proper solution to the OP's question.

> Then you added: the array is not initialized.
>
> I do not care if the variable is used initially to the left of a = or
> passed to a function that intends to write in it. If the function does
> not work or works but writes bad data to my array, then it's a badly
> written function.

You seem to misunderstand the problem with uninitialized data. It has
nothing to do with something going wrong when the data does eventually
get initialized. It has to do with reading data before it has been
initialized, which your code will do when it passes that value to
printf(). The standard says that the elements of such an array have
indeterminate values, and that any evaluation of such objects has
undefined behavior (11.6p12). It lists some exceptions, and one of those
exceptions covers your code. However, I get the distinct impression that
you don't know what those exceptions are, so the fact that your code has
defined behavior is purely an accident.

...

> Add again: there may be text files where the first byte indicates the
> length from the line.

More accurately, I said that there were systems where the standard text
file format uses fixed-sized blocks, with each line starting a new
block, and there's a count at the start of each block indicating how
many bytes of that block are used; a count smaller than the block size
occurs only in the block containing the end of the line.

> First of all it should not be called a text file because this should be
> readable by any text editor / viewer and would not be the case with this

> file. ...

Text editors targeted for such systems will be written to understand
files in the format described. On some such systems that is the only
format allowed for text files - a text editor targeting such a system
would NOT work properly when viewing files you would be willing to call
"text files". What you would be willing to call "text files" would need
to be converted when ported to such systems, to use the correct text
format for those systems.

> Honestly I do not understand if you answer me for your pleasure to
> contradict me or if you are trying to explain something that, in your

> opinion, I do not know or do not understand. ...

The latter - and this message tends to confirm that opinion.