istream replace

[Christopher Pisz]

I need to replace any occurance of "\r\n" occurring in an istream with
just "\n".

I could so this by copying the stream contents to a string, then
replacing the occurances within the string, and creating another stream
object from that string.

Is there a more efficient way?

[Victor Bazarov]

First you need to ask yourself, what is inefficient about it?
Second, why bother with creating another stream when you can simply
incorporate that action into your stream parser?

V
--
I do not respond to top-posted replies, please don't ask

[Christopher Pisz]

On 12/3/2014 2:34 PM, Victor Bazarov wrote:
> On 12/3/2014 3:26 PM, Christopher Pisz wrote:
>> I need to replace any occurance of "\r\n" occurring in an istream with
>> just "\n".
>>
>> I could so this by copying the stream contents to a string, then
>> replacing the occurances within the string, and creating another stream
>> object from that string.
>>
>> Is there a more efficient way?
>
> First you need to ask yourself, what is inefficient about it?
> Second, why bother with creating another stream when you can simply
> incorporate that action into your stream parser?
>
> V

I'd figure the copies were inefficient. Although I know creating a
stringstream from a const string the docs say it just moves the
stringbuf pointer, I don't think the reverse works the same way. I could
be wrong.

But then, I don't even see how to create a string from an istream unless
it was a stringstream and I cast it, but I can't be sure what kind of
stream I am receiving as a parameter.

My parser orginally just worked with string, but then I made it work
with istream so that I could work with filestream and stringstream both
for on disk or in memory text. I did this just by fudging someone else's
3000 line unreadable chunk of poopy where they had taken a string and
created a stream and then used getline on it. It passed testing there,
so I didn't bother messing with it.

Now I come across the windows newline scenario in the data and have to
fix that.

[Christopher Pisz]

On 12/3/2014 2:34 PM, Victor Bazarov wrote:

> On 12/3/2014 3:26 PM, Christopher Pisz wrote:
>> I need to replace any occurance of "\r\n" occurring in an istream with
>> just "\n".
>>
>> I could so this by copying the stream contents to a string, then
>> replacing the occurances within the string, and creating another stream
>> object from that string.
>>
>> Is there a more efficient way?
>
> First you need to ask yourself, what is inefficient about it?
> Second, why bother with creating another stream when you can simply
> incorporate that action into your stream parser?
>
> V

Seems very inefficient to be copying string to stream and back again.
Here is a compilable example of what I have:

// Standard Includes
#include <iostream>
#include <sstream>
#include <vector>

// Existing function to replace occurances of a string within a string
std::string ReplaceAllOccurrences(const std::string & original, const
std::string & search, const std::string & replacement)
{
std::string temp = original;
size_t position = 0;

while((position = temp.find(search, position)) != std::string::npos)
{
temp.replace(position, search.length(), replacement);
position += replacement.length();
}

return temp;
}

// Function I am working on
void Test(std::istream & stream)
{
// Verify the stream is good
if(!stream)
{
// Error - stream was given in an error state
throw std::exception("Stream to be parsed was given with an
error state set");
}

// Replace all occurances of "\r\n" with "\n" so the newline can be
handled the same way
// TODO - This can't be the most efficient way
stream.seekg (0, stream.end);
const unsigned length = stream.tellg();
stream.seekg (0, stream.beg);

std::string temp('\0', length);
temp = ReplaceAllOccurrences(temp, "\r\n", "\n");
std::istringstream formattedStream(temp);

// Snip the actual work
}

// Test
int main()
{
std::istringstream testData("Hello\r\nI am a Windows
string\r\nBecause I like carriage returns\r\n");
Test(testData);
}

[Christopher Pisz]

On 12/3/2014 3:11 PM, Christopher Pisz wrote:
> On 12/3/2014 2:34 PM, Victor Bazarov wrote:
>> On 12/3/2014 3:26 PM, Christopher Pisz wrote:
>>> I need to replace any occurance of "\r\n" occurring in an istream with
>>> just "\n".
>>>
>>> I could so this by copying the stream contents to a string, then
>>> replacing the occurances within the string, and creating another stream
>>> object from that string.
>>>
>>> Is there a more efficient way?
>>
>> First you need to ask yourself, what is inefficient about it?
>> Second, why bother with creating another stream when you can simply
>> incorporate that action into your stream parser?
>>
>> V
>
> Seems very inefficient to be copying string to stream and back again.
> Here is a compilable example of what I have:
>

Whoops, left a few mistakes in the former listing.
Corrected code:

std::string temp(length, '\0');
stream.read(&temp[0], length);

[Geoff]

I was under the impression that \r\n only appears in disk files and
the system automatically converts them to \n in memory as the file is
read in text mode. Do streams behave differently?

Why not just do the replacement in the stream as you read it in?

Many moons ago I wrote a small and dirty utility in C for Windows that
converted \n to \r\n on files and I had to use fopen(inpath, "rb") to
accomplish it on Windows. I never wrote an equivalent \r\n to \n tool
and I never tried doing it in C++.

[Victor Bazarov]

OK, this is how I do it when I need to (and that's not often), use it if
you think it suits you, or maybe it'll give you an idea of your own.

I would extract the lines from the stream using std::getline and \n as
the delimiter. Once the line (a string) is in my possession, I check
whether it ends with \r\n (I don't remember if the delimiter is also put
in the string or not, if not then check for \r only), if so I simply
change the ending to remove \r and proceed with extracting the rest of
information from that string.

If you need multiple strings, then the next tool in the pipeline should
be the concatenator. If not, the parser.

Since you have the somebody's piece of code that already uses 'getline',
all you need to add is a small function to do the \r clean-up in each
line you get. Call your function right after 'getline' returns a string.

If you need more information, do ask.

[Jorgen Grahn]

On Wed, 2014-12-03, Victor Bazarov wrote:
> On 12/3/2014 3:42 PM, Christopher Pisz wrote:
>> On 12/3/2014 2:34 PM, Victor Bazarov wrote:
>>> On 12/3/2014 3:26 PM, Christopher Pisz wrote:
>>>> I need to replace any occurance of "\r\n" occurring in an istream with
>>>> just "\n".
>>>>
>>>> I could so this by copying the stream contents to a string, then
>>>> replacing the occurances within the string, and creating another stream
>>>> object from that string.
>>>>
>>>> Is there a more efficient way?
>>>
>>> First you need to ask yourself, what is inefficient about it?
>>> Second, why bother with creating another stream when you can simply
>>> incorporate that action into your stream parser?

...

>> My parser orginally just worked with string, but then I made it work
>> with istream so that I could work with filestream and stringstream both
>> for on disk or in memory text. I did this just by fudging someone else's
>> 3000 line unreadable chunk of poopy where they had taken a string and
>> created a stream and then used getline on it. It passed testing there,
>> so I didn't bother messing with it.
>>
>> Now I come across the windows newline scenario in the data and have to
>> fix that.
>
> OK, this is how I do it when I need to (and that's not often), use it if
> you think it suits you, or maybe it'll give you an idea of your own.
>
> I would extract the lines from the stream using std::getline and \n as
> the delimiter. Once the line (a string) is in my possession, I check
> whether it ends with \r\n (I don't remember if the delimiter is also put
> in the string or not, if not then check for \r only), if so I simply
> change the ending to remove \r and proceed with extracting the rest of
> information from that string.
>
> If you need multiple strings, then the next tool in the pipeline should
> be the concatenator. If not, the parser.
>
> Since you have the somebody's piece of code that already uses 'getline',
> all you need to add is a small function to do the \r clean-up in each
> line you get. Call your function right after 'getline' returns a string.

Seconded. It's not technically elegant, but it's a simple,
maintainable solution -- at least for people like me who prefer to
read streams line by line.

On the other hand, writing an istream which acts as a filter stacked
on top of another istream can be an interesting exercise. Doesn't
Boost have that kind of stuff?

/Jorgen

--
// Jorgen Grahn <grahn@ Oo o. . .
\X/ snipabacken.se> O o .

[Jorgen Grahn]

On Wed, 2014-12-03, Geoff wrote:
> On Wed, 03 Dec 2014 14:26:48 -0600, Christopher Pisz
> <nos...@notanaddress.com> wrote:
>
>>I need to replace any occurance of "\r\n" occurring in an istream with
>>just "\n".
>>
>>I could so this by copying the stream contents to a string, then
>>replacing the occurances within the string, and creating another stream
>>object from that string.
>>
>>Is there a more efficient way?
>
> I was under the impression that \r\n only appears in disk files and
> the system automatically converts them to \n in memory as the file is
> read in text mode. Do streams behave differently?

That's the typical behavior on some systems (MS-DOS etc) but e.g. on
Unix there's no such mechanism ... and sometimes you get DOS text
files on Unix, and someone expects you to handle them sensibly anyway.

[Ian Collins]

Write your own streambuf and do the filtering in its underflow() member?

--
Ian Collins

[Geoff]

On Wed, 03 Dec 2014 16:43:57 -0500, Victor Bazarov
<v.ba...@comcast.invalid> wrote:

>I would extract the lines from the stream using std::getline and \n as
>the delimiter. Once the line (a string) is in my possession, I check
>whether it ends with \r\n (I don't remember if the delimiter is also put
>in the string or not, if not then check for \r only), if so I simply
>change the ending to remove \r and proceed with extracting the rest of
>information from that string.

std::getline strips the delimiter by default. Reading a Windows \r\n
from a file stream yields a string that has no carriage return or
newline in it.

std::getline(input, str, '\n') and std::getline(input, str) behave
identically so there is really no need to explicitly delimit with
newline unless you prefer it for style/maintenance purposes.

[Geoff]

On 3 Dec 2014 22:00:19 GMT, Jorgen Grahn <grahn...@snipabacken.se>
wrote:

I see. I am often the recipient of the converse on Windows, converting
Unix eol's to DOS. :) Somehow I got the mis-impression that
Christopher was running his test on Windows.

[Luca Risolia]

> main.cpp: In function 'void Test(std::istream&)':

g++ -std=c++14 -O2 -Wall -pedantic -pthread main.cpp && ./a.out

main.cpp:28:85: error: no matching function for call to
'std::exception::exception(const char [54])'

[Luca Risolia]

Il 03/12/2014 22:18, Christopher Pisz ha scritto:

> Whoops, left a few mistakes in the former listing.
> Corrected code:

g++ -std=c++14 -O2 -Wall -pedantic -pthread main.cpp && ./a.out

main.cpp:28:85: error: no matching function for call to
'std::exception::exception(const char [54])'

[Ian Collins]

That probably should be std::runtime_error.

--
Ian Collins

[Luca Risolia]

Il 03/12/2014 23:48, Ian Collins ha scritto:
> That probably should be std::runtime_error.

Not only that...
I wonder why people don't try to compile their code before posting.

[Christopher Pisz]

They do.
They just don't use your compiler.

[Christopher Pisz]

On 12/3/2014 3:42 PM, Geoff wrote:
> On Wed, 03 Dec 2014 14:26:48 -0600, Christopher Pisz
> <nos...@notanaddress.com> wrote:
>
>> I need to replace any occurance of "\r\n" occurring in an istream with
>> just "\n".
>>
>> I could so this by copying the stream contents to a string, then
>> replacing the occurances within the string, and creating another stream
>> object from that string.
>>
>> Is there a more efficient way?
>
> I was under the impression that \r\n only appears in disk files and
> the system automatically converts them to \n in memory as the file is
> read in text mode. Do streams behave differently?
>
> Why not just do the replacement in the stream as you read it in?

I don't read it in. Someone else somewhere else does. I suppose we could
call it a bug in their code, but you know how it goes. Fix this crap
quick, deadline deadline. All I know is I have an istream from somewhere
that may or may not contain \r\n which the parser after my code cannot
handle.

[Christopher Pisz]

Good stuff Victor. You are the man.

[Geoff]

On Wed, 03 Dec 2014 17:47:48 -0600, Christopher Pisz

This seems to work:
(Mind the line wraps.)

// Standard Includes
#include <iostream>
#include <sstream>
#include <vector>

// Function I am working on
void Test(std::istream & stream)
{
// Verify the stream is good
if(!stream)
{
// Error - stream was given in an error state

throw std::runtime_error("Stream to be parsed was given with

an error state set");
}

std::string stro = "\r"; // old string to find
std::string strn = "\n"; // new string to replace it
std::string str;

while (std::getline(stream, str, '\n'))
{
std::string::iterator iter = str.begin();
while (iter != str.end())
{
std::string::size_type pos = str.find(stro);
if(pos != str.npos) // found a match
{
str.replace(pos, strn.length(), strn);
}
iter++;
}
std::istringstream formattedStream(str);

[Luca Risolia]

To tell you the truth, there is no need of any compiler to see that your
code may not compile, as it's evident you did not include the necessary
header for std::exception. Not to mention the fact that
std::exception(const char*) is not standard.

All the above tells me you are using the VS C++ compiler.

For the above reasons, I suggest that you change your compiler and try
to be less arrogant next time.

[Christopher Pisz]

On 12/3/2014 7:31 PM, Luca Risolia wrote:
> Il 04/12/2014 00:45, Christopher Pisz ha scritto:
>> On 12/3/2014 4:56 PM, Luca Risolia wrote:
>>> Il 03/12/2014 23:48, Ian Collins ha scritto:
>>>> That probably should be std::runtime_error.
>>>
>>> Not only that...
>>> I wonder why people don't try to compile their code before posting.
>>>
>>
>>
>> They do.
>> They just don't use your compiler.
>
> To tell you the truth, there is no need of any compiler to see that your
> code may not compile, as it's evident you did not include the necessary
> header for std::exception. Not to mention the fact that
> std::exception(const char*) is not standard.
>
> All the above tells me you are using the VS C++ compiler.

How intuitive!

> For the above reasons, I suggest that you change your compiler and try
> to be less arrogant next time.
>

I suppose I should install Linux and tell all my customers to go to hell
too.

I am sorry if it took you more than 0.5 seconds to figure out that
#include<exception> was omitted, which somehow stopped you from
understanding the code at all, leaving it impossible for you to follow
the post.

I am also wearing mismatching socks today.

Arrogance begets arrogance.

The code was compiled and ran. I am so sorry that I did not have the
foreknowledge that msvc didn't require me to include the header and that
it ruined your week.

[Richard]

[Please do not mail me a copy of your followup]

Luca Risolia <luca.r...@linux-projects.org> spake the secret code
<547FB95E...@linux-projects.org> thusly:

>To tell you the truth, there is no need of any compiler to see that your
>code may not compile, as it's evident you did not include the necessary
>header for std::exception.

Many C++ programmers make this mistake of getting the appropriate
header by accident instead of by design. This can happen with any
compiler because standard library headers include other standard
library headers by virtue of their implementation. You may get the
necessary header by accident and no compile error ensues. Every
implementation of the standard library has this going on.

That's why a team at Google created a tool called "include what you
use" to alert you that you are using identifiers that aren't defined
in headers that you explicitly included.
<https://code.google.com/p/include-what-you-use/>

I recommend using that regularly on your code base, no matter what
compiler you are using.

>Not to mention the fact that
>std::exception(const char*) is not standard.

Yeah, this has bugged me for a while[*] that the standard library that
ships with VS lets you make this mistake and you don't find out its
non-portable until you try to compile your code somewhere else or with
another implementation of the standard library.

I also recommend building your C++ code on multiple platforms with
multiple compilers in order to avoid this mistake. The reverse is
also true that it is too easy to write some non-portable, non-standard
code that happens to compile on gcc. Compiling your code on multiple
platforms (even if you never intend to ship on them), will keep you
honest regardless of which platform is your preferred platform.

[*] Since the late 1990s!
--
"The Direct3D Graphics Pipeline" free book <http://tinyurl.com/d3d-pipeline>
The Computer Graphics Museum <http://computergraphicsmuseum.org>
The Terminals Wiki <http://terminals.classiccmp.org>
Legalize Adulthood! (my blog) <http://legalizeadulthood.wordpress.com>

[Geoff]

On Thu, 04 Dec 2014 10:42:59 -0600, Christopher Pisz
<nos...@notanaddress.com> wrote:

[snip]

>#include<exception> was omitted [...]

[snip]

<exception> is a Microsoft header. <stdexcept> is the standard header.

[Christopher Pisz]

Fair enough. I shall enter that into the noggin. MS sure does like to
make me memorize nuances. Thanks.

[red floyd]

No it isn't MS only. It's part of the Standard.

See ISO/IEC 14882:2003 18.6 [lib.support.exception]

[Öö Tiib]

Some standard C++ (or some other standard like POSIX) headers, functions,
types or classes being different and behaving slightly differently is
typical everyday case for anyone who is using two different versions of
*same* compiler for *same* platform.

On the other hand people post here code that compiles nowhere and even if
taken as pseudo-code has serious errors in it.

So conjuring a shit-storm out of such a little trivial difference is
perhaps going too far?

[Luca Risolia]

Öö Tiib wrote:

> Some standard C++ (or some other standard like POSIX) headers, functions,
> types or classes being different and behaving slightly differently is
> typical everyday case for anyone who is using two different versions of
> *same* compiler for *same* platform.
>
> On the other hand people post here code that compiles nowhere and even if
> taken as pseudo-code has serious errors in it.

That's why online compilers come in handy these days to share the same code,
compiler and preferred platform.