Problem with printing to a file

[megaziomek]

Hi,
I wrote a program in which the main part is a loop with a big number
of iterations (the program takes about 1 day to perform the
calculations i want) that imitates a flow of time ("professionally
speaking": time evolution). After each iteration of the loop I send
some data to files. To make the long story short, u can simply imagine
a two columns file with time values (proportional to the the loop
iterator) in the 1st column and some number/quantity (say, energy) in
the 2nd column.

So, at the very beginning of the code I open the file:

ofstream EnergyStream;
EnergyStream.open(OutpuStreamName(Results_Location,"EnergyValues.dat"),
ios::app);

And i run the loop:

for(int t=0; t<=Nt; t++){
/*Here i calculate a quantity t_Energy and i print it to the file; dt
and RealTimeUnit are some inputs:*/
EnergyStream<<setw(15)<<fixed<<setprecision(9)<<dt*t*RealTimeUnit<<"\t"<<fixed<<setprecision(9)<<t_Energy<<endl;

}

And close the file:
EnergyStream.close();

However by following this method I came across some problem. When I
open my file EnergyValues.dat while the program is running (same
phenomenon when i open the file only after it'll finish to run) what I
see are the two columns (as I expected) say for abt 5000-6000 lines
but then, out of nowhere, i see one very long line (takes abt 1/5 of
my emacs window when maximized to fullscreen) of "^@" signs, so I see
smth like:
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
Then, again several thousands of lines of the correct results and
again this long line of "^@". I had a closer look to that but there's
no periodicity at all in that. Different occurrences of "the" line
have different lengths, also the "bad lines" are separated by
different number of lines with the actual results.

I dont' have to tell u that all plotting programs are going crazy when
u load such a file as an input.

If, on contrary, I open, write and close the file separately for each
and single iteration of the loop, this problem does not occur and
everything is as it should be! Unfortunately, I expect that this
approach is way more time consuming (if u imagine opening, writing and
closing Nt=10^8 times). So, I would definitely go for the first
solution (i.e. opening only once at the beginning, closing once at the
very end) if only I could avoid this "^@" stuff.

I would be very grateful for any suggestions that would help me to
solve this problem.
Thank u in advance.
--
regards,
okurczeblade

[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]

[Francis Glassborow]

Well it looks as if your program involves some form of undefined
behaviour that is only manifesting once in a while. The randomness you
report (I wonder if there is any relationship between the number of
repeats of ^@ and the number of correctly stored lines, and what about
the time intervals between the last correct output before failure and
the next correct output after failure.

I think that keeping a file open for an entire day is taking unnecessary
risks with the system. However if you cannot open and close for each
entry consider closing and reopening after a block (say of 1000 data lines)

I have njo idea how many data items you have in a complete run, but to
avoid the risk of losing all your data in a system crash I would
seriously consider using a new file for each block of data (say of 1000
items) and concatenating them at the end of the complete run.

I know this does not fix the programming problem but in the real world
getting something that works reliably is often more important and you
can then fix the problem at your leisure.

If the rogue lines are not massive failures to write data, you can also
consider writing a short program to filter the file after the completion
of a run and then your other programs will not have problems with the data.

Part of the art of programming in the real world is to learn how to
side-step problems until you have the time and/or insight to fix them.

--

[ThosRTanner]

On Aug 12, 7:56 am, megaziomek <okurczebl...@gmail.com> wrote:
> Hi,
> I wrote a program in which the main part is a loop with a big number
> of iterations (the program takes about 1 day to perform the
> calculations i want) that imitates a flow of time ("professionally
> speaking": time evolution). After each iteration of the loop I send
> some data to files. To make the long story short, u can simply imagine
> a two columns file with time values (proportional to the the loop
> iterator) in the 1st column and some number/quantity (say, energy) in
> the 2nd column.
>

[snip]

> However by following this method I came across some problem. When I
> open my file EnergyValues.dat while the program is running (same
> phenomenon when i open the file only after it'll finish to run) what I
> see are the two columns (as I expected) say for abt 5000-6000 lines
> but then, out of nowhere, i see one very long line (takes abt 1/5 of
> my emacs window when maximized to fullscreen) of "^@" signs, so I see
> smth like:

[snip]

> Then, again several thousands of lines of the correct results and
> again this long line of "^@". I had a closer look to that but there's
> no periodicity at all in that. Different occurrences of "the" line
> have different lengths, also the "bad lines" are separated by
> different number of lines with the actual results.

I'd reckon the ^@ are actually the editor displaying nulls in the only
way it can.

>
> I dont' have to tell u that all plotting programs are going crazy when
> u load such a file as an input.
>
> If, on contrary, I open, write and close the file separately for each
> and single iteration of the loop, this problem does not occur and
> everything is as it should be! Unfortunately, I expect that this
> approach is way more time consuming (if u imagine opening, writing and
> closing Nt=10^8 times). So, I would definitely go for the first
> solution (i.e. opening only once at the beginning, closing once at the
> very end) if only I could avoid this "^@" stuff.
>
> I would be very grateful for any suggestions that would help me to
> solve this problem.

A few questions you might wish to consider:
1) Do you see the issue if you let the program run to completion and
then open the file
2) Does your generating program fork to do the work - I've seen the
same sort of thing when running 2 programs which both have stdout
redirected to the same file - text from one, then nulls, then text
from the other
3) Do you manipulate the file pointer anywhere in your program (lseek)

If the answer to all 3 of the above is no, then the issue is probably
something to do with the editor trying to read a file when someone
else is writing to it.

If you *need* to start processing the output from the file before it's
fully generated of course, you'd probably do better to product
multiple output files.

--