"In progress" saving of data collected using Reap/Sow
giacom...@gmail.com
I'm quite new to Mathematica, so please be patient if I ask something
trivial.
I wrote a little simulation program that need to iterate a calculation
thousand of times, storing the result of each iteration. At first, I
used the "Append" function to store data in a list at every iteration,
but I noticed that this operation become slower and slower as the list
increases in size, so that appending a single data at the end of a
quite big list takes a lot of time.
So I migrated to the reap/sow functions that are written exactly for
this purpose, ad gives much better performances. My code now is
something like:
results = Reap[Do[a lot of iterations with almost a call to Sow in
each]]];
The problem now is that, since the collected data are returned by the
Reap function only whern it terminates, I have no way to access that
data if the calculation is in progress, ie if the Do loop is not
completed. This way, I can't check the status of the calculation nor
provide backup save of data during the calculation itself, that can
last for days...
As you can imagine, using the Append function both this (and others)
tasks where easily accomplishable, since in every moment I had a list
whith all the results obtained so far...
Any idea on how I can solve the problem?
Thanks a lot
Giacomo
Chris Chiasson
status line?
I found this code in an old thread:
ShowStatus[status_] := If[
Head[$FrontEnd]===FrontEndObject,
LinkWrite[
$ParentLink,
SetNotebookStatusLine[FrontEnd`EvaluationNotebook[],
ToString[status]]
]
,
Print[status];
];
bghi...@ucdavis.edu
I am not familiar with your actual code, but if you could break up your
problem into a set of subproblems that can be restarted then you could
write blocks of your data to file. Here is how one could go about
doing it:
First, open up a file to write to:
strm = OpenWrite["/Users/brian/Desktop/temps.dat"]
Now suppose my problem was to add a random number to the previous
element in a list and do it 300 times. The Reap/Sow strategy saves the
intremediate results in a list
a=0;Reap[Do[a = a + Random[];Sow[a], {i, 1, 300}]]
Let us suppose we want to back-up intermediate results to a file after
every 100 iterations Again we implement a reap/sow strategy but now
break up the problem into 3 sub blocks of 100 iterations nad place the
Reap/Sow stategy inside the ionner loop
a = 0; Do[Reap[Do[a = a + Random[]; Sow[a, strm], {i, 1, 100}], _,
Write], {3}]
The key idea is to used the channel name "strm" as a tag for Sow, and
then reference this tag in Reap, as shown above. In this way after
every 100 interations of the inner Do loop, data is written to my file.
Hope this gives you some ideas- this is all I could think of. Perhaps
others on the mathgroup have a better idea.
Cheers,
Brian
Ray Koopman
In[1]:= n = 1*^4
Timing[y = {}; Do[AppendTo[y,i],{i,n}];]
Timing[z = {}; Do[z = {z,i},{i,n}];]
i = Random[Integer,{1,n}]
y[[i]]
Last@Nest[First,z,n-i]
Out[1]= 10000
Out[2]= {1.86 Second,Null}
Out[3]= {0.02 Second,Null}
Out[4]= 9353
Out[5]= 9353
Out[6]= 9353
Giacomo Ciani
of collecting in a variable data generated in a loop; to do this, using
Reap/Saw is a good solution, except that there is no direct way of
saving data till all the iteration are finished.
Anyway I already found convenient solution fot that. In addition, your
suggestion solves onether problem of mines, that is report the
calculation progress quite often without filling the notebook's output
whit a lot of lines like:
0.1 % done
0.2 % done
0.3 % done
...
Thank a lot!
Giacomo
Giacomo Ciani
in the list?
Very interesting approach... there is a limit on the number of levels
of a list? And what about the amount of memory needed to store the data
with such an approach?
Thaks
Giacomo
Ray Koopman
a simple list of all the past results. The first way is fastest, but
doesn't work if each result itself is a list.
In[1]:= n = 8
y = {}; Do[AppendTo[y,result[i]],{i,n}]; y
z = {}; Do[z = {z,result[i]},{i,n}]; z
Flatten[z] === y
Nest[FlattenAt[#,1]&,z,n] === y
Table[Last@Nest[First,z,n-i],{i,n}] === y
Out[1]= 8
Out[2]= {result[1],result[2],result[3],result[4],
result[5],result[6],result[7],result[8]}
Out[3]= {{{{{{{{{},result[1]},result[2]},result[3]},result[4]},
result[5]},result[6]},result[7]},result[8]}
Out[4]= True
Out[5]= True
Out[6]= True
In[7]:= n = 8
y = {}; Do[AppendTo[y,{i,-i}],{i,n}]; y
z = {}; Do[z = {z,{i,-i}},{i,n}]; z
Flatten[z] === y
Nest[FlattenAt[#,1]&,z,n] === y
Table[Last@Nest[First,z,n-i],{i,n}] === y
Out[7]= 8
Out[8]= {{1,-1},{2,-2},{3,-3},{4,-4},
{5,-5},{6,-6},{7,-7},{8,-8}}
Out[9]= {{{{{{{{{},{1,-1}},{2,-2}},{3,-3}},{4,-4}},
{5,-5}},{6,-6}},{7,-7}},{8,-8}}
Out[10]= False
Out[11]= True
Out[12]= True
Giacomo Ciani
wondering if there was a more direct way of collecting intermediate
data from Reap function. Anyway, I thik I will implemet your
suggestion.
Only one more question: I can't understand exactly why you introduced
the "strm" tag to collect data...
Thanks a lot. Giacomo
Ray Koopman
> Are you suggesting me just to store every new result as a new
> "level" in the list?
Yes. See the end of section 2.5.4 in version 5 (or 2.4.4 in v.3)
of The Mathematica Book.
Bill Rowe
>Expanding on my previous post, here are three ways to unravel z into
>a simple list of all the past results. The first way is fastest, but
>doesn't work if each result itself is a list.
You had suggested two methods for dealing with things when the result is a list, i.e.,
> Nest[FlattenAt[#,1]&,z,n] === y
> Table[Last@Nest[First,z,n-i],{i,n}] === y
If the result is a list with known constant length then it is much faster to do
Partition[Flatten@z, len] where len is the known constant len. For example,
In[27]:=
n = 10^3;
z = {}; Timing[Do[z = {z, {i, -i}}, {i, n}]; ]
Timing[a=Nest[FlattenAt[#1, 1] & , z, n]; ]
Timing[Table[Last[Nest[First, z, n - i]], {i, n}]; ]
Timing[b=Partition[Flatten[z], 2]; ]
a==b
Out[28]=
{0.003924 Second,Null}
Out[29]=
{0.062477 Second,Null}
Out[30]=
{0.387071 Second,Null}
Out[31]=
{0.001232 Second,Null}
Out[32]=
True
--
To reply via email subtract one hundred and four
Maxim
wrote:
Two heads are better than one:
In[1]:= z = h[]; Do[z = h[z, {i, -i}], {i, 10^5}] // Timing
Out[1]= {0.188*Second, Null}
In[2]:= (z = List @@ Flatten[z, Infinity, h];) // Timing
Out[2]= {0.062*Second, Null}
Maxim Rytin
m...@inbox.ru