Tracking progress in ParallelDo

[Brian Beckage]

I'm applying a function repeatedly using ParallelDo. I would like to keep track of its progress. I've tried using both my own counter variable as well as using the increments within the ParallelDo function (below). Neither provides a way of sequentially tracking how many of the iterations have been spawned (not completed--but just kicked off).

How could I track that iteration 1, 2, 3, 4, 5 etc. have been started? Is there a better way of monitoring progress of the ParalleDo?

Thanks for your help!

Brian


USING MY OWN COUNTER: myCounter:

parallelBootstrapFireMultiplier[nBoot_, fireMultiplierRange_, parmList1_,
nYears_, initLandscapeProbs_] := Module[{outIter, myCounter},

ParallelEvaluate[Clear[myCounter]];
myCounter = 0;
SetSharedVariable[myCounter];
Reap[
ParallelDo[
myCounter++;
Print["Starting parallelBootstrapMultiplier step ", myCounter, " out of ",
nBoot];
outIter =
iterFireMultiplier[fireMultiplierRange, parmList1, nYears,
initLandscapeProbs];
ParallelSow[outIter],
{nBoot}]
]
]

RESULTS in:

(kernel 2) Starting parallelBootstrapMultiplier step 2 out of 6
(kernel 1) Starting parallelBootstrapMultiplier step 2 out of 6
(kernel 1) Starting parallelBootstrapMultiplier step 4 out of 6
(kernel 2) Starting parallelBootstrapMultiplier step 4 out of 6
(kernel 2) Starting parallelBootstrapMultiplier step 5 out of 6
(kernel 1) Starting parallelBootstrapMultiplier step 6 out of 6

USING THE INCREMENT i FROM ParallelDo:

parallelBootstrapFireMultiplier[nBoot_,fireMultiplierRange_,parmList1_,nYears_,initLandscapeProbs_]:= Module[{outIter},
Reap[
ParallelDo[
Print["Starting parallelBootstrapMultiplier step ",i, " out of ",nBoot];
outIter=iterFireMultiplier[fireMultiplierRange,parmList1,nYears,initLandscapeProbs]; (*This returns a list*)
ParallelSow[outIter],
{i,nBoot}]
]
]

RESULTS in:
Starting parallelBootstrapMultiplier step 1 out of 6
Starting parallelBootstrapMultiplier step 4 out of 6
Starting parallelBootstrapMultiplier step 5 out of 6
Starting parallelBootstrapMultiplier step 2 out of 6
Starting parallelBootstrapMultiplier step 3 out of 6
Starting parallelBootstrapMultiplier step 6 out of 6

[Szabolcs Horvát]

There was a very similar question asked on StackExchange. You might be
interested in the answers there.

http://mathematica.stackexchange.com/questions/1548/monitor-doesnt-work-with-paralleltable

Here's a slightly modified version of Leonid's solution:

SetSharedVariable[progress]

progress = 0;
Monitor[
ParallelTable[progress++; Pause[0.5]; i, {i, 100}],
progress
]

The caveat here is that SetSharedVariable causes 'progress' to be always
evaluated on the main kernel. Accessing the main kernel is expensive
(it takes a long time). So this only makes sense if:

- it is reasonable to break the parallel evaluation into the smallest
possible pieces (equivalent of Method -> "FinestGrained")

- each single evaluation in your ParallelDo takes considerably longer
than the overhead due to the callback to the main kernel

Otherwise doing this will kill performance. Personally I do not
recommend doing this unless you are experienced with parallelization in
Mathematica and understand what factors affect performance.

Do check my link for more complex but also more efficient variations on
the same idea (Eli Lansey's answer).

Szabolcs