Reworking of Accumulators and Aggregators

[Markus Holzemer]

Hi!
I'm currently working on an unification or extension of Accumulators and Aggregators.
Both have several advantages and disadvantages, that's why it would be cool to combine them to get the best of both. I am not sure what is the best approach to take and would like to start a discussion on this.

Aggregators are used inside of iterations mainly to serve as the termination criterion. After each superstep there is a barrier, and when all workers are finished the Aggregators are passed to the start node.
The current implementation is absolutely bound to iterations, so I think it is not possible to make aggregators usable for tasks outside of iterations.

The big disadvantage of Accumulators is that the result is only accessible after a job has finished. That is the main reasons why they currently cannot serve as a termination criterion in iterations.
Accumulators are implemented quite simple: Every worker sends RPC calls with its updates to the master node where the aggregate is built and saved. These calls are asynchron, which is fast but brings also a few problems.

I see several alternative ways to proceed:
1) Make the result of accumulators available at  each superstep of an iteration on a best effort way. This would be quite easy to implement: At the start of a new superstep the current result of the Accumulators would be fetched and made available.
The problem is, that it is very likely that not all updates of the last superstep have arrived at the master yet. That could lead to some big problems when formulating a termination criterion.

2) Handle Accumulators the same way as currently Aggregators are handled when inside an iteration.
The problem here is, that in that approach still a barrier would be needed before the iteration starts, in case of an Accumulator that is needed both inside and ouside of an iteration. Currently it is not possible to tell if or when all RPC calls have been transmitted.
This approach probably would also be problematic for the future development of iterations (iterative plan rollout...)

3) Make somehow sure that all RPC calls are transmitted and processed at the start of each superstep. That would probably be the best solution but I don't know how to implement this. Probably a bunch of changes on the current RPC implementation would be necessary.


So what do you think? Is a best effort approach good enough? If not, what would be a good way for a different implementation?

cheers,
Markus

[Vasia Kalavri]

Hi,

I'm not exactly sure if I understood correctly what you would like to do, but here are some thoughts:

In my opinion, the aggregators work quite well with the current model of the iterations. The barrier is already there at the end of the superstep, so I don't see a big overhead there.

If at some point we support asynchronous iterations, then we might want to reconsider.

I see how the "best effort" approach could be helpful in some cases, for example, when a termination criterion depends on a threshold, you might not want to wait for all the updates. However, this might also cost you one (or more) extra iteration(s). 

I believe that if you go for this approach, it would be nice to have both choices, i.e. something like a "blocking" and a "non-blocking / best-effort" implementation and let the user choose, depending on application requirements.

The same could apply for non-iterative programs and accumulators. If you want to retrieve the partial result of an accumulator, before job completion, then you could have a "best-effort" call, that will return immediately the current aggregated value or provide a "blocking" call, that would introduce a barrier there.

What do you think?

Cheers,

V.

[Stephan Ewen]

Hi!

Thanks for the good comments!

Here are some thoughts from my side:

  - In theory, if we could make the Aggregators available at the end of the Job and make them usable outside iterations, we would basically have what we need. We could adjust their interface a bit - I believe Vasia already started switching from classes to objects.

  - I would like it very much if not only the end result of the aggregators is available on the job manager, but also the intermediate results after each superstep. That way we could plot them in the Web GUI for people to track convergence.

  - If the aggregators go to the job manager anyways, we could just as well do the suberstep synchronization via the jobmanager RPC calls, rather than through data channel events.

  - For asynchronous iterations (once we get to those), we would not have the intermediate results, since there are no supersteps. I think that is fair.

  - We could think about sending the aggregators/accumulators periodically (with heart beats) to the job manager. That would give you also a progress indicator for jobs.

My suggestion to proceed would be the following:

  - We merge and extend Vasia's rework of the aggregators towards using objects. That makes the aggregators similar to the accumulators.

  - Let's rework the iteration superstep barrier to go through a jobmanager RPC call. In the call that notifies the job manager of a worker's end-of-superstep, we would transmit the aggregators. In the call that kicks of the next superstep (after all workers have reported end of superstep), we send back the combined aggregators

  - The final step is to see how to best initialize the aggregators lazily (like the accumulators)

  - The periodic sending of the aggregators comes later ;-)

Greetings,

Stephan

[Markus Holzemer]

Hi!
Thanks Stephan and Vasia for your extensive feedback! It helped me a lot and I think I have now a quite good idea of how this can be implemented.
I will at first follow Stephans suggestion and change the superstep barrier to RPC calls instead of channel events. Then the aggregators are synced at the JobManager and can easily be displayed in the webinterface.

regards,
Markus