design proposal: celery and redis

[Axel Hecht]

Hi,

here's some thoughts on a possible replacement for twisted/buildbot with
celery and redis:

Everything stays in python.

Twisted Poller:
That one mostly stays, but instead of calling handlePushes, kick off a
celery task. That allows for more simultaneous connections, and retries
of failed jobs. I'd sacrifice the id <-> push_date ordering, possibly.

ChangeSource: does the handlePushes task, triggers scheduler task.

Scheduler:
The buildbot scheduler would be replaced with a celery task. It'd spawn
tasks per tree and locale. The revisions to process would be stored in
redis. The latter has the benefit that we can continue to collapse builds.

Buildmaster: X

Buildslave: celery task touching disk. Needs synchronization (X).
Verify and update hg revs
Run compare-locales in-process in python.
Update ES with results.
Update elmo with Run

Celery supports per-host concurrency and multi-host setups. That should
allow us to scale OK.

One caveat that comes with is that if we want to make use of OS resource
caching, in particular for mozilla-central and friends, we need to
synchronize multiple tasks such that only one revision for each repo is
currently active.

Synchronization:
Target: Ensure that all currently running tasks refer to the same
revision of the same repo.
No-goal: enable different revisions on different machines. There's
little to loose here, too.
Proposal: Use redis to store: repo, revision, refcount, timeout. Every
new refcount updates the EXPIRE of the redis hash. If a task comes in
and requests a different hash for the repo, block.
To avoid race conditions, tasks request locks in sorted order for the
repo names. That ensures you're not waiting for a repo that's currently
in line to get another repo.

We may also use some lua scripting here.

Comments?

Axel