Periodic jobs

[Armen Zambrano G.]

Hello,
I recently filed bug 1319439 [1] to schedule Valgrind jobs on a 12 hour
schedule.

I'm looking to add it as hook but I have several questions:
* Should it be put under "project-releng" [2] since there are few
services that schedule jobs?
* Should we use a different name for non "Release" specific hooks? (e.g.
"tree periodic jobs"?
* Is there a guide for developers/sheriffs to create this on their own?
* Should these kind of requests be Releng requests? or Ateam? or
TaskCluster?

regards,
Armen

[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1319439
[2] https://tools.taskcluster.net/hooks/#project-releng

--
Zambrano Gasparnian, Armen
Engineering productivity
http://armenzg.blogspot.ca

[Jonas Finnemann Jensen]

I don't we ever got around to building periodic tasks in-tree.
Back then we talked about creating a cron.yml file that would list tasks
that should run periodically.
Then taskcluster-hooks would periodically kick off a decision task that
would create tasks from cron.yml.

But perhaps a better design is to "optimize" out tasks that should only run
periodically.
Perhaps using a transform like:
when:
every: '24 hours'

Then we just need taskcluster-hooks to run a task that kicks off a decision
task for the latest revision on mozilla-central with some try flag
indicating that periodic tasks should be scheduled.
If taskcluster-hooks could kick off such a task every 3 hours, we could
reasonably schedule things to run every: 3, 6, 12, 24, 48 or 72 hours.
In-tree we would just add a YAML file like we do for most other tasks, and
give it a "when.every" clause, similar to how the 'when.files-changed'
clause is used to only run tasks if certain files have changed.

Note: I say taskcluster-hooks would run an auxiliary task that kicks off
the decision task.
Because the auxiliary task would need find latest revision on
mozilla-central and set appropriate routes
for the decision tasks to show up on treeherder.

@dustin, what do you think? I suspect the task-kind optimization logic can
lookup the task-label in the index, to determine whether to schedule or not.


--
Regards Jonas Finnemann Jensen.

2016-11-22 8:43 GMT-08:00 Armen Zambrano G. <arm...@mozilla.com>:

> I wonder if instead of hooks, the gecko decision task could determine
> that it should run a periodic job by checking when was the last time one
> of those jobs run.
>
> In other words, I'm proposing that there is an in-tree mechanism to say
> "run this task if it has passed more than X seconds since it has run".
>
> I could also potentially implement this inside of SETA, however, I believe
> this is a common feature that we should support via TaskCluster.
>
> regards,
> Armen

> _______________________________________________
> tools-taskcluster mailing list
> tools-ta...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/tools-taskcluster
>

[Gregory Szorc]

On Tue, Nov 22, 2016 at 2:48 PM, Jonas Finnemann Jensen <jop...@gmail.com>
wrote:

> I don't we ever got around to building periodic tasks in-tree.
> Back then we talked about creating a cron.yml file that would list tasks
> that should run periodically.
> Then taskcluster-hooks would periodically kick off a decision task that
> would create tasks from cron.yml.
>
> But perhaps a better design is to "optimize" out tasks that should only run
> periodically.
> Perhaps using a transform like:
> when:
> every: '24 hours'
>

I like this idea. I like having all the tasks in a single DAG instead of
potentially a separate DAG derived from e.g. a cron.yml file. This allows
us to do things like running a task if files changed *or* sufficient time
has passed (this is something I need for the CI for `mach bootstrap.`)

One could also imagine the scheduling frequency defined in terms of push
count. e.g. "every 4 pushes." That starts to infringe on SETA's territory
though, so we'd have to think long and hard about that.

[Jonas Finnemann Jensen]

Also keeping it in the same DAG means that actions tasks for:
- backfilling
- retriggering
, will work as they work for normal tasks.
Hence, it's possible to force manually schedule a periodic task.
Similar to how you backfill SETA or retrigger a failed task.

--
Regards Jonas Finnemann Jensen.

[Dustin Mitchell]

++ to all that :)

IMHO a lot of SETA could be implemented with this scheme as well,
especially if we're comfortable with the "every 4 pushes" being
probabalistic (with the pushdate or something used as random seed, of
course).

You're right that this is a common request, so it would be great to build
the stuff to support the next person who wants this kind of feature.

Dustin

[Chris Cooper]

Dustin: Since the periodic file updates (HSTS, HPKP, blocklist)
builders are still to migrate, this functionality also counts as
something the TC could work on to assist the migration process.

+1 from me too.

cheers,
--
coop

> _______________________________________________
> dev-tree-management mailing list
> dev-tree-...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/dev-tree-management

[Dustin Mitchell]

Fair enough :)

Dustin

[Dustin Mitchell]

In fact,
https://bugzilla.mozilla.org/show_bug.cgi?id=1288993
is already blocked on
https://bugzilla.mozilla.org/show_bug.cgi?id=1252948

I've duped things. I also added this as a migration item in

https://github.com/taskcluster/migration/pull/30

Dustin