RFC/intent-to-implement: Exposing actions defined in-tree through treeherder (actions.json)

[Jonas Finnemann Jensen]

Having read some of the thread on RFC/intent-to-implement, discussion, here
my first:
---

*== Summary ==*
The in-tree decision task generates an 'public/actions.json' artifact that
can be consumed by treeherder and task/task-group inspectors. The
'public/actions.json' artifact contains a list of actions that can be
triggered, as well as how to trigger such actions. This allows in-tree
definition of actions such as: retriggering, backfilling, bisection,
one-click-loaners, etc.

*== Motivation ==*
This is a about making how actions are defined completely rooted in-tree,
such that actions are decoupled from treeherder.
This will allow us to implement an action when people ask "how do I trigger
with low priority", or who do I create a one-click-loaner.

As we add more platforms and workerTypes things like retriggering and
one-click-loaners is going to become more complicated. There is likely
going to be a tight coupling between workerType and how create a
one-click-loaner, this can probably be gracefully handled in the decision
task.

*== Design Outline ==*

Decision task creates an artifact public/actions.json with a list of
entries on the following form:
{
*version*: 1,
*actions*: [
{
*kind*: 'task', // An action backed by a task (only kind for now)
*title*: '...',
*description*: '...',
*context*: [
// The action is relevant for a task, if task.tags is a super-set
of the tags specified
// in at-least one of the tag-sets listed in the 'context' property
{tag1: 'val1', ...}, // matches a task with task.tags.tag1 = 'val1'
and ...
// If context is an empty list, or not present, the action is
relevant in the task-group context
],
*schema*: { // optional JSON schema specifying `input` variable
type: 'string',
enum: ['low', 'normal', 'high'],
description: 'task priority',
},
*task*: { // task template that can be parameterized using
following variables:
// input, taskId, task, taskGroupId
created: {$fromNow: ''},
deadline: {$fromNow: '1 hour 20 minutes'},
priority: {$eval: 'input'}, // replaced by input variable
payload: {
// '${...}' does string interpolation of variable
command: ['./mach', 'action', 'retrigger', '--task-id=${taskId}'],
env: { // {$dumps: x}, renders x to a JSON string
TASK_DEF: {$dumps: {$eval: 'task'}},
}
...
},
...
}
},
...
],
}

Proposed JSON schema for actions.json is available here:
https://gist.github.com/jonasfj/3e9e5d16dee44e908a74597d4ec05c43
(Document includes detailed descriptions of all properties)

Any supporting UI (treeherder, task-inspector, task-group inspector) would:

- load public/actions.json from the decision task (the task with taskId
= taskGroupId)
- actions with context: [{...}, ...] shows up in a menu under each
task (if task.tags match one of tag-sets)
- actions with context: [] shows up in a menu for the entire
task-group/result-set
- When an action is selected:
- the user is presented with a form generated from the JSON schema
- the task template is parameterized with {input, taskId,
taskGroupId, task}
- the parameterized task is created using the users taskcluster
credentials
- the UI displays the livelog and waits for the task to be completed.
- if the task is completed then the action was successfully executed
- if the task failed then the actions failed to be executed.

For some commonly used actions that takes a lot complex options a form
auto-generated from JSON schema might not be ideal.
In this case we can defined the schema for the action in treeherder, and
develop a custom UI for the schema that produces data that matches the
schema.
Hence, the action could use a schema such as:
{$ref: "https://treeherder.mozilla.org/schemas/my-complex-input-v1.json"},
And treeherder UI will recognize this schema, and use a custom UI rather
than an auto-generated form.
This way, if we need a custom UI for a new action we can build it in
treeherder, but keep the action definition in-tree.
Maintaining a loose coupling between treeherder and action definitions,
while facilitating a well integrated UI.

Further more if we later change the complex action to use a different input
schema, we just fallback to the an auto-generated form, until such time
that treeherder implements a custom UI for it. Commonly used actions
probably needs a custom UI. But many rarely used actions like
one-click-loaner, will probably fine with a auto-generated form.

*== Drawbacks ==*
This could definitely be construed as over-engineering, it's also quite
possible that some complex use-cases can't implemented with this.

*== Alternatives ==*
We currently have actions.yaml, example:

https://public-artifacts.taskcluster.net/JCGbx3HJRtWskOG-seK9NQ/0/public/action.yml
But it has a fairly tight coupling between treeherder and the gecko tree,
and it doesn't allow for custom input.

*== Unresolved Questions ==*
The templating language for parameterization of the JSON task definition
isn't fully defined.
We are working to make something similar to
https://github.com/jonasfj/json-parameterization,
but we don't want to use safeEval (for javascript) as this is hard to
re-implement in python.

We know for sure that we want:
{$fromNow: '1 day'} -> '2017-01-18T18:15:49.249Z'
"${input}-world" -> 'hello-world', if variables are {input: 'hello'}
{$dumps: [1,2,3]} -> '[1,2,3]'
{$eval: 'mylist'} -> [1,2,3], if variables are {mylist: [1, 2, 3]}
We'll probably want a slightly more complicated expression based language
for use, as well as allowing for things like:
{$if: 'input == "hello"', then: 'input was hello', else: 'input was not
hallo'}
Hammad (contributor, cc'ed here) is working on this structured template
language: https://github.com/hammad13060/json-e
But a small set of features like {$fromNow: ...}, "${...}", {$dumps: ...},
{$eval: ...} should be enough to get this off the ground.

Note: Once the template language is fairly solid and feature-complete we'll
probably freeze it.

*== Timeline ==*
Status: No commitment.

Following discussion with wlach, dustin and Eli, I have committed to
proposing a schema and writing in-tree documentation for this convention.
And it seems that there is significant interest between wlach, bstack,
dustin and Eli to implement the treeherder-ui and port existing tasks.
So with any luck this isn't that far away :)

--
Regards Jonas Finnemann Jensen.

[Ted Mielczarek]

On Wed, Jan 18, 2017, at 05:52 PM, Jonas Finnemann Jensen wrote:
> *== Motivation ==*
> This is a about making how actions are defined completely rooted in-tree,
> such that actions are decoupled from treeherder.
> This will allow us to implement an action when people ask "how do I
> trigger
> with low priority", or who do I create a one-click-loaner.

This sounds great, thanks for doing the design work here! I agree that
having these actions defined in-tree will make a lot of people's lives
easier.

> *== Design Outline ==*
>
> Decision task creates an artifact public/actions.json with a list of
> entries on the following form:
> {
> *version*: 1,
> *actions*: [
> {
> *kind*: 'task', // An action backed by a task (only kind for now)
> *title*: '...',
> *description*: '...',
> *context*: [
> // The action is relevant for a task, if task.tags is a super-set
> of the tags specified
> // in at-least one of the tag-sets listed in the 'context'
> property
> {tag1: 'val1', ...}, // matches a task with task.tags.tag1 =
> 'val1'
> and ...
> // If context is an empty list, or not present, the action is
> relevant in the task-group context

Is the intention here that you'll only be able to match on task.tags? We
don't seem to use tags very much currently. We do have a lot of metadata
in "extra" that might be useful, and I could also see wanting to
restrict actions to specific worker types, like the "one click loaner"
almost certainly only works with docker-worker right now. (I'm not sure
that we have any task metadata that would let us match on that
currently.)

> *task*: { // task template that can be parameterized using
> following variables:
> // input, taskId, task, taskGroupId
> created: {$fromNow: ''},
> deadline: {$fromNow: '1 hour 20 minutes'},
> priority: {$eval: 'input'}, // replaced by input variable
> payload: {
> // '${...}' does string interpolation of variable
> command: ['./mach', 'action', 'retrigger',
> '--task-id=${taskId}'],
> env: { // {$dumps: x}, renders x to a JSON string
> TASK_DEF: {$dumps: {$eval: 'task'}},
> }

So the intention here is that actions that want to manipulate the
existing task graph (like for retriggers) would spawn a task that would
fetch the existing task graph and start new tasks using the Taskcluster
API? That seems totally reasonable, I just think it should be stated up
front that that's the expected mode of operation for those sorts of
things. We should be able to have some in-tree helper code to make this
straightforward, like a "schedule a copy of this existing task with
modifications".

> For some commonly used actions that takes a lot complex options a form
> auto-generated from JSON schema might not be ideal.
> In this case we can defined the schema for the action in treeherder, and
> develop a custom UI for the schema that produces data that matches the
> schema.

I like this part a lot! Not having to define custom UI for everything
will make it easy to experiment and get things off the ground, but being
able to customize the UI for commonly-used tasks will let us make a nice
UX as well.

-Ted

[Jonas Finnemann Jensen]

>
> Is the intention here that you'll only be able to match on task.tags?

Yes, the idea is to match on task.tags.
Expressing filters over all properties of the task definition would be
complicated.
So the idea is to only filter on task.tags, then we'll likely add tags like:
task.tags.workerKind = 'docker-worker',
Hence, the action for one-click-loaner can have context: [{workerKind:
'docker-worker'}],
ensuring that the one-click-loaner action is only available for
docker-worker backed tasks.

> So the intention here is that actions that want to manipulate the
> existing task graph (like for retriggers) would spawn a task that would
> fetch the existing task graph and start new tasks using the Taskcluster
> API?

Initially that's how it'll work. Maybe when the template language becomes
more powerful
it'll be possible to do some of the simple transformations without an
intermediary action task.
But for now an in-tree an action might look like this:

/taskcluster/actions/loaner_action.py:
from . import register_action

@register_action(
title = "one-click-loaner",
description = """
Create an **interactive task** with same environment as the selected
task.
""",
context = [{"workerKind": "docker-worker"}],
schema = {
"type": "object",
"properties": {
"priority": {

"type": "string",
"enum": ["low", "normal", "high"],

"description": "Task Priority",
"default": "high",
},
},
"additionalProperties": false,
"required": ["priority"],
},
)
def create_loaner(input, taskGroupId, taskId, task):
del task.routes
task['created'] = taskcluster.fromNowJSON('')
task['deadline'] = taskcluster.fromNowJSON('2 h')
task['payload']['features']['interactive'] = True
task['priority'] = input['priority']
...
newTaskId = slugid()
queue.createTask(newTaskId, task)
print("Created interactive task: {}".format(newTaskId))

Exactly how actions are defined and registered in-tree can easily be
changed later.
Initially it'll probably be something where you register a callback (like
create_loaner above)
which then gets called by the intermediary action task.
That way you just have to write a simple script doing whatever it you want.

--
Regards Jonas Finnemann Jensen.

> _______________________________________________
> tools mailing list
> to...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/tools
>

[William Lachance]

Thanks Jonas, this looks awesome.

FWIW I planning on using this feature to develop bug 1322433 ("Make it
easier to retrigger a job with failing test with extra logging and
debugging options") which many developers seem to be excited about.

I'm going to start working on some of the Treeherder bits right away for
this specific feature just to prove out the concept, but am happy to
help on the gecko side or whatever else needs doing to make this happen.

Will

On 2017-01-18 5:52 PM, Jonas Finnemann Jensen wrote:
> Having read some of the thread on RFC/intent-to-implement, discussion, here
> my first:
> ---
>

> ...

[Dustin Mitchell]

To give due credit, this is an extension of the work begun by Kalpesh
Krishna to create action tasks. This is an upgrade to and
formalization of the file format, but built on the same foundation.

We would likely continue to support `actions.yml` in parallel with
`actions.json` until the latter is fully supported, then remove
creation of `actions.yml`. Since this is in-tree, though, we will
have decision tasks creating `actions.yml` for a year or two afterward
(admittedly just ESR for most of that time).

Dustin