How to schedule tasks with personal credentials?

[Armen Zambrano G.]

Hello all,
I've been trying to schedule a Linux 64 task by using my personal
credentials (obtained through TC's web auth approach), however, I don't
get all the necessary scopes to do so.

I've written a script that does this [1] (notice that's not the 'master'
branch).

What is the right approach? Do I need to first create a new set of
credentials in the web UI and use that instead? [2]

regards,
Armen

[1]
https://github.com/armenzg/TC_developer_scheduling_experiments/blob/schedule_linux_task/schedule_linux64_task.py
[2] https://tools.taskcluster.net/auth/clients/
--
Zambrano Gasparnian, Armen
Automation & Tools Engineer
http://armenzg.blogspot.ca

[Armen Zambrano G.]

On 2016-05-12 12:12 PM, Armen Zambrano G. wrote:
> Hello all,
> I've been trying to schedule a Linux 64 task by using my personal
> credentials (obtained through TC's web auth approach), however, I don't
> get all the necessary scopes to do so.
>
> I've written a script that does this [1] (notice that's not the 'master'
> branch).
>
> What is the right approach? Do I need to first create a new set of
> credentials in the web UI and use that instead? [2]
>
> regards,
> Armen
>
> [1]
> https://github.com/armenzg/TC_developer_scheduling_experiments/blob/schedule_linux_task/schedule_linux64_task.py
>
> [2] https://tools.taskcluster.net/auth/clients/

We've spoken in the past that if TH used TaskCluster's auth, it would
allow us to make Treeherder schedule tasks for the developer using their
credentials directly from the UI.

What I'm trying to determine in here is if we can do the same from the CLI.

regards,
Armen

[Dustin Mitchell]

I can't see the task you are trying to create, or the error message, so I'm
guessing, but I bet you're missing the scopes for the index routes.

We've left those scopes off of users' credentials, and only added them for
decision tasks derived from the branch in question. For example, oak's
role [1] contains several oak-specific routes. The idea here is to avoid
polluting those routes with random tasks submitted by users, and to prevent
tasks on one branch from being indexed on another due to errors in task
configuration.

For the moment, your script should probably strip task.routes down to [].
When we allow users to trigger tasks from treeherder, we'll need to revisit
this plan.

Dustin

On Thu, May 12, 2016 at 12:16 PM, Armen Zambrano G. <arm...@mozilla.com>
wrote:

> On 2016-05-12 12:12 PM, Armen Zambrano G. wrote:
>

>> Hello all,
>> I've been trying to schedule a Linux 64 task by using my personal
>> credentials (obtained through TC's web auth approach), however, I don't
>> get all the necessary scopes to do so.
>>
>> I've written a script that does this [1] (notice that's not the 'master'
>> branch).
>>
>> What is the right approach? Do I need to first create a new set of
>> credentials in the web UI and use that instead? [2]
>>
>> regards,
>> Armen
>>
>> [1]
>>
>> https://github.com/armenzg/TC_developer_scheduling_experiments/blob/schedule_linux_task/schedule_linux64_task.py
>>
>> [2] https://tools.taskcluster.net/auth/clients/
>>
>

> We've spoken in the past that if TH used TaskCluster's auth, it would
> allow us to make Treeherder schedule tasks for the developer using their
> credentials directly from the UI.
>
> What I'm trying to determine in here is if we can do the same from the CLI.
>
> regards,
> Armen
>
>

> --
> Zambrano Gasparnian, Armen
> Automation & Tools Engineer
> http://armenzg.blogspot.ca

> _______________________________________________
> tools-taskcluster mailing list
> tools-ta...@lists.mozilla.org
> https://lists.mozilla.org/listinfo/tools-taskcluster
>

[Armen Zambrano Gasparnian]

Your trick seems to work [1] but that takes away from being able to see the
job on Treeherder.
task['routes'] only contained this:

> [u'tc-treeherder.try.cc896d5d2241a2da1ed42c9bd29013c87ef403c6',
> u'tc-treeherder-stage.try.cc896d5d2241a2da1ed42c9bd29013c87ef403c6']
>

Is there a way that I can still make the job show up on Treeherder?

[1] https://tools.taskcluster.net/task-inspector/#LffmegUYTRaA5GlkL8k63A/0

On 12 May 2016 at 13:06, Armen Zambrano Gasparnian <arm...@mozilla.com>
wrote:

> I will try that.
>
> This is the error I get:
> "code": "InsufficientScopes",
> "details": {
> "scopes": [
> "assume:hook-id:garbage/*",
> "assume:moz-tree:level:1",
> "assume:moz-tree:level:2",
> "assume:moz-tree:level:3",
> "assume:mozilla-group:ateam",
> "assume:mozilla-group:scm_level_1",
> "assume:mozilla-group:scm_level_2",
> "assume:mozilla-group:scm_level_3",
> "assume:mozilla-group:team_moco",
> "assume:mozilla-group:vpn_tooltooleditor",
> "assume:mozilla-user:arm...@mozilla.com",
> "assume:project-admin:ateam",
> "assume:project:taskcluster:level-1-sccache-buckets",
> "assume:project:taskcluster:tutorial",
> "assume:worker-id:*",
>
> "auth:aws-s3:read-write:taskcluster-level-1-sccache-us-east-1/*",
>
> "auth:aws-s3:read-write:taskcluster-level-1-sccache-us-west-1/*",
>
> "auth:aws-s3:read-write:taskcluster-level-1-sccache-us-west-2/*",
> "auth:create-client:mozilla-ldap/arm...@mozilla.com/*",
> "auth:create-client:project/ateam/*",
> "auth:create-role:hook-id:project-ateam/*",
> "auth:create-role:project:ateam:*",
> "auth:delete-client:mozilla-ldap/arm...@mozilla.com/*",
> "auth:delete-client:project/ateam/*",
> "auth:delete-role:hook-id:project-ateam/*",
> "auth:delete-role:project:ateam:*",
> "auth:disable-client:project/ateam/*",
> "auth:enable-client:project/ateam/*",
> "auth:reset-access-token:mozilla-ldap/arm...@mozilla.com/*",
> "auth:reset-access-token:project/ateam/*",
> "auth:update-client:mozilla-ldap/arm...@mozilla.com/*",
> "auth:update-client:project/ateam/*",
> "auth:update-role:hook-id:project-ateam/*",
> "auth:update-role:project:ateam:*",
> "docker-worker:cache:level-1-*",
> "docker-worker:cache:level-2-*",
> "docker-worker:cache:level-3-*",
> "docker-worker:cache:tooltool-cache",
> "docker-worker:capability:device:loopbackAudio",
> "docker-worker:capability:device:loopbackVideo",
> "docker-worker:capability:device:phone",
> "docker-worker:capability:privileged",
> "docker-worker:feature:allowPtrace",
> "docker-worker:feature:balrogVPNProxy",
> "docker-worker:image:quay.io/mozilla/builder:*",
> "docker-worker:image:quay.io/mozilla/decision:*",
> "docker-worker:image:taskcluster/builder:*",
> "docker-worker:image:taskcluster/tester:*",
> "docker-worker:image:taskclusterprivate/phone-builder:*",
>
> "docker-worker:image:taskclusterprivate/taskcluster-vpn-proxy:*",
> "docker-worker:image:taskclusterprivate/tester-device:*",
> "docker-worker:image:taskclusterprivate/upload_symbols:*",
> "docker-worker:relengapi-proxy:tooltool.download.internal",
> "docker-worker:relengapi-proxy:tooltool.download.public",
> "docker-worker:relengapi-proxy:tooltool.upload.*",
> "hooks:modify-hook:garbage/*",
> "hooks:modify-hook:project-ateam/*",
> "index:insert-task:project.ateam.*",
> "project:ateam:*",
> "queue:create-task:aws-provisioner-v1/ami-test*",
> "queue:create-task:aws-provisioner-v1/android-api-*",
> "queue:create-task:aws-provisioner-v1/b2g-desktop-*",
> "queue:create-task:aws-provisioner-v1/b2gbuild*",
> "queue:create-task:aws-provisioner-v1/b2gtest*",
> "queue:create-task:aws-provisioner-v1/balrog",
> "queue:create-task:aws-provisioner-v1/build-c4-2xlarge",
> "queue:create-task:aws-provisioner-v1/dbg-*",
> "queue:create-task:aws-provisioner-v1/desktop-test*",
> "queue:create-task:aws-provisioner-v1/dolphin",
> "queue:create-task:aws-provisioner-v1/emulator-*",
> "queue:create-task:aws-provisioner-v1/flame-kk*",
> "queue:create-task:aws-provisioner-v1/gecko-decision",
> "queue:create-task:aws-provisioner-v1/gecko-talos-c3large",
> "queue:create-task:aws-provisioner-v1/gecko-talos-c4large",
> "queue:create-task:aws-provisioner-v1/mulet-debug",
> "queue:create-task:aws-provisioner-v1/mulet-opt",
> "queue:create-task:aws-provisioner-v1/opt-*",
> "queue:create-task:aws-provisioner-v1/rustbuild",
> "queue:create-task:aws-provisioner-v1/spidermonkey",
> "queue:create-task:aws-provisioner-v1/symbol-upload",
> "queue:create-task:aws-provisioner-v1/taskcluster-images",
> "queue:create-task:aws-provisioner-v1/test-c4-2xlarge",
> "queue:create-task:aws-provisioner-v1/testdroid-device",
> "queue:create-task:aws-provisioner-v1/tutorial",
> "queue:create-task:aws-provisioner-v1/win*",
> "queue:create-task:dummy-test-provisioner/dummy-test-type",
> "queue:create-task:packetnet/*",
> "queue:define-task:aws-provisioner-v1/build-c4-2xlarge",
> "queue:define-task:aws-provisioner-v1/taskcluster-images",
> "queue:define-task:aws-provisioner-v1/test-c4-2xlarge",
> "queue:define-task:dummy-test-provisioner/dummy-test-type",
> "queue:get-artifact:private/*",
> "queue:get-artifact:project/ateam/*",
> "queue:rerun-task",
> "queue:resolve-task",
> "queue:route:index.project.ateam.*",
> "scheduler:create-task-graph",
> "scheduler:extend-task-graph",
> "scheduler:extend-task-graph:*",
> "secrets:get:garbage/*",
> "secrets:get:project/ateam/*",
> "secrets:get:project/releng/gecko/build/level-1/*",
> "secrets:get:project/releng/gecko/build/level-2/*",
> "secrets:get:project/releng/gecko/build/level-3/*",
> "secrets:get:project/taskcluster/gecko/build/level-2/*",
> "secrets:get:project/taskcluster/gecko/build/level-3/*",
> "secrets:get:project/taskcluster/gecko/hgfingerprint",
> "secrets:set:garbage/*",
> "secrets:set:project/ateam/*"
> ],
> "scopesets": [
> [
> "docker-worker:feature:allowPtrace",
> "docker-worker:cache:level-1-try-test-workspace",
>
> "queue:route:tc-treeherder.try.cc896d5d2241a2da1ed42c9bd29013c87ef403c6",
>
> "queue:route:tc-treeherder-stage.try.cc896d5d2241a2da1ed42c9bd29013c87ef403c6"
> ]
> ]

> }
> }
>
> On 12 May 2016 at 12:32, Dustin Mitchell <dus...@mozilla.com> wrote:
>
>> I can't see the task you are trying to create, or the error message, so
>> I'm guessing, but I bet you're missing the scopes for the index routes.
>>
>> We've left those scopes off of users' credentials, and only added them
>> for decision tasks derived from the branch in question. For example, oak's
>> role [1] contains several oak-specific routes. The idea here is to avoid
>> polluting those routes with random tasks submitted by users, and to prevent
>> tasks on one branch from being indexed on another due to errors in task
>> configuration.
>>
>> For the moment, your script should probably strip task.routes down to
>> []. When we allow users to trigger tasks from treeherder, we'll need to
>> revisit this plan.
>>
>> Dustin
>>
>> On Thu, May 12, 2016 at 12:16 PM, Armen Zambrano G. <arm...@mozilla.com>
>> wrote:
>>
>>> On 2016-05-12 12:12 PM, Armen Zambrano G. wrote:
>>>

>>>> Hello all,
>>>> I've been trying to schedule a Linux 64 task by using my personal
>>>> credentials (obtained through TC's web auth approach), however, I don't
>>>> get all the necessary scopes to do so.
>>>>
>>>> I've written a script that does this [1] (notice that's not the 'master'
>>>> branch).
>>>>
>>>> What is the right approach? Do I need to first create a new set of
>>>> credentials in the web UI and use that instead? [2]
>>>>
>>>> regards,
>>>> Armen
>>>>
>>>> [1]
>>>>
>>>> https://github.com/armenzg/TC_developer_scheduling_experiments/blob/schedule_linux_task/schedule_linux64_task.py
>>>>
>>>> [2] https://tools.taskcluster.net/auth/clients/
>>>>
>>>

>>> We've spoken in the past that if TH used TaskCluster's auth, it would
>>> allow us to make Treeherder schedule tasks for the developer using their
>>> credentials directly from the UI.
>>>
>>> What I'm trying to determine in here is if we can do the same from the
>>> CLI.
>>>
>>> regards,
>>> Armen
>>>
>>>

>>> --
>>> Zambrano Gasparnian, Armen
>>> Automation & Tools Engineer
>>> http://armenzg.blogspot.ca

>>> _______________________________________________
>>> tools-taskcluster mailing list
>>> tools-ta...@lists.mozilla.org
>>> https://lists.mozilla.org/listinfo/tools-taskcluster
>>>
>>
>>
>
>

> --
> Zambrano Gasparnian, Armen
> Engineering productivy engineer - #ateam
> http://armenzg.blogspot.ca
>



--
Zambrano Gasparnian, Armen
Engineering productivy engineer - #ateam
http://armenzg.blogspot.ca

[Dustin Mitchell]

Ah, I forgot treeherder listens to a different set of routes. So, yes, we
could give all scm_level_1's `queue:route:tc-treeherder.*` (and stage).
The risk is that treeherder results can then easily be forged. I don't
know how much of a concern that is.

Another option may be to change those routes to include a level (which
would require teaching treeherder about levels, probably), and allow
scm_level_1 only `queue:route:tc-treeherder.level-1.*`.

Dustin

On Thu, May 12, 2016 at 1:14 PM, Armen Zambrano Gasparnian <

>>>>> Hello all,
>>>>> I've been trying to schedule a Linux 64 task by using my personal
>>>>> credentials (obtained through TC's web auth approach), however, I don't
>>>>> get all the necessary scopes to do so.
>>>>>
>>>>> I've written a script that does this [1] (notice that's not the
>>>>> 'master'
>>>>> branch).
>>>>>
>>>>> What is the right approach? Do I need to first create a new set of
>>>>> credentials in the web UI and use that instead? [2]
>>>>>
>>>>> regards,
>>>>> Armen
>>>>>
>>>>> [1]
>>>>>
>>>>> https://github.com/armenzg/TC_developer_scheduling_experiments/blob/schedule_linux_task/schedule_linux64_task.py
>>>>>
>>>>> [2] https://tools.taskcluster.net/auth/clients/
>>>>>
>>>>

>>>> We've spoken in the past that if TH used TaskCluster's auth, it would
>>>> allow us to make Treeherder schedule tasks for the developer using their
>>>> credentials directly from the UI.
>>>>
>>>> What I'm trying to determine in here is if we can do the same from the
>>>> CLI.
>>>>
>>>> regards,
>>>> Armen
>>>>
>>>>

>>>> --
>>>> Zambrano Gasparnian, Armen
>>>> Automation & Tools Engineer
>>>> http://armenzg.blogspot.ca

[Jonas Finnemann Jensen]

Giving "queue:route:tc-treeherder.try.*" is probably less of a problem...

But note that the decision task shouldn't get anything but: "queue:route:
tc-treeherder.try.cc896d5d2241a2da1ed42c9bd29013c87ef403c6"
So it's can't fake the result-set it attaches to...

With "queue:route:tc-treeherder.try.*" you can add tasks to any treeherder
result-set for try.


--
Regards Jonas Finnemann Jensen.

> >>>>> Hello all,
> >>>>> I've been trying to schedule a Linux 64 task by using my personal
> >>>>> credentials (obtained through TC's web auth approach), however, I
> don't
> >>>>> get all the necessary scopes to do so.
> >>>>>
> >>>>> I've written a script that does this [1] (notice that's not the
> >>>>> 'master'
> >>>>> branch).
> >>>>>
> >>>>> What is the right approach? Do I need to first create a new set of
> >>>>> credentials in the web UI and use that instead? [2]
> >>>>>
> >>>>> regards,
> >>>>> Armen
> >>>>>
> >>>>> [1]
> >>>>>
> >>>>>
> https://github.com/armenzg/TC_developer_scheduling_experiments/blob/schedule_linux_task/schedule_linux64_task.py
> >>>>>
> >>>>> [2] https://tools.taskcluster.net/auth/clients/
> >>>>>
> >>>>

> >>>> We've spoken in the past that if TH used TaskCluster's auth, it would
> >>>> allow us to make Treeherder schedule tasks for the developer using
> their
> >>>> credentials directly from the UI.
> >>>>
> >>>> What I'm trying to determine in here is if we can do the same from the
> >>>> CLI.
> >>>>
> >>>> regards,
> >>>> Armen
> >>>>
> >>>>

> >>>> --
> >>>> Zambrano Gasparnian, Armen
> >>>> Automation & Tools Engineer
> >>>> http://armenzg.blogspot.ca

[Dustin Mitchell]

Jonas is referring to an extra level of assurance, where one decision
task could not submit results for a different push to the same branch.

I don't think this is a terribly hard problem to solve -- we just need
to select our level of paranoia :)

Dustin

On Thu, May 12, 2016 at 2:41 PM, Armen Zambrano Gasparnian
<arm...@mozilla.com> wrote:
> What kind of forging are you thinking of? Submit jobs incorrectly?
>
> Should I go back to the idea that a privileged app can do scheduling on
> behalf of the user?
> I could then stick to a model that I know it works.

>>>>>>> Hello all,
>>>>>>> I've been trying to schedule a Linux 64 task by using my personal
>>>>>>> credentials (obtained through TC's web auth approach), however, I
>>>>>>> don't
>>>>>>> get all the necessary scopes to do so.
>>>>>>>
>>>>>>> I've written a script that does this [1] (notice that's not the
>>>>>>> 'master'
>>>>>>> branch).
>>>>>>>
>>>>>>> What is the right approach? Do I need to first create a new set of
>>>>>>> credentials in the web UI and use that instead? [2]
>>>>>>>
>>>>>>> regards,
>>>>>>> Armen
>>>>>>>
>>>>>>> [1]
>>>>>>>
>>>>>>> https://github.com/armenzg/TC_developer_scheduling_experiments/blob/schedule_linux_task/schedule_linux64_task.py
>>>>>>>
>>>>>>> [2] https://tools.taskcluster.net/auth/clients/
>>>>>>
>>>>>>

>>>>>> We've spoken in the past that if TH used TaskCluster's auth, it would
>>>>>> allow us to make Treeherder schedule tasks for the developer using their
>>>>>> credentials directly from the UI.
>>>>>>
>>>>>> What I'm trying to determine in here is if we can do the same from the
>>>>>> CLI.
>>>>>>
>>>>>> regards,
>>>>>> Armen
>>>>>>
>>>>>>

>>>>>> --
>>>>>> Zambrano Gasparnian, Armen
>>>>>> Automation & Tools Engineer
>>>>>> http://armenzg.blogspot.ca

[Armen Zambrano Gasparnian]

>>>>>> Hello all,
>>>>>> I've been trying to schedule a Linux 64 task by using my personal
>>>>>> credentials (obtained through TC's web auth approach), however, I
>>>>>> don't
>>>>>> get all the necessary scopes to do so.
>>>>>>
>>>>>> I've written a script that does this [1] (notice that's not the
>>>>>> 'master'
>>>>>> branch).
>>>>>>
>>>>>> What is the right approach? Do I need to first create a new set of
>>>>>> credentials in the web UI and use that instead? [2]
>>>>>>
>>>>>> regards,
>>>>>> Armen
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>> https://github.com/armenzg/TC_developer_scheduling_experiments/blob/schedule_linux_task/schedule_linux64_task.py
>>>>>>
>>>>>> [2] https://tools.taskcluster.net/auth/clients/
>>>>>>
>>>>>

>>>>> We've spoken in the past that if TH used TaskCluster's auth, it would
>>>>> allow us to make Treeherder schedule tasks for the developer using their
>>>>> credentials directly from the UI.
>>>>>
>>>>> What I'm trying to determine in here is if we can do the same from the
>>>>> CLI.
>>>>>
>>>>> regards,
>>>>> Armen
>>>>>
>>>>>

>>>>> --
>>>>> Zambrano Gasparnian, Armen
>>>>> Automation & Tools Engineer
>>>>> http://armenzg.blogspot.ca

[Armen Zambrano Gasparnian]

>>> Hello all,
>>> I've been trying to schedule a Linux 64 task by using my personal
>>> credentials (obtained through TC's web auth approach), however, I don't
>>> get all the necessary scopes to do so.
>>>
>>> I've written a script that does this [1] (notice that's not the 'master'
>>> branch).
>>>
>>> What is the right approach? Do I need to first create a new set of
>>> credentials in the web UI and use that instead? [2]
>>>
>>> regards,
>>> Armen
>>>
>>> [1]
>>>
>>> https://github.com/armenzg/TC_developer_scheduling_experiments/blob/schedule_linux_task/schedule_linux64_task.py
>>>
>>> [2] https://tools.taskcluster.net/auth/clients/
>>>
>>

>> We've spoken in the past that if TH used TaskCluster's auth, it would
>> allow us to make Treeherder schedule tasks for the developer using their
>> credentials directly from the UI.
>>
>> What I'm trying to determine in here is if we can do the same from the
>> CLI.
>>
>> regards,
>> Armen
>>
>>

>> --
>> Zambrano Gasparnian, Armen
>> Automation & Tools Engineer
>> http://armenzg.blogspot.ca