Updating roles on project update/install

[Mike Biancaniello]

I am running AWX inside of some docker containers and can't get the project_update job to install roles from my requirements.yml file. It successfully finds the file, but, for some reason, scm_full_checkout is always set to false. 

I modified /usr/lib/python2.7/site-packages/awx/playbooks/project_update.yml to add this:

- name: detect requirements.yml

  stat: path={{project_path|quote}}/roles/requirements.yml

  register: doesRequirementsExist

+ - name: show me stat

+   debug:

+   msg: "{{ doesRequirementsExist }}"

+

+ - name: show me scm_full_checkout

+   debug:

+   msg: "{{ scm_full_checkout }}"

- name: fetch galaxy roles from requirements.yml

  command: ansible-galaxy install -r requirements.yml -p {{project_path|quote}}/roles/ --force

  args:

    chdir: "{{project_path|quote}}/roles"

  when: doesRequirementsExist.stat.exists and scm_full_checkout|bool

Which results in the following displayed in the job output (I removed most of the file stats for brevity):

TASK [detect requirements.yml] *************************************************

ok: [localhost]

TASK [show me stat] ************************************************************

ok: [localhost] => {

"msg": {

"changed": false,

"failed": false,

"stat": {

"executable": false,

"exists": true,

"mode": "0644",

"readable": true,

"uid": 0,

"writeable": true,

}

}

}

TASK [show me scm_full_checkout] ***********************************************

ok: [localhost] => {

"msg": false

}

TASK [fetch galaxy roles from requirements.yml] ********************************

skipping: [localhost]

I've tried every option I can find for defining the project and in every case, scm_full_checkout is *always* false!

Any ideas how I can get this to work?

thanks,

-MikeB

 

[Matthew Jones]

It'll only sync the roles when a job is run using that project, until then it won't sync the roles from the requirements file.

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project+unsubscribe@googlegroups.com.
To post to this group, send email to awx-p...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/46f8b7ed-a615-4b06-b457-47b4c7b47de0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Matt Jones

Principal Software Engineer

Ansible Tower

[Mike Biancaniello]

I'm getting closer!

So I was able to get this to at least try to install the roles when I run the job. However I am running into a few issues:

1. The roles aren't installing with error: "SCM Update failed with ID 37".

    I assume this is because the roles require creds. Unfortunately, the job template doesn't allow for adding "Source Control" creds.

    I tried setting the "machine" creds to use my key, but that failed with "ID 39" (I didn't actually expect this to work).

2. The output display is empty (even at highest verbosity). 

    I assume this is because the output display is only going to show the output from the playbook that I specified for the job.

[Mike Biancaniello]

Is there a way to assign creds to the job template to use when grabbing new roles from a requirements.yml? In this case, it can use the same creds that it used to get the project (same git repo).

[Matthew Jones]

We have thought about having JTs generated from projects but not currently.

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project+unsubscribe@googlegroups.com.
To post to this group, send email to awx-p...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/0fafb4f4-4156-4d6e-a31d-34500d2d3001%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Chris Church]

Note that there are two different modes for running project updates (job_type == "check" vs. job_type == "run", which also show up as launch_type == "dependency" vs. launch_type == "sync"). The check/dependency job runs project_update.yml in check mode and sets scm_full_checkout = false, whereas the run/sync mode sets scm_full_checkout = true.

The default jobs list hides the sync updates:  /#/jobs?job_search=page_size:20;order_by:-finished;not__launch_type:sync

If you modify that default filter in the URL, you should see both SCM updates that ran before your playbook run: /#/jobs?job_search=page_size:20;order_by:-finished

What Matt meant by "It'll only sync the roles when a job is run using that project" is that the the requirements.yml will only be pulled down by the second run/sync SCM update prior to running your job template. The SCM credentials specified for your project (if SSH keys) should be available for use by galaxy when checking out roles, but if you're using username/password instead, we won't be able to inject that into the SCM URLs as we do for normal SCM updates (https://github.com/ansible/awx/blob/devel/awx/main/tasks.py#L1346).

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/CAHaX%3DYq%3DwsPOOnQtLWR7%2BoN%2BZFHcSyjYOnuV5xbT7_Lp2pz__A%40mail.gmail.com.

[Mike Biancaniello]

Thanks for the tip about the filter! 

That helped me find the "SCM Update" task that had the actual error in it. Turns out, the urls in the requirements file were ssh host aliases and (obviously) those aliases aren't configured in the container.

I'll give it another shot tomorrow.

On Thursday, September 21, 2017 at 9:54:32 PM UTC-4, Chris Church wrote:

Note that there are two different modes for running project updates (job_type == "check" vs. job_type == "run", which also show up as launch_type == "dependency" vs. launch_type == "sync"). The check/dependency job runs project_update.yml in check mode and sets scm_full_checkout = false, whereas the run/sync mode sets scm_full_checkout = true.

The default jobs list hides the sync updates:  /#/jobs?job_search=page_size:20;order_by:-finished;not__launch_type:sync

If you modify that default filter in the URL, you should see both SCM updates that ran before your playbook run: /#/jobs?job_search=page_size:20;order_by:-finished

What Matt meant by "It'll only sync the roles when a job is run using that project" is that the the requirements.yml will only be pulled down by the second run/sync SCM update prior to running your job template. The SCM credentials specified for your project (if SSH keys) should be available for use by galaxy when checking out roles, but if you're using username/password instead, we won't be able to inject that into the SCM URLs as we do for normal SCM updates (https://github.com/ansible/awx/blob/devel/awx/main/tasks.py#L1346).

On Thu, Sep 21, 2017 at 4:31 PM, Matthew Jones <maj...@redhat.com> wrote:

We have thought about having JTs generated from projects but not currently.

On Thu, Sep 21, 2017 at 3:07 PM, Mike Biancaniello <chep...@gmail.com> wrote:

Is there a way to assign creds to the job template to use when grabbing new roles from a requirements.yml? In this case, it can use the same creds that it used to get the project (same git repo).

On Wednesday, September 20, 2017 at 10:05:52 AM UTC-4, Mike Biancaniello wrote:

I'm getting closer!

So I was able to get this to at least try to install the roles when I run the job. However I am running into a few issues:

1. The roles aren't installing with error: "SCM Update failed with ID 37".

    I assume this is because the roles require creds. Unfortunately, the job template doesn't allow for adding "Source Control" creds.

    I tried setting the "machine" creds to use my key, but that failed with "ID 39" (I didn't actually expect this to work).

2. The output display is empty (even at highest verbosity). 

    I assume this is because the output display is only going to show the output from the playbook that I specified for the job.

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.

To post to this group, send email to awx-p...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/0fafb4f4-4156-4d6e-a31d-34500d2d3001%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Matt Jones

Principal Software Engineer

Ansible Tower

--

You received this message because you are subscribed to the Google Groups "AWX Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.

To post to this group, send email to awx-p...@googlegroups.com.

[Mike Biancaniello]

Ok, so my issue turned out to be the username.

- I am using SSH-key creds with git scm.
- I also have to send a username (because awx in the container runs as root and user:root isn't a valid git user).

- I specify this username (no password, just the key) in the project definition.
- Task: "update project using git and accept hostkey" works fine (meaning it sends ssh://user...@git.url), presumably because it builds the URL.
- Task: "fetch galaxy roles from requirements.yml" just runs galaxy as `-r requirements`, so uses the URL from the file, which doesn't contain a username.

hence, failure.

I don't see a good AWX or galaxy way around this, though. And, if I put the username in the URL in the requirements file, then nobody else can use that file to install the requirements on their own machine (for dev/testing/etc) without modifying the file to either remove the username or use their own.

I also tested this when in check mode and it also tried (and failed) to install the requirements.

(fwiw, updating the requirements when in check mode is good. Otherwise, every playbook in the project that "requires" a role in the "requirements" file would fail)

anyway, I got everything installed, but then the playbooks can't find the roles!

My tree:

[root@awx projects]# tree -L 4

.

└── _6__my_projectmikeb_test

    ├── ansible.cfg

    ├── inventory_scripts

    │   └── cmdb_inventory.py

    ├── playbooks

    │   └── test

    │       ├── helloworld.yml

    │       └── network_config.yml

    ├── requirements.txt

    └── roles

        ├── my.libs

        ├── my.tasks

        └── requirements.yml

So ... I had to modify my project's ansible.cfg to set roles_path=roles/ (relative path).

Which leaves me with 2 modifications to my project's repo to work within AWX, but not anywhere else.

I may just stick with mounting a volume into awx_task:/etc/ansible/roles/ and installing the roles on the host. I'll have to sleep on it.