Send / Receive now in ansible-tower-cli on PyPI

[Alan Rominger]

We have released tower-cli / awx-cli to PyPI.

pip install ansible-tower-cli==3.3.0

See the help text for the commands related to the new send / receive feature that has been discussed on this mailing list before:

tower-cli --help

Commands:

  config   Read or write tower-cli configuration.

  empty    Empties assets from Tower.

  login    Retrieves and stores an OAuth2 personal auth...

  logout   Removes an OAuth2 personal auth token from...

  receive  Export assets from Tower.

  send     Import assets into Tower.

This can copy resources & permissions from an AWX server to file and back again.

Alan

github: AlanCoding

[Andrew Crisp]

Alan will this allow you to download from awx version 1.0.1 and send to 1.0.5 ? ie get round the problem of schema changes 

Andy

[Marcus von Gerhardt]

additionally to Andrew's question, I've noticed this wont bring any sensitive data (e.g.: primary keys, passwords) - is there anything we can run to export these? 

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project+unsubscribe@googlegroups.com.
To post to this group, send email to awx-p...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/9727a0a5-4ded-4cce-b06a-54456d416296%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Andrew Crisp]

Oh, yeah we also need the ability to extract credentials (not decrypt them) 

On Thursday, April 26, 2018 at 11:54:22 AM UTC+1, Marcus von Gerhardt wrote:

additionally to Andrew's question, I've noticed this wont bring any sensitive data (e.g.: primary keys, passwords) - is there anything we can run to export these? 

On 26 April 2018 at 11:45, Andrew Crisp <cri...@atonhill.com> wrote:

Alan will this allow you to download from awx version 1.0.1 and send to 1.0.5 ? ie get round the problem of schema changes 

Andy

On Wednesday, April 25, 2018 at 6:41:15 PM UTC+1, Alan Rominger wrote:

We have released tower-cli / awx-cli to PyPI.

pip install ansible-tower-cli==3.3.0

See the help text for the commands related to the new send / receive feature that has been discussed on this mailing list before:

tower-cli --help

Commands:

  config   Read or write tower-cli configuration.

  empty    Empties assets from Tower.

  login    Retrieves and stores an OAuth2 personal auth...

  logout   Removes an OAuth2 personal auth token from...

  receive  Export assets from Tower.

  send     Import assets into Tower.

This can copy resources & permissions from an AWX server to file and back again.

Alan

github: AlanCoding

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.

To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.

To post to this group, send email to awx-p...@googlegroups.com.

[Alan Rominger]

Right, I should have mentioned that, secrets management is not part of what the send/receive commands do.

There are many cases like that where you don't want to move secret data around for a migration. Since the send command is idempotent, that means that you can migrate once, manually put in the secrets, and then subsequent migrations will not alter those. I can see some scenarios where this is ideal, but it's not all cases.

Alan

github: AlanCoding

To unsubscribe from this group and stop receiving emails from it, send an email to awx-project+unsubscribe@googlegroups.com.

To post to this group, send email to awx-p...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/48a0d126-434a-49df-a361-03269ed368a3%40googlegroups.com.

[Andrew Crisp]

Ok but whey about the scenario where you want to migrate everything between releases - given we know we can’t upgrade between releases - we need ability to move everything. So shouldn’t this be an option to do credentials ?

Andy

Sent from my iPhone

[Andrew Crisp]

Alan I have just done a test receive and although it’s getting credentials it’s not actually retrieving the private ssh key itself. Is that actually possible via the api ? Or do I need to directly extract from dB ? We don’t need to decrypt it merely want it downloaded so I can import it as part of migration 

Andy

Sent from my iPhone

On 26 Apr 2018, at 13:55, Alan Rominger <arom...@redhat.com> wrote:

[Alan Rominger]

It is not possible to retrieve the private ssh key from the API - so this is a fundamental limitation of the approach of the send / receive commands.

Also, even if you could get the encrypted value, you can't just copy it from one server to another. Only the decryption key on that particular server can unlock the actual secrets, and the decryption key will be unique and random for each server. I can see how you might want this for purposes of a single-server migration, in this case, it's not the right tool.

Alan

github: AlanCoding

[Andrew Crisp]

Ugh !! That’s painful then - because for security we deliberately delete the private key once added into awx. Not being able to copy this from one install to another means we are stuck given upgrades break between releases.

Seems to me we need to be able to execute the decryption key on the source against each credential  - extract the keys before uploading directly into the new tower 

Sent from my iPhone

[Robin Miller]

If only there was a tool that would let you deploy new public keys to servers... ;)

(I kid. I'm in the same boat with wanting a good way to upgrade AWX without losing all this stuff. Although, pushing out a new key would seem like a not-too-painful workaround).

-Robin

Alan

github: AlanCoding

Alan

github: AlanCoding

[Andrew Crisp]

Well the tower-cli receive seems to get everything apart from credentials. Now we have figured out where the encryption key is - it’s fairly obvious ! - we will probably write some python to extract them. 

Alan suggested using the use the `awx-manage shell_plus` entry point, from there, you need to use the Django ORM syntax, and our utilities to manage encryption in code

Andy

Sent from my iPhone

You received this message because you are subscribed to a topic in the Google Groups "AWX Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/awx-project/1f77RNcihZc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to awx-project...@googlegroups.com.

To post to this group, send email to awx-p...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/3ce8763e-bb00-4db0-9197-36571cea8210%40googlegroups.com.

[dnc92301]

Hi Andrew ,

Have you validated you can download from an older release - 1.0.1* and Send it to the latest 1.0.6.* ?
And if you can supply the commands used to perform this activity that would be appreciated.

- don

[Martin Juhl]

I see some issues in using awx-cli:

Users (export is blank), and therefore user permissions isn't set
Log/History is not exported (not high priority in the short run)
Inventory Groups (custom created groups fails for me, going from 1.0.5.31->1.0.6.1
Credential passwords (there should be an option to include them)
LDAP/Auth config (is just not included)

Got any ideas to solve these???

/Martin

[Andrew Crisp]

Not yet - that’s the next piece of work I need to do. I will be sending to 1.0.5.32 btw

Sent from my iPhone

> --
> You received this message because you are subscribed to a topic in the Google Groups "AWX Project" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/awx-project/1f77RNcihZc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to awx-project...@googlegroups.com.
> To post to this group, send email to awx-p...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/9b944d81-a2e6-4eb9-b1ab-8152facfe1af%40googlegroups.com.

[Andrew Crisp]

Guys already said it’s not possible to get credential passwords via api. Suggestion is to look at awx-manage shell_plus.

I might try going via python and using the secret key from inventory to decrypt

Sent from my iPhone

> On 1 May 2018, at 13:00, dnc92301 <dhu...@gmail.com> wrote:
>

[jowe...@redhat.com]

Hi Martin,

    Users and Inventory Groups should be exporting with this tool. What command did you use to export?

    Also, was it the export that failed or the import? Can you check the file that was created by the export to see if it contains the data.

    The rest of the items (logs/history, credential passwords, and LDAP/AWX config) are not part of this tool.

-John

[Andrew Crisp]

Just been testing importing ldap settings via curl to post to the Api v2. Having downloaded the config vid curl, made some changes to the default ldap settings we found on upload  It doesn’t seem to like ldap1-5 settings not that we use them - get server error. So we removed them from the file - just using default. If you leave an option out, ie ldap bind password for example then it deletes the entry in awx ie removes the password. If you have it with $encrypted$ in your file your trying to upload then its fine. 

It also doesn’t seem too complain if you pass it an empty or not existent file - it just happily removes all your settings. We found that by accident when we passed in a file with an incorrect path ie - d @/tmp/file on curl command - /tmp/file didn’t exist - result all settings blown away.

Sent from my iPhone

--

You received this message because you are subscribed to a topic in the Google Groups "AWX Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/awx-project/1f77RNcihZc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to awx-project...@googlegroups.com.
To post to this group, send email to awx-p...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/1f020a3c-a2d1-41cd-9253-fdeee57ecad3%40googlegroups.com.

[dnc92301]

A problem faced is the ldap users imported (send) to new instance - have become Local users . The users were imported prior to configurig LDAP . I tried re-running tower-cli send but it didn’t work. All imported users are missing the LDAP flag. We’d have to delete imported users , and people log back in .

Importing some ~20K host was painfully slow so ended up killing it . It seems to making API call serially . We ended up deleting inventory file with the bulk of the machines - since we have a way to recreating it dynamically .

Lastly , to recreate the job templates - it requires the git credentials ? Most of job templates - except for 1 , did not get imported since git scm update failed due to missing password /credentials . This applies to upgrading from 1.0.4 to 1.0.6* . However it seems to have worked fine from 1.0.1.* to 1.0.6.*
. Question - can the job template be recreated without the need of git project credentials? I would hope it does . If not , we will would to recreate all passwords - git/machine/vault . After the git project is reset , re-run the tower_cli send , to pull down the playbooks and recreate the job template .

[John Westcott]

> A problem faced is the ldap users imported (send) to new instance - have become Local users . The users were imported prior to configurig LDAP . I tried re-running tower-cli send but it didn’t work. All imported users are missing the LDAP flag. We’d have to delete imported users , and people log back in .

Are the users tied to object roles you are importing? Do they need to be or can you use teams to drive the objects? One implementation I did with this includes removing users from roles before the import. That way only the teams own objects and when users log into the system (via LDAP) they are added to the appropriate teams.

> Importing some ~20K host was painfully slow so ended up killing it . It seems to making API call serially . We ended up deleting inventory file with the bulk of the machines - since we have a way to recreating it dynamically .

I have a follow up task to try and determine if machines are generated by dynamic inventories or not. Nodes that are built by dynamic scripts should not be part of the export.

> Lastly , to recreate the job templates - it requires the git credentials ? Most of job templates - except for 1 , did not get imported since git scm update failed due to missing password /credentials . This applies to upgrading from 1.0.4 to 1.0.6* . However it seems to have worked fine from 1.0.1.* to 1.0.6.*
> . Question - can the job template be recreated without the need of git project credentials? I would hope it does . If not , we will would to recreate all passwords - git/machine/vault . After the git project is reset , re-run the tower_cli send , to pull down the playbooks and recreate the job template .

To build a job template the system has to be able to find the ansible playbook on the file system. This is why the import performs a sync before building the template (to make sure the latest content has been pulled down). If your AWX server some how had a cache of the git pulls already on the filesystem (check in /var/lib/awx/projects for the folders the sync creates) then you in theory could rebuild the job templates even if the sync failed. You should only have to put in the git credentials and not all of the rest (machine/vault) for the syncs to work. Instead of doing one bulk migration you may have to send/receive your credentials first, fix the passwords in the git credentials on the new instance and then send/receive the rest of the assets.

-John

[dnc92301]

John - appreciate the detailed explanation .
A few manual steps is needed for restoring to an existing version . Tower-cli receive -all - won’t do us any good since all Users will be converted to Local Accounts . Can you elaborate on - ‘One implementation I did with this includes removing users from roles before the import.’ Yes we will go with Teams as they are mapped to Organzation / RBAC.

I’m trying to map out what’s proper steps with the minimal number of manual steps - since the ‘send all’ option won’t work for us -

- ldap.json
- inventory.json (without dynamic inventory)
- organization.json
- team.json
- credential.json
- project.json
- job_template.son
- workflow.json

[John Westcott]

I can’t elaborate too far on that particular project but essentially you have JSON from the receive so you could manipulate it as you see fit before sending it to the send command. For example, you could loop over all of the objects and remove any users populated in the roles sections.

-John

> --
> You received this message because you are subscribed to the Google Groups "AWX Project" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.

> To post to this group, send email to awx-p...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/cf65626a-e9f6-416d-8dc7-54743af9677e%40googlegroups.com.

[driCs]

Hello !

Has anyone successfully restored the credentials and the corresponding ssh keys since 2018/06/11 ? :)

Regards,

driCs