Migrating and upgrading from AWX 17.1.0 to AWX 19.x with Operator approach

[Cnu k]

Hi,

We currently have 6 different AWX 17.1.0 installs running in old Kubernetes clusters. We're planning to move the AWX instances to latest OpenShift platform and wondering what steps we need to take or any specific issues we need to be aware of.

Would like to upgrade to AWX 19.3.0 and use AWX Operator for installing new instances since Ansible playbook approach is no longer supported.

One of the approaches I saw was to use AWX CLI(https://docs.ansible.com/ansible-tower/latest/html/towercli/examples) to export and import existing configuration and data into the new AWX instance. Would this be an easier approach when compared to PostgresSQL database backup and restore?

Thanks

[Nik Tetteh-Lartey]

I found using sql dump of old env into new env works very easily - let Alex do migrations. Remember to ensure you use correct secret_key for your Postgres or will only find pain.

--
You received this message because you are subscribed to the Google Groups "AWX Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/e5153845-36e6-4516-9d95-850a45b701b5n%40googlegroups.com.

[Nik Tetteh-Lartey]

Alex = awx 

[Edson Jeancarlo Rueda Socola]

Thanks Nik.. please i could sharing the procedure or we sharing the references links ??

[Антон Невзоров]

In common:

1. Create a backup on the old system, find the secret_key
2. Add secret_key info in the installation yaml file
3. Deploy new AWX install
4. Scale operator and awx to 0, drop DB, create new DB
5. Restore database from backup

If someone need more datailed step-by-step migration manual just let me know

четверг, 8 сентября 2022 г. в 17:50:28 UTC+2, edsonje...@gmail.com:

[Edson Jeancarlo Rueda Socola]

Thanks.

We have right now the ansible awx version 17.1.0 and we want to migrate to a current version in k8s on premise that we have put together. 

According to what you indicate:

1. Create a backup on the old system, find the secret_key --> the backup is using de postgresql backup in the container ?? are there other tool free ??  because I have tried to do it with what is indicated in several forums but it is not fully specified... and an error comes out...
2. Add secret_key info in the installation yaml file -->  yes we have it .. but where do I add it?? Apologies for the perhaps silly question.
3. Deploy new AWX install --> ok.. we use the AWX operator to display the new version. 
4. Scale operator and awx to 0, drop DB, create new DB -->   I don't understand this part, how from awx to zero... does it mean a new installation?
5. Restore database from backup -->  It is not necessary to change any table index or something. It is just restore and that's it? the same name of the BD is kept?

Please, if it wasn't too much trouble, could you help us with a manual or where to locate it? Thank you.

[Антон Невзоров]

My instruction is about minikube setup, but I believe you can apply it on k3s also (with small adjustments).. Spent a few weeks trying to upgrade AWX from version 15 (docker compose) to version 21.5 (minikube) with an existing database migration and encountered a lack of step-by-step documentation on the upgrade process. Unfortunately, the suggested migration playbooks didn't work properly, so I had to do the update process manually. As a result of the upgrade, I got a step-by-step upgrade instruction, which may be useful to others who want to migrate from old version to the new one.  In my case I didn't preform any additional steps for DB migration - just reimport the DB from old one to the new one. 

In common:

1. Create a backup on the old system, find the secret_key
2. Add secret_key info in the installation yaml file
3. Deploy new AWX install
4. Scale operator and awx to 0, drop DB, create new DB
5. Restore database from backup

In details:

1. Make the old DB backup

docker ps <--find the postgresql container name

Check mount points:

docker inspect %postgresql-container-name%

And find the Mounts section:

  "Mounts": [

            {

                "Type": "bind",

                "Source": "/app/.awx/pgdocker/10/data",

                "Destination": "/var/lib/postgresql/data",

 Connect to posgresql container

docker exec -it %postgresql-container-name% /bin/bash

 1.3 Check DB name:

psql -U awx

\l 

 1.4 Create backup for the DB from previous step

From shell:

cd /var/lib/posgtgresql/data/

pg_dump -U awx > backup.sql

 

2. Find the secret_key data from old setup

In docker compose it could be /app/.awx/awxcompose/SECRET_KEY

 3. Change the settings in awx.yaml

---

   apiVersion: awx.ansible.com/v1beta1

   kind: AWX

   metadata:

    name: awx

   spec:

    service_type: nodeport

    nodeport_port: 30080

    secret_key_secret: custom-awx-secret-key

 ---

   apiVersion: v1

   kind: Secret

   metadata:

    name: custom-awx-secret-key

    namespace: awx

   stringData:

    secret_key: <KEY FROM OLD SECRET_KEY>

   type: Opaque

 3. Deploy the AWX

 4. Compare the SECRET_KEY info in the pod awx-web

kubectl get pods

kubectl exec -it awx-%podname% -c awx-web /bin/bash 

cat /etc/tower/SECRET_KEY

 5. Scale awx operator and awx deployment to 0 execute the commands

kubectl scale deployment awx-operator-controller-manager --replicas=0 

kubectl scale deployment awx --replicas=0

 6. Copy the backup file from old system to the new system (in my case I use SCP). Backup file should be at /app/.awx/pgdocker/10/data on the old system

 7. Find the mount point for minikube setup:

docker  inspect minikube

Find the Mounts section:

"Mounts": [

                   "Source": "/var/lib/docker/volumes/minikube/_data",

And copy the backup file to  /var/lib/docker/volumes/minikube/_data/hostpath-provisioner/awx/postgres-13-awx-postgres-13-0/data

 8. Connect do postgresql-pod (you can find the DB pod name with kubectl get pods command)

kubectl exec -it awx-postgres-13-0 bash -n=awx

 9. Drop old database

psql -U awx -d posgres

Check DBs list

\l

DROP DATABASE awx;

Create new DB

CREATE DATABASE awx

 10. Restore the backup form shell

psql -U awx < backup.sql

And check DB structure (amount of tables and etc):

Psql -U awx

\d

 11. Scale deployments back 

kubectl scale deployment awx-operator-controller-manager --replicas=1 

kubectl scale deployment awx --replicas=1

 12. Check logs in awx-web container

kubectl logs -f awx-%podname% -c awx-web -n=awx

 13. Check application, job history and etc.

If you have any Q about these instruction - ask :)

понедельник, 17 октября 2022 г. в 17:43:51 UTC+2, edsonje...@gmail.com:

[Антон Невзоров]

May be you can put that info on the migration page?

---------- Пересылаемое сообщение ---------
От: Антон Невзоров <Неизвестный>
Дата: понедельник, 17 октября 2022 г., 17:59:16 UTC+2
Тема: Re: [awx-project] Migrating and upgrading from AWX 17.1.0 to AWX 19.x with Operator approach
Кому: AWX Project <Неизвестный>

[Paul garzaro]

I have a question, i've followed your tutorial, but the awx-web keep staying in the state "wait-for-migration". 
I'm trying to do a migration from 17.1 to 22. Is the jump to high ? 

[Paul garzaro]

I've found how to resolve this.

When it's frozen in this state, just run "awx-manage migrate --noinput".

Since the database has already been restored by the pgsql cli, all data is available.

[Michał Chruściel]

Great job! I needed to adjust a couple of things but that was really, really helpful :) thanks!