AWX Deployment does not connect to RDS Database

812 views
Skip to first unread message

Adrian Cortinas

unread,
Apr 28, 2021, 6:49:38 PM4/28/21
to AWX Project
Hi Team.

My Environment:
Amazon EKS 1.19 w/Managed hosts
Amazon RDS Postgres 12.5R1
AWX Operator 0.8.0

AWX deployment yml file:
##File.yml##
---
apiVersion: v1
kind: Secret
metadata:
  name: awx-postgres-configuration
  namespace: default
stringData:
  port: "5432"
  database: "awx"
  username: "postgres"
  password: "SomeStrongPassword"
type: Opaque

---
kind: AWX
metadata:
  name: awx
  namespace: default
spec:
  tower_ingress_type: Ingress
  tower_image_pull_policy: Always
  tower_admin_user: admin
  tower_admin_email: email_at_address.com

##File.yml##

Issue:
I created an EKS cluster on a private network, created a corresponding RDS database, deployed AWX operator to the eks cluster successfully, but whenever I deploy AWX it simply doesn't seem to connect to the database and doesn't create the database objects. I tested the same procedure in a personal (full access and privs) AWS environment and had no issues whatsoever, but this other environment ( I am admin but not AWS environment owner) everything but the object creation succeeds.

I see the pods and services up and running:
$ kubectl get pods
NAME                          READY   STATUS    RESTARTS   AGE
awx-cf###88b6-jjrlg           4/4     Running   0          3m23s
awx-operator-f###499d-mwzpz   1/1     Running   0          24h

$ kubectl get svc
NAME                   TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
awx-operator-metrics   ClusterIP   172.##.138.243   <none>        8383/TCP,8686/TCP   24h
awx-service            NodePort    172.##.88.126    <none>        80:32595/TCP        3m28s
kubernetes             ClusterIP   172.##.0.1       <none>        443/TCP             24h

I thought that maybe connectivity to the database is not working from within the pods, but that is working fine:

kubectl exec -it pod/awx-cf###88b6-jjrlg -c awx-task /bin/bash

psql -U postgres postgresql://awx-tower.some.location.rds.amazonaws.com/awx
Password for user postgres: 
psql (12.5)
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

awx=> \dt
Did not find any relations.

I checked the operator log and see no failed tasks (attached).

Any clue what am I be doing wrong or missing? Is there a way to debug the deployment to validate database connectivity from the ansible playbook? any ideas what else I can test/try?

Thanks

awx-42821-1.log

Adrian Cortinas

unread,
May 4, 2021, 10:56:12 AM5/4/21
to AWX Project
well... operator 0.9.0 made the trick, that version fixes connectivity to external (unmanaged) PostgreSQL databases.

Aaron Spiller

unread,
May 6, 2021, 10:00:34 AM5/6/21
to AWX Project
I'm having similar issues, but have upgraded to 0.9.0 with no success:

Amazon RDS Postgres 12.5R1
minikube 1.12.1 (but will be running on rancher in production)
AWX Operator 0.9.0

After the awx deploy, I see the following in the logs:

django.db.utils.OperationalError: SSL error: certificate verify failed
FATAL:  no pg_hba.conf entry for host "x.x.x.x", user "user_i_created", database "awx", SSL off

where x.x.x.x is the IP of the host I am not and not the DB. I have used this RDS instance for my existing (16.0.0) implementation so the host/user/db/password are all correct. 
Not sure where else I can look or what I can set to get this working

Adrian Cortinas

unread,
May 6, 2021, 1:33:58 PM5/6/21
to AWX Project
I didn't have that particular issue, in my case it wouldn't even try to connect to the database. For the new operator version to work I made some changes to my yaml file, in bold below. Did you try those? I also made sure that the security group in RDS accepts connections from the k8s cluster network.

---
apiVersion: v1
kind: Secret
metadata:
  name: awx-postgres-configuration
  namespace: default
stringData:
  port: "5432"
  database: "awx"
  username: "postgres"
  password: "SomeStrongPassword"
  sslmode: prefer
  type: unmanaged
type: Opaque


Try to login to the database using psql from within the awx-web/task containers, probably there's something else going on in your setup that is not too obvious.

Hope this helps.

Aaron Spiller

unread,
May 6, 2021, 2:57:03 PM5/6/21
to AWX Project
Yes, I have been using those options in bold and they don't help.
I can connect just fine via psql and awx-manage dbshell, that's what is confusing me.

Juan C.

unread,
Dec 21, 2021, 9:54:08 AM12/21/21
to AWX Project
Did you create the database in RDS?

I had the same issue, then I found out the awx-* containers were trying to connect to a database that didn't exist in the RDS host.

Sai Krishna

unread,
Feb 16, 2023, 6:54:52 AM2/16/23
to AWX Project
Hi folks,

I am trying to create AWX tower in EKS fargate and would like to use the external rds database connection. 

And following the below GitHub documentation

As was able to successfully run the awx-operator control manager.

But when I tried to deploy the AWX yaml file it creates a awx-postgres pod which will be in pending state with pod doesn't support fargate and when I deep dive into it it's looking for persistantvolume and persistent volume claim.


Questions here:
When I am trying to say to use external db, why it's again deploying db pod.

If it's mandatory that even after mentioning to use external db, it needs the and db pod running. 

Please help me out here i struggling to find the information
Reply all
Reply to author
Forward
0 new messages