AWX Applications - Internal server error

Elia Curti

unread,

Sep 21, 2022, 6:15:55 AM9/21/22

to AWX Project

Hello,

I've deployed in my aks cluster AWX with awx-operator 0.28.0

It's not a clean installation, I'm using a db from a previous AWX version (quite old version).

When I click on Application menù I receive an internal server error:

From container "awx-web" I see this log error:

2022-09-21 10:08:28,819 ERROR    [2f9c4cc6f70d45b4af66e8983b5f0624] django.request Internal Server Error: /api/v2/applications/
Traceback (most recent call last):
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/cryptography/fernet.py", line 120, in _verify_signature
    h.verify(data[-32:])
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/cryptography/hazmat/primitives/hmac.py", line 69, in verify
    ctx.verify(signature)
  File "/var/lib/awx/venv/awx/lib64/python3.9/site-packages/cryptography/hazmat/backends/openssl/hmac.py", line 76, in verify
    raise InvalidSignature("Signature did not match digest.")

cryptography.exceptions.InvalidSignature: Signature did not match digest.

Is it possible that some tables as old data with a different signature? Can I clean this tables and re-insert data with new AWX version?

Thank you,

Elia

HG

unread,

Sep 21, 2022, 9:03:52 AM9/21/22

to AWX Project

Hi Elia

Have you set the awx-secret-key as it was on your old environment?

It must be the same as far as I have discovered.
When I import data from an old environment in a new environment I have to edit that secret key.

Regards Hans

Op woensdag 21 september 2022 om 12:15:55 UTC+2 schreef elia....@gmail.com:

Elia Curti

unread,

Sep 21, 2022, 9:25:04 AM9/21/22

to AWX Project

Hi Hans,

no I haven't set the "awx-secret-key" as it was on my old environment.

I've been working with the new AWX environment and the new secret key for months.
I configured the new AWX with the new secret-key I think that now I can no longer replace it with the one recovered from the old environment.

I hope it will be possible to delete all the data related to the applications (I don't know in which tables) and re-enter everything with the new secret-key.
It's possible?

Regards,

Elia

HG

unread,

Sep 21, 2022, 10:15:31 AM9/21/22

to awx-p...@googlegroups.com

Well it is confusing for me as well.

But when I import data from my current production environment into a newly setup one.

I do the following.

1. Drop de awx database in postgres.

2. Import the data into postgres

3. Set awx-secret-key.

This worked until 0.21 of the awx operator.

After that I have had no luck so far.

--
You received this message because you are subscribed to a topic in the Google Groups "AWX Project" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/awx-project/xX4vUBM-iEI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to awx-project...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/ff4069fb-e8e8-4ca4-b16c-196140a4a871n%40googlegroups.com.

Elia Curti

unread,

Sep 21, 2022, 11:04:59 AM9/21/22

to AWX Project

I've checked in my old environment (AWX 3.0.1.0) and "awx-secret-key" secret it's differently formatted.

I can't find "secret_key" as in my new enviroment... It's like this:

apiVersion: v1
data:
admin_password: xxx
confd_contents: xxx
pg_password: xxx
rabbitmq_erlang_cookie: xxx
rabbitmq_password: xxx
kind: Secret

...

Which is the "secret_key" data?

Elia

HG

unread,

Sep 23, 2022, 1:21:25 AM9/23/22

to AWX Project

HI Elia,

What do you get when you do : kubectl get secret -n <namespace>

Is there no awx-secret-key?

Regards Hans

Op woensdag 21 september 2022 om 17:04:59 UTC+2 schreef elia....@gmail.com:

Elia Curti

unread,

Sep 23, 2022, 2:56:21 AM9/23/22

to AWX Project

Hi Hans,

no there isn't. In this older AWX version there is only a secret "secret_key" and it's not the same secret.

Regards,

Elia

Elia Curti

unread,

Sep 26, 2022, 3:39:51 AM9/26/22

to AWX Project

Anyone from the staff who can suggest me how to clean up the db to fix "Application" error?

Thank you very much,

Elia

HG

unread,

Sep 26, 2022, 5:39:31 AM9/26/22

to AWX Project

Hello Elia

Although I am not staff, perhaps you need to elaborate on what exactly you are doing.
You have deployed 0.28 but it is not a new installation.

What does that mean?

Regards Hans-Peter

Op maandag 26 september 2022 om 09:39:51 UTC+2 schreef elia....@gmail.com:

HG

unread,

Sep 26, 2022, 5:40:24 AM9/26/22

to AWX Project

Is it an external database?

Op maandag 26 september 2022 om 11:39:31 UTC+2 schreef HG:

Elia Curti

unread,

Sep 26, 2022, 6:05:45 AM9/26/22

to AWX Project

Hi Hans,

I mean that I restored an external database from a previous AWX version.

Maybe some secret in the database should have a different signature but only for "Application" I have the issue.

HG

unread,

Sep 26, 2022, 6:11:33 AM9/26/22

to awx-p...@googlegroups.com

But when you installed 0.28, did you check the awx-operator-controller-manager logging?

I discovered that some tables have additional columns in new versions.

Regards Hans-Peter

Op ma 26 sep. 2022 om 12:05 schreef Elia Curti <elia....@gmail.com>:

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/72423903-a7eb-48b1-bffb-1c2ccce86b99n%40googlegroups.com.

HG

unread,

Sep 26, 2022, 8:31:58 AM9/26/22

to awx-p...@googlegroups.com

Hi,

But have you watched the logging of awx-controller-manager after you did the : make deploy command?

What kind of kubernetes do you use?

Regards Hans

Op ma 26 sep. 2022 om 12:05 schreef Elia Curti <elia....@gmail.com>:

Hi Hans,

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/72423903-a7eb-48b1-bffb-1c2ccce86b99n%40googlegroups.com.

Elia Curti

unread,

Sep 26, 2022, 8:49:51 AM9/26/22

to AWX Project

Yes, I saw the awx-controller-manager log but I didn't notice anything strange.

K8s version: 1.22.6

regards Elia

HG

unread,

Sep 26, 2022, 9:12:42 AM9/26/22

to AWX Project

I think that you need to:

1. install the old version of awx-operator and have that used the database

2. upgrade it to 0.28

IMHO you need to have awx-secret-key and set it just as in the old environment.

Op maandag 26 september 2022 om 14:49:51 UTC+2 schreef elia....@gmail.com:

Elia Curti

unread,

Sep 26, 2022, 9:23:49 AM9/26/22

to AWX Project

As I show you in previous messages in my old enviroment there is no secret key. Or it is in a different secret... there is no "secret_key" field.

Elia

HG

unread,

Sep 26, 2022, 9:43:40 AM9/26/22

to AWX Project

What is the old awx-operator version?

Op maandag 26 september 2022 om 15:23:49 UTC+2 schreef elia....@gmail.com:

Elia Curti

unread,

Sep 26, 2022, 9:50:22 AM9/26/22

to AWX Project

I've checked in my old environment (AWX 3.0.1.0) and "awx-secret-key" secret it's differently formatted.

I can't find "secret_key" as in my new enviroment... It's like this:

apiVersion: v1
data:
admin_password: xxx
confd_contents: xxx
pg_password: xxx
rabbitmq_erlang_cookie: xxx
rabbitmq_password: xxx
kind: Secret

...

Which is the "secret_key" data?

Elia

Elia Curti

unread,

Sep 27, 2022, 9:38:42 AM9/27/22

to AWX Project

I've found a solution clearing "Applications" tables on awx database.

Main table is "main_oauth2application" and first related tables "main_oauth2accesstoken", "oauth2_provider_refreshtoken" only for the records affected.

Now "Applications" works fine, Internal server error disappeared.

Regards,

Elia

HG

unread,

Sep 27, 2022, 10:11:09 AM9/27/22

to awx-p...@googlegroups.com

Hi

Super.

Did you find it somewhere on the web?

If so can you put it here for the reference?

Regards Hans

Op di 27 sep. 2022 om 15:38 schreef Elia Curti <elia....@gmail.com>:

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/49112a24-3e11-4b32-a7c8-bef904c4f1e3n%40googlegroups.com.

Elia Curti

unread,

Sep 27, 2022, 11:42:03 AM9/27/22

to AWX Project

No. I opened awx database and searched the main table for "Applications"... then trying to delete records from there I've discovered related tables.

Regards,

Elia

AWX Project

unread,

Sep 28, 2022, 2:53:04 PM9/28/22

to AWX Project

Elia, nice digging, could you open an issue on AWX detailing the internal 500 error and how you resolved it? at the very least there should be better error messages in this scenario

Thanks!

AWX Team

Elia Curti

unread,

Sep 29, 2022, 4:42:26 AM9/29/22

to AWX Project

Ok. Done.

Regards,

Elia

Andrei Mihai

unread,

Nov 9, 2022, 9:02:28 AM11/9/22

to AWX Project

nice one. Thanks Elia. This helped.

On thing before I go on. Could you please post your github issue number here, for others to find and refer to? I couldn't really find any issues raised for this on the awx-operator or awx github issues page.

I had the same issue as you. Only difference is that this is a new deployment. I did not migrate the database prior to it. Still using an external database.

My issue was generated by something in the activity stream table for activities
/api/v2/activity_stream/140/ and /api/v2/activity_stream/139/

There was a token created at some point for the bash application. I suspect that this is a token I created for myself, to run API calls that…went bad, somehow.

To resolve this, I deleted all relations for the toke from the following tables:


main_activitystream_o_auth2_application

main_oauth2accesstoken

main_oauth2application 

oauth2_provider_refreshtoken

main_activitystream_o_auth2_access_token

This is also the API information as well as the token entry from the DB. Nothing looks bad…Also note that I had just upgraded AWX today and I was accessing applications after the upgrade. Before today, I can’t remember when I accessed that page for the last time, but it was probably on the 21nd of October. :)


HTTP 200 OK Allow: GET, HEAD, OPTIONS Content-Type: application/json Vary: Accept X-API-Node: loki-7cbd4888d6-hph4k X-API-Product-Name: AWX X-API-Product-Version: 21.8.0 X-API-Time: 0.034s  {     "id": 139,     "type": "activity_stream",     "url": "/api/v2/activity_stream/139/",     "related": {         "actor": "/api/v2/users/1/"     },     "summary_fields": {         "actor": {             "id": 1,             "username": "admin",             "first_name": "",             "last_name": ""         }     },     "timestamp": "2022-10-21T11:30:45.043907Z",     "operation": "create",     "changes": {         "name": "bash",         "description": "",         "client_type": "confidential",         "redirect_uris": "https://ansible.aimcorp.co.uk/",         "authorization_grant_type": "authorization-code",         "skip_authorization": false,         "organization": "Default-1",         "id": 1     },     "object1": "o_auth2_application",     "object2": "",     "object_association": "",     "action_node": "loki-dd4c549f4-pngzh",     "object_type": "" }

HTTP 200 OK Allow: GET, HEAD, OPTIONS Content-Type: application/json Vary: Accept X-API-Node: loki-7cbd4888d6-hph4k X-API-Product-Name: AWX X-API-Product-Version: 21.8.0 X-API-Time: 0.036s  {     "id": 140,     "type": "activity_stream",     "url": "/api/v2/activity_stream/140/",     "related": {         "actor": "/api/v2/users/1/"     },     "summary_fields": {         "actor": {             "id": 1,             "username": "admin",             "first_name": "",             "last_name": ""         }     },     "timestamp": "2022-10-21T11:31:30.549299Z",     "operation": "create",     "changes": {         "description": "",         "application": "bash",         "scope": "write",         "id": 1,         "token": "************"     },     "object1": "o_auth2_access_token",     "object2": "",     "object_association": "",     "action_node": "loki-dd4c549f4-pngzh",     "object_type": "" }

 id |             token              | access_token_id | application_id | user_id |            created            |            updated            | revoked 
----+--------------------------------+-----------------+----------------+---------+-------------------------------+-------------------------------+---------
  1 | CyiHOuzUhy4y25L4PnKRmFzuTnEbFX |               1 |              1 |       1 | 2022-10-21 13:31:30.568923+02 | 2022-10-21 13:31:30.568939+02 |

Elia Curti

unread,

Nov 9, 2022, 9:52:29 AM11/9/22

to AWX Project

Github: https://github.com/ansible/awx/issues/12978

Regards,

Elia

Andrei Mihai

unread,

Nov 9, 2022, 10:39:05 AM11/9/22

to awx-p...@googlegroups.com

Ah. It was Closed. That's why i did not find it.

Mille grazie.

Get Outlook for Android

From: awx-p...@googlegroups.com <awx-p...@googlegroups.com> on behalf of Elia Curti <elia....@gmail.com>
Sent: Wednesday, November 9, 2022 3:52:28 PM
To: AWX Project <awx-p...@googlegroups.com>
Subject: Re: [awx-project] Re: AWX Applications - Internal server error

To view this discussion on the web visit https://groups.google.com/d/msgid/awx-project/cb509853-ee1e-4b29-912d-7e1d011d2645n%40googlegroups.com.

Reply all

Reply to author

Forward