<class 'django.db.utils.InternalError'>

[sonorvinh vinh]

Hi Guys, need your input as our netbox seems to be stop working out of the blue. we deployed it on AWS=ECS by using Fargate and RDS Aurora PostgreSQL. it was workin for long time and out of sudden it throwing error.

*************

 Server Error

There was a problem with your request. Please contact an administrator.

The complete exception is provided below:

<class 'django.db.utils.InternalError'>
cannot execute DELETE in a read-only transaction

 Python version: 3.8.8
NetBox version: 2.10.8

If further assistance is required, please post to the NetBox mailing list.

*************

Any one can help ?

[Brian Candler]

> we deployed it on AWS=ECS by using Fargate and RDS Aurora PostgreSQL

Note that AWS has two different services:

- RDS for Postgresql is a real Postgresql database instance, fully managed

- RDS Aurora is an AWS-in-house designed database, which provides Mysql and Postgres "compatible" APIs.  But it's neither Mysql nor Postgres internally, as far as I understand.

It sounds to me like you're coming across some compatibility issue with Aurora, and you'd be better served by RDS for Postgresql.

I think it would be a good idea in any case to configure your system to send out the full backtrace when an exception occurs: configure ADMINS and EMAIL in configuration.py.  This would at least show what Netbox was trying to do at the time the error occurred.  It probably won't help much in fixing the issue with Aurora though.

[sonorvinh vinh]

Hi Brian;

i retrieved these log from the log event in aws. i'm not sure how much it help. the netbox system is working fine with browing and read only, but write access doesnt work.

: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

2021-11-30 14:50:25 UTC::@:[11897]:LOG: received SIGHUP, reloading configuration files

2021-11-30 14:50:25 UTC::@:[11897]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

2021-11-30 14:50:25 UTC::@:[11897]:LOG: parameter "unix_socket_permissions" cannot be changed without restarting the server

2021-11-30 14:50:25 UTC::@:[11897]:LOG: configuration file "/rdsdbdata/config/postgresql.conf" contains errors; unaffected changes were applied

2021-11-30 14:50:25 UTC::@:[11901]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

2021-11-30 14:50:25 UTC::@:[12485]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

2021-11-30 14:50:25 UTC::@:[11899]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

2021-11-30 14:50:25 UTC::@:[12529]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

2021-11-30 14:50:25 UTC::@:[12528]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

2021-11-30 14:50:31 UTC:10.1.8.37(44060):netbox@netboxdb:[27729]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

2021-11-30 14:50:34 UTC:10.1.7.68(59982):netbox@netboxdb:[30343]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

2021-11-30 14:50:41 UTC:10.1.8.37(44148):netbox@netboxdb:[30248]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

2021-11-30 14:50:44 UTC:10.1.7.68(59728):netbox@netboxdb:[23395]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

2021-11-30 14:51:21 UTC:10.1.8.37(44130):netbox@netboxdb:[29509]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

2021-11-30 14:51:34 UTC:10.1.7.68(59940):netbox@netboxdb:[29044]:LOG: skipping missing configuration file "/rdsdbdata/db/postgresql.auto.conf"

----------------------- END OF LOG ----------------------

2021-11-30 19:44:02 UTC:10.1.8.37(40322):netbox@netboxdb:[4476]:ERROR: cannot execute INSERT in a read-only transaction

2021-11-30 19:44:02 UTC:10.1.8.37(40322):netbox@netboxdb:[4476]:STATEMENT: INSERT INTO "ipam_ipaddress" ("created", "last_updated", "custom_field_data", "address", "vrf_id", "tenant_id", "status", "role", "assigned_object_type_id", "assigned_object_id", "nat_inside_id", "dns_name", "description") VALUES ('2021-11-30'::date, '2021-11-30T19:44:02.408383+00:00'::timestamptz, '{"Hostname": "", "Ping": null}', '74.117.12.117/32', NULL, NULL, 'reserved', '', NULL, NULL, NULL, '', 'PAT for labitp[a-a2, d-d2, e-e2,f-f2].lab servers') RETURNING "ipam_ipaddress"."id"

2021-11-30 19:45:13 UTC:10.1.7.68(57708):netbox@netboxdb:[8840]:ERROR: cannot execute INSERT in a read-only transaction

2021-11-30 19:45:13 UTC:10.1.7.68(57708):netbox@netboxdb:[8840]:STATEMENT: INSERT INTO "ipam_ipaddress" ("created", "last_updated", "custom_field_data", "address", "vrf_id", "tenant_id", "status", "role", "assigned_object_type_id", "assigned_object_id", "nat_inside_id", "dns_name", "description") VALUES ('2021-11-30'::date, '2021-11-30T19:45:13.396693+00:00'::timestamptz, '{"Hostname": "", "Ping": null}', '74.117.12.117/32', NULL, NULL, 'active', '', NULL, NULL, NULL, '', 'PAT for labitp[a-a2, d-d2, e-e2,f-f2].lab servers') RETURNING "ipam_ipaddress"."id"

----------------------- END OF LOG ----------------------

[Brian Candler]

I suggest that you either

1. raise this with AWS as an issue with Aurora (which will take however long it takes to resolve); or

2. switch to using real Postgres as your database (which could simply be changing from RDS Aurora to RDS Postgres)

[sonorvinh vinh]

Yes, i'm already bringing this up to AWS support. the postgres DB seems to be in recovery mode.

netboxdb=> select pg_is_in_recovery();

 pg_is_in_recovery

-------------------

 t

(1 row)

[sonorvinh vinh]

problem identfied.. it was due to the AWS DB failover. it messed up the write and reader endpoint.
issue resolved by updated the "DB_host" with the cluster endpoint.

[Brian Candler]

Cool, thank you for reporting the resolution!