Database failure

[Billy Ashmall]

Our production environment server (https://tethys.servirglobal.net/) rebooted itself last night.  Upon reboot the tethys db start will not work.  It has been running for over a year, no modifications were made to it.  The error being reported in the psql log file is:

2020-12-02 19:27:17.324 UTC [5427] FATAL:  database files are incompatible with server

2020-12-02 19:27:17.324 UTC [5427] DETAIL:  The data directory was initialized by PostgreSQL version 11, which is not compatible with this version 12.2.

I am a little confused as I believe it was installed originally at version 12.  Do you know how to get Tethys back running?

[swainn]

Hey Billy:

Sorry you are having trouble. To clarify, you are using the PostgreSQL database that is installed by the Tethys conda environment? That's the one that is controlled by the `tethys db` command. The data directory for this database defaults to something like ~/.tethys/psql or ~/.tethys/<env_name>/psql. Is that where you found the log?

I'd recommend you check the version of postgresql installed in Tethys conda environment:

conda list | grep postgres

It may be possible that the package was upgraded during installation of another package (conda is sneaky that way). I've learned to read through the list of upgrades, downgrades, etc when installing packages.

If instead you are using the postgresql database installed by the system (i.e. apt-get) it is possible that it was upgraded during a server update/upgrade. Do you know why the server rebooted itself? Was it performing updates?

--

Nathan

[Billy Ashmall]

Yes, it's the one installed /home/tethys/miniconda/envs/tethys/bin/postgres "-D" "/home/tethys/.tethys/psql/data" "-p" "5436"

It says it is postgresql 12.2 , should i just downgrade it to 11?

We are not sure why the servers were rebooted, it seems possible that the entire cluster went down, there were several machines that we could tell were rebooted.

[Scott Christensen]

Yes, downgrading your postgres package to the same one that the database was created with should enable you to use the tethys db command to start your database again. You should be able to do that with something like:

conda install -c conda-forge postgresql=11

 

Just a reminder that the production documentation recommends not using the conda version of the database: "For a production installation we recommend that you DO NOT use the database that is installed via conda. Instead install PostgreSQL using the system package manager..."

(http://docs.tethysplatform.org/en/stable/installation/production/system_dependencies.html#postgresql)

You may want to consider migrating your database to a dedicated DB server. That will give you a more stable production environment.

--
You received this message because you are subscribed to the Google Groups "Tethys Platform" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tethysplatfor...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tethysplatform/4740f485-15eb-4b34-9dc7-71a33fa7921dn%40googlegroups.com.

[Billy Ashmall]

I tried that and it resolved for several hours and then spit out pages and pages of conflicts.  I copied as many as the terminal would cache but i hesitate to post it in here as it would be a super long post for no apparent reason...