[postgis-users] Help to upgrade postgresql10 with postgis 2.5

pham lan

unread,

Mar 15, 2023, 5:00:22 PM3/15/23

to PostGIS Users Discussion

Hello,

I have very less experience with postgres and postgis. However i receive a task to upgrade an old database which has postgresql10 with postgis 2.5. The database has some extensions: postgis, postgis_sfcgal, postgis_topology, raster and have a lot of materialized views which depends on functions on postgis and sfcgal libraries. Could someone please instruct me to upgrade my postgres DB to a newer postgres and postgis version without having to drop all those materialized views? Idealy to postgis 3.x?

Thanks in advance.

Best regards

Lan Pham

Regina Obe

unread,

Mar 16, 2023, 9:15:15 AM3/16/23

to PostGIS Users Discussion

It depends on what platform you are on.

Can you output the following from your postgis enabled databases:

SELECT postgis_full_version(), version();

And send us the output of what each says. That will give a clue of at least what platform you are running.

You’ll need to run this on each database you have with postgis installed on, as it is possible to have 2 versions of postgis installed in separate databases.

As to whether you’d need to drop and recreate any materialized views, it would depend on what functions they are using. If they are using deprecated or removed functions, then eventually you will need to drop and recreate, but you could do that at a later time. The PostGIS 3+ upgrade will notify you of those issues, but generally can just rename the functions in use, so you can drop and recreate at a more convenient time.

For materialized views that take a long time to build, I generally build them under a new name, and do a swap after the new one has been built. That would reduce the downtime from hours to 1-2 minutes.

Thanks,

Regina

pham lan

unread,

Mar 17, 2023, 3:56:30 AM3/17/23

to PostGIS Users Discussion

Hi Regina,

Thank you for your reply. Below is the output of the command. Do you know a good way to migrate all the definitions of materialized views because we have a lot of them and they are quite complex?

# SELECT postgis_full_version(), version();
postgis_full_version
| version
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------+----------------------------------------------------------------------------------------------------------
POSTGIS="2.5.5" [EXTENSION] PGSQL="100" GEOS="3.9.2-CAPI-1.14.3" SFCGAL="1.4.1" PROJ="Rel. 7.2.1, January 1st, 2021" GDAL="GDAL 3.2.3, released 2021/04/27" LIBXML="2.9.7" LIBJSON="0.13.1" LIBPROTOBUF="
1.3.0" TOPOLOGY RASTER | PostgreSQL 10.23 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 8.5.0 20210514 (Red Hat 8.5.0-15), 64-bit

Thanks,

Lan

_______________________________________________
postgis-users mailing list
postgi...@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/postgis-users

Regina Obe

unread,

Mar 17, 2023, 4:25:58 AM3/17/23

to PostGIS Users Discussion

You’d probably want to upgrade PostgreSQL too since PostgreSQL 10 is EOL’d.

Below are some instructions I had written for Centos. Should be more or less the same for Redhat.

https://www.bostongis.com/blog/index.php?/archives/278-Using-pg_upgrade-to-upgrade-PostgreSQL-9.6-PostGIS-2.4-to-PostgreSQL-15-3.3-on-Yum.html

pham lan

unread,

Mar 17, 2023, 5:27:42 AM3/17/23

to PostGIS Users Discussion

Yes, actually I followed your instructions.

After drop extension postgis_sfcgal whichs required me to drop cascade 13 materialized views, the upgrade to postgres15 went well.

Afterward, I wanted to update the postgis extension:

ALTER EXTENSION postgis UPDATE;

it requests me to drop cascade around 200 materialized views. For the test DB, I did that, the update to postgis 3.3 went well but afterward, the vacuum step failed.

In the end, I talked to the teams and they can not afford to have those materialized views deleted and then being created again by them.

That's why I wanna ask if there is a simple way of migrating the materialized views? I am not familiar with this.

Thanks,

Lan

pham lan

unread,

Mar 18, 2023, 1:21:21 AM3/18/23

to PostGIS Users Discussion

Hi Regina,

So I tried to pg_upgrade at least from postgresql 10 (postgis 2.5) to postgresql 12 (postgis 2.5) and get the following error. Do you have any idea why? (I tried this several times, I even pg_basebackup to another server with postgresql 10 and pg_upgrade to postgres12 but got the same error)

> /usr/pgsql-12/bin/pg_upgrade -b /usr/pgsql-10/bin/ -B /usr/pgsql-12/bin/ -d /var/lib/pgsql/10/data/ -D /var/lib/pgsql/12/data/ -p 5432 -P 5433
Performing Consistency Checks
-----------------------------
Checking cluster versions ok
Checking database user is the install user ok
Checking database connection settings ok
Checking for prepared transactions ok
Checking for system-defined composite types in user tables ok
Checking for reg* data types in user tables ok
Checking for contrib/isn with bigint-passing mismatch ok
Checking for tables WITH OIDS ok
Checking for invalid "sql_identifier" user columns ok
Creating dump of global objects ok
Creating dump of database schemas
ok
Checking for presence of required libraries ok
Checking database user is the install user ok
Checking for prepared transactions ok
Checking for new cluster tablespace directories ok

If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.

Performing Upgrade
------------------
Analyzing all rows in the new cluster
*failure*

Consult the last few lines of "pg_upgrade_utility.log" for
the probable cause of the failure.
Failure, exiting
1 postgres@host:~> cat pg_upgrade_utility.log

-----------------------------------------------------------------
pg_upgrade run on Sat Mar 18 06:16:00 2023
-----------------------------------------------------------------

command: "/usr/pgsql-12/bin/pg_dumpall" --host /var/lib/pgsql --port 5432 --username postgres --globals-only --quote-all-identifiers --binary-upgrade -f pg_upgrade_dump_globals.sql >> "pg_upgrade_utility.log" 2>&1

command: "/usr/pgsql-12/bin/vacuumdb" --host /var/lib/pgsql --port 5433 --username postgres --all --analyze >> "pg_upgrade_utility.log" 2>&1
vacuumdb: vacuuming database "postgres"
vacuumdb: vacuuming database "template1"
WARNING: terminating connection because of crash of another server process
DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT: In a moment you should be able to reconnect to the database and repeat your command.
vacuumdb: error: vacuuming of table "pg_catalog.pg_proc" in database "template1" failed: server closed the connection unexpectedly
This probably means the server terminated abnormally

before or while processing the request.

Thanks,

Lan

Regina Obe

unread,

Mar 18, 2023, 2:04:03 AM3/18/23

to PostGIS Users Discussion

I’ve never seen that error before.

It might be some OOM killer getting in the way as it’s failing on vacuuming of pg_catalog.pg_proc in template1, which is a really strange place to be failing since the template database have pretty much nothing, unless you installed some extension in them.

What does output – is it low?

free –h

It might be something in your postgresql.conf or postgresql.auto.conf like your maintenance_work_mem is set too high for your server.

Since you are doing non-link mode pg_upgrade, you should have 2 instances off pg now. I think if it got to the point of vacuumdb it’s already loaded all the data in the PG12 instance and you can check if everything is there.

Can you connect to your PostgreSQL 12, you might have to manually start it up

And then see if you can run below without it crashing. That is essentially what vacuumdb does:

psql -p 5433 -d template1 -c "vacuum (analyze,verbose);"

Also check what extensions you have installed in template1.

psql -p 5433 -d template1 -c "SELECT * FROM pg_extension;"

For template1, there is usually only one extension plpgsql.

If you have more, you might want to drop the others to rule out it’s some extension getting in the way.

pham lan

unread,

Mar 18, 2023, 3:37:33 AM3/18/23

to PostGIS Users Discussion

Hi Regina,

The postgresql.conf and pg_hba.conf for postgres12 are the default one. I just changed the port to 5433 and all authentication to trust. Nothing special.

Please find in attachments those files.

Swap: 2.0Gi 95Mi 1.9Gi

Let me know if you see something wrong.

Thanks,

Lan

vacuum.txt

pg_hba.conf

postgresql.conf

Regina Obe

unread,

Mar 18, 2023, 9:28:52 AM3/18/23

to PostGIS Users Discussion

Don’t see anything out of the ordinary.

You don’t have a postgresql.auto.conf file or is it empty? The custom settings might be in that one. The auto is set when you use

ALTER SYSTEM

and that would generally be copied over from your old instance.

Also are all your databases on PG12 and full as I would expect, since I think the failure happened after all the data was copied over already.

The fact vacuum(full, analyze) didn’t fail is good and perhaps suggests nothing to worry about.

Whatever killed vacuumdb sounds like another process that was running at the same time. The fact it happened even on the other server you are testing might be just something you consistently install. I would try the vacuum on each of your databases to see if any cause a crash.

Regina Obe

unread,

Mar 18, 2023, 9:44:03 AM3/18/23

to PostGIS Users Discussion

I should add that ultimately it isn’t the vacuuming crashing, but something else running at the same time.

This article details how to find the root cause

https://pganalyze.com/docs/log-insights/server/S1

I have heard on occasion postgis raster crashing because of a bad library dependency (but this usually happens when on create extension of postgis (or postgis_raster) or running a raster process. I suppose it could happen on backend load of postgis libraries.

If you don’t need postgis raster, then the easy fix for that is just to

After you upgrade each database in your PG12 that has postgis installed with :

SELECT postgis_extensions_upgrade();

Do

DROP EXTENSION postgis_raster;

Dropping will fail if you have raster tables, so no need for concern accidentally dropping data.

From: Regina Obe [mailto:l...@pcorp.us]

Sent: Saturday, March 18, 2023 9:29 AM
To: 'PostGIS Users Discussion' <postgi...@lists.osgeo.org>

Subject: RE: [postgis-users] Help to upgrade postgresql10 with postgis 2.5

Don’t see anything out of the ordinary.

You don’t have a postgresql.auto.conf file or is it empty? The custom settings might be in that one. The auto is set when you use

ALTER SYSTEM

and that would generally be copied over from your old instance.

Also are all your databases on PG12 and full as I would expect, since I think the failure happened after all the data was copied over already.

The fact vacuum(full, analyze) didn’t fail is good and perhaps suggests nothing to worry about.

Whatever killed vacuumdb sounds like another process that was running at the same time. The fact it happened even on the other server you are testing might be just something you consistently install. I would try the vacuum on each of your databases to see if any cause a crash.

pham lan

unread,

Mar 23, 2023, 7:41:03 AM3/23/23

to PostGIS Users Discussion

Hi,

So my last test was to drop extension postgis cascade (from one test system) which drops more than 200 materialized views. Only afterward the pg_upgrade works without being crashed.

So it seems the problem is the postgis extension, though the version on postgres10 and postgres 12 are both postgis 2.5. I don't understand why it caused the problem then. Do you have any idea?

If so, in this case, there is no other way for us to upgrade postgres besides dropping those views and recreate them again, right?

Thanks,

Lan

Regina Obe

unread,

Mar 23, 2023, 5:02:33 PM3/23/23

to PostGIS Users Discussion

Well if you drop cascade postgis, you may be dropping table columns as well. So that is not a good idea.

You could try upgrading your PostgreSQL 10 to 3.something first and then upgrade that.

pham lan

unread,

Mar 26, 2023, 12:00:13 AM3/26/23

to PostGIS Users Discussion

I did try to upgrade to 3. and it also failed unless I drop many maetirialized views. So I did drop all views/matviews and upgraded directly to postgres15 with postgis3.3 following the link you sent, then recreated these views/matviews. All look fine.

However the last step is to vacuum the DBs and it throws me this error. Do you have any idea what causes that and is it harmful or we can use the DB like it is?

postgres@lgngeodmsv04:~> /usr/pgsql-15/bin/vacuumdb --all --analyze-in-stages --verbose

vacuumdb: error: processing of database "geodm" failed: ERROR: transform: Point outside of projection domain (2050)

Reply all

Reply to author

Forward