[GENERAL] pg_upgrade: "pg_ctl failed to start the new server"

Evan D. Hoffman

unread,

Jun 13, 2012, 11:19:41 AM6/13/12

to

I'm trying to upgrade Postgres 9.0 to 9.1 with pg_upgrade. Both
versions are installed from the PGDG Yum repo:

-bash-4.1$ /usr/pgsql-9.0/bin/postgres -V
postgres (PostgreSQL) 9.0.8
-bash-4.1$ /usr/pgsql-9.1/bin/postgres -V
postgres (PostgreSQL) 9.1.4

I can successfully start and connect to both 9.0 and 9.1. When I run
pg_upgrade, it fails with the error "pg_ctl failed to start the new
server," however it apparently does start the new server (output
below). Any ideas? This is a freshly installed CentOS 6.2 machine
with the 9.0 DB copied over from another machine via streaming
replication (which I have since ended and the server is now
standalone).

Thanks.

# su - postgres
-bash-4.1$ ps ax | grep postg
4354 pts/0 S 0:00 su - postgres
4389 pts/0 S+ 0:00 grep postg
-bash-4.1$ /usr/pgsql-9.1/bin/pg_upgrade -c -b /usr/pgsql-9.0/bin/ -B
/usr/pgsql-9.1/bin/ -d /var/lib/pgsql/9.0/data/ -D
/var/lib/pgsql/9.1/data/
Performing Consistency Checks
-----------------------------
Checking current, bin, and data directories ok
Checking cluster versions ok
Checking database user is a superuser ok
Checking for prepared transactions ok
Checking for reg* system oid user data types ok
Checking for contrib/isn with bigint-passing mismatch ok

pg_ctl failed to start the new server
Failure, exiting
-bash-4.1$ ps ax | grep postg
4354 pts/0 S 0:00 su - postgres
4445 pts/0 S 0:00 /usr/pgsql-9.1/bin/postgres -D
/var/lib/pgsql/9.1/data -p 5432 -b
4446 ? Ss 0:00 postgres: logger process
4448 ? Ss 0:00 postgres: writer process
4449 ? Ss 0:00 postgres: wal writer process
4450 ? Ss 0:00 postgres: stats collector process
4453 pts/0 S+ 0:00 grep postg

--
Sent via pgsql-general mailing list (pgsql-...@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-general

Bruce Momjian

unread,

Jun 13, 2012, 10:26:55 PM6/13/12

to

On Wed, Jun 13, 2012 at 11:19:41AM -0400, Evan D. Hoffman wrote:
> I'm trying to upgrade Postgres 9.0 to 9.1 with pg_upgrade. Both
> versions are installed from the PGDG Yum repo:
>
> -bash-4.1$ /usr/pgsql-9.0/bin/postgres -V
> postgres (PostgreSQL) 9.0.8
> -bash-4.1$ /usr/pgsql-9.1/bin/postgres -V
> postgres (PostgreSQL) 9.1.4
>
> I can successfully start and connect to both 9.0 and 9.1. When I run
> pg_upgrade, it fails with the error "pg_ctl failed to start the new
> server," however it apparently does start the new server (output
> below). Any ideas? This is a freshly installed CentOS 6.2 machine
> with the 9.0 DB copied over from another machine via streaming
> replication (which I have since ended and the server is now
> standalone).

Have you looked at the logs from pg_upgrade? Does "pg_ctl -w start"
work? My guess is that the -w (wait) is failing for some reason. We
did improve the -w handling in PG 9.1 so I didn't think there were any
failure cases, but you might have found a new one.

--
Bruce Momjian <br...@momjian.us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Bruce Momjian

unread,

Jun 13, 2012, 10:49:17 PM6/13/12

to

On Wed, Jun 13, 2012 at 10:41:37PM -0400, Evan D. Hoffman wrote:
> Actually I found the solution right after I sent that email (of
> course):
>
> https://wiki-bsse.ethz.ch/download/attachments/55283107/PostgreSQL_9_M
> aintenance_Backup_and_Recovery_final.docx
>
> Has to do with the order in which shared libs are loaded when both 9.0
> and 9.1 are installed. Renaming the config file as described in that
> doc resolved it.

I didn't find out which file you renamed. Why would the server show as
started while pg_ctl -w returned an error? I am guessing that you could
not connect somehow.

Evan D. Hoffman

unread,

Jun 14, 2012, 8:03:49 AM6/14/12

to

Woops. The file that had to be renamed was
/etc/ld.so.conf.d/postgresql-9.0-libs.conf =>
postgresql-9.old-libs.conf . Prior to rename, ldconfig output was:

# ldconfig -p | grep pq
libpqwalreceiver.so (libc6) => /usr/pgsql-9.0/lib/libpqwalreceiver.so
libpqwalreceiver.so (libc6) => /usr/pgsql-9.1/lib/libpqwalreceiver.so
libpq.so.5 (libc6) => /usr/pgsql-9.0/lib/libpq.so.5
libpq.so.5 (libc6) => /usr/pgsql-9.1/lib/libpq.so.5
libpq.so (libc6) => /usr/pgsql-9.0/lib/libpq.so
libpq.so (libc6) => /usr/pgsql-9.1/lib/libpq.so

After the rename:

# cd /etc/ld.so.conf.d
# mv postgresql-9.0-libs.conf postgresql-9.old-libs.conf
# ldconfig
# ldconfig -p | grep pq
libpqwalreceiver.so (libc6) => /usr/pgsql-9.1/lib/libpqwalreceiver.so
libpqwalreceiver.so (libc6) => /usr/pgsql-9.0/lib/libpqwalreceiver.so
libpq.so.5 (libc6) => /usr/pgsql-9.1/lib/libpq.so.5
libpq.so.5 (libc6) => /usr/pgsql-9.0/lib/libpq.so.5
libpq.so (libc6) => /usr/pgsql-9.1/lib/libpq.so
libpq.so (libc6) => /usr/pgsql-9.0/lib/libpq.so

So it appears the problem was inability to connect, although
pg_upgrade reported that it couldn't start the server (I assume
ability to connect is how it determines whether or not the server was
started).

Evan D. Hoffman

unread,

Jun 13, 2012, 10:41:37 PM6/13/12

to

Actually I found the solution right after I sent that email (of course):

https://wiki-bsse.ethz.ch/download/attachments/55283107/PostgreSQL_9_Maintenance_Backup_and_Recovery_final.docx

Has to do with the order in which shared libs are loaded when both 9.0 and 9.1 are installed. Renaming the config file as described in that doc resolved it.

Bruce Momjian

unread,

Jun 15, 2012, 8:01:51 PM6/15/12

to

On Thu, Jun 14, 2012 at 08:03:49AM -0400, Evan D. Hoffman wrote:
> So it appears the problem was inability to connect, although
> pg_upgrade reported that it couldn't start the server (I assume
> ability to connect is how it determines whether or not the server was
> started).

Ah, OK, so it was the connection that failed. I wasn't aware pg_ctl
start could fail in that case (we added PGping), but obviously it can.

The attached patch applied to PG 9.2 and current will give a more
appropriate error message saying it might be a connection problem.

pg_upgrade.diff