[Dspace-tech] flyway migration error

1,037 views
Skip to first unread message

Wally Grotophorst

unread,
Aug 26, 2015, 2:22:50 PM8/26/15
to dspace-tech
Trying to install 5.1 on a 4.1 database...on a new server (as a test)
but using my 4.1 database that's imported into Postgres before I launch
tomcat/dspace.

Starts to make the schema conversion, then bombs. This part of the log
looks like it's trying to tell me the problem:

at org.flywaydb.core.Flyway.migrate(Flyway.java:811)
at
org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:419)
... 21 more
Caused by: org.postgresql.util.PSQLException: ERROR: constraint
"metadatavalue_item_id_fkey" of relation "metadatavalue" does not exist
at
org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103)
at
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836)
at
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)


I saw some comment about this issue in a few earlier posts but couldn't
figure out how to fix this from the info in the message traffic. Is
there a simple SQL fix I can run prior to trying again? Or maybe
something else?

Platform: OSX 10.10.1
tomcat: 7.0.59
postgres: 9.3
dspace (trying 5.1)

- Wally

Wally Grotophorst
Associate University Librarian
George Mason University
Fairfax, Virginia 22030
(703) 993-9005




Wally Grotophorst

unread,
Aug 26, 2015, 2:22:55 PM8/26/15
to Bill Tantzen, dspace-tech
BINGO! A virtual beer for you!

Thanks for the clear statement (no pun intended) of what I needed to
do. Worked like a charm.

- Wally



Bill Tantzen wrote:
> Wally,
> Here's what I did:
>
> dspace=> \d metadatavalue
> Table "public.metadatavalue"
> ...
> ...
> ...
>
> Foreign-key constraints:
> "$1" FOREIGN KEY (item_id) REFERENCES item(item_id)
> "$2" FOREIGN KEY (metadata_field_id) REFERENCES
> metadatafieldregistry(metadata_field_id)
>
> If your Foreign-key constraints look like the above, do the following:
>
> dspace=> ALTER TABLE metadatavalue DROP CONSTRAINT "$1";
> ALTER TABLE
> dspace=> ALTER TABLE metadatavalue ADD CONSTRAINT
> metadatavalue_item_id_fkey FOREIGN KEY(item_id) REFERENCES
> item(item_id);
> ALTER TABLE
>
> Then, try the conversion!
> Cheers,
> Bill
>> ------------------------------------------------------------------------------
>> Dive into the World of Parallel Programming The Go Parallel Website, sponsored
>> by Intel and developed in partnership with Slashdot Media, is your hub for all
>> things parallel software development, from weekly thought leadership blogs to
>> news, videos, case studies, tutorials and more. Take a look and join the
>> conversation now. http://goparallel.sourceforge.net/
>> _______________________________________________
>> DSpace-tech mailing list
>> DSpac...@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>> List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette

Bill Tantzen

unread,
Aug 26, 2015, 2:22:55 PM8/26/15
to Wally Grotophorst, dspace-tech
Wally,
Here's what I did:

dspace=> \d metadatavalue
Table "public.metadatavalue"
...
...
...

Foreign-key constraints:
"$1" FOREIGN KEY (item_id) REFERENCES item(item_id)
"$2" FOREIGN KEY (metadata_field_id) REFERENCES
metadatafieldregistry(metadata_field_id)

If your Foreign-key constraints look like the above, do the following:

dspace=> ALTER TABLE metadatavalue DROP CONSTRAINT "$1";
ALTER TABLE
dspace=> ALTER TABLE metadatavalue ADD CONSTRAINT
metadatavalue_item_id_fkey FOREIGN KEY(item_id) REFERENCES
item(item_id);
ALTER TABLE

Then, try the conversion!
Cheers,
Bill

On Wed, Apr 1, 2015 at 9:47 AM, Wally Grotophorst <wal...@gmu.edu> wrote:

Peter Dietz

unread,
Aug 26, 2015, 2:31:28 PM8/26/15
to Wally Grotophorst, dspace-tech
Virtual Beer for all as well. I've just ran into this, (Tim pointed me to it from IRC).

My version of the SQL, wrapping in a transaction, to be safe, is:

BEGIN;

ALTER TABLE metadatavalue DROP CONSTRAINT "$1"; 
ALTER TABLE metadatavalue ADD CONSTRAINT metadatavalue_item_id_fkey FOREIGN KEY(item_id) REFERENCES item(item_id); 

COMMIT;



We're not sure of what the cause is. 
12:47 PM <tdonohue> yea, I'm not sure either. :)  It's definitely a "glitch" that happens sometimes (since others have seen it too). But, according to the PostgreSQL docs, these contraints are SUPPOSED to be named [table]_[column]_fkey
12:47 PM <tdonohue> So, it seems like something has changed in Postgres, or Postgres doesn't always follow it's own "rules"

________________
Peter Dietz
Longsight
www.longsight.com
pe...@longsight.com
p: 740-599-5005 x809

Tim Donohue

unread,
Aug 26, 2015, 2:31:30 PM8/26/15
to Peter Dietz, Wally Grotophorst, dspace-tech
Hi all,

Just a note to say I've now logged this as a bug:
https://jira.duraspace.org/browse/DS-2577

In that ticket, I've tracked down the area of the code that makes the
assumption on the name of a PostgreSQL constraint (but it makes that
assumption based on the default PostgreSQL naming scheme for
constraints). So, we may just need to enhance that area of the code to
no longer assume any particular name for a PostgreSQL constraint.

In the meantime, the "workaround" is also documented in that ticket
(thanks to you all in this thread!).

- Tim

On 5/14/2015 12:04 PM, Peter Dietz wrote:
> Virtual Beer for all as well. I've just ran into this, (Tim pointed me
> to it from IRC).
>
> My version of the SQL, wrapping in a transaction, to be safe, is:
>
> BEGIN;
>
> ALTER TABLE metadatavalue DROP CONSTRAINT "$1";
> ALTER TABLE metadatavalue ADD CONSTRAINT metadatavalue_item_id_fkey
> FOREIGN KEY(item_id) REFERENCES item(item_id);
>
> COMMIT;
>
>
>
> We're not sure of what the cause is.
> 12:47 PM <tdonohue> yea, I'm not sure either. :) It's definitely a
> "glitch" that happens sometimes (since others have seen it too). But,
> according to the PostgreSQL docs, these contraints are SUPPOSED to be
> named [table]_[column]_fkey
> 12:47 PM <tdonohue> So, it seems like something has changed in Postgres,
> or Postgres doesn't always follow it's own "rules"
>
> ________________
> Peter Dietz
> Longsight
> www.longsight.com <http://www.longsight.com>
> pe...@longsight.com <mailto:pe...@longsight.com>
> <mailto:DSpac...@lists.sourceforge.net>
> >> https://lists.sourceforge.net/lists/listinfo/dspace-tech
> >> List Etiquette:
> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
>
> ------------------------------------------------------------------------------
> Dive into the World of Parallel Programming The Go Parallel Website,
> sponsored
> by Intel and developed in partnership with Slashdot Media, is your
> hub for all
> things parallel software development, from weekly thought leadership
> blogs to
> news, videos, case studies, tutorials and more. Take a look and join the
> conversation now. http://goparallel.sourceforge.net/
> _______________________________________________
> DSpace-tech mailing list
> DSpac...@lists.sourceforge.net
> <mailto:DSpac...@lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
> List Etiquette:
> https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
>
>
>
>
> ------------------------------------------------------------------------------
> One dashboard for servers and applications across Physical-Virtual-Cloud
> Widest out-of-the-box monitoring support with 50+ applications
> Performance metrics, stats and reports that give you Actionable Insights
> Deep dive visibility with transaction tracing using APM Insight.
> http://ad.doubleclick.net/ddm/clk/290420510;117567292;y

Tim Donohue

unread,
Aug 26, 2015, 2:31:32 PM8/26/15
to dspac...@lists.sourceforge.net
Digging into this Constraint naming issue a bit more, it looks like in
PostgreSQL you can look up constraints (on columns) by name from the
"key_column_usage" view:

http://www.postgresql.org/docs/9.4/static/infoschema-key-column-usage.html

So, it seems reasonable to change the code to simply query for the
constraint name (rather than assuming the name is the default value).
Something similar to this *should work*

SELECT * FROM information_schema.key_column_usage WHERE
table_name='[tablename]' AND column_name='[columname]';

For this specific issue with finding the name of the (old) constraint on
table "metadatavalue" and column "item_id" that query would be:

SELECT * FROM information_schema.key_column_usage WHERE
table_name='metadatavalue' AND column_name='item_id';

Usually, that query will return "metadatavlua_item_id_fkey" (on a
pre-5.0 database). But, obviously in some scenarios this constraint
seems to be named "$1" (or similar) instead.

(I'll add these notes to the DS-2577 ticket)

- Tim


On 5/14/2015 12:42 PM, Brian Freels-Stendel wrote:
> Good morning,
>
> This is a problem from way back, see: http://sourceforge.net/p/dspace/mailman/message/28781652/. If I remember right, it was because In The Beginning, the constraints weren't being given names, so they were auto-generated by the db engine. For those of us with old databases, this will be a recurring problem. If it would help, I can grab a schema dump (I know that's not the right terminology, but I'm hoping the meaning will be clear) from our db to compare with a current one. We've been up since 1.3, so we should have all of the offenders.
>
> B--

Brian Freels-Stendel

unread,
Aug 26, 2015, 2:31:32 PM8/26/15
to dspace-tech
Good morning,

This is a problem from way back, see: http://sourceforge.net/p/dspace/mailman/message/28781652/. If I remember right, it was because In The Beginning, the constraints weren't being given names, so they were auto-generated by the db engine. For those of us with old databases, this will be a recurring problem. If it would help, I can grab a schema dump (I know that's not the right terminology, but I'm hoping the meaning will be clear) from our db to compare with a current one. We've been up since 1.3, so we should have all of the offenders.

B--

-----Original Message-----
From: Tim Donohue [mailto:tdon...@duraspace.org]
Sent: Thursday, May 14, 2015 11:17 AM
To: Peter Dietz; Wally Grotophorst
Cc: dspace-tech
Subject: Re: [Dspace-tech] flyway migration error

Tim Donohue

unread,
Aug 26, 2015, 2:31:33 PM8/26/15
to dspac...@lists.sourceforge.net
Hi Brian,

I definitely think a dump of what your constraints look like could be
useful. We definitely want to avoid this being a recurring problem in
any flyway migrations involving contraints.

As noted in my previous message, you also might want to simply query
your "information_schema.key_column_usage" table for oddly named
constraints, as that seems to be the easiest way in PostgreSQL (that I
can find) to query for column constraint names:

SELECT * from information_schema.key_column_usage;

In my "newer" PostgreSQL databases (that I've tried), the constraint
names returned all are logically named based on the table and/or column
names (as they are by default these days). But, it sounds like in
databases that have been around for some time, some of these constraints
may have less logical names like $1 or $2 or something else.

I'd just be curious to know if fixing our code to simply querying the
"key_column_usage" view seems like it'd resolve this issue long term?
This is a tough scenario in that I don't have any "test data" to try
this out from, even if I've found what *looks* to be the right solution.

- Tim

On 5/14/2015 12:42 PM, Brian Freels-Stendel wrote:

Brian Freels-Stendel

unread,
Aug 26, 2015, 2:31:35 PM8/26/15
to Tim Donohue, dspac...@lists.sourceforge.net
Well, I was able to pull the column names out into a spreadsheet, which may be helpful. I've also dumped our db structure. It's v. 5.1, so it won't have the constraint name the v.5 upgrade tripped over, but going forward it might be useful.

I'm also wondering if you might rename the constraint in one of your databases from the logical "metadatavalue_item_id_fkey" to "$1" for before trying to upgrade with flyway.
dspace.backup
information_schema.key_column_usage.xlsx

Tim Donohue

unread,
Aug 26, 2015, 2:31:36 PM8/26/15
to Brian Freels-Stendel, dspac...@lists.sourceforge.net
Hi Brian,

Thanks, actually that spreadsheet is very useful. It shows that this
problem is one we may repeat in the future, as that DB has several
non-standard names for constraints: $1, $2, all the way through $7.

I'll take your advice and try a 4.x -> 5.x upgrade after renaming that
constraint to $1 ;) Seems like this should be a fixable bug by querying
that key_column_usage table.

- Tim

Freddy Guerrero

unread,
Aug 26, 2015, 2:38:05 PM8/26/15
to dspac...@lists.sourceforge.net
Hi Bill,

Forgive thousand bad English, please I need your help I am trying to update dspace of version 1.8 to version 5.2, but I have a mistake, I ask you please help me, look what I get:


+----------------+----------------------------+---------------------+---------+
| Version        | Description                | Installed on        | State   |
+----------------+----------------------------+---------------------+---------+
| 1.1            | Initial DSpace 1.1 databas |                     | PreInit |
| 1.2            | Upgrade to DSpace 1.2 sche |                     | PreInit |
| 1.3            | Upgrade to DSpace 1.3 sche |                     | PreInit |
| 1.3.9          | Drop constraint for DSpace |                     | PreInit |
| 1.4            | Upgrade to DSpace 1.4 sche |                     | PreInit |
| 1.5            | Upgrade to DSpace 1.5 sche |                     | PreInit |
| 1.5.9          | Drop constraint for DSpace |                     | PreInit |
| 1.6            | Upgrade to DSpace 1.6 sche |                     | PreInit |
| 1.7            | Upgrade to DSpace 1.7 sche |                     | PreInit |
| 1.8            | Upgrade to DSpace 1.8 sche |                     | PreInit |
| 3.0            | Upgrade to DSpace 3.x sche |                     | PreInit |
| 4.0            | Initializing from DSpace 4 | 2015-06-05 09:53:47 | Success |
| 5.0.2014.08.08 | DS-1945 Helpdesk Request a |                     | Pending |
| 5.0.2014.09.25 | DS 1582 Metadata For All O |                     | Pending |
| 5.0.2014.09.26 | DS-1582 Metadata For All O |                     | Pending |
+----------------+----------------------------+---------------------+---------+


The error:


2015-06-05 11:02:20,700 ERROR org.dspace.storage.rdbms.DatabaseManager @ Failed to initialize the database:  
java.sql.SQLException: Flyway migration error occurred
at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:430)
at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:338)
at org.dspace.storage.rdbms.DatabaseManager.initialize(DatabaseManager.java:1373)
at org.dspace.storage.rdbms.DatabaseManager.getDbName(DatabaseManager.java:1491)
at org.dspace.storage.rdbms.DatabaseUtils.main(DatabaseUtils.java:191)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.dspace.app.launcher.ScriptLauncher.runOneCommand(ScriptLauncher.java:226)
at org.dspace.app.launcher.ScriptLauncher.main(ScriptLauncher.java:78)
Caused by: org.flywaydb.core.internal.dbsupport.FlywaySqlScriptException: Error executing statement at line 20: ALTER TABLE requestitem ADD COLUMN request_message TEXT
at org.flywaydb.core.internal.dbsupport.SqlScript.execute(SqlScript.java:91)
at org.flywaydb.core.internal.resolver.sql.SqlMigrationExecutor.execute(SqlMigrationExecutor.java:73)
at org.flywaydb.core.internal.command.DbMigrate$5.doInTransaction(DbMigrate.java:287)
at org.flywaydb.core.internal.command.DbMigrate$5.doInTransaction(DbMigrate.java:285)
at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:72)
at org.flywaydb.core.internal.command.DbMigrate.applyMigration(DbMigrate.java:285)
at org.flywaydb.core.internal.command.DbMigrate.access$800(DbMigrate.java:46)
at org.flywaydb.core.internal.command.DbMigrate$2.doInTransaction(DbMigrate.java:207)
at org.flywaydb.core.internal.command.DbMigrate$2.doInTransaction(DbMigrate.java:156)
at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:72)
at org.flywaydb.core.internal.command.DbMigrate.migrate(DbMigrate.java:156)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:864)
at org.flywaydb.core.Flyway$1.execute(Flyway.java:811)
at org.flywaydb.core.Flyway.execute(Flyway.java:1171)
at org.flywaydb.core.Flyway.migrate(Flyway.java:811)
at org.dspace.storage.rdbms.DatabaseUtils.updateDatabase(DatabaseUtils.java:419)
... 10 more
Caused by: org.postgresql.util.PSQLException: ERROR: must be owner of relation requestitem
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2103)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1836)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:257)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:512)
at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:374)
at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:366)
at org.apache.commons.dbcp.DelegatingStatement.execute(DelegatingStatement.java:264)
at org.apache.commons.dbcp.DelegatingStatement.execute(DelegatingStatement.java:264)
at org.apache.commons.dbcp.DelegatingStatement.execute(DelegatingStatement.java:264)
at org.flywaydb.core.internal.dbsupport.JdbcTemplate.executeStatement(JdbcTemplate.java:235)
at org.flywaydb.core.internal.dbsupport.SqlScript.execute(SqlScript.java:89)
... 25 more


Pls helpme

Tim Donohue

unread,
Aug 26, 2015, 2:41:45 PM8/26/15
to Freddy Guerrero, dspac...@lists.sourceforge.net
Hi Freddy,

Sorry for the delayed response to your issue. It sounds like, in your situation, the database migration has failed because the database tables are "owned" by a different database user account?  The main error seems to be:

Caused by: org.postgresql.util.PSQLException: ERROR: must be owner of relation requestitem

So, essentially, the "requestitem" relation does not seem to be owned by the database user configured in your "db.username" field of the 'dspace.cfg'.  

So, you may need to investigate whether the "db.username" is set properly. If it is set properly, you should investigate the ownership of your database tables, and ensure they are all owned by the user account specified in "db.username". It's possible that you are running the migration against a database without full "owner" access rights.

You should be able to use either the "ALTER DATABASE [database] OWNER TO [owner]" or "ALTER TABLE [table] OWNER TO [owner]" commands to fix your ownership:

* ALTER DATABASE: http://www.postgresql.org/docs/9.1/static/sql-alterdatabase.html

-- 
Tim Donohue
Technical Lead for DSpace & DSpaceDirect
DuraSpace.org | DSpace.org | DSpaceDirect.org


------------------------------------------------------------------------------ _______________________________________________ DSpace-tech mailing list DSpac...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech List Etiquette: https://wiki.duraspace.org/display/DSPACE/Mailing+List+Etiquette
Reply all
Reply to author
Forward
0 new messages