dspace-cris 5.x upgrade to dspace-cris 7.x

[Wan Mohamad Azren Wan Mohamad Yusoff]

Hi I'm doing a test upgrade of dspace-cris 5.x to dspace-cris 7.x and having issues during data migration from dspace-cris 5.

My current setup are as follows:

1. dump PostgreSQL database from dspace-cris 5 (old server)

2. Setup a new server (Ubuntu 20.04) and configure all backend dspace-cris 7 requirements

3. restore dspace-cris 5 PostgreSQL database into the new server

4. Completed dspace-cris 7 backend installation

5. Migrate database successfully

    ./dspace database migrate ignore-missing

6. Using Pentaho Data Integration (PDI) Community Edition migrate data from dspace-cris 5.x to 7 but failed with the following error messages:

2022/05/10 13:47:42 - dspace_cris_migration - Starting entry [funding migration]
2022/05/10 13:47:43 - funding migration - Using run configuration [Pentaho local]
2022/05/10 13:47:43 - funding migration - Running transformation using the Kettle execution engine
2022/05/10 13:47:43 - entity_migration - Dispatching started for transformation [entity_migration]
2022/05/10 13:47:43 - insert into imp_record.0 - Connected to database [dspace] (commit=1000)
2022/05/10 13:47:43 - insert into imp_metadatavalue.0 - Connected to database [dspace] (commit=1000)
2022/05/10 13:47:43 - insert into imp_bitstream_metadatavalue.0 - Connected to database [dspace] (commit=1000)
2022/05/10 13:47:43 - insert into imp_bitstream.0 - Connected to database [dspace] (commit=1)
2022/05/10 13:47:44 - entities.0 - Finished reading query, closing connection.
2022/05/10 13:47:44 - entities.0 - Finished processing (I=82, O=0, R=0, W=82, U=0, E=0)
2022/05/10 13:47:44 - entities metadata.0 - Finished reading query, closing connection.
2022/05/10 13:47:44 - entities nested metadata.0 - Finished reading query, closing connection.
2022/05/10 13:47:44 - entities metadata.0 - Finished processing (I=81, O=0, R=0, W=81, U=0, E=0)
2022/05/10 13:47:45 - Join rows 4.0 - No rows coming from step [entities nested metadata]
2022/05/10 13:47:45 - metadata configuration.0 - Finished processing (I=83, O=0, R=0, W=83, U=0, E=0)
2022/05/10 13:47:45 - Get ENTITY_TYPE from variables.0 - Finished processing (I=0, O=0, R=83, W=83, U=0, E=0)
2022/05/10 13:47:45 - collections config.0 - Finished processing (I=6, O=0, R=0, W=6, U=0, E=0)
2022/05/10 13:47:46 - Get ENTITY_TYPE from variables 2.0 - Finished processing (I=0, O=0, R=6, W=6, U=0, E=0)
2022/05/10 13:47:46 - orcid scopes configuration.0 - Finished processing (I=4, O=0, R=0, W=4, U=0, E=0)
2022/05/10 13:47:46 - filter by entity type.0 - Finished processing (I=0, O=0, R=6, W=1, U=0, E=0)
2022/05/10 13:47:46 - orcid authentication configuration.0 - Finished processing (I=1, O=0, R=0, W=1, U=0, E=0)
2022/05/10 13:47:46 - get only collection uuid.0 - Finished processing (I=0, O=0, R=1, W=1, U=0, E=0)
2022/05/10 13:47:46 - metadata visibility configuration.0 - Finished processing (I=2, O=0, R=0, W=2, U=0, E=0)
2022/05/10 13:47:46 - Rename to metadata_visibility.0 - Finished processing (I=0, O=0, R=2, W=2, U=0, E=0)
2022/05/10 13:47:46 - nested metadata configuration.0 - Finished processing (I=11, O=0, R=0, W=11, U=0, E=0)
2022/05/10 13:47:46 - filter by entity type 2.0 - Finished processing (I=0, O=0, R=83, W=27, U=0, E=0)
2022/05/10 13:47:46 - Join rows 4.0 - Finished processing (I=0, O=0, R=11, W=0, U=0, E=0)
2022/05/10 13:47:46 - Select values 5.0 - Finished processing (I=0, O=0, R=27, W=27, U=0, E=0)
2022/05/10 13:47:46 - insert into imp_record.0 - ERROR (version 9.0.0.0-423, build 9.0.0.0-423 from 2020-01-31 04.53.04 by buildguy) : Because of an error, this step can't continue:
2022/05/10 13:47:46 - entity_migration - ERROR (version 9.0.0.0-423, build 9.0.0.0-423 from 2020-01-31 04.53.04 by buildguy) : Errors detected!
2022/05/10 13:47:46 - insert into imp_record.0 - ERROR (version 9.0.0.0-423, build 9.0.0.0-423 from 2020-01-31 04.53.04 by buildguy) : org.pentaho.di.core.exception.KettleException:
2022/05/10 13:47:46 - insert into imp_record.0 - Error inserting row into table [imp_record] with values: [null], [Funding], [1], [pj00001], [null], [1], [insert], [z], [1], [efd2ee0c-a0d8-4163-bd24-20b6f27c7cd0]
2022/05/10 13:47:46 - insert into imp_record.0 -
2022/05/10 13:47:46 - insert into imp_record.0 - Error inserting/updating row
2022/05/10 13:47:46 - insert into imp_record.0 - ERROR: null value in column "imp_collection_uuid" violates not-null constraint
2022/05/10 13:47:46 - insert into imp_record.0 -   Detail: Failing row contains (1, pj00001, efd2ee0c-a0d8-4163-bd24-20b6f27c7cd0, null, z, insert, null, null, null).
2022/05/10 13:47:46 - insert into imp_record.0 -
2022/05/10 13:47:46 - insert into imp_record.0 -
2022/05/10 13:47:46 - insert into imp_record.0 -        at org.pentaho.di.trans.steps.tableoutput.TableOutput.writeToTable(TableOutput.java:384)
2022/05/10 13:47:46 - insert into imp_record.0 -        at org.pentaho.di.trans.steps.tableoutput.TableOutput.processRow(TableOutput.java:125)
2022/05/10 13:47:46 - insert into imp_record.0 -        at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2022/05/10 13:47:46 - insert into imp_record.0 -        at java.lang.Thread.run(Thread.java:748)
2022/05/10 13:47:46 - insert into imp_record.0 - Caused by: org.pentaho.di.core.exception.KettleDatabaseException:
2022/05/10 13:47:46 - insert into imp_record.0 - Error inserting/updating row
2022/05/10 13:47:46 - insert into imp_record.0 - ERROR: null value in column "imp_collection_uuid" violates not-null constraint
2022/05/10 13:47:46 - insert into imp_record.0 -   Detail: Failing row contains (1, pj00001, efd2ee0c-a0d8-4163-bd24-20b6f27c7cd0, null, z, insert, null, null, null).
2022/05/10 13:47:46 - insert into imp_record.0 -
2022/05/10 13:47:46 - insert into imp_record.0 -        at org.pentaho.di.core.database.Database.insertRow(Database.java:1329)
2022/05/10 13:47:46 - insert into imp_record.0 -        at org.pentaho.di.trans.steps.tableoutput.TableOutput.writeToTable(TableOutput.java:262)
2022/05/10 13:47:46 - insert into imp_record.0 -        ... 3 more
2022/05/10 13:47:46 - insert into imp_record.0 - Caused by: org.postgresql.util.PSQLException: ERROR: null value in column "imp_collection_uuid" violates not-null constraint
2022/05/10 13:47:46 - insert into imp_record.0 -   Detail: Failing row contains (1, pj00001, efd2ee0c-a0d8-4163-bd24-20b6f27c7cd0, null, z, insert, null, null, null).
2022/05/10 13:47:46 - insert into imp_record.0 -        at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
2022/05/10 13:47:46 - insert into imp_record.0 -        at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
2022/05/10 13:47:46 - insert into imp_record.0 -        at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
2022/05/10 13:47:46 - insert into imp_record.0 -        at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
2022/05/10 13:47:46 - insert into imp_record.0 -        at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
2022/05/10 13:47:46 - insert into imp_record.0 -        at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:143)
2022/05/10 13:47:46 - insert into imp_record.0 -        at org.postgresql.jdbc.PgPreparedStatement.executeUpdate(PgPreparedStatement.java:120)
2022/05/10 13:47:46 - insert into imp_record.0 -        at org.pentaho.di.core.database.Database.insertRow(Database.java:1296)
2022/05/10 13:47:46 - insert into imp_record.0 -        ... 4 more
2022/05/10 13:47:46 - Add imp_record_id.0 - Finished processing (I=0, O=0, R=21, W=21, U=0, E=0)
2022/05/10 13:47:46 - Add insert operation and status.0 - Finished processing (I=0, O=0, R=24, W=24, U=0, E=0)
2022/05/10 13:47:46 - insert into imp_record.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=1)
2022/05/10 13:47:46 - Get variables.0 - Finished processing (I=0, O=0, R=15, W=15, U=0, E=0)
2022/05/10 13:47:46 - entity_migration - Transformation detected one or more steps with errors.
2022/05/10 13:47:46 - entity_migration - Transformation is killing the other steps!
2022/05/10 13:47:46 - Join rows.0 - Finished processing (I=0, O=0, R=83, W=82, U=0, E=0)
2022/05/10 13:47:46 - Join rows 7.0 - Finished processing (I=0, O=0, R=1, W=0, U=0, E=0)
2022/05/10 13:47:46 - Join rows 8.0 - Finished processing (I=0, O=0, R=4, W=0, U=0, E=0)
2022/05/10 13:47:46 - Join rows 6.0 - Finished processing (I=0, O=0, R=27, W=0, U=0, E=0)
2022/05/10 13:47:48 - entity_migration - ERROR (version 9.0.0.0-423, build 9.0.0.0-423 from 2020-01-31 04.53.04 by buildguy) : Errors detected!
2022/05/10 13:47:48 - dspace_cris_migration - Finished job entry [funding migration] (result=[false])
2022/05/10 13:47:48 - dspace_cris_migration - Finished job entry [funding migration setup] (result=[false])
2022/05/10 13:47:48 - dspace_cris_migration - Finished job entry [set funding variables] (result=[false])
2022/05/10 13:47:48 - dspace_cris_migration - Finished job entry [publications migration] (result=[false])
2022/05/10 13:47:48 - dspace_cris_migration - Finished job entry [set database variables] (result=[false])
2022/05/10 13:47:48 - dspace_cris_migration - Job execution finished
2022/05/10 13:47:48 - Kitchen - Finished!
2022/05/10 13:47:48 - Kitchen - ERROR (version 9.0.0.0-423, build 9.0.0.0-423 from 2020-01-31 04.53.04 by buildguy) : Finished with errors
2022/05/10 13:47:48 - Kitchen - Start=2022/05/10 13:47:34.536, Stop=2022/05/10 13:47:48.265
2022/05/10 13:47:48 - Kitchen - Processing ended after 13 seconds.
localadmin@dspace7:~/source/pentaho/data-integration$

Any idea on how should I go about resolving this?

Thank you.

Best regards,

Azren