Experiencing migration issues from 2.0.25 to 2.5.15

[Gene Paul Quevedo]

Hi,

I'ts my first time to post here, and I hope someone can point me to the right way of solving this issue, even though it seems like there were issues like this before

I'm trying to do a DB upgrade of Review from 2.0.25 to 2.5.15 on a test server with a copy of our Database, and seems like it has bumped into this issue

Rebuilding directory structure
Upgrading site settings_local.py
Updating database. This may take a while.

The log output below, including warnings and errors,
can be ignored unless upgrade fails.

------------------ <begin log output> ------------------
Creating tables ...
There are unapplied evolutions for accounts.
There are unapplied evolutions for attachments.
There are unapplied evolutions for diffviewer.
There are unapplied evolutions for reviews.
There are unapplied evolutions for webapi.
Project signature has changed - an evolution is required
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)

This may take a while. Please be patient, and do not cancel the upgrade!

Applying database evolutions for accounts...
Database evolutions for accounts failed!
The SQL statement was: (u'ALTER TABLE `accounts_reviewrequestvisit` ADD COLUMN `visibility` varchar(1) NOT NULL DEFAULT %s;', [u'V'])
The database error was: (1060, "Duplicate column name 'visibility'")

CommandError: Error applying evolution for accounts: (1060, "Duplicate column name 'visibility'")

There are probably 4 variations of that 'duplicate column' error, and the other one is a 'duplicate index' .
My initial solution was to delete the duplicates and run the upgrade procedure again, but seems like the duplication seems to be repeating (!), as if Django Evolution is creating those duplicate columns/indices

Any ideas on how to debug this further? I'm attaching our DB schema here

thank you!
- Gene

[Christian Hammond]

Hi Gene,

Looks like there was a partial attempt at an evolution that failed once before. The "visibility" column was applied and then something apparently went wrong, or upgrading was cancelled. Deleting the duplicates and running again would lead to this, unless carefully handled in a way compatible with django-evolution's internals.

Is this database schema the initial one applied from 2.0.25, or what you have on the test server after the upgrade attempts?

Christian

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[Gene Paul Quevedo]

hi Christian

The schema is from 2.0.25. It was from a snapshot of our test server before the upgrade was performed. If there's a suspicion I could generate again the schema coming from ou production server, which runs 2.0.25

To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gene Paul Quevedo]

Oops looks like you might be right. Anyway here's our schema direct coming from our production site.

Previously we were running 2.0.20, and the upgrade went smooth if I recall.

[Christian Hammond]

Hi Gene,

The schema looks healthy at first glance. It seems like a normal 2.0.x schema, and one that would be safe to upgrade to 2.5.x.

I don't know what the original error was that you hit (the one in the original post here was due to a subsequent attempt at upgrading a database that already failed in a way we couldn't recover from), but one possible cause of failures would be the table engine type.

Old versions of MySQL used MyISAM as the table type (the ENGINE= lines in the schema). Newer MySQL versions (from the past several years onward) default to InnoDB for any new tables (and 2.5.x would introduce new tables). If you have a mix of table types, MySQL can end up throwing confusing errors when attempting to create any relations between those tables. It's something I'd love for us to be able to catch, but it's pretty much a MySQL responsibility.

While I don't know if you hit an error related to that, what you might want to try doing is converting your schema to use InnoDB for all tables. You can do a full database dump from the production server, change all ENGINE=MyISAM entries to ENGINE=InnoDB, and then load the database into the test server. Then attempt an upgrade, and see if it goes through successfully.

If it doesn't succeed, show me the full output, and I'll probably be able to narrow it down.

Christian

To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gene Paul Quevedo]

Hi Christian,

I (re) tried this procedure of creating a dump from production, creating a database on a test server, changing the engine to InnoDB for 64 tables and, running the rb-site upgrade.

Still, it's failing somewhere, but I'm able to capture the log message from the upgrade failure, the first after using a pristine DB copy from the production.

root@test:~# rb-site upgrade /var/www/netcomm-codereviews

Rebuilding directory structure
Upgrading site settings_local.py
Updating database. This may take a while.

The log output below, including warnings and errors,
can be ignored unless upgrade fails.

------------------ <begin log output> ------------------
Creating tables ...

Creating table accounts_trophy
Creating table attachments_fileattachmenthistory
Creating table diffviewer_rawfilediffdata
Creating table notifications_webhooktarget_repositories
Creating table notifications_webhooktarget
Creating table webapi_webapitoken
Upgrading Review Board from 2.0.25 to 2.5.15

There are unapplied evolutions for accounts.
There are unapplied evolutions for attachments.
There are unapplied evolutions for diffviewer.

There are unapplied evolutions for notifications.

There are unapplied evolutions for reviews.
There are unapplied evolutions for webapi.

Adding baseline version for new models
Evolutions in notifications baseline: webhooktarget_extra_state, webhooktarget_extra_data_null

Project signature has changed - an evolution is required
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)

This may take a while. Please be patient, and do not cancel the upgrade!

Applying database evolutions for accounts...

Applying database evolutions for attachments...
Applying database evolutions for diffviewer...
Database evolutions for diffviewer failed!
The SQL statement was: CREATE INDEX `diffviewer_filediff_4d02a56a` ON `diffviewer_filediff` (`raw_diff_hash_id`);
The database error was: (1061, "Duplicate key name 'diffviewer_filediff_4d02a56a'")

CommandError: Error applying evolution for diffviewer: (1061, "Duplicate key name 'diffviewer_filediff_4d02a56a'")

-----------------------------------------------------------------------------------------------------------

By the time it failed it has created at least 5 tables. Seems like it's hard to tell what is causing the duplicate key name on table indexes.

Attaching here are the latest schema post-upgrade failure

but hey! thanks so much for helping me out on this one :)

-Gene

You received this message because you are subscribed to a topic in the Google Groups "reviewboard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/NZ33huj8f8A/unsubscribe.
To unsubscribe from this group and all its topics, send an email to reviewboard...@googlegroups.com.

[Christian Hammond]

Hi Gene,

Was the database fully recreated between attempts? It seems like there may be some stale indexes sitting around.

Christian

To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Christian Hammond

President/CEO of Beanbag

Makers of Review Board

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---

You received this message because you are subscribed to a topic in the Google Groups "reviewboard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/NZ33huj8f8A/unsubscribe.

To unsubscribe from this group and all its topics, send an email to reviewboard+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.

To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Gene Paul Quevedo]

Hi Christian,

I did drop the database before creating it again using the DB Dump. Was that sufficient? What would be a proper way of recreating it?

You received this message because you are subscribed to a topic in the Google Groups "reviewboard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/NZ33huj8f8A/unsubscribe.

To unsubscribe from this group and all its topics, send an email to reviewboard...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Christian Hammond

President/CEO of Beanbag

Makers of Review Board

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---

You received this message because you are subscribed to a topic in the Google Groups "reviewboard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/NZ33huj8f8A/unsubscribe.

To unsubscribe from this group and all its topics, send an email to reviewboard...@googlegroups.com.

[Peter Simms]

I am getting this same error after doing a restore of our current/old reviewboard db and then rb-site upgrade

Applying database evolutions for diffviewer...

Database evolutions for diffviewer failed!

The SQL statement was: CREATE INDEX `diffviewer_filediff_4d02a56a` ON `diffviewer_filediff` (`raw_diff_hash_id`);

The database error was: (1061, "Duplicate key name 'diffviewer_filediff_4d02a56a'")

CommandError: Error applying evolution for diffviewer: (1061, "Duplicate key name 'diffviewer_filediff_4d02a56a'")

[Peter Simms]

If I drop the index and try the upgrade again I get the below

Applying database evolutions for accounts...

Database evolutions for accounts failed!

The SQL statement was: ALTER TABLE `accounts_profile` ADD COLUMN `settings` longtext NULL;

The database error was: (1060, "Duplicate column name 'settings'")

CommandError: Error applying evolution for accounts: (1060, "Duplicate column name 'settings'")