Reviewboard upgrade failure

[Ben Cooksley]

Hi everyone,

While upgrading from Reviewboard 2.0.17 to 2.5.2 i've encountered a
few SQL errors.
Reviewboard itself appears to boot and run fine though, based on the
nature of the errors I suspect it's a case of index name collisions.

Output from the upgrade process is below:

(virtualenv)reviewboard@mimi:~$ rb-site upgrade
/srv/www/reviewboard/git.reviewboard.kde.org/
Rebuilding directory structure
Updating database. This may take a while.

The log output below, including warnings and errors,
can be ignored unless upgrade fails.

------------------ <begin log output> ------------------
Creating tables ...
Creating table accounts_trophy

[!] There was an error synchronizing the database. Make sure the
database is created and has the appropriate permissions, and then
continue.
[!] Details: (1005, "Can't create table 'reviewboard_git.#sql-
331d_214215' (errno: 150)")

Press Enter to continue
Creating tables ...
Creating table attachments_fileattachmenthistory
Creating table diffviewer_rawfilediffdata
Creating table notifications_webhooktarget_repositories

[!] There was an error synchronizing the database. Make sure the
database is created and has the appropriate permissions, and then
continue.
[!] Details: (1005, "Can't create table 'reviewboard_git.#sql-
331d_214215' (errno: 150)")

Press Enter to continue
Creating tables ...
Creating table notifications_webhooktarget

[!] There was an error synchronizing the database. Make sure the
database is created and has the appropriate permissions, and then
continue.
[!] Details: (1005, "Can't create table 'reviewboard_git.#sql-
331d_214215' (errno: 150)")

Press Enter to continue
Creating tables ...
Creating table webapi_webapitoken

[!] There was an error synchronizing the database. Make sure the
database is created and has the appropriate permissions, and then
continue.
[!] Details: (1005, "Can't create table 'reviewboard_git.#sql-
331d_214215' (errno: 150)")

Press Enter to continue
Creating tables ...
Upgrading Review Board from 2.0.17 to 2.5.2
There are unapplied evolutions for accounts.
There are unapplied evolutions for attachments.
There are unapplied evolutions for diffviewer.
There are unapplied evolutions for notifications.
There are unapplied evolutions for reviews.
There are unapplied evolutions for webapi.
Adding baseline version for new models
Evolutions in notifications baseline: webhooktarget_extra_state,
webhooktarget_extra_data_null
Project signature has changed - an evolution is required
Installing custom SQL ...
Installing indexes ...
Installed 0 object(s) from 0 fixture(s)
CommandError: Error applying evolution: (1005, "Can't create table
'reviewboard_git.#sql-331d_214238' (errno: 150)")

Any pointers?

Cheers,
Ben Cooksley
KDE Sysadmin

[Christian Hammond]

Hi Ben,

This is due to a mismatch between MySQL table types. The existing tables are likely MyISAM, with MySQL now defaulting to InnoDB for new ones. You'll need to either migrate all the existing tables, or tell MySQL to use the existing type for new tables.

(It's a pretty terrible error, but unfortunately, beyond our control. I just recognize this sort of problem.)

Christian

-- 
Christian Hammond - chri...@beanbaginc.com
Review Board - https://www.reviewboard.org

Beanbag, Inc. - https://www.beanbaginc.com

--

---
You received this message because you are subscribed to the Google Groups "reviewboard-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard-d...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Cooksley]

On Fri, Jan 15, 2016 at 11:04 PM, Christian Hammond
<chri...@beanbaginc.com> wrote:
> Hi Ben,
>
> This is due to a mismatch between MySQL table types. The existing tables are
> likely MyISAM, with MySQL now defaulting to InnoDB for new ones. You'll need
> to either migrate all the existing tables, or tell MySQL to use the existing
> type for new tables.
>
> (It's a pretty terrible error, but unfortunately, beyond our control. I just
> recognize this sort of problem.)

Argh. Our systems usually have InnoDB as default, guess that isn't the
case when we originally had Reviewboard provisioned.
I shouldn't see any issues migrating all tables into InnoDB correct?

>
> Christian

Cheers,
Ben

[Ben Cooksley]

On Fri, Jan 15, 2016 at 11:09 PM, Ben Cooksley <bcoo...@kde.org> wrote:
> On Fri, Jan 15, 2016 at 11:04 PM, Christian Hammond
> <chri...@beanbaginc.com> wrote:
>> Hi Ben,
>>
>> This is due to a mismatch between MySQL table types. The existing tables are
>> likely MyISAM, with MySQL now defaulting to InnoDB for new ones. You'll need
>> to either migrate all the existing tables, or tell MySQL to use the existing
>> type for new tables.
>>
>> (It's a pretty terrible error, but unfortunately, beyond our control. I just
>> recognize this sort of problem.)
>
> Argh. Our systems usually have InnoDB as default, guess that isn't the
> case when we originally had Reviewboard provisioned.
> I shouldn't see any issues migrating all tables into InnoDB correct?

Seems it is safe.
Unfortunately it looks like one of the evolutions got part way through
the process.

CommandError: Error applying evolution: (1060, "Duplicate column name
'visibility'")

Any suggestions (I could restore from backups, but if I can avoid it...)?

>
>>
>> Christian
>
> Cheers,
> Ben

Thanks,

[Christian Hammond]

Hi Ben,

You'll have a much easier time restoring from a backup. It's hard to say how far it went through the evolution process, and unfortunately today it doesn't keep track of how far it got and what it'd have to do to recover. You'd have a lot of trial and error to fix it manually. You can try it, though.

Basically, you'll need to dump the SQL that the evolutions want to apply, and go through and hand-undo each thing it did until you get back to the point of where it was. You'd definitely want to do a backup first, though.

Christian

--
Christian Hammond - chi...@chipx86.com

Review Board - https://www.reviewboard.org

Beanbag, Inc. - https://www.beanbaginc.com

[Ben Cooksley]

On Sat, Jan 16, 2016 at 6:09 AM, Christian Hammond <chi...@chipx86.com> wrote:
> Hi Ben,
>
> You'll have a much easier time restoring from a backup. It's hard to say how
> far it went through the evolution process, and unfortunately today it
> doesn't keep track of how far it got and what it'd have to do to recover.
> You'd have a lot of trial and error to fix it manually. You can try it,
> though.
>
> Basically, you'll need to dump the SQL that the evolutions want to apply,
> and go through and hand-undo each thing it did until you get back to the
> point of where it was. You'd definitely want to do a backup first, though.

Unfortunately people had started using it already so this was the
easiest approach :(

Would it be possible to get a copy of a normally, safely upgraded
schema so I can double check I haven't clobbered anything?

The queries I ended up having to run to revert things to a state where
the upgrade process would work was:

160115 19:34:12 2197763 Query ALTER TABLE
accounts_reviewrequestvisit DROP COLUMN visibility
160115 19:34:24 2197763 Query DROP INDEX
`accounts_reviewrequestvisit_05ee5d21` ON
`accounts_reviewrequestvisit`
160115 19:34:45 2197763 Query ALTER TABLE attachments_fileattachment
DROP COLUMN attachment_revision, DROP COLUMN attachment_history_id
160115 19:34:58 2197763 Query ALTER TABLE diffviewer_filediff DROP
COLUMN raw_diff_hash_id, DROP COLUMN raw_parent_diff_hash_id
160115 19:36:18 2197763 Query ALTER TABLE `reviews_group` DROP
COLUMN `email_list_only`, DROP COLUMN is_default_group
160115 19:36:25 2197763 Query DROP TABLE
reviews_reviewrequest_file_attachment_histories

Note that I observed that the Reviewboard process tries to reverse
it's failed upgrade by doing a rollback. It is noted in the case of
InnoDB that schema changes cannot be rolled back (see
http://www.sitepoint.com/mysql-transaction-gotchas-good-parts/)

>
> Christian

Cheers,

[Daniel Laird]

I too had this issue (MySQL default changing from MyISAM to InnoDB).

Although I understand that this is caused by MySQL version being updated at the same time as a Reviewboard Update (and migrate).

But would there be any way for the 'upgrade' logic of Reviewboard to post a more useful error message (DB Table formats dont match) or something like that.

As I am sure it will save a lot of time for some people (but I dont have the skills to make these changes sadly)

Dan

[Christian Hammond]

Hi Dan,

This sort of information actually comes at a level lower than what we deal with. It's during Django's table building process, and we have no insight or control into any of that. I agree that this would be *very* nice (and we need to document it), but as for creating a better error or knowing ahead of time, there's no easy solution to that right now.

I'll think about this..

Christian

-- 
Christian Hammond - chri...@beanbaginc.com
Review Board - https://www.reviewboard.org

Beanbag, Inc. - https://www.beanbaginc.com

--

[Daniel Laird]

All,

I am trying to upgrade from 2.0.20 -> 2.5.3.

I do this by taking a backup of the 2.0.20 database then importing into MySQL and then running the rb-site upgrade.

However I get the following output:

....

Updating database. This may take a while.

The log output below, including warnings and errors,

can be ignored unless upgrade fails.

------------------ <begin log output> ------------------

Creating tables ...

There are unapplied evolutions for accounts.

There are unapplied evolutions for attachments.

There are unapplied evolutions for diffviewer.

There are unapplied evolutions for reviews.

There are unapplied evolutions for webapi.

Project signature has changed - an evolution is required

Installing custom SQL ...

Installing indexes ...

Installed 0 object(s) from 0 fixture(s)

/usr/local/lib/python2.7/dist-packages/ReviewBoardPowerPack-1.4-py2.7.egg/rbpowerpack/scmtools/tfs.py:9: DeprecationWarning: django.utils.simplejson is deprecated; use json instead.

CommandError: Error applying evolution: (1060, "Duplicate column name 'visibility'")

...

Any ideas on why I am getting this error or how I can get more debug to help?

Much appreciated

Dan

[Christian Hammond]

Hey Dan,

That looks like it's due to a half-updated database, where some changes were applied and then there was a failure. Or, due to a forced hinted evolution that took place at some point.

To fix it, you'd have to figure out exactly what SQL statements had applied from which evolutions, how far all that got, and what entries in the schema storage may be incorrect, and manually alter portions of the database to get it back into the state it should be in to undo that upgrade. It's a lot of work.

It's also the kind of thing we're good at. You've likely seen our other threads about database repair. We can get you a working database within hours for an hourly fee, if you wanted to go that route. We charge $200/hour for the repair, and it usually takes only a few hours. We also include this service as part of our Premium Support contracts, which I can talk about more if you're interested.

Christian

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[朱皓]

Hi Daniel,

This problem is caused by some bugs in the database upgrade script. You can fix it by dropping all duplicate columns and indexes. I have fixed it by keeping running this script (you may need to run it again and again, and make sure all these are dropped):

ALTER TABLE accounts_reviewrequestvisit DROP COLUMN visibility;

ALTER TABLE accounts_reviewrequestvisit DROP INDEX accounts_reviewrequestvisit_05ee5d21;

ALTER TABLE attachments_fileattachment DROP COLUMN attachment_history_id;

ALTER TABLE attachments_fileattachment DROP COLUMN attachment_revision;

ALTER TABLE diffviewer_filediff DROP COLUMN raw_parent_diff_hash_id;

ALTER TABLE diffviewer_filediff DROP COLUMN raw_diff_hash_id;

ALTER TABLE diffviewer_filediff DROP INDEX diffviewer_filediff_4d02a56a;

ALTER TABLE diffviewer_filediff DROP INDEX diffviewer_filediff_79f4fcf6;

ALTER TABLE attachments_fileattachment DROP COLUMN attachment_revision;

ALTER TABLE diffviewer_filediff DROP COLUMN raw_parent_diff_hash_id;

[Christian Hammond]

Please don't hand-apply SQL. Generally if you need to, something deeper has gone wrong. Hand-applying SQL without a deep understanding of how Django Evolution works can easily result in more failures going forward.

If your database is not upgrading, work with us instead. We offer database repair services (standalone and as part of a support contract) and can usually get you in good shape within a few hours.

Christian

To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--