Database upgrade issues - Possible fix

[Christian Hammond]

Hi everyone,

So some of you have hit database upgrade issues before when upgrading to 1.7 (or, in some cases, 1.6). We've managed to figure out solutions for some of you, but not all.

I've spent a lot of time trying to figure out solutions, find the common cases, and figure out a fix. Today, I have one that should fix it for at least some of you. What I really need are volunteers to help test it.

If you're having issues now, or have had issues in the past and still have a broken database dump handy, I'd appreciate your help. Heck, even if you have had no issues, and have an old database dump to test.

I have a test version of Django Evolution up at http://downloads.reviewboard.org/temp/django-evolution/

This version has some new logic that tries to avoid applying database upgrades for things that are already in the database, which was the core problem most of you hit.

If you can give this a try, here's what I'd like you to do:

1) Get an installation going with your broken or pre-upgrade database.

2) Install django_evolution from the above URL, using:

    $ sudo easy_install -U django_evolution -f http://downloads.reviewboard.org/temp/django-evolution/

3) Attempt a Review Board upgrade (rb-site manage /path/to/site upgrade)

4) Assuming it works, try to access the site and make sure things generally work on it.

Please then reply to this post with the following information:

1) The complete log output.

2) Whether you were testing with a dump from before a past broken upgrade attempt, or a dump made after that past upgrade attempt failed.

For extra credit, if you have did have a working database dump, sanity-checking that you don't have any upgrade problems and that the site runs would also really help me out.

Thanks! Sooner I get some feedback, the sooner I can put this problem to rest.

- Christian

-- 
Christian Hammond - chi...@chipx86.com
Review Board - http://www.reviewboard.org
VMware, Inc. - http://www.vmware.com

[Nachiketh G]

Hi Christian,

I was able to upgrade now successfully! I have tried migrating an old database(1.0) into the latest RB Server, please find below the result of the tests.

Cheers!

Nachiketh

OUTPUT:

root@nachiketh-virtual-machine:~# easy_install -f http://downloads.reviewboard.org/temp/django-evolution/ -U django_evolution

Searching for django-evolution

Reading http://downloads.reviewboard.org/temp/django-evolution/

Reading http://pypi.python.org/simple/django_evolution/

Reading http://code.google.com/p/django-evolution/

Best match: django-evolution 0.6.9alpha0.dev

Downloading http://downloads.reviewboard.org/temp/django-evolution/django_evolution-0.6.9alpha0.dev-py2.7.egg

Processing django_evolution-0.6.9alpha0.dev-py2.7.egg

creating /usr/local/lib/python2.7/dist-packages/django_evolution-0.6.9alpha0.dev-py2.7.egg

Extracting django_evolution-0.6.9alpha0.dev-py2.7.egg to /usr/local/lib/python2.7/dist-packages

Removing django-evolution 0.6.8 from easy-install.pth file

Adding django-evolution 0.6.9alpha0.dev to easy-install.pth file

Installed /usr/local/lib/python2.7/dist-packages/django_evolution-0.6.9alpha0.dev-py2.7.egg

Processing dependencies for django-evolution

Finished processing dependencies for django-evolution

root@nachiketh-virtual-machine:~# rb-site upgrade /var/www/csm

Rebuilding directory structure

Updating database. This may take a while.

The log output below, including warnings and errors,

can be ignored unless upgrade fails.

------------------ <begin log output> ------------------

Creating tables ...

Creating table extensions_registeredextension

Creating table accounts_localsiteprofile

Creating table attachments_fileattachment

Creating table diffviewer_filediffdata

Creating table hostingsvcs_hostingserviceaccount

Creating table reviews_fileattachmentcomment

Creating table site_localsite_users

Creating table site_localsite_admins

Creating table site_localsite

There are unapplied evolutions for auth.

There are unapplied evolutions for sessions.

There are unapplied evolutions for accounts.

There are unapplied evolutions for changedescs.

There are unapplied evolutions for diffviewer.

There are unapplied evolutions for reviews.

There are unapplied evolutions for scmtools.

Adding baseline version for new models

Project signature has changed - an evolution is required

Installing custom SQL ...

Installing indexes ...

Installed 0 object(s) from 0 fixture(s)

Registering new SCM Tool Plastic SCM (reviewboard.scmtools.plastic.PlasticTool) in database

Registering new SCM Tool ClearCase (reviewboard.scmtools.clearcase.ClearCaseTool) in database

Evolution could not be simulated, possibly due to raw SQL mutations

Evolution successful.

------------------- <end log output> -------------------

Resetting in-database caches.

Upgrade complete!

root@nachiketh-virtual-machine:~# 

[Steve]

Sorry for the long, messy output, here's what happened when I tested the fix. This is an attempt to upgrade a 1.5.5 database to 1.7.6.

[root@crush]~/ReviewBoard# easy_install -U django_evolution -f http://downloads.reviewboard.org/temp/django-evolution/

Searching for django-evolution

Reading http://pypi.python.org/simple/django_evolution/

Reading http://code.google.com/p/django-evolution/

Best match: django-evolution 0.6.8

Downloading http://pypi.python.org/packages/2.6/d/django_evolution/django_evolution-0.6.8-py2.6.egg#md5=7527e0ca398639c87b68969ceb1cd2d7

Processing django_evolution-0.6.8-py2.6.egg

removing '/usr/lib/python2.6/site-packages/django_evolution-0.6.8-py2.6.egg' (and everything under it)

creating /usr/lib/python2.6/site-packages/django_evolution-0.6.8-py2.6.egg

Extracting django_evolution-0.6.8-py2.6.egg to /usr/lib/python2.6/site-packages

django-evolution 0.6.8 is already the active version in easy-install.pth

Installed /usr/lib/python2.6/site-packages/django_evolution-0.6.8-py2.6.egg

Processing dependencies for django-evolution

Finished processing dependencies for django-evolution

Searching for -f

Reading http://pypi.python.org/simple/-f/

Couldn't find index page for '-f' (maybe misspelled?)

Scanning index of all packages (this may take a while)

Reading http://pypi.python.org/simple/

  C-c C-c^Cinterrupted

I don't know why easy_install got confused by '-f', but I did get a new django_evolution, so I tried updating my broken site:

[root@crush]~/ReviewBoard# cat rb-upgrade.log

Error: Model 'reviews.FileAttachmentComment' already has a field named 'extra_data'

Rebuilding directory structure

Updating database. This may take a while.

The log output below, including warnings and errors,

can be ignored unless upgrade fails.

------------------ <begin log output> ------------------

Creating tables ...

There are unapplied evolutions for auth.

There are unapplied evolutions for accounts.

There are unapplied evolutions for changedescs.

There are unapplied evolutions for diffviewer.

There are unapplied evolutions for reviews.

There are unapplied evolutions for scmtools.

Project signature has changed - an evolution is required

Installing custom SQL ...

Installing indexes ...

Installed 0 object(s) from 0 fixture(s)

Now when I log in to my site, I get this:

(1054, "Unknown column 'accounts_profile.is_private' in 'field list'")

Request Method:	GET
Request URL:	https://crush/
Django Version:	1.4.5
Exception Type:	DatabaseError
Exception Value:	(1054, "Unknown column 'accounts_profile.is_private' in 'field list'")
Exception Location:	/usr/lib/python2.6/site-packages/MySQL_python-1.2.3-py2.6-linux-x86_64.egg/MySQLdb/connections.py in defaulterrorhandler, line 36
Python Executable:	/usr/bin/python
Python Version:	2.6.6
Python Path:	['/var/www/reviewboard/conf', '/usr/lib64/python2.6/site-packages/JCC-2.14-py2.6-linux-x86_64.egg', '/usr/lib64/python2.6/site-packages/lucene-3.6.1-py2.6-linux-x86_64.egg', '/usr/lib/python2.6/site-packages/distribute-0.6.27-py2.6.egg', '/usr/lib/python2.6/site-packages/MySQL_python-1.2.3-py2.6-linux-x86_64.egg', '/usr/lib/python2.6/site-packages/recaptcha_client-1.0.6-py2.6.egg', '/usr/lib/python2.6/site-packages/python_dateutil-1.5-py2.6.egg', '/usr/lib/python2.6/site-packages/flup-1.0.3.dev_20110405-py2.6.egg', '/usr/lib/python2.6/site-packages/Pygments-1.5-py2.6.egg', '/usr/lib/python2.6/site-packages/PIL-1.1.7-py2.6-linux-x86_64.egg', '/usr/lib/python2.6/site-packages/pycrypto-2.6-py2.6-linux-x86_64.egg', '/usr/lib/python2.6/site-packages/feedparser-5.1.3-py2.6.egg', '/usr/lib/python2.6/site-packages/ReviewBoard-1.7.6-py2.6.egg', '/usr/lib/python2.6/site-packages/pytz-2012j-py2.6.egg', '/usr/lib/python2.6/site-packages/paramiko-1.9.0-py2.6.egg', '/usr/lib/python2.6/site-packages/mimeparse-0.1.3-py2.6.egg', '/usr/lib/python2.6/site-packages/Markdown-2.2.1-py2.6.egg', '/usr/lib/python2.6/site-packages/docutils-0.10-py2.6.egg', '/usr/lib/python2.6/site-packages/django_pipeline-1.2.24-py2.6.egg', '/usr/lib/python2.6/site-packages/Djblets-0.7.11-py2.6.egg', '/usr/lib/python2.6/site-packages/Django-1.4.5-py2.6.egg', '/usr/lib/python2.6/site-packages/django_evolution-0.6.8-py2.6.egg', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/gst-0.10', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib64/python2.6/site-packages/webkit-1.0', '/usr/lib/python2.6/site-packages']
Server time:	Fri, 1 Mar 2013 19:08:48 +0000

I may try it again on a pristine 1.5.5 database.

--Steve

[Christian Hammond]

That's the wrong Django Evolution. Can you try downloading the one from that URL and just passing it directly to easy_install?

Christian

-- 
Christian Hammond - chi...@chipx86.com
Review Board - http://www.reviewboard.org
VMware, Inc. - http://www.vmware.com

--
Want to help the Review Board project? Donate today at http://www.reviewboard.org/donate/
Happy user? Let us know at http://www.reviewboard.org/users/
-~----------~----~----~----~------~----~------~--~---
To unsubscribe from this group, send email to reviewboard...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/reviewboard?hl=en
---
You received this message because you are subscribed to the Google Groups "reviewboard" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Steve]

Now I get this error:

(1054, "Unknown column 'reviews_reviewrequest.local_site_id' in 'on clause'")

Request Method:	GET
Request URL:	https://crush/dashboard/

Django Version:	1.4.5
Exception Type:	DatabaseError

Exception Value:	(1054, "Unknown column 'reviews_reviewrequest.local_site_id' in 'on clause'")

Python Path:

['/var/www/reviewboard/conf', '/usr/lib64/python2.6/site-packages/JCC-2.14-py2.6-linux-x86_64.egg', '/usr/lib64/python2.6/site-packages/lucene-3.6.1-py2.6-linux-x86_64.egg', '/usr/lib/python2.6/site-packages/distribute-0.6.27-py2.6.egg', '/usr/lib/python2.6/site-packages/MySQL_python-1.2.3-py2.6-linux-x86_64.egg', '/usr/lib/python2.6/site-packages/recaptcha_client-1.0.6-py2.6.egg', '/usr/lib/python2.6/site-packages/python_dateutil-1.5-py2.6.egg', '/usr/lib/python2.6/site-packages/flup-1.0.3.dev_20110405-py2.6.egg', '/usr/lib/python2.6/site-packages/Pygments-1.5-py2.6.egg', '/usr/lib/python2.6/site-packages/PIL-1.1.7-py2.6-linux-x86_64.egg', '/usr/lib/python2.6/site-packages/pycrypto-2.6-py2.6-linux-x86_64.egg', '/usr/lib/python2.6/site-packages/feedparser-5.1.3-py2.6.egg', '/usr/lib/python2.6/site-packages/ReviewBoard-1.7.6-py2.6.egg', '/usr/lib/python2.6/site-packages/pytz-2012j-py2.6.egg', '/usr/lib/python2.6/site-packages/paramiko-1.9.0-py2.6.egg', '/usr/lib/python2.6/site-packages/mimeparse-0.1.3-py2.6.egg', '/usr/lib/python2.6/site-packages/Markdown-2.2.1-py2.6.egg', '/usr/lib/python2.6/site-packages/docutils-0.10-py2.6.egg', '/usr/lib/python2.6/site-packages/django_pipeline-1.2.24-py2.6.egg', '/usr/lib/python2.6/site-packages/Djblets-0.7.11-py2.6.egg', '/usr/lib/python2.6/site-packages/Django-1.4.5-py2.6.egg', '/usr/lib/python2.6/site-packages/django_evolution-0.6.9alpha0.dev-py2.6.egg', '/usr/lib64/python26.zip', '/usr/lib64/python2.6', '/usr/lib64/python2.6/plat-linux2', '/usr/lib64/python2.6/lib-tk', '/usr/lib64/python2.6/lib-old', '/usr/lib64/python2.6/lib-dynload', '/usr/lib64/python2.6/site-packages', '/usr/lib64/python2.6/site-packages/gst-0.10', '/usr/lib64/python2.6/site-packages/gtk-2.0', '/usr/lib64/python2.6/site-packages/webkit-1.0', '/usr/lib/python2.6/site-packages']

Maybe I should try on a fresh database.

--Steve

[Nachiketh G]

Steve,

The order of arguments passed to easy_install in your output is incorrect, you need to execute "easy_install -f http://downloads.reviewboard.org/temp/django-evolution/ -U django_evolution", hope it helps!

Thanks & Regards,

Nachiketh

You received this message because you are subscribed to a topic in the Google Groups "reviewboard" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/kfHroDRjytc/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to reviewboard...@googlegroups.com.

[Christian Hammond]

Did you revert to your state before the previous test, or is this on top of the previous? It's possible the previous test will have caused a problem for this one (though unlikely).

Christian

-- 
Christian Hammond - chi...@chipx86.com
Review Board - http://www.reviewboard.org
VMware, Inc. - http://www.vmware.com

[Steve]

I went back to a pristine version of the 1.5.5 database and ran rb-site again:

# rb-site upgrade /var/www/reviewboard 

Rebuilding directory structure

Updating database. This may take a while.

The log output below, including warnings and errors,

can be ignored unless upgrade fails.

------------------ <begin log output> ------------------

Creating tables ...

Creating table extensions_registeredextension

Creating table accounts_localsiteprofile

Creating table attachments_fileattachment

Creating table diffviewer_filediffdata

Creating table hostingsvcs_hostingserviceaccount

Creating table reviews_fileattachmentcomment

Creating table site_localsite_users

Creating table site_localsite_admins

Creating table site_localsite

Upgrading Review Board from 1.5.5 to 1.7.6

There are unapplied evolutions for auth.

There are unapplied evolutions for accounts.

There are unapplied evolutions for changedescs.

There are unapplied evolutions for diffviewer.

There are unapplied evolutions for reviews.

There are unapplied evolutions for scmtools.

Adding baseline version for new models

Project signature has changed - an evolution is required

Installing custom SQL ...

Installing indexes ...

Installed 0 object(s) from 0 fixture(s)

Registering new SCM Tool Plastic SCM (reviewboard.scmtools.plastic.PlasticTool) in database

Evolution could not be simulated, possibly due to raw SQL mutations

...

It's been sitting at that point for about 15 minutes. It's a 7GB database, so it might really be taking this long.  Or it might be hung.  I'll just leave it for a while.

--Steve

[Christian Hammond]

Hmm, longer than I'd expect, but it's a fairly big upgrade I suppose, given how old the DB is. Shame we don't have logging set up to show what it's doing. Still, no errors in the simulation stage, which is promising. Hopefully the other issue you hit won't crop up. If it does, we'll need to investigate further.

Christian

[Steve]

It's still sitting at the same place and appears to be stuck.  I see the rb-site process in ps, but no child processes so I have no idea what it's doing.  Sorry I've been more trouble than help in testing this fix.

--steve

[Christian Hammond]

Yeah that sounds stuck.

Maybe try doing it one more time, to be sure?

What type of database is this?

I can try to get you a build later tonight that has some extra debugging so we can at least see what it's trying to do before it gets stuck.

Christian

-- 
Christian Hammond - chi...@chipx86.com
Review Board - http://www.reviewboard.org
VMware, Inc. - http://www.vmware.com

[Steve]

It's a MySQL database. I was able to migrate the same database to 1.6 with no difficulty.  I'll give it another try later after my head clears a bit to make sure I'm not doing something goofy.

--steve

[Steve]

Well, sometimes procrastination pays off.  When I came back to my session, I saw this:

Evolution successful.

------------------- <end log output> -------------------

Resetting in-database caches.

Upgrade complete!

The location of static media files (CSS, JavaScript, images)

has changed. You will need to make manual changes to

your web server configuration.

For Apache, you will need to add:

    <Location "/static">

        SetHandler None

    </Location>

    Alias /static "/var/www/reviewboard/htdocs/static"

For lighttpd:

    alias.url = (

        ...

        "/static" => "/var/www/reviewboard/htdocs/static",

        ...

    )

    url.rewrite-once = (

        ...

        "^(/static/.*)$" => "$1",

        ...

    )

Once you have made these changes, type the following

to resolve this:

    $ rb-site manage /var/www/reviewboard resolve-check static-media

So that's good, right?  I'll respond back when I've made those changes and had a chance to test out the server.

Thanks

--Steve

[Steve]

I'm happy to report the patch you provided seems to have worked. I'm able to log on to my 1.7.6 site and browse reviews.  I'll test it more today, but so far it looks good.

Thanks Chris!

--Steve