Backup and Restore of Reviewboard 2.0 failing (mysqldump issues)

[Daniel Laird]

All,

Since RB 1.5 I have backed up the RB installation everynight be by using

mysqldump user=x password=y reviewboard > backup.sql

I have also backed up all attachments etc.

I could then restore the data easily with another script.

I used this method to upgrade from 1.7.25 -> 2.0.11 and with the additional rb-site upgrade etc, it all worked great.

However, after 2 days I noticed an error on my setup and re-created the site and re-imported the data.

At this point it was a RB 2.0.11 backup being restored to a RB2.0.11 database.

It all appeared to go ok, but when I clicked on any review i got

Error 500, something broke, I also got the following email:

Traceback (most recent call last):

  File "/usr/local/lib/python2.7/dist-packages/Django-1.6.8-py2.7.egg/django/core/handlers/base.py", line 112, in get_response

    response = wrapped_callback(request, *callback_args, **callback_kwargs)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.11-py2.7.egg/reviewboard/accounts/decorators.py", line 23, in _check

    return view_func(*args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.11-py2.7.egg/reviewboard/site/decorators.py", line 35, in _check

    return view_func(request, local_site=local_site, *args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.11-py2.7.egg/reviewboard/reviews/views.py", line 710, in review_detail

    'screenshots': screenshots,

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.11-py2.7.egg/reviewboard/reviews/context.py", line 92, in make_review_request_context

    upload_diff_form = UploadDiffForm(review_request, request=request)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.11-py2.7.egg/reviewboard/reviews/forms.py", line 95, in __init__

    data, *args, **kwargs)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.11-py2.7.egg/reviewboard/diffviewer/forms.py", line 41, in __init__

    if self.repository.get_scmtool().get_diffs_use_absolute_paths():

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.11-py2.7.egg/reviewboard/scmtools/models.py", line 230, in get_scmtool

    return cls(self)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.11-py2.7.egg/reviewboard/scmtools/svn/__init__.py", line 73, in __init__

    local_site_name)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.11-py2.7.egg/reviewboard/scmtools/svn/__init__.py", line 415, in build_client

    client = Client(config_dir, repopath, username, password)

  File "/usr/local/lib/python2.7/dist-packages/ReviewBoard-2.0.11-py2.7.egg/reviewboard/scmtools/svn/pysvn.py", line 41, in __init__

    self.client.set_default_password(six.text_type(password))

UnicodeDecodeError: 'ascii' codec can't decode byte 0xd5 in position 0: ordinal not in range(128)

Has something changed in the internal data structures?

Should I be using some options like "--default-character-set=utf8" when dumping/importing the database?

Hope you can help

Dan

[Christian Hammond]

Hi Daniel,

It does look like an encoding conflict. We recommend using UTF-8 for all database encodings, but MySQL generally defaults to latin1. My guess is that this password uses some sort of latin1 character (Õ, from the looks of it), which is failing to be converted to UTF-8.

For most of the database, you can convert the tables and content to use UTF-8. There's a bunch of articles online for doing this. However, it won't affect the repository passwords, as we store those encrypted. Instead, you'll need to go in and actually re-enter the passwords for the repositories.

Another option is to write a little Python script that uses this. You'd do something like:

    $ rb-site manage /path/to/site shell
    >>> from reviewboard.scmtools.models import Repository
    >>> for repo in Repository.objects.all():
    >>>     repo.password = repo.password.decode('latin1')
    >>>     repo.save()

I would test this first on a copy of the database in case it doesn't do what we'd expect.

If it's not a big deal to just re-type the passwords, or just re-save the repository configuration (that may be sufficient), then I'd just do that. Certainly, you can test it with that failing repository first.

Hope that helps!

Christian

--
Christian Hammond - chri...@beanbaginc.com
Review Board - https://www.reviewboard.org
Beanbag, Inc. - https://www.beanbaginc.com

> --
> Get the Review Board Power Pack at http://www.reviewboard.org/powerpack/
> ---
> Sign up for Review Board hosting at RBCommons: https://rbcommons.com/
> ---
> Happy user? Let us know at http://www.reviewboard.org/users/
> ---
> You received this message because you are subscribed to the Google Groups "reviewboard"
> group.
> To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[er...@tibco.com]

Turns out I've been fighting this issue on and off for several days now.

This looks like it is one of those particularly evil bugs. I've captured a lot of details about it below:

Situation very similar - I'm using mysqldump to backup a database. In my case, I'm trying to restore to a different machine for testing purposes.

When I bring up the test instance, I get an error 500 when I attempt to examine the details of a configured repository.

Email contains this (trimmed somewhat to avoid the long stack trace):

  File "/usr/lib64/python2.7/site-packages/django/template/base.py", line 897, in render
    return render_value_in_context(output, context)

  File "/usr/lib64/python2.7/site-packages/django/template/base.py", line 875, in render_value_in_context
    value = force_text(value)

  File "/usr/lib64/python2.7/site-packages/django/utils/encoding.py", line 116, in force_text
    raise DjangoUnicodeDecodeError(s, *e.args)

DjangoUnicodeDecodeError: 'utf8' codec can't decode byte 0xad in position 0: invalid start byte. You passed in <django.forms.forms.BoundField object at 0x7effbb2e0510> (<class 'django.forms.forms.BoundField'>)

----

So, seems obvious - an encoding issue. Except, it isn't.

I've been very careful to ensure that MySQL uses UTF-8 everywhere (on both production and testing). A mysqldump of the tables on the production database, and the test instance database shows that the scmtools_repository tables are identical, including the schema, and show use of UTF-8 as the charset. I even did a mysql --xml -e query of the tables, and found that they're identical. A hexdump of the results shows no non-ASCII characters. In fact, the only odd character is a 09 at the beginning of the password field (which looks to be intentional?). Mind you, this is as expected, because the password field looks to be a base64 encoding of an actual password, so it necessarily wouldn't encode any non-ASCII characters.

I've confirmed that changing the password field using SQL, then resetting the password in the web UI makes the problem go away (for how long?).

So now for the really messed up part. To try to get more / better diagnostics, I went to the "conf" folder for my ReviewBoard instance. I changed "Debug" flag to "True". Restart Apache (hosted via wsgi, FYI).

Ready to look at details problem reports. Except ... problem gone....

I've now repeated this experiment several times. Set "Debug" to "False", problem occurs. "Debug = True", and problem goes away.

This has all the classic signs of being an uninitialized variable. Arghhh.

Any suggestions you have for further debugging would be most welcome. I fear that this is a bug in Django treatment of password fields.

Eric.

[er...@tibco.com]

I figured out what my issue was. Copying the database from one machine to another isn't sufficient.

ReviewBoard has a "SECRET_KEY" that is used when encrypting passwords into the database. If you're restoring your database backup to a different machine (as I was doing), and you don't copy over the SECRET_KEY, then stuff will break.

If you re-save the passwords from the web UI, it uses the SECRET_KEY when re-obfuscating the password.

Might that be the problem?

Eric.

[harish varadarajan]

Hi Eric,

Thanks for your post, that I could understand the issue that was happening at my reviewboard instance.

Did you figure out the solution for this.

With your approach to turning the Debug = True, I am currently able to access the repositories / enter the edit screen.

But unfortunately, the password field for the user who access on repositories is not editable.

But, still while creating newer reviews for older repositories, it states that the content / change is not available.

Is the only way to make it work, is to copy the SECRET_KEY from the older instance ? 

any ideas on how this could be solved, if access to the older host / instance is not available.

Thanks.

Harish

[Christian Hammond]

Hi Harish,

I'm not sure I fully understand what problems you're hitting, but a couple notes:

1) If you're using a database that was set up on an older instance, then yes, SECRET_KEY must be copied over. If it was not, a number of things like stored database passwords or hosting service auth tokens will be inaccessible.

Entering a new password for each repository should take care of that. It sounds like you're hitting a problem doing this? Define "not editable?"

2) You should not have DEBUG = True on, for any reason, really. This is intended for development environments, and will alter behaviors that are not suitable for production environments.

Can you describe what you're trying to do and exactly what's going wrong?

Thanks,

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/

[Christian Hammond]

Ah, I see you have another thread on this. I'll respond there.

Christian

-- 
Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[harish varadarajan]

Hi Christian,

Thanks for looking into this.

As suggested by Eric, and per update in your mail, I copied over the SECRET_KEY, and restarted httpd, and now the repositories are accessible and also, the previous review requests created in the older server show up their diff's correcty, which was not the case before.

Thanks for your inputs and certainly going over the contents in this group, help solved a lot of initial niggles.

--

Harish