ReviewBoard 3.0 is too slow

Hema

unread,

Apr 24, 2018, 11:26:25 PM4/24/18

to Review Board Community

Hi,

We've upgraded ReviewBoard from 2.0.12 to 3.0.3 and we use mysql as back-end. After the upgrade, we see the performance of the site drastically come down especially when we access filediff table. I've tried the optimization tips of memcache and mysql, recommended by ReviewBoard team. But I still see the slowness in performance.

Please let me know the steps to fine tune the performance.

Regards,

Hema.

Christian Hammond

unread,

Apr 24, 2018, 11:31:26 PM4/24/18

to revie...@googlegroups.com

Hi Hema,

If anything else, Review Board 3.0 should be a lot faster than 2.0.12. Can you verify that Review Board can see the memcached server in the admin UI -> Server Cache, and that it's properly storing data?

Christian

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Christian Hammond

President/CEO of Beanbag

Makers of Review Board

Hema

unread,

Apr 25, 2018, 5:19:40 AM4/25/18

to Review Board Community

Hi Christian,

Yes. ReviewBoard can see memcached servers. Cause I can see messages getting printed when I use "-vv" switch with memcached.

A query especially the following one gets stuck at the db end.

SELECT `diffviewer_filediffdata`.`binary_hash`, `diffviewer_filediffdata`.`binary`, `diffviewer_filediffdata`.`extra_data` FROM `diffviewer_filediffdata` WHERE (`diffviewer_filediffdata`.`binary_hash` IN ('de97317bfd39313d60016c365948bbb5f108b412') AND NOT ((`diffviewer_filediffdata`.`binary_hash` IN (SELECT U1.`diff_hash_id` FROM `diffviewer_filediff` U1 WHERE (U1.`id` > 0 AND U1.`diff_hash_id` IS NOT NULL)) OR `diffviewer_filediffdata`.`binary_hash` IN (SELECT U1.`parent_diff_hash_id` FROM `diffviewer_filediff` U1 WHERE (U1.`id` > 0 AND U1.`parent_diff_hash_id` IS NOT NULL))))) ;

Do you have any idea on this?

Regards,

Hema.

Christian Hammond

unread,

Apr 25, 2018, 5:26:38 AM4/25/18

to revie...@googlegroups.com

I saw that behavior on an older version of MySQL locally a long time back. If I remember correctly, rebooting MySQL fixed it, and newer versions didn't seem to exhibit the problem. It had to do with indexing the primary key hashes. For some reason, it was being incredibly slow with this. However, I never encountered this when just using the product. I only saw it when performing SQL statements by hand for testing I was doing.

Review Board 3.0 stopped adding to that table, instead storing new diffs in diffviewer_rawfilediffdata, which uses a different design. Accessing existing diffs performs a migration for that diff, and all diffs can be migrated in one go using `rb-site manage /path/to/sitedir condensediffs`.

Do you see slowdown when posting brand new diffs for review?

Can you tell me more about the MySQL version, table types, and general hardware specs and setup?

Christian

--

Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hema

unread,

Apr 25, 2018, 6:15:42 AM4/25/18

to Review Board Community

Hi Christian,

The condensediffs command takes days and days together. Hence, I stopped it in between. After seeing your reply, started it again.

--------------------------------------------------------------------------------------------------------------------------

[root@reviewboard html]# /opt/software/bin/rb-site manage /var/www/reviewboard.com condensediffs

Processing 6054597 diffs for duplicates...

This may take a while. It is safe to continue using Review Board while this is

processing, but it may temporarily run slower.

[0%] 57600/6054597 - 0 seconds remaining

--------------------------------------------------------------------------------------------------------------------------

Here are the details of the setup.

App server:

No. of processors: 12

Processor Speed: 2.7 GHz

Memory: 24 GB

Memcache size: 2GB

DB server:

No. of processors: 2

Processor Speed: 2.13GHz

Memory: 10GB

--------------------------------------------------------------------------------------------------------

MySQL [ReviewBoard]> SHOW VARIABLES LIKE "%version%";

+-------------------------+---------------------+

| Variable_name | Value |

+-------------------------+---------------------+

| protocol_version | 10 |

| version | 5.1.52-log |

| version_comment | Source distribution |

| version_compile_machine | x86_64 |

| version_compile_os | redhat-linux-gnu |

+-------------------------+---------------------+

5 rows in set (0.00 sec)

MySQL [ReviewBoard]> SHOW TABLE STATUS WHERE Name = 'diffviewer_filediffdata';

+-------------------------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+----------------+---------+

+-------------------------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+----------------+---------+

| diffviewer_filediffdata | MyISAM | 10 | Dynamic | 4216198 | 11479 | 48398494548 | 281474976710655 | 264371200 | 252356 | NULL | 2018-04-09 09:27:12 | 2018-04-25 02:33:16 | 2018-04-10 09:13:02 | latin1_swedish_ci | NULL | | |

+-------------------------+--------+---------+------------+---------+----------------+-------------+-----------------+--------------+-----------+----------------+---------------------+---------------------+---------------------+-------------------+----------+----------------+---------+

1 row in set (0.00 sec)

--------------------------------------------------------------------------------------------------------

And yes. I don't see any slowdown when posting new diffs.

Please let me know if we have any work-around other than running condensediffs.

Regards,

Hema.

Christian Hammond

unread,

Apr 25, 2018, 4:16:16 PM4/25/18

to revie...@googlegroups.com

It does look like there's some extremely large amounts of data stored in there, so it might just simply take time to do the migration, but I think it's being slower than it should be.

One aspect of the slowdown is going to be caused by MyISAM table-level locking, which would cause the table to be locked every time we migrate one diff out of it, blocking all other queries (and page loads) that reference the table. The other problem is that the default index type is possibly just not being used efficiently.

I'm going to play around with this locally on a large test database, see if I can come up with a suitable workaround, but I'm massively swamped this week (only able to allocate sufficient time for companies with support contracts -- we can discuss that if you're interested). So it might be a little while until I have something to recommend.

For now, let me get some additional information from you.

The table indicates it was created earlier this month. Was this created as part of importing data from another server, or is this a test server? Trying to get a sense of how big the server is and what the best approach might be.

I want to also make sure the indexes are in there and get some stats. Can you run:

SHOW INDEXES FROM diffviewer_filediffdata\G

EXPLAIN EXTENDED SELECT COUNT(binary_hash) FROM diffviewer_filediffdata\G

ANALYZE TABLE diffviewer_filediffdata\G

Christian

--

Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hemapriya

unread,

Apr 26, 2018, 5:33:12 AM4/26/18

to revie...@googlegroups.com

We created a copy of the actual database and working on it. Once it works perfect, we'll upgrade to ReviewBoard3.0 in production environment.

Here are the requested detail.

----------------------------------------------------------------------------------------------------------------

MySQL [ReviewBoard]> SHOW INDEXES FROM diffviewer_filediffdata;

+-------------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+

+-------------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+

+-------------------------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+

1 row in set (1.18 sec)

MySQL [ReviewBoard]> EXPLAIN EXTENDED SELECT COUNT(binary_hash) FROM diffviewer_filediffdata;

+----+-------------+-------+------+---------------+------+---------+------+------+----------+------------------------------+

+----+-------------+-------+------+---------------+------+---------+------+------+----------+------------------------------+

+----+-------------+-------+------+---------------+------+---------+------+------+----------+------------------------------+

1 row in set, 1 warning (0.03 sec)

MySQL [ReviewBoard]> ANALYZE TABLE diffviewer_filediffdata;

+------------------------------------------+---------+----------+----------+

+------------------------------------------+---------+----------+----------+

+------------------------------------------+---------+----------+----------+

1 row in set (5 min 23.74 sec)

----------------------------------------------------------------------------------------------------------------

Do you think upgrading mysql to a higher version will help here?

Regards,

Hemapriya.

Christian Hammond

unread,

Apr 26, 2018, 5:34:37 PM4/26/18

to revie...@googlegroups.com

An upgrade might help, but I'm not sure whether it'll solve the issue.

I want to narrow down whether it's traversal over the index that's slowing down some of our lookups. How long does this take? (If the answer is more than a couple minutes, Control-C it).

SELECT COUNT(binary_hash) FROM diffviewer_filediffdata;

By the way, using \G instead of ; at the end of a statement will result in more readable output for e-mail (doesn't rely on fixed-width text). Useful trick.

Christian

--

Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hemapriya

unread,

Apr 27, 2018, 12:26:18 AM4/27/18

to revie...@googlegroups.com

Thanks for the \G tip. I wasn't aware of it.

MySQL [ReviewBoard]> SELECT COUNT(binary_hash) FROM diffviewer_filediffdata\G

*************************** 1. row ***************************

COUNT(binary_hash): 4216198

1 row in set (0.00 sec)

The above select query doesn't take even a second. Where else could be the issue?

By the way, please let me know if you have any recommendation of mysql version, engine type,... that we need to use for ReviewBoard.

Thanks,

Hema.

You received this message because you are subscribed to a topic in the Google Groups "Review Board Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/HZP5szW5_-M/unsubscribe.
To unsubscribe from this group and all its topics, send an email to reviewboard+unsubscribe@googlegroups.com.

Hemapriya

unread,

Apr 27, 2018, 6:50:29 AM4/27/18

to revie...@googlegroups.com

Is there any way that we can migrate only the open review requests or the review requests created over a specific period of time?

Regards,

Hema.

manoj m

unread,

Jun 25, 2018, 1:47:20 PM6/25/18

to Review Board Community

FWIW we have a large installation of reviewboard as well and it was running MyISAM engine.

We upgraded from 1.X some to 2.0 recently and also moved to Innodb which has row level locking. This is what we did:

1. Turned off foreign key constraints

2. Ran alter table query on on all tables and change the engine to innodb

3. Ran the Reviewboard update

4. Turned on the key constraints again.

Needless to say we scheduled a downtime for this and also backed up the data .

--

To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
Christian Hammond
President/CEO of Beanbag
Makers of Review Board

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to a topic in the Google Groups "Review Board Community" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/reviewboard/HZP5szW5_-M/unsubscribe.

To unsubscribe from this group and all its topics, send an email to reviewboard...@googlegroups.com.

Reply all

Reply to author

Forward