How to find the bottleneck?

[Vlad S.]

Hi,


Generating a diff file info for interdiffsettakes too much time ~30 seconds. How do I know why?
Is there a command line method to debug this and see all the calls made?
===
CRITICAL - None - ... - /rb/api/review-requests/27122/diff-context/ - .. - Generating diff file info for interdiffset ids 66619-66797 took 29.881996 seconds
===
Thanks!

/v

[Christian Hammond]

Hi Vlad,

It's often a delay in getting data back from the repository. Are there any other log messages? Is the repository in the same network?

How many files are in each diff, and are any excessively large?

Also, make sure that Review Board is seeing the memcached server. Check the Admin UI -> Cache information to be sure.

Christian

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "Review Board Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/reviewboard/47fd0081-0d5d-4c50-8fd9-5a2fe036806d%40googlegroups.com.

--

Christian Hammond

President/CEO of Beanbag

Makers of Review Board

[Vlad S.]

Hi Christian,

There are messages like this

  File "/venvs/rb/lib/python2.7/site-packages/reviewboard/scmtools/git.py", line 702, in _cat_file
    raise SCMError(errmsg)
SCMError: fatal: git cat-file 1ba156144fe754340c79e0f871a52f750ad8f6b7: bad file

The repository is on the same network, NFS mount. memcached runs, 2G, cache hit ratio ~95%, 4 cores CPU.

Only mysql server is distant with 100ms latency. Tried to turn off syntax highlighting. It does not help much.

163K diff file, 116 files, C text. So it does not look large.

#                                 Old fast host - New slow host

CPU                            4 cores            4 cores

                                    both have plenty on RAM

memcached              v1.5* 2GB       v1.4* 2GB

memcached memory used   ~400MB            ~90MB 

mysql latency             0.2ms               100ms

repo host latency       both on the same network.

Memory usage:

411.7 MB

пятница, 25 октября 2019 г., 11:44:25 UTC+3 пользователь Christian Hammond написал:

Hi Vlad,

It's often a delay in getting data back from the repository. Are there any other log messages? Is the repository in the same network?

How many files are in each diff, and are any excessively large?

Also, make sure that Review Board is seeing the memcached server. Check the Admin UI -> Cache information to be sure.

Christian

On Fri, Oct 25, 2019 at 1:32 AM Vlad S. <the...@gmail.com> wrote:

Hi,


Generating a diff file info for interdiffsettakes too much time ~30 seconds. How do I know why?
Is there a command line method to debug this and see all the calls made?
===
CRITICAL - None - ... - /rb/api/review-requests/27122/diff-context/ - .. - Generating diff file info for interdiffset ids 66619-66797 took 29.881996 seconds
===
Thanks!

/v

--
Supercharge your Review Board with Power Pack: https://www.reviewboard.org/powerpack/
Want us to host Review Board for you? Check out RBCommons: https://rbcommons.com/
Happy user? Let us know! https://www.reviewboard.org/users/
---
You received this message because you are subscribed to the Google Groups "Review Board Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to revie...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/reviewboard/47fd0081-0d5d-4c50-8fd9-5a2fe036806d%40googlegroups.com.

[Christian Hammond]

Hi Vlad,

Is this problem only occurring on the new host and not the old one?

Do you ever see those `git cat-file` errors on the old host?

Any differences in NFS configuration between the hosts?

Christian

To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/reviewboard/a8aa658c-748b-4e7d-99e7-c858220eaf4f%40googlegroups.com.

[Vlad S.]

Only new one. The old host is 10x times faster. Ten times.

Generates diffs one-by-one in 200-300ms.

Old host uses NFS v3, dd reading a large file from repo at ~450MB/s

New host uses NFS v3,  dd reading a large file from repo at ~255 MB/s

SO the old has slower CPU but faster NFS, the new host has faster CPU but still fast NFS..

I want to point both hosts at the same database and compare to find a problem. The RB manual says it is supported.

May I use the same SECRET_KEY in /var/www/rb-site1/conf/settings_local.py  across the hosts?

/v

пятница, 25 октября 2019 г., 13:39:19 UTC+3 пользователь Christian Hammond написал:

To view this discussion on the web visit https://groups.google.com/d/msgid/reviewboard/a8aa658c-748b-4e7d-99e7-c858220eaf4f%40googlegroups.com.

[Christian Hammond]

Hi Vlad,

The problem is definitely looking like NFS. A Git repository is comprised of a great many files, which must be read and parsed and often point to other files. You want fast filesystem access to work with a Git repository directly.

If you’re seeing that level of slowdown just from a dd test, I bet you’re having more issues than you realize. Directory scans and file seeking are probably slow as well. Every time we fetch a file (which will happen for each file in a diff, at upload and diff viewer time), it’s going to be performing a number of slow steps, resulting in the problems you’re seeing.

What I’d suggest is moving away from NFS access to Git. Instead, install our RB Gateway service on the system hosting the repositories, configure it to point to each repository, and then configure those existing repositories in Review Board to access through RB Gateway instead of over NFS.

https://www.reviewboard.org/downloads/rbgateway/

This is going to provide deeper integration with Git overall. RB Gateway can take care of some Git management that Review Board doesn’t attempt. It’s the preferred way if you want to use a plain Git repository not backed by a service like GitHub or GitLab.

Regarding the database question, yes, you can point multiple Review Board servers to the same database. You must use the same SECRET_KEY on each server.

Christian

To unsubscribe from this group and stop receiving emails from it, send an email to reviewboard...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/reviewboard/c57a4342-d868-4e62-adfc-6e5fffcc9b96%40googlegroups.com.

--

[Vlad S.]

Hi Christian,

Thank you for your suggestions and  RB Gateway link! I will examine all the options.

/v

суббота, 26 октября 2019 г., 7:15:42 UTC+3 пользователь Christian Hammond написал:

To view this discussion on the web visit https://groups.google.com/d/msgid/reviewboard/c57a4342-d868-4e62-adfc-6e5fffcc9b96%40googlegroups.com.