heads up, master may be slower than before

[Shawn Pearce]

I submitted the new /changes/ JSON API code to master this morning,
and also submitted the changes to use it in the search results web UI,
and the user dashboard web UI.

This afternoon I merged into my environment to start looking at
upgrading gerrit-review.googlesource.com to latest master.
Unfortunately its a lot slower than before. I am seeing page loads
that should be ~300ms taking ~4000ms. Locally I did not see this
negative performance hit in my local testing environment... maybe
because that was using the H2 database with a small data set? :-)

I'll continue to work on this tomorrow, but just thought folks
considering master should know it may be slower. The slowdown might be
unique to my environment. Or it might not.

[Martin Fick]

Do we need to have an easy mechanism to perform the queries
asynchronously (assuming they can be done in parallel)?

-Martin

--
Employee of Qualcomm Innovation Center, Inc. which is a
member of Code Aurora Forum

[Shawn Pearce]

That is the general approach to "fix" this sort of problem, perform
queries in parallel to hide latency. gwtorm already supports loading
multiple entities at once by primary key using get(Set<K>) on any
access object. But I think the problem is the patch set approvals, we
load them using the byChange(Change.Id) query, or
byPatchSet(PatchSet.Id) query, neither of which has a Set overload to
perform multiple queries at once.

I'm not yet sure that is exactly where the performance problem is. I
tried some easy stuff yesterday evening, and it made a small impact,
but not enough.

[Shawn Pearce]

On Wed, May 2, 2012 at 9:10 AM, Shawn Pearce <s...@google.com> wrote:
> On Wed, May 2, 2012 at 8:51 AM, Martin Fick <mf...@codeaurora.org> wrote:
>> On Tuesday, May 01, 2012 08:17:17 pm Shawn Pearce wrote:
>>> I submitted the new /changes/ JSON API code to master
>>> this morning, and also submitted the changes to use it
>>> in the search results web UI, and the user dashboard web
>>> UI.
>>>
>>> This afternoon I merged into my environment to start
>>> looking at upgrading gerrit-review.googlesource.com to
>>> latest master. Unfortunately its a lot slower than
>>> before. I am seeing page loads that should be ~300ms
>>> taking ~4000ms. Locally I did not see this negative
>>> performance hit in my local testing environment... maybe
>>> because that was using the H2 database with a small data
>>> set?  :-)

Yay, I declare success. I have a 7 change topic query-perf that I
think fixes the problems, mostly by doing more stuff in batch where
the APIs allow it, or async where the backend allows it, or bypassing
it where its maybe sort of kind of reasonable to allow bypassing it in
the name of faster responses.

With those 7 changes applied, master may be faster than the old RPC
system was for showing search results or user dashboards. E.g. in my
testing environment, ChangeListService was taking 500ms to show the
first 25 "status:open", and another 413ms to show PatchDetailService's
results for the CR and V status columns on the table. Now its (just
barely) sub 500ms to do the entire thing. So we get the CR/V status
data back "instantly".

[Markus Duft]

Hey!

Are there any open performance issues with 2.5-rc0? I observed frontpage load times ~30 seconds (!) on my local machine without any load and me as only user. The repo is rather large (6 months history, 10000 commits, ~7000 review items).

With 2.2.2.1 the UI was really snappy (load time for the same pages under one second).

Regards,

Markus

[Shawn Pearce]

On Fri, Sep 14, 2012 at 6:30 AM, Markus Duft <duft....@gmail.com> wrote:
> Are there any open performance issues with 2.5-rc0?

I am not aware of any. I know some things were slower on the first
iterations of the new /changes/ API, but I thought I had addressed
most of these performance issues.

> I observed frontpage
> load times ~30 seconds (!) on my local machine without any load and me as
> only user. The repo is rather large (6 months history, 10000 commits, ~7000
> review items).
>
> With 2.2.2.1 the UI was really snappy (load time for the same pages under
> one second).

This is bad. We run 2.5-rc0-13-g3153be5 and have no such problems. I
don't think our configuration is really that far from others, our
database is known to be slower for example.

The first load may take some time to cache user account data. But once
its cached, it should be hot and not as expensive to lookup user
information for the Owner, CR and V columns. So I am a bit puzzled as
to why you would see such a large load time.

[duft.markus]

If i can help gather data somehow, please tell me. Tomorrow i will try to migrate another snapshot of the 2.2.2.1 Gerrit to 2.4.2, and check how it behaves. i guess since 2.5 won't we ready in time this will be my target version for the switch on 1st of October :)

Markus

[Markus Duft]

I copied the configuration from the 2.5 instance to a freshly migrated 2.4.2 (from 2.2.2.1) instance, and it loads blazingly fast :) So it is definitely something with 2.5.

I'll upgrade to 2.4.2 then.

Regards,

Markus

[Shawn Pearce]

On Fri, Sep 14, 2012 at 6:30 AM, Markus Duft <duft....@gmail.com> wrote:

> Are there any open performance issues with 2.5-rc0? I observed frontpage
> load times ~30 seconds (!) on my local machine without any load and me as
> only user. The repo is rather large (6 months history, 10000 commits, ~7000
> review items).

By frontpage, you mean the All > Open view? How many approval
categories are configured on the server?

[Markus Duft]

Yes, All > Open. The configuration is pretty much default - verified and code review only. Also in other places, gerrit is not much customized (the database was not touched manually at all) configuration is something like this:

 1 [gerrit]

 2     basePath = /fast/wamas5/gerrit/git/

 3     canonicalWebUrl = http://<...>:2502/

 4 [database]

 5     type = H2

 6     database = db/ReviewDB

 7 [auth]

 8     type = LDAP

 9 [ldap]

<skipped>

19 [sendemail]

20     smtpServer = mailhost

21 [container]

22     user = git

23     javaHome = /opt/oracle-jdk-bin-1.7.0.7/jre

24     heapLimit = 512m

25 [sshd]

26     listenAddress = *:2501

27 [httpd]

28     listenUrl = http://*:2502/

29 [core]

30     packedGitLimit = 100m

31     streamFileThreshold = 5120m

32 [cache]

33     directory = cache

34 [commentlink "bugzilla"]

35     match = ([Bb]ug\\s*:\\s*)(\\d+)

36     link = http://bugzilla/show_bug.cgi?id=$2

37 [user]

38     name = Gerrit Code Review

39     email = git@<...>

40 [gitweb]

41     type = custom

42     url = http://<...>:2503/

43     revision = commit/${project}/${commit}

44     project = summary/${project}

45     branch = metrics/${project}?h=${branch}

46 

47     fileHistory = blob/${project}/${branch}/${file}

48 

49 [download]

50     scheme = ssh

51 [cache "web_sessions"]

52     maxAge = 30d

53 [theme]

54     trimColor = d8d6ec

55 [changeMerge]

56     test = true

Maybe you can tell some from that :)

Regards, and thanks for the help!

Markus

[Shawn Pearce]

On Sun, Oct 14, 2012 at 11:27 AM, Arnaud <fabre....@gmail.com> wrote:

> On Monday, October 1, 2012 4:37:29 PM UTC+2, Markus Duft wrote:
>>
>> Yes, All > Open. The configuration is pretty much default - verified and
>> code review only. Also in other places, gerrit is not much customized (the
>> database was not touched manually at all) configuration is something like
>> this:
>
>

> I have encountered similar performance issue when upgrading to 2.5-rc0,
> though it was on the dashboard page.
>
> After some analysis, I found some issue with an sql query. I've proposed a
> fix on gerrit-review [1], perhaps will this help you?
>
> [1] https://gerrit-review.googlesource.com/38460

That would be a problem. :-)

Our database requires an index on anything that a query is on, so we
have an index on this field. Must have been missed in the SQL
database.

It is certainly odd that the change_id column is already indexed and
must be equal to patchset_change_id, but they are two different
columns.

[Markus Duft]

It seems that /something/ (maybe this change ;)) fixed the problem. i tried current stable-2.5 branch, and it works like a charm, blazingly fast again :D (at least on some first tries).

Thanks,

Markus