Accounts super-slow after upgrading to 2.15

[lucamilanesio]

Hi all,

for those who have already upgraded to Gerrit v2.15: have you noticed super-slow performance on the accounts?

What I have noticed on GerritHub.io (*a lot* of accounts there, with *a lot* of external-ids) is that the show-caches was showing that the account cache took over 50s to load a single account in memory.

Then, after running a GC on the All-Users repo, all came back to normal.

Is it worth including a JGit GC in the Accounts migration to NoteDb in v2.15?

Luca.

[Matthew Webber]

Is it worth including a JGit GC in the Accounts migration to NoteDb in v2.15?

Unfortunately I didn't record the numbers, but after I did a test offline migration from 2.14.7 to 2.15, I did a 

cd $site_path/git
find -mindepth 1 -maxdepth 2 -type d -name "*.git" | sort | xargs -i --verbose git -C {} gc --aggressive --prune=now

And found that recovered a significant amount of space (for some value of significant). That's not really surprising, when you think about it.

We've talked before about whether Gerrit should do a JGit GC after a migration, and then general feeling has been that, if Gerrit gc is not enabled, the migration should respect that. Some folks still don't trust JGit GC.

So, my feeling is that the migration should not gc, but maybe the release notes should mention the value of doing it.

[Luca Milanesio]

On 19 Mar 2018, at 08:14, Matthew Webber <mat...@unsolvable.org> wrote:

Is it worth including a JGit GC in the Accounts migration to NoteDb in v2.15?

Unfortunately I didn't record the numbers, but after I did a test offline migration from 2.14.7 to 2.15, I did a 

cd $site_path/git
find -mindepth 1 -maxdepth 2 -type d -name "*.git" | sort | xargs -i --verbose git -C {} gc --aggressive --prune=now

And found that recovered a significant amount of space (for some value of significant). That's not really surprising, when you think about it.

We've talked before about whether Gerrit should do a JGit GC after a migration, and then general feeling has been that, if Gerrit gc is not enabled, the migration should respect that. Some folks still don't trust JGit GC.

Hold on, one thing is "doing regular GC of your repos" but another is the All-Users repo.

I suspect that the migration of all account external ids to the All-Users repo (or something else?) creates too many loose objects and refs and *that* is putting the FS too much under pressure.

After the migration to v2.15 was not only *slow* but really *unusable*.

So, my feeling is that the migration should not gc, but maybe the release notes should mention the value of doing it.

Yes, definitely. And the init step should at least have an option to trigger it with a *strongly suggested* note for encouraging doing it :-)

Luca.

[Dave Borowitz]

I agree with Luca, All-Users is a special case. The reason it's so bad is explained in the code:

https://gerrit.googlesource.com/gerrit/+/f4fb10467f93bf8eaf262d70694193e8bbb2acb1/gerrit-server/src/main/java/com/google/gerrit/server/notedb/rebuild/NoteDbMigrator.java#836

Let's add a step to just GC All-Users after the migration.

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dave Borowitz]

https://gerrit-review.googlesource.com/166910

[Dave Borowitz]

I'm not super thrilled that it adds an extra minute of time to the migration to GC All-Users on a site the size of gerrit-review. Possibly this could be optimized, for example if we skip creating bitmaps, but with the way the code works now it's not very easy to pass in a one-off PackConfig. So I'm inclined to leave it as-is.

[2018-03-19 08:42:20,922] [main] INFO  com.google.gerrit.server.config.ScheduleConfig : gc schedule parameter "gc.interval" is not configured

[2018-03-19 08:42:20,923] [main] INFO  gc_log : [All-Users] gc config: gc.aggressive=false;

[2018-03-19 08:42:20,924] [main] INFO  gc_log : [All-Users] pack config: maxDeltaDepth=50, deltaSearchWindowSize=10, deltaSearchMemoryLimit=0, deltaCacheSize=52428800, deltaCacheLimit=100, compressionLevel=-1, indexVersion=2, bigFileThreshold=52428800, threads=0, reuseDeltas=true,

reuseObjects=true, deltaCompress=true, buildBitmaps=true, bitmapContiguousCommitCount=100, bitmapRecentCommitCount=20000, bitmapRecentCommitSpan=100, bitmapDistantCommitSpan=5000, bitmapExcessiveBranchCount=100, bitmapInactiveBranchAge=90, singlePack=false

[2018-03-19 08:42:21,020] [main] INFO  gc_log : [All-Users] before: sizeOfPackedObjects=23453689, sizeOfLooseObjects=1080205, numberOfPackedObjects=84307, numberOfPackFiles=1, numberOfPackedRefs=12031, numberOfLooseRefs=693, numberOfLooseObjects=4225

collecting garbage for "All-Users":

Pack refs:              100% (12724/12724)

Counting objects:       88532

Finding sources:        100% (88532/88532)

Getting sizes:          100% (36406/36406)

Compressing objects:    100% (2411318/2411318)

Writing objects:        100% (88532/88532)

Selecting commits:      100% (37334/37334)

Building bitmaps:       100% (13132/13132)

Prune loose objects also found in pack files: 100% (258/258)

Prune loose, unreferenced objects: 100% (258/258)

[2018-03-19 08:43:29,559] [main] INFO  gc_log : [All-Users] after:  sizeOfPackedObjects=24153101, sizeOfLooseObjects=0, numberOfPackedObjects=88532, numberOfPackFiles=1, numberOfPackedRefs=12724, numberOfLooseRefs=0, numberOfLooseObjects=0