Qualcomm 2.7 -> 3.4+ Data Upgrade Status, Issues, and Plans

187 views
Skip to first unread message

Martin Fick

unread,
Aug 18, 2021, 4:59:27 PM8/18/21
to repo-d...@googlegroups.com
I am happy to report that we (Qualcomm) have made great progress in the
upgrade speeds of older Gerrit versions. More specifically, we have made enough
improvements that we currently can now perform an offline notedb migration of
our largest instance's latest dataset (3.6M+ changes, 16K+ projects, 4+TB) on
NFS in less than 3 hours using the latest tip version of Gerrit v2.16! This is
particularly great since our previous attempt was on an older data set (about
1/3 smaller) and on local disks (not NFS), and it took over 3 days!!!

------------- HIGHLIGHT -------------
For anyone considering upgrading from anything before 2.16, we STRONGLY
suggest trying the latest 2.16-stable tip to see if upgrades and migration is
much better for you now!

------------- PERFORMANCE OBJECTIVE -------------
As mentioned in the previous thread, our objective is to perform the entire
2.7-3.4+ upgrade in under 4 hours. While many improvements are still needed
to get there, this is huge leap towards that goal, and we believe we still
have many things we can potentially improve. The current full upgrade timeline
for the following 3 phases, for us, looks like :

* PHASE 1 - Upgrade from 2.7 to 2.16 (schema migrations)
-> ~3.5 hours

* PHASE 2 - NoteDB migration
-> ~2.5hours + 1 hour to repack All-Users (we hope to inline this in the
migration soon)

* PHASE 3 - Upgrade from 2.16 to 3.4+ (schema migrations + indexing)
-> ~22hrs

Which is still over a day and not realistically fast enough yet even if we
relax our objective a bit. However, note that the previously longest phases is
now the shortest phase, so I am optimistic that there are some low hanging
fruit in those other phases still.

------------- FUTURE IMPROVEMENTS -------------
--- Workflow ---
We would like to reduce the amount of user steps required to perform such a
long upgrade and we will likely be submitting changes soon to add additional
switches to make things simpler. For example:

-An "init" switch to make draft changes become private changes instead of the
WIP default (just completed this)

-An "init" switch to control the thread counts for indexing since it seems to
default to 1 currently

--- Performance ---
To meet our 4 hour objective we would like the to have something like:

* 0.5 hrs -> PHASE 1 - Upgrade from 2.7 to 2.16 (schema migrations)
* 2.0 hrs -> PHASE 2 - NoteDB migration
* 1.5 hrs -> PHASE 3 - Upgrade from 2.16 to 3.4+ (schema migrations +
indexing)

To get there, we will try:

* PHASE 1 - Upgrade from 2.7 to 2.16 (schema migrations)

-making schema 139 create an initial empty commit so that the user ref
does not get rewritten in schema 146
-not creating an empty commit in schema 146 if it is not going to be used
-continuing to optimize All-Users.git inefficiencies
-parallelizing some of the longer schema upgrades

PHASE 2 - NoteDB migration:

-adding repacking for All-Users to the noteDb migration after a certain number
of draft changes being written to the repo (currently repos are only repacked
after they have had many changes written to them, so All-Users is generally
unlikely to get repacked)

-ordering the "migration" phase slices the same way that the rebuild phase
slices are ordered (by project, instead of by change#s) to hopefully get some
better caching results (for reading change ref values).

-investigate why the diff caches seem to get overwritten during the migration
to see if we can eventually gain some more speed by pre-populating them

* PHASE 3 - Upgrade from 2.16 to 3.4+ (schema migrations + indexing)

-parallelizing the initial per project change ref scanning used just to split
the indexing up into slices (this alone can take more than an hour)
-re-using the results of the initial change ref scanning for each slice
instead of rescanning the changes for each slice
-investigating using the diff caches better
-investigating if there are bulk indexing options or APIs that can speed
things up

We have spent very little time so far on phases 1 and 3, hopefully we will get
more ideas, especially on phase 3! We welcome other suggestions!

Thanks,

-Martin

--
The Qualcomm Innovation Center, Inc. is a member of Code
Aurora Forum, hosted by The Linux Foundation

doug.r...@wandisco.com

unread,
Aug 19, 2021, 3:16:32 PM8/19/21
to Repo and Gerrit Discussion
Congratulations!  That's a GREAT reduction in NoteDB migration timing!

Patrick Hiesel

unread,
Aug 24, 2021, 7:27:38 AM8/24/21
to Martin Fick, repo-d...@googlegroups.com
Fantastic results, congratulations!

I recently looked at how we do index version upgrades and was wondering if instead of re-indexing all changes from scratch for every new schema version we could just index the new fields and add them to existing docs as well as drop fields that were dropped from the schema.

Some fields - for example anything related to diffs - are expensive to compute but also don't change often.

WDYT?

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/2046916.jXAeu0JPfY%40mfick-lnx.

Martin Fick

unread,
Aug 24, 2021, 3:59:47 PM8/24/21
to Patrick Hiesel, repo-d...@googlegroups.com
On 2021-08-24 05:27, Patrick Hiesel wrote:
> Fantastic results, congratulations!

Thanks!

> I recently looked at how we do index version upgrades and was
> wondering if instead of re-indexing all changes from scratch for every
> new schema version we could just index the new fields and add them to
> existing docs as well as drop fields that were dropped from the
> schema.

If that is faster, it would be nice. It may depend on which index
(Lucene/other) is being used whether this would help.

> Some fields - for example anything related to diffs - are expensive to
> compute but also don't change often.

It would be good if we could ensure that the diff caches are able to be
used, we have not been able to figure out how to get pre-populated
diff caches to be re-used on indexing or on migration.
Reply all
Reply to author
Forward
0 new messages