2.16.13 Offline reindex for projects stuck at the very end
269 views
Skip to first unread message
Nasser Grainawi
Dec 5, 2019, 11:42:45 PM12/5/19
to Repo and Gerrit Discussion
Hi everyone,
I’m trying to run an offline reindex for the ‘projects’ index on a site for the very first time (upgrading from 2.7…). Before this step, all that I’ve done is run ‘gerrit-2.9.6.war init’ and ‘gerrit-2.16.13 init --no-reindex --no-auto-start’.
Now I’m running:
$ java -server -verbose:gc -XX:+PrintGCDateStamps -Xloggc:/tmp/jvm_logs/jvm.log -XX:+PrintGCDetails -Xms96679m -Xmn31904m -XX:+UseParallelOldGC -XX:ParallelGCThreads=24 -XX:MetaspaceSize=128m -XX:+UseNUMA -XX:+UseBiasedLocking -XX:+AggressiveOpts -Xmx96679m -jar bin/gerrit-2.16.13.war reindex --index projects --threads 1
And it’s hung at the very end (progress output shows 1 project not complete).
Most jstacks show one index thread doing work and look like:
"Index-Batch-9" #57 prio=5 os_prio=0 tid=0x00007f38c8aba800 nid=0xea4 runnable [0x00007f20bb8f6000]
java.lang.Thread.State: RUNNABLE
at org.eclipse.jgit.internal.storage.file.WindowCache.removeAll(WindowCache.java:491)
at org.eclipse.jgit.internal.storage.file.WindowCache.purge(WindowCache.java:197)
at org.eclipse.jgit.internal.storage.file.PackFile.close(PackFile.java:330)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.reuseMap(ObjectDirectory.java:955)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.scanPacksImpl(ObjectDirectory.java:886)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.scanPacks(ObjectDirectory.java:877)
- locked <0x00007f2795c052c0> (a java.util.concurrent.atomic.AtomicReference)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.searchPacksAgain(ObjectDirectory.java:777)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:495)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:447)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:438)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:165)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:236)
at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:889)
at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:799)
at com.google.gerrit.server.git.meta.VersionedMetaData.load(VersionedMetaData.java:178)
at com.google.gerrit.server.git.meta.VersionedMetaData.load(VersionedMetaData.java:151)
at com.google.gerrit.server.git.meta.VersionedMetaData.load(VersionedMetaData.java:129)
at com.google.gerrit.server.project.ProjectCacheImpl$Loader.load(ProjectCacheImpl.java:280)
at com.google.gerrit.server.project.ProjectCacheImpl$Loader.load(ProjectCacheImpl.java:261)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154)
- locked <0x00007f28b837b418> (a com.google.common.cache.LocalCache$StrongAccessWriteEntry)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044)
at com.google.common.cache.LocalCache.get(LocalCache.java:3952)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3974)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4958)
at com.google.gerrit.server.project.ProjectCacheImpl.strictCheckedGet(ProjectCacheImpl.java:168)
at com.google.gerrit.server.project.ProjectCacheImpl.checkedGet(ProjectCacheImpl.java:148)
at com.google.gerrit.server.project.ProjectCacheImpl.get(ProjectCacheImpl.java:135)
at com.google.gerrit.server.index.project.ProjectIndexerImpl.index(ProjectIndexerImpl.java:75)
at com.google.gerrit.server.project.ProjectCacheImpl.evict(ProjectCacheImpl.java:187)
at com.google.gerrit.server.index.project.AllProjectsIndexer.lambda$reindexProjects$0(AllProjectsIndexer.java:76)
at com.google.gerrit.server.index.project.AllProjectsIndexer$$Lambda$154/1578026015.call(Unknown Source)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:125)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:57)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:78)
at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:83)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:646)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
If I run jstacks quickly back-to-back, I can catch this stack trace:
"Index-Batch-9" #57 prio=5 os_prio=0 tid=0x00007f38c8aba800 nid=0xea4 runnable [0x00007f20bb8f6000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.fillInStackTrace(Throwable.java:784)
- locked <0x00007f31c29a24a8> (a org.eclipse.jgit.errors.PackMismatchException)
at java.lang.Throwable.<init>(Throwable.java:266)
at java.lang.Exception.<init>(Exception.java:66)
at java.io.IOException.<init>(IOException.java:58)
at org.eclipse.jgit.errors.PackMismatchException.<init>(PackMismatchException.java:61)
at org.eclipse.jgit.internal.storage.file.PackFile.onOpenPack(PackFile.java:812)
at org.eclipse.jgit.internal.storage.file.PackFile.doOpen(PackFile.java:693)
- locked <0x00007f31c299ca08> (a java.lang.Object)
at org.eclipse.jgit.internal.storage.file.PackFile.beginWindowCache(PackFile.java:672)
- locked <0x00007f31c299c968> (a org.eclipse.jgit.internal.storage.file.PackFile)
at org.eclipse.jgit.internal.storage.file.WindowCache.load(WindowCache.java:295)
at org.eclipse.jgit.internal.storage.file.WindowCache.getOrLoad(WindowCache.java:379)
- locked <0x00007f24914ae898> (a org.eclipse.jgit.internal.storage.file.WindowCache$Lock)
at org.eclipse.jgit.internal.storage.file.WindowCache.get(WindowCache.java:184)
at org.eclipse.jgit.internal.storage.file.WindowCursor.pin(WindowCursor.java:360)
at org.eclipse.jgit.internal.storage.file.WindowCursor.copy(WindowCursor.java:259)
at org.eclipse.jgit.internal.storage.file.PackFile.readFully(PackFile.java:648)
at org.eclipse.jgit.internal.storage.file.PackFile.load(PackFile.java:830)
at org.eclipse.jgit.internal.storage.file.PackFile.get(PackFile.java:318)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedObject(ObjectDirectory.java:489)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openPackedFromSelfOrAlternate(ObjectDirectory.java:447)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:438)
at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:165)
at org.eclipse.jgit.lib.ObjectReader.open(ObjectReader.java:236)
at org.eclipse.jgit.revwalk.RevWalk.parseAny(RevWalk.java:889)
at org.eclipse.jgit.revwalk.RevWalk.parseCommit(RevWalk.java:799)
at com.google.gerrit.server.git.meta.VersionedMetaData.load(VersionedMetaData.java:178)
[snip, same as above]
There’s no exceptions being printed to the console. Anyone know how I can move forward on getting this to finish?
Thanks,
Nasser
Saša Živkov
Dec 6, 2019, 9:41:49 AM12/6/19
to Nasser Grainawi, Repo and Gerrit Discussion
On Fri, Dec 6, 2019 at 5:42 AM Nasser Grainawi <nas...@codeaurora.org> wrote:
Hi everyone,I’m trying to run an offline reindex for the ‘projects’ index on a site for the very first time (upgrading from 2.7…). Before this step, all that I’ve done is run ‘gerrit-2.9.6.war init’ and ‘gerrit-2.16.13 init --no-reindex --no-auto-start’.Now I’m running:$ java -server -verbose:gc -XX:+PrintGCDateStamps -Xloggc:/tmp/jvm_logs/jvm.log -XX:+PrintGCDetails -Xms96679m -Xmn31904m -XX:+UseParallelOldGC -XX:ParallelGCThreads=24 -XX:MetaspaceSize=128m -XX:+UseNUMA -XX:+UseBiasedLocking -XX:+AggressiveOpts -Xmx96679m -jar bin/gerrit-2.16.13.war reindex --index projects --threads 1And it’s hung at the very end (progress output shows 1 project not complete).Most jstacks show one index thread doing work and look like:
Have you observed this same thread in several subsequent thread dumps? Does it change?
This looks like some corrupt pack file?
--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en
---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/0101016ed9844feb-5f8e6b7c-a629-4c82-a3a2-471e38619b9c-000000%40us-west-2.amazonses.com.
Nasser Grainawi
Dec 6, 2019, 9:55:45 AM12/6/19
to Saša Živkov, Repo and Gerrit Discussion
On Dec 6, 2019, at 7:41 AM, Saša Živkov <ziv...@gmail.com> wrote:On Fri, Dec 6, 2019 at 5:42 AM Nasser Grainawi <nas...@codeaurora.org> wrote:Hi everyone,I’m trying to run an offline reindex for the ‘projects’ index on a site for the very first time (upgrading from 2.7…). Before this step, all that I’ve done is run ‘gerrit-2.9.6.war init’ and ‘gerrit-2.16.13 init --no-reindex --no-auto-start’.Now I’m running:$ java -server -verbose:gc -XX:+PrintGCDateStamps -Xloggc:/tmp/jvm_logs/jvm.log -XX:+PrintGCDetails -Xms96679m -Xmn31904m -XX:+UseParallelOldGC -XX:ParallelGCThreads=24 -XX:MetaspaceSize=128m -XX:+UseNUMA -XX:+UseBiasedLocking -XX:+AggressiveOpts -Xmx96679m -jar bin/gerrit-2.16.13.war reindex --index projects --threads 1And it’s hung at the very end (progress output shows 1 project not complete).Most jstacks show one index thread doing work and look like:Have you observed this same thread in several subsequent thread dumps? Does it change?
Yes, I have. It mostly looks the same, but I suspect it’s just in a loop. I occasionally have caught it at this state too:
"Index-Batch-9" #57 prio=5 os_prio=0 tid=0x00007f38c8aba800 nid=0xea4 runnable [0x00007f20bb8f6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.fs.UnixNativeDispatcher.lstat0(Native Method)
at sun.nio.fs.UnixNativeDispatcher.lstat(UnixNativeDispatcher.java:300)
at sun.nio.fs.UnixFileAttributes.get(UnixFileAttributes.java:72)
at sun.nio.fs.UnixFileAttributeViews$Basic.readAttributes(UnixFileAttributeViews.java:52)
at sun.nio.fs.UnixFileSystemProvider.readAttributes(UnixFileSystemProvider.java:144)
at sun.nio.fs.LinuxFileSystemProvider.readAttributes(LinuxFileSystemProvider.java:99)
at java.nio.file.Files.readAttributes(Files.java:1737)
at org.eclipse.jgit.util.FileUtils.fileAttributes(FileUtils.java:703)
at org.eclipse.jgit.util.FS.fileAttributes(FS.java:1036)
at org.eclipse.jgit.internal.storage.file.FileSnapshot.<init>(FileSnapshot.java:261)
at org.eclipse.jgit.internal.storage.file.FileSnapshot.<init>(FileSnapshot.java:238)
at org.eclipse.jgit.internal.storage.file.PackFileSnapshot.<init>(PackFileSnapshot.java:76)
at org.eclipse.jgit.internal.storage.file.PackFileSnapshot.save(PackFileSnapshot.java:67)
at org.eclipse.jgit.internal.storage.file.PackFile.<init>(PackFile.java:175)
at org.eclipse.jgit.internal.storage.file.ObjectDirectory.scanPacksImpl(ObjectDirectory.java:922)
Ok. Any suggestions on how I find which one? I have 11,000 projects.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CACrmCORGp_6mBD2jexWYM7_1qXrTUsO2cBs2nnDj0HXEjoPNFw%40mail.gmail.com.
Nasser Grainawi
Dec 6, 2019, 11:06:24 AM12/6/19
to Saša Živkov, Repo and Gerrit Discussion
I went through and set core.trustFolderStat to ’true’ instead of false and now I have some pack mismatch exceptions printed to the console and it doesn’t hang forever. This looks like a legit bug though based on how I’m reading the code in org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/ObjectDirectory.java. searchPacksAgain() will always return true when trustFolderStat is false and we end up in a loop forever.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/0101016edbb57faf-f1c9d058-5730-4f95-afb5-7d5fe039c4b4-000000%40us-west-2.amazonses.com.
Luca Milanesio
Dec 6, 2019, 11:07:50 AM12/6/19
to Nasser Grainawi, Luca Milanesio, Saša Živkov, Repo and Gerrit Discussion
I believe that bug was fixed in the latest v2.16 :-(
Luca.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/0101016edbf63b97-2089617a-eb78-4a4a-8c44-d53a9b5c13a1-000000%40us-west-2.amazonses.com.
Matthias Sohn
Dec 6, 2019, 11:33:50 AM12/6/19
to Nasser Grainawi, Saša Živkov, Repo and Gerrit Discussion
On Fri, Dec 6, 2019 at 3:55 PM Nasser Grainawi <nas...@codeaurora.org> wrote:
On Dec 6, 2019, at 7:41 AM, Saša Živkov <ziv...@gmail.com> wrote:On Fri, Dec 6, 2019 at 5:42 AM Nasser Grainawi <nas...@codeaurora.org> wrote:Hi everyone,I’m trying to run an offline reindex for the ‘projects’ index on a site for the very first time (upgrading from 2.7…). Before this step, all that I’ve done is run ‘gerrit-2.9.6.war init’ and ‘gerrit-2.16.13 init --no-reindex --no-auto-start’.
I think doing a jump upgrade from 2.7 to 2.9.6 and from there to 2.16.13 is not supported.
How do you ensure all the migrations in between these version are run ?
I'd expect you have to run gerrit init for each minor version above 2.7 to upgrade to 2.8 then to 2.9 etc ... all the way up to 2.16.13
Maybe you are hitting some exception in ObjectDirectory which causes repeated rescanning for packfiles.
You can try to configure the logger for ObjectDirectory to WARNING or DEBUG to get more tracing.
The PackMismatchException should have the packfile path in the error message, do you find that in the error_log ?
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/0101016edbb57faf-f1c9d058-5730-4f95-afb5-7d5fe039c4b4-000000%40us-west-2.amazonses.com.
Luca Milanesio
Dec 6, 2019, 11:43:28 AM12/6/19
to Nasser Grainawi, Luca Milanesio, Saša Živkov, Repo and Gerrit Discussion, Matthias Sohn
On 6 Dec 2019, at 16:33, Matthias Sohn <matthi...@gmail.com> wrote:On Fri, Dec 6, 2019 at 3:55 PM Nasser Grainawi <nas...@codeaurora.org> wrote:On Dec 6, 2019, at 7:41 AM, Saša Živkov <ziv...@gmail.com> wrote:On Fri, Dec 6, 2019 at 5:42 AM Nasser Grainawi <nas...@codeaurora.org> wrote:Hi everyone,I’m trying to run an offline reindex for the ‘projects’ index on a site for the very first time (upgrading from 2.7…). Before this step, all that I’ve done is run ‘gerrit-2.9.6.war init’ and ‘gerrit-2.16.13 init --no-reindex --no-auto-start’.I think doing a jump upgrade from 2.7 to 2.9.6 and from there to 2.16.13 is not supported.How do you ensure all the migrations in between these version are run ?I'd expect you have to run gerrit init for each minor version above 2.7 to upgrade to 2.8 then to 2.9 etc ... all the way up to 2.16.13
That is what the init does already automatically.
The problem is the size of the setup and the number of schema migrations: I reckon the init took a *veeeery long time*.
@Nasser can you share the output and the timings of the init?
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAKSZd3Riuek8gmTwvz4Thwv9bq%2Bw3deg8%2BJePh2xfHci9rxdUQ%40mail.gmail.com.
Nasser Grainawi
Dec 6, 2019, 12:26:12 PM12/6/19
to Luca Milanesio, Saša Živkov, Repo and Gerrit Discussion, Matthias Sohn
On Dec 6, 2019, at 9:43 AM, Luca Milanesio <luca.mi...@gmail.com> wrote:On 6 Dec 2019, at 16:33, Matthias Sohn <matthi...@gmail.com> wrote:On Fri, Dec 6, 2019 at 3:55 PM Nasser Grainawi <nas...@codeaurora.org> wrote:On Dec 6, 2019, at 7:41 AM, Saša Živkov <ziv...@gmail.com> wrote:On Fri, Dec 6, 2019 at 5:42 AM Nasser Grainawi <nas...@codeaurora.org> wrote:Hi everyone,I’m trying to run an offline reindex for the ‘projects’ index on a site for the very first time (upgrading from 2.7…). Before this step, all that I’ve done is run ‘gerrit-2.9.6.war init’ and ‘gerrit-2.16.13 init --no-reindex --no-auto-start’.I think doing a jump upgrade from 2.7 to 2.9.6 and from there to 2.16.13 is not supported.How do you ensure all the migrations in between these version are run ?I'd expect you have to run gerrit init for each minor version above 2.7 to upgrade to 2.8 then to 2.9 etc ... all the way up to 2.16.13That is what the init does already automatically.The problem is the size of the setup and the number of schema migrations: I reckon the init took a *veeeery long time*.@Nasser can you share the output and the timings of the init?
Took about 1 hour, so I thought it worked, but I’m guessing it didn’t. I did see the ALTER TABLE command running for Schema 129, so I assumed it got past that successfully. It does have a stack trace at the end, but it seemed like something it got past successfully… maybe not?
This output is from a re-run just now, so the timing doesn’t match the original run. The exception at the end does.
$ date; time java -server -verbose:gc -XX:+PrintGCDateStamps -Xloggc:/tmp/jvm_logs/jvm.log -XX:+PrintGCDetails -Xms96679m -Xmn31904m -XX:+UseParallelOldGC -XX:ParallelGCThreads=24 -XX:MetaspaceSize=128m -XX:+UseNUMA -XX:+UseBiasedLocking -XX:+AggressiveOpts -Xmx96679m -jar bin/gerrit-2.16.13.war init --no-auto-start --no-reindex -d . ; date
Fri Dec 6 17:13:38 UTC 2019
Using secure store: com.google.gerrit.server.securestore.DefaultSecureStore
*** Gerrit Code Review 2.16.13
***
*** Git Repositories
***
Location of Git repositories [/srv/quic/data/prod/git/repos/android]:
*** SQL Database
***
Database server type [postgresql]:
Server hostname [database]:
Server port [(postgresql default)]:
Database name [reviewdb]:
Database username [gerrit2]:
gerrit2's password :
confirm password :
*** Index
***
Type [lucene/?]:
The index must be rebuilt before starting Gerrit:
java -jar gerrit.war reindex -d site_path
*** User Authentication
***
Authentication method [ldap/?]:
Git/HTTP authentication [http/?]:
LDAP server [ldaps://<ldap server>]:
LDAP username [<ldap user>]:
<ldap user>'s password :
confirm password :
Account BaseDN [<account basedn>]:
Group BaseDN [<group basedn>]:
Enable signed push support [y/N]?
*** Email Delivery
***
SMTP server hostname [localhost]:
SMTP server port [(default)]:
SMTP encryption [none/?]:
SMTP username :
*** Container Process
***
Run as [nasserg]:
Java runtime [/usr/local/openjdk-8]:
Upgrade bin/gerrit.war [Y/n]?
Copying gerrit-2.16.13.war to bin/gerrit.war
*** SSH Daemon
***
Listen on address [*]:
Listen on port [29418]:
Generating SSH host key ... rsa...No user exists for uid 66082 #running inside a container where I didn’t run add-user, ignore this
ed25519...No user exists for uid 66082
ecdsa 256...No user exists for uid 66082
ecdsa 384...No user exists for uid 66082
ecdsa 521...No user exists for uid 66082
done
*** HTTP Daemon
***
Behind reverse proxy [Y/n]?
Proxy uses SSL (https://) [Y/n]?
Subdirectory on proxy server [/]:
Listen on address [*]:
Listen on port [8081]:
Canonical URL [https://<testserver>/]:
*** Cache
***
Delete cache file /srv/quic/apps/gerrit/disk_cache/diff_intraline.h2.db [y/N]?
Delete cache file /srv/quic/apps/gerrit/disk_cache/diff.lock.db [y/N]?
Delete cache file /srv/quic/apps/gerrit/disk_cache/git_tags.h2.db [y/N]?
Delete cache file /srv/quic/apps/gerrit/disk_cache/diff_summary.h2.db [y/N]?
Delete cache file /srv/quic/apps/gerrit/disk_cache/mergeability.h2.db [y/N]?
Delete cache file /srv/quic/apps/gerrit/disk_cache/change_kind.h2.db [y/N]?
*** Plugins
***
Installing plugins.
Install plugin codemirror-editor version v2.16.13 [Y/n]?
codemirror-editor v2.16.13 is already installed, overwrite it [Y/n]?
Updated codemirror-editor to v2.16.13
Install plugin commit-message-length-validator version v2.16.13 [Y/n]?
commit-message-length-validator v2.16.13 is already installed, overwrite it [Y/n]?
Updated commit-message-length-validator to v2.16.13
Install plugin download-commands version v2.16.13 [Y/n]?
download-commands v2.16.13 is already installed, overwrite it [Y/n]?
Updated download-commands to v2.16.13
Install plugin hooks version v2.16.13 [Y/n]?
hooks v2.16.13 is already installed, overwrite it [Y/n]?
Updated hooks to v2.16.13
Install plugin replication version v2.16.13 [Y/n]?
replication v2.16.13 is already installed, overwrite it [Y/n]?
Updated replication to v2.16.13
Install plugin reviewnotes version v2.16.13 [Y/n]?
reviewnotes v2.16.13 is already installed, overwrite it [Y/n]?
Updated reviewnotes to v2.16.13
Install plugin singleusergroup version v2.16.13 [Y/n]?
singleusergroup v2.16.13 is already installed, overwrite it [Y/n]?
Updated singleusergroup to v2.16.13
Initializing plugins.
Upgrading schema to 94 ...
Upgrading schema to 95 ...
Upgrading schema to 96 ...
Upgrading schema to 97 ...
Upgrading schema to 98 ...
Upgrading schema to 99 ...
Upgrading schema to 100 ...
Upgrading schema to 101 ...
Upgrading schema to 102 ...
Upgrading schema to 103 ...
Upgrading schema to 104 ...
Upgrading schema to 105 ...
Upgrading schema to 106 ...
Upgrading schema to 107 ...
Upgrading schema to 108 ...
Upgrading schema to 109 ...
Upgrading schema to 110 ...
Upgrading schema to 111 ...
Upgrading schema to 112 ...
Upgrading schema to 113 ...
Upgrading schema to 114 ...
Upgrading schema to 115 ...
Upgrading schema to 116 ...
Upgrading schema to 117 ...
Upgrading schema to 118 ...
Upgrading schema to 119 ...
Upgrading schema to 120 ...
Upgrading schema to 121 ...
Upgrading schema to 122 ...
Upgrading schema to 123 ...
Upgrading schema to 124 ...
Upgrading schema to 125 ...
Upgrading schema to 126 ...
Upgrading schema to 127 ...
Upgrading schema to 128 ...
Upgrading schema to 129 ...
Upgrading schema to 130 ...
Upgrading schema to 131 ...
Upgrading schema to 132 ...
Upgrading schema to 133 ...
Upgrading schema to 134 ...
Upgrading schema to 135 ...
Upgrading schema to 136 ...
Upgrading schema to 137 ...
Upgrading schema to 138 ...
Upgrading schema to 139 ...
Upgrading schema to 140 ...
Upgrading schema to 141 ...
Upgrading schema to 142 ...
Upgrading schema to 143 ...
Upgrading schema to 144 ...
Upgrading schema to 145 ...
Upgrading schema to 146 ...
Upgrading schema to 147 ...
Upgrading schema to 148 ...
Upgrading schema to 149 ...
Upgrading schema to 150 ...
Upgrading schema to 151 ...
Upgrading schema to 152 ...
Upgrading schema to 153 ...
Upgrading schema to 154 ...
Upgrading schema to 155 ...
Upgrading schema to 156 ...
Upgrading schema to 157 ...
Upgrading schema to 158 ...
Upgrading schema to 159 ...
Upgrading schema to 160 ...
Upgrading schema to 161 ...
Upgrading schema to 162 ...
Upgrading schema to 163 ...
Upgrading schema to 164 ...
Upgrading schema to 165 ...
Upgrading schema to 166 ...
Upgrading schema to 167 ...
Upgrading schema to 168 ...
Upgrading schema to 169 ...
Upgrading schema to 170 ...
Migrating data to schema 94 ...
Couldn't upgrade schema. Expected if slave and read-only database
ERROR com.google.gerrit.pgm.init.BaseInit : Couldn't upgrade schema. Expected if slave and read-only database
com.google.gwtorm.server.OrmException: Cannot upgrade schema
at com.google.gerrit.server.schema.SchemaUpdater.update(SchemaUpdater.java:113)
at com.google.gerrit.pgm.init.BaseInit$SiteRun.upgradeSchema(BaseInit.java:389)
at com.google.gerrit.pgm.init.BaseInit.run(BaseInit.java:145)
at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:225)
at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:121)
at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:65)
at Main.main(Main.java:28)
Caused by: org.postgresql.util.PSQLException: ERROR: relation "patch_sets_byrevision" already exists
at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2440)
at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2183)
at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:308)
at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:266)
at com.google.gerrit.server.schema.Schema_94.migrateData(Schema_94.java:32)
at com.google.gerrit.server.schema.SchemaVersion.migrateData(SchemaVersion.java:155)
at com.google.gerrit.server.schema.SchemaVersion.upgradeFrom(SchemaVersion.java:94)
at com.google.gerrit.server.schema.SchemaVersion.check(SchemaVersion.java:85)
at com.google.gerrit.server.schema.SchemaUpdater.update(SchemaUpdater.java:111)
... 11 more
Initialized <site>
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/9E0E54AF-7CD3-40A4-A28B-B1E79B99FBE2%40gmail.com.
luca.mi...@gmail.com
Dec 6, 2019, 12:53:15 PM12/6/19
to Nasser Grainawi, Saša Živkov, Repo and Gerrit Discussion, Matthias Sohn
On Dec 6, 2019, at 9:43 AM, Luca Milanesio <luca.mi...@gmail.com> wrote:On 6 Dec 2019, at 16:33, Matthias Sohn <matthi...@gmail.com> wrote:On Fri, Dec 6, 2019 at 3:55 PM Nasser Grainawi <nas...@codeaurora.org> wrote:On Dec 6, 2019, at 7:41 AM, Saša Živkov <ziv...@gmail.com> wrote:On Fri, Dec 6, 2019 at 5:42 AM Nasser Grainawi <nas...@codeaurora.org> wrote:Hi everyone,I’m trying to run an offline reindex for the ‘projects’ index on a site for the very first time (upgrading from 2.7…). Before this step, all that I’ve done is run ‘gerrit-2.9.6.war init’ and ‘gerrit-2.16.13 init --no-reindex --no-auto-start’.I think doing a jump upgrade from 2.7 to 2.9.6 and from there to 2.16.13 is not supported.How do you ensure all the migrations in between these version are run ?I'd expect you have to run gerrit init for each minor version above 2.7 to upgrade to 2.8 then to 2.9 etc ... all the way up to 2.16.13That is what the init does already automatically.The problem is the size of the setup and the number of schema migrations: I reckon the init took a *veeeery long time*.@Nasser can you share the output and the timings of the init?Took about 1 hour, so I thought it worked, but I’m guessing it didn’t. I did see the ALTER TABLE command running for Schema 129, so I assumed it got past that successfully. It does have a stack trace at the end, but it seemed like something it got past successfully… maybe not?This output is from a re-run just now, so the timing doesn’t match the original run. The exception at the end does.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/0101016edc3f4013-932f1e09-f3db-41eb-9f10-64be4c2f24f9-000000%40us-west-2.amazonses.com.
Nasser Grainawi
Dec 6, 2019, 1:24:45 PM12/6/19
to Luca Milanesio, Saša Živkov, Repo and Gerrit Discussion
I’m using 2.16.13 which is the latest released version so it does not appear to be fixed.
Nasser Grainawi
Dec 6, 2019, 1:26:59 PM12/6/19
to Luca Milanesio, Saša Živkov, Repo and Gerrit Discussion, Matthias Sohn
On Dec 6, 2019, at 9:43 AM, Luca Milanesio <luca.mi...@gmail.com> wrote:On 6 Dec 2019, at 16:33, Matthias Sohn <matthi...@gmail.com> wrote:On Fri, Dec 6, 2019 at 3:55 PM Nasser Grainawi <nas...@codeaurora.org> wrote:On Dec 6, 2019, at 7:41 AM, Saša Živkov <ziv...@gmail.com> wrote:On Fri, Dec 6, 2019 at 5:42 AM Nasser Grainawi <nas...@codeaurora.org> wrote:Hi everyone,I’m trying to run an offline reindex for the ‘projects’ index on a site for the very first time (upgrading from 2.7…). Before this step, all that I’ve done is run ‘gerrit-2.9.6.war init’ and ‘gerrit-2.16.13 init --no-reindex --no-auto-start’.I think doing a jump upgrade from 2.7 to 2.9.6 and from there to 2.16.13 is not supported.How do you ensure all the migrations in between these version are run ?I'd expect you have to run gerrit init for each minor version above 2.7 to upgrade to 2.8 then to 2.9 etc ... all the way up to 2.16.13That is what the init does already automatically.
Looks like init is not doing it’s job correctly. I went back and started again with 2.9, then did 2.10, then 2.11, now 2.12. 2.12 is taking a while “Updating groups for open changes” in schema 108, but the others were fast.
I started down this path after I hit https://bugs.chromium.org/p/gerrit/issues/detail?id=3123 and saw the note in https://gerrit-review.googlesource.com/c/gerrit/+/63678 and then even after 2.10 was successful, I hit more errors trying 2.16 again (using latest patch versions always).
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/9E0E54AF-7CD3-40A4-A28B-B1E79B99FBE2%40gmail.com.
Matthias Sohn
Dec 6, 2019, 6:25:25 PM12/6/19
to Nasser Grainawi, Saša Živkov, Repo and Gerrit Discussion
why ?
boolean searchPacksAgain(PackList old) {
// Whether to trust the pack folder's modification time. If set
// to false we will always scan the .git/objects/pack folder to
// check for new pack files. If set to true (default) we use the
// lastmodified attribute of the folder and assume that no new
// pack files can be in this folder if his modification time has
// not changed.
boolean trustFolderStat = config.getBoolean(
ConfigConstants.CONFIG_CORE_SECTION,
ConfigConstants.CONFIG_KEY_TRUSTFOLDERSTAT, true);
return ((!trustFolderStat) || old.snapshot.isModified(packDirectory))
&& old != scanPacks(old);
}
if trustFolderStat == false the first term in ((!trustFolderStat) || old.snapshot.isModified(packDirectory)) is true
This means it does not evaluate the second term which would aks the packDirectory's FileSnapshot if the meta data
for the packDirectory says the directory was modified. This is the purpose of trustFolderStat, if we don't trust the
directory metaData to be able to detect changes in the list of files contained in the directory
we have to scan the packDirectory again to see if the list of packfiles changed which is done in the last term
&& old != scanPacks(old)
scanPacksImpl lists the files in the pack directory and looks for .idx files, checks if a corresponding .pack file exists and
if the packFile was modified by looking at its cached FileSnapshot
this is of course slower than checking the meta data of the packDirectory to detect if we have to rescan the packDirectory
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/0101016edbf63b97-2089617a-eb78-4a4a-8c44-d53a9b5c13a1-000000%40us-west-2.amazonses.com.
Nasser Grainawi
Dec 9, 2019, 12:28:45 AM12/9/19
to Matthias Sohn, Saša Živkov, Repo and Gerrit Discussion
But if there is a pack with a mismatched idx, we keep finding that problem and never exit the loop in openPackedObject() (just searches packs again each time PackMismatchException is thrown, but the list of packs doesn’t change I think).
Nasser Grainawi
Dec 10, 2019, 6:11:18 PM12/10/19
to Luca Milanesio, Saša Živkov, Repo and Gerrit Discussion, Matthias Sohn
On Dec 6, 2019, at 11:26 AM, Nasser Grainawi <nas...@codeaurora.org> wrote:On Dec 6, 2019, at 9:43 AM, Luca Milanesio <luca.mi...@gmail.com> wrote:On 6 Dec 2019, at 16:33, Matthias Sohn <matthi...@gmail.com> wrote:On Fri, Dec 6, 2019 at 3:55 PM Nasser Grainawi <nas...@codeaurora.org> wrote:On Dec 6, 2019, at 7:41 AM, Saša Živkov <ziv...@gmail.com> wrote:On Fri, Dec 6, 2019 at 5:42 AM Nasser Grainawi <nas...@codeaurora.org> wrote:Hi everyone,I’m trying to run an offline reindex for the ‘projects’ index on a site for the very first time (upgrading from 2.7…). Before this step, all that I’ve done is run ‘gerrit-2.9.6.war init’ and ‘gerrit-2.16.13 init --no-reindex --no-auto-start’.I think doing a jump upgrade from 2.7 to 2.9.6 and from there to 2.16.13 is not supported.How do you ensure all the migrations in between these version are run ?I'd expect you have to run gerrit init for each minor version above 2.7 to upgrade to 2.8 then to 2.9 etc ... all the way up to 2.16.13That is what the init does already automatically.Looks like init is not doing it’s job correctly. I went back and started again with 2.9, then did 2.10, then 2.11, now 2.12. 2.12 is taking a while “Updating groups for open changes” in schema 108, but the others were fast.I started down this path after I hit https://bugs.chromium.org/p/gerrit/issues/detail?id=3123 and saw the note in https://gerrit-review.googlesource.com/c/gerrit/+/63678 and then even after 2.10 was successful, I hit more errors trying 2.16 again (using latest patch versions always).The problem is the size of the setup and the number of schema migrations: I reckon the init took a *veeeery long time*.
Ok, made it to 2.16 (inits only)! Took ~3 hours. There are some outliers that took much longer than expected, so we’re going to see if we can speed those up some. I’ll send separate mails on those as we investigate further.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/0101016edc76ea09-e226d06c-a335-4921-86a2-4d4bb83a5fd7-000000%40us-west-2.amazonses.com.
Reply all
Reply to author
Forward
0 new messages