error.log message (old version 2.8.1)

Miten Mehta

unread,

Jan 29, 2024, 8:42:14 AM1/29/24

to Repo and Gerrit Discussion

Hi,

I am currently supporting 2.8.1 version and cannot upgrade.

1. I see below warning in error log. is there some thing I need to fix ? as such at times it continously prints this and then I have to restart the gerrit process else it might not serve requests from users.

[2024-01-25 16:30:34,116] WARN org.eclipse.jetty.io.nio : Dispatched Failed! SCEP@78401eca{l(/10.20.191.237:51257)<->r(/10.135.128.153:8080),d=false,open=true,ishut=false,oshut=false,rb=false,wb=false,w=true,i=1r}-{AsyncHttpConnection@196ae0f1,g=HttpGenerator{s=0,h=-1,b=-1,c=-1},p=HttpParser{s=-14,l=0,c=0},r=0} to org.eclipse.jetty.server.nio.SelectChannelConnector$ConnectorSelectorManager@224bf198

2. Also one more error. do I need to remove some thing from some db table ?:

javax.naming.PartialResultException: [LDAP: error code 10 - 0000202B: RefErr: DSID-0310075F, data 0, 1 access points
ref 1: 'rjil.ril.com'
^@]; remaining name 'CN=PRODESEQ2507CP01APPSUPPO,OU=CyberArk,DC=rjil,DC=ril,DC=com'
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:3081)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2998)
at com.sun.jndi.ldap.LdapCtx.c_getAttributes(LdapCtx.java:1357)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:235)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:141)
at javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:152)
at javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:147)
at com.google.gerrit.server.auth.ldap.Helper.recursivelyExpandGroups(Helper.java:267)
at com.google.gerrit.server.auth.ldap.Helper.recursivelyExpandGroups(Helper.java:285)
at com.google.gerrit.server.auth.ldap.Helper.queryForGroups(Helper.java:217)
at com.google.gerrit.server.auth.ldap.LdapRealm.authenticate(LdapRealm.java:232)
at com.google.gerrit.server.account.AccountManager.authenticate(AccountManager.java:107)
at com.google.gerrit.httpd.auth.ldap.LdapLoginServlet.doPost(LdapLoginServlet.java:128)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:641)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:278)
at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:268)
at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:180)
at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:93)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
at com.google.gerrit.pgm.http.jetty.GetUserFilter.doFilter(GetUserFilter.java:76)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gwtexpui.server.CacheControlFilter.doFilter(CacheControlFilter.java:70)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:113)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:64)

at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:57)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:75)
at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:120)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:132)
at com.google.inject.servlet.GuiceFilter$1.call(GuiceFilter.java:129)
at com.google.inject.servlet.GuiceFilter$Context.call(GuiceFilter.java:206)
at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:129)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:67)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
at org.eclipse.jetty.server.Server.handle(Server.java:365)
at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:937)
at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:998)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:856)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627)
at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
at java.lang.Thread.run(Thread.java:750)

3. Also postfix mails are not coming from gerrit and do not see any thing in logs either. I tested postfix from command line and am getting mails.

any thing to do with secure.config:

[auth]
registerEmailPrivateKey = masked*****=

Regards,

Miten.

Matthias Sohn

unread,

Jan 29, 2024, 10:01:48 AM1/29/24

to Miten Mehta, Repo and Gerrit Discussion

On Mon, Jan 29, 2024 at 2:42 PM Miten Mehta <india...@gmail.com> wrote:

Hi,

I am currently supporting 2.8.1 version and cannot upgrade.

This version was released more than 10 years ago and has been EOL for a long time.
The latest release is 3.9.1, community support [1] for versions older than 3.7 ended [2].

Why can't you upgrade ?

[1] https://www.gerritcodereview.com/support.html#community-support

[2] https://www.gerritcodereview.com/support.html and https://endoflife.date/gerrit

-Matthias

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/d349ad6f-052b-48a0-b848-3b4f0c366820n%40googlegroups.com.

Miten Mehta

unread,

Jan 30, 2024, 2:17:20 AM1/30/24

to Repo and Gerrit Discussion

Hi,

I had tried upgrading but it did was not simple as replacing the war and may be running sql scripts. I would expect that install dir repo area should get picked up without any issues with new upgrade and also database changes should not cause any issues for user access/projects list etc. but it was not so. if you have some dedicated spoc for it then I can retry.

Regards,

Miten.

Matthias Sohn

unread,

Jan 30, 2024, 2:53:48 AM1/30/24

to Miten Mehta, Repo and Gerrit Discussion

On Tue, Jan 30, 2024 at 8:17 AM Miten Mehta <india...@gmail.com> wrote:

Hi,

I had tried upgrading but it did was not simple as replacing the war and may be running sql scripts. I would expect that install dir repo area should get picked up without any issues with new upgrade and also database changes should not cause any issues for user access/projects list etc. but it was not so. if you have some dedicated spoc for it then I can retry.

Please avoid top posting on this list, we prefer interleaved style [1].

Follow the release notes [2] for upgrades, go minor release by minor release (e.g. from 2.8.1 to 2.9.5).

Always use the latest service release per minor release (e.g. 2.9.5 for 2.9.x).

First read the "Important notes" for each version bump to get an overview what needs to be done.

10 years after the fact it might be tricky to find all prerequisites, e.g. old Java and database versions.

[1] https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

[2] https://www.gerritcodereview.com/releases-readme.html

[3] https://www.gerritcodereview.com/2.9.html

https://gerrit-documentation.storage.googleapis.com/ReleaseNotes/ReleaseNotes-2.9.html

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/7dc9466d-0678-4223-a3a8-4ca43dae4a1cn%40googlegroups.com.

Nasser Grainawi

unread,

Feb 1, 2024, 12:32:55 AM2/1/24

to Matthias Sohn, Miten Mehta, Repo and Gerrit Discussion

On Mon, Jan 29, 2024 at 11:53 PM Matthias Sohn <matthi...@gmail.com> wrote:

On Tue, Jan 30, 2024 at 8:17 AM Miten Mehta <india...@gmail.com> wrote:
Hi,

I had tried upgrading but it did was not simple as replacing the war and may be running sql scripts. I would expect that install dir repo area should get picked up without any issues with new upgrade and also database changes should not cause any issues for user access/projects list etc. but it was not so. if you have some dedicated spoc for it then I can retry.

Please avoid top posting on this list, we prefer interleaved style [1].

Follow the release notes [2] for upgrades, go minor release by minor release (e.g. from 2.8.1 to 2.9.5).
Always use the latest service release per minor release (e.g. 2.9.5 for 2.9.x).
First read the "Important notes" for each version bump to get an overview what needs to be done.

10 years after the fact it might be tricky to find all prerequisites, e.g. old Java and database versions.

Yes, there's some challenges there. We successfully upgraded from 2.7 (a forked version) to 3.5 a little over 5 months ago. I'm overdue on a write up concluding the journey, but you can read some of the earlier progress we made here: https://groups.google.com/g/repo-discuss/c/WVwvngCkRMs/

You can do a minimal set of upgrades instead of doing every minor release. From 2.7, we needed 2.9, 2.11, 2.16, then the NoteDb migration on 2.16, then 3.5, then offline reindexing with 3.5. Starting with 2.8 you might be able to get directly to 2.11, but I'm not sure.

We did these commands effectively:
java -jar gerrit-war-2.9.5.war init --no-auto-start --batch -d /path/to/gerrit/site_dir

java -jar gerrit-war-2.11.12.war init --no-auto-start --batch -d /path/to/gerrit/site_dir

# build 2.16.war from the latest stable-2.16 commit

java -jar gerrit-war-2.16.war init --no-auto-start --no-reindex --batch --migrate-draft-to PRIVATE -d /path/to/gerrit/site_dir

java -jar gerrit-war-2.16.war migrate-to-note-db --reindex=false --threads 24 -d /path/to/gerrit/site_dir

# run 3.5 (our target version) init without reindex so that we can git gc/repack repos after all writes and before indexing

java -jar gerrit-war-3.5.war init --no-auto-start --reindex-threads=-1 -d /path/to/gerrit/site_dir

java -jar gerrit-war-3.5.war reindex --threads 60 -d /path/to/gerrit/site_dir

Be sure to add all your "normal" Gerrit java options, adjust the values above used for --threads to ones appropriate for your setup, and definitely TEST on a non-production environment first.

Nasser

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/CAKSZd3TBjTEJ0AbmQYxb9wd20z-NP5PtX0_aYVyUvv0c4hdkTg%40mail.gmail.com.

Miten Mehta

unread,

Feb 1, 2024, 8:07:44 AM2/1/24

to Repo and Gerrit Discussion

On Thursday 1 February 2024 at 11:02:55 UTC+5:30 Nasser Grainawi wrote:

On Mon, Jan 29, 2024 at 11:53 PM Matthias Sohn <matthi...@gmail.com> wrote:
On Tue, Jan 30, 2024 at 8:17 AM Miten Mehta <india...@gmail.com> wrote:
Hi,

I had tried upgrading but it did was not simple as replacing the war and may be running sql scripts. I would expect that install dir repo area should get picked up without any issues with new upgrade and also database changes should not cause any issues for user access/projects list etc. but it was not so. if you have some dedicated spoc for it then I can retry.

Please avoid top posting on this list, we prefer interleaved style [1].

did not quite understand but just did Reply to All and then hit the ... to expand it and started to write in between. I hope this is right else please instruct.

Follow the release notes [2] for upgrades, go minor release by minor release (e.g. from 2.8.1 to 2.9.5).
Always use the latest service release per minor release (e.g. 2.9.5 for 2.9.x).
First read the "Important notes" for each version bump to get an overview what needs to be done.

I had already tried this and also posted errors in past where I was getting stuck but did not find solutions so had to abort.

10 years after the fact it might be tricky to find all prerequisites, e.g. old Java and database versions.

Yes, there's some challenges there. We successfully upgraded from 2.7 (a forked version) to 3.5 a little over 5 months ago. I'm overdue on a write up concluding the journey, but you can read some of the earlier progress we made here: https://groups.google.com/g/repo-discuss/c/WVwvngCkRMs/

Thanks for sharing your experience on it. I assume that below init command on existing site_dir will be OK to do and does not need empty site_dir etc. I mean its going to update under site_dir but will not touch site_dir/repo. Also I have only one instance / env of gerrit so need advise how to clone it to my laptop. can I copy the structure below without repositories folder to debian vm on laptop and export import mysql to laptop vm ? you can see repositories is huge so copying it from data center server to laptop will be difficult.

[jioapp@NVMBD01VSR634 data]$ ls -l gerrit
total 232
drwxrwxr-x 2 gerrit2 gerrit2 4096 Nov 2 21:48 bin
drwxrwxr-x 2 gerrit2 gerrit2 4096 Jan 25 17:23 cache
drwxrwxr-x 2 gerrit2 gerrit2 4096 Mar 17 2016 data
drwxrwxr-x 3 gerrit2 gerrit2 4096 Jun 7 2023 etc
drwxr-xr-x 2 gerrit2 gerrit2 4096 Mar 17 2016 lib
drwxrwxr-x 2 gerrit2 gerrit2 176128 Feb 1 10:42 logs
drwxrwxr-x 2 gerrit2 gerrit2 4096 Jun 9 2017 plugins
drwxrwxr-x 244 gerrit2 gerrit2 16384 May 26 2023 repositories
drwxrwxr-x 2 gerrit2 gerrit2 4096 Mar 17 2016 static
drwx------ 10 gerrit2 gerrit2 12288 Jan 25 17:23 tmp
[jioapp@NVMBD01VSR634 data]$ du -sm gerrit/repositories/
63077 gerrit/repositories/
[jioapp@NVMBD01VSR634 data]$ pwd
/data

[gerrit2@NVMBD01VSR634 data]$ du -sm gerrit
63785 gerrit

once I copy to all this and have running setup on laptop I can try below commands you shared downloading the war files from download folders (https://gerrit-releases.storage.googleapis.com/index.html). Also note on idc server we have to go via proxy (no direct internet) so please advise what exact site I need to whitelist just in case we get to use it. I assume /data/gerrit <=> /path/to/gerrit/site_dir.

You can do a minimal set of upgrades instead of doing every minor release. From 2.7, we needed 2.9, 2.11, 2.16, then the NoteDb migration on 2.16, then 3.5, then offline reindexing with 3.5. Starting with 2.8 you might be able to get directly to 2.11, but I'm not sure.

We did these commands effectively:
java -jar gerrit-war-2.9.5.war init --no-auto-start --batch -d /path/to/gerrit/site_dir
java -jar gerrit-war-2.11.12.war init --no-auto-start --batch -d /path/to/gerrit/site_dir
# build 2.16.war from the latest stable-2.16 commit
java -jar gerrit-war-2.16.war init --no-auto-start --no-reindex --batch --migrate-draft-to PRIVATE -d /path/to/gerrit/site_dir
java -jar gerrit-war-2.16.war migrate-to-note-db --reindex=false --threads 24 -d /path/to/gerrit/site_dir
# run 3.5 (our target version) init without reindex so that we can git gc/repack repos after all writes and before indexing
java -jar gerrit-war-3.5.war init --no-auto-start --reindex-threads=-1 -d /path/to/gerrit/site_dir
java -jar gerrit-war-3.5.war reindex --threads 60 -d /path/to/gerrit/site_dir

Be sure to add all your "normal" Gerrit java options, adjust the values above used for --threads to ones appropriate for your setup, and definitely TEST on a non-production environment first.

Nasser

I do not know what java options might have been used as I inherited this setup and am just supporting. may be you can guide on what value of threads I need to use based on lscpu:

lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 14
On-line CPU(s) list: 0-13
Thread(s) per core: 1
Core(s) per socket: 14
Socket(s): 1
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 62
Model name: Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz
Stepping: 4
CPU MHz: 2493.748
BogoMIPS: 4987.49
Hypervisor vendor: Microsoft
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 25600K
NUMA node0 CPU(s): 0-13
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx lm constant_tsc rep_good nopl cpuid pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx f16c rdrand hypervisor lahf_lm pti fsgsbase smep erms xsaveopt

Regards,

Miten.

Nasser Grainawi

unread,

Feb 1, 2024, 10:19:33 AM2/1/24

to Miten Mehta, Repo and Gerrit Discussion

On Thu, Feb 1, 2024 at 5:07 AM Miten Mehta <india...@gmail.com> wrote:

On Thursday 1 February 2024 at 11:02:55 UTC+5:30 Nasser Grainawi wrote:
On Mon, Jan 29, 2024 at 11:53 PM Matthias Sohn <matthi...@gmail.com> wrote:
On Tue, Jan 30, 2024 at 8:17 AM Miten Mehta <india...@gmail.com> wrote:
Hi,

I had tried upgrading but it did was not simple as replacing the war and may be running sql scripts. I would expect that install dir repo area should get picked up without any issues with new upgrade and also database changes should not cause any issues for user access/projects list etc. but it was not so. if you have some dedicated spoc for it then I can retry.

Please avoid top posting on this list, we prefer interleaved style [1].
did not quite understand but just did Reply to All and then hit the ... to expand it and started to write in between. I hope this is right else please instruct.

Follow the release notes [2] for upgrades, go minor release by minor release (e.g. from 2.8.1 to 2.9.5).
Always use the latest service release per minor release (e.g. 2.9.5 for 2.9.x).
First read the "Important notes" for each version bump to get an overview what needs to be done.

I had already tried this and also posted errors in past where I was getting stuck but did not find solutions so had to abort.

10 years after the fact it might be tricky to find all prerequisites, e.g. old Java and database versions.

Yes, there's some challenges there. We successfully upgraded from 2.7 (a forked version) to 3.5 a little over 5 months ago. I'm overdue on a write up concluding the journey, but you can read some of the earlier progress we made here: https://groups.google.com/g/repo-discuss/c/WVwvngCkRMs/

Thanks for sharing your experience on it. I assume that below init command on existing site_dir will be OK to do and does not need empty site_dir etc. I mean its going to update under site_dir but will not touch site_dir/repo. Also I have only one instance / env of gerrit so need advise how to clone it to my laptop. can I copy the structure below without repositories folder to debian vm on laptop and export import mysql to laptop vm ? you can see repositories is huge so copying it from data center server to laptop will be difficult.

There's some good information on how to make backups in https://gerrit-documentation.storage.googleapis.com/Documentation/3.9.1/backup.html (and that still includes information on how to do backups with 2.x versions).

[jioapp@NVMBD01VSR634 data]$ ls -l gerrit
total 232
drwxrwxr-x 2 gerrit2 gerrit2 4096 Nov 2 21:48 bin
drwxrwxr-x 2 gerrit2 gerrit2 4096 Jan 25 17:23 cache
drwxrwxr-x 2 gerrit2 gerrit2 4096 Mar 17 2016 data
drwxrwxr-x 3 gerrit2 gerrit2 4096 Jun 7 2023 etc
drwxr-xr-x 2 gerrit2 gerrit2 4096 Mar 17 2016 lib
drwxrwxr-x 2 gerrit2 gerrit2 176128 Feb 1 10:42 logs
drwxrwxr-x 2 gerrit2 gerrit2 4096 Jun 9 2017 plugins
drwxrwxr-x 244 gerrit2 gerrit2 16384 May 26 2023 repositories
drwxrwxr-x 2 gerrit2 gerrit2 4096 Mar 17 2016 static
drwx------ 10 gerrit2 gerrit2 12288 Jan 25 17:23 tmp
[jioapp@NVMBD01VSR634 data]$ du -sm gerrit/repositories/
63077 gerrit/repositories/
[jioapp@NVMBD01VSR634 data]$ pwd
/data
[gerrit2@NVMBD01VSR634 data]$ du -sm gerrit
63785 gerrit

once I copy to all this and have running setup on laptop I can try below commands you shared downloading the war files from download folders (https://gerrit-releases.storage.googleapis.com/index.html). Also note on idc server we have to go via proxy (no direct internet) so please advise what exact site I need to whitelist just in case we get to use it. I assume /data/gerrit <=> /path/to/gerrit/site_dir.

I'm not sure how to provide that info for you, sorry.

You can do a minimal set of upgrades instead of doing every minor release. From 2.7, we needed 2.9, 2.11, 2.16, then the NoteDb migration on 2.16, then 3.5, then offline reindexing with 3.5. Starting with 2.8 you might be able to get directly to 2.11, but I'm not sure.

We did these commands effectively:
java -jar gerrit-war-2.9.5.war init --no-auto-start --batch -d /path/to/gerrit/site_dir
java -jar gerrit-war-2.11.12.war init --no-auto-start --batch -d /path/to/gerrit/site_dir
# build 2.16.war from the latest stable-2.16 commit
java -jar gerrit-war-2.16.war init --no-auto-start --no-reindex --batch --migrate-draft-to PRIVATE -d /path/to/gerrit/site_dir
java -jar gerrit-war-2.16.war migrate-to-note-db --reindex=false --threads 24 -d /path/to/gerrit/site_dir
# run 3.5 (our target version) init without reindex so that we can git gc/repack repos after all writes and before indexing
java -jar gerrit-war-3.5.war init --no-auto-start --reindex-threads=-1 -d /path/to/gerrit/site_dir
java -jar gerrit-war-3.5.war reindex --threads 60 -d /path/to/gerrit/site_dir

Be sure to add all your "normal" Gerrit java options, adjust the values above used for --threads to ones appropriate for your setup, and definitely TEST on a non-production environment first.

Nasser

I do not know what java options might have been used as I inherited this setup and am just supporting. may be you can guide on what value of threads I need to use based on lscpu:

You can check your etc/gerrit.config file and see if you have any `javaOptions` lines under the [container] section.

If you're not sure on thread values, you can try the defaults (just remove the --threads option from those 2 commands).

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/1e5bc652-715b-40ca-8fc6-ab79f2c73251n%40googlegroups.com.

Miten Mehta

unread,

Feb 2, 2024, 4:41:49 AM2/2/24

to Repo and Gerrit Discussion

On Thursday 1 February 2024 at 20:49:33 UTC+5:30 Nasser Grainawi wrote:

On Thu, Feb 1, 2024 at 5:07 AM Miten Mehta <india...@gmail.com> wrote:

On Thursday 1 February 2024 at 11:02:55 UTC+5:30 Nasser Grainawi wrote:
On Mon, Jan 29, 2024 at 11:53 PM Matthias Sohn <matthi...@gmail.com> wrote:
On Tue, Jan 30, 2024 at 8:17 AM Miten Mehta <india...@gmail.com> wrote:
Hi,

I had tried upgrading but it did was not simple as replacing the war and may be running sql scripts. I would expect that install dir repo area should get picked up without any issues with new upgrade and also database changes should not cause any issues for user access/projects list etc. but it was not so. if you have some dedicated spoc for it then I can retry.

Please avoid top posting on this list, we prefer interleaved style [1].
did not quite understand but just did Reply to All and then hit the ... to expand it and started to write in between. I hope this is right else please instruct.

Follow the release notes [2] for upgrades, go minor release by minor release (e.g. from 2.8.1 to 2.9.5).
Always use the latest service release per minor release (e.g. 2.9.5 for 2.9.x).
First read the "Important notes" for each version bump to get an overview what needs to be done.

I had already tried this and also posted errors in past where I was getting stuck but did not find solutions so had to abort.

10 years after the fact it might be tricky to find all prerequisites, e.g. old Java and database versions.

Yes, there's some challenges there. We successfully upgraded from 2.7 (a forked version) to 3.5 a little over 5 months ago. I'm overdue on a write up concluding the journey, but you can read some of the earlier progress we made here: https://groups.google.com/g/repo-discuss/c/WVwvngCkRMs/

Thanks for sharing your experience on it. I assume that below init command on existing site_dir will be OK to do and does not need empty site_dir etc. I mean its going to update under site_dir but will not touch site_dir/repo. Also I have only one instance / env of gerrit so need advise how to clone it to my laptop. can I copy the structure below without repositories folder to debian vm on laptop and export import mysql to laptop vm ? you can see repositories is huge so copying it from data center server to laptop will be difficult.

There's some good information on how to make backups in https://gerrit-documentation.storage.googleapis.com/Documentation/3.9.1/backup.html (and that still includes information on how to do backups with 2.x versions).

I am doing full backup of /data/gerrit (tar) and mysql db dump. My query was if I copy /data/gerrit without repositories folder will the upgrade steps you mentioned be fine to do. Due to size issue and access issue I suggested to experiment with setup on my laptop. Its java and debian instead of redhat (in data center). Once I apply all upgrade then I can just copy it back on /data/gerrit in data center. does it sound OK ?

[container]
user = gerrit2

That is all I have in gerrit.config so as you said will remove thread options from commands.

Nasser Grainawi

unread,

Feb 2, 2024, 11:29:02 AM2/2/24

to Miten Mehta, Repo and Gerrit Discussion

On Fri, Feb 2, 2024 at 1:41 AM Miten Mehta <india...@gmail.com> wrote:

On Thursday 1 February 2024 at 20:49:33 UTC+5:30 Nasser Grainawi wrote:
On Thu, Feb 1, 2024 at 5:07 AM Miten Mehta <india...@gmail.com> wrote:

On Thursday 1 February 2024 at 11:02:55 UTC+5:30 Nasser Grainawi wrote:
On Mon, Jan 29, 2024 at 11:53 PM Matthias Sohn <matthi...@gmail.com> wrote:
On Tue, Jan 30, 2024 at 8:17 AM Miten Mehta <india...@gmail.com> wrote:
Hi,

I had tried upgrading but it did was not simple as replacing the war and may be running sql scripts. I would expect that install dir repo area should get picked up without any issues with new upgrade and also database changes should not cause any issues for user access/projects list etc. but it was not so. if you have some dedicated spoc for it then I can retry.

Please avoid top posting on this list, we prefer interleaved style [1].
did not quite understand but just did Reply to All and then hit the ... to expand it and started to write in between. I hope this is right else please instruct.

Follow the release notes [2] for upgrades, go minor release by minor release (e.g. from 2.8.1 to 2.9.5).
Always use the latest service release per minor release (e.g. 2.9.5 for 2.9.x).
First read the "Important notes" for each version bump to get an overview what needs to be done.

I had already tried this and also posted errors in past where I was getting stuck but did not find solutions so had to abort.

10 years after the fact it might be tricky to find all prerequisites, e.g. old Java and database versions.

Yes, there's some challenges there. We successfully upgraded from 2.7 (a forked version) to 3.5 a little over 5 months ago. I'm overdue on a write up concluding the journey, but you can read some of the earlier progress we made here: https://groups.google.com/g/repo-discuss/c/WVwvngCkRMs/

Thanks for sharing your experience on it. I assume that below init command on existing site_dir will be OK to do and does not need empty site_dir etc. I mean its going to update under site_dir but will not touch site_dir/repo. Also I have only one instance / env of gerrit so need advise how to clone it to my laptop. can I copy the structure below without repositories folder to debian vm on laptop and export import mysql to laptop vm ? you can see repositories is huge so copying it from data center server to laptop will be difficult.

There's some good information on how to make backups in https://gerrit-documentation.storage.googleapis.com/Documentation/3.9.1/backup.html (and that still includes information on how to do backups with 2.x versions).

I am doing full backup of /data/gerrit (tar) and mysql db dump. My query was if I copy /data/gerrit without repositories folder will the upgrade steps you mentioned be fine to do. Due to size issue and access issue I suggested to experiment with setup on my laptop. Its java and debian instead of redhat (in data center). Once I apply all upgrade then I can just copy it back on /data/gerrit in data center. does it sound OK ?

No. You must have a copy of the repositories when you run the upgrade commands.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/bb75c308-7f05-41a5-88f7-743a34067a92n%40googlegroups.com.

Miten Mehta

unread,

Feb 22, 2024, 4:27:59 AM2/22/24

to Repo and Gerrit Discussion

On Friday 2 February 2024 at 21:59:02 UTC+5:30 Nasser Grainawi wrote:

On Fri, Feb 2, 2024 at 1:41 AM Miten Mehta <india...@gmail.com> wrote:

On Thursday 1 February 2024 at 20:49:33 UTC+5:30 Nasser Grainawi wrote:
On Thu, Feb 1, 2024 at 5:07 AM Miten Mehta <india...@gmail.com> wrote:

On Thursday 1 February 2024 at 11:02:55 UTC+5:30 Nasser Grainawi wrote:
On Mon, Jan 29, 2024 at 11:53 PM Matthias Sohn <matthi...@gmail.com> wrote:
On Tue, Jan 30, 2024 at 8:17 AM Miten Mehta <india...@gmail.com> wrote:
Hi,

I had tried upgrading but it did was not simple as replacing the war and may be running sql scripts. I would expect that install dir repo area should get picked up without any issues with new upgrade and also database changes should not cause any issues for user access/projects list etc. but it was not so. if you have some dedicated spoc for it then I can retry.

Please avoid top posting on this list, we prefer interleaved style [1].
did not quite understand but just did Reply to All and then hit the ... to expand it and started to write in between. I hope this is right else please instruct.

Follow the release notes [2] for upgrades, go minor release by minor release (e.g. from 2.8.1 to 2.9.5).
Always use the latest service release per minor release (e.g. 2.9.5 for 2.9.x).
First read the "Important notes" for each version bump to get an overview what needs to be done.

I had already tried this and also posted errors in past where I was getting stuck but did not find solutions so had to abort.

10 years after the fact it might be tricky to find all prerequisites, e.g. old Java and database versions.

Yes, there's some challenges there. We successfully upgraded from 2.7 (a forked version) to 3.5 a little over 5 months ago. I'm overdue on a write up concluding the journey, but you can read some of the earlier progress we made here: https://groups.google.com/g/repo-discuss/c/WVwvngCkRMs/

Thanks for sharing your experience on it. I assume that below init command on existing site_dir will be OK to do and does not need empty site_dir etc. I mean its going to update under site_dir but will not touch site_dir/repo. Also I have only one instance / env of gerrit so need advise how to clone it to my laptop. can I copy the structure below without repositories folder to debian vm on laptop and export import mysql to laptop vm ? you can see repositories is huge so copying it from data center server to laptop will be difficult.

There's some good information on how to make backups in https://gerrit-documentation.storage.googleapis.com/Documentation/3.9.1/backup.html (and that still includes information on how to do backups with 2.x versions).

I am doing full backup of /data/gerrit (tar) and mysql db dump. My query was if I copy /data/gerrit without repositories folder will the upgrade steps you mentioned be fine to do. Due to size issue and access issue I suggested to experiment with setup on my laptop. Its java and debian instead of redhat (in data center). Once I apply all upgrade then I can just copy it back on /data/gerrit in data center. does it sound OK ?

No. You must have a copy of the repositories when you run the upgrade commands.

I will use unix split and cat command to get the huge 66GB gerrit dir tar to my laptop. now do I need to have gerrit process running to do the upgrade (meaning as I do your git init and other commands you provided and any other) ? If I have to run it then I will also need to get db setup on local (export/import mysql) and then can I just change the db section details in gerrit config like ip ? Also I have ldap authentication against corporate directory so can I turn that off instead and do some harcoded credentials to get it running just for upgrade on local ? how ?

Message has been deleted

Nasser Grainawi

unread,

Feb 22, 2024, 10:46:58 AM2/22/24

to Miten Mehta, Repo and Gerrit Discussion

On Thu, Feb 22, 2024 at 1:28 AM Miten Mehta <india...@gmail.com> wrote:

On Friday 2 February 2024 at 21:59:02 UTC+5:30 Nasser Grainawi wrote:
On Fri, Feb 2, 2024 at 1:41 AM Miten Mehta <india...@gmail.com> wrote:

On Thursday 1 February 2024 at 20:49:33 UTC+5:30 Nasser Grainawi wrote:
On Thu, Feb 1, 2024 at 5:07 AM Miten Mehta <india...@gmail.com> wrote:

On Thursday 1 February 2024 at 11:02:55 UTC+5:30 Nasser Grainawi wrote:
On Mon, Jan 29, 2024 at 11:53 PM Matthias Sohn <matthi...@gmail.com> wrote:
On Tue, Jan 30, 2024 at 8:17 AM Miten Mehta <india...@gmail.com> wrote:
Hi,

I had tried upgrading but it did was not simple as replacing the war and may be running sql scripts. I would expect that install dir repo area should get picked up without any issues with new upgrade and also database changes should not cause any issues for user access/projects list etc. but it was not so. if you have some dedicated spoc for it then I can retry.

Please avoid top posting on this list, we prefer interleaved style [1].
did not quite understand but just did Reply to All and then hit the ... to expand it and started to write in between. I hope this is right else please instruct.

Follow the release notes [2] for upgrades, go minor release by minor release (e.g. from 2.8.1 to 2.9.5).
Always use the latest service release per minor release (e.g. 2.9.5 for 2.9.x).
First read the "Important notes" for each version bump to get an overview what needs to be done.

I had already tried this and also posted errors in past where I was getting stuck but did not find solutions so had to abort.

10 years after the fact it might be tricky to find all prerequisites, e.g. old Java and database versions.

Yes, there's some challenges there. We successfully upgraded from 2.7 (a forked version) to 3.5 a little over 5 months ago. I'm overdue on a write up concluding the journey, but you can read some of the earlier progress we made here: https://groups.google.com/g/repo-discuss/c/WVwvngCkRMs/

Thanks for sharing your experience on it. I assume that below init command on existing site_dir will be OK to do and does not need empty site_dir etc. I mean its going to update under site_dir but will not touch site_dir/repo. Also I have only one instance / env of gerrit so need advise how to clone it to my laptop. can I copy the structure below without repositories folder to debian vm on laptop and export import mysql to laptop vm ? you can see repositories is huge so copying it from data center server to laptop will be difficult.

There's some good information on how to make backups in https://gerrit-documentation.storage.googleapis.com/Documentation/3.9.1/backup.html (and that still includes information on how to do backups with 2.x versions).

I am doing full backup of /data/gerrit (tar) and mysql db dump. My query was if I copy /data/gerrit without repositories folder will the upgrade steps you mentioned be fine to do. Due to size issue and access issue I suggested to experiment with setup on my laptop. Its java and debian instead of redhat (in data center). Once I apply all upgrade then I can just copy it back on /data/gerrit in data center. does it sound OK ?

No. You must have a copy of the repositories when you run the upgrade commands.

I will use unix split and cat command to get the huge 66GB gerrit dir tar to my laptop. now do I need to have gerrit process running to do the upgrade (meaning as I do your git init and other commands you provided and any other) ? If I have to run it then I will also need to get db setup on local (export/import mysql) and then can I just change the db section details in gerrit config like ip ? Also I have ldap authentication against corporate directory so can I turn that off instead and do some harcoded credentials to get it running just for upgrade on local ? how ?

No, it's expected that Gerrit isn't running while you do an upgrade. You should make sure that any production-specific gerrit.config values (like db) are updated to reference your test setup on your laptop.

I don't think you'll need or want to change the authentication settings while testing the upgrade. If you can't use the LDAP settings on your laptop, then for your post-upgrade testing there's a special auth.type=DEVELOPMENT_BECOME_ANY_ACCOUNT value [1] that allows you to run a test server in a mode where you can log in as any account.

[1] https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#auth.type

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/909c4495-cdf3-4ec0-9cc3-9888b6625059n%40googlegroups.com.

Miten Mehta

unread,

Mar 4, 2024, 5:33:19 AM3/4/24

to Repo and Gerrit Discussion

4th March 2024
--------------
I used jdk17 to run below successfully

java --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-2.9.5.war init --no-auto-start --batch -d $GHOME
Upgrading schema to 85 ...
Upgrading schema to 86 ...
Upgrading schema to 87 ...
Upgrading schema to 88 ...
Upgrading schema to 89 ...
Upgrading schema to 90 ...
Upgrading schema to 91 ...
Upgrading schema to 92 ...
Upgrading schema to 93 ...
Migrating data to schema 85 ...
Migrating data to schema 86 ...
Migrating data to schema 87 ...
Migrating data to schema 88 ...
Migrating data to schema 89 ...
Migrating data to schema 90 ...
Migrating data to schema 91 ...
Migrating data to schema 92 ...
Migrating data to schema 93 ...
Execute the following SQL to drop unused objects:

DROP TABLE tracking_ids;
ALTER TABLE account_groups DROP COLUMN group_type;
ALTER TABLE patch_set_approvals DROP COLUMN change_open;
ALTER TABLE patch_set_approvals DROP COLUMN change_sort_key;
Initialized /media/sf_C_DRIVE/work/software/gerrit/gerrit_backup/data/gerrit

when I tried running gerrit.sh I faced some errors so I thought to better install jdk1.8 and proceed.

Now I updated gerrit.config to use 2.9.5 war by setting war property under container section to point to path of 2.9.5 war as below assuming it will use that war.
war = /media/sf_C_DRIVE/work/software/gerrit/releases/gerrit-2.9.5.war

I also then installed jdk 1.8 and set it in gerrit.config under container section:
javaHome = /opt/jdk1.8.0_202

Now in old 2.8.5 install area itself where I had bin dir I used gerrit.sh start command to start as earlier but am getting below output/error.

[2024-03-04 15:50:39,166] INFO com.google.gerrit.server.cache.h2.H2CacheFactory : Enabling disk cache /media/sf_C_DRIVE/work/software/gerrit/gerrit_backup/data/gerrit/cache
[2024-03-04 15:50:39,954] WARN com.google.gerrit.sshd.SshDaemon : Disabling cipher arcfour256: Parameters not supported
[2024-03-04 15:50:39,954] WARN com.google.gerrit.sshd.SshDaemon : Disabling cipher arcfour128: Parameters not supported
[2024-03-04 15:50:39,970] WARN com.google.gerrit.httpd.GitWebConfig : gitweb not installed (no /usr/lib/cgi-bin/gitweb.cgi found)
[2024-03-04 15:50:43,810] INFO com.google.gerrit.server.git.LocalDiskRepositoryManager : Defaulting core.streamFileThreshold to 218m
[2024-03-04 15:50:43,842] INFO com.google.gerrit.server.plugins.PluginLoader : Loading plugins from /media/sf_C_DRIVE/work/software/gerrit/gerrit_backup/data/gerrit/plugins
[2024-03-04 15:50:43,967] WARN com.googlesource.gerrit.plugins.replication.ReplicationQueue : No /media/sf_C_DRIVE/work/software/gerrit/gerrit_backup/data/gerrit/etc/replication.config; not replicating
[2024-03-04 15:50:43,968] INFO com.google.gerrit.server.plugins.PluginLoader : Loaded plugin replication, version v2.8.1
[2024-03-04 15:50:44,005] INFO com.google.gerrit.server.plugins.PluginLoader : Loaded plugin reviewnotes, version v2.8.1
[2024-03-04 15:50:44,040] INFO com.google.gerrit.server.plugins.PluginLoader : Loaded plugin download-commands, version v2.8.1
[2024-03-04 15:50:44,070] WARN com.google.gerrit.server.plugins.PluginLoader : Cannot load plugin deleteproject
java.lang.NoClassDefFoundError: com/google/gerrit/server/util/PluginLogFile
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at com.googlesource.gerrit.plugins.deleteproject.Module.configure(Module.java:38)
at com.google.inject.AbstractModule.configure(AbstractModule.java:62)
at com.google.inject.spi.Elements$RecordingBinder.install(Elements.java:230)
at com.google.inject.spi.Elements.getElements(Elements.java:103)
at com.google.inject.internal.InjectorShell$Builder.build(InjectorShell.java:136)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:104)
at com.google.inject.internal.InjectorImpl.createChildInjector(InjectorImpl.java:230)
at com.google.inject.internal.InjectorImpl.createChildInjector(InjectorImpl.java:237)
at com.google.gerrit.server.plugins.JarPlugin.startPlugin(JarPlugin.java:144)
at com.google.gerrit.server.plugins.JarPlugin.start(JarPlugin.java:127)
at com.google.gerrit.server.plugins.PluginLoader.runPlugin(PluginLoader.java:425)
at com.google.gerrit.server.plugins.PluginLoader.rescan(PluginLoader.java:395)
at com.google.gerrit.server.plugins.PluginLoader.start(PluginLoader.java:306)
at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:74)
at com.google.gerrit.pgm.Daemon.start(Daemon.java:289)
at com.google.gerrit.pgm.Daemon.run(Daemon.java:201)
at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:166)
at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:93)
at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:50)
at Main.main(Main.java:25)
Caused by: java.lang.ClassNotFoundException: com.google.gerrit.server.util.PluginLogFile
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 36 more
[2024-03-04 15:50:44,073] ERROR com.google.gerrit.pgm.Daemon : Unable to start daemon
com.google.inject.ProvisionException: Guice provision errors:

1) No index versions ready; run Reindex

1 error
at com.google.gerrit.lucene.LuceneVersionManager.start(LuceneVersionManager.java:118)
at com.google.gerrit.lifecycle.LifecycleManager.start(LifecycleManager.java:74)
at com.google.gerrit.pgm.Daemon.start(Daemon.java:289)
at com.google.gerrit.pgm.Daemon.run(Daemon.java:201)
at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:63)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:166)
at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:93)
at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:50)
at Main.main(Main.java:25)

I read some where that to install delete-project plugin just copy jar file to plugins dir but am not sure where to get it from.

Matthias Sohn

unread,

Mar 4, 2024, 10:40:44 AM3/4/24

to Miten Mehta, Repo and Gerrit Discussion

The oldest delete-project plugin version available on the archive server is for 2.10: https://archive-ci.gerritforge.com/job/

Either build the plugin yourself or delete it, you can add it again after upgrading to a still supported release.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/b9dae503-25ed-4109-83cd-2d112e5e1175n%40googlegroups.com.

Miten Mehta

unread,

Mar 6, 2024, 6:22:14 AM3/6/24

to Repo and Gerrit Discussion

Hi,

I am using mysql 5.1.73. I have upgraded gerrit war from 2.8.1 to 3.9.2 on laptop.

now reviewdb is only database i need to export and import into datacenter mysql db ?

mysqldump -u root -p "reviewdb" > dump.sql

do I need to login to datacenter mysql db and drop reviewdb first before importing it? my dba uses some mysqlbkp user to take backups can he use same instead of root in below command? I guess he only will be doing it and might know what to do but in general is below command correct ?
mysql -u root -p reviewdb < dump.sql

Now also copy laptop gerrit dir to datacenter.

currently dataceneter gerrit is in use. It might have got more commits.

1. To upgrade datacenter gerrit with my laptop gerrit I am thinking to block writes to datacenter gerrit for few hours.
how do I prevent writes to gerrit in datacenter but users can read and clone fine?

take copy of gerrit site dir in datacenter to my laptop and db dump also and repeat upgrade steps on my laptop and then update datacenter gerrit as above to copy db and gerrit site.

2. or if users try to push with more commits to upgraded gerrit that I have on laptop which I will copy to datacenter will it be fine ?

I have done below commands based on what was shared to me in this group:

GHOME is /data/gerrit
and where we see --add-opens I used jdk17. basically I started with jdk17 but then 2.9.5 to 2.16.28 I used jdk8 as soon as I encountered errors doing first reindex on 2.9.5. Then 3.5.6 I had to switch back to jdk17 as jdk8 is not supported.

java --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-2.9.5.war init --no-auto-start --batch -d $GHOME

java -jar gerrit-2.9.5.war reindex -d $GHOME
java -jar gerrit-2.11.12.war init --no-auto-start --batch -d $GHOME
java -jar gerrit-2.11.12.war reindex -d $GHOME
java -jar gerrit-2.11.12.war reindex -d $GHOME
java -jar gerrit-2.16.28.war init --no-auto-start --no-reindex --batch -d $GHOME
java -jar gerrit-2.16.28.war reindex --index changes -d $GHOME #then repeated for index groups, accounts, projects

java -jar ./gerrit-2.16.28.war migrate-to-note-db --reindex=false --threads 24 -d /data/gerrit/
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.5.6.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME
java -jar ./gerrit-3.5.6.war reindex -d /data/gerrit/
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.6.8.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.6.8.war reindex --index changes -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.7.8.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.7.8.war reindex --index changes -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.8.4.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.9.2.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME

is there a way to do init and reindex both in one command as it seems whenever there are any db schema changes reindex was needed. I guess if I leave out --reindex-threads option it will by default do reindex or if I use postive value of it then I might not need to reindex after init.

In few steps I guess reindex ones I saw some exception/errors like missing commit and projects like below. I hope they are ignorable.

com.google.gwtorm.server.OrmException: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: Cannot open repository jionetm

com.google.gwtorm.server.OrmException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 7ff2fc9b614e1faaf712256bb8aa978d547639c1

Regards,

Miten.

Kenyon Ralph

unread,

Mar 6, 2024, 4:58:55 PM3/6/24

to Repo and Gerrit Discussion

On Wednesday, March 6, 2024 at 3:22:14 AM UTC-8 Miten Mehta wrote:

Hi,

I am using mysql 5.1.73. I have upgraded gerrit war from 2.8.1 to 3.9.2 on laptop.

now reviewdb is only database i need to export and import into datacenter mysql db ?
mysqldump -u root -p "reviewdb" > dump.sql

I also used the --all-databases command line option.

do I need to login to datacenter mysql db and drop reviewdb first before importing it? my dba uses some mysqlbkp user to take backups can he use same instead of root in below command? I guess he only will be doing it and might know what to do but in general is below command correct ?
mysql -u root -p reviewdb < dump.sql

You don't need to drop any databases.

Now also copy laptop gerrit dir to datacenter.

currently dataceneter gerrit is in use. It might have got more commits.

1. To upgrade datacenter gerrit with my laptop gerrit I am thinking to block writes to datacenter gerrit for few hours.
how do I prevent writes to gerrit in datacenter but users can read and clone fine?
take copy of gerrit site dir in datacenter to my laptop and db dump also and repeat upgrade steps on my laptop and then update datacenter gerrit as above to copy db and gerrit site.

Not sure if there's a good way to prevent writes but allow reads. When I went through this, I just stopped the old Gerrit instances. I was able to do this on a weekend. You'll probably need more than a few hours, depending on how large of an instance this Gerrit is. You'll definitely want to script as much of the process as possible, and iterate on the process until it works perfectly.

Matthias Sohn

unread,

Mar 6, 2024, 6:01:19 PM3/6/24

to Miten Mehta, Repo and Gerrit Discussion

On Wed, Mar 6, 2024 at 12:22 PM Miten Mehta <india...@gmail.com> wrote:

Hi,

I am using mysql 5.1.73. I have upgraded gerrit war from 2.8.1 to 3.9.2 on laptop.

now reviewdb is only database i need to export and import into datacenter mysql db ?
mysqldump -u root -p "reviewdb" > dump.sql

After the migration to notedb on 2.16.28 the reviewdb is not used anymore and

all primary data is stored in the git repositories. Only file reviewed flags are stored

in a relational database after the migration, see

https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#accountPatchReviewDb

do I need to login to datacenter mysql db and drop reviewdb first before importing it? my dba uses some mysqlbkp user to take backups can he use same instead of root in below command? I guess he only will be doing it and might know what to do but in general is below command correct ?
mysql -u root -p reviewdb < dump.sql

Now also copy laptop gerrit dir to datacenter.

currently dataceneter gerrit is in use. It might have got more commits.

If you do the upgrade on a copy of the production system you need to ensure

the production system is not modified until it was replaced by the upgraded copy.

The simplest way to achieve that is to shutdown the production system during the upgrade.

1. To upgrade datacenter gerrit with my laptop gerrit I am thinking to block writes to datacenter gerrit for few hours.
how do I prevent writes to gerrit in datacenter but users can read and clone fine?

The readonly plugin might be a solution, but not on 2.8.1, it's only available starting from 2.14.

https://gerrit-review.googlesource.com/admin/repos/plugins/readonly,general

I don't have experience using it on these old versions.

take copy of gerrit site dir in datacenter to my laptop and db dump also and repeat upgrade steps on my laptop and then update datacenter gerrit as above to copy db and gerrit site.

I wouldn't upgrade a production system on my laptop but only on some controlled staging server

or directly on the production server. Don't forget to take a full backup before upgrading.

And test restoring the system from a backup before you start the upgrade.

2. or if users try to push with more commits to upgraded gerrit that I have on laptop which I will copy to datacenter will it be fine ?

I have done below commands based on what was shared to me in this group:

GHOME is /data/gerrit
and where we see --add-opens I used jdk17. basically I started with jdk17 but then 2.9.5 to 2.16.28 I used jdk8 as soon as I encountered errors doing first reindex on 2.9.5. Then 3.5.6 I had to switch back to jdk17 as jdk8 is not supported.

java --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-2.9.5.war init --no-auto-start --batch -d $GHOME
java -jar gerrit-2.9.5.war reindex -d $GHOME
java -jar gerrit-2.11.12.war init --no-auto-start --batch -d $GHOME
java -jar gerrit-2.11.12.war reindex -d $GHOME
java -jar gerrit-2.11.12.war reindex -d $GHOME
java -jar gerrit-2.16.28.war init --no-auto-start --no-reindex --batch -d $GHOME
java -jar gerrit-2.16.28.war reindex --index changes -d $GHOME #then repeated for index groups, accounts, projects
java -jar ./gerrit-2.16.28.war migrate-to-note-db --reindex=false --threads 24 -d /data/gerrit/
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.5.6.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME
java -jar ./gerrit-3.5.6.war reindex -d /data/gerrit/
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.6.8.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.6.8.war reindex --index changes -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.7.8.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.7.8.war reindex --index changes -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.8.4.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.9.2.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME

is there a way to do init and reindex both in one command as it seems whenever there are any db schema changes reindex was needed. I guess if I leave out --reindex-threads option it will by default do reindex or if I use postive value of it then I might not need to reindex after init.

I think it's not necessary to reindex after each intermediate upgrade.

It should be sufficient to do a full reindex after the last upgrade to 3.9.2.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/d9c392d5-04cd-4d43-9570-53b66ac69d42n%40googlegroups.com.

Miten Mehta

unread,

Mar 7, 2024, 2:21:12 AM3/7/24

to Repo and Gerrit Discussion

On Thursday 7 March 2024 at 04:31:19 UTC+5:30 Matthias Sohn wrote:

On Wed, Mar 6, 2024 at 12:22 PM Miten Mehta <india...@gmail.com> wrote:
Hi,

I am using mysql 5.1.73. I have upgraded gerrit war from 2.8.1 to 3.9.2 on laptop.

now reviewdb is only database i need to export and import into datacenter mysql db ?
mysqldump -u root -p "reviewdb" > dump.sql

After the migration to notedb on 2.16.28 the reviewdb is not used anymore and
all primary data is stored in the git repositories. Only file reviewed flags are stored
in a relational database after the migration, see
https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#accountPatchReviewDb

But I still need to copy db dump from laptop to datacenter mysql right ? what I understand is minimal but cannot ignore that data and still need to be in place. I did not see any notedb database in mysql when I did show databases. It still shows reviewdb only.

do I need to login to datacenter mysql db and drop reviewdb first before importing it? my dba uses some mysqlbkp user to take backups can he use same instead of root in below command? I guess he only will be doing it and might know what to do but in general is below command correct ?
mysql -u root -p reviewdb < dump.sql

Now also copy laptop gerrit dir to datacenter.

currently dataceneter gerrit is in use. It might have got more commits.

If you do the upgrade on a copy of the production system you need to ensure
the production system is not modified until it was replaced by the upgraded copy.
The simplest way to achieve that is to shutdown the production system during the upgrade.

ok best is to take datacenter gerrit down and then copy site folder and db. bring up datacenter gerrit and most probably it should work. you meant that while file copy and db dump import is happening there should be no modifications to both which if system is not down could happen.

1. To upgrade datacenter gerrit with my laptop gerrit I am thinking to block writes to datacenter gerrit for few hours.
how do I prevent writes to gerrit in datacenter but users can read and clone fine?

The readonly plugin might be a solution, but not on 2.8.1, it's only available starting from 2.14.
https://gerrit-review.googlesource.com/admin/repos/plugins/readonly,general
I don't have experience using it on these old versions.

I will better bring datacenter gerrit down. Also for earlier deleteproject plugin I was unable to make it work so when I upgrading and reached a release for which there was compatiable deleteproject plugin jar avaiable I started to use it fine.

take copy of gerrit site dir in datacenter to my laptop and db dump also and repeat upgrade steps on my laptop and then update datacenter gerrit as above to copy db and gerrit site.

I wouldn't upgrade a production system on my laptop but only on some controlled staging server
or directly on the production server. Don't forget to take a full backup before upgrading.
And test restoring the system from a backup before you start the upgrade.

Yes I have backups of site folder and db and using that only on my laptop I have upgraded to latest gerrit on laptop so sounds like I have needed backup and it works. that is one reason I stick to 5.1.75 version of mysql even though was tempted to use latest one.

2. or if users try to push with more commits to upgraded gerrit that I have on laptop which I will copy to datacenter will it be fine ?

I have done below commands based on what was shared to me in this group:

GHOME is /data/gerrit
and where we see --add-opens I used jdk17. basically I started with jdk17 but then 2.9.5 to 2.16.28 I used jdk8 as soon as I encountered errors doing first reindex on 2.9.5. Then 3.5.6 I had to switch back to jdk17 as jdk8 is not supported.

java --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-2.9.5.war init --no-auto-start --batch -d $GHOME
java -jar gerrit-2.9.5.war reindex -d $GHOME
java -jar gerrit-2.11.12.war init --no-auto-start --batch -d $GHOME
java -jar gerrit-2.11.12.war reindex -d $GHOME
java -jar gerrit-2.11.12.war reindex -d $GHOME
java -jar gerrit-2.16.28.war init --no-auto-start --no-reindex --batch -d $GHOME
java -jar gerrit-2.16.28.war reindex --index changes -d $GHOME #then repeated for index groups, accounts, projects
java -jar ./gerrit-2.16.28.war migrate-to-note-db --reindex=false --threads 24 -d /data/gerrit/
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.5.6.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME
java -jar ./gerrit-3.5.6.war reindex -d /data/gerrit/
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.6.8.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.6.8.war reindex --index changes -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.7.8.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.7.8.war reindex --index changes -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.8.4.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/java.net=ALL-UNNAMED -jar gerrit-3.9.2.war init --no-auto-start --reindex-threads=-1 --batch -d $GHOME

is there a way to do init and reindex both in one command as it seems whenever there are any db schema changes reindex was needed. I guess if I leave out --reindex-threads option it will by default do reindex or if I use postive value of it then I might not need to reindex after init.

I think it's not necessary to reindex after each intermediate upgrade.
It should be sufficient to do a full reindex after the last upgrade to 3.9.2.

Actually I was trying to brind up gerrit after each release upgrade and noticed that when there were db changes in release it would not come up. should I have assumed that is expected and ignore it and move on to do next release upgrade and only at very last release do full reindex and expect it to work fine ?

In few steps I guess reindex ones I saw some exception/errors like missing commit and projects like below. I hope they are ignorable.

com.google.gwtorm.server.OrmException: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: Cannot open repository jionetm

com.google.gwtorm.server.OrmException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 7ff2fc9b614e1faaf712256bb8aa978d547639c1

will above be any issue ? I assume the misisng repo might be like I must have deleted it from file system as way to do deleteproject since that plugin was not doing it works for me in 2.8.1. Also any missing commits would be issue ?

Matthias Sohn

unread,

Mar 7, 2024, 7:30:36 PM3/7/24

to Miten Mehta, Repo and Gerrit Discussion

On Thu, Mar 7, 2024 at 8:21 AM Miten Mehta <india...@gmail.com> wrote:

On Thursday 7 March 2024 at 04:31:19 UTC+5:30 Matthias Sohn wrote:
On Wed, Mar 6, 2024 at 12:22 PM Miten Mehta <india...@gmail.com> wrote:
Hi,

I am using mysql 5.1.73. I have upgraded gerrit war from 2.8.1 to 3.9.2 on laptop.

now reviewdb is only database i need to export and import into datacenter mysql db ?
mysqldump -u root -p "reviewdb" > dump.sql

After the migration to notedb on 2.16.28 the reviewdb is not used anymore and
all primary data is stored in the git repositories. Only file reviewed flags are stored
in a relational database after the migration, see
https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#accountPatchReviewDb

But I still need to copy db dump from laptop to datacenter mysql right ? what I understand is minimal but cannot ignore that data and still need to be in place. I did not see any notedb database in mysql when I did show databases. It still shows reviewdb only.

As I said already Gerrit 3.x does not use a relational database anymore but stores all primary data in git repositories

including review metadata like comments, votes etc. Some of this metadata is stored in git notes hence we call

this notedb, see https://gerrit-review.googlesource.com/Documentation/note-db.html

do I need to login to datacenter mysql db and drop reviewdb first before importing it? my dba uses some mysqlbkp user to take backups can he use same instead of root in below command? I guess he only will be doing it and might know what to do but in general is below command correct ?
mysql -u root -p reviewdb < dump.sql

Now also copy laptop gerrit dir to datacenter.

currently dataceneter gerrit is in use. It might have got more commits.

If you do the upgrade on a copy of the production system you need to ensure
the production system is not modified until it was replaced by the upgraded copy.
The simplest way to achieve that is to shutdown the production system during the upgrade.

ok best is to take datacenter gerrit down and then copy site folder and db. bring up datacenter gerrit and most probably it should work. you meant that while file copy and db dump import is happening there should be no modifications to both which if system is not down could happen.

No, the production site needs to be down, or at least read-only starting before you create the backup you copy to another

machine to run the upgrade there until you copy the upgraded gerrit site back to the production machine.

If you don't ensure that you will loose all changes and updates done on the production machine after the backup was taken

since there is no way to merge these updates (done on the old version 2.8.1) with the upgraded copy which uses Gerrit version 3.9.2

after the upgrade.

1. To upgrade datacenter gerrit with my laptop gerrit I am thinking to block writes to datacenter gerrit for few hours.
how do I prevent writes to gerrit in datacenter but users can read and clone fine?

The readonly plugin might be a solution, but not on 2.8.1, it's only available starting from 2.14.
https://gerrit-review.googlesource.com/admin/repos/plugins/readonly,general
I don't have experience using it on these old versions.

I will better bring datacenter gerrit down. Also for earlier deleteproject plugin I was unable to make it work so when I upgrading and reached a release for which there was compatiable deleteproject plugin jar avaiable I started to use it fine.

You don't need to bring the datacenter down, it should be sufficient to shutdown gerrit.

The indexes are only used to serve queries. This means if you don't reindex after each upgrade step

only queries will not work or will have incomplete results but the server should still start.

After the last upgrade you then need to run a full offline reindexing to create the index versions

matching your target release 3.9.2.

In few steps I guess reindex ones I saw some exception/errors like missing commit and projects like below. I hope they are ignorable.

com.google.gwtorm.server.OrmException: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: Cannot open repository jionetm

com.google.gwtorm.server.OrmException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 7ff2fc9b614e1faaf712256bb8aa978d547639c1

will above be any issue ? I assume the misisng repo might be like I must have deleted it from file system as way to do deleteproject since that plugin was not doing it works for me in 2.8.1. Also any missing commits would be issue ?

I don't have access to your data hence you need to check yourself.

If you can't find the missing repository under $gerrit_site/git then it's obviously missing.

Use git show objectId on the respective repository to find out if this object is really missing.

You may consider to run git fsck on all repositories to check if any of them is corrupted.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/80962a36-acf3-4bb0-b3dc-45a361af6960n%40googlegroups.com.

Björn Pedersen

unread,

Mar 8, 2024, 6:35:02 AM3/8/24

to Repo and Gerrit Discussion

com.google.gwtorm.server.OrmException: org.eclipse.jgit.errors.RepositoryNotFoundException: repository not found: Cannot open repository jionetm

com.google.gwtorm.server.OrmException: org.eclipse.jgit.errors.MissingObjectException: Missing unknown 7ff2fc9b614e1faaf712256bb8aa978d547639c1

will above be any issue ? I assume the misisng repo might be like I must have deleted it from file system as way to do deleteproject since that plugin was not doing it works for me in 2.8.1. Also any missing commits would be issue ?

One main resaons for these missing unknowns are deleted private(draft) changes. In that case the error is not fatal.

Reply all

Reply to author

Forward