IOException: Failed to update ref refs/users/27/27: LOCK_FAILURE

[John S.]

I ran into this exception when trying to initialize gerrit-2.15.2.war. Our system is currently running on v2.14.8. 

Here is the migration output and StackTrace:

Upgrading schema to 143 ...
Upgrading schema to 144 ...
Upgrading schema to 145 ...
Upgrading schema to 146 ...
Upgrading schema to 147 ...
Upgrading schema to 148 ...
Upgrading schema to 149 ...
Upgrading schema to 150 ...
Upgrading schema to 151 ...
Upgrading schema to 152 ...
Upgrading schema to 153 ...
Upgrading schema to 154 ...
Upgrading schema to 155 ...
Upgrading schema to 156 ...
Upgrading schema to 157 ...
Upgrading schema to 158 ...
Upgrading schema to 159 ...
Upgrading schema to 160 ...
Upgrading schema to 161 ...
Migrating data to schema 143 ...
        > Done (0.007 s)
Migrating data to schema 144 ...
        > Done (0.303 s)
Migrating data to schema 145 ...
        > Done (0.120 s)
Migrating data to schema 146 ...
Exception in thread "main" com.google.gwtorm.server.OrmException: Failed to rewrite user branches.
        at com.google.gerrit.server.schema.Schema_146.migrateData(Schema_146.java:93)
        at com.google.gerrit.server.schema.SchemaVersion.migrateData(SchemaVersion.java:153)
        at com.google.gerrit.server.schema.SchemaVersion.upgradeFrom(SchemaVersion.java:92)
        at com.google.gerrit.server.schema.SchemaVersion.check(SchemaVersion.java:83)
        at com.google.gerrit.server.schema.SchemaUpdater.update(SchemaUpdater.java:118)
        at com.google.gerrit.pgm.init.BaseInit$SiteRun.upgradeSchema(BaseInit.java:389)
        at com.google.gerrit.pgm.init.BaseInit.run(BaseInit.java:146)
        at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:223)
        at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:119)
        at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:63)
        at Main.main(Main.java:24)
Caused by: java.io.IOException: Failed to update ref refs/users/27/27: LOCK_FAILURE
        at com.google.gerrit.server.schema.Schema_146.rewriteUserBranch(Schema_146.java:139)
        at com.google.gerrit.server.schema.Schema_146.migrateData(Schema_146.java:87)
        ... 15 more

I checked our database and ID=27 is me (woo!). I was unable to find any relevant information online so I tabled upgrading to v2.15.2 and got back to some coding tasks for my day-to-day work. However, I was unable to pull our master branch. Eclipse gave me a message regarding SSH authentication failure. So I went to check the Gerrit's sshd_log.  

AUTH FAILURE FROM <ip-address> no-matching-key

My initial reaction was to regenerate my SSH keys and configure Eclipse and Gerrit to use them. Eclipse has the private key, Gerrit has the public key. I did so but, unfortunately, I was still got an authentication failure message. I imagine that being unable to be authenticated might have to do with my user and this LOCK_FAILURE, but I cannot seem to find any answers online.

Any help is greatly appreciated.

[Gert van Dijk]

On Monday, 11 June 2018 08:30:57 UTC+2, John S. wrote:

I ran into this exception when trying to initialize gerrit-2.15.2.war. Our system is currently running on v2.14.8. 

Here is the migration output and StackTrace:

[...]

I checked our database and ID=27 is me (woo!). I was unable to find any relevant information online so I tabled upgrading to v2.15.2 and got back to some coding tasks for my day-to-day work. However, I was unable to pull our master branch. Eclipse gave me a message regarding SSH authentication failure. So I went to check the Gerrit's sshd_log.  

AUTH FAILURE FROM <ip-address> no-matching-key

My initial reaction was to regenerate my SSH keys and configure Eclipse and Gerrit to use them. Eclipse has the private key, Gerrit has the public key. I did so but, unfortunately, I was still got an authentication failure message. I imagine that being unable to be authenticated might have to do with my user and this LOCK_FAILURE, but I cannot seem to find any answers online.

Any help is greatly appreciated.

I'm not sure whether I entirely got this right, but does, what you did, boil down to this below?

1. Perform an upgrade to 2.15.2 on a 2.14.8 system using the init command. While performing that the schema migration error occurred and you ignored it at that point.
   
2. Just start 2.14.8 again on the half-migrated data.
3. Running into the AUTH FAILURE error now.

To get your Gerrit starting again, I'd suggest to revert to a backup of all data (database + Git repository) of a known consistent state before the upgrade, *before* starting it with 2.14.x again. And avoid starting Gerrit again on a half-migrated site - it's likely to run into any kind of odd errors at this point.

I presume there's some problem with migrating some user data from the ReviewDb database to the All-Users Git repository. Can you verify ...:

• that you stopped a running Gerrit (or other process) using the repositories on your filesystem (at the Gerrit server) before starting the upgrade. You can't run 'init' on a site directory where Gerrit is running from at the same time (or at least you shouldn't).
  
• that the filesystem permissions for your Git repositories on the server are OK? Can the user running as Gerrit *and* the init command write to the All-Users git repository?
• that you run the init command as the same user as you start Gerrit as. So, if your Gerrit usually runs as user mygerrituser, then perform e.g. 'su -l mygerrituser -s /bin/bash' to become that user and perform the init command in that shell.

Is it possible for you to revert to a known working state? If not, you may be lucky to perform the upgrade again on your site with crossed-fingers to see if that it finishes the schema migration until v161.

HTH

[John S.]

Thanks for your reply

I assumed leaving my data in a mid-migrate state was a bad way to run, so I actually restored a backup before restarting the Gerrit service again. After reading your reply, I think it is also important to note that Gerrit is running on a Windows Server 2012 VM.

Here's what I did

1. Stop the Gerrit service 
2. Perform an upgrade to 2.15.2 from 2.14.8 using the init command. 
1. Schema migration fails at schema v146. 
2. StackTrace is output to the command prompt as shown in OP. 
3. It didn't seem that anything was recovering, so I closed the command prompt at this point.
3. Assuming my data was in a bad state, I reverted the backing MySQL db to schema v142. I had exported the data prior to starting the 2.15.2 upgrade process.
4. Ran the init command on 2.14.8. No schema migration occurred (as expected).
5. Reindexed the database
6. Deployed the newly initialized 2.14.8 in Tomcat
7. Started the Windows service

So to answer your question: Yes, I am currently reverted to a known working state. 

To address your bullet points

• Yes, I did stop the running Gerrit service prior to starting the upgrade
• This may be the problem point. I am logging into the Server '12 VM with a user who is shown to have administrative credentials when looking at Control Panel > User Accounts > MyUser. However, when I check the read/write access for the site path as well as the All-Users parent directory, MyUser only has Special Permissions and not Full Control.
• I am logging into the VM with the same user that starts the Gerrit service when I am running the init command. 

I am going to try to update the user access on the site path and All-Users.git parent directory to see if that works. I will be surprised if it does though. I was able to upgrade from 2.11.x -> 2.12.x -> 2.13.x -> 2.14.8 very smoothly, and these were the directory permissions for each of those upgrades. But, I will give it a shot...and this time, I will make sure to opt to create a new war while initializing instead of overwriting the existing 2.14.8 war.

Thanks for your help. Will report back

- JS

[John S.]

Also, I am able to pull master in Eclipse. This appeared to be an issue with my Eclipse Workspace.

[John S.]

Reporting back: File permissions do not appear to be an issue.