Gerrit 2.15.2 fails when submitting related changes

[Mateusz Grzechociński]

Hi,

I use Gerrit 2.15.2 as a Docker image, provided by this repo https://github.com/openfrontier/docker-gerrit. I store whole Gerrit review site as a Docker volume on my machine.

After upgrading from Gerrit 2.14.x to 2.15.2 I'm now struggling with an issue when working with the related changes.

Steps to reproduce

1. Create two commits in your local repository:

commit 2f74faa64b0bf680faf147db721427d8a03c42ad (HEAD -> master)
Author: Mateusz Grzechociński <...>
Date:   Thu Jul 26 14:24:16 2018 +0200

    Second change

    Change-Id: I9a56f3d29aa3db4e86a3c4b0c4967fa4a38fba36

commit 57d20db51a6a2cbd2b953653dfc891f5e530eca7
Author: Mateusz Grzechociński <...>
Date:   Thu Jul 26 14:24:03 2018 +0200

    First change

    Change-Id: I4c7defd5db80fe3d2fe72fb930b5de262556ed41

2. Push them to Gerrit:

↪  git push origin HEAD:refs/for/master
Counting objects: 6, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 668 bytes | 668.00 KiB/s, done.
Total 6 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3)
remote: Processing changes: new: 2, done
remote:
remote: New Changes:
remote:   https://localhost/gerrit/#/c/training/+/1904 First change
remote:   https://localhost/gerrit/#/c/training/+/1905 Second change
remote:
To ssh://localhost:29418/training
 * [new branch]      HEAD -> refs/for/master

3. Mark both of them as Code-review +2, Verified +1. On the following screenshot, "CM" is my custom label which marks commit message as well-formatted by the script running on my Jenkins.

4. Try to submit first or second (including first) change. Gerrit fails and shows a message in the UI:

For 1904 ("First change") submit, Gerrit shows: 

Could not perform action: Failed to submit 1 change due to the following problems: Change 1904: Revision 57d20db51a6a2cbd2b953653dfc891f5e530eca7 of patch set 1 does not match refs/changes/04/1904/1 for change

For 1905 ("Second change") submit, Gerrit shows:

Could not perform action: Failed to submit 2 changes due to the following problems: Change 1904: Revision 57d20db51a6a2cbd2b953653dfc891f5e530eca7 of patch set 1 does not match refs/changes/04/1904/1 for change Change 1905: Revision 2f74faa64b0bf680faf147db721427d8a03c42ad of patch set 1 does not match refs/changes/05/1905/1 for change

Workaround #1

Issue occurs only via web interface in both PolyGerrit and the old UI. When I try to push them directly to refs/heads/master, it accepts both of them:

↪  git push origin HEAD:refs/heads/master
Counting objects: 6, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (6/6), done.
Writing objects: 100% (6/6), 668 bytes | 668.00 KiB/s, done.
Total 6 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3)
remote: Processing changes: done
To ssh://localhost:29418/training
   e22700d..2f74faa  HEAD -> master

After that, both PolyGerrit and the old UI consider those changes as merged:

Workaround #2

When I look into refs/changes/ in my Git repo in Gerrit I see:

bash-4.4# tree
.
├── 04
│   └── 1904
└── 05
    └── 1905

There are no files in this directory. When I create a file named "1" (as it's the first patchset), containing commit tree-ish, I can successfully merge both changes from the UI and everything works as expected.

bash-4.4# tree
.
├── 04
│   └── 1904
│       └── 1
└── 05
    └── 1905
        └── 1

bash-4.4# cat 04/1904/1
57d20db51a6a2cbd2b953653dfc891f5e530eca7

bash-4.4# cat 05/1905/1
2f74faa64b0bf680faf147db721427d8a03c42ad

Unfortunately, I'm not able to reproduce it on a clean Gerrit 2.15.2 install so far. Both workarounds don't work on a scale which my project has. I'd appreciate your hints or any advice fow to fix this. Or maybe it a bug in Gerrit itself and should be reported to the issue tracker?

Thanks in advance for any help.

-- 

MG

[Gert van Dijk]

On Thursday, 26 July 2018 15:58:56 UTC+2, Mateusz Grzechociński wrote:

I use Gerrit 2.15.2 as a Docker image, provided by this repo https://github.com/openfrontier/docker-gerrit. I store whole Gerrit review site as a Docker volume on my machine.

After upgrading from Gerrit 2.14.x to 2.15.2 I'm now struggling with an issue when working with the related changes.

Just guessing here, but I believe this has to do with the NoteDb migration or it might be some file system permission issue. Getting some more details will probably help:

• How exactly did you perform the upgrade?
  
• Did you perform the optional NoteDb migration of change data [1]?
• If so, did you use the online or offline upgrade?
  
• If so, did you encounter any issues during the migration?
  
• What is the current content of the following file: <sitepath>/etc/notedb.config
• What do you have as configuration settings in your <sitepath>/etc/gerrit.config for noteDb? perform e.g.:
  
• git config --file <sitepath>/etc/gerrit.config --get-all | grep -Fi notedb
• Could you verify the file system level permissions for the repositories? Can you spot any deviations in owner/group and permissions for those refs before and after?
• Do you see any server error logs lines appearing when pushing the changes?

For the last two items, I presume it should not report to you that it has created new changes, but you never know... and perhaps I'm looking in the wrong direction here.

(I once created a Docker image from a Dockerfile which had an implicit user ID assigned for the Gerrit user on build time. When it changed in this image due to base image changes, and I upgraded the image using site data created with the older version, it resulted in quite some unexpected behavior. I know have checks in place to ensure this doesn't happen anymore. I see that your Docker image is also very implicit and could potentially fall for this [2], as adduser just increments uids/gids from /etc/passwd. Hint: use 'ls -n' to list files with numeric IDs to compare against.)

[1]: https://www.gerritcodereview.com/releases/2.15.md#notedb-for-changes

[2]: https://github.com/openfrontier/docker-gerrit/blob/8e6cd40e9dd20907b0482d02c9f88652c41f3678/Dockerfile#L14

[Mateusz Grzechociński]

Hi, thanks for the response. I answer inline.

W dniu czwartek, 26 lipca 2018 20:08:13 UTC+2 użytkownik Gert van Dijk napisał:

Just guessing here, but I believe this has to do with the NoteDb migration or it might be some file system permission issue. Getting some more details will probably help:

• How exactly did you perform the upgrade?
  

I've just changed the Docker image from openfrontier/gerrit:2.14.9 to openfrontier/gerrit:2.15.2. Nothing special here, I guess.

• Did you perform the optional NoteDb migration of change data [1]?
• If so, did you use the online or offline upgrade?
  
• If so, did you encounter any issues during the migration?
  

According to the Docker image project docs:

Since Gerrit 2.15, NoteDB is recommended to store account data, group data and change data.
Accounts and Groups are migrated offline to NoteDB automatically during the start up of the container.
Change data can be migrated to NoteDB offline via the MIGRATE_TO_NOTEDB_OFFLINE environment variable.

I did not enable this flag and stayed with H2 for changes. AFAIK, this means I have only account/group data migrated to NoteDB.

 

• What is the current content of the following file: <sitepath>/etc/notedb.config

I do not have such file:

bash-4.4# find review_site/etc/ -name "notedb.config"
bash-4.4#

 

• What do you have as configuration settings in your <sitepath>/etc/gerrit.config for noteDb? perform e.g.:
  
• git config --file <sitepath>/etc/gerrit.config --get-all | grep -Fi notedb

Pasted command doesn't work so I had to tune it a bit, but it still seems like I do not have any specific config regarding notedb, am I?

bash-4.4# git config -f review_site/etc/gerrit.config --get-all notedb
error: key does not contain a section: notedb
bash-4.4# git config -f review_site/etc/gerrit.config -l | grep notedb
bash-4.4#  

• Could you verify the file system level permissions for the repositories? Can you spot any deviations in owner/group and permissions for those refs before and after?

 I can't see any differency actually. The only thing which look suspicious is that some folders are created by root, and some by gerrit2:

drwxr-xr-x    3 root     root            96 Jul 27 10:26 01

drwxr-xr-x    3 root     root            96 Jul 27 10:26 02

drwxr-xr-x    4 gerrit2  root           128 Jul 27 10:26 04

drwxr-xr-x    4 gerrit2  root           128 Jul 27 10:26 05

drwxr-xr-x    3 gerrit2  root            96 Jul 26 13:35 06

drwxr-xr-x    3 gerrit2  root            96 Jul 26 13:35 07

drwxr-xr-x    4 gerrit2  root           128 Jul 27 10:26 08

drwxr-xr-x    5 gerrit2  root           160 Jul 27 10:26 09

drwxr-xr-x    5 gerrit2  root           160 Jul 27 10:26 10

drwxr-xr-x    4 root     root           128 Jul 27 10:26 11

drwxr-xr-x    3 root     root            96 Jul 27 10:26 12

drwxr-xr-x    3 root     root            96 Jul 27 10:26 21

drwxr-xr-x    3 root     root            96 Jul 27 10:26 22

drwxr-xr-x    5 root     root           160 Jul 27 10:26 39

drwxr-xr-x    4 root     root           128 Jul 27 10:26 40

drwxr-xr-x    4 root     root           128 Jul 27 10:26 41

drwxr-xr-x    4 root     root           128 Jul 27 10:26 42

This might be the issue?

• Do you see any server error logs lines appearing when pushing the changes?

Nothing fancy. When I try to submit (this is a log from a different, production instance):

[2018-07-27 09:51:51,464] [HTTP-3093] ERROR com.google.gerrit.server.git.MergeOp : Revision f06f91d1e3000e9d62f0669b653aabb8c431f527 of patch set 1 does not match refs/changes/94/1894/1 for change 1894
[2018-07-27 09:51:51,464] [HTTP-3093] ERROR com.google.gerrit.server.git.MergeOp : Revision a789b41668a91b29698086d4661d37b7af979d3e of patch set 5 does not match refs/changes/83/1883/5 for change 1883
[2018-07-27 09:51:51,464] [HTTP-3093] ERROR com.google.gerrit.server.git.MergeOp : Revision 9dd9e1f456ec1a03b8b10e8091a1e3bff5d1a688 of patch set 7 does not match refs/changes/82/1882/7 for change 1882

 

For the last two items, I presume it should not report to you that it has created new changes, but you never know... and perhaps I'm looking in the wrong direction here.

(I once created a Docker image from a Dockerfile which had an implicit user ID assigned for the Gerrit user on build time. When it changed in this image due to base image changes, and I upgraded the image using site data created with the older version, it resulted in quite some unexpected behavior. I know have checks in place to ensure this doesn't happen anymore. I see that your Docker image is also very implicit and could potentially fall for this [2], as adduser just increments uids/gids from /etc/passwd. Hint: use 'ls -n' to list files with numeric IDs to compare against.)

[1]: https://www.gerritcodereview.com/releases/2.15.md#notedb-for-changes

[2]: https://github.com/openfrontier/docker-gerrit/blob/8e6cd40e9dd20907b0482d02c9f88652c41f3678/Dockerfile#L14

Let's say I have pushed two related changes: 1808 and 1809 (one on another) and one non related change 1810. The issue seems pretty easy to reproduce. In such a scenario, Gerrit doesn't create files with commit tree-ish:

bash-4.4# tree

.

├── 08

│   └── 1908

├── 09

│   └── 1909

├── 10

│   └── 1910

│       └── 1

AFAIK, that's the main reason why I can easily merge non-related change 1910, but can't do that for neither 1908 nor 1909.

My main question is - how it should actually work for related changes - should there be files with commit tree-ish created or not. Is this file structure ok, or not?

bash-4.4# tree

.

├── 08

│   └── 1908 # Related to 1909. No file here.

├── 09

│   └── 1909 # Depends on 1908. No file here.

├── 10

│   └── 1910 # Non related to 1908 or 1909. File is generated. Submit works.

│       └── 1 

[Gert van Dijk]

On Friday, 27 July 2018 12:34:04 UTC+2, Mateusz Grzechociński wrote:

Hi, thanks for the response. I answer inline.

[...]

I can't see any differency actually. The only thing which look suspicious is that some folders are created by root, and some by gerrit2:

drwxr-xr-x    3 root     root            96 Jul 27 10:26 01

drwxr-xr-x    3 root     root            96 Jul 27 10:26 02

drwxr-xr-x    4 gerrit2  root           128 Jul 27 10:26 04

drwxr-xr-x    4 gerrit2  root           128 Jul 27 10:26 05

drwxr-xr-x    3 gerrit2  root            96 Jul 26 13:35 06

drwxr-xr-x    3 gerrit2  root            96 Jul 26 13:35 07

drwxr-xr-x    4 gerrit2  root           128 Jul 27 10:26 08

drwxr-xr-x    5 gerrit2  root           160 Jul 27 10:26 09

drwxr-xr-x    5 gerrit2  root           160 Jul 27 10:26 10

drwxr-xr-x    4 root     root           128 Jul 27 10:26 11

drwxr-xr-x    3 root     root            96 Jul 27 10:26 12

drwxr-xr-x    3 root     root            96 Jul 27 10:26 21

drwxr-xr-x    3 root     root            96 Jul 27 10:26 22

drwxr-xr-x    5 root     root           160 Jul 27 10:26 39

drwxr-xr-x    4 root     root           128 Jul 27 10:26 40

drwxr-xr-x    4 root     root           128 Jul 27 10:26 41

drwxr-xr-x    4 root     root           128 Jul 27 10:26 42

This might be the issue?

Yeah, that looks like an issue, definitely. Did you ever run manual commands with a shell in the container? By default it runs as root and the entrypoint appears responsible for su-ing to the gerrit2 user. If you ever run arbitrary commands, you're on your own... It could also be an issue/bug with the Docker image (entrypoint/scripting - a single omission of a su-exec prefix can cause this).

Files managed and supposed to be writable by Gerrit (such as git repositories), should be owned by gerrit2 only, in this Docker image if I'm correct. If they are root-owned, they must have been created by a process running as root.

I'd suggest to do something like the following:

• Find the numeric uid&gid of the 'gerrit2' user in the container. (e.g. run 'id' in shell inside the container as user gerrit2)
• Stop the container.
  
• Run a "chown -R <uid>:<gid> <site-path>" on the data.
• Restart the container.

(Oh and welcome to the world of Docker and broken images put on the internet. Running an SQL database next to Gerrit Java in a single container, really...? That scares the hell out of me - Docker should have a single process to manage.)

HTH

[Mateusz Grzechociński]

W dniu piątek, 27 lipca 2018 13:03:00 UTC+2 użytkownik Gert van Dijk napisał:

Yeah, that looks like an issue, definitely. Did you ever run manual commands with a shell in the container? By default it runs as root and the entrypoint appears responsible for su-ing to the gerrit2 user. If you ever run arbitrary commands, you're on your own... It could also be an issue/bug with the Docker image (entrypoint/scripting - a single omission of a su-exec prefix can cause this).

Files managed and supposed to be writable by Gerrit (such as git repositories), should be owned by gerrit2 only, in this Docker image if I'm correct. If they are root-owned, they must have been created by a process running as root.

I'd suggest to do something like the following:

• Find the numeric uid&gid of the 'gerrit2' user in the container. (e.g. run 'id' in shell inside the container as user gerrit2)
• Stop the container.
  
• Run a "chown -R <uid>:<gid> <site-path>" on the data.
• Restart the container.

(Oh and welcome to the world of Docker and broken images put on the internet. Running an SQL database next to Gerrit Java in a single container, really...? That scares the hell out of me - Docker should have a single process to manage.)

Thanks. Meanwhile I followed the path of migrating to NoteDB. It succeeded after around half a minute (no issues in the startup logs) but now I cannot push any single change:

↪  git push origin HEAD:refs/for/master

Counting objects: 6, done.

Delta compression using up to 8 threads.

Compressing objects: 100% (5/5), done.

Writing objects: 100% (6/6), 634 bytes | 634.00 KiB/s, done.

Total 6 (delta 3), reused 0 (delta 0)

remote: Resolving deltas: 100% (3/3)

remote: Processing changes: new: 1, refs: 1, done

remote:

remote: New Changes:

remote:   https://localhost/gerrit/#/c/training/+/1945 Test3

remote:

To ssh://localhost:29418/training

 ! [remote rejected] HEAD -> refs/for/master (internal server error: Error inserting change/patchset)

error: failed to push some refs to 'ssh://mateusz.grzechocinski@localhost:29418/training'

error_log contains a lot of stack traces complainign about missing change: https://gist.github.com/mgrzechocinski/8349c8bce4981a2e61222339b8911fd9. Seems like I'm not the only one having this issue [1]. 

That change is actually pushed into refs/changes, but again, no file is created. I still don't know whether that's ok or not. Should there be a file with patchset name?

bash-4.4# tree changes/45
changes/45
└── 1945

Trying to push this commit again results with "No new changes":

↪  git push origin HEAD:refs/for/master
Counting objects: 6, done.
Delta compression using up to 8 threads.

Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 634 bytes | 634.00 KiB/s, done.

Total 6 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3)

remote: Processing changes: refs: 1, done
To ssh://localhost:29418/training
 ! [remote rejected] HEAD -> refs/for/master (no new changes)
error: failed to push some refs to 'ssh://mateusz.grzechocinski@localhost:29418/training'

Fun fact is that when I restart my container and Gerrit reindexes itself during startup, change appears in the PolyGerrit and I'm able to submit it.

Should it be connected with the permission mess as well? I will try to fix it, according to your hints and get back to you ASAP. Thanks for your support.

[1] https://groups.google.com/forum/#!topic/repo-discuss/KahIBGkW2XA

[Gert van Dijk]

On Friday, 27 July 2018 13:24:00 UTC+2, Mateusz Grzechociński wrote:

Should it be connected with the permission mess as well?

Probably related, yes.

The permission mess is partially caused by the confusing way this Docker image is working. I believe I can rant for quite a while looking at the approach the developer of that container has taken. ;)

Also a general remark: have you looked at Gerrit server logs as well? You don't seem to inspect them, reading your earlier messages. They are usually pretty relevant in such cases to get clues.

[Mateusz Grzechociński]

W dniu piątek, 27 lipca 2018 13:31:52 UTC+2 użytkownik Gert van Dijk napisał 

The permission mess is partially caused by the confusing way this Docker image is working. I believe I can rant for quite a while looking at the approach the developer of that container has taken. ;)

Fine. Then what's your recommended approach to run Gerrit in an environment where all other services like Jekins, Nginx etc are dockerized? Is there any better approach to Dockerize Gerrit or should I just use it on a host machine?

 

Also a general remark: have you looked at Gerrit server logs as well? You don't seem to inspect them, reading your earlier messages. They are usually pretty relevant in such cases to get clues.

Well, I cannot find any server log. When Gerrit runs as daemon, it outputs all the logs to stdout so that's what I see. When I run it as supervise, it created error_log file and AFAIK, redirect its output there. That what the docs says: https://github.com/openfrontier/docker-gerrit#override-the-default-startup-action 

Last but no least, Gert, could you please answer the question about the file with patchset name? Should they always be created for every single change or not? 

[Gert van Dijk]

On Friday, 27 July 2018 13:40:48 UTC+2, Mateusz Grzechociński wrote:

Then what's your recommended approach to run Gerrit in an environment where all other services like Jekins, Nginx etc are dockerized? Is there any better approach to Dockerize Gerrit or should I just use it on a host machine?

In general, Docker containers should run a single process. That means each service should have their own container. It's out of scope for this mailing list to discuss that in detail, I think, so I'll keep it to this paragraph. I would definitely recommend running separate containers for the database/PostgreSQL, the reverse HTTP proxy and Gerrit. (However, I have switched to Gerrit/master, dropping the SQL dependency completely.) Managing multiple containers to run a single "service" in your network can be achieved by using tooling that ranges from simple orchestration with docker-compose to advanced/complex setups with Kubernetes and Docker Swarm - many options out there!

I believe all of the services involved here are pretty easy to run containerized. Jenkins, Gerrit, PostgreSQL, nginx/Apache, I had them all running "properly" in Docker with relative ease. Whether or not to run it in a container is a choice you'll have to make yourself; it's very opinionated as well as depending on your situation. Personally, I like it from a security point of view as welll as resource efficiency (compared to segregation with virtual machines), but it comes with some downsides too.

 

Well, I cannot find any server log. When Gerrit runs as daemon, it outputs all the logs to stdout so that's what I see. When I run it as supervise, it created error_log file and AFAIK, redirect its output there. That what the docs says: https://github.com/openfrontier/docker-gerrit#override-the-default-startup-action

If this image produces the logs to stdout/stderr, use "docker logs <container name>" to access the logs, in a default Docker daemon configuration that should return you the contents.

 

Last but no least, Gert, could you please answer the question about the file with patchset name? Should they always be created for every single change or not? 

On Friday, 27 July 2018 13:24:00 UTC+2, Mateusz Grzechociński wrote:

That change is actually pushed into refs/changes, but again, no file is created. I still don't know whether that's ok or not. Should there be a file with patchset name?

bash-4.4# tree changes/45
changes/45
└── 1945

My git filesystem layout knowledge is limited, but I believe that is OK, as there's a 'packed-refs' file having this information. This is a plain textfile with hashes pointing to refs, e.g.:

<sha1> refs/changes/10/210/1
<sha1> refs/changes/10/210/2
<sha1> refs/changes/10/210/meta

Perhaps check the contents of that file as well!

[Mateusz Grzechociński]

W dniu piątek, 27 lipca 2018 14:00:34 UTC+2 użytkownik Gert van Dijk napisał:

On Friday, 27 July 2018 13:40:48 UTC+2, Mateusz Grzechociński wrote:

Then what's your recommended approach to run Gerrit in an environment where all other services like Jekins, Nginx etc are dockerized? Is there any better approach to Dockerize Gerrit or should I just use it on a host machine?

In general, Docker containers should run a single process. That means each service should have their own container. It's out of scope for this mailing list to discuss that in detail, I think, so I'll keep it to this paragraph. I would definitely recommend running separate containers for the database/PostgreSQL, the reverse HTTP proxy and Gerrit. (However, I have switched to Gerrit/master, dropping the SQL dependency completely.) Managing multiple containers to run a single "service" in your network can be achieved by using tooling that ranges from simple orchestration with docker-compose to advanced/complex setups with Kubernetes and Docker Swarm - many options out there!

I believe all of the services involved here are pretty easy to run containerized. Jenkins, Gerrit, PostgreSQL, nginx/Apache, I had them all running "properly" in Docker with relative ease. Whether or not to run it in a container is a choice you'll have to make yourself; it's very opinionated as well as depending on your situation. Personally, I like it from a security point of view as welll as resource efficiency (compared to segregation with virtual machines), but it comes with some downsides too.

Basically, that's how my setup look like. I use docker-compose with few containers - one for each service like Jenkins, Gerrit, Nginx, Swagger and more. As far as I understand, the only thing you're complaining about it that Gerrit and H2 DB are one Docker container, right?. Whichever attitude I choose, I'll always have to have a volume to store my Gerrit state in some kind of a local storage to save them between container restarts.

 

 

Well, I cannot find any server log. When Gerrit runs as daemon, it outputs all the logs to stdout so that's what I see. When I run it as supervise, it created error_log file and AFAIK, redirect its output there. That what the docs says: https://github.com/openfrontier/docker-gerrit#override-the-default-startup-action

If this image produces the logs to stdout/stderr, use "docker logs <container name>" to access the logs, in a default Docker daemon configuration that should return you the contents.

By default I use docker-compose up, without "-d" which AFAIK redirects all the output from all containers to stdout. That's how I'm browsing them.

 

 

Last but no least, Gert, could you please answer the question about the file with patchset name? Should they always be created for every single change or not? 

On Friday, 27 July 2018 13:24:00 UTC+2, Mateusz Grzechociński wrote:

That change is actually pushed into refs/changes, but again, no file is created. I still don't know whether that's ok or not. Should there be a file with patchset name?

bash-4.4# tree changes/45
changes/45
└── 1945

My git filesystem layout knowledge is limited, but I believe that is OK, as there's a 'packed-refs' file having this information. This is a plain textfile with hashes pointing to refs, e.g.:

<sha1> refs/changes/10/210/1
<sha1> refs/changes/10/210/2
<sha1> refs/changes/10/210/meta

Perhaps check the contents of that file as well!

I did. Indeed, when there are no files in refs/changes, there is a new entry in this file.

After spending almost entire day on this issue, I came into the conclusion that it has to be connected with the permissions issue. As long as all files are owned by gerrit2, everything works fine. When either refs/changes/* or "packed-refs" changes its owner, things starting to break. Unfortunately, I have no idea how to reproduce it, why and when some files are generated by Gerrit with "root" owner if java process is run as gerrit2 user. I opened an issue/question on docker-image GitHub repo so that  maybe someone would explain it to me. 

Thank you Gert for your help. I really appreciate it.