Gerrit restart fails on container

[Shad Perwez]

Hi experts,

gerrit restart failing even if it is run as root user or gerrit user

Please suggest any solution.

Error:

bash-4.2$ ./gerrit.sh restart

Stopping Gerrit Code Review: OK

Starting Gerrit Code Review: WARNING: Could not adjust Gerrit's process for the kernel's out-of-memory killer.

         This may be caused by /var/gerrit/bin/./gerrit.sh not being run as root.

         Consider changing the OOM score adjustment manually for Gerrit's PID=579 with e.g.:

         echo '-1000' | sudo tee /proc/579/oom_score_adj

FAILED

Note: gerrit hosted as docker container 

output of status

bash-4.2$ ./gerrit.sh status

Checking arguments to Gerrit Code Review:

  GERRIT_SITE            =  /var/gerrit

  GERRIT_CONFIG          =  /var/gerrit/etc/gerrit.config

  GERRIT_PID             =  /var/gerrit/logs/gerrit.pid

  GERRIT_TMP             =  /var/gerrit/tmp

  GERRIT_WAR             =  /var/gerrit/bin/gerrit.war

  GERRIT_FDS             =  1024

  GERRIT_USER            =  root

  GERRIT_STARTUP_TIMEOUT =  90

  JAVA                   =  /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre/bin/java

  JAVA_OPTIONS           =   -Dflogger.backend_factory=com.google.common.flogger.backend.log4j.Log4jBackendFactory#getInstance -Dflogger.logging_context=com.google.gerrit.server.logging.LoggingContext#getInstance -Djava.security.egd=file:/dev/./urandom

  RUN_EXEC               =  /usr/bin/perl -e '$x=$ENV{JAVA};exec $x @ARGV;die $!' -- GerritCodeReview

  RUN_ARGS               =   -Dflogger.backend_factory=com.google.common.flogger.backend.log4j.Log4jBackendFactory#getInstance -Dflogger.logging_context=com.google.gerrit.server.logging.LoggingContext#getInstance -Djava.security.egd=file:/dev/./urandom  -jar /var/gerrit/bin/gerrit.war daemon -d /var/gerrit

[Marco Miller]

-I'd have these 4 points below.

/Marco

On Thursday, April 16, 2020 at 11:13:26 AM UTC-4, Shad Perwez wrote:

Hi experts,

gerrit restart failing even if it is run as root user or gerrit user

Please suggest any solution.

Error:

bash-4.2$ ./gerrit.sh restart

1. Is this command line truly done as root, here or in the container?

(Just in case it isn't.)

Stopping Gerrit Code Review: OK

Starting Gerrit Code Review: WARNING: Could not adjust Gerrit's process for the kernel's out-of-memory killer.

         This may be caused by /var/gerrit/bin/./gerrit.sh not being run as root. 

         Consider changing the OOM score adjustment manually for Gerrit's PID=579 with e.g.:

         echo '-1000' | sudo tee /proc/579/oom_score_adj

2. So if not effectively root, you will likely need to follow that last instruction.

In either case, you may need to check if that command manually fails, and if so, what's the error.

FAILED

Note: gerrit hosted as docker container

3. Using which Dockerfile, yours or an official Gerrit one? (Which one, the version?)

I'm asking mostly because that Dockerfile may misconfigure the USER starting Gerrit, if custom.

 output of status

bash-4.2$ ./gerrit.sh status

Checking arguments to Gerrit Code Review:

  GERRIT_SITE            =  /var/gerrit

  GERRIT_CONFIG          =  /var/gerrit/etc/gerrit.config

  GERRIT_PID             =  /var/gerrit/logs/gerrit.pid

  GERRIT_TMP             =  /var/gerrit/tmp

  GERRIT_WAR             =  /var/gerrit/bin/gerrit.war

  GERRIT_FDS             =  1024

  GERRIT_USER            =  root

4. Says root here but still wondering if this matches the user effectively running gerrit.sh above.

[Luca Milanesio]

My 2c: you are not mounting /dev/urandom isn’t it?

Luca.

4. Says root here but still wondering if this matches the user effectively running gerrit.sh above.

 

  GERRIT_STARTUP_TIMEOUT =  90

  JAVA                   =  /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64/jre/bin/java

  JAVA_OPTIONS           =   -Dflogger.backend_factory=com.google.common.flogger.backend.log4j.Log4jBackendFactory#getInstance -Dflogger.logging_context=com.google.gerrit.server.logging.LoggingContext#getInstance -Djava.security.egd=file:/dev/./urandom

  RUN_EXEC               =  /usr/bin/perl -e '$x=$ENV{JAVA};exec $x @ARGV;die $!' -- GerritCodeReview

  RUN_ARGS               =   -Dflogger.backend_factory=com.google.common.flogger.backend.log4j.Log4jBackendFactory#getInstance -Dflogger.logging_context=com.google.gerrit.server.logging.LoggingContext#getInstance -Djava.security.egd=file:/dev/./urandom  -jar /var/gerrit/bin/gerrit.war daemon -d /var/gerrit

-- 
-- 
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/ca505f2f-e7e3-46ab-a611-1f893692ab78%40googlegroups.com.

[Shad Perwez]

Hi Miller/Luca,

I am using gerrit image official one from dockerhub.

1. The script has run by using gerrit user and facing this error message and stopping is not the issue, but the starting is failing.

    When I did it with root user, faced permission denied message, then changed the permissions, and faced the same issue even in root user.

2. As we clearly can see GERRIT_USER            =  root , so I did it using root. even also did with gerrit user.

3. But the when it comes to costume dockerfile, the restart, stop, start service is working perfectly. 

    Do not worry about the WARNING output, I have seen the script , it is just a warning output.

 4. But the thing is why is it failing on official docker container, Is there anything that we are missing to mount, or to configure.

Note: we need to restart the service from outside of the container to overcome some issue,  It is a production blocker for us.

So please suggest , that will be very helpful.

@Luca Milanesio , Nope we are not mounting /dev/urandom  as mount volume or any kind of volume. But the file is there in the container .

So can you please elaborate , why /how should we mount the mentioned file and how is it related to resarting the gerrit service inside the container.

Thanks in advance... 

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/76047EB6-E28B-47C0-8C0A-D6E44513DABD%40gmail.com.

[Sven Selberg]

On Friday, April 17, 2020 at 4:21:32 AM UTC+2, Shad Perwez wrote:

Hi Miller/Luca,

I am using gerrit image official one from dockerhub.

1. The script has run by using gerrit user and facing this error message and stopping is not the issue, but the starting is failing.

    When I did it with root user, faced permission denied message, then changed the permissions, and faced the same issue even in root user.

2. As we clearly can see GERRIT_USER            =  root , so I did it using root. even also did with gerrit user.

3. But the when it comes to costume dockerfile, the restart, stop, start service is working perfectly. 

    Do not worry about the WARNING output, I have seen the script , it is just a warning output.

 4. But the thing is why is it failing on official docker container, Is there anything that we are missing to mount, or to configure.

Note: we need to restart the service from outside of the container to overcome some issue,  It is a production blocker for us.

Are you running gerrit.sh from outside of the container? If so I'm not surprised that it doesn't start.
Why can't you just restart the container?

 

So please suggest , that will be very helpful.

...@Luca Milanesio , Nope we are not mounting /dev/urandom  as mount volume or any kind of volume. But the file is there in the container .

So can you please elaborate , why /how should we mount the mentioned file and how is it related to resarting the gerrit service inside the container.

Thanks in advance... 

To unsubscribe, email rep...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

--- 
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/ca505f2f-e7e3-46ab-a611-1f893692ab78%40googlegroups.com.

--
--
To unsubscribe, email repo-d...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-d...@googlegroups.com.

[Shad Perwez]

Hi Sven,

I am doign restart from inside the container.

I am raising the question because of the restart failure , it is impacting the lfs configuration that we are trying to do.

As I mentioned , on a custom docker container the service is restarting , In which our lfs configuration working fine after the restart.

But in the official container , the lfs part is not working, because of the service restart failure.

Note: Restarting the container is not effecting the LFS configuration .

So, we will be frequently using the official gerrit container , thats why I need to understand why the restart process is failing in a official container.

To unsubscribe, email repo-discuss...@googlegroups.com

More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.

To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/360ae963-62d0-4cf8-88a3-a566e16a5b31%40googlegroups.com.

[Sven Selberg]

Do you get some more output if you run:

./gerrit.sh run

/Sven

To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/360ae963-62d0-4cf8-88a3-a566e16a5b31%40googlegroups.com.

[Luca Milanesio]

Hi Shad,

Some comments below.

On 17 Apr 2020, at 03:21, Shad Perwez <tech....@gmail.com> wrote:

Hi Miller/Luca,

I am using gerrit image official one from dockerhub.

The official image does not support the restart use-case as the container would die if Gerrit goes down.

P.S. In the “containers world” the restart of a service is the restart of the container: you do not restart processes inside the container. Your use-case is not typical at all for containers.

1. The script has run by using gerrit user and facing this error message and stopping is not the issue, but the starting is failing.

    When I did it with root user, faced permission denied message, then changed the permissions, and faced the same issue even in root user.

Yes, the message is a warning on a different issue, the OOM killer.

2. As we clearly can see GERRIT_USER            =  root , so I did it using root. even also did with gerrit user.

3. But the when it comes to costume dockerfile, the restart, stop, start service is working perfectly. 

You mean a “custom Dockerfile” ? Is that your custom Dockerfile? How is different from the official one?

    Do not worry about the WARNING output, I have seen the script , it is just a warning output.

 4. But the thing is why is it failing on official docker container, Is there anything that we are missing to mount, or to configure.

The official container is suitable for a “playground” environment and then, together with external mounts and configuration, can be used in production environments.

There is also an example on how to use the official container in an AWS container environment on:

https://gerrit.googlesource.com/aws-gerrit

Note: we need to restart the service from outside of the container to overcome some issue,  It is a production blocker for us.

See my earlier feedback that says you *cannot use* the standard container as-is for production, as you would need *a lot more stuff* for a real production setup:

- Proper authentication (LDAP or similar)

- Volume persistence

- Monitoring

- Logs collection

- DNS

- SSL

So please suggest , that will be very helpful.

@Luca Milanesio , Nope we are not mounting /dev/urandom  as mount volume or any kind of volume.

*DO NOT* use the internal Docker volumes for any production data, as they will be lost after the container is shutdown.

Also, the Docker filesystem isn’t fast at all and is degrading in performance with the increase of data.

But the file is there in the container .

So can you please elaborate , why /how should we mount the mentioned file and how is it related to resarting the gerrit service inside the container.

You need to mount external volumes, including the random number generator.

E.g. /dev/urandom:/dev/random

HTH

Luca.