Gerrit service failed to start as root

[Masaru Oonuma]

Hi,

I'm facing issue during Gerrit service start up.
The issue is related to systemd, and I do not sure this issue is common for other OS.
Could yoou please let me know if you faced similar issue or if you have more better workaround/solutions for this issue?

Environment
Gerrit: 3.4.8
OS: AmazonLinux 2023

Gerrit was installed on my server and then I run that by gerrit.service.
Gerrit process itself can be started and I can see WebUI for that but Gerrit sevice said Failed to start.

/etc/systemd/system/gerrit.service
---------------------------
# Systemd unit file for Gerrit

[Unit]
Description=Gerrit Code Review
After=syslog.target network.target remote-fs.target

[Service]
Type=forking
User=root
EnvironmentFile=/etc/default/gerritcodereview
ExecStart=<GERRIT-USER-HOME>/review_site/bin/gerrit.sh start
ExecStop=<GERRIT-USER-HOME>/review_site/bin/gerrit.sh stop
PIDFile=<GERRIT-USER-HOME>/review_site/logs/gerrit.pid
TimeoutStartSec = 3000

[Install]
WantedBy=multi-user.target
---------------------------

Error
There is no error mesage in error_logs(review_site/logs/error_log)

Found error during gerrit service start
# systemctl start gerrit
Job for gerrit.service failed because the service did not take the steps required by its unit configuration.
See "systemctl status gerrit.service" and "journalctl -xeu gerrit.service" for details.

Check details for that like following.
# journalctl -xeu gerrit.service
----------------------------------------
(to gerrit) root on none
pam_unix(su-l:session): session opened for user gerrit(uid=xxxx) by (uid=0)
pam_unix(su-l:session): session closed for user gerrit
gerrit.sh[2289]: Starting Gerrit Code Review: OK
systemd[1]: gerrit.service: Refusing to accept PID outside of service control group, acquired through symlink chain: <GERRIT-USER-HOME>/review_site/logs/gerrit.pid
systemd[1]: gerrit.service: Refusing to accept PID outside of service control group, acquired through symlink chain: <GERRIT-USER-HOME>/review_site/logs/gerrit.pid
systemd[1]: gerrit.service: Failed with result 'protocol'.
----------------------------------------
That seems failed due to systemd, gerrit.pid was created once but removed by systemd.


I found 2 workaround for this issue

1. run gerrit service by gerrit user
2. modify gerrit.sh to run as root

both are worked but I do not sure which is better way..  which sould I take...

1. run gerrit service by gerrit user
Update gerrit.service to run gerrit by gerrit user.
like bellow.
------------------------------------------
change from
User=root

change to
User=gerrit
------------------------------------------
This worked with warning and it seems there is limitation regarding out of memory killer...
That said
------------------------------------------
gerrit.sh[2015]: <GERRIT-USER-HOME>/review_site/bin/gerrit.sh: line 347: ulimit: open files: cannot modify limit: Operation not permitted
gerrit.sh[2015]: Starting Gerrit Code Review: WARNING: Could not adjust Gerrit's process for the kernel's out-of-memory killer.
gerrit.sh[2015]:          This may be caused by <GERRIT-USER-HOME>/review_site/bin/gerrit.sh not being run as root.
gerrit.sh[2015]:          Consider changing the OOM score adjustment manually for Gerrit's PID=2062 with e.g.:
gerrit.sh[2015]:          echo '-1000' | sudo tee /proc/2062/oom_score_adj
gerrit.sh[2015]: OK
systemd[1]: Started gerrit.service - Gerrit Code Review.
------------------------------------------

2.modify gerrit.sh to run as root
Update gerrit.service to change path of .pid file to the new location which is owned by root
Add several lines into "gerrit.sh" to change owner of .pid file to root
This worked fine without any error/warning.

But I need to take care this point if I upgrade Gerrit version in future...

I do not sure this issue is common for other OS.
Please let me know if you faced similar issue or if you have more better workaround/solutions for this.

[Kenyon Ralph]

That gerrit.sh script is a relic of a pre-systemd era, and shouldn't be used with systemd. My gerrit.service looks like this:

[Unit]

Description=Gerrit Code Review

After=syslog.target network.target

[Service]

Type=simple

WorkingDirectory=/opt/gerrit

ExecStart=/usr/bin/java -jar /opt/gerrit/bin/gerrit.war daemon --site-path /opt/gerrit --console-log --show-stack-trace

User=gerrit

SyslogIdentifier=gerrit

StandardInput=socket

StandardOutput=journal

LimitNOFILE=20000

SuccessExitStatus=143

Restart=on-failure

[Install]

WantedBy=multi-user.target 

With a gerrit.socket like so:

[Unit]

Description=Gerrit HTTP socket

[Socket]

ListenStream=8080

BindToDevice=lo

[Install]

WantedBy=sockets.target

We use nginx as the reverse proxy in front of Gerrit's HTTP service.

[Matthias Sohn]

Did you set container.user (https://gerrit-review.googlesource.com/Documentation/config-gerrit.html#container.user) 

in $gerrit_site/etc/gerrit.config to the OS user which should own the gerrit process ?

 

--
--
To unsubscribe, email repo-discuss...@googlegroups.com
More info at http://groups.google.com/group/repo-discuss?hl=en

---
You received this message because you are subscribed to the Google Groups "Repo and Gerrit Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to repo-discuss...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/repo-discuss/2baa76a5-ed67-418c-99f8-33e59317f851n%40googlegroups.com.

[Masaru Oonuma]

Hi Ralph,

Thank you for your info.

I will update gerrit.service based on your contents.

I wondering "out-of-memory killer" works fine in your environment?

[Masaru Oonuma]

Hi Sohn,

Yes container user is defined in my configuration like below.

[container]
        user = gerrit

And then I run gerrit.sh by root acount.

As a result, gerrit.sh run by root and process is owned by gerrit user in my understanding.

[Masaru Oonuma]

Hi Ralph,

I could start Gerrit with gerrit.service which is based on you provided one.

And also starting Gerrit by gerrit.sh also works with gerrit user.

So I'm wondering which way should I take...

You said,

> That gerrit.sh script is a relic of a pre-systemd era, and shouldn't be used with systemd.

Is that Gerrit's official position?

On Tuesday, July 30, 2024 at 2:16:03 AM UTC+9 Kenyon Ralph wrote:

[Kenyon Ralph]

On Monday, August 5, 2024 at 2:10:51 AM UTC-7 Masaru Oonuma wrote:

Hi Ralph,

I could start Gerrit with gerrit.service which is based on you provided one.

And also starting Gerrit by gerrit.sh also works with gerrit user.

So I'm wondering which way should I take...

It's totally up to you.

 

You said,

> That gerrit.sh script is a relic of a pre-systemd era, and shouldn't be used with systemd.

Is that Gerrit's official position?

No, that's just my opinion.

[Masaru Oonuma]

Hi Ralph,

Thank you for your comments.

I will share that with team and dissucus which way should I take.

Thanks.