Problem using tracd daemonized (Trac 1.4)

[Falconieri Falconieri]

Hi all.

I'm getting a critical error using tracd. I'm serving Trac for multiple envs, using Ubuntu and MySQL 8. The performance is very good, everything is running ok and performing very well. However during the day, something happens and tracd stops to attend new requests. It is not deterministic. Suddenly, one time per day tracd stops. And it is very strange because it keeps running but nobody can access Trac because it does not respond any request. It is incredible because the pidfile is still there and the process alive. The resolution is to call "sudo systemctl restart tracd". Immediately, after restarting, tracd starts to serve Trac again.

Below is my systemd configuration:

[Unit]
Description = Agilo Tracd Service
After=network.target

[Service]
Type = forking
PIDFile=/tmp/tracd.pid
ExecStart = /usr/local/bin/tracd -p 443 --protocol https --certfile /xxx/conf/server.crt --keyfile /xxx/conf/server.key --basic-auth='*,/xxx/conf/trac.htpasswd, Trac' --hostname xxx.yyy.com -d --pidfile /tmp/tracd.pid -e /xxx/trac-envs
KillMode=process
Restart=on-failure

[Install]
WantedBy=multi-user.target

Many people use this Trac. Could anyone give a hint on how to solve this problem?

Thanks.

Falconieri

[Jun Omae]

Try to check systemd logs for tracd.service after reproducing the issue.

$ systemctl status tracd.service
$ journalctl -u tracd

Also, I recommend using WSGI server (e.g. mod_wsgi) for production and
tracd is not.

--
Jun Omae <jun...@gmail.com> (大前 潤)

[Falconieri Falconieri]

Hi.

Thanks for the answer. I will check it out and post here again. About the web server what server do you recommend?

Tks

[Jun Omae]

Apache with mod_wsgi (daemon mode)

[Falconieri Falconieri]

Hi Jun.

See below:

adm-grff@ip-172-31-155-44:~$ systemctl status tracd.service
● tracd.service - Agilo Tracd Service
Loaded: loaded (/etc/systemd/system/tracd.service; enabled; vendor preset: en
Active: active (running) since Thu 2020-02-06 17:59:36 -03; 5h 18min ago
Process: 20175 ExecStart=/usr/local/bin/tracd -p 443 --protocol https --certfi
Main PID: 20204 (tracd)
Tasks: 16 (limit: 4915)
CGroup: /system.slice/tracd.service
└─20204 /usr/bin/python /usr/local/bin/tracd -p 443 --protocol https
lines 1-8/8 (END)


journal
-- Logs begin at Fri 2019-12-20 08:05:56 -03, end at Thu 2020-02-06 23:52:34 -03. --
Feb 06 23:16:10 ip-172-31-155-44 sudo[22409]: root : unable to resolve host ip-172-31-155-44
Feb 06 23:16:10 ip-172-31-155-44 sudo[22409]: root : problem with defaults entries ; TTY=pts/0 ; PWD=/home/adm-grff ; USER=root ;
Feb 06 23:16:16 ip-172-31-155-44 sshd[22234]: pam_unix(sshd:session): session closed for user adm-grff
Feb 06 23:16:16 ip-172-31-155-44 su[22384]: pam_unix(su:session): session closed for user root
Feb 06 23:16:16 ip-172-31-155-44 sudo[22383]: pam_unix(sudo:session): session closed for user root
Feb 06 23:16:16 ip-172-31-155-44 systemd-logind[842]: Removed session 1442.
Feb 06 23:17:01 ip-172-31-155-44 CRON[22414]: pam_unix(cron:session): session opened for user root by (uid=0)
Feb 06 23:17:01 ip-172-31-155-44 CRON[22415]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Feb 06 23:17:01 ip-172-31-155-44 CRON[22414]: pam_unix(cron:session): session closed for user root
Feb 06 23:17:37 ip-172-31-155-44 sshd[22418]: Accepted publickey for adm-grff from 192.168.98.5 port 65010 ssh2: RSA SHA256:ZTP/CZ2bONKe0KsEZZcOY2CFgR/Qftp9RCVd0dcHjdI
Feb 06 23:17:37 ip-172-31-155-44 sshd[22418]: pam_unix(sshd:session): session opened for user adm-grff by (uid=0)
Feb 06 23:17:37 ip-172-31-155-44 systemd[1]: Started Session 1444 of user adm-grff.
Feb 06 23:17:37 ip-172-31-155-44 systemd-logind[842]: New session 1444 of user adm-grff.
Feb 06 23:19:39 ip-172-31-155-44 systemd-timesyncd[11487]: Network configuration changed, trying to establish connection.
Feb 06 23:19:39 ip-172-31-155-44 systemd-timesyncd[11487]: Synchronized to time server 10.0.0.73:123 (10.0.0.73).

========= PROBABLY the problem occurred at this time ==========


Feb 06 23:19:58 ip-172-31-155-44 sudo[22522]: adm-grff : unable to resolve host ip-172-31-155-44
Feb 06 23:19:58 ip-172-31-155-44 sudo[22522]: adm-grff : problem with defaults entries ; TTY=pts/0 ; PWD=/home/adm-grff ; USER=root ;
Feb 06 23:19:58 ip-172-31-155-44 sudo[22522]: adm-grff : TTY=pts/0 ; PWD=/home/adm-grff ; USER=root ; COMMAND=/bin/su
Feb 06 23:19:58 ip-172-31-155-44 sudo[22522]: pam_unix(sudo:session): session opened for user root by adm-grff(uid=0)
Feb 06 23:19:58 ip-172-31-155-44 su[22523]: Successful su for root by root
Feb 06 23:19:58 ip-172-31-155-44 su[22523]: + /dev/pts/0 root:root
Feb 06 23:19:58 ip-172-31-155-44 su[22523]: pam_unix(su:session): session opened for user root by adm-grff(uid=0)
Feb 06 23:19:58 ip-172-31-155-44 su[22523]: pam_systemd(su:session): Cannot create session: Already running in a session
Feb 06 23:34:31 ip-172-31-155-44 systemd[1]: Starting Daily apt download activities...
Feb 06 23:34:32 ip-172-31-155-44 systemd[1]: Started Daily apt download activities.
Feb 06 23:49:38 ip-172-31-155-44 systemd-timesyncd[11487]: Network configuration changed, trying to establish connection.
Feb 06 23:49:38 ip-172-31-155-44 systemd-timesyncd[11487]: Synchronized to time server 10.0.0.73:123 (10.0.0.73).
Feb 06 23:49:45 ip-172-31-155-44 sshd[22418]: pam_unix(sshd:session): session closed for user adm-grff
Feb 06 23:49:45 ip-172-31-155-44 su[22523]: pam_unix(su:session): session closed for user root
Feb 06 23:49:45 ip-172-31-155-44 sudo[22522]: pam_unix(sudo:session): session closed for user root
Feb 06 23:49:45 ip-172-31-155-44 systemd-logind[842]: Removed session 1444.
Feb 06 23:50:02 ip-172-31-155-44 sshd[22685]: Accepted publickey for adm-grff from 192.168.98.5 port 61468 ssh2: RSA SHA256:ZTP/CZ2bONKe0KsEZZcOY2CFgR/Qftp9RCVd0dcHjdI
Feb 06 23:50:02 ip-172-31-155-44 sshd[22685]: pam_unix(sshd:session): session opened for user adm-grff by (uid=0)
Feb 06 23:50:02 ip-172-31-155-44 systemd[1]: Started Session 1445 of user adm-grff.
Feb 06 23:50:02 ip-172-31-155-44 systemd-logind[842]: New session 1445 of user adm-grff.
Feb 06 23:52:34 ip-172-31-155-44 sudo[22795]: adm-grff : unable to resolve host ip-172-31-155-44
Feb 06 23:52:34 ip-172-31-155-44 sudo[22795]: adm-grff : problem with defaults entries ; TTY=pts/0 ; PWD=/home/adm-grff ; USER=root ;
Feb 06 23:52:34 ip-172-31-155-44 sudo[22795]: adm-grff : TTY=pts/0 ; PWD=/home/adm-grff ; USER=root ; COMMAND=/bin/su
Feb 06 23:52:34 ip-172-31-155-44 sudo[22795]: pam_unix(sudo:session): session opened for user root by adm-grff(uid=0)
Feb 06 23:52:34 ip-172-31-155-44 su[22796]: Successful su for root by root
Feb 06 23:52:34 ip-172-31-155-44 su[22796]: + /dev/pts/0 root:root
Feb 06 23:52:34 ip-172-31-155-44 su[22796]: pam_unix(su:session): session opened for user root by adm-grff(uid=0)
Feb 06 23:52:34 ip-172-31-155-44 su[22796]: pam_systemd(su:session): Cannot create session: Already running in a session

[Jun Omae]

Hi,

On Fri, Feb 7, 2020 at 12:10 PM Falconieri Falconieri
<gfalc...@gmail.com> wrote:
>
> Hi Jun.
>
> See below:
>
> adm-grff@ip-172-31-155-44:~$ systemctl status tracd.service
> ● tracd.service - Agilo Tracd Service
> Loaded: loaded (/etc/systemd/system/tracd.service; enabled; vendor preset: en
> Active: active (running) since Thu 2020-02-06 17:59:36 -03; 5h 18min ago
> Process: 20175 ExecStart=/usr/local/bin/tracd -p 443 --protocol https --certfi
> Main PID: 20204 (tracd)
> Tasks: 16 (limit: 4915)
> CGroup: /system.slice/tracd.service
> └─20204 /usr/bin/python /usr/local/bin/tracd -p 443 --protocol https
>

> journal
> -- Logs begin at Fri 2019-12-20 08:05:56 -03, end at Thu 2020-02-06 23:52:34 -03. --
> Feb 06 23:16:10 ip-172-31-155-44 sudo[22409]: root : unable to resolve host ip-172-31-155-44

The tracd has no logging option. The issue is not logged if an issue
occurs on tracd
process. Also, the stderr is redirected to /dev/null if --daemonize
option is used.
Therefore, we cannot find any issues from systemd logs.

Instead, try to use Type=simple and remove -d option for [Service]
section in tracd.service.

[Service]
Type=simple

ExecStart = /usr/local/bin/tracd -p 443 --protocol https --certfile
/xxx/conf/server.crt --keyfile /xxx/conf/server.key
--basic-auth='*,/xxx/conf/trac.htpasswd, Trac' --hostname xxx.yyy.com

-e /xxx/trac-envs
ExecStop=/bin/kill ${MAINPID}
Restart=on-failure

[Falconieri Falconieri]

Hi Jun.

About the log, I already have done the changes to daemon.py to track the actions. But it is weird: nothing is logged when the process *freezes*. This is why I call this behavior *freeze*. When the process is frozen, it gives no responses to connections. Today I have tried to telnet the port and I got a connection refused message.

Earlier, before my post, I started tracd in daemon mode, without systemd, and I'm waiting the results.

During the day, I will take the opportunity in the lunch time, to change de configs as you said in our post. I come back with other news.

Thanks a lot.
Gabriel

[Falconieri Falconieri]

Hi Jun.

I changed my serviced file and we are experiencing the new configuration today. Last Saturday I rebooted the Ubuntu because OS was sending messages to apply some patches and updates. In parallel we have created a new local environment in order to test our trac envs and plugins using Apache + mod-wsgi. In the local environment the results were very good. Probably in the end of today we will deploy the Apache server in production as you have advised.
I will get you in touch with news latter.