There was a problem on
api.opentreeoflife.org today: The oti process was running as root, and therefore couldn't be killed by the deployment scripts. I'm not interested in knowing how this came about, because it's a perfectly sensible mistake to make, but going forward I would like anyone who deals with the servers, especially the production servers, to know that nearly all of the files and processes specifically involved in the operation of the server are owned by the 'opentree' user, and if things are done as root that shouldn't be, that messes up the operation of deployment scripts and leads to other headaches like incorrect file ownership.
You should try to use the deployment scripts to manipulate the servers, and if they're not adequate, try doing whatever needs doing as user 'opentree'. The only reasons I use admin or root are (1) checking the apache logs, (2) installing software using apt-get, (3) very rare system administration or troubleshooting tasks. The scripts all run as user opentree except for 'as-admin.sh', which only installs software. In general, running as much as possible under an unprivileged user helps make the system more secure.
Thanks
Jonathan