Atom worker issue

[brzic...@gmail.com]

I have the following test setup:

Archivematica 1.8.1 running on Ubuntu 18.04

AtoM 2.2.1 with a fully working copy of our production box running on Ubuntu 14.04

qtSwordPlugin 1.0.0 and arRestApiPlugin 1.0.0 enabled in AtoM

gearman-job-server installed and running on AtoM

upstart service for the AtoM worker configured in AtoM

When I run initctl list on AtoM I get:

.............

gearman-job-server start/running, process 665

atom-worker stop/waiting

..............

When I upload DIP to AtoM I get a 500 error. The files DO get written to the destination directory but not processed into AtoM. The Access tab in Archivematica keeps saying "...sending..."

However, when I explicitly start the worker using sudo php symfony jobs:worker I get no errors, the files also get written to the destination directory AND processed into AtoM. The Access tab in Archivematica says that files have been uploaded and are being processed asynchronously.

No Nginx errors or atom-worker log errors.

What could be wrong with my worker setup ?

Thanks

[Karl Goetz]

Hi,

AtoM doesn’t cope with its workers being unavailable so the 500 error is almost to be expected if you are having worker problems (Yes its a terrible failure mode).

Is your worker run via upstart job or init script?

Karl

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/d42ad481-1e6f-4711-a020-9d80f65ec1bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 

Karl Goetz,  Senior Library Officer (Library Systems)

University of Tasmania, Private Bag 25, Hobart 7001

Available Tuesday, Wednesday, Thursday

University of Tasmania Electronic Communications Policy (December, 2014).
This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.

[brzic...@gmail.com]

Hi Karl,

I configured the worker via upstart job as per the documentation.

[brzic...@gmail.com]

I have attached my atom-worker.conf file from/etc/init

[Dan Gillean]

Hi there, 

As a first recommendation: if you are able, I would strongly recommend that you consider upgrading AtoM to the latest version (2.4.1) - you may be encountering compatibility issues between your Archivematica and AtoM versions that are further complicating the DIP upload process. 

I will ask our system administrators to take a look and see if they can offer any suggestions on why the job scheduler is not working. In the meantime, I would like to gather more information about your AtoM installation that will help our team provide further suggestions. The following commands should be run from your root AtoM installation directory (e.g. /usr/share/nginx/atom if you have followed our recommended instructions. And if you have made changes from those instructions, please let us know!).  Please share the output of the folllowing: 

First, let's get the full AtoM version, including the database schema version: 

• php symfony tools:get-version
  

Let's check the PHP version: 

• php -v

Now let's check on the PHP extensions you have installed, and make sure there isn't something missing: 

• php -m

Checking the user accounts active on your system: 

• getent passwd

Finally, let's check the permissions on your root AtoM directory: 

• ls -l

And the permission of the atom-worker.conf file: 

• ls -l /etc/init/atom-worker.conf

Hopefully this information will help our team spot any obvious inconsistencies, and provide further suggestions. 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

On Thu, Feb 28, 2019 at 12:44 PM <brzic...@gmail.com> wrote:

I have attached my atom-worker.conf file from/etc/init

--

You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/b908830f-fd99-4152-bd86-7d1adcd32d6e%40googlegroups.com.

[brzic...@gmail.com]

Here you go...

innopac@ATOM221:~$ php /usr/share/nginx/atom/symfony tools:get-version
2.2.1 v123
**************************************************************************************************
    php -m
[PHP Modules]
apc
apcu
bcmath
bz2
calendar
Core
ctype
curl
date
dba
dom
ereg
exif
fileinfo
filter
ftp
gettext
hash
iconv
json
ldap
libxml
mbstring
mhash
mysql
mysqli
openssl
pcntl
pcre
PDO
pdo_mysql
Phar
posix
readline
Reflection
session
shmop
SimpleXML
soap
sockets
SPL
standard
sysvmsg
sysvsem
sysvshm
tokenizer
wddx
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache
**************************************************************************************************
    getent passwd
innopac@ATOM221:~$ getent passwd
root:x:0:0:root:/root:/bin/bash
daemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologin
bin:x:2:2:bin:/bin:/usr/sbin/nologin
sys:x:3:3:sys:/dev:/usr/sbin/nologin
sync:x:4:65534:sync:/bin:/bin/sync
games:x:5:60:games:/usr/games:/usr/sbin/nologin
man:x:6:12:man:/var/cache/man:/usr/sbin/nologin
lp:x:7:7:lp:/var/spool/lpd:/usr/sbin/nologin
mail:x:8:8:mail:/var/mail:/usr/sbin/nologin
news:x:9:9:news:/var/spool/news:/usr/sbin/nologin
uucp:x:10:10:uucp:/var/spool/uucp:/usr/sbin/nologin
proxy:x:13:13:proxy:/bin:/usr/sbin/nologin
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
backup:x:34:34:backup:/var/backups:/usr/sbin/nologin
list:x:38:38:Mailing List Manager:/var/list:/usr/sbin/nologin
irc:x:39:39:ircd:/var/run/ircd:/usr/sbin/nologin
gnats:x:41:41:Gnats Bug-Reporting System (admin):/var/lib/gnats:/usr/sbin/nologin
nobody:x:65534:65534:nobody:/nonexistent:/usr/sbin/nologin
libuuid:x:100:101::/var/lib/libuuid:
syslog:x:101:104::/home/syslog:/bin/false
messagebus:x:102:106::/var/run/dbus:/bin/false
landscape:x:103:109::/var/lib/landscape:/bin/false
sshd:x:104:65534::/var/run/sshd:/usr/sbin/nologin
innopac:x:1000:1000:innopac,,,:/home/innopac:/bin/bash
xrdp:x:105:113::/var/run/xrdp:/bin/false
usbmux:x:106:46:usbmux daemon,,,:/home/usbmux:/bin/false
lightdm:x:107:116:Light Display Manager:/var/lib/lightdm:/bin/false
dnsmasq:x:108:65534:dnsmasq,,,:/var/lib/misc:/bin/false
ntp:x:109:118::/home/ntp:/bin/false
avahi-autoipd:x:110:119:Avahi autoip daemon,,,:/var/lib/avahi-autoipd:/bin/false
colord:x:111:121:colord colour management daemon,,,:/var/lib/colord:/bin/false
pulse:x:112:122:PulseAudio daemon,,,:/var/run/pulse:/bin/false
kernoops:x:113:65534:Kernel Oops Tracking Daemon,,,:/:/bin/false
rtkit:x:114:124:RealtimeKit,,,:/proc:/bin/false
speech-dispatcher:x:115:29:Speech Dispatcher,,,:/var/run/speech-dispatcher:/bin/sh
whoopsie:x:116:125::/nonexistent:/bin/false
hplip:x:117:7:HPLIP system user,,,:/var/run/hplip:/bin/false
avahi:x:118:126:Avahi mDNS daemon,,,:/var/run/avahi-daemon:/bin/false
saned:x:119:127::/home/saned:/bin/false
mysql:x:120:128:MySQL Server,,,:/nonexistent:/bin/false
elasticsearch:x:121:129::/home/elasticsearch:/bin/false
gearman:x:122:130:Gearman Job Server,,,:/var/lib/gearman:/bin/false
memcache:x:123:131:Memcached,,,:/nonexistent:/bin/false
archivematica:x:1001:1001:archivematica,,,,:/home/archivematica:/bin/bash

**************************************************************************************************
    ls -l
innopac@ATOM221:/usr/share/nginx/atom$ ls -l
total 300
drwxr-xr-x  3 www-data www-data   4096 Jan 24 16:10 apps
drwxrwxrwx  3 www-data www-data   4096 Feb 26 13:56 cache
drwxr-xr-x  3 www-data www-data   4096 Feb 11 10:08 config
-rw-rw-r--  1 www-data www-data    490 Feb  3  2016 CONTRIBUTING
-rw-rw-r--  1 www-data www-data   5199 Feb  3  2016 COPYRIGHT
drwxr-xr-x  2 www-data www-data   4096 Jan 24 16:10 css
drwxr-xr-x  9 www-data www-data   4096 Feb 26 13:47 data
-rw-rw-r--  1 www-data www-data 182068 Feb  3  2016 ead.dtd
-rw-rw-r--  1 www-data www-data   1150 Feb  3  2016 favicon.ico
drwxr-xr-x  5 www-data www-data   4096 Jan 25 09:28 images
-rw-rw-r--  1 www-data www-data    229 Feb  3  2016 index.php
drwxr-xr-x  2 www-data www-data   4096 Jan 24 16:10 js
drwxr-xr-x 16 www-data www-data   4096 Jan 24 16:10 lib
-rw-rw-r--  1 www-data www-data  34519 Feb  3  2016 LICENSE
drwxr-xr-x  2 www-data www-data   4096 Feb 11 13:04 log
drwxr-xr-x 30 www-data www-data   4096 Jan 24 16:10 plugins
-rw-rw-r--  1 www-data www-data    751 Feb  3  2016 qubit_dev.php
-rw-rw-r--  1 www-data www-data    339 Feb  3  2016 README
-rw-rw-r--  1 www-data www-data     26 Feb  3  2016 robots.txt
lrwxrwxrwx  1 www-data www-data     48 Jan 24 16:13 sf -> /usr/share/nginx/atom/vendor/symfony/data/web/sf
-rwxrwxr-x  1 www-data www-data    445 Feb  3  2016 symfony
drwxr-xr-x  4 www-data www-data   4096 Dec  5  2015 uploads
drwxr-xr-x 14 www-data www-data   4096 Jan 24 16:10 vendor

**************************************************************************************************
innopac@ATOM221:/usr/share/nginx/atom$ ls -l /etc/init/atom-worker.conf
-rw-rw-r-- 1 root root 420 Feb 28 10:56 /etc/init/atom-worker.conf

**************************************************************************************************

Thanks !

[brzic...@gmail.com]

Missed this one :

innopac@ATOM221:~$ php -v
PHP 5.5.9-1ubuntu4.26 (cli) (built: Sep 17 2018 13:46:30)
Copyright (c) 1997-2014 The PHP Group
Zend Engine v2.5.0, Copyright (c) 1998-2014 Zend Technologies
    with Zend OPcache v7.0.3, Copyright (c) 1999-2014, by Zend Technologies

[brzic...@gmail.com]

Syslog has this part :

Feb 28 09:51:01 ATOM221 kernel: [    3.477132] init: atom-worker main process (224) terminated with status 127
Feb 28 09:51:01 ATOM221 kernel: [    3.478202] init: atom-worker main process ended, respawning
Feb 28 09:51:01 ATOM221 kernel: [    3.482197] init: atom-worker main process (239) terminated with status 127
Feb 28 09:51:01 ATOM221 kernel: [    3.483271] init: atom-worker main process ended, respawning
Feb 28 09:51:01 ATOM221 kernel: [    3.487085] init: atom-worker main process (241) terminated with status 127
Feb 28 09:51:01 ATOM221 kernel: [    3.488174] init: atom-worker main process ended, respawning
Feb 28 09:51:01 ATOM221 kernel: [    3.491243] init: atom-worker main process (244) terminated with status 127
Feb 28 09:51:01 ATOM221 kernel: [    3.492332] init: atom-worker main process ended, respawning
Feb 28 09:51:01 ATOM221 kernel: [    3.495257] init: atom-worker main process (246) terminated with status 127
Feb 28 09:51:01 ATOM221 kernel: [    3.496344] init: atom-worker main process ended, respawning
Feb 28 09:51:01 ATOM221 kernel: [    3.499291] init: atom-worker main process (248) terminated with status 127

Now, from what I have read status 127 usually means path not found - therefore it may be a permissions issue...

[David Hume]

Hi there,

Per that error, indeed I think you may be on to something with the permissions issue

Does /usr/share/nginx/atom/log/atom-worker.log have content? If you're comfortable doing so, you could attach that, or provide output of `tail -30 /usr/share/nginx/atom/log/atom-worker.log` say (at least, it should elaborate on the error).

Also I'd be interested in the basic output of good old 'ps' vis-a-vis

ps -elf | egrep gearma[n] ; ps -elf | egrep job[s]

Thanks,

    David Hume, Artefactual Systems Administrator

[brzic...@gmail.com]

atom-worker log is empty

-rw-rw-rw- 1 www-data www-data 0 Feb 11 10:47 atom-worker.log
-rw-rw-rw- 1 root root 0 Feb 11 10:47 qubit_cli.log
-rw-rw-rw- 1 www-data www-data 0 Feb 11 10:14 qubit_prod.log

innopac@ATOM221:/usr/share/nginx/atom/log$ ps -elf | egrep gearman
4 S gearman    672     1  0  80   0 - 120386 -     10:33 ?        00:00:00 /usr/sbin/gearmand --log-file=/var/log/gearman-job-server/gearman.log
0 S innopac   5067  5016  0  80   0 -  2365 pipe_w 14:39 pts/0    00:00:00 egrep --color=auto gearman

innopac@ATOM221:/usr/share/nginx/atom/log$ ps -elf | egrep job
4 S gearman    672     1  0  80   0 - 120386 -     10:33 ?        00:00:00 /usr/sbin/gearmand --log-file=/var/log/gearman-job-server/gearman.log
0 S innopac   5074  5016  0  80   0 -  2365 pipe_w 14:40 pts/0    00:00:00 egrep --color=auto job

Thanks !

[brzic...@gmail.com]

/var/log/gearman-job-server/gearman.log is also empty..

Thanks !

[David Hume]

Hmmm - maybe try the following

$ cd /usr/share/nginx/atom

make sure that is indeed the directory you are in, then

$ sudo chown www-data.www-data ./

when that is done, maybe these two commands - in theory there _shouldn't_ be any jobs yet if you haven't got the atom-worker running, but I'm curious

$ sudo -i -u www-data php symfony jobs:list

$ sudo -i -u www-data php symfony jobs:clear

Then

$ sudo restart gearman-job-server

wait 30 seconds, maybe confirm has started anew with 'ps' then

$ sudo restart atom-worker

$ sudo php symfony cc

Then I guess do the ps / logs content check I suggested above, see if any difference, see if you're still getting "respawing" messages in syslog

Definitely interested in any output, error or otherwise, you see along the way.

Thanks,

    Dave H., Artefactual Systems

[brzic...@gmail.com]

$ sudo -i -u www-data php symfony jobs:list

$ sudo -i -u www-data php symfony jobs:clear

When I run these two commands I get

This account is currently not available

[David Hume]

Yes, realized afterwards if www-data has "nologin" shell you might get that result - sorry about that

Simple

$ sudo php symfony jobs:list

and assuming nothing listed there you're concerned about keeping, then

$ sudo php symfony jobs:clear

from /usr/share/nginx/atom directory should do the trick.

      Dave H.

[brzic...@gmail.com]

Thanks Dave. Will try that too.

But I found another way to make it work.

Edited atom-worker.conf

description "AtoM worker (gearmand) upstart service"
 
start on (started mysql)
stop on shutdown
 
respawn
respawn limit 5 10
 
setuid root
setgid root
 
exec bash /usr/bin/worker.sh

Created bash script with these two lines :

#!bin/bash
php /usr/share/nginx/atom/symfony jobs:worker

Made sure file was executable

Bingo ! It now works !

If anybody has any comments or concerns with this resolution please let me know....

Thanks

[David Hume]

Glad you got it working!

Off the top of my head, you _may_ run into issues later if you want to modify / re-generate / delete any files associated with the atom worker (root file ownership conflicting with www-data ownership e.g.), and I'd be curious if the worker restarts okay if you do a clean reboot of the system, but if otherwise your AtoM seems fully functional now... (does that include writing to the logs?)

Cheers,

     Dave H.

[Karl Goetz]

Hi,

Glad to see you got it working (sorry I dissapeare half way through).

The bash script below replaces the following lines in your upstart configuration

  php \

    -d memory_limit=-1 \

    -d error_reporting="E_ALL" \

      ${LOCATION}/symfony jobs:worker >> ${LOGFILE} 2>&1

Would you be willing to re-add the removed lines one by one and see if things break?

Specifically, update

php /usr/share/nginx/atom/symfony jobs:worker

To

php -d memory_limit=-1 /usr/share/nginx/atom/symfony jobs:worker

Then

php -d memory_limit=-1 -d error_reporting="E_ALL"  /usr/share/nginx/atom/symfony jobs:worker

then

php -d memory_limit=-1 -d error_reporting="E_ALL"  /usr/share/nginx/atom/symfony jobs:worker >> ${LOGFILE} 2>&1

karl.

On 5 Mar 2019, at 9:23 am, brzic...@gmail.com wrote:

Edited atom-worker.conf

description "AtoM worker (gearmand) upstart service" 
 
start on (started mysql) 
stop on shutdown 
 
respawn 
respawn limit 5 10 
 
setuid root 
setgid root 
 
exec bash /usr/bin/worker.sh

Created bash script with these two lines :

#!bin/bash
php /usr/share/nginx/atom/symfony jobs:worker

-- 

Karl Goetz

Mon, Tue, Wed, Technical Services Officer - eResearch

Wed, Thu, Fri Senior Library Officer (Library Systems)

University of Tasmania, Private Bag 25, Hobart 7001

[brzic...@gmail.com]

Hi Karl,

made the following changes:

-added the full command line to the bash script

-set the log file path variable

-set user back to www-data (from root)

atom-worker.conf looks like this now :

description "AtoM worker (gearmand) upstart service"
start on (started mysql)
stop on shutdown
respawn
respawn limit 5 10

env LOGFILE=/usr/share/nginx/atom/log/atom-worker.log
setuid www-data
setgid www-data
exec bash /usr/bin/worker.sh

bash script looks like this now :

#!bin/bash

php -d memory_limit=-1 -d error_reporting="E_ALL"  /usr/share/nginx/atom/symfony jobs:worker >> ${LOGFILE} 2>&1

Everything works fine now. And for the first time I am getting the log file being populated.

I think I will leave it as is...

Thanks for the help !