DIP upload from Archivematica to Atom fails

Andy Steven

unread,

Oct 30, 2018, 12:34:20 PM10/30/18

to AtoM Users

Hi.

CLEAN UBUNTU 16 INSTALLATION - installation followed online instructions - data in ATOM restored by mySQL database restore and ATOM works standalone in every other way as far as we can see.

Archivematica is also clean Ubuntu 16 installation and appears to work in every way apart from the DIP Upload "500 error" scenario posted below.

I know there has been a few similar posts on this topic however I have read and digested them all and have fully reviewed the installation - specifically relating to Gearman workers etc. I am still receiving the same issues and decided to seek some support from the user forum.

In addition a new system was created by a 3rd party following the same instructions and the issue continues on that platform too. The dataset used on both systems originates from a common database backup however this does not appears to be the issue and the file system permissions have also been reset too sudo chown -R www-data:www-data /usr/share/nginx/atom

We know RSYNC works fine and the DIP does appears in the atom.server:/tmp directory root - but not in the subdirectory as indicated in the error messages from Archivematica i.e. /tmp/tmp8cINrq

Content of atom.server /tmp - as you can see the DIP upload gets to this directory via Rsync

root@atom:/tmp# ls

Demo_2-062df233-3a20-4c93-bc03-34c9d2c5fa75 search_export_209330 testY.txt

Demo_3-631a8a69-3d8f-4e9a-aca4-68425bb7352a search_export_209331 testZ.txt

hsperfdata_elasticsearch systemd-private-14444d5d85504090ad184a3d45697b3f-systemd-timesyncd.service-YPCJAW

search_export_209329 testX.txt

root@atom:/tmp#

Here are the pile permissions on the Demo_2 and Demo_3 DIP Uploads

root@atom:/tmp# ls -al

total 68

drwxrwxrwt 14 root root 4096 Oct 30 16:17 .

drwxr-xr-x 23 root root 4096 Oct 20 12:38 ..

drwxrwxrwt 2 root root 4096 Oct 20 15:48 .ICE-unix

drwxrwxrwt 2 root root 4096 Oct 20 15:48 .Test-unix

drwxrwxrwt 2 root root 4096 Oct 20 15:48 .X11-unix

drwxrwxrwt 2 root root 4096 Oct 20 15:48 .XIM-unix

drwxrwxrwt 2 root root 4096 Oct 20 15:48 .font-unix

drwxr-xr-x 4 archivematica archivematica 4096 Oct 29 16:13 Demo_2-062df233-3a20-4c93-bc03-34c9d2c5fa75

drwxr-xr-x 4 archivematica archivematica 4096 Oct 30 14:21 Demo_3-631a8a69-3d8f-4e9a-aca4-68425bb7352a

drwxr-xr-x 2 elasticsearch elasticsearch 4096 Oct 20 15:48 hsperfdata_elasticsearch

drwxr-xr-x 2 www-data www-data 4096 Oct 23 13:50 search_export_209329

drwxr-xr-x 2 www-data www-data 4096 Oct 23 13:50 search_export_209330

drwxr-xr-x 2 www-data www-data 4096 Oct 25 08:38 search_export_209331

drwx------ 3 root root 4096 Oct 20 15:48 systemd-private-14444d5d85504090ad184a3d45697b3f-systemd-timesyncd.service-YPCJAW

-rw-r--r-- 1 archivematica archivematica 12 Oct 20 22:57 testX.txt

-rw-r--r-- 1 archivematica archivematica 12 Oct 29 14:31 testY.txt

-rw-r--r-- 1 archivematica archivematica 12 Oct 29 14:51 testZ.txt

root@atom:/tmp#

root@atom:/tmp# cd Demo_3-631a8a69-3d8f-4e9a-aca4-68425bb7352a/

root@atom:/tmp/Demo_3-631a8a69-3d8f-4e9a-aca4-68425bb7352a# ls -al

total 136

drwxr-xr-x 4 archivematica archivematica 4096 Oct 30 14:21 .

drwxrwxrwt 14 root root 4096 Oct 30 16:17 ..

-rw-r--r-- 1 archivematica archivematica 115368 Oct 30 14:21 METS.631a8a69-3d8f-4e9a-aca4-68425bb7352a.xml

drwxr-xr-x 2 archivematica archivematica 4096 Oct 30 14:20 objects

-rw-r--r-- 1 archivematica archivematica 991 Oct 30 14:21 processingMCP.xml

drwxr-xr-x 2 archivematica archivematica 4096 Oct 30 14:21 thumbnails

root@atom:/tmp/Demo_3-631a8a69-3d8f-4e9a-aca4-68425bb7352a#

Here is the NGINX errors

root@atom:/var/log/nginx# more error.log

2018/10/30 13:58:51 [notice] 29766#29766: signal process started

2018/10/30 14:14:00 [error] 29768#29768: *1875 FastCGI sent in stderr: "PHP message: No Gearman worker available that can handle the job qtSwordPluginWorke

r" while reading response header from upstream, client: 94.237.44.115, server: atom.shetlandamenity.org, request: "POST /index.php/sword/deposit/tp1 HTTP/1

.1", upstream: "fastcgi://unix:/run/php7.0-fpm.atom.sock:", host: "atom.shetlandamenity.org"

2018/10/30 15:52:21 [error] 29768#29768: *1923 FastCGI sent in stderr: "PHP message: No Gearman worker available that can handle the job qtSwordPluginWorke

r" while reading response header from upstream, client: 94.237.44.115, server: atom.shetlandamenity.org, request: "POST /index.php/sword/deposit/tp1 HTTP/1

.1", upstream: "fastcgi://unix:/run/php7.0-fpm.atom.sock:", host: "atom.shetlandamenity.org"

root@atom:/var/log/nginx#

NGINX ATOM permissions:

root@atom:/usr/share/nginx/atom# ls -al

total 206772

drwxr-x--- 15 www-data www-data 4096 Oct 23 13:50 .

drwxr-xr-x 5 root root 4096 Oct 20 15:12 ..

-rw-rw-r-- 1 www-data www-data 5304 Sep 4 2017 COPYRIGHT

-rw-rw-r-- 1 www-data www-data 34519 Sep 4 2017 LICENSE

-rw-rw-r-- 1 www-data www-data 336 Sep 4 2017 README

drwxr-xr-x 3 www-data www-data 4096 Oct 20 12:51 apps

drwxrwxrwx 3 www-data www-data 4096 Oct 20 17:20 cache

drwxr-xr-x 3 www-data www-data 4096 Oct 20 12:57 config

drwxr-xr-x 2 www-data www-data 4096 Oct 20 12:51 css

drwxr-xr-x 9 www-data www-data 4096 Oct 20 16:41 data

drwxr-xr-x 3 www-data www-data 4096 Oct 23 13:50 downloads

-rw-rw-r-- 1 www-data www-data 182068 Sep 4 2017 ead.dtd

-rw-rw-r-- 1 www-data www-data 1150 Sep 4 2017 favicon.ico

drwxr-xr-x 5 www-data www-data 4096 Oct 20 12:51 images

-rw-rw-r-- 1 www-data www-data 229 Sep 4 2017 index.php

drwxr-xr-x 2 www-data www-data 4096 Oct 20 12:51 js

drwxr-xr-x 17 www-data www-data 4096 Oct 20 12:51 lib

drwxr-xr-x 2 www-data www-data 4096 Oct 20 16:41 log

drwxr-xr-x 29 www-data www-data 4096 Oct 20 12:51 plugins

-rw-rw-r-- 1 www-data www-data 751 Sep 4 2017 qubit_dev.php

-rw-rw-r-- 1 www-data www-data 26 Sep 4 2017 robots.txt

lrwxrwxrwx 1 www-data www-data 48 Oct 20 12:57 sf -> /usr/share/nginx/atom/vendor/symfony/data/web/sf

-rwxrwxr-x 1 www-data www-data 445 Sep 4 2017 symfony

drwxr-xr-x 4 www-data www-data 4096 Feb 2 2018 uploads

-rw-r--r-- 1 www-data www-data 211412303 Oct 20 13:46 uploads.tar.gz

drwxr-xr-x 18 www-data www-data 4096 Oct 20 12:51 vendor

root@atom:/usr/share/nginx/atom#

Here is the ARCHIVEMATICA ERRORS

upload-qubit.py: ERROR 2018-10-30 15:52:20,267 archivematica.upload.qubit:log:64: [uploadDIP] Target: tp1

upload-qubit.py: ERROR 2018-10-30 15:52:20,267 archivematica.upload.qubit:log:64: [uploadDIP] rsync --protect-args -rltz -P --chmod=ugo=rwX /var/archivematica/sharedDirectory/watchedDirectories/uploadDIP/Demo_3-631a8a69-3d8f-4e9a-aca4-68425bb7352a archiv...@atom.server.org:/tmp

upload-qubit.py: ERROR 2018-10-30 15:52:20,269 archivematica.upload.qubit:log:64: [uploadDIP] Rsync output is being saved in /tmp/tmp8cINrq

upload-qubit.py: ERROR 2018-10-30 15:52:21,276 archivematica.upload.qubit:log:64: [uploadDIP] Sending... 100 (ETA: 0:00:00)

upload-qubit.py: ERROR 2018-10-30 15:52:21,278 archivematica.upload.qubit:log:64: [uploadDIP] About to deposit to: https://atom.server.org/index.php/sword/deposit/tp1

upload-qubit.py: DEBUG 2018-10-30 15:52:21,281 urllib3.connectionpool:_new_conn:824: Starting new HTTPS connection (1): atom.server.org

upload-qubit.py: DEBUG 2018-10-30 15:52:21,311 urllib3.connectionpool:_make_request:396: https://atom.server.org:443 "POST /index.php/sword/deposit/tp1 HTTP/1.1" 500 None

upload-qubit.py: ERROR 2018-10-30 15:52:21,313 archivematica.upload.qubit:log:64: [uploadDIP] > Response code: 500

upload-qubit.py: ERROR 2018-10-30 15:52:21,313 archivematica.upload.qubit:log:64: [uploadDIP] > Location: None

upload-qubit.py: ERROR 2018-10-30 15:52:21,313 archivematica.upload.qubit:log:64: [uploadDIP] > Content received: <!DOCTYPE html>

Gearman services restarted along with most of the ATOM services and Gearman worker appears to be configured correctly and showing ok from the service status command:

root@atom:/var/log/nginx# sudo systemctl status atom-worker

* atom-worker.service - AtoM worker

Loaded: loaded (/usr/lib/systemd/system/atom-worker.service; enabled; vendor preset: enabled)

Active: active (running) since Tue 2018-10-30 13:59:52 UTC; 2h 13min ago

Process: 29785 ExecStop=/bin/kill -s TERM $MAINPID (code=exited, status=0/SUCCESS)

Main PID: 29788 (php)

Tasks: 1

Memory: 15.0M

CPU: 1.529s

CGroup: /system.slice/atom-worker.service

`-29788 /usr/bin/php -d memory_limit=-1 -d error_reporting=E_ALL symfony jobs:worker

Oct 30 13:59:52 atom.server.org php[29788]: 2018-10-30 06:59:52 > New ability: arUpdatePublicationStatusJob

Oct 30 13:59:52 atom.server.org php[29788]: 2018-10-30 06:59:52 > New ability: arFileImportJob

Oct 30 13:59:52 atom.server.org php[29788]: 2018-10-30 06:59:52 > New ability: arInformationObjectXmlExportJob

Oct 30 13:59:52 atom.server.org php[29788]: 2018-10-30 06:59:52 > New ability: arXmlExportSingleFileJob

Oct 30 13:59:52 atom.server.org php[29788]: 2018-10-30 06:59:52 > New ability: arGenerateReportJob

Oct 30 13:59:52 atom.server.org php[29788]: 2018-10-30 06:59:52 > New ability: arActorCsvExportJob

Oct 30 13:59:52 atom.server.org php[29788]: 2018-10-30 06:59:52 > New ability: arActorXmlExportJob

Oct 30 13:59:52 atom.server.org php[29788]: 2018-10-30 06:59:52 > New ability: arRepositoryCsvExportJob

Oct 30 13:59:52 atom.server.org php[29788]: 2018-10-30 06:59:52 > Running worker...

Oct 30 13:59:52 atom.server.org php[29788]: 2018-10-30 06:59:52 > PID 29788

root@atom:/var/log/nginx#

Any support to get this working and more importantly to show what I did wrong during the installation process to create the problem in the first place.

I can provide SSH and web access via DM/email if this becomes necessary.

Best wishes

Andy

Dan Gillean

unread,

Oct 30, 2018, 1:01:20 PM10/30/18

to ica-ato...@googlegroups.com

Hi Andy,

Thanks for including so much detail in your post, it helps!

Unfortunately, SSHing into your system and providing individual support is a bit beyond the level of assistance we can provide via this free community forum. If you are interested in paid support options, please feel free to contact me off-list.

In the meantime, though it does sound like you've installed and configured everything correctly, did you try restarting the job scheduler and redoing your DIP upload to see if that resolves the issue?

sudo systemctl restart atom-worker

There is also a known issue where the job scheduler seems to put the atom-wokers to sleep after a long period of inactivity. It's possible you are encountering this. We have not tested this, but a community user in the forum has previously shared a configuration solution that you might want to investigate - see this thread:

https://groups.google.com/d/msg/ica-atom-users/X-07V3RUusw/mvHbLMg9BwAJ

I will ask someone from the Archivematica team to take a look at this thread as well, and see if they can offer any further suggestions.

Regards,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/090b3144-f402-4cda-8699-2d38ab6a2b2b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Andy Steven

unread,

Oct 30, 2018, 1:48:32 PM10/30/18

to AtoM Users

Thanks Dan,

What is the best way to contact you directly? Via the website?

I did restart the job scheduler with the command a few times however I also did it about 30 minutes before we attempted to upload the DIP so I am not sure what s long period of activity is relating to the known issue but I will check out the thread and check again.

Best wishes

Andy

Dan Gillean

unread,

Oct 30, 2018, 5:32:49 PM10/30/18

to ica-ato...@googlegroups.com

Hi Andy,

I can be reached at dan [at] artefactual [dot] com.

Let us know how retrying the upload goes.

Cheers,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/350c1b63-1595-4504-b987-6e2190ec8862%40googlegroups.com.

Dan Gillean

unread,

Nov 1, 2018, 1:43:45 PM11/1/18

to ica-ato...@googlegroups.com

Hi Andy,

I'm trying to get more input from our team. So far, I have a request for any related logs, and one suggestion for you.

In terms of logs, can you try the following, and share any relevant results that you find?

sudo journalctl -u atom-worker

Second, I had a suggestion from one of our team for you to doublecheck the atom-worker.service configuration. Specifically, make sure the workingDirectory is correct, and that the user/group are correct as well. You can compare it against this file:

https://github.com/artefactual/atom/blob/qa/2.5.x/init/atom-worker.service

Let us know what you find.

Cheers,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

jarrod...@gmail.com

unread,

Dec 13, 2018, 12:12:04 AM12/13/18

to AtoM Users

Hi Dan,

I'm getting this exact same error.

One possible relevant line from sudo journalctl -u:

Dec 13 03:29:20 atom php[1111]: 2018-12-12 19:29:20 > Ability not defined: qtSwordPluginWorker. Please ensure the job is in the lib/task/job directory or that the plugin is enabled.

The SWORD plugin is enabled, though!

- Jarrod

jarrod...@gmail.com

unread,

Dec 13, 2018, 1:30:32 AM12/13/18

to AtoM Users

Hi Dan/Andy,

I was able to resolve my issue, and in the spirit of being a good community member I thought I'd reveal my fix.

I wasn't the first person to discover this fix, it was the user Olav in this forum two years ago:

https://groups.google.com/forum/#!searchin/ica-atom-users/sword$20bad$20request%7Csort:date/ica-atom-users/3Z1pEW3z_pM/m8V3VWRnDQAJ

What Olav did first was create an uploads directory. I don't know why we didn't get one, Andy - maybe we missed something somewhere?

sudo mkdir /usr/share/nginx/atom/uploads

Olav then set permissions on the new uploads directory:

sudo chown -R www-data:www-data /usr/share/nginx/atom/uploads

Finally, Olav enabled the worker using symfony instead of SystemD:

sudo php symfony jobs:worker

At that point I pushed my objects through archivematica and got the following outputs:

2018-12-12 21:53:13 > Job started.

2018-12-12 21:53:14 > A package was deposited by reference.

2018-12-12 21:53:14 > Location: file:///benjamin_mcjobsWorker-9af3cf58-7e97-4150-8e51-738a5cb28cda

2018-12-12 21:53:14 > Processing...

2018-12-12 21:53:14 > Object slug: gadsden-description

identify: not authorized `/usr/share/nginx/atom/uploads/r/null/1/2/5/12580ec9e830ec3d3b16145e326d4751d3f9d2846021139385844ffd1f44bd5a/d08f8c6d-0c5a-439a-8ec0-a6bdd58e72e2-SN_Feb_2001.pdf' @ error/constitute.c/ReadImage/412.

2018-12-12 21:53:17 > Job finished.

The job completed successfully and the objects are in AtoM. The PDF isn't rendering in AtoM, however from the logs this looks like it's an imagemagick issue and beyond the scope of this thread.

- Jarrod

Dan Gillean

unread,

Dec 13, 2018, 12:17:23 PM12/13/18

to ICA-AtoM Users

Hi Jarrod,

I love entering into forum threads where users have managed to resolve their issues by searching the forum before I even arrive!

Thanks so much for updating the thread and sharing exactly how you resolved the issue.

I think I know what's happening here, but perhaps you can help me confirm my hunch. Have you previously uploaded digital objects to AtoM via the user interface, or were you trying to add your first digital objects to AtoM via an Archivematica DIP upload?

You didn't miss a step in creating the uploads directory. Normally, in a fresh install the uploads directory does not exist- instead, it is automatically created the first time a user uploads a digital object to AtoM.

We had previously discovered that this wasn't working for DIP uploads, and we filed a ticket, here:

https://projects.artefactual.com/issues/9799

We had thought the issue resolved, but based on your response, I'm wondering if the manner in which this was tested didn't actually properly confirm the fix - i.e. this may still be an issue. Once I hear back from you, I can look into this further and re-open the ticket if needed.

Regarding the PDF derivatives issue: it's possible you are running into this issue? See the following thread for an outline of the issue and a solution:

https://groups.google.com/d/msg/ica-atom-users/AwNUifxWsS4/BvfV6x9uBQAJ

Once you make the necessary ImageMagick policy change, you could try recreating the derivatives with this command-line task:

https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/cli-tools/#regenerating-derivatives

Let us know how it goes!

Regards,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/77eecb5b-1c1d-45b5-aff2-9a2fef525b4c%40googlegroups.com.

jarrod...@gmail.com

unread,

Dec 13, 2018, 8:42:53 PM12/13/18

to AtoM Users

Hi Dan,

I hadn't added anything previously through AtoM! This explains what was happening, then.

Thank you for your advice on the ImageMagick policy change, it worked well.

Regards,

Jarrod

Reply all

Reply to author

Forward