Uploads from Archivematica to Atom fail

[Harald Hjønnevåg]

Hi

I've set up Archivematica 1.5.0 and Atom 2.3.1 138 on two new servers running Ubuntu 14.0.4. Even though I have tried to follow the installation instructions to the letter, I've been failing for days trying to upload DIPS from Archivematica to Atom.

I've searched through all the documentation and the web without finding any solution to my problem. As far as I can tell from the log, the error must have something to do with  file system permissions. I've checked the permissions on /usr/share/nginx/atom www-data:www:data 755 and /tmp 1777.

I've also verified that the user 'archivematica' is able to login to the atom server from the archivematica server with ssh keys and write to /tmp.

Surely it's just a tiny detail I've overlooked in order to get the uploads working, but as it stands I'm stuck.I would appreciate if somebody could point me in the right direction. Please have a look at the output from my atom-worker log file:

2017-02-17 00:50:06 > Job 478 "qtSwordPluginWorker": A new job has started to being processed.
2017-02-17 00:50:06 > Job 478 "qtSwordPluginWorker": A package was deposited by reference.
2017-02-17 00:50:06 > Job 478 "qtSwordPluginWorker": Location: file:///A-eae28b37-4002-46c0-8842-b5ca8fd52525
2017-02-17 00:50:06 > Job 478 "qtSwordPluginWorker": Processing...
2017-02-17 00:50:06 > Job 478 "qtSwordPluginWorker": Object slug: a
2017-02-17 00:50:06 > Job 478 "qtSwordPluginWorker": Exception: File/directory does not exist or is not readable: /tmp/A-eae28b37-4002-46c0-8842-b5ca8fd52525.
2017-02-17 00:50:06 > Job failed: File/directory does not exist or is not readable: /tmp/A-eae28b37-4002-46c0-8842-b5ca8fd52525.

Many thanks in advance!

Best regards
Harald

[Dan Gillean]

Hi Harald,

Thanks for checking the forum for previous answers first, and letting us know what you tried. For anyone else reading this thread and getting started with configuring AtoM and Archivematica together, these posts include the basic things to check first, and where to find the relevant documentation:

• https://groups.google.com/d/msg/ica-atom-users/shM3sPuvSMA/-3mWFX94CQAJ
• https://groups.google.com/d/msg/ica-atom-users/1GsLemVvCe8/0obTdhPQDgAJ

Harald, something I noticed in the output of your atom-worker file and Gessy's, who previously had a similar issue, is that in the 3rd line, it's reporting that the location of the file is file:///A-eae28b37-4002-46c0-8842-b5ca8fd52525, but then the task proceeds to look for the file at /tmp/A-eae28b37-4002-46c0-8842-b5ca8fd52525. I wonder if this might be an indication of what's going on?

Just to confirm, here's how the Admin settings in Archivematica should be configured for upload. If your AtoM site is located at http://www.example.com, then this is exactly what should go in the Upload URL part of the page. For the rsync target value, you would then enter: www.example.com:/tmp. I'm guessing you've got this configured already, but I just wanted to double-check :)

I'm going to ask one of our Archivematica team members to take a look at this post and see if they have any ideas. Hopefully we can help you figure this out!

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/e14caa2f-9460-499a-922b-e9f2a603867e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Harald Hjønnevåg]

Hi Dan

Thank you for replying to my request.

In our Archivematica admin settings I had entered "/tmp" as the "rsync target". Changing this to "http://atom.arbark.no:/tmp", as you suggest, did not solve the problem. After the edit nothing happens on the Atom side. I also tried with "atom.arbark.no:/tmp" with the same result:

[uploadDIP] Rsync quit unexpectedly (exit 12), the upload script will be stopped here

Last week, before I posted my message, I visited another institution with a working system, and had a look at their setup. Like us,they also had simply "/tmp" as "rsync target". Unfortunately I wasn't given access to look any further under the hood, in order to replicate more of  their setup.

This is the command that's run which gets in contact with the atom server, and reports the upload as successful:

upload-qubit_v0.0 --url="http://atom.arbark.no" --email="x...@xxx.xx" --password="xxxxxxxx" --rsync-target="http://atom.arbark.no:/tmp" --rsync-command="ssh -i /var/lib/archivematica/.ssh/id_rsa archiv...@atom.arbark.no" --version="2" --debug --uuid="2c1934e8-d1c0-4e10-8a39-90ef2e5e3cf5"

Attached is a screen dump of our Archivematica settings.I would be most grateful for any further assistance in solving this issue.

Many thanks in advance!

best regards

Harald

To post to this group, send email to ica-ato...@googlegroups.com.

Visit this group at https://groups.google.com/group/ica-atom-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/e14caa2f-9460-499a-922b-e9f2a603867e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZLsEO51LZLgx2YEMoFY1X652acs0gPjrctXQaDZ4JwJzw%40mail.gmail.com.

[Dan Gillean]

Hi Harald,

I tried searching in the Archivematica user forum to see if I could find resolved threads related to this issue, and unfortunately didn't find much that you haven't already checked, from the sounds of it. One suggestion here, about ensuring that rysnc is enabled if you are using rssh:

• https://groups.google.com/d/msg/archivematica/xujvoVkvhO4/-2yYbnY12WcJ

In the meantime, I'm still trying to get an Archivematica team member to take a look at this thread for you.

Cheers,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZLsEO51LZLgx2YEMoFY1X652acs0gPjrctXQaDZ4JwJzw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CANCWY_f8M9nB1Os3g-%2BwmOAE67i9AA_RTQgTf5SWF9mCpbv8Yw%40mail.gmail.com.

[Harald Hjønnevåg]

Hi Dan

I have verified that “allowrsync” is uncommented in /etc/rssh.conf on the atom server. But I think you´re definitely pointing me in the right direction as nothing appears in the /tmp directory on the atom server, although archivematica reports the upload as successful.

As it’s getting late here, I don´t have time to look any further into the issue at the moment, but I will return to it tomorrow. I´m sure it´s just a silly detail I´ve overlooked. 

Have a nice day, and thanks again for your support!

Best regards

Harald

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.

To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZKmNuL7mXsyj%3DkN55_JjrfftHgGhggHJdAQK2NUEhcHgA%40mail.gmail.com.

[David at Artefactual]

Hi Harald,

It might be helpful if I clarify how the DIP upload is working behind the scenes here.  When doing DIP upload Archivematica attempts to copy the DIP from the local file system to the "rsync target".  

If the AtoM installation is on the same server as the Archivematica installation, then a local filepath f (e.g. rsync target="/tmp") is sufficient, as both Archivematica and AtoM can access the local "/tmp" directory. 

If the AtoM installation is on a *different* server than Archivematica, then a remote file path is required (e.g. rsync target="archiv...@192.168.1.10:/tmp").  An important point to note here is that when copying the DIP to a remote server the SSH protocol is used behind the scenes to securely connect from the source server (Archivematica) to the target server (AtoM) In  and the rsync target must be of the format "{username}@{hostname}:{target directory}".  In your case your attempt to use "archivematica @atom.arbark.no:/tmp" is probably the rsync target you want.

For the upload to work between two different servers several pieces may be in place:

1. You need to have rsync installed on your Archivematica server
2. Your Archivematica server must be able to translate the "hostname" into an IP. You can test this by opening a remote shell in your Archivematica server and typing: 
   host atom.arbark.no
   If you get an error like the one below, then you will need to either use the AtoM server IP address or find a hostname that resolves.
   Host atom.arbark.no not found: 3(NXDOMAIN)
3. You will need to configure your AtoM server to allow SSH connections from your Archivematica server.  You can test this is working by opening a remote shell in to your Archivematica server, and typing 
   sudo -u archivematica ssh archivematica@atom.arbark.no
   
   If you get prompted for a password, or the connection fails for any reason then DIP upload will not work.
4. You need to ensure that the archivematica user on the AtoM server has write access to the target directory, or you will get a permission denied error
5. You need to ensure that the AtoM worker is installed and running on the AtoM 
   

If you are having problems with any of the above, write back to the list and we will try and help. :)

Best regards,
David

To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/e14caa2f-9460-499a-922b-e9f2a603867e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.

To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZLsEO51LZLgx2YEMoFY1X652acs0gPjrctXQaDZ4JwJzw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Harald Hjønnevåg]

Hi again Dan

Your last reply is most clarifying. I believe the rsync target is at the root of my problem. I have now checked with archivematica@atom.arbark.no:/tmp as rsync target, but still no luck.

I have verified that I can login to the Atom server from the Archivematica server with the command "sudo -u archivematica ssh archivematica@atom.arbark.no".

Anyway – hopefully now I have all the information I need in order to get the upload working. I´ll let you know tomorrow how it works out.

Thank again for taking your time helping me out.

Best regards

Harald

On 20 Feb 2017, at 21:35, David at Artefactual <da...@artefactual.com> wrote:

archivematica@atom.arbark.no

[Dan Gillean]

Let us know how it goes, Harald,

One last test, to make sure that you have the proper permissions configured - can you SSH from your Archivematica server and create a file in the /tmp location? From the command-line of the Archivematica server, try something like the following, with your proper credentials and locations entered:

• AM_srvr% sudo -u archivematica ssh archivematica@atom.arbark.no "touch /tmp/preDipTestFile"

If that fails, then something about your permissions or the access key required is not properly configured.

Good luck!

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/82AC10D2-C8C5-4C93-B794-F8D3313E12EB%40gmail.com.

[Harald Hjønnevåg]

Hi again.

Wrong permissions was my first suspicion. So yes, I have verified that I’m able to write to the /tmp directory on the atom server via ssh from the archivematica server, but thanks for the tip. I keep my fingers crossed, and hope I’ll get it sorted out tomorrow.

Best regards

Harald

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.

Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZJ7vSmXNGOxORP-oEb8e2oupHdi9wbs3QSAB_rADW8M0g%40mail.gmail.com.

[Harald Hjønnevåg]

Hi Dan

I’ve been through the checklist you sent me yesterday, and after some rounds with trial and error I was finally able to upload. The source of the problem was the "sync target" as you suspected. 

There must have been some misunderstanding regarding how many servers the institution I visited last week is running the applications on, as they have made it work with just “/tmp/“ as the rsync target. I was told that they had a consultant in to get it working. He probably has probably dumped one of their original servers and installed both Archivematica and Atom on the same machine.

Your suggestions have been invaluable in solving the issue. Thank you very much for your help and patience!

Best regards Harald

On 20 Feb 2017, at 22:05, Dan Gillean <d...@artefactual.com> wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To post to this group, send email to ica-ato...@googlegroups.com.

Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZJ7vSmXNGOxORP-oEb8e2oupHdi9wbs3QSAB_rADW8M0g%40mail.gmail.com.

[Dan Gillean]

Glad to hear you have it working, Harald - thanks for updating us!

Cheers,

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

To post to this group, send email to ica-ato...@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/82AC10D2-C8C5-4C93-B794-F8D3313E12EB%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/CAC1FhZJ7vSmXNGOxORP-oEb8e2oupHdi9wbs3QSAB_rADW8M0g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-users+unsubscribe@googlegroups.com.
To post to this group, send email to ica-atom-users@googlegroups.com.
Visit this group at https://groups.google.com/group/ica-atom-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/4E6C8BEA-BC91-4FAD-B08C-C475A7EF097A%40gmail.com.