Content not showing up in AToM-server

[Paul Bergström]

When i upload DIP to AToM there is no obvious error. It all appears correct, but nothing shows up in the target description.

The log looks like this:

[uploadDIP] Target: daniel
[uploadDIP] rsync -rltz -P --chmod=ugo=rwX /var/archivematica/sharedDirectory/watchedDirectories/uploadDIP/wwww-da3eebeb-8823-41e2-934c-011903c4e74d /tmp
[uploadDIP] Rsync output is being saved in /tmp/tmpupgF8q
[uploadDIP] About to deposit to: http://localhost/atom/index.php/;sword/deposit/daniel
[uploadDIP] > Response code: 302
[uploadDIP] > Location: /atom/index.php/daniel;isad
[uploadDIP] > Content received: <?xml version="1.0" encoding="utf-8" ?><entry xmlns="http://www.w3.org/2005/Atom"
       xmlns:sword="http://purl.org/net/sword/">

  <title>daniel</title>

  
  <id>327 / 2013-12-01T10:22:56</id>

  <updated>2013-12-01T10:22:56</updated>
  <author>
    <name>arkivarie</name>
  </author>

  <generator uri="http://localhost/atom/index.php/" version="1.4.0">Qubit 1.4.0</generator>

  <content type="text/html" src="http://localhost/atom/index.php/daniel;isad" />

  <sword:noOp>false</sword:noOp>

  <sword:packaging>http://purl.org/net/sword-types/METSArchivematicaDIP</sword:packaging>

  <sword:userAgent>Archivematica</sword:userAgent>

  
</entry>

[uploadDIP] Deposited asynchronously, Qubit is processing the DIP in the job queue


Any ideas what might be wrong?

[Courtney Mumma]

Hi Paul,

DIP upload to AtoM is verified as working with AtoM 1.3.1, but it looks like you're uploading to 1.4.0. We have a bug filed (https://projects.artefactual.com/issues/6063) for DIP upload to AtoM 2.0 which we are currently working on. Please follow the progress of our work, as that is likely the same problem,

Thanks,
Courtney

--
Courtney C. Mumma, MAS/MLIS
Archivematica Product Manager/Systems Analyst 
Artefactual Systems, Inc.
604-527-2056

--
You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To post to this group, send email to archiv...@googlegroups.com.
Visit this group at http://groups.google.com/group/archivematica.
For more options, visit https://groups.google.com/groups/opt_out.

[Paul Bergström]

Thank you Courtney!

Well, I tried the AtoM 1.3.1, but the result was the same. The sample objects don't show up in the server. Maybe it is a sword-problem?

Another thing. Is there any wiki for translating Archivematica? I would like to do my share by submitting translations to some scandinavian languages.

Thanks, again:)

/Paul

[Courtney Mumma]

Ok, Paul, thank you for the update. Have you already created a target description in AtoM? And have you configured AtoM for DIP upload?

As far as translation, we are interested in translation but are looking into a big change in our documentation over the next several months, so we won't likely be ready for that until late 2014. Thanks for offering, though. We hope you're still up for it next year.

Cheers,
Courtney

--

You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To post to this group, send email to archiv...@googlegroups.com.
Visit this group at http://groups.google.com/group/archivematica.
For more options, visit https://groups.google.com/groups/opt_out.

[Paul Bergström]

Courtney,

Yes, I created a target description in AtoM. However since the AtoM is running on the same machine as Archivematica, the steps described in "configure AtoM for DIP upload", shouldn't they be obsolete? Both servers are on "localhost" and the transfer of the DIP occurs within that same machine. Or am I getting it all wrong here?

I found an error in /var/log/apache2/error.log

[Tue Dec 03 20:09:00 2013] [error] [client 127.0.0.1] Action "qtSwordPlugin/deposit" does not exist.

[Courtney Mumma]

Hi again, Paul,

Although I linked to the administrative manual for AtoM configuration and the user manual for DIP upload, I wanted to make sure to ask you specifically whether you enabled job scheduling and the Sword plugin in AtoM.

Courtney

[Courtney Mumma]

Yes, Paul, please see my previous message (looks like our messages crossed paths) - it looks like you do need to enable the Sword plugin and turn on job scheduling as indicated.

Let me know how it goes,
Courtney

--
You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To post to this group, send email to archiv...@googlegroups.com.
Visit this group at http://groups.google.com/group/archivematica.
For more options, visit https://groups.google.com/groups/opt_out.

[Paul Bergström]

Courtney,

they are both enabled and has always been in every attempt I previously made.

qtSwordPlugin SWORD plugin.

1.0.0

Name	Value
Enable job scheduler	yes  no

The thing I find odd in the manual regarding "configure AtoM for DIP upload" is how to do when it's on the same machine?

On the Archivematica-server there is a user "archivematica" with a home directory "/var/lib/archivematica". Then it is suggested to create a NEW user on the AtoM-server also named "archivematica" which has a home directory "/home/archivemativa/". This is not a problem if we are talking about two different machines, but it can't be done on the same machine since there will be a home directory "conflict".

I did this on the same machine by creating a different user "archivematica-rsync".

But the bottom line is whether this is necessary all together when both servers are hosted on the same machine with the same hostname and IP?

/Paul

[Courtney Mumma]

Hi Paul, 

I'm a little stumped. We are currently fixing a couple of bugs that have to do with DIP upload for the 1.0 packages release, so I am hoping that they resolve your problems. I will try to have a developer take a look at this issue over the next few days, too.

Courtney

--
You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To post to this group, send email to archiv...@googlegroups.com.
Visit this group at http://groups.google.com/group/archivematica.
For more options, visit https://groups.google.com/groups/opt_out.

[Paul Bergström]

Courtney,

I made a new setup of both Archivematica and AtoM and this time "by the book" on two different machines.

It stops with:

[uploadDIP] Rsync quit unexpectedly (exit 12), the upload script will be stopped here

Now it unfortunately goes worse. The migrationprocess of the "upload DIP to AtoM" stops in an early stage. This is the log on the Archicematica side:

This account is restricted by rssh.
This user is locked out.

If you believe this is in error, please contact your system administrator.

rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(605) [sender=3.0.9]
 
As what I have read the first two lines are just normal and expected since the user "archivematica" on the AtoM-server doesn't have a shell - just "/usr/bin/rssh".

The last two lines are problematic though. This seems to be an Ubuntu bug that has been discussed in many places for quite a long time without seemingly without becoming resolved.

It looks challenging:)

/Paul

[Paul Bergström]

And on the AtoM side it looks like this in the /var/log/auth.log

Dec  6 20:39:56 daylyn-PC systemd-logind[732]: New session 18 of user archivematica.
Dec  6 20:39:56 daylyn-PC sshd[6366]: Received disconnect from 192.168.1.10: 11: disconnected by user
Dec  6 20:39:56 daylyn-PC sshd[6330]: pam_unix(sshd:session): session closed for user archivematica
Dec  6 20:40:01 daylyn-PC systemd-logind[732]: Removed session 18.
Dec  6 20:40:05 daylyn-PC sshd[6369]: Accepted publickey for archivematica from 192.168.1.10 port 33437 ssh2
Dec  6 20:40:05 daylyn-PC sshd[6369]: pam_unix(sshd:session): session opened for user archivematica by (uid=0)
Dec  6 20:40:05 daylyn-PC systemd-logind[732]: New session 19 of user archivematica.
Dec  6 20:40:05 daylyn-PC sshd[6405]: Received disconnect from 192.168.1.10: 11: disconnected by user
Dec  6 20:40:05 daylyn-PC sshd[6369]: pam_unix(sshd:session): session closed for user archivematica
Dec  6 20:40:06 daylyn-PC systemd-logind[732]: Removed session 19.
Dec  6 20:56:17 daylyn-PC sshd[6446]: Accepted publickey for archivematica from 192.168.1.10 port 34060 ssh2
Dec  6 20:56:17 daylyn-PC sshd[6446]: pam_unix(sshd:session): session opened for user archivematica by (uid=0)
Dec  6 20:56:17 daylyn-PC systemd-logind[732]: New session 20 of user archivematica.
Dec  6 20:56:19 daylyn-PC sshd[6482]: Received disconnect from 192.168.1.10: 11: disconnected by user
Dec  6 20:56:19 daylyn-PC sshd[6446]: pam_unix(sshd:session): session closed for user archivematica


It seems as if the termination of the session is initiated from the Archivematica side, thus causing rsync to close "unexpectedly". The question is "why" or "what"?

/Paul

[Jesús García Crespo]

Hi Paul,

It seems that you are using rssh. Have you updated /etc/rssh.conf in order to allow rsync? I think that you could fix the problem by adding allowrsync to that file.

Let us know how it goes!

Best regards,

--

You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To post to this group, send email to archiv...@googlegroups.com.
Visit this group at http://groups.google.com/group/archivematica.
For more options, visit https://groups.google.com/groups/opt_out.

--
Jesús García Crespo,
Software Engineer, Artefactual Systems Inc.
http://www.artefactual.com | +1.604.527.2056

[Paul Bergström]

Courtney and Jesus,

Yes, I had forgotten to enable "allowrsync" in /etc/rssh.conf and I really thought that was it, but sadly there is more to it:(

The upload really rolls on for a while and at one point I was sure I would succeed, but then finally it turned red "Failed".

[uploadDIP] Target: bild
[uploadDIP] rsync -rltz -P --chmod=ugo=rwX /var/archivematica/sharedDirectory/watchedDirectories/uploadDIP/olki-8fe23cfe-6de7-4d04-b126-41dba52d4f3d archiv...@192.168.1.6:/tmp
[uploadDIP] Rsync output is being saved in /tmp/tmp4d4R7v
[uploadDIP] About to deposit to: http://192.168.1.6/index.php/;sword/deposit/bild
[uploadDIP] > Response code: 500
[uploadDIP] > Location: None
[uploadDIP] > Content received: 


[uploadDIP] Response code not expected

This lead me through suggestions in this group to rebuild the Qubit Search Index. I tried commands like these:

rm -rf data/index/*
rm -rf cache/*
php symfony cc
php symfony search:populate QubitSearch
php symfony search:optimize QubitSearch

I also installed php5-gd which was suggested.

Unfortunately it lead to the exact same result.

The above mentioned tmp-file looks like this:

sending incremental file list
olki-8fe23cfe-6de7-4d04-b126-41dba52d4f3d/
olki-8fe23cfe-6de7-4d04-b126-41dba52d4f3d/METS.8fe23cfe-6de7-4d04-b126-41dba52d4f3d.xml
      132941 100%   23.88MB/s    0:00:00 (xfer#1, to-check=6/8)
olki-8fe23cfe-6de7-4d04-b126-41dba52d4f3d/objects/
olki-8fe23cfe-6de7-4d04-b126-41dba52d4f3d/objects/d4c54eeb-2d83-40d0-99fc-9f0c8c1d4736-MARBLES.jpg
      137263 100%    4.85MB/s    0:00:00 (xfer#2, to-check=3/8)
olki-8fe23cfe-6de7-4d04-b126-41dba52d4f3d/objects/d8a8782f-6b03-4cfa-8651-b6533cc5dd43-Landing_zone.jpg
     1361321 100%    6.52MB/s    0:00:00 (xfer#3, to-check=2/8)
olki-8fe23cfe-6de7-4d04-b126-41dba52d4f3d/thumbnails/
olki-8fe23cfe-6de7-4d04-b126-41dba52d4f3d/thumbnails/d4c54eeb-2d83-40d0-99fc-9f0c8c1d4736.jpg
        5120 100%   25.00kB/s    0:00:00 (xfer#4, to-check=1/8)
olki-8fe23cfe-6de7-4d04-b126-41dba52d4f3d/thumbnails/d8a8782f-6b03-4cfa-8651-b6533cc5dd43.jpg
        3079 100%   15.03kB/s    0:00:00 (xfer#5, to-check=0/8)

sent 1515300 bytes  received 119 bytes  233141.38 bytes/sec
total size is 1639724  speedup is 1.08

I did it a third time, now with "apparmor" torn down, but still the exact same result. There is nothing showing up in the logs that is explaining this:(

/Paul

[Hutchinson, Tim]

Hi Paul,

I may have missed this, but I wondered if you'd tried uploading manually on the AtoM side? - to try to eliminate other issues specific to the AtoM installation. And with a response code of 500, is there anything in the server logs on the AtoM side?

Tim

---

From: archiv...@googlegroups.com [archiv...@googlegroups.com] on behalf of Paul Bergström [paulhenry...@gmail.com]
Sent: December 7, 2013 5:57 AM
To: archiv...@googlegroups.com
Subject: Re: [archivematica] Content not showing up in AToM-server

[Paul Bergström]

Hi Tom,

Well i found this in /var/log/apache2/access.log:

192.168.1.10 - - [07/Dec/2013:14:50:28 +0100] "GET /index.php/bild HTTP/1.1" 200 3787 "-" "python-requests/0.8.3"
192.168.1.10 - arki...@trafikstyrelsen.se [07/Dec/2013:14:50:34 +0100] "POST /index.php/%3Bsword/deposit/bild HTTP/1.1" 500 396 "-" "Archivematica"

Is it really supposed to be /%3Bsword/ in that link?

Could it be a LOCATE related problem? I have both swedish and english on the system.

What other logs are there supposed to be? And how to do the upload manually?

/Paul

[Hutchinson, Tim]

Hi Paul,

Is there anything in the apache error log? I'm not sure what the path is supposed to be or anything about LOCATE, we've ventured outside my expertise :)

To try the upload manually, log into your AtoM site and edit the description. At the bottom of the screen you should see both "link digital object" (for a single object) and "import digital objects" (for multiple objects). It occurs to me now that the process is probably different, but we might as well eliminate that angle.

Tim

---

From: archiv...@googlegroups.com [archiv...@googlegroups.com] on behalf of Paul Bergström [paulhenry...@gmail.com]

Sent: December 7, 2013 8:20 AM

To: archiv...@googlegroups.com
Subject: Re: [archivematica] Content not showing up in AToM-server

[Paul Bergström]

Tom,

Yes there are some errors, but they aren't saying that much:

[Sat Dec 07 16:16:44.136269 2013] [:error] [pid 6783] [client 127.0.0.1:45759]
[Sat Dec 07 16:16:44.691557 2013] [:error] [pid 6769] [client 127.0.0.1:45760]
[Sat Dec 07 16:16:45.155329 2013] [:error] [pid 6770] [client 127.0.0.1:45761]
[Sat Dec 07 16:16:45.489438 2013] [:error] [pid 6771] [client 127.0.0.1:45762]
[Sat Dec 07 16:16:45.671849 2013] [:error] [pid 6772] [client 127.0.0.1:45763]
[Sat Dec 07 16:16:46.013735 2013] [:error] [pid 6773] [client 127.0.0.1:45764]
[Sat Dec 07 16:16:47.757643 2013] [:error] [pid 6783] [client 127.0.0.1:45765]
[Sat Dec 07 16:16:48.328050 2013] [:error] [pid 6769] [client 127.0.0.1:45766]
[Sat Dec 07 16:17:31.491576 2013] [:error] [pid 6844] [client 127.0.0.1:45780]
[Sat Dec 07 16:17:52.920745 2013] [:error] [pid 6845] [client 127.0.0.1:45781]

When I upload one file locally it works just fine, but multiple halts immediately:

Import multiple digital objects

Fonds Bilder - bild

×

• Required.

Title

The "%dd%" placeholder will be replaced with a incremental number (e.g. 'image 01', 'image 02')

Level of description

Digital objects

Waiting...

Filename:Personligt_brev.pdf

Filesize:33940 bytes

Start

Upload error, retry?

Filename:skiftesprotokoll.pdf

Filesize:24067 bytes

Cancel

Select files

Actions

• Cancel
• 
  

There is an upload error directly. I tried to set the whole Apache2-documentroot open by doing "sudo chmod -R 777 /var/www/*", but it's still the same error. I wonder what "Required" refers to here?

There is another thing I should mention. The AtoM-servers runs in a 64-bits machine.

/Paul

[Hutchinson, Tim]

Hi Paul,

The multiple file issue sounds like a common setup issue; see especially re the cookies directive:
https://www.qubit-toolkit.org/wiki/Multiple_file_upload

But I think it's probably unrelated to the transfer from Archivematica. Hopefully one of the developers will be able to shed some light on the log entries etc.; I'm out of my depth on the rest of this.

Tim

---

From: archiv...@googlegroups.com [archiv...@googlegroups.com] on behalf of Paul Bergström [paulhenry...@gmail.com]

Sent: December 7, 2013 9:27 AM

To: archiv...@googlegroups.com
Subject: Re: [archivematica] Content not showing up in AToM-server

[Paul Bergström]

Tom,

Thanks for your effort:)

It made some progress. I can now upload multiple files from the local machine.

This is what I did:

In the file /etc/php5/apache2/php.ini i changed the directive session.use-only-cookies=1 to session.use-only-cookies=0.

However the upload issue between Archivematica and AtoM remains:(

When uploading locally, is the plugin Sword involved? If not, that's perhaps where the problem is?

/Paul

[Jesús García Crespo]

On Sat, Dec 7, 2013 at 8:46 AM, Paul Bergström <paulhenry...@gmail.com> wrote:

When uploading locally, is the plugin Sword involved? If not, that's perhaps where the problem is?

Yes, it is.

Do you have AtoM and Archivematica running in different machines?

[Hutchinson, Tim]

Paul, I wonder if it's worth going back to the default setup (with passwords, without rsync), and see if you can get that working first.

Tim

--

[Paul Bergström]

Jesus,

Yes they are running on two different machines.

Archivematica runs on 32-bits Ubuntu 12.04 LTS
AtoM runs on 64-bits Ubuntu 13.04.

/Paul

[Paul Bergström]

Tom,

I will try that too.

/Paul

[Jesús García Crespo]

On Sat, Dec 7, 2013 at 9:39 AM, Paul Bergström <paulhenry...@gmail.com> wrote:

Yes they are running on two different machines.

Do you have the atom-worker service up and running? There should be a file under /tmp/atom-worker.log with some details about its status.

[Paul Bergström]

Jesus,

I think that might be it:)

That file is not present in /tmp.

So is the atom-worker started by "service atom-worker start" or some other way?

/Paul

[Jesús García Crespo]

Either that or "sudo start atom-worker", but you need to copy the upstart config file to /etc/init first.

dev-installer should be able to do all that in a single-node environment.

--

You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To post to this group, send email to archiv...@googlegroups.com.
Visit this group at http://groups.google.com/group/archivematica.
For more options, visit https://groups.google.com/groups/opt_out.

[Paul Bergström]

Hi everyone,

Well what can I say? The circle is closed now and I'm back in the exact same situation as in the beginning of the thread - now having the same situation with two nodes involved instead of one.

After setting up the /etc/init/atom-worker.conf correctly and with the help of dev-installer also installing gearman as suggested by Jesus, a system restart, then a confirmed running of the AtoM-service, I tried once more to "Upload to AtoM" from Archivematica.

It went fine - nothing appears to be wrong - but there is nothing showing up on the AtoM-server.

Here is Archivematica log that finished with exitcode 0:

[uploadDIP] Target: bild
[uploadDIP] rsync -rltz -P --chmod=ugo=rwX /var/archivematica/sharedDirectory/watchedDirectories/uploadDIP/rukko-2e8f4d27-92ee-4643-a663-642c6326f96d archiv...@192.168.1.6:/tmp
[uploadDIP] Rsync output is being saved in /tmp/tmpIGlrKL
[uploadDIP] About to deposit to: http://192.168.1.6/index.php/;sword/deposit/bild
[uploadDIP] > Response code: 302
[uploadDIP] > Location: /index.php/bild;isad
[uploadDIP] > Content received: <?xml version="1.0" encoding="utf-8" ?><entry xmlns="http://www.w3.org/2005/Atom"
       xmlns:sword="http://purl.org/net/sword/">

  <title>bild</title>

  
  <id>313 / 2013-12-05T11:43:45</id>

  <updated>2013-12-05T11:43:45</updated>
  <author>
    <name>arkivarie</name>
  </author>

  <generator uri="http://192.168.1.6/index.php/" version="1.3.1">Qubit 1.3.1</generator>

  <content type="text/html" src="http://192.168.1.6/index.php/bild;isad" />

  <sword:noOp>false</sword:noOp>

  <sword:packaging>http://purl.org/net/sword-types/METSArchivematicaDIP</sword:packaging>

  <sword:userAgent>Archivematica</sword:userAgent>

  
</entry>

[uploadDIP] Deposited asynchronously, Qubit is processing the DIP in the job queue

Here is the /tmp/atom-worker.log:

>> gearman-worker New ability: qtSwordPluginWorker
>> gearman-worker Running worker...
>> gearman-worker PID 9215
>> gearman-worker A new job has started to being processed.
>> gearman-worker A package was deposited by reference.
>> gearman-worker Location: file:///rukko-2e8f4d27-92ee-4643-a663-642c6326f96d
>> gearman-worker Processing...
>> gearman-worker Object slug: bild
>> gearman-worker Job finished.

I wonder where in the filestructure of the AtoM-server, the uploaded files are supposed to be stored? There is a directory "uploads" - is it there?

/Paul

[Paul Bergström]

Praise the Lord:)

I logged out from the AtoM-server and then in again.....and VOILÀ.....the uploaded files were there:)))))))

Thank you all for your efforts on this one.

Knowledge is painstaking.....

/Paul

[Courtney Mumma]

I literally just WooHoo'd for you in the middle of my living room! Great news,
Courtney

--
You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To post to this group, send email to archiv...@googlegroups.com.
Visit this group at http://groups.google.com/group/archivematica.
For more options, visit https://groups.google.com/groups/opt_out.

--