separate Storage Service
163 views
Skip to first unread message
Jamen McGranahan
Oct 2, 2019, 10:46:41 AM10/2/19
to archivematica
Archivematica 1.10.0
Archivematica Storage Service 0.15.0
DEBUG 2019-10-02 14:35:31 archivematica.mcp.server:taskGroupRunner:_monitor_running_jobs:258: 0 jobs pending; 0 jobs running; 0 known task groups
DEBUG 2019-10-02 14:35:34 archivematica.mcp.server:package:create_package:315: Transfer object created: 225f198c-9afc-41d4-b161-4ef4a2b645ea
DEBUG 2019-10-02 14:35:34 archivematica.mcp.server:package:_start:325: Package 225f198c-9afc-41d4-b161-4ef4a2b645ea: starting transfer (('test123', u'standard', 'e4f7d748-f676-4e1c-92e3-adab814520f4:/apps/Images', '/var/archivematica/sharedDirectory/tmp/tmpyNrKeU'))
DEBUG 2019-10-02 14:35:34 archivematica.mcp.server:package:_start_package_transfer_with_auto_approval:409: Package 225f198c-9afc-41d4-b161-4ef4a2b645ea: determined vars transfer_rel=tmp/tmpyNrKeU/test123, filepath=/var/archivematica/sharedDirectory/tmp/tmpyNrKeU/test123, path=e4f7d748-f676-4e1c-92e3-adab814520f4:/apps/Images/.
DEBUG 2019-10-02 14:35:34 archivematica.mcp.server:package:_start_package_transfer_with_auto_approval:416: Package 225f198c-9afc-41d4-b161-4ef4a2b645ea: copying chosen contents from transfer sources (from=e4f7d748-f676-4e1c-92e3-adab814520f4:/apps/Images/., to=tmp/tmpyNrKeU/test123)
DEBUG 2019-10-02 14:35:34 archivematica.common:storageService:get_location:192: Storage locations returned: [{u'uuid': u'194600f8-0cd4-4f0c-89c3-534723a89b49', u'pipeline': [u'/api/v2/pipeline/42991a9e-30d8-4ded-b556-85168fb0e1ea/'], u'used': u'0', u'description': None, u'space': u'/api/v2/space/72376017-c0ab-41b3-a5bc-84927cb3e515/', u'enabled': True, u'quota': None, u'relative_path': u'var/archivematica/sharedDirectory/', u'purpose': u'CP', u'path': u'/var/archivematica/sharedDirectory', u'resource_uri': u'/api/v2/location/194600f8-0cd4-4f0c-89c3-534723a89b49/'}, {u'uuid': u'c7dde1d4-fffe-40c3-a4c9-eddfeb049fb1', u'pipeline': [u'/api/v2/pipeline/42991a9e-30d8-4ded-b556-85168fb0e1ea/'], u'used': u'0', u'description': u'processing', u'space': u'/api/v2/space/d811f7b6-dd27-4242-a545-4b7d0d422533/', u'enabled': True, u'quota': None, u'relative_path': u'/var/archivematica/sharedDirectory', u'purpose': u'CP', u'path': u'/var/archivematica/sharedDirectory', u'resource_uri': u'/api/v2/location/c7dde1d4-fffe-40c3-a4c9-eddfeb049fb1/'}]
DEBUG 2019-10-02 14:35:34 archivematica.common:storageService:get_location:192: Storage locations returned: [{u'uuid': u'e4f7d748-f676-4e1c-92e3-adab814520f4', u'pipeline': [u'/api/v2/pipeline/42991a9e-30d8-4ded-b556-85168fb0e1ea/'], u'used': u'0', u'description': u'Apps', u'space': u'/api/v2/space/72376017-c0ab-41b3-a5bc-84927cb3e515/', u'enabled': True, u'quota': None, u'relative_path': u'apps', u'purpose': u'TS', u'path': u'/apps', u'resource_uri': u'/api/v2/location/e4f7d748-f676-4e1c-92e3-adab814520f4/'}]
DEBUG 2019-10-02 14:35:34 archivematica.mcp.server:package:_copy_from_transfer_sources:220: source: Images/., destination: /var/archivematica/sharedDirectory/tmp/tmpyNrKeU/test123/.
WARNING 2019-10-02 14:35:35 archivematica.common:storageService:copy_files:257: Unable to move files with {'files': [{'source': u'Images/.', 'destination': '/var/archivematica/sharedDirectory/tmp/tmpyNrKeU/test123/.'}], 'pipeline': u'/api/v2/pipeline/42991a9e-30d8-4ded-b556-85168fb0e1ea/', 'origin_location': u'/api/v2/location/e4f7d748-f676-4e1c-92e3-adab814520f4/'} because 500 Server Error: INTERNAL SERVER ERROR for url: http://archivematica-storage.library.vanderbilt.edu/api/v2/location/194600f8-0cd4-4f0c-89c3-534723a89b49/
ERROR 2019-10-02 14:35:35 archivematica.mcp.server:package:wrap:371: Exception: The following errors occured: 500 Server Error: INTERNAL SERVER ERROR for url: http://archivematica-storage.library.vanderbilt.edu/api/v2/location/194600f8-0cd4-4f0c-89c3-534723a89b49/
Traceback (most recent call last):
File "/usr/lib/archivematica/MCPServer/package.py", line 350, in wrap
fn(*args, **kwargs)
File "/usr/lib/archivematica/MCPServer/package.py", line 418, in _start_package_transfer_with_auto_approval
_copy_from_transfer_sources([path], transfer_rel)
File "/usr/lib/archivematica/MCPServer/package.py", line 232, in _copy_from_transfer_sources
% {"message": ", ".join(message)}
Exception: The following errors occured: 500 Server Error: INTERNAL SERVER ERROR for url: http://archivematica-storage.library.vanderbilt.edu/api/v2/location/194600f8-0cd4-4f0c-89c3-534723a89b49/
^C
I'm not sure how to fix this. Any advice/guidance is greatly appreciated!
Sincerely,
Jamen McGranahan
Jorik van Kemenade
Oct 2, 2019, 10:52:54 AM10/2/19
to archiv...@googlegroups.com
Hi Jamen,
Have you checked the (debug) logs of the Archivematica Storage service?
They are located in /var/log/archivematica/storage-service/ on your storage server.
If you share them with us we might be able to help you further.
Regards,
Jorik
Have you checked the (debug) logs of the Archivematica Storage service?
They are located in /var/log/archivematica/storage-service/ on your storage server.
If you share them with us we might be able to help you further.
Regards,
Jorik
--
You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archivematica/b95b2041-c165-44b0-9d64-84f06125874d%40googlegroups.com.
Jamen McGranahan
Oct 2, 2019, 11:05:37 AM10/2/19
to archivematica
That just gives a bunch of lines like this:
DEBUG 2019-10-02 07:59:57 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:00:02 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:00:07 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:00:12 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:00:17 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:00:22 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:00:27 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:00:32 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:00:37 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:00:42 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:00:47 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:00:52 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:00:57 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:01:02 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:01:07 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 08:01:12 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
--Jamen
To unsubscribe from this group and stop receiving emails from it, send an email to archiv...@googlegroups.com.
Jamen McGranahan
Oct 2, 2019, 1:22:40 PM10/2/19
to archivematica
I just tested it again, keeping an eye on both logs from both servers. Here is what I found:
on Archivematica Storage Service - from /var/log/archivematica/storage-service/storage_service_debug.log -
DEBUG 2019-10-02 10:18:51 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
DEBUG 2019-10-02 10:18:51 locations.api.resources:resources:_move_files_between_locations:476: Moving files from this location is not allowed
DEBUG 2019-10-02 10:18:51 locations.models.space:space:move_to_storage_service:340: TO: src: apps/Images/.
DEBUG 2019-10-02 10:18:51 locations.models.space:space:move_to_storage_service:341: TO: dst: /var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123/.
DEBUG 2019-10-02 10:18:51 locations.models.space:space:move_to_storage_service:342: TO: staging: /var/archivematica/storage_service
INFO 2019-10-02 10:18:51 locations.models.space:space:move_rsync:546: Moving from /apps/Images/. to /var/archivematica/storage_service/var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123/.
INFO 2019-10-02 10:18:51 locations.models.space:space:move_rsync:588: rsync command: ['rsync', '-t', '-O', '--protect-args', '-vv', '--chmod=Fug+rw,o-rwx,Dug+rwx,o-rwx', '-r', '/apps/Images/.', '/var/archivematica/storage_service/var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123/.']
DEBUG 2019-10-02 10:18:52 locations.models.space:space:move_from_storage_service:417: FROM: src: /var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123/.
DEBUG 2019-10-02 10:18:52 locations.models.space:space:move_from_storage_service:418: FROM: dst: /var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123/.
INFO 2019-10-02 10:18:52 locations.models.space:space:move_rsync:546: Moving from /var/archivematica/storage_service/var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123/./ to /var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123/.
DEBUG 2019-10-02 10:18:52 locations.models.space:space:move_rsync:560: os.rename failed, trying with normalized paths
DEBUG 2019-10-02 10:18:56 locations.models.async_manager:async_manager:_watchdog_loop:122: Watchdog sees 0 tasks running
on Archivematica server - from /var/log/archivematica/MCPServer/MCPServer.debug.log -
DEBUG 2019-10-02 17:18:50 archivematica.mcp.server:taskGroupRunner:_monitor_running_jobs:258: 0 jobs pending; 0 jobs running; 0 known task groups
DEBUG 2019-10-02 17:18:51 archivematica.mcp.server:package:create_package:315: Transfer object created: a6e799eb-8809-445c-be84-64d61a51ceeb
DEBUG 2019-10-02 17:18:51 archivematica.mcp.server:package:_start:325: Package a6e799eb-8809-445c-be84-64d61a51ceeb: starting transfer (('test123', u'standard', 'e4f7d748-f676-4e1c-92e3-adab814520f4:/apps/Images', '/var/archivematica/sharedDirectory/tmp/tmpZOugOu'))
DEBUG 2019-10-02 17:18:51 archivematica.mcp.server:package:_start_package_transfer_with_auto_approval:409: Package a6e799eb-8809-445c-be84-64d61a51ceeb: determined vars transfer_rel=tmp/tmpZOugOu/test123, filepath=/var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123, path=e4f7d748-f676-4e1c-92e3-adab814520f4:/apps/Images/.
DEBUG 2019-10-02 17:18:51 archivematica.mcp.server:package:_start_package_transfer_with_auto_approval:416: Package a6e799eb-8809-445c-be84-64d61a51ceeb: copying chosen contents from transfer sources (from=e4f7d748-f676-4e1c-92e3-adab814520f4:/apps/Images/., to=tmp/tmpZOugOu/test123)
DEBUG 2019-10-02 17:18:51 archivematica.common:storageService:get_location:192: Storage locations returned: [{u'uuid': u'194600f8-0cd4-4f0c-89c3-534723a89b49', u'pipeline': [u'/api/v2/pipeline/42991a9e-30d8-4ded-b556-85168fb0e1ea/'], u'used': u'0', u'description': None, u'space': u'/api/v2/space/72376017-c0ab-41b3-a5bc-84927cb3e515/', u'enabled': True, u'quota': None, u'relative_path': u'var/archivematica/sharedDirectory/', u'purpose': u'CP', u'path': u'/var/archivematica/sharedDirectory', u'resource_uri': u'/api/v2/location/194600f8-0cd4-4f0c-89c3-534723a89b49/'}, {u'uuid': u'c7dde1d4-fffe-40c3-a4c9-eddfeb049fb1', u'pipeline': [u'/api/v2/pipeline/42991a9e-30d8-4ded-b556-85168fb0e1ea/'], u'used': u'0', u'description': u'processing', u'space': u'/api/v2/space/d811f7b6-dd27-4242-a545-4b7d0d422533/', u'enabled': True, u'quota': None, u'relative_path': u'/apps', u'purpose': u'CP', u'path': u'/apps', u'resource_uri': u'/api/v2/location/c7dde1d4-fffe-40c3-a4c9-eddfeb049fb1/'}]
DEBUG 2019-10-02 17:18:51 archivematica.common:storageService:get_location:192: Storage locations returned: [{u'uuid': u'e4f7d748-f676-4e1c-92e3-adab814520f4', u'pipeline': [u'/api/v2/pipeline/42991a9e-30d8-4ded-b556-85168fb0e1ea/'], u'used': u'0', u'description': u'Apps', u'space': u'/api/v2/space/72376017-c0ab-41b3-a5bc-84927cb3e515/', u'enabled': True, u'quota': None, u'relative_path': u'apps', u'purpose': u'TS', u'path': u'/apps', u'resource_uri': u'/api/v2/location/e4f7d748-f676-4e1c-92e3-adab814520f4/'}]
DEBUG 2019-10-02 17:18:51 archivematica.mcp.server:package:_copy_from_transfer_sources:220: source: Images/., destination: /var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123/.
DEBUG 2019-10-02 17:18:52 archivematica.mcp.server:package:_start_package_transfer_with_auto_approval:422: Package a6e799eb-8809-445c-be84-64d61a51ceeb: moving package to processing directory
ERROR 2019-10-02 17:18:52 archivematica.mcp.server:package:wrap:371: Exception: Filepath /var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123 does not exist.
Traceback (most recent call last):
File "/usr/lib/archivematica/MCPServer/package.py", line 350, in wrap
fn(*args, **kwargs)
File "/usr/lib/archivematica/MCPServer/package.py", line 424, in _start_package_transfer_with_auto_approval
filepath, django_settings.PROCESSING_DIRECTORY, transfer
File "/usr/lib/archivematica/MCPServer/package.py", line 247, in _move_to_internal_shared_dir
raise Exception(error)
Exception: Filepath /var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123 does not exist.
DEBUG 2019-10-02 17:19:00 archivematica.mcp.server:taskGroupRunner:_monitor_running_jobs:258: 0 jobs pending; 0 jobs running; 0 known task groups
Karl Goetz
Oct 2, 2019, 7:33:06 PM10/2/19
to Jamen McGranahan, archiv...@googlegroups.com
On Wed, 2 Oct 2019 10:22:40 -0700 (PDT)
Jamen McGranahan <jmcgr...@gmail.com> wrote:
> I just tested it again, keeping an eye on both logs from both servers. Here
> is what I found:
Jamen McGranahan <jmcgr...@gmail.com> wrote:
> I just tested it again, keeping an eye on both logs from both servers. Here
> is what I found:
> locations.models.space:space:move_rsync:588: rsync command: ['rsync',
> '-t', '-O', '--protect-args', '-vv', '--chmod=Fug+rw,o-rwx,Dug+rwx,o-rwx',
> '-r', '/apps/Images/.',
> '/var/archivematica/storage_service/var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123/.']
> DEBUG 2019-10-02 17:18:51
> '-t', '-O', '--protect-args', '-vv', '--chmod=Fug+rw,o-rwx,Dug+rwx,o-rwx',
> '-r', '/apps/Images/.',
> '/var/archivematica/storage_service/var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123/.']
> archivematica.mcp.server:package:_copy_from_transfer_sources:220: source:
> Images/., destination:
> /var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123/.
> DEBUG 2019-10-02 17:18:52
> archivematica.mcp.server:package:_start_package_transfer_with_auto_approval:422:
> Package a6e799eb-8809-445c-be84-64d61a51ceeb: moving package to processing
> directory
> ERROR 2019-10-02 17:18:52 archivematica.mcp.server:package:wrap:371:
> Exception: Filepath
> /var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123 does not exist.
> Traceback (most recent call last):
> File "/usr/lib/archivematica/MCPServer/package.py", line 350, in wrap
> fn(*args, **kwargs)
> File "/usr/lib/archivematica/MCPServer/package.py", line 424, in
> _start_package_transfer_with_auto_approval
> filepath, django_settings.PROCESSING_DIRECTORY, transfer
> File "/usr/lib/archivematica/MCPServer/package.py", line 247, in
> _move_to_internal_shared_dir
> raise Exception(error)
> Exception: Filepath
> /var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123 does not exist.
> DEBUG 2019-10-02 17:19:00
> archivematica.mcp.server:taskGroupRunner:_monitor_running_jobs:258: 0 jobs
> pending; 0 jobs running; 0 known task groups
>
Does /var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123 exist? :)
> Images/., destination:
> /var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123/.
> DEBUG 2019-10-02 17:18:52
> archivematica.mcp.server:package:_start_package_transfer_with_auto_approval:422:
> Package a6e799eb-8809-445c-be84-64d61a51ceeb: moving package to processing
> directory
> ERROR 2019-10-02 17:18:52 archivematica.mcp.server:package:wrap:371:
> Exception: Filepath
> /var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123 does not exist.
> Traceback (most recent call last):
> File "/usr/lib/archivematica/MCPServer/package.py", line 350, in wrap
> fn(*args, **kwargs)
> File "/usr/lib/archivematica/MCPServer/package.py", line 424, in
> _start_package_transfer_with_auto_approval
> filepath, django_settings.PROCESSING_DIRECTORY, transfer
> File "/usr/lib/archivematica/MCPServer/package.py", line 247, in
> _move_to_internal_shared_dir
> raise Exception(error)
> Exception: Filepath
> /var/archivematica/sharedDirectory/tmp/tmpZOugOu/test123 does not exist.
> DEBUG 2019-10-02 17:19:00
> archivematica.mcp.server:taskGroupRunner:_monitor_running_jobs:258: 0 jobs
> pending; 0 jobs running; 0 known task groups
>
We have a similar issue - a full /var/archivematica hierachy nested inside /var/archivematica.
:~$ ls -lhd /var/archivematica/storage_service/var/archivematica/sharedDirectory/tmp/
drwxr-xr-x 21 archivematica archivematica 4.0K Jun 2 21:39 /var/archivematica/storage_service/var/archivematica/sharedDirectory/tmp/
:~$ ls -lhd /var/archivematica/sharedDirectory/tmp/
drwxr-x--- 14 archivematica archivematica 4.0K Sep 30 05:01 /var/archivematica/sharedDirectory/tmp/
If you don't have the second folder you will definitely see the problem from above because you rsynced in to the first location but are
trying to process it from the second.
Not much of a clue but hopefully gets you on the right path.
thanks,
--
Karl Goetz
Technical Services Officer - eResearch, Information Technology Services
University of Tasmania & Tasmanian Partnership for Advanced Computing
Mail: University of Tasmania, Private Bag 69, Hobart, Tasmania 7001
Delivery: TT Flynn Street, Sandy Bay, Tasmania 7005
University of Tasmania Electronic Communications Policy (December, 2014).
This email is confidential, and is for the intended recipient only. Access, disclosure, copying, distribution, or reliance on any of it by anyone outside the intended recipient organisation is prohibited and may be a criminal offence. Please delete if obtained in error and email confirmation to the sender. The views expressed in this email are not necessarily the views of the University of Tasmania, unless clearly intended otherwise.
Jamen McGranahan
Oct 2, 2019, 9:22:10 PM10/2/19
to archivematica
Karl,
Both servers have /var/archivematica directories.
Archivematica - /var/archivematica/sharedDirectory/tmp/tmpSaQQDn - this is the one that is unable to add the files
Archivematica Storage Service - /var/archivematica/sharedDirectory/tmp/tmpSaQQDn - this one has created the test3 directory & the files are there.
* Original location of the files is located at /apps/Pdfs/
* /var/archivematica/storage_service/var/archivematica/sharedDirectory/tmp -- has the tmpSaQQDn directory but 0 files
I'm kind of at a loss here. Please advise. Thank you.
--Jamen
Jorik van Kemenade
Oct 3, 2019, 3:14:42 AM10/3/19
to archiv...@googlegroups.com
Hi Jamen,
The Archivematica Storage Service has different types of access protocols for different types of file systems.
For some more information on the different types please check out:
https://www.archivematica.org/en/docs/storage-service-0.15/administrators/#access-protocols
By default the Storage Service uses the Local Filesystem type.
This means that all operations by the Storage Service are done on disk using "normal" file system operations where possible.
This works well in the standard case where the Storage Service is collocated with all the other services.
However, you have a different case.
In your case the Storage Service moves the files to on it's own disk, but not on the disk visible by the dashboard service.
As far as I know there are two possible solutions to your problem.
Number 1 is the simplest solution, which I would recommend at this point.
I am currently experimenting with 2, so let me know if you want more information.
1) Use a Pipeline Local Filesystem for /var/archivematica/sharedDirectory
On the Storage Service create a new space of the Pipeline Local Filesystem type.
Path and staging path can remain /.
Remote user should be 'archivematica'.
The remote name should be the name of your dashboard server.
Once you create the Pipeline Local Filesystem, you click "Create Location Here"
There you select the "Currently Processing" purpose with relative path: "var/archivematica/sharedDirectory/"
Save everything and it should work.
2) Use a shared file system for the shared directory
I am currently working on using a shared file system for the shared directory.
However, this requires some extra configuration etcetera and might come with some unintended side effects.
If you are interested let me know, I can share my experiences later this week.
Hopefully this helps, if you have any questions please let me know.
Good luck!
Jorik
The Archivematica Storage Service has different types of access protocols for different types of file systems.
For some more information on the different types please check out:
https://www.archivematica.org/en/docs/storage-service-0.15/administrators/#access-protocols
By default the Storage Service uses the Local Filesystem type.
This means that all operations by the Storage Service are done on disk using "normal" file system operations where possible.
This works well in the standard case where the Storage Service is collocated with all the other services.
However, you have a different case.
In your case the Storage Service moves the files to on it's own disk, but not on the disk visible by the dashboard service.
As far as I know there are two possible solutions to your problem.
Number 1 is the simplest solution, which I would recommend at this point.
I am currently experimenting with 2, so let me know if you want more information.
1) Use a Pipeline Local Filesystem for /var/archivematica/sharedDirectory
On the Storage Service create a new space of the Pipeline Local Filesystem type.
Path and staging path can remain /.
Remote user should be 'archivematica'.
The remote name should be the name of your dashboard server.
Once you create the Pipeline Local Filesystem, you click "Create Location Here"
There you select the "Currently Processing" purpose with relative path: "var/archivematica/sharedDirectory/"
Save everything and it should work.
2) Use a shared file system for the shared directory
I am currently working on using a shared file system for the shared directory.
However, this requires some extra configuration etcetera and might come with some unintended side effects.
If you are interested let me know, I can share my experiences later this week.
Hopefully this helps, if you have any questions please let me know.
Good luck!
Jorik
--
You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archivematica/2c4e6052-38ff-47c2-978d-d2f65fb4a4ef%40googlegroups.com.
Jamen McGranahan
Oct 3, 2019, 11:58:17 AM10/3/19
to archivematica
Unfortunately, it's still not working. I tried the #1 method again (had done so earlier, based on another forum post). This is very frustrating and once again, makes me NOT want to use Archivematica. But I will keep working at it because it is something my institution wants to proceed with, so I have to get this working.
The Storage Service appears to be putting files where they belong on it's system. It's just Archivematica is not copying the file there to process it. The error log - even in debug mode - is no help at all. I see this error:
ERROR 2019-10-03 15:35:29 archivematica.mcp.server:package:wrap:371: Exception: Filepath /var/archivematica/sharedDirectory/tmp/tmpfXZaeI/testing does not exist.
And I'm like, no duh - but WHY does it not exist? It DOES create tmpfXZaeI, but DOES NOT create the testing folder that contains the files to be processed. That's what it does not tell me so I am just shooting in the dark here. Any other suggestions? Guidance? Advice? Or should I just chuck it and start all over?
And I'm like, no duh - but WHY does it not exist? It DOES create tmpfXZaeI, but DOES NOT create the testing folder that contains the files to be processed. That's what it does not tell me so I am just shooting in the dark here. Any other suggestions? Guidance? Advice? Or should I just chuck it and start all over?
To unsubscribe from this group and stop receiving emails from it, send an email to archiv...@googlegroups.com.
Miguel Angel Medinilla Luque
Oct 3, 2019, 12:41:17 PM10/3/19
to archivematica
Hi Jamen,
Jorik van Kemenade has explained perfectly the 2 alternatives to configure SS and pipeline in separate servers.
It seems you are going to use the option #1 (I think it is the better option). Please, can you copy again a screenshot with the locations? (please, add the pipeline column)
Thanks.
Jamen McGranahan
Oct 3, 2019, 3:03:55 PM10/3/19
to archivematica
Here it is:
Miguel Angel Medinilla Luque
Oct 3, 2019, 5:14:56 PM10/3/19
to archivematica
Hi Jamen,
You have 2 enabled Currently Processing locations for the pipeline. The first step is to disable (or remove, or unselect the pipeline from the location) the local currently processing. You need to use as Currently Processing the one created in the "Pipeline Local Filesystem" Space.
Please, can you try it?
Thanks
Jamen McGranahan
Oct 3, 2019, 7:03:47 PM10/3/19
to archiv...@googlegroups.com
No change in behavior:
DEBUG 2019-10-03 23:01:07 archivematica.mcp.server:package:create_package:315: Transfer object created: 100f3026-c70b-471b-9603-743463f5b0d3
DEBUG 2019-10-03 23:01:07 archivematica.mcp.server:package:_start:325: Package 100f3026-c70b-471b-9603-743463f5b0d3: starting transfer (('test1234', u'standard', 'e4f7d748-f676-4e1c-92e3-adab814520f4:/apps/Images', '/var/archivematica/sharedDirectory/tmp/tmpZfBHad'))
DEBUG 2019-10-03 23:01:07 archivematica.mcp.server:package:_start_package_transfer_with_auto_approval:409: Package 100f3026-c70b-471b-9603-743463f5b0d3: determined vars transfer_rel=tmp/tmpZfBHad/test1234, filepath=/var/archivematica/sharedDirectory/tmp/tmpZfBHad/test1234, path=e4f7d748-f676-4e1c-92e3-adab814520f4:/apps/Images/.
DEBUG 2019-10-03 23:01:07 archivematica.mcp.server:package:_start_package_transfer_with_auto_approval:416: Package 100f3026-c70b-471b-9603-743463f5b0d3: copying chosen contents from transfer sources (from=e4f7d748-f676-4e1c-92e3-adab814520f4:/apps/Images/., to=tmp/tmpZfBHad/test1234)
DEBUG 2019-10-03 23:01:07 archivematica.common:storageService:get_location:192: Storage locations returned: [{u'uuid': u'194600f8-0cd4-4f0c-89c3-534723a89b49', u'pipeline': [u'/api/v2/pipeline/42991a9e-30d8-4ded-b556-85168fb0e1ea/'], u'used': u'0', u'description': None, u'space': u'/api/v2/space/72376017-c0ab-41b3-a5bc-84927cb3e515/', u'enabled': True, u'quota': None, u'relative_path': u'var/archivematica/sharedDirectory/', u'purpose': u'CP', u'path': u'/var/archivematica/sharedDirectory', u'resource_uri': u'/api/v2/location/194600f8-0cd4-4f0c-89c3-534723a89b49/'}]
DEBUG 2019-10-03 23:01:07 archivematica.common:storageService:get_location:192: Storage locations returned: [{u'uuid': u'e4f7d748-f676-4e1c-92e3-adab814520f4', u'pipeline': [u'/api/v2/pipeline/42991a9e-30d8-4ded-b556-85168fb0e1ea/'], u'used': u'0', u'description': u'Apps', u'space': u'/api/v2/space/72376017-c0ab-41b3-a5bc-84927cb3e515/', u'enabled': True, u'quota': None, u'relative_path': u'apps', u'purpose': u'TS', u'path': u'/apps', u'resource_uri': u'/api/v2/location/e4f7d748-f676-4e1c-92e3-adab814520f4/'}]
DEBUG 2019-10-03 23:01:07 archivematica.mcp.server:package:_copy_from_transfer_sources:220: source: Images/., destination: /var/archivematica/sharedDirectory/tmp/tmpZfBHad/test1234/.
DEBUG 2019-10-03 23:01:08 archivematica.mcp.server:package:_start_package_transfer_with_auto_approval:422: Package 100f3026-c70b-471b-9603-743463f5b0d3: moving package to processing directory
ERROR 2019-10-03 23:01:08 archivematica.mcp.server:package:wrap:371: Exception: Filepath /var/archivematica/sharedDirectory/tmp/tmpZfBHad/test1234 does not exist.
DEBUG 2019-10-03 23:01:07 archivematica.mcp.server:package:_start:325: Package 100f3026-c70b-471b-9603-743463f5b0d3: starting transfer (('test1234', u'standard', 'e4f7d748-f676-4e1c-92e3-adab814520f4:/apps/Images', '/var/archivematica/sharedDirectory/tmp/tmpZfBHad'))
DEBUG 2019-10-03 23:01:07 archivematica.mcp.server:package:_start_package_transfer_with_auto_approval:409: Package 100f3026-c70b-471b-9603-743463f5b0d3: determined vars transfer_rel=tmp/tmpZfBHad/test1234, filepath=/var/archivematica/sharedDirectory/tmp/tmpZfBHad/test1234, path=e4f7d748-f676-4e1c-92e3-adab814520f4:/apps/Images/.
DEBUG 2019-10-03 23:01:07 archivematica.mcp.server:package:_start_package_transfer_with_auto_approval:416: Package 100f3026-c70b-471b-9603-743463f5b0d3: copying chosen contents from transfer sources (from=e4f7d748-f676-4e1c-92e3-adab814520f4:/apps/Images/., to=tmp/tmpZfBHad/test1234)
DEBUG 2019-10-03 23:01:07 archivematica.common:storageService:get_location:192: Storage locations returned: [{u'uuid': u'194600f8-0cd4-4f0c-89c3-534723a89b49', u'pipeline': [u'/api/v2/pipeline/42991a9e-30d8-4ded-b556-85168fb0e1ea/'], u'used': u'0', u'description': None, u'space': u'/api/v2/space/72376017-c0ab-41b3-a5bc-84927cb3e515/', u'enabled': True, u'quota': None, u'relative_path': u'var/archivematica/sharedDirectory/', u'purpose': u'CP', u'path': u'/var/archivematica/sharedDirectory', u'resource_uri': u'/api/v2/location/194600f8-0cd4-4f0c-89c3-534723a89b49/'}]
DEBUG 2019-10-03 23:01:07 archivematica.common:storageService:get_location:192: Storage locations returned: [{u'uuid': u'e4f7d748-f676-4e1c-92e3-adab814520f4', u'pipeline': [u'/api/v2/pipeline/42991a9e-30d8-4ded-b556-85168fb0e1ea/'], u'used': u'0', u'description': u'Apps', u'space': u'/api/v2/space/72376017-c0ab-41b3-a5bc-84927cb3e515/', u'enabled': True, u'quota': None, u'relative_path': u'apps', u'purpose': u'TS', u'path': u'/apps', u'resource_uri': u'/api/v2/location/e4f7d748-f676-4e1c-92e3-adab814520f4/'}]
DEBUG 2019-10-03 23:01:07 archivematica.mcp.server:package:_copy_from_transfer_sources:220: source: Images/., destination: /var/archivematica/sharedDirectory/tmp/tmpZfBHad/test1234/.
DEBUG 2019-10-03 23:01:08 archivematica.mcp.server:package:_start_package_transfer_with_auto_approval:422: Package 100f3026-c70b-471b-9603-743463f5b0d3: moving package to processing directory
ERROR 2019-10-03 23:01:08 archivematica.mcp.server:package:wrap:371: Exception: Filepath /var/archivematica/sharedDirectory/tmp/tmpZfBHad/test1234 does not exist.
Traceback (most recent call last):
File "/usr/lib/archivematica/MCPServer/package.py", line 350, in wrap
fn(*args, **kwargs)
File "/usr/lib/archivematica/MCPServer/package.py", line 424, in _start_package_transfer_with_auto_approval
filepath, django_settings.PROCESSING_DIRECTORY, transfer
File "/usr/lib/archivematica/MCPServer/package.py", line 247, in _move_to_internal_shared_dir
raise Exception(error)
filepath, django_settings.PROCESSING_DIRECTORY, transfer
File "/usr/lib/archivematica/MCPServer/package.py", line 247, in _move_to_internal_shared_dir
raise Exception(error)
Exception: Filepath /var/archivematica/sharedDirectory/tmp/tmpZfBHad/test1234 does not exist.
--
You received this message because you are subscribed to the Google Groups "archivematica" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archivematic...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archivematica/ae3d5c4e-2a37-4274-831a-d3ac4b221b4e%40googlegroups.com.
Jamen
Miguel Angel Medinilla Luque
Oct 3, 2019, 8:43:48 PM10/3/19
to archivematica
Hi again,
The above log says:
The above log says:
DEBUG 2019-10-03 23:01:07 archivematica.common:storageService:get_location:192: Storage locations returned: [{u'uuid': u'194600f8-0cd4-4f0c-89c3-534723a89b49',
This is the UUID of the Currently Processing location in the Local Filesystem space, it should be the other Currently processing location (Pipeline Filesystem Space).
Thanks
To unsubscribe from this group and stop receiving emails from it, send an email to archiv...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archivematica/ae3d5c4e-2a37-4274-831a-d3ac4b221b4e%40googlegroups.com.
--Jamen
Jamen McGranahan
Oct 3, 2019, 8:44:44 PM10/3/19
to archivematica
I have finally given up and have resorted to adding the Storage Service to the Archivematica server itself. It's not ideal, but at least it is working now. Thanks for the advice all.
-Jamen
On Wednesday, October 2, 2019 at 9:46:41 AM UTC-5, Jamen McGranahan wrote:
Reply all
Reply to author
Forward
0 new messages