Total digital object upload amount not showing up in under archival institution
39 views
Skip to first unread message
Dallas Hanbury
Nov 4, 2019, 11:52:03 AM11/4/19
to AtoM Users
Hi all,
I cannot see the total amount (in terms of gigabytes and megabytes) of digital uploads under my archival institution. It shows a total amount of .10 gigabytes uploaded. The actual amount uploaded is somewhere around 3 gigs. For a few collections we erroneously inputted the repository information (small differences here and there). That led to the creation of multiple repositories. We are cleaning up our data, and as part of that process, have edited the repository information in each collection to be identical. I have scanned the group, the wiki, and everything else I can think of, but cannot find the answer. Any help would be greatly appreciated.
Using the newest version of Internet Explorer
Not sure what version of AtoM we are using, I believe 2.4 (I am not sure how to access this information as I am a relatively new AtoM user)
AtoM is the host.
Corinne Rogers
Nov 4, 2019, 12:28:06 PM11/4/19
to AtoM Users
Hi Dallas,
When you log in to your site as an administrator, go to the gear icon on the top menu bar: Admin > Settings. On that global settings page you will see the version of AtoM you are using, and about 2/3 down the page you will see the amount of digital object storage used - currently showing as 2.3 GB of 10. This matches the reports that our Ops team get on digital object storage. I am not sure why this total (which is correct) doesn't agree with the total on the repository page, although perhaps this has something to do with the data cleanup you have been doing. I recommend you check the admin page for accurate information. If you are having any trouble logging in, let us know.
best regards,
Corinne
Artefactual Systems
Dan Gillean
Nov 4, 2019, 1:01:33 PM11/4/19
to ICA-AtoM Users
Hi Dallas,
I checked in with one of our developers on how all of this works, and wanted to supply a bit more information, for future reference on related questions.
First, of all, as noted in the documentation, the total amount of available storage shown on the Global Settings page is determined by a value set in one of AtoM's configuration files. Specifically, this is set in the config/app.yml file, via the upload_limit variable. If this value is set to -1, then there is no set limit in AtoM on the max file size, and AtoM's settings will just say "Unlimited".
Note that upload file sizes can be limited by other installation variables, such as the PHP execution limits, and individual repository upload limits - see this previous forum post for further links:
In terms of how AtoM actually calculates the size of the files:
First, it's important to note that this calculation is only based on the uploads directory in AtoM - so it does not include uploaded finding aids for example, which (somewhat confusingly), are stored in AtoM's downloads directory. Additionally, if you are using a URL to link a digital object in AtoM, AtoM will not store a copy of the master digital object - it will generate local derivatives (such as a thumbnail and reference display copy, but it will just save the URL to the master, rather than copying and storing it.
Another thing to note: if you have deleted your duplicate repositories and their related descriptions, then it is also possible that the attached digital objects have been deleted as well.
When calculating the total space used, AtoM is relying on PHP functions to get the value. Specifically, the call is made in the code here:
It uses a function called getDirectorySize, which is here:
These operations make use of the following PHP function:
I don't see any changes in this PHP function in recent versions, so I don't think there is a regression there. As far as I'm aware, the application cache and the index are not involved in this operation as well, so I don't think the usual maintenance tasks would change anything. If you are seeing inaccurate or unexpected amounts, the only theories I have at this point are:
- There could be a permissions issue that is preventing the function from properly calculating and returning the correct value. You can always try resetting the permissions to what AtoM needs by running the following: sudo chown -R www-data:www-data /usr/share/nginx/atom
- It can take a while for this calculation to be run if you have a lot of digital objects. You could try waiting a bit and doing a hard refresh on the settings page to see if the used total changes.
Another option, for an independent check, would be to run a calculation against the uploads folder yourself via the command line. You could do this for example with the following command:
- du -sh /usr/share/nginx/atom/uploads/
Alternatively, to see the size of all of AtoM's subdirectories (including both uploads and downloads), you could navigate to AtoM's root installation directory (typically /usr/share/nginx/atom if you have followed our recommended instructions) and then run:
- du -sh *
See:
--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/edd06c26-acc3-4141-93fd-ac4def90f5f7%40googlegroups.com.
Dan Gillean
Nov 4, 2019, 1:06:07 PM11/4/19
to ICA-AtoM Users
One more thing to mention:
In the uploads directory, digital objects are organized first by repository, and then by a number of subdirectories based on a hash value. I've outlined this here in the forum previously:
However, if you change the linked repository, as far as I'm aware, the digital objects are NOT moved to a new subdirectory. So if you accidentally created some duplicate repositories, and then changed which repo your descriptions with digital objects were linked to, this could be why the per-repository uploads total is incorrect.
Cheers,
Reply all
Reply to author
Forward
0 new messages