Re: [atom-users] Thumbnails missing after move to new server and upgrade

[Dan Gillean]

Hi Jimmy, 

I'm not sure what has caused the issue but you could: 

• Ensure that the filesystem permissions are properly set for all levels with: sudo chown -R www-data:www-data /usr/share/nginx/atom
• Run the derivatives regeneration command - there are several options, so if for example the reference display copies (shown on the view page of related descriptions) are okay, you can limit the task to just regenerating thumbnails. See: https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/cli-tools/#regenerating-derivatives
• Run the standard tasks after like clearing the application cache, restarting PHP-FPM, etc. Remember that your web browser has its own cache, so you might want to clear this or else test in an incognito/private browser window (where the cache is typically disabled by default) first! 

Hopefully that might resolve the issue? Let us know how it goes!

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

On Tue, Nov 2, 2021 at 4:41 AM Jimmy Thelander <jimmy.t...@gmail.com> wrote:

Hi,

We just recently moved our AtoM installation to a new server and upgraded to 2.6.4 (v184). Everything went fine, except that some of our thumbnails no longer show up properly under digital objects. On the old installation it works fine, on the new one only some show up properly. 

The actual files in /uploads/ are there, but they don't seem to be associated to the object somehow. I've run all the usual symfony tasks but the issue is still there. 

Any ideas what could cause this?

Best regards,

Jimmy 

--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/0c4e2060-79ab-4572-900f-5039255b3a26n%40googlegroups.com.

[Dan Gillean]

Hi Jimmy, 

Let  me just try to rule out a few obvious things. 

When trying to access those images and their related URIs, are you logged into AtoM? Are the related descriptions published?

Have any changes been made to the default group permissions (for the anonymous group, the authenticated group, etc) in Admin > Groups that might be affecting this? What about the use of PREMIS rights as a way of restricting digital object access?

I noticed that the two examples you provided are associated with different archival institutions. Does the problem delineate clearly along these lines - i.e. are all the images that are not working associated with descriptions from one particular archival institution? Has the related repository record name changed at all recently (wondering if there could be some kind of mismatch causing the issue)?

In the user interface on one of the related descriptions, if you open the "More" menu and select Edit digital object, you should have the option to delete the related thumbnail. If you return to the Edit digital object page after, there should be an option to either upload your own thumbnail, or automatically re-generate the thumbnail from the original upload. Do these options work as expected for you?

If no, are there any related messages in the webserver error logs? See: 

• https://www.accesstomemory.org/docs/latest/admin-manual/maintenance/logging/#web-server-logs
  

What version were you upgrading from during this server migration? Is there any chance that some step in the upgrade process was missed? The two most common missed steps are dropping and recreating the database, and running the upgrade task after loading it. Missing one of these may lead to an installation that appears to be working properly at first, but can lead to unexpected issues down the road! 

Hopefully by asking these questions, we might shake out some further details that will help our team provide further ideas for troubleshooting. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

On Tue, Nov 2, 2021 at 12:49 PM Jimmy Thelander <jimmy.t...@gmail.com> wrote:

Hi Dan,

Thanks for taking the time to reply!

I made sure

• the permissions properly set
  
• the derivatives are regenerated
• and that I rerun the standard tasks, restarted the services, etc and I've tried with with the cache cleared in my browser

I've found out that the images that don't work can't be accessed directly, even though the file exists on the server on the exact path (i.e. https://arken.kb.se/uploads/r/kungliga-bibliotekets-handskriftssamling/b/2/0/b204892e00b4a41bc0dc0659d1e13494f7b530d4903dcb9de9d1f63c79eb3631/002696458_1_i_1.jpg), but images that do work in Access to Memory can be accessed in the same way (i.e. https://arken.kb.se/uploads/r/umea-universitetsbibliotek/0/6/5/065c580f55b8de6fd92e4f3c78063b8baab867f1257fe1213078ca85811b6007/handskrift_12_1.jpg). Both files exist at the path in the URL. 

Maybe this is a clue to why this is? Any other ideas?

Thanks again!

Best regards,

Jimmy

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/836cfb3d-18fe-4fbf-869a-b52bd6aa872fn%40googlegroups.com.

[Dan Gillean]

Hi Jimmy, 

I will consult my team and see if we can come up with further suggestions. Given the example provided, I'm now not surprised there is no thumbnail (more on this in a moment), but the fact that you can't get to the digital object edit page suggests something else is going on as well. 

Regarding the example here: https://arken.kb.se/se-q-handskrift-7a-f-3-a 

This digital object appears to be linked via URI to: https://digital.ub.umu.se/resolve?urn=urn:gha_000031

However, this does not meet the documented requirements that AtoM expects from remote digital objects to be able to successfully link the content and generate a derivative. For AtoM to be able to do so, the URI must: 

1. Be available on the public web - no passwords, VPNs, confirmation screens, etc. should be required or else AtoM will not be able to fetch the digital object
2. Use HTTP or HTTPS - local share drive links, FTP or SFTP links, etc will not work
3. End in the file extension of the digital object (e.g. .jpg, etc). That is, the URI needs to link directly to the target digital object, and not to a landing page

It is this third criteria that your URI appears to be missing. When I follow the link, I am taken to a landing page for a compound digital object made up of individual scans, with separate JPGs for the first 40 pages showing on page 1 of this landing page. 

To create a derivative, AtoM needs to be able to fetch a copy of the target digital object, store it locally in a temporary location, and then use that to create the reference display copy and thumbnail via tools like imagemagick. In this case, there are 40+ choices on the page. AtoM's link digital object via URI functionality doesn't have the capacity to semantically parse the contents of the provided URI, find a digital object, and then follow that link - it's very simple, and if the provided URI doesn't point directly to the target digital object, then the derivative generation will fail. 

Instead, you would need a URI that points to the target itself - for example, if the PDF option on the landing page were pre-generated rather than created by the site on the fly, then linking to the PDF version might work, since it's possible to get to a URI that ends in .pdf: 

• https://digital.ub.umu.se/profiles/nainuwa/libraries/pdf.js/web/viewer.html?file=/admin/nainuwa/pdf/download/118987%3F%E2%80%9DGrindbygubbens%20minnen%E2%80%9D%20Avskrift.%20Gunnar%20Hallstr%C3%B6m%20-%20120dpi.pdf
  

Alternatively (although I doubt this would be desirable for a multi-page digital object), it appears possible to get to suitable URIs for the individual pages, like this one: 

• https://digital.ub.umu.se/image_server_router/gha_000031/0005.jpg
  

This third criteria is flagged in the related documentation in an "Important" admonition below step 5 of the "Link a single digital object" instructions, in this section: 

• https://www.accesstomemory.org/docs/latest/user-manual/import-export/upload-digital-object/#link-a-single-digital-object-to-an-archival-description
  

I will review our documentation and consider how to make the criteria for remote digital objects more explicit for 2.7. 

It's possible that in the past, someone uploaded their own custom derivatives - e.g. a screenshot of the target landing page, or of the first page of the document. However, running the derivative regeneration task would mean that any previous custom manual derivative uploads would be overwritten by AtoM's attempt to regenerate the derivatives. So: if it was working before, it may be because the old site still has custom derivatives in place, but your upgrade process removed those. My theory is then that AtoM is trying to access the JPG derivative that was previously there, but it no longer is - hence the 404 in the error logs. 

Manually adding custom image derivatives to external digital object URLs that don't meet the linking criteria is a workaround I have seen some community members use before - for example, linking to a YouTube video would have the same problems as what you are experiencing (YouTube purposefully obfuscates access to the URL pointing directly to the MP4 to prevent unauthorized downloads, so a YouTube link doesn't work in AtoM in the same way), but I've seen users add video links and then provide their own screenshots of the video to use as the reference copy and thumbnail. There is a tradeoff to this approach however - the media type of the digital object will be wrong if you want your thumbnail to show, which can impact search and browse: 

• If you manually set the media type to "video" then AtoM will try to display the media player on the description view page - meaning your custom thumbnail won't show, and the video player will be empty, and only the authenticated users will see a "Download" link that will actually link out to the YouTube page
• To get the custom derivatives to show, you need to set the media type to "Image" (or if I recall correctly, "Other" will work too) - but then when using the Media Type facet in search and browse, your description will not be included in results limited to "video." 
• Public users will also need to either use the URI in the digital object metadata section (which some public users will not see), or else the "anonymous" user group (i.e. public users) need to be given  "View master" permissions so they can click through the derivatives to the source link

So: yes, in this particular case, it's possible that running the derivative regeneration as part of your upgrade process has caused this issue. 

If you were happy with how things were working previously and you still have your 2.5 installation and backup data, then one thing you could try now would be replacing the current 2.6 uploads directory with a fresh copy from your 2.5 installation, and not running the derivative regeneration task this time. You would still want to run several other maintenance tasks after replacing it - I would clear the application cache, restart PHP-FPM, and possibly reindex the site as well to see if the replacement worked. 

In the meantime, I will see if our team has any other suggestions. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056

@accesstomemory

he / him

On Wed, Nov 3, 2021 at 6:46 AM Jimmy Thelander <jimmy.t...@gmail.com> wrote:

Hi Dan,

Logged in or not the images are not accessible. The descriptions are published.

As far as I can tell, no changes have been made to the default group permissions. The permissions are the same as on the old server running version 2.5.3 (172). The images are accessible on the old server but not on the new one. The files themselves are located in the same place on both servers. The database was dumped from the old server and imported on the new one and the the database upgrade task was run. Nothing else has changed.

It more ore less delineates clearly along these lines, but not entirely. Tthis description for example is from the "working" archival institution, but is not working on the new installation: https://arken.kb.se/se-q-handskrift-7a-f-3-a, but is working on the old one.

I'm able to delete the related thumbnail (the generic thumbnail). I'm not able to return to the "Edit digital object" page after deleting it. I can only link a digital object or import a digital object to the description. I can't find any option to automatically regenerate the thumbnail.

When trying to retrieve an image that isn't working, the error log shows the following:

2021/11/03 11:23:37 [error] 2134#0: *724 FastCGI sent in stderr: "PHP message: This request has been forwarded to a 404 error page by the action "digitalobject/view"" while reading response header from upstream, client: 192.168.0.101, server: arken.kb.se, request: "GET /uploads/r/Kungl-bibliotekets-handskriftssamling/9/1/e/91e0e8a87d0c85d59bec49709e2cb89c97498463200e1c3baa7a9a10de3edaa6/002696456_1_i_1.jpg HTTP/2.0", upstream: "fastcgi://unix:/run/php7.2-fpm.atom.sock:", host: "arken.kb.se"

We're upgrading from version 2.5.3 (172) to 2.6.4 (184), and at the same time moved to a new server running a newer version of the operating system as required. The installation was done by following the instruction for a new installation on Linux, and then following the instructions for upgrading. The asynchronous workers are running as systemd services. I ran "php symfony digitalobject:regen-derivatives" as part of the upgrade process, could this somehow have messed up the images?

Thanks once again!

Best regards,

Jimmy

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/7e5ee76b-d421-4397-9058-b9bf5048afe4n%40googlegroups.com.