Reference representation not created automatically

238 views
Skip to first unread message

Josef Vítovec

unread,
Oct 27, 2020, 4:40:58 PM10/27/20
to AtoM Users

Hi,

we use Archivematica to generate DIPs and upload them to AtoM which works well itself but in AtoM a reference representation is not automatically created. DIPs contain mostly a video file (.mov) and on created archival description's page we can see just a huge icon representing a file and "Download movie" link below. By clicking the link our video is played but in a separate window which is not part of AtoM anymore. Yes we can still manually upload a reference representation on "Edit digital object" page and then the video can be played straight from AtoM but that's not how the process should be I think. Is to possible to make it automatically somehow? Also our upload limit in AtoM is currently 512 MB and the testing .mov video file is about 130 MB so that should not be the issue.

Thanks

Josef


Dan Gillean

unread,
Oct 27, 2020, 5:03:29 PM10/27/20
to ICA-AtoM Users
Hi Josef, 

The .mov format is an Apple QuickTime format, and I believe that AtoM's media player does not natively support this format. You can see a list of supported formats here in our documentation: 
I would suggest that you use Archivematica's Format Policy Registry to generate MP4 files as your normalized video access copies instead. The same underlying structure and compression algorithms are used (in fact I believe that MOV was the basis of MP4's development), but MP4 has become a more widely supported de facto industry standard for video container formats. If you need help configuring this, I recommend making a post in the Archivematica user forum. 

One thing you could try is trying to regenerate the access derivatives for video files in AtoM - this may resolve your issues without having to alter or re-upload existing content. See: 
For example, you could first use the --slug option to test it on one single description with a MOV file attached: 
  • php symfony digitalobject:regen-derivatives --index --slug="the-jane-doe-fonds"
Be sure to replace "the-jane-doe-fonds" with the actual slug of your target description. If it works well, then you can run the general command, targeting only videos, like so: 
  • php symfony digitalobject:regen-derivatives --media-type="video"
Note that, with this second command example, you would need to reindex your site after. If you would like to index progressively as the task progresses, the task will take longer, but it will save you that extra step, which ultimately could take longer to run if you have a lot of other content in your site. To include indexing in the command, include the --index option, like so: 
  • php symfony digitalobject:regen-derivatives --index --media-type="video"
Let us know how it goes!

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him


--
You received this message because you are subscribed to the Google Groups "AtoM Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to ica-atom-user...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/d703f8d4-098e-44eb-bea0-f51a75c028a5n%40googlegroups.com.

Josef Vítovec

unread,
Oct 27, 2020, 9:16:38 PM10/27/20
to AtoM Users
Thanks for the quick reply,

sorry I was mistaken about the formats. Actually, we use .movs only as source files but then we normalize them to .mp4s to be easily accessed in AtoM. So basically there are only .mp4s in AtoM, yet still it's not possible to play these .mp4s straight on the archival description detail page since the reference representation is empty. I tried to regenerate the access derivatives as you suggested, it regenerates them successfully but with no effect on the reference representation – it's still empty.

Dne úterý 27. října 2020 v 22:03:29 UTC+1 uživatel Dan Gillean napsal:

Dan Gillean

unread,
Oct 28, 2020, 6:09:24 PM10/28/20
to ICA-AtoM Users
Hi Josef, 

Thank you for the clarification. 

What version of AtoM are you using?

Also, is the behavior any different if you try uploading a small mp4 directly via AtoM?

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/54a9a796-1617-40a8-aa53-d4f245dd79d0n%40googlegroups.com.

Josef Vítovec

unread,
Oct 28, 2020, 7:01:20 PM10/28/20
to AtoM Users
Hi Dan,

we are using version 2.6.0.

When I upload .mp4 video file (~ 12 MB) using Link digital object option, the reference representation is still empty. It behaves the same way as it was created via Archivematica. Thumbnail representation is always generated successfully.

FYI when I click on Import digital objects option it gives me this flash player error:

Your browser does not have Flash player installed. To import digital objects, please Download Adobe Flash Player (requires version 9.0.45 or higher)

which is weird since I thought you introduced an HTML5 player in version 2.6.

Thank you for your help,

Josef

Dne středa 28. října 2020 v 23:09:24 UTC+1 uživatel Dan Gillean napsal:

Dan Gillean

unread,
Oct 29, 2020, 9:22:12 AM10/29/20
to ICA-AtoM Users
Hi Josef, 

I've run a few tests in a local stable/2.6.x test instance, and in each case with local file uploads AtoM has successfully created a display copy that I can play immediately in AtoM's media player. Thus far, I don't think this is a bug in 2.6 - I suspect there might be some configuration issue in your installation we can identify and solve. 

Have you followed our recommended installation instructions for 2.6 (e.g. PHP 7.2, Ubuntu 18.04, MySQL 8.0, Elasticsearch 5.6, etc)? If no, what have you changed?

Did you install all the additional packages used for managing digital objects during installation? See: 
Another thing we could check are the filesystem permissions. AtoM expects all its subdirectories and files to be owned by the www-data user. You can reset the filesystem permissions with the following command, if you've followed our recommended installation instructions: 
It may also make sense to double-check your user group permissions if you've made any adjustments to the default groups. For example, does the authenticated group have permission to view the reference display copies? Does the administrator's group? 

authenticated-view-reference.png

See: 
Finally, have you tried clearing the application cache and restarting PHP-FPM to see if there's a caching issue preventing you from seeing the latest version of the description with the digital object? From AtoM's root installation directory (typically /usr/share/nginx/atom), try: 
  • php symfony cc
  • sudo systemctl restart php7.2-fpm
See: 
Remember as well that your web browser also has its own cache. You may want to clear this, or else test in an incognito / private browser window! 

Let us know if any of this resolves the issue. If not, then at least we've ruled out some common solutions! 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/266556b3-f102-494a-84ab-68a42a5c9b9fn%40googlegroups.com.

Josef Vítovec

unread,
Oct 30, 2020, 6:28:12 PM10/30/20
to AtoM Users
Hi Dan,

our installation setup includes:
I checked all the things you mentioned – additional packages, filesystem permission, user group permissions and also tried to clear the application cache and restarting PHP-FPM. However, nothing has changed, reference representation is still empty.

Regards,

Josef

Dne čtvrtek 29. října 2020 v 14:22:12 UTC+1 uživatel Dan Gillean napsal:

Dan Gillean

unread,
Nov 2, 2020, 1:52:48 PM11/2/20
to ICA-AtoM Users
Hi Josef, 

I'm not sure what to suggest next, to be honest.  It's possible that there is some CentOS library/package compatibility issue occurring somewhere - we don't test CentOS installations, so there are some limits to the support we can provide. However, there are several community-supplied installation guides listed on our wiki - perhaps you could check your installation against those and see if you notice any differences, missing packages, etc. See: 
Regards, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/fd627bf7-13da-4825-9cc7-d6be58439081n%40googlegroups.com.

Josef Vítovec

unread,
Feb 17, 2021, 11:47:08 AM2/17/21
to AtoM Users
Hi Dan,

I have some updates on this topic and also new issues that we encountered. I don't want to create a separate thread since it's still connected to generating reference representations/thumbnails.

Regarding the video reference representations, we found out that for some videos (.mkv and some .mp4) AtoM successfully generates a reference representation. Though for some .mp4s and .movs it's still not working and we have to click on the "file icon" and then a new tab with a video player is opened. So it's still possible to view them at least. However, we'd like to see them right on the archival description page. When I try to generate the derivatives manually it shows:

>> php72 symfony digitalobject:regen-derivatives --slug="dist-ar-more-v-zrcadle-01-mov-2"
                                                                              
  Continuing will regenerate the derivatives for ALL descendants of           
  "dist-ar-more-v-zrcadle-01-mov-2"                                           
  This will PERMANENTLY DELETE existing derivatives you chose to regenerate   
                                                                              
  Continue? (y/N)                                                             
                                                                              
y 

>> digital object Regenerating derivatives for d84ee940-ea84-46cb-a3f0-bba3a657f49e-DIST-AR-more_v_zrcadle_01.mp4... (1.51s)
>> digital object Please update the search index manually to reflect any changes
>> digital object Done!

Regenerating itself took only 1.51s which is quite weird with the 1.2 GiB file. Anyway, then I update the search index:

>> php72 symfony cc && php72 symfony search:populate

And after all these steps the description is still without reference representation. Why did the regenerating complete without any errors when obviously it's not working correctly? Is there any place where we can at least check logs regarding the generating or something?


We're also struggling with generating reference representation for TIFF files. We have ImageMagick installed on our server:

>> /usr/bin/convert --version
Version: ImageMagick 6.9.10-68 Q16 x86_64 2020-12-15 https://imagemagick.org
Copyright: © 1999-2019 ImageMagick Studio LLC
License: https://imagemagick.org/script/license.php
Features: Cipher DPC Modules OpenMP(3.1)  
Delegates (built-in): bzlib cairo fontconfig freetype gslib jng jp2 jpeg lcms ltdl lzma openexr pangocairo png ps rsvg tiff wmf x xml zlib

Also, it shows that the TIFF delegate is present. However, when I try to regenerate derivatives for TIFF it shows this error:

>> php72 symfony digitalobject:regen-derivatives --slug="tiff-test"
                                                                              
  Continuing will regenerate the derivatives for ALL descendants of           
  "tiff-test"                                                                 
  This will PERMANENTLY DELETE existing derivatives you chose to regenerate   
                                                                              
  Continue? (y/N)                                                             
                                                                              
y
>> digital object Regenerating derivatives for file_example_TIFF_1MB.tiff... (2.13s)
identify: no decode delegate for this image format `TIFF' @ error/constitute.c/ReadImage/562.
convert: no decode delegate for this image format `TIFF' @ error/constitute.c/ReadImage/562.
convert: no images defined `jpeg:-' @ error/convert.c/ConvertImageCommand/3285.
identify: no decode delegate for this image format `TIFF' @ error/constitute.c/ReadImage/562.
convert: no decode delegate for this image format `TIFF' @ error/constitute.c/ReadImage/562.
convert: no images defined `jpeg:-' @ error/convert.c/ConvertImageCommand/3285.
>> digital object Please update the search index manually to reflect any changes
>> digital object Done!

It seems that the TIFF delegate is not present but as shown above, it is actually there. So I'm curious is AtoM using the right ImageMagick or are there any other versions available on our server? Is it possible to somehow check which ImageMagick is AtoM using? Or do you have any ideas why the error occurred?


Thank you for your time.

Regards,

Josef

Dne pondělí 2. listopadu 2020 v 19:52:48 UTC+1 uživatel Dan Gillean napsal:

Dan Gillean

unread,
Feb 23, 2021, 5:44:31 PM2/23/21
to ICA-AtoM Users
Hi Josef, 

I did a quick test in my local AtoM instance and I could upload a TIFF and have JPGs generated for the derivatives without issue. The imagemagick version I have installed is the same as yours (6.9.10). I can only assume that there is some missing dependencies for CentOS. 

I took a look at one of the guides available on our wiki for CentOS installations. The RedHat 7.5 guide includes mention of an additional dependency, rpmfusion, that is needed for ffmpeg to work as expected in RHEL/CentOS environments. See: 
Alternatively, an older guide listed there for RHEL/CentOS 7.1, with a PDF version linked in the user forum (here), includes a number of additional packages and configurations for ffmpeg on page 5. 

So... hopefully one of these suggestions might help with the video derivatives?  

 As for the issue with the TIFF derivatives, I'm really not sure! I can't see any additional packages involved in most of the community guides, and as I said, our ImageMagick versions are the same. It seems to work in a typical installation that follows our recommended installation guidelines, so I assume there's some configuration issue going on, but I'm not sure what. 

Perhaps check the filesystem permissions? Make sure that all PHP extensions (including php-gd) are properly installed? I will see if our team has any other ideas, but hopefully looking into the above might help move you towards a solution. 

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/ec603a42-e807-4990-8084-76b590424f10n%40googlegroups.com.

Josef Vítovec

unread,
Feb 25, 2021, 12:11:31 PM2/25/21
to AtoM Users
Hey Dan,

thanks a lot for your recommendations, we tried to install some of the additional ffmpeg dependencies mentioned in both guides and derivatives generating now works!

And regarding the ImageMagick, it was a misconfiguration on our side. We really had two versions running on the server and AtoM was using the wrong version with almost no delegates. So we just deleted the wrong version and it works too!

So thank you again for this, you saved us a lot of time :)

Regards,

Josef Vítovec

Dne úterý 23. února 2021 v 23:44:31 UTC+1 uživatel Dan Gillean napsal:

Dan Gillean

unread,
Feb 25, 2021, 3:24:01 PM2/25/21
to ICA-AtoM Users
Hi Josef, 

I'm really glad to hear that - thank you for posting an update and letting us know what worked for you!

Cheers, 

Dan Gillean, MAS, MLIS
AtoM Program Manager
Artefactual Systems, Inc.
604-527-2056
@accesstomemory
he / him

To view this discussion on the web visit https://groups.google.com/d/msgid/ica-atom-users/0b244b28-bca6-40ef-aad2-09673b3864cfn%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages