regen-derivatives log

[PHXHoward]

Searched around before asking this question.

Hello I'm upgrading from AtoM 2.7 to 2.10.1 on a new server and decided to run

php symfony digitalobject:regen-derivatives 

I read that it is not necessary when going from version 2.7  but I thought why not do it anyway. Maybe it will fix something or uncover a problem.

It takes a very long time (whoops) but that's ok.  Problem is it generating some errors:

4 x 

openjpeg warning: unspec CS. 1 component so assuming gray

6 x 

Syntax Error: Expected the default config, but wasn't able to find it, or it isn't a Dictionary

There are thousands of digital objects that are regenerating successfully with only about 10 giving this notice. I could move on but curious if these errors are fixable?

In previous upgrades (years ago) I have a note to comment out a line in the ImageMagik-6 policy.xml pertaining to PDF to fix a similar problem but that note doesn't seem to apply here.

Thanks,

Howard

[Johan Pieterse]

Hi Howard,

Good news: neither of these is an AtoM problem or an upgrade regression, they're warnings from the underlying imaging tools about a handful of slightly non-standard source files, and they're non-fatal (the derivatives still generate). Thousands succeeding and ~10 warning is exactly the signature of a few odd files, not a config or version issue.

- openjpeg warning: unspec CS. 1 component so assuming gray, from OpenJPEG (ImageMagick's JPEG-2000 delegate). Those JP2 files declare no colourspace in the header and have a single channel, so it assumes grayscale.
Harmless if the originals are grayscale; only worth a look if one is actually colour and its thumbnail came out gray (then the source JP2 is malformed).
- Syntax Error: Expected the default config… isn't a Dictionary, from Poppler (pdftoppm), parsing a malformed object in a PDF. Also non-fatal; it normally still renders the page.

Your old ImageMagick policy.xml note doesn't apply here because that one fixes a different problem, ImageMagick refusing PDFs entirely because the PDF coder is disabled by the security policy, which fails hard on every PDF rather than warning on a few. So your instinct that it doesn't apply is right.

What I'd do: find those ~10 objects (the regen output names the object next to each warning) and eyeball their thumbnails/reference images. If they look right, ignore the warnings. If a specific one is blank or wrong, repair the source file rather than AtoM, then re-run regen for just that object:

# ── Locate & repair the ~10 non-conformant source files ──
# Point this at your digital-object storage root (e.g. uploads/r):
UPLOADS=/path/to/atom/uploads

# 1) Malformed PDFs  → the "Syntax Error: Expected the default config …" (Poppler) warnings
find "$UPLOADS" -type f -iname '*.pdf' -exec sh -c 'qpdf --check "$1" >/dev/null 2>&1 || echo "BAD PDF: $1"' _ {} \;

# 2) JP2s with no/odd colourspace  → the "openjpeg: unspec CS … assuming gray" warnings
find "$UPLOADS" -type f -iname '*.jp2' -exec sh -c 'identify -format "%[colorspace]  ch=%[channels]  %f\n" "$1" 2>/dev/null' _ {} \;

# 3) Repair ONE malformed PDF (normalise via Ghostscript; qpdf is the fallback)
gs -o /tmp/fixed.pdf -sDEVICE=pdfwrite -dPDFSETTINGS=/prepress "$BAD_PDF" && cp /tmp/fixed.pdf "$BAD_PDF"
# alt: qpdf --replace-input "$BAD_PDF"

# 4) Re-encode ONE JP2 with an explicit colourspace (or re-derive from the master)
convert "$BAD_JP2" -colorspace sRGB "$BAD_JP2"

# 5) Re-run derivative regen for just the affected object(s), by slug, and reindex.
#    Run as the web user, NOT root, so you don't leave root-owned files that break later writes:
sudo -u www-data php symfony digitalobject:regen-derivatives --slug=<object-slug> --index

Bottom line — safe to move on; nothing about the 2.7→2.10 upgrade is at fault.

Regards
Johan Pieterse
The Archive and Heritage Digital Commons Group (Pty) Ltd
https://theahg.co.za

[PHXHoward]

This is wonderful information Johan.  Thank you so much. We will move forward and later run those repair steps. Our organization is very happy with AtoM. My mother is the archivist. She promotes this software to everyone who will listen.  I've been the "tech guy" upgrading AtoM as needed each time the OS reaches its end of support for the past 14 years. Looking forward to this redeployment with version 2.10.1. We are now using a friendly login portal with Authelia and lldap to give our users access to AtoM.

Sincerely,

Howard