Problem generating derivatives for some drush ingests, Islandora 7

[Howard Ding]

Hi,

We are experiencing problems with drush ingests that appears to be tied to the size of the ingest (we haven't narrowed down whether this means number of input items, number of objects created, or byte size, though), though it happens for relatively small ones.

This is an ISLE 1.5.0 installation (though I'll leave out the dockerish stuff below).

We are trying to ingest a set of ten pdf/xml pairs, the largest being under 10MB, with:

drush -v -u 1 --uri=https://digitalcollections.tricolib.brynmawr.edu islandora_paged_content_pdf_batch_preprocess --scan_target=/ingest/bryn_mawr/MCThomas_InHOR/batch1 --content_model=islandora:bookCModel --parent=bmc:mcthomas --namespace=bmc --text=none --type=directory

and

drush -v ibi --ingest_set=<set number>

This fails as explained below when doing it on all ten items, but succeeds with derivative generation if we split it into two parts of five each.

The first command above succeeds but the latter fails immediately at the start of derivative generation with a 401 error coming from Fedora.

Ingested bmc:2415. [ok] WD islandora: Failed to ingest datastream TECHMD on object: [error] bmc:2339code: 401message: WD islandora: Failed to ingest datastream OCR on object: [error] bmc:2339code: 401message: HttpConnectionException in [error] /var/www/html/site/sites/all/libraries/tuque/HttpConnection.php:486 Stack trace: #0 /var/www/html/site/sites/all/libraries/tuque/HttpConnection.php(615): CurlConnection->doCurlRequest()

This is reminiscent of this issue, though we have confirmed that we do not have the conditions that caused it:

https://jira.lyrasis.org/browse/ISLANDORA-1648

The failure at the beginning of derivative generation with the same error suggests that it could be coming from a similar problem with the connection. I have trouble conceiving why this would pop up for the single ingest but not for the split ones.

We find that if we run the same ingests via the UI they work, but our librarians require the ability to do drush ingests and have little interest in splitting each into tiny parts.

Any ideas most welcome.

Thanks,

Howard

--

Howard Ding

TriCo Digital Library Developer

610-526-7554; hdi...@brynmawr.edu; had...@gmail.com