Batch ingest error

[Brandon Weigel]

Hello all,

If anyone is an expert at batch ingests via the command line, I'd appreciate some advice. I'm testing out a batch ingest, following the directions on the islandora_batch Github page: https://github.com/Islandora/islandora_batch

I'm in my Vagrant machine, and I have my zip file in the directory i'm currently in (/var/www/drupal/sites). Zip file is "unbc.zip".

I run my preprocessing command:

drush -v -u 1 --uri=http://localhost islandora_batch_scan_preprocess --type=zip --target=unbc.zip

And I get the following results:

ZipArchive::open(): Empty string as source                           [warning]

islandora_scan_batch.inc:87

Invalid argument supplied for foreach() islandora_scan_batch.inc:134 [warning]

SetId: 38                                                            [ok]

Command dispatch complete                                               [notice]

When I check the batch sets for SetID 38, the set is there, but it has zero items. Continuing the process results in nothing happening, of course. I checked my zip file, and it definitely has items in it: 1.1 GB worth. File contains a large set of files, one XML and one PDF, all named appropriately, all in the root of the file.

What am I missing/doing wrong?

[Brandon Weigel]

Also misunderstood the documentation, but fixing the command to include all the pieces I needed still resulted in same error.

Modified command with more parameters:

drush -v -u 1 --uri=http://localhost islandora_batch_scan_preprocess --content_models=ir:thesisCModel --parent=unbc:dtcs --parent_relationship_pred=isMemberOfCollection --type=zip --target=unbc.zip

ZipArchive::open(): Empty string as source                           [warning]

islandora_scan_batch.inc:87

Invalid argument supplied for foreach() islandora_scan_batch.inc:134 [warning]

SetId: 39                                                            [ok]

Command dispatch complete                                               [notice]

[Brandon Weigel]

Sorry, fixed my own problem. Turns out the command requires the full path to the file, not relative to where you are.

Correct command was:

drush -v -u 1 --uri=http://localhost islandora_batch_scan_preprocess --content_models=ir:thesisCModel --parent=unbc:dtcs --parent_relationship_pred=isMemberOfCollection --type=zip --target=/var/www/drupal/sites/unbc.zip

[Peter Murray]

I remember falling into that problem myself; it it worth adding a note in the documentation?  (https://wiki.duraspace.org/display/ISLANDORA/Islandora+Batch)

Peter

--
For more information about using this group, please read our Listserv Guidelines: http://islandora.ca/content/welcome-islandora-listserv
---
You received this message because you are subscribed to the Google Groups "islandora" group.
To unsubscribe from this group and stop receiving emails from it, send an email to islandora+...@googlegroups.com.
Visit this group at https://groups.google.com/group/islandora.
To view this discussion on the web visit https://groups.google.com/d/msgid/islandora/bd632f12-e237-4f26-88d7-a27bc8529384%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Mark Jordan]

Brandon, your objects must be at the root of your zip; in other words, a directory within the zip that contains object will probably result in 0 objects. Check that your zip does not contain any directories (assuming a single-file content model like PDF, large image, etc), which it sounds like you're using since you're asking about Batch and not Newspaper Batch or Book Batch.

Mark

--

For more information about using this group, please read our Listserv Guidelines: http://islandora.ca/content/welcome-islandora-listserv
---
You received this message because you are subscribed to the Google Groups "islandora" group.
To unsubscribe from this group and stop receiving emails from it, send an email to islandora+...@googlegroups.com.
Visit this group at https://groups.google.com/group/islandora.

To view this discussion on the web visit https://groups.google.com/d/msgid/islandora/8a02b968-769d-4813-a5af-8a0520981ed4%40googlegroups.com.

[Brandon Weigel]

Yep, they are all at the root -- the problem was just that the tool required the full path to the zip file, rather than a relative path from where I was executing the command as I had assumed.

Peter -- I think it would be a useful note in the documentation. It's an easy trap to fall into.

[dp...@metro.org]

Hi, good you solved it! I agree with Peter, this needs some extra documentation! and also, maybe, some sanity checks on the code. Not cool that if some value is  not exactly as expected you end getting foreach loops on non existing arrays and that empty string as source. A "file not found" message would be better.

Would encourage you to open some Jira tickets. Is this the latest version? 1.9VM?

Cheers

Diego Pino

metro.org

[Brandon Weigel]

This is on Vagrant. I don't think changes have been made since my last update. I've made a little adjustment to the wiki's documentation of the --target parameter, but it should be documented on the Readme and, even better, make the code saner as you said. Sure, I'll open a ticket or two.

[Brandon Weigel]

Tickets created:

Documentation: https://jira.duraspace.org/browse/ISLANDORA-1932

Error handling: https://jira.duraspace.org/browse/ISLANDORA-1933

[dp...@metro.org]

Brandon++

[Brandon Weigel]

Hi all,

Once again, batch ingest error rears its ugly head. This time I really can't figure out why it's not working.

Command:

drush -v -u 1 --uri=http://localhost islandora_batch_scan_preprocess --content_models=ir:thesisCModel --parent=unbc:dtc --parent_relationship_pred=isMemberOfCollection --type=zip --target=/home/brandonw/stagedata/fullset/6-UNBC.zip

Same error:

Initialized Drupal 7.54 root directory at /var/www/drupal7              [notice]

Initialized Drupal site localhost at sites/default                      [notice]

Invalid argument supplied for foreach() islandora_scan_batch.inc:134 [warning]

SetId: 28                                                            [ok]

Command dispatch complete                                               [notice]

And a set containing zero files is generated.

ZIP file is correclty structured. Target path is correct, and begins at the root. I can't identify anything else that might be wrong with this command; aside from the slightly changed path to my source zip file, it's exactly the same as the command that worked before. The only difference I can note is that we recently moved to a new server, which included a Fedora version upgrade.

Any ideas what I'm doing wrong?

[Peter MacDonald]

I think if you are using drush version 7+,  --target= needs to be --scan_target=

Peter

--

For more information about using this group, please read our Listserv Guidelines: http://islandora.ca/content/welcome-islandora-listserv
---
You received this message because you are subscribed to the Google Groups "islandora" group.

To unsubscribe from this group and stop receiving emails from it, send an email to islandora+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/islandora.

To view this discussion on the web visit https://groups.google.com/d/msgid/islandora/19d77d03-36a4-4fea-9c25-ff2948dc3ee9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Peter MacDonald,
Library Information Systems Specialist
Hamilton College Library

Clinton, New York
315 859-4493

pmacdona-hamilton (Skype)

[p37]

Is your version of drush the same?   newer drush's use "--scan_target="

--

For more information about using this group, please read our Listserv Guidelines: http://islandora.ca/content/welcome-islandora-listserv
---
You received this message because you are subscribed to the Google Groups "islandora" group.

To unsubscribe from this group and stop receiving emails from it, send an email to islandora+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/islandora.

To view this discussion on the web visit https://groups.google.com/d/msgid/islandora/19d77d03-36a4-4fea-9c25-ff2948dc3ee9%40googlegroups.com.

[Brandon Weigel]

--scan_target fails; "Unknown option: --scan_target."

To unsubscribe from this group and stop receiving emails from it, send an email to islandora+...@googlegroups.com.

[Brandon Weigel]

Looks like the person who sent me the packages set up these ones differently from how he'd set up all the others that I had already ingested successfully. Unzipping them and processing directories instead of zips fixed the trouble.

Thanks for all the advice, folks.