creating newspapers with Tuque

[Aaron Collie]

I'm creating a newspaper issue with tuque using the same method that I have used before to create PDFs, but I'm getting something odd.

1. Grab a PID

2. Create some datastreams and add content to those streams

3. Define RELS-EXT by mimicking an issue created through the GUI

4. ingest the new object

here is the pertinent code:

$arguments = $_SERVER['argv'];

$dcFile = $arguments[4];

$modsFile = $arguments[5]; // I'm passing in datastream paths this way for now

$marcxmlFile = $arguments[6];

$collection_pid = $arguments[7]; // I use this to select a namespace, then later to override for my test object

$connection = new RepositoryConnection($FEDORAURL, $FEDORAUSER, $FEDORAPASSWORD);

$connection->reuseConnection = TRUE;

$repository = new FedoraRepository( new FedoraApi($connection), new SimpleCache());

$my_islandora_tuque = new IslandoraTuque();

$repository = $my_islandora_tuque->repository;

content_models = array(array('pid' => 'islandora:newspaperCModel'));

$namespace = $collection_pid;

$collection_pid = "islandora:1"; // little overridding

$newspaper_issue = $repository->constructObject($namespace);

//$newspaper_issue['models'] = array('islandora:newspaperIssueCModel'); // old method?

$fedora_object->models = array('islandora:newspaperIssueCModel');

$newspaper_issue->label = 'change this string to mods/dc title variable'; // I actually add the label in a slightly different script

$newspaper_issue->owner = "fedoraAdmin"; 

$datastream_id = "DC";

$controlGroup = "X";

$dc_datastream = $newspaper_issue->constructDatastream($datastream_id, $controlGroup);

$dc_datastream->label = 'Dublin Core record of this object.';

$dc_datastream->mimetype = 'text/xml';

$dc_datastream->setContentFromFile($dcFile);

$newspaper_issue->ingestDatastream($dc_datastream);

$datastream_id = "MODS";

$controlGroup = "M";

$mods_datastream = $newspaper_issue->constructDatastream($datastream_id, $controlGroup);

$mods_datastream->label = 'MODS record of this object.';

$mods_datastream->mimetype = 'text/xml';

$mods_datastream->setContentFromFile($modsFile);

$newspaper_issue->ingestDatastream($mods_datastream);

$newspaper_issue->relationships->add(FEDORA_MODEL_URI, 'hasModel', 'islandora:newspaperIssueCModel');

$newspaper_issue->relationships->add(FEDORA_RELS_EXT_URI, 'isMemberOf', $collection_pid);

$new_fedora_object = islandora_add_object($newspaper_issue);

$pid = $newspaper_issue->id;

echo $pid;

When I navigate to the object I get this

RepositoryException: in RepositoryConnection->parseFedoraExceptions() (line 227 of /var/www/drupal-7.24/sites/all/libraries/tuque/RepositoryConnection.php).

This error doesn't occur with issues created through the GUI. I suspect some of my tuque code is outdated.

[Nigel Banks]

Other than the line below which shouldn't compile, I don't see any problems with your code.

content_models = array(array('pid' => 'islandora:newspaperCModel'));

 

Perhaps its the content of the datastream that is causing the problem? Can you provide the full error, it should include a stack trace.

[Aaron Collie]

It's taken me a while to get back to working on this newspaper collection. Here is the full error:

ERROR 2014-02-20 11:55:04.203 [http-8080-6] (RISearchServlet) Unexpected error servicing API-A request

org.trippi.TrippiException: Query failed: No such variable $issued in tuples [$pid, $label, $sequence] (class org.mulgara.resolver.AppendAggregateTuples)

at org.trippi.impl.mulgara.MulgaraSession.query(MulgaraSession.java:167) ~[trippi-mulgara-1.5.8.jar:na]

at org.trippi.impl.base.ConcurrentTriplestoreReader.findTuples(ConcurrentTriplestoreReader.java:80) ~[trippi-core-1.5.8.jar:na]

at org.fcrepo.server.resourceIndex.ResourceIndexImpl.findTuples(ResourceIndexImpl.java:280) ~[fcrepo-server-3.7.0.jar:na]

at org.fcrepo.server.resourceIndex.ResourceIndexModule.findTuples(ResourceIndexModule.java:313) ~[fcrepo-server-3.7.0.jar:na]

at org.trippi.server.TrippiServer.find(TrippiServer.java:119) ~[trippi-core-1.5.8.jar:na]

at org.fcrepo.server.access.RISearchServlet.doFind(RISearchServlet.java:356) [fcrepo-server-3.7.0.jar:na]

at org.fcrepo.server.access.RISearchServlet.doSearch(RISearchServlet.java:257) [fcrepo-server-3.7.0.jar:na]

at org.fcrepo.server.access.RISearchServlet.doGet(RISearchServlet.java:206) [fcrepo-server-3.7.0.jar:na]

at org.fcrepo.server.access.RISearchServlet.doGet(RISearchServlet.java:125) [fcrepo-server-3.7.0.jar:na]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) ~[servlet-api.jar:na]

at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) ~[servlet-api.jar:na]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) ~[catalina.jar:6.0.35]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) ~[catalina.jar:6.0.35]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369) ~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.fcrepo.server.security.jaas.AuthFilterJAAS.doFilter(AuthFilterJAAS.java:329) ~[fcrepo-security-jaas-3.7.0.jar:na]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) ~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:109) ~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381) ~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:168) ~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) ~[spring-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) ~[spring-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) ~[catalina.jar:6.0.35]

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) ~[catalina.jar:6.0.35]

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) ~[catalina.jar:6.0.35]

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) ~[catalina.jar:6.0.35]

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) ~[catalina.jar:6.0.35]

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) ~[catalina.jar:6.0.35]

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) ~[catalina.jar:6.0.35]

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:293) ~[catalina.jar:6.0.35]

at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859) ~[tomcat-coyote.jar:6.0.35]

at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:602) ~[tomcat-coyote.jar:6.0.35]

at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) ~[tomcat-coyote.jar:6.0.35]

at java.lang.Thread.run(Thread.java:701) ~[na:1.6.0_27]

Caused by: org.mulgara.query.QueryException: Query failed: No such variable $issued in tuples [$pid, $label, $sequence] (class org.mulgara.resolver.AppendAggregateTuples)

at org.mulgara.resolver.DatabaseSession.execute(DatabaseSession.java:746) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.DatabaseSession.query(DatabaseSession.java:464) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.LocalJRDFDatabaseSession.query(LocalJRDFDatabaseSession.java:67) ~[mulgara-core-2.1.12.jar:na]

at org.trippi.impl.mulgara.MulgaraSession.query(MulgaraSession.java:165) ~[trippi-mulgara-1.5.8.jar:na]

... 32 common frames omitted

Caused by: org.mulgara.query.MulgaraTransactionException: Transaction rollback triggered

at org.mulgara.resolver.MulgaraInternalTransaction.implicitRollback(MulgaraInternalTransaction.java:516) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.MulgaraInternalTransaction.execute(MulgaraInternalTransaction.java:629) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.DatabaseSession.execute(DatabaseSession.java:743) ~[mulgara-core-2.1.12.jar:na]

... 35 common frames omitted

Caused by: org.mulgara.query.QueryException: Can't order by $issued

at org.mulgara.resolver.OrderByRowComparator.<init>(OrderByRowComparator.java:149) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.LocalQueryResolver.orderResult(LocalQueryResolver.java:385) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.LocalQueryResolver.resolveE(LocalQueryResolver.java:300) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.DatabaseOperationContext.doQuery(DatabaseOperationContext.java:800) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.QueryOperation.execute(QueryOperation.java:136) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.MulgaraInternalTransaction.execute(MulgaraInternalTransaction.java:625) ~[mulgara-core-2.1.12.jar:na]

... 36 common frames omitted

Caused by: org.mulgara.query.TuplesException: No such variable $issued in tuples [$pid, $label, $sequence] (class org.mulgara.resolver.AppendAggregateTuples)

at org.mulgara.store.tuples.AbstractTuples.getColumnIndex(AbstractTuples.java:363) ~[mulgara-core-2.1.12.jar:na]

at org.mulgara.resolver.OrderByRowComparator.<init>(OrderByRowComparator.java:146) ~[mulgara-core-2.1.12.jar:na]

... 41 common frames omitted

--
You received this message because you are subscribed to the Google Groups "islandora" group.
To unsubscribe from this group and stop receiving emails from it, send an email to islandora+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Jordan Dukart]

Hey Aaron,

The error is when the issue navigator is trying to render where an RI query is used to go find all issues of a newspaper. Are you running 7.x-release code? I have a feeling that it's looking to sort by ?issued which could blow up if it does not find a date issued as it's an OPTIONAL portion of the query. ?issued used to be required but has since been fixed by this commit: https://github.com/Islandora/islandora_solution_pack_newspaper/commit/cb21502df746c7b64fc3cbc2638d5e48ef384c82.

Jordan

[Aaron Collie]

I was trying switching back and forth with git checkout, but I'm not sure I knew what I was doing, so I switched to the 7.x-1.2 zip from the Release Notes page https://wiki.duraspace.org/display/ISLANDORA712/Release+Notes+and+Downloads

Is the appropriate way to grab the current release to:

git clone https://github.com/Islandora/islandora_solution_pack_newspaper.git

git checkout -b 7.x-1.2

[Jordan Dukart]

Hey Aaron,

Honestly at this point the 7.x-1.2 code is really outdated. There is an Islandora 7.x-1.3 release on the way but I'm not fully sure on the timetable for that. All of dgi's production boxes run HEAD (bleeding edge) or near HEAD commits from Git. The vast majority of the Islandora repositories use Travis CI to run the module tests before new code gets merged. At this point, I'd say that HEAD is much, much more stable than what 7.x-1.2 is. If you do decide to move forward please note that you have to update all your modules to the implementation of new features/refactors of old things. If you do decide to move forward and still see this issue/any other issues it will be easier for us to help pinpoint what's going wrong.

Jordan

[Aaron Collie]

Cool, well I checked out HEAD and cleared caches and now the issues display and order correctly. This let's me continue working on my ingest script, so that is good. I'm getting something which is probably due to my not-all-on-head configuration

Notice: Use of undefined constant ISLANDORA_VIEW_OBJECTS - assumed 'ISLANDORA_VIEW_OBJECTS' in islandora_newspaper_issue_page_access() (line 664 of/var/www/drupal-7.26/sites/all/modules/islandora_solution_pack_newspaper/islandora_newspaper.module).

Thanks for your ideas. I'll look into a more systematic way of keeping the code up to date, but that's a limitation of my knowledge currently :)

[Jordan Dukart]

Yep, the constants got re-named to conform to Drupal coding standards.

If you update your islandora module it'll be fine but that'll definitely mean you need to cascade everything.

Jordan

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:369)~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.fcrepo.server.security.jaas.AuthFilterJAAS.doFilter(AuthFilterJAAS.java:329) ~[fcrepo-security-jaas-3.7.0.jar:na]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.security.web.access.channel.ChannelProcessingFilter.doFilter(ChannelProcessingFilter.java:109)~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:381)~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:168)~[spring-security-web-3.0.7.RELEASE.jar:3.0.7.RELEASE]

[Don]

Hi Aaron:

not sure if it helps, but we use drush (which uses tuque) to batch load our newspaper ingests [1]. It requires a spreadsheet of metadata and a folder of images. Derivative creation happens via our microservices [2]. 

Donald

[1] https://github.com/roblib/scripts/blob/master/drush/drupal7/islandora_newspapers.drush.inc

[2] https://github.com/roblib/php_listeners

[Aaron Collie]

Don,

Thanks for the example. I'm bent on finishing my script as it is a learning process for me :) I am able to create a barebones structure, but my pages (content model) are currently not triggering the derivative microservices. I think it is because I've not built the appropriate relationships (I'm not sure how to count issues, for example). When I did this with PDFs it was pretty seamless but the newspapers are a little more complex.

I need to learn how to use the islandora hooks for say the paged content model, I think, in order to count issues/sequence, etc.

-Aaron

--

[Aaron Collie]

Awesome. Thanks to Don's example and digging around in the newspaper pack, my ingest script now works in creating the appropriate relationships, content models, and datastreams -- including triggering the microservices. However, one (set of) relationship(s) is not being created: isPageNumber and isSection

I think these have to do with the paged content solution pack. How might I go about adding these? 

Here is the section of code I'm working with:

$iterator = new DirectoryIterator($tiff_directory);

foreach ($iterator as $fileinfo) {

    if ($fileinfo->isFile()) {

        $tiff_path = $fileinfo->getPathname();

        $newspaper_page = $repository->constructObject($namespace);

        $fedora_object->models = array('islandora:newspaperPageCModel');

        $datastream_id = "OBJ";

        $controlGroup = "M";

        $obj_datastream = $newspaper_page->constructDatastream($datastream_id, $controlGroup);

        $obj_datastream->label = 'Image file of this object.';

        $obj_datastream->mimetype = 'image/tiff';

        $obj_datastream->setContentFromFile($tiff_path);

        $selected_issue_object = islandora_object_load($newspaper_pid);

        $issue_seq = islandora_newspaper_get_issues($selected_issue_object);

        $num_issue_seq = count($issue_seq) + 1;

        $newspaper_page->label = $fileinfo->getFilename();  // will probably get changed

        $newspaper_page->owner = "fedoraAdmin";

        $newspaper_page->relationships->add(FEDORA_MODEL_URI, 'hasModel', 'islandora:newspaperPageCModel');

        $newspaper_page->relationships->add(FEDORA_RELS_EXT_URI, 'isMemberOf', $newspaper_issue_pid);

        $newspaper_page->relationships->add(ISLANDORA_RELS_EXT_URI, 'isPageOf', $newspaper_issue_pid);

        $newspaper_page->relationships->add(ISLANDORA_RELS_EXT_URI, 'hasLanguage', 'eng');

        $newspaper_page->relationships->add(ISLANDORA_RELS_EXT_URI, 'isSequenceNumber', (string) $num_issue_seq, TRUE);

        $newspaper_page->ingestDatastream($obj_datastream);

        $new_page_object = islandora_add_object($newspaper_page);

    }

}

-Aaron

p.s. sorry about posting big code snippets in email.. we currently use a private repo for the project, though that might change.

[Amanda Lehman]

Hi Aaron,

I know this thread/work was a while ago, but I came here while looking for ways to manipulate the RELS-EXT relationships to compound objects via tuque, but I also have newspapers in my future.  Have you figured out how to correctly load your isPageNumber, etc? 

And by the way, I appreciate seeing your code, the more we see, the more we learn!

Thanks!
Amanda Lehman
University of Wyoming Libraries