Fwd: Mint NLA configuration

[Grant Jackson]

Hi all,

Oops, I'll try the redbox-dev list this time!
Continuing with the recent NLA theme...

I have access to an appropriate ISIL for our university, have registered with Trove-test & Trove-production & have requested access to TIM (see references below). Shortly I intend to configure Mint 1.5.1/1.5.2.2 dev-handle institutional build to integrate with NLA IDs.

Can someone please confirm if the following are the appropriate steps to connect to Trove-test (or point me to some doco if I've missed it)? In particular, I'm keen to get confirmation that Mint "agencyCode" should be set to our ISIL.


1) Update Mint home/system-config.json:

    "nlaIntegration": {
        "enabled": true,
        "pidProperty": "nlaPid",
        "agencyCode": "MyIsil",
        "agencyName": "My University",
        ...

where MyIsil = AU-MyNucSymbol (for Australia)

2) Update portal/default/mint/scripts/hkjobs/nla.py as per comment "If using the NLA's test server, comment out the line above and uncomment the line below". That is, replace line with:
   sru = SRUClient("http://www-test.nla.gov.au/apps/srw/search/peopleaustralia")

3) Update portal/default/mint/scripts/search/nlaLookup.py as per comment "If using the NLA's test server, comment out the line above and uncomment the line below". That is, replace line with:
   sru = SRUClient("http://www-test.nla.gov.au/apps/srw/search/peopleaustralia")

4) Restart ReDBox-Mint.

5) If required, add firewall rule for NLA harvesting (see references below).

Have I missed anything? Thank you.


References:
 - http://ands.org.au/guides/ardc-party-infrastructure-awareness.html
 - http://ands.org.au/training/trove/trove-module1-a.html
 - http://ands.org.au/training/trove/trove-module1-b.html
 - http://metadata-stores.blogspot.com.au/2012/08/testing-your-nlatrove-party.html
 - Mint NLA info; https://groups.google.com/forum/?fromgroups=&hl=en#!searchin/redbox-dev/nla/redbox-dev/odS2GDPG5Sg/mGk7hS7WaAEJ
 - Mint NLA workflow; https://groups.google.com/forum/?fromgroups=#!topic/redbox-repo/-nGTI7wUWXM
 - NLA firewall info; https://groups.google.com/forum/?fromgroups=&hl=en#!searchin/redbox-dev/nla/redbox-dev/4GvPoyR6gfw/QaZEIJ1ySqYJ

Cheers, Grant

[Mike Jones]

Hi Grant,

I have a couple of comments, but you've got most of the steps the same as I have.

1) Your system-config looks good, but I have extra values for inside the "nlaIntegration" section, i.e.:

        "nlaIntegration": {
            "enabled": true,
            "pidProperty": "nlaPid",

            "agencyCode": "AU-XYZ",
            "agencyName": "The University of XYZ",
            "includeTest": {
                "repository.name": "People"
            }
        }

2) Correct.

3) I haven't done this, so it might not be needed. This script sounds like one that gets used when the user clicks on the search button next on the People tab. If you really want to search NLA-test for Party records, you can do this, but I don't know that you'd really need to.

4) You can just restart Mint, but restart both if it makes you feel better :)

5) Yes, makes sure your firewall allows:
    - outgoing HTTP (port 80) traffic to the NLA server(s) and
    - incoming traffic (on the port that you're exposing your OAI-PMH feed on) from the NLA server(s)

Good luck!

Mike

[Grant Jackson]

Excellent, thanks Mike.

Thanks for reminding me about both directions for the firewall rules. Will test this once the NLA access is all setup.

Cheers, Grant

--
You received this message because you are subscribed to the Google Groups "ReDBox Development" group.
To post to this group, send an email to redbo...@googlegroups.com.
To unsubscribe from this group, send email to redbox-dev+...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msg/redbox-dev/-/bCVCaRmRRf0J.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Dave Huthnance]

Hi Grant,

This is ours from Dev:

"nlaIntegration": {
"enabled": true,
"pidProperty": "nlaPid",

"agencyCode": "AU-NNCU",
"agencyName": "University of Newcastle",
"includeTest": {
"repository.name": "People"

Note agencyName is the name of the feed and must match what you have sent to NLA.

I had probs because I didn't know what was sent and used "The University of Newcastle"

When uncommenting the files ensure you get the indents right as it is python, ( found the hard way..)

We need the scripts. One is for the lookup in Redbox Parties/People for their NLA id and the other is used by the Housekeeping job to look up and return the NLA id after it goes through trove to put it into the Mint record.

In our case curation of the redbox record can't occur until this is returned and added into the Mint record.

Once this is done the 'published' flag is set and the record goes in the published feed for RDA.


Regards,

Dave








Dave Huthnance
eResearch Consultant,
Academic & Research Computing Support
Client Services - IT Services
The University of Newcastle
University Drive, Callaghan NSW 2308 AUSTRALIA

T: +61 2 4921 5469
F: +61 2 4921 7087
dave.hu...@newcastle.edu.au

www.newcastle.edu.au
CRICOS Provider Number: 00109J


>>> Grant Jackson <grant...@gmail.com> 29/11/2012 2:30 pm >>>

Hi all,

Oops, I'll try the redbox-dev list this time!
Continuing with the recent NLA theme...

I have access to an appropriate ISIL for our university, have registered
with Trove-test & Trove-production & have requested access to TIM (see
references below). Shortly I intend to configure Mint

1.5.1/1.5.2.2dev-handle institutional build to integrate with NLA IDs.

--
You received this message because you are subscribed to the Google Groups "ReDBox Development" group.
To post to this group, send an email to redbo...@googlegroups.com.
To unsubscribe from this group, send email to redbox-dev+...@googlegroups.com.

[Grant Jackson]

Excellent, thanks for the pointers Dave.

The agencyName issue sounds like it could be tricky to avoid, so perhaps I'll fall into the same trap!  :(

Cheers, Grant

[Grant Jackson]

Hi All,

The instructions in the post seemed to work well in terms of the workflow, ie.
 - exposing the EAC-CPF party record to the NLA
 - SRU house keeping job finding the NLA ID (after matching the record to an identity in TIM Beta)
 - NLA ID appearing in the Mint party-person record
 - the ReDBox-Mint curation process completing & exposing all the related RIF-CS records for RDA (Demo environment) harvest

So thanks all, this is great.

However as far as I can see, the NLA ID does not appear in the Mint 1.5.1 RIF-CS party-person record for RDA harvest. I guess I was expecting an extra <relatedObject> key and that there would be some interaction between RDA-Demo & Trove-Test (as per http://metadata-stores.blogspot.com.au/2012/11/new-link-between-ands-demo-and-trove.html)

Have I made a wrong assumption? Have I missed a step? Thanks.

Cheers, Grant

[Grant Jackson]

Oops, wrong assumption. The NLA ID is in the <relatedObject> element of the collection.

Cheers, Grant

[sue...@utas.edu.au]

Hi All,

Just wonder anybody already have party records with matched NLA ID back from NLA successfully so far?

So in you MINT can you see NLA ID yet? and also Collection <relatedObject> element?

Thanks!

Cheers,

Sue

[Grant Jackson]

Hi Sue,

I have not done this in the production environment but have 1 matched record in the Trove test environment.

The Mint Preview page for the party-person shows the NLA ID, but the Mint RIF-CS portal shows the party-person *handle* in both the key & identifier elements (and does not show the NLA ID at all).

The ReDBox Preview page for the related collection shows the NLA ID in the Curation section, and the ReDBox RIF-CS portal shows the NLA ID in the relatedObject element as follows.

<rif:relatedObject>
  <rif:key>http://nla.gov.au/nla.party-999999</rif:key>
  <rif:relation type="hasCollector" ></rif:relation>
</rif:relatedObject>

Note that the trove-test page for the person has a link which points to the (Mint generated) handle for the person. Hope this helps.

Cheers, Grant

--

You received this message because you are subscribed to the Google Groups "ReDBox Development" group.
To post to this group, send an email to redbo...@googlegroups.com.
To unsubscribe from this group, send email to redbox-dev+...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msg/redbox-dev/-/T4jYmqwn1UAJ.

[sue...@utas.edu.au]

Hi Grant,

Thanks for your reply. That is good news for me. So may I ask does all these processes happen automatically or do you have to configure something to make it happen?

Actually we had three records manually matched in TIM but don't know how MINT get NLA ID update from NLA. Do you have more information/details about the process?

Thank you!

Cheers,

Sue

[Grant Jackson]

Hi Sue,

> So may I ask does all these processes happen automatically or do you have to configure something to make it happen?

I think the short answer is that I configured a bunch of stuff then the ReDBox-Mint steps happened automatically (but manual matching in TIM-beta was needed before ReDBox-Mint curation was able to complete).

Rough steps for me  were:

 1) Install ReDBox-Mint 1.5.1 dev-handle build into a test environment

 2) Get handles working for all metadata records (but particularly ReDBox collection & Mint party-person records) with NLA Integration disabled (ie. "nlaIntegration" ... "enabled": false, in Mint system-config.json).

 - Since handles are supposed to be persistent, I use something like the following in dataset.json (redbox) and Parties_People.json, Parties_Groups.json, Services.json (mint) to produce test handles in a production handle server:
     "template": "test-[[INC]]",

 3) Configure NLA integration as per posts on this thread (ie. https://groups.google.com/forum/?fromgroups=&hl=en#!topic/redbox-dev/BLQSzYBB5Zw). There are many good Trove related articles in the references in the first post.

 4) Create & publish a new dataset with a related Mint person who is not yet published in Mint.

 4a) The person configured for Mint admin email should get a Notification email saying: "This email is confirming that the object linked below has completed curation and is ready to be harvested by the National Library. NOTE: This object is not ready for publication until after the NLA has harvested it."

 4b) Mint main.log should indicate it is searching for a matching NLA ID every 15 minutes. Eg.

 - DEBUG  namicPageServiceImpl NLA housekeeping executing
 - DEBUG  namicPageServiceImpl Processing '1' records
 - DEBUG  SRUResponse          SRU Search found 0 results(s)
 - WARN   SRUClient            This identifier matches no records.
 - DEBUG  namicPageServiceImpl 98a5644e9ead143b7fcf715bc8586ad6 => http://hdl.handle.net/xxxx/yyyy (null)
 - DEBUG  namicPageServiceImpl Object '98a5644e9ead143b7fcf715bc8586ad6' does not yet have a national Identity in NLA
 - DEBUG  namicPageServiceImpl Failed to process record

 5) Ask your NLA contact to harvest your records into the Trove test environment.

 6) Perform matching with TIM-beta so that your record/identify is assigned an NLA ID.

 7) Your Mint (& ReDBox) records should complete curation automatically when the next NLA housekeeping job runs (within 15 minutes). main.log should say:

 - DEBUG  namicPageServiceImpl NLA housekeeping executing
 - DEBUG  namicPageServiceImpl Processing '1' records
 - DEBUG  SRUResponse          SRU Search found 1 results(s)
 - DEBUG  namicPageServiceImpl 98a5644e9ead143b7fcf715bc8586ad6 => http://hdl.handle.net/xxxx/yyyy (http://nla.gov.au/nla.party-999999)
 - DEBUG  namicPageServiceImpl Object '98a5644e9ead143b7fcf715bc8586ad6' has a new national Identity in NLA (http://nla.gov.au/nla.party-999999)
 - DEBUG  namicPageServiceImpl Record processed

 8) The NLA ID will be visible in Mint preview & ReDBox preview & RIF-CS as per my previous post.

In addition to keeping an eye on the ReDBox-Mint logs, viewing the following in a web browser can be useful for debugging.

ReDBox RIFCS portal:
https or http://YOUR_SERVER/redbox/published/feed/oai?verb=ListRecords&metadataPrefix=rif

Mint RIFCS portal:
https or http://YOUR_SERVER/mint/published/feed/oai?verb=ListRecords&metadataPrefix=rif

Mint EAC-CPF portal:
https or http://YOUR_SERVER/mint/NLA_Harvest/feed/oai?verb=ListRecords&metadataPrefix=eac-cpf

Hope this helps.

Cheers, Grant

Cheers,

Sue

--

You received this message because you are subscribed to the Google Groups "ReDBox Development" group.
To post to this group, send an email to redbo...@googlegroups.com.
To unsubscribe from this group, send email to redbox-dev+...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msg/redbox-dev/-/3sxiVai2zzQJ.

[Grant Jackson]

Sue, I forgot to mention... if you are getting:

 - WARN   SRUClient            This identifier matches no records.

every 15 minutes in your main.log, & you are using trove-test, perhaps you haven't updated portal/default/mint/scripts/hkjobs/nla.py to point to the NLA test server as per item 2 of my first post on this thread (ie. https://groups.google.com/forum/?fromgroups=&hl=en#!topic/redbox-dev/BLQSzYBB5Zw)?

Cheers, Grant

On Fri, Jan 11, 2013 at 12:23 PM, <sue...@utas.edu.au> wrote:

Cheers,

Sue

--

You received this message because you are subscribed to the Google Groups "ReDBox Development" group.
To post to this group, send an email to redbo...@googlegroups.com.
To unsubscribe from this group, send email to redbox-dev+...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msg/redbox-dev/-/3sxiVai2zzQJ.

[Duncan Dickinson]

Hi All,

A quick note to let you know that I've been working through a doco to-do list and this may be handy: http://www.redboxresearchdata.com.au/documentation/drafts/nla-integration

Cheers,

Duncan

--
Cheers,


Duncan


Duncan Dickinson
QCIF Project Manager 
Central Queensland University

Contact me:

monday to thursday

ph: 07 3138 2084
m: 0432 402 511
skype: de.dickinson

website | calendar | LinkedIn

[Grant Jackson]

Hi Sue,

That log entry appears to indicate:
 - you have 2 records ready for NLA harvesting
 - the SRU query is unable to find an NLA ID for the first of these
 - the SRU query *did* find an NLA ID for the second of these

Are the 2 party records part of the same set being curated at the same time (ie. both related to the same collection which is being curated)?

Lets concentrate on the second party record. I confirm I can see the NLA record at http://trove-test.nla.gov.au/people/1489022?c=people

I confirm I cannot see the corresponding NLA ID in Mint http://115.146.85.179/mint/published/detail/1401e6faf060003e96be766327ea6de1/

I don't know why the NLA ID is not present in the Mint record. Perhaps you could login to Mint & look at "Object History" to see if it gives you a clue. The object history for my Mint record says (most recent first):

 - Curation     Publication flag set
 - ...EmailNotificationConsumer     notify
 - Curation     This object is ready for publication
 - Curation     Curation completed.
 - Curation     This object is ready to go to the NLA
 - ...EmailNotificationConsumer     notify
 - Curation     Object curation requested.

Other ideas:

 - Any clues from the object histories for other party, collection, etc records being curated with this party record? Eg. Does the collection record Object History say "Publication flag set" & "Curation completed"?

 - I presume all of the records being curated with this party record (in particular the party record itself) were not previously published prior to NLA integration?

 - It appears you are not using handles for persistent IDs. I don't know if this is important for NLA integration. We are using handles here.

Cheers, Grant

On Wed, Jan 16, 2013 at 8:12 AM, <sue...@utas.edu.au> wrote:

Hi Grant,

Now I have the message as below from Mint main log

DEBUG  namicPageServiceImpl NLA housekeeping executing

DEBUG  SolrSearcher         URL:http://localhost:9001/solr/fascinator/select, POSTDATA:[name=q, value=ready_for_nla:ready AND NOT nlaId:http*, name=facet, value=false, name=wt, value=json, name=rows, value=20]

DEBUG  namicPageServiceImpl Processing '2' records

DEBUG  SRUResponse          SRU Search found 0 results(s)

WARN   SRUClient            This identifier matches no records.

DEBUG  namicPageServiceImpl 84af982d2016a851a9034fb66a012e5c => http://115.146.85.179/mint/published/detail/84af982d2016a851a9034fb66a012e5c (null)

DEBUG  namicPageServiceImpl Object '84af982d2016a851a9034fb66a012e5c' does not yet have a national Identity in NLA

DEBUG  namicPageServiceImpl Failed to process record

2013-01-15 21:30:01,288          DEBUG  SRUResponse          SRU Search found 1 results(s)

2013-01-15 21:30:01,290          DEBUG  namicPageServiceImpl 1401e6faf060003e96be766327ea6de1 => http://115.146.85.179/mint/published/detail/1401e6faf060003e96be766327ea6de1 (http://nla.gov.au/nla.party-1489022)

DEBUG  namicPageServiceImpl Object '1401e6faf060003e96be766327ea6de1' has a new national Identity in NLA (http://nla.gov.au/nla.party-1489022)

DEBUG  namicPageServiceImpl Record processed

But I still can't see NLAID in my Mint and RedBox records preview.

Any idea?

Thank you!

Cheers,

Sue

--
You received this message because you are subscribed to the Google Groups "ReDBox Development" group.
To post to this group, send an email to redbo...@googlegroups.com.
To unsubscribe from this group, send email to redbox-dev+...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msg/redbox-dev/-/BU4MgLmC9bAJ.

[sue...@utas.edu.au]

Hi Grant,

It is strange, after restarted my MINT, then I found the NLAID updated to my MINT party record.

Thank you for your help!

Cheers,

Sue

[Mike Jones]

Hi Sue,

 

I think your message broker error message might be related to this behaviour. If the message broker stops working then it is likely to have a negative impact on functionality.

 

Mike

 

 

--

You received this message because you are subscribed to the Google Groups "ReDBox Development" group.
To post to this group, send an email to redbo...@googlegroups.com.
To unsubscribe from this group, send email to redbox-dev+...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msg/redbox-dev/-/L1R5eUmrEDcJ.

[sue...@utas.edu.au]

Hi Mike,

Yes I think you are right. I got the NLA SRU message in main.log later yesterday, and the BrokerMonitor message stopped later yesterday as well.

Thanks all your help here, really helpful.

Cheers,

Sue