Importing marc into Blacklight using solr-marc
152 views
Skip to first unread message
Simon Lamb
Jun 19, 2012, 3:41:08 AM6/19/12
to blacklight-...@googlegroups.com
Hi all,
Firstly apologies for duplicating this post from the solr-marc community, however I've not had much in the way of a response so wanted to try here - I've made the assumption that there will be a lot of people on this board also using solr-marc for catalog updates.
My original post on solr-marc tech is here - https://groups.google.com/d/msg/solrmarc-tech/OGtA46Zco4g/oYN6MJDJgoUJ
Briefly again, I'm having problems with the solr-marc routine for Deleting records out of solr-marc. I'm using 'customDeleteRecordIfFieldEmpty' routine to remove any records that have been deemed 'suppressed' in my nightly Millennium export. The routine behaves correctly, in that it will not add any suppressed/deleted records to the index, however, I've read on the solr-marc board that the 'DeleteRecordIf...' method will call a Delete on the SOLR index to remove the records (whether they exist or not). This is good for when publicly accessible resources, get deleted or suppressed for whatever reason and we want blacklight to reflect this.
The output from solr-marc gives me:-
INFO: Deleted record (record count 287)
19-Jun-2012 08:35:13 org.solrmarc.marc.MarcImporter importRecords
INFO: Deleted record 23b18951314 (record count 288)
19-Jun-2012 08:35:13 org.solrmarc.marc.MarcImporter handleAll
INFO: Adding 0 of 288 documents to index
19-Jun-2012 08:35:13 org.solrmarc.marc.MarcImporter handleAll
INFO: Deleting 0 documents from index
19-Jun-2012 08:35:13 org.solrmarc.marc.MarcImporter finish
INFO: Calling commit (with optimize set to false)
Which as expected adds '0' records, however it doesn't Delete the said the records.
Given there is log line that states 'Deleting _ documents from index', I'm guessing delete functionality is part of solr-marc. I'm just wondering if I'm missing something very obvious (usually the case!).
Thanks, and again apologies for cross-posting!
Simon Lamb
Simon Lamb
Jun 19, 2012, 10:58:54 AM6/19/12
to blacklight-...@googlegroups.com
For general information, I may have figured the cause of this issue. It looks like its down to us using a different ID (we use 907a) for our solr records over the standard one (001) that solr-marc defines. It seems that solr-marc defaults to looking for the 001 based IDs when Deleting.
More here - https://groups.google.com/d/msg/solrmarc-tech/OGtA46Zco4g/avd9U4vlvh0J
Thanks,
Simon
Simon
Jonathan Rochkind
Jun 19, 2012, 11:24:32 AM6/19/12
to blacklight-...@googlegroups.com, Simon Lamb
You might want to send this to the SolrMarc listserv, since it's a
SolrMarc problem. I'm afraid none of us may be experts in SolrMarc.
BL itself just displays what's in your Solr index, no more, no less.
If you want something in your Solr index that are nonetheless suppressed
by BL, there are ways to do that with BL, that we could help you with.
But if your problem is that you can't get SolrMarc to delete things from
your Solr index and you want to -- might try asking for help on the
SolrMarc listserv.
Note also that you aren't required to use SolrMarc to delete records,
there are other ways to delete records from a Solr index. You could
write your own script that uses ordinary Solr HTTP api to delete records
from the solr index, if you like. BL just displays what's in the Solr
index, you can use any method you want to add or remove documents from it.
On 6/19/2012 3:41 AM, Simon Lamb wrote:
> Hi all,
>
> Firstly apologies for duplicating this post from the solr-marc
> community, however I've not had much in the way of a response so wanted
> to try here - I've made the assumption that there will be a lot of
> people on this board also using solr-marc for catalog updates.
>
> My original post on solr-marc tech is here -
> https://groups.google.com/d/msg/solrmarc-tech/OGtA46Zco4g/oYN6MJDJgoUJ
> *
> *
> You received this message because you are subscribed to the Google
> Groups "Blacklight Development" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/blacklight-development/-/9X9jwckyjEsJ.
> To post to this group, send email to
> blacklight-...@googlegroups.com.
> To unsubscribe from this group, send email to
> blacklight-develo...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/blacklight-development?hl=en.
SolrMarc problem. I'm afraid none of us may be experts in SolrMarc.
BL itself just displays what's in your Solr index, no more, no less.
If you want something in your Solr index that are nonetheless suppressed
by BL, there are ways to do that with BL, that we could help you with.
But if your problem is that you can't get SolrMarc to delete things from
your Solr index and you want to -- might try asking for help on the
SolrMarc listserv.
Note also that you aren't required to use SolrMarc to delete records,
there are other ways to delete records from a Solr index. You could
write your own script that uses ordinary Solr HTTP api to delete records
from the solr index, if you like. BL just displays what's in the Solr
index, you can use any method you want to add or remove documents from it.
On 6/19/2012 3:41 AM, Simon Lamb wrote:
> Hi all,
>
> Firstly apologies for duplicating this post from the solr-marc
> community, however I've not had much in the way of a response so wanted
> to try here - I've made the assumption that there will be a lot of
> people on this board also using solr-marc for catalog updates.
>
> My original post on solr-marc tech is here -
> https://groups.google.com/d/msg/solrmarc-tech/OGtA46Zco4g/oYN6MJDJgoUJ
> *
> Briefly again, I'm having problems with the solr-marc routine for
> Deleting records out of solr-marc. I'm using
> 'customDeleteRecordIfFieldEmpty' routine to remove any records that have
> been deemed 'suppressed' in my nightly Millennium export. The routine
> behaves correctly, in that it will not add any suppressed/deleted
> records to the index, however, I've read on the solr-marc board that the
> 'DeleteRecordIf...' method will call a Delete on the SOLR index to
> remove the records (whether they exist or not). This is good for when
> publicly accessible resources, get deleted or suppressed for whatever
> reason and we want blacklight to reflect this.
>
> The output from solr-marc gives me:-
>
> INFO: Deleted record (record count 287)
> 19-Jun-2012 08:35:13 org.solrmarc.marc.MarcImporter importRecords
> INFO: Deleted record 23b18951314 (record count 288)
> 19-Jun-2012 08:35:13 org.solrmarc.marc.MarcImporter handleAll
> INFO: *Adding 0 of 288 documents to index*
> Deleting records out of solr-marc. I'm using
> 'customDeleteRecordIfFieldEmpty' routine to remove any records that have
> been deemed 'suppressed' in my nightly Millennium export. The routine
> behaves correctly, in that it will not add any suppressed/deleted
> records to the index, however, I've read on the solr-marc board that the
> 'DeleteRecordIf...' method will call a Delete on the SOLR index to
> remove the records (whether they exist or not). This is good for when
> publicly accessible resources, get deleted or suppressed for whatever
> reason and we want blacklight to reflect this.
>
> The output from solr-marc gives me:-
>
> INFO: Deleted record (record count 287)
> 19-Jun-2012 08:35:13 org.solrmarc.marc.MarcImporter importRecords
> INFO: Deleted record 23b18951314 (record count 288)
> 19-Jun-2012 08:35:13 org.solrmarc.marc.MarcImporter handleAll
> 19-Jun-2012 08:35:13 org.solrmarc.marc.MarcImporter handleAll
> INFO: *Deleting 0 documents from index*
> 19-Jun-2012 08:35:13 org.solrmarc.marc.MarcImporter finish
> INFO: Calling commit (with optimize set to false)
>
>
> Which as expected adds '0' records, however it doesn't Delete the said
> the records.
>
> Given there is log line that states 'Deleting _ documents from index',
> I'm guessing delete functionality is part of solr-marc. I'm just
> wondering if I'm missing something very obvious (usually the case!).
>
> Thanks, and again apologies for cross-posting!
>
> Simon Lamb
>
> --
> INFO: Calling commit (with optimize set to false)
>
>
> Which as expected adds '0' records, however it doesn't Delete the said
> the records.
>
> Given there is log line that states 'Deleting _ documents from index',
> I'm guessing delete functionality is part of solr-marc. I'm just
> wondering if I'm missing something very obvious (usually the case!).
>
> Thanks, and again apologies for cross-posting!
>
> Simon Lamb
>
> You received this message because you are subscribed to the Google
> Groups "Blacklight Development" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/blacklight-development/-/9X9jwckyjEsJ.
> To post to this group, send email to
> blacklight-...@googlegroups.com.
> To unsubscribe from this group, send email to
> blacklight-develo...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/blacklight-development?hl=en.
Simon Lamb
Jun 19, 2012, 11:41:28 AM6/19/12
to blacklight-...@googlegroups.com
Thanks for the response Jonathan.
I did indeed post it to the solr-marc tech mailing list, unfortunately it didn't get much in the way of a response. So I thought I'd try in here just on the off chance that somebody had hit the same problem populating/depopulating their solr index.
The suggestion of using a script to directly interact with the Solr HTTP API sounds like an a good alternative if I hit a dead-end with this issue.
Thanks for the suggestions,
Simon
To unsubscribe from this group, send email to
--
You received this message because you are subscribed to the Google Groups "Blacklight Development" group.
To post to this group, send email to blacklight-development@googlegroups.com.
To unsubscribe from this group, send email to blacklight-development+unsub...@googlegroups.com.
Reply all
Reply to author
Forward
0 new messages