Avoid Solr commit

10 views
Skip to first unread message

Michael Levy

unread,
Dec 13, 2019, 8:25:59 PM12/13/19
to solrmarc-tech
Is it possible to configure SolrMarc so as not to perform a Solr commit?

I'd like to run SolrMarc and then do some other things before doing a Solr commit, which I'd like to do myself later.


Demian Katz

unread,
Dec 14, 2019, 12:04:42 PM12/14/19
to solrma...@googlegroups.com

It looks like SolrMarc is hard-coded to always commit here:

 

https://github.com/solrmarc/solrmarc/blob/966aae7e90e60c4774a97e74df5b8dc4ef655093/src/org/solrmarc/driver/Indexer.java#L581

 

It might be useful to add a Java property to disable this behavior; that might be worth a pull request if you’re interested. (Or, for your own purposes, you could just comment this out and recompile the .jar as a short-term solution).

 

(Bob, please correct me if I’m overlooking a way of controlling this – but it doesn’t look like there is one at present).

 

- Demian

--
You received this message because you are subscribed to the Google Groups "solrmarc-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to solrmarc-tec...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/solrmarc-tech/521a650a-0c32-4cbd-a60a-cc2e289738fe%40googlegroups.com.

Michael Levy

unread,
Dec 14, 2019, 3:20:22 PM12/14/19
to solrmarc-tech
Thanks, Demian.

In my case, I can either work around this requirement, and/or I can recompile as you suggest, and/or I could think of a god way to configure a switch and do a pull request.

In case it's not obvious why I am looking into this: Solr commits act against all non-committed updates in a core. We have several update processes running at various times. SolrMarc is one. If any of these update processes executes a commit while other processes are partway done, it can leave the Solr core in an unintended state. So it's nice to have maximum flexibility about when the update is performed.

Tod Olson

unread,
Dec 14, 2019, 3:28:41 PM12/14/19
to solrma...@googlegroups.com, Tod Olson
We also have some use cases where it would be useful to defer the commit. So I would encourage submitting a PR!

-Tod

--
You received this message because you are subscribed to the Google Groups "solrmarc-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to solrmarc-tec...@googlegroups.com.

Michael Levy

unread,
Dec 14, 2019, 4:26:37 PM12/14/19
to solrmarc-tech
OK, I will take a run at it!

On Friday, December 13, 2019 at 8:25:59 PM UTC-5, Michael Levy wrote:

Michael Levy

unread,
Dec 17, 2019, 1:15:12 PM12/17/19
to solrmarc-tech
After building the jar with commenting out 
 solrProxy.commit(false);
as suggested by Demian,

I found that our Solr id lost the prefix "bib" although everything else seemed to work OK. I try to clone a SolrMarc a couple weeks old to see whether this is a side effect of some other recent change or was caused by that one line. Unfortunately I'm not sure but thought I'd mention it now.

This is what is being ignored.

id = 001, (pattern_map.id_prefix), first
pattern_map.id_prefix.pattern_0 = (.+)=>bib$1

Demian Katz

unread,
Dec 17, 2019, 3:36:36 PM12/17/19
to solrma...@googlegroups.com

Michael,

 

It seems unlikely to me that commenting out the commit would change the translation map behavior. I just tested your SolrMarc configuration with the latest VuFind master branch, and it works as expected there… so either there’s a problem with the custom configuration failing to load, or some recently-introduced bug. Let me know if you need me to do any further testing on this end!

 

- Demian

 

From: solrma...@googlegroups.com <solrma...@googlegroups.com> On Behalf Of Michael Levy
Sent: Tuesday, December 17, 2019 1:15 PM
To: solrmarc-tech <solrma...@googlegroups.com>

--

You received this message because you are subscribed to the Google Groups "solrmarc-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to solrmarc-tec...@googlegroups.com.

Levy, Michael

unread,
Dec 17, 2019, 5:00:02 PM12/17/19
to solrma...@googlegroups.com
Many thanks, Robert and Demian.

I had checked out an earlier version from Github, but I then used the wrong jar file for testing, which led to my confusion about the cause of that pattern_map error. All good on that account. The pattern_map error does seem to have been introduced very recently. I had done git checkout 9d2ff6ba65730eae5da14120edfa122cc97172de and that version supports the pattern_map OK. I didn't have time today to confirm that the changes Robert pointed out were the direct cause.

I had started to play with a -Dproperty=value approach to overriding the commit. I'll take your approach using Boolean.parseBoolean(System.getProperty() and keep playing with it.

Demian Katz

unread,
Dec 18, 2019, 7:13:39 AM12/18/19
to solrma...@googlegroups.com
Good luck, and let me know if I can be of any further assistance. If we need to narrow down the exact commit causing the problem, the "git bisect" command can be really helpful for that!

- Demian


From: solrma...@googlegroups.com <solrma...@googlegroups.com> on behalf of Levy, Michael <ml...@ushmm.org>
Sent: Tuesday, December 17, 2019 4:59 PM
To: solrma...@googlegroups.com <solrma...@googlegroups.com>
Subject: Re: [EXTERNAL] [solrmarc-tech] Re: Avoid Solr commit
 

Levy, Michael

unread,
Dec 18, 2019, 10:49:07 AM12/18/19
to solrma...@googlegroups.com
I executed a PR. It will only skip committing if passed exactly:
-Dsolrmarc.commit.at.end=false
If there's anything non-idiosyncratic or if you'd like me to add to some documentation, let me know.

Michael Levy

unread,
Dec 18, 2019, 3:08:34 PM12/18/19
to solrmarc-tech
I intended to write "idiomatic," not "idiosyncratic."   :-)   By this I mean I'd be happy to implement any suggestions or hints.

Demian Katz

unread,
Dec 19, 2019, 8:41:51 AM12/19/19
to solrma...@googlegroups.com

Ha! Thanks, Michael. I took a glance at this and it looks fine to me, but I’ll defer to Bob for better feedback, since he’s much more familiar with the code than I am, and I’m not sure whether or not there’s a preferable way to relate arguments and properties.

 

thanks,

Demian

 

From: solrma...@googlegroups.com <solrma...@googlegroups.com> On Behalf Of Michael Levy
Sent: Wednesday, December 18, 2019 3:09 PM
To: solrmarc-tech <solrma...@googlegroups.com>
Subject: Re: [EXTERNAL] [solrmarc-tech] Re: Avoid Solr commit

 

I intended to write "idiomatic," not "idiosyncratic."   :-)   By this I mean I'd be happy to implement any suggestions or hints.

--

You received this message because you are subscribed to the Google Groups "solrmarc-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to solrmarc-tec...@googlegroups.com.

Reply all
Reply to author
Forward
0 new messages