alpha-browse

[Hannah]

Hi together,

we are upgrading from Solr 4 to Solr 7.

Everything works fine - but I can't create the  alphabetical browse db.

We are using Solr 7 and the solrmarc from VuFind. We also changed from bsh-scripts to the Java Classes.

I got the newest browse-handler.jar and browse-indexer.jar from vufind.

This is one of the outputs when I start the script:

++ dirname ./index-alphabetic-browse-ubfr.sh
+ cd .
+ bib_index=/data/rdsindex2/solr/ubfr/index
+ auth_index=/data/rdsindex2/solr/authority_ubfr/index
+ index_dir=/data/rdsindex2/solr/alphabetical_browse
+ CLASSPATH='browse-indexing.jar:../solr/jars/*'
+ mkdir -p /data/rdsindex2/solr/alphabetical_browse
+ build_browse 25author author_browse
+ browse=25author
+ field=author_browse
+ skip_authority=
+ extra_jvm_opts=
+ '[' '' = 1 ']'

+ java -Xms2048m -Xmx2048m -Dfile.encoding=UTF-8 -Dfield.preferred=heading -Dfield.insteadof=use_for -cp 'browse-indexing.jar:../solr/jars/*' PrintBrowseHeadings /data/rdsindex2/solr/ubfr/index author_browse /data/rdsindex2/solr/authority_ubfr/index 25author.tmp
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/lucene/search/Query
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.search.Query
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more

sometimes this message:

+ java -Xms2048m -Xmx2048m -Dfile.encoding=UTF-8 -Dfield.preferred=heading -Dfield.insteadof=use_for -cp 'browse-indexing.jar:../solr/jars/*' PrintBrowseHeadings /data/rdsindex2/solr/ubfr/index author_browse /data/rdsindex2/solr/authority_ubfr/index 25author.tmp
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/lucene/index/IndexNotFoundException
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.lucene.index.IndexNotFoundException
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more

Both indexes exist...

What did I forget?

Thanks in advance

Hannah

[Demian Katz]

Hannah,

 

Sorry for the slow response – I was out of the office for the American Thanksgiving holiday.

 

The problem is that the browse index tool uses Solr’s Java APIs, and these frequently change in backward-incompatible ways. A browse indexer built for Solr 4 will not work with Solr 7.

 

You should be able to grab the indexer jars from a more recent version of VuFind (5.0 or newer) or simply rebuild them from source (https://github.com/vufind-org/vufind-browse-handler) to resolve the problem.

 

Please let me know if you need further assistance!

 

- Demian

--
You received this message because you are subscribed to the Google Groups "solrmarc-tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to solrmarc-tec...@googlegroups.com.
To post to this group, send email to solrma...@googlegroups.com.
Visit this group at https://groups.google.com/group/solrmarc-tech.
For more options, visit https://groups.google.com/d/optout.

[Hannah]

Hi Demian,

I still downloaded the newest browse-handler.jar and browse-indexer.jar from vufind 5.1.... but this did not work.

We don't use the import functions from vufind. VuFind and Solrmarc with Solr are separate Servers...

Index is build with Solr 7.

What else can I do?

- Hannah

[Demian Katz]

When you say that the newer versions did not work, exactly what happened? Same error as before? Are you using the latest version of the index-alphabetic-browse.sh script in combination with browse-indexer.jar? Do you have all of the environment variables (JAVA_HOME/VUFIND_HOME/VUFIND_LOCAL_DIR) set to appropriate values? What Java version are you using?

[Hannah]

Hi Demian,

yes, the errors I posted are the one with the newest jars.

We are using our own browse.sh, because we don't use the 'import' directory from VuFind.

Solrmarc from git with Solr 7 and a seperate VuFind instance.

Using java: openjdk version "1.8.0_192"

Please find attached our index-alphabetic-browse script. It works fine with our Solr 4.6.1 .

Path ../solr/jars/* includes: browsehandler.jar, MarcImporter.jar, sqlite-jdbc-3.7.15-SNAPSHOT.jar, sqlitejdbc-v053.jar

Path ../solrjetty/dist: are different jars like solr-core-7.3.1, solr-analytics-7.3.1 (perhaps we don't have to include this path)

regards Hannah

[Demian Katz]

Hannah,

 

Given that the error seems to be related to a missing class, and that VuFind’s bundled Solr has a different directory layout now than it did several versions back, I wonder if this boils down to a missing .jar file somewhere. It might be worth double-checking the long classpath definition used by the default VuFind script (CLASSPATH="browse-indexing.jar:${SOLR_HOME}/jars/*:${SOLR_HOME}/../vendor/contrib/analysis-extras/lib/*:${SOLR_HOME}/../vendor/server/solr-webapp/webapp/WEB-INF/lib/*") and making sure that all of the .jar files represented there are copied into your solr/jars directory.

 

Does that make any difference?

[Hannah]

Demian,

thanks for the hint. I tried it again and at least it works.

But now we have the next problem...(sorry, the configuration of alpha-browse is new for us, our experienced college is no longer here)

when I try: http://localhost:8080/solr/ubfr/browse?source=author

I get:

[   x:ubfr] o.a.s.h.RequestHandlerBase java.lang.NullPointerException
    at org.vufind.solr.handler.BrowseRequestHandler.handleRequestBody(BrowseRequestHandler.java:879)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)

........

2018-12-11 11:48:23.019 INFO  (qtp2114889273-20) [   x:ubfr] o.a.s.c.S.Request [ubfr]  webapp=/solr path=/browse params={offset=0&source=author&rows=20} status=500 QTime=1
2018-12-11 11:48:23.020 ERROR (qtp2114889273-20) [   x:ubfr] o.a.s.s.HttpSolrCall null:java.lang.NullPointerException
    at org.vufind.solr.handler.BrowseRequestHandler.handleRequestBody(BrowseRequestHandler.java:879)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:195)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:2503)

solrconfig.xml of core ubfr: 

<requestHandler name="/browse" class="org.vufind.solr.handler.BrowseRequestHandler">

<str name="preferredHeadingField">heading</str>
    <str name="useInsteadHeadingField">use_for</str>
    <str name="seeAlsoHeadingField">see_also</str>
    <str name="scopeNoteField">scope_note</str>

    <str name="sources">author,topic,corporation,callnumber</str>

    <lst name="topic">
      <str name="DBpath">/data/rdsindex2/solr/alphabetical_browse/25topic_browse.db</str>
      <str name="field">topic_browse</str>
    </lst>
    <lst name="author">
      <str name="DBpath">/data/rdsindex2/solr/alphabetical_browse/25author_browse.db</str>
      <str name="field">author_browse</str>
    </lst>....    

</requestHandler>

in /data/rdsindex2/solr/alphabetical_browse/ we have : 25author_browse.db-ready and 25author_browse.db-updated

Do you have a hint for us, where we have to search?

regards 

Hannah

sorry, I answered 1 hour ago, but it seems that I  accidentally deleted my answer ...

[Demian Katz]

I'm at a conference today but will attempt a more detailed answer tomorrow. In the meantime, have you tried looking at the code line cited in the backtrace to see if it offers any clues? You can find the source here, and the tags show where the code was at any given vufind release:

https://github.com/vufind-org/vufind-browse-handler

Based solely on the error message it sounds like the code may be trying unsuccessfully to determine a solr url, but I'm not sure why it would be doing that at all...

I'm also copying Mark Triggs into this thread in case this rings a bell for him.

- Demian

Get Outlook for Android

---

From: solrma...@googlegroups.com <solrma...@googlegroups.com> on behalf of Hannah <krook...@gmail.com>
Sent: Tuesday, December 11, 2018 7:33:08 AM
To: solrmarc-tech
Subject: Re: [solrmarc-tech] alpha-browse

 

[Hannah]

Yes i looked at the code.

SolrCore authCore = cc.getCore(authCoreName);

//Must decrement RefCounted when finished!

RefCounted<SolrIndexSearcher> authSearcherRef = authCore.getSearcher();

But I don't know, why the Code cannot find the core...

regards Hannah

[Demian Katz]

Hannah,

 

Is it possible the core does not exist? Have you indexed any authority records? If not, does accessing the core through the web-based Solr admin to initialize it change anything?

 

There used to be a problem in the browse code that would cause a failure if the authority core had not been initialized. That should be fixed in the latest code, but perhaps because you are not using this in the typical way, some variation of the problem is still occurring.

 

If that doesn’t help, please let me know and I’ll dig deeper!

[Hannah]

Hi Demian,

yes the authority core exists.

I thought the index-alphabetic-browse script cannot create the sqlite db files without the authority core?

regards Hannah

[Demian Katz]

Hannah,

 

That is correct. I am not aware of a known issue with the handler failing without the authority core, but that’s what the backtrace you shared seemed to suggest might be the problem.

 

Is there anything unusual about your Solr deployment, relative to a standard VuFind install, that might be a factor here? Are you using non-default core names or anything like that?

 

Is there a particular reason you are using the 25 prefix on your database filename? I don’t think it should matter, but I wonder if there’s any chance the code makes some assumption about correlation between filename and field name that’s causing a problem. (Again, I have no real reason to suspect this – just trying to think about what might be unique to your instance that could potentially cause problems).

[Tod Olson]

This is really puzzling. The authority core name is hard-coded in BrowseRequestHandler, so it shouldn't be a typo. When you start Solr, does the authority core look like it loads correctly?

If not, I would  wonder what cores the CoreContainer thinks exist? If you're okay with recompiling the browse handler, you could try adding something like:

        CoreContainer cc = core.getCoreContainer();

        

        StringBuilder msg = new StringBuilder();

        msg.append("Solr Home: ").append(cc.getSolrHome());

        msg.append("  Known core names: ");

        for (String cname : cc.getAllCoreNames()) {

        msg.append('"').append(cname).append('"');

        }

        Log.info(msg.toString());

That would at least confirm what cores the CoreContainer sees.

-Tod

[Hannah]

Renaming the authority index from authority_ubfr to the default name authority didn't make any change.

I try to recompile the browse-handler.jar but I'm not sure, if I am right.

cloning the vufind-browse-handler git repository and gone to the solr7 branch - recompile with: ant jars -Dvufind.dir=/VuFind/vufind-5.0.1/ (test directory with vufind, but no config , it is only a git pull)

The recompiling works (I added the text from Tod)  , but after copying the new browse-handler.jar to my Solr-Instance it makes no difference. Same error message and  I can't find any logging Info in my solr.log...

Our VuFind Instance and the Solr/Solrjetty  Instance are separate applications.

I assumed all configurations from our Solr 4 Instance for the alpha-browse... perhaps I forgot something, but I can't find it..

regards Hannah

[Demian Katz]

It certainly seems like you are building correctly.

 

As a simple sanity check, what if you add a log statement above the line that’s triggering the backtrace. If your recompiled version is being used, then the line number in the backtrace should change, since you’ve pushed everything down by a line. Then you need to figure out why the log is not going anywhere. On the other hand, if the number doesn’t change, then maybe there is a problem with compiling or deployment.

 

Is that any help?

[Hannah]

Hi,

back from long weekend I tried again the recompiling with:

StringBuilder msg = new StringBuilder();
        msg.append("Solr Home: ").append(cc.getSolrHome());

        msg.append(" authCoreName: ").append(authCoreName);

        msg.append(" Known core names: ");
        for (String cname : cc.getAllCoreNames()) {
                msg.append('"').append(cname).append('"');
        }

        SolrCore authCore = cc.getCore(authCoreName);
        msg.append(" authCore: ").append(cc.getCore(authCoreName));
        Log.info(msg.toString());

I found the logging for the handler:

Dez 17, 2018 9:45:47 AM org.vufind.solr.handler.Log info
INFORMATION: Solr Home: /home/hannah/data/rdsindex2/solr authCoreName: /home/hannah/data/rdsindex2/solr/authority/  Known core names: "authority""authority_blb""authority_phfr""blb""phfr""tue""ubfr" authCore: null

Path information is right, the core names also ... but the message:

msg.append(" authCore: ").append(cc.getCore(authCoreName)); yields me null.

After hard coding the core name in the BrowseRequestHandler.java ( SolrCore authCore = cc.getCore("authority"); ) the alpha-browse give me a result:

Dez 17, 2018 12:17:59 PM org.vufind.solr.handler.Log info
INFORMATION: Solr Home: /home/hannah/data/rdsindex2/solr authCoreName: /home/hannah/data/rdsindex2/solr/authority Known core names: "authority""authority_blb""authority_phfr""blb""phfr""tue""ubfr" authCore: null
Dez 17, 2018 12:17:59 PM org.vufind.solr.handler.Log info
INFORMATION: constructor: HeadingsDB (/home/hannah/data/rdsindex2/solr/alphabetical_browse/ubfrauthor_browse.db, null)
Dez 17, 2018 12:17:59 PM org.vufind.solr.handler.Log info
INFORMATION: new browse source with HeadingsDB (/home/hannah/data/rdsindex2/solr/alphabetical_browse/ubfrauthor_browse.db, null)
Dez 17, 2018 12:17:59 PM org.vufind.solr.handler.Log info
INFORMATION: Index update event detected!
Dez 17, 2018 12:17:59 PM org.vufind.solr.handler.Log info
INFORMATION: Browsing from: 209198

- Hannah

[Hannah]

Ok, the alpha-browse for one core works... but not for the other core.

I found a hard coded path in the solrconfig.xml and changed it. Then the browse works.

But I changed it back and it still works with it.... I don't know why.

I will try experimenting again with the configuration and will ask you again if I don't make any progress....

Reading the Browse-Handler code and recompiling helps me to understand a bit how it works. Thanks for the tips.

regards Hannah

[Demian Katz]

It sounds like you’re making progress! Please let us know if you need more help, or if you find anything that looks like a bug we should fix to prevent future confusion for other users working with this code outside of the typical VuFind context.

[Hannah]

Hi Demian,

After long evaluation, sampling and testing I made a  dirty fix in the BrowseRequestHandler.java , but I can't find another way:

I changed the static final String DFLT_AUTH_CORE_NAME from authority to authority_ubfr (it is one of our cores) in line 791.

With this recompiled browse-handler.jar the alpha-browse works....

Now we have to test, if the browse-handler really searches in the right index.

Then the next problem was, that we can't made simultaneous searches in different browse.db / cores.

This was the error:

java.lang.UnsatisfiedLinkError: Native Library /tmp/sqlite-3.7.15-amd64-libsqlitejdbc.so already loaded in another classloader
Dez 19, 2018 9:07:21 AM org.vufind.solr.handler.Log info

The solution was  replacing the sqlite-jdbc-3.7.15-SNAPSHOT.jar (this came from vufind/solr/ )  with the newest one - sqlite-jdbc-3.25.2.jar

This is our current status

- Hannah

[Demian Katz]

Hannah,

 

It looks like the DFLT_AUTH_CORE_NAME is used as a default value when no

 

<str name="authCoreName">core</str>

 

is found in the <requestHandler> definition for the browse handler in solrconfig.xml.

 

It looks like VuFind relies on this default since the authCoreName is currently not set in the default configuration.

 

I think you may have mentioned trying to set this value in the past, but I could be misremembering. In any case, it seems to me that either you need to adjust your configuration, or else there’s a bug in the code that is preventing the configuration from working as expected (which would not be entirely surprising, since it has quite possibly never been used before). If you think there is a bug at work here, please let me know and I can spend a little time investigating.

 

I also haven’t seen that simultaneous search problem before, but it’s good to know that upgrading the sqlite-jdbc jar file is a workaround. I’m copying Mark and Tod into this reply in case they have any thoughts on pros and cons of upgrading the default jar included with the project. I’m actually a little confused about the current state of things, since the vufind-browse-handler project itself appears to come with a 10-year-old sqlitejdbc-v053.jar file, while VuFind ships with the sqlite-jdbc-3.7.15-SNAPSHOT.jar that you mentioned. I’m not sure how this came about, but it seems like a situation in need of cleaning up.

 

In any case, I’m glad you’ve found a way to get things up and running, and hopefully next time will be easier. Thanks for bringing our attention to these issues along the way!

[Tod Olson]

[Re-sending from my Gmail account...]

On the SQLite driver, that is a situation in need of cleanup. And I'm glad to see that the driver has been updated, as it now includes shared object libraries for all of Linux, Windows, Mac OS X, FreeBSD. Clearly we will want to upgrade this JDBC driver.

I don't know how we want to coordinate, though. I could upgrade the driver as part of my current work, or do it as a separate PR once this round of optimizing is done. That would at least make it clear in the history when the SQLite JDBC driver was upgraded.

-Tod

[Demian Katz]

I would recommend doing the SQLite upgrade as a separate PR for clarity. Does it need to wait until after the current work is done, or is it a completely independent issue? And any idea if the sqlite code found in the vufind-browse-handler repo is even being used for anything since it’s so old? Is it possible that we really just want to delete that and upgrade the one bundled with VuFind?

 

- Demian

 

From: Tod Olson <t...@uchicago.edu>
Sent: Wednesday, December 19, 2018 10:25 AM
To: Demian Katz <demia...@villanova.edu>
Cc: Tod Olson <t...@uchicago.edu>; solrma...@googlegroups.com; Mark Triggs <m...@dishevelled.net>
Subject: Re: [solrmarc-tech] alpha-browse

 

On the SQLite driver, that is a situation in need of cleanup. And I'm glad to see that the driver has been updated, as it now includes shared object libraries for all of Linux, Windows, Mac OS X, FreeBSD. Clearly we will want to upgrade this JDBC driver.

 

I don't know how we want to coordinate, though. I could upgrade the driver as part of my current work, or do it as a separate PR once this round of optimizing is done. That would at least make it clear in the history when the SQLite JDBC driver was upgraded.

 

-Tod

On Dec 19, 2018, at 7:26 AM, Demian Katz <demia...@villanova.edu> wrote:

 

[Tod Olson]

It's independent. I'd kind of like to do it after, just for simplicity. I think the current work is pretty close to done.

-Tod

[Demian Katz]

Fine with me!

[Hannah]

I'm back again, with the next problem...

I try to explain:

We have 2 Index - Servers - on each Server we have the same Solr-Solrmarc installation from our git.

So each Server works with the same library-classes and jar files - java version...the only obvious different is the service pack...

But I have different views / conduct of the AlphaBrowse presentation on our VuFind Server.

####

First Server with SLES 12 SP3 - the view is correct with

Goethe

Use instead: Goethe, Johann Wolfgang von 1749-1832

2019-02-12 10:14:32.792 INFO  (qtp1060830840-20) [   x:ubfr] o.a.s.c.S.Request [ubfr]  webapp=/solr path=/browse params={json.nl=arrarr&offset=0&from=goethe&source=author&rows=20&wt=json} status=0 QTime=1344

"items":[{
        "note":"",
        "heading":"Goethe",
        "useInstead":["Goethe, Johann Wolfgang von 1749-1832"],
        "count":0,
        "ids":[],
        "extras":{"":[]},
        "sort_key":"Goethe",
        "seeAlso":[]},
      {

#####

Second Server with SLES 12 SP4 - view is not correct:

the "use instead" is missing.

{
        "note":"",
        "heading":"Goethe",
        "useInstead":[],
        "count":0,
        "ids":[],
        "extras":{"":[]},
        "sort_key":"Goethe",
        "seeAlso":[]},
      {

2019-02-12 10:09:57.293 INFO  (qtp1060830840-22) [   x:ubfr] o.a.s.c.S.Request [ubfr]  webapp=/solr path=/browse params={json.nl=arrarr&offset=0&from=goethe&source=author&rows=20&wt=json} status=0 QTime=26

The output in solr.log is the same...

When I copy the browse.db files from one server to the other server , the effect is the same.

Its not relevant, where we make the browse.db files, but relevant where we use it.

On my archlinux I have the view with "use instead", other test server with OpenSuse not.

Do you have any tip, what we can do? We have to go productive with the new solr version - but without alphabrowse it makes no sense....

regards

Hannah

[Demian Katz]

That's quite odd! A few questions to think about:

1. How easily can you build a new test environment? It might be interesting to clone the working environment, then upgrade the service pack and see if it breaks.

2. Have you looked at the service pack changelog to see if there are relevant changes?

3. Have you compared sqlite versions in case that's a factor?

- Demian

Get Outlook for Android

---

From: solrma...@googlegroups.com <solrma...@googlegroups.com> on behalf of Hannah <krook...@gmail.com>

Sent: Tuesday, February 12, 2019 3:52:25 AM
To: solrmarc-tech
Subject: [solrmarc-tech] Re: alpha-browse

 

[Tod Olson]

And one more question:

4. The "Use Instead" and other syndetic relations in the author browse list come from the authority core. Do the authority cores have the same information for that heading?

-Tod

[Hannah]

my answers below

Am Dienstag, 12. Februar 2019 19:52:15 UTC+1 schrieb Tod Olson:

And one more question:

4. The "Use Instead" and other syndetic relations in the author browse list come from the authority core. Do the authority cores have the same information for that heading?

Yes - the Indexing routines are using (no matter on which server) the same solrconfig.xml and schema

-Tod

On Feb 12, 2019, at 9:08 AM, Demian Katz <demia...@villanova.edu> wrote:

That's quite odd! A few questions to think about:

1. How easily can you build a new test environment? It might be interesting to clone the working environment, then upgrade the service pack and see if it breaks.

this was our idea too, but we didn't want to spend so much time...

I will ask our admin if he can make a snapshot of the server and then upgrade to sp4

 

2. Have you looked at the service pack changelog to see if there are relevant changes?

yes we did. we found changes in curl and sqlite. not sure if this is the reason for the wrong view...

 

3. Have you compared sqlite versions in case that's a factor?

we did this also.

SP3:

rpm -qa | fgrep -i sqlite | sort
libqt4-sql-sqlite-4.8.7-8.6.1.x86_64
libsqlite3-0-3.8.10.2-8.1.x86_64
libsqlite3-0-32bit-3.8.10.2-8.1.x86_64
php7-sqlite-7.0.7-50.56.2.x86_64
sqlite3-3.8.10.2-8.1.x86_64

SP4:

rpm -qa | fgrep -i sqlite | sort
libqt4-sql-sqlite-4.8.7-8.8.1.x86_64
libsqlite3-0-3.8.10.2-8.1.x86_64
libsqlite3-0-32bit-3.8.10.2-8.1.x86_64
perl-DBD-SQLite-1.40-3.175.x86_64
php7-sqlite-7.0.7-50.56.2.x86_64
sqlite3-3.8.10.2-8.1.x86_64

there are differences. But we have assumed, the alphabrowse uses the squlite jar files from the solr instance?

- Hannah

[Demian Katz]

Hannah,

Regarding Tod's question, have you confirmed that the data in the indexes is identical by querying the offending records through the Solr admin panel? It's worth checking whether the data is getting lost at index creation time or at data retrieval time.

Regarding the SQLite changes, I don't see major differences in the output you shared (apart from the addition of a connector library that is probably not relevant). However, if there really are changes there, they might be significant -- I'm pretty sure (though not 100% certain) that the SQLite jar included with the package is used to connect to the SQLite software installed on the server, so if the underlying software changes, it could have an impact even though the same connector jar is being used.

- Demian

---

From: solrma...@googlegroups.com <solrma...@googlegroups.com> on behalf of Hannah <krook...@gmail.com>

Sent: Wednesday, February 13, 2019 7:14 AM
To: solrmarc-tech
Subject: Re: [solrmarc-tech] Re: alpha-browse

 

[Hannah]

Hello Demian,

I looked at both authority indexes.We use on every server the same schema and solrconfig and properties.

Below you can find an example:

SLES 12 SP 3

"id":"161142842",
        "source":"Unknown",
        "record_type":"Unknown",
        "heading":"Goethe, Johann Wolfgang von 1749-1832",
        "use_for":["Goethe, Johann Wolfgang",
          "Goethe, Johan Wolfgang von",
          "Goethe, Johan Wolphgang",
          "Goethe, Johan W. von",
          "Goethe, Joh. Wolfg. v.",
          "Goethe, J. Wolfgang",
          "Goethe, J. W. v.",
          "Goethe, J. W.",
          ............
          .....

SLES 12 SP4:

"id":"161142842",
        "source":"Unknown",
        "record_type":"Unknown",
        "heading":"Goethe, Johann Wolfgang von 1749-1832",
        "use_for":["Goethe, Johann Wolfgang",
          "Goethe, Johan Wolfgang von",
          "Goethe, Johan Wolphgang",
          "Goethe, Johan W. von",
          "Goethe, Joh. Wolfg. v.",
          "Goethe, J. Wolfgang",
          "Goethe, J. W. v.",
          "Goethe, J. W.",
           .....
           .....

marc_auth.properties:

id = 001
source = "Unknown"
record_type = "Unknown"

fullrecord = FullRecordAsMarc
allfields = custom, getAllSearchableFields(100, 900)

heading = custom,getHeading
use_for = 400abctx:410abctx:411a:430abcdtx:448abcdtx:450abvtx:451atxg:455avtx
see_also = 880a

managed-schema:

  <fieldType name="string" class="solr.StrField" omitNorms="true" sortMissingLast="true"/>

  <fieldType name="text" class="solr.TextField" positionIncrementGap="100">

  <field name="heading" type="string" indexed="true" stored="true"/>

  <field name="heading_keywords" type="text" indexed="true" stored="false"/>

  <field name="id" type="string" indexed="true" stored="true"/>

  <copyField source="heading" dest="heading_keywords"/>

  <copyField source="see_also" dest="see_also_keywords"/>

  <copyField source="use_for" dest="use_for_keywords"/>

regards

Hannah

[Demian Katz]

Hmm, this is quite a mysterious situation!

Have you tried rebuilding the browse handler on the problematic machine to see if that makes a difference? I wonder if the newest master code (which is also included in release 5.1) will behave any differently from the version you're using (though be careful in case of backward incompatibilities). It might also be worth putting some logging into the code to try to trace this, if other methods of troubleshooting don't prove productive. I'm not sure where best to begin (maybe Tod has a suggestion), but I can take a look when I have a bit more free time if you need suggestions!

- Demian

---

From: solrma...@googlegroups.com <solrma...@googlegroups.com> on behalf of Hannah <krook...@gmail.com>

Sent: Thursday, February 14, 2019 3:57 AM

[Hannah]

Our latest status:

we upgraded the server from SP3 to SP4.

Updated and checked all software packages and  both servers are now identical.

Testing the alpha-browse: nothing has changed - the upgraded Server still shows the right result list with "use instead".

a) Then we copied again (only) the core - alphabrowse-index and the alphabrowse.db files from the older server to the problematic server - nothing changed. the view is still wrong.

b) Then we copied with rsync the complete structure (with jetty-solr-solrmarc ; all indexes ) from the server with the correct view to the other problematic server.

After a reboot we got the right answer with "use instead"  on the problematic server - but why? we can't find any difference between the folders.

This is our project structure:

rdsindex2 -

   -- solr

       -- solr.xml

       -- alphabetical_browse ( e.g. cbfrauthor_browse.db )

       -- authority_cbfr ( conf  core.properties  index snapshot_metadata  tlog)

       -- jars

       -- cbfr ( conf  core.properties  index  snapshot_metadata  spellShingle  spellchecker  tlog )

   -- solrjetty ( bin  contrib  dist  docs  example  licenses  server )

   -- solrmarc ( bin index_java index_scripts lib lib_local logs translation_maps xsl browse-indexing.jar log4j.properties marc.properties import.properties ... )

What can we do now? Is there any debug option that we can use?

On our index-server we have no vufind instance, so its problematic to rebuild the browse-handler.

-Hannah

[Demian Katz]

Hannah,

 

Is it possible that there were differences in the Solr indexes between the boxes? The generated alphabrowse index is only part of what the browse handler does – it also does some real-time Solr lookups. So if the index files are the same, and the browse handler code is the same, but the results are different, the next logical candidate to look at for inconsistency would be the Solr index itself.

 

- Demian

 

From: solrma...@googlegroups.com <solrma...@googlegroups.com> On Behalf Of Hannah

[kro...@gmx.net]

Hi together,

I have problems with the right boosting for our library catalogue.
We are using VuFind with searchspecs.yaml

Example:

Searching 'ta:(Psychologie heute)' - this ta field is an exact title search.

and 'ta' is a multi valued field

in the result_list are a.o. this titles:

Titel 1

[ti_short] => Mode
...
[ta] => Array
    (
    [0] => Mode
    [1] => Psychologie heute
   )

Titel 2:

[ti_short] => Psychologie heute
...
[ta] => Array
  (
    [0] => Psychologie heute
    [1] => Taschenbuch
  )

We want Titel 2 before Titel 1.

is it possible to define a boosting in the DismaxParams, like
if( ti_short == query ) --> boost^500
or
if ( ta[0] == query ) --> boost^500 ?


I tried with:

- [bq, ti_short:"Psychologie heute"^500] and the order of the hits is correct.

but this: - [bq, ti_short:"{!q}"^500] don't work



Any help would be greatly appreciated
regards Hannah