[Dspace-tech] Search for date in srw/u

26 views
Skip to first unread message

Klaus Hessellund

unread,
Aug 25, 2015, 11:11:46 AM8/25/15
to dspac...@lists.sourceforge.net
Hi List,

I'm running the lastest dspace with srw.

I want to search records from a specific date in srw. ie 2008-09-10

I did try to make a search index in dspace.cfg .
search.index.15 = datecreated:dc.date.created

and then use datecreated in srw. It seems like srw/lucene only likes
words and not dates. ?

Regards,
Klaus



Mika Stenberg

unread,
Aug 25, 2015, 11:11:48 AM8/25/15
to Klaus Hessellund, dspac...@lists.sourceforge.net
I tried the same with an identifier we've added. We configured it in
dspace.cfg and ran index-inits. I am unable to search using it with srw.


-Mika
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> DSpace-tech mailing list
> DSpac...@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/dspace-tech
>
>


LeVan,Ralph

unread,
Aug 25, 2015, 11:11:53 AM8/25/15
to Mika Stenberg, Klaus Hessellund, dspac...@lists.sourceforge.net
The questions that leap to mind are: do you have an SRW configuration
problem or a Lucene problem? Did the indexes get built and are just not
searchable? Did the indexes get built, but the terms extracted don't
actually look like the terms you are using to search.

Fortunately, there's an easy test for these questions.

Bring up the screen with the rendered user interface to SRW. On the
right side is an area for Browse. This feature lets you look at the
terms in your index.

First, pull down the list of indexes. Is your index on the list? If
so, then you've got the SRW configuration right. If not, then you've
got a configuration problem.

Next, if your index is on the list, then select it. Put a seed term
into the term box. Browse is trying to return a moving window onto your
index. So, the numbers it wants from you are the number of terms to
return and where your seed term should appear in the list. Do you want
your seed term at the top or in the middle or even at the bottom?
Sadly, Lucene does not support browsing backwards through its indexes,
so the only legal values for response position are 1 or less. I don't
remember if there's a limit on the maximum number of terms returned. I
know I have applications where I ask for a thousand terms at a time.
You should probably pick something sensible like 10 for this test.

Do you get any terms back? If so, do they look like what you expected?
If so, can you click on them? The interface has rendered the returned
terms as hot links with the term embedded in a query. It is a useful
way to see how to formulate those queries. If the term doesn't look
like you expected, perhaps you didn't understand what was actually in
the record or perhaps you used an inappropriate indexing method in
Lucene. If the term does look like you expected and you don't get a
good result when you click on the term, then something is messed up and
you and I probably need to have a more detailed conversation. If you
get no terms back, then there still might be a configuration problem.
Whether the problem is in the SRW server or the Lucene database would
require deeper investigation.

I'm ralphlevan on AOL and Yahoo chat if you want to talk about this more
over there or we can continue this via email. (I find email tedious for
the quick back-and-forth questions of remote debugging.) If we do this
via chat, I promise we'll report something back to the list.

Ralph
-----------------------------------------------------------------------

Klaus Hessellund

unread,
Aug 25, 2015, 11:12:11 AM8/25/15
to LeVan,Ralph, dspac...@lists.sourceforge.net, Mika Stenberg
I get a NPE and sometimes other faults like
"java.lang.ArrayStoreException" when I try using the browse/term box on
the right side, so I don't get far..
Actually I get an exception no matter what what I search for in the
browse box.

Btw, It works fine when I setup searchindex for other fields which
contains words like "FLOS" . So I think I have pretty good understanding
of setting up srw :p

I guess its not srw fault as I can't search for at date in dspace either.

I don't have AOL or Yahoo :-( ICQ or msn i have.

My config for setting up search index for date.

dspace.cfg :

search.index.15 = dateissued:dc.date.issued


DSpace.SRWDatabase.props :


indexSynonym.dc.date.issued=datecreated
.
.
qualifier.dc.date.issued = 1=1005


Exception output :

<soapenv:Envelope>

<soapenv:Body>

<soapenv:Fault>
<faultcode>soapenv:Server.userException</faultcode>

<faultstring>
java.rmi.RemoteException: null; nested exception is:
java.lang.ArrayStoreException
</faultstring>

<detail>

<ns1:stackTrace>
java.rmi.RemoteException: null; nested exception is:
java.lang.ArrayStoreException
at
gov.loc.www.zing.srw.srw_bindings.SRWSoapBindingImpl.scanOperation(SRWSoapBindingImpl.java:250)
at
gov.loc.www.zing.srw.srw_bindings.SRWSoapBindingSkeleton.scanOperation(SRWSoapBindingSkeleton.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
at
org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
at org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
at ORG.oclc.os.SRW.SRWServlet.processMethodRequest(SRWServlet.java:1397)
at ORG.oclc.os.SRW.SRWServlet.doGet(SRWServlet.java:296)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at
org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.ArrayStoreException
at java.lang.System.arraycopy(Native Method)
at java.util.ArrayList.toArray(ArrayList.java:304)
at
java.util.Collections$UnmodifiableCollection.toArray(Collections.java:1002)
at org.dspace.browse.BrowseInfo.getStringResults(BrowseInfo.java:489)
at
ORG.oclc.os.SRW.DSpaceLucene.SRWLuceneDatabase.getTermList(SRWLuceneDatabase.java:277)
at ORG.oclc.os.SRW.SRWDatabase.doRequest(SRWDatabase.java:287)
at
gov.loc.www.zing.srw.srw_bindings.SRWSoapBindingImpl.scanOperation(SRWSoapBindingImpl.java:212)
... 33 more
</ns1:stackTrace>
<ns2:hostname>dspacetest</ns2:hostname>
</detail>
</soapenv:Fault>
</soapenv:Body>
</soapenv:Envelope>

LeVan,Ralph

unread,
Aug 25, 2015, 11:12:12 AM8/25/15
to Klaus Hessellund, dspac...@lists.sourceforge.net, Mika Stenberg
Can you browse on any index? It is critical that you set the value in "response position" to 1 for Lucene. Omitting that value will result in an error.

I've got an ICQ account now too. 356480764 or ralphlevan (assuming I understand what they mean by nicknames).

Ralph

> -----Original Message-----
> From: Klaus Hessellund [mailto:kh....@cbs.dk]
> Sent: Friday, September 12, 2008 8:27 AM
> To: LeVan,Ralph
> Cc: Mika Stenberg; dspac...@lists.sourceforge.net
> Subject: Re: [Dspace-tech] Search for date in srw/u
>

Klaus Hessellund

unread,
Aug 25, 2015, 11:12:13 AM8/25/15
to LeVan,Ralph, dspac...@lists.sourceforge.net, Mika Stenberg
I can't browse any indexes. If I search dc.creator for "Hansen" and set
Response position" to 1, I get this exception :

<soapenv:Envelope>

<soapenv:Body>

<soapenv:Fault>
<faultcode>soapenv:Server.userException</faultcode>
<faultstring>java.lang.reflect.InvocationTargetException</faultstring>

<detail>

<ns1:stackTrace>
java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.axis.providers.java.RPCProvider.invokeMethod(RPCProvider.java:397)
at
org.apache.axis.providers.java.RPCProvider.processMessage(RPCProvider.java:186)
at
org.apache.axis.providers.java.JavaProvider.invoke(JavaProvider.java:323)
at
org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)
at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)
at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)
at
org.apache.axis.handlers.soap.SOAPService.invoke(SOAPService.java:454)
at org.apache.axis.server.AxisServer.invoke(AxisServer.java:281)
at ORG.oclc.os.SRW.SRWServlet.processMethodRequest(SRWServlet.java:1397)
at ORG.oclc.os.SRW.SRWServlet.doGet(SRWServlet.java:296)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:690)
at
org.apache.axis.transport.http.AxisServletBase.service(AxisServletBase.java:327)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:200)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:283)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:773)
at
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
at
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
at
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685)
at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.NoClassDefFoundError: com/ibm/icu/text/Normalizer
at
org.dspace.text.filter.DecomposeDiactritics.filter(DecomposeDiactritics.java:54)
at
org.dspace.sort.AbstractTextFilterOFD.makeSortString(AbstractTextFilterOFD.java:116)
at org.dspace.sort.OrderFormat.makeSortString(OrderFormat.java:125)
at
org.dspace.browse.BrowseEngine.normalizeJumpToValue(BrowseEngine.java:720)
at org.dspace.browse.BrowseEngine.browseByValue(BrowseEngine.java:487)
at org.dspace.browse.BrowseEngine.browse(BrowseEngine.java:128)
at
ORG.oclc.os.SRW.DSpaceLucene.SRWLuceneDatabase.getTermList(SRWLuceneDatabase.java:276)
at ORG.oclc.os.SRW.SRWDatabase.doRequest(SRWDatabase.java:287)
at
gov.loc.www.zing.srw.srw_bindings.SRWSoapBindingImpl.scanOperation(SRWSoapBindingImpl.java:212)
at
gov.loc.www.zing.srw.srw_bindings.SRWSoapBindingSkeleton.scanOperation(SRWSoapBindingSkeleton.java:74)
... 32 more

LeVan,Ralph

unread,
Aug 25, 2015, 11:12:15 AM8/25/15
to Klaus Hessellund, dspac...@lists.sourceforge.net, Mika Stenberg
I'm not denying responsibility for this problem, but it looks like a DSpace issue. If you walk down that stack trace far enough, you run into this:
Caused by: java.lang.NoClassDefFoundError: com/ibm/icu/text/Normalizer
at
org.dspace.text.filter.DecomposeDiactritics.filter(DecomposeDiactritics.java:54)
at
org.dspace.sort.AbstractTextFilterOFD.makeSortString(AbstractTextFilterOFD.java:116)
at org.dspace.sort.OrderFormat.makeSortString(OrderFormat.java:125)
at
org.dspace.browse.BrowseEngine.normalizeJumpToValue(BrowseEngine.java:720)
at org.dspace.browse.BrowseEngine.browseByValue(BrowseEngine.java:487)
at org.dspace.browse.BrowseEngine.browse(BrowseEngine.java:128)
at
ORG.oclc.os.SRW.DSpaceLucene.SRWLuceneDatabase.getTermList(SRWLuceneDatabase.java:276)

In other words, I've asked DSpace to return a list of terms to me and in doing that it's invoked its DecomposeDiacritics filter and that filter tried to call com.ibm.icu.text.Normalizer, which it couldn't find.

Now, as it happens, I believe that names are handled internally by DSpace and not left to Lucene to handle. You might want to try browsing on yet another index and see what happens.

Ralph


> -----Original Message-----
> From: Klaus Hessellund [mailto:kh....@cbs.dk]
> Sent: Friday, September 12, 2008 10:12 AM
> To: LeVan,Ralph
> Cc: Mika Stenberg; dspac...@lists.sourceforge.net
> Subject: Re: [Dspace-tech] Search for date in srw/u
>
> I can't browse any indexes. If I search dc.creator for "Hansen" and
> set
> Response position" to 1, I get this exception :
>
> <soapenv:Envelope>
> −
> <soapenv:Body>
> −
> <soapenv:Fault>
> <faultcode>soapenv:Server.userException</faultcode>
> <faultstring>java.lang.reflect.InvocationTargetException</faultstring>
> −
> <detail>
> −
> <ns1:stackTrace>
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> Caused by: java.lang.NoClassDefFoundError: com/ibm/icu/text/Normalizer
> at
> org.dspace.text.filter.DecomposeDiactritics.filter(DecomposeDiactritics
> .java:54)
> at
> org.dspace.sort.AbstractTextFilterOFD.makeSortString(AbstractTextFilter

Mika Stenberg

unread,
Aug 25, 2015, 11:12:29 AM8/25/15
to dspac...@lists.sourceforge.net
Should DSpace update the RSS feeds in realtime (1.5.1) ?
It seems that the archive-wide RSS 1.0 is up to date, but I can only see
older submissions using RSS 2.0 no matter how much I refresh / reload
the screen.

Whats the catch here?


Thanks,
Mika


Klaus Hessellund

unread,
Aug 25, 2015, 11:12:29 AM8/25/15
to LeVan,Ralph, dspac...@lists.sourceforge.net, Mika Stenberg
Hi Ralph,

I've tried all searches indexes and they all come with an exception.

I've just installed 1.5.1 release instead of svn version and get same
result.

Is there any way, where I can search for year and month as I can in dspace ?

f.ex if I want all post from 2005-08, i can use :
http://hostname/xmlui/browse?rpp=20&etal=-1&type=dateissued&sort_by=2&order=ASC&month=12&year=2005

So year and month are some how indexed, and the question is if I can do
something simelar in srw ?

Unfortnately I can't specify day in month and dspace gives me posts that
is newer than the specified date. That is, I get posts from the
specified date and later. It is better that nothing and I can strip
newer posts in another script. That would be a workaround for me ;-)


/Klaus

LeVan,Ralph

unread,
Aug 25, 2015, 11:12:33 AM8/25/15
to Klaus Hessellund, dspac...@lists.sourceforge.net, Mika Stenberg
SRW Index Browse does not work in DSpace 1.5. Mark Diggory has a programmer working on that now.

The example URL that you attached does not look like a search to me. It looks like a browse with qualifiers. Your comment that it doesn't actually return only documents with those dates reinforces that impression.

Are you really sure there are year and month date indexes in Lucene? Anyone closer to the application have an opinion?

Ralph

> -----Original Message-----
> From: Klaus Hessellund [mailto:kh....@cbs.dk]
> Sent: Monday, September 15, 2008 4:05 AM
> To: LeVan,Ralph
> Cc: Mika Stenberg; dspac...@lists.sourceforge.net
> Subject: Re: [Dspace-tech] Search for date in srw/u
>
> Hi Ralph,
>
> I've tried all searches indexes and they all come with an exception.
>
> I've just installed 1.5.1 release instead of svn version and get same
> result.
>
> Is there any way, where I can search for year and month as I can in
> dspace ?
>
> f.ex if I want all post from 2005-08, i can use :
> http://hostname/xmlui/browse?rpp=20&etal=-
> 1&type=dateissued&sort_by=2&order=ASC&month=12&year=2005
>
> So year and month are some how indexed, and the question is if I can do
> something simelar in srw ?
>
> Unfortnately I can't specify day in month and dspace gives me posts
> that
> is newer than the specified date. That is, I get posts from the
> specified date and later. It is better that nothing and I can strip
> newer posts in another script. That would be a workaround for me ;-)
>
>
> /Klaus
>
>
>
>
>
>
> LeVan,Ralph wrote:
> > I'm not denying responsibility for this problem, but it looks like a
> DSpace issue. If you walk down that stack trace far enough, you run
> into this:
> > Caused by: java.lang.NoClassDefFoundError:
> com/ibm/icu/text/Normalizer
> > at
> >

Klaus Hessellund

unread,
Aug 25, 2015, 11:12:39 AM8/25/15
to LeVan,Ralph, dspac...@lists.sourceforge.net, Mika Stenberg
No, Actually I have no idea if its in lucene or not. :-$ . I meant it as
a question. To me it just looked like it was indexed somehow.

Bottom line. Search for a specific date in srw would make the day :-)

/Klaus
Reply all
Reply to author
Forward
0 new messages