Fwd: [CODE4LIB] Millions of Harvard Library Catalog Records Publicly Available

[Tom Cramer]

Here's an ample source of full records that could be added to the BL demo app's index. 

- Tom

Begin forwarded message:

From: Will Kurt <wck...@GMAIL.COM>

Date: April 24, 2012 1:43:04 PM PDT

To: CODE...@LISTSERV.ND.EDU

Subject: [CODE4LIB] Millions of Harvard Library Catalog Records Publicly Available

Reply-To: Code for Libraries <CODE...@LISTSERV.ND.EDU>

Apologies if this is old news, but I was very excited to see Harvard
making all this data public:
http://isites.harvard.edu/icb/icb.do?keyword=k77982&pageid=icb.page498373

Tons of cool data analysis / machine learning work to be done here!
Warm up your SVMs ;)

--Will

[Louis St-Amour]

I'm interested in getting started with the Harvard data and Blacklight, and ultimately in helping others to do so. 

Apologies in advance if this seems silly, I've only read webpages and haven't yet tried installing and running Blacklight locally. But the Issues on Github makes it sound as if Blacklight is less of a product and more of a "starter kit" in Rails + SolrMarc, and that configuring Blacklight for the catalogue data you have in Solr (the schema) could be tricky.

While not employed at a library, but wanting to hack away at systems like Blacklight or VuFind for curiosity's sake, I downloaded the Harvard dataset but once extracted I found it hard to begin. Newer activities around NoSQL like BibServer seem to require BibJSON and don't scale well yet. Past experience showed me that the last thing I want to be doing is writing my own Marc interpreter, particularly if there are any charset issues. But my goal is to try out some machine learning algorithms or new visualizations of catalogue data, and I love Ruby and Rails.

Do you think it would be feasible to work with 12 million records on a single (8GB RAM) Mac mini with an SSD? Any pointers or tips for someone trying out Blacklight with data on that scale? Should I be using JRuby via RVM? I take it that most of the work I'll be doing in regards to the data will be with SolrMarc, given the 2009 date is this ancient or still good advice? http://code.google.com/p/solrmarc/wiki/BlacklightHowTo

I don't mind contributing back to the community with a write-up on my experience. Heck, I'd even turn it into a homebrew script or import wizard. I'm just looking for any advice / up-to-date documentation on getting started with a pre-existing Marc21 open dataset.

Thanks very much in advance,

Louis St-Amour

[Bess Sadler]

Hi, Louis, welcome! Try the basic blacklight install & take a look at the index process. It comes packaged with a small test set, and a slightly larger set of Marc records from the library of congress set. If you get those running the way you want, you'll be in good shape to troubleshoot any weirdness you run into from the harvard set. 

You'll be able to do lots with 8G of ram. I highly recommend starting with a small, known good data set, get a feel for working with Marc, and then tackle the Harvard records. You will almost certainly uncover malformed records and other weirdness (google "marcthulu" if you want to help slay those beasts).

I was so pleased to hear that Harvard had released this data set and I'm super excited to see someone running with it. Please keep us posted on your project!

Best wishes,

Bess

--
You received this message because you are subscribed to the Google Groups "Blacklight Development" group.
To view this discussion on the web visit https://groups.google.com/d/msg/blacklight-development/-/FYLUXp66D3YJ.
To post to this group, send email to blacklight-...@googlegroups.com.
To unsubscribe from this group, send email to blacklight-develo...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/blacklight-development?hl=en.

[Louis St-Amour]

Well it wasn't half as hard as I expected, though Solr is a big black box of mystery. About all I can figure out from Google is I query using XML and sending <delete><query>*:*</query></delete> with a commit param does a nice job cleaning the slate for me to try a new import.

Works pretty darn fast between the RAM, SSD and data files stored on an external drive. Still I'm impressed with the speed. It's going at about 1000 reconds a second and I could probably get it going even faster by moving the files on to the SSD itself. At this rate it shouldn't take more than 3.5 hours to index 12 million records ... assuming they're valid and parse right. The hardest thing to set up was a ln -s between the volume and a data subdirectory, since the rake importer can't handle marc filenames with spaces (or whatever it was...)

As expected, the marcthulu bug hit me from the Harvard dataset. I was expecting it given the trials and tribulations of others' - e.g. "This is the first version to parse the entire dataset without crashing" commit in https://github.com/aristus/copymine-harvard/commit/c84b0c373cfc7a59cdcf5a50a768815c5df45a1a

A sampling of errors, these were the most common fields. How should I go about fixing these? Edit the importer to prevent multiple values for fields being submitted to Solr, if that's possible, or is there an automatic tool to fix Marc records? As I understand it, I should think of Marc as a document syntax (markup language) rather than a formal specification, so perhaps there are no easy answers, except to ignore the invalid entries given the size of the dataset and scope of my interest in it (I'm not Harvard!)

SEVERE: org.apache.solr.common.SolrException: ERROR: [doc=000001189-4] multiple values encountered for non multiValued field subtitle_display: [in history, religion, legend and myth, brochure]

SEVERE: org.apache.solr.common.SolrException: ERROR: [doc=000003217-4] multiple values encountered for non multiValued field subtitle_display: [drama giocoso per musica, da rappresentarsi nel nobils. pubblico teatro di Clagenfurt nella primavera dell'anno 1778]

SEVERE: org.apache.solr.common.SolrException: ERROR: [doc=000003256-5] multiple values encountered for non multiValued field subtitle_display: [dramma giocoso in due atti = La femme capricieuse, La femme capricieuse : opera buffon en deux actes]

...

SEVERE: org.apache.solr.common.SolrException: ERROR: [doc=000006191-3] multiple values encountered for non multiValued field title_vern_display: [Berit rishonim, ???? ???????]

...

SEVERE: org.apache.solr.common.SolrException: ERROR: [doc=000009604-0] multiple values encountered for non multiValued field author_vern_display: [Bernsh?en, Tsevi, 1914-, ???????, ???, 1914?]

...

SEVERE: org.apache.solr.common.SolrException: ERROR: [doc=000089405-2] multiple values encountered for non multiValued field author_display: [Colombi, Arturo, Partito comunista italiano]

...

SEVERE: org.apache.solr.common.SolrException: ERROR: [doc=000106736-2] multiple values encountered for non multiValued field title_display: [Letters, Oxford]

Finally, mysteriously, it halts indexing after 183,999 records:

 INFO [main] (MarcImporter.java:308) - Added record 183999 read from file: 000194278-6

Apr 30, 2012 9:38:54 AM org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer addToQueue

SEVERE: Timeout inserting documents into queue (5 minutes ) - problem occured - exiting

ERROR [main] (MarcImporter.java:363) - Unable to index record 000194279-4 (record count 184000) -- Timeout inserting documents into queue (5 minutes )

java.lang.RuntimeException: Timeout inserting documents into queue (5 minutes )

at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.addToQueue(StreamingUpdateSolrServer.java:184)

at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.add(StreamingUpdateSolrServer.java:152)

at org.solrmarc.solr.SolrServerProxy.addDoc(SolrServerProxy.java:46)

at org.solrmarc.marc.MarcImporter.addToIndex(MarcImporter.java:439)

at org.solrmarc.marc.MarcImporter.addToIndex(MarcImporter.java:383)

at org.solrmarc.marc.MarcImporter.importRecords(MarcImporter.java:304)

at org.solrmarc.marc.MarcImporter.handleAll(MarcImporter.java:572)

at org.solrmarc.marc.MarcImporter.main(MarcImporter.java:832)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at com.simontuffs.onejar.Boot.run(Boot.java:334)

at com.simontuffs.onejar.Boot.main(Boot.java:170)

ERROR [main] (MarcImporter.java:366) - ******** Halting indexing! ********

... and there's no sign of error in my Solr terminal window. Well, I'll run it once more again to see if it was a fluke, or if it happens again, or worse.

Hmm, it seems to have happened again, this time on 173,800 -- 

Exception in thread "main" java.lang.reflect.InvocationTargetException

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

at java.lang.reflect.Method.invoke(Method.java:597)

at com.simontuffs.onejar.Boot.run(Boot.java:334)

at com.simontuffs.onejar.Boot.main(Boot.java:170)

Caused by: java.lang.RuntimeException: Timeout inserting documents into queue (5 minutes )

at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.addToQueue(StreamingUpdateSolrServer.java:184)

at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.flush(StreamingUpdateSolrServer.java:277)

at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.blockUntilFinished(StreamingUpdateSolrServer.java:255)

at org.apache.solr.client.solrj.impl.StreamingUpdateSolrServer.request(StreamingUpdateSolrServer.java:125)

at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:105)

at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:86)

at org.apache.solr.client.solrj.SolrServer.commit(SolrServer.java:75)

at org.solrmarc.solr.SolrServerProxy.commit(SolrServerProxy.java:77)

at org.solrmarc.marc.MarcImporter.finish(MarcImporter.java:457)

at org.solrmarc.marc.MarcImporter.handleAll(MarcImporter.java:585)

at org.solrmarc.marc.MarcImporter.main(MarcImporter.java:832)

... 6 more

ERROR [main] (MarcImporter.java:366) - ******** Halting indexing! ********

Is this something I should take to the Solr groups, or would anyone have any tips to debug this? Is there a way to start from a record in the middle of a MARC_FILE?

Louis.

[Sean Hannan]

At the stage you are at, what you are really dealing with is SolrMarc, which Blacklight bundles to help folks get MARC records into Solr. There is a discussion list for dealing with SolrMarc specifically: http://groups.google.com/group/solrmarc-tech/

-Sean

http://isites.harvard.edu/icb/icb.do?keyword=k77982&pageid=icb.page498373 <http://isites.harvard.edu/icb/icb.do?keyword=k77982&pageid=icb.page498373>

[Louis St-Amour]

Great, thanks Sean. I'll try there, and take a look at SolrMarc's issue log.

Louis.

[Jonathan Rochkind]

Easiest way to handle "multiple values encountered for non multiValued
field" errors is simply to change the field to a multi-valued field in
your Solr schema.

Another way is to change your SolrMarc config to only put the _first_
field value found in the field, not try to put them all there.

If you want to use the field for _sorting_ in Solr, it needs to be
non-multiValued. Sometimes I put things in a multiValued field, and
then duplicatively put the _first_ value in a single valued field for
sorting.

(Really, I think the out-of-the-box BL SolrMarc configs and Solr configs
ought to do one or both of these things, this is a problem that will hit
almost anyone with real world Marc data).

Can't say about your indexer crashing. "Timeout inserting documents into
queue (5 minutes )" kind of sounds like Solr is crashing, which can be
an annoyingly difficult thing to debug and fix. The SolrMarc listserv
might be helpful there. Or the Solr-user listserv.

> Hi, Louis, welcome! Try the basic blacklight install& take a look at the index process. It comes packaged with a small test set, and a slightly larger set of Marc records from the library of congress set. If you get those running the way you want, you'll be in good shape to troubleshoot any weirdness you run into from the harvard set.

[Chris Beer]

Hi Louis,

I was tinkering with the Harvard data last week too and ran into the
same problems. The multivalued issue has to do with the
Blacklight-distributed schema.xml (which can be easily modified),
although from what I gather, this may indicate some non-standard(?)
MARC.

I also had the problem with SolrMarc hanging, but persistence seemed
to be sufficient to index the whole dataset.

[Louis St-Amour]

Thanks for the multivalued tips, Chris, Jonathan. I've no idea if it'll cause trouble down the road, but as I'm using the jetty-solr provided by the generator, I edited jetty/solr/conf/schema.xml to set marc_display through author_vern_display to multiValued="true" and I haven't yet encountered an error. Yay!

I also seem to have fixed, or delayed, the crashing, by editing config/SolrMarc/config.properties to have absolute paths set for solrmarc.solr.war.path and solr.path. Doing so should allow direct access via SolrMarc to the Solr data store, which seems a bit slower, but is perhaps more reliable. I also gave the Java more RAM with export SOLRMARC_MEM_ARGS='-server -Xms2048m -Xmx7000m -XX:+UseParallelGC -XX:NewRatio=5' 

And it imported the first file with no problems. Awesome. Let's see what the end result looks like in 4 hours ... How long has it taken other people to import records? If I want to adjust the schema down the road, it'd be nice to get imports down to an hour or less. Then again, it probably doesn't matter, and Googling further reveals that the process appears to be CPU-bound or something, perhaps because of the Marc to Solr mapping... and people often get 200-500 a sec. So I'm doing well. If I want it faster, I suppose I should set up a central Solr server then distribute Marc files to different computers and run it over HTTP instead of the embedded Solr instance I'm using now to avoid that bug.

Louis.

[Jonathan Rochkind]

I'm only getting 90 records a second at best, so you're doing way better
than me already!

The process could very well be either CPU bound or disk bound.

However, to map from Marc to the solr fields we need, we need to do a
lot more logic than the "hello world" Solr examples that brag of fast
indexing times, so it could very well be unavoidably CPU bound in
SolrMarc. My own local example uses even more complex logic than the
out of the box example. It's possible SolrMarc could be optimized, but
it's also possible we're just doing a lot of work, and it's going to
take time.

(There are also various different ways for an indexing client to talk to
Solr, with possibly different performance implications. binary vs XML,
streaming server vs not, how many updates you batch together in a single
POST. I get confused figuring out what SolrMarc does by default, and
what the config is to change it. But it's possible you could change some
of these configs in your solrmarc to improve speed -- although,
actually, probably not relevant if you are using the 'direct to disk'
SolrMarc indexing method, it's not actually talking to an external
solrserver at all, it's just writing solr/lucene-compatible indexes
directly to disk itself. I myself only use HTTP -- it's _not_ faster
than what you're doing, but it is more bug-free in _my_ experience. Not
sure what you were running into.)

I (and many others) use the Solr replication feature to have an indexing
server and a search serving server. Then, if I need to re-index because
I've changed the schema, I reindex in my indexing server (without
bothering the search serving server at all, or causing any downtime),
and then replicate to the search serving server when I'm done.
(replication can send the new schema.xml over too). This is one way to
deal with long reindexing times. In development, I use a different
development solr index, and when I can avoid it, I don't work on a full
index, so I can change my schema and reindex quickly to see changes,
iteratively. I only do a full reindex when I really need to, or when I'm
ready to make the switch in production.

I guess this whole discussion at this point would really be better on
the SolrMarc list, it's mostly about SolrMarc, a little about Solr
itself, and not at all about Blacklight at this level we're talking about.

On 4/30/2012 1:52 PM, Louis St-Amour wrote:
> Thanks for the multivalued tips, Chris, Jonathan. I've no idea if it'll
> cause trouble down the road, but as I'm using the jetty-solr provided by

> the generator, I edited *jetty/solr/conf/schema.xml* to set
> *marc_display* through *author_vern_display* to *multiValued="true"* and

> I haven't yet encountered an error. Yay!
>
> I also seem to have fixed, or delayed, the crashing, by editing

> *config/SolrMarc/config.properties* to have absolute paths set for
> *solrmarc.solr.war.path* and *solr.path*. Doing so should allow direct

> access via SolrMarc to the Solr data store, which seems a bit slower,

> but is perhaps more reliable. I also gave the Java more RAM with *export

> SOLRMARC_MEM_ARGS='-server -Xms2048m -Xmx7000m -XX:+UseParallelGC

> -XX:NewRatio=5'*

>
> And it imported the first file with no problems. Awesome. Let's see what
> the end result looks like in 4 hours ... How long has it taken other
> people to import records? If I want to adjust the schema down the road,
> it'd be nice to get imports down to an hour or less. Then again, it
> probably doesn't matter, and Googling further reveals that the process
> appears to be CPU-bound or something, perhaps because of the Marc to
> Solr mapping... and people often get 200-500 a sec. So I'm doing well.
> If I want it faster, I suppose I should set up a central Solr server
> then distribute Marc files to different computers and run it over HTTP
> instead of the embedded Solr instance I'm using now to avoid that bug.
>
>
> Louis.
>
> On Monday, 30 April 2012 10:58:37 UTC-4, Chris Beer wrote:
>
> Hi Louis,
>
> I was tinkering with the Harvard data last week too and ran into the
> same problems. The multivalued issue has to do with the
> Blacklight-distributed schema.xml (which can be easily modified),
> although from what I gather, this may indicate some non-standard(?)
> MARC.
>
> I also had the problem with SolrMarc hanging, but persistence seemed
> to be sufficient to index the whole dataset.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Blacklight Development" group.

> To view this discussion on the web visit

> https://groups.google.com/d/msg/blacklight-development/-/7fPjG-m0Q98J.

[Louis St-Amour]

Well, to keep it a bit Blacklight-related, I found two things helpful in working with Harvard data just now, and it's caused me to go back and re-index the first two Marc files:

1. In app/models/solr_document.rb, set :marc_format_type to :marc21 ... reading source code, it appears that's the difference (or part of it) between the default non-multivalued title, subtitle and author fields and the multivalued ones the Harvard data requires.

2. And this is the one that made me redo the indexing -- don't make marc_display multivalued! It's what Blacklight appears to use when actually showing information from the Marc record. The rest, I assume, is for indexing/search results.

Louis.

> blacklight-development@googlegroups.com.

> To unsubscribe from this group, send email to

> blacklight-development+unsub...@googlegroups.com.

> For more options, visit this group at
> http://groups.google.com/group/blacklight-development?hl=en.

> blacklight-development@googlegroups.com.

> To unsubscribe from this group, send email to

> blacklight-development+unsub...@googlegroups.com.

[Chris Beer]

Hi Louis,

Does it make sense to collect all this information onto a new
Blacklight wiki page, it seems like there's some good information in
this thread?

Our wiki is public editable, so if you want to take a stab at it,
that'd be great. I'm sure others would provide additional details (esp
about MARC indexing strategies)

Thanks,
Chris

>> > blacklight-...@googlegroups.com.

>> > To unsubscribe from this group, send email to

>> > blacklight-develo...@googlegroups.com.

>> > blacklight-...@googlegroups.com.

>> > To unsubscribe from this group, send email to

>> > blacklight-develo...@googlegroups.com.

>> > For more options, visit this group at
>> > http://groups.google.com/group/blacklight-development?hl=en.
>

> --
> You received this message because you are subscribed to the Google Groups
> "Blacklight Development" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/blacklight-development/-/LrzEgE0aihsJ.

>
> To post to this group, send email to

> blacklight-...@googlegroups.com.

> To unsubscribe from this group, send email to

> blacklight-develo...@googlegroups.com.

[Jonathan Rochkind]

On 4/30/2012 2:23 PM, Louis St-Amour wrote:
> Well, to keep it a bit Blacklight-related, I found two things helpful in
> working with Harvard data just now, and it's caused me to go back and
> re-index the first two Marc files:
>

> 1. In *app/models/solr_document.rb*, set *:marc_format_type* to
> *:marc21* ... reading source code, it appears that's the difference (or

> part of it) between the default non-multivalued title, subtitle and
> author fields and the multivalued ones the Harvard data requires.

Hmm, if you are telling SolrMarc to store marc fields in marc binary (we
really should have called it iso_2709 not 'marc21'), then you need to
tell Blacklight to read it out that way.

That ought to have nothing to do with the multivalued issue though, I
can't think of why it would.

But did you find that our default SolrMarc config (write to marc
binary?) does not match our defualt blacklight config (read as marcxml?).

If that's so, we should fix it, to match. Please feel free to file an
issue if that's what you've found!

>
> 2. And this is the one that made me redo the indexing -- *don't make
> marc_display multivalued!* It's what Blacklight appears to use when

> actually showing information from the Marc record. The rest, I assume,
> is for indexing/search results.

Hmm, I _think_ that 'marc_display' may be the field we actually put the
raw solrmarc record in, for later retrieval. We probably ought to make
BL not raise if it's multi-valued, in case you want to put more than one
marc record in there, although default BL wont' _do_ anything with more
than one, and for non-advanced use cases you probably don't want to.