Stuck on rake solr:marc:index_test_data

[Paul Hoffman]

I'm struggling to get Blacklight running on an Ubuntu 14.04 box under
Tomcat 8. I've managed to get Tomcat and Solr running, but when I index
the sample MARC records and try to access http://localhost:3000/catalog
on the server -- using w3m within an ssh session, because my server is a
couple hundred miles away --I get the following error:

------------------------------------------------------------------------
Blacklight::Exceptions::InvalidRequest in CatalogController#index

RSolr::Error::Http - 404 Not Found Error: Not Found URI:
http://127.0.0.1:8983/solr/blacklight-core/select?wt=ruby&qt
=search&facet.field=format&facet.field=%7B%21ex%3Dpub_date_single%7Dpub_date&facet.field=subject_topic_facet&
facet.field=language_facet&facet.field=lc_1letter_facet&facet.field=subject_geo_facet&facet.field=subject_era_facet&
facet.query=pub_date%3A%5B2010+TO+*%5D&facet.query=pub_date%3A%5B2005+TO+*%5D&facet.query=pub_date%3A%5B1990+TO+*%5D&
facet.pivot=format%2Clanguage_facet&&&rows=10&facet=true&f.subject_topic_facet.facet.limit=21&
f.language_facet.facet.limit=11&sort=score+desc%2C+pub_date_sort+desc%2C+title_sort+asc
Backtrace: /var/lib/gems/
1.9.1/gems/rsolr-1.0.12/lib/rsolr/client.rb:284:in `adapt_response'
/var/lib/gems/1.9.1/gems/rsolr-1.0.12/lib/rsolr/
client.rb:190:in `execute'
/var/lib/gems/1.9.1/gems/rsolr-1.0.12/lib/rsolr/client.rb:176:in
`send_and_receive' /var/
lib/gems/1.9.1/gems/blacklight-5.14.0/lib/blacklight/solr_repository.rb:44:in
`block in send_and_receive' /usr/lib/
ruby/vendor_ruby/active_support/benchmarkable.rb:41:in `block in
benchmark' /usr/lib/ruby/vendor_ruby/active_support/
core_ext/benchmark.rb:12:in `block in ms'
/usr/lib/ruby/1.9.1/benchmark.rb:295:in `realtime' /usr/lib/ruby/
vendor_ruby/active_support/core_ext/benchmark.rb:12:in `ms'
/usr/lib/ruby/vendor_ruby/active_support/
benchmarkable.rb:41:in `benchmark'
/var/lib/gems/1.9.1/gems/blacklight-5.14.0/lib/blacklight/solr_repository.rb:42:in
`send_and_receive'
/var/lib/gems/1.9.1/gems/blacklight-5.14.0/lib/blacklight/solr_repository.rb:28:in
`search'

Rails.root: /home/paul/proj/my_new_blacklightapp
[...]
------------------------------------------------------------------------

I'm also seeing the following when running the test data indexing:

------------------------------------------------------------------------
# rake solr:marc:index_test_data
java -Xmx512m -Dsolr.hosturl=http://127.0.0.1:8983/solr/blacklight-core -jar /var/lib/gems/1.9.1/gems/blacklight-marc-5.10.0/lib/SolrMarc.jar /home/paul/proj/my_new_blacklightapp/config/SolrMarc/config.properties /var/lib/gems/1.9.1/gems/blacklight-marc-5.10.0/test_support/data/test_data.utf8.mrc

INFO [main] (MarcImporter.java:816) - Starting SolrMarc indexing.
INFO [main] (Utils.java:191) - Opening file: /home/paul/proj/my_new_blacklightapp/config/SolrMarc/config.properties
INFO [main] (MarcImporter.java:749) - Connecting to remote Solr server at URL http://127.0.0.1:8983/solr/blacklight-core/update
java.io.IOException: Server returned HTTP response code: 500 for URL: http://127.0.0.1:8983/solr/blacklight-core/admin/registry.jsp
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1839)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1440)
at java.net.URL.openStream(URL.java:1038)
at org.solrmarc.solr.SolrCoreLoader.loadRemoteSolrServer(SolrCoreLoader.java:351)
at org.solrmarc.marc.MarcImporter.getSolrServerProxy(MarcImporter.java:750)
at org.solrmarc.marc.MarcImporter.loadLocalProperties(MarcImporter.java:193)
at org.solrmarc.marc.MarcHandler.loadProperties(MarcHandler.java:172)
at org.solrmarc.marc.MarcHandler.init(MarcHandler.java:118)
at org.solrmarc.marc.MarcImporter.main(MarcImporter.java:822)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.simontuffs.onejar.Boot.run(Boot.java:334)
at com.simontuffs.onejar.Boot.main(Boot.java:170)
Warning: Specifying binary request handling from a Solr Server that doesn't support it. Enable it in the solrconfig file for that server.
Solrversion UNKNOWN using xml response writer
INFO [main] (MarcHandler.java:357) - Attempting to open data file: /var/lib/gems/1.9.1/gems/blacklight-marc-5.10.0/test_support/data/test_data.utf8.mrc
------------------------------------------------------------------------

(Yes, I'm running everything as root for now. No, I won't keep doing
that once things are working OK!)

I'm following the quick-start instructions[*] "the hard way" but with a
few differences:

(1) I didn't add tzinfo-data to my Gemfile, because it trips something
up (I've lost track of what exactly).

(2) I ran _rails generate blacklight:install --devise --marc_ without
--jettywrapper because I'm using Tomcat 8, not Jetty. I would have
liked to use Jetty, but I ran into problems that an experienced Java
developer would no doubt be able to get around.

Can anyone give me a hint as to what I might be doing wrong?

Thanks in advance,

Paul.

[*] https://github.com/projectblacklight/blacklight/wiki/Quickstart

--
Paul Hoffman <pa...@flo.org>
Systems Librarian
Fenway Libraries Online
c/o Wentworth Institute of Technology
550 Huntington Ave.
Boston, MA 02115
(617) 442-2384 (FLO main number)

[Benjamin Armintor]

Hello Paul:

Having departed from the instructions, it falls to you to make sure the last part of step 6 in that Quickstart:

"You should be able to navigate to Solr at http://localhost:8983/solr/#/blacklight-core/query"

... is true. If it is not (and this is likely if you are using Tomcat, which would typically change the port at minimum) you will need to sort out where your Solr core is running and update the config/blacklight.yml file accordingly. The quickstart also assumes that the running Solr has a compatible configuration, see also the would-be installed Solr config at https://github.com/projectblacklight/blacklight-jetty/tree/v4.10.3/solr/blacklight-core/conf

Regards,

Ben

--
You received this message because you are subscribed to the Google Groups "Blacklight Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blacklight-develo...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Paul Hoffman]

On Wed, Oct 14, 2015 at 02:49:22AM -0400, Benjamin Armintor wrote:
> Having departed from the instructions, it falls to you to make sure the
> last part of step 6 in that Quickstart:

Thanks for your reply. I'll start over, this time following the
instructions faithfully. Bear with me here, as there's a question at
the end but I want to show the intervening progress...

First, here's my environment:

------------------------------------------------------------------------
$ uname -a
Linux lola 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

$ id
uid=2009(paul) gid=2009(paul) groups=2009(paul),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),117(lpadmin),118(sambashare)

$ ruby --version
ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]

$ rails --version
Rails 4.0.2

$ java -version
openjdk version "1.8.0_45-internal"
OpenJDK Runtime Environment (build 1.8.0_45-internal-b14)
OpenJDK 64-Bit Server VM (build 25.45-b02, mixed mode)
------------------------------------------------------------------------

Next, I start the rails project -- I'll snip most of the output but
include info about the dependencies, all of which I had already
satisfied.

------------------------------------------------------------------------
$ cd

$ mkdir projects

$ cd projects

$ rails new search_app -m https://raw.github.com/projectblacklight/blacklight/master/template.demo.rb
[snip]
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies...............................................................................................
Using rake 10.4.2
Using i18n 0.7.0
Using minitest 4.7.5
Using multi_json 1.11.2
Using thread_safe 0.3.5
Using tzinfo 0.3.45
Using activesupport 4.0.2
Using builder 3.1.4
Using erubis 2.7.0
Using rack 1.5.5
Using rack-test 0.6.3
Using actionpack 4.0.2
Using mime-types 1.25.1
Using polyglot 0.3.5
Using treetop 1.4.15
Using mail 2.5.4
Using actionmailer 4.0.2
Using activemodel 4.0.2
Using activerecord-deprecated_finders 1.0.4
Using arel 4.0.2
Using activerecord 4.0.2
Using execjs 2.6.0
Using json 1.8.3
Using autoprefixer-rails 6.0.3
Using sass 3.2.19
Using bootstrap-sass 3.3.5
Using deprecation 0.2.2
Using kaminari 0.16.3
Using mini_portile 0.6.2
Using nokogiri 1.6.6.2
Using bundler 1.10.6
Using thor 0.19.1
Using railties 4.0.2
Using hike 1.2.3
Using tilt 1.4.1
Using sprockets 2.12.4
Using sprockets-rails 2.0.1
Using rails 4.0.2
Using rsolr 1.0.12

Your user account isn't allowed to install to the system Rubygems.
You can cancel this installation and run:

bundle install --path vendor/bundle

to install the gems into ./vendor/bundle/, or you can enter your password
and install the bundled gems to Rubygems using sudo.
------------------------------------------------------------------------

Which of the two options would you recommend?

Thanks again,

Paul.

[Benjamin Armintor]

I'd personally try to use a local (non-root) bundle, but either should work.

Am I correct in assuming that, despite the interruption, you have a Rails app in a directory called search_app, that has blacklight in the Gemfile?

- Ben

[Paul Hoffman]

On Wed, Oct 14, 2015 at 11:16:49AM -0400, Benjamin Armintor wrote:
> I'd personally try to use a local (non-root) bundle, but either should work.
>
> Am I correct in assuming that, despite the interruption, you have a Rails
> app in a directory called search_app, that has blacklight in the Gemfile?

That's right:

$ fgrep blacklight ~/projects/search_app/Gemfile
gem "blacklight", ">= 5.3.0"

I decided to try the non-root alternative first, so I interrupted the
`rails new' command and ran `bundle install --path vendor/bundle', then
proceeded with `Easy or Hard: After creating your new application' (step
6). Note the error at the end:

------------------------------------------------------------------------
$ cd ~/projects/search_app

$ bundle install --path vendor/bundle

Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving
dependencies...............................................................................................

Installing rake 10.4.2
[snip]
Installing uglifier 2.7.2
Bundle complete! 12 Gemfile dependencies, 56 gems now installed.
Bundled gems are installed into ./vendor/bundle.
Post-install message from rdoc:
Depending on your version of ruby, you may need to install ruby rdoc/ri data:

<= 1.8.6 : unsupported
= 1.8.7 : gem install rdoc-data; rdoc-data --install
= 1.9.1 : gem install rdoc-data; rdoc-data --install
>= 1.9.2 : nothing to do! Yay!

$ ruby -v

ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]

$ rake jetty:start
`x64_mingw` is not a valid platform. The available options are: [:ruby, :ruby_18, :ruby_19, :ruby_20, :mri, :mri_18, : mri_19, :mri_20, :rbx, :jruby, :mswin, :mingw, :mingw_18, :mingw_19, :mingw_20]

$ fgrep -r x64_mingw .
./vendor/bundle/ruby/1.9.1/gems/blacklight-5.15.0/template.demo.rb:gem 'tzinfo-data', platforms: [:mingw, :mswin, :x64_mingw]
./Gemfile:gem "tzinfo-data", platforms: [:mingw, :mswin, :x64_mingw]

$ uname -a
Linux lola 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

------------------------------------------------------------------------

Should I delete tzinfo-data from my Gemfile, since I'm not on a Windows
platform? Or should I modify that line somehow?

Thanks in advance,

[Benjamin Armintor]

Paul,

When you interrupt there, you would need to make sure the rest of template commands are executed as well:

https://raw.githubusercontent.com/projectblacklight/blacklight/master/template.demo.rb

I think you may need a newer version of bundler: This platform option was added in 1.5.0:

https://github.com/bundler/bundler/commit/facdbf0a962

We ought also to ensure that the Blacklight generator checks the bundler version. In the meantime if you run:

bundle platform

... and the output does not match any of the listed options ([:mingw, :mswin, :x64_mingw]), then you should be able to:

1. remove that gem entirely from the Gemfile

2. run bundle install

3. run rails g blacklight:install --devise --marc --jettywrapper

4. run the sequence of rake commands listed in the template

- Ben

[Paul Hoffman]

On Wed, Oct 14, 2015 at 12:37:43PM -0400, Benjamin Armintor wrote:
> When you interrupt there, you would need to make sure the rest of template
> commands are executed as well:
>
> https://raw.githubusercontent.com/projectblacklight/blacklight/master/template.demo.rb

Thanks -- I wouldn't have realized that. I'm stumped as to *how* to
execute them, however; my naive attempts yield the error message below.
Do I have to run the commands using rake, or using ruby itself? Or in
some other way?

------------------------------------------------------------------------
$ url=https://raw.githubusercontent.com/projectblacklight/blacklight/master/template.demo.rb

$ curl -s $url > /tmp/template.demo.rb

$ perl -ne '$p++, next if /^run/; print if !/tzinfo-data/' < /tmp/template.demo.rb > /tmp/Rakefile

$ /usr/local/bin/rake /tmp/Rakefile
rake aborted!
NoMethodError: undefined method `generate' for main:Object
/tmp/Rakefile:8:in `<top (required)>'
(See full trace by running task with --trace)

$ /usr/bin/ruby1.9.1 /tmp/Rakefile
/tmp/Rakefile:8:in `<main>': undefined method `generate' for main:Object (NoMethodError)

$ grep '[^ ]' /tmp/Rakefile

gem "blacklight", ">= 5.3.0"

gem "jettywrapper", ">= 2.0"
# run the blacklight install generator
options = ENV.fetch("BLACKLIGHT_INSTALL_OPTIONS", '--devise --marc --jettywrapper')
generate 'blacklight:install', options
# run the database migrations
rake "db:migrate"
# index some data
if options =~ /jettywrapper/
rake "jetty:clean"
rake "jetty:start"
rake "blacklight:index:seed"
rake "jetty:stop"
end

------------------------------------------------------------------------

> I think you may need a newer version of bundler: This platform option was
> added in 1.5.0:
>
> https://github.com/bundler/bundler/commit/facdbf0a962

How do I get this new version?

> We ought also to ensure that the Blacklight generator checks the bundler
> version. In the meantime if you run:
>
> bundle platform
>
>
> ... and the output does not match any of the listed options ([:mingw,

> :mswin, :x64_mingw]), [...]

It didn't match any of them, so I went ahead and followed your
instructions, and they worked great up to the point where I run the rest
of the rake commands in template.demo.rb. Just for the record, it went
like this:

> [...] then you should be able to:

>
> 1. remove that gem entirely from the Gemfile
>
> 2. run bundle install
>
> 3. run rails g blacklight:install --devise --marc --jettywrapper
>
> 4. run the sequence of rake commands listed in the template

------------------------------------------------------------------------
$ bundle platform
Your platform is: x86_64-linux

Your app has gems that work on these platforms:
* ruby

Your Gemfile does not specify a Ruby version requirement.

$ mv Gemfile Gemfile.old

$ fgrep -v tzinfo-data Gemfile.old > Gemfile

$ bundle install --path vendor/bundle

Resolving dependencies...
Using rake 10.4.2
[snip]
Using uglifier 2.7.2
Bundle complete! 11 Gemfile dependencies, 56 gems now installed.

Bundled gems are installed into ./vendor/bundle.

$ rails g blacklight:install --devise --marc --jettywrapper
/home/paul/projects/search_app/vendor/bundle/ruby/1.9.1/gems/execjs-2.6.0/lib/execjs/runtimes.rb:48:in `autodetect': Could not find a JavaScript runtime. See https://github.com/rails/execjs for a list of available runtimes. (ExecJS::RuntimeUnavailable)
[snip]

$ echo "gem 'therubyracer'" >> Gemfile

$ bundle install --path vendor/bundle
Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..
Resolving dependencies.....

Using rake 10.4.2
[snip]
Using jquery-rails 3.1.4
Installing libv8 3.16.14.11
Using rdoc 4.2.0
Installing ref 2.0.0
Using sass-rails 4.0.5
Using sdoc 0.4.1
Using sqlite3 1.3.11
Installing therubyracer 0.12.2 with native extensions
Using turbolinks 2.5.3
Using uglifier 2.7.2
Bundle complete! 12 Gemfile dependencies, 59 gems now installed.

Bundled gems are installed into ./vendor/bundle.

$ rails g blacklight:install --devise --marc --jettywrapper
[snip]
Bundle complete! 16 Gemfile dependencies, 70 gems now installed.

Bundled gems are installed into ./vendor/bundle.

generate blacklight:marc:install
create config/SolrMarc
create config/SolrMarc/config-test.properties
create config/SolrMarc/config.properties
create config/SolrMarc/index.properties
create config/SolrMarc/index_scripts/dewey.bsh
create config/SolrMarc/index_scripts/format.bsh
create config/SolrMarc/translation_maps/README_MAPS
create config/SolrMarc/translation_maps/callnumber_map.properties
create config/SolrMarc/translation_maps/composition_era_map.properties
create config/SolrMarc/translation_maps/country_map.properties
create config/SolrMarc/translation_maps/format_map.properties
create config/SolrMarc/translation_maps/instrument_map.properties
create config/SolrMarc/translation_maps/language_map.properties
insert app/models/solr_document.rb
insert app/controllers/catalog_controller.rb
route Blacklight::Marc.add_routes(self)

------------------------------------------------------------------------

Thanks again,

[Paul Hoffman]

On Wed, Oct 14, 2015 at 04:51:47PM -0400, Paul Hoffman wrote:
> On Wed, Oct 14, 2015 at 12:37:43PM -0400, Benjamin Armintor wrote:
> > When you interrupt there, you would need to make sure the rest of template
> > commands are executed as well:
> >
> > https://raw.githubusercontent.com/projectblacklight/blacklight/master/template.demo.rb
>
> Thanks -- I wouldn't have realized that. I'm stumped as to *how* to
> execute them, however; my naive attempts yield the error message below.
> Do I have to run the commands using rake, or using ruby itself? Or in
> some other way?

I decided to start over, this time using sudo and actually *thinking*
about the output of the rails command -- it shows me which steps are
failing because of the invalid platform code. I made it much further in
the quickstart guide, but when attempting to index the sample MARC
recors I hit a "SolrCore Initialization Failures" warning that says "No
cores available". Is Solr perhaps not actually running?

Here's what I did this time, for posterity and in case it illuminates
the underlying cause of the problem...

------------------------------------------------------------------------
$ cd ~/projects

$ sudo -v

$ rails new search_app -m https://raw.github.com/projectblacklight/blacklight/master/template.demo.rb
[snip]

apply https://raw.github.com/projectblacklight/blacklight/master/template.demo.rb
gemfile blacklight (>= 5.3.0)
gemfile jettywrapper (>= 2.0)
gemfile tzinfo-data
run bundle install from "."

Fetching gem metadata from https://rubygems.org/..........
Fetching version metadata from https://rubygems.org/...
Fetching dependency metadata from https://rubygems.org/..

Resolving dependencies...............................................................................................

Using rake 10.4.2
[snip]
Using uglifier 2.7.2

Bundle complete! 12 Gemfile dependencies, 56 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
generate blacklight:install

`x64_mingw` is not a valid platform. The available options are: [:ruby, :ruby_18, :ruby_19, :ruby_20, :mri, :mri_18, :mri_19, :mri_20, :rbx, :jruby, :mswin, :mingw, :mingw_18, :mingw_19, :mingw_20]

rake db:migrate

`x64_mingw` is not a valid platform. The available options are: [:ruby, :ruby_18, :ruby_19, :ruby_20, :mri, :mri_18, :mri_19, :mri_20, :rbx, :jruby, :mswin, :mingw, :mingw_18, :mingw_19, :mingw_20]

rake jetty:clean

`x64_mingw` is not a valid platform. The available options are: [:ruby, :ruby_18, :ruby_19, :ruby_20, :mri, :mri_18, :mri_19, :mri_20, :rbx, :jruby, :mswin, :mingw, :mingw_18, :mingw_19, :mingw_20]
rake jetty:start
`x64_mingw` is not a valid platform. The available options are: [:ruby, :ruby_18, :ruby_19, :ruby_20, :mri, :mri_18, :mri_19, :mri_20, :rbx, :jruby, :mswin, :mingw, :mingw_18, :mingw_19, :mingw_20]

rake blacklight:index:seed

`x64_mingw` is not a valid platform. The available options are: [:ruby, :ruby_18, :ruby_19, :ruby_20, :mri, :mri_18, :mri_19, :mri_20, :rbx, :jruby, :mswin, :mingw, :mingw_18, :mingw_19, :mingw_20]

rake jetty:stop

`x64_mingw` is not a valid platform. The available options are: [:ruby, :ruby_18, :ruby_19, :ruby_20, :mri, :mri_18, :mri_19, :mri_20, :rbx, :jruby, :mswin, :mingw, :mingw_18, :mingw_19, :mingw_20]

run bundle install

`x64_mingw` is not a valid platform. The available options are: [:ruby, :ruby_18, :ruby_19, :ruby_20, :mri, :mri_18, :mri_19, :mri_20, :rbx, :jruby, :mswin, :mingw, :mingw_18, :mingw_19, :mingw_20]

$ cd search_app

$ vim Gemfile
[remove the tzinfo-data gem]

$ sudo -v

$ bundle install

Bundle complete! 11 Gemfile dependencies, 56 gems now installed.

Use `bundle show [gemname]` to see where a bundled gem is installed.

$ rails generate blacklight:install --devise --marc --jettywrapper
/var/lib/gems/1.9.1/gems/execjs-2.6.0/lib/execjs/runtimes.rb:48:in `autodetect': Could not find a JavaScript runtime. See https://github.com/rails/execjs for a list of available runtimes. (ExecJS::RuntimeUnavailable)

$ echo 'gem "therubyracer"' >> Gemfile

$ rails generate blacklight:install --devise --marc --jettywrapper
gemfile jettywrapper (>= 2.0)
create config/jetty.yml
append Rakefile
gemfile rsolr (~> 1.0.6)
run bundle install from "."

Resolving dependencies...
Using rake 10.4.2
[snip]
Using uglifier 2.7.2

Bundle complete! 13 Gemfile dependencies, 59 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
generate blacklight:assets
create app/assets/stylesheets/blacklight.css.scss
insert app/assets/javascripts/application.js
generate blacklight:document
create app/models/solr_document.rb
generate blacklight:search_builder
create app/models/search_builder.rb
generate blacklight:models
create config/blacklight.yml
rake blacklight:install:migrations
Copied migration 20151015201029_create_searches.blacklight.rb from blacklight
Copied migration 20151015201030_create_bookmarks.blacklight.rb from blacklight
Copied migration 20151015201031_add_polymorphic_type_to_bookmarks.blacklight.rb from blacklight
generate blacklight:user
gemfile devise
gemfile devise-guests (~> 0.3)
run bundle install from "."

Resolving dependencies...
Using rake 10.4.2
[snip]
Using uglifier 2.7.2

Bundle complete! 15 Gemfile dependencies, 65 gems now installed.
Use `bundle show [gemname]` to see where a bundled gem is installed.
generate devise:install
create config/initializers/devise.rb
create config/locales/devise.en.yml

===============================================================================

Some setup you must do manually if you haven't yet:

[snip -- see below]

===============================================================================
generate devise
invoke active_record
create db/migrate/20151015201051_devise_create_users.rb
create app/models/user.rb
invoke test_unit
create test/models/user_test.rb
create test/fixtures/users.yml
insert app/models/user.rb
route devise_for :users
generate devise_guests
invoke active_record
create db/migrate/20151015201059_add_devise_guests_to_users.rb
insert app/models/user.rb
gsub config/initializers/devise.rb
insert app/models/user.rb
generate blacklight:controller
insert app/controllers/application_controller.rb
create app/controllers/catalog_controller.rb
route blacklight_for :catalog
route root to: "catalog#index"
insert config/application.rb
create config/locales/blacklight.en.yml
create config/initializers/blacklight_initializer.rb
gemfile blacklight-marc (~> 5.0)
run bundle install from "."

Resolving dependencies...
Using rake 10.4.2
[snip]
Using uglifier 2.7.2

Bundle complete! 16 Gemfile dependencies, 70 gems now installed.

Use `bundle show [gemname]` to see where a bundled gem is installed.

generate blacklight:marc:install
create config/SolrMarc
create config/SolrMarc/config-test.properties
create config/SolrMarc/config.properties
create config/SolrMarc/index.properties
create config/SolrMarc/index_scripts/dewey.bsh
create config/SolrMarc/index_scripts/format.bsh
create config/SolrMarc/translation_maps/README_MAPS
create config/SolrMarc/translation_maps/callnumber_map.properties
create config/SolrMarc/translation_maps/composition_era_map.properties
create config/SolrMarc/translation_maps/country_map.properties
create config/SolrMarc/translation_maps/format_map.properties
create config/SolrMarc/translation_maps/instrument_map.properties
create config/SolrMarc/translation_maps/language_map.properties
insert app/models/solr_document.rb
insert app/controllers/catalog_controller.rb
route Blacklight::Marc.add_routes(self)

$ rake db:migrate
[snip -- looks good]

$ rake jetty:clean
I, [2015-10-15T16:15:47.379304 #17157] INFO -- : Downloading jetty at https://github.com/projectblacklight/blacklight-jetty/archive/v4.10.3.zip ...
I, [2015-10-15T16:16:22.418087 #17157] INFO -- : Unpacking tmp/v4.10.3.zip...
------------------------------------------------------------------------

So far, so good. The manual setup instructions that I snipped above are
pretty clear so I do those now, then continue with step 6 (i.e., the
beginning of the `Easy or Hard: After creating your new application'
section):

------------------------------------------------------------------------
$ vim config/environments/development.rb
[insert:]
config.action_mailer.default_url_options = { host: 'localhost', port: 3000 }

$ vim config/routes.rb
[notice that _root to:_ is already set, so leave it untouched]

$ vim app/views/layouts/application.html.erb
[insert:]
<p class="notice"><%= notice %></p>
<p class="alert"><%= alert %></p>

$ rake jetty:start
D, [2015-10-15T16:40:08.056673 #17260] DEBUG -- : Starting jetty with these values:
D, [2015-10-15T16:40:08.056846 #17260] DEBUG -- : jetty_home: /home/paul/projects/search_app/jetty
D, [2015-10-15T16:40:08.056953 #17260] DEBUG -- : jetty_command: java -Djetty.port=8983 -Dsolr.solr.home=/home/paul/projects/search_app/jetty/solr -jar start.jar
W, [2015-10-15T16:40:08.058012 #17260] WARN -- : Logging jettywrapper stdout to /home/paul/projects/search_app/jetty/jettywrapper.log
D, [2015-10-15T16:40:08.066449 #17260] DEBUG -- : Wrote pid file to /home/paul/projects/search_app/tmp/pids/_home_paul_projects_search_app_jetty_development.pid with value 17265
I, [2015-10-15T16:40:16.133549 #17260] INFO -- : Started jetty (8075.0ms)
jetty started at PID 17265
------------------------------------------------------------------------

All looks good, so I try importing the test MARC records (step 7):

------------------------------------------------------------------------
$ rake solr:marc:index_test_data
java -Xmx512m -Dsolr.hosturl=http://127.0.0.1:8983/solr/blacklight-core -jar /var/lib/gems/1.9.1/gems/blacklight-marc-5.10.0/lib/SolrMarc.jar /home/paul/projects/search_app/config/SolrMarc/config.properties /var/lib/gems/1.9.1/gems/blacklight-marc-5.10.0/test_support/data/test_data.utf8.mrc

log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException: /var/lib/gems/1.9.1/gems/blacklight-marc-5.10.0/lib/solrmarc.log (Permission denied)
[snip]
------------------------------------------------------------------------

OK, so I'll run it as root and sort out the ownership/permission details
later:

------------------------------------------------------------------------
$ sudo rake solr:marc:index_test_data
java -Xmx512m -Dsolr.hosturl=http://127.0.0.1:8983/solr/blacklight-core -jar /var/lib/gems/1.9.1/gems/blacklight-marc-5.10.0/lib/SolrMarc.jar /home/paul/projects/search_app/config/SolrMarc/config.properties /var/lib/gems/1.9.1/gems/blacklight-marc-5.10.0/test_support/data/test_data.utf8.mrc

INFO [main] (MarcImporter.java:816) - Starting SolrMarc indexing.

INFO [main] (Utils.java:191) - Opening file: /home/paul/projects/search_app/config/SolrMarc/config.properties

INFO [main] (MarcImporter.java:749) - Connecting to remote Solr server at URL http://127.0.0.1:8983/solr/blacklight-core/update
java.io.IOException: Server returned HTTP response code: 500 for URL: http://127.0.0.1:8983/solr/blacklight-core/admin/registry.jsp

[snip]
------------------------------------------------------------------------

This is what I hit earlier. The end of step 6 says:

| You should be able to navigate to Solr at http://localhost:8983/solr/#/blacklight-core/query

When I use w3m to access that URL I get an error message:

| Apache SOLR
|
| SolrCore Initialization Failures
|
| Please check your logs for more information
| [snip]
| No cores available Go and create one
| [snip]
| Connection lost …

So I gather that Solr is not running? My jetty-wrapper log doesn't show
any errors, nor does my solr.log:

------------------------------------------------------------------------
$ fgrep -c -i err $(find . -name \*.log)
./log/development.log:0
./jetty/jettywrapper.log:0
./jetty/logs/solr.log:0
------------------------------------------------------------------------

Any ideas what I need to do at this point?

Thanks in advance,

[Paul Hoffman]

On Thu, Oct 15, 2015 at 05:00:06PM -0400, Paul Hoffman wrote:
> I decided to start over, this time using sudo and actually *thinking*
> about the output of the rails command -- it shows me which steps are
> failing because of the invalid platform code. I made it much further in
> the quickstart guide, but when attempting to index the sample MARC
> recors I hit a "SolrCore Initialization Failures" warning that says "No
> cores available". Is Solr perhaps not actually running?

Does anyone have any idea what might be wrong here?

[Jack Reed]

It sounds like that core isn¹t available. Are you able to hit the solr
admin page?

http://127.0.0.1:8983/solr/admin

If so check and see what cores are available:

http://localhost:8983/solr/admin/cores?action=STATUS


You can also create core from the solr interface called ³blacklight-core²
but it seems like it wasn¹t created for you.

If you are still having issues I would try:

`bundle exec rake jetty:stop`

`rm -rf jetty`

`bundle exec rake jetty:clean`

-Jack

On 10/15/15, 2:00 PM, "blacklight-...@googlegroups.com on behalf
of Paul Hoffman" <blacklight-...@googlegroups.com on behalf of

>| Connection lost Š

>
>So I gather that Solr is not running? My jetty-wrapper log doesn't show
>any errors, nor does my solr.log:
>
>------------------------------------------------------------------------
>$ fgrep -c -i err $(find . -name \*.log)
>./log/development.log:0
>./jetty/jettywrapper.log:0
>./jetty/logs/solr.log:0
>------------------------------------------------------------------------
>
>Any ideas what I need to do at this point?
>
>Thanks in advance,
>
>Paul.
>
>--
>Paul Hoffman <pa...@flo.org>
>Systems Librarian
>Fenway Libraries Online
>c/o Wentworth Institute of Technology
>550 Huntington Ave.
>Boston, MA 02115
>(617) 442-2384 (FLO main number)
>

[Paul Hoffman]

On Fri, Oct 16, 2015 at 09:20:20PM +0000, Jack Reed wrote:
> It sounds like that core isn¹t available. Are you able to hit the solr
> admin page?
>
> http://127.0.0.1:8983/solr/admin

No, I get the same error:

$ w3m 'http://localhost:8983/solr/#/blacklight-core/query'

SolrCore Initialization Failures
Please check your logs for more information

By `logs' does it mean jetty/logs/solr.log and jetty/jettywrapper.log?
There are no errors or exceptions in either one, and the 7 warnings in
solr.log don't seem relevant.

Here's something very odd, though. My project's gems are supposedly
installed in two different places:

$ bundle list --paths | sed 's:/[^/]*$::' | sort | uniq -c
11 /usr/share/rubygems-integration/1.9.1/gems
59 /var/lib/gems/1.9.1/gems

When I run `bundle exec rake jetty:start' under strace I see more than
70,000 ENOENT (no such file) errors:

$ strace -e open,stat,stat64 bundle exec rake jetty:start > /tmp/jetty_start.strace 2>&1

$ fgrep -cw ENOENT /tmp/jetty_start.strace
71479

Every attempt to open or stat a file within
/usr/share/rubygems-integration/1.9.1/gems failed:

$ fgrep -c /usr/share/rubygems-integration/1.9.1/gems /tmp/jetty_start.strace
8869

$ fgrep /usr/share/rubygems-integration/1.9.1/gems /tmp/jetty_start.strace | fgrep -cv ENOENT
0

Which is not surprising, because that directory does not exist.

Here's what is supposed to be there:

$ bundle list --paths | fgrep /usr/share/rubygems-integration/1.9.1/gems
/usr/share/rubygems-integration/1.9.1/gems/actionmailer-4.0.2
/usr/share/rubygems-integration/1.9.1/gems/actionpack-4.0.2
/usr/share/rubygems-integration/1.9.1/gems/activemodel-4.0.2
/usr/share/rubygems-integration/1.9.1/gems/activerecord-4.0.2
/usr/share/rubygems-integration/1.9.1/gems/activesupport-4.0.2
/usr/share/rubygems-integration/1.9.1/gems/erubis-2.7.0
/usr/share/rubygems-integration/1.9.1/gems/mail-2.5.4
/usr/share/rubygems-integration/1.9.1/gems/rails-4.0.2
/usr/share/rubygems-integration/1.9.1/gems/railties-4.0.2
/usr/share/rubygems-integration/1.9.1/gems/sprockets-rails-2.0.1
/usr/share/rubygems-integration/1.9.1/gems/tilt-1.4.1

Should I try installing as root?

Feel free to skip the rest, since the answer to that question is
probably YES...

> If so check and see what cores are available:
>
> http://localhost:8983/solr/admin/cores?action=STATUS

Here's what I get; it looks to me as though core "collection1" is the
default but no such core exists:

$ curl -s 'http://localhost:8983/solr/admin/cores?action=STATUS' | sed 's:><:>\n<:g'
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader">...</lst>
<str name="defaultCoreName">collection1</str>
<lst name="initFailures"/>
<lst name="status">
<lst name="blacklight-core">
<str name="name">blacklight-core</str>
<bool name="isDefaultCore">false</bool>
<str name="instanceDir">/home/paul/projects/search_app/jetty/solr/blacklight-core/</str>
<str name="dataDir">/home/paul/projects/search_app/jetty/solr/blacklight-core/data/</str>
<str name="config">solrconfig.xml</str>
<str name="schema">schema.xml</str>
<date name="startTime">2015-10-15T20:40:14.114Z</date>
<long name="uptime">321697392</long>
<lst name="index">
<int name="numDocs">30</int>
<int name="maxDoc">30</int>
<int name="deletedDocs">0</int>
<long name="indexHeapUsageBytes">42104</long>
<long name="version">7</long>
<int name="segmentCount">1</int>
<bool name="current">true</bool>
<bool name="hasDeletions">false</bool>
<str name="directory">org.apache.lucene.store.MMapDirectory:MMapDirectory@/home/paul/projects/search_app/jetty/solr/blacklight-core/data/index lockFactory=NativeFSLockFactory@/home/paul/projects/search_app/jetty/solr/blacklight-core/data/index</str>
<lst name="userData">
<str name="commitTimeMSec">1444941847356</str>
</lst>
<date name="lastModified">2015-10-15T20:44:07.356Z</date>
<long name="sizeInBytes">143214</long>
<str name="size">139.86 KB</str>
</lst>
</lst>
</lst>
</response>

What's weird is that grep doesn't find any occurrence of "collection1"
within ~/projects/search_app.

Furthermore, jetty/solr/blacklight-core exists and it contains
core.properties and conf/solrconfig.yml and data/index on which a java
process is holding a write lock:

$ lsof jetty/solr/blacklight-core/data/index/write.lock
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 24568 paul 99wW REG 252,0 0 2491929 jetty/solr/blacklight-core/data/index/write.lock

> You can also create core from the solr interface called ³blacklight-core²
> but it seems like it wasn¹t created for you.
>
> If you are still having issues I would try:
>
> `bundle exec rake jetty:stop`

OK, that ran successfully.

> `rm -rf jetty`
>
> `bundle exec rake jetty:clean`

OK, that ran successfully so I started jetty using bundle -- btw, what's
the diff between `bundle exec rake' and just `rake'?

$ bundle exec rake jetty:start

But that doesn't seem to have done the trick:

$ w3m 'http://localhost:8983/solr/#/blacklight-core/query'

SolrCore Initialization Failures
Please check your logs for more information

And the solr cores status page is essentially the same, except the 30
MARC records have been deleted (of course).

On a hunch, I reindexed the MARC records using bundle exec:

$ bundle exec rake solr:marc:index_test_data

No luck -- same errors, back to square one. Does anyone have any more
ideas? I think I've been following the quickstart guide as closely as
possible.

Here are the warnings I see in the log files, just in case I'm missing
something:

$ egrep -hi 'warn' $(find ~/projects/search_app -name \*.log)
4283 [coreLoadExecutor-5-thread-1] WARN org.apache.solr.core.SolrResourceLoader – Solr loaded a deprecated plugin/analysis class [solr.IntField]. Please consult documentation how to replace it accordingly.
4290 [coreLoadExecutor-5-thread-1] WARN org.apache.solr.core.SolrResourceLoader – Solr loaded a deprecated plugin/analysis class [solr.LongField]. Please consult documentation how to replace it accordingly.
4296 [coreLoadExecutor-5-thread-1] WARN org.apache.solr.core.SolrResourceLoader – Solr loaded a deprecated plugin/analysis class [solr.FloatField]. Please consult documentation how to replace it accordingly.
4303 [coreLoadExecutor-5-thread-1] WARN org.apache.solr.core.SolrResourceLoader – Solr loaded a deprecated plugin/analysis class [solr.DoubleField]. Please consult documentation how to replace it accordingly.
4307 [coreLoadExecutor-5-thread-1] WARN org.apache.solr.core.SolrResourceLoader – Solr loaded a deprecated plugin/analysis class [solr.DateField]. Please consult documentation how to replace it accordingly.
4774 [coreLoadExecutor-5-thread-1] WARN org.apache.solr.core.SolrCore – [blacklight-core] Solr index directory '/home/paul/projects/search_app/jetty/solr/blacklight-core/data/index' doesn't exist. Creating new index...
5368 [coreLoadExecutor-5-thread-1] WARN org.apache.solr.core.RequestHandlers – Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler
WARN - 2015-10-19 10:11:32.193; org.apache.solr.core.SolrResourceLoader; Solr loaded a deprecated plugin/analysis class [solr.IntField]. Please consult documentation how to replace it accordingly.
WARN - 2015-10-19 10:11:32.200; org.apache.solr.core.SolrResourceLoader; Solr loaded a deprecated plugin/analysis class [solr.LongField]. Please consult documentation how to replace it accordingly.
WARN - 2015-10-19 10:11:32.206; org.apache.solr.core.SolrResourceLoader; Solr loaded a deprecated plugin/analysis class [solr.FloatField]. Please consult documentation how to replace it accordingly.
WARN - 2015-10-19 10:11:32.213; org.apache.solr.core.SolrResourceLoader; Solr loaded a deprecated plugin/analysis class [solr.DoubleField]. Please consult documentation how to replace it accordingly.
WARN - 2015-10-19 10:11:32.217; org.apache.solr.core.SolrResourceLoader; Solr loaded a deprecated plugin/analysis class [solr.DateField]. Please consult documentation how to replace it accordingly.
WARN - 2015-10-19 10:11:32.684; org.apache.solr.core.SolrCore; [blacklight-core] Solr index directory '/home/paul/projects/search_app/jetty/solr/blacklight-core/data/index' doesn't exist. Creating new index...
WARN - 2015-10-19 10:11:33.278; org.apache.solr.core.RequestHandlers; Multiple requestHandler registered to the same name: /update ignoring: org.apache.solr.handler.UpdateRequestHandler

[Paul Hoffman]

On Mon, Oct 19, 2015 at 12:12:46PM -0400, Paul Hoffman wrote:
> On Fri, Oct 16, 2015 at 09:20:20PM +0000, Jack Reed wrote:

> > It sounds like that core isnšt available. Are you able to hit the solr

> > admin page?
> >
> > http://127.0.0.1:8983/solr/admin
>
> No, I get the same error:
>
> $ w3m 'http://localhost:8983/solr/#/blacklight-core/query'
>
> SolrCore Initialization Failures
> Please check your logs for more information

I've fixed the problem. I downloaded template.demo.rb from GitHub,
removed the tzinfo-data gem (and added therubyracer), then installed
normally as root using the local copy of template.demo.rb. I also
upgraded Sass and associated gems, though it's not entirely clear that
that was necessary. As a bonus, I no longer have gems "installed" in a
non-existent directory. :-)