RE: [Bangalore RUG] [Solr + Rails] - rake sunspot:reindex is not workingcj

[Satish Kota]

Puneeth,

 

Some of the steps you could do…

 

1.       Stop Solr

2.       Remove SOLR folder

3.       Start Solr

4.       Ensure your Config file is correct (solr/conf/schema.xml)

5.       Run sunspot:reindex

 

Many times the error occurs because the schema file is not properly generated… we too have faced this kind of issue…

 

Thanks and Regards

Satish N Kota

Director, Heurion

Creator, TestimonialsFor

@phone: +91 98862 03255

@testifor: http://www.testifor.com/satishnkota

www.testifor.com – What others say, Matters!!!

 

From: bangal...@googlegroups.com [mailto:bangal...@googlegroups.com] On Behalf Of Puneet Pandey
Sent: Wednesday, April 03, 2013 11:34 AM
To: bangalorerug
Subject: Re: [Bangalore RUG] [Solr + Rails] - rake sunspot:reindex is not working

 

Tried with creating fresh instance and after deploying when I am trying to execute sunspot:reindex, I am facing the same issue again.

Also, checked the permissions for /data folder and after changing it to 777 when I executed the above command, nothing worked out :(

Any other suggestions?

 

On Tue, Apr 2, 2013 at 8:39 AM, Vamsee Kanakala <vkan...@gmail.com> wrote:

On Mon, Apr 01, 2013 at 04:01:39PM +0530, Puneet Pandey wrote:
> From the logs, it is clear that re-indexing is failing because of 2 reasons:
>
> A. Unable to create core: collection1
> B. Can't find resource 'solrconfig.xml' in classpath
>
> Based on the errors above, below are the tricks I tried:
>
> A. Checked the folder permission for solr
> B. Manually created collection1 inside solr and copied solrconfig.xml
> inside /solr/collection1/conf.
> C. copied the lib and conf folders from
> /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr
> and pasted it in APP_NAME/solr/sunspot_rails folder.
>

As far as I can remember, sunspot should do most of the hard work of creating solr configs for you.
It still looks like a permissions issue to me. Did you check if the /data folder is writable by
the web server process that runs solr? If it's Apache, the folder permissions should be writable
by the www-data user/group.

Vamsee.


--
You received this message because you are subscribed to the Google Groups "BANGALORE RUG-Ruby Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bangalorerug...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Puneet Pandey
Associate Consultant
NOIDA/INDIA - 201301
E: ad...@puneetpandey.com
B: My Blog
Linkedin Profile | GitHub Profile

********Where there is a will, there is a way***********

--
You received this message because you are subscribed to the Google Groups "BANGALORE RUG-Ruby Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bangalorerug...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Puneet Pandey]

@Satish, tried with the steps you'd provided. But still no luck :-(

Please check the details, maybe this will help you to understand if I am heading towards the right direction.

1. Stopped solr

${PATH_TO_APP}: bundle exec rake sunspot:solr:stop

2. remove solr folder

${PATH_TO_APP}: rm -rf solr

3. Start solr
${PATH_TO_APP}: bundle exec rake sunspot:solr:start

--backtrace
java version "1.6.0_30"
Java(TM) SE Runtime Environment (build 1.6.0_30-b12)
Java HotSpot(TM) 64-Bit Server VM (build 20.5-b03, mixed mode)
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/con
f/solrconfig.xml => /data/APP_NAME/releases/20130402102051/solr/conf/solrconfig.xml
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/con
f/spellings.txt => /data/APP_NAME/releases/20130402102051/solr/conf/spellings.txt
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/con
f/synonyms.txt => /data/APP_NAME/releases/20130402102051/solr/conf/synonyms.txt
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/con
f/elevate.xml => /data/APP_NAME/releases/20130402102051/solr/conf/elevate.xml
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/con
f/admin-extra.html => /data/APP_NAME/releases/20130402102051/solr/conf/admin-extra.html
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/con
f/schema.xml => /data/APP_NAME/releases/20130402102051/solr/conf/schema.xml
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/con
f/protwords.txt => /data/APP_NAME/releases/20130402102051/solr/conf/protwords.txt
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/con
f/scripts.conf => /data/APP_NAME/releases/20130402102051/solr/conf/scripts.conf
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/con
f/stopwords.txt => /data/APP_NAME/releases/20130402102051/solr/conf/stopwords.txt
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/con
f/mapping-ISOLatin1Accent.txt => /data/APP_NAME/releases/20130402102051/solr/conf/mapping-ISOLatin1Accent.txt
Copying /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bundler/gems/sunspot-87e29212527f/sunspot_solr/solr/solr/sol
r.xml => /data/APP_NAME/releases/20130402102051/solr

Successfully started Solr ...

4. Ensure config file is correct (solr/conf/schema.xml)

I used this link for schema.xml: http://stackoverflow.com/a/14700918/649868 and overwrite schema.xml file under {PATH_TO_APP}/solr/conf

5. Run Sunspot re-index

${PATH_TO_APP}: bundle exec rake sunspot:reindex

--backtrace
rake aborted!
RSolr::Error::Http - 404 Not Found
Error:     Not Found

Request Data: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><delete><query>type:JobSeeker</query></delete>"

One thing I noticed is, when I executed bundle exec rake sunspot:solr:start command and checked the log files, I found this:

SEVERE: null:org.apache.solr.common.SolrException: Unable to create core: development
Caused by: org.apache.solr.common.SolrException: Cannot create directory: /data/APP_NAME/current/solr/data/index

But when I visited this directory index folder was there.

Interesting to see that solr creates solr.xml file in Production mode, whereas it's not there in my staging environment. Please see solr.xml (after changing it):

<?xml version="1.0" encoding="UTF-8" ?>
<solr persistent="true">
  <cores adminPath="/admin/cores" host="${host:}" hostPort="${jetty.port:}">
    <core name="default"     instanceDir="/data/APP_NAME/current/solr" dataDir="/data/APP_NAME/current/solr/data"/>
    <core name="development" instanceDir="/data/APP_NAME/current/solr" dataDir="/data/APP_NAME/current/solr/data"/>
    <core name="test"        instanceDir="/data/APP_NAME/current/solr" dataDir="/data/APP_NAME/current/solr/data"/>
  </cores>
</solr>

Any idea about this?

Thanks for the help.

[Satish Kota]

Hi Puneet,

 

I am not sure why you are replacing the schema.xml…I think you might want to try using the basic schema created..

 

Also I tried looking at the errors…it says…

 

SEVERE: null:org.apache.solr.common.SolrException: Unable to create core: development

Why does it have to create development core on Production???? can you try doing these steps

 

Replace all  “bundle exec rake ….” Tasks with

 

“RAILS_ENV=production bundle exec rake…. “

 

This will make your solr to execute as production instead of development mode….

 

Secondly I also see the error

 

Caused by: org.apache.solr.common.SolrException: Cannot create directory: /data/APP_NAME/current/solr/data/index

 

This could be  when the user running solr doesnot have necessary previleges…. This many times happens because Your app is running on deploy and solr running or root or vice versa…. Can you verify that too….

[Sharath Chander Punthambekar]

Superb Satish.

with best regards
Sharath Punthambekar

"Leadership is the capacity to translate vision into reality."
-Warren G. Bennis

[Puneet Pandey]

Seems like I am going through the worst experience of SOLR :-(

Tried to run all the sunspot rake tasks by specifying RAILS_ENV=production and cross checked if SOLR is running with ROOT or deploy, and got this:

/data/APP_NAME/current $ ps -ef | grep solr
deploy    2342     1  0 Apr04 ?        00:00:30 /usr/lib/jvm/sun-jdk-1.6/bin/java -Djetty.port=8983 -Dsolr.data.dir=/data/APP_NAME/releases/20130404095808/solr/data/production -Dsolr.solr.home=/data/APP_NAME/releases/20130404095808/solr -Djava.util.logging.config.file=/tmp/logging.properties20130404-2342-17k2tsp -jar start.jar
deploy    9770  9461  0 06:42 pts/0    00:00:00 grep --colour=auto solr

It tells me that SOLR is running on deploy.

I am not even in a situation to provide SOLR logs, because failing sunspot:reindex is not generating any log level information as well. :-(

Only thing I can get is this:

$ RAILS_ENV=production bundle exec rake sunspot:reindex --trace

** Invoke sunspot:reindex (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute sunspot:reindex
Skipping progress bar: for progress reporting, add gem 'progress_bar' to your Gemfile

rake aborted!
RSolr::Error::Http - 404 Not Found
Error:     Not Found

Request Data: "<?xml version=\"1.0\" encoding=\"UTF-8\"?><delete><query>type:OccupationData</query></delete>"
Backtrace: /data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:230:in `adapt_response'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:167:in `execute'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:161:in `send_and_receive'
(eval):2:in `post'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:67:in `update'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:131:in `delete_by_query'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/indexer.rb:55:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `block in remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session_proxy/abstract_session_proxy.rb:11:in `remove_all'/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:230:in `adapt_response'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:167:in `execute'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:161:in `send_and_receive'
(eval):2:in `post'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:67:in `update'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rsolr-1.0.8/lib/rsolr/client.rb:131:in `delete_by_query'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/indexer.rb:55:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `block in remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session.rb:181:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session_proxy/abstract_session_proxy.
rb:11:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session_proxy/retry_5xx_session_proxy.rb:17:in `method_missing'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/session_proxy/abstract_session_proxy.rb:11:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot.rb:464:in `remove_all'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot_rails-1.3.3/lib/sunspot/rails/searchable.rb:178:in `solr_remove_all_from_index'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot_rails-1.3.3/lib/sunspot/rails/searchable.rb:197:in `solr_reindex'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot_rails-1.3.3/lib/sunspot/rails/tasks.rb:58:in `block (3levels) in <top (required)>'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/class_set.rb:16:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot-1.3.3/lib/sunspot/class_set.rb:16:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/sunspot_rails-1.3.3/lib/sunspot/rails/tasks.rb:57:in `block (2levels) in <top (required)>'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:246:in `call'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:246:in `block in execute'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:241:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:241:in `execute'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:184:in `block in invoke_with_call_chain'
/usr/lib64/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:177:in `invoke_with_call_chain'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/task.rb:170:in `invoke'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:143:in `invoke_task'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:101:in `block (2 levels) in top_level'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:101:in `each'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:101:in `block in top_level'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:110:in `run_with_threads'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:95:in `top_level'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:73:in `block in run'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:160:in `standard_exception_handling'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/lib/rake/application.rb:70:in `run'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/gems/rake-10.0.4/bin/rake:33:in `<top (required)>'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bin/rake:19:in `load'
/data/APP_NAME/shared/bundled_gems/ruby/1.9.1/bin/rake:19:in `<main>'
Tasks: TOP => sunspot:reindex

Below is my model:
class OccupationData < ActiveRecord::Base
  attr_accessible :onetsoc_code, :title, :description

  searchable :auto_index => false do
    text :title
    text :description
  end
end

Any help on this would be highly appreciated.

[Puneet Pandey]

One interesting fact I came up with is sunspot_solr doesn't work on Production environment. That means whenever I run:

$ RAILS_ENV=production bundle exec rake sunspot:reindex it will not work.

Source: 1. https://github.com/outoftime/sunspot
2. http://sunspot.github.com/sunspot/docs/

So, I removed the gem from the Gemfile, but how this indexing works now? Because I am no more able to access typical solr commands.

Any help!

[Leena S N]

Puneet,

We had used Sunspot for couple of our projects and it seem very straight forward. It abstracts the Solr APIs very well. I am trying to get a good picture of the problem to check whether I can help you somehow. So couple of questions:

• Are you trying to say that the problem with reindex exists only with the production environment? Does it work fine in Development environment?
  
• Does the problem exists only with Reindex rake task? What about while indexing after every save? Does that work well from the application?
• Have you tried Model.reindex method? Does that work?

Leena S N
http://www.multunus.com/

[Puneet Pandey]

Thanks Leena for looking into this and please find the details you've asked for:

On Sat, Apr 6, 2013 at 12:24 AM, Leena S N <lee...@gmail.com> wrote:

Puneet,

We had used Sunspot for couple of our projects and it seem very straight forward. It abstracts the Solr APIs very well. I am trying to get a good picture of the problem to check whether I can help you somehow. So couple of questions:

• Are you trying to say that the problem with reindex exists only with the production environment? Does it work fine in Development environment?
  

Yes, it exists only with Production environment and working fine for development and staging environments.

• Does the problem exists only with Reindex rake task? What about while indexing after every save? Does that work well from the application?

We are not executing indexing after every save. We are only re-indexing after 8 hours and that too Cron is handling this operation.

Additionally, after every deploy we run re-index manually and then let cron allow to handle this.

Yes, the problem exists only with re-index rake task. While starting and stopping solr rake commands are working fine.

• Have you tried Model.reindex method? Does that work?

No, is this something like:

{PATH_TO_APP}$ RAILS_ENV=production bundle exec User.reindex ?

[Puneet Pandey]

Tried with the below command to run Model.reindex:

$ rake sunspot:reindex[,JobSeeker]

but ended up with the same issue.

[Puneet Pandey]

Hi Everyone,

After struggling a lot with re-indexing, I finally found a solution of this. The error was occurring due to wrong gem mentioned in Gemfile.

Here's the old copy of my Gemfile:

gem "sunspot", "~> 1.3.3"
gem 'sunspot_rails', '~> 1.3.0'
gem 'sunspot_cell', :git => 'git://github.com/zheileman/sunspot_cell.git'
gem 'sunspot_cell_jars', :git => 'https://github.com/mrcsparker/sunspot_cell_jars.git'
gem 'sunspot_solr', :git => "git://github.com/sunspot/sunspot.git"

When I browsed this link: https://github.com/mrcsparker/sunspot_cell_jars, I came to know that I have to use the updated sunspot_solr gem in order to keep it work.

so I am using:

gem 'sunspot_solr', :git => "https://github.com/mrcsparker/sunspot.git"

and then I created JAR files again with "rails g command" and when ran sunspot:reindex everything worked. Hope this will help someone.

[Sharath Chander Punthambekar]

Thank you.

with best regards
Sharath Punthambekar

"Leadership is the capacity to translate vision into reality."
-Warren G. Bennis