Developers: setup Solr, switch from db:fetch to data:fetch, learn about additional rake tasks
Igal Koshevoy
developing the Calagator source code:
= Setup Solr
The Solr library and acts_as_solr plugin provide the search feature for
Calagator. You must setup Solr to use Calagator on your development system:
* Install Java from http://www.java.com/getjava
* Update your Calagator checkout and run "rake solr:start". This runs
the Solr search server, which is needed by the Rails application.
* There's no need to restart Solr when you restart your Rails app.
However, you'll probably want to run "rake solr:stop" to shut down the
Solr server when you're done working on Calagator for the day so that
the Java process can stop using up memory and resources.
* Warning to Windows users: There's a bug in the code which prevents the
"rake solr:stop" task from working. You'll need to go to the Task
Manager and manually kill the Java process yourself.
= Switch from db:fetch to data:fetch
* "rake db:fetch" is the old task that downloads the production database
to your local machine. However, this causes a problem with the new Solr
code. The issue is that replacing the database confuses Solr since the
index and data no longer match. To regenerate the index, run "rake
solr:reset", but be warned that this will take a long time. However, you
could instead run....
* "rake data:fetch" is a new task that downloads the production server's
database and Solr index, and installs them both, and restarts Solr.
Because it gets the indexes and their matching database, this task runs
very quickly compared to regenerating the indexes. This "data:fetch"
task will be the preferred way to fetch data off the server.
* Warning to Windows users: You'll need to manually restart Solr per the
earlier instructions since the rake task can't do it itself yet. I don't
have a working copy of Windows at the time, so I can't really confirm
how well this works otherwise.
= Additional rake tasks
I've added a bunch of rake tasks to support the above functionality:
* solr:destroy_index -- The acts_as_solr implementation was defective
and caused the server to break, I've fixed it.
* solr:reset -- Stops solr, destroys the indexes, starts solr, builds
indexes. This is necessary if you drop/recreate/fetch a database because
the indexes will contain records that aren't in your data. The
solr:build_index only adds additional entries, but doesn't remove
orphans. However, if you used data:fetch, you already have the current
indexes and don't need to use solr:reset.
* solr:dump and solr:restore -- Dumps/restores Solr's indexes to/from a
file.
* db:dump and db:restore -- Dumps/restores database as SQL to/from a
file. This is used by data:fetch and shoul improve compatibility of the
dumps and make it easier to exclude data from them, as opposed to the
db:fetch approach of transferring the raw database binary itself.
* data:dump and data:restore -- Dumps/restores the solr and db files. In
the future, it'll probably bundle some other state too.
Annoyances
* Solr logs to the terminal: I want it to log to a file, but can't
figure out how. I've created issue 196 which describes some of the
gymnastics I've tried.
* acts_as_solr start/stop sucks: Besides not working with Windows,
there's a lot wrong with this, although I think I've resolved most of
this other than figuring out how to make Windows stop the process. The
logic for doing start/stop really needs to be rewritten from scratch
because it's so awful and I've had to make major changes to it just to
get it working.
* acts_as_solr changes: Please be careful if updating the vendor/plugins
version of acts_as_solr because it has many of my fixes. When we have
time, we should clean them up and submit patches.
-igal
Michael Bunsen
should know how much effort Igal put into finding workarounds so that
Solr could be actually usable in a practical production environment.
We heard lots of curses target at the Solr code while at the Green
Dragon last Saturday!
Michael
Scott Becker
time to write up this helpful info. Calagator is the bomb. Not
just the site itself, but the community effort around it.
Also big shout outs to Reid and Audrey for their continued hard
work.
- Scott
On Jun 13, 8:06 am, Igal Koshevoy <i...@pragmaticraft.com> wrote:
> Here are some changes to the environment that affect those using and
> developing the Calagator source code:
>
> = Setup Solr
>
> The Solr library and acts_as_solr plugin provide the search feature for
> Calagator. You must setup Solr to use Calagator on your development system:
>
Scott Becker
Just saw this:
http://www.elctech.com/blog/speedy-solr-xml-libraries
The gist of it: if you are using acts_as_solr you should make
sure you have the libxml-ruby gem installed, because acts_as_solr
uses libxml or rexml as a failover. If libxml is properly installed,
its 7X faster.
I don't know if we already have that installed, but that is
certainly good to know.
> developing the Calagator source code:
>
> = Setup Solr
>
> The Solr library and acts_as_solr plugin provide the search feature for
> Calagator. You must setup Solr to use Calagator on your development system:
>
Joe Cohen
does not get along with Windows
c:\ruby\rails_apps\calagator>rake data:fetch
(in c:/ruby/rails_apps/calagator)
* Downloading http://calagator.org/export.data...
* Replacing data...
rake aborted!
invalid bit length repeat
> developing the Calagator source code:
>
> = Setup Solr
>
> The Solr library and acts_as_solr plugin provide the search feature for
> Calagator. You must setup Solr to use Calagator on your development system:
>
Igal Koshevoy
data:fetch", I've changed the encoding format used by the production
server.
Scott and Michael: Thanks for the kind words, and suggested library.
I'll check it out soon.
Joe and other Windows users: The "rake data:fetch" should now work on
Windows, please give it a try. I've resurrected my Windows virtual
machine and it works correctly there. Having this virtual machine will
also help me debug issues with that OS more easily in the future. The
error messages you sent were helpful, but in the future please also
include the backtrace debugging information provided by running the rake
tasks with the "--trace" flag (e.g. "rake data:fetch --trace").
Anyway, here's what I changed:
r798: Fixed data and solr marhsalling so they'll work on Windows. Fixed
all File.open calls so that they use binary mode to avoid short-reads on
Windows. Fixed how database is restored on Windows, it's now done
in-place rather than through unlink because Windows can't disconnect to
release the SQLite file. Deleted EasyTgz and Archive::Tar::Minitar and
switched all code to RubyZip because it's a better long-term solution.
Fixed acts_as_solr tasks so that they display useful, human-oriented
error messages when failing on Windows rather than throwing exceptions
until we have a way to stop/restart Solr on Windows.
-igal
Joe Cohen
I tried it on one machine and it works
I wasn't actually expecting you to fix it (especially with everything
else on your plate), which is why I didn't include the full trace.
Thanks again for going way beyond the call of duty to deal with
Windows problems.
--Joe
On Jun 15, 4:30 am, Igal Koshevoy <i...@pragmaticraft.com> wrote:
.....