Fulltext indexing and french accents! argggg
Frahugo
MySQL supports accent insensitive searches when using utf8_general_ci
collation. Great, and it does work. You just have to make sure to
have UTF8 support all the way.
Now what about Ferret? It doesn't seem to support accent insensitive
searches. I would have to filter the strings and remove the
diacritics before indexing and before searching.
Anyone has a suggestion on how to solve this? Any other fulltext
search engine for RoR that you would recommend?
Thanks (see you tonight at MoR)
hugo
Carl Mercier
I don't have a real solution to your problem, but you might want to
use Solr (and acts_as_solr) instead of Ferret as it's known to scale
much better. Obviously, it depends on your app.
Frahugo
accent out of the box.
seeya
Matt Aimonetti
-Matt
Sylvain Carle
indexing speed on large datasets is orders of magnitude faster that what
we saw previously with Ferret and Solr. Two thumbs up for Sphinx.
--
Sylvain Carle / CTO
Praized Media / www.praized.com
> Unless you want to deal with Java, I would strongly recommend you use
> Sphinx and Ultrasphinx
>
> -Matt
>
> On 2/19/08, *Frahugo* <hugo.f...@gmail.com
> <mailto:hugo.f...@gmail.com>> wrote:
>
>
> Thanks Carl... I will look at Solr right now and see if it supports
> accent out of the box.
>
> seeya
>
> On 19 fév, 14:38, Carl Mercier <cmerc...@gmail.com
> <mailto:cmerc...@gmail.com>> wrote:
> > Easy solution: don't allow french text :) lol
> >
> > I don't have a real solution to your problem, but you might want to
> > use Solr (and acts_as_solr) instead of Ferret as it's known to scale
> > much better. Obviously, it depends on your app.
> >
> > On Feb 19, 2:30 pm, Frahugo <hugo.frapp...@gmail.com
Carl Mercier
I been?). I'll have to look at it!
On Feb 20, 10:40 am, Sylvain Carle <sylvainca...@gmail.com> wrote:
> Another vote for Sphinx, if the way it works fits your model, the
> indexing speed on large datasets is orders of magnitude faster that what
> we saw previously with Ferret and Solr. Two thumbs up for Sphinx.
>
> --
> Sylvain Carle / CTO
> Praized Media /www.praized.com
>
> > Unless you want to deal with Java, I would strongly recommend you use
> > Sphinx and Ultrasphinx
>
> > -Matt
>
Luc Boudreau
If I could add my two cents, Sphinx and Solr are REALLY not the same thing. Comparing these is ludicrous.
Solr is an enterprise search server, while Sphinx looks more like a library. I've worked with Solr for a while and it's a must for corporate environments. We used it as a central indexing web service for thousands and thousands of documents and about 50 applications. But don't get me wrong, I ain't advocating for Solr over Sphinx. If you have a single application which needs search functionality, definitely go for Sphinx. If you want serious architecture with high scalability and 5-9 availability, you need Solr.
Cheers
Luc
Sylvain Carle
For a few large datasets and super-fast indexing Sphinx is great for our
needs.
Check out the "Powered by Sphinx" page to get an idea of what type of
applications are a good fit for Sphinx -
http://www.sphinxsearch.com/powered.html
Solr is better at indexing "business type" documents. Facets are another
great feature of Solr, so if your app can benefit from faceted browsing
and filtering it's a good fit.
> If I could add my two cents, Sphinx and Solr are REALLY not the same
> thing. Comparing these is ludicrous.
>
> Solr is an enterprise search server, while Sphinx looks more like a
> library. I've worked with Solr for a while and it's a must for corporate
> environments. We used it as a central indexing web service for thousands
> and thousands of documents and about 50 applications. But don't get me
> wrong, I ain't advocating for Solr over Sphinx. If you have a single
> application which needs search functionality, *definitely *go for
> Sphinx. If you want serious architecture with high scalability and 5-9
> availability, you *need* Solr.
>
> Cheers
>
> Luc
>
>
> On Feb 20, 2008 10:44 AM, Carl Mercier <cmer...@gmail.com
> <mailto:cmer...@gmail.com>> wrote:
>
>
> this thing looks hot! i never heard of it until yesterday (where have
> I been?). I'll have to look at it!
>
>
> On Feb 20, 10:40 am, Sylvain Carle <sylvainca...@gmail.com
> <mailto:sylvainca...@gmail.com>> wrote:
> > Another vote for Sphinx, if the way it works fits your model, the
> > indexing speed on large datasets is orders of magnitude faster
> that what
> > we saw previously with Ferret and Solr. Two thumbs up for Sphinx.
> >
> > --
> > Sylvain Carle / CTO
> > Praized Media /www.praized.com
> >
> > > Unless you want to deal with Java, I would strongly recommend
> you use
> > > Sphinx and Ultrasphinx
> >
> > > -Matt
> >
> > > On 2/19/08, *Frahugo* <hugo.frapp...@gmail.com
> <mailto:hugo.frapp...@gmail.com>
> > > <mailto:hugo.frapp...@gmail.com
> <mailto:hugo.frapp...@gmail.com>>> wrote:
> >
> > > Thanks Carl... I will look at Solr right now and see if it
> supports
> > > accent out of the box.
> >
> > > seeya
> >
> > > On 19 fév, 14:38, Carl Mercier <cmerc...@gmail.com
> <mailto:cmerc...@gmail.com>
> > > <mailto:cmerc...@gmail.com <mailto:cmerc...@gmail.com>>> wrote:
> > > > Easy solution: don't allow french text :) lol
> >
> > > > I don't have a real solution to your problem, but you
> might want to
> > > > use Solr (and acts_as_solr) instead of Ferret as it's
> known to scale
> > > > much better. Obviously, it depends on your app.
> >
> > > > On Feb 19, 2:30 pm, Frahugo <hugo.frapp...@gmail.com
> <mailto:hugo.frapp...@gmail.com>
> > > <mailto:hugo.frapp...@gmail.com
Frahugo
thread on the subject: http://www.ruby-forum.com/topic/137629
I reduced my choice to Solr and Sphinx. I really like the fact that
Solr is tied to the model and not the database, that way I can index a
virtual attribute. I installed Solr and tweaked the schema.xml to
filter the diacritics (french accents) and it works well.
On the other hand, Sphinx seems to be the fastest one, but it does not
support live updates and you have to "rotate" the index via a cron job
(they have a multi-delta file concept that would allow refreshing the
index every minute for example). A major drawback of Sphinx is the
fact that it will only index data found in MySQL, but in my case I
could use the :concatenate function on the association. I will
install Sphinc and see how it works and if I can filter the diacritics
like in Solr.
Will post results.
On Feb 20, 11:08 am, Sylvain Carle <sylvainca...@gmail.com> wrote:
> Indeed. That's why I added "if the way it works fits your model".
>
> For a few large datasets and super-fast indexing Sphinx is great for our
> needs.
>
> Check out the "Powered by Sphinx" page to get an idea of what type of
> Solr is better at indexing "business type" documents. Facets are another
> great feature of Solr, so if your app can benefit from faceted browsing
> and filtering it's a good fit.
>
> > If I could add my two cents, Sphinx and Solr are REALLY not the same
> > thing. Comparing these is ludicrous.
>
> > Solr is an enterprise search server, while Sphinx looks more like a
> > library. I've worked with Solr for a while and it's a must for corporate
> > environments. We used it as a central indexing web service for thousands
> > and thousands of documents and about 50 applications. But don't get me
> > wrong, I ain't advocating for Solr over Sphinx. If you have a single
> > application which needs search functionality, *definitely *go for
> > Sphinx. If you want serious architecture with high scalability and 5-9
> > availability, you *need* Solr.
>
> > Cheers
>
> > Luc
>
Colin Surprenant
Sphinx is a great/solid full-text search engine, used by top sites/
applications, with a great community and very well supported by its
developer.
The advantage of Sphinx is its blasting speed both for indexing and
performing search. For example, indexing ~20M MySQL rows takes about
30 minutes. There are numerous client libraries, in particular for
Ruby/Rails (and PHP, etc...).
Before using Sphinx you have to make sure it can fit in your context -
it is not a 'universal' enterprise search engine. It currently only
support MySQL, PostgreSQL and XML data sources and the indexes are
static - when you have new data you have to re-index. To avoid
reindexing everything you can use two indexes; a main and a delta
index where the delta index will contain only the 'new' data since the
last main index refresh. A typical scenario is to refresh the main
once a day and the delta every minute or so - you can ajust the
periods for your data context.
Another great feature is it distributed search feature. You can define
distributed indexes that will parallelize queries over multiple remote
Sphinx engines. This can be used both to scale and/or to do failover.
Have fun!
Colin.
Thierry
Nice discussion here. We were looking for a search solution for a
small web app and I came accross acts_as_fulltextable
(http://blog.wonsys.net/posts/26-our-first-plugin-
acts_as_fulltextable/). Anyone heard about it tested it?
Thanks,
Thierry
On Mar 4, 9:05 pm, Colin Surprenant <colin.surpren...@gmail.com>
wrote:
Frahugo
We actually went with SOLR (acts_as_solr), which is closer to our
needs than Sphinx (primarily because it can index virtual attributes
in models and updates its index on model save). Although it requires
installing JRE (java virtual machine), it runs pretty fast (it's a lot
better than ferret) and it has a web admin interface which is pretty
useful for testing the search/index features.
It does not support accent insensitive searches out of the box. So I
had to hack the schema.xml and add some filters to remove accents when
indexing and querying. I also had to add some code in my models to
make sure the text sent to SOLR is valid UTF8 text (our users are
pasting text from other applications and we sometimes end up with
problematic characters from other encodings, like the famous single
quotation mark (http://www.fileformat.info/info/unicode/char/2019/
index.htm).
acts_as_solr does not have a rake task for reindexing, but it's easy
to build one (and someone wrote one, just do a quick search on the
internet for that).
The latest Advanced Rails Recipes (http://www.pragprog.com/titles/
fr_arr) has a chapter on that (and on Sphinx as well) to get you
started.
Hoping this will help some of you guys,
hugo
Colin Surprenant
A few other thoughts I forgot to mention in my last post; the whole
thread started about utf-8 and french accents - so for Sphinx, (and I
actually just went through this) unicode/utf-8 is well supported. If
your DB data is utf-8, you can use the character remapping feature
which by default simply remaps to lowercases (for case insensitive
search) and add you own rules to remap accented characters to non-
accented ones. The nice thing is that this allows you to support many
languages/character sets.
If using MySQL in utf-8 - do not forget to specify SET
CHARACTER_SET_RESULTS=utf8 before your query!
Also, it is worth noting that in the latest 8.3 release of PostgreSQL,
full text search has been integrated. There is the acts_as_tsearch
plugin http://code.google.com/p/acts-as-tsearch/ but I don't know if
it integrates with the new 8.3 feature.
Colin.