Re: [jcms] Joomla CMS General Search Needs Some More Improvement

[Mark Dexter]

Are you using Search or Smart Search? If you haven't tried Smart Search, I would try that. Search is very outdated and probably will be dropped in a future version. I believe the plan is that our future efforts will go into Smart Search. Mark

On Wed, May 15, 2013 at 12:24 AM, Mohamed Akram - Sl Joomla Developer <akram...@gmail.com> wrote:

Dear Everyone,
First of all  I want to brief me about ,I am a joomla developer and Joomla lover from Sri Lanka, (you may find about me in http://www.akramwahid.com)

I am Really sorry for late message, I am pretty worried that I couldnot send My proposal for this year
 GSOC and I dont know whether its has been finalized, with my busy schedule in launching my personal
buy and sell site based on Joomla 3 (http://www.autodeal.lk), I couldnt focus on this.

I have been getting some negative feed back from 100 of our clients in the way current Joomla
Genral search work since last couple of months, Finally When I go through this meter with other developer in our team
we found it that the way the current Joomla general search work is not more relevant and need some
more improvements, and I feel I this should be discussed with the Joomla expert in the developer team,
Can I submit my proposal regarding Improving the General Search in Joomla CMS hereafter?

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.
To post to this group, send an email to joomla-...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-cms?hl=en-GB.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[elin]

It's too late for GSOC for this year but there was a project idea for Smart Search improvements (I have a list of things for that) that it would be great to have someone or a group of people work on. 

Elin

[Mohamed Akram]

Dear Elin, I am really worried, anyway you can just consider this metter, this is not really regarding the improvements in smart search, its about improvement in General search of joomla, the problem is with search result in current joomla com_search, for some of the client I have hacked core joomla search component to fix this problem, but I feel its not a right solution and I do want to expect to bring  a right solution

[wdburgdorf]

While it is good news that you plan to put efforts into Smart Search, I hope you are not going to drop standard search any time soon. I recently worked on a project (J!2.5) where Smart Search did not work well at all. Various errors in the results, duplicate results etc. Probably compatibility issues with K2/sh404sef/Falang. Standard search worked well enough for us ...

Ralf.

[Mohamed Akram]

Dear Everyone , wait I will explain the problems we and our client face,

 In one of Joomla site, in general search there is a specific search area for election results as well , but when a search is performed in the general search with match is set to all words, and the string is "election 2013" and assume no any specific search area is selected , in the result set it returns without a proper ordered result, in the result set top there are some articles which has "selection" string in the body but the title is some thing that not relevant to searched string "election 2013", this happens Because there is no any order that results sets are returned by each search plugin, in my case the content search plugin fired before the election search plugin run, its all about problem with search result relevancy, the most relevant results should be in top although its a all word search,

the whole problem is the undefined order in which each search plugin run and return a result set to Search Model,  As a temporary solution , I had to hack joomla search model bit, I write entire results set returned on line number 131 in SearchModel, to  a temporary database table and perform mysql full text search and some Like queries to get most relevant result to top and rest to last,

 

--

[Chris Davenport]

It's unlikely that anyone will make any significant changes to the old search, one of the reasons being that you can't realistically do things like relevancy search with it (well, not in a scalable way at least).

There is a working group covering search improvements here: http://docs.joomla.org/Search_Working_Group

If you know of any issues with Smart Search not already listed there then please open a bug tracker item for them.

Thanks,

Chris.

--
Chris Davenport
Joomla Production Leadership Team

[elin]

General search is a very simple search that just returns results in the order it finds them.  That's why we have smart search, which takes care of all of those things. It is quite easy to make a smart search plugin for most extensions, you just need to identify the table and fields to be be indexed.

Unfortunately the core cms cannot be responsible for the implementations done by third parties or their interoperability. All we can do is suggest that you as a customer strongly urge  the extension developers you work with to improve their support of smart search (and tags, and acl and other feature of the core).  Hopefully  smart search will become even easier to support over time, perhaps getting to the point that we don't need plugins, but it's really the work of an hour or two to make a plugin.

2.5 is a LTS release so it will have simple search forever. I would think it is here for the duration of the 3 series as well though certainly at some point the default search module will probably be changed over (but not until we solve all issues in Japanese, Thai and similar languages). Simple search does not meet the needs of a modern CMS. That is why unless you are using a multibyte language or extensions that for some reason aren't working you should be using smart search.

Elin

[Janich Rasmussen]

Sorry if this is offtopic, but I got triggered by the talk of search vs finder.

Finder generates a lot of data, which can have a deadly impact on some lowcost web hosts/shared hosts, which I believe is pretty common for a lot of joomla users.

Are there any performance reports on the difference of these two components (with some real world statistics)?

[Mark Dexter]

In theory, Finder should be slower when saving an item (to generate the more detailed indexes) and faster and much more precise when returning search results. I am not aware of any real-world stats that have been shared. Also, I don't think we have any information about how many sites use Finder. Mark

[Janich Rasmussen]

In theory, yes, but witnessing a site with 500k rows on a shared host really makes one wonder.

Maybe this would be an interesting point to shed some light on for the WG.

Anyway - Just wondering. :)

[Mark Dexter]

I would expect that the old com_search would likely be extremely slow in that scenario, since it searches the medium text column of every item with no help from indexes. Mark

On Sat, May 18, 2013 at 3:12 PM, Janich Rasmussen <jan...@gmail.com> wrote:

In theory, yes, but witnessing a site with 500k rows on a shared host really makes one wonder.

Maybe this would be an interesting point to shed some light on for the WG.

Anyway - Just wondering. :)

Mvh,

Janich Rasmussen
Freelance Web Enthusiast

Direct: +45 31 13 61 01
E-mail: jan...@gmail.com

Twitter: @janichdk

Skype: janich.rasmussen

2013/5/19 Mark Dexter <dexter...@gmail.com>

--
You received this message because you are subscribed to a topic in the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/joomla-dev-cms/UuRkNr1BiLA/unsubscribe?hl=en-GB.
To unsubscribe from this group and all of its topics, send an email to joomla-dev-cm...@googlegroups.com.

To post to this group, send an email to joomla-...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-cms?hl=en-GB.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Node-0]

Finder is useful for what it is a general purpose indexer one can just deploy on any commodity LAMP host.
Is it fast? For small scale (less than 1000 new rows per half hour) indexing it's fine, for large scale 10,000+ new rows per hour it's unsuitable as a solution. As a test I imported 12,000 rows into the content table (simple account and contact data featuring name + address + phone number) back in 10/2012 during a comparison of Joomla's finder engine vs Sphinx on the backend.

Finder took 15 minutes to index 12,000 rows of simplistic content whereas Sphinx tackled the chore in 100ms.
Now this is an unscientific comparison but it speaks to your question about whether finer is "web scale" capable. It is not (nor do I believe was it ever intended to be) "web scale" capable. Any site owner sporting databases with half a million records on a shared host is basically asking for exactly what they're going to get, hideous latency.

It's apples vs oranges frankly.

In my opinion the new indexer is a blessing, but it should be made clear that it's no magic genie for searching massive volumes of data.

[Chris Davenport]

I agree.  Smart Search is never going to scale as well as a native search engine, however there are improvements that can be made and these are noted on the WG page.  They just need people to step forward and code them.

In particular, the number of terms per phrase is currently hard-wired at 3.  This is fine for small to medium sites and gives a good user experience for autocompletion, but the index files can become really big and insertions/deletions on big indexes can be very slow.  To help alleviate that issue we could make the terms per phrase a parameter that can be reduced to 2 or even 1 for larger sites.

The other issue is that someone saving a very large content item can experience a long delay while the content is (re-)indexed.  It can sometimes get so excessive that the server can time-out and the user will most likely get a white screen of death with the content left partially indexed.  The solution there is to have the indexer run as a background process so that on save a request to re-index is placed on a message queue.  A background process then processes the requests from the queue without having to worry about latency and timeouts.  I started work on this over Christmas but haven't had chance to get it finished yet.

Chris.

--
You received this message because you are subscribed to the Google Groups "Joomla! CMS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-cm...@googlegroups.com.
To post to this group, send an email to joomla-...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-cms?hl=en-GB.
For more options, visit https://groups.google.com/groups/opt_out.

--

[piotr_cz]

While I have no issues with search performance, I think it's worth
discussing how to use new UCM tables for results, just like the tags
component does.

I mean, to use different criteria for searching, and/ or showing
results.


On May 19, 10:51 am, Chris Davenport <chris.davenp...@joomla.org>
wrote:

> I agree.  Smart Search is never going to scale as well as a native search
> engine, however there are improvements that can be made and these are noted
> on the WG page.  They just need people to step forward and code them.
>
> In particular, the number of terms per phrase is currently hard-wired at
> 3.  This is fine for small to medium sites and gives a good user experience
> for autocompletion, but the index files can become really big and
> insertions/deletions on big indexes can be very slow.  To help alleviate
> that issue we could make the terms per phrase a parameter that can be
> reduced to 2 or even 1 for larger sites.
>
> The other issue is that someone saving a very large content item can
> experience a long delay while the content is (re-)indexed.  It can
> sometimes get so excessive that the server can time-out and the user will
> most likely get a white screen of death with the content left partially
> indexed.  The solution there is to have the indexer run as a background
> process so that on save a request to re-index is placed on a message
> queue.  A background process then processes the requests from the queue
> without having to worry about latency and timeouts.  I started work on this
> over Christmas but haven't had chance to get it finished yet.
>
> Chris.
>

> > For more options, visithttps://groups.google.com/groups/opt_out.

[Chris Davenport]

I'm not sure I follow. Surely the search results should be entirely independent of the underlying data models?

Chris.

[elin]

Like smart search ucm has a concept of types which can be used to limit  results.  In the long run we should be able to merge the two types tables and maintain that behavior.   Smart search has many, many work arounds /solutions for the same issues ucm does such as it stores both state and published for every item thus sidestepping that issue. 

Also, with ucm maintaining a list of all content items and unique ids smart search will no longer need to create that list each time it does a complete index. With unique ids and a routing solution it will also no longer need to maintain its own routing information for each item or the blob field. In other words, it can  essentially stop providing a ucm solution which is what it does now and focus totally on providing a search solution.  Add version history to the mix and if you are using the cli to index you no longer need to reindex unchanged items. 

We did find a couple of issues causing very very slow indexing prior to 3.1 when some notices were being generated when debug was enabled and I'm sure there are more things like that also.   It really helps is developers who are using smart search try to identify where problems are even if they can't solve them.  

Elin