Issue with Joomla! 3 smart search model's getListQuery method

[Clubnite]

Hi,

i am a user of smart search since it was distributed by its original author Rob Schley as jxFinder and since then missing one important feature - hooking into the query creation process before it is executed to fetch results.

The reason for this wish lies within the limitation of smart search model's query building function. Everything relies on the database fields and there is no evaluation for some fields to be combined via AND OR OR. It is only possible to define this glue in general defaulting to AND, but this sometimes creates where clauses that are not wished.

I integrated a location search for items having a continent id, a country id and a region id. All 3 ids are indexed with every item. The search filters are radiogroups rather than dropdown lists. Now, when it comes to search, the following problem occurs - a user selects:

1. continent 'Europe' (10 indexed items total)

2. countries 'France' (1 indexed item total), 'Austria' (1 indexed item total), 'Germany' (7 indexed items total), 'Ireland' (1 indexed item total)

3. region 'Bavaria' (1 indexed item total)

The expected output would be 4 results:

France + Austria + Bavaria + Ireland

But instead only 1 result will be returned - Bavaria. Why? Because all filters are combined via AND. In other words, the query is:

Select all items that are in Europe AND France AND Austria AND Ireland AND Germany AND Bavaria.

What i really want is:

all items from France + all items from Austria + all items from Ireland + 1 item from Germany.

This extra processing is not part of finders getListQuery function but required for this scenario. So one is forced to hack the core to get the extra processing hooked into the query build.

For this reason i manipulated the getListQuery method to dispatch a new event which is then handled by a smart search plugin - onFinderGetListQuery($context, FinderModelSearch $model, FinderIndexerQuery $query, $sql). This handler, in my case, inspects the FinderIndexerQuery for geo filters selected and then processes the dependencies into the correct relations. Afterwards it removes wrong where statements from the passed in list query and injects the correct ones. When it is done, the query will be returned to be executed.

For me this is the only way to get hands on the query creation process and to inject custom clauses.

I can imagine that there are more scenarios like this that require a possibility to get hands on the query to be executed just before it is executed.

I think, this component should integrate a stage for devs to hook in. So i suggest to integrate the dispatching of this event.

What do you think about this idea?

[Matt Thomas]

This sounds very much like the generic query plugin events that have been discussed before (Sorry, I can't find a link at the moment). K2 has something like this and it is very powerful for manipulating the queries contextually. I'd love to see core adopt this universally.

+1

Best,

Matt Thomas

Founder betweenbrain™

Lead Developer Construct Template Development Framework

Phone: 203.632.9322
Twitter: @betweenbrain

Github: https://github.com/betweenbrain

--
You received this message because you are subscribed to the Google Groups "Joomla! General Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-gene...@googlegroups.com.
To post to this group, send an email to joomla-de...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-general.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Chris Davenport]

If memory serves you should be able to do this kind of search by submitting a search query string instead of (or as well as) requesting by taxonomy ids.  In theory you should be able to submit a search query like: "Country:France OR Country:Austria OR Country:Germany".  It won't handle bracketted clauses or anything complex like that, but it should work for your use-case (if it doesn't it's a bug).  The code for parsing the search query is in the FinderIndexerQuery class.

I don't think that allowing developers to interfere with the internals of the search queries with a plugin is a good idea.  What is needed is an improved and comprehensive search API and that's something I'm hoping to work on for Joomla 4.x.

Chris.

--
Chris Davenport
Joomla Production Leadership Team

[Clubnite]

Thanks for pointing us to this interesting capability of FinderIndexerQuery class. I wonder where and how this query construct should be created on search form submit. Correct me if i'm wrong, but i can see this happening in a JS function only, which makes the whole process somehow unreliable, as it depends on JS and wouldn't work for users with JS disabled, right?

This seems way more complicated and requiring more pre-processing just to get the query string created than the integration of an event allowing for pre-executional sql-manipulation. But mabe i'm totally wrong. Any correction is highly welcome.

Beside this argument this solution doesn't fit my needs as i have to execute 1-2 extra database queries while processing the hirarchical structure of the geofilter items. This processing isn't trivial and can and should not be applied in a JS function.

Why do you think that interfering is no good idea? The interference is triggered by events - system internal. I also thought if potential dangers through bad code injection, but, if you think of the sql query contruction chain than all input sanitization already happend up to this stage, meaning, the input is clean. The sql query object passed to the handler is not accessable from outside. So far i cannot find any potential leak for bad code injection.

But i see a big improvement in this feature. As a second argument: The search query, which is finally executed is not only getting fixed in sense of the geo filter items, but also limited to a specific link_id range, which is also hooked in by the mentioned handler. The reason for this is, that i have to work with real-time date ranges from an availability calendar of a booking system. I cannot use the start-/end_date capability of finder to cover this need, as the dates must not be indexed. Every query having a date range involved must check in realtime for current availability states and limit the previously fetched link_ids coming from getListQuery.

So, the need is absolutely and much bigger than any problems that i cannot see (yet). And i'm almost sure that i'm not alone with this opinion. But, again, please correct me, if i'm wrong.

[Chris Davenport]

If you don't want to rely on client-side construction of the URL then you could write a little system plugin that modifies the URL on the server instead.

The problem with interfering with the internal SQL queries is that you will create something that is fragile and will very probably break if Smart Search is updated (although that seems unlikely during the 3.x cycle).

Chris.

--

You received this message because you are subscribed to the Google Groups "Joomla! General Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-gene...@googlegroups.com.
To post to this group, send an email to joomla-de...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-general.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Clubnite]

The problem with interfering with the internal SQL queries is that you will create something that is fragile and will very probably break if Smart Search is updated (although that seems unlikely during the 3.x cycle).

I see your point, Chris, but isn't it the case, that due to ongoing platform- or cms-changes every extension developer is forced to update his extensions to keep them compatible ... ongoing? It is nothing new, that things changes. This isn't specific to this case only. So, i see no problem in modifying my query processing routine after something changed in Smart Search or the underlying platform. As long as the ability to hook in stays available (e.g. through a dispatched event) there should be no problem for any extension developer to update its extension code to keep it working and compatible to platform changes.

Wouldn't you agree?

So i repeat my vote for an event dispatched after when list query is created. Mabe you could discuss this valuable improvement in the leadership team. ;)

[Chris Davenport]

I would be reluctant to encourage bad practice especially when I'm not (yet) convinced that your problem requires it.  Re-reading your original post I think that one way you could approach this would be to set up catch-all entries in the region taxonomy which are selected in your UI by default.

So for the example you gave, a user who selected France together with Bavaria would generate the query (Country: France OR Country: Germany) AND (Region: All France OR Region: Bavaria) which is equivalent to France + Bavaria.

In this case there is no need to manipulate the query string because you can get the correct results using just the taxonomy ids.

There are probably other ways of solving this particular problem without hacking into Smart Search's internals, but this is the first to spring to mind.

Chris.

--

You received this message because you are subscribed to the Google Groups "Joomla! General Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-gene...@googlegroups.com.
To post to this group, send an email to joomla-de...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-general.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Clubnite]

Re-reading your original post I think that one way you could approach this would be to set up catch-all entries in the region taxonomy which are selected in your UI by default.

Unfortunately the UI does not allow for a "select all" toggle. It is not wished by the customer.

So for the example you gave, a user who selected France together with Bavaria would generate the query (Country: France OR Country: Germany) AND (Region: All France OR Region: Bavaria) which is equivalent to France + Bavaria.  

 I still can not imagine where your query is constructed. So far i never saw this kind of smart search query in action to get a feeling for it and understand how it works. Also, does the search model semantically understand "All France"? I think, it must be teached to understand this "language", mustn't it? The filters use numerical taxonomies instead of translated filter names, which is much more reliable in processing.

In this case there is no need to manipulate the query string because you can get the correct results using just the taxonomy ids.

Sorry, i don't see, how. Again, can you provide me with an example in action, that i could try and play around to understand it? Seriously, i would like to understand this kind of query, but i don't get it the abstract way. I'm more the "understanding by doin'" type.

[Chris Davenport]

You can hide the "Select All" options in CSS.

The plugin determines the taxonomies that each item belongs to.  So when it sees an item in Bavaria it will add it to Region:Bavaria and Region:All Germany as well as Country:Germany and Continent:Europe.  Presumably you already have some tables that will allow the plugin to determine which regions belong to which countries and which countries belong to which continents.

To generate the user interface your module/component will need to look up the taxonomy ids.  Unfortunately there isn't an API to do that (another item on the to-do list) but it's very easy to grab the ids directly from the database with a simple query in combination with your continent/country/region database.

I can't imagine how this is going to look without JavaScript.  Personally I wouldn't even attempt to do something like this without it.  I'd probably use Ajax to pull in the different lists as the user selected different items, but whatever.  For each region, render your radiobutton group with the default "All <region>" option pre-selected but hidden in CSS.  As soon as a user selects a region, the "All <region>" option is deselected (because it's a radio button).  When the user clicks "Search" all the correct taxonomy ids have been set and Smart Search does the rest.

In general, when designing search applications, I find that it is best to start by designing the user interface (ask the question: "What kinds of searches will the user need to perform") then design the taxonomies that will be required to perform those searches efficiently.  It's the job of the plugin to build those taxonomies.

Hope that hasn't confused you even more!

Chris.

--

You received this message because you are subscribed to the Google Groups "Joomla! General Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to joomla-dev-gene...@googlegroups.com.
To post to this group, send an email to joomla-de...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-general.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Clubnite]

I get your point regarding the pre-checked "select all" option. Indeed i had these master checkboxes in use for countries and regions, before i reverted to not use 'em, as with so many params (all countries ~ 280 and all regions ~ 600) i encountered a max query params issue with the server. My thinking was also, why to pass that much overhead to the server on every request. This ends up in a miles long query string and forces the application to calculate more input than necessary.

The plugin determines the taxonomies that each item belongs to.

Another thing left unclear is the several times mentioned plugin. To pass a job to a plugin an event is required. But, there are no events like e.g. 'onFinderBeforeDisplayResults' or 'onFinderBeforeShowResults' that i could hook into. This is exactly the issue i am talking about. The Smart Search component doesn't provide this kind of entry point for external access. Only indexing and state change trigger events. So, what kind of plugin and processing stage are you refering to in your idea?

Presumably you already have some tables that will allow the plugin to determine which regions belong to which countries and which countries belong to which continents.

These tables are available.

To generate the user interface your module/component will need to look up the taxonomy ids. Unfortunately there isn't an API to do that (another item on the to-do list) but it's very easy to grab the ids directly from the database with a simple query in combination with your continent/country/region database.

My general intention is to create performant code and prevent unnecessary database queries, which is also, why i'd prefer to hook into the query creation stage rather than any other.

I can't imagine how this is going to look without JavaScript. Personally I wouldn't even attempt to do something like this without it.

See, i'd like to keep this kind of dependency off my component. In sense of progressive enhancement it should always be possible to do something natively before improving it the AJAX way.

For each region, render your radiobutton group with the default "All <region>" option pre-selected but hidden in CSS. As soon as a user selects a region, the "All <region>" option is deselected (because it's a radio button). When the user clicks "Search" all the correct taxonomy ids have been set and Smart Search does the rest.

Not a solution in my case. All geo-filters but the continent are checkboxes as it must be possible for a user to select more than one filter a time. Sure, i could control and (un)check the master checkbox via JS, but from my experience i know that errors caused by other scripts loaded may prevent own scripts from proper work. This makes the whole thing very unreliable, which is why i want to avoid a JS-based solution. Disabling unselected checkboxes via JS as it is of now isn't dramatic if not working (due to an error from another script). But the whole thing not working anymore is dramatic.

I have to repeat myself. This geo-filter isn't the only hook i have to apply. In fact there are many more.

The availability dates mentioned, which are not indexed, since they come from another database and must be fetched in real time.

This 'favourite' flag for a result. A user can mark a result as favourite. On query build it can then easily be injected as another where clause to join more information from the favourites table and thus fetch all required information in one run.

Also i have to process a taxonomy like 'num travellers allowed'. Indexed is the number of allowed travellers per item. The dropdown list shows all indexed values. The query will then be created with a where clause like 'AND num_travellers = %d', which looks for results with only this number allowed. But in fact all numbers >= the selected must be allowed. This circumstance must be checked and fixed on query building.

In general, when designing search applications, I find that it is best to start by designing the user interface (ask the question: "What kinds of searches will the user need to perform")

then design the taxonomies that will be required to perform those searches efficiently. It's the job of the plugin to build those taxonomies.

Sure, this is how i did it. Nevertheless this process is very limited in my case.

Hope that hasn't confused you even more!

Not really. Much of your thinking was part of my thinking too, when i pored over the best implementation of this search page. And i thank you very much for sharing your experience and provide solutions, Chris.