Why this expression is not found by xapian

[Mario Menezes]

Hi,

   I'm indexing a product whose ProductIndex is constructed in this way:

Class ProductIndex( indexes.SearchIndex, indexes.Indexable ):
    text = indexes.CharField( document = True, use_template = True )
    name = indexes.CharField(model_attr='name')
    description = indexes.CharField( model_attr = 'description' )
    creation_date = indexes.DateTimeField( model_attr = 'creation_date' )
    codigo = indexes.CharField(model_attr='codigo')
    codigoUnitario = indexes.CharField(model_attr='codigoUnitario')
    codigoAgregado = indexes.CharField(model_attr='codigoAgregado')
    brand = indexes.CharField(model_attr='brand',faceted=True)
    category = indexes.CharField(model_attr='get_category',faceted=True)

    content_auto=indexes.EdgeNgramField(model_attr='name')
     
    def get_model( self ):
        return Product

    def index_queryset( self ):
        """Used when the entire index for model is updated."""
        return self.get_model().objects.filter( creation_date__lte = datetime.datetime.now(),active=True )
   
    def prepare_category(self,obj):
        return obj.get_category().name

 
 In my model, description is an TextField and I've put this lines in the template:

{{ object.name }}
{{ object.name }}
{{ object.name }}
{{ object.description }}
{{ object.description }}
{{ object.description }}
{{ object.description }}
{{ object.slug }}
{{ object.brand.name }}
{{ object.category.name }}
{{ object.codigo }}
{{ object.codigoUnitario }}
{{ object.codigoAgregado }}



     Now, I've several products whose description contains the words:  "tinta" "acrílica" "ambiente" "externo". These words are spread in the description. For instance, a description could be:
  "É uma tinta acrílica, que por ter acabamento semibrilho proporciona alta impermeabilidade quando aplicada em ambientes externos e em superfícies internas oferece grande facilidade de limpeza."

          Sorry for the portuguese text, but the point is that the words are in the description but if I search for "tinta acrílica ambiente externo", Xapian returns zero docs.

          How to make it find this? Should I improve the template? I have no idea how to do this - please help me!  Is there any configuration parameter to help with this? Help me here too!

           Thanks again!

[David Sauve]

If you search for the terms "tint ambiente externo" does it find them?  I wonder if it's having an issue with the accents.

Another possibility.  How are you doing the search?  If you use SearchQuerySet.filter(content='ambiente', content='externos', content='tint', content='acrílica') do you get results?  This isn't the same as SearchQuerySet().filter('tint acrílica ambiente externos')

The former is four terms ANDed together and the latter is a phrase search so it'd only find documents that have the phrase in it.

--
You received this message because you are subscribed to the Google Groups "django-haystack" group.
To view this discussion on the web visit https://groups.google.com/d/msg/django-haystack/-/Ii17kQXbbNwJ.
To post to this group, send email to django-...@googlegroups.com.
To unsubscribe from this group, send email to django-haysta...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-haystack?hl=en.

[Mario Menezes]

Hi David,

    I'm doing the search without accents: "tinta acrilica ambiente externo". But this doesn't help. No results found.

    Regarding the second possibility, I've tried to search directly using the shell but I'm getting some errors in the SearchQuerySet expression:]

>>> from haystack.query import SearchQuerySet

>>> SearchQuerySet.filter(content='ambiente',content='externos',content='tint',content='acrilica')

File "<ipython-input-2-7fcbc4a653e2>", line 1

SyntaxError: keyword argument repeated (<ipython-input-2-7fcbc4a653e2>, line 1)

  Using the second expression you suggested doesn't solve the problem too.

>>> SearchQuerySet.filter('ambiente externos tint acrilica')

TypeError        Traceback (most recent call last)

/home/mario/progs/python/qmaterial/trunk/qmaterial/<ipython-input-2-9c4084699dfc> in <module>()

----> 1 SearchQuerySet.filter('ambiente externos tint acrilica')

TypeError: unbound method filter() must be called with SearchQuerySet instance as first argument (got str instance instead)

As I'm not good with haystack I couldn't find the problem with the expressions above. Sorry.

   Thanks

2012/4/23 David Sauve <dns...@gmail.com>

--
--
Mario O.de Menezes, Ph.D.
"Many are the plans in a man's heart, but is the Lord's purpose that prevails" Pv 19.21
http://www.momenezes.com     -    LinuxUser: #24626

[David Sauve]

Doh, I'm rusty and was going from memory.

That should be:

  SearchQuerySet().filter(content='ambiente').filter(content='externos').filter(content='acrílica').filter(content='tint')

and

  SearchQuerySet().filter(content='ambiente externos tint acrilica')

[Mario Menezes]

Hi,
 
  It seems that the queries from my form are being done using the last form: SearchQuerySet().filter(content="ambiente externos tinta acrilica"). This brings no results.
  
    Using the concatenated filters

>>> result = SearchQuerySet().filter(content='ambiente').filter(content='externos').filter(content='tinta').filter(content='acrilica')

In [12]: result
Out[12]: [<SearchResult: catalog.product (pk=1)>, <SearchResult: catalog.product (pk=2)>, <SearchResult: catalog.product (pk=3)>, <SearchResult: catalog.product (pk=4)>, <SearchResult: catalog.product (pk=5)>, <SearchResult: catalog.product (pk=6)>, <SearchResult: catalog.product (pk=7)>, <SearchResult: catalog.product (pk=8)>, <SearchResult: catalog.product (pk=9)>, <SearchResult: catalog.product (pk=10)>]

     Now, how do I implement this? Im my SearchView? Should I split the content and build the queryset according the principle above?

     Thanks

Em segunda-feira, 23 de abril de 2012 15h37min12s UTC-3, notanumber escreveu:

Doh, I'm rusty and was going from memory.

That should be:

  SearchQuerySet().filter(content='ambiente').filter(content='externos').filter(content='acrílica').filter(content='tint')

and

  SearchQuerySet().filter(content='ambiente externos tint acrilica')

On Monday, 23 April, 2012 at 11:33 AM, Mario Menezes wrote:

Hi David,

    I'm doing the search without accents: "tinta acrilica ambiente externo". But this doesn't help. No results found.

    Regarding the second possibility, I've tried to search directly using the shell but I'm getting some errors in the SearchQuerySet expression:]

>>> from haystack.query import SearchQuerySet

>>> SearchQuerySet.filter(content='ambiente',content='externos',content='tint',content='acrilica')

File "<ipython-input-2-7fcbc4a653e2>", line 1

SyntaxError: keyword argument repeated (<ipython-input-2-7fcbc4a653e2>, line 1)

  Using the second expression you suggested doesn't solve the problem too.

>>> SearchQuerySet.filter('ambiente externos tint acrilica')

TypeError        Traceback (most recent call last)

/home/mario/progs/python/qmaterial/trunk/qmaterial/<ipython-input-2-9c4084699dfc> in <module>()

----> 1 SearchQuerySet.filter('ambiente externos tint acrilica')

TypeError: unbound method filter() must be called with SearchQuerySet instance as first argument (got str instance instead)

As I'm not good with haystack I couldn't find the problem with the expressions above. Sorry.

   Thanks

2012/4/23 David Sauve

If you search for the terms "tint ambiente externo" does it find them?  I wonder if it's having an issue with the accents.

To post to this group, send email to django-haystack@googlegroups.com.
To unsubscribe from this group, send email to django-haystack+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/django-haystack?hl=en.

--
You received this message because you are subscribed to the Google Groups "django-haystack" group.

To post to this group, send email to django-haystack@googlegroups.com.
To unsubscribe from this group, send email to django-haystack+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/django-haystack?hl=en.

--
--
Mario O.de Menezes, Ph.D.
"Many are the plans in a man's heart, but is the Lord's purpose that prevails" Pv 19.21
http://www.momenezes.com     -    LinuxUser: #24626

--
You received this message because you are subscribed to the Google Groups "django-haystack" group.

To post to this group, send email to django-haystack@googlegroups.com.
To unsubscribe from this group, send email to django-haystack+unsubscribe@googlegroups.com.

[David Sauve]

If you're using the built-in SearchForm, it should be splitting the terms via the `auto_query` method (https://github.com/toastdriven/django-haystack/blob/master/haystack/forms.py#L46 and http://django-haystack.readthedocs.org/en/latest/searchqueryset_api.html#auto-query).

You can derive your own form from SearchForm or use `auto_query` yourself to split the query string in to terms.

Also, if you have access to and #irc client, you might want to jump in to #haystack for even quicker responses!

To view this discussion on the web visit https://groups.google.com/d/msg/django-haystack/-/tBGNV8aKp3gJ.
To post to this group, send email to django-...@googlegroups.com.
To unsubscribe from this group, send email to django-haysta...@googlegroups.com.

[Mario Menezes]

AutoQuery seems is not working here:

In[7]: from haystack.inputs import AutoQuery

In [8]: sqs = SearchQuerySet().filter(content=AutoQuery('tinta acrilica ambiente externo'))

In [9]: sqs
Out[9]: []


Yes, I've built a custom form, but switching back to ModelSearchForm didn't help too.

My urls.py looks like:

from haystack.views import SearchView, search_view_factory

sqs = SearchQuerySet().facet('brand').facet('category')

urlpatterns += patterns('haystack.views',
    url(r'^busca$', FacetedSearchView(form_class=FacetedSearchForm , searchqueryset=sqs), name='haystack_search_marca'),
    url(r'^busca$', search_view_factory(
        view_class=SearchView,
        template='lfs/search/search.html',
        form_class=ProductSearchForm,
    ), name='haystack_search'),
)


  Thanks for your attention and patience. I'll try to use irc, Just need to install a client and learn to use it.

  Best regards,

[Mario Menezes]

Hi,

   I've being trying to understand AutoQuery but could not succeed at all.

   No matter what I try, AutoQuery just give me a empty result.

   When trying to debug it on shell, what is the appropriate format of the query string to be passed?

   For instance, I've tried:

  q = "tinta acrilica ambiente externo"
 
   Then: 
  
  SearchQuerySet().filter(content=AutoQuery(q))

   And the result is []

   Then I tried:

  q = {"q": "tinta acrilica ambiente externo"}
 
  SearchQuerySet().filter(content=AutoQuery(q))

   And again, the result is []

   If I try:

  SearchQuerySet().filter(content="tinta").filter(content="acrilica").filter(content="ambiente").filter(content="externo")

   I got lots of results, properly indexed.

   I did some further experiments, trying to see which methods or attributes AutoQuery exposes:

>>> q = AutoQuery(content="tinta acrilica ambiente externo")
>>> q
<AutoQuery 'tinta acrilica ambiente externo'>

>>> q.query_string
'tinta acrilica ambiente externo'

>>> q.kwargs
{}

  
   Now, what else should I try to debug what's going on with my haystack installation?

   Why AutoQuery isn't breaking the phrase properly?

   Please I do need some help to fix this!

   BTW, I can not access IRC as it's blocked by my Institute firewall policy. Too bad!

   Thanks in advance for any suggestion.

Mario  

[David Sauve]

What about `SearchQuerySet().auto_query('tinta acrilica ambiente externo')`?

It's bizarre that you're not getting any results as the latest code on Github for the Xapian backend passes all tests.  IT doesn't explicitly make use of AutoQuery in any test, but does call it indirectly through the `SearchQuerySet.auto_query` method.

Perhaps step in to https://github.com/toastdriven/django-haystack/blob/master/haystack/inputs.py#L85 to see what's going on?

Just as a sanity check, you have ran the `rebuild_index` command prior to searching, correct?

You might also try inspecting the Xapian database itself directly via the command line tools: delve and quest.  This doc explains further: http://xapian.org/docs/admin_notes.html#inspecting-a-database

To view this discussion on the web visit https://groups.google.com/d/msg/django-haystack/-/OUNE9eXEoT0J.
To post to this group, send email to django-...@googlegroups.com.
To unsubscribe from this group, send email to django-haysta...@googlegroups.com.

[Mario Menezes]

Hi David,

   Yes, I've tried this also:

In [30]: q = 'tinta acrilica ambiente externo'

In [31]: sqs = SearchQuerySet().auto_query(q)

In [32]: sqs
Out[32]: []


   And yes, I've rebuilt the index database.

   I think the problem here is that the distance between these words in the description field is too large and xapian isn't counting them as a match.

   I'm experimenting with spliting the query string if sqs lenght is zero and them seaching again on a subset of the original string. I'll restirct the second search only to two terms. I think this will give me some relevant results.

   I'm also trying to increase the distance (NEAR) between the terms. I'm trying to figure out the corect form of the parameters to Raw to see if this can help in any way.

   Thanks again for your answer and  suggestions.

   Best regards,

Mario Menezes.