Facet problems with ForeignKey

[PatrickB]

Hello,

I'm trying to use a faceted search for my new website. I have my
search_indexes.py setup as seen in 1. This should get the fields
'author' and 'post_date' from my database. The template used for the
text field can be seen at 2. I read the tutorial on faceted search on
the haystack site, and copied the author facet from there. The author
facet works.

Then I decided that I also wanted to facet on the category and tags
which I give to the articles. I modified my search_indexes.py for
this, and got a error saying:
> TypeError at /search/
> unhashable type: 'list'
Alright, so a list cannot be hashed, so there must be something wrong
in the prepare functions I use to populate the 'tag_exact' and
'category_exact' field. But what am I doing wrong?

I got these prepare function from somewhere on the net (I believe it
was this group, but I'm not sure). I have put my search urls.py
online, this can be seen at 3. 4 is a link to my website, where I am
having this problem.

I don't know if it matters, but the search backend is Xapian, and the
xapian backend was just updated from GITHub.

Does anyone have an idea what is going wrong here?

1. search_indexes.py: http://metadirc.nl/MetaBin/show/385
2. http://metadirc.nl/MetaBin/show/388
3. urls.py: http://metadirc.nl/MetaBin/show/387
4. site: http://patrickbregman.eu/search/

[David Sauve]

Hi,

Judging from the code supplied, I believe the issue is that you're
trying to use a list to facet (prepare_category

> --
>
> You received this message because you are subscribed to the Google
> Groups "django-haystack" group.
> To post to this group, send email to django-...@googlegroups.com.
> To unsubscribe from this group, send email to django-haysta...@googlegroups.com
> .
> For more options, visit this group at http://groups.google.com/group/django-haystack?hl=en
> .
>
>

[dns...@gmail.com]

Oops. Hit send before I was done typing. Anyways, here's what I meant to say.

Judging from the code supplied, I believe the issue is that you're trying to use a list to facet (prepare_category_exact, prepare_tag_exact). I think what you mean to do here is:

def prepare_tag_exact(self, object):
return ' '.join([v for v in object.tags.all())

or you may want to instead use a MultiValueField.

[PatrickB]

Thanks, now I got rid of the error message. Next problem is that
multiple categories appear in 1 line. This is visible on my site (see
link 1). I have a category 'Algemeen', a category 'Latijn' and a
category 'Spam', which are all put in 1 facet.

I hope this is fixed when I use MultiValue fields. I tried this, and
modified my search_indexes.py as seen in 2. When I try to do a ./
manage.py rebuild_index, I get a traceback saying '# TypeError:
'ManyRelatedManager' object is not iterable '. I guess I am doing
something wrong, but what?

The complete traceback can be seen in 3.

1. Wrong facets: http://patrickbregman.eu/search/?q=lorem&selected_facets=
2. New search_indexes.py: http://metadirc.nl/MetaBin/show/390
3. Traceback: http://metadirc.nl/MetaBin/show/389

On Nov 28, 1:57 pm, dnsa...@gmail.com wrote:
> Oops. Hit send before I was done typing. Anyways, here's what I meant to  
> say.
>
> Judging from the code supplied, I believe the issue is that you're trying  
> to use a list to facet (prepare_category_exact, prepare_tag_exact). I think  
> what you mean to do here is:
>
> def prepare_tag_exact(self, object):
> return ' '.join([v for v in object.tags.all())
>
> or you may want to instead use a MultiValueField.
>

> On , David Sauve <dnsa...@gmail.com> wrote:
>
> > Hi,
> > Judging from the code supplied, I believe the issue is that you're
> > trying to use a list to facet (prepare_category

> > On 2009-11-28, at 7:04 AM, PatrickB patrick.breg...@gmail.com> wrote:
> > > Hello,
>
> > > I'm trying to use a faceted search for my new website. I have my
> > > search_indexes.py setup as seen in 1. This should get the fields
> > > 'author' and 'post_date' from my database. The template used for the
> > > text field can be seen at 2. I read the tutorial on faceted search on
> > > the haystack site, and copied the author facet from there. The author
> > > facet works.
>
> > > Then I decided that I also wanted to facet on the category and tags
> > > which I give to the articles. I modified my search_indexes.py for
> > > this, and got a error saying:
> > >> TypeError at /search/
> > >> unhashable type: 'list'
> > > Alright, so a list cannot be hashed, so there must be something wrong
> > > in the prepare functions I use to populate the 'tag_exact' and
> > > 'category_exact' field. But what am I doing wrong?
>
> > > I got these prepare function from somewhere on the net (I believe it
> > > was this group, but I'm not sure). I have put my search urls.py
> > > online, this can be seen at 3. 4 is a link to my website, where I am
> > > having this problem.
>
> > > I don't know if it matters, but the search backend is Xapian, and the
> > > xapian backend was just updated from GITHub.
>
> > > Does anyone have an idea what is going wrong here?
>
> > > 1. search_indexes.py:http://metadirc.nl/MetaBin/show/385

> > > 2.http://metadirc.nl/MetaBin/show/388

[David Sauve]

Sorry if my response wasn't clear initially, it was early morning here.

Using the ' '.join() trick will put all of the tags/categories in one long string and so you wouldn't be able to facet properly. In this case, what you want (I believe) is to return the category/tag list and use a MultiValueField.  Basically like you're first search_indexes was setup, but instead of a CharField, a MultiValueField.  Hope that clears it up.

[PatrickB]

It was clear, I just didn't quite get the join trick :)

But I fixed it with a MultiValue field, the trick that I didn't get
was that a MultiValueField didn't have a model_attr. With a little
digging through this group and a bit of documentation, I figured it
out.

Documentation: http://haystacksearch.org/docs/searchindex_api.html#prepare-foo-self-object
Final search_indexes.py: http://metadirc.nl/MetaBin/show/391

The new site is not yet visible on http://patrickbregman.eu/, but it
will be shortly. Thanks a lot!

And I must say, django-haystack works very nice and easily! Keep up
the good work!

On Nov 28, 5:04 pm, David Sauve <dnsa...@gmail.com> wrote:
> Sorry if my response wasn't clear initially, it was early morning here.
>
> Using the ' '.join() trick will put all of the tags/categories in one long
> string and so you wouldn't be able to facet properly. In this case, what you
> want (I believe) is to return the category/tag list and use a
> MultiValueField.  Basically like you're first search_indexes was setup, but
> instead of a CharField, a MultiValueField.  Hope that clears it up.
>

> > > > django-haysta...@googlegroups.com<django-haystack%2Bunsu...@googlegroups.com>

> > > > > .
> > > > > For more options, visit this group at
> > > >http://groups.google.com/group/django-haystack?hl=en
> > > > > .
>
> > --
>
> > You received this message because you are subscribed to the Google Groups
> > "django-haystack" group.
> > To post to this group, send email to django-...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > django-haysta...@googlegroups.com<django-haystack%2Bunsu...@googlegroups.com>

[PatrickB]

Hmm... Almost there. I all facets show up now, but it is impossible to
facet on either 'Categorie' or 'Tag'... This results in the message
'No results found'. I guess this is some kind of problem in my
template... I uploaded my template to 1, but since this comes directly
from the faceting tutorial (2).

Strangest thing is, it works when the thing I facet on (be it a
category, be it a tag) has a space in the name, then it works. This is
visible when going to 3. Faceting on either 'To Aru Kagaku no Railgun'
or 'To Aru Majutsu no Index' works, but when faceting on 'Index' or
'Railgun', it gives the message that no results could be found.

In the console I nicely get all facets (author_exact, tag_exact and
category_exact), see 4.

What is going on here, and how do I fix this?
If you need more information or a certain source file, please ask.

1. http://metadirc.nl/MetaBin/show/392
2. http://haystacksearch.org/docs/faceting.html
3. http://patrickbregman.eu/search/?q=railgun&selected_facets=
4. http://metadirc.nl/MetaBin/show/393

On Nov 28, 5:25 pm, PatrickB <patrick.breg...@gmail.com> wrote:
> It was clear, I just didn't quite get the join trick :)
>
> But I fixed it with a MultiValue field, the trick that I didn't get
> was that a MultiValueField didn't have a model_attr. With a little
> digging through this group and a bit of documentation, I figured it
> out.
>

> Documentation:http://haystacksearch.org/docs/searchindex_api.html#prepare-foo-self-...
> Final search_indexes.py:http://metadirc.nl/MetaBin/show/391
>
> The new site is not yet visible onhttp://patrickbregman.eu/, but it

[PatrickB]

Alright, I took a look at haystack/forms.py at line 97, and noticed
something strange, namely:
- The non-cleared version has some content, which seems like the
correct result
- The cleared version is... empty :O

I "fixed" this, like in 1. But... Faceting doesn't work at all after
this "fix", so I guess this is wrong :P
What is happening here? And why do facets that HAVE spaces in them
work, while facets WITHOUT spaces fail?

1. http://metadirc.nl/MetaBin/show/394

On Nov 28, 7:09 pm, PatrickB <patrick.breg...@gmail.com> wrote:
> Hmm... Almost there. I all facets show up now, but it is impossible to
> facet on either 'Categorie' or 'Tag'... This results in the message
> 'No results found'. I guess this is some kind of problem in my
> template... I uploaded my template to 1, but since this comes directly
> from the faceting tutorial (2).
>
> Strangest thing is, it works when the thing I facet on (be it a
> category, be it a tag) has a space in the name, then it works. This is
> visible when going to 3. Faceting on either 'To Aru Kagaku no Railgun'
> or 'To Aru Majutsu no Index' works, but when faceting on 'Index' or
> 'Railgun', it gives the message that no results could be found.
>
> In the console I nicely get all facets (author_exact, tag_exact and
> category_exact), see 4.
>
> What is going on here, and how do I fix this?
> If you need more information or a certain source file, please ask.
>
> 1.http://metadirc.nl/MetaBin/show/392
> 2.http://haystacksearch.org/docs/faceting.html
> 3.http://patrickbregman.eu/search/?q=railgun&selected_facets=

> 4.http://metadirc.nl/MetaBin/show/393

[PatrickB]

Well, I hate to kick topics/threads, but does anyone have an idea why
the faceting doesn't work?
With spaces it returns *something* but without it doesn't return
anything. The strange thing is, that even when it returns something,
it appears that the facets are not actually used.

For example:
- Go to http://patrickbregman.eu/search/?q=railgun
- Select a facet with spaces
- - Nothing changes, all search results stay exactly the same
- Go back and select a facet without spaces
- - There are no search results

I've setup a Mercurial repository which is free to browse for all at
http://patrickbregman.eu/hg_pb_eu/pb_eu/. It doesn't have any
revisions, because I normally use another repository.
You are allowed to take this code and use it for your own site.

Does anyone has an idea what is going wrong here?

[Daniel Lindsley]

Patrick,


I think I can explain why but ran out of time last night to
respond. I'll try to put together a response tonight.


Daniel

> --
>
> You received this message because you are subscribed to the Google Groups "django-haystack" group.
> To post to this group, send email to django-...@googlegroups.com.

> To unsubscribe from this group, send email to django-haysta...@googlegroups.com.

[Daniel Lindsley]

Patrick,


Sorry for the delay. The reason why facets with spaces (but not
individual words) works is because you're doing ``indexed=False``.
This causes the backend to store the exact string you pass it and, in
turn, do exact matches when faceting. If you want the individual words
to be faceted, you'll need to tokenize (whitespace split is the
easiest) them in your ``prepare_FOO`` methods. So for example:


===========
class PostIndex(indexes.SearchIndex):
# Indexed stuff
text = indexes.CharField(document=True, use_template=True)
author = indexes.CharField(model_attr='author')
post_date = indexes.DateTimeField(model_attr='post_date')

# Non-indexed stuff, used for facets
author_exact = indexes.CharField(model_attr='author', indexed=False)
category_exact = indexes.MultiValueField(indexed=False)
tag_exact = indexes.MultiValueField(indexed=False)

def prepare_category_exact(self, obj):
cat_list = []

for category in obj.categories.all():
cat_words = category.split()
cat_list.extend(cat_words)

return cat_list
===========

This ought to split things down and make individual words from the
category work. If you want the full category as well, simply append
that on to the list in addition to the words. Hope this helps,


Daniel

[PatrickB]

The problem that I can't facet is now replaced by an error., namely

TypeError at /search/
unhashable type: 'list'

Full traceback is at 1. I also put the thing online, which is at 2.
This looks like the same error that I got before, but I can't figure
out why this is happening now. I tried the things I tried before (see
above), but those didn't help.

Any ideas?

1. http://metadirc.nl/MetaBin/show/397
2. http://patrickbregman.eu/

I updated the mercurial repoitory mentioned above.

> On Tue, Dec 1, 2009 at 12:52 PM, Daniel Lindsley <polarc...@gmail.com> wrote:
> > Patrick,
>
> >   I think I can explain why but ran out of time last night to
> > respond. I'll try to put together a response tonight.
>
> > Daniel
>

> > On Tue, Dec 1, 2009 at 12:32 PM, PatrickB <patrick.breg...@gmail.com> wrote:
> >> Well, I hate to kick topics/threads, but does anyone have an idea why
> >> the faceting doesn't work?
> >> With spaces it returns *something* but without it doesn't return
> >> anything. The strange thing is, that even when it returns something,
> >> it appears that the facets are not actually used.
>
> >> For example:

> >> - Go tohttp://patrickbregman.eu/search/?q=railgun

[Daniel Lindsley]

Patrick,


Sorry, I'm not familiar enough with the Xapian backend and how it
does faceting. I'm guess that it's some sort of data issue (maybe
something getting stored wrong) but I'll have to defer on this to
David (notanumber).


Daniel

[David Sauve]

Hi Patrick,

Can you confirm that the xapian_backend is running the latest code from GitHub?

As an FYI, I'm planning on making fairly large change to the code base in the next day or two with merging of the `next` branch into `master`.  I'm hoping this will fix a lot of the more obscure issues people have been experiencing.  You're welcome to give the `next` branch a try if you'd like.

David

[PatrickB]

Hi David,

I am currently using the latest GIT code from the master branch. I
would like to try the 'next' branch, but I can't figure out howto
switch to this branch using GIT. Google isn't too helpfull with this
either. Any tips on how I can switch the branch from master/HEAD to
'next'?

On Dec 5, 4:50 pm, David Sauve <dnsa...@gmail.com> wrote:
> Hi Patrick,
>
> Can you confirm that the xapian_backend is running the latest code from
> GitHub?
>
> As an FYI, I'm planning on making fairly large change to the code base in
> the next day or two with merging of the `next` branch into `master`.  I'm
> hoping this will fix a lot of the more obscure issues people have been
> experiencing.  You're welcome to give the `next` branch a try if you'd like.
>
> David
>

> On Fri, Dec 4, 2009 at 10:29 PM, Daniel Lindsley <polarc...@gmail.com>wrote:
>
> > Patrick,
>
> >   Sorry, I'm not familiar enough with the Xapian backend and how it
> > does faceting. I'm guess that it's some sort of data issue (maybe
> > something getting stored wrong) but I'll have to defer on this to
> > David (notanumber).
>
> > Daniel
>

> > On Wed, Dec 2, 2009 at 11:49 AM, PatrickB <patrick.breg...@gmail.com>

> > wrote:
> > > The problem that I can't facet is now replaced by an error., namely
> > >  TypeError at /search/
> > >  unhashable type: 'list'
> > > Full traceback is at 1. I also put the thing online, which is at 2.
> > > This looks like the same error that I got before, but I can't figure
> > > out why this is happening now. I tried the things I tried before (see
> > > above), but those didn't help.
>
> > > Any ideas?
>
> > > 1.http://metadirc.nl/MetaBin/show/397

> > > 2.http://patrickbregman.eu/

> > django-haysta...@googlegroups.com<django-haystack%2Bunsu...@googlegroups.com>
> > .

> > >> >> For more options, visit this group athttp://
> > groups.google.com/group/django-haystack?hl=en.
>

> > > --
>
> > > You received this message because you are subscribed to the Google Groups
> > "django-haystack" group.
> > > To post to this group, send email to django-...@googlegroups.com.
> > > To unsubscribe from this group, send email to

> > django-haysta...@googlegroups.com<django-haystack%2Bunsu...@googlegroups.com>
> > .

> > > For more options, visit this group at
> >http://groups.google.com/group/django-haystack?hl=en.
>
> > --
>
> > You received this message because you are subscribed to the Google Groups
> > "django-haystack" group.
> > To post to this group, send email to django-...@googlegroups.com.
> > To unsubscribe from this group, send email to

> > django-haysta...@googlegroups.com<django-haystack%2Bunsu...@googlegroups.com>
> > .

[dns...@gmail.com]

Ok, I think I see what the issue is. `tag_exact` and `author_exact` are both lists of lists. This is why you're getting the hash error.

Here's what the traceback is showing for `tag_exact`: 'tag_exact': [[u'Railgun'], [u'Index'], [u'To', u'Aru', u'Kagaku', u'no', u'Railgun'], [u'To', u'Aru', u'Majutsu', u'no', u'Index']]

Is this what you intended? I'll need to confirm with Daniel whether this is something Haystack is supposed to support. If it is, I'll see about extending the Xapian code to support it. In the meantime, would it be possible to collapse the tags down into a single list? For example, make `[u'To', u'Aru', u'Kagaku', u'no', u'Railgun']` 'To Aru Kagaku no Railgun'.

On Dec 5, 2009 8:21pm, dns...@gmail.com wrote:
> Thanks, Patrick. I'll take a look at this and let you know if I can find the source of your issue.
>
> On , Patrick Bregman patrick...@gmail.com> wrote:
> > I've put the code onlie at a mercurial repository. This repository is located at http://patrickbregman.eu/hg_pb_eu/pb_eu/. I just pushed the last changes to there, so it should be completely up to date. I've also enabled pulling, so you can use a mercurial client if you like.
> >
> >
> > On Sun, Dec 6, 2009 at 00:19, dns...@gmail.com> wrote:
> >
> > Are the indexes still online somewhere that I could have a peak at?
> >
> >
> > On , Patrick Bregman patrick...@gmail.com> wrote:
> >
> > > I tried, and I couldn't pull it from GIT, so I downloaded it from the GITHub page. I just saw that I could switch branches there :)
> > > But this code doesn't work either. It keeps complaining about the same error, TypeError at /search/ unhashable type: 'list'.
> >
> > >
> > > And I don't mind running unstable code at all if it fixes a problem I'm having.
> > >
> > > On Sun, Dec 6, 2009 at 00:04, dns...@gmail.com> wrote:
> >
> > >
> > > Sure thing, Patrick. You'd probably want to checkout and track a remote branch: git checkout -b next notanumber/next
> > >
> > >
> > > This depends on how you got the code to begin with, though. This link should be a good start: http://github.com/guides/showing-and-tracking-remote-branches
> >
> > >
> > >
> > > Please, keep in mind that this is experimental code, although it has pretty good test coverage. If you're not bothered by running potentially unstable code, I'd love to hear how it works for you.

> > > > To unsubscribe from this group, send email to django-haysta...@googlegroups.com.

[Patrick Bregman]

Strange... I didn't intend to make it a list of lists. Let's see how I can fix that.

Thanks!

[Patrick Bregman]

I tried a few things, but it all results in the same error.
How did you get this detailed traceback? I couldn't find anything like it in the standard Django errorpage.

[dns...@gmail.com]

Hi Patrick,

I simply visited this link: http://patrickbregman.eu/search/ and expanded the "Local Vars" on the second to last item in the traceback. You should be able to see the contents of `model_data` here. You can see how this is used in the last item of the traceback by expanding "Local Vars" again. Here, you'll see that `field_value` is a two-dimensional list, same as in `model_data`.

Hope that helps.

David

&gt%

[PatrickB]

To find out if this was a Xapian specific issue, I decided to try this
with Solr. Unfortunately, I can't get Solr to facet, so I couldn't
verify this. Did you find out anything yet?

Anyway, I'll try to look into it this weekend. Didn't have any time to
spare this week :(

On Dec 6, 10:51 pm, dnsa...@gmail.com wrote:
> Hi Patrick,
>

> I simply visited this link:http://patrickbregman.eu/search/and expanded  

> the "Local Vars" on the second to last item in the traceback. You should be  
> able to see the contents of `model_data` here. You can see how this is used  
> in the last item of the traceback by expanding "Local Vars" again. Here,  
> you'll see that `field_value` is a two-dimensional list, same as in  
> `model_data`.
>
> Hope that helps.
>
> David
>

> On Dec 6, 2009 3:37pm, Patrick Bregman <patrick.breg...@gmail.com> wrote:
>
> > I tried a few things, but it all results in the same error.
> > How did you get this detailed traceback? I couldn't find anything like it  
> > in the standard Django errorpage.

> > On Sun, Dec 6, 2009 at 20:02, Patrick Bregman patrick.breg...@gmail.com>  

> > wrote:
> > Strange... I didn't intend to make it a list of lists. Let's see how I  
> > can fix that.
> > Thanks!

> > On Sun, Dec 6, 2009 at 19:56, dnsa...@gmail.com> wrote:
> > Ok, I think I see what the issue is. `tag_exact` and `author_exact` are  
> > both lists of lists. This is why you're getting the hash error.
> > Here's what the traceback is showing for `tag_exact`: 'tag_exact':  
> > [[u'Railgun'], [u'Index'], [u'To', u'Aru', u'Kagaku', u'no', u'Railgun'],  
> > [u'To', u'Aru', u'Majutsu', u'no', u'Index']]
> > Is this what you intended? I'll need to confirm with Daniel whether this  
> > is something Haystack is supposed to support. If it is, I'll see about  
> > extending the Xapian code to support it. In the meantime, would it be  
> > possible to collapse the tags down into a single list? For example, make  
> > `[u'To', u'Aru', u'Kagaku', u'no', u'Railgun']` 'To Aru Kagaku no  
> > Railgun'.

> > On Dec 5, 2009 8:21pm, dnsa...@gmail.com wrote:
> > > Thanks, Patrick. I'll take a look at this and let you know if I can  
> > find the source of your issue.
>

> > > On , Patrick Bregman patrick.breg...@gmail.com> wrote:
> > > > I've put the code onlie at a mercurial repository. This repository is  

> > located athttp://patrickbregman.eu/hg_pb_eu/pb_eu/. I just pushed the  

> > last changes to there, so it should be completely up to date. I've also  
> > enabled pulling, so you can use a mercurial client if you like.
>

> > > > On Sun, Dec 6, 2009 at 00:19, dnsa...@gmail.com> wrote:
>
> > > > Are the indexes still online somewhere that I could have a peak at?
>

> > > > On , Patrick Bregman patrick.breg...@gmail.com> wrote:
>
> > > > > I tried, and I couldn't pull it from GIT, so I downloaded it from  
> > the GITHub page. I just saw that I could switch branches there :)
> > > > > But this code doesn't work either. It keeps complaining about the  
> > same error, TypeError at /search/ unhashable type: 'list'.
>
> > > > > And I don't mind running unstable code at all if it fixes a problem  
> > I'm having.
>

> > > > > On Sun, Dec 6, 2009 at 00:04, dnsa...@gmail.com> wrote:
>
> > > > > Sure thing, Patrick. You'd probably want to checkout and track a  
> > remote branch: git checkout -b next notanumber/next
>
> > > > > This depends on how you got the code to begin with, though. This  
> > link should be a good start:  
> >http://github.com/guides/showing-and-tracking-remote-branches
>
> > > > > Please, keep in mind that this is experimental code, although it  
> > has pretty good test coverage. If you're not bothered by running  
> > potentially unstable code, I'd love to hear how it works for you.
>

> ...
>
> read more »

[PatrickB]

And before I forget, I did update Haystack and xapian-backend to the
last versions, and I'm using Xapian version 1.0.17.

Besides, is there any kind of a Xapian (web) interface so I can try to
enter queries by hand?