Performance problem when updating the index

25 views
Skip to first unread message

Salvatore Iovene

unread,
Nov 9, 2016, 1:23:43 AM11/9/16
to django-haystack
Hello,
I use django-haystack 2.4.1 and Solr 4.9.1.

Unfortunately, I'm in the situation in which my search_index.py kept growing and growing, and so did my data, and now doing a full indexing every day takes up to 18 hours! I'm indexing about 200k documents.

The reason is that I have A LOT of "prepare_" functions, and they hit the database.

I have lots of "prepare_" functions for two reasons:

  1. I store numerical data in the index, because I need advanced searches like "find all objects for which property FOO is greater between the integer values X and Y"
  2. I'm probably abusing the search index here, but I'm keeping things like "number of likes, number of comments, number of views" so that I can display them in search results (I should be updating a database column instead?)

The code is here if you want, but it ain't pretty:

Can you suggest a strategy to improve this situation?

Thanks in advance!
Salvatore

Reply all
Reply to author
Forward
0 new messages