Performance problem when updating the index

[Salvatore Iovene]

Hello,

I use django-haystack 2.4.1 and Solr 4.9.1.

Unfortunately, I'm in the situation in which my search_index.py kept growing and growing, and so did my data, and now doing a full indexing every day takes up to 18 hours! I'm indexing about 200k documents.

The reason is that I have A LOT of "prepare_" functions, and they hit the database.

I have lots of "prepare_" functions for two reasons:

1. I store numerical data in the index, because I need advanced searches like "find all objects for which property FOO is greater between the integer values X and Y"
2. I'm probably abusing the search index here, but I'm keeping things like "number of likes, number of comments, number of views" so that I can display them in search results (I should be updating a database column instead?)

The code is here if you want, but it ain't pretty:

https://bitbucket.org/siovene/astrobin/src/e573f788711cff18fe2ecc489a5b28c871949954/astrobin/search_indexes.py?at=master&fileviewer=file-view-default

Can you suggest a strategy to improve this situation?

Thanks in advance!

Salvatore