Search Slow
wavesound
somewhat slow. Is there an effective way to speed these searches up?
Are there things I should do in the MySQL to "tune" it for Streeme?
Rich
definitely something I'm going to be working on using a real indexer
like Lucene. Right now searches use a really inefficient LIKE/ILIKE
command over your artist/songs/albums tables, which means dumb, full
table scans every time you keyup. A couple fixes I can think of would
be to put a timeout on the keyups so it's not searching so often,
adding an indexer to keep a separate table of word matches and logic
handling and tightening up the indexing to make everything join
faster. Just out of curiosity, how big is your library, what ordering
(eg. Newest/Artist Name/Length) do you like to use, and what kind of
hardware are you using to deploy Streeme? I could use the info to help
benchmark and track improvement. If you feel like adding a bug to the
bug tracker - please do.
-Rich
wavesound
I currently have ~65,000 tracks consuming around 500GB of space. I
deployed the server on VM in our datacenter space with 2GB of RAM on a
Xeon 5160 Processor. I usually use the default ordering which I
believe is "Modified."
I just noticed some Lucene check-ins and I got excited.
Rich
wasn't happy with the performance of the Zend PHP implementation of
Lucene once the library got over about 10,000 tracks, so I'm going to
try a Java version of SOLR/Lucene and see if it's much better. I'll do
my stress tests with a 65k track library instead of 20k. I'll let you
know when there'a branch to test out.
-Rich
Rich
the integration with the media scanner before I integrate it into
trunk - here are the steps to try it out immediately:
svn switch ^/branches/rh-solr-0.5.0
cd path/to/streeme/plugins/
git clone git://github.com/rande/sfSolrPlugin.git
open path/to/streeme/config/projectConfiguration.class.php and
uncomment the sfsolrplugin line
open path/to/streeme/apps/client/config/app.yml.template and copy the
new indexer configs at the bottom to your app.yml file.
type: ./symfony cc
./symfony solr start
./symfony media-index --env=prod
Now go into the desktop or mobile app and try searching. let me know
how it goes. I'm also vaguely interested in making a mysql fulltext
search as well - I'll let you know when that one's ready too.
One other thing to note is that Solr starts on port 8983, has a web
interface and is not password protected. You should make sure port
8983 is not accessible through your router.
That's it - hope it significantly boosts the speed of searching - it
sure did on my end.
-Rich
Rich
testing for speed. It searches a little differently by casting wider
net across all metadata tokens at once. You'll typically get a lot
more results varied results as you add words to the search.
To use this indexer (most of the same steps as the solr version
above):
svn switch ^/branches/rh-solr-0.5.0
open path/to/streeme/apps/client/config/app.yml.template and copy the
new indexer configs at the bottom to your app.yml file.
change class: StreemeIndexerSolr to StreemeIndexerMysql
type: ./symfony cc
./symfony mysql initialize (this will add the indexer table
and fulltext index to your existing mysql project)
./symfony media-index --env=prod
again, go to the desktop or mobile app and try searching
You may want to capture the speeds of your searches using firebug or
chrome web inspector. That's probably the easiest way to benchmark the
two fulltext implementations.
-Rich
wavesound
Thank you! Thank you! Thank you!
I have not tested Solr yet, however I wanted to let you know that
MySQL full-text search is such an improvement. Its almost like
iTunes. :)
wavesound
search indexers.
Solr seems to return results more like what I was accustomed to with
Streeme before adding these indexers. MySQL also performs well but
returns more irrelevant results (results that match a single word in
the search string as opposed to all of the words). I think I prefer
the implicit "and" treatment in Streeme's original search feature.
I didn't benchmark the results yet, but they both seem incredibly
fast.
I added the following to my iptables for solr:
sudo iptables -A INPUT -p tcp -s 127.0.0.1 --dport 8983 -j ACCEPT
sudo iptables -A INPUT -p tcp --dport 8983 -j DROP
wavesound
search behavior to return exact phrases by modifying the query in /
path/to/streeme/apps/client/lib/indexer/StreemeIndexerMysql.class.php:
I inserted: $keywords = '"'.$keywords.'"';
above: $keywords = sprintf('*%s*', $keywords);
This search results are now consistent with results presented before
the indexer was enabled.
Rich
making the mysql one a lot better than it is at the moment. I can use
the fulltext stuff to really reduce the size of the query plan, so I
should probably separate the fields into artists/albums/genres/songs
then use LIKE after the bulk of the results have been targeted. That
should make it so you can do exact matches on all the named elements
of a song like iTunes does. Solr looks like it can do a fairly similar
search, so I'll toy with them a bit more before I release the code to
trunk.
I think I prefer the search style like this:
j
---
would show:
Artist: James Zabiela
or
Album: Jasmine
or
Song Name: Jar of Hearts
or
Genre: Jams
or
Song Name: Jekyll of the desert
---
ja
would show:
Artist: James Zabiela
or
Album: Jasmine
or
Song Name: Jar of Hearts
or
Genre: Jams
BUT EXCLUDE
Song Name: Jekyll of the desert
---
jam
would show:
Artist: James Zabiela
or
Genre: Jams
BUT EXCLUDE
Album: Jasmine
and
Song Name: Jar of Hearts
further letters would continue this left hand pattern matching until
the user stopped typing.
Does that sound right to you? I wonder if it might be good to even
have some really fancy matching from things like SOLR/lucene, where
you can do logical, pattern and fuzzy matches. Seems like it's missing
from iTunes completely
-Rich
wavesound
I agree, I think your suggestion is better. I've never written a
search query to in doing those tweaks, I learned a lot about search
complexity.
I look forward to seeing what you come up with!