Introduction

[Sean Gallagher]

Hello all!

 I'm Sean Gallagher, and I develop Watsonsim, a class project at UNC Charlotte targeting Jeopardy questions. We're on Github and we have some status updates. As it stands right now, we have 35.5% accuracy, and about 52% recall, but it's been changing regularly (i.e. our accuracy yesterday was 31.2%). Our system is a bit different, and tries to be conceptually lean; it does not use UIMA, but it does use Bing. It's main drawbacks right now are that it's slow (it takes about a minute to answer a question), and it is too difficult to setup; in particular we have trouble distributing the requisite data because of the bandwidth. We have made inroads with the speed; only a few weeks ago it averaged around 150 sec/question. There are some oddities though; in our experience, logistic regression is not nearly as effective as RBF SVM.

I do have some questions though. According to my advisor, when IBM was using Indri, it's query latency was within a few percent of Lucene's. But for all the tweaking I have done I cannot seem to make that happen. Indri queries once took 15 seconds per question on a ~28 million record index, and I have been able to get it to 3 seconds per query, but in about the same amount of time Lucene performs about 60 to 120 queries. Has anyone else experienced this?

[Jack Park]

Hi Sean!

I built YodaQA and have building Watsonsim on the agenda. Many thanks for your updates. I do not have any answers for you, but do have a generalized question: I'm interested in a comparison between Lucene and Indri; what drives a choice between the two?

Cheers

Jack

ps: please note that you may see bounces on one email that went stale if it wasn't upgraded yet.

--
You received this message because you are subscribed to the Google Groups "qa-oss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qa-oss+un...@googlegroups.com.
To post to this group, send email to qa-...@googlegroups.com.
Visit this group at http://groups.google.com/group/qa-oss.
To view this discussion on the web visit https://groups.google.com/d/msgid/qa-oss/d2173078-3995-4c85-96cf-d61b94619fc2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Leonid Boytsov]

I haven't work a lot with Indri, but I think that simple Indri queries should be blazingly fast. However if you run complex structured queries those can be painfully slow. If this is the case, here is the advice from Indri manual:

Note: if child, parent, or ancestor queries are slow, you may want to be certain to index the specified fields explicitly as an ordinal. This speeds things up at the cost of a minimal amount of disk space. In the example above ("title"), the following would be placed in the build index parameters:

  <field>
    <name>title</name>
    <ordinal>true</ordinal>
  </field>

[Leonid Boytsov]

Another question would be: why do you need Indri at all?

On Thursday, April 16, 2015 at 11:49:54 AM UTC-4, Sean Gallagher wrote:

[Jack Park]

I would like to follow Leonid's question with this suggestion: I think we have an opportunity here to begin to talk in some detail about all aspects of architectures, from indexing, to NLP, to question answering, hypothesis formation, and response ranking, and more.

It is my hope that we can find ways to share ideas and explore options.

Cheers

Jack

--

You received this message because you are subscribed to the Google Groups "qa-oss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qa-oss+un...@googlegroups.com.
To post to this group, send email to qa-...@googlegroups.com.
Visit this group at http://groups.google.com/group/qa-oss.

To view this discussion on the web visit https://groups.google.com/d/msgid/qa-oss/1a4caaf8-1de5-4daa-b6ce-87e18872e4ab%40googlegroups.com.

[Leonid Boytsov]

Sure,
no problem.

[Sean Gallagher]

We don't have very complex queries. It's one #combine (in QE, RQ) with about 5 to 20 terms, having stopwords removed. (Removing them gave a ~5 fold speedup.) There are only two fields as well: docno and text, and we only need to know the docno of a document that we retrieved by text. The text is also rather clean, unformatted plain text, and the documents are paragraph-sized.

And to answer both you and Jack, we use Indri because in our early experiments, it produced measurably better results. We also compared it with Bing. This brings me to the question though, of whether our Lucene query is accurate. We use skip-bigram conjuncts for candidate answer generation, and plain unigram conjuncts for supporting evidence retrieval, because we found the skip-bigram queries improved accuracy but required longer query times. (This was before we started the blog.)

And I too look forward to speaking about all parts of the architecture! For one, I would be interested to know what activity there is with NLP in question answering; I don't know of any OSS parsers for ESG for the candidate answers as suggested in the Watson papers so we are trying to match Prolog rules against Stanford dependencies; we don't have any really convincing results one way or the other yet (it had very little effect). It does work for really specific constructions like "X[, X2] is a Y1 [[,][and][,] a Y2]", were we use it to read lexical type's but we haven't gone much further yet.

- Sean

[Leonid Boytsov]

Hi Sean,

Stopwords can definitely make things much harder. A simple #combine operator shouldn't be especially slow. However, if it is processed in the OR-mode (which I suspect is a case), it is going to be slower than the AND Lucene query that can be efficiently processed using leapfrogging.

I don't know anything about the skipped bigram queries. However, in our experiments there was little difference between Lucene and Indri. So, I suspect that there is something wrong with these skipped bigrams.

You probably need to consider different types of Indri and Lucene queries (e.g., weak AND, OR, etc...), measure their speed and the performance of respective pipelines. Then, we will know better about relative performance of these two engines.