Sensei term vector options not being respected? (Can only get back term-frequencies)

[Jack Dunham]

Hi there -- I'm using sensei 1.6.0, and having some difficulty w/ term vectors. I want to get back term-positions for tokens in a text-column, so I've set the options in schema.xml, like so:

<column name="taggedText" type="text" index="ANALYZED" store="YES" termvector="WITH_POSITIONS"/>

However, the term-vectors I'm getting back from my query are still term-frequency pairs, instead of the desired term-positions. Is there some other ways of specifying the org.apache.lucene.document.Field.TermVector.WITH_POSITIONS option to sensei?

[John Wang]

Hi Jack:

    Can you create a ticket at: https://senseidb.atlassian.net/secure/Dashboard.jspa

Thanks

John

--
You received this message because you are subscribed to the Google Groups "Sensei" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sensei-searc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Yonghui Zhao]

Hi Jack，

How do you get the term vector result.

Can you give me the detail query you used？

[Jack Dunham]

Hi there -- Sorry for the delay in responding (and thanks, btw!). Here's the json query I'm using.

{

    "query": {

        "query_string": {

            "query": "taggedText:hello"

        }

    },

    "from": 0,

    "size": 10,

    "explain": false,

    "fetchStored": true,

    "termVectors": ["taggedText"]

[Jack Dunham]

FYI -- Created issue  SENSEI-306

On Tuesday, November 19, 2013 8:24:45 AM UTC-8, yozhao wrote:

[Yonghui Zhao]

Hi Jack,

Thank you for report this bug.

Sensei doesn't return offsets and positions info, now I have done a fix in sensei 2.0.1-SNAPSHOT and bobo 4.0.1-SNAPSHOT.

Before the fix sensei only return term and frequency  in Map<String, List<TermFrequency>>.

Now sensei will return termvector in Map<String, List<FieldTerm>>, while FieldTerm contains:

  private String term;
  private Integer freq;
  private List<Integer> positions;
  private List<Integer> startOffsets;
  private List<Integer> endOffsets;

If index option is termvector="YES" , then positions, startOffsets and endOffsets are null.
If index option is termvector="WITH_POSITIONS" , then positions have real value while startOffsets and endOffsets are all -1.
If index option is termvector="WITH_OFFSETS" , then positions are all -1 while startOffsets and endOffsets are real value.
If index option is termvector="WITH_POSITIONS_OFFSETS" , then positions startOffsets and endOffsets are all real value.

Here is the changes:

Bobo:
https://github.com/yozhao/bobo/commit/aa4a1fbcffe375533e5f601b441896a7695febf3

Sensei:
https://github.com/yozhao/sensei/commit/efb6daa1cf2c20b3f8bd23d31e7d8db923a1f61d
https://github.com/yozhao/sensei/commit/17c9d6e90a0c36fb526fb2fd1f5ded303c1350df

Which version are you using now, I can prepare a fixed version for you.

2013/11/21 Jack Dunham <jack....@gmail.com>