Specifying per-field TermProcessor

[Mohamed Yahya]

Is there a way to have per-field term processors in MG4J?

[Sebastiano Vigna]

On 11 Apr 2014, at 3:28 PM, Mohamed Yahya <yahya....@gmail.com> wrote:

> Is there a way to have per-field term processors in MG4J?

Yes, in principle. Every Scan instance (which analyzes a single field) has a separate term processor. However, there is presently no way to set the processor from outside.

The current workaround is to do as many indexing round as your term processors, subsetting the index construction to those field that must be parsed by the specific term processor.

Ciao,

seba

[Mohamed Yahya]

If I understand correctly, the solution you propose is to have an
independent indexing round per field. Is this correct?

> --
> You received this message because you are subscribed to a topic in the Google Groups "MG4J" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/mg4j/TPiko0cKq4A/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to mg4j+uns...@googlegroups.com.
> To post to this group, send email to mg...@googlegroups.com.
> Visit this group at http://groups.google.com/group/mg4j.
> For more options, visit https://groups.google.com/d/optout.



--
Mohamed Yahya

[Sebastiano Vigna]

On 22 Apr 2014, at 4:46 PM, Mohamed Yahya <yahya....@gmail.com> wrote:

> If I understand correctly, the solution you propose is to have an
> independent indexing round per field. Is this correct?

Well, I would rather gather the fields by term processor and index in each round all fields using the same term processor. But, yes, you can have one round per field. How is large the collection?

Ciao,

seba

[Mohamed Yahya]

I see the point about working per term processor. We're talking about
4 million documents, and 8 fields per document.

[Sebastiano Vigna]

On 22 Apr 2014, at 5:40 PM, Mohamed Yahya <yahya....@gmail.com> wrote:

> I see the point about working per term processor. We're talking about
> 4 million documents, and 8 fields per document.

Oh well. You can definitely do several rounds :).

Ciao,

seba

[Mohamed Yahya]

With everything being read and written from the same disk on a single
machine with 20GB of memory available to MG4J, how much time would you
expect indexing something like the text of Wikipedia (20GB after
cleaning, two fields: text, title) to take, running with the options
"--downcase -s 1000000".

[Sebastiano Vigna]

On 22 Apr 2014, at 8:04 PM, Mohamed Yahya <yahya....@gmail.com> wrote:

> With everything being read and written from the same disk on a single
> machine with 20GB of memory available to MG4J, how much time would you
> expect indexing something like the text of Wikipedia (20GB after
> cleaning, two fields: text, title) to take, running with the options
> "--downcase -s 1000000".

I'd say a few hours, but it depends on several factors.

Ciao,

seba