> ditch my idea of deriving from Tokenizer because line number information is
> not passed to Tokenizer.getLineTokens(), and I absolutely need line number
> information to know which part of the document I'm processing.
for things that need row number i'd suggest to look into deriving from
background_tokenizer since it manages token cache and there is one
background_tokenizer for each session.