net.sf.eos.analyzer
Interface Tokenizer

All Known Subinterfaces:

DictionaryBasedEntityRecognizer, EntityRecognizer, ResettableTokenizer

All Known Implementing Classes:

AbstractDictionaryBasedEntityRecognizer, CaseTokenFilter, LuceneTokenizerWrapper, ResettableTokenFilter, SentenceTokenizer, SimpleLongestMatchDictionaryBasedEntityRecognizer, StopTokenFilter, SurroundingTokenFilter, TokenFilter, WhitespaceTokenizer

public interface Tokenizer

An implementation splits text data into its Token.

Author:

Sascha Kohlmann

Method Summary

Token

next() The next token or null.

Method Detail

next

Token next()
           throws TokenizerException

The next token or null.

Returns:

the next token or null

Throws:

TokenizerException