public class NoopPreTokenizer extends AbstractPreTokenizer implements PreTokenizer
alwaysSeparators, alwaysSeparatorsReplacer, asterisks, commaSeparator, commaSeparatorReplacer, hyphens, logger, periods
Constructor and Description |
---|
NoopPreTokenizer()
Create a "no-op" preTokenizer.
|
Modifier and Type | Method and Description |
---|---|
java.lang.String |
pretokenize(java.lang.String line)
Prepares a string for tokenization.
|
getLogger, setLogger
close
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
close
public java.lang.String pretokenize(java.lang.String line)
pretokenize
in interface PreTokenizer
pretokenize
in class AbstractPreTokenizer
line
- The string to prepare for tokenization.