public class DefaultPostTokenizer extends AbstractPostTokenizer implements PostTokenizer
logger
Constructor and Description |
---|
DefaultPostTokenizer()
Create a default postTokenizer.
|
Modifier and Type | Method and Description |
---|---|
java.lang.String[] |
postTokenize(java.lang.String token)
Process a token after tokenization.
|
getLogger, setLogger
close
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
close
public DefaultPostTokenizer()
public java.lang.String[] postTokenize(java.lang.String token)
postTokenize
in interface PostTokenizer
postTokenize
in class AbstractPostTokenizer
token
- The token to process after tokenization.The minimally processed token is typically results in an original spelling.
The maximally processed token typically results in a partially or completely standardized spelling.
These may be identical.