edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class DefaultPreTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPreTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.DefaultPreTokenizer

All Implemented Interfaces:

PreTokenizer, UsesLogger

public class DefaultPreTokenizer
extends AbstractPreTokenizer
implements PreTokenizer

Default pretokenizes which prepares a string for tokenization.

Field Summary

Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPreTokenizer

alwaysSeparators, alwaysSeparatorsReplacer, asterisks, commaSeparator, commaSeparatorReplacer, hyphens, logger, periods

Constructor Summary

Constructors

Constructor and Description
DefaultPreTokenizer() Create a default preTokenizer.

Method Summary

Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractPreTokenizer

getLogger, pretokenize, setLogger

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.PreTokenizer

close, pretokenize

Constructor Detail

DefaultPreTokenizer

public DefaultPreTokenizer()

Create a default preTokenizer.