edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class ContractionTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.ContractionTokenizer

All Implemented Interfaces:

WordTokenizer, IsCloseable, UsesLogger

public class ContractionTokenizer
extends AbstractWordTokenizer
implements WordTokenizer

Split text containing contraction into separate tokens.

Field Summary

Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

abbreviations, aposTokens, apostropheCanBeQuote, coalesceAsterisks, coalesceHyphens, contractions, contractionsURL, hyphensMatcher, hyphensPattern, logger, preTokenizer

Constructor Summary

Constructors

Constructor and Description
ContractionTokenizer() Create a contraction tokenizer.

Method Summary

Methods

Modifier and Type	Method and Description
java.util.List<java.lang.String>	extractWords(java.lang.String text) Break text into word tokens.
static java.lang.String	prepareTextForTokenization(java.lang.String str)

Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

addWordToSentence, findWordOffsets, getLogger, getPreTokenizer, isClosingQuote, isLetterOrSingleQuote, isMultipleHyphens, isSingleOpeningQuote, loadContractions, preprocessToken, setAbbreviations, setAposTokens, setLogger, setPreTokenizer, splitToken

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.WordTokenizer

addWordToSentence, close, findWordOffsets, getPreTokenizer, preprocessToken, setAbbreviations, setAposTokens, setPreTokenizer

Methods inherited from interface edu.northwestern.at.utils.IsCloseable

close

Constructor Detail

ContractionTokenizer

public ContractionTokenizer()

Create a contraction tokenizer.

Method Detail

prepareTextForTokenization

public static java.lang.String prepareTextForTokenization(java.lang.String str)

extractWords

public java.util.List<java.lang.String> extractWords(java.lang.String text)

Break text into word tokens.

Specified by:

extractWords in interface WordTokenizer

Specified by:

extractWords in class AbstractWordTokenizer

Parameters:

text - Text to break into word tokens.

Returns:

List of word tokens.

Word tokens may be words, numbers, punctuation, etc.