edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class NoopWordTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.NoopWordTokenizer

All Implemented Interfaces:

WordTokenizer, IsCloseable, UsesLogger

public class NoopWordTokenizer
extends AbstractWordTokenizer
implements WordTokenizer

Word tokenizer which leaves original text untokenized.

Field Summary

Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

abbreviations, aposTokens, apostropheCanBeQuote, coalesceAsterisks, coalesceHyphens, contractions, contractionsURL, hyphensMatcher, hyphensPattern, logger, preTokenizer

Constructor Summary

Constructors

Constructor and Description
NoopWordTokenizer() Create a word tokenizer that leaves the text untokenized.

Method Summary

Methods

Modifier and Type	Method and Description
java.util.List<java.lang.String>	extractWords(java.lang.String text) Break text into word tokens.

Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

addWordToSentence, findWordOffsets, getLogger, getPreTokenizer, isClosingQuote, isLetterOrSingleQuote, isMultipleHyphens, isSingleOpeningQuote, loadContractions, preprocessToken, setAbbreviations, setAposTokens, setLogger, setPreTokenizer, splitToken

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.WordTokenizer

addWordToSentence, close, findWordOffsets, getPreTokenizer, preprocessToken, setAbbreviations, setAposTokens, setPreTokenizer

Methods inherited from interface edu.northwestern.at.utils.IsCloseable

close

Constructor Detail

NoopWordTokenizer

public NoopWordTokenizer()

Create a word tokenizer that leaves the text untokenized.

Method Detail

extractWords

public java.util.List<java.lang.String> extractWords(java.lang.String text)

Break text into word tokens.

Specified by:

extractWords in interface WordTokenizer

Specified by:

extractWords in class AbstractWordTokenizer

Parameters:

text - Text to break into word tokens.

Returns:

Input text unchanged as a single list item.