edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class PennTreebankTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.PennTreebankTokenizer

All Implemented Interfaces:

WordTokenizer, IsCloseable, UsesLogger

public class PennTreebankTokenizer
extends AbstractWordTokenizer
implements WordTokenizer

Split text into tokens according the Penn Treebank tokenization rules.

Based upon the sed script written by Robert McIntyre at http://www.cis.upenn.edu/~treebank/tokenizer.sed .

Field Summary

Fields

Modifier and Type	Field and Description
protected static java.util.List<PatternReplacer>	pennPatterns Replacement patterns for transforming original text.

Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

abbreviations, aposTokens, apostropheCanBeQuote, coalesceAsterisks, coalesceHyphens, contractions, contractionsURL, hyphensMatcher, hyphensPattern, logger, preTokenizer

Constructor Summary

Constructors

Constructor and Description
PennTreebankTokenizer() Create a simple word tokenizer.

Method Summary

Methods

Modifier and Type	Method and Description
java.util.List<java.lang.String>	extractWords(java.lang.String text) Break text into word tokens.
static java.lang.String	prepareTextForTokenization(java.lang.String s)

Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

addWordToSentence, findWordOffsets, getLogger, getPreTokenizer, isClosingQuote, isLetterOrSingleQuote, isMultipleHyphens, isSingleOpeningQuote, loadContractions, preprocessToken, setAbbreviations, setAposTokens, setLogger, setPreTokenizer, splitToken

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.WordTokenizer

addWordToSentence, close, findWordOffsets, getPreTokenizer, preprocessToken, setAbbreviations, setAposTokens, setPreTokenizer

Methods inherited from interface edu.northwestern.at.utils.IsCloseable

close

Field Detail

pennPatterns

protected static java.util.List<PatternReplacer> pennPatterns

Replacement patterns for transforming original text.

Constructor Detail

PennTreebankTokenizer

public PennTreebankTokenizer()

Create a simple word tokenizer.

Method Detail

prepareTextForTokenization

public static java.lang.String prepareTextForTokenization(java.lang.String s)

extractWords

public java.util.List<java.lang.String> extractWords(java.lang.String text)

Break text into word tokens.

Specified by:

extractWords in interface WordTokenizer

Specified by:

extractWords in class AbstractWordTokenizer

Parameters:

text - Text to break into word tokens.

Returns:

List of word tokens.

Word tokens may be words, numbers, punctuation, etc.