edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Class EEBOWordTokenizer

java.lang.Object

edu.northwestern.at.utils.IsCloseableObject

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.DefaultWordTokenizer

edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.EEBOWordTokenizer

All Implemented Interfaces:

WordTokenizer, IsCloseable, UsesLogger

public class EEBOWordTokenizer
extends DefaultWordTokenizer
implements WordTokenizer

Word tokenizer for EEBO texts.

Do not use this when EEBO texts have been converted to TEIAnalytics format.

Field Summary

Fields

Modifier and Type	Field and Description
protected static java.util.regex.Matcher	numberDotSpellingMatcher
protected static java.util.regex.Pattern	numberDotSpellingPattern Pattern to match number.word
protected static java.util.regex.Matcher	underlineCapCapMatcher
protected static java.util.regex.Pattern	underlineCapCapPattern Pattern to match _CapCap

Fields inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

abbreviations, aposTokens, apostropheCanBeQuote, coalesceAsterisks, coalesceHyphens, contractions, contractionsURL, hyphensMatcher, hyphensPattern, logger, preTokenizer

Constructor Summary

Constructors

Constructor and Description
EEBOWordTokenizer() Create EEBO word tokenizer.

Method Summary

Methods

Modifier and Type	Method and Description
java.lang.String	preprocessToken(java.lang.String token, java.util.List<java.lang.String> tokenList) Preprocess a word token.

Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.DefaultWordTokenizer

addWordToSentence, extractWords

Methods inherited from class edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.AbstractWordTokenizer

findWordOffsets, getLogger, getPreTokenizer, isClosingQuote, isLetterOrSingleQuote, isMultipleHyphens, isSingleOpeningQuote, loadContractions, setAbbreviations, setAposTokens, setLogger, setPreTokenizer, splitToken

Methods inherited from class edu.northwestern.at.utils.IsCloseableObject

close

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer.WordTokenizer

addWordToSentence, close, extractWords, findWordOffsets, getPreTokenizer, setAbbreviations, setAposTokens, setPreTokenizer

Methods inherited from interface edu.northwestern.at.utils.IsCloseable

close

Field Detail

numberDotSpellingPattern

protected static final java.util.regex.Pattern numberDotSpellingPattern

Pattern to match number.word

numberDotSpellingMatcher

protected static final java.util.regex.Matcher numberDotSpellingMatcher

underlineCapCapPattern

protected static java.util.regex.Pattern underlineCapCapPattern

Pattern to match _CapCap

underlineCapCapMatcher

protected static final java.util.regex.Matcher underlineCapCapMatcher

Constructor Detail

EEBOWordTokenizer

public EEBOWordTokenizer()

Create EEBO word tokenizer.

Method Detail

preprocessToken

public java.lang.String preprocessToken(java.lang.String token,
                               java.util.List<java.lang.String> tokenList)

Preprocess a word token.

Specified by:

preprocessToken in interface WordTokenizer

Overrides:

preprocessToken in class AbstractWordTokenizer

Parameters:

token - Token to preprocess.

tokenList - List of previous tokens already issued.

Returns:

Preprocessed token. The token list may also have been modified.