edu.northwestern.at.morphadorner

Class IDFixerFilter

java.lang.Object

org.xml.sax.helpers.XMLFilterImpl

edu.northwestern.at.utils.xml.ExtendedXMLFilterImpl

edu.northwestern.at.morphadorner.IDFixerFilter

All Implemented Interfaces:

org.xml.sax.ContentHandler, org.xml.sax.DTDHandler, org.xml.sax.EntityResolver, org.xml.sax.ErrorHandler, org.xml.sax.XMLFilter, org.xml.sax.XMLReader

public class IDFixerFilter
extends ExtendedXMLFilterImpl

XML filter which updates tag IDs and attributes.

Field Summary

Fields

Modifier and Type	Field and Description
protected java.lang.String	baseFileName Base XML file name for generating ID values.
protected QueueStack<java.lang.String>	divStack Div tag stack.
protected java.lang.String	elementURI URI for elements without one.
protected int	emittedWordCount Current number of words emitted.
protected java.lang.String	facsFromPB "facs" or "ref" from current element.
protected QueueStack<java.lang.String>	foreignStack Foreign language attribute stack.
protected int	gapCount Gap count.
protected java.lang.String	id Current ID value as string.
protected static java.text.NumberFormat	ID_FORMATTER Word ID formatters.
protected java.lang.String	idAttrName ID attribute name.
protected int	idSpacing ID Spacing.
protected MorphAdornerSettings.XMLIDType	idType ID Type.
protected boolean	isFirstWord True if we're processing first word in a sentence.
protected QueueStack<XMLWriterState>	jumpStack Jump tag stack.
protected static java.util.Map<java.lang.String,java.lang.String>	languageTags Map of XML language tags to language name.
protected java.lang.String	lastID Previous ID value.
protected java.lang.String	lastIDString
protected MorphAdornerSettings	morphAdornerSettings MorphAdorner settings.
protected java.lang.String	nFromPB "n" from current element.
protected boolean	outputNonredundantAttributesOnly True to output non-redundant attributes only.
protected boolean	outputNonredundantEosAttribute True to output non-redundant eos attributes only.
protected boolean	outputNonredundantPartAttribute True to output non-redundant part attributes only.
protected boolean	outputNonredundantTokenAttribute True to output non-redundant token attributes only.
protected boolean	outputPseudoPageBoundaryMilestones True to output page boundary milestones.
protected boolean	outputSentenceBoundaryMilestones True to output sentence boundary milestones.
protected boolean	outputWhitespace True to output whitespace elements.
protected boolean	outputWordOrdinal Output word ordinal.
protected static java.text.NumberFormat	PAGE_FORMATTER Page number formatter.
protected int	pageColumn Column (0-based) within current page based upon repeated facs= value of pb.
protected int	pageNumber Running page number.
protected PendingElement	pendingWordElement Pending word element.
protected PartOfSpeechTags	posTags Part of speech tags used in XML file.
protected java.lang.String	prevFacsFromPB "facs" or "ref" from previous element.
protected java.util.Set<java.lang.String>	pseudoPageContainerDivTypes Pseudo-page ending div types.
protected int	pseudoPageCount Current pseudo page count.
protected int	pseudoPageSize Page size in number of tokens.
protected boolean	pseudoPageStarted True if pseudo page started.
protected int	pseudoPageWordCount Current pseudo page word count.
protected XMLSentenceMelder	sentenceMelder XML sentence melder.
protected SortedArrayList<SentenceAndWordNumber>	sortedWords Sorted sentence and word number information.
protected java.util.Map<java.lang.Integer,java.lang.Integer>	splitWords Split words map of word ID to # of word parts.
protected java.util.Map<java.lang.Integer,java.lang.Integer>	splitWordsCopy Copy of split words map.
protected boolean	tokenizingOnly True if only doing tokenization.
protected int	totalWordsToEmit Total number of words to emit.
protected boolean	usePCToMarkEndOfSentence True to use element to mark end of sentence.
protected static java.text.NumberFormat	WORD_FORMATTER Word within page formatter.
protected int	wordNumberWithinPage Word within page number.
protected int	wordOrdinal Word ordinal.
protected XMLWriter	writer XML writer.
protected java.lang.String	xmlTokenLabelAttribute Token label attribute.
protected boolean	xmlTokenLabelEmit Emit token label?
protected boolean	xmlTokenLabelPrependWorkName Token label prepend work name flag.
protected int	xmlTokenLabelSpacing Token label spacing.
protected boolean	zzzzljTagSeen "zzzzlj" tag found.

Constructor Summary

Constructors

Constructor and Description
IDFixerFilter(org.xml.sax.XMLReader reader, PartOfSpeechTags posTags, java.lang.String outFile, int maxID, SortedArrayList<SentenceAndWordNumber> sortedWords, java.util.Map<java.lang.Integer,java.lang.Integer> splitWords, int totalWords, int totalPageBreaks, MorphAdornerSettings morphAdornerSettings, boolean tokenizingOnly) Create ID filter.

Method Summary

Methods

Modifier and Type	Method and Description
void	characters(char[] ch, int start, int length) Handle character data.
PendingElement	createPseudoPageElement(java.lang.String uri, boolean forcedEmit, boolean start, java.lang.String path) Create a pseudo page milestone.
void	emitPseudoPageElement(PendingElement pseudoPageElement) Emit a pseudo page milestone.
void	emitWordElement(java.lang.String uri, java.lang.String localName, java.lang.String qName, org.xml.sax.helpers.AttributesImpl atts, java.lang.String wordText, boolean allowOutputWhitespace, boolean forceEOS) Emit a word element.
void	endElement(java.lang.String uri, java.lang.String localName, java.lang.String qName) Handle end of an element.
protected java.lang.String	generateTokenLabel() Generate token label based upon current pb element attributes.
protected static java.lang.String	getDisplayableAttributes(org.xml.sax.Attributes atts) Get displayable list of element attributes and values.
java.lang.String	getForeignLanguageTag(java.lang.String qName, org.xml.sax.Attributes atts) Get the foreign language tag for XML element.
XMLSentenceMelder	getSentenceMelder() Get associated sentence melder.
protected void	setIDFormat(java.lang.String outFile, int maxID, int maxPageBreaks) Set word ID format.
void	setPosTags(PartOfSpeechTags posTags) Set the part of speech tags.
void	setSentenceMelder(XMLSentenceMelder sentenceMelder) Set associated sentence melder.
protected void	setSplitWords(java.util.Map<java.lang.Integer,java.lang.Integer> splitWords) Set split words.
void	setWriter(XMLWriter writer) Set associated XML writer.
void	startElement(java.lang.String uri, java.lang.String localName, java.lang.String qName, org.xml.sax.Attributes atts) Handle start of an XML element.

Methods inherited from class edu.northwestern.at.utils.xml.ExtendedXMLFilterImpl

removeAttribute, setAttributeValue, setAttributeValue, setAttributeValue

Methods inherited from class org.xml.sax.helpers.XMLFilterImpl

endDocument, endPrefixMapping, error, fatalError, getContentHandler, getDTDHandler, getEntityResolver, getErrorHandler, getFeature, getParent, getProperty, ignorableWhitespace, notationDecl, parse, parse, processingInstruction, resolveEntity, setContentHandler, setDocumentLocator, setDTDHandler, setEntityResolver, setErrorHandler, setFeature, setParent, setProperty, skippedEntity, startDocument, startPrefixMapping, unparsedEntityDecl, warning

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ID_FORMATTER

protected static final java.text.NumberFormat ID_FORMATTER

Word ID formatters.

PAGE_FORMATTER

protected static final java.text.NumberFormat PAGE_FORMATTER

Page number formatter.

WORD_FORMATTER

protected static final java.text.NumberFormat WORD_FORMATTER

Word within page formatter.

wordOrdinal

protected int wordOrdinal

Word ordinal.

lastID

protected java.lang.String lastID

Previous ID value.

lastIDString

protected java.lang.String lastIDString

id

protected java.lang.String id

Current ID value as string.

idAttrName

protected java.lang.String idAttrName

ID attribute name.

baseFileName

protected java.lang.String baseFileName

Base XML file name for generating ID values.

posTags

protected PartOfSpeechTags posTags

Part of speech tags used in XML file.

elementURI

protected java.lang.String elementURI

URI for elements without one.

outputWhitespace

protected boolean outputWhitespace

True to output whitespace elements.

outputNonredundantAttributesOnly

protected boolean outputNonredundantAttributesOnly

True to output non-redundant attributes only.

outputNonredundantTokenAttribute

protected boolean outputNonredundantTokenAttribute

True to output non-redundant token attributes only.

outputNonredundantPartAttribute

protected boolean outputNonredundantPartAttribute

True to output non-redundant part attributes only.

outputNonredundantEosAttribute

protected boolean outputNonredundantEosAttribute

True to output non-redundant eos attributes only.

outputSentenceBoundaryMilestones

protected boolean outputSentenceBoundaryMilestones

True to output sentence boundary milestones.

usePCToMarkEndOfSentence

protected boolean usePCToMarkEndOfSentence

True to use element to mark end of sentence.

outputPseudoPageBoundaryMilestones

protected boolean outputPseudoPageBoundaryMilestones

True to output page boundary milestones.

pseudoPageSize

protected int pseudoPageSize

Page size in number of tokens.

pseudoPageCount

protected int pseudoPageCount

Current pseudo page count.

pseudoPageWordCount

protected int pseudoPageWordCount

Current pseudo page word count.

pseudoPageStarted

protected boolean pseudoPageStarted

True if pseudo page started.

emittedWordCount

protected int emittedWordCount

Current number of words emitted.

sentenceMelder

protected XMLSentenceMelder sentenceMelder

XML sentence melder.

isFirstWord

protected boolean isFirstWord

True if we're processing first word in a sentence.

pendingWordElement

protected PendingElement pendingWordElement

Pending word element.

splitWords

protected java.util.Map<java.lang.Integer,java.lang.Integer> splitWords

Split words map of word ID to # of word parts.

splitWordsCopy

protected java.util.Map<java.lang.Integer,java.lang.Integer> splitWordsCopy

Copy of split words map.

foreignStack

protected QueueStack<java.lang.String> foreignStack

Foreign language attribute stack.

jumpStack

protected QueueStack<XMLWriterState> jumpStack

Jump tag stack. Saves state across jump tags.

divStack

protected QueueStack<java.lang.String> divStack

Div tag stack.

pseudoPageContainerDivTypes

protected java.util.Set<java.lang.String> pseudoPageContainerDivTypes

Pseudo-page ending div types.

sortedWords

protected SortedArrayList<SentenceAndWordNumber> sortedWords

Sorted sentence and word number information.

writer

protected XMLWriter writer

XML writer.

totalWordsToEmit

protected int totalWordsToEmit

Total number of words to emit.

pageNumber

protected int pageNumber

Running page number. Starts at 0.

wordNumberWithinPage

protected int wordNumberWithinPage

Word within page number. Starts at 1.

nFromPB

protected java.lang.String nFromPB

"n" from current element.

facsFromPB

protected java.lang.String facsFromPB

"facs" or "ref" from current element.

prevFacsFromPB

protected java.lang.String prevFacsFromPB

"facs" or "ref" from previous element.

pageColumn

protected int pageColumn

Column (0-based) within current page based upon repeated facs= value of pb.

idSpacing

protected int idSpacing

ID Spacing.

idType

protected MorphAdornerSettings.XMLIDType idType

ID Type.

xmlTokenLabelEmit

protected boolean xmlTokenLabelEmit

Emit token label?

xmlTokenLabelAttribute

protected java.lang.String xmlTokenLabelAttribute

Token label attribute.

xmlTokenLabelSpacing

protected int xmlTokenLabelSpacing

Token label spacing.

xmlTokenLabelPrependWorkName

protected boolean xmlTokenLabelPrependWorkName

Token label prepend work name flag.

outputWordOrdinal

protected boolean outputWordOrdinal

Output word ordinal.

languageTags

protected static java.util.Map<java.lang.String,java.lang.String> languageTags

Map of XML language tags to language name.

gapCount

protected int gapCount

Gap count.

morphAdornerSettings

protected MorphAdornerSettings morphAdornerSettings

MorphAdorner settings.

tokenizingOnly

protected boolean tokenizingOnly

True if only doing tokenization.

zzzzljTagSeen

protected boolean zzzzljTagSeen

"zzzzlj" tag found.

Constructor Detail

IDFixerFilter

public IDFixerFilter(org.xml.sax.XMLReader reader,
             PartOfSpeechTags posTags,
             java.lang.String outFile,
             int maxID,
             SortedArrayList<SentenceAndWordNumber> sortedWords,
             java.util.Map<java.lang.Integer,java.lang.Integer> splitWords,
             int totalWords,
             int totalPageBreaks,
             MorphAdornerSettings morphAdornerSettings,
             boolean tokenizingOnly)

Create ID filter.

Parameters:

reader - The XML reader to filter.

posTags - The part of speech tags.

outFile - The output file name.

maxID - The maximum integer word ID.

sortedWords - Sentence and word numbers sorted by word ID.

splitWords - Split words.

totalWords - Total words.

totalPageBreaks - Total page breaks.

tokenizingOnly - True if only emitting tokenization.

Method Detail

generateTokenLabel

protected java.lang.String generateTokenLabel()

Generate token label based upon current pb element attributes.

startElement

public void startElement(java.lang.String uri,
                java.lang.String localName,
                java.lang.String qName,
                org.xml.sax.Attributes atts)
                  throws org.xml.sax.SAXException

Handle start of an XML element.

Specified by:

startElement in interface org.xml.sax.ContentHandler

Overrides:

startElement in class org.xml.sax.helpers.XMLFilterImpl

Throws:

org.xml.sax.SAXException

characters

public void characters(char[] ch,
              int start,
              int length)
                throws org.xml.sax.SAXException

Handle character data.

Specified by:

characters in interface org.xml.sax.ContentHandler

Overrides:

characters in class org.xml.sax.helpers.XMLFilterImpl

Parameters:

ch - Array of characters.

start - The starting position in the array.

length - The number of characters.

Throws:

org.xml.sax.SAXException - If there is an error.

emitWordElement

public void emitWordElement(java.lang.String uri,
                   java.lang.String localName,
                   java.lang.String qName,
                   org.xml.sax.helpers.AttributesImpl atts,
                   java.lang.String wordText,
                   boolean allowOutputWhitespace,
                   boolean forceEOS)
                     throws org.xml.sax.SAXException

Emit a word element.

Parameters:

uri - The word element's URI.

localName - The word element's local name.

qName - The word element's qname.

atts - The word element's attributes.

wordText - The word element's text.

allowOutputWhitespace - True to allow outputting whitespace element for word.

forceEOS - True to force end of sentence for this word.

Throws:

org.xml.sax.SAXException

endElement

public void endElement(java.lang.String uri,
              java.lang.String localName,
              java.lang.String qName)
                throws org.xml.sax.SAXException

Handle end of an element.

Specified by:

endElement in interface org.xml.sax.ContentHandler

Overrides:

endElement in class org.xml.sax.helpers.XMLFilterImpl

Parameters:

uri - The XML element's URI.

localName - The XML element's local name.

qName - The XML element's qname.

Throws:

org.xml.sax.SAXException

createPseudoPageElement

public PendingElement createPseudoPageElement(java.lang.String uri,
                                     boolean forcedEmit,
                                     boolean start,
                                     java.lang.String path)

Create a pseudo page milestone.

Parameters:

uri - Element URI.

forcedEmit - Emit pseudo page milestone even if not enough words accumulated, as long as at least one word in current block.

start - true if starting milestone, false if ending.

path - Path attribute. May be null.

Returns:

The pseudo page element.

emitPseudoPageElement

public void emitPseudoPageElement(PendingElement pseudoPageElement)

Emit a pseudo page milestone.

Parameters:

pseudoPageElement - The pseudo page element to emit.

setPosTags

public void setPosTags(PartOfSpeechTags posTags)

Set the part of speech tags.

Parameters:

posTags - The part of speech tags.

setSplitWords

protected void setSplitWords(java.util.Map<java.lang.Integer,java.lang.Integer> splitWords)

Set split words.

Parameters:

splitWords - Map of split words.

setIDFormat

protected void setIDFormat(java.lang.String outFile,
               int maxID,
               int maxPageBreaks)

Set word ID format.

Parameters:

outFile - Output file name used to derive word IDs.

maxID - Maximum integer word ID value.

maxPageBreaks - Maximum number of page breaks.

setWriter

public void setWriter(XMLWriter writer)

Set associated XML writer.

Parameters:

writer - XML writer.

setSentenceMelder

public void setSentenceMelder(XMLSentenceMelder sentenceMelder)

Set associated sentence melder.

Parameters:

sentenceMelder - Sentence melder.

getSentenceMelder

public XMLSentenceMelder getSentenceMelder()

Get associated sentence melder.

Returns:

Sentence melder.

getForeignLanguageTag

public java.lang.String getForeignLanguageTag(java.lang.String qName,
                                     org.xml.sax.Attributes atts)

Get the foreign language tag for XML element.

Parameters:

qName - XML element name.

atts - XML element attributes.

Returns:

The language part of speech tag for this element.

getDisplayableAttributes

protected static java.lang.String getDisplayableAttributes(org.xml.sax.Attributes atts)

Get displayable list of element attributes and values.

Parameters:

atts - Attributes.

Returns:

Displayable string in name=value format.