edu.northwestern.at.morphadorner.tools.annolex

Class MergeAnnolexCorrectionsIntoAdornedXML

java.lang.Object

edu.northwestern.at.morphadorner.tools.annolex.MergeAnnolexCorrectionsIntoAdornedXML

public class MergeAnnolexCorrectionsIntoAdornedXML
extends java.lang.Object

Merge Annolex corrections into MorphAdorned XML file.

Usage:

MergeAnnolexCorrectionsIntoAdornedXML correctionsdirectory outputdirectory inputfiles

• correctionsdirectory -- the input directory with corrections files in tabular format.
• outputdirectory -- the output directory for the corrected adorned XML files.
• inputfiles -- the input adorned XML files to be corrected.

The corrections file is a tab-separated utf-8 file containing the following columns.

• Work ID.
• Word ID.
• Old spelling.
• Corrected spelling.
• Standard spelling.
• Corrected lemmata.
• Corrected parts of speech.
• Operation: 1=update, 2=insert, 3=delete, 5=delete nearest gap.

The corrected spelling, lemmata, and parts of speech may all be empty when the operation is 3 (delete).

The value of the "ord" (word ordinal) attribute for each word is adjusted to account for inserted and deleted words. The value of the "reg" (standard spelling) and "tok" attributes (original token) are generated as needed for updated and inserted words.

Whitespace markers "<c> </c>" are added and deleted as needed when tokens are added or deleted. In general, most added punctuation and symbols do not require added whitespace markers. When tokens are deleted, sequences of "<c> </c><c> </c> ..." are compressed to a single "<c> </c>" entry.

Field Summary

Fields

Modifier and Type	Field and Description
protected static int	addedWords Words added, deleted, or modified in current document.
protected static java.util.Set<java.lang.String>	badPosTags Holds bad pos tags.
protected static org.jdom2.Element	clonableCElement "c" element for cloning.
protected static java.util.Set<java.lang.String>	combinedBadPosTags Holds combined bad pos tags.
protected static java.util.Set<java.lang.String>	combinedMismatches Holds combined mismatches.
protected static java.util.Map<java.lang.String,CorrectedWord>	correctedWordsMap Map word IDs to corrected words.
protected static int	currentDocNumber Current document.
protected static boolean	debug True to produce debugging output.
protected static int	deletedGaps
protected static int	deletedWords
protected static int	docsToProcess Number of documents to process.
protected static org.jdom2.Document	document DOM document.
protected static java.util.List<org.jdom2.Element>	gapElementsToDelete List of gaps to delete.
protected static java.util.Map<java.lang.String,org.jdom2.Element>	gapIDsToElements Map gap IDs to gap elements in DOM document.
protected static int	INITPARAMS # params before input file specs.
protected static java.lang.String	inputCorrectionsDirectory Input directory for corrections files.
protected static java.lang.String	inputXMLDirectory Input directory for adorned files.
protected static java.util.Set<java.lang.String>	mismatches Holds individual mismatches.
protected static int	modifiedWords
protected static java.lang.String	outputDirectory Output directory for corrected adorned files.
protected static java.io.PrintStream	outputFileStream Output file stream.
protected static PartOfSpeechTags	posTags Pos tags.
protected static java.io.PrintStream	printStream Wrapper for printStream to allow utf-8 output.
protected static boolean	verbose True to produce verbose output.
protected static java.util.List<org.jdom2.Element>	wordElementsToDelete List of word elements to delete.
protected static java.util.Map<java.lang.String,org.jdom2.Element>	wordIDsToElements Map word IDs to word elements in DOM document.

Constructor Summary

Constructors

Constructor and Description
MergeAnnolexCorrectionsIntoAdornedXML()

Method Summary

Methods

Modifier and Type	Method and Description
protected static void	applyCorrections(org.jdom2.Document document) Apply corrections to XML document from corrections file.
protected static boolean	changeAttribute(org.jdom2.Element element, java.lang.String attrName, java.lang.String oldValue, java.lang.String newValue) Change attribute value.
protected static void	compressCElements(org.jdom2.Document document) Compress "" elements.
protected static int	countSeparators(java.lang.String s, char sep) Count separators in string.
protected static void	deleteGapElement(org.jdom2.Element gapElement) Delete a gap element.
protected static void	deleteGapElements(java.util.List<org.jdom2.Element> gapElementsToDelete) Delete gaps in XML file.
protected static void	deleteWordElement(org.jdom2.Element wordElement, java.util.List<java.lang.String> sortedWordIDs) Delete a word element.
protected static void	deleteWordElements(java.util.List<org.jdom2.Element> wordElementsToDelete, java.util.List<java.lang.String> sortedWordIDs) Delete words in XML file.
protected static void	deleteWordOld(org.jdom2.Element wordElement, java.util.List<java.lang.String> correctedWordIDs, int i) Delete a word.
protected static org.jdom2.Element	extractCElement(org.jdom2.Document document) Extract a "c" element from XML file to use for cloning copies.
protected static java.util.Map<java.lang.String,org.jdom2.Element>	extractGaps(org.jdom2.Document document) Extract gaps from "gap" elements in XML document file.
protected static java.util.Map<java.lang.String,org.jdom2.Element>	extractWords(org.jdom2.Document document) Extract words specified by "w" elements in XML document file.
protected static void	fixEOSAttributes(java.util.Map<java.lang.String,org.jdom2.Element> wordIDsToElements) Fix EOS attributes.
protected static void	fixSplitWordIDs(org.jdom2.Document document) Fix split word IDs.
protected static void	gapToWord(org.jdom2.Element gapElement, CorrectedWord correctedWord) Replace a gap element by a word element.
static java.util.List<java.lang.String>	getRelatedWordIDs(java.lang.String wordID) Get related adorned word IDs for a word ID.
protected static java.util.List<java.lang.String>	getSortedWordIDs() Get sorted word IDs.
protected static boolean	initialize(java.lang.String[] args) Initialize.
protected static int	insertWord(java.lang.String idToInsert, CorrectedWord correctedWord, java.util.List<java.lang.String> correctedWordIDs, int i) Insert word in XML file.
protected static CorrectedWordsFileReader	loadCorrectionsFile(java.lang.String correctionFileName) Load word corrections file.
protected static org.jdom2.Document	loadXML(java.lang.String inputXMLFileName) Load XML document file.
static void	main(java.lang.String[] args) Main program.
protected static <K,V> void	printMap(java.lang.String mapLabel, java.util.Map<K,V> map) Print contents of a map (HashMap, TreeMap, etc.).
protected static void	printMismatches() Print mismatches.
protected static <V> void	printSet(java.lang.String setLabel, java.util.Set<V> set) Print contents of a set (HashSet, TreeSet, etc.).
protected static int	processFiles(java.lang.String[] args) Process files.
protected static void	processOneFile(java.lang.String xmlInputFileName) Process corrections for one XML file.
protected static void	replaceGapElementsWithWords(java.util.List<org.jdom2.Element> gapElementsToUpdate) Replace gaps in XML file with real words.
protected static void	resplit(java.lang.String id, java.lang.String[] spellingParts, java.lang.String oldJoinedSpelling, java.lang.String updatedSpelling) Fix spelling parts of updated split word.
protected static void	terminate(int filesProcessed, long processingTime) Terminate.
protected static boolean	updateWord(org.jdom2.Document document, org.jdom2.Element wordElement, CorrectedWord correctedWord, java.util.List<java.lang.String> correctedWordIDs, int i, java.lang.String correctedSpelling) Update word in XML file.
protected static void	updateWordOrdinals(java.util.Map<java.lang.String,org.jdom2.Element> wordIDsToElements) Update word ordinals.
protected static int	validateCorrections(java.util.Map<java.lang.String,org.jdom2.Element> wordIDsToElements) Validate entries in corrections file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

docsToProcess

protected static int docsToProcess

Number of documents to process.

currentDocNumber

protected static int currentDocNumber

Current document.

inputXMLDirectory

protected static java.lang.String inputXMLDirectory

Input directory for adorned files.

inputCorrectionsDirectory

protected static java.lang.String inputCorrectionsDirectory

Input directory for corrections files.

outputDirectory

protected static java.lang.String outputDirectory

Output directory for corrected adorned files.

outputFileStream

protected static java.io.PrintStream outputFileStream

Output file stream.

printStream

protected static java.io.PrintStream printStream

Wrapper for printStream to allow utf-8 output.

document

protected static org.jdom2.Document document

DOM document.

wordIDsToElements

protected static java.util.Map<java.lang.String,org.jdom2.Element> wordIDsToElements

Map word IDs to word elements in DOM document.

gapIDsToElements

protected static java.util.Map<java.lang.String,org.jdom2.Element> gapIDsToElements

Map gap IDs to gap elements in DOM document.

correctedWordsMap

protected static java.util.Map<java.lang.String,CorrectedWord> correctedWordsMap

Map word IDs to corrected words.

INITPARAMS

protected static final int INITPARAMS

# params before input file specs.

See Also:

Constant Field Values

posTags

protected static PartOfSpeechTags posTags

Pos tags.

badPosTags

protected static java.util.Set<java.lang.String> badPosTags

Holds bad pos tags.

combinedBadPosTags

protected static java.util.Set<java.lang.String> combinedBadPosTags

Holds combined bad pos tags.

mismatches

protected static java.util.Set<java.lang.String> mismatches

Holds individual mismatches.

combinedMismatches

protected static java.util.Set<java.lang.String> combinedMismatches

Holds combined mismatches.

addedWords

protected static int addedWords

Words added, deleted, or modified in current document.

deletedWords

protected static int deletedWords

modifiedWords

protected static int modifiedWords

deletedGaps

protected static int deletedGaps

clonableCElement

protected static org.jdom2.Element clonableCElement

"c" element for cloning.

wordElementsToDelete

protected static java.util.List<org.jdom2.Element> wordElementsToDelete

List of word elements to delete.

gapElementsToDelete

protected static java.util.List<org.jdom2.Element> gapElementsToDelete

List of gaps to delete.

verbose

protected static boolean verbose

True to produce verbose output.

debug

protected static boolean debug

True to produce debugging output.

Constructor Detail

MergeAnnolexCorrectionsIntoAdornedXML

public MergeAnnolexCorrectionsIntoAdornedXML()

Method Detail

main

public static void main(java.lang.String[] args)

Main program.

Parameters:

args - Program parameters.

initialize

protected static boolean initialize(java.lang.String[] args)
                             throws java.lang.Exception

Initialize.

Throws:

java.lang.Exception

processOneFile

protected static void processOneFile(java.lang.String xmlInputFileName)
                              throws java.io.IOException

Process corrections for one XML file.

Parameters:

xmlInputFileName - XML input file.

Throws:

java.io.IOException

loadXML

protected static org.jdom2.Document loadXML(java.lang.String inputXMLFileName)
                                     throws org.jdom2.JDOMException,
                                            java.io.IOException

Load XML document file.

Parameters:

inputXMLFileName - Input XML file name.

Returns:

Parsed XML file as document.

Throws:

org.jdom2.JDOMException - java.io.IOException

java.io.IOException

extractWords

protected static java.util.Map<java.lang.String,org.jdom2.Element> extractWords(org.jdom2.Document document)

Extract words specified by "w" elements in XML document file.

Parameters:

document - Parsed XML document.

Returns:

Map from word ID to parsed "w" element for each word in XML document.

extractGaps

protected static java.util.Map<java.lang.String,org.jdom2.Element> extractGaps(org.jdom2.Document document)

Extract gaps from "gap" elements in XML document file.

Parameters:

document - Parsed XML document.

Returns:

Map from word ID to parsed "gap" element for each gap in XML document.

extractCElement

protected static org.jdom2.Element extractCElement(org.jdom2.Document document)

Extract a "c" element from XML file to use for cloning copies.

Parameters:

document - Parsed XML document.

Returns:

A parsed "c" element.

validateCorrections

protected static int validateCorrections(java.util.Map<java.lang.String,org.jdom2.Element> wordIDsToElements)

Validate entries in corrections file.

Parameters:

wordIDsToElements - Map of document word IDs to document "w" elements.

Returns:

Number of bad corrections.

applyCorrections

protected static void applyCorrections(org.jdom2.Document document)
                                throws java.lang.Exception

Apply corrections to XML document from corrections file.

Parameters:

document - Document to update.

Throws:

java.lang.Exception

resplit

protected static void resplit(java.lang.String id,
           java.lang.String[] spellingParts,
           java.lang.String oldJoinedSpelling,
           java.lang.String updatedSpelling)

Fix spelling parts of updated split word.

compressCElements

protected static void compressCElements(org.jdom2.Document document)

Compress "" elements.

Parameters:

document - Document in which to compress "" elements.

Deleting words may have left sequences of " ..." elements. Each such sequence should be compressed to a single " " element.

fixSplitWordIDs

protected static void fixSplitWordIDs(org.jdom2.Document document)

Fix split word IDs.

Parameters:

document - Parsed XML document.

fixEOSAttributes

protected static void fixEOSAttributes(java.util.Map<java.lang.String,org.jdom2.Element> wordIDsToElements)

Fix EOS attributes.

Parameters:

wordIDsToElements - Map of word ID to parsed "w" elements.

Adding or deleting words may have left sequences of "w" elements which all have the "eos" attribute set to "1". This method updates the "w" elements in such as sequence so that only the last "w" element in the sequence has "eos" set to "1".

updateWordOrdinals

protected static void updateWordOrdinals(java.util.Map<java.lang.String,org.jdom2.Element> wordIDsToElements)

Update word ordinals.

Parameters:

wordIDsToElements - Map of document word IDs to document "w" elements.

countSeparators

protected static int countSeparators(java.lang.String s,
                  char sep)

Count separators in string.

Parameters:

s - The string in which to count separators.

sep - The separator character.

Returns:

The separator count.

changeAttribute

protected static boolean changeAttribute(org.jdom2.Element element,
                      java.lang.String attrName,
                      java.lang.String oldValue,
                      java.lang.String newValue)

Change attribute value.

Parameters:

element - DOM element.

attrName - Attribute name.

oldValue - Old attribute value.

newValue - New attribute value.

Returns:

true if attribute changed, false otherwise.

getRelatedWordIDs

public static java.util.List<java.lang.String> getRelatedWordIDs(java.lang.String wordID)

Get related adorned word IDs for a word ID.

Parameters:

wordID - Word ID for which related IDs are wanted.

Returns:

List of related word IDs.

Related word IDs are the word IDs for the other parts of a split word. The returned list includes the given wordID.

For unsplit words, the single given wordID is returned in the list.

Null is returned when the wordID does not exist.

processFiles

protected static int processFiles(java.lang.String[] args)
                           throws java.lang.Exception

Process files.

Throws:

java.lang.Exception

printMismatches

protected static void printMismatches()

Print mismatches.

printMap

protected static <K,V> void printMap(java.lang.String mapLabel,
                  java.util.Map<K,V> map)

Print contents of a map (HashMap, TreeMap, etc.).

Parameters:

mapLabel - Label for map.

map - The map to print.

N.B. This method assumes both the keys and values have toString() methods.

printSet

protected static <V> void printSet(java.lang.String setLabel,
                java.util.Set<V> set)

Print contents of a set (HashSet, TreeSet, etc.).

Parameters:

setLabel - Label for set.

set - The set to print.

N.B. This method assumes set values have toString() methods.

terminate

protected static void terminate(int filesProcessed,
             long processingTime)

Terminate.

Parameters:

filesProcessed - Number of files processed.

processingTime - Processing time in seconds.

loadCorrectionsFile

protected static CorrectedWordsFileReader loadCorrectionsFile(java.lang.String correctionFileName)
                                                       throws java.lang.Exception

Load word corrections file.

Parameters:

correctionFileName - Name of word correction file.

Throws:

java.lang.Exception

updateWord

protected static boolean updateWord(org.jdom2.Document document,
                 org.jdom2.Element wordElement,
                 CorrectedWord correctedWord,
                 java.util.List<java.lang.String> correctedWordIDs,
                 int i,
                 java.lang.String correctedSpelling)

Update word in XML file.

Parameters:

wordElement - Word to update.

correctedWord - Correction for word.

correctedWordIDs - List of corrected word IDs.

i - Index of word in list of word IDs.

correctedSpelling - Corrected spelling.

Returns:

True if word updated.

insertWord

protected static int insertWord(java.lang.String idToInsert,
             CorrectedWord correctedWord,
             java.util.List<java.lang.String> correctedWordIDs,
             int i)

Insert word in XML file.

Parameters:

idToInsert - Word ID to insert.

correctedWord - Data for word to insert.

correctedWordIDs - List of corrected word IDs.

i - Index of word in list of word IDs.

Returns:

Number of new word elements added.

deleteWordElements

protected static void deleteWordElements(java.util.List<org.jdom2.Element> wordElementsToDelete,
                      java.util.List<java.lang.String> sortedWordIDs)

Delete words in XML file.

Parameters:

wordElementsToDelete - Word elements to delete.

sortedWordIDs - Sorted word IDs in document.

deleteGapElements

protected static void deleteGapElements(java.util.List<org.jdom2.Element> gapElementsToDelete)

Delete gaps in XML file.

Parameters:

gapElementsToDelete - Gap elements to delete.

replaceGapElementsWithWords

protected static void replaceGapElementsWithWords(java.util.List<org.jdom2.Element> gapElementsToUpdate)

Replace gaps in XML file with real words.

Parameters:

gapElementsToUpdate - Gap elements to update.

gapToWord

protected static void gapToWord(org.jdom2.Element gapElement,
             CorrectedWord correctedWord)

Replace a gap element by a word element.

Parameters:

gapElement - Gap element to replace.

correctedWord - Word to replace gap element.

deleteGapElement

protected static void deleteGapElement(org.jdom2.Element gapElement)

Delete a gap element.

Parameters:

gapElement - Gap element to delete.

deleteWordElement

protected static void deleteWordElement(org.jdom2.Element wordElement,
                     java.util.List<java.lang.String> sortedWordIDs)

Delete a word element.

Parameters:

wordElement - Word element to delete.

getSortedWordIDs

protected static java.util.List<java.lang.String> getSortedWordIDs()

Get sorted word IDs.

Returns:

List of sorted word IDs.

deleteWordOld

protected static void deleteWordOld(org.jdom2.Element wordElement,
                 java.util.List<java.lang.String> correctedWordIDs,
                 int i)

Delete a word.

Parameters:

wordElement - Word to delete.

correctedWordIDs - List of corrected word IDs.

i - Index of word in list of word IDs.