|
MergeBrillLexicon
merges the contents of a Brill format lexicon with a MorphAdorner
format lexicon into a combined MorphAdorner lexicon.
Usage:
mergebrilllexicon lexicon.lex brilllexicon.txt mergedlexicon.lex
where
- lexicon.lex -- input MorphAdorner format word lexicon.
- brilllexicon.txt -- input Brill format word lexicon to be merged
with MorphAdorner word lexicon.
- mergedlexicon.lex -- output merged lexicon in MorphAdorner format.
A Brill lexicon is a simple utf-8 formatted text file containing
words and their possible part of speech tags. Each word
appears on a separate line. The first token on each line is the
word. The remaining tokens are the potential parts of speech
for the word, separated by blanks or tab characters.
The most commonly occurring part of speech should be the first one listed.
word pos1 pos2 pos3 ...
This type of lexicon was popularized by Eric Brill's part of speech tagger
in the early 1990s.
The Brill entries are merged with the input MorphAdorner lexicon
to produce an updated output MorphAdorner format lexicon. The first
part of speech for each word is added with a could of two, while the
remaining words are added with a count of one. The default
English lemmatizer is used to determine lemmata for the Brill
words. When a word to be added already exists in the MorphAdorner
lexicon, only the new parts of speech are added to the existing
lexicon entry.
Brill lexicons are convenient for adding large lists of words
such as proper and place names, foreign language words, and
so on. Here is a small section of a sample Brill lexicon.
Yellott np1
Yellowby np1
Yellville np1
Yelton np1
Yelverton np1
lieu fw-fr
lieux fw-fr
lire fw-fr
lit fw-fr
literary j
livre fw-fr
livres fw-fr
loi fw-fr
lois fw-fr
loix fw-fr
MorphAdorner also defines an enhanced Brill lexicon which provides the
lemmata for each word's parts of speech.
MergeEnhancedBrillLexicon
allows you to merge an enhanced Brill lexicon into a MorphAdorner lexicon.
|
