Adorning a string

Let's extend the example of adorning a string with parts of speech to add lemma forms and standardized spellings for each word in the string.

Creating a default lemmatizer and spelling standardizer

We will use the default English lemmatizer and the default spelling standardizer.

                                //  Get the default English
                                //  lemmatizer.
        Lemmatizer lemmatizer       =  new DefaultLemmatizer();
                                //  Get the default spelling
                                //  standardizer.
        SpellingStandardizer standardizer   =
            new DefaultSpellingStandardizer();

Adding lemmata and standardized spellings to the output

The process of adding parts of the speech is the same as in PosTagString. We call two new auxiliary methods to determine the lemmata and standard spelling for each part-of-speech tagged spelling.

            for ( int j = 0 ; j < sentence.size() ; j++ )
            {
                AdornedWord adornedWord = sentence.get( j );
                                //  Get the standard spelling
                                //  given the original spelling
                                //  and part of speech.
                setStandardSpelling
                (
                    adornedWord ,
                    standardizer ,
                    partOfSpeechTags
                );
                                //  Set the lemma.
                setLemma
                (
                    adornedWord ,
                    wordLexicon ,
                    lemmatizer ,
                    partOfSpeechTags ,
                    spellingTokenizer
                );
                                //  Display the adornments.
                System.out.println
                (
                    StringUtils.rpad( ( j + 1 ) + "" , 3  ) + ": " +
                    StringUtils.rpad( adornedWord.getSpelling() , 20 ) +
                    StringUtils.rpad(
                        adornedWord.getPartsOfSpeech() , 8 ) +
                    StringUtils.rpad(
                        adornedWord.getStandardSpelling() , 20 ) +
                    adornedWord.getLemmata()
                );
            }

Getting the lemma form

We start by setting the lemma form to the spelling. If the spelling belongs to a word class which should not be further lemmatized, we do nothing further. We test for this by checking if the lemmatization class for the spelling's associated part of speech tag is "none" or if the language specific lemmatizer indicates that tag should not be lemmatized.

If the spelling should be lemmatized, we next check if there are multiple parts of speech in the spelling. If so, we try to find the lemma form for each part separately, and join them into a compound lemma, separating the individual pieces with the lemma form separator character. If the spelling has only a single part of speech, we find the lemma form that best fits the combination of spelling and part of speech.

    /** Get lemma for a word.
     *
     *  @param  adornedWord         The adorned word.
     *  @param  lexicon             The word lexicon.
     *  @param  lemmatizer          The lemmatizer.
     *  @param  partOfSpeechTags    The part of speech tags.
     *  @param  spellingTokenizer   Tokenizer for spelling.
     *
     *  <p>
     *  On output, sets the lemma field of the adorned word
     *  We look in the word lexicon first for the lemma.
     *  If the lexicon does not contain the lemma, we
     *  use the lemmatizer.
     *  </p>
     */
    public static void setLemma
    (
        AdornedWord adornedWord  ,
        Lexicon lexicon ,
        Lemmatizer lemmatizer ,
        PartOfSpeechTags partOfSpeechTags ,
        WordTokenizer spellingTokenizer
    )
    {
        String spelling     = adornedWord.getSpelling();
        String partOfSpeech = adornedWord.getPartsOfSpeech();
        String lemmata      = spelling;
                                //  Get lemmatization word class
                                //  for part of speech.
        String lemmaClass   =
            partOfSpeechTags.getLemmaWordClass( partOfSpeech );
                                //  Do not lemmatize words which
                                //  should not be lemmatized,
                                //  including proper names.
        if  (   lemmatizer.cantLemmatize( spelling ) ||
                lemmaClass.equals( "none" )
            )
        {
        }
        else
        {
                                //  Try compound word exceptions
                                //  list first.
            lemmata = lemmatizer.lemmatize( spelling , "compound" );
                                //  If lemma not found, keep trying.
            if ( lemmata.equals( spelling ) )
            {
                                //  Extract individual word parts.
                                //  May be more than one for a
                                //  contraction.
                List wordList   =
                    spellingTokenizer.extractWords( spelling );
                                //  If just one word part,
                                //  get its lemma.
                if  (   !partOfSpeechTags.isCompoundTag( partOfSpeech ) ||
                        ( wordList.size() == 1 )
                    )
                {
                    if ( lemmaClass.length() == 0 )
                    {
                        lemmata = lemmatizer.lemmatize( spelling );
                    }
                    else
                    {
                        lemmata =
                            lemmatizer.lemmatize( spelling , lemmaClass );
                    }
                }
                                //  More than one word part.
                                //  Get lemma for each part and
                                //  concatenate them with the
                                //  lemma separator to form a
                                //  compound lemma.
                else
                {
                    lemmata             = "";
                    String lemmaPiece   = "";
                    String[] posTags    =
                        partOfSpeechTags.splitTag( partOfSpeech );
                    if ( posTags.length == wordList.size() )
                    {
                        for ( int i = 0 ; i < wordList.size() ; i++ )
                        {
                            String wordPiece    = (String)wordList.get( i );
                            if ( i > 0 )
                            {
                                lemmata = lemmata + lemmaSeparator;
                            }
                            lemmaClass  =
                                partOfSpeechTags.getLemmaWordClass
                                (
                                    posTags[ i ]
                                );
                            lemmaPiece  =
                                lemmatizer.lemmatize
                                (
                                    wordPiece ,
                                    lemmaClass
                                );
                            lemmata = lemmata + lemmaPiece;
                        }
                    }
                }
            }
        }
        adornedWord.setLemmata( lemmata );
    }
}

Getting the standardized spelling

We start by setting the standardized form to the original spelling. If the spelling belongs to a word class which should not be standardized, we do nothing further. This includes spellings that are tagged as numbers, proper nouns, and foreign words.

If the spelling can be standardized, we ask the spelling standardizer to give us the best standardized form it can. We try to match the case of the original spelling in the standardized form. Alternatively we could always set the standardized form to a lower case version, except possibly for proper nouns and adjectives, and the pronoun "I".

    /** Get standard spelling for a word.
     *
     *  @param  adornedWord     The adorned word.
     *  @param  standardizer        The spelling standardizer.
     *  @param  partOfSpeechTags    The part of speech tags.
     *
     *  <p>
     *  On output, sets the standard spelling field of the adorned word
     *  </p>
     */
    public static void setStandardSpelling
    (
        AdornedWord adornedWord  ,
        SpellingStandardizer standardizer ,
        PartOfSpeechTags partOfSpeechTags
    )
    {
                                //  Get the spelling.
        String spelling         = adornedWord.getSpelling();
        String standardSpelling = spelling;
        String partOfSpeech     = adornedWord.getPartsOfSpeech();
                                //  Leave proper nouns alone.
        if ( partOfSpeechTags.isProperNounTag( partOfSpeech ) )
        {
        }
                                //  Leave nouns with internal
                                //  capitals alone.
        else if (   partOfSpeechTags.isNounTag( partOfSpeech )  &&
                    CharUtils.hasInternalCaps( spelling ) )
        {
        }
                                //  Leave foreign words alone.
        else if ( partOfSpeechTags.isForeignWordTag( partOfSpeech ) )
        {
        }
                                //  Leave numbers alone.
        else if ( partOfSpeechTags.isNumberTag( partOfSpeech ) )
        {
        }
                                //  Anything else -- call the
                                //  standardizer on the spelling
                                //  to get the standard spelling.
        else
        {
            standardSpelling    =
                standardizer.standardizeSpelling
                (
                    adornedWord.getSpelling() ,
                    partOfSpeechTags.getMajorWordClass
                    (
                        adornedWord.getPartsOfSpeech()
                    )
                );
                                //  If the standard spelling
                                //  is the same as the original
                                //  spelling except for case,
                                //  use the original spelling.
            if ( standardSpelling.equalsIgnoreCase( spelling ) )
            {
                standardSpelling    = spelling;
            }
        }
                                //  Set the standard spelling.
        adornedWord.setStandardSpelling( standardSpelling );
    }

Putting it altogether

You can peruse the Java source code for AdornAString which puts all the above code together in a runnable sample program. You will also find the source code in the src/edu/northwestern/at/examples/ directory in the MorphAdorner release.