edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Interface PreTokenizer

All Known Implementing Classes:

AbstractPreTokenizer, DefaultPreTokenizer, EccoPreTokenizer, EEBOPreTokenizer, NoopPreTokenizer

public interface PreTokenizer

Interface for preparing a string for tokenization.

Method Summary

Methods

Modifier and Type	Method and Description
void	close() Close down the preTokenizer.
java.lang.String	pretokenize(java.lang.String s) Prepares a string for tokenization.

Method Detail

pretokenize

java.lang.String pretokenize(java.lang.String s)

Prepares a string for tokenization.

Parameters:

s - The string to prepare for tokenization.

Returns:

The modified string ready for tokenization.

close

void close()

Close down the preTokenizer.