edu.northwestern.at.morphadorner.corpuslinguistics.tokenizer

Interface CanTokenizeWhitespace

All Known Implementing Classes:

ICU4JBreakIteratorWordTokenizer

public interface CanTokenizeWhitespace

Interface for tokenizer which can tokenize whitespace.

Method Summary

Methods

Modifier and Type	Method and Description
boolean	getMergeWhitespaceTokens() Get merge whitespace tokens.
boolean	getStoreWhitespaceTokens() Get store whitespace tokens.
void	setMergeWhitespaceTokens(boolean mergeWhitespaceTokens) Set merge whitespace tokens.
void	setStoreWhitespaceTokens(boolean storeWhitespaceTokens) Set store whitespace tokens.

Method Detail

getStoreWhitespaceTokens

boolean getStoreWhitespaceTokens()

Get store whitespace tokens.

setStoreWhitespaceTokens

void setStoreWhitespaceTokens(boolean storeWhitespaceTokens)

Set store whitespace tokens.

getMergeWhitespaceTokens

boolean getMergeWhitespaceTokens()

Get merge whitespace tokens.

setMergeWhitespaceTokens

void setMergeWhitespaceTokens(boolean mergeWhitespaceTokens)

Set merge whitespace tokens.