edu.northwestern.at.morphadorner.corpuslinguistics.textsegmenter.texttiling

Class TextTiling

java.lang.Object

edu.northwestern.at.morphadorner.corpuslinguistics.textsegmenter.texttiling.TextTiling

public class TextTiling
extends java.lang.Object

An implementation of Marti Hearst's text tiling algorithm.

Author:

Freddy Choi, Philip R. Burns. Modified for integration in MorphAdorner.

Use of this code is free for academic, education, research and other non-profit making uses only.

Field Summary

Fields

Modifier and Type	Field and Description
protected RawText	C Collection for segmentation.
protected double[]	depth_score Depth scores.
protected static java.io.PrintStream	printStream Wrapper for printStream to allow utf-8 output.
protected int	s Step size.
protected StopWords	S Stopwords for noise reduction.
protected java.util.List<java.lang.Integer>	segmentation Segment boundaries.
protected double[]	sim_score Similarity scores and the corresponding locations.
protected int[]	site_loc
protected java.util.Map<java.lang.String,java.lang.String>	stemOf Token -> stem dictionary
protected int	w Size of sliding window.

Constructor Summary

Constructors

Constructor and Description
TextTiling() Create text tiler.
TextTiling(RawText c, StopWords s) Create text tiler from text collection and stop word set.

Method Summary

Methods

Modifier and Type	Method and Description
protected void	blockAdd(java.lang.String term, java.util.Map<java.lang.String,java.lang.Integer> B) Add a term to a block Creation date: (07/12/99 01:41:24)
protected double	blockCosine(java.util.Map<java.lang.String,java.lang.Integer> B1, java.util.Map<java.lang.String,java.lang.Integer> B2) Compute the cosine similarity measure for two blocks Creation date: (07/12/99 01:49:16)
protected void	blockRemove(java.lang.String term, java.util.Map<java.lang.String,java.lang.Integer> B) Remove a term from the block.
void	boundaryIdentification() Identify the boundaries
void	depthScore() Compute depth score after applying similarityDetermination()
java.util.List<java.lang.Integer>	getSegmentation() Return segmentation list.
java.util.Map<java.lang.String,java.lang.String>	getStemOf() Return stem dictionary.
StopWords	getStopWords() Return stop words.
protected boolean	include(int i) Decide whether word i is worth using as feature for segmentation.
protected void	preprocess() Perform some preprocessing to save execution time
void	setStepSize(int stepSize)
void	setWindowSize(int windowSize)
void	similarityDetermination() Compute the similarity score.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

printStream

protected static java.io.PrintStream printStream

Wrapper for printStream to allow utf-8 output.

w

protected int w

Size of sliding window.

s

protected int s

Step size.

C

protected RawText C

Collection for segmentation.

S

protected StopWords S

Stopwords for noise reduction.

stemOf

protected java.util.Map<java.lang.String,java.lang.String> stemOf

Token -> stem dictionary

sim_score

protected double[] sim_score

Similarity scores and the corresponding locations.

site_loc

protected int[] site_loc

depth_score

protected double[] depth_score

Depth scores.

segmentation

protected java.util.List<java.lang.Integer> segmentation

Segment boundaries.

Constructor Detail

TextTiling

public TextTiling()

Create text tiler.

TextTiling

public TextTiling(RawText c,
          StopWords s)

Create text tiler from text collection and stop word set.

Parameters:

c - Text collection to segment.

s - Stop word set.

Method Detail

getSegmentation

public java.util.List<java.lang.Integer> getSegmentation()

Return segmentation list.

Returns:

segmentation as a list of integer sentence indices.

getStemOf

public java.util.Map<java.lang.String,java.lang.String> getStemOf()

Return stem dictionary.

Returns:

stem dictionary.

getStopWords

public StopWords getStopWords()

Return stop words.

Returns:

Stop words.

blockAdd

protected void blockAdd(java.lang.String term,
            java.util.Map<java.lang.String,java.lang.Integer> B)

Add a term to a block Creation date: (07/12/99 01:41:24)

Parameters:

term - java.lang.String

B - java.util.HashMap

setWindowSize

public void setWindowSize(int windowSize)

setStepSize

public void setStepSize(int stepSize)

blockCosine

protected double blockCosine(java.util.Map<java.lang.String,java.lang.Integer> B1,
                 java.util.Map<java.lang.String,java.lang.Integer> B2)

Compute the cosine similarity measure for two blocks Creation date: (07/12/99 01:49:16)

Parameters:

B1 - java.util.HashMap

B2 - java.util.HashMap

Returns:

double

blockRemove

protected void blockRemove(java.lang.String term,
               java.util.Map<java.lang.String,java.lang.Integer> B)

Remove a term from the block.

Parameters:

term - java.lang.String

B - java.util.HashMap

boundaryIdentification

public void boundaryIdentification()

Identify the boundaries

depthScore

public void depthScore()

Compute depth score after applying similarityDetermination()

include

protected boolean include(int i)

Decide whether word i is worth using as feature for segmentation.

Parameters:

i - int

Returns:

boolean

preprocess

protected void preprocess()

Perform some preprocessing to save execution time

similarityDetermination

public void similarityDetermination()

Compute the similarity score.