cz.cvut.felk.newsgroup.preprocess
Class StringUtils

java.lang.Object
  extended by cz.cvut.felk.newsgroup.preprocess.StringUtils

public abstract class StringUtils
extends Object

Different string-processing utilities.


Constructor Summary
StringUtils()
           
 
Method Summary
static String processWord(String word)
          Converts a raw word read from a file into a normalized form.
private static String simplify(String word)
          Removes all characters, which are not letters.
private static String stem(String word)
          Returns the "stem" of a word.
static String[] tokenize(String lineWithWords)
          Tokenizer splits a line into words.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

StringUtils

public StringUtils()
Method Detail

tokenize

public static String[] tokenize(String lineWithWords)
Tokenizer splits a line into words.

Parameters:
lineWithWords - a line in a document
Returns:
words on the given line

processWord

public static String processWord(String word)
Converts a raw word read from a file into a normalized form.

Parameters:
word - the word read from a file.
Returns:
normalized form of the given word.

stem

private static String stem(String word)
Returns the "stem" of a word.

A stemmer for English should identify the string "cats" (and possibly "catlike", "catty" etc.) as based on the root "cat", and "stemmer", "stemming", "stemmed" as based on "stem". A stemming algorithm reduces the words "fishing", "fished", "fish", and "fisher" to the root word, "fish".

You can read and consult article on wikipedia.

Parameters:
word - an English word in lower-case
Returns:
the stem of the given word

simplify

private static String simplify(String word)
Removes all characters, which are not letters.

Parameters:
word - a lower-case string
Returns:
the word stripped of non-letter charactes