Spam filter - step 4

Create 3 simple non-adaptive filters, paranoid, naive, and random, and evaluate their quality.

You should think about and write down on a piece of paper:

• How is a spam filter actually used?
• What is the difference (from the implementation standpoint) between a learning filter and a non-learning filter?
• Is there any part which all of the spam filters have in common?
• Is it better to create a spam filter as a function or as a class with methods and properties?
• What are the minimal requirements for such an implementation? What does it have to be able to do, and what inputs and what outputs does it have to have?

Tasks:

• In module simplefilters.py, create 3 classes representing 3 simple filters:
  • NaiveFilter which classifies all the emails as OK,
  • ParanoidFilter which classifies all the emails as SPAM, and
  • RandomFilter which assigns the lables OK and SPAM randomly.
  • If these 3 filters have some functionality in common, try to extract it into a common ancestor called BaseFilter in module basefilter.py.

Why do we need it?

• These simple filters will demonstrate the skeleton of the filter and will show the parts common to all filters. We will also have some baseline filters to compare using the functions from step 3.

A filter will be represented by a class with at least 2 public methods: train() and test(). Filters unable to learn from data will probably have the method train() empty. The rest of the class structure is up to you.

Methods train():

Method test():

Create a simple script that computes the quality of a specified filter. The script shall:

• import the class of the chosen filter,
• call method train() on the first dataset,
• call method test() on the second dataset,
• call function compute_quality_for_corpus() for the second corpus,
• print out the quality, and
• remove the file !prediction.txt from the corpus.