Stemming and Lemmatization

When working with text data in Natural Language Processing (NLP), one of the essential preprocessing steps is to reduce words to their base form. This helps machines more easily process and analyze text by standardizing variations of words like "running" and "ran" to their root form, "run."

Two common techniques for achieving this are stemming and lemmatization. In this tutorial, we'll explore what these techniques are, how they differ, and how to implement them in Python using the Natural Language Toolkit (NLTK) library.

What is Stemming?

Stemming is the process of reducing words to their root form, known as the "stem," by chopping off word endings. Although the resulting "stem" may not always be a valid word, stemming is effective in grouping related word forms that convey the same meaning.

Example:

Words like "running," "runs," and "ran" can all be reduced to the common stem, "run."

There are several algorithms for stemming, each with a unique approach to cutting words down to their stems. Let's explore a few popular ones in NLTK.

Types of Stemmers in NLTK

1. Porter Stemmer

The Porter Stemmer is one of the oldest and most widely used stemming algorithms. It works by iteratively applying a set of rules to remove word suffixes.

from nltk.stem import PorterStemmer

# Initialize the Porter Stemmer
porter = PorterStemmer()

# Example words to stem
words = ["running", "runner", "ran", "runs", "fairly", "generously"]

# Stemming each word
for word in words:
    print(f"{word} -> {porter.stem(word)}")

Output:

running -> run
runner -> runner
ran -> ran
runs -> run
fairly -> fairli
generously -> gener

Note how words like "fairly" and "generously" are reduced to meaningless forms like "fairli" and "gener," highlighting some of the limitations of this approach.

2. Lancaster Stemmer

The Lancaster Stemmer is more aggressive than the Porter Stemmer, often cutting off more of the word, which can lead to over-stemming.

from nltk.stem import LancasterStemmer

# Initialize the Lancaster Stemmer
lancaster = LancasterStemmer()

# Example words to stem
for word in words:
    print(f"{word} -> {lancaster.stem(word)}")

Output:

running -> run
runner -> run
ran -> ran
runs -> run
fairly -> fair
generously -> gen

Here, "runner" is reduced to "run," and "generously" becomes "gen," which may not be desirable in many contexts.

3. Snowball Stemmer

The Snowball Stemmer (also known as Porter2) is an improved version of the Porter Stemmer and is more linguistically accurate. It also supports multiple languages.

from nltk.stem import SnowballStemmer

# Initialize the Snowball Stemmer for English
snowball = SnowballStemmer("english")

# Stemming each word
for word in words:
    print(f"{word} -> {snowball.stem(word)}")

Output:

running -> run
runner -> runner
ran -> ran
runs -> run
fairly -> fair
generously -> generous

The Snowball Stemmer is more precise, as seen with "generously" being reduced to "generous" instead of "gener."

Problems with Stemming

While stemming is quick and efficient, it has some significant limitations:

1. Over-Stemming

Stemming can sometimes be too aggressive, leading to words being chopped down too much and losing their meaning.

words = ["university", "universities"]

for word in words:
    print(f"{word} -> {porter.stem(word)}")

Output:

university -> univers
universities -> univers

Both "university" and "universities" are reduced to "univers," which is not a valid word.

2. Under-Stemming

On the flip side, stemming can sometimes be too conservative, leaving words that should be treated as the same in their original form.

words = ["ran", "running", "runner"]

for word in words:
    print(f"{word} -> {porter.stem(word)}")

Output:

ran -> ran
running -> run
runner -> runner

While "running" is reduced to "run," "ran" and "runner" remain unchanged, resulting in inconsistencies.

What is Lemmatization?

Lemmatization is a more sophisticated approach that reduces words to their lemma—their base or dictionary form. Unlike stemming, lemmatization results in valid words, making it more accurate in many NLP applications.

For example, lemmatization would reduce "running" to "run" and "better" to "good."

Lemmatization requires knowledge of the word's part of speech (POS) to make the right transformation. Let’s see how it works using NLTK’s WordNetLemmatizer.

WordNet Lemmatizer

In NLTK, the WordNetLemmatizer uses the WordNet lexical database to find the correct base form of a word.

from nltk.stem import WordNetLemmatizer
import nltk
nltk.download('wordnet')
# Initialize the WordNet Lemmatizer
lemmatizer = WordNetLemmatizer()

# Example words to lemmatize
words = ["running", "ran", "runner", "studies", "studying", "easily", "fairly"]

for word in words:
    print(f"{word} -> {lemmatizer.lemmatize(word, pos='v')}")

Output:

running -> run
ran -> run
runner -> runner
studies -> study
studying -> study
easily -> easily
fairly -> fairly

Notice how "running" and "ran" are correctly reduced to "run," while "fairly" and "easily" remain unchanged since their base forms are already correct.

Using POS Tags in Lemmatization

For better accuracy, it's important to specify the correct part of speech (POS) when lemmatizing words.

word = "better"

# Lemmatizing as an adjective
print(f"better (adjective) -> {lemmatizer.lemmatize(word, pos='a')}")

# Lemmatizing as an adverb
print(f"better (adverb) -> {lemmatizer.lemmatize(word, pos='r')}")

Output:

better (adjective) -> good
better (adverb) -> better

By specifying the POS, the lemmatizer can return the appropriate base form.

Real-World Applications of Lemmatization

Lemmatization is particularly useful in tasks like:

• Search Engines: It helps match different forms of a word to ensure relevant search results.

• Sentiment Analysis: Lemmatization captures the underlying meaning of words, improving sentiment prediction.

• Machine Translation: Lemmatization simplifies translation by reducing words to their base forms.

• Text Summarization: It ensures that different forms of a word are treated the same, resulting in more accurate summaries.

Differences Between Stemming and Lemmatization

Practice Exercises

Now that you’ve learned the basics of stemming and lemmatization, try these exercises:

1. Porter Stemmer: Stem the words "nationalities," "running," and "jumping."

2. Snowball Stemmer: Use the Snowball Stemmer on the sentence: "He was jumping and running at the park."

3. WordNet Lemmatizer: Lemmatize the sentence: "The studies were focusing on running and jumping."

These exercises will solidify your understanding of how to apply these techniques in real-world NLP tasks. While stemming is faster, lemmatization offers greater accuracy, especially for complex language structures.