Basic Terminologies in NLP
Transformative Tech Leader | Serial Entrepreneur & Machine Learning Engineer Leveraging 3+ years of expertise in Machine Learning and a background in Web Development, I drive innovation through building, mentoring, and educating. Passionate about harnessing AI to solve real-world problems."
Natural Language Processing (NLP) is a field of artificial intelligence that focuses on the interaction between computers and human language. It involves teaching machines to read, interpret, and generate text, enabling them to "understand" and process language the way humans do.
Before diving into advanced concepts, it’s important to grasp some basic terms. These terms lay the groundwork for working with textual data.
Corpus
A corpus is a large collection of text data used for training and evaluating NLP models. It is essentially a dataset of textual information.
Example:
- A collection of 10,000 news articles used to train a text classification model would be considered a corpus.
Document
A document refers to a single piece of text within a corpus. It could be a paragraph, a full article, or a book depending on the context of your analysis.
Example: In a collection of news articles, each individual article is considered a document. If you're analyzing social media data, each tweet or post might be treated as a document.
Documents are important because they allow you to analyze language at a slightly larger scale than individual words.
Vocabulary
The vocabulary of a corpus is the set of unique words that appear across all documents. In NLP, building a vocabulary is one of the first steps, as it defines the scope of words that your model will recognize.
Example: If your corpus consists of the following two documents:
Doc 1: "Python is fun"
Doc 2: "I love Python"
The vocabulary is the set of unique words: {"Python", "is", "fun", "I", "love"}.
Vocabulary size can impact model performance. A large vocabulary gives the model access to a wide range of words, but it can also slow down training. On the other hand, a small vocabulary might limit the model’s ability to understand more complex texts.
Words
Words are the fundamental units of text. In NLP, we often think of words as tokens, which are the smallest pieces of text that we analyze. They are separated by spaces or punctuation in a sentence.
Example: In the sentence, "NLP is fun," the words (tokens) are "NLP," "is," and "fun."
Different languages treat words differently. For example, Chinese doesn’t separate words with spaces, so identifying words involves additional processing.
Putting It All Together:
Let’s summarize how these terms work together in a typical NLP task.
Corpus: The dataset you're working with, such as a collection of reviews, articles, or books.
Document: Each individual review, article, or chapter from that corpus.
Words: The individual tokens that make up each document.
Vocabulary: The collection of all unique words found in the entire corpus.
For example, if you’re building a model to predict the sentiment of movie reviews:
Your corpus might be a dataset of 10,000 reviews.
Each document is a single movie review.
The words in a review are individual units like "good," "bad," "acting."
The vocabulary would consist of all the unique words across the 10,000 reviews, like {"good", "bad", "acting", "plot", "soundtrack"}.
Practical Exercise:
Let’s try a simple exercise to solidify your understanding.
Task:
Given the following text: "I love machine learning. Machine learning is powerful."
Identify the corpus.
Break it down into documents.
List the words in each document.
Build the vocabulary.
