Day 82: Natural Language Processing — Turn Words Into Numbers

Why Should I Care?

Every time Google autocompletes your search, WhatsApp detects spam, or YouTube generates subtitles — NLP is working behind the scenes. Natural Language Processing is the reason machines can read, understand, and respond to human language. Without NLP there would be no ChatGPT, no Google Translate, no Siri. This is where language meets AI — and it all starts with one simple idea: text must become numbers before any model can touch it.

Core Concept

NLP is like being an exchange student learning Hindi from scratch. On Day 1 everything is noise — random sounds with no meaning. By Day 30 patterns start emerging — you notice certain words appear near happy emotions, others near sad ones. By Day 90 you are fluent — text is no longer noise, it carries meaning. A machine goes through the exact same three stages when trained on millions of sentences. Patterns emerge automatically from the data.

How It Works

The text pipeline has four steps. Raw text comes in first. Then tokenization breaks it into individual words. Then vectorization converts those words into numbers using a word count matrix. Finally those numbers go into the model. The most important step is vectorization — models only understand numbers, never strings. CountVectorizer is the tool that does this job. It builds a vocabulary from all unique words and then represents each sentence as a row of counts showing how many times each word appears.

Text Pipeline:

Raw Text --> Tokenize --> Vectorize --> Numbers --> Model

Example sentences:
  "Rohith loves Python"
  "Python is powerful"
  "Rohith builds AI"

After CountVectorizer — Word Count Matrix:

         ai  bl  is  lo  pw  py  ro  to
Text 1:   0   0   0   1   0   1   1   0   ("loves Python Rohith")
Text 2:   0   0   1   0   1   1   0   0   ("Python is powerful")
Text 3:   1   1   0   0   0   0   1   1   ("Rohith builds AI")

Each sentence = one row of numbers
Each column   = one unique word from the vocabulary
Model can now read and learn from this

Real World Connection

When you type a review on Zomato — "food was amazing, delivery was fast" — NLP converts that sentence into numbers, checks which words appear near positive ratings, and classifies it as positive sentiment. When you get a suspicious message on WhatsApp saying "you won a free iPhone, click here" — NLP detects words that appear near spam patterns and blocks it before you even see it. Every app that reads, filters, or responds to human text is running an NLP pipeline underneath.

Examples

from sklearn.feature_extraction.text import CountVectorizer

# Three simple sentences
texts = [
    "Rohith loves Python",
    "Python is powerful",
    "Rohith builds AI tools"
]

# CountVectorizer builds vocabulary from all words
# and converts sentences into number arrays in one call
vec    = CountVectorizer()
matrix = vec.fit_transform(texts).toarray()

print("— Vocabulary —")
print(vec.get_feature_names_out())

print("\n— Word Count Matrix —")
for i, row in enumerate(matrix):
    print(f"Text {i+1}: {row}")

# OUTPUT:
# — Vocabulary —
# [''ai'' ''builds'' ''is'' ''loves'' ''powerful'' ''python'' ''rohith'' ''tools'']
#
# — Word Count Matrix —
# Text 1: [0 0 0 1 0 1 1 0]   --> loves Python Rohith
# Text 2: [0 0 1 0 1 1 0 0]   --> Python is powerful
# Text 3: [1 1 0 0 0 0 1 1]   --> Rohith builds AI tools
#
# Each row is one sentence as a number array.
# Now a model can train on this data.

Common Mistakes

Two mistakes beginners make when starting with NLP. First, trying to feed raw text strings directly into a model. Second, thinking NLP is only useful for building chatbots and missing its real power.

-- MISTAKE 1: Feeding raw text into a model --

WRONG:
  model.fit(["Rohith loves Python", "Python is powerful"])
  Result: TypeError — models don''t read strings

CORRECT:
  vec    = CountVectorizer()
  X      = vec.fit_transform(texts).toarray()
  model.fit(X, labels)

NOTE: "ML models only understand numbers —
       text must be vectorized first."
RULE: Text must become numbers before any model can touch it.


-- MISTAKE 2: Thinking NLP is just chatbots --

WRONG:
  "NLP = just chatbots"

CORRECT:
  NLP powers: translation, summarization, sentiment analysis,
  search, spam detection, autocomplete, and much more.

NOTE: "Limiting NLP to chatbots means
       missing 90% of its real applications."

Mini Challenge

Quick Quiz

Q1: Why can''t you feed raw text strings directly into a machine learning model? A1: Because ML models only understand numbers — text must be tokenized and vectorized into a number array first. Q2: What does CountVectorizer build from the input sentences? A2: A vocabulary of all unique words, and a word count matrix where each row is a sentence and each column is a word count. Q3: Name three real-world NLP applications beyond chatbots. A3: Translation, spam detection, sentiment analysis, autocomplete, summarization, and search — any three of these are correct.

Bonus Knowledge

CountVectorizer is powerful but has one big weakness — it treats every word as equally important. The word "is" appears in almost every sentence but tells you almost nothing useful. There is a smarter technique called TF-IDF — Term Frequency Inverse Document Frequency — that gives lower scores to words that appear everywhere and higher scores to words that are rare and meaningful. This is what Google uses when ranking search results. Words that appear on every website get ignored. Words unique to a specific page get boosted. You will build on this understanding when you reach embeddings in the next lesson.

Key Takeaways