Regular Expressions: Extracting and Cleaning Text Data

A lot of real data is not in tidy columns. Customer feedback, support tickets, log files, product descriptions and email bodies are all free text, and hidden inside them are the facts you want: an order ID, a phone number, a refund reference, "42 minutes". Regular expressions (regex) are a compact language for describing text patterns, and Python's built-in re module lets you search for, extract and replace anything that matches a pattern.

Regex looks cryptic at first because it is written in dense symbols. The secret is that there are only a handful of building blocks (character classes, quantifiers, groups and anchors) and you combine them like Lego. This lesson introduces them one at a time on FreshBasket's feedback.txt, a file of 40 customer comments in the format date | order | rating:N | text. We parse it into a DataFrame, pull out emails, phone numbers and minutes, mask personal data before sharing, and classify comments into themes.

Two warnings up front. Always write patterns as raw strings (r"..."), and test them on real examples, because regex fails silently by matching too much or too little.

Worked examples

Example 1: Read the raw lines

Question: What does the feedback file look like?

from pathlib import Path

lines = Path("feedback.txt").read_text(encoding="utf-8").splitlines()
print(len(lines))
for line in lines[:3]:
    print(line)

40
2025-12-12 | O04344 | rating:3 | Late delivery by almost 50 minutes. Very disappointed. Call me on +91 92699 33648.
2026-01-05 | O00290 | rating:2 | Late delivery by almost 12 minutes. Very disappointed. Call me on +91 95177 90844.
2025-09-06 | O03999 | rating:3 | Rider could not find my address. Reach me at karan.rao481@example.com.

How it works: read_text().splitlines() returns the file as a list of lines without the trailing newline. Each line packs four fields separated by | . We could split on that separator, but the last field has free text with phone numbers and emails inside, which is where regex shines.

Reading the result: The file has 40 comments. The first has a date, an order ID, a rating and a free-text message.

Example 2: Search for a pattern

Question: Which comments mention a delivery time in minutes?

import re

pattern = r"\d+ minutes"
hits = [l for l in lines if re.search(pattern, l)]
print(len(hits))
print(re.search(pattern, hits[0]))
print(re.search(pattern, hits[0]).group())

8
<re.Match object; span=(57, 67), match='50 minutes'>
50 minutes

How it works: re.search(pattern, text) scans the text and returns a match object for the first match, or None if there is none. \d means "one digit" and + means "one or more", so \d+ minutes matches things like 42 minutes. match.group() gives the matched text. Prefix patterns with r so Python does not interpret \d as a string escape.

Reading the result: 8 of 40 comments mention minutes. In the first of them the pattern matched the text "50 minutes".

Example 3: Pull out the number with a group

Question: How do we get just the number of minutes, not the whole phrase?

minutes = []
for line in lines:
    m = re.search(r"(\d+) minutes", line)
    if m:
        minutes.append(int(m.group(1)))

print(minutes)
print(re.findall(r"(\d+) minutes", " ".join(lines)) == [str(x) for x in minutes])

[50, 12, 42, 53, 14, 43, 41, 32]
True

How it works: Parentheses create a group. The match as a whole is group(0), and the first parenthesised part is group(1), here just the digits. When a pattern contains one group, re.findall returns the group's text for every match in one call, which is often the neatest way to extract many values from a string.

Reading the result: We extracted 8 values, from 12 to 53 minutes, with a mean of 35.9.

Example 4: Parse every line with named groups

Question: Can we turn the whole file into a DataFrame?

import pandas as pd

line_re = re.compile(
    r"^(?P<date>\d{4}-\d{2}-\d{2}) \| (?P<order_id>O\d{5}) \| rating:(?P<rating>[1-5]) \| (?P<text>.*)$"
)

rows = []
for line in lines:
    m = line_re.match(line)
    if m is None:
        print("could not parse:", line)
        continue
    rows.append(m.groupdict())

fb = pd.DataFrame(rows)
fb["date"] = pd.to_datetime(fb["date"])
fb["rating"] = fb["rating"].astype(int)
print(fb.dtypes.to_dict())
print(fb.head(3))

{'date': dtype('<M8[us]'), 'order_id': <StringDtype(storage='python', na_value=nan)>, 'rating': dtype('int64'), 'text': <StringDtype(storage='python', na_value=nan)>}
        date order_id  rating                                               text
0 2025-12-12   O04344       3  Late delivery by almost 50 minutes. Very disap...
1 2026-01-05   O00290       2  Late delivery by almost 12 minutes. Very disap...
2 2025-09-06   O03999       3  Rider could not find my address. Reach me at k...

How it works: re.compile prepares the pattern once so you can reuse it. ^ and $ anchor the match to the start and end of the line. \d{4} means exactly four digits and [1-5] means one character from 1 to 5. (?P<name>...) gives a group a name, and groupdict() returns all named groups as a dictionary, ready for a DataFrame. Printing the lines that fail to parse is a quick data-quality check, since unmatched lines tell you the pattern or the data has a surprise.

Reading the result: All 40 lines parsed, covering dates from 2025-01-22 to 2026-01-27. The average rating is 3.05 out of 5.

Example 5: Extract emails and phone numbers

Question: How many comments contain personal contact details?

email_re = re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+")
phone_re = re.compile(r"\+91 \d{5} \d{5}")

fb["emails"] = fb["text"].apply(email_re.findall)
fb["phones"] = fb["text"].apply(phone_re.findall)

print("with email:", (fb["emails"].str.len() > 0).sum())
print("with phone:", (fb["phones"].str.len() > 0).sum())
print("with both :", ((fb["emails"].str.len() > 0) & (fb["phones"].str.len() > 0)).sum())
print(fb.loc[fb["emails"].str.len() > 0, "emails"].iloc[0])

with email: 20
with phone: 12
with both : 5
['karan.rao481@example.com.']

How it works: The email pattern says: one or more word characters, dots, plus or hyphen, then @, then a domain, a dot and a suffix. It is a practical pattern, not a full validation of the email standard (which is famously complex). The phone pattern describes the Indian format +91 followed by two five-digit groups. Series.apply(func) runs the compiled pattern's findall on every comment and stores a list per row.

Reading the result: 20 comments contain an email address and 12 a phone number; 5 contain both. Anyone sharing this data needs to remove them first.

Example 6: Mask personal data with re.sub

Question: Create a version that is safe to share.

def mask(text):
    text = email_re.sub("[email removed]", text)
    text = phone_re.sub("[phone removed]", text)
    return text

fb["safe_text"] = fb["text"].apply(mask)
idx = fb.index[fb["emails"].str.len() > 0][0]
print(fb.loc[idx, "text"])
print(fb.loc[idx, "safe_text"])

leftover = fb["safe_text"].apply(lambda t: bool(email_re.search(t) or phone_re.search(t))).sum()
print("rows still containing contact details:", leftover)

Rider could not find my address. Reach me at karan.rao481@example.com.
Rider could not find my address. Reach me at [email removed]
rows still containing contact details: 0

How it works: pattern.sub(replacement, text) replaces every match. Always verify the result, here by re-searching the masked text for leftovers: privacy tools that are not tested are not privacy tools. (If the data is sensitive, regex masking is a first line of defence, not a guarantee: names and addresses need other methods.)

Reading the result: After masking, 0 rows still contain an email or phone. The sentence structure is preserved, so the text remains useful for analysis.

Example 7: Greedy versus lazy

Question: Why does a pattern sometimes match too much?

html = "<b>Paneer</b> and <b>Curd</b>"

print(re.findall(r"<b>.*</b>", html))      # greedy: grabs as much as possible
print(re.findall(r"<b>.*?</b>", html))     # lazy: stops at the first closing tag
print(re.findall(r"<b>(.*?)</b>", html))   # lazy with a group: just the text

['<b>Paneer</b> and <b>Curd</b>']
['<b>Paneer</b>', '<b>Curd</b>']
['Paneer', 'Curd']

How it works: Quantifiers such as * and + are greedy: they match as much text as they can. In the first pattern .* runs all the way to the last </b>, swallowing both items. Adding ? makes the quantifier lazy so it stops at the first possible end. Greedy over-matching is the single most common regex bug, so test patterns on text with more than one occurrence.

Reading the result: The greedy pattern returns 1 match (everything in one chunk), while the lazy one finds 2 separate items: ['Paneer', 'Curd'].

Example 8: Flags, alternation and verbose patterns

Question: Classify comments into themes, ignoring upper and lower case.

themes = {
    "late delivery": r"late|slow|took \d+ minutes",
    "wrong or missing item": r"wrong item|missing item",
    "quality": r"not fresh|quality|stale",
    "address problem": r"find my address|address",
    "praise": r"super quick|awesome|great|excellent|love",
}

def classify(text):
    return [name for name, pat in themes.items() if re.search(pat, text, flags=re.IGNORECASE)]

fb["themes"] = fb["text"].apply(classify)
exploded = fb.explode("themes")
summary = exploded.groupby("themes").agg(comments=("order_id", "count"), avg_rating=("rating", "mean")).sort_values("comments", ascending=False)
print(summary.round(2))
print("unclassified:", int((fb["themes"].str.len() == 0).sum()))

                       comments  avg_rating
themes                                     
praise                       11        4.64
wrong or missing item        11        1.91
address problem               9        2.78
quality                       7        2.71
late delivery                 4        3.50
unclassified: 0

How it works: The | symbol means "or". re.IGNORECASE makes matching case-insensitive. A comment can match several themes, so classify returns a list and explode turns each list item into its own row for counting. Always look at what was not classified, which is a quick measure of how well your rules cover the data.

Reading the result: The most common theme is "praise" with 11 comments. Unclassified comments: 0. Average rating is lowest for "wrong or missing item" (1.91) and highest for "praise" (4.64).

Example 9: Extract product names and link to the catalogue

Question: Which products do customers complain about?

products = pd.read_csv("products.csv")

complaint_re = re.compile(
    r"""
    (?:The\ (?P<a>.+?)\ was\ not\ fresh)     # 'The X was not fresh'
    |
    (?:instead\ of\ (?P<b>.+?)\.)              # 'instead of X.'
    """,
    re.VERBOSE,
)

found = []
for text in fb["text"]:
    m = complaint_re.search(text)
    if m:
        found.append(m.group("a") or m.group("b"))

named = pd.Series(found, name="product_name").to_frame().merge(products[["product_name", "category"]], on="product_name", how="left")
print(len(found), "complaints name a product")
print(named["category"].value_counts(dropna=False).to_dict())
print("unmatched names:", int(named["category"].isna().sum()))

10 complaints name a product
{'Household': 2, 'Dairy': 2, 'Staples': 2, 'Personal Care': 2, 'Beverages': 1, 'Fruits & Veg': 1}
unmatched names: 0

How it works: re.VERBOSE lets you spread a pattern over several lines with comments (whitespace inside the pattern is ignored, so literal spaces are escaped). (?:...) is a group that does not capture. The alternation | handles two sentence templates, and exactly one of the named groups is filled for each match, so m.group("a") or m.group("b") picks it. We then merge with the product table to see which categories draw complaints, and count the names that failed to match as a final check.

Reading the result: 10 comments name a specific product. The categories mentioned are {'Household': 2, 'Dairy': 2, 'Staples': 2, 'Personal Care': 2, 'Beverages': 1, 'Fruits & Veg': 1}, with 0 names not found in the catalogue.

Example 10: Compare the text with the system record

Question: Do the minutes customers mention match the recorded delivery times?

orders = pd.read_csv("orders.csv")
fb["claimed_min"] = fb["text"].str.extract(r"(\d+) minutes")[0].astype(float)

check = fb.dropna(subset=["claimed_min"]).merge(orders[["order_id", "delivery_minutes"]], on="order_id")
check["gap"] = check["claimed_min"] - check["delivery_minutes"]
print(check[["order_id", "claimed_min", "delivery_minutes", "gap"]])
print("exact matches:", int((check["gap"] == 0).sum()), "of", len(check))
print("mean absolute gap:", round(check["gap"].abs().mean(), 1))

  order_id  claimed_min  delivery_minutes   gap
0   O04344         50.0              24.0  26.0
1   O00290         12.0              23.0 -11.0
2   O01543         42.0              27.0  15.0
3   O05414         53.0              43.0  10.0
4   O02732         14.0              18.0  -4.0
5   O00327         43.0              26.0  17.0
6   O01097         41.0              27.0  14.0
7   O05388         32.0              27.0   5.0
exact matches: 0 of 8
mean absolute gap: 12.8

How it works: Series.str.extract(pattern) is the pandas shortcut: it applies a regex with a group to each element and returns a DataFrame of the groups (NaN where there is no match). We then join to the orders table by order ID, a typical use of extracted text: turn it into a key or a number and combine it with structured data.

Reading the result: Of 8 comments with a time, 0 match the recorded delivery time exactly, and the mean absolute gap is 12.8 minutes. What customers say and what the system recorded often differ, so treat free-text numbers as perception unless you can verify them.

Common mistakes

1. Forgetting the raw-string prefix, so "\d" or "\b" is misread by Python before re sees it. Write r"\d+".

2. Greedy over-matching with .*. Use .*? or a more specific class such as [^<]*.

3. Unescaped special characters: ., +, ?, (, ), [, $ have special meanings. To match them literally, use a backslash or re.escape(text).

4. Using re.match when you meant re.search. match only looks at the start of the string.

5. Over-trusting a pattern after testing on one example. Test it on lines that should match and lines that should not.

re.compile("(unclosed group")

Traceback (most recent call last):
  File "example.py", line 1, in <module>
    re.compile("(unclosed group")
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "example.py", line 289, in compile
    return _compile(pattern, flags)
  File "example.py", line 350, in _compile
    p = _compiler.compile(pattern, flags)
  File "example.py", line 748, in compile
    p = _parser.parse(p, flags)
  File "example.py", line 980, in parse
    p = _parse_sub(source, state, flags & SRE_FLAG_VERBOSE, 0)
  File "example.py", line 459, in _parse_sub
    itemsappend(_parse(source, state, verbose, nested + 1,
                ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                       not nested and not items))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "example.py", line 865, in _parse
    raise source.error("missing ), unterminated subpattern",
                       source.tell() - start)
re.PatternError: missing ), unterminated subpattern at position 0

Practice questions

Practice 1 (Easy): Extract reference numbers

Some comments contain a refund reference such as REF-43163. Extract all references and count them.

refs = [r for t in fb["text"] for r in re.findall(r"REF-\d+", t)]
print(len(refs), refs[:3])

6 ['REF-43163', 'REF-99185', 'REF-56532']

Practice 2 (Medium): Find comments with a rating mismatch

Some comments say "Rated 5 stars" in the text. Find those whose structured rating column disagrees.

claimed = fb["text"].str.extract(r"Rated (\d) stars")[0].astype(float)
mismatch = fb[claimed.notna() & (claimed != fb["rating"])]
print(int(claimed.notna().sum()), len(mismatch))
print(mismatch[["order_id", "rating"]])

4 1
   order_id  rating
11   O04376       4

Practice 3 (Hard): Normalise phone numbers

Phone numbers in the wild come as +91 92699 33648, 9269933648, +919269933648 and 92699-33648. Write one function that accepts all four formats and returns +91 XXXXX XXXXX, and test it.

def normalise_phone(raw):
    digits = re.sub(r"\D", "", raw)           # keep only digits
    if len(digits) == 12 and digits.startswith("91"):
        digits = digits[2:]
    if not re.fullmatch(r"[6-9]\d{9}", digits):
        return None
    return f"+91 {digits[:5]} {digits[5:]}"

tests = ["+91 92699 33648", "9269933648", "+919269933648", "92699-33648", "12345", "+91 12345 67890"]
results = [normalise_phone(t) for t in tests]
for t, r_ in zip(tests, results):
    print(f"{t:<18} -> {r_}")

+91 92699 33648    -> +91 92699 33648
9269933648         -> +91 92699 33648
+919269933648      -> +91 92699 33648
92699-33648        -> +91 92699 33648
12345              -> None
+91 12345 67890    -> None

Key takeaways

• Write patterns as raw strings and test them on both matching and non-matching examples.
• search finds the first match anywhere, match only at the start, fullmatch requires the whole string, findall returns all.
• Groups extract parts; named groups plus groupdict() turn lines into DataFrame rows.
• Quantifiers are greedy by default; add ? for lazy matching.
• re.sub replaces matches (including with a function); always verify masking by searching the output again.
• In pandas, str.extract, str.contains and str.replace apply regex to whole columns.