Web Scraping with requests and BeautifulSoup

Sometimes the data you need has no API and no download button: it is just on a web page. A competitor's price list, a government notice board, a product catalogue or a table of delivery fees. Web scraping means fetching the page's HTML and pulling out the pieces you want with code. Python makes this easy with requests to download the page and BeautifulSoup (the bs4 package) to navigate the HTML.

Scraping is a last resort, not a first choice. If a site offers an API or a data export, use that, because pages change without warning and scrapers break. Scraping also comes with responsibilities: check a site's terms of use and robots.txt, identify your script honestly, do not hammer servers, and never scrape personal data you have no right to. We practise on a safe, local page: the FreshBasket shop page served by mock_server.py (start it with python mock_server.py in a second terminal), so you can experiment freely with no ethical or legal worries.

By the end you will be able to turn a messy HTML page into a clean DataFrame and write a scraper that fails loudly when the page changes.

Worked examples

Example 1: Download a page and look at the HTML

Question: What does the shop page contain?

import requests
from bs4 import BeautifulSoup

BASE = "http://127.0.0.1:8000"
resp = requests.get(f"{BASE}/shop", timeout=5)
print(resp.status_code, resp.headers["Content-Type"])

soup = BeautifulSoup(resp.text, "html.parser")
print(soup.title.text)
print(soup.h1.text)
print(len(resp.text), "characters of HTML")
print(soup.find("div", class_="card").prettify()[:300])

200 text/html; charset=utf-8
FreshBasket Shop
FreshBasket Shop
3340 characters of HTML
<div class="card" data-id="P001">
 <h3 class="name">
  Banana Robusta
 </h3>
 <p class="brand">
  Farm Fresh
 </p>
 <span class="category">
  Fruits &amp; Veg
 </span>
 <span class="price">
  ₹48
 </span>
 <span class="stock">
  In stock
 </span>
 <a class="more" href="/shop/P001">
  Details
 </a>
<

How it works: resp.text is the HTML as a string. BeautifulSoup(html, "html.parser") parses it into a tree you can search. Tags are accessed as attributes (soup.title), and .text gives the text inside. prettify() re-indents a fragment, which is helpful for reading unfamiliar markup. (The faster lxml parser also works if installed, but html.parser ships with Python.)

Reading the result: The page title is "FreshBasket Shop", the main heading is "FreshBasket Shop", and the page is 3340 characters long. Each product sits in a div with the class card.

Example 2: find and find_all

Question: How many products are listed, and what are their names?

cards = soup.find_all("div", class_="card")
print(len(cards))

first = cards[0]
print(first.find("h3", class_="name").text)
print(first.find("span", class_="price").text)
print(first["data-id"], first.get("data-id"))
print(first.find("a")["href"])

names = [c.find("h3", class_="name").text for c in cards]
print(names[:4])

12
Banana Robusta
₹48
P001 P001
/shop/P001
['Banana Robusta', 'Tomato Hybrid', 'Onion', 'Potato']

How it works: find_all(tag, class_=...) returns a list of every matching element and find returns the first. (class_ has a trailing underscore because class is a Python keyword.) Text comes from .text, and attributes are read like dictionary keys: tag["href"] raises KeyError if missing, while tag.get("href") returns None. A list comprehension over cards collects one field from every product.

Reading the result: The page lists 12 products, starting with ['Banana Robusta', 'Tomato Hybrid', 'Onion']. The first card's data-id attribute is "P001" and its details link goes to /shop/P001.

Example 3: CSS selectors

Question: Is there a shorter way to say "the price inside each card"?

prices = [el.text for el in soup.select("div.card span.price")]
print(prices[:3])

title = soup.select_one("#title").text
print(title)

in_stock_cards = soup.select("div.card:has(span.stock)")
print(len(in_stock_cards))
print(soup.select_one("#catalog > div.card:nth-of-type(2) .name").text)

['₹48', '₹36', '₹42']
FreshBasket Shop
12
Tomato Hybrid

How it works: select takes a CSS selector, the same language used to style web pages. div.card means a div with class card, #title means the element with that id, a space means "inside", > means "direct child" and :nth-of-type(2) picks the second. select returns a list and select_one returns the first match or None. Browser developer tools (right-click, Inspect) can copy a selector for you.

Reading the result: The selector div.card span.price returned 12 prices, such as ['₹48', '₹36', '₹42']. The second product's name is "Tomato Hybrid".

Example 4: Build a DataFrame and clean it

Question: Turn the cards into a table with proper numeric types.

import pandas as pd
import re

rows = []
for c in soup.select("div.card"):
    rows.append({
        "product_id": c["data-id"],
        "name": c.select_one(".name").text,
        "brand": c.select_one(".brand").text,
        "category": c.select_one(".category").text,
        "price_text": c.select_one(".price").text,
        "stock_text": c.select_one(".stock").text,
    })

scraped = pd.DataFrame(rows)
scraped["price"] = scraped["price_text"].str.replace(r"[^\d.]", "", regex=True).astype(float)
scraped["in_stock"] = scraped["stock_text"].eq("In stock")
print(scraped[["product_id", "name", "price_text", "price", "in_stock"]].head(4))
print(scraped.dtypes.to_dict())

  product_id            name price_text  price  in_stock
0       P001  Banana Robusta        ₹48   48.0      True
1       P002   Tomato Hybrid        ₹36   36.0      True
2       P003           Onion        ₹42   42.0      True
3       P004          Potato        ₹38   38.0      True
{'product_id': <StringDtype(storage='python', na_value=nan)>, 'name': <StringDtype(storage='python', na_value=nan)>, 'brand': <StringDtype(storage='python', na_value=nan)>, 'category': <StringDtype(storage='python', na_value=nan)>, 'price_text': <StringDtype(storage='python', na_value=nan)>, 'stock_text': <StringDtype(storage='python', na_value=nan)>, 'price': dtype('float64'), 'in_stock': dtype('bool')}

How it works: Scraped values are always strings: "₹48" is text, not a number. We strip everything that is not a digit or dot ([^\d.]) with a regex and convert to float, and turn "In stock" / "Out of stock" into a boolean. Keeping the raw text columns next to the cleaned ones makes debugging easy, and you can drop them at the end.

Reading the result: The table has 12 products and prices from ₹25 to ₹420. 2 products are out of stock.

Example 5: Parse an HTML table

Question: The page has a table of delivery fees. How do we read it?

from io import StringIO

fees = pd.read_html(StringIO(resp.text), attrs={"id": "delivery-fees"})[0]
print(fees)

# the manual way, for comparison
manual = [[td.text for td in tr.find_all(["th", "td"])] for tr in soup.select("#delivery-fees tr")]
print(manual)

fees["fee_rupees"] = fees["Fee"].str.replace("₹", "").astype(int)
print(fees["fee_rupees"].sum())

        City  Fee
0  Bengaluru  ₹25
1     Mumbai  ₹30
2      Delhi  ₹30
[['City', 'Fee'], ['Bengaluru', '₹25'], ['Mumbai', '₹30'], ['Delhi', '₹30']]
85

How it works: pd.read_html finds <table> elements and returns a list of DataFrames, the fastest route for tabular data. We pass attrs to pick the table by id and wrap the text in StringIO (pandas expects a file-like object). The manual approach loops over rows (tr) and cells (th or td), which you need when a table is irregular. Either way, convert the fee text to a number before using it.

Reading the result: The fee table has 3 cities and the fees are {'Bengaluru': 25, 'Mumbai': 30, 'Delhi': 30}. The manual parse returned the same 4 rows including the header.

Example 6: Following links safely

Question: Each product has a "Details" link. Do they work?

from urllib.parse import urljoin

links = [urljoin(BASE + "/shop", a["href"]) for a in soup.select("a.more")]
print(links[:2])

statuses = {}
for url in links[:3]:
    r = requests.get(url, timeout=5)
    statuses[url.rsplit("/", 1)[1]] = r.status_code
print(statuses)

['http://127.0.0.1:8000/shop/P001', 'http://127.0.0.1:8000/shop/P002']
{'P001': 404, 'P002': 404, 'P003': 404}

How it works: Links in HTML are often relative (/shop/P001), and urljoin(base, href) builds the full URL correctly whether the link is relative, absolute or even uses ... Then we fetch each linked page and look at the status code. On a real site, always check for 404s and other failures before parsing: a scraper that assumes every page exists will crash in the middle of a long run.

Reading the result: The first link resolves to http://127.0.0.1:8000/shop/P001. Fetching the first three detail pages returned {'P001': 404, 'P002': 404, 'P003': 404}, so this demo shop has no product detail pages and a real scraper would need to skip them.

Example 7: Handling missing elements

Question: What if a field is missing from some cards?

messy = BeautifulSoup(
    '<div class="card"><h3 class="name">Paneer 200g</h3><span class="price">₹95</span></div>'
    '<div class="card"><h3 class="name">Curd 400g</h3></div>',
    "html.parser",
)

def text_or_none(tag, selector):
    found = tag.select_one(selector)
    return found.get_text(strip=True) if found else None

parsed = [{"name": text_or_none(c, ".name"), "price": text_or_none(c, ".price")} for c in messy.select(".card")]
print(parsed)

try:
    messy.select(".card")[1].select_one(".price").text
except AttributeError as e:
    print("AttributeError:", e)

[{'name': 'Paneer 200g', 'price': '₹95'}, {'name': 'Curd 400g', 'price': None}]
AttributeError: 'NoneType' object has no attribute 'text'

How it works: select_one and find return None when nothing matches, so calling .text on the result raises AttributeError, one of the most common scraping crashes. A small helper that returns None when an element is absent lets the rest of the data come through, and the missing values show up as NaN in pandas where you can count them. get_text(strip=True) removes surrounding whitespace.

Reading the result: The second card has no price, so its entry is None rather than a crash. Missing fields are data too: count them with df.isna().sum() and decide what to do.

Example 8: Navigating the tree

Question: How do we find an element relative to another one?

html = """
<ul id="menu">
  <li>Fruits <ul><li>Banana</li><li>Apple</li></ul></li>
  <li>Dairy <ul><li>Milk</li><li>Curd</li><li>Paneer</li></ul></li>
</ul>
"""
tree = BeautifulSoup(html, "html.parser")

milk = tree.find("li", string="Milk")
print("parent section:", milk.find_parent("li").contents[0].strip())
print("next sibling   :", milk.find_next_sibling("li").text)
print("all dairy items:", [li.text for li in milk.find_parent("ul").find_all("li")])
print("top-level      :", [li.contents[0].strip() for li in tree.select("#menu > li")])

parent section: Dairy
next sibling   : Curd
all dairy items: ['Milk', 'Curd', 'Paneer']
top-level      : ['Fruits', 'Dairy']

How it works: Elements form a tree: parents, children and siblings. find_parent, find_next_sibling and find_all walk the tree from any starting point, which is useful when the text you can identify (a label such as "Milk") sits near the data you want. .contents lists a tag's children, including bare text.

Reading the result: Starting from "Milk" we found its section "Dairy", the next item "Curd" and all 3 dairy items.

Example 9: Scrape politely

Question: What should a well-behaved scraper do before and during a run?

import time
from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url(f"{BASE}/robots.txt")
rp.read()
print("robots.txt allows /shop:", rp.can_fetch("skillancy-tutorial", f"{BASE}/shop"))

HEADERS = {"User-Agent": "skillancy-tutorial/1.0 (learning project; contact: you@example.com)"}
cache = {}

def polite_get(url, delay=0.2):
    if url in cache:
        return cache[url]
    time.sleep(delay)
    r = requests.get(url, headers=HEADERS, timeout=5)
    cache[url] = r
    return r

a = polite_get(f"{BASE}/shop")
b = polite_get(f"{BASE}/shop")
print(a is b, len(cache))

robots.txt allows /shop: True
True 1

How it works: A polite scraper (1) reads the site's robots.txt, which states what automated tools may fetch (Python's RobotFileParser interprets it for you), (2) identifies itself with an honest User-Agent so the owner can contact you, (3) waits between requests, and (4) caches pages so it never downloads the same one twice. Here the mock server has no robots.txt (a 404), which the parser treats as "everything allowed". Real sites may forbid scraping entirely, and then the answer is to ask for permission or find another source.

Reading the result: The robots check returned True. The second call to polite_get served the page from the cache (True), so the cache holds 1 entry and the server saw one request.

Example 10: Reconcile scraped data with the API

Question: Can we trust the scraped prices?

api = pd.DataFrame(requests.get(f"{BASE}/api/products", params={"per_page": 100}, timeout=5).json()["results"])
check = scraped.merge(api[["product_id", "unit_price", "category"]], on="product_id", suffixes=("_web", "_api"))

price_diff = (check["price"] - check["unit_price"]).abs()
print("rows compared:", len(check))
print("max price difference:", price_diff.max())
print("categories match:", bool((check["category_web"] == check["category_api"]).all()))

rows compared: 12
max price difference: 0.0
categories match: True

How it works: Scraping can silently go wrong: a changed layout, a currency symbol, a promotional price. When a trusted source (an API, a database) exists for even a sample of the data, compare the two. Here we merge on product ID and compare prices and categories. Any non-zero difference deserves investigation before you rely on the scraped data.

Reading the result: Across 12 products the largest price difference is ₹0.00, and the categories all match. The scraper's cleaning logic checks out.

Common mistakes

1. Scraping when an API exists. Check for an API or a data download first.

2. Calling .text on None when an element is missing. Guard with a helper.

3. Forgetting that scraped values are strings. Convert prices, dates and counts explicitly.

4. Ignoring robots.txt, rate limits and terms of use, or hammering a server with rapid-fire requests.

5. Writing brittle selectors like div > div > div:nth-child(3). Prefer ids and meaningful class names, which change less often.

BeautifulSoup("<p>hi</p>", "html.parser").find("table").text

Traceback (most recent call last):
  File "example.py", line 1, in <module>
    BeautifulSoup("<p>hi</p>", "html.parser").find("table").text
AttributeError: 'NoneType' object has no attribute 'text'

Practice questions

Practice 1 (Easy): Out-of-stock products

List the names of the products that are out of stock.

oos = scraped.loc[~scraped["in_stock"], "name"].tolist()
print(oos)

['Apple Shimla', 'Curd 400g']

Practice 2 (Medium): Average price by category

Compute the mean scraped price per category and the number of products in each.

by_cat = scraped.groupby("category")["price"].agg(["count", "mean"]).round(1).sort_values("mean", ascending=False)
print(by_cat)

              count   mean
category                  
Fruits & Veg      8  102.4
Dairy             4   67.5

Practice 3 (Hard): A complete scraper function

Write scrape_shop() that downloads the page, returns a clean DataFrame with product_id, name, price, in_stock, and raises a clear error if no product cards are found (a sign that the page layout changed). Test it on the real page and on broken HTML.

def parse_shop(html):
    soup_ = BeautifulSoup(html, "html.parser")
    cards_ = soup_.select("div.card")
    if not cards_:
        raise ValueError("no product cards found: has the page layout changed?")
    out = pd.DataFrame([{
        "product_id": c["data-id"],
        "name": c.select_one(".name").get_text(strip=True),
        "price": float(re.sub(r"[^\d.]", "", c.select_one(".price").text)),
        "in_stock": c.select_one(".stock").text == "In stock",
    } for c in cards_])
    return out

def scrape_shop():
    r = requests.get(f"{BASE}/shop", headers=HEADERS, timeout=5)
    r.raise_for_status()
    return parse_shop(r.text)

result = scrape_shop()
print(result.shape)
try:
    parse_shop("<html><body><p>Under maintenance</p></body></html>")
except ValueError as e:
    print("caught:", e)

(12, 4)
caught: no product cards found: has the page layout changed?

Key takeaways

• Download with requests, parse with BeautifulSoup(html, "html.parser").
• Use find_all, select and CSS selectors; read text with .text and attributes with tag["attr"].
• Scraped values are strings: clean them into numbers, booleans and dates.
• pd.read_html is the fastest way to read tables.
• Guard against missing elements and 404s, and validate the output.
• Be polite: robots.txt, honest user agent, delays and caching.
• Prefer an API when one exists, and reconcile scraped data with any trusted source.