Generators, Iterators and Decorators

Two features separate everyday scripts from professional Python: laziness and wrapping. Laziness means handling data one piece at a time instead of loading everything into memory: that is what iterators and generators do, and it is how a program can process a 50 GB log file on a laptop. Wrapping means adding behaviour to a function without editing it: timing it, retrying it, caching its results, logging its calls. That is what decorators do.

Both ideas look exotic at first (what does yield mean? what is @?) but they are built from things you already know: functions, loops and the fact that functions are objects that can be passed around. In this lesson we build up from the iterator protocol behind every for loop, to generator functions, generator pipelines, itertools, chunked reading with pandas, and then to decorators (with and without arguments), functools.lru_cache, and context managers (the machinery behind the with statement).

All examples use FreshBasket data and small, testable functions; there are no hidden magic tricks.

Worked examples

Example 1: What a for loop really does

Question: How does Python loop over a list?

cities = ["Delhi", "Mumbai", "Pune"]

it = iter(cities)
print(next(it))
print(next(it))
print(next(it))
try:
    next(it)
except StopIteration:
    print("StopIteration: nothing left")

print(iter(cities) is cities, iter(it) is it)

Delhi
Mumbai
Pune
StopIteration: nothing left
False True

How it works: A for loop calls iter() on the object to get an iterator, then calls next() repeatedly until the iterator raises StopIteration. An iterator remembers where it is, so each next returns the following item. Lists are iterable (you can get an iterator from them) but are not iterators themselves, while an iterator is its own iterator (iter(it) is it). Once exhausted, an iterator stays empty.

Reading the result: We pulled 3 items one at a time, and the fourth next raised StopIteration, which is exactly how a for loop knows to stop.

Example 2: A generator function

Question: How do we write our own lazy sequence?

def countdown(n):
    print("starting")
    while n > 0:
        yield n
        n -= 1
    print("finished")

gen = countdown(3)
print(type(gen).__name__)
print("nothing has run yet")
print(next(gen))
print(next(gen))
print(list(gen))

generator
nothing has run yet
starting
3
2
finished
[1]

How it works: A function containing yield is a generator function: calling it does not run the body; it returns a generator object. Each next() runs the body until the next yield, hands out that value, and pauses with all local variables intact. The code after the last yield runs on the final next, then StopIteration is raised. The "starting" message appears only when we first ask for a value, which proves the work is lazy.

Reading the result: The generator produced 3 and 2 one at a time; list(gen) then gathered only what was left ([1]), because a generator remembers its position.

Example 3: Memory: list versus generator

Question: How much does laziness save?

import sys

def squares_list(n):
    return [i * i for i in range(n)]

def squares_gen(n):
    for i in range(n):
        yield i * i

n = 1_000_000
as_list = squares_list(n)
as_gen = squares_gen(n)
print("list size     :", sys.getsizeof(as_list) // 1024, "KB")
print("generator size:", sys.getsizeof(as_gen), "bytes")
print(sum(as_list) == sum(squares_gen(n)))

list size     : 8250 KB
generator size: 200 bytes
True

How it works: The list stores a million integers; the generator stores only its current state, a few hundred bytes, however long the sequence. Both give the same sum. Use a list when you need to index, slice or loop several times; use a generator when you only need one pass over a large or endless sequence.

Reading the result: The list occupies about 8250 KB (just the pointers; the integers add more), while the generator takes 200 bytes and the results are identical.

Example 4: A lazy file reader

Question: How do we process a large CSV line by line without loading it?

import csv
from pathlib import Path

def read_orders(path):
    with open(path, newline="", encoding="utf-8") as f:
        for row in csv.DictReader(f):
            yield row

def only_status(rows, status):
    for row in rows:
        if row["order_status"] == status:
            yield row

def to_amounts(rows):
    for row in rows:
        yield row["order_id"], float(row["delivery_minutes"] or 0)

pipeline = to_amounts(only_status(read_orders("orders.csv"), "Cancelled"))
print(next(pipeline))
print(next(pipeline))
print(sum(1 for _ in only_status(read_orders("orders.csv"), "Cancelled")))

('O02449', 0.0)
('O00568', 0.0)
370

How it works: Each stage is a small generator that takes an iterable and yields transformed items. Chained together they form a pipeline: a row is read, filtered and converted before the next row is even read, so memory use stays constant however large the file is. Nothing runs until something asks for a value (next or a loop). Each stage is simple to test on its own.

Reading the result: The first cancelled order is O02449, with a delivery time of 0.0 because cancelled orders have none. Counting them lazily gives 370 cancelled orders, the same count you would get from pandas.

Example 5: Generator expressions and short-circuiting

Question: Can we write a generator in one line, and why would we?

rows = list(read_orders("orders.csv"))

total_minutes = sum(float(r["delivery_minutes"]) for r in rows if r["delivery_minutes"])
has_cancelled = any(r["order_status"] == "Cancelled" for r in rows)
all_have_ids = all(r["order_id"].startswith("O") for r in rows)
first_big = next((r["order_id"] for r in rows if r["discount_pct"] == "20"), None)

print(round(total_minutes), has_cancelled, all_have_ids, first_big)
print(max((r for r in rows if r["delivery_minutes"]), key=lambda r: float(r["delivery_minutes"]))["delivery_minutes"])

133260 True True O04499
54.0

How it works: A generator expression looks like a list comprehension with round brackets: (x for x in items if condition). It produces values lazily, and functions such as sum, any, all, min, max and next consume it directly, with no brackets needed inside a call. any and all short-circuit: they stop as soon as the answer is known, so any on a million rows may inspect only the first few. next(gen, default) fetches the first match or a default.

Reading the result: The sum of delivery minutes is 133260. Cancelled orders exist (True), every ID starts with O (True), the first 20%-discount order is O04499 and the slowest delivery took 54 minutes.

Example 6: itertools

Question: What tools exist for common lazy patterns?

import itertools as it

ids = (r["order_id"] for r in rows)
print(list(it.islice(ids, 3)))                       # first three, lazily

print(list(it.pairwise([10, 15, 12, 20])))           # consecutive pairs
print(list(it.batched(range(7), 3)))                 # chunks of 3

merged = it.chain(["Delhi", "Mumbai"], ["Pune"], ("Kolkata",))
print(list(merged))

by_status = {k: len(list(g)) for k, g in it.groupby(sorted(rows, key=lambda r: r["order_status"]), key=lambda r: r["order_status"])}
print(by_status)

print(list(it.islice(it.cycle("AB"), 5)))

['O02299', 'O04533', 'O00778']
[(10, 15), (15, 12), (12, 20)]
[(0, 1, 2), (3, 4, 5), (6,)]
['Delhi', 'Mumbai', 'Pune', 'Kolkata']
{'Cancelled': 370, 'Delivered': 5391, 'Returned': 239}
['A', 'B', 'A', 'B', 'A']

How it works: islice takes a slice of any iterator (even an infinite one), pairwise yields overlapping pairs (handy for differences), batched cuts a sequence into fixed-size chunks (new in Python 3.12), chain joins several iterables, groupby groups consecutive equal keys (so sort first!) and cycle repeats forever. They all work lazily, so they combine well with generator pipelines.

Reading the result: groupby counted {'Cancelled': 370, 'Delivered': 5391, 'Returned': 239} orders per status, pairwise turned four numbers into 3 pairs and batched split 7 items into 3 chunks, the last one shorter.

Example 7: Read a big CSV in chunks with pandas

Question: How do we aggregate a file that does not fit in memory?

import pandas as pd

totals = {}
n_chunks = 0
for chunk in pd.read_csv("orders.csv", chunksize=1500):
    n_chunks += 1
    for status, count in chunk["order_status"].value_counts().items():
        totals[status] = totals.get(status, 0) + count

print(n_chunks, "chunks")
print(totals)
print(totals == pd.read_csv("orders.csv")["order_status"].value_counts().to_dict())

4 chunks
{'Delivered': 5391, 'Cancelled': 370, 'Returned': 239}
True

How it works: With chunksize, read_csv returns an iterator of DataFrames of that many rows each. You compute a partial result per chunk and combine them (here, adding the counts), so memory use depends on the chunk size and not the file size. This works for sums, counts and other results that combine easily; statistics such as the median need more care. The last line proves the chunked answer equals the all-at-once answer.

Reading the result: The file was processed in 4 chunks of up to 1,500 rows, giving {'Delivered': 5391, 'Cancelled': 370, 'Returned': 239}, identical to loading everything at once.

Example 8: A decorator

Question: How do we add timing and logging to any function without changing it?

import functools

calls = []

def logged(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        result = func(*args, **kwargs)
        calls.append((func.__name__, args, result))
        return result
    return wrapper

@logged
def revenue(price, quantity):
    """Revenue for one line."""
    return price * quantity

print(revenue(120, 3))
print(revenue(48, 10))
print(calls)
print(revenue.__name__, "|", revenue.__doc__)

360
480
[('revenue', (120, 3), 360), ('revenue', (48, 10), 480)]
revenue | Revenue for one line.

How it works: A decorator is a function that takes a function and returns a new function that wraps it. @logged above def revenue is shorthand for revenue = logged(revenue). The wrapper accepts any arguments (*args, **kwargs), calls the original, records the call and returns the result. functools.wraps copies the original's name and docstring onto the wrapper; without it, every decorated function would be called "wrapper" in tracebacks and help text.

Reading the result: After two calls the log contains 2 entries: [('revenue', (120, 3), 360), ('revenue', (48, 10), 480)]. Thanks to wraps, revenue.__name__ is still "revenue".

Example 9: A decorator with arguments, and caching

Question: How do we retry a flaky function, and avoid recomputing slow ones?

import time

def retry(times=3, delay=0.0):
    def decorator(func):
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(1, times + 1):
                try:
                    return func(*args, **kwargs)
                except ConnectionError as e:
                    print(f"attempt {attempt} failed: {e}")
                    if attempt == times:
                        raise
                    time.sleep(delay)
        return wrapper
    return decorator

state = {"calls": 0}

@retry(times=3)
def fetch_price(product):
    state["calls"] += 1
    if state["calls"] < 3:
        raise ConnectionError("server busy")
    return {"Milk": 56}[product]

print(fetch_price("Milk"), "after", state["calls"], "calls")

@functools.lru_cache(maxsize=None)
def slow_lookup(product_id):
    time.sleep(0.01)
    return product_id.lower()

for pid in ["P001", "P002", "P001", "P001", "P002"]:
    slow_lookup(pid)
print(slow_lookup.cache_info())

attempt 1 failed: server busy
attempt 2 failed: server busy
56 after 3 calls
CacheInfo(hits=3, misses=2, maxsize=None, currsize=2)

How it works: A decorator with arguments needs one more layer: retry(times=3) returns the actual decorator, which returns the wrapper. This one catches only ConnectionError (retrying everything would hide bugs) and re-raises on the last attempt. functools.lru_cache is a built-in decorator that memoises a function: it remembers results by arguments, so repeated calls with the same input return instantly. It reports hits and misses. Use it only for pure functions whose result depends only on their arguments.

Reading the result: fetch_price succeeded on call 3 after two failures. The cache saw 3 hits and 2 misses for five calls with only two distinct products.

Example 10: Context managers

Question: How do we guarantee set-up and clean-up, like closing a file?

from contextlib import contextmanager
import tempfile
import shutil

@contextmanager
def scratch_folder():
    folder = Path(tempfile.mkdtemp(prefix="fb_"))
    print("created folder")
    try:
        yield folder
    finally:
        shutil.rmtree(folder)
        print("cleaned up")

with scratch_folder() as tmp:
    (tmp / "note.txt").write_text("temporary")
    print(sorted(p.name for p in tmp.iterdir()))
    kept = tmp

print("folder still exists:", kept.exists())

try:
    with scratch_folder() as tmp:
        raise RuntimeError("something went wrong")
except RuntimeError as e:
    print("caught:", e, "| cleaned:", not tmp.exists())

created folder
['note.txt']
cleaned up
folder still exists: False
created folder
cleaned up
caught: something went wrong | cleaned: True

How it works: The with statement guarantees clean-up even if an exception occurs. You can write your own with @contextmanager: code before yield is the set-up, the yielded value is bound by as, and code after yield (put in a finally block) is the tear-down. It is a generator used for a specific job. Files, database connections, locks and temporary folders all use this pattern.

Reading the result: The folder existed inside the block and was gone afterwards (True), including when the code inside raised an error, which is the whole point.

Common mistakes

1. Iterating a generator twice. It is exhausted after one pass; the second loop sees nothing. Recreate it or use a list.

2. Forgetting functools.wraps, which makes debugging and documentation confusing.

3. Using groupby on unsorted data. It groups only consecutive items.

4. Caching impure functions, such as lru_cache on a function that reads a file or the current time: you will serve stale results.

5. Putting a return value in a generator and expecting a list. Generators yield items; call list() to collect them.

gen = (x for x in range(3))
print(list(gen))
print(gen[0])

[0, 1, 2]
Traceback (most recent call last):
  File "example.py", line 3, in <module>
    print(gen[0])
          ~~~^^^
TypeError: 'generator' object is not subscriptable

Practice questions

Practice 1 (Easy): A generator of order IDs

Write a generator order_ids(start, count) that yields IDs like O06001, and print the first four values for start=6001.

def order_ids(start, count):
    for i in range(start, start + count):
        yield f"O{i:05d}"

first4 = list(order_ids(6001, 4))
print(first4)

['O06001', 'O06002', 'O06003', 'O06004']

Practice 2 (Medium): Count lazily by city

Using the lazy readers, count delivered orders per store ID without loading the file into a list.

from collections import Counter

per_store = Counter(row["store_id"] for row in only_status(read_orders("orders.csv"), "Delivered"))
print(per_store.most_common(3))
print(sum(per_store.values()))

[('S07', 616), ('S01', 601), ('S05', 587)]
5391

Practice 3 (Hard): A timing decorator that records durations

Write a decorator timed that stores the duration of each call in a list attached to the wrapper (wrapper.durations), apply it to a function, call it three times and show that three durations were recorded and all are non-negative.

def timed(func):
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        start = time.perf_counter()
        try:
            return func(*args, **kwargs)
        finally:
            wrapper.durations.append(time.perf_counter() - start)
    wrapper.durations = []
    return wrapper

@timed
def summarise(n):
    return sum(i * i for i in range(n))

for n in (1000, 10000, 100000):
    summarise(n)

print(len(summarise.durations), all(d >= 0 for d in summarise.durations))
print(summarise.durations[0] < summarise.durations[2])

3 True
True

Key takeaways

• A for loop uses iter() and next(); an iterator is consumed once.
• A function with yield is a generator: lazy, memory-light and perfect for pipelines.
• Generator expressions feed sum, any, all, min, max and next without building lists; itertools adds islice, chain, groupby, pairwise and batched.
• Use read_csv(chunksize=...) for files bigger than memory and combine partial results.
• A decorator wraps a function; always use functools.wraps; add a layer to give a decorator arguments.
• lru_cache memoises pure functions; @contextmanager builds with blocks with guaranteed clean-up.