Python Sets: Unique Values and Set Operations

Analysts constantly ask questions that are really about groups: Which customers ordered in both January and February? Who ordered in January but never came back? How many different products were sold? Which cities appear in one dataset but not in the other? A set is the data structure made for exactly these questions.

A set is an unordered collection of unique items. Add the same value twice and it is stored once. Because of the way sets are built, checking "is this value in the set?" is extremely fast. And sets support the same operations you learned in school as Venn diagrams: union, intersection and difference. In this lesson you will use them on FreshBasket customer data, including a real retention question.

Creating sets

cities = {"Pune", "Delhi", "Pune", "Mumbai"}   # duplicates removed automatically
empty = set()                                   # NOT {}, which creates an empty dictionary
from_list = set(["UPI", "Card", "UPI"])

A set has no order and no indexing: you cannot ask for cities[0]. Elements must be hashable, so strings, numbers and tuples are fine but lists are not. Printing a set may show its elements in a different order from the one you typed, and that order can differ between runs for strings.

Worked examples

Example 1: Duplicates disappear

Question: A list of payment methods has repeats. How many distinct methods are there?

payments = ["UPI", "Card", "UPI", "COD", "UPI", "Wallet", "Card"]

unique = set(payments)
print(len(payments), len(unique))
print(sorted(unique))
print(type(unique))

7 4
['COD', 'Card', 'UPI', 'Wallet']
<class 'set'>

How it works: Converting a list to a set keeps only one of each value. We sort the set to show it in a stable order, because the set itself does not guarantee one.

Reading the result: Of 7 entries, 4 are distinct: ['COD', 'Card', 'UPI', 'Wallet']. len(set(x)) is the quickest way to count unique values in plain Python, and pandas calls it nunique().

Example 2: Adding and removing elements

Question: Maintain a set of cities that are currently live.

live = {"Bengaluru", "Mumbai"}

live.add("Pune")
live.add("Pune")
live.update(["Delhi", "Chennai"])
print(sorted(live))

live.remove("Chennai")
live.discard("Kolkata")
print(sorted(live))

removed = live.pop()
print(len(live))

['Bengaluru', 'Chennai', 'Delhi', 'Mumbai', 'Pune']
['Bengaluru', 'Delhi', 'Mumbai', 'Pune']
3

How it works: add inserts one element (and does nothing if it is already present), and update inserts many. remove raises a KeyError if the element is missing, whereas discard stays silent. pop removes and returns an arbitrary element, since sets have no order.

Reading the result: After the changes, the live set had 4 cities before the pop and 3 after it. Prefer discard when you do not care whether the element was present.

Example 3: Fast membership

Question: Why do people say sets are faster than lists for "is it in there?"

vip_list = [f"C{i:04d}" for i in range(1, 601)]
vip_set = set(vip_list)

print("C0599" in vip_list)
print("C0599" in vip_set)
print("C9999" in vip_set)
print(len(vip_set))

True
True
False
600

How it works: Checking a list must compare the item with elements one by one until it finds a match, so the time grows with the list's length. A set uses hashing and jumps straight to where the item would be, taking about the same time however large the set is. For 600 items the difference is invisible, but for millions of items it can be the difference between a second and many minutes.

Reading the result: Both structures give the same answers: True for a customer that exists and False for one that does not. Convert a big list to a set once, then test membership against the set repeatedly.

Example 4: Union, intersection and difference

Question: Using customer ids who ordered in January and in February, find who ordered in both, either, and only one month.

jan = {"C01", "C02", "C03", "C04", "C05"}
feb = {"C04", "C05", "C06", "C07"}

print(sorted(jan | feb))
print(sorted(jan & feb))
print(sorted(jan - feb))
print(sorted(feb - jan))
print(sorted(jan ^ feb))

['C01', 'C02', 'C03', 'C04', 'C05', 'C06', 'C07']
['C04', 'C05']
['C01', 'C02', 'C03']
['C06', 'C07']
['C01', 'C02', 'C03', 'C06', 'C07']

How it works: The operators map to Venn diagram regions: | is union (in either), & is intersection (in both), - is difference (in the left but not the right), and ^ is symmetric difference (in exactly one). The equivalent methods are union, intersection, difference and symmetric_difference.

Reading the result: 7 different customers ordered across the two months. 2 ordered in both, 3 ordered only in January (they did not come back in February), and 2 are new in February.

Example 5: A retention calculation

Question: What share of January customers returned in February?

retained = jan & feb
retention_rate = len(retained) / len(jan)
churned = jan - feb

print(f"Retained: {len(retained)} of {len(jan)} = {retention_rate:.0%}")
print("Lost:", sorted(churned))

Retained: 2 of 5 = 40%
Lost: ['C01', 'C02', 'C03']

How it works: The retained customers are those present in both sets, and the rate divides by the size of the starting group. The churned set is the difference. This tiny calculation is the core of every retention analysis, and in the pandas case study you will repeat it for thousands of customers.

Reading the result: 40% of January's customers returned, and 3 were lost. Always make sure the denominator is the starting group, not everyone.

Example 6: Subset, superset and disjoint

Question: Are all premium products also on the main catalogue, and do two sets have anything in common?

catalogue = {"P001", "P002", "P003", "P004", "P005"}
premium = {"P002", "P005"}
discontinued = {"P009"}

print(premium <= catalogue)
print(premium.issubset(catalogue))
print(catalogue >= premium)
print(catalogue.isdisjoint(discontinued))
print(discontinued <= catalogue)

True
True
True
True
False

How it works: A <= B asks whether every element of A is in B (a subset), and A >= B asks the reverse (a superset). isdisjoint is true when the two sets share nothing.

Reading the result: Premium is a subset of the catalogue (True), the catalogue and the discontinued set share nothing (True), and the discontinued product is not in the catalogue (False). These checks are handy for data validation: "are all order product ids present in the products table?"

Example 7: Data validation with real data

Question: Are all product ids in the order items table present in the products table?

import pandas as pd

products = pd.read_csv("products.csv")
items = pd.read_csv("order_items.csv")

product_ids = set(products["product_id"])
sold_ids = set(items["product_id"])

print(len(product_ids), len(sold_ids))
print(sold_ids <= product_ids)
print(sorted(product_ids - sold_ids))

50 50
True
[]

How it works: Converting a pandas column to a set gives the distinct values. We then compare the two sets: if every sold product id is in the products table, sold_ids <= product_ids is true. The difference tells us which products have never sold.

Reading the result: The catalogue has 50 products and 50 of them have appeared in orders, so every order line refers to a valid product (True). This pattern, called a referential integrity check, is one of the first things to run on any new dataset.

Example 8: Remove duplicates, and keep order

Question: De-duplicate a list of order ids, once with a set and once preserving the first-seen order.

ids = ["O3", "O1", "O3", "O2", "O1", "O4"]

print(sorted(set(ids)))
print(list(dict.fromkeys(ids)))

seen, ordered = set(), []
for x in ids:
    if x not in seen:
        seen.add(x)
        ordered.append(x)
print(ordered)

['O1', 'O2', 'O3', 'O4']
['O3', 'O1', 'O2', 'O4']
['O3', 'O1', 'O2', 'O4']

How it works: set(ids) loses the original order. A dictionary keeps insertion order and ignores repeated keys, so dict.fromkeys(ids) removes duplicates while keeping the first occurrence of each. The loop version does the same explicitly using a seen set for fast membership tests, which is useful when you need extra logic inside the loop.

Reading the result: The two order-preserving methods give identical results, ['O3', 'O1', 'O2', 'O4']. Pandas offers drop_duplicates() for the same job on tables.

Example 9: Frozenset

Question: Can a set be a dictionary key or a member of another set?

pair_a = frozenset({"Milk", "Bread"})
pair_b = frozenset({"Bread", "Milk"})

print(pair_a == pair_b)

basket_counts = {pair_a: 12}
print(basket_counts[pair_b])

try:
    {{"Milk", "Bread"}: 1}
except TypeError as e:
    print("set:", e)

True
12
set: unhashable type: 'set'

How it works: A normal set is mutable and therefore unhashable. A frozenset is an immutable set, so it can be a key or an element. Because sets ignore order, {"Milk", "Bread"} and {"Bread", "Milk"} are the same, which is perfect for counting how often two items are bought together regardless of order.

Reading the result: The two frozensets compare equal and share one dictionary entry, 12. This is the building block of simple market-basket analysis.

Common mistakes

1. Creating an empty set with {}.

print(type({}), type(set()))

<class 'dict'> <class 'set'>

2. Expecting order or indexing.

cities = {"Pune", "Delhi"}
print(cities[0])

Traceback (most recent call last):
  File "example.py", line 2, in <module>
    print(cities[0])
          ~~~~~~^^^
TypeError: 'set' object is not subscriptable

3. Putting a list inside a set. Lists are unhashable; use a tuple.

4. Using remove when the element might be absent. Use discard.

5. Assuming set printing order is stable. Sort when you need a predictable display.

Practice questions

Practice 1 (Easy): Count distinct cities

From ["Pune", "Delhi", "pune", "Delhi", "Mumbai"], count the distinct cities ignoring capitalisation.

cities = ["Pune", "Delhi", "pune", "Delhi", "Mumbai"]
print(len({c.lower() for c in cities}))

3

Practice 2 (Medium): Two-month overlap

Given march = {"C1", "C2", "C3", "C4", "C5", "C6"} and april = {"C2", "C4", "C7", "C8"}, print the number of customers who ordered in both months, the number new in April, and April's retention rate relative to March to one decimal place as a percentage.

march = {"C1", "C2", "C3", "C4", "C5", "C6"}
april = {"C2", "C4", "C7", "C8"}

both = march & april
new_in_april = april - march
rate = len(both) / len(march) * 100

print(len(both), len(new_in_april), f"{rate:.1f}%")

2 2 33.3%

Practice 3 (Hard): Never-reordered customers from real data

Using orders.csv, find customers whose orders all happened in 2025 and who placed no order in 2026, and report how many there are.

orders = pd.read_csv("orders.csv")

year = orders["order_date"].str[:4]
in_2025 = set(orders.loc[year == "2025", "customer_id"])
in_2026 = set(orders.loc[year == "2026", "customer_id"])

lapsed = in_2025 - in_2026
print(len(in_2025), len(in_2026), len(lapsed))

472 555 33

Key takeaways

• A set holds unique, hashable items in no particular order; create an empty one with set().
• len(set(values)) counts distinct values, and a set removes duplicates (use dict.fromkeys to keep order).
• |, &, - and ^ give union, intersection, difference and symmetric difference.
• Subset, superset and isdisjoint tests are excellent for data validation checks.
• Membership tests on sets are very fast, so convert large lists to sets before repeated lookups.
• Retention and churn questions are intersections and differences of customer-id sets.