Shrink a DataFrame's memory footprint

Question

A 40-million-row extract will not fit in memory with default dtypes.

Write `optimise(df)` that returns a copy where integer columns are downcast to the smallest safe integer type, float columns to the smallest safe float type, and object columns with fewer than 50% unique values become `category`.

Accepted Answer

Step 1: `pd.to_numeric(..., downcast='integer')` picks the smallest integer type that still holds every value — int8 here instead of int64, an 8x saving. Step 2: Float downcasting to float32 halves the memory but also halves the precision; that trade is fine for scores, not for money. Step 3: The 50% cardinality rule is the standard heuristic: `category` wins big on repeated strings and loses on near-unique ones like free-text notes. Step 4: Working on a copy keeps the original frame available if a downcast turns out to be too aggressive.

Shrink a DataFrame's memory footprint

More Pandas — Cleaning questions

Asked at Amazon