Why Is My pandas Loop So Slow? iterrows vs Vectorize

Because it’s a loop.

Pandas was built to operate on whole columns at once — not on one row at a time. New users see a DataFrame, see something that looks like a spreadsheet, and reach for for row in df.iterrows(). It works. It also turns a 1-millisecond calculation into a 300-millisecond one.

This post measures the same calculation three ways on the same 28,000-row OHLC table. The numbers do the talking.

The setup

Daily return is straightforward: (Close - Open) / Open. Compute it on a 28,140-row prices table — 14 tickers, eight years of trading days. Time each approach with time.perf_counter.

import time
import pandas as pd

df = pd.read_parquet("prices.parquet")
print(f"rows: {len(df):>6}")
# rows:  28140

Same input. Same answer. Three drastically different speeds.

Method 1 — iterrows (the textbook anti-pattern)

t0 = time.perf_counter()
returns = []
for idx, row in df.iterrows():
    returns.append((row["Close"] - row["Open"]) / row["Open"])
df["ret_iterrows"] = returns
print(f"iterrows: {(time.perf_counter() - t0) * 1000:.1f} ms")
# iterrows: 278.0 ms

What’s slow here isn’t the arithmetic — it’s the wrapping. On every single one of those 28,140 iterations, pandas takes the underlying row and constructs a fresh Series for you. That involves a dict allocation, copying values into a new object, attaching a name, plumbing the dtype. You never asked for any of that — you just wanted to read two numbers — but you pay for it every row.

The cost isn’t visible in the code. It’s hidden in how iterrows works internally.

Method 2 — itertuples (the right escape hatch)

t0 = time.perf_counter()
returns = []
for row in df.itertuples(index=False):
    returns.append((row.Close - row.Open) / row.Open)
df["ret_itertuples"] = returns
print(f"itertuples: {(time.perf_counter() - t0) * 1000:.1f} ms")
# itertuples: 25.2 ms   (11x faster than iterrows)

Same loop shape. The only change: iterrows returns (index, Series) pairs; itertuples returns named tuples. Named tuples are cheap — they’re just tuple subclasses with attribute access. No dict allocation, no Series construction.

You read the same two values (row.Close, row.Open) and run the same math. The loop is just as long. The body is the same. But the per-iteration cost dropped by an order of magnitude.

itertuples is the right tool when you genuinely need to walk row by row — a stateful sequence that depends on the previous row’s result, a path-dependent calculation that can’t be expressed in column-math. Those cases exist. They’re rare.

Method 3 — vectorize (the answer)

t0 = time.perf_counter()
df["ret_vector"] = (df["Close"] - df["Open"]) / df["Open"]
print(f"vectorize: {(time.perf_counter() - t0) * 1000:.2f} ms")
# vectorize: 0.26 ms   (1080x faster than iterrows)

There’s no loop. There’s no body. There’s no row variable. There’s a single expression that operates on whole columns at once. Pandas pushes that expression down into NumPy, NumPy pushes it down into C, and what would have been 28,140 Python iterations becomes one contiguous machine-level operation over a contiguous array of doubles.

The number to remember: 1000× faster than the textbook iterrows version. Same calculation. One line.

Why this matters

The 28k case is annoying. The 1M case is the difference between “instant” and “go get a coffee”. The 100M case is the difference between feasible and infeasible.

This isn’t an optimization you do at the end. It’s the way pandas wants to be used.

The mental shift

Stop thinking row-by-row. Start thinking column-by-column.

• Filter rows? Boolean mask: df[df["Close"] > 100]. No loop.
• Compute a new column? Column expression: df["pct_up"] = df["Close"] / df["Open"] - 1. No loop.
• Apply a function? Look for the vectorized equivalent: df["log"] = np.log(df["Close"]). No loop.
• Compare to previous row? .shift(), .diff(), .pct_change(). No loop.
• Rolling window? .rolling(20).mean(). No loop.

If you find yourself writing a for loop over a DataFrame, treat it as a red flag. Ninety percent of the time there’s a one-line column expression that replaces it and runs hundreds of times faster.

What about .apply(axis=1)?

.apply(f, axis=1) is a loop in disguise. It’s faster than iterrows because pandas inlines some of the row-wrapping, but it’s still iterating in Python under the hood. Anywhere you can rewrite df.apply(f, axis=1) as a column expression, you should.

.apply(f, axis=0) (column-wise apply) is different — that hands an entire column to your function. If f is itself vectorized, this is fast. If f is a Python loop inside, you’ve just moved the slowness one level down.

What you’re not going to do

A few patterns that show up on Stack Overflow but you should avoid:

• for i in range(len(df)): then df.iloc[i]. Even worse than iterrows — every .iloc[i] is a fresh row-construction call. Slowest possible iteration.
• for index, row in df.iterrows(): for anything that can be expressed as a column operation. The default assumption should be “there’s a vectorized version” — only fall back to iteration when there genuinely isn’t.
• Reaching for apply(axis=1) when a direct column expression exists. df.apply(lambda row: row["Close"] / row["Open"], axis=1) is slow. df["Close"] / df["Open"] is fast.

Takeaways

1. iterrows is almost always wrong. Series-per-row dict allocations dominate the cost.
2. itertuples is the right loop when you truly need one. ~10× faster, named tuples instead of Series.
3. Vectorize first. Column arithmetic, boolean masks, NumPy ufuncs, pandas built-ins. 1000× faster than the loop you would have written.
4. .apply(axis=1) is a loop in disguise. Faster than iterrows, slower than true vectorize. Rewrite as column expression when possible.
5. Row loop = red flag. First place to look for a speedup when someone else’s pandas script is slow.

Watch the video for the timing demo on the full 28k-row dataset.