pandas: three speedups in the workhorse of data science
We ran our Auto Engineer on pandas to make common operations faster while preserving correctness. It found, implemented, and verified three optimizations, which you can explore below.
pandas supports many data types, missing-value conventions, and memory layouts. That flexibility can add conversion, masking, and copying work even when the data is already ready to use. The three optimizations share an approach: check the data first, then skip unnecessary work when it is safe. Each uses a fast path for inputs that meet specific conditions; other inputs follow pandas’ existing implementation.
Explore 3 optimizations
Optimization 1Row-wise math on nullable dtypes
pandas' nullable integer, float, and boolean columns, including their Arrow-backed variants, take their own route for row-wise sum, prod, min, and max: the frame is flattened and run through the groupby kernels, which meant first building a "row label" for every single element, a throwaway array as large as the frame itself, just to tell the kernel which row each value belongs to. AutoR&D's patch views the existing data and mask buffers in the layout the kernel already expects and reduces with a single group, so that label array is never allocated and the kernels and reduction order stay exactly as they were. Across pandas' own 40-cell benchmark that's 1.22–1.84× faster (median 1.38×) with every confidence interval above parity, and 12–15% lower peak memory on a 10-million-element frame.
Technical details: formulation & implementation
Formulation
Take a frame X with n rows and c columns that share one nullable dtype. To evaluate X.sum(axis=1), pandas concatenates the columns into a single one-dimensional extension array A of length n·c, laid out column-major, and hands the groupby aggregation kernel a label vector L so that group r collects row r:
That label vector is an n·c array of 8-byte integers (on 64-bit builds) whose only job is to tell the kernel which row each element belongs to. The new route views the same data and mask buffers as a (c, n) matrix, transposes to (n, c), and calls the kernel with one group and c ids. Each of the kernel's n "columns" is then one frame row, aggregated over its c observations. Every output cell consumes the same values in the same column order as the labelled formulation, so results are bit-identical; the only thing that disappears is the label array.
Implementation
BaseMaskedArray._groupby_op_axis1(how, nrows, ncols, min_count, skipna)reshapes_dataand_maskas views and callsWrappedCythonOp._cython_op_ndim_compatwithngroups=1; it raisesNotImplementedErrorfor anything it does not handle so the caller falls back.ArrowExtensionArray._groupby_op_axis1covers the eleven Arrow int, uint, float and bool types by delegating through_to_masked()and converting the result back. String, decimal and timestamp Arrow types fall through.DataFrame._reducetries the route first forsum,prod,minandmax, and otherwise continues to the existingnp.tilelabel path. Two ASV benchmark classes (FrameOpsAxis1EA,FrameOpsAxis1EAPeakmem) cover it.
# pandas/core/arrays/masked.py (condensed: the real guard also checks ndim and positive shape)
def _groupby_op_axis1(self, *, how, nrows, ncols, min_count, skipna, **kwargs):
if how not in ("sum", "prod", "min", "max") or self.size != nrows * ncols:
raise NotImplementedError # caller falls back to _groupby_op
kind = WrappedCythonOp.get_kind_from_how(how)
op = WrappedCythonOp(how=how, kind=kind, has_dropped_na=False)
values = self._data.reshape(ncols, nrows).T # views of the existing buffers
mask = self._mask.reshape(ncols, nrows).T
ids = np.zeros(ncols, dtype=np.intp) # one group: c ids, no n*c label array
result_mask = np.zeros((nrows, 1), dtype=bool)
result = op._cython_op_ndim_compat(values, min_count=min_count, ngroups=1, comp_ids=ids,
mask=mask, result_mask=result_mask, skipna=skipna, **kwargs)
return self._maybe_mask_result(result.squeeze(axis=1), result_mask.squeeze(axis=1))Evidence
- 134,064 differential cases (141,120 calls) across every supported dtype, both
skipnasettings,min_count0 through 4, degenerate shapes, infinities and NA patterns: zero result, mask, dtype, index-dtype or exception differences. 1,584 of 1,584 routing checks used the new kernel, and a test that monkeypatches_groupby_opto raise proves the old path is never reached. - ASV: 29 of 40 timing cells flagged improved (ratios 0.51 to 0.82) and 10 more with faster point estimates. A paired CPU-pinned study gives 1.215× to 1.837× (median 1.377×) with every bootstrap interval above 1.
- Peak memory on a 2,000,000 × 5 frame: 449 → 396 MB (sum, Int64) and 472 → 401 MB (sum, float64[pyarrow]). 437 new tests and 44,458 existing tests pass.
Optimization 2Counting and de-duplicating strings
For nullable string columns, duplicated() and value_counts() now hand their backing array straight to pandas' own hashing kernel: 4.4–12.2% faster on million-value workloads, with peak memory for value_counts down by about a third. Total change: a 28-line diff.
Technical details: formulation & implementation
Formulation
Write the baseline operator as F0 and the candidate as F1. A route is a guard g, a specialized kernel K and a reconstruction Q, with P a zero-copy (or bounded-copy) projection of the physical representation:
The obligation is stronger than equal values. With Π the public projection (values or exception type, value dtype, shape, index dtype, index values and names), the route must satisfy Π(F1(x)) = Π(F0(x)) for every input, so the guard has to cover aliasing and mutability as well as dtype.
Here the representation is a StringArray whose backing store is an object ndarray with pd.NA as the sentinel. The inherited duplicated first materializes an NA mask, and the generic value_counts exports through to_numpy, hashes, then converts the result index back to the extension dtype. All of that is adapter work around a kernel that already accepts the backing array as it is.
Implementation
One file, pandas/core/arrays/string_.py: 28 lines inserted, 2 deleted. The guard for duplicated is that the sentinel is exactly pd.NA; value_counts additionally requires dropna=True. No new hash table, encoding or NA rule is introduced, so the review surface is just the guard and the reconstruction.
# pandas/core/arrays/string_.py
def duplicated(self, keep="first"):
if self.dtype.na_value is libmissing.NA:
return duplicated(self._ndarray, keep=keep) # algorithms-layer kernel, no NA-mask pass
return super().duplicated(keep=keep)
def value_counts(self, dropna=True):
if self.dtype.na_value is libmissing.NA and dropna:
result = value_counts_internal(self._ndarray, sort=False, dropna=True)
index_arr = self._from_backing_data(np.asarray(result.index._data))
index = Index(index_arr, name=result.index.name, copy=False)
result = Series(result._values, index=index, name=result.name, copy=False)
else:
result = super().value_counts(dropna=dropna)Evidence
- A 130-case differential matrix over object, Python-string and Arrow-string representations; ASCII, empty, embedded-NUL, composed and decomposed Unicode, CJK and emoji payloads; every NA variant; all
keepmodes andvalue_countsoptions. Baseline and candidate serializations share the same SHA-256. - 30 paired timings per case in alternating order on a pinned core: all four activated routes have simultaneous 95% upper ratio bounds below parity (0.899 to 0.970). A deliberate Unicode fallback control behaves exactly as the unchanged path should, confirming the route counter.
- Fresh-process memory:
value_countspeak RSS increase 37.13 → 24.45 MB and traced peak 31.65 → 24.66 MB; duplicate detection is non-increasing on both metrics.
Optimization 3Row-wise math on numeric frames
sum, prod, min, and max across rows used to allocate fresh intermediate arrays for every internal block; the patch folds blocks into one reused accumulator. That's 12–28% faster on a typical 100,000-row mixed frame, and up to 64.6% on the fragmented frames real pipelines accumulate.
Technical details: formulation & implementation
Formulation
For sum, prod, min and max with axis=1, the baseline avoids a transpose. Each physical block is reduced to a length-n vector middle, the first is copied into result, and every later block produces a fresh ufunc(result, middle). With B blocks that is B block temporaries, one copy and B − 1 combine temporaries; nullable float sum and prod add mask and filled-copy allocations inside nanops. The candidate reduces the first eligible block straight into an owned accumulator and folds each later block through one reused scratch array, combining in place: one accumulator and at most one scratch, whatever B is.
Bitwise fidelity is the constraint. Float sum and prod are identity-seeded in the baseline, and that seed is observable:
The fold therefore seeds with 0 or 1, visits rows in order, and substitutes the identity for NaN only on rows that contain one. Integer operations and min/max use ufunc.reduce(..., out=).
Implementation
pandas/core/frame.py, 151 lines added, most of them guards. A block is eligible when it is a C-contiguous, native-byte-order, two-dimensional ndarray of int64, uint64, float32 or float64 whose ufunc.resolve_dtypes resolves without a cast; the accumulator must be owned, writable, shape-compatible and non-aliasing. Unlike dtypes are accepted only when one side dominates promotion (the accumulator is promoted once); promotions that change both operands, such as int64 + uint64 → float64, stay on the generic path. min and max use the route from 4,096 rows; sum and prod at any length.
# pandas/core/frame.py, inside the block loop of DataFrame._reduce_axis1
if result is None and _can_fuse_axis1_block(vals, ufunc):
result = np.empty(vals.shape[1], dtype=vals.dtype)
_fold_axis1_block(vals, result, name, skipna, ufunc) # reduce straight into the accumulator
continue
elif _can_fuse_axis1_into(vals, result, ufunc):
if np.promote_types(result.dtype, vals.dtype) == vals.dtype:
result = result.astype(vals.dtype) # promote the accumulator once
if vals.shape[0] == 1 and singleton_is_bitwise_safe:
ufunc(result, vals[0], out=result)
else:
scratch = ensure_scratch(dtype=vals.dtype) # one scratch array, reused
_fold_axis1_block(vals, scratch, name, skipna, ufunc)
ufunc(result, scratch, out=result) # combine in place
continueEvidence
- A 900-case broad matrix (four dtypes, four operations, both
skipnasettings,min_countboundaries, several shapes) plus an 880-case edge matrix (both orders of 12 promotion pairs at 17, 4,095 and 4,096 rows; signed-zero, NaN and infinity patterns; near-limit integers), compared by value-byte hash, dtype, index dtype, shape and name: zero mismatches. - Primary 100,000-row mixed frame over 11 paired samples: sum 556.5 → 469.3 µs, prod 675.3 → 486.5 µs, min 437.8 → 382.2 µs, max 439.0 → 384.8 µs.
- A 32-cell topology sweep (8 frame layouts × 4 operations, 9 pairs each, bootstrap 95% intervals) shows where routing overhead dominates: homogeneous float frames sit near parity, homogeneous integer frames lose up to 9%, 1,024-row mixed frames lose 4–16%, and the 4,096-row min/max gate admits one narrow case that costs about 31%. Those cells are reported as the data for a production dispatch policy over rows, blocks, columns per block, dtype and NA density.
The verification is what makes this upstream-ready: 1,910 differential test cases for the first two patches and 134,064 more for the nullable-dtype patch, each comparing patched and unpatched pandas on identical inputs, from Unicode and emoji to integer overflow, infinities, min_count edge cases, and the sign bit of zero. Every output matched exactly, and anything unusual simply takes the original code path.
