Learning Machines: Why random_state=42 Isn't Enough

I used to think random_state=42 was enough. Set the seed, get the same split, move on. It felt reproducible. It felt safe.

Then I added 200 rows to a dataset and watched every data point shuffle into a different bucket. Same seed. Completely different split. I'd been committing this mistake for longer than I'd like to admit — the "reproducibility" I was relying on was a house of cards that only held up if nothing changed. And in production, everything changes.

The Problem With Random Seeds

Here's what random_state actually does: it initializes a pseudorandom number generator that shuffles your data before splitting. Given the exact same array, in the exact same order, with the exact same length, you'll get the exact same split. Remove a row, add a row, reorder anything — the entire shuffle changes.

from sklearn.model_selection import train_test_split

data_v1 = list(range(10))
train, test = train_test_split(data_v1, test_size=0.2, random_state=42)
print(test)  # [1, 5]

data_v2 = list(range(11))  # one new record
train, test = train_test_split(data_v2, test_size=0.2, random_state=42)
print(test)  # [4, 1, 10] — completely different

Same seed. One extra row. Every assignment reshuffled.

This is fine for a notebook where the dataset is frozen. It breaks down the moment your data is alive.

Picture this: you train a model, get 95% accuracy, and deploy it. Two months later, you notice some model drift. New data has come in, patterns have shifted, so you retrain. But now your dataset has grown. With random_state=42, the split is completely different — data points that were in your test set last time might now be in training, and vice versa.

You get 96% accuracy. Is that a real improvement? Or did you just get lucky with a more favorable split?

There's no way to know. The comparison is poisoned.

Deterministic Splitting With Hashing

The fix is to make the split a property of each data point, not a property of the dataset's current shape. Every record should know whether it belongs in training or testing — and that assignment shouldn't budge when new records show up.

Hashing gives us exactly this. A hash function takes an input and produces a fixed-size output — always the same output for the same input. Hash a stable identifier for each data point, and the bucket assignment stays constant no matter what else happens to the dataset.

Three requirements make this work:

1. The identifier must be unique — no two data points share the same ID.
2. The identifier must be immutable — it doesn't change over time.
3. The hash function must distribute outputs uniformly across its output space.

With those three in place, every data point gets a deterministic, evenly distributed position in the hash space. Splitting becomes a matter of drawing a line through that space.

Drawing the Line

Say we want an 80/20 train/test split using a 32-bit hash function. The output space has 2^32 possible values (about 4.3 billion), ranging from 0 to 2^32 − 1.

We set a threshold:

threshold = 0.2 × 2^32

If the hash of a data point's ID falls below that threshold, it goes into the test set. Otherwise, training. Since the hash function distributes values uniformly, roughly 20% of data points land below the line and 80% land above it.

The Law of Large Numbers does the rest — as your dataset grows, the actual ratio converges toward the target.

Here's how it looks in Python using MurmurHash3:

import mmh3

MAX_HASH = 2**32  # unsigned 32-bit output space


def is_id_in_test_set(identifier, test_ratio):
    hash_value = mmh3.hash(str(identifier), signed=False)
    threshold = test_ratio * MAX_HASH
    return hash_value < threshold


def deterministic_split(data, test_ratio, id_col):
    """Split a DataFrame into train/test using stable hash-based assignment."""
    in_test_set = data[id_col].apply(
        lambda id: is_id_in_test_set(id, test_ratio)
    )
    return data.loc[~in_test_set], data.loc[in_test_set]

Add a million new users tomorrow. Every existing user stays in the same bucket. That's the whole point.

Why MurmurHash3 and Not CRC32?

You'll see examples online using zlib.crc32. It works for quick scripts, but CRC32 was designed for error detection (checksums), not for distributing values evenly. It can exhibit clustering — certain regions of the output space get more hits than others.

MurmurHash3 (mmh3) was built specifically for hash-based distribution. It has a strong avalanche effect, meaning a tiny change in the input (flipping a single character in the ID) produces a wildly different output. For production splitting where even distribution matters, it's the better tool.

When Your Hash Space Gets Crowded

There's a ceiling to watch for. Hash functions have a finite output space, and as you add more items, the probability of two items getting the same hash (a collision) goes up faster than intuition suggests.

The birthday paradox tells us that collisions become likely once the number of items approaches the square root of the output space:

N ≈ √S

For a 32-bit hash, that's √(2^32) = 65,536 items. Past that threshold, collision risk climbs rapidly. For most ML datasets, 65K records is not a lot.

Collisions won't corrupt your split — two records landing in the same bucket is harmless — but they erode the precision of your target ratio, and at scale, that drift adds up.

A 64-bit hash pushes the threshold to √(2^64) = 2^32 ≈ 4.3 billion — far more comfortable for production-scale data. If your dataset is large, use a hash function with a wider output space. MurmurHash3 supports 128-bit output, which effectively eliminates collision concerns for any realistic dataset size.

The Oxymoron That Works

"Deterministic randomness" sounds like a contradiction. But that's exactly what hash-based splitting gives you: an assignment that looks random (uniform distribution, no pattern to the buckets) but is deterministic (same input, same output, every time).

You get the statistical properties of a random split — even distribution, no systematic bias in bucket assignment — with the stability of a fixed rule. Retrain on a bigger dataset six months from now, and every old data point is still in its original bucket. Your performance comparisons are valid. Your test set hasn't been contaminated by data that used to be in training.

What This Doesn't Solve

Stable splits guarantee that a data point's assignment is permanent. They don't guarantee that the split is representative.

If your data has structure — clusters, time dependencies, class imbalances — a uniform hash doesn't know about any of that. You could end up with a test set that's technically stable but statistically skewed. Imagine a medical dataset where 90% of your positive cases happened to hash below the threshold. Your test set now looks nothing like production traffic.

That's a different problem: sampling bias. And it's worth its own post.