Learning Machines: Stable Isn't the Same as Representative

The last post ended with a debt I said I'd pay. I'd walked through why random_state=42 breaks when your dataset changes, solved it with hash-based splitting, and then left one problem standing: sampling bias. This is that post. If you haven't read the last one yet, I'd go there first; the context carries over. But even if you haven't, what follows stands on its own.

Here's the thing about a perfectly stable, hash-based split: it tells you a data point will be in the same bucket every time. It says nothing about whether that bucket looks anything like the world your model is supposed to understand.

When Your Training Data Has a Blind Spot

Sampling bias happens when the data you train on doesn't properly represent the population your model will face in production.

Say you're building a churn prediction model. Your training data happens to skew heavily toward enterprise customers — long-tenured, deeply integrated with your product, with strong financial incentive to stay. Your model learns what churn looks like for them. Then you deploy it on free-tier users with completely different behavior patterns and completely different reasons for leaving. The model underperforms. It's not that the model learned wrong things. It just never learned about this population in the first place.

Or: a job satisfaction model trained entirely on survey responses from employees at stable, well-paying companies. The model has no signal about what satisfaction looks like when compensation isn't a given, or when the environment is genuinely difficult. It learned from a filtered view of the question and now answers a different question than the one you're asking.

Neither dataset is bad data. Both introduce bias — because neither reflects the full population the model needs to understand. The fix isn't better data quality. It's better sampling strategy.

The Fix (and How It Actually Works)

Stratified sampling works by dividing the data into groups based on attributes that matter (called strata), then sampling from each group in proportion to how it appears in the target population.

The clearest way to see it: say you're building a model to predict average monthly spending per person, and you have good reason to think age plays a role. Without stratification, a random shuffle might give you a training set that's 75% people aged 19–25 (not because that's the population breakdown, but because that's where the shuffle landed). Stratified sampling prevents that.

You'd:

1. Define your age classes — 13–18, 19–25, 26–35, 36–50, etc.
2. Split the dataset into groups by age class.
3. Sample from each group in proportion to how that group actually appears in the population you're modeling.

The result is a dataset that looks like the world, not like whatever the random draw happened to produce.

In scikit-learn, this is one additional argument to a function you're already calling:

from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,
    random_state=42,  # or better — hash-based. See the previous post.
    stratify=y        # preserves the class distribution across both splits
)

stratify=y tells the split to maintain the proportion of each class in both training and test. For a dataset that's 70% class A and 30% class B, both splits come out at that same ratio. Small addition. Real difference.

The Part No Library Can Do For You

Here's where domain knowledge becomes load-bearing.

You can hand stratify=y to scikit-learn and it will dutifully preserve whatever class distribution exists in your target variable. But it has no opinion about what you should be stratifying on. That's entirely on you.

If you're predicting political satisfaction, a perfect representation of height distribution probably doesn't matter much. A good representation of gender and income bracket almost certainly does. If you're predicting medical outcomes, age and pre-existing conditions are going to matter more than ZIP code — probably. The probably is the whole point. You need to understand the problem well enough to identify which attributes, if skewed in your sample, would quietly warp what the model learns.

Stratified sampling only helps if you're stratifying on the right things. Get that wrong and you've traded a visible problem (your data is obviously skewed) for an invisible one (your data looks balanced, but the balance is on the wrong axis).

Sampling Bias and Class Imbalance Are Different Problems

People mix these up often enough that it's worth saying clearly: stratified sampling and class imbalance techniques are not the same thing, and they don't solve the same problem. They do tend to show up together in a pipeline, which is probably why the confusion sticks.

Stratified sampling is about your split: making sure both train and test sets reflect the real-world distribution of your data. Class imbalance handling is about your model: making sure it has enough signal from underrepresented classes to actually learn from them.

A fraud detection scenario makes this concrete. Your dataset is 95% legitimate transactions, 5% fraudulent. That ratio isn't a mistake — it's what the real world looks like. So your first goal is a split that preserves it:

X_train, X_test, y_train, y_test = train_test_split(
    X, y,
    test_size=0.2,
    stratify=y  # train: ~95/5. test: ~95/5. Both reflect reality.
)

Your test set is now representative. Good. But the model still has a problem.

5% is not a lot of examples to learn from. A model trained naively on 95/5 data will often just predict "not fraud" on everything, hit 95% accuracy, and be completely useless at the one job it was given. Technically impressive. Practically worthless.

That's where SMOTE comes in — and critically, only on the training set:

# pip install imbalanced-learn
from imblearn.over_sampling import SMOTE

smote = SMOTE(random_state=42)
X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)

# X_test stays untouched. It still represents reality.

SMOTE (Synthetic Minority Over-sampling Technique) generates synthetic minority-class examples by interpolating between existing ones, giving the model more fraud signal to learn from during training. Your test set doesn't move — it keeps reflecting the actual distribution. The model trains on augmented data; it gets evaluated on an honest one.

If you apply SMOTE to the test set too, you poison the evaluation. Test accuracy will look great because the test set no longer represents what the model faces in production. It's an easy mistake to make, and the output looks like good news.

The pipeline runs in sequence, not in parallel:

1. Stratified split — both sets reflect the actual class distribution of your population.
2. SMOTE (or equivalent) on the training set only — enough minority-class signal for the model to generalize.

Skip the first step and your test set misrepresents the world. Skip the second and your model never properly learns the edge cases that matter most.

What a Clean, Stable, Representative Split Still Doesn't Fix

A stable split means the same data points land in the same buckets every time. A representative split means those buckets reflect the population you care about. Together, they're necessary. They're not sufficient.

We've been assuming the data is clean enough to split in the first place. What happens when it isn't — when labels are missing, when attributes have null values, when you need to decide whether to drop an instance, drop a feature, or fill in the gap with a mean and hope for the best?

That's the next problem. And it has its own process.