Difference between StratifiedKFold and StratifiedShuffleSplit in sklearn
Quick answer
Both preserve class proportions, but they partition differently. StratifiedKFold splits the data into k non-overlapping folds and uses each fold as the test set exactly once, so every sample is tested exactly once. StratifiedShuffleSplit generates independent random train/test splits, so samples may repeat across splits or never appear at all. Use StratifiedKFold for cross-validation, StratifiedShuffleSplit when you want a specific test size or many random repeats.
Short answer: Both keep each class's proportion in every split. The difference is coverage: StratifiedKFold partitions the data into k non-overlapping folds and uses each as the test set exactly once, so every sample is tested exactly once. StratifiedShuffleSplit draws each split independently at random, so test sets can overlap and some samples may appear in several — or none. Use KFold for cross-validation; use ShuffleSplit when you want an exact test_size or many random repeats.
Both are stratified splitters — they preserve the class distribution in every train/test split, which matters most with imbalanced data, where an ordinary random split can leave a fold with no minority-class samples at all. What differs is how they partition.
The key distinction: disjoint folds vs independent random splits
StratifiedKFolddivides the dataset into k mutually exclusive folds. Across the k iterations, each fold serves as the test set exactly once, so the test sets never overlap and together cover the whole dataset. Test size is fixed at ≈ 1/k.StratifiedShuffleSplitignores that partition. Each of itsn_splitsis a fresh, independent shuffle-and-sample, so a sample can land in the test set of several splits or of none.n_splitsandtest_sizeare set independently.
StratifiedKFold (k=5): test sets are disjoint, cover everything
split1 [T][ ][ ][ ][ ]
split2 [ ][T][ ][ ][ ] every sample tested exactly once
split3 [ ][ ][T][ ][ ]
StratifiedShuffleSplit (n_splits=5, test_size=0.2): independent draws
split1 ....T..T.....T... test sets may overlap;
split2 T........T....T.. some samples appear many times, some neverStratifiedKFold — for cross-validation
Because the folds cover the whole dataset with no overlap, the k scores average into an unbiased performance estimate:
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import accuracy_score
X, y = load_iris(return_X_y=True)
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = []
for train_idx, test_idx in skf.split(X, y):
model = RandomForestClassifier(random_state=0)
model.fit(X[train_idx], y[train_idx])
scores.append(accuracy_score(y[test_idx], model.predict(X[test_idx])))
print("mean CV accuracy:", sum(scores) / len(scores))(In practice cross_val_score(model, X, y, cv=5) uses StratifiedKFold automatically for classifiers.)
StratifiedShuffleSplit — for an exact test size or many repeats
Reach for it when you want a specific test proportion decoupled from the number of repeats — for example a single 80/20 split, or 100 random 80/20 repeats for a stability estimate:
from sklearn.model_selection import StratifiedShuffleSplit
sss = StratifiedShuffleSplit(n_splits=10, test_size=0.2, random_state=42)
for train_idx, test_idx in sss.split(X, y):
X_train, X_test = X[train_idx], X[test_idx]
y_train, y_test = y[train_idx], y[test_idx]
# ... fit and evaluate ...Which to use
| You want… | Use |
|---|---|
| Standard cross-validation, every sample tested once | StratifiedKFold |
An exact test_size (e.g. 20%) regardless of split count | StratifiedShuffleSplit |
| Many random repeats for a variance estimate | StratifiedShuffleSplit |
A common error
Both need at least one sample per class per split, so the smallest class must have at least n_splits members. Otherwise scikit-learn raises "The least populated class ... has too few members" — reduce n_splits, or merge/drop very rare classes.
Related guides
Sources
Key takeaways
- •StratifiedKFold guarantees every sample is in the test set exactly once; StratifiedShuffleSplit gives no such guarantee.
- •StratifiedShuffleSplit's splits are independent and may overlap, so a sample can appear in several test sets or none.
- •Use StratifiedKFold for standard k-fold cross-validation and model comparison — it uses all the data with no wasted samples.
- •Use StratifiedShuffleSplit when you need an exact test proportion (for example test_size=0.2) independent of the number of splits.
- •With StratifiedKFold the test size is fixed at roughly 1/k; with StratifiedShuffleSplit you set n_splits and test_size independently.
- •Both require enough members in each class — stratification fails if any class has fewer samples than the number of folds.
Frequently asked questions
What is the main difference between StratifiedKFold and StratifiedShuffleSplit?
StratifiedKFold partitions the dataset into k mutually exclusive folds, so the test sets never overlap and every sample is tested exactly once. StratifiedShuffleSplit creates each split by independently shuffling and sampling, so test sets across splits can overlap and coverage is not guaranteed.
Which should I use for cross-validation?
StratifiedKFold. Because the folds are non-overlapping and cover the whole dataset, the k scores can be averaged into an unbiased estimate of model performance. StratifiedShuffleSplit is better suited to repeated random validation or when you need a specific train/test ratio.
Do both keep the class distribution balanced?
Yes. Both are stratified, so the proportion of each class in every train and test split approximately matches the original dataset. This matters most with imbalanced classes, where an unstratified split can produce folds missing a minority class entirely.
Can I control the test set size with StratifiedKFold?
Only indirectly — the test size is approximately 1/k of the dataset, so n_splits=5 gives a 20% test set. If you need an exact test proportion decoupled from the number of splits, use StratifiedShuffleSplit with its test_size parameter.
Why do I get 'The least populated class has too few members' error?
Stratification needs at least one sample per class per fold, so the smallest class must have at least n_splits members. Either reduce n_splits, merge or drop very rare classes, or gather more samples for the minority class.
Software Engineering Leader & Technical Author · Updated September 9, 2026