Selection Bias in Python
A backtesting distortion arising from non-random selection of assets, time periods, or signals that unconsciously favor positive results.
Definition
Selection Bias is the systematic error introduced when the data, time period, asset universe, or hypotheses chosen for backtesting are not randomly selected but have been — consciously or unconsciously — selected because they are known to produce favorable results. Unlike Survivorship Bias (which affects which assets are included) or Data Mining Bias (which affects which strategies are reported), Selection Bias is broader and more subtle: it includes selecting a test period that happened to suit the strategy, choosing only markets where the strategy is known to have worked historically, or reporting results from a favorable starting date.
Why It Matters in Backtesting
Selection Bias is pervasive and difficult to detect precisely because it often occurs subconsciously. A researcher who has heard that a momentum strategy worked particularly well in US equities from 2010–2020 and then backtests exclusively on that market and period has introduced selection bias before writing a single line of code. The remedy is pre-registration of hypotheses before seeing data, testing on multiple asset classes and time periods simultaneously, and requiring consistent performance across diverse market regimes rather than cherry-picked conditions.
Python Implementation
import numpy as np
import pandas as pd
def audit_selection_bias(strategy_fn, returns_by_market: dict,
returns_by_period: dict) -> dict:
"""
Stress-tests a strategy across multiple markets and periods to expose selection bias.
strategy_fn: callable(returns: pd.Series) -> float (e.g., Sharpe ratio)
returns_by_market: dict of {market_name: pd.Series of returns}
returns_by_period: dict of {period_label: pd.Series of returns}
"""
market_results = {
market: strategy_fn(returns)
for market, returns in returns_by_market.items()
}
period_results = {
period: strategy_fn(returns)
for period, returns in returns_by_period.items()
}
market_values = list(market_results.values())
period_values = list(period_results.values())
positive_markets = sum(1 for v in market_values if v > 0)
positive_periods = sum(1 for v in period_values if v > 0)
return {
"results_by_market": market_results,
"results_by_period": period_results,
"market_consistency": positive_markets / len(market_values) if market_values else 0,
"period_consistency": positive_periods / len(period_values) if period_values else 0,
"cross_market_std": np.std(market_values),
"cross_period_std": np.std(period_values),
"selection_bias_risk": "HIGH" if np.std(market_values) > 1.0 or np.std(period_values) > 1.0 else "LOW"
}Test this in a live environment
Stop running Jupyter notebooks locally. Paste this Selection Bias code directly into Valetha's Strategy Lab and run a full historical backtest in seconds.
Open the Python Strategy Lab