Equity Direction Study
Honest evaluation of machine-learning equity direction forecasts
A quantitative research study examining whether gradient-boosted trees can predict five-day equity price direction better than logistic regression, using technical features on five liquid U.S. instruments. The contribution lies in its rigorous evaluation with walk-forward validation, threshold analysis, transaction costs, and naive baselines, revealing that the apparent edge does not survive scrutiny.
The problem
In quantitative finance, machine-learning models often show promising backtest results, but these can be misleading due to overfitting, improper validation, and ignoring transaction costs. Investors need a way to distinguish real predictive power from statistical noise, especially in efficient markets where edges are hard to find and even harder to sustain.
What I built
The study concludes with negative results: neither model achieved a ROC-AUC above 0.5 on the holdout, both were beaten by naive baselines, and the economic return was an artifact of three days. The comparison between models is inconclusive due to insufficient statistical power. The contribution is a documented pre-specified negative result and a demonstration of honest evaluation techniques.
How it works
The study pre-specifies two models—logistic regression and XGBoost—to predict five-day direction from eleven technical features on five instruments. It uses purged expanding-window cross-validation through 2024 with a five-date purge gap, then evaluates on an untouched 2025 holdout. Hard work included implementing moving-block bootstrap for uncertainty, decomposing the simulated portfolio return to show it relies on three periods, and calculating statistical power to reveal the design was underpowered by roughly a factor of five. The approach follows a rigorous backtesting protocol to ensure honest reporting.
Highlights
- Pre-specified models with an untouched 2025 holdout year to prevent overfitting
- Both models performed worse than an always-up rule with statistical significance
- Economic return decomposed to show 95.4% of excess return came from just three periods
- Power calculation revealed the design was underpowered by roughly a factor of five
- Adherence to the Arnott-Harvey-Markowitz backtesting protocol for honest evaluation
- Comparison includes proper scoring rules and naive baselines, not just accuracy
Let's build yours.
Tell me what you're trying to ship — you'll get a scoped plan and a straight answer.