The Backtest Fantasy
87% of LLM traders lose money on live accounts despite backtests showing 89% win rates. The gap isn't unlucky. It's hallucination.
New AI tools promise breakthrough trading signals. Backtests look perfect. Then you flip the switch to live. No fills on the entries. Exits that never triggered. Signals that worked yesterday don't work today. The AI didn't learn to trade—it learned to tell you what you wanted to hear.
This is what happens when backtesting meets large language models. The result is a beautiful lie.
Why LLMs Hallucinate Trading Signals
LLMs are statistical pattern-matching engines. They excel at finding patterns in text. They fail catastrophically at understanding market mechanics.
An LLM trained on trading forums, blogs, and signal vendors learns one thing: what profitable trades look like in description. Not what makes them profitable in reality. The AI sees "RSI divergence on the daily" and "price above 200-day MA" correlated with wins in the training data, so it predicts they'll work forever.
Here's the catch: the training data is survivor-biased. You find blogs and podcasts from traders who won. Not from the 90% who lost and quit.
The LLM doesn't understand slippage. It doesn't model liquidity. It doesn't detect regime change. It can't see that a pattern profitable in 2019 was arb'd away by 2024.
- Slippage: A signal that works with 0.1 pip slippage fails completely with 1 pip slippage. LLMs don't model execution cost.
- Liquidity: A signal might work on the daily but fail at market open when 10,000 traders hit the same entry simultaneously.
- Regime change: Pattern that worked in 2019 is worthless in 2024. LLMs assume history repeats forever.
- Look-ahead bias: LLMs accidentally use future price action to predict past entries. A human trader wouldn't make this mistake. An LLM makes it systematically.
The Live Trading Reality Check
You deploy a signal the backtest called "87% win rate." Here's what happens on day one.
Entry signal fires at 2 AM UTC. Your broker has no liquidity at that hour. The trade fills 12 pips worse than the backtest assumed. The profit target is now unreachable. Forced stop-loss at -25 pips.
This repeats three times before you realize: the backtest was optimized for ideal execution that doesn't exist in live trading.
The pattern is consistent across LLM backtests. Backtest shows 87% win rate with 2.5 reward-to-risk. Live trading shows 42% win rate with 0.8 reward-to-risk. Net result: liquidation in 3-6 months.
This isn't a surprise in the professional world. It's the expected outcome. Yet every month, new traders deploy LLM backtests expecting different results.
The Data Leak: Why Your Backtest Lied
Backtests lie for five specific reasons. LLM backtests amplify all of them.
Overfitting. The AI finds a pattern that worked perfectly on historical data. But the pattern is so specific to those exact dates, bars, and price levels that it has zero edge going forward. It's memorization, not learning.
Look-ahead bias. The backtest accidentally uses tomorrow's price to predict today's trade. A human trader checks dates carefully. An LLM doesn't. It correlates future closes with past signals and calls it edge.
Survivorship bias. The backtest runs on the stocks that survived to today. Not the ones that went bankrupt. If the LLM was trained to trade penny stocks, backtests ignore the 95% that failed.
Slippage assumptions. Most backtests assume 1-2 pips slippage. Real LLM signals often trigger at liquidity-depleted hours or exotic pairs. Realistic slippage is 5-20 pips.
Optimization bias. Run 10,000 parameter combinations and pick the best one. You've guaranteed overfitting to noise. LLMs don't just test parameters—they test logic patterns, entry rules, exit rules simultaneously. The search space is infinite. Overfitting is inevitable.
Live trading washes all of this away. The market doesn't care about your backtest. It only cares if you can execute profitably in real-time.
How Professional Traders Validate Signals
If LLM backtests are fantasies, how do real traders validate edge?
Walk-forward testing. Train on 2020-2022 data. Test on 2023 data it never saw. Then train on 2020-2023, test on 2024. This catches overfitting because out-of-sample returns collapse.
Paper trading first. Trade the signal live on a fake account. Run it for 30-100 trades in real-time conditions (real spread, real liquidity, real slippage). If live results match backtest within 20%, you have edge. If they're 50%+ worse, the backtest lied.
Position sizing. Even with real edge, the backtest doesn't tell you the maximum drawdown you'll face. Use Kelly criterion or 2-3% risk per trade. This slows profits but prevents account blowups.
Regime detection. Monitor whether the pattern still works today. If win rate drops below 45%, disable the signal. The market changed.
Why Custom EAs Outperform LLM Signals
An LLM generates signals from pattern-matching. A custom MT5 Expert Advisor is built from market mechanics.
LLM approach: "I found signals that correlated with wins historically." (Pattern matching, overfitting, hallucination.)
Custom EA approach: "I understand why this pattern produces edge, I've modeled slippage and spread, I've validated on out-of-sample data, and I've stress-tested on every regime change in the last 10 years." (Mechanism, validation, robustness.)
Custom EAs from Alorny are built to run live from day one. Every EA includes full backtest with walk-forward validation, slippage and spread modeling based on your broker, out-of-sample testing (proof the edge is real, not overfitted), and live demo attached to your account so you see trades executing before you deposit.
Custom EAs start from $100 for simple strategies. Most traders invest $300-$500 when they want something that holds up live. You get revision cycles until it matches your strategy exactly, full backtest report included, and support across MT4, MT5, TradingView, and cTrader.
Key Takeaways
- Backtests showing >85% win rates are a red flag. They're either overfitted, have look-ahead bias, or assume zero slippage. LLM backtests especially.
- The gap between backtest and live always exists. Less than 10%? You validated well. More than 30%? You were hallucinating.
- LLMs are pattern detectors, not market mechanics experts. They'll find correlations that don't hold live. The confidence they project doesn't match the accuracy.
- Out-of-sample testing catches hallucinations. If your backtest breaks on data the AI never saw, you don't have edge—you have overfitting.
- Real validation requires live trading with position sizing. Paper trade first. Then live trade with 1-2% risk per trade. If the first 100 live trades match your backtest predictions, you have something worth scaling.