Your Backtest Shows 15%, But Reality Shows 5%
You built a trading bot. Backtested it. Got 15% annual returns. You're ready to go live.
Three months later, you're down 8%. By month six, the EA is closed and you're manually trading again. The backtest wasn't lying. Your backtest was incomplete.
This is survivor bias—and it's destroying retail traders every single day.
What Survivor Bias Actually Is
Survivor bias happens when you only count the winners and pretend the losers don't exist.
Here's the mechanism: You test 50 trading strategies. 47 fail. 3 succeed. You publish the 3 that won. You got a 94% failure rate invisibly, but your results look perfect.
The trader reading your backtest has no idea you tested 47 failures. They only see the 3 winners. They think you're a genius. You're not—you're just the guy who kept testing until he got lucky.
This happens at scale. A broker publishes "top-performing EAs." What they're not publishing: the 400 EAs that got delisted after blowing up accounts.
Survivor bias isn't cheating. It's just how backtesting works unless you explicitly prevent it.
How DIY Traders Cherry-Pick Without Realizing It
You don't set out to lie. Here's what actually happens:
1. You test a strategy on 5 years of data. It wins 60% of trades but loses 4% annually. You move on.
2. You adjust the parameters. New settings: 55% win rate, 8% annual return. Better. Keep adjusting.
3. You test 200 parameter combinations. Most are garbage. Three combinations hit 15%+ returns.
4. You publish the best one. You're not hiding anything—you genuinely tested it.
But here's what actually happened: You tested 200 times until you found one that fit the historical data perfectly. The odds that exact combination will work on future data? Almost zero.
You didn't find a strategy that works. You found the combination that overfits the past.
The Math: 60-70% Haircut From Backtest to Reality
Professional audits consistently show the same pattern: real returns are 60-70% lower than backtest claims.
Your 15% backtest? Realistically worth 4.5-6% live.
Here's why:
Look-ahead bias (20-30% loss): Your backtest accidentally had access to future data. Maybe your indicator used tomorrow's close to calculate today's signal. Maybe your backtest assumed you could enter at the exact low of the day. Real trading doesn't work that way.
Curve fitting (15-25% loss): You optimized parameters on historical data. When you submit 200 parameter combinations, one will fit the data perfectly. That's math, not skill. On future data, it fails.
Market selection (10-20% loss): You tested on the years the market went up. You didn't test through 2008, 2020, or crypto winter 2022. Backtests that skip the crashes always look profitable.
Slippage assumptions (10-15% loss): Your backtest assumes 0.5 pips slippage. Real execution: 2-3 pips on retail brokers. Your backtest assumed 1:100 leverage always available. Real trading: spreads widen in volatility, liquidity disappears.
Add these up. 15% - (25% + 20% + 15% + 15%) = 4.75% real expected return.
That's the math every professional trader knows. That's the math most DIY traders skip.
Why Your Perfect Backtest Feels Real
Your EA closed 1,000 trades. Won 612. Lost 388. Profit factor 1.85. Drawdown 12%. On your 5-year backtest, it's flawless.
None of that means it'll work tomorrow.
The problem: you're looking at a backtest built on the one market regime that worked. Your EA is tuned to catch 2015-2020 bull market moves. It doesn't know how to handle 2022 sideways chop.
Your parameters are optimized for EURUSD 4H timeframe. Change to 1H? Breaks. Test on GBPUSD? Loses money. That's not robustness—that's overfitting.
Professional traders call this "the law of induction." Your backtest proves the EA worked in the past. It says absolutely nothing about whether it works in the future. Yet 99% of traders treat it as a promise.
What Professional Audits Actually Check
When you deploy an EA without verification, you're betting on an incomplete test. Real audits use walk-forward validation and out-of-sample testing.
Out-of-sample testing: Train the EA on years 1-3. Test it on years 4-5 it never saw. If it works, great. If it crashes, you know before you deploy live.
Walk-forward validation: Test on 2015, validate on 2016. Test on 2015-2016, validate on 2017. Roll forward one year at a time. Real market performance, real verification.
Monte Carlo testing: Run the EA on randomized bar sequences. If it works on random data, the edge wasn't real—you were just curve-fitting. If it fails on randomized bars but wins on real bars, you've got an edge.
Slippage modeling: Don't assume perfect fills. Model real broker slippage, real spreads, real liquidity. We test with 2-3 pip slippage minimum. If the EA dies at realistic slippage, you know before deploying.
Benchmark comparison: Your EA returns 15% over 5 years. S&P 500 returned 80%. Your EA is underperformance dressed as success.
A real backtest report shows all of this. If you get a backtest without walk-forward validation or out-of-sample testing, it's not an audit—it's a sales pitch.
The Cost of Trading on a Fake Backtest
You deploy the EA live. First week: +1.2%. You're convinced. Week two: -0.8%. Normal variance, you think. Month two: -5%. Month three: -8%.
Now you're questioning everything. The EA is closed. You've manually traded instead, losing another 3%. Total damage: $4,000+ on a $300 EA that looked perfect.
That's the actual cost of survivor bias. Not the commission. The missed opportunity cost. The emotional damage of watching a "15% strategy" become a -15% reality.
The traders who survive and scale aren't the ones who built the perfect backtest. They're the ones who built the verified one.
Getting a Real Backtest Instead of a Sales Pitch
You can't eliminate backtest bias—it's built into how testing works. But you can audit it away.
A real EA development process includes:
- Multiple parameter sets tested, not just the best one cherry-picked
- Out-of-sample validation on data the EA never saw
- Walk-forward testing across multiple market cycles
- Real slippage and spread modeling, not optimistic assumptions
- Comparison to benchmarks (S&P, buy-and-hold, risk-free rate)
- Written report showing how the backtest was verified
At Alorny, every EA we build includes a full backtest report. Not just the glossy numbers—the validation, the methodology, the worst-case scenarios. You get to see exactly how we verified that 5% real return, not promised that fictional 15%. Built in 45 minutes, delivered in hours.
Key Takeaways
Survivor bias makes your backtest look better than reality. Test 50 strategies, publish 3 winners, and suddenly you're a genius. The other 47 are invisible.
60-70% of backtest returns typically vanish when you go live. Look-ahead bias, curve fitting, market selection, and slippage assumptions erode returns predictably.
A real backtest is validated, not published. Out-of-sample testing, walk-forward analysis, and realistic slippage modeling separate real edges from curve-fit luck.
The cost of a fake backtest isn't the commission—it's the losses. Deploy an overfit EA and you're not just losing the backtest return. You're losing your confidence and your capital.