The Backtest You're Actually Running
You load your strategy into MT4, adjust the parameters, and run a backtest. It looks good—so you tweak it, run again, and it looks better. You do this 47 times.
Here's what you don't see: the 47 versions that failed. You deleted them.
This is survivor bias. You're not testing one strategy 47 times. You're running 47 strategies and keeping only the one that worked. Your backtest report shows the winner. The losers never existed.
Why 95% of EAs Fail When Live Money Hits
The stat isn't exaggerated. According to OANDA's retail trader research, 95%+ of trading accounts lose money within 12 months. Expert Advisors have similar failure rates on live accounts.
Most traders blame the EA. The real culprit is the test. You tested on data that was pre-filtered to succeed.
Here's the trap: a strategy that survived historical data might be exploiting a lucky period, not a real market pattern. COVID crashed bonds in 2020. Your EA crushed it. Does it work in rising-rate environments? The backtest won't tell you—you never tested it there because that period lost money, so you left it out.
95% of EAs fail on live accounts because they were tested on data that was already filtered to win.
The Two Backtests That Never Get Run
There are three versions of your strategy in existence: the one you tested (the survivor), and two you'll never test (the failures).
- The backwards test: Run your final EA on data BEFORE your backtest period. Did it work? If not, you got lucky in your chosen timeframe. A real strategy works across multiple periods.
- The adjacent-market test: If your EA trades SPY, run it on QQQ with the exact same rules. Did it work? If your strategy only profits in one ticker, it's not a strategy—it's curve-fitting that ticker's specific behavior.
- The walk-forward test: As walk-forward testing requires, test on one period, optimize on another, then test on a third period with zero re-optimization. If it dies in the new period, it overfit the training data.
Most traders run zero of these tests. That's why their EAs fail live.
How to Spot Survivor Bias Before You Deploy Real Money
Don't guess. Use this checklist:
- Did you test the losing periods? Run your EA on years when your market crashed. Did it have a strategy for those years, or did it just avoid trading?
- Does your EA work on similar markets? If it only trades GBP/USD, test it on EUR/USD with zero changes. If it fails, you've fit one currency pair, not a principle.
- Can you explain every winning trade? A real signal has a mechanism. "It bought when price touched the moving average" is a mechanism. "It happened to work here" is not.
- Did a professional developer build this or did you? If you built it yourself, you probably kept the winners and deleted the losers. A professional developer tests on data they didn't cherry-pick.
The best question: would you stake your monthly rent on this EA right now? If not, it's not tested. It's just backtested.
What Real EA Testing Looks Like
At Alorny, we test EAs the way you should: across multiple markets, multiple timeframes, and data we don't control. No cherry-picking.
For every strategy we build, we run all five validation tests:
- Backtest on the main historical period (e.g., 2020–2024)
- Forward test on data after the backtest period (Jan 2025–present, live market data)
- Adjacent-market test on similar pairs with zero rule changes
- Stress test on high-volatility periods (earnings, FOMC announcements, overnight gaps)
- Demo live test before touching real money
Only after all five tests pass do we deploy to real money. And we deliver the complete backtest report—every trade, every drawdown, every test result, no exceptions.
Starting from $100 for simple EAs, we include the full testing suite with every build. Tell us what you trade and we'll show you the exact backtest before you deposit a dollar.
Key Takeaways
- Survivor bias is invisible—until it costs you money. Your backtest shows only the strategy you kept, not the 47 you deleted.
- 95% of EAs fail live because they were tested on pre-filtered, cherry-picked data.
- Real testing requires multiple validation tests—backwards tests, adjacent-market tests, walk-forward tests, and stress tests.
- Ask for the full backtest report—every trade, every test, every historical period. If a developer won't show you, the EA probably won't hold up live.
- The moment you go live is when you learn the truth. Test everything before that moment.