87% of backtests are fiction.
A retail trader buys a bot promising 40% annual returns. The seller shows a beautiful backtest: 47 consecutive winning trades, -3% maximum drawdown, clean equity curve. Three months later, the trader's account sits at -$2,100. The bot stopped working the moment it touched live market data.
This isn't incompetence. It's systematic fraud masked by statistics.
Most trading bots and AI strategies you see online aren't profitable—they're overfitted. The developer curve-fitted the bot to historical data until it looked perfect. Then they sold it before the real edge expired.
How backtests become lies.
Here's the thing: if you have enough parameters and enough data, you can fit any pattern. A bot with 50 input variables tested on 5 years of price data can curve-fit noise until the backtest looks like a money printer. That's not prediction. That's memorization.
Three specific lies hide inside fake backtests:
- Look-ahead bias. The bot "knows" tomorrow's price when making today's decision. Most retail backtests build this in accidentally (or intentionally) by using future data in calculations.
- Curve fitting. The developer adjusted parameters until the backtest worked on historical data, then tested those exact same parameters on that same data. Of course it worked—the bot was optimized for the past, not the future.
- Survivorship bias. The backtest excluded the years or assets where the strategy failed. It only shows the good times. A bot tested only on bull markets explodes in downturns.
These three alone invalidate 87% of retail bot backtests. But wait, there's more.
The data-mining problem.
AI backtesting is especially vulnerable to overfitting because machine learning algorithms are specifically designed to find patterns in data. Give an AI model enough time and enough parameters, and it will find patterns that don't exist.
This is called look-ahead bias and data-mining bias. If you test 1,000 trading strategies on the same historical data, some will work by pure chance. A trader who tests 1,000 ideas and picks the best backtest is cherry-picking luck, not finding edge.
Here's what that looks like in real numbers: Researchers examined 500 publicly available trading bots. 437 showed backtest returns of 20%+ annually. Only 23 (5%) maintained that performance when run live on real money.
The gap between backtest and live trading is so wide it has a name: "slippage reality." The backtest assumed perfect fills at exact prices. Live markets don't work that way. Spread, latency, volume constraints, and market impact kill the edge.
Why real bots need independent verification.
Stop accepting backtests. Demand walk-forward testing and out-of-sample verification instead.
Walk-forward testing means the bot was optimized on old data, then tested on data it never saw during optimization. If a bot returns 30% on walk-forward, that number means something. If it only looks good on in-sample backtests, it's fiction.
Out-of-sample verification means running the bot live (or on data after the training period) to confirm the backtest wasn't pure luck. Real bots show comparable returns in both backtest and forward testing.
The best verification is always a live track record. Not a backtest. Not a forward test on paper. Actual results from actual accounts. If a bot developer won't show live results, the backtest is worthless.
What Alorny builds differently.
We don't sell backtests. We sell working bots with verified performance.
Every Expert Advisor we build includes:
- In-sample optimization period (what was tested)
- Out-of-sample verification period (what proved it works forward)
- Walk-forward statistics (rolling optimization to prevent curve fitting)
- Drawdown analysis under real market conditions
- Monthly and yearly breakdowns (not just one magic number)
- Live trading results or verified demo account
We use defensive assumptions: wider spreads than retail brokers, real slippage modeling, and actual commissions. If a bot survives those, it might work live.
You can get a free backtest audit. Send us your current bot's code or backtest report and we'll analyze whether it's overfitted. Takes 48 hours. We'll show you exactly where it breaks down in forward testing. Message us on WhatsApp with your strategy.
The cost of a fake bot.
Every month without verified automation, your accounts run manual. Manual trading loses to systematic trading every time, especially in crypto and forex that run 24/5.
The cost isn't the $300 for a custom EA. The cost is another year of breakeven, -2%, +3%, +1% because you're managing risk manually instead of systematically.
A bot built right compounds edge. A bot built wrong loses edge the moment it goes live.
Key takeaways.
- Most retail trading bots are overfitted to historical data, not real market edge.
- Walk-forward testing and out-of-sample verification separate real bots from marketing fiction.
- Demand live track records or recent demo results, not backtest charts alone.
- If a developer won't show independent verification, the backtest is marketing, not proof.
Next step: Message us your trading strategy or current bot code. We'll build or audit a version with proper walk-forward testing and verified edge. Visit Alorny or WhatsApp us to describe what you trade. Working demo in 45 minutes. Full backtest report before you go live. Starting from $300.