7 Backtesting Mistakes That Make Your Results Worthless

Your backtest says the strategy wins 70% of the time. Your live account says otherwise. That gap isn't variance and it isn't bad luck — it's the distance between testing a strategy and quietly cheating on the test without noticing you're doing it.

Almost every backtesting mistake pushes your results in the same direction: up. That's the tell. If your testing were honest, its errors would scatter both ways. Instead they all flatter you, which is why the live number is always the disappointing one. Here are the seven mistakes that make backtesting results worthless — and you're probably making at least three.

Why most backtesting doesn't work

The uncomfortable truth about why backtesting doesn't work for most people: they're not testing a strategy, they're auditioning for a conclusion they've already reached. They want the setup to work, so every ambiguous decision breaks in its favour. None of the mistakes below feel like cheating in the moment. That's exactly what makes them dangerous — each one is a small, reasonable-seeming shortcut that inflates the result, and they stack.

Mistake 1: You marked up a chart you could already see

This is the big one, and nearly everyone does it. You scroll back on a chart where the entire move is already printed, spot the setup sitting right before a clean 200-point run, and mark it as a winner. Of course it worked — you found it because you already knew price went up.

That's not a backtest. That's hindsight bias with a drawing tool. On a fully-loaded chart your eye is drawn to the setups that resolved cleanly and slides right past the identical-looking ones that failed. You can't un-see the outcome, so you can't judge the entry honestly. The only fix is to not have the outcome visible when you make the call.

Mistake 2: You only counted the setups that worked

Go back through your journal. How many "invalid" setups did you quietly not log because they lost? Cherry-picking is rarely deliberate — it's the ugly loss you decide "didn't really count" because the entry was a bit late, or the one you rationalise away because "I wouldn't have taken that live."

Every setup you exclude on a technicality is a thumb on the scale. A strategy's real win rate includes the losses that hurt, the ones that stopped you out by a tick, and the messy ones you're embarrassed by. Log every valid setup by your written rules or don't bother — a backtest of only the pretty trades is a highlight reel, not data.

Mistake 3: Your sample is far too small

Twenty trades tell you almost nothing. A genuinely edgeless strategy will show a 65%+ win rate over 20 setups roughly one time in eight, purely by chance — so a good-looking small sample is exactly what a bad strategy produces on a lucky run. Thirty trades isn't proof of anything; it's a coin flip with commentary.

You need a few hundred logged setups across varied conditions before the numbers separate signal from noise — and because uncertainty falls with the square root of your sample, getting reliable takes far more trades than people expect. This one has its own article: How Many Backtests Do You Need Before Trusting a Strategy? walks through the actual math.

Mistake 4: You skipped the periods where it bled

You tested your setup across a three-month stretch — and that stretch happened to be a clean trend that suited it perfectly. You never tested the choppy range that followed, or the news week that ran every stop. So you measured one market regime and assumed it was the whole market.

A setup that prints money in a trend can hand every dollar back in a range. If your backtest doesn't include the conditions that hurt your strategy, you haven't found your edge — you've found the weather it happens to like. Test across trends, ranges, and ugly news-driven sessions, or your win rate only describes a market that isn't here anymore.

Mistake 5: You changed the rules halfway through

You started testing "enter on the sweep." Fifty setups in, you noticed the ones with a confirmation candle did better, so you started requiring that. Now your sample is a blend of two different strategies and your results describe neither.

Every mid-test tweak resets the count — you just don't restart the counter, so the old trades quietly contaminate the new rule. If you spot an improvement, good: write it down, then test it from scratch as its own strategy. Optimising the rules while you measure them is how you curve-fit a system to noise and call it an edge.

Mistake 6: You ignored the trades you'd have missed in real time

Here's the one almost nobody accounts for. In your backtest you catch every setup, because you're calmly scrolling through history. Live, you were asleep at 3am when the London setup fired. You were already in another position. The alert didn't trigger. You hesitated and missed the entry.

A backtest that assumes flawless, omnipresent execution is testing a version of you that doesn't exist. The setups you'd realistically have missed — and the bad fills, slippage and spread you'd realistically have eaten — are part of the strategy's true performance. Leave them out and you're testing a fantasy trader with perfect attention and zero costs.

Mistake 7: You tested two weeks and called it proof

Two weeks of data can't prove anything, for two reasons at once: it's too few setups to be statistically meaningful, and it's almost always a single market condition. You cannot see how a strategy survives a correction if there wasn't one in your window.

Real testing needs real history — years, not weeks. Enough to contain the trends, ranges, reversals and news events you'll actually trade through. Anything less and you're not measuring an edge, you're measuring a fortnight's mood.

How to backtest so your results actually mean something

Every mistake above has the same root cause and the same fix. Hindsight, cherry-picking, missed trades — they all come from seeing too much and testing too little. Fix it with honest mechanics:

• Replay candle by candle. Step through history one bar at a time so you only ever see what you'd have seen live. This single change kills hindsight bias, forces you to log the setups you'd actually have caught, and makes cherry-picking obvious. It's the difference between marking up the past and reliving it. That's the whole idea behind candle-by-candle replay.
• Test on years of real data. Enough history to cover every regime, so the losing periods are in the sample whether you like it or not.
• Write the rules first, log every setup, and don't move the goalposts. Including the ugly ones. Especially the ugly ones.

If you trade Candle Range Theory or ICT, the step-by-step is already mapped out in How to Backtest the CRT Strategy and How to Backtest ICT Concepts — and if the strategy itself is still fuzzy, start with What Is CRT (Candle Range Theory)?.

You can do all of this for free in CRTLAB: pick a market, replay it bar by bar on real history, and build a sample you can actually trust. Stop backtesting in a way that lies to you.

FAQ

Why don't my backtest results match my live trading? Almost always because your backtest was too flattering, not because live is unlucky. Hindsight bias, cherry-picked setups, missed live trades and untested losing periods all inflate backtest results upward, so the honest live number comes in lower. Fix the testing and the gap shrinks.

What's the most common backtesting mistake? Hindsight bias — marking up a chart where the outcome is already visible. You unconsciously pick the setups that resolved cleanly and skip the identical ones that failed. Bar-by-bar replay, where you can't see the future, is the only real cure.

How many trades do I need for a backtest to be reliable? At least 100 logged setups to see a signal and ideally 300–400 across varied conditions before you trust it. Twenty or thirty trades can't tell a real edge from a lucky streak — the sample-size breakdown covers the math.

Does backtesting actually work? Yes — when it's honest. Done properly (bar-by-bar, full rules logged, years of data, every setup counted) it's the only way to know your win rate and expectancy before you risk money. Done the common way, it's just self-deception that gives you false confidence.

How much data should I use to backtest a strategy? Years, not weeks. Your sample needs to contain the trends, ranges and news events you'll actually trade through. A couple of weeks is both too small a sample and a single market regime, so it can't tell you how the strategy behaves when conditions change.