7 Backtesting Mistakes That Make Your Results Worthless

Your backtest says the strategy wins 70% of the time. Your live account says otherwise. That gap isn't variance and it isn't bad luck — it's the distance between testing a strategy and quietly cheating on the test without noticing you're doing it.

Almost every backtesting mistake pushes your results in the same direction: up. That's the tell. If your testing were honest, its errors would scatter both ways. Instead they all flatter you, which is why the live number is always the disappointing one. Here are the seven mistakes that make backtesting results worthless — and you're probably making at least three.

Each one below comes with a test you can run against your own log. Reading a list of mistakes and nodding is easy; finding out which three you're actually making takes about twenty minutes with the trades you've already recorded.

The twenty-minute self-audit

Open your backtest log. You don't need to redo anything yet — every mistake here leaves a fingerprint in numbers you already have.

#	Mistake	What it looks like in your log	The test
1	Hindsight	Win rate far above what the strategy makes live	Re-test 20 setups bar-by-bar and compare
2	Cherry-picking	Suspiciously few losses; gaps in the dates	Count skipped setups — if it's zero, you're not logging them
3	Small sample	Under ~100 trades	Ask what an edgeless strategy scores this often
4	One regime	All trades inside one trend or one quarter	Sort by month and look for the empty ones
5	Moving rules	Later trades outperform earlier ones	Split the log in half and compare the two win rates
6	Missed trades	No "skipped" or "no-fill" rows at all	Every real strategy misses trades; a log with none is fiction
7	Short window	Date range under a few months	Check whether your sample contains a drawdown

Mistake 5 has the sneakiest fingerprint and it's worth stating plainly: if the second half of your log outperforms the first half, that is not you getting better. You are the only variable that changed. Either you tightened the rules as you went, or you started unconsciously skipping the setups that had been losing. Both mean the sample describes two different strategies.

Why most backtesting doesn't work

The uncomfortable truth about why backtesting doesn't work for most people: they're not testing a strategy, they're auditioning for a conclusion they've already reached. They want the setup to work, so every ambiguous decision breaks in its favour. None of the mistakes below feel like cheating in the moment. That's exactly what makes them dangerous — each one is a small, reasonable-seeming shortcut that inflates the result, and they stack.

Mistake 1: You marked up a chart you could already see

This is the big one, and nearly everyone does it. You scroll back on a chart where the entire move is already printed, spot the setup sitting right before a clean 200-point run, and mark it as a winner. Of course it worked — you found it because you already knew price went up.

That's not a backtest. That's hindsight bias with a drawing tool. On a fully-loaded chart your eye is drawn to the setups that resolved cleanly and slides right past the identical-looking ones that failed. You can't un-see the outcome, so you can't judge the entry honestly. The only fix is to not have the outcome visible when you make the call.

How to tell: take twenty setups you already marked as winners, go back to the bar where you'd have entered, and hide everything to the right of it. Would you have taken it there, with no idea what came next? Most people lose three or four out of twenty on that test alone. That's your hindsight tax, and it applies to the whole sample.

Mistake 2: You only counted the setups that worked

Go back through your trading journal. How many "invalid" setups did you quietly not log because they lost? Cherry-picking is rarely deliberate — it's the ugly loss you decide "didn't really count" because the entry was a bit late, or the one you rationalise away because "I wouldn't have taken that live."

Every setup you exclude on a technicality is a thumb on the scale. A strategy's real win rate includes the losses that hurt, the ones that stopped you out by a tick, and the messy ones you're embarrassed by. Log every valid setup by your written rules or don't bother — a backtest of only the pretty trades is a highlight reel, not data.

How to tell: count the setups you rejected. Not the losses — the ones you looked at and decided didn't qualify. If that number is zero, or you can't produce it, you weren't logging rejections, which means every borderline call went in whichever direction the outcome suggested. A healthy log has a rejection pile roughly the size of the trade pile.

Mistake 3: Your sample is far too small

Twenty trades tell you almost nothing. A genuinely edgeless strategy will show a 65%+ win rate over 20 setups roughly one time in eight, purely by chance — so a good-looking small sample is exactly what a bad strategy produces on a lucky run. Thirty trades isn't proof of anything; it's a coin flip with commentary.

You need a few hundred logged setups across varied conditions before the numbers separate signal from noise — and because uncertainty falls with the square root of your sample, getting reliable takes far more trades than people expect. This one has its own article: How Many Backtests Do You Need Before Trusting a Strategy? walks through the actual math, and the backtest sample size calculator runs it on your own win rate — including the line that stings, which is how often a strategy with no edge would have scored as well as yours did.

There is a second reason the count matters, and it survives even when the win rate turns out to be real. A small sample almost never contains the worst losing streak the strategy is capable of, so the drawdown you plan around is the drawdown you happened to see rather than the one waiting for you. Run the numbers through the risk of ruin calculator and it simulates thousands of accounts on your win rate and reward:risk — the streak in the roughest five percent of them is usually a lot longer than anything in a hundred-trade backtest.

How to tell: this one is just a count. Under 100 logged setups, treat the win rate as a rumour. The trap is that fixing this mistake first feels productive and isn't — a bigger sample collected the same dishonest way is a more confident lie. Fix 1 and 2, then grow the sample.

Mistake 4: You skipped the periods where it bled

You tested your setup across a three-month stretch — and that stretch happened to be a clean trend that suited it perfectly. You never tested the choppy range that followed, or the news week that ran every stop. So you measured one market regime and assumed it was the whole market.

A setup that prints money in a trend can hand every dollar back in a range. If your backtest doesn't include the conditions that hurt your strategy, you haven't found your edge — you've found the weather it happens to like. Test across trends, ranges, and ugly news-driven sessions, or your win rate only describes a market that isn't here anymore.

How to tell: group your trades by month and print the win rate for each. A real edge is uneven but present across most of them. If two months carry the entire result and the rest hover around break-even, you don't have a strategy with a 60% win rate — you have a strategy that works in one condition and you happened to test during it. The months missing entirely are the more interesting question: why didn't the setup appear at all?

Mistake 5: You changed the rules halfway through

You started testing "enter on the sweep." Fifty setups in, you noticed the ones with a confirmation candle did better, so you started requiring that. Now your sample is a blend of two different strategies and your results describe neither.

Every mid-test tweak resets the count — you just don't restart the counter, so the old trades quietly contaminate the new rule. If you spot an improvement, good: write it down as the next version and test it from scratch as its own strategy — versioning your rules is what makes that possible instead of aspirational. Optimising the rules while you measure them is how you curve-fit a system to noise and call it an edge.

The procedure that catches this even when discipline slips is out-of-sample testing: freeze the rules, then run them over a block of history you haven't studied. A rule that was invented to explain the losses in one sample will always look brilliant on that sample — the only way to find out whether it's real is to test it somewhere it had no hand in the result.

How to tell: split your log down the middle by date and compare the two halves. If the second half is meaningfully better, the honest reading is that the rules drifted. Nobody's discretion improves that cleanly inside one sample — and if it did, the first half is now testing a strategy you no longer trade.

Mistake 6: You ignored the trades you'd have missed in real time

Here's the one almost nobody accounts for. In your backtest you catch every setup, because you're calmly scrolling through history. Live, you were asleep at 3am when the London setup fired. You were already in another position. The alert didn't trigger. You hesitated and missed the entry.

A backtest that assumes flawless, omnipresent execution is testing a version of you that doesn't exist. The setups you'd realistically have missed — and the bad fills, slippage and spread you'd realistically have eaten — are part of the strategy's true performance. Leave them out and you're testing a fantasy trader with perfect attention and zero costs.

How to tell: search your log for rows marked skipped, missed or no-fill. If there are none across hundreds of trades, that's the finding. Add two columns — one for why a setup wasn't taken, one for whether the entry price was actually available — and the gap between your test and your account starts becoming measurable instead of mysterious.

Mistake 7: You tested two weeks and called it proof

Two weeks of data can't prove anything, for two reasons at once: it's too few setups to be statistically meaningful, and it's almost always a single market condition. You cannot see how a strategy survives a correction if there wasn't one in your window.

Real testing needs real history — years, not weeks. Enough to contain the trends, ranges, reversals and news events you'll actually trade through. Anything less and you're not measuring an edge, you're measuring a fortnight's mood.

How to tell: find the worst losing run in your sample. If your biggest drawdown is three trades, your window was too short — every strategy worth trading has stretches that make you want to abandon it, and if yours hasn't shown you one, you haven't met it yet. Better to live through the bad patch in replay than to meet it for the first time with money on it.

Fix them in this order

They aren't equal, and the order matters more than people expect.

1 and 2 first, always. Hindsight and cherry-picking are the only two that corrupt every row in the log, so nothing measured before you fix them can be salvaged — not by a bigger sample, not by a longer window. This is why "I'll just backtest more" so rarely helps: it scales the error.

Then 5, because moving rules mean you can't even say what was tested. A blended sample can't be split, filtered or trusted; it has to be rerun.

Then 3, 4 and 7 together — they're the same problem wearing three hats. Not enough trades, not enough conditions, not enough calendar. Once the mechanics are honest, these get fixed by doing more of the same thing rather than something different.

6 last, not because it matters least but because it's the only one you can retrofit. Missed trades and realistic fills can be added as columns to an existing honest log and re-summed. You cannot retrofit honesty onto a log built by scrolling a chart you'd already seen.

How to backtest so your results actually mean something

Every mistake above has the same root cause and the same fix. Hindsight, cherry-picking, missed trades — they all come from seeing too much and testing too little. Fix it with honest mechanics:

• Replay candle by candle. Step through history one bar at a time so you only ever see what you'd have seen live. This single change kills hindsight bias, forces you to log the setups you'd actually have caught, and makes cherry-picking obvious. It's the difference between marking up the past and reliving it. That's the whole idea behind candle-by-candle replay.
• Test on years of real data. Enough history to cover every regime, so the losing periods are in the sample whether you like it or not.
• Write the rules first, log every setup, and don't move the goalposts. Including the ugly ones. Especially the ugly ones. Rules vague enough to argue with are how most of the mistakes above get in.

If you want the whole procedure in order rather than a list of things to avoid, how to backtest a trading strategy walks through all six stages. And if you trade Candle Range Theory or ICT, the step-by-step is already mapped out in How to Backtest the CRT Strategy and How to Backtest ICT Concepts — and if the strategy itself is still fuzzy, start with What Is CRT (Candle Range Theory)?.

You can do all of this for free in CRTLAB: pick a market, replay it bar by bar on real history, and build a sample you can actually trust. Stop backtesting in a way that lies to you.

FAQ

Why don't my backtest results match my live trading? Almost always because your backtest was too flattering, not because live is unlucky. Hindsight bias, cherry-picked setups, missed live trades and untested losing periods all inflate backtest results upward, so the honest live number comes in lower. Fix the testing and the gap shrinks. If your test survives every mistake on this page and live is still worse, the remaining gap has four specific sources — costs, fills, data and the trades you skip — and why backtest results differ from live trading breaks down how to measure each one.

What's the most common backtesting mistake? Hindsight bias — marking up a chart where the outcome is already visible. You unconsciously pick the setups that resolved cleanly and skip the identical ones that failed. Bar-by-bar replay, where you can't see the future, is the only real cure.

How many trades do I need for a backtest to be reliable? At least 100 logged setups to see a signal and ideally 300–400 across varied conditions before you trust it. Twenty or thirty trades can't tell a real edge from a lucky streak — the sample-size breakdown covers the math.

Does backtesting actually work? Yes — when it's honest. Done properly (bar-by-bar, full rules logged, years of data, every setup counted) it's the only way to know your win rate and expectancy before you risk money — and once you're live, the position size calculator is what keeps a normal losing streak from ending the account. Done the common way, it's just self-deception that gives you false confidence.

How do I know which of these mistakes I'm making? Run the twenty-minute audit near the top of this page against the log you already have. Each mistake leaves a fingerprint in numbers you've already recorded: an unrealistic win rate, no rejected setups, under 100 trades, every result inside one quarter, a second half that outperforms the first, no skipped rows, or a date range with no drawdown in it. You don't need to redo the testing to find out which ones apply — only to fix them.

Is a bigger backtest sample always better? Only once the mechanics are honest. A larger sample collected with hindsight and cherry-picking is a more confident wrong answer, not a better one, because the error scales with the trade count. Fix how you're testing first, then grow the sample — which is why sample size comes fourth in the fix order above, not first.

Can I fix a backtest I've already done, or do I have to start over? It depends which mistakes are in it. Missed trades, fills and costs can be retrofitted: add the columns and re-sum. Too few conditions or too short a window can be extended by testing more history with the same rules. But hindsight, cherry-picking and mid-test rule changes corrupt the rows themselves, so a log with any of those three has to be rerun. Run the twenty-setup test before you decide — if the number holds up, keep the log.

How much data should I use to backtest a strategy? Years, not weeks. Your sample needs to contain the trends, ranges and news events you'll actually trade through. A couple of weeks is both too small a sample and a single market regime, so it can't tell you how the strategy behaves when conditions change.