Out-of-Sample Testing: How to Prove Your Backtest Isn't Just a Story

Here's a test you can run right now. Take the strategy you're most confident in, and ask yourself one question: was every rule in it written before or after I looked at the charts I tested it on?

If any rule was added because of something you saw in the data — a session that kept losing, a filter that cleaned up the equity curve, a "this only works when…" clause — then your backtest result is not a measurement. It's a description of the past, and descriptions of the past have a 100% hit rate on the past.

Out-of-sample testing is the fix. It's the difference between a strategy that worked on the charts you studied and a strategy that works on charts in general — and it's the step almost every retail backtest skips.

What in-sample and out-of-sample actually mean

Two definitions, and they're the whole article:

• In-sample data is the price history you used to build the strategy. Everything you looked at while deciding the rules, plus everything you tweaked the rules in response to.
• Out-of-sample data is price history the strategy has never touched. You wrote the rules, you froze them, and only then did you run them over this data.

The result that means something is the out-of-sample one. The in-sample result is best understood as a floor — the best your strategy will ever look — because you had a hand in producing it.

This is not the same idea as the sample-size question. How many backtests you need is about having enough trades for the number to be stable. Out-of-sample testing is about whether the number is measuring anything at all. You can have 500 trades and still be looking at pure hindsight, and that combination is the most convincing wrong answer in trading — a big sample makes a fitted result look authoritative.

Why manual backtesters need this more than algo traders do

The usual objection: "I don't optimise anything. I'm a discretionary trader — there are no parameters to curve-fit."

There are. You just are them.

An algorithmic trader curve-fits by running an optimiser across parameter values and picking the peak. A discretionary trader curve-fits by scrolling through charts, noticing what would have worked, and adjusting the mental model accordingly. Same process, no log file. And it's worse in two ways: it happens continuously, and you never see the parameter you changed, so you can't count how many you tried.

Every one of these is a fitted parameter, even though none of them has a number attached:

• "I only take this in the London session" — added after London happened to work in the data you studied.
• "I need a body close beyond the level, not just a wick" — added after wick entries lost in one sample.
• "Not on NFP week."
• "The range has to be at least as big as the last few candles."

None of these are wrong. Any of them might be a genuine improvement. The point is that you cannot tell which ones are real using the data that generated them, and there's no optimiser output to remind you that you tried fourteen variations. Mistake 5 in the backtesting mistakes guide covers changing the rules mid-test as an error of discipline. This article is the procedure that makes it detectable when it happens anyway.

Look-ahead bias: the leak that makes any split useless

Splitting your data does nothing if information from the future leaks into your decisions. Look-ahead bias is when your test uses knowledge that wasn't available at the moment of the trade. In manual testing it arrives through four doors, and the first is the one everyone uses:

1. The chart is already loaded. You open a chart, scroll left, and mark up setups. Every one of those decisions is made with the outcome visible in your peripheral vision. You can't unsee it, and you will systematically mark the sweeps that worked. This is the reason candle-by-candle replay exists as a category at all — not convenience, integrity of the test.

2. Unclosed higher-timeframe candles. You take a 5-minute entry because the 4-hour candle "is bullish". Live, that 4-hour candle had three hours left to run and may have finished bearish. If your rules reference a higher-timeframe candle, they must specify closed, and your test must respect it.

3. Hindsight-chosen instruments and periods. Testing your breakout strategy on the instrument you already know trended that year is look-ahead bias at the level of test design. You knew the answer before you picked the question.

4. Indicators and levels that repaint. Anything drawn from a swing that is only identifiable after the fact — including most "structure" markings — needs a rule for when the swing becomes confirmed, or you'll place levels that a live trader could not have drawn yet.

Get the leaks closed before you bother splitting anything. An out-of-sample test that peeks is just an in-sample test with extra steps.

How to split your data

Three approaches. Pick by how much history you have, not by preference.

Approach	How it works	Best when	Main weakness
Simple holdout	Split history in time. Build on the first ~70%, test on the last ~30%, untouched.	You're testing one strategy version and have a decent stretch of history.	One verdict from one market period. A regime change inside the holdout can sink a good strategy or flatter a bad one.
Walk-forward	Build on a window, test on the window that follows, then roll both forward and repeat.	You have years of data and want to know whether the edge persists over time.	Slower by hand. Requires real discipline to avoid carrying knowledge between windows.
Instrument holdout	Build on two or three instruments, test on a different one with similar characteristics.	Your strategy claims to be structural rather than instrument-specific.	Only valid if the claim is genuinely structural — don't test an index strategy on a currency pair and call it a failure.

Two rules that apply to all three:

Split by time, never randomly. Randomly sampling trades across the whole history is a standard technique in other fields and it is wrong here. Markets have regimes and price is serially correlated — random sampling puts January's answer in your test set while February's near-identical conditions sit in your training set, and the "out-of-sample" result is contaminated by construction.

The out-of-sample period must be recent, and must be last. Building on 2024 and testing on 2022 tells you your strategy would have worked on data you can no longer trade. Keep the newest block as the holdout.

Running a walk-forward test by hand

Walk-forward has a reputation as an algo-only technique. It isn't — it's just a discipline about the order you look at things in, and a manual trader can run it with a spreadsheet:

1. Divide your history into blocks — quarters work well for intraday strategies, half-years for higher-timeframe ones.
2. Study block 1. Write the rules. Freeze them in a document with today's date on it.
3. Test on block 2 without changing anything. Record the result. This result counts.
4. Now you're allowed to study block 2 as well, and revise the rules.
5. Test the revised rules on block 3. Record. That result counts too.
6. Repeat to the end of your history.

What you end up with is a series of results that were each produced before the strategy saw that data. Stitch them together and you have something far closer to an honest equity curve than any single backtest produces. It also answers a question a simple holdout can't: is the edge stable, or was it real in 2023 and gone by 2025? Two strategies with identical total returns and different stability profiles are not equally tradeable.

How much data to hold out

Reason in trades, not in months. A six-month holdout that produces eleven setups has told you nothing — you've correctly avoided one bias and walked straight into a sample size problem.

Work backwards: decide the minimum number of out-of-sample trades you'd need to draw any conclusion, then hold out however much calendar time produces that. The backtest sample size calculator will give you the number for your win rate and the margin of error you're willing to live with; for most retail strategies the honest answer is bigger than the holdout people intend to keep.

If your entire history can't produce a meaningful in-sample and a meaningful out-of-sample set, that's information too. It means you're not yet in a position to validate the strategy, and the correct next step is more data or a more frequent setup — not a smaller holdout.

Reading the result

Expect degradation. Out-of-sample performance is almost always worse than in-sample, and a small drop is normal rather than damning — some of your in-sample result was fitting, some was luck, and both come out in the wash.

There is no magic pass mark, and anyone quoting one is inventing it. What you can do is set the criteria before you look, which is the entire discipline here:

• Write down what "pass" means in advance. For example: positive expectancy, at least n trades, and a win rate that clears the break-even rate for your average R. Decide it before the test, or you will negotiate with yourself afterwards.
• Compare expectancy, not win rate. A win rate that falls while average R rises is a strategy behaving differently, not failing.
• Check that the trades are distributed. An out-of-sample period that passes on the back of two enormous winners has told you about two trades.
• A negative out-of-sample expectancy is a result, not a setback. It's the test doing its job. The alternative was finding out with money.

The rule that makes all of this work

Out-of-sample data is single-use. The moment you look at the results, that block is in-sample forever. If you tweak the rules and re-run them on the same block, the second number is no longer an out-of-sample number — you've just fitted to it, one iteration at a time. This is the most commonly broken rule in the whole discipline, and it's broken innocently: "I'll just try it with the session filter and see."

So when a strategy fails out-of-sample, you have exactly three honest options:

1. Bin it. Cheapest outcome available, and it just saved you real money.
2. Revise, then test on a fresh block you haven't used. This is precisely what walk-forward formalises, and it's why walk-forward is worth the extra effort — it gives you a legitimate way to keep iterating.
3. Revise and forward-test. If you're out of historical data, the honest next block is the future: run it live at minimum size or on demo, and treat that as your out-of-sample set. Slower, but uncontaminated. Backtesting vs paper trading covers how these fit together as a sequence.

Budget your history like a resource, because that's what it is. Every block you burn is one you can never validate on again.

The mistakes worth naming

• Testing on the same data you learned the strategy from. If you learned the setup from a course that used 2023 examples, 2023 is in-sample for you personally, whether or not you ran a test on it.
• Splitting after the fact. Testing everything, then labelling the last third "out-of-sample" once you've already seen it. The label doesn't change what you know.
• Running out-of-sample until it passes. Twenty variations against one holdout will find a passing one by chance. That's optimisation against your test set with more steps.
• Holding out too little to matter. A holdout with fifteen trades in it isn't a validation, it's a formality.
• Treating in-sample as the headline. The in-sample equity curve is the one that's screenshot-worthy and the one that means the least.

How to actually do this in a replay tool

Practically, the workflow that keeps the split honest:

• Pick your blocks by date first, before you look at any of them, and write the dates down. Deciding the boundary after browsing is how contamination starts.
• Do your building on the in-sample block with the candle-by-candle replay running forward only — never scrolled ahead — so even your in-sample work is free of look-ahead bias. That matters more than it sounds: a clean in-sample process means the gap between your two results is small enough to interpret.
• Freeze the rules in a document with a date on it. If it's not written down, it will drift, and you'll never be able to say what the out-of-sample test was a test of.
• Run the holdout in one pass without stopping to adjust anything. Log every trade, including the ones you'd rather not.
• Then, and only then, compare.

CRTLAB is built for exactly this loop — candle-by-candle replay across the markets we cover, so you can jump to a specific date range, step forward one candle at a time without ever seeing what's next, and run a frozen ruleset over a holdout period properly. It's the same tooling as any other backtesting workflow; the discipline is what turns it into validation rather than storytelling.

For a worked example of freezing a ruleset before testing it, how to backtest the CRT strategy step by step walks through pinning down an entire model — which is the prerequisite for any of this, because you can't hold rules constant across a split if they were never written down.

FAQ

What is out-of-sample testing in trading? Running a strategy over price history it played no part in creating. You build and refine the rules on one block of data, freeze them, then test on a separate block you haven't studied. The out-of-sample result is the one worth believing, because your knowledge of that data didn't contribute to it.

How much data should I hold out for out-of-sample testing? Enough to produce a meaningful number of trades — think in trades, not months. A common starting point is around 70% for building and 30% for testing, split by time with the most recent block held out, but the real constraint is that the holdout must generate enough setups to draw a conclusion from.

What's the difference between out-of-sample testing and walk-forward testing? A simple out-of-sample test splits your history once and gives you one verdict. Walk-forward repeats the process in rolling windows — build on one block, test on the next, roll forward — so you get a series of honest results and can see whether the edge persists across market regimes rather than existing in one period.

Can I do out-of-sample testing as a discretionary trader? Yes, and you need it more than an algo trader does. Discretionary traders curve-fit by adjusting their mental model as they scroll, with no record of how many variations they tried. Write the rules down, freeze them, and test on data you haven't looked at — the procedure is the same, only the optimiser is invisible.

What is look-ahead bias in a backtest? Using information in a decision that wouldn't have been available at that moment. In manual testing it usually means marking up a chart whose outcome is already on screen, referencing a higher-timeframe candle that hadn't closed yet, or picking the instrument and period because you already knew they behaved well.

What if my strategy fails the out-of-sample test? That's the test working. You can abandon the strategy, revise it and test on a fresh block you haven't used, or revise it and forward-test live at minimum size. What you can't do is tweak it and re-run on the same holdout — once you've seen those results, that block is in-sample forever.

How much worse should out-of-sample results be? Some degradation is normal, because part of any in-sample result is fitting and luck. There's no universal threshold, which is why you set your pass criteria — expectancy, minimum trades, break-even win rate for your average R — before running the test rather than after seeing the number.

Does out-of-sample testing replace a big sample size? No, they're separate requirements and you need both. Sample size determines whether your result is stable; out-of-sample determines whether it's measuring anything real. A large, fitted backtest is more misleading than a small one, because the size makes it persuasive.