Verdict
Reject, retest, observe, or advance: one of four, stated plainly.
Approach
A verdict is only worth what the method behind it can withstand.
A validation is not a second opinion on a backtest. It is an adversarial process designed to find the reasons a result should not be believed, applied before anyone commits capital or engineering to it.
Method
Six things that have to be true.
-
Register the claim before testing it
The hypothesis, the universe, the horizon, the cost assumptions and the bar for success are written down first. A result graded against a bar that moved is not a result, and the only way to prove the bar did not move is to have published it.
-
Compute significance honestly
On a k-day hold, daily observations overlap, so the naive t-statistic is inflated by roughly the square root of k. Significance is recomputed on non-overlapping blocks with a serial-correlation correction, and the Sharpe is deflated for the number of attempts actually made.
-
Test against the right null
A candidate is compared against a shuffled twin and a plausible baseline, not against zero. Beating zero is easy. Beating a permutation of your own data, and a naive forecast that already works, is the question worth asking.
-
Separate underpowered from absent
A result below the bar is not automatically a null. If the test could not have resolved an effect large enough to matter, the finding is 'underpowered', not 'no effect'. Absence of evidence and evidence of absence are recorded differently.
-
Keep what failed
Rejected candidates are registered rather than discarded, with the reason and the scope of the rejection. A search that remembers only its successes will keep rediscovering the same dead ends and will overstate its own hit rate.
-
Let forward time be the judge
Surviving candidates are frozen and evaluated forward, point-in-time. A forward track cannot be refitted, which is precisely what makes it slow, unflattering, and worth more than any backtest.
Deliverable
What a Strategy Validation Report contains.
Bounded on purpose. It answers one question about one claim, and it attaches everything needed to disagree with it.
Evidence
The tests run, the nulls used, and the numbers as computed, including the unflattering ones.
Limits
What the result does not establish: the universe it was not tested on, the regimes it did not see, the costs it assumed.
Failure map
Where the claim breaks, and which conditions would change the verdict.
A report describes what the evidence supports about a specific claim under stated conditions. It is not investment advice, not a recommendation, and not a representation that any strategy will perform in future.