Ask what the test
can actually establish.
A successful software check, a historical simulation and an independent future observation answer different questions. Their conclusions should stay separate.
01 / Three kinds of evidence
| Evidence | Useful for | Does not establish |
|---|---|---|
| Historical replay | Inspecting behaviour on recorded data | Unseen-sample performance |
| Artificial scenarios | Testing controlled paths and branches | Real-world frequency or returns |
| Forward simulation | Recording decisions as observations arrive | Live fills or a proven investment edge |
Historical comparisons need consistent inputs, sensible data cutoffs, and explicit treatment of costs, corporate actions and execution assumptions. A result can remain dependent on the selected instruments or period even when the calculation is reproducible.
A formal out-of-sample evaluation requires a separately defined window, frozen rules and observations not used to select the model. An ordinary simulation record should not be relabelled as that evaluation after the results are known.
02 / Why use artificial scenarios?
Controlled paths make it possible to inspect behaviour that a particular historical sample may rarely contain. The scenario framework covers six families: a sustained rise; a rise, sell-off and rebound; a decline, rebound and renewed decline; a sustained fall; choppy appreciation; and choppy depreciation.
Comparing versions across these paths can reveal inactive branches, unexpected timing or sensitivity to reversals. All defined branches should remain visible, including unfavourable or unchanged outcomes. A scenario is a diagnostic example, with no implied probability of occurring.
Artificial tests complement historical research. They cannot determine expected returns or replace an independently observed market sample.
03 / What makes a record reviewable?
Date the input and the conclusion
State which sessions were complete, what was still forming, and when the interpretation was written. An old observation keeps its original date when it moves into the archive.
Keep assumptions and versions together
Retain the inputs, model version, comparison setup and accounting assumptions that produced a result. Checksums help detect changed files; they do not prove that the underlying data or investment thesis is correct.
Show the gaps
Separate a generated decision, a simulated order and a confirmed fill. Preserve missing observations and failed runs. Re-reading the same saved record does not create another independent observation.
The practical standard is to report what was checked, what failed, and what remains unresolved. Technical readiness, evidence completeness and strategy effectiveness need separate conclusions.
A guide to reading evidence.
This guide describes research principles and selected practices documented in AiTrader’s internal comparison and validation work through October 17, 2026. It does not certify that every study has met every standard or that formal out-of-sample validation is complete.
No current market assessment, performance guarantee or live execution claim is made. Internal datasets, model parameters and execution rules are not published here.