Testing
Five tiers. A test belongs to exactly one, and the tier determines what a failure means.
| Tier | Directory | What it proves | Exists |
|---|---|---|---|
| Unit | tests/unit/ |
One algorithm against a closed-form answer | yes |
| Property | tests/property/ |
Invariants over many generated inputs, seeded | yes |
| Integration | tests/integration/ |
One pipeline stage against a frozen capability contract | yes |
| Validation | tests/validation/ |
Output against published reference data | yes |
| Determinism | tests/determinism/ |
Bit-identical repeat runs; cross-platform agreement | yes |
| Evals | evals/ |
Agent behaviour against checkable outcomes | no |
The "Exists" column is the honest state of this repository right now. A
directory appears in tests/CMakeLists.txt in the same commit that adds its
first test, never in advance.
Rules that make the tiers mean something
Reference values come from published sources, never from the implementation. A test that captures current output as its expected value proves only that the code still does what it did, which is worth having but is not validation. Such a test is labelled a regression-lock in its own name and carries a comment naming the validated case it is anchored to. Every other test cites a document.
Integration tests are written without reading the implementation. They are written against the capability's documented contract. A test written by reading the code tests the code's opinion of itself.
Property-based tests carry the invariants that example-based tests miss. Round-trip conversions, quaternion normalisation, and the frame relationships that must hold at every attitude — that the wind and body rotations are mutual inverses, that the stability frame never moves the body y-axis, that elementary rotations add their angles. These are the tests that catch a transposed rotation matrix, which no single hand-picked example reliably does. Composition associativity is not among them; it would be worth adding.
No number in the V&V report is typed. Every figure it states about galata's
own behaviour is measured at render time and referred to in prose by name. The
generator refuses to produce a document if a case note contains a value it also
computes, and scripts/gen-verification.sh fails on an unresolved placeholder —
which is left visible rather than dropped, because a sentence with the number
silently removed still reads fine and says nothing.
Anything shared between a test and the report lives in one place for the same reason: the determinism battery, the perturbation-amplification study and the hand-assembled NT-33A matrices are all libraries, not copies. Two implementations would be two answers to the same question.
The V&V report's summary table is generated, and its claims are checked.
docs/VERIFICATION.md is produced from tools/validation/case_registry.cpp,
which declares every validation case: its reference, its status, and the tests
that stand behind it. Four tests reconcile that declaration with reality —
evidence must name a test that is actually registered in the binary it claims
to live in; a capability declaring implemented and validated must be backed
by a case that validates it; a case marked not implemented may not name a
capability that exists. The reasoning is in that file's header, and it comes
down to this: a hand-typed status line is a claim nothing checks, and this one
drifted three times before it was generated.
Coverage is a diagnostic, not a target. The project aims at a test-to-source line ratio of 0.3 or better and treats a sudden drop as a signal to look, not as a gate to satisfy.
Running them
cmake --preset dev
cmake --build --preset dev
ctest --preset dev
ctest --preset dev -L unit runs one tier. Labels match the table above.