NodetteHQ/PureYAML
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

test(validation): extend the finding-coverage law to both layers

Browse Source 
The finding-coverage meta-test governed only the model validator's default
checks. Bring the tagged validator's defaults under the same law, and make
the law precise as either layer grows past one check.

Each check now runs in isolation, in a validator holding only it, built from
the check's own AnyCheck. A fixture therefore cannot lean on a sibling check
to fail, which the previous model test (running the whole default validator)
would have allowed once a second model check shipped. Set-equality asserts
the fixtures cover exactly each layer's default checks.

The tagged set ships failing and near-miss fixtures for both built-in checks:
duplicate versus one-character-apart keys, and a built-in tag on the wrong
node kind versus a supported, kind-consistent one. A planted orphan fixture
was confirmed to fail both the set-equality and the isolation tests, then
removed. 405 tests pass.
This commit is contained in:
Mihaela M 2026-06-28 17:10:03 +02:00
parent 47960a85c8
commit 8f87a1ecd0
2 changed files with 83 additions and 34 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
CHANGELOG.md
View File

@ -18,12 +18,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
coverage reason is corrected to note that merges and aliases are resolved at parse time, so they
are not a validation-phase invariant: every `validated` manifest entry now names an invariant a
check actually enforces.
- A finding-coverage meta-test makes validation coverage a tested law: every rule in the default
validator must ship a failing fixture and a near-miss (just-valid) fixture, and the meta-test
asserts the fixture registry covers exactly the default rule set, naming any default rule without
fixtures or any fixture for a rule that no longer ships. Each rule is also exercised in isolation,
failing its failing fixture and passing its near miss. A planted orphan fixture was confirmed to
fail the meta-test.
- A finding-coverage meta-test makes validation coverage a tested law, now across both validating
layers: every default check, in the model validator and in the tagged validator, must ship a
failing fixture and a near-miss (just-valid) fixture. The meta-test asserts the fixture set covers
exactly each layer's default checks, naming any default check without fixtures or any fixture for a
check that no longer ships. Each check is exercised in isolation, in a validator holding only it, so
a fixture cannot lean on a sibling check to fail and the law stays precise as a layer grows past one
check. A planted orphan fixture was confirmed to fail both the set-equality and the isolation tests.
- Type-directed validation, the core of the validation-rules idiom: a `Check<Subject, Document>`
carries a positive description, a `when` predicate, and a Bool or multi-error check, and fires only
on values of its `Subject` type wherever they occur in the tree. `AnyCheck` erases the subject type

104
Tests/ValidationCoverageMetaTests.swift
View File

@ -1,49 +1,97 @@
@testable import PureYAML
import Testing
/// "Coverage is itself a tested law" (validation-rules.md): every rule in the default validator must
/// ship a failing fixture and a near-miss succeeding fixture, and a meta-test asserts the registry
/// covers exactly the default rule set, naming any gap. Adding a default rule without its fixtures,
/// or leaving a fixture for a rule that no longer ships, fails a test before it is deliberate.
/// "Coverage is itself a tested law" (validation-rules.md), applied to both validating layers. Every
/// default check, in the model validator and in the tagged validator, must ship a failing fixture and
/// a near-miss (just-valid) fixture. A meta-test asserts the fixture set covers exactly the layer's
/// default checks, naming any default check without fixtures or any fixture for a check that no longer
/// ships. Each check is exercised in isolation (a validator holding only it), so a fixture cannot lean
/// on another check to fail and the law stays precise as each layer grows past one check.
@Suite("Validation finding-coverage law")
struct ValidationCoverageMetaTests {
/// One failing document and one near-miss (just-valid) document per default rule, keyed by the
/// rule's description (the same identity the validator removes and lists by).
private static let fixtures: [String: (failing: PureYAML.Model.Value, nearMiss: PureYAML.Model.Value)] = [
"Mapping keys are unique": (
private struct ModelFixture {
let check: PureYAML.Validation.AnyCheck<PureYAML.Model.Value>
let failing: PureYAML.Model.Value
let nearMiss: PureYAML.Model.Value
}
private static let modelFixtures: [ModelFixture] = [
.init(
check: .init(PureYAML.Validation.BuiltinCheck.mappingKeysAreUnique),
failing: .mapping(.init([
.init(key: "a", value: .int(1)),
.init(key: "a", value: .int(2)),
])),
// The boundary the rule almost trips: two keys one character apart.
// The boundary the check almost trips: two keys one character apart.
nearMiss: .mapping(.init([
.init(key: "a", value: .int(1)),
.init(key: "b", value: .int(2)),
]))
])),
),
]
@Test("Every default rule has a failing and a near-miss fixture, and no fixture is orphaned")
func test_fixturesCoverExactlyTheDefaultRuleSet() {
let ruled = Set(PureYAML.Validation.Validator().validationDescriptions)
let covered = Set(Self.fixtures.keys)
let missing = ruled.subtracting(covered)
let orphaned = covered.subtracting(ruled)
#expect(missing.isEmpty, "default rules without coverage fixtures: \(missing.sorted())")
#expect(orphaned.isEmpty, "coverage fixtures for rules that no longer ship: \(orphaned.sorted())")
private struct TaggedFixture {
let check: PureYAML.Validation.AnyCheck<PureYAML.Tagged.Node>
let failing: PureYAML.Tagged.Node
let nearMiss: PureYAML.Tagged.Node
}
@Test("Each default check fails its failing fixture and passes its near-miss")
func test_eachCheckFailsAndNearMisses() throws {
// The default validator carries exactly the default checks, each keyed by a description the
// fixture registry is asserted (above) to cover one-for-one.
let validator = PureYAML.Validation.Validator()
for (description, fixture) in Self.fixtures {
#expect(throws: PureYAML.Validation.Issue.Collection.self, "check '\(description)' did not fail its failing fixture") {
try validator.validate(fixture.failing)
private static let taggedFixtures: [TaggedFixture] = [
.init(
check: .init(PureYAML.Tagged.BuiltinCheck.mappingKeysAreUnique),
failing: .mapping(.init(pairs: [
.init(key: "a", value: .scalar(.init(rawValue: "1", value: .int(1)))),
.init(key: "a", value: .scalar(.init(rawValue: "2", value: .int(2)))),
])),
nearMiss: .mapping(.init(pairs: [
.init(key: "a", value: .scalar(.init(rawValue: "1", value: .int(1)))),
.init(key: "b", value: .scalar(.init(rawValue: "2", value: .int(2)))),
])),
),
.init(
check: .init(PureYAML.Tagged.BuiltinCheck.builtInTagsAreSupportedAndKindConsistent),
// A built-in tag on the wrong node kind: !!map applied to a scalar.
failing: .scalar(.init(rawValue: "x", value: .string("x"), tag: .mapping)),
// The boundary: a built-in tag that is supported and kind-consistent, !!str on a scalar.
nearMiss: .scalar(.init(rawValue: "x", value: .string("x"), tag: .string)),
),
]
@Test("Model fixtures cover exactly the model validator's default checks")
func test_modelFixturesCoverDefaults() {
let ruled = Set(PureYAML.Validation.Validator().validationDescriptions)
let covered = Set(Self.modelFixtures.map(\.check.description))
#expect(ruled.subtracting(covered).isEmpty, "model default checks without fixtures: \(ruled.subtracting(covered).sorted())")
#expect(covered.subtracting(ruled).isEmpty, "model fixtures for checks that no longer ship: \(covered.subtracting(ruled).sorted())")
}
@Test("Each model check fails its failing fixture and passes its near miss in isolation")
func test_modelChecksFailAndNearMiss() throws {
for fixture in Self.modelFixtures {
let isolated = PureYAML.Validation.Validator(checks: [fixture.check])
#expect(throws: PureYAML.Validation.Issue.Collection.self, "check '\(fixture.check.description)' did not fail its failing fixture") {
try isolated.validate(fixture.failing)
}
// The near miss must pass: nothing proves the check stays quiet on valid input otherwise.
_ = try validator.validate(fixture.nearMiss)
_ = try isolated.validate(fixture.nearMiss)
}
}
@Test("Tagged fixtures cover exactly the tagged validator's default checks")
func test_taggedFixturesCoverDefaults() {
let ruled = Set(PureYAML.Tagged.Validator().validationDescriptions)
let covered = Set(Self.taggedFixtures.map(\.check.description))
#expect(ruled.subtracting(covered).isEmpty, "tagged default checks without fixtures: \(ruled.subtracting(covered).sorted())")
#expect(covered.subtracting(ruled).isEmpty, "tagged fixtures for checks that no longer ship: \(covered.subtracting(ruled).sorted())")
}
@Test("Each tagged check fails its failing fixture and passes its near miss in isolation")
func test_taggedChecksFailAndNearMiss() throws {
for fixture in Self.taggedFixtures {
let isolated = PureYAML.Tagged.Validator(checks: [fixture.check])
#expect(throws: PureYAML.Validation.Issue.Collection.self, "check '\(fixture.check.description)' did not fail its failing fixture") {
try isolated.validate(fixture.failing)
}
_ = try isolated.validate(fixture.nearMiss)
}
}
}