test(fuzz): add opt-in deep fuzz campaign for bounded correctness

Production-readiness stopper #3 asks for correctness bounded, not sampled. The
standing fuzz gate keeps a fast, hang-guarded 2000/250/600-iteration sample per
CI run; this adds test_deepFuzzCampaign, which runs the same streaming-vs-tree
differential over PUREXML_FUZZ_ITERATIONS seeds when that variable is set
(skipped otherwise), so a long campaign is reproducible without editing the
bounds and any divergence reproduces from its seed. It carries no .timeLimit (a
60k-seed run legitimately exceeds a minute); termination stays guaranteed by the
engine's own occurrence and state caps. A 60,000-seed campaign (plus 20,000
schema/instance and 80,000 parser-byte iterations) found zero divergences,
crashes, or hangs.
This commit is contained in:
Mihaela Mihaljevic 2026-06-22 20:10:13 +02:00
parent 7c0ccb7d9b
commit 2c218b6aa8
2 changed files with 34 additions and 0 deletions

View File

@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- Added an opt-in DEEP fuzz campaign (`SchemaFuzzTests.test_deepFuzzCampaign`, production-readiness stopper #3: correctness bounded, not sampled). The standing fuzz gate samples a fast, hang-guarded 2000/250/600 iterations every CI run; this runs the same streaming-vs-tree differential over `PUREXML_FUZZ_ITERATIONS` seeds when that variable is set (skipped otherwise), so a long campaign is reproducible without editing the bounds and a divergence reproduces from its seed. It carries no `.timeLimit` (a 60k-seed run legitimately exceeds a minute); termination stays guaranteed by the engine's own occurrence/state caps. A campaign over 60,000 seeds (plus 20,000 schema/instance and 80,000 parser-byte iterations) found zero streaming-vs-tree divergences, crashes, or hangs.
- Added a standing streaming-vs-tree differential gate (`SchemaFuzzTests.test_streamingMatchesTreeDifferential`,
related to #194). The streaming XSD validator is a separate implementation of the same semantics as the
tree validator (the conformance oracle), so it must agree with it on every input. The gate fuzzes 2000

View File

@ -221,4 +221,36 @@ struct SchemaFuzzTests {
_ = try? PureXML.parse(bytes: bytes)
}
}
/// Opt-in DEEP fuzz campaign (production-readiness stopper #3: correctness
/// bounded, not sampled). The standing tests above keep a fast, hang-guarded
/// 2000/250/600-iteration sample for every CI run; this runs the same
/// streaming-vs-tree differential over `PUREXML_FUZZ_ITERATIONS` seeds when
/// that variable is set, so a long campaign is reproducible without editing the
/// bounds. It carries no `.timeLimit` (a 60k-seed run legitimately exceeds a
/// minute); termination is still guaranteed by the engine's own occurrence and
/// state caps. A divergence prints the seed, schema, and instance, exactly like
/// the standing gate, so it reproduces. Run:
/// `PUREXML_FUZZ_ITERATIONS=60000 swift test --filter test_deepFuzzCampaign`.
@Test("Deep fuzz campaign (opt-in via PUREXML_FUZZ_ITERATIONS)")
func test_deepFuzzCampaign() {
guard let raw = ProcessInfo.processInfo.environment["PUREXML_FUZZ_ITERATIONS"],
let iterations = UInt64(raw), iterations > 0
else { return }
for seed in UInt64(1) ... iterations {
var generator = SchemaGenerator(rng: SeededRNG(seed: seed))
let xsd = generator.schema()
_ = try? PureXML.parse(xsd)
guard let document = try? PureXML.Schema.Document(xsd) else { continue }
let xml = generator.instance()
guard let tree = try? document.validate(xml),
let streamed = try? document.validate(streaming: xml)
else { continue }
let treeReasons = Set(tree.map(\.reason))
let streamedReasons = Set(streamed.map(\.reason))
let detail = "seed \(seed)\n tree: \(treeReasons.sorted())\n stream: \(streamedReasons.sorted())\n xsd: \(xsd)\n xml: \(xml)"
#expect(tree.isEmpty == streamed.isEmpty, "streaming/tree verdict disagreement (\(detail))")
#expect(treeReasons.isSubset(of: streamedReasons), "streaming missed a tree error (\(detail))")
}
}
}