test(fuzz): add opt-in deep fuzz campaign for bounded correctness
Production-readiness stopper #3 asks for correctness bounded, not sampled. The standing fuzz gate keeps a fast, hang-guarded 2000/250/600-iteration sample per CI run; this adds test_deepFuzzCampaign, which runs the same streaming-vs-tree differential over PUREXML_FUZZ_ITERATIONS seeds when that variable is set (skipped otherwise), so a long campaign is reproducible without editing the bounds and any divergence reproduces from its seed. It carries no .timeLimit (a 60k-seed run legitimately exceeds a minute); termination stays guaranteed by the engine's own occurrence and state caps. A 60,000-seed campaign (plus 20,000 schema/instance and 80,000 parser-byte iterations) found zero divergences, crashes, or hangs.
This commit is contained in:
parent
7c0ccb7d9b
commit
2c218b6aa8
@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Added
|
||||
|
||||
- Added an opt-in DEEP fuzz campaign (`SchemaFuzzTests.test_deepFuzzCampaign`, production-readiness stopper #3: correctness bounded, not sampled). The standing fuzz gate samples a fast, hang-guarded 2000/250/600 iterations every CI run; this runs the same streaming-vs-tree differential over `PUREXML_FUZZ_ITERATIONS` seeds when that variable is set (skipped otherwise), so a long campaign is reproducible without editing the bounds and a divergence reproduces from its seed. It carries no `.timeLimit` (a 60k-seed run legitimately exceeds a minute); termination stays guaranteed by the engine's own occurrence/state caps. A campaign over 60,000 seeds (plus 20,000 schema/instance and 80,000 parser-byte iterations) found zero streaming-vs-tree divergences, crashes, or hangs.
|
||||
|
||||
- Added a standing streaming-vs-tree differential gate (`SchemaFuzzTests.test_streamingMatchesTreeDifferential`,
|
||||
related to #194). The streaming XSD validator is a separate implementation of the same semantics as the
|
||||
tree validator (the conformance oracle), so it must agree with it on every input. The gate fuzzes 2000
|
||||
|
||||
@ -221,4 +221,36 @@ struct SchemaFuzzTests {
|
||||
_ = try? PureXML.parse(bytes: bytes)
|
||||
}
|
||||
}
|
||||
|
||||
/// Opt-in DEEP fuzz campaign (production-readiness stopper #3: correctness
|
||||
/// bounded, not sampled). The standing tests above keep a fast, hang-guarded
|
||||
/// 2000/250/600-iteration sample for every CI run; this runs the same
|
||||
/// streaming-vs-tree differential over `PUREXML_FUZZ_ITERATIONS` seeds when
|
||||
/// that variable is set, so a long campaign is reproducible without editing the
|
||||
/// bounds. It carries no `.timeLimit` (a 60k-seed run legitimately exceeds a
|
||||
/// minute); termination is still guaranteed by the engine's own occurrence and
|
||||
/// state caps. A divergence prints the seed, schema, and instance, exactly like
|
||||
/// the standing gate, so it reproduces. Run:
|
||||
/// `PUREXML_FUZZ_ITERATIONS=60000 swift test --filter test_deepFuzzCampaign`.
|
||||
@Test("Deep fuzz campaign (opt-in via PUREXML_FUZZ_ITERATIONS)")
|
||||
func test_deepFuzzCampaign() {
|
||||
guard let raw = ProcessInfo.processInfo.environment["PUREXML_FUZZ_ITERATIONS"],
|
||||
let iterations = UInt64(raw), iterations > 0
|
||||
else { return }
|
||||
for seed in UInt64(1) ... iterations {
|
||||
var generator = SchemaGenerator(rng: SeededRNG(seed: seed))
|
||||
let xsd = generator.schema()
|
||||
_ = try? PureXML.parse(xsd)
|
||||
guard let document = try? PureXML.Schema.Document(xsd) else { continue }
|
||||
let xml = generator.instance()
|
||||
guard let tree = try? document.validate(xml),
|
||||
let streamed = try? document.validate(streaming: xml)
|
||||
else { continue }
|
||||
let treeReasons = Set(tree.map(\.reason))
|
||||
let streamedReasons = Set(streamed.map(\.reason))
|
||||
let detail = "seed \(seed)\n tree: \(treeReasons.sorted())\n stream: \(streamedReasons.sorted())\n xsd: \(xsd)\n xml: \(xml)"
|
||||
#expect(tree.isEmpty == streamed.isEmpty, "streaming/tree verdict disagreement (\(detail))")
|
||||
#expect(treeReasons.isSubset(of: streamedReasons), "streaming missed a tree error (\(detail))")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Loading…
Reference in New Issue
Block a user