fix(schema,regex): bound NFA state explosion and class-subtraction crash; land XSTS baseline

Running the full W3C XSTS archive to completion exposed three crash and hang
classes, now fixed with regression tests, which unblocked the #129 first
measurement.

- Content-model and regex NFA builders now cap total state count (2^20),
  degrading further occurrence/quantifier repetition to star, so nested
  high-maxOccurs schemas no longer OOM (was 8 GB on the MS Particles set).
- Content-model validation advances one active state-set per child instead of
  re-walking the prefix each time, removing the O(children^2) spin.
- Regex class subtraction [abc-[b]] no longer traps on a reversed range, and a
  reversed range such as [z-a] is rejected cleanly.

XSTS first measurement baselined: 75 valid schemas rejected, 2467 invalid
schemas accepted, 604 valid instances rejected, 582 invalid instances accepted
across 14383 groups. The suite runs green in 8s, opt-in via XSTS_ROOT.

Refs #129, #122.
This commit is contained in:
Mihaela Mihaljevic 2026-06-13 16:11:28 +02:00
parent 1254755733
commit 85d435d035
10 changed files with 217 additions and 46 deletions

View File

@ -7,13 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Fixed
- Three crash and hang classes uncovered by running the full XSTS archive to completion (#129). (1) Nested occurrence explosion: the content-model NFA capped each particle's `maxOccurs` unroll at 16384 but not the total state count, so nested high-`maxOccurs` particles multiplied (a sequence of 16384 holding a choice of 16384 is 2.7e8 states) and OOM-killed the process at 8 GB on the Microsoft Particles set; the builder now also caps total states (2^20), degrading further repetition to star. The regex `{n,m}` quantifier builder had the same unbounded unroll and the same cap is applied. (2) Quadratic content-model validation: `sequenceStructureErrors` called `follow(after: prefix)` once per child and `follow` re-walked the whole prefix from the start each time, so a content model over many children spun at 100% CPU; matching is now incremental (one active state-set advanced per child), turning O(children squared) into O(children). (3) Character-class subtraction trap: the regex pattern `[abcd-[d]]` (XSD class subtraction) was parsed as a `d` to `[` range, and since `d` (100) exceeds `[` (91) the reversed `ClosedRange` trapped (SIGBUS); `-[` now always opens a subtraction, and any reversed range (such as `[z-a]`) is a clean rejection rather than a trap.
### Changed
- Scanner-side validity findings are now located structured values (validation-framework audit follow-up): `DocumentType.validityFindings` carries `ValidityFinding` values (reason plus the declared name they are about) instead of strings, `DTDSchema.declarationErrors` holds ready `ValidationError`s, and every declaration-level finding renders at its declaration's coding path (`x`, `r/@a`) instead of "at root of document", matching the XSD consistency-errors precedent. Finding texts are unchanged.
### Added
- W3C XSTS runner, first measurement (#129, #122): the W3C XML Schema Test Suite (2006-11-06 release, ~13k test groups across NIST, Sun, Microsoft, and Boeing contributions; W3C Document License, never vendored) runs opt-in via XSTS_ROOT: every schemaTest must compile when expected valid and be rejected when expected invalid, and every instanceTest validates against its group's schema with the expected outcome; hrefs resolve against the referencing file and include/import locations from disk relative to the schema. The four disagreement counts are asserted exactly; the baselines are pending the first measurement (scheduled 2026-06-14), whose failure messages carry the counts to transcribe, and current failures are written to /tmp/xsts-failures.txt for the burn-down. Building the runner immediately caught a denial-of-service bug: a schema with maxOccurs in the trillions (groupF009v) OOM-killed the process because the content-model automaton unrolled occurrence counts literally; unrolling is now capped at 16384 with larger repetitions treated as unbounded, the libxml2 posture. The roadmap mermaid records the closed behavioral-parity set (#141-#144, collation) and the #139 performance epic.
- W3C XSTS runner, first measurement (#129, #122): the W3C XML Schema Test Suite (2006-11-06 release, ~13k test groups across NIST, Sun, Microsoft, and Boeing contributions; W3C Document License, never vendored) runs opt-in via XSTS_ROOT: every schemaTest must compile when expected valid and be rejected when expected invalid, and every instanceTest validates against its group's schema with the expected outcome; hrefs resolve against the referencing file and include/import locations from disk relative to the schema. The four disagreement counts are asserted exactly; the first measurement landed 2026-06-13 and is baselined at 75 valid schemas rejected, 2467 invalid schemas accepted, 604 valid instances rejected, and 582 invalid instances accepted across 14383 groups, with per-case deviations written to /tmp/xsts-failures.txt for the burn-down. Building the runner immediately caught a denial-of-service bug: a schema with maxOccurs in the trillions (groupF009v) OOM-killed the process because the content-model automaton unrolled occurrence counts literally; unrolling is now capped at 16384 with larger repetitions treated as unbounded, the libxml2 posture. The roadmap mermaid records the closed behavioral-parity set (#141-#144, collation) and the #139 performance epic.
### Added

View File

@ -99,7 +99,7 @@ flowchart TB
EncodingMismatch["#137 encoding-declaration mismatches: complete"]:::done
BaseURI["#138 per-entity base URIs: complete"]:::done
Epic122["#122 official spec suites"]:::epic
XSTS["#129 W3C XSD test suite (XSTS): runner in, occurrence-cap DoS fix; first measurement 2026-06-14"]:::review
XSTS["#129 W3C XSD test suite (XSTS): first measurement 2026-06-13 baselined (deviations 75/2467/604/582); occurrence-explosion and reversed-range crashes fixed, suite green in 8s"]:::review
XSLTTS["#130 XSLT 1.0 suite (xalan-test): runner + baselines in, burning down (125)"]:::review
Parity100["Behavioral parity closed: #141 top-level params, #142 public streaming, #143 XPath budget, #144 WASI runtime proof, pl/ru sort collation"]:::done
Epic139["#139 faster than libxml2, measured + streaming (benchmarks in docs/benchmarks.md; quadratics killed; unsafe authorized)"]:::epic

View File

@ -62,6 +62,20 @@ extension PureXML.Regex {
return states.count - 1
}
/// `{n,m}` quantifiers unroll into automaton states. Mirroring the XSD
/// content-model builder (`Schema.ContentNFABuilder`), each repetition is
/// capped and the total state count is bounded: past the ceiling a
/// repetition degrades to star, the libxml2 posture. Without this an
/// `xs:pattern` such as `a{1000000000}` (the XSTS msMeta Regex set)
/// allocates until the process is OOM-killed (#129). Real patterns are
/// tens of states, so only pathological quantifiers reach the ceiling.
private static let repeatUnrollCap = 16384
private static let totalStateCap = 1 << 20
private var stateBudgetExhausted: Bool {
states.count >= Self.totalStateCap
}
private mutating func fragment(for node: Node) -> (start: Int, accept: Int) {
switch node {
case .empty:
@ -109,14 +123,22 @@ extension PureXML.Regex {
private mutating func repeatedFragment(_ node: Node, minimum: Int, maximum: Int?) -> (start: Int, accept: Int) {
let start = addState()
var current = start
for _ in 0 ..< minimum {
let boundedMin = Swift.min(minimum, Self.repeatUnrollCap)
var unrolled = 0
for _ in 0 ..< boundedMin {
if stateBudgetExhausted { break }
let part = fragment(for: node)
states[current].epsilon.append(part.start)
current = part.accept
unrolled += 1
}
let accept = addState()
if let maximum {
appendOptional(node, count: maximum - minimum, from: current, to: accept)
if stateBudgetExhausted || unrolled < boundedMin || minimum > Self.repeatUnrollCap {
// An absurd minimum, or a build that hit the total state ceiling
// mid-unroll, repeats freely (star) past the cap.
appendStar(node, from: current, to: accept)
} else if let maximum, maximum - boundedMin <= Self.repeatUnrollCap {
appendOptional(node, count: maximum - boundedMin, from: current, to: accept)
} else {
appendStar(node, from: current, to: accept)
}
@ -126,6 +148,11 @@ extension PureXML.Regex {
private mutating func appendOptional(_ node: Node, count: Int, from: Int, to accept: Int) {
var current = from
for _ in 0 ..< max(0, count) {
if stateBudgetExhausted {
// Ran out of state budget: let the remainder repeat freely.
appendStar(node, from: current, to: accept)
return
}
let part = fragment(for: node)
states[current].epsilon.append(part.start)
states[current].epsilon.append(accept)

View File

@ -197,9 +197,14 @@ extension PureXML.Regex {
add(member, to: &cls)
return
}
if peek() == "-", peek(1) != "]" {
// A "-" immediately before "[" opens a class subtraction (`[abc-[b]]`),
// not a range, so it is left for parseClass; "-]" is a literal hyphen.
// Otherwise "-" introduces a range whose endpoints must be ordered: a
// reversed range like [z-a] is an error, not a trapping `low ... high`.
if peek() == "-", peek(1) != "]", peek(1) != "[" {
advance()
guard case let .scalar(high) = try nextClassMember() else { throw RegexError.badClass }
guard low <= high else { throw RegexError.badClass }
cls.ranges.append(low ... high)
} else {
cls.ranges.append(low ... low)

View File

@ -313,19 +313,19 @@ extension PureXML.Schema.ComplexValidator {
func sequenceStructureErrors(_ particle: PureXML.Schema.Particle, children: [PureXML.Model.Element], at path: XSDPath, into errors: inout [XSDFailure]) {
let nfa = PureXML.Schema.ContentNFABuilder.build(particle)
let steps = Self.childSteps(children)
var prefix: [PureXML.Model.QualifiedName] = []
// Advance one active state-set across the children rather than re-walking
// the prefix per child (which is quadratic over the content model, #129).
var current = nfa.startStates()
for (index, child) in children.enumerated() {
let allowed = nfa.follow(after: prefix).allowed
if allowed.contains(where: { $0.matches(child.name) }) {
prefix.append(child.name)
} else {
guard let next = nfa.step(current, over: child.name) else {
let allowed = nfa.admissible(from: current)
errors.append(XSDFailure(reason: "element '\(child.name.localName)' is not allowed here\(Self.expectation(allowed))", at: path + [steps[index]]))
return
}
current = next
}
let (allowed, complete) = nfa.follow(after: prefix)
if !complete {
errors.append(XSDFailure(reason: "content is incomplete\(Self.expectation(allowed))", at: path))
if !nfa.isComplete(current) {
errors.append(XSDFailure(reason: "content is incomplete\(Self.expectation(nfa.admissible(from: current)))", at: path))
}
}

View File

@ -29,18 +29,12 @@ extension PureXML.Schema {
var accept: Int
func matchesWhole(_ names: [PureXML.Model.QualifiedName]) -> Bool {
var current = closure([start])
var current = startStates()
for name in names {
var next: Set<Int> = []
for state in current {
if let label = states[state].label, let target = states[state].target, label.matches(name) {
next.insert(target)
}
}
if next.isEmpty { return false }
current = closure(next)
guard let next = step(current, over: name) else { return false }
current = next
}
return current.contains(accept)
return isComplete(current)
}
/// After consuming `names`, the labels the automaton can accept next and
@ -49,22 +43,53 @@ extension PureXML.Schema {
/// whether something is still required. Returns an empty, not-complete
/// result when the prefix is already invalid.
func follow(after names: [PureXML.Model.QualifiedName]) -> (allowed: [TermLabel], complete: Bool) {
var current = closure([start])
var current = startStates()
for name in names {
var next: Set<Int> = []
for state in current {
if let label = states[state].label, let target = states[state].target, label.matches(name) {
next.insert(target)
}
}
if next.isEmpty { return ([], false) }
current = closure(next)
guard let next = step(current, over: name) else { return ([], false) }
current = next
}
return (admissible(from: current), isComplete(current))
}
// MARK: Incremental matching
//
// The active set is the epsilon-closure of the states reachable after some
// prefix of children. Callers that scan a child sequence advance it one
// child at a time (`startStates` then `step` per child) rather than calling
// `follow(after:)` on a growing prefix, which re-walks the whole prefix each
// time and is quadratic in the child count over a content model (#129).
/// The active set before any child: the closure of the start state.
func startStates() -> Set<Int> {
closure([start])
}
/// The labels admissible from `current`, for diagnostics and completions.
func admissible(from current: Set<Int>) -> [TermLabel] {
var labels: [TermLabel] = []
for state in current where states[state].target != nil {
if let label = states[state].label { labels.append(label) }
}
return (labels, current.contains(accept))
return labels
}
/// Advances `current` by consuming one element `name`, or nil when `name`
/// is not admissible there.
func step(_ current: Set<Int>, over name: PureXML.Model.QualifiedName) -> Set<Int>? {
var next: Set<Int> = []
for state in current {
if let label = states[state].label, let target = states[state].target, label.matches(name) {
next.insert(target)
}
}
if next.isEmpty { return nil }
return closure(next)
}
/// Whether the content may legally end at `current`.
func isComplete(_ current: Set<Int>) -> Bool {
current.contains(accept)
}
private func closure(_ seed: Set<Int>) -> Set<Int> {
@ -101,18 +126,38 @@ extension PureXML.Schema {
/// (found by XSTS groupF009v, #129).
private static let occursUnrollCap = 16384
/// The per-particle cap bounds one repetition, but nested particles
/// multiply: a sequence of 16384 holding a choice of 16384 is 2.7e8
/// states even though each particle is individually capped. So the
/// total state count is also capped; once an NFA reaches the ceiling
/// every further repetition degrades to star, the same posture the
/// per-particle cap already takes. Legitimate content models are tens
/// of states, so only the pathological XSTS particle schemas (the
/// msMeta Particles set, which OOM-killed the suite at 8 GB, #129)
/// reach it. The ceiling sits well above a single maxed particle
/// (~33k states) so capped-but-not-nested schemas are unaffected.
private static let totalStateCap = 1 << 20
private var stateBudgetExhausted: Bool {
states.count >= Self.totalStateCap
}
private mutating func particle(_ particle: PureXML.Schema.Particle) -> (start: Int, accept: Int) {
let start = addState()
var current = start
let boundedMin = Swift.min(particle.minOccurs, Self.occursUnrollCap)
var unrolledMin = 0
for _ in 0 ..< boundedMin {
if stateBudgetExhausted { break }
let part = term(particle.term)
states[current].epsilon.append(part.start)
current = part.accept
unrolledMin += 1
}
let accept = addState()
if particle.minOccurs > Self.occursUnrollCap {
// The tail of an absurd minOccurs is approximated as star.
if stateBudgetExhausted || unrolledMin < boundedMin || particle.minOccurs > Self.occursUnrollCap {
// The tail of an absurd minOccurs, or a run that hit the total
// state ceiling mid-unroll, is approximated as star.
appendStar(particle.term, from: current, to: accept)
} else if let maximum = particle.maxOccurs, maximum - boundedMin <= Self.occursUnrollCap {
appendOptional(particle.term, count: maximum - boundedMin, from: current, to: accept)
@ -125,6 +170,12 @@ extension PureXML.Schema {
private mutating func appendOptional(_ term: PureXML.Schema.Term, count: Int, from: Int, to accept: Int) {
var current = from
for _ in 0 ..< Swift.max(0, count) {
if stateBudgetExhausted {
// Ran out of state budget: let the remainder repeat freely
// (star) rather than keep allocating bounded optionals.
appendStar(term, from: current, to: accept)
return
}
let part = self.term(term)
states[current].epsilon.append(part.start)
states[current].epsilon.append(accept)

View File

@ -65,7 +65,7 @@ extension PureXML.XSLT.XSLTParser {
/// the namespaces of the prefixes listed in `exclude-result-prefixes`
/// and `extension-element-prefixes` in scope.
static func copiedNamespaces(_ node: XSLTTree) -> [String: String] {
var bindings = inScopeNamespaces(node)
let bindings = inScopeNamespaces(node)
var excludedURIs: Set<String> = [XSLTNode.namespace]
for prefix in excludedPrefixTokens(node) {
let key = prefix == "#default" ? "" : prefix

View File

@ -0,0 +1,52 @@
@testable import PureXML
import Testing
/// Guards the content-model NFA builder against occurrence-explosion (#129).
/// The per-particle unroll cap bounds one repetition, but nested high-`maxOccurs`
/// particles multiply: the XSTS msMeta Particles set drove the builder to 8 GB
/// and OOM-killed the suite. A total state ceiling caps the whole automaton,
/// degrading further repetition to star. This locks that ceiling in: the
/// pathological model must build, stay bounded, and still match without
/// crashing, while ordinary models are untouched.
@Suite("Content NFA state budget")
struct ContentNFAStateBudgetTests {
private func element(_ name: String, min: Int, max: Int?) -> PureXML.Schema.Particle {
.init(minOccurs: min, maxOccurs: max, term: .element(name: .init(name), type: nil))
}
/// A sequence repeated `min` times, each repeat holding an element repeated
/// `min` times: naively `min * min` automaton states.
private func nested(min: Int) -> PureXML.Schema.Particle {
let inner = element("a", min: min, max: min)
return .init(
minOccurs: min,
maxOccurs: min,
term: .group(.init(compositor: .sequence, particles: [inner])),
)
}
@Test("A nested occurrence explosion stays bounded instead of exhausting memory")
func test_nestedExplosionBounded() {
// 16384 * 16384 ~ 2.7e8 states without a total cap; the ceiling is 2^20.
let nfa = PureXML.Schema.ContentNFABuilder.build(nested(min: 16384))
// Bounded to the ceiling plus at most one in-flight term expansion,
// far below the naive product that previously allocated until death.
#expect(nfa.states.count < 1_500_000)
// The degraded automaton must still answer queries without crashing.
let names = Array(repeating: PureXML.Model.QualifiedName("a"), count: 32)
_ = nfa.matchesWhole(names)
}
@Test("Ordinary content models are unaffected by the ceiling")
func test_ordinaryModelUnchanged() {
let particle = PureXML.Schema.Particle(
minOccurs: 1,
maxOccurs: nil,
term: .group(.init(compositor: .sequence, particles: [element("a", min: 1, max: 1)])),
)
let nfa = PureXML.Schema.ContentNFABuilder.build(particle)
#expect(nfa.states.count < 64)
#expect(nfa.matchesWhole([.init("a"), .init("a"), .init("a")]))
#expect(!nfa.matchesWhole([.init("b")]))
}
}

View File

@ -54,6 +54,22 @@ struct RegexTests {
#expect(try matches("[abc]{2}", "ca"))
}
@Test("A '-[' opens class subtraction even after a class member")
func test_classSubtractionAfterMember() throws {
// `[abcd-[d]]` is {a,b,c,d} minus {d} = {a,b,c}; the `d-[` must read as
// subtraction, not a `d`..`[` range that traps on reversed bounds (#129,
// XSTS RegexTest_322).
#expect(try matches("[abcd-[d]]+", "abcabc"))
#expect(try !matches("[abcd-[d]]+", "dddaabbccddd"))
#expect(try matches("[a-z-[aeiou]]+", "bcdfg"))
#expect(try !matches("[a-z-[aeiou]]+", "bcde"))
}
@Test("A reversed range is rejected, not a trap")
func test_reversedRangeRejected() {
#expect(throws: PureXML.Regex.RegexError.self) { _ = try PureXML.Regex.Pattern("[z-a]") }
}
@Test("Class escapes \\d \\w \\s and their negations")
func test_classEscapes() throws {
#expect(try matches("\\d{4}", "2026"))
@ -83,4 +99,17 @@ struct RegexTests {
#expect(throws: PureXML.Regex.RegexError.self) { _ = try PureXML.Regex.Pattern("*ab") }
#expect(throws: PureXML.Regex.RegexError.self) { _ = try PureXML.Regex.Pattern("") }
}
@Test("Pathological quantifiers compile bounded instead of exhausting memory")
func test_quantifierExplosionBounded() throws {
// `a{1000000000}` would unroll a billion automaton states without a
// ceiling; the nested form multiplies. Both must build and answer
// without the OOM that killed the XSTS msMeta Regex set (#129).
_ = try matches("a{1000000000}", String(repeating: "a", count: 64))
_ = try matches("(a{100000}){100000}", "aaaa")
// Ordinary bounded quantifiers keep their exact semantics.
#expect(try matches("a{2,4}", "aaa"))
#expect(try !matches("a{2,4}", "a"))
#expect(try !matches("a{2,4}", "aaaaa"))
}
}

View File

@ -18,15 +18,18 @@ struct XSTSSuiteTests {
ProcessInfo.processInfo.environment["XSTS_ROOT"]
}
/// Exact-count baselines. These are PENDING the first measurement
/// (scheduled 2026-06-14): until it lands, a run against XSTS_ROOT
/// fails these assertions by design, and the failure messages carry
/// the measured counts to transcribe here. The suite is opt-in, so
/// CI and plain `swift test` are unaffected.
private let knownSchemaValidRejected = 0
private let knownSchemaInvalidAccepted = 0
private let knownInstanceValidRejected = 0
private let knownInstanceInvalidAccepted = 0
/// Exact-count baselines from the first measurement against the
/// 2006-11-06 archive (2026-06-13, over 14383 groups; schema ok 11786,
/// instance ok 23804). A case must leave these counts as conformance work
/// closes them, and a regression shows up as a count rising. The suite is
/// opt-in, so CI and plain `swift test` are unaffected; run it with
/// `swift test -c release --filter XSTS` (the debug build is far slower and
/// the corpus is large). Per-case deviations are written to
/// /tmp/xsts-failures.txt for the burn-down.
private let knownSchemaValidRejected = 75
private let knownSchemaInvalidAccepted = 2467
private let knownInstanceValidRejected = 604
private let knownInstanceInvalidAccepted = 582
@Test("Every XSTS case behaves: compile, reject, validate, invalidate")
func test_suite() throws {