From 85d435d03528237021cfdf50752a5dad7551cc58 Mon Sep 17 00:00:00 2001 From: Mihaela Mihaljevic Date: Sat, 13 Jun 2026 16:11:28 +0200 Subject: [PATCH] fix(schema,regex): bound NFA state explosion and class-subtraction crash; land XSTS baseline Running the full W3C XSTS archive to completion exposed three crash and hang classes, now fixed with regression tests, which unblocked the #129 first measurement. - Content-model and regex NFA builders now cap total state count (2^20), degrading further occurrence/quantifier repetition to star, so nested high-maxOccurs schemas no longer OOM (was 8 GB on the MS Particles set). - Content-model validation advances one active state-set per child instead of re-walking the prefix each time, removing the O(children^2) spin. - Regex class subtraction [abc-[b]] no longer traps on a reversed range, and a reversed range such as [z-a] is rejected cleanly. XSTS first measurement baselined: 75 valid schemas rejected, 2467 invalid schemas accepted, 604 valid instances rejected, 582 invalid instances accepted across 14383 groups. The suite runs green in 8s, opt-in via XSTS_ROOT. Refs #129, #122. --- CHANGELOG.md | 6 +- README.md | 2 +- Sources/Regex/RegexAutomaton.swift | 33 ++++++++- Sources/Regex/RegexParser.swift | 7 +- Sources/Schema/ComplexValidator.swift | 16 ++--- Sources/Schema/ContentMatcher.swift | 95 +++++++++++++++++++------ Sources/XSLT/XSLTParserStructural.swift | 2 +- Tests/ContentNFAStateBudgetTests.swift | 52 ++++++++++++++ Tests/RegexTests.swift | 29 ++++++++ Tests/XSTSSuiteTests.swift | 21 +++--- 10 files changed, 217 insertions(+), 46 deletions(-) create mode 100644 Tests/ContentNFAStateBudgetTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 8d7d5ca..3a806af 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,13 +7,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Fixed + +- Three crash and hang classes uncovered by running the full XSTS archive to completion (#129). (1) Nested occurrence explosion: the content-model NFA capped each particle's `maxOccurs` unroll at 16384 but not the total state count, so nested high-`maxOccurs` particles multiplied (a sequence of 16384 holding a choice of 16384 is 2.7e8 states) and OOM-killed the process at 8 GB on the Microsoft Particles set; the builder now also caps total states (2^20), degrading further repetition to star. The regex `{n,m}` quantifier builder had the same unbounded unroll and the same cap is applied. (2) Quadratic content-model validation: `sequenceStructureErrors` called `follow(after: prefix)` once per child and `follow` re-walked the whole prefix from the start each time, so a content model over many children spun at 100% CPU; matching is now incremental (one active state-set advanced per child), turning O(children squared) into O(children). (3) Character-class subtraction trap: the regex pattern `[abcd-[d]]` (XSD class subtraction) was parsed as a `d` to `[` range, and since `d` (100) exceeds `[` (91) the reversed `ClosedRange` trapped (SIGBUS); `-[` now always opens a subtraction, and any reversed range (such as `[z-a]`) is a clean rejection rather than a trap. + ### Changed - Scanner-side validity findings are now located structured values (validation-framework audit follow-up): `DocumentType.validityFindings` carries `ValidityFinding` values (reason plus the declared name they are about) instead of strings, `DTDSchema.declarationErrors` holds ready `ValidationError`s, and every declaration-level finding renders at its declaration's coding path (`x`, `r/@a`) instead of "at root of document", matching the XSD consistency-errors precedent. Finding texts are unchanged. ### Added -- W3C XSTS runner, first measurement (#129, #122): the W3C XML Schema Test Suite (2006-11-06 release, ~13k test groups across NIST, Sun, Microsoft, and Boeing contributions; W3C Document License, never vendored) runs opt-in via XSTS_ROOT: every schemaTest must compile when expected valid and be rejected when expected invalid, and every instanceTest validates against its group's schema with the expected outcome; hrefs resolve against the referencing file and include/import locations from disk relative to the schema. The four disagreement counts are asserted exactly; the baselines are pending the first measurement (scheduled 2026-06-14), whose failure messages carry the counts to transcribe, and current failures are written to /tmp/xsts-failures.txt for the burn-down. Building the runner immediately caught a denial-of-service bug: a schema with maxOccurs in the trillions (groupF009v) OOM-killed the process because the content-model automaton unrolled occurrence counts literally; unrolling is now capped at 16384 with larger repetitions treated as unbounded, the libxml2 posture. The roadmap mermaid records the closed behavioral-parity set (#141-#144, collation) and the #139 performance epic. +- W3C XSTS runner, first measurement (#129, #122): the W3C XML Schema Test Suite (2006-11-06 release, ~13k test groups across NIST, Sun, Microsoft, and Boeing contributions; W3C Document License, never vendored) runs opt-in via XSTS_ROOT: every schemaTest must compile when expected valid and be rejected when expected invalid, and every instanceTest validates against its group's schema with the expected outcome; hrefs resolve against the referencing file and include/import locations from disk relative to the schema. The four disagreement counts are asserted exactly; the first measurement landed 2026-06-13 and is baselined at 75 valid schemas rejected, 2467 invalid schemas accepted, 604 valid instances rejected, and 582 invalid instances accepted across 14383 groups, with per-case deviations written to /tmp/xsts-failures.txt for the burn-down. Building the runner immediately caught a denial-of-service bug: a schema with maxOccurs in the trillions (groupF009v) OOM-killed the process because the content-model automaton unrolled occurrence counts literally; unrolling is now capped at 16384 with larger repetitions treated as unbounded, the libxml2 posture. The roadmap mermaid records the closed behavioral-parity set (#141-#144, collation) and the #139 performance epic. ### Added diff --git a/README.md b/README.md index 1923677..25ee94a 100644 --- a/README.md +++ b/README.md @@ -99,7 +99,7 @@ flowchart TB EncodingMismatch["#137 encoding-declaration mismatches: complete"]:::done BaseURI["#138 per-entity base URIs: complete"]:::done Epic122["#122 official spec suites"]:::epic - XSTS["#129 W3C XSD test suite (XSTS): runner in, occurrence-cap DoS fix; first measurement 2026-06-14"]:::review + XSTS["#129 W3C XSD test suite (XSTS): first measurement 2026-06-13 baselined (deviations 75/2467/604/582); occurrence-explosion and reversed-range crashes fixed, suite green in 8s"]:::review XSLTTS["#130 XSLT 1.0 suite (xalan-test): runner + baselines in, burning down (125)"]:::review Parity100["Behavioral parity closed: #141 top-level params, #142 public streaming, #143 XPath budget, #144 WASI runtime proof, pl/ru sort collation"]:::done Epic139["#139 faster than libxml2, measured + streaming (benchmarks in docs/benchmarks.md; quadratics killed; unsafe authorized)"]:::epic diff --git a/Sources/Regex/RegexAutomaton.swift b/Sources/Regex/RegexAutomaton.swift index bbcdab9..9ec9860 100644 --- a/Sources/Regex/RegexAutomaton.swift +++ b/Sources/Regex/RegexAutomaton.swift @@ -62,6 +62,20 @@ extension PureXML.Regex { return states.count - 1 } + /// `{n,m}` quantifiers unroll into automaton states. Mirroring the XSD + /// content-model builder (`Schema.ContentNFABuilder`), each repetition is + /// capped and the total state count is bounded: past the ceiling a + /// repetition degrades to star, the libxml2 posture. Without this an + /// `xs:pattern` such as `a{1000000000}` (the XSTS msMeta Regex set) + /// allocates until the process is OOM-killed (#129). Real patterns are + /// tens of states, so only pathological quantifiers reach the ceiling. + private static let repeatUnrollCap = 16384 + private static let totalStateCap = 1 << 20 + + private var stateBudgetExhausted: Bool { + states.count >= Self.totalStateCap + } + private mutating func fragment(for node: Node) -> (start: Int, accept: Int) { switch node { case .empty: @@ -109,14 +123,22 @@ extension PureXML.Regex { private mutating func repeatedFragment(_ node: Node, minimum: Int, maximum: Int?) -> (start: Int, accept: Int) { let start = addState() var current = start - for _ in 0 ..< minimum { + let boundedMin = Swift.min(minimum, Self.repeatUnrollCap) + var unrolled = 0 + for _ in 0 ..< boundedMin { + if stateBudgetExhausted { break } let part = fragment(for: node) states[current].epsilon.append(part.start) current = part.accept + unrolled += 1 } let accept = addState() - if let maximum { - appendOptional(node, count: maximum - minimum, from: current, to: accept) + if stateBudgetExhausted || unrolled < boundedMin || minimum > Self.repeatUnrollCap { + // An absurd minimum, or a build that hit the total state ceiling + // mid-unroll, repeats freely (star) past the cap. + appendStar(node, from: current, to: accept) + } else if let maximum, maximum - boundedMin <= Self.repeatUnrollCap { + appendOptional(node, count: maximum - boundedMin, from: current, to: accept) } else { appendStar(node, from: current, to: accept) } @@ -126,6 +148,11 @@ extension PureXML.Regex { private mutating func appendOptional(_ node: Node, count: Int, from: Int, to accept: Int) { var current = from for _ in 0 ..< max(0, count) { + if stateBudgetExhausted { + // Ran out of state budget: let the remainder repeat freely. + appendStar(node, from: current, to: accept) + return + } let part = fragment(for: node) states[current].epsilon.append(part.start) states[current].epsilon.append(accept) diff --git a/Sources/Regex/RegexParser.swift b/Sources/Regex/RegexParser.swift index b824e72..1ee4727 100644 --- a/Sources/Regex/RegexParser.swift +++ b/Sources/Regex/RegexParser.swift @@ -197,9 +197,14 @@ extension PureXML.Regex { add(member, to: &cls) return } - if peek() == "-", peek(1) != "]" { + // A "-" immediately before "[" opens a class subtraction (`[abc-[b]]`), + // not a range, so it is left for parseClass; "-]" is a literal hyphen. + // Otherwise "-" introduces a range whose endpoints must be ordered: a + // reversed range like [z-a] is an error, not a trapping `low ... high`. + if peek() == "-", peek(1) != "]", peek(1) != "[" { advance() guard case let .scalar(high) = try nextClassMember() else { throw RegexError.badClass } + guard low <= high else { throw RegexError.badClass } cls.ranges.append(low ... high) } else { cls.ranges.append(low ... low) diff --git a/Sources/Schema/ComplexValidator.swift b/Sources/Schema/ComplexValidator.swift index 6522891..fc88575 100644 --- a/Sources/Schema/ComplexValidator.swift +++ b/Sources/Schema/ComplexValidator.swift @@ -313,19 +313,19 @@ extension PureXML.Schema.ComplexValidator { func sequenceStructureErrors(_ particle: PureXML.Schema.Particle, children: [PureXML.Model.Element], at path: XSDPath, into errors: inout [XSDFailure]) { let nfa = PureXML.Schema.ContentNFABuilder.build(particle) let steps = Self.childSteps(children) - var prefix: [PureXML.Model.QualifiedName] = [] + // Advance one active state-set across the children rather than re-walking + // the prefix per child (which is quadratic over the content model, #129). + var current = nfa.startStates() for (index, child) in children.enumerated() { - let allowed = nfa.follow(after: prefix).allowed - if allowed.contains(where: { $0.matches(child.name) }) { - prefix.append(child.name) - } else { + guard let next = nfa.step(current, over: child.name) else { + let allowed = nfa.admissible(from: current) errors.append(XSDFailure(reason: "element '\(child.name.localName)' is not allowed here\(Self.expectation(allowed))", at: path + [steps[index]])) return } + current = next } - let (allowed, complete) = nfa.follow(after: prefix) - if !complete { - errors.append(XSDFailure(reason: "content is incomplete\(Self.expectation(allowed))", at: path)) + if !nfa.isComplete(current) { + errors.append(XSDFailure(reason: "content is incomplete\(Self.expectation(nfa.admissible(from: current)))", at: path)) } } diff --git a/Sources/Schema/ContentMatcher.swift b/Sources/Schema/ContentMatcher.swift index d81ec6a..61f2194 100644 --- a/Sources/Schema/ContentMatcher.swift +++ b/Sources/Schema/ContentMatcher.swift @@ -29,18 +29,12 @@ extension PureXML.Schema { var accept: Int func matchesWhole(_ names: [PureXML.Model.QualifiedName]) -> Bool { - var current = closure([start]) + var current = startStates() for name in names { - var next: Set = [] - for state in current { - if let label = states[state].label, let target = states[state].target, label.matches(name) { - next.insert(target) - } - } - if next.isEmpty { return false } - current = closure(next) + guard let next = step(current, over: name) else { return false } + current = next } - return current.contains(accept) + return isComplete(current) } /// After consuming `names`, the labels the automaton can accept next and @@ -49,22 +43,53 @@ extension PureXML.Schema { /// whether something is still required. Returns an empty, not-complete /// result when the prefix is already invalid. func follow(after names: [PureXML.Model.QualifiedName]) -> (allowed: [TermLabel], complete: Bool) { - var current = closure([start]) + var current = startStates() for name in names { - var next: Set = [] - for state in current { - if let label = states[state].label, let target = states[state].target, label.matches(name) { - next.insert(target) - } - } - if next.isEmpty { return ([], false) } - current = closure(next) + guard let next = step(current, over: name) else { return ([], false) } + current = next } + return (admissible(from: current), isComplete(current)) + } + + // MARK: Incremental matching + + // + // The active set is the epsilon-closure of the states reachable after some + // prefix of children. Callers that scan a child sequence advance it one + // child at a time (`startStates` then `step` per child) rather than calling + // `follow(after:)` on a growing prefix, which re-walks the whole prefix each + // time and is quadratic in the child count over a content model (#129). + + /// The active set before any child: the closure of the start state. + func startStates() -> Set { + closure([start]) + } + + /// The labels admissible from `current`, for diagnostics and completions. + func admissible(from current: Set) -> [TermLabel] { var labels: [TermLabel] = [] for state in current where states[state].target != nil { if let label = states[state].label { labels.append(label) } } - return (labels, current.contains(accept)) + return labels + } + + /// Advances `current` by consuming one element `name`, or nil when `name` + /// is not admissible there. + func step(_ current: Set, over name: PureXML.Model.QualifiedName) -> Set? { + var next: Set = [] + for state in current { + if let label = states[state].label, let target = states[state].target, label.matches(name) { + next.insert(target) + } + } + if next.isEmpty { return nil } + return closure(next) + } + + /// Whether the content may legally end at `current`. + func isComplete(_ current: Set) -> Bool { + current.contains(accept) } private func closure(_ seed: Set) -> Set { @@ -101,18 +126,38 @@ extension PureXML.Schema { /// (found by XSTS groupF009v, #129). private static let occursUnrollCap = 16384 + /// The per-particle cap bounds one repetition, but nested particles + /// multiply: a sequence of 16384 holding a choice of 16384 is 2.7e8 + /// states even though each particle is individually capped. So the + /// total state count is also capped; once an NFA reaches the ceiling + /// every further repetition degrades to star, the same posture the + /// per-particle cap already takes. Legitimate content models are tens + /// of states, so only the pathological XSTS particle schemas (the + /// msMeta Particles set, which OOM-killed the suite at 8 GB, #129) + /// reach it. The ceiling sits well above a single maxed particle + /// (~33k states) so capped-but-not-nested schemas are unaffected. + private static let totalStateCap = 1 << 20 + + private var stateBudgetExhausted: Bool { + states.count >= Self.totalStateCap + } + private mutating func particle(_ particle: PureXML.Schema.Particle) -> (start: Int, accept: Int) { let start = addState() var current = start let boundedMin = Swift.min(particle.minOccurs, Self.occursUnrollCap) + var unrolledMin = 0 for _ in 0 ..< boundedMin { + if stateBudgetExhausted { break } let part = term(particle.term) states[current].epsilon.append(part.start) current = part.accept + unrolledMin += 1 } let accept = addState() - if particle.minOccurs > Self.occursUnrollCap { - // The tail of an absurd minOccurs is approximated as star. + if stateBudgetExhausted || unrolledMin < boundedMin || particle.minOccurs > Self.occursUnrollCap { + // The tail of an absurd minOccurs, or a run that hit the total + // state ceiling mid-unroll, is approximated as star. appendStar(particle.term, from: current, to: accept) } else if let maximum = particle.maxOccurs, maximum - boundedMin <= Self.occursUnrollCap { appendOptional(particle.term, count: maximum - boundedMin, from: current, to: accept) @@ -125,6 +170,12 @@ extension PureXML.Schema { private mutating func appendOptional(_ term: PureXML.Schema.Term, count: Int, from: Int, to accept: Int) { var current = from for _ in 0 ..< Swift.max(0, count) { + if stateBudgetExhausted { + // Ran out of state budget: let the remainder repeat freely + // (star) rather than keep allocating bounded optionals. + appendStar(term, from: current, to: accept) + return + } let part = self.term(term) states[current].epsilon.append(part.start) states[current].epsilon.append(accept) diff --git a/Sources/XSLT/XSLTParserStructural.swift b/Sources/XSLT/XSLTParserStructural.swift index 8c8e41c..ab17aea 100644 --- a/Sources/XSLT/XSLTParserStructural.swift +++ b/Sources/XSLT/XSLTParserStructural.swift @@ -65,7 +65,7 @@ extension PureXML.XSLT.XSLTParser { /// the namespaces of the prefixes listed in `exclude-result-prefixes` /// and `extension-element-prefixes` in scope. static func copiedNamespaces(_ node: XSLTTree) -> [String: String] { - var bindings = inScopeNamespaces(node) + let bindings = inScopeNamespaces(node) var excludedURIs: Set = [XSLTNode.namespace] for prefix in excludedPrefixTokens(node) { let key = prefix == "#default" ? "" : prefix diff --git a/Tests/ContentNFAStateBudgetTests.swift b/Tests/ContentNFAStateBudgetTests.swift new file mode 100644 index 0000000..cfef106 --- /dev/null +++ b/Tests/ContentNFAStateBudgetTests.swift @@ -0,0 +1,52 @@ +@testable import PureXML +import Testing + +/// Guards the content-model NFA builder against occurrence-explosion (#129). +/// The per-particle unroll cap bounds one repetition, but nested high-`maxOccurs` +/// particles multiply: the XSTS msMeta Particles set drove the builder to 8 GB +/// and OOM-killed the suite. A total state ceiling caps the whole automaton, +/// degrading further repetition to star. This locks that ceiling in: the +/// pathological model must build, stay bounded, and still match without +/// crashing, while ordinary models are untouched. +@Suite("Content NFA state budget") +struct ContentNFAStateBudgetTests { + private func element(_ name: String, min: Int, max: Int?) -> PureXML.Schema.Particle { + .init(minOccurs: min, maxOccurs: max, term: .element(name: .init(name), type: nil)) + } + + /// A sequence repeated `min` times, each repeat holding an element repeated + /// `min` times: naively `min * min` automaton states. + private func nested(min: Int) -> PureXML.Schema.Particle { + let inner = element("a", min: min, max: min) + return .init( + minOccurs: min, + maxOccurs: min, + term: .group(.init(compositor: .sequence, particles: [inner])), + ) + } + + @Test("A nested occurrence explosion stays bounded instead of exhausting memory") + func test_nestedExplosionBounded() { + // 16384 * 16384 ~ 2.7e8 states without a total cap; the ceiling is 2^20. + let nfa = PureXML.Schema.ContentNFABuilder.build(nested(min: 16384)) + // Bounded to the ceiling plus at most one in-flight term expansion, + // far below the naive product that previously allocated until death. + #expect(nfa.states.count < 1_500_000) + // The degraded automaton must still answer queries without crashing. + let names = Array(repeating: PureXML.Model.QualifiedName("a"), count: 32) + _ = nfa.matchesWhole(names) + } + + @Test("Ordinary content models are unaffected by the ceiling") + func test_ordinaryModelUnchanged() { + let particle = PureXML.Schema.Particle( + minOccurs: 1, + maxOccurs: nil, + term: .group(.init(compositor: .sequence, particles: [element("a", min: 1, max: 1)])), + ) + let nfa = PureXML.Schema.ContentNFABuilder.build(particle) + #expect(nfa.states.count < 64) + #expect(nfa.matchesWhole([.init("a"), .init("a"), .init("a")])) + #expect(!nfa.matchesWhole([.init("b")])) + } +} diff --git a/Tests/RegexTests.swift b/Tests/RegexTests.swift index abf1dac..04db049 100644 --- a/Tests/RegexTests.swift +++ b/Tests/RegexTests.swift @@ -54,6 +54,22 @@ struct RegexTests { #expect(try matches("[abc]{2}", "ca")) } + @Test("A '-[' opens class subtraction even after a class member") + func test_classSubtractionAfterMember() throws { + // `[abcd-[d]]` is {a,b,c,d} minus {d} = {a,b,c}; the `d-[` must read as + // subtraction, not a `d`..`[` range that traps on reversed bounds (#129, + // XSTS RegexTest_322). + #expect(try matches("[abcd-[d]]+", "abcabc")) + #expect(try !matches("[abcd-[d]]+", "dddaabbccddd")) + #expect(try matches("[a-z-[aeiou]]+", "bcdfg")) + #expect(try !matches("[a-z-[aeiou]]+", "bcde")) + } + + @Test("A reversed range is rejected, not a trap") + func test_reversedRangeRejected() { + #expect(throws: PureXML.Regex.RegexError.self) { _ = try PureXML.Regex.Pattern("[z-a]") } + } + @Test("Class escapes \\d \\w \\s and their negations") func test_classEscapes() throws { #expect(try matches("\\d{4}", "2026")) @@ -83,4 +99,17 @@ struct RegexTests { #expect(throws: PureXML.Regex.RegexError.self) { _ = try PureXML.Regex.Pattern("*ab") } #expect(throws: PureXML.Regex.RegexError.self) { _ = try PureXML.Regex.Pattern("") } } + + @Test("Pathological quantifiers compile bounded instead of exhausting memory") + func test_quantifierExplosionBounded() throws { + // `a{1000000000}` would unroll a billion automaton states without a + // ceiling; the nested form multiplies. Both must build and answer + // without the OOM that killed the XSTS msMeta Regex set (#129). + _ = try matches("a{1000000000}", String(repeating: "a", count: 64)) + _ = try matches("(a{100000}){100000}", "aaaa") + // Ordinary bounded quantifiers keep their exact semantics. + #expect(try matches("a{2,4}", "aaa")) + #expect(try !matches("a{2,4}", "a")) + #expect(try !matches("a{2,4}", "aaaaa")) + } } diff --git a/Tests/XSTSSuiteTests.swift b/Tests/XSTSSuiteTests.swift index f0acfb1..9f65c60 100644 --- a/Tests/XSTSSuiteTests.swift +++ b/Tests/XSTSSuiteTests.swift @@ -18,15 +18,18 @@ struct XSTSSuiteTests { ProcessInfo.processInfo.environment["XSTS_ROOT"] } - /// Exact-count baselines. These are PENDING the first measurement - /// (scheduled 2026-06-14): until it lands, a run against XSTS_ROOT - /// fails these assertions by design, and the failure messages carry - /// the measured counts to transcribe here. The suite is opt-in, so - /// CI and plain `swift test` are unaffected. - private let knownSchemaValidRejected = 0 - private let knownSchemaInvalidAccepted = 0 - private let knownInstanceValidRejected = 0 - private let knownInstanceInvalidAccepted = 0 + /// Exact-count baselines from the first measurement against the + /// 2006-11-06 archive (2026-06-13, over 14383 groups; schema ok 11786, + /// instance ok 23804). A case must leave these counts as conformance work + /// closes them, and a regression shows up as a count rising. The suite is + /// opt-in, so CI and plain `swift test` are unaffected; run it with + /// `swift test -c release --filter XSTS` (the debug build is far slower and + /// the corpus is large). Per-case deviations are written to + /// /tmp/xsts-failures.txt for the burn-down. + private let knownSchemaValidRejected = 75 + private let knownSchemaInvalidAccepted = 2467 + private let knownInstanceValidRejected = 604 + private let knownInstanceInvalidAccepted = 582 @Test("Every XSTS case behaves: compile, reject, validate, invalidate") func test_suite() throws {