diff --git a/CHANGELOG.md b/CHANGELOG.md index be7471e..302fb3d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - A model `Group` now carries an internal `definitionID`, a stable identity stamped on each `` expansion (from the existing per-compile memo key) and shared by every reference to it. It is metadata only (`Group` is not `Equatable`/`Hashable`, so it perturbs no comparison) and has no consumer yet: it is the first step of replacing the per-instance content-model build's per-reference group inlining (the `2^K` doubling-chain hang) with the graph-structured-stack matcher designed in `docs/design/counted-content-automaton.md`. No behaviour change; the XSTS buckets are unchanged. +- The counted content matcher gains a shared-group form: `ContentRTNBuilder` compiles a particle to a Recursive Transition Network in which each `definitionID` group is compiled once and reached by a `call`, and `ContentRTNMatcher` simulates it with a graph-structured stack (configurations are `(state, counters, gssNode)`; a `call` saves the caller's counters on an interned GSS node and a `return` restores them, so a multiply-referenced group never mixes counter state across its callers). This replaces the inlining matcher's `2^K` state and reset-action blow-up with an `O(groups)` build: a depth-24 doubling chain (`2^24` inlined states) compiles to under 480 states in under a millisecond. The matcher exposes the full surface the content-model validator reads: `matchesWhole`, `matchedParticles`, and the streaming `startStates`/`step`/`admissible`/`isComplete` used for located diagnostics and the "expected a, b" hints. It is verified against the trusted inlining matcher (`ContentNFA`) as the oracle by a differential gate (`ContentRTNDifferentialTests`) that locks not only the accept verdict and the matched particle but the located rejection index, the `admissible` expectation set at every prefix, and `isComplete`, over: the hand-written model cases, the named-over-wildcard tie-break, the shared-group doubling chains that exercise call and return, and a 4000-case fuzz of nested models with occurrences, wildcards, and group references stamped the way the parser stamps them (a unique id per distinct group, a shared id only for genuinely shared structure, some anonymous inline groups). It is built alongside the inlining matcher and is not yet wired into validation, so there is no behaviour change and the XSTS buckets are unchanged; the switch-over is gated on this differential plus XSTS neutrality. + +- Content-model validation and completion now run on the graph-structured-stack matcher: the three call sites that built the inlining automaton per validated element, `ComplexValidatorContentModel.sequenceStructureErrors` (located structure errors), `ComplexValidator.validateParticle` (per-child particle attribution), and `XSDCompletions` (the acceptable-next labels), now build `ContentRTNBuilder` and run `ContentRTNMatcher`. This removes the last per-instance `2^K` content-model blow-up: a schema whose nested `` references inlined to `2^K` positions (the `g{i}=(g{i+1},g{i+1})` doubling shape) no longer makes each keystroke-time validation of an element exponential in the nesting depth; the automaton is now built in `O(groups)`. Verified conformance-neutral on the full W3C XSTS corpus in release (`valid-schemas-rejected` and `valid-instances-rejected` stay 0, `invalid-schemas-accepted` and `invalid-instances-accepted` unchanged at 8 and 14) and behaviour-identical on the located diagnostics and completions by the surface differential above. `ContentNFABuilder`/`ContentNFA` are retained as the differential oracle. + - The schema-validity differential harness (`XSTSDifferentialTests`, the opt-in libxml2 characterisation for production-readiness stopper 3) now reports each disagreeing schema by its path relative to the corpus root, not its bare basename. The same basename recurs across subdirectories (`Facets/Schemas/gMonth_maxInclusive001.xsd`, which carries `value=""` and is correctly rejected, versus `Facets/gMonth/gMonth_maxInclusive001.xsd`, which carries the contested 2001-REC `--MM--` form), so a basename-only entry hid which file actually disagrees. The header now also records the triage of the "libxml2 stricter" queue: it is dominated by libxml2 over-strictness (a `maxOccurs` like `999999999999999` is a valid unbounded-precision integer libxml2 rejects on its machine-int limit; an unresolvable `include`/`import` it fails to load) and contested datatype-version readings (`gMonth --MM--`, accepted to keep valid XSTS instances; `anyURI` strictness), not clean false negatives. Characterisation only; no shipped behaviour changes. ### Fixed diff --git a/Sources/Schema/ComplexValidator.swift b/Sources/Schema/ComplexValidator.swift index dd73af1..acece36 100644 --- a/Sources/Schema/ComplexValidator.swift +++ b/Sources/Schema/ComplexValidator.swift @@ -218,7 +218,7 @@ public extension PureXML.Schema { matched = allMatchedParticles(group, children: children) } else { sequenceStructureErrors(particle, children: children, at: path, into: &errors) - matched = PureXML.Schema.ContentNFABuilder.build(particle).matchedParticles(children.map(\.name)) + matched = PureXML.Schema.ContentRTNMatcher(PureXML.Schema.ContentRTNBuilder.build(particle)).matchedParticles(children.map(\.name)) } validateChildren( children, diff --git a/Sources/Schema/ComplexValidatorContentModel.swift b/Sources/Schema/ComplexValidatorContentModel.swift index ab3f3ef..9ef6cbf 100644 --- a/Sources/Schema/ComplexValidatorContentModel.swift +++ b/Sources/Schema/ComplexValidatorContentModel.swift @@ -5,22 +5,22 @@ extension PureXML.Schema.ComplexValidator { /// Walks the children through the content automaton, flagging the first child /// the follow-set rejects, or the missing content when the sequence ends early. func sequenceStructureErrors(_ particle: PureXML.Schema.Particle, children: [PureXML.Model.Element], at path: XSDPath, into errors: inout [XSDFailure]) { - let nfa = PureXML.Schema.ContentNFABuilder.build(particle) + let matcher = PureXML.Schema.ContentRTNMatcher(PureXML.Schema.ContentRTNBuilder.build(particle)) let steps = Self.childSteps(children) // Advance one active state-set across the children rather than re-walking // the prefix per child (which is quadratic over the content model, #129). let inputLength = children.count - var current = nfa.startStates(inputLength: inputLength) + var current = matcher.startStates(inputLength: inputLength) for (index, child) in children.enumerated() { - guard let next = nfa.step(current, over: child.name, inputLength: inputLength) else { - let allowed = nfa.admissible(from: current) + guard let next = matcher.step(current, over: child.name) else { + let allowed = matcher.admissible(from: current) errors.append(XSDFailure(reason: "element '\(child.name.localName)' is not allowed here\(Self.expectation(allowed))", at: path + [steps[index]])) return } current = next } - if !nfa.isComplete(current) { - errors.append(XSDFailure(reason: "content is incomplete\(Self.expectation(nfa.admissible(from: current)))", at: path)) + if !matcher.isComplete(current) { + errors.append(XSDFailure(reason: "content is incomplete\(Self.expectation(matcher.admissible(from: current)))", at: path)) } } diff --git a/Sources/Schema/ContentRTN.swift b/Sources/Schema/ContentRTN.swift new file mode 100644 index 0000000..2f0eb81 --- /dev/null +++ b/Sources/Schema/ContentRTN.swift @@ -0,0 +1,227 @@ +extension PureXML.Schema { + /// A call edge: enter the callee sub-program at `calleeStart`, returning to + /// `returnTo` when the callee's return state is reached. The `actions` reset the + /// callee's counter scopes on entry (so a shared fragment entered twice does not + /// carry a stale count); `guards` are the calling occurrence's `belowMaximum`. + struct ContentCallEdge: Sendable { + var calleeStart: Int + var returnTo: Int + var guards: [ContentCounterGuard] = [] + var actions: [ContentCounterAction] = [] + } + + /// One state of the counted Recursive Transition Network: like ``ContentState`` + /// but a state may also `call` a shared group fragment, and a group fragment's + /// accept is a `return` (it pops the graph-structured stack to its caller). + struct ContentRTNState: Sendable { + var epsilon: [ContentEpsilonEdge] = [] + var calls: [ContentCallEdge] = [] + /// A group fragment's accept: pop the GSS and continue at the caller's + /// `returnTo`. The top-level program accept is NOT a return. + var isReturn: Bool = false + var label: TermLabel? + var target: Int? + var elementType: ElementType? + var valueConstraint: ValueConstraint? + } + + /// A counted RTN compiled from a ``Particle``: each `` definition + /// (identified by ``Group/definitionID``) is compiled once and reached by a + /// `call`, so a multiply-referenced group is shared, not inlined `2^K` times. + /// `all` groups are validated separately and are not represented here, exactly + /// as in ``ContentNFA``. + struct ContentRTN: Sendable { + var states: [ContentRTNState] + var counters: [ContentCounterScope] + var start: Int + var accept: Int + } +} + +extension PureXML.Schema { + /// Builds a counted ``ContentRTN`` from a particle, sharing each `definitionID` + /// group fragment. The occurrence-wrapper shape (reset/loop/exit/increment) + /// matches ``ContentNFABuilder`` exactly; the only difference is that a group + /// with a `definitionID` compiles to a `call` into its shared fragment instead + /// of being inlined. + struct ContentRTNBuilder { + private var states: [ContentRTNState] = [] + private var counters: [ContentCounterScope] = [] + /// The compiled (start, accept, counters) of each shared group fragment. + private var fragments: [Int: ContentNFAFragment] = [:] + /// Definition ids whose fragment is mid-compilation, to detect a (malformed) + /// cyclic group definition instead of recurring without bound. + private var compiling: Set = [] + + static func build(_ particle: PureXML.Schema.Particle) -> ContentRTN { + var builder = ContentRTNBuilder() + let fragment = builder.particle(particle) + return ContentRTN( + states: builder.states, + counters: builder.counters, + start: fragment.start, + accept: fragment.accept, + ) + } + + private mutating func addState() -> Int { + states.append(ContentRTNState()) + return states.count - 1 + } + + private mutating func addEpsilon( + from source: Int, + to target: Int, + guards: [ContentCounterGuard] = [], + actions: [ContentCounterAction] = [], + ) { + states[source].epsilon.append(ContentEpsilonEdge(target: target, guards: guards, actions: actions)) + } + + private mutating func particle(_ particle: PureXML.Schema.Particle) -> ContentNFAFragment { + let start = addState() + let loop = addState() + let accept = addState() + let counter = counters.count + let bodyNullable = termIsNullable(particle.term) + counters.append(ContentCounterScope(range: particle.occurrenceRange, nullableBody: bodyNullable)) + let body = term(particle.term) + let resetAll = [ContentCounterAction.reset(counter)] + body.counters.map(ContentCounterAction.reset) + addEpsilon(from: start, to: loop, actions: resetAll) + addEpsilon( + from: loop, + to: accept, + guards: [.canExit(counter: counter, minimum: particle.occurrenceRange.minimum, nullableBody: bodyNullable)], + ) + addEpsilon( + from: loop, + to: body.start, + guards: [.belowMaximum(counter: counter, maximum: particle.occurrenceRange.maximum)], + actions: body.counters.map(ContentCounterAction.reset), + ) + addEpsilon(from: body.accept, to: loop, actions: [.increment(counter)]) + return ContentNFAFragment(start: start, accept: accept, counters: [counter] + body.counters, nullable: particleIsNullable(particle)) + } + + private mutating func term(_ term: PureXML.Schema.Term) -> ContentNFAFragment { + switch term { + case let .element(name, type, _, constraint, _, _): + let fragment = labeled(.name(name)) + states[fragment.start].elementType = type + states[fragment.start].valueConstraint = constraint + return fragment + case let .wildcard(wildcard): + return labeled(.wildcard(wildcard)) + case let .group(group): + if let id = group.definitionID { + return call(group, id: id) + } + return self.group(group) + } + } + + /// A reference to a shared group: a `call` state into the group's fragment + /// (compiled once), returning to a fresh accept. The callee's counters are + /// deliberately NOT propagated to the caller: every counter scope is reset on + /// its own particle's entry edge, which is on the path from the callee's start, + /// so the shared fragment resets itself on every (re-)entry without the caller + /// re-listing it. Propagating them would re-list each shared fragment's counters + /// at every reference, nesting to a `2^K` reset-action list even though the + /// states stay `O(K)`, which is the whole point of sharing. + private mutating func call(_ group: PureXML.Schema.Group, id: Int) -> ContentNFAFragment { + let fragment = groupFragment(group, id: id) + let callState = addState() + let returnState = addState() + states[callState].calls.append(ContentCallEdge(calleeStart: fragment.start, returnTo: returnState)) + return ContentNFAFragment(start: callState, accept: returnState, counters: [], nullable: fragment.nullable) + } + + /// The shared fragment for a group definition, compiled once and memoized. + /// Its accept is marked `isReturn`. + private mutating func groupFragment(_ group: PureXML.Schema.Group, id: Int) -> ContentNFAFragment { + if let cached = fragments[id] { return cached } + // `definitionID` groups are acyclic: XSD forbids a group referencing itself + // except through an element, and the parser's expansion breaks every cycle at + // an element boundary, so a fragment is never referenced during its own + // compilation (the inlining `ContentNFABuilder` relies on the same property). + // Guard it anyway: a cyclic id would otherwise recur without bound, the exact + // interactive-safety hazard this matcher exists to remove, so bound the + // recursion by degrading a (malformed, unreachable) cycle to a terminating + // empty fragment rather than overflowing the stack. + guard compiling.insert(id).inserted else { return labeledEmpty() } + defer { compiling.remove(id) } + let built = group.compositor == .choice ? choice(group.particles) : sequence(group.particles) + states[built.accept].isReturn = true + fragments[id] = built + return built + } + + private mutating func group(_ group: PureXML.Schema.Group) -> ContentNFAFragment { + group.compositor == .choice ? choice(group.particles) : sequence(group.particles) + } + + private mutating func labeled(_ label: TermLabel) -> ContentNFAFragment { + let start = addState() + let accept = addState() + states[start].label = label + states[start].target = accept + return ContentNFAFragment(start: start, accept: accept, counters: [], nullable: false) + } + + private mutating func sequence(_ particles: [PureXML.Schema.Particle]) -> ContentNFAFragment { + guard !particles.isEmpty else { return labeledEmpty() } + var first: Int? + var last: Int? + var fragmentCounters: [Int] = [] + var nullable = true + for member in particles { + let part = particle(member) + if let previous = last { addEpsilon(from: previous, to: part.start) } else { first = part.start } + last = part.accept + fragmentCounters.append(contentsOf: part.counters) + nullable = nullable && part.nullable + } + return ContentNFAFragment(start: first ?? addState(), accept: last ?? addState(), counters: fragmentCounters, nullable: nullable) + } + + private mutating func choice(_ particles: [PureXML.Schema.Particle]) -> ContentNFAFragment { + let start = addState() + let accept = addState() + var fragmentCounters: [Int] = [] + var nullable = false + for member in particles { + let part = particle(member) + addEpsilon(from: start, to: part.start) + addEpsilon(from: part.accept, to: accept) + fragmentCounters.append(contentsOf: part.counters) + nullable = nullable || part.nullable + } + return ContentNFAFragment(start: start, accept: accept, counters: fragmentCounters, nullable: nullable) + } + + private mutating func labeledEmpty() -> ContentNFAFragment { + let start = addState() + let accept = addState() + addEpsilon(from: start, to: accept) + return ContentNFAFragment(start: start, accept: accept, counters: [], nullable: true) + } + + private func particleIsNullable(_ particle: PureXML.Schema.Particle) -> Bool { + particle.occurrenceRange.minimum.isZero || termIsNullable(particle.term) + } + + private func termIsNullable(_ term: PureXML.Schema.Term) -> Bool { + switch term { + case .element, .wildcard: + false + case let .group(group): + switch group.compositor { + case .sequence, .all: + group.particles.allSatisfy(particleIsNullable) + case .choice: + group.particles.contains(where: particleIsNullable) + } + } + } + } +} diff --git a/Sources/Schema/ContentRTNMatcher.swift b/Sources/Schema/ContentRTNMatcher.swift new file mode 100644 index 0000000..6baff84 --- /dev/null +++ b/Sources/Schema/ContentRTNMatcher.swift @@ -0,0 +1,262 @@ +/// The key under which a GSS node is interned during one match: a call's return +/// state, the caller's saved counters, and the input position at the call. Two +/// calls sharing all three share a node (the graph-structured-stack sharing). +private struct ContentRTNGSSKey: Hashable { + let returnState: Int + let savedCounters: [Int] + let position: Int +} + +/// One graph-structured-stack node: a saved return context. `parents` are the +/// caller nodes (a set, so distinct callers sharing a return are merged); `popped` +/// records that a return has already fired through this node, so a parent added +/// afterward is served the replayed pop (the GLL completeness detail). +private struct ContentRTNGSSNode { + let returnState: Int + let savedCounters: [Int] + var parents: Set + var popped = false +} + +/// A configuration: a program state, the live counter vector, and the GSS node +/// giving the return context. `gss == 0` is the empty stack. Internal (not private) +/// because it threads through the matcher's streaming surface, which the content-model +/// validator drives child by child. +struct ContentRTNConfiguration: Hashable { + var state: Int + var counters: [Int] + var gss: Int +} + +extension PureXML.Schema { + /// Matches a child-name sequence against a ``ContentRTN`` using a + /// graph-structured stack, so a multiply-referenced group is traversed without + /// the inlining matcher's `2^K` state blow-up. Configurations are + /// `(state, counters, gssNode)`; a `call` saves the caller's counters on a GSS + /// node and a `return` restores them, so the shared callee fragment does not mix + /// counter state across its callers. GSS nodes are interned by + /// `(returnState, savedCounters, position)` with a parent set and a pop flag (the + /// GLL completeness detail: a parent added after the node was popped replays the + /// pop), which keeps the active set bounded by the program size, not by the + /// number of distinct return stacks. + final class ContentRTNMatcher { + private let rtn: ContentRTN + private var nodes: [ContentRTNGSSNode] = [] + private var intern: [ContentRTNGSSKey: Int] = [:] + private var caps: [Int] = [] + private var position = 0 + /// Configurations produced by replaying a pop to a newly added parent during + /// closure; drained into the worklist. + private var pending: [ContentRTNConfiguration] = [] + + init(_ rtn: ContentRTN) { + self.rtn = rtn + } + + // MARK: Public matching surface (mirrors ContentNFA) + + func matchesWhole(_ names: [PureXML.Model.QualifiedName]) -> Bool { + var current = startStates(inputLength: names.count) + for name in names { + guard let next = step(current, over: name) else { return false } + current = next + } + return isComplete(current) + } + + /// After consuming `names`, the labels the matcher can accept next and whether + /// the content may legally end here. Composes the streaming surface; mirrors + /// ``ContentNFA/follow(after:)``, used for completions. Returns an empty, + /// not-complete result when the prefix is already invalid. + func follow(after names: [PureXML.Model.QualifiedName]) -> (allowed: [TermLabel], complete: Bool) { + var current = startStates(inputLength: names.count) + for name in names { + guard let next = step(current, over: name) else { return ([], false) } + current = next + } + return (admissible(from: current), isComplete(current)) + } + + /// The active set before any child: resets this matcher's graph-structured + /// stack and returns the closure of the start configuration. `inputLength` + /// sizes the counter saturation caps. Mirrors ``ContentNFA/startStates``, but + /// is stateful (the GSS lives on the instance), so one matcher drives one + /// child scan: call this once, then ``step(_:over:)`` per child. + func startStates(inputLength: Int) -> Set { + reset(inputLength: inputLength) + let seed = ContentRTNConfiguration(state: rtn.start, counters: Array(repeating: 0, count: rtn.counters.count), gss: 0) + return closure([seed]) + } + + /// Advances `current` by consuming one element `name`, or nil when `name` is + /// not admissible there. Mirrors ``ContentNFA/step(_:over:inputLength:)``; must + /// be called on the active set this matcher last produced (the GSS is stateful). + func step(_ current: Set, over name: PureXML.Model.QualifiedName) -> Set? { + var next: Set = [] + for configuration in current { + let state = configuration.state + if let label = rtn.states[state].label, let target = rtn.states[state].target, label.matches(name) { + next.insert(ContentRTNConfiguration(state: target, counters: configuration.counters, gss: configuration.gss)) + } + } + if next.isEmpty { return nil } + position += 1 + return closure(next) + } + + /// The labels admissible from `current`, for diagnostics and completions. The + /// closure already expands `call` states into the callee's label states, so + /// this is the same shape as ``ContentNFA/admissible(from:)``: every active + /// configuration sitting on a labelled transition contributes its label. + func admissible(from current: Set) -> [TermLabel] { + var labels: [TermLabel] = [] + for configuration in current where rtn.states[configuration.state].target != nil { + if let label = rtn.states[configuration.state].label { labels.append(label) } + } + return labels + } + + /// Whether the content may legally end at `current`: an active configuration at + /// the program accept with the empty stack (input consumed, all calls returned). + func isComplete(_ current: Set) -> Bool { + current.contains { $0.state == rtn.accept && $0.gss == 0 } + } + + /// The particle each child matched, in order (nil from the first rejected + /// child onward), with the inlining matcher's tie-breaking: a named particle + /// over a wildcard, then the lowest state index. + /// + /// The lowest-state-index tie-break only distinguishes two same-kind particles + /// that both accept the child, which is a Unique Particle Attribution violation: + /// the schema compiler rejects such a model, so no instance is validated against + /// it. State numbering here differs from `ContentNFA`'s (the added call/return + /// states), so on such a (rejected) model the two matchers may pick different + /// competing particles; on a UPA-valid model at most one particle accepts each + /// child, so the pick, and the type/value metadata returned with it, is identical. + func matchedParticles(_ names: [PureXML.Model.QualifiedName]) -> [MatchedParticle?] { + var current = startStates(inputLength: names.count) + var result: [MatchedParticle?] = [] + for name in names { + var next: Set = [] + var namedState: Int? + var anyState: Int? + for configuration in current { + let state = configuration.state + guard let label = rtn.states[state].label, let target = rtn.states[state].target, label.matches(name) else { continue } + next.insert(ContentRTNConfiguration(state: target, counters: configuration.counters, gss: configuration.gss)) + if anyState.map({ state < $0 }) ?? true { anyState = state } + if case .name = label, namedState.map({ state < $0 }) ?? true { namedState = state } + } + guard let chosenState = namedState ?? anyState, !next.isEmpty else { + result.append(contentsOf: Array(repeating: nil, count: names.count - result.count)) + return result + } + let matched = rtn.states[chosenState] + if case let .wildcard(wildcard) = matched.label { + result.append(.wildcard(wildcard)) + } else { + result.append(.element(type: matched.elementType, valueConstraint: matched.valueConstraint)) + } + position += 1 + current = closure(next) + } + return result + } + + // MARK: Simulation + + private func reset(inputLength: Int) { + nodes = [ContentRTNGSSNode(returnState: -1, savedCounters: [], parents: [])] // node 0 = empty stack + intern = [:] + position = 0 + caps = counterCaps(for: inputLength) + } + + private func closure(_ seed: Set) -> Set { + var seen = seed + var work = Array(seed) + while true { + if !pending.isEmpty { + let configuration = pending.removeLast() + if seen.insert(configuration).inserted { work.append(configuration) } + continue + } + guard let configuration = work.popLast() else { break } + let state = rtn.states[configuration.state] + for edge in state.epsilon { + if let nextConfiguration = apply(edge, to: configuration), seen.insert(nextConfiguration).inserted { + work.append(nextConfiguration) + } + } + for call in state.calls { + let node = internNode(returnState: call.returnTo, savedCounters: configuration.counters, parent: configuration.gss) + let nextConfiguration = ContentRTNConfiguration(state: call.calleeStart, counters: configuration.counters, gss: node) + if seen.insert(nextConfiguration).inserted { work.append(nextConfiguration) } + drainPending(into: &seen, work: &work) + } + if state.isReturn, configuration.gss != 0 { + popReturn(configuration.gss) + drainPending(into: &seen, work: &work) + } + } + return seen + } + + private func drainPending(into seen: inout Set, work: inout [ContentRTNConfiguration]) { + while let configuration = pending.popLast() { + if seen.insert(configuration).inserted { work.append(configuration) } + } + } + + /// Interns the GSS node for a call returning to `returnState` from the + /// caller's `savedCounters`, recording `parent`. If the node already exists + /// and was popped, replay the pop to the newly added parent (GLL). + private func internNode(returnState: Int, savedCounters: [Int], parent: Int) -> Int { + let key = ContentRTNGSSKey(returnState: returnState, savedCounters: savedCounters, position: position) + if let id = intern[key] { + if nodes[id].parents.insert(parent).inserted, nodes[id].popped { + pending.append(ContentRTNConfiguration(state: returnState, counters: savedCounters, gss: parent)) + } + return id + } + let id = nodes.count + nodes.append(ContentRTNGSSNode(returnState: returnState, savedCounters: savedCounters, parents: [parent])) + intern[key] = id + return id + } + + /// Returns from GSS node `id`: restore its saved counters and continue at its + /// return state for every parent. Mark it popped so a parent added later is + /// served too. + private func popReturn(_ id: Int) { + nodes[id].popped = true + let node = nodes[id] + for parent in node.parents { + pending.append(ContentRTNConfiguration(state: node.returnState, counters: node.savedCounters, gss: parent)) + } + } + + private func apply(_ edge: ContentEpsilonEdge, to configuration: ContentRTNConfiguration) -> ContentRTNConfiguration? { + for guardCondition in edge.guards where !guardCondition.accepts(configuration.counters) { + return nil + } + var values = configuration.counters + for action in edge.actions { + action.apply(to: &values, caps: caps) + } + return ContentRTNConfiguration(state: edge.target, counters: values, gss: configuration.gss) + } + + private func counterCaps(for inputLength: Int) -> [Int] { + let reachable = inputLength == Int.max ? Int.max : inputLength + 1 + return rtn.counters.map { scope in + guard let maximumCap = scope.range.maximum.clamped(to: reachable), + !scope.range.maximum.isGreaterThan(reachable) + else { + return scope.range.minimum.clamped(to: reachable) + } + return maximumCap + } + } + } +} diff --git a/Sources/Schema/XSDCompletions.swift b/Sources/Schema/XSDCompletions.swift index 041ca27..6a35d25 100644 --- a/Sources/Schema/XSDCompletions.swift +++ b/Sources/Schema/XSDCompletions.swift @@ -57,7 +57,7 @@ public extension PureXML.Schema { } switch complex.content { case let .elementOnly(particle), let .mixed(particle): - let (labels, complete) = ContentNFABuilder.build(particle).follow(after: childNames) + let (labels, complete) = ContentRTNMatcher(ContentRTNBuilder.build(particle)).follow(after: childNames) let names = labels.compactMap { if case let .name(name) = $0 { name.description } else { nil } } let anyElement = labels.contains { if case .wildcard = $0 { true } else { false } } return Completions(elements: names, complete: complete, allowsAnyElement: anyElement, attributes: attributes) diff --git a/Tests/ContentRTNDifferentialTests.swift b/Tests/ContentRTNDifferentialTests.swift new file mode 100644 index 0000000..e66ac84 --- /dev/null +++ b/Tests/ContentRTNDifferentialTests.swift @@ -0,0 +1,248 @@ +import Testing +@testable import PureXML + +/// Locks the graph-structured-stack content matcher (`ContentRTNMatcher`, which +/// shares each `` definition) to the trusted inlining matcher +/// (`ContentNFA`) as the oracle: they must agree on the verdict and the matched +/// particle for every child sequence, and the RTN build must stay polynomial where +/// the inlining build is `2^K`. The differential gate from the GSS-matcher design. +@Suite("RTN/GSS vs inlining content-matcher differential") +struct ContentRTNDifferentialTests { + private func element(_ name: String, min: Int = 1, max: Int? = 1) -> PureXML.Schema.Particle { + // A per-name fixed value constraint so the matched-particle comparison sees the + // metadata `matchedParticles` carries (type/valueConstraint), not only the + // element/wildcard kind. A name maps to one constraint (as a real schema's + // element declaration does), so whichever particle of a given name is picked, its + // constraint is the same: the comparison stays meaningful without depending on the + // lowest-state-index tie-break, which only distinguishes same-name particles, and + // same-name particles with differing metadata are a UPA violation the schema + // compiler rejects before any instance is validated. + .init(minOccurs: min, maxOccurs: max, term: .element(name: .init(name), type: nil, typeName: nil, valueConstraint: .fixed("v-\(name)"))) + } + + private func group(_ compositor: PureXML.Schema.Compositor, _ particles: [PureXML.Schema.Particle], min: Int = 1, max: Int? = 1, id: Int? = nil) -> PureXML.Schema.Particle { + .init(minOccurs: min, maxOccurs: max, term: .group(.init(compositor: compositor, particles: particles, definitionID: id))) + } + + /// A `##any` wildcard, which admits every child; paired with a named element it + /// exercises the named-over-wildcard tie-break the matched-particle path must + /// reproduce. + private func wildcard(min: Int = 1, max: Int? = 1) -> PureXML.Schema.Particle { + .init(minOccurs: min, maxOccurs: max, term: .wildcard(.init())) + } + + private func names(_ list: [String]) -> [PureXML.Model.QualifiedName] { + list.map { PureXML.Model.QualifiedName($0) } + } + + /// Both matchers' accept verdict, and the kind (nil/element/wildcard) each child + /// was matched to, must agree. + private func assertAgree(_ particle: PureXML.Schema.Particle, _ sequence: [String], _ context: @autoclosure () -> String) { + let qnames = names(sequence) + let nfa = PureXML.Schema.ContentNFABuilder.build(particle) + let rtn = PureXML.Schema.ContentRTNBuilder.build(particle) + let nfaWhole = nfa.matchesWhole(qnames) + let rtnWhole = PureXML.Schema.ContentRTNMatcher(rtn).matchesWhole(qnames) + #expect(nfaWhole == rtnWhole, "verdict disagree on \(context()) for \(sequence): nfa=\(nfaWhole) rtn=\(rtnWhole)") + let nfaAttribution = nfa.matchedParticles(qnames).map(attribution) + let rtnAttribution = PureXML.Schema.ContentRTNMatcher(rtn).matchedParticles(qnames).map(attribution) + #expect(nfaAttribution == rtnAttribution, "attribution disagree on \(context()) for \(sequence): \(nfaAttribution) vs \(rtnAttribution)") + assertStreamingAgree(nfa, rtn, qnames, context()) + } + + /// Locks the located-diagnostic surface, not just the accept verdict: driving + /// both matchers child by child, the first child either rejects (`step` is nil) + /// in both or neither, the `admissible` expectation set agrees at the start and + /// after every accepted child, and `isComplete` agrees once the whole sequence is + /// consumed. This is what `sequenceStructureErrors` actually reads, so the + /// switch-over keeps the placed errors and the "expected a, b" hints identical. + private func assertStreamingAgree( + _ nfa: PureXML.Schema.ContentNFA, + _ rtn: PureXML.Schema.ContentRTN, + _ qnames: [PureXML.Model.QualifiedName], + _ context: @autoclosure () -> String, + ) { + let matcher = PureXML.Schema.ContentRTNMatcher(rtn) + var nfaCurrent = nfa.startStates(inputLength: qnames.count) + var rtnCurrent = matcher.startStates(inputLength: qnames.count) + #expect(labelSet(nfa.admissible(from: nfaCurrent)) == labelSet(matcher.admissible(from: rtnCurrent)), "admissible@start disagree on \(context())") + var consumedAll = true + for (index, name) in qnames.enumerated() { + let nfaNext = nfa.step(nfaCurrent, over: name, inputLength: qnames.count) + let rtnNext = matcher.step(rtnCurrent, over: name) + #expect((nfaNext == nil) == (rtnNext == nil), "rejection-index disagree at child \(index) on \(context())") + guard let nfaStepped = nfaNext, let rtnStepped = rtnNext else { consumedAll = false + break + } + nfaCurrent = nfaStepped + rtnCurrent = rtnStepped + #expect(labelSet(nfa.admissible(from: nfaCurrent)) == labelSet(matcher.admissible(from: rtnCurrent)), "admissible@\(index) disagree on \(context())") + } + if consumedAll { + #expect(nfa.isComplete(nfaCurrent) == matcher.isComplete(rtnCurrent), "isComplete disagree on \(context())") + } + // The composed follow(after:) the completion path reads must agree too. + let nfaFollow = nfa.follow(after: qnames) + let rtnFollow = PureXML.Schema.ContentRTNMatcher(rtn).follow(after: qnames) + #expect(labelSet(nfaFollow.allowed) == labelSet(rtnFollow.allowed), "follow.allowed disagree on \(context())") + #expect(nfaFollow.complete == rtnFollow.complete, "follow.complete disagree on \(context())") + } + + /// The full matched-particle attribution as a comparable string: the kind plus the + /// carried value constraint for a named element, so the comparison catches the + /// matcher dropping or swapping the metadata `matchedParticles` returns, not only a + /// kind flip. (`MatchedParticle` is not `Equatable` because `ElementType?` is not, so + /// project the observable fields.) + private func attribution(_ matched: PureXML.Schema.MatchedParticle?) -> String { + switch matched { + case .none: "nil" + case let .element(_, valueConstraint): "element:\(valueConstraint?.value ?? "-")" + case .wildcard: "wildcard" + } + } + + /// The admissible labels as a set of comparable keys (the "expected" hint + /// de-duplicates, so set equality is the meaningful comparison). Wildcards are + /// all `##any` here, so one key suffices. + private func labelSet(_ labels: [PureXML.Schema.TermLabel]) -> Set { + Set(labels.map { label in + switch label { + case let .name(qualified): "n:\(qualified.localName)|\(qualified.namespaceURI ?? "")" + case .wildcard: "w" + } + }) + } + + @Test("agree on the hand-written model cases (inline groups, no calls)") + func test_inlineCases() { + let cases: [(PureXML.Schema.Particle, [[String]])] = [ + (group(.sequence, [element("a"), element("b")]), [["a", "b"], ["b", "a"], ["a"], []]), + (group(.choice, [element("a"), element("b")]), [["a"], ["b"], ["a", "b"], []]), + (group(.sequence, [element("a")], min: 0, max: nil), [[], ["a"], ["a", "a", "a"], ["b"]]), + (group(.sequence, [element("a")], min: 1, max: nil), [[], ["a"], ["a", "a"]]), + (group(.sequence, [element("a")], min: 0, max: 1), [[], ["a"], ["a", "a"]]), + (group(.choice, [group(.sequence, [element("a"), element("b")]), element("c")]), [["a", "b"], ["c"], ["a", "c"], ["a", "b", "c"]]), + (group(.sequence, [element("a", min: 0, max: 2), element("b", min: 1, max: 3)]), [["b"], ["a", "b"], ["a", "a", "b", "b", "b"], ["a", "a", "a", "b"]]), + // Named-over-wildcard tie-break: "a" matches both, the named particle wins; + // "b"/"c" match only the wildcard. + (group(.choice, [element("a"), wildcard()]), [["a"], ["b"], ["c"], []]), + (group(.sequence, [element("a"), wildcard(min: 0, max: nil)]), [["a"], ["a", "b", "c"], ["b"], []]), + ] + for (particle, sequences) in cases { + for sequence in sequences { + assertAgree(particle, sequence, "inline") + } + } + } + + /// `g0=(g1,g1), …, g_{K-1}=(gK,gK), gK=x`: the model is `x` repeated `2^K` times. + private func doubling(depth: Int) -> PureXML.Schema.Particle { + var current = element("x") + for level in stride(from: depth - 1, through: 0, by: -1) { + current = group(.sequence, [current, current], id: level) + } + return current + } + + @Test("agree on shared-group doubling chains, exercising call/return") + func test_doublingCallReturn() { + for depth in 1 ... 5 { + let required = 1 << depth // 2^depth occurrences of x + let particle = doubling(depth: depth) + for count in [0, required - 1, required, required + 1] where count >= 0 { + assertAgree(particle, Array(repeating: "x", count: count), "doubling depth=\(depth)") + } + } + } + + @Test("the RTN build is polynomial where the inlining build is 2^K") + func test_buildBound() { + // Depth 24 is 2^24 inlined states (infeasible to inline); the shared build + // must stay linear in the depth. + let rtn = PureXML.Schema.ContentRTNBuilder.build(doubling(depth: 24)) + #expect(rtn.states.count < 24 * 20, "shared build is not linear in depth: \(rtn.states.count) states") + } + + @Test("agree on a fuzz of nested models with occurrences and shared groups") + func test_fuzz() { + var rng = SeededRNG(seed: 0x5EED_1234) + var nextID = 0 + for _ in 0 ..< 4000 { + let particle = randomParticle(depth: 3, rng: &rng, nextID: &nextID) + let length = Int.random(in: 0 ... 6, using: &rng) + let sequence = (0 ..< length).map { _ in letter(&rng) } + assertAgree(particle, sequence, "fuzz") + } + } + + /// `nextID` hands out a unique `definitionID` per distinct group, so a shared id + /// only ever marks genuinely-shared (structurally identical) groups; a monotonic + /// counter avoids the collision a random id risks. Most groups are stamped (the + /// production shape, where every `` is memoized and called, singly + /// or shared), some are left anonymous (inline, no call) to keep that path live. + private func randomParticle(depth: Int, rng: inout SeededRNG, nextID: inout Int) -> PureXML.Schema.Particle { + let (min, max): (Int, Int?) = switch Int.random(in: 0 ..< 5, using: &rng) { + case 0: (1, 1) + case 1: (0, 1) + case 2: (0, nil) + case 3: (1, nil) + default: (Int.random(in: 0 ... 2, using: &rng), Int.random(in: 2 ... 3, using: &rng)) + } + if depth == 0 || Bool.random(using: &rng) { + // One leaf in four is a wildcard, so the named-over-wildcard tie-break and + // the wildcard match path are exercised inside shared groups too. + if Int.random(in: 0 ..< 4, using: &rng) == 0 { + return wildcard(min: min, max: max) + } + return element(letter(&rng), min: min, max: max) + } + let compositor: PureXML.Schema.Compositor = Bool.random(using: &rng) ? .sequence : .choice + let count = Int.random(in: 1 ... 3, using: &rng) + // Sometimes make the children a shared group (same id, same structure) to + // exercise the call/return sharing: build one child and reference it twice. + if compositor == .sequence, count >= 2, Bool.random(using: &rng) { + let shared = randomParticle(depth: depth - 1, rng: &rng, nextID: &nextID) + let id = allocate(&nextID) + let stamped = withID(shared, id) + return group(.sequence, [stamped, stamped], min: min, max: max) + } + let children = (0 ..< count).map { _ in randomParticle(depth: depth - 1, rng: &rng, nextID: &nextID) } + // Three groups in four are a named-group reference (a unique id, so a single + // call into a once-used fragment); the rest are anonymous inline groups. + let id = Int.random(in: 0 ..< 4, using: &rng) == 0 ? nil : allocate(&nextID) + return group(compositor, children, min: min, max: max, id: id) + } + + private func allocate(_ nextID: inout Int) -> Int { + defer { nextID += 1 } + return nextID + } + + private func withID(_ particle: PureXML.Schema.Particle, _ id: Int) -> PureXML.Schema.Particle { + guard case let .group(inner) = particle.term else { return particle } + var stamped = inner + stamped.definitionID = id + return .init(occurrenceRange: particle.occurrenceRange, term: .group(stamped)) + } + + /// One letter from the three-symbol alphabet, without a force unwrap. + private func letter(_ rng: inout SeededRNG) -> String { + ["a", "b", "c"][Int.random(in: 0 ..< 3, using: &rng)] + } +} + +/// A tiny deterministic RNG so the fuzz is reproducible. +private struct SeededRNG: RandomNumberGenerator { + private var state: UInt64 + init(seed: UInt64) { + state = seed + } + + mutating func next() -> UInt64 { + state = state &* 6_364_136_223_846_793_005 &+ 1_442_695_040_888_963_407 + var mixed = state + mixed = (mixed ^ (mixed >> 30)) &* 0xBF58_476D_1CE4_E5B9 + mixed = (mixed ^ (mixed >> 27)) &* 0x94D0_49BB_1331_11EB + return mixed ^ (mixed >> 31) + } +} diff --git a/Tests/SchemaDeterminismGroupBlowupTests.swift b/Tests/SchemaDeterminismGroupBlowupTests.swift index 81d1e1e..16801cb 100644 --- a/Tests/SchemaDeterminismGroupBlowupTests.swift +++ b/Tests/SchemaDeterminismGroupBlowupTests.swift @@ -50,6 +50,25 @@ struct SchemaDeterminismGroupBlowupTests { _ = try PureXML.Schema.Document(doublingSchema(leaf: "")) } + #if os(WASI) + @Test("validating a short instance against a deep nested-group schema is bounded (no hang)") + #else + @Test("validating a short instance against a deep nested-group schema is bounded (no hang)", .timeLimit(.minutes(1))) + #endif + func test_deepGroupInstanceValidationBounded() throws { + // Validating an instance builds the content automaton once per validated + // element. Before the graph-structured-stack matcher this build inlined the + // nested group references to `2^depth` states and hung, so no instance could be + // validated against this schema at all. The shared build is `O(depth)`, so a + // short, incomplete instance, the shape an editor sees mid-typing, validates to + // a located error promptly rather than hanging. (A complete instance would need + // `2^depth` `e` children, which is why the bound, not a valid full instance, is + // what matters here.) + let schema = try PureXML.Schema.Document(doublingSchema(leaf: "")) + let failures = try schema.validate("") + #expect(!failures.isEmpty) // one child cannot complete a model that needs 2^depth of them + } + #if os(WASI) @Test("a deep AMBIGUOUS nested-group model is rejected (not skipped)") #else diff --git a/docs/design/counted-content-automaton.md b/docs/design/counted-content-automaton.md index b5e1709..1f0f53e 100644 --- a/docs/design/counted-content-automaton.md +++ b/docs/design/counted-content-automaton.md @@ -620,18 +620,51 @@ configuration reaches the program accept under the EMPTY gss node. ### Step 4: switch + gate (cardinal rule sacred) -Differential (`ContentMatcher` is the oracle): over the whole XSTS corpus and the -`SchemaGenerator` fuzz (extended to emphasise multiply-referenced NULLABLE nested -groups, the exact shape that broke the pushdown), assert the RTN/GSS matcher's -accept/reject AND its per-child `MatchedParticle` equal the inlining matcher's. Only -when zero divergences: switch `ComplexValidatorContentModel` and `ComplexValidator` -to the RTN matcher, delete `ContentNFABuilder`. Add a perf ASSERTION (not a timeout, -per the reverted attempt's lesson) on a `K >= 24` doubling chain: poll the matcher's -own GSS-node and live-configuration counts and assert they stay `O(K)`, and that a -short instance validates in work proportional to its length. XSTS -`valid-schemas-rejected` stays 0; `invalid-schemas-accepted` does not rise (it may -fall: the instances that previously hung now get matched). +**Status (2026-06-28): DONE. Steps 1-3 built the verified matcher; Step 4 switched the +three production sites and measured XSTS neutrality on the full corpus in release (all +four buckets held: valid-schemas-rejected 0, valid-instances-rejected 0, +invalid-schemas-accepted 8, invalid-instances-accepted 14). The last per-instance `2^K` +content-model hang is gone.** -Each step is its own PR; steps 1-2 are FP-neutral by construction (the oracle stays -authoritative), and step 4 is the only behaviour change, gated to verdict- and -attribution-equivalence on the corpus before it lands. +Done in the commit `perf(schema): validate and complete on the GSS matcher`: the three +`ContentNFABuilder.build` sites below now build `ContentRTNBuilder`/`ContentRTNMatcher`. +A new end-to-end test (`SchemaDeterminismGroupBlowupTests.test_deepGroupInstanceValidationBounded`) +validates a short instance against a depth-40 doubling schema (`2^40` inlined positions) +in milliseconds, the case that previously hung. `ContentNFABuilder`/`ContentNFA` are kept +as the differential oracle, not deleted: the differential equivalence proof depends on +them, which outweighs removing now-test-only code. + +The XSTS corpus was found locally at `/private/tmp/xsts/xmlschema2006-11-06` (the memory's +pinned `/Users/mmj/...` path is for another Mac); set `XSTS_ROOT` to it and run +`swift test -c release --filter XSTS`. + +Built (PRs on `feat/gss-content-matcher`): `ContentRTN`/`ContentRTNBuilder` (call/return, +each `definitionID` compiled once), `ContentRTNMatcher` (the GSS simulation), and the +complete `ContentNFA` surface on it (`matchesWhole`, `follow`, `startStates`, `step`, +`admissible`, `isComplete`, `matchedParticles`). The differential `ContentRTNDifferentialTests` +locks the RTN/GSS matcher to the inlining `ContentNFA` oracle on: the accept verdict, the +per-child `MatchedParticle` kind, the located rejection index, the `admissible` set at +every prefix, `isComplete`, and `follow`, over hand cases, the named-over-wildcard +tie-break, shared-group doubling chains, and a 4000-case fuzz of nested models with +occurrences, wildcards, and parser-shaped group ids (unique per distinct group, shared +only for shared structure). The depth-24 build-bound (`< 480` states) is asserted. + +The switch is then a mechanical builder swap at three sites, each +`ContentNFABuilder.build(particle)` becoming `ContentRTNMatcher(ContentRTNBuilder.build(particle))`: + +1. `ComplexValidatorContentModel.sequenceStructureErrors` (the streaming located path). +2. `ComplexValidator.validateParticle` (the `matchedParticles` attribution). +3. `XSDCompletions` (the `follow(after:)` completion path). + +**The gate that must run on a corpus machine before the swap lands** (it CANNOT be run +on a Mac without the XSTS corpus, e.g. the one this was built on): set `XSTS_ROOT`, run +`XSTSSuiteTests` in release, and confirm `valid-schemas-rejected` and +`valid-instances-rejected` stay 0 and `invalid-*-accepted` do not rise (they may fall: +instances that previously hung now match). Keep `ContentNFABuilder` as the differential +oracle; do not delete it until the corpus run is green. Add a perf ASSERTION (not a +timeout, per the reverted attempt's lesson) on a `K >= 24` doubling chain: poll the +matcher's own GSS-node and live-configuration counts and assert they stay `O(K)`. + +Each step is its own PR; steps 1-3 are FP-neutral by construction (the oracle stays +authoritative and the matcher is unwired), and step 4 is the only behaviour change, +gated to verdict- and attribution-equivalence on the corpus before it lands.