PureXML/Sources/Schema/ContentRTNMatcher.swift
Mihaela Mihaljevic 5d38f0c82f
perf(schema): graph-structured-stack content matcher, killing the last 2^K content-model hang (#364)
* perf(schema): graph-structured-stack content matcher, sharing each xs:group (alongside)

The per-instance content-model build inlines every xs:group reference, so a
multiply-referenced nested group (g0=(g1,g1), g1=(g2,g2), ...) explodes to 2^K
states and reset actions: a depth-16 doubling chain already takes seconds, depth
20+ hangs. This compiles the same particle to a counted Recursive Transition
Network instead, where each definitionID group is compiled once and reached by a
call, and simulates it with a graph-structured stack.

ContentRTNBuilder mirrors ContentNFABuilder exactly except that a group with a
definitionID becomes a call into its shared, compiled-once fragment (memoized by
id; the fragment's accept is a return). The callee's counters are deliberately not
propagated to the caller: every counter is reset on its own particle's entry edge,
which is on the path from the callee's start, so a shared fragment resets itself on
every re-entry without the caller re-listing it. Re-listing was the residual
blow-up: it nested the reset-action lists to 2^K even when the states stayed O(K).

ContentRTNMatcher runs configurations (state, counters, gssNode): a call saves the
caller's counters on a GSS node interned by (returnState, savedCounters, position)
and a return restores them, so the shared callee never mixes counter state across
its callers; parents are a set and a popped node replays its pop to a parent added
later (the GLL completeness detail). The build drops to O(groups): a depth-24
doubling chain (2^24 inlined states) compiles to under 480 states in under a
millisecond.

It is gated by a differential (ContentRTNDifferentialTests) against the trusted
inlining matcher ContentNFA as the oracle: the two agree on the accept verdict and
the matched particle over the hand model cases, the named-over-wildcard tie-break,
the shared-group doubling chains that exercise call and return, and a 4000-case
fuzz of nested models with occurrences, wildcards, and shared groups, plus a bound
assertion. Built alongside the inlining matcher and not yet wired into validation,
so no behaviour change and the XSTS buckets are unchanged; the switch-over is gated
on this differential.

* perf(schema): add the streaming/located surface to the GSS matcher, lock it differentially

The content-model validator reads two things off the inlining matcher: matchedParticles
for per-child attribution, and the streaming startStates/step/admissible/isComplete for
located diagnostics (which child breaks the model, and the "expected a, b" hint). The GSS
matcher only had matchesWhole/matchedParticles, so it could not yet replace the inlining
matcher. This adds the streaming surface: admissible works because the closure already
expands call states into the callee's label states, so the follow set has the same shape
as ContentNFA's.

The differential gate now locks the whole surface, not just the accept verdict: driving
both matchers child by child, the first rejected child (step is nil) agrees, the
admissible expectation set agrees at the start and after every accepted child, and
isComplete agrees once the sequence is consumed. The fuzz now stamps group references the
way the parser does, a unique definitionID per distinct group (so a singly-referenced
named group also takes the call path, not only doubled groups), a shared id only for
genuinely shared structure, and some anonymous inline groups, over occurrences and
wildcards. Still alongside, not wired; no behaviour change, XSTS buckets unchanged.

* perf(schema): complete the GSS matcher surface with follow(after:), verified

follow(after:) is the last ContentNFA method the GSS matcher lacked: the completion
path (XSDCompletions) reads it for the acceptable-next labels and whether the content
may end. It composes the streaming surface, and the differential now locks its allowed
set and complete flag to ContentNFA too. The RTN matcher is now a complete drop-in for
the ContentNFA surface the validator and completions read, so the switch-over is a
mechanical builder swap gated on XSTS neutrality. Still alongside; no behaviour change.

* docs(design): record GSS Steps 1-3 done, the switch as a corpus-gated mechanical swap

Updates the counted-content-automaton Step 4 to the current state: the GSS matcher and
its full differential gate are built and green in-repo, and the production switch is now
a three-site builder swap blocked only on the XSTS neutrality measurement, which must run
on a machine that has the corpus. Records the exact swap sites and the gate so the corpus
machine can execute it mechanically.

* perf(schema): validate and complete on the GSS matcher, killing the last 2^K content-model hang

The three call sites that built the inlining content automaton once per validated
element, ComplexValidatorContentModel.sequenceStructureErrors (located structure
errors), ComplexValidator.validateParticle (per-child particle attribution), and
XSDCompletions (acceptable-next labels), now build ContentRTNBuilder and run
ContentRTNMatcher. A schema whose nested <xs:group> references inline to 2^K positions
(the g{i}=(g{i+1},g{i+1}) doubling shape) no longer makes each keystroke-time validation
of an element exponential in the nesting depth: the automaton is built in O(groups).

Validating a short instance against a depth-40 doubling schema (2^40 inlined positions),
the shape an editor sees mid-typing, now completes in milliseconds with a located error
instead of hanging; a new end-to-end test pins it.

Verified conformance-neutral on the full W3C XSTS corpus in release: valid-schemas-rejected
and valid-instances-rejected stay 0, invalid-schemas-accepted and invalid-instances-accepted
unchanged at 8 and 14. The located diagnostics, per-child attribution, and completions are
behaviour-identical by the surface differential (ContentRTNDifferentialTests), which locks
the verdict, matched particle, rejection index, admissible set, isComplete, and follow to
the inlining ContentNFA. ContentNFABuilder/ContentNFA are retained as that oracle.

* docs(design): mark GSS content-matcher Step 4 done, XSTS-neutral in production

Records that the three production sites are switched to the graph-structured-stack
matcher and the switch was measured conformance-neutral on the full W3C XSTS corpus in
release (all four buckets held). Notes the local corpus path and that ContentNFABuilder
is kept as the differential oracle rather than deleted.

* test(schema): compare full matched-particle attribution; bound a cyclic group id

From the adversarial review of the GSS matcher:

- The differential compared only the matched-particle kind (element/wildcard), so it
  could not observe the type/value metadata matchedParticles carries. Every fuzz and
  hand element now carries a per-name fixed value constraint and the comparison projects
  it, so the gate verifies the metadata is carried identically through call/return, not
  only that the kind agrees. A name maps to one constraint, so the comparison does not
  depend on the lowest-state-index tie-break (which only distinguishes same-name
  particles, a UPA violation the compiler rejects before instance validation); the
  matchedParticles doc now records that caveat.

- ContentRTNBuilder.groupFragment memoizes after compiling, so a (malformed) cyclic
  definitionID would recur without bound, the interactive-safety hazard this matcher
  removes. definitionID groups are acyclic by construction (the parser breaks cycles at
  element boundaries, the inlining builder relies on the same), but bound it anyway:
  a cycle degrades to a terminating empty fragment instead of recurring.

No production behaviour change; the differential, full suite, and the full XSTS corpus
in release (16/16, all four buckets held) stay green.
2026-06-28 05:07:37 +02:00

263 lines
13 KiB
Swift

/// The key under which a GSS node is interned during one match: a call's return
/// state, the caller's saved counters, and the input position at the call. Two
/// calls sharing all three share a node (the graph-structured-stack sharing).
private struct ContentRTNGSSKey: Hashable {
let returnState: Int
let savedCounters: [Int]
let position: Int
}
/// One graph-structured-stack node: a saved return context. `parents` are the
/// caller nodes (a set, so distinct callers sharing a return are merged); `popped`
/// records that a return has already fired through this node, so a parent added
/// afterward is served the replayed pop (the GLL completeness detail).
private struct ContentRTNGSSNode {
let returnState: Int
let savedCounters: [Int]
var parents: Set<Int>
var popped = false
}
/// A configuration: a program state, the live counter vector, and the GSS node
/// giving the return context. `gss == 0` is the empty stack. Internal (not private)
/// because it threads through the matcher's streaming surface, which the content-model
/// validator drives child by child.
struct ContentRTNConfiguration: Hashable {
var state: Int
var counters: [Int]
var gss: Int
}
extension PureXML.Schema {
/// Matches a child-name sequence against a ``ContentRTN`` using a
/// graph-structured stack, so a multiply-referenced group is traversed without
/// the inlining matcher's `2^K` state blow-up. Configurations are
/// `(state, counters, gssNode)`; a `call` saves the caller's counters on a GSS
/// node and a `return` restores them, so the shared callee fragment does not mix
/// counter state across its callers. GSS nodes are interned by
/// `(returnState, savedCounters, position)` with a parent set and a pop flag (the
/// GLL completeness detail: a parent added after the node was popped replays the
/// pop), which keeps the active set bounded by the program size, not by the
/// number of distinct return stacks.
final class ContentRTNMatcher {
private let rtn: ContentRTN
private var nodes: [ContentRTNGSSNode] = []
private var intern: [ContentRTNGSSKey: Int] = [:]
private var caps: [Int] = []
private var position = 0
/// Configurations produced by replaying a pop to a newly added parent during
/// closure; drained into the worklist.
private var pending: [ContentRTNConfiguration] = []
init(_ rtn: ContentRTN) {
self.rtn = rtn
}
// MARK: Public matching surface (mirrors ContentNFA)
func matchesWhole(_ names: [PureXML.Model.QualifiedName]) -> Bool {
var current = startStates(inputLength: names.count)
for name in names {
guard let next = step(current, over: name) else { return false }
current = next
}
return isComplete(current)
}
/// After consuming `names`, the labels the matcher can accept next and whether
/// the content may legally end here. Composes the streaming surface; mirrors
/// ``ContentNFA/follow(after:)``, used for completions. Returns an empty,
/// not-complete result when the prefix is already invalid.
func follow(after names: [PureXML.Model.QualifiedName]) -> (allowed: [TermLabel], complete: Bool) {
var current = startStates(inputLength: names.count)
for name in names {
guard let next = step(current, over: name) else { return ([], false) }
current = next
}
return (admissible(from: current), isComplete(current))
}
/// The active set before any child: resets this matcher's graph-structured
/// stack and returns the closure of the start configuration. `inputLength`
/// sizes the counter saturation caps. Mirrors ``ContentNFA/startStates``, but
/// is stateful (the GSS lives on the instance), so one matcher drives one
/// child scan: call this once, then ``step(_:over:)`` per child.
func startStates(inputLength: Int) -> Set<ContentRTNConfiguration> {
reset(inputLength: inputLength)
let seed = ContentRTNConfiguration(state: rtn.start, counters: Array(repeating: 0, count: rtn.counters.count), gss: 0)
return closure([seed])
}
/// Advances `current` by consuming one element `name`, or nil when `name` is
/// not admissible there. Mirrors ``ContentNFA/step(_:over:inputLength:)``; must
/// be called on the active set this matcher last produced (the GSS is stateful).
func step(_ current: Set<ContentRTNConfiguration>, over name: PureXML.Model.QualifiedName) -> Set<ContentRTNConfiguration>? {
var next: Set<ContentRTNConfiguration> = []
for configuration in current {
let state = configuration.state
if let label = rtn.states[state].label, let target = rtn.states[state].target, label.matches(name) {
next.insert(ContentRTNConfiguration(state: target, counters: configuration.counters, gss: configuration.gss))
}
}
if next.isEmpty { return nil }
position += 1
return closure(next)
}
/// The labels admissible from `current`, for diagnostics and completions. The
/// closure already expands `call` states into the callee's label states, so
/// this is the same shape as ``ContentNFA/admissible(from:)``: every active
/// configuration sitting on a labelled transition contributes its label.
func admissible(from current: Set<ContentRTNConfiguration>) -> [TermLabel] {
var labels: [TermLabel] = []
for configuration in current where rtn.states[configuration.state].target != nil {
if let label = rtn.states[configuration.state].label { labels.append(label) }
}
return labels
}
/// Whether the content may legally end at `current`: an active configuration at
/// the program accept with the empty stack (input consumed, all calls returned).
func isComplete(_ current: Set<ContentRTNConfiguration>) -> Bool {
current.contains { $0.state == rtn.accept && $0.gss == 0 }
}
/// The particle each child matched, in order (nil from the first rejected
/// child onward), with the inlining matcher's tie-breaking: a named particle
/// over a wildcard, then the lowest state index.
///
/// The lowest-state-index tie-break only distinguishes two same-kind particles
/// that both accept the child, which is a Unique Particle Attribution violation:
/// the schema compiler rejects such a model, so no instance is validated against
/// it. State numbering here differs from `ContentNFA`'s (the added call/return
/// states), so on such a (rejected) model the two matchers may pick different
/// competing particles; on a UPA-valid model at most one particle accepts each
/// child, so the pick, and the type/value metadata returned with it, is identical.
func matchedParticles(_ names: [PureXML.Model.QualifiedName]) -> [MatchedParticle?] {
var current = startStates(inputLength: names.count)
var result: [MatchedParticle?] = []
for name in names {
var next: Set<ContentRTNConfiguration> = []
var namedState: Int?
var anyState: Int?
for configuration in current {
let state = configuration.state
guard let label = rtn.states[state].label, let target = rtn.states[state].target, label.matches(name) else { continue }
next.insert(ContentRTNConfiguration(state: target, counters: configuration.counters, gss: configuration.gss))
if anyState.map({ state < $0 }) ?? true { anyState = state }
if case .name = label, namedState.map({ state < $0 }) ?? true { namedState = state }
}
guard let chosenState = namedState ?? anyState, !next.isEmpty else {
result.append(contentsOf: Array(repeating: nil, count: names.count - result.count))
return result
}
let matched = rtn.states[chosenState]
if case let .wildcard(wildcard) = matched.label {
result.append(.wildcard(wildcard))
} else {
result.append(.element(type: matched.elementType, valueConstraint: matched.valueConstraint))
}
position += 1
current = closure(next)
}
return result
}
// MARK: Simulation
private func reset(inputLength: Int) {
nodes = [ContentRTNGSSNode(returnState: -1, savedCounters: [], parents: [])] // node 0 = empty stack
intern = [:]
position = 0
caps = counterCaps(for: inputLength)
}
private func closure(_ seed: Set<ContentRTNConfiguration>) -> Set<ContentRTNConfiguration> {
var seen = seed
var work = Array(seed)
while true {
if !pending.isEmpty {
let configuration = pending.removeLast()
if seen.insert(configuration).inserted { work.append(configuration) }
continue
}
guard let configuration = work.popLast() else { break }
let state = rtn.states[configuration.state]
for edge in state.epsilon {
if let nextConfiguration = apply(edge, to: configuration), seen.insert(nextConfiguration).inserted {
work.append(nextConfiguration)
}
}
for call in state.calls {
let node = internNode(returnState: call.returnTo, savedCounters: configuration.counters, parent: configuration.gss)
let nextConfiguration = ContentRTNConfiguration(state: call.calleeStart, counters: configuration.counters, gss: node)
if seen.insert(nextConfiguration).inserted { work.append(nextConfiguration) }
drainPending(into: &seen, work: &work)
}
if state.isReturn, configuration.gss != 0 {
popReturn(configuration.gss)
drainPending(into: &seen, work: &work)
}
}
return seen
}
private func drainPending(into seen: inout Set<ContentRTNConfiguration>, work: inout [ContentRTNConfiguration]) {
while let configuration = pending.popLast() {
if seen.insert(configuration).inserted { work.append(configuration) }
}
}
/// Interns the GSS node for a call returning to `returnState` from the
/// caller's `savedCounters`, recording `parent`. If the node already exists
/// and was popped, replay the pop to the newly added parent (GLL).
private func internNode(returnState: Int, savedCounters: [Int], parent: Int) -> Int {
let key = ContentRTNGSSKey(returnState: returnState, savedCounters: savedCounters, position: position)
if let id = intern[key] {
if nodes[id].parents.insert(parent).inserted, nodes[id].popped {
pending.append(ContentRTNConfiguration(state: returnState, counters: savedCounters, gss: parent))
}
return id
}
let id = nodes.count
nodes.append(ContentRTNGSSNode(returnState: returnState, savedCounters: savedCounters, parents: [parent]))
intern[key] = id
return id
}
/// Returns from GSS node `id`: restore its saved counters and continue at its
/// return state for every parent. Mark it popped so a parent added later is
/// served too.
private func popReturn(_ id: Int) {
nodes[id].popped = true
let node = nodes[id]
for parent in node.parents {
pending.append(ContentRTNConfiguration(state: node.returnState, counters: node.savedCounters, gss: parent))
}
}
private func apply(_ edge: ContentEpsilonEdge, to configuration: ContentRTNConfiguration) -> ContentRTNConfiguration? {
for guardCondition in edge.guards where !guardCondition.accepts(configuration.counters) {
return nil
}
var values = configuration.counters
for action in edge.actions {
action.apply(to: &values, caps: caps)
}
return ContentRTNConfiguration(state: edge.target, counters: values, gss: configuration.gss)
}
private func counterCaps(for inputLength: Int) -> [Int] {
let reachable = inputLength == Int.max ? Int.max : inputLength + 1
return rtn.counters.map { scope in
guard let maximumCap = scope.range.maximum.clamped(to: reachable),
!scope.range.maximum.isGreaterThan(reachable)
else {
return scope.range.minimum.clamped(to: reachable)
}
return maximumCap
}
}
}
}