PureXML/docs/design/counted-content-automaton.md
Mihaela Mihaljevic 5d38f0c82f
perf(schema): graph-structured-stack content matcher, killing the last 2^K content-model hang (#364)
* perf(schema): graph-structured-stack content matcher, sharing each xs:group (alongside)

The per-instance content-model build inlines every xs:group reference, so a
multiply-referenced nested group (g0=(g1,g1), g1=(g2,g2), ...) explodes to 2^K
states and reset actions: a depth-16 doubling chain already takes seconds, depth
20+ hangs. This compiles the same particle to a counted Recursive Transition
Network instead, where each definitionID group is compiled once and reached by a
call, and simulates it with a graph-structured stack.

ContentRTNBuilder mirrors ContentNFABuilder exactly except that a group with a
definitionID becomes a call into its shared, compiled-once fragment (memoized by
id; the fragment's accept is a return). The callee's counters are deliberately not
propagated to the caller: every counter is reset on its own particle's entry edge,
which is on the path from the callee's start, so a shared fragment resets itself on
every re-entry without the caller re-listing it. Re-listing was the residual
blow-up: it nested the reset-action lists to 2^K even when the states stayed O(K).

ContentRTNMatcher runs configurations (state, counters, gssNode): a call saves the
caller's counters on a GSS node interned by (returnState, savedCounters, position)
and a return restores them, so the shared callee never mixes counter state across
its callers; parents are a set and a popped node replays its pop to a parent added
later (the GLL completeness detail). The build drops to O(groups): a depth-24
doubling chain (2^24 inlined states) compiles to under 480 states in under a
millisecond.

It is gated by a differential (ContentRTNDifferentialTests) against the trusted
inlining matcher ContentNFA as the oracle: the two agree on the accept verdict and
the matched particle over the hand model cases, the named-over-wildcard tie-break,
the shared-group doubling chains that exercise call and return, and a 4000-case
fuzz of nested models with occurrences, wildcards, and shared groups, plus a bound
assertion. Built alongside the inlining matcher and not yet wired into validation,
so no behaviour change and the XSTS buckets are unchanged; the switch-over is gated
on this differential.

* perf(schema): add the streaming/located surface to the GSS matcher, lock it differentially

The content-model validator reads two things off the inlining matcher: matchedParticles
for per-child attribution, and the streaming startStates/step/admissible/isComplete for
located diagnostics (which child breaks the model, and the "expected a, b" hint). The GSS
matcher only had matchesWhole/matchedParticles, so it could not yet replace the inlining
matcher. This adds the streaming surface: admissible works because the closure already
expands call states into the callee's label states, so the follow set has the same shape
as ContentNFA's.

The differential gate now locks the whole surface, not just the accept verdict: driving
both matchers child by child, the first rejected child (step is nil) agrees, the
admissible expectation set agrees at the start and after every accepted child, and
isComplete agrees once the sequence is consumed. The fuzz now stamps group references the
way the parser does, a unique definitionID per distinct group (so a singly-referenced
named group also takes the call path, not only doubled groups), a shared id only for
genuinely shared structure, and some anonymous inline groups, over occurrences and
wildcards. Still alongside, not wired; no behaviour change, XSTS buckets unchanged.

* perf(schema): complete the GSS matcher surface with follow(after:), verified

follow(after:) is the last ContentNFA method the GSS matcher lacked: the completion
path (XSDCompletions) reads it for the acceptable-next labels and whether the content
may end. It composes the streaming surface, and the differential now locks its allowed
set and complete flag to ContentNFA too. The RTN matcher is now a complete drop-in for
the ContentNFA surface the validator and completions read, so the switch-over is a
mechanical builder swap gated on XSTS neutrality. Still alongside; no behaviour change.

* docs(design): record GSS Steps 1-3 done, the switch as a corpus-gated mechanical swap

Updates the counted-content-automaton Step 4 to the current state: the GSS matcher and
its full differential gate are built and green in-repo, and the production switch is now
a three-site builder swap blocked only on the XSTS neutrality measurement, which must run
on a machine that has the corpus. Records the exact swap sites and the gate so the corpus
machine can execute it mechanically.

* perf(schema): validate and complete on the GSS matcher, killing the last 2^K content-model hang

The three call sites that built the inlining content automaton once per validated
element, ComplexValidatorContentModel.sequenceStructureErrors (located structure
errors), ComplexValidator.validateParticle (per-child particle attribution), and
XSDCompletions (acceptable-next labels), now build ContentRTNBuilder and run
ContentRTNMatcher. A schema whose nested <xs:group> references inline to 2^K positions
(the g{i}=(g{i+1},g{i+1}) doubling shape) no longer makes each keystroke-time validation
of an element exponential in the nesting depth: the automaton is built in O(groups).

Validating a short instance against a depth-40 doubling schema (2^40 inlined positions),
the shape an editor sees mid-typing, now completes in milliseconds with a located error
instead of hanging; a new end-to-end test pins it.

Verified conformance-neutral on the full W3C XSTS corpus in release: valid-schemas-rejected
and valid-instances-rejected stay 0, invalid-schemas-accepted and invalid-instances-accepted
unchanged at 8 and 14. The located diagnostics, per-child attribution, and completions are
behaviour-identical by the surface differential (ContentRTNDifferentialTests), which locks
the verdict, matched particle, rejection index, admissible set, isComplete, and follow to
the inlining ContentNFA. ContentNFABuilder/ContentNFA are retained as that oracle.

* docs(design): mark GSS content-matcher Step 4 done, XSTS-neutral in production

Records that the three production sites are switched to the graph-structured-stack
matcher and the switch was measured conformance-neutral on the full W3C XSTS corpus in
release (all four buckets held). Notes the local corpus path and that ContentNFABuilder
is kept as the differential oracle rather than deleted.

* test(schema): compare full matched-particle attribution; bound a cyclic group id

From the adversarial review of the GSS matcher:

- The differential compared only the matched-particle kind (element/wildcard), so it
  could not observe the type/value metadata matchedParticles carries. Every fuzz and
  hand element now carries a per-name fixed value constraint and the comparison projects
  it, so the gate verifies the metadata is carried identically through call/return, not
  only that the kind agrees. A name maps to one constraint, so the comparison does not
  depend on the lowest-state-index tie-break (which only distinguishes same-name
  particles, a UPA violation the compiler rejects before instance validation); the
  matchedParticles doc now records that caveat.

- ContentRTNBuilder.groupFragment memoizes after compiling, so a (malformed) cyclic
  definitionID would recur without bound, the interactive-safety hazard this matcher
  removes. definitionID groups are acyclic by construction (the parser breaks cycles at
  element boundaries, the inlining builder relies on the same), but bound it anyway:
  a cycle degrades to a terminating empty fragment instead of recurring.

No production behaviour change; the differential, full suite, and the full XSTS corpus
in release (16/16, all four buckets held) stay green.
2026-06-28 05:07:37 +02:00

34 KiB
Raw Blame History

Counted content automaton

This is the design for replacing the content-model NFA's occurrence unrolling with a counted automaton. It is intentionally separate from XSTS conformance fixes: this work closes the production-readiness resource-bound stopper, not one schema-validity bucket.

Problem

ContentNFABuilder currently compiles a particle by copying the repeated term minOccurs times and then adding optional copies up to maxOccurs. That makes automaton size proportional to the numeric occurrence value, not to the schema text. A short schema containing maxOccurs="1000000000" can request a billion states.

The current safety valves are:

  • occursUnrollCap = 16384
  • totalStateCap = 2^20
  • "degrade to star" when a cap is hit

Those caps prevent OOM, but they change the language. A finite {m,n} can become {m,unbounded} or, after an exhausted required prefix, effectively *. That is a silent false negative: invalid instances can pass because the matcher widened a bounded content model.

ContentModelDeterminism has a smaller version of the same issue: positionCap = 4096 skips the UPA check when the raw position automaton grows too large. Skipping a check is bounded as an under-rejection, but it is still silent debt.

Sources

Research notes read for this design:

  • /Volumes/Code/DeveloperExt/private/PureXML-research/notes/strange-cases-literature.md
  • /Volumes/Code/DeveloperExt/private/PureXML-research/notes/xsd-content-model-implementations-survey.md
  • /Volumes/Code/DeveloperExt/private/PureXML-research/research/scientific-literature.md

The relevant literature line is:

  • Brüggemann-Klein and Wood: UPA is one-unambiguity; the Glushkov position automaton is the canonical decision tool.
  • Kilpeläinen and Tuhkanen: numeric occurrence indicators make naive one-unambiguity algorithms erroneous or exponential; a polynomial method exists.
  • Gelade, Gyssens, and Martens: counting separates weak and strong determinism; unrolling can be exponentially less succinct than the counted representation.
  • Kilpeläinen 2011: the principled XML Schema determinism tool is a counting automaton, not occurrence unrolling.

These are witnesses for the algorithm family. PureXML's instance-time matcher now implements the counted shape against its own Particle model; counted UPA remains separate.

Correctness claims

Every claim below is labeled per proof discipline.

Claim Status Meaning
Counted matcher accepts exactly the same child-name language as the XSD particle tree for element, wildcard, sequence, and choice with minOccurs/maxOccurs. theorem target, implemented ContentMatcher.swift now uses counted configurations; the remaining work is writing the full structural proof against nullable nested particles.
Counted matcher keeps per-child matched-particle attribution. theorem target under UPA, implemented For UPA-valid content models, each consumed element is matched to exactly one particle; that particle's type/value metadata is the one used by child validation.
Counted matcher build size is independent of numeric occurrence magnitudes. theorem, implemented ContentNFABuilder allocates one counter scope per particle occurrence and a constant number of states/edges per particle; high-bound tests assert state count stays structural.
Counted matcher runtime is bounded for interactive validation. theorem target For N child elements, S program states, and D counter scopes, validation is bounded by O(N * S * D) time and O(S * D) live matcher memory. The constants are schema-structural, not occurrence-magnitude structural.
all groups remain exact. theorem already implemented, out of scope for NFA replacement all is validated by direct member counts in ComplexValidator.matchesAll and allStructureErrors; the counted automaton does not need to encode it initially.
Counted UPA/determinism replaces positionCap. DONE (2026-06-22) CompositionalDeterminism computes first/followlast over particle identities, summarizing each group once; ContentModelDeterminism delegates to it and positionCap is deleted. Proven verdict-equivalent to the inlining automaton by a differential over the whole XSTS corpus (zero divergences).

Bound representation

Do not store occurrence bounds as only Int.

XSD occurrence values are lexical nonNegativeInteger; users can write values larger than machine integer range. The parser needs an internal representation that preserves the bound and compares it without overflow:

enum OccurrenceUpper: Sendable, Equatable {
    case finite(NonNegativeDecimal)
    case unbounded
}

struct OccurrenceRange: Sendable, Equatable {
    var minimum: NonNegativeDecimal
    var maximum: OccurrenceUpper
}

NonNegativeDecimal is a normalized ASCII decimal string, with leading zeroes removed except for 0. It supports:

  • equality by string equality;
  • ordering by digit count, then lexicographic comparison;
  • isGreaterThan(_ limit: Int) without constructing a large integer;
  • clamped(to limit: Int) -> Int, used only when the input length gives a semantic cap.

For a concrete instance with N child elements:

  • any finite upper bound above N is equivalent to N + 1 for rejection/acceptance;
  • any minimum above N means the model cannot complete after this child list;
  • counters never need to hold values above N + 1.

That is the key bridge from arbitrary-size lexical numbers to machine-size runtime counters without changing semantics.

Program model

Compile the particle tree to a counted epsilon program:

struct CountedContentProgram: Sendable {
    var states: [State]
    var counters: [CounterScope]
    var start: StateID
    var accept: StateID
}

struct State: Sendable {
    var epsilon: [EpsilonEdge]
    var consuming: ConsumingEdge?
}

struct EpsilonEdge: Sendable {
    var target: StateID
    var guards: [CounterGuard]
    var actions: [CounterAction]
}

struct ConsumingEdge: Sendable {
    var label: TermLabel
    var target: StateID
    var particle: MatchedParticle
}

A CounterScope belongs to one particle occurrence range. The scope records:

  • its minimum;
  • its maximum (finite or unbounded);
  • which nested counters must be reset at the start of each iteration.

The edge language is small:

  • reset(counter) when entering a repeated particle's scope;
  • increment(counter) when one iteration of the repeated body completes;
  • counter >= minimum guard to allow exit;
  • counter < maximum guard to allow another iteration;
  • unconditional epsilon edges for sequence/choice structure.

No operation copies the body minimum or maximum times.

Compilation rules

The compiler is structural.

Element and wildcard

An element or wildcard compiles to one consuming edge. The consuming edge carries the same TermLabel and MatchedParticle metadata that ContentState carries today.

Sequence

Compile each child. Link the previous child's accept state to the next child's start state with an unconditional epsilon edge. Empty sequence is one epsilon edge from start to accept.

Choice

Compile each branch. Add unconditional epsilon edges from the choice start to each branch start, and from each branch accept to the choice accept.

Occurrence wrapper

Every particle gets an occurrence wrapper around its term program:

  1. On entry, reset the particle counter and every counter nested in its term.
  2. If minimum == 0, add an exit edge guarded by counter >= minimum.
  3. If counter < maximum, enter the body.
  4. On body accept, increment the counter.
  5. After increment, either exit when counter >= minimum or loop when counter < maximum.

For maxOccurs="unbounded", the upper guard is always true. For finite maximum, the guard uses decimal comparison against the current counter value.

The exact one-iteration case {1,1} still creates a counter scope initially. A later optimization may erase scopes whose range is statically {1,1}; the proof does not depend on that optimization.

Nested occurrence

Nested scopes are reset when their containing scope begins a new iteration. This is required for models like (a{2,3}){4,5}: the inner count is per outer iteration, not cumulative over the whole match.

This reset relation is computed during compilation by giving each particle term a set of descendant counter IDs. When an outer scope loops, its entry edge resets all descendants before the body is entered.

Runtime algorithm

A runtime configuration is:

struct Configuration: Hashable {
    var state: StateID
    var counters: CounterStore
}

CounterStore holds only counters whose value differs from zero. Values are machine Ints clamped to N + 1, where N is the child-count being validated.

Validation:

  1. Compute epsilon closure from the start configuration.
  2. For each child name:
    • inspect consuming edges from the current closure;
    • select matching edges by TermLabel.matches;
    • emit the matched particle for the chosen edge when the model is UPA-valid;
    • advance to each target configuration;
    • compute epsilon closure again.
  3. At the end, accept iff any closed configuration is at the accept state.

Closure applies guards and actions atomically per edge. A guard that fails drops that edge. An increment above N + 1 stores N + 1; no later decision can distinguish values above that threshold for a length-N input.

Determinism and attribution

For valid XSD content models, UPA says no input element can be attributed to two different particles at the same position without lookahead. The counted matcher should preserve the current API:

  • matchesWhole(_:)
  • follow(after:)
  • matchedParticles(_:)
  • incremental startStates / step

Internally, those APIs become wrappers over counted configurations instead of plain NFA state sets.

When more than one consuming edge matches a child, that is either:

  • an invalid schema that UPA should have rejected; or
  • a known UPA under-rejection frontier.

The implementation may keep the current deterministic tie-breaker for recovery diagnostics, but it must not call that tie-breaker a proof. The proof of attribution applies only when the content model is UPA-valid.

Follow sets and completions

follow(after:) returns labels available from the current closed configurations. The counted closure means these labels respect counters exactly:

  • after consuming a once in a{1,1}, <a> is not offered again;
  • after consuming a once in a{1,2}, <a> is offered once more;
  • after consuming a twice in a{1,2}, <a> is no longer offered;
  • after consuming fewer than minOccurs, completion is false even if the current state is structurally close to accept.

This is the visible user-facing benefit for IDE completions: numeric occurrence diagnostics become exact without unrolling large bounds.

Resource bound

Let:

  • P be the number of particle nodes in the compiled content model;
  • S be the number of counted program states;
  • D be the number of occurrence scopes/counters;
  • N be the number of child elements being matched.

Compilation creates a constant number of states and edges per particle and a constant number of counters per particle:

  • S = O(P)
  • D = O(P)
  • build memory O(P)

During matching, each active configuration has one state and at most D live counter values. For UPA-valid models the closure frontier is bounded by the program size. Each child step evaluates each reachable consuming edge and closure edge at most once per distinct reachable configuration after deduplication.

Target bound:

  • time O(N * S * D);
  • live memory O(S * D);
  • counter values bounded by N + 1;
  • no factor depends on the numeric magnitude of minOccurs or maxOccurs.

This is the proof obligation the implementation must satisfy before deleting the current caps.

Migration plan

  1. Done 2026-06-19: add NonNegativeDecimal, OccurrenceUpper, and OccurrenceRange while preserving the existing Particle API for call sites that only need small integer comparisons.
  2. Done 2026-06-19: replace ContentNFABuilder's unrolled body copies with a counted program behind the existing ContentNFA surface.
  3. Keep differential tests comparing DTD and XSD matchers on ordinary models below the current caps.
  4. Done 2026-06-19: add high-bound tests that are exact and previously impossible without language widening:
    • a{2,2} rejects one a and three as;
    • a{2,100000000000000000000} accepts two as and rejects one;
    • a{0,2} rejects three as;
    • (a{2,3}){2,2} accepts four to six as and rejects three/seven;
    • sequence(a{1000000000000}, b) never offers b before enough as.
  5. Done 2026-06-19: switch ComplexValidatorContentModel, XSDCompletions, and matchedParticles to the counted program.
  6. Done 2026-06-19: delete occursUnrollCap and totalStateCap.
  7. Done 2026-06-19: replace ContentNFAStateBudgetTests with exact high-bound tests. The old state-ceiling assertion should disappear because there is no state ceiling.
  8. Design and land the counted UPA pass; then delete positionCap.

Non-goals for the first implementation slice

  • Exact Particle Valid (Restriction) language inclusion. That remains the ParticleRestriction arc; inclusion for counted regular expressions is a different and harder problem.
  • Regex counted quantifier replacement. RegexAutomaton has the same cap posture, but XSD content models are the IDE resource-bound blocker. Regex can reuse the counted-bound representation later.
  • Changing all group semantics. all is already matched by counters over its members and has XSD-specific occurrence limits.

Done criteria

This arc is complete only when all are true:

  • Done 2026-06-19: ContentMatcher.swift contains no occursUnrollCap, totalStateCap, or cap-triggered "degrade to star" path.
  • ContentModelDeterminism.swift contains no positionCap skip.
  • Done 2026-06-19: tests include exact high-bound acceptance/rejection cases whose numeric bounds exceed any practical unroll count.
  • The proof labels in this document are upgraded from "theorem target" to "theorem" with code references.
  • docs/release-hurdles.md no longer lists content-model occurrence unrolling as an interactive-safety blocker.

Counted UPA / deleting positionCap: the compositional determinism algorithm

This section specifies the algorithm that deletes ContentModelDeterminism.positionCap, turning the "blocked design frontier" (table row above) into a ready-to-implement, FP-safe plan. Derived 2026-06-22 from a first-principles analysis of the current engine.

Why positionCap exists (root cause, confirmed in code)

ContentModelDeterminism builds a Glushkov position automaton; groupReference (ContentModelDeterminism.swift) inlines every <xs:group ref> via build(model,…), with the visiting set guarding only CYCLES, not repeated references. So g0=seq(ref g1, ref g1), g1=seq(ref g2, ref g2), … expands to 2^K positions: position count is EXPONENTIAL in schema text via multiply-referenced nested groups. positionCap = 4096 caps that blowup and, when hit, return nil (silently skips the UPA check = an under-rejection = the remaining stopper-3/4 debt). It is NOT primarily an overlap-cost issue (overlap is O(set²) but within the cap that is ~16M cheap ops, sub-second at compile time, not a bottleneck): the blocker is the exponential position COUNT.

Why naive fixes are unsound

Deduplicating positions by (label, particle) so a group's elements get ONE position regardless of reference count is UNSOUND: a group referenced in context A (followed by X) and context B (followed by Y) would merge followpos = X ∪ Y, manufacturing a decision set {X,Y} that occurs in no single context: a spurious conflict = a FALSE POSITIVE (violates stopper #1). Inlining is sound precisely because each reference site gets fresh positions with context-pure followpos.

The sound compositional algorithm (no inlining, bounded O(particles²))

Compute determinism over PARTICLE identities (ObjectIdentifier of the element/wildcard node), of which there are O(schema text size), never over inlined positions. For each sub-model node compute a summary that is CONTEXT-INDEPENDENT:

  • nullable: Bool
  • firstParticles: Set<(particle, label)>: distinct particles that can start it
  • lastParticles: Set<(particle, label)>: distinct particles that can end it
  • followlast: Set<{particle pair}>: particle pairs that share a decision set INTERNALLY (its own followpos-induced co-occurrences), plus the node's internal verdict
  • internalConflict: TermLabel?: a conflicting label if the sub-model is itself ambiguous

Composition rules (a conflict = two DISTINCT particles whose labels labelsOverlap sharing a decision set):

  • leaf (element/any): first = last = {(particle,label)}; no internal conflict.
  • choice(c…): decision set = ∪ firstParticles(cᵢ); conflict if that set holds a distinct-particle overlapping-label pair, or any child conflicts. first/last = unions.
  • sequence(c…): walk the nullable runs; at each gap the decision set is the firstParticles of the maximal nullable run starting there (joined to the preceding run's lastParticles via followpos): check each such set; first = firstParticles of the leading nullable prefix; last = lastParticles of the trailing nullable suffix.
  • all(c…): every member may follow every other and all share one decision set (mirrors the current all): the decision set is ∪ firstParticles(cᵢ) AND each member pair contributes cross-followpos; check the union for conflicts.
  • repetition R{m,n} with n>1 or unbounded: the Brüggemann-Klein star condition. followlast(R) ∪ firstParticles(R) must be conflict-free (the last->first repetition edge). {n,n} clamps to two copies exactly as today (determinism-equivalent); {0,1} adds no edge.
  • group ref: use the group's MEMOIZED summary (computed once: context-independent). Cross-boundary conflicts (the group's lastParticles followed by the next context's firstParticles, and the previous context's lastParticles followed by the group's firstParticles) are computed by the PARENT at each reference site, so two distinct reference sites never merge their continuations. This is the soundness key: each group is summarized once (no 2^K blowup) while cross-boundary decision sets stay context-pure.

Bound: each group summarized once; each composition checks O(particles²) pairs; total O(nodes × particles²), polynomial in schema text: a PROVEN bound, no cap.

FP-safe rollout (cardinal rule sacred)

  1. Implement the compositional checker ALONGSIDE the existing inlining violation; do NOT make it authoritative.
  2. Add a differential: for every fuzzed schema where positionCap is not hit (so the inlining oracle is exact) AND for the whole XSTS corpus, assert the compositional verdict EQUALS the inlining verdict. Extend SchemaGenerator to emphasize multiply-referenced nested groups.
  3. Iterate the compositional checker until it agrees everywhere (the inlining check is the trusted oracle; any divergence is a bug in the new checker, fixed toward agreement). This makes the change FP-safe by construction: nothing ships until proven equivalent on the oracle.
  4. Only then switch violation to the compositional checker and DELETE positionCap. Add a perf test on the 2^K-group pattern proving polynomial (not exponential) time.

Acceptance criteria (supersedes the positionCap row in the table above)

  • ContentModelDeterminism.swift contains no positionCap and no silent UPA skip.
  • The compositional/inlining differential passes on XSTS + a deep group-ref fuzz.
  • A perf test compiles a 2^K-nested-group schema (K ≥ 20) within a tight time bound.
  • XSTS valid-schemas-rejected stays 0 and invalid-schemas-accepted does not rise (it may fall, as previously-skipped pathological models now get checked).

Status: DONE (2026-06-22)

Implemented as CompositionalDeterminism (Sources/Schema/ContentModelDeterminismCompositional.swift); ContentModelDeterminism delegates to it and the inlining position automaton and positionCap are deleted. The followlast detail (a repeating end-item's loop-back combined with the suffix-first as ONE decision set, and an all member's cross to the other members combined likewise) was essential: a first attempt that emitted the loop and the suffix as SEPARATE decision sets under-detected 13 XSTS content models; the differential caught every one. Acceptance met: no positionCap and no silent skip; verdict-equivalent on the XSTS corpus (zero divergences); SchemaDeterminismGroupBlowupTests compiles a 2^20-position model without hanging and rejects an ambiguous one; XSTS valid-schemas-rejected 0 and invalid-schemas-accepted unchanged at 15.

The matcher's group-sharing residual (the open resource bound)

The counted automaton above removed the occurrence-magnitude blow-up (a maxOccurs="1000000000" no longer allocates a billion states). It did NOT remove the group-sharing blow-up. ContentNFABuilder.build is called per instance (ComplexValidator and ComplexValidatorContentModel), and its group(_:) recurses into a group's body once per reference. The compiled Particle tree is a DAG (a multiply-referenced group is one shared, copy-on-write Particle), but the NFA build flattens that DAG by tree traversal, so a doubling chain

g0 = (g1, g1)   g1 = (g2, g2)   ...   g_{K-1} = (gK, gK)   gK = <element X/>

builds 2^K states. There is no cap on this path; validating any instance against such a model hangs on the first validate. This is the last place the production-readiness resource bound (stopper 4) is a hang, not a proven bound. The determinism check escaped the same 2^K by summarizing each group once (CompositionalDeterminism, DONE above); the matcher must escape it too.

Why the reverted pushdown did not close it

The natural fix is to compile the group DAG to a Recursive Transition Network: a group reference becomes a call(returnTo:) epsilon edge into the group's single shared sub-program, and the group's accept becomes a pop. Then the program is O(groups), not 2^K. That rewrite was built and reverted (2026-06-23): it was verdict-correct (full suite + XSTS + the streaming differential all passed) but did not bound anything. It keyed each live configuration on (state, counters, stack) and enumerated configurations explicitly. Over a nullable doubling chain the start-state epsilon-closure alone reaches exponentially many distinct return stacks (measured ~3^depth: startStates().count was 1.72M at depth 10), because a nullable group can be entered and exited by epsilon, and each distinct path leaves a distinct explicit stack. The exponential moved from the build to the closure; arguably worse, because it now fires on every keystroke, not just on compile.

The lesson (recorded against this design): an RTN with explicit stacks is not a bound. The configuration count must be made independent of the number of distinct stacks.

The fix: a graph-structured stack (the stack DAG)

Represent the return stack as a graph-structured stack (GSS), the device GLL and Earley parsing use for exactly this. A GSS node is a pair (returnState, sourceSet) where sourceSet is a set of parent GSS nodes rather than a single parent. Two computations that reach the same returnState by different paths share that one GSS node, recording both parents in its sourceSet; they do not fork two stacks. A pop from a GSS node fans out to all its parents.

The live configuration set is then keyed on (state, counters, gssNode), and the number of distinct GSS nodes at a given input position is bounded by the number of distinct returnStates, which is O(states), NOT by the number of distinct explicit stacks (2^K). The nullable doubling chain that produced 3^depth distinct explicit stacks produces O(K) distinct GSS nodes: every entry into the shared g_i sub-program that wants to return to the same site reuses the one GSS node for that site.

This is the precise meaning of "collapse matching-equivalent stack-distinct configurations": configurations that agree on (state, counters) and on the return continuation (the GSS node, which is the shared tail of every stack that returns there) are one configuration. Different prefixes of the stack are already shared as the GSS's parent edges.

Bound (theorem target, to discharge before implementing)

Let S be the number of program states (O(schema text), since groups compile once), D the number of counter scopes (O(S)), and N the instance child count. Counters are clamped to N+1 (the bridge above), so a counter vector has O(D) entries each in 0...N+1.

  • GSS size. A GSS node is created per (returnState, inputPosition), so there are O(S·N) GSS nodes over a full match, each with at most O(S) parents: O(S²·N) GSS edges. Claim: independent of group-nesting depth, hence no 2^K.
  • Epsilon-closure termination. Group definitions are non-recursive (a transitive self-reference is already a schema error), so the call graph is a DAG; combined with the counter guards (a counter cannot be incremented past its clamp, and a nullable body's empty iteration is taken at most once per GSS node), the closure at one input position visits each (state, counters, gssNode) once.
  • Per-position work. O(S·D·(N+1)^{?}) configurations bounded by deduplication on (state, counters, gssNode); for a UPA-valid model (the only models the matcher ever runs on, since a non-deterministic model is a rejected schema) the live set at each position has size O(S) because determinism leaves at most one matching consuming edge per element. So for the conformant case the match is O(N·S·D); the general (defensive) bound is polynomial via the GSS, never 2^K.

The acceptance discipline mirrors the determinism rollout, which the cardinal rule makes non-negotiable:

FP-safe rollout (cardinal rule sacred)

  1. Build the GSS matcher ALONGSIDE the inlining ContentNFABuilder; the inlining matcher stays authoritative.
  2. Differential: for every fuzzed schema whose inlining build stays under a size the oracle can afford (so the inlining matcher is exact) AND over the whole XSTS corpus, assert the GSS matcher's accept/reject AND its per-child matched particle equal the inlining matcher's. Extend SchemaGenerator to emphasize multiply-referenced nullable nested groups (the exact shape that broke the pushdown). Reuse the #246 deep-group differential.
  3. Iterate toward agreement; the inlining matcher is the trusted oracle, any divergence is a GSS bug fixed toward it. Nothing ships until equivalent.
  4. Switch the matcher and delete the inlining build. Add a perf ASSERTION (not a wall-clock limit, per the reverted attempt's lesson): on the 2^K doubling chain at K ≥ 24, assert the live configuration set and GSS node count stay O(K) (poll the matcher's own counts), and that a short instance validates in work proportional to its length, not 2^K.

Acceptance criteria

  • ContentNFABuilder's per-reference group expansion is gone; a group compiles once.
  • The GSS/inlining differential passes on XSTS + a deep nullable-group fuzz.
  • A perf test on a K ≥ 24 doubling chain asserts O(K) configuration/GSS counts (the bound is observed, not timed).
  • XSTS valid-schemas-rejected stays 0 and invalid-schemas-accepted does not rise (it may fall: instances that previously hung now get matched).

Status: DESIGNED (2026-06-28), not yet implemented

The reverted pushdown failed for lack of this proof; the GSS is the representation that discharges the bound. Implementation is a separate, FP-critical effort gated exactly as the compositional-determinism change was. Until it lands, the residual stays disclosed: a deeply-nested, multiply-referenced group model is the one content shape whose per-instance match is not yet polynomial.

Implementation blueprint (code-level, for the focused build)

The algorithm and bound above are settled; this is the concrete plan so the build is mechanical, not exploratory. It is the full scope of the reverted RTN attempt (its sharing was correct) with the explicit stack replaced by the GSS.

Step 1: group-definition identity (no behaviour change)

Sources/Schema/ComplexType.swift: Term/Group are value types, so a multiply-referenced group has no exposed identity. Add var definitionID: Int? to Group. Sources/Schema/XSDParserParticles.swift groupReferenceParticle already keys the shared expansion by GroupParticleMemoKey(definition: ObjectIdentifier(definition), visiting:) (line ~105). Assign one stable Int per distinct key (a counter on XSDContext, beside groupMemo) and stamp the cached group's definitionID when it is first compiled, so every reference to the same expansion carries the same id and two unrelated groups differ. Verify with a unit test on the parsed Particle tree (same id for sibling refs, distinct for distinct definitions). Used by nothing yet, so it ships green and FP-neutral on its own.

Step 2: call/return compilation (alongside, unused)

A new ContentRTNBuilder (do NOT mutate ContentNFABuilder, the trusted oracle) compiles each definitionID to ONE fragment, memoized in a [Int: (start, accept)]. A .group(group) term whose group.definitionID is set compiles to a call epsilon edge (target = the group fragment start, carrying the return state) wrapped by the reference's own occurrence scope; the fragment's accept is a return. Add two epsilon-edge kinds, call(returnTo:) and pop, to a new ContentState variant or a parallel program type. Counters stay in the configuration vector and are reset on entry exactly as today, so a shared fragment entered from two live configurations does not alias (the reverted attempt proved this part sound). A size test asserts the doubling chain compiles to O(K) states.

Step 3: GSS simulation (the new, FP-critical core)

A GSSNode is (returnState: Int, parents: Set<GSSNodeID>), interned per (returnState, inputPosition) so paths that return to the same site share one node. The live configuration is (state, counters, gssNode). The epsilon-closure: on a call(returnTo: B) from gss node P, intern Q = (B, {P}) (union P into an existing Q's parents) and continue at the callee start under Q; on a pop under Q, fan out to (returnState, parent) for each parent. Deduplicate the worklist on (state, counters, gssNode). The consuming step and isComplete/admissible/ matchedParticles carry the gss node through unchanged in shape. Accept iff a configuration reaches the program accept under the EMPTY gss node.

Step 4: switch + gate (cardinal rule sacred)

Status (2026-06-28): DONE. Steps 1-3 built the verified matcher; Step 4 switched the three production sites and measured XSTS neutrality on the full corpus in release (all four buckets held: valid-schemas-rejected 0, valid-instances-rejected 0, invalid-schemas-accepted 8, invalid-instances-accepted 14). The last per-instance 2^K content-model hang is gone.

Done in the commit perf(schema): validate and complete on the GSS matcher: the three ContentNFABuilder.build sites below now build ContentRTNBuilder/ContentRTNMatcher. A new end-to-end test (SchemaDeterminismGroupBlowupTests.test_deepGroupInstanceValidationBounded) validates a short instance against a depth-40 doubling schema (2^40 inlined positions) in milliseconds, the case that previously hung. ContentNFABuilder/ContentNFA are kept as the differential oracle, not deleted: the differential equivalence proof depends on them, which outweighs removing now-test-only code.

The XSTS corpus was found locally at /private/tmp/xsts/xmlschema2006-11-06 (the memory's pinned /Users/mmj/... path is for another Mac); set XSTS_ROOT to it and run swift test -c release --filter XSTS.

Built (PRs on feat/gss-content-matcher): ContentRTN/ContentRTNBuilder (call/return, each definitionID compiled once), ContentRTNMatcher (the GSS simulation), and the complete ContentNFA surface on it (matchesWhole, follow, startStates, step, admissible, isComplete, matchedParticles). The differential ContentRTNDifferentialTests locks the RTN/GSS matcher to the inlining ContentNFA oracle on: the accept verdict, the per-child MatchedParticle kind, the located rejection index, the admissible set at every prefix, isComplete, and follow, over hand cases, the named-over-wildcard tie-break, shared-group doubling chains, and a 4000-case fuzz of nested models with occurrences, wildcards, and parser-shaped group ids (unique per distinct group, shared only for shared structure). The depth-24 build-bound (< 480 states) is asserted.

The switch is then a mechanical builder swap at three sites, each ContentNFABuilder.build(particle) becoming ContentRTNMatcher(ContentRTNBuilder.build(particle)):

  1. ComplexValidatorContentModel.sequenceStructureErrors (the streaming located path).
  2. ComplexValidator.validateParticle (the matchedParticles attribution).
  3. XSDCompletions (the follow(after:) completion path).

The gate that must run on a corpus machine before the swap lands (it CANNOT be run on a Mac without the XSTS corpus, e.g. the one this was built on): set XSTS_ROOT, run XSTSSuiteTests in release, and confirm valid-schemas-rejected and valid-instances-rejected stay 0 and invalid-*-accepted do not rise (they may fall: instances that previously hung now match). Keep ContentNFABuilder as the differential oracle; do not delete it until the corpus run is green. Add a perf ASSERTION (not a timeout, per the reverted attempt's lesson) on a K >= 24 doubling chain: poll the matcher's own GSS-node and live-configuration counts and assert they stay O(K).

Each step is its own PR; steps 1-3 are FP-neutral by construction (the oracle stays authoritative and the matcher is unwired), and step 4 is the only behaviour change, gated to verdict- and attribution-equivalence on the corpus before it lands.