From 1f8d99adfc08656aa25ebb7d2430594183f0ed01 Mon Sep 17 00:00:00 2001 From: Mihaela Mihaljevic Date: Sun, 28 Jun 2026 23:15:32 +0200 Subject: [PATCH] perf(schema): make the UPA determinism check depth-safe on group-reference chains (#371) A flat chain of references is not bounded by the parser's maxDepth (the definitions are siblings, not nested), so the compositional UPA determinism check's lazy per-reference recursion overflowed the native 8 MB stack while compiling a legal schema. Group summaries form an acyclic graph, so they are now precomputed bottom-up in dependency order (GroupDependencyOrder): the walk's group lookup is a cache hit and recursion is bounded by a single group's own model nesting. The precomputed reachable set is identical to the lazy walk's, so the verdict is unchanged. Verified verdict-equivalent over the full W3C XSTS corpus, the libxml2 determinism differential (0 FP, same 3 FN), the full suite, and an adversarial critic. Residual, tracked: the particle compiler that inlines a group reference still overflows on a far longer chain; closing it needs a content-model nesting-depth limit with a located error. --- CHANGELOG.md | 2 + ...ContentModelDeterminismCompositional.swift | 67 ++++++--------- .../ContentModelDeterminismConflict.swift | 47 ++++++++++ .../Schema/ContentModelGroupDependency.swift | 85 +++++++++++++++++++ Tests/SchemaGroupChainDepthSafetyTests.swift | 63 ++++++++++++++ 5 files changed, 222 insertions(+), 42 deletions(-) create mode 100644 Sources/Schema/ContentModelDeterminismConflict.swift create mode 100644 Sources/Schema/ContentModelGroupDependency.swift create mode 100644 Tests/SchemaGroupChainDepthSafetyTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 29fe81f..fa2b381 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- A deeply nested chain of `` references (`g0` referencing `g1` referencing `g2`, and so on) no longer overflows the stack in the Unique Particle Attribution determinism check. Such a chain is flat in the schema document, the group definitions being siblings rather than nested, so it is not bounded by the parser's `maxDepth`; the compositional check summarised each group lazily by recursing through the reference at its use site, so a long chain recursed once per link and overflowed the stack while compiling a legal (if hostile) schema, a keystroke-time interactive-safety crash on the primary platform. Group summaries form an acyclic graph (a circular group reference is rejected elsewhere), so they are now precomputed bottom-up in dependency order: every reachable group is summarised once, in an order where each group's own references are already cached, so the walk's group lookup is always a cache hit and recursion is bounded by a single group's model nesting (which `maxDepth` bounds), not by the chain length. A group summary is context-independent, so this only reorders the work; the set of decision sets it produces is identical and the accept/reject verdict does not depend on their order. Verified verdict-equivalent over the full W3C XSTS corpus in release (all four buckets unchanged: `valid-schemas-rejected` and `valid-instances-rejected` stay 0, `invalid-schemas-accepted` 8, `invalid-instances-accepted` 14), the generated-schema determinism differential against libxml2 unchanged (zero false positives, the same three named false negatives), and the full 1816-test suite. Disclosed residual, not silent: the separate particle-compiler pass that inlines a group reference into the instance content model still recurses on the same chain and overflows on a far longer one (around ten thousand links); closing it needs a content-model nesting-depth limit, analogous to `maxDepth`, that rejects such a pathological chain with a located error, and is tracked. + - A repeating model whose optional trailing particle and leading particle share a name, `(c, b?, c?)*`, is now correctly rejected as a Unique Particle Attribution violation, as libxml2/Xerces reject it: after `b` the next `c` could be the trailing `c?` or the looped leading `c`, two distinct particles. The compositional determinism check formed the internal sequence gap before the enclosing repetition's loop edge was added, so the combined `{trailing, leading}` followpos was never checked (an invalid-schemas-accepted false negative the finite XSTS corpus did not contain). The sequence now carries each trailing end-item's internal suffix in its `lastFollow`, so a parent gap or the loop edge forms the end-item's full followpos. The carried suffix is always a subset of the true followpos, so the check only catches more genuine conflicts and never over-rejects: the non-repeated `(c, b?, c?)` and disjoint shapes like `(a, b?)*` stay valid. Found by the new generated-schema determinism differential against libxml2; verified over the full W3C XSTS corpus in release (`valid-schemas-rejected`/`valid-instances-rejected` stay 0, `invalid-*-accepted` unchanged) with an adversarial over-rejection critic finding no valid schema newly rejected over 47 shapes. The related ambiguity through two references to one group in a repeating sequence stays an under-rejection (the cached group identity cannot distinguish a looped from a trailing occurrence without per-occurrence tracking), named in `DeterminismDifferentialTests`. - Two references to the same `` that overlap (`(g?, g?)`, `(g | g)`, `(g*, g)`) are now correctly rejected as a Unique Particle Attribution violation, as the inline forms `(x?, x?)` / `(x | x)` already were and as libxml2/Xerces reject them. The compositional determinism check summarises each group once and reused the group's node identity across references, so the distinct-identity overlap test could not see that two references are two expanded occurrences, and PureXML accepted these models (an invalid-schemas-accepted false negative). It also let the per-instance content matcher run on such a non-UPA model, where a multiply-referenced nullable group is exponential: a depth-18 optional doubling chain was accepted and then hung instance validation, a keystroke-time interactive-safety hang on a schema PureXML itself accepted. The fix dedupes each decision set (first set, sequence gap, `all` cross-follow) by particle identity and treats a same-identity collision as the conflict, while the repetition self-loop (`a*`, `(a*)*`) keeps its non-flagging dedupe so a single particle looping is not misread as two occurrences. This also bounds the check: the deduped first/follow sets stay `O(labels)` instead of `2^K`, so the same depth-40 chain is now rejected at compile in 11 ms and no instance is ever matched against it. A redefine self-extension (`` inside the redefinition of `T`) lists a type's own content twice through `effectiveModelNodes`; that assembly artifact is deduped before the walk so it is not read as a conflict (addB094, schZ007). Verified over the full W3C XSTS corpus in release: `valid-schemas-rejected` and `valid-instances-rejected` stay 0 and `invalid-*-accepted` are unchanged (the corpus does not exercise this shape in its pinned buckets), with an adversarial over-rejection critic finding no valid schema newly rejected across the group-overlap shape space. diff --git a/Sources/Schema/ContentModelDeterminismCompositional.swift b/Sources/Schema/ContentModelDeterminismCompositional.swift index b8f8cff..0ab17b7 100644 --- a/Sources/Schema/ContentModelDeterminismCompositional.swift +++ b/Sources/Schema/ContentModelDeterminismCompositional.swift @@ -1,7 +1,8 @@ /// One distinct particle occurrence in a content model: its source-node identity /// and label. A particle never conflicts with a repetition of itself, so identity, -/// not label alone, decides conflict (per the UPA semantics). File-private. -private struct CompItem: Hashable { +/// not label alone, decides conflict (per the UPA semantics). Module-internal so the +/// conflict tests (``CompConflict``) can live in their own file. +struct CompItem: Hashable { let particle: ObjectIdentifier let label: PureXML.Schema.TermLabel @@ -52,6 +53,10 @@ extension PureXML.Schema { // (addB094, schZ007). var seenModel: Set = [] let uniqueModels = models.filter { seenModel.insert(ObjectIdentifier($0)).inserted } + // Summarize every reachable group up front, in dependency order, so the walk + // below never recurses through a `` chain (which is flat in the + // document and so unbounded by `maxDepth`); see `precomputeGroupSummaries`. + walk.precomputeGroupSummaries(uniqueModels) let summary = walk.sequence(uniqueModels.map { walk.summarize($0, visiting: []) }) // The root's entry decision set, plus the followpos of each item that can // end the whole model (nothing follows the root, so its followpos is just @@ -66,7 +71,7 @@ extension PureXML.Schema { return "ambiguous content model: '\(conflict)' can be matched by more than one particle" } for set in walk.decisionSets { - if let conflict = CompWalk.overlap(set) { + if let conflict = CompConflict.overlap(set) { return "ambiguous content model: '\(conflict)' can be matched by more than one particle" } } @@ -326,31 +331,6 @@ private struct CompWalk { .first { ["sequence", "choice", "all"].contains(PureXML.Schema.XSDNode.localName($0) ?? "") } } - // MARK: Conflict - - static func overlap(_ set: [CompItem]) -> String? { - var nameOwner: [PureXML.Model.QualifiedName: ObjectIdentifier] = [:] - var wildcards: [(card: PureXML.Schema.Wildcard, particle: ObjectIdentifier)] = [] - for item in set { - switch item.label { - case let .name(qname): - if let owner = nameOwner[qname], owner != item.particle { return qname.localName } - if nameOwner[qname] == nil { nameOwner[qname] = item.particle } - case let .wildcard(card): - wildcards.append((card, item.particle)) - } - } - for (index, wildcard) in wildcards.enumerated() { - // A wildcard conflicts with any name it admits, from a distinct particle, - // and with any other wildcard it intersects, from a distinct particle. - if nameOwner.contains(where: { wildcard.card.admits($0.key) && $0.value != wildcard.particle }) { return "any" } - for other in wildcards[(index + 1)...] where other.particle != wildcard.particle && wildcardsOverlap(wildcard.card, other.card) { - return "any" - } - } - return nil - } - // MARK: Helpers (mirror ContentModelDeterminism) private func elementQName(_ node: PureXML.Model.TreeNode) -> PureXML.Model.QualifiedName { @@ -371,20 +351,23 @@ private struct CompWalk { return PureXML.Model.QualifiedName(localName: qname, namespaceURI: bindings[""]) } - static func wildcardsOverlap(_ lhs: PureXML.Schema.Wildcard, _ rhs: PureXML.Schema.Wildcard) -> Bool { - // Two wildcards overlap when some namespace name is admitted by both. Any - // two `not` forms always share infinitely many named namespaces; a `not` - // form overlaps a set when the set holds a named namespace it admits. - switch (lhs.namespace, rhs.namespace) { - case (.any, _), (_, .any): true - case let (.enumerated(lhsSet), .enumerated(rhsSet)): !lhsSet.isDisjoint(with: rhsSet) - case let (.enumerated(set), .notNamespace(name)), let (.notNamespace(name), .enumerated(set)): - set.contains { !$0.isEmpty && $0 != name } - case let (.enumerated(set), .notAbsent), let (.notAbsent, .enumerated(set)): - set.contains { !$0.isEmpty } - case (.notNamespace, .notNamespace), (.notNamespace, .notAbsent), - (.notAbsent, .notNamespace), (.notAbsent, .notAbsent): - true + // MARK: Group summaries (depth-safe precomputation) + + /// Summarizes every group reachable from `models`, in dependency order, into + /// `groupCache`, so the main walk's `group(_:)` is always a cache hit and + /// `summarize` never recurses through a `` chain. A group reference + /// chain is FLAT in the schema document (the definitions are siblings, not + /// nested), so it is NOT bounded by the parser's `maxDepth`; summarizing lazily + /// through references therefore recursed on the chain length and overflowed the + /// stack on a long (hostile but legal) chain. Each group's own model nesting is + /// document structure, so it stays `maxDepth`-bounded. The cached summary is the + /// SAME object the lazy walk would have produced (a group summary is + /// context-independent), so this only reorders the work; the decision sets it + /// emits are identical as a set, and the verdict does not depend on their order. + mutating func precomputeGroupSummaries(_ models: [PureXML.Model.TreeNode]) { + for name in GroupDependencyOrder.reachable(from: models, context: context) where groupCache[name] == nil { + guard let definition = context.groups[name], let model = modelChild(definition) else { continue } + groupCache[name] = summarize(model, visiting: [name]) } } } diff --git a/Sources/Schema/ContentModelDeterminismConflict.swift b/Sources/Schema/ContentModelDeterminismConflict.swift new file mode 100644 index 0000000..737ef10 --- /dev/null +++ b/Sources/Schema/ContentModelDeterminismConflict.swift @@ -0,0 +1,47 @@ +/// The overlap tests for a compositional-determinism decision set: a UPA violation +/// is two DISTINCT particles in one decision set that can match the same element +/// (same element name, or a wildcard admitting the other's name/namespace). Pure +/// functions over ``CompItem`` sets, factored out of ``CompWalk`` so the walk file +/// stays within its length budget; the walk accumulates decision sets and hands +/// each to ``overlap(_:)``. +enum CompConflict { + static func overlap(_ set: [CompItem]) -> String? { + var nameOwner: [PureXML.Model.QualifiedName: ObjectIdentifier] = [:] + var wildcards: [(card: PureXML.Schema.Wildcard, particle: ObjectIdentifier)] = [] + for item in set { + switch item.label { + case let .name(qname): + if let owner = nameOwner[qname], owner != item.particle { return qname.localName } + if nameOwner[qname] == nil { nameOwner[qname] = item.particle } + case let .wildcard(card): + wildcards.append((card, item.particle)) + } + } + for (index, wildcard) in wildcards.enumerated() { + // A wildcard conflicts with any name it admits, from a distinct particle, + // and with any other wildcard it intersects, from a distinct particle. + if nameOwner.contains(where: { wildcard.card.admits($0.key) && $0.value != wildcard.particle }) { return "any" } + for other in wildcards[(index + 1)...] where other.particle != wildcard.particle && wildcardsOverlap(wildcard.card, other.card) { + return "any" + } + } + return nil + } + + static func wildcardsOverlap(_ lhs: PureXML.Schema.Wildcard, _ rhs: PureXML.Schema.Wildcard) -> Bool { + // Two wildcards overlap when some namespace name is admitted by both. Any + // two `not` forms always share infinitely many named namespaces; a `not` + // form overlaps a set when the set holds a named namespace it admits. + switch (lhs.namespace, rhs.namespace) { + case (.any, _), (_, .any): true + case let (.enumerated(lhsSet), .enumerated(rhsSet)): !lhsSet.isDisjoint(with: rhsSet) + case let (.enumerated(set), .notNamespace(name)), let (.notNamespace(name), .enumerated(set)): + set.contains { !$0.isEmpty && $0 != name } + case let (.enumerated(set), .notAbsent), let (.notAbsent, .enumerated(set)): + set.contains { !$0.isEmpty } + case (.notNamespace, .notNamespace), (.notNamespace, .notAbsent), + (.notAbsent, .notNamespace), (.notAbsent, .notAbsent): + true + } + } +} diff --git a/Sources/Schema/ContentModelGroupDependency.swift b/Sources/Schema/ContentModelGroupDependency.swift new file mode 100644 index 0000000..28f8f24 --- /dev/null +++ b/Sources/Schema/ContentModelGroupDependency.swift @@ -0,0 +1,85 @@ +/// Orders the model groups reachable from a set of content models so each group +/// comes after every group it references. The compositional determinism check +/// summarizes groups in this order, so its recursive walk is always a cache hit at a +/// `` and never recurses through a reference chain. A chain is FLAT in the +/// schema document (group definitions are siblings, not nested), so it is not bounded +/// by the parser's `maxDepth`; ordering and scanning here are iterative so neither +/// step recurses on the chain length either. +enum GroupDependencyOrder { + /// Reachable group names in dependency order (dependencies first), by an iterative + /// post-order DFS from the groups the `models` reference. A name already on the + /// current path is not re-entered, so a circular reference (illegal, rejected + /// elsewhere) terminates. + static func reachable(from models: [PureXML.Model.TreeNode], context: PureXML.Schema.XSDContext) -> [String] { + var roots: [String] = [] + var seen: Set = [] + for model in models { + for ref in referencedGroupNames(in: model) where seen.insert(ref).inserted { + roots.append(ref) + } + } + var order: [String] = [] + var visited: Set = [] + var onPath: Set = [] + var work: [(name: String, exit: Bool)] = roots.reversed().map { ($0, false) } + while let (name, exit) = work.popLast() { + if exit { + onPath.remove(name) + order.append(name) + continue + } + if visited.contains(name) { continue } + visited.insert(name) + onPath.insert(name) + work.append((name, true)) + for ref in dependencies(of: name, context: context).reversed() where !visited.contains(ref) && !onPath.contains(ref) { + work.append((ref, false)) + } + } + return order + } + + /// The group names referenced within `node`, by an iterative tree walk (so scanning + /// a deep model does not itself recurse on depth). It descends compositors but stops + /// at an ``, exactly as the determinism walk's `summarize` treats an element + /// as terminal: a group inside an element's inline complex type belongs to THAT type's + /// own content model (checked separately), not this one, and must not be drawn in. + private static func referencedGroupNames(in node: PureXML.Model.TreeNode) -> [String] { + var names: [String] = [] + var stack = [node] + while let current = stack.popLast() { + let local = PureXML.Schema.XSDNode.localName(current) + if local == "group", let ref = PureXML.Schema.XSDNode.attribute(current, "ref") { + names.append(PureXML.Schema.XSDNode.stripPrefix(ref)) + } + if local != "element" { + stack.append(contentsOf: PureXML.Schema.XSDNode.elementChildren(current)) + } + } + return names + } + + /// The groups `name` references. A redefined group's base-model references are added + /// ONLY when the redefinition actually self-references (`` inside the + /// redefinition of `name`): that reference expands to the base model + /// (`CompWalk.redefineSelfReference`), making the base's groups part of the effective + /// content model, so they must be ordered first. When the redefinition does NOT + /// self-reference, the base is fully replaced and dead, and pulling its refs in would + /// summarise unreachable groups: surfacing a conflict in a dead group is a false + /// positive, so it is deliberately excluded. This keeps the reachable set identical + /// to the lazy walk's. + private static func dependencies(of name: String, context: PureXML.Schema.XSDContext) -> [String] { + guard let definition = context.groups[name], let model = modelChild(definition) else { return [] } + var refs = referencedGroupNames(in: model) + if refs.contains(name), context.redefinedGroups.contains(name), let base = context.baseGroups[name], let baseModel = modelChild(base) { + refs += referencedGroupNames(in: baseModel) + } + return refs + } + + /// The `sequence`/`choice`/`all` model child of a group definition, if any. + private static func modelChild(_ definition: PureXML.Model.TreeNode) -> PureXML.Model.TreeNode? { + PureXML.Schema.XSDNode.elementChildren(definition) + .first { ["sequence", "choice", "all"].contains(PureXML.Schema.XSDNode.localName($0) ?? "") } + } +} diff --git a/Tests/SchemaGroupChainDepthSafetyTests.swift b/Tests/SchemaGroupChainDepthSafetyTests.swift new file mode 100644 index 0000000..959ad0b --- /dev/null +++ b/Tests/SchemaGroupChainDepthSafetyTests.swift @@ -0,0 +1,63 @@ +#if !os(WASI) + import Foundation + import Testing + @testable import PureXML + + /// A `` reference chain (`g0 = (e0, g1)`, `g1 = (e1, g2)`, ...) is FLAT in + /// the schema document: the group definitions are siblings, not nested, so the chain + /// is NOT bounded by the parser's `maxDepth`, and compiling it recurses once per + /// link. The Unique Particle Attribution determinism check summarised groups lazily + /// by recursing through each reference, so a long chain overflowed the stack while + /// compiling a legal (if hostile) schema. It now summarises every reachable group + /// once in dependency order, so the walk's group lookup is a cache hit and recursion + /// is bounded by a single group's own model nesting. + /// + /// Runs on an 8 MB stack, the native main-thread budget a real caller has: swift + /// testing executes a test on a much smaller Task stack, where the separately + /// tracked particle-compiler group expansion (which still recurses on the chain) + /// would overflow first and mask the determinism fix. WASI cannot size a thread + /// stack, so the case is gated off there; its budget is a link-time setting. + @Suite("Schema group-chain depth safety") + struct SchemaGroupChainDepthSafetyTests { + @Test("a deep group-reference chain compiles without overflowing the stack") + func test_deepGroupChainCompiles() { + let depth = 1500 + var groups = "" + for index in 0 ..< depth { + let next = index < depth - 1 ? "" : "" + groups += "" + + "\(next)" + } + let schema = "\(groups)" + + "" + + "" + + // The semaphore makes the worker's write happen-before the read, so the box's + // unchecked-Sendable mutation is safe (the closure must be Sendable for + // Foundation's Thread on Linux). + let result = ResultBox() + runOnNativeStack { result.compiled = (try? PureXML.Schema.Document(schema)) != nil } + + // The chain is deterministic, so it must compile (not be rejected) and, above + // all, must not crash: before the dependency-order precompute the determinism + // check overflowed the stack on this shape well before this depth. + #expect(result.compiled) + } + + private final class ResultBox: @unchecked Sendable { + var compiled = false + } + + /// Runs `work` on a thread with the native main-thread stack budget and waits. + private func runOnNativeStack(_ work: @escaping @Sendable () -> Void) { + let done = DispatchSemaphore(value: 0) + let thread = Thread { + work() + done.signal() + } + thread.stackSize = 8 << 20 + thread.start() + done.wait() + } + } +#endif