diff --git a/CHANGELOG.md b/CHANGELOG.md index fa2b381..c37c7d8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,7 +25,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed -- A deeply nested chain of `` references (`g0` referencing `g1` referencing `g2`, and so on) no longer overflows the stack in the Unique Particle Attribution determinism check. Such a chain is flat in the schema document, the group definitions being siblings rather than nested, so it is not bounded by the parser's `maxDepth`; the compositional check summarised each group lazily by recursing through the reference at its use site, so a long chain recursed once per link and overflowed the stack while compiling a legal (if hostile) schema, a keystroke-time interactive-safety crash on the primary platform. Group summaries form an acyclic graph (a circular group reference is rejected elsewhere), so they are now precomputed bottom-up in dependency order: every reachable group is summarised once, in an order where each group's own references are already cached, so the walk's group lookup is always a cache hit and recursion is bounded by a single group's model nesting (which `maxDepth` bounds), not by the chain length. A group summary is context-independent, so this only reorders the work; the set of decision sets it produces is identical and the accept/reject verdict does not depend on their order. Verified verdict-equivalent over the full W3C XSTS corpus in release (all four buckets unchanged: `valid-schemas-rejected` and `valid-instances-rejected` stay 0, `invalid-schemas-accepted` 8, `invalid-instances-accepted` 14), the generated-schema determinism differential against libxml2 unchanged (zero false positives, the same three named false negatives), and the full 1816-test suite. Disclosed residual, not silent: the separate particle-compiler pass that inlines a group reference into the instance content model still recurses on the same chain and overflows on a far longer one (around ten thousand links); closing it needs a content-model nesting-depth limit, analogous to `maxDepth`, that rejects such a pathological chain with a located error, and is tracked. +- The schema compiler no longer overflows the stack on a content model whose expanded depth is unbounded. Two constructs are flat in the schema document, so they escape the parser's element-nesting `maxDepth` (256) yet make a content model arbitrarily deep once compiled: a `` chain (the group definitions are siblings, not nested) and a `complexContent` derivation chain (`B0` extends `B1` extends `B2`, and so on, the type definitions also siblings). The particle compiler inlines each group reference and composes each derivation once per link, so a long chain recursed once per link until the stack overflowed while compiling a legal, if hostile, schema, a keystroke-time interactive-safety crash. The expanded depth is now bounded by a single counter (`XSDContext.expansionDepth`, incremented at each compositor level, group-reference expansion, and derivation step) against `maximumContentModelExpansionDepth` (512, twice the element-nesting limit: room for a maximally nested model plus as much again for expansion). A model past the bound is rejected with a located error (`content model nesting ... exceeds the maximum depth of 512`, or the derivation form), not crashed: a bound, not a band-aid. The bound is on total expanded depth, so it covers both a long chain of shallow links and a short chain of internally deep ones. Together with the determinism-check fix below, this closes the group-reference-chain crash on every compiler pass that recursed on the chain. Verified conformance-neutral over the full W3C XSTS corpus in release (all four buckets unchanged; no valid schema newly rejected, the corpus's content models nesting far below the bound), the libxml2 determinism differential unchanged, and the full suite. An adversarial over-rejection critic confirmed the counter lives in a value-type context whose copies make it track depth, not width, so a wide-but-shallow model (hundreds of sibling particles) is never falsely rejected, and that the rejection is harvested into the schema's errors rather than silently truncating the model. + +- A deeply nested chain of `` references (`g0` referencing `g1` referencing `g2`, and so on) no longer overflows the stack in the Unique Particle Attribution determinism check. Such a chain is flat in the schema document, the group definitions being siblings rather than nested, so it is not bounded by the parser's `maxDepth`; the compositional check summarised each group lazily by recursing through the reference at its use site, so a long chain recursed once per link and overflowed the stack while compiling a legal (if hostile) schema, a keystroke-time interactive-safety crash on the primary platform. Group summaries form an acyclic graph (a circular group reference is rejected elsewhere), so they are now precomputed bottom-up in dependency order: every reachable group is summarised once, in an order where each group's own references are already cached, so the walk's group lookup is always a cache hit and recursion is bounded by a single group's model nesting (which `maxDepth` bounds), not by the chain length. A group summary is context-independent, so this only reorders the work; the set of decision sets it produces is identical and the accept/reject verdict does not depend on their order. Verified verdict-equivalent over the full W3C XSTS corpus in release (all four buckets unchanged: `valid-schemas-rejected` and `valid-instances-rejected` stay 0, `invalid-schemas-accepted` 8, `invalid-instances-accepted` 14), the generated-schema determinism differential against libxml2 unchanged (zero false positives, the same three named false negatives), and the full 1816-test suite. The particle-compiler pass that inlines a group reference, which this left recursing on the same chain, is now bounded as well (see the content-model expansion-depth entry above). - A repeating model whose optional trailing particle and leading particle share a name, `(c, b?, c?)*`, is now correctly rejected as a Unique Particle Attribution violation, as libxml2/Xerces reject it: after `b` the next `c` could be the trailing `c?` or the looped leading `c`, two distinct particles. The compositional determinism check formed the internal sequence gap before the enclosing repetition's loop edge was added, so the combined `{trailing, leading}` followpos was never checked (an invalid-schemas-accepted false negative the finite XSTS corpus did not contain). The sequence now carries each trailing end-item's internal suffix in its `lastFollow`, so a parent gap or the loop edge forms the end-item's full followpos. The carried suffix is always a subset of the true followpos, so the check only catches more genuine conflicts and never over-rejects: the non-repeated `(c, b?, c?)` and disjoint shapes like `(a, b?)*` stay valid. Found by the new generated-schema determinism differential against libxml2; verified over the full W3C XSTS corpus in release (`valid-schemas-rejected`/`valid-instances-rejected` stay 0, `invalid-*-accepted` unchanged) with an adversarial over-rejection critic finding no valid schema newly rejected over 47 shapes. The related ambiguity through two references to one group in a repeating sequence stays an under-rejection (the cached group identity cannot distinguish a looped from a trailing occurrence without per-occurrence tracking), named in `DeterminismDifferentialTests`. diff --git a/Sources/Schema/XSDComplexContent.swift b/Sources/Schema/XSDComplexContent.swift index fb7acb9..6aae0d3 100644 --- a/Sources/Schema/XSDComplexContent.swift +++ b/Sources/Schema/XSDComplexContent.swift @@ -158,6 +158,21 @@ extension PureXML.Schema.XSDParser { return nil } guard let baseNode = baseTypeNode(named: baseName, context) else { return nil } + // A `complexContent` derivation chain (`B0` extends `B1` extends ... `B_N`) is, like + // a group-reference chain, flat in the document and so not bounded by the parser's + // `maxDepth`; composing it recurses once per link. Count each link on the shared + // expansion-depth budget (which group/compositor nesting also uses, so a type whose + // base chain and content model are both deep is bounded by their sum) and reject a + // chain past the bound with a located error rather than overflowing the stack. + var context = context + context.expansionDepth += 1 + guard context.expansionDepth <= PureXML.Schema.XSDParser.maximumContentModelExpansionDepth else { + context.diagnostics.report( + "type derivation nesting exceeds the maximum content model depth of \(PureXML.Schema.XSDParser.maximumContentModelExpansionDepth)", + at: derivationNode, + ) + return nil + } let scoped = context.scoped(for: XSDContentNode.schemaOwner(baseNode)) return complexType(baseNode, scoped.visitingType(baseName)) } diff --git a/Sources/Schema/XSDContext.swift b/Sources/Schema/XSDContext.swift index 0e47509..0a4b1db 100644 --- a/Sources/Schema/XSDContext.swift +++ b/Sources/Schema/XSDContext.swift @@ -340,6 +340,9 @@ extension PureXML.Schema { /// reference to an abstract head expands to its members but not the head. var abstractElements: Set = [] var visitingGroups: Set = [] + /// Particle-compiler expansion depth (each compositor level and `` + /// expansion adds one); `groupParticle` bounds it (`maximumContentModelExpansionDepth`). + var expansionDepth = 0 /// Named complex types being resolved up the current `complexContent` /// derivation chain, a guard against a cyclic base reference. var visitingTypes: Set = [] diff --git a/Sources/Schema/XSDParserParticles.swift b/Sources/Schema/XSDParserParticles.swift index 84e3145..b086559 100644 --- a/Sources/Schema/XSDParserParticles.swift +++ b/Sources/Schema/XSDParserParticles.swift @@ -20,16 +20,35 @@ extension PureXML.Schema.XSDParser { return nil } + /// The deepest a content model may nest once every `` is expanded. + /// Twice the parser's default element-nesting `maxDepth` (256): a maximally nested + /// content model, plus as much again for group-reference expansion. A `` + /// chain is flat in the document (the definitions are siblings), so it is NOT bounded + /// by `maxDepth`, and expanding a long one recursed once per link until it overflowed + /// the stack. A model deeper than this is rejected (a located error, below), not + /// compiled: a bound, not a band-aid. No real schema approaches it (the W3C XSTS + /// corpus stays far below); it only stops hostile or generated input. + static let maximumContentModelExpansionDepth = 512 + private static func groupParticle( _ node: PureXML.Model.TreeNode, _ compositor: Compositor, _ context: XSDContext, ) -> Particle { + let occurrence = XSDNode.occurrenceRange(node) + var context = context + context.expansionDepth += 1 + guard context.expansionDepth <= maximumContentModelExpansionDepth else { + context.diagnostics.report( + "content model nesting, including expanded group references, exceeds the maximum depth of \(maximumContentModelExpansionDepth)", + at: node, + ) + return Particle(occurrenceRange: occurrence, term: .group(Group(compositor: compositor, particles: []))) + } var particles: [Particle] = [] for child in XSDNode.elementChildren(node) { if let member = particle(child, context) { particles.append(member) } } - let occurrence = XSDNode.occurrenceRange(node) // An `all` group's own `minOccurs="0"` makes the GROUP optional (it occurs // zero or one times), not its members optional: when the group is present // (any child appears) every member keeps its own `minOccurs`. The members diff --git a/Tests/SchemaContentModelDepthSafetyTests.swift b/Tests/SchemaContentModelDepthSafetyTests.swift new file mode 100644 index 0000000..6f19e90 --- /dev/null +++ b/Tests/SchemaContentModelDepthSafetyTests.swift @@ -0,0 +1,123 @@ +#if !os(WASI) + import Foundation + import Testing + @testable import PureXML + + /// Two flat-in-the-document constructs escape the parser's element-nesting `maxDepth` + /// and so let a content model's EXPANDED depth grow without bound: a `` + /// chain (`g0 = (e0, g1)`, `g1 = (e1, g2)`, ...) and a `complexContent` derivation chain + /// (`B0` extends `B1` extends ...). The particle compiler inlines a group reference and + /// composes a derivation once per link, so a long chain recursed until the stack + /// overflowed while compiling a legal (if hostile) schema. Both are now bounded by one + /// `maximumContentModelExpansionDepth` (512): within it compiles, past it is rejected + /// with a LOCATED error, neither crashes. + /// + /// Runs on an 8 MB stack, the native main-thread budget: reaching the bound is hundreds + /// of levels deep (a couple of MB), more than swift-testing's small Task stack, and WASI + /// cannot size a thread stack (its budget is a link-time setting), so the cases are gated + /// off there. + @Suite("Schema content-model expansion depth safety") + struct SchemaContentModelDepthSafetyTests { + @Test("a group-reference chain within the depth bound compiles") + func test_groupChainWithinBoundCompiles() { + #expect(compile(groupChain(depth: 400)).compiled) + } + + @Test("a group-reference chain past the bound is rejected with a located error, not a crash") + func test_groupChainOverBoundRejectsLocated() { + // Past the bound and past the depth at which the unbounded compiler overflowed. + assertRejectedForDepth(compile(groupChain(depth: 2500))) + } + + @Test("a complexContent derivation chain within the depth bound compiles") + func test_derivationChainWithinBoundCompiles() { + #expect(compile(derivationChain(depth: 10)).compiled) + } + + @Test("a complexContent derivation chain past the bound is rejected with a located error, not a crash") + func test_derivationChainOverBoundRejectsLocated() { + assertRejectedForDepth(compile(derivationChain(depth: 600))) + } + + // MARK: Helpers + + private func assertRejectedForDepth(_ outcome: Outcome) { + #expect(!outcome.compiled) + let depthFinding = outcome.findings.first { $0.reason.contains("maximum") && $0.reason.contains("depth") } + #expect(depthFinding != nil) // rejected for depth, not some unrelated reason + #expect(!(depthFinding?.codingPath.isEmpty ?? true)) // located: an IDE can underline it + } + + private func groupChain(depth: Int) -> String { + var parts: [String] = [] + for index in 0 ..< depth { + let next = index < depth - 1 ? "" : "" + parts.append("" + + "\(next)") + } + return "\(parts.joined())" + + "" + + "" + } + + private func derivationChain(depth: Int) -> String { + var parts: [String] = [] + for index in 0 ..< depth { + if index < depth - 1 { + parts.append("" + + "" + + "" + + "") + } else { + parts.append("" + + "") + } + } + return "\(parts.joined())" + + "" + } + + private struct Outcome { + var compiled: Bool + var findings: [PureXML.Validation.ValidationError] + } + + /// Compiles `schema` on a native-sized stack and reports whether it compiled and, + /// if not, the located findings. + private func compile(_ schema: String) -> Outcome { + let result = ResultBox() + runOnNativeStack { + do { + _ = try PureXML.Schema.Document(schema) + result.outcome = Outcome(compiled: true, findings: []) + } catch let error as PureXML.Schema.SchemaError { + if case let .inconsistent(findings) = error { + result.outcome = Outcome(compiled: false, findings: findings) + } else { + result.outcome = Outcome(compiled: false, findings: []) + } + } catch { + result.outcome = Outcome(compiled: false, findings: []) + } + } + return result.outcome + } + + private final class ResultBox: @unchecked Sendable { + var outcome = Outcome(compiled: false, findings: []) + } + + /// Runs `work` on a thread with the native main-thread stack budget and waits. The + /// semaphore orders the worker's write before the read, so the box is safe. + private func runOnNativeStack(_ work: @escaping @Sendable () -> Void) { + let done = DispatchSemaphore(value: 0) + let thread = Thread { + work() + done.signal() + } + thread.stackSize = 8 << 20 + thread.start() + done.wait() + } + } +#endif diff --git a/Tests/SchemaGroupChainDepthSafetyTests.swift b/Tests/SchemaGroupChainDepthSafetyTests.swift deleted file mode 100644 index 959ad0b..0000000 --- a/Tests/SchemaGroupChainDepthSafetyTests.swift +++ /dev/null @@ -1,63 +0,0 @@ -#if !os(WASI) - import Foundation - import Testing - @testable import PureXML - - /// A `` reference chain (`g0 = (e0, g1)`, `g1 = (e1, g2)`, ...) is FLAT in - /// the schema document: the group definitions are siblings, not nested, so the chain - /// is NOT bounded by the parser's `maxDepth`, and compiling it recurses once per - /// link. The Unique Particle Attribution determinism check summarised groups lazily - /// by recursing through each reference, so a long chain overflowed the stack while - /// compiling a legal (if hostile) schema. It now summarises every reachable group - /// once in dependency order, so the walk's group lookup is a cache hit and recursion - /// is bounded by a single group's own model nesting. - /// - /// Runs on an 8 MB stack, the native main-thread budget a real caller has: swift - /// testing executes a test on a much smaller Task stack, where the separately - /// tracked particle-compiler group expansion (which still recurses on the chain) - /// would overflow first and mask the determinism fix. WASI cannot size a thread - /// stack, so the case is gated off there; its budget is a link-time setting. - @Suite("Schema group-chain depth safety") - struct SchemaGroupChainDepthSafetyTests { - @Test("a deep group-reference chain compiles without overflowing the stack") - func test_deepGroupChainCompiles() { - let depth = 1500 - var groups = "" - for index in 0 ..< depth { - let next = index < depth - 1 ? "" : "" - groups += "" - + "\(next)" - } - let schema = "\(groups)" - + "" - + "" - - // The semaphore makes the worker's write happen-before the read, so the box's - // unchecked-Sendable mutation is safe (the closure must be Sendable for - // Foundation's Thread on Linux). - let result = ResultBox() - runOnNativeStack { result.compiled = (try? PureXML.Schema.Document(schema)) != nil } - - // The chain is deterministic, so it must compile (not be rejected) and, above - // all, must not crash: before the dependency-order precompute the determinism - // check overflowed the stack on this shape well before this depth. - #expect(result.compiled) - } - - private final class ResultBox: @unchecked Sendable { - var compiled = false - } - - /// Runs `work` on a thread with the native main-thread stack budget and waits. - private func runOnNativeStack(_ work: @escaping @Sendable () -> Void) { - let done = DispatchSemaphore(value: 0) - let thread = Thread { - work() - done.signal() - } - thread.stackSize = 8 << 20 - thread.start() - done.wait() - } - } -#endif