perf(schema): bound content-model expansion depth so deep chains cannot overflow the stack (#372)
Two constructs are flat in the schema document, so they escape the parser's element-nesting maxDepth (256) yet make a content model arbitrarily deep once compiled: a <xs:group ref> chain and a complexContent derivation chain (the definitions are siblings). The particle compiler inlines each group reference and composes each derivation once per link, overflowing the stack on a long chain. The expanded depth is now bounded by one counter (XSDContext.expansionDepth, incremented at each compositor level, group-reference expansion, and derivation step) against maximumContentModelExpansionDepth (512); a model past the bound is rejected with a located error, not crashed. The counter lives in a value-type context so it tracks depth not width (no width false positive), and the rejection is harvested into the schema's errors (no silent truncation). Verified conformance-neutral over the full W3C XSTS corpus (all four buckets unchanged), the libxml2 determinism differential unchanged, and the full suite; an adversarial critic found the derivation-chain gap, which this fixes with the same counter.
This commit is contained in:
GitHub
parent
1f8d99adfc
commit
3e4ca0cce3
@ -25,7 +25,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Fixed
|
||||
|
||||
- A deeply nested chain of `<xs:group>` references (`g0` referencing `g1` referencing `g2`, and so on) no longer overflows the stack in the Unique Particle Attribution determinism check. Such a chain is flat in the schema document, the group definitions being siblings rather than nested, so it is not bounded by the parser's `maxDepth`; the compositional check summarised each group lazily by recursing through the reference at its use site, so a long chain recursed once per link and overflowed the stack while compiling a legal (if hostile) schema, a keystroke-time interactive-safety crash on the primary platform. Group summaries form an acyclic graph (a circular group reference is rejected elsewhere), so they are now precomputed bottom-up in dependency order: every reachable group is summarised once, in an order where each group's own references are already cached, so the walk's group lookup is always a cache hit and recursion is bounded by a single group's model nesting (which `maxDepth` bounds), not by the chain length. A group summary is context-independent, so this only reorders the work; the set of decision sets it produces is identical and the accept/reject verdict does not depend on their order. Verified verdict-equivalent over the full W3C XSTS corpus in release (all four buckets unchanged: `valid-schemas-rejected` and `valid-instances-rejected` stay 0, `invalid-schemas-accepted` 8, `invalid-instances-accepted` 14), the generated-schema determinism differential against libxml2 unchanged (zero false positives, the same three named false negatives), and the full 1816-test suite. Disclosed residual, not silent: the separate particle-compiler pass that inlines a group reference into the instance content model still recurses on the same chain and overflows on a far longer one (around ten thousand links); closing it needs a content-model nesting-depth limit, analogous to `maxDepth`, that rejects such a pathological chain with a located error, and is tracked.
|
||||
- The schema compiler no longer overflows the stack on a content model whose expanded depth is unbounded. Two constructs are flat in the schema document, so they escape the parser's element-nesting `maxDepth` (256) yet make a content model arbitrarily deep once compiled: a `<xs:group ref>` chain (the group definitions are siblings, not nested) and a `complexContent` derivation chain (`B0` extends `B1` extends `B2`, and so on, the type definitions also siblings). The particle compiler inlines each group reference and composes each derivation once per link, so a long chain recursed once per link until the stack overflowed while compiling a legal, if hostile, schema, a keystroke-time interactive-safety crash. The expanded depth is now bounded by a single counter (`XSDContext.expansionDepth`, incremented at each compositor level, group-reference expansion, and derivation step) against `maximumContentModelExpansionDepth` (512, twice the element-nesting limit: room for a maximally nested model plus as much again for expansion). A model past the bound is rejected with a located error (`content model nesting ... exceeds the maximum depth of 512`, or the derivation form), not crashed: a bound, not a band-aid. The bound is on total expanded depth, so it covers both a long chain of shallow links and a short chain of internally deep ones. Together with the determinism-check fix below, this closes the group-reference-chain crash on every compiler pass that recursed on the chain. Verified conformance-neutral over the full W3C XSTS corpus in release (all four buckets unchanged; no valid schema newly rejected, the corpus's content models nesting far below the bound), the libxml2 determinism differential unchanged, and the full suite. An adversarial over-rejection critic confirmed the counter lives in a value-type context whose copies make it track depth, not width, so a wide-but-shallow model (hundreds of sibling particles) is never falsely rejected, and that the rejection is harvested into the schema's errors rather than silently truncating the model.
|
||||
|
||||
- A deeply nested chain of `<xs:group>` references (`g0` referencing `g1` referencing `g2`, and so on) no longer overflows the stack in the Unique Particle Attribution determinism check. Such a chain is flat in the schema document, the group definitions being siblings rather than nested, so it is not bounded by the parser's `maxDepth`; the compositional check summarised each group lazily by recursing through the reference at its use site, so a long chain recursed once per link and overflowed the stack while compiling a legal (if hostile) schema, a keystroke-time interactive-safety crash on the primary platform. Group summaries form an acyclic graph (a circular group reference is rejected elsewhere), so they are now precomputed bottom-up in dependency order: every reachable group is summarised once, in an order where each group's own references are already cached, so the walk's group lookup is always a cache hit and recursion is bounded by a single group's model nesting (which `maxDepth` bounds), not by the chain length. A group summary is context-independent, so this only reorders the work; the set of decision sets it produces is identical and the accept/reject verdict does not depend on their order. Verified verdict-equivalent over the full W3C XSTS corpus in release (all four buckets unchanged: `valid-schemas-rejected` and `valid-instances-rejected` stay 0, `invalid-schemas-accepted` 8, `invalid-instances-accepted` 14), the generated-schema determinism differential against libxml2 unchanged (zero false positives, the same three named false negatives), and the full 1816-test suite. The particle-compiler pass that inlines a group reference, which this left recursing on the same chain, is now bounded as well (see the content-model expansion-depth entry above).
|
||||
|
||||
- A repeating model whose optional trailing particle and leading particle share a name, `(c, b?, c?)*`, is now correctly rejected as a Unique Particle Attribution violation, as libxml2/Xerces reject it: after `b` the next `c` could be the trailing `c?` or the looped leading `c`, two distinct particles. The compositional determinism check formed the internal sequence gap before the enclosing repetition's loop edge was added, so the combined `{trailing, leading}` followpos was never checked (an invalid-schemas-accepted false negative the finite XSTS corpus did not contain). The sequence now carries each trailing end-item's internal suffix in its `lastFollow`, so a parent gap or the loop edge forms the end-item's full followpos. The carried suffix is always a subset of the true followpos, so the check only catches more genuine conflicts and never over-rejects: the non-repeated `(c, b?, c?)` and disjoint shapes like `(a, b?)*` stay valid. Found by the new generated-schema determinism differential against libxml2; verified over the full W3C XSTS corpus in release (`valid-schemas-rejected`/`valid-instances-rejected` stay 0, `invalid-*-accepted` unchanged) with an adversarial over-rejection critic finding no valid schema newly rejected over 47 shapes. The related ambiguity through two references to one group in a repeating sequence stays an under-rejection (the cached group identity cannot distinguish a looped from a trailing occurrence without per-occurrence tracking), named in `DeterminismDifferentialTests`.
|
||||
|
||||
|
||||
@ -158,6 +158,21 @@ extension PureXML.Schema.XSDParser {
|
||||
return nil
|
||||
}
|
||||
guard let baseNode = baseTypeNode(named: baseName, context) else { return nil }
|
||||
// A `complexContent` derivation chain (`B0` extends `B1` extends ... `B_N`) is, like
|
||||
// a group-reference chain, flat in the document and so not bounded by the parser's
|
||||
// `maxDepth`; composing it recurses once per link. Count each link on the shared
|
||||
// expansion-depth budget (which group/compositor nesting also uses, so a type whose
|
||||
// base chain and content model are both deep is bounded by their sum) and reject a
|
||||
// chain past the bound with a located error rather than overflowing the stack.
|
||||
var context = context
|
||||
context.expansionDepth += 1
|
||||
guard context.expansionDepth <= PureXML.Schema.XSDParser.maximumContentModelExpansionDepth else {
|
||||
context.diagnostics.report(
|
||||
"type derivation nesting exceeds the maximum content model depth of \(PureXML.Schema.XSDParser.maximumContentModelExpansionDepth)",
|
||||
at: derivationNode,
|
||||
)
|
||||
return nil
|
||||
}
|
||||
let scoped = context.scoped(for: XSDContentNode.schemaOwner(baseNode))
|
||||
return complexType(baseNode, scoped.visitingType(baseName))
|
||||
}
|
||||
|
||||
@ -340,6 +340,9 @@ extension PureXML.Schema {
|
||||
/// reference to an abstract head expands to its members but not the head.
|
||||
var abstractElements: Set<String> = []
|
||||
var visitingGroups: Set<String> = []
|
||||
/// Particle-compiler expansion depth (each compositor level and `<group ref>`
|
||||
/// expansion adds one); `groupParticle` bounds it (`maximumContentModelExpansionDepth`).
|
||||
var expansionDepth = 0
|
||||
/// Named complex types being resolved up the current `complexContent`
|
||||
/// derivation chain, a guard against a cyclic base reference.
|
||||
var visitingTypes: Set<String> = []
|
||||
|
||||
@ -20,16 +20,35 @@ extension PureXML.Schema.XSDParser {
|
||||
return nil
|
||||
}
|
||||
|
||||
/// The deepest a content model may nest once every `<group ref>` is expanded.
|
||||
/// Twice the parser's default element-nesting `maxDepth` (256): a maximally nested
|
||||
/// content model, plus as much again for group-reference expansion. A `<group ref>`
|
||||
/// chain is flat in the document (the definitions are siblings), so it is NOT bounded
|
||||
/// by `maxDepth`, and expanding a long one recursed once per link until it overflowed
|
||||
/// the stack. A model deeper than this is rejected (a located error, below), not
|
||||
/// compiled: a bound, not a band-aid. No real schema approaches it (the W3C XSTS
|
||||
/// corpus stays far below); it only stops hostile or generated input.
|
||||
static let maximumContentModelExpansionDepth = 512
|
||||
|
||||
private static func groupParticle(
|
||||
_ node: PureXML.Model.TreeNode,
|
||||
_ compositor: Compositor,
|
||||
_ context: XSDContext,
|
||||
) -> Particle {
|
||||
let occurrence = XSDNode.occurrenceRange(node)
|
||||
var context = context
|
||||
context.expansionDepth += 1
|
||||
guard context.expansionDepth <= maximumContentModelExpansionDepth else {
|
||||
context.diagnostics.report(
|
||||
"content model nesting, including expanded group references, exceeds the maximum depth of \(maximumContentModelExpansionDepth)",
|
||||
at: node,
|
||||
)
|
||||
return Particle(occurrenceRange: occurrence, term: .group(Group(compositor: compositor, particles: [])))
|
||||
}
|
||||
var particles: [Particle] = []
|
||||
for child in XSDNode.elementChildren(node) {
|
||||
if let member = particle(child, context) { particles.append(member) }
|
||||
}
|
||||
let occurrence = XSDNode.occurrenceRange(node)
|
||||
// An `all` group's own `minOccurs="0"` makes the GROUP optional (it occurs
|
||||
// zero or one times), not its members optional: when the group is present
|
||||
// (any child appears) every member keeps its own `minOccurs`. The members
|
||||
|
||||
123
Tests/SchemaContentModelDepthSafetyTests.swift
Normal file
123
Tests/SchemaContentModelDepthSafetyTests.swift
Normal file
@ -0,0 +1,123 @@
|
||||
#if !os(WASI)
|
||||
import Foundation
|
||||
import Testing
|
||||
@testable import PureXML
|
||||
|
||||
/// Two flat-in-the-document constructs escape the parser's element-nesting `maxDepth`
|
||||
/// and so let a content model's EXPANDED depth grow without bound: a `<xs:group ref>`
|
||||
/// chain (`g0 = (e0, g1)`, `g1 = (e1, g2)`, ...) and a `complexContent` derivation chain
|
||||
/// (`B0` extends `B1` extends ...). The particle compiler inlines a group reference and
|
||||
/// composes a derivation once per link, so a long chain recursed until the stack
|
||||
/// overflowed while compiling a legal (if hostile) schema. Both are now bounded by one
|
||||
/// `maximumContentModelExpansionDepth` (512): within it compiles, past it is rejected
|
||||
/// with a LOCATED error, neither crashes.
|
||||
///
|
||||
/// Runs on an 8 MB stack, the native main-thread budget: reaching the bound is hundreds
|
||||
/// of levels deep (a couple of MB), more than swift-testing's small Task stack, and WASI
|
||||
/// cannot size a thread stack (its budget is a link-time setting), so the cases are gated
|
||||
/// off there.
|
||||
@Suite("Schema content-model expansion depth safety")
|
||||
struct SchemaContentModelDepthSafetyTests {
|
||||
@Test("a group-reference chain within the depth bound compiles")
|
||||
func test_groupChainWithinBoundCompiles() {
|
||||
#expect(compile(groupChain(depth: 400)).compiled)
|
||||
}
|
||||
|
||||
@Test("a group-reference chain past the bound is rejected with a located error, not a crash")
|
||||
func test_groupChainOverBoundRejectsLocated() {
|
||||
// Past the bound and past the depth at which the unbounded compiler overflowed.
|
||||
assertRejectedForDepth(compile(groupChain(depth: 2500)))
|
||||
}
|
||||
|
||||
@Test("a complexContent derivation chain within the depth bound compiles")
|
||||
func test_derivationChainWithinBoundCompiles() {
|
||||
#expect(compile(derivationChain(depth: 10)).compiled)
|
||||
}
|
||||
|
||||
@Test("a complexContent derivation chain past the bound is rejected with a located error, not a crash")
|
||||
func test_derivationChainOverBoundRejectsLocated() {
|
||||
assertRejectedForDepth(compile(derivationChain(depth: 600)))
|
||||
}
|
||||
|
||||
// MARK: Helpers
|
||||
|
||||
private func assertRejectedForDepth(_ outcome: Outcome) {
|
||||
#expect(!outcome.compiled)
|
||||
let depthFinding = outcome.findings.first { $0.reason.contains("maximum") && $0.reason.contains("depth") }
|
||||
#expect(depthFinding != nil) // rejected for depth, not some unrelated reason
|
||||
#expect(!(depthFinding?.codingPath.isEmpty ?? true)) // located: an IDE can underline it
|
||||
}
|
||||
|
||||
private func groupChain(depth: Int) -> String {
|
||||
var parts: [String] = []
|
||||
for index in 0 ..< depth {
|
||||
let next = index < depth - 1 ? "<xs:group ref=\"g\(index + 1)\"/>" : ""
|
||||
parts.append("<xs:group name=\"g\(index)\"><xs:sequence>"
|
||||
+ "<xs:element name=\"e\(index)\" type=\"xs:string\"/>\(next)</xs:sequence></xs:group>")
|
||||
}
|
||||
return "<xs:schema xmlns:xs=\"http://www.w3.org/2001/XMLSchema\">\(parts.joined())"
|
||||
+ "<xs:element name=\"doc\"><xs:complexType><xs:sequence>"
|
||||
+ "<xs:group ref=\"g0\"/></xs:sequence></xs:complexType></xs:element></xs:schema>"
|
||||
}
|
||||
|
||||
private func derivationChain(depth: Int) -> String {
|
||||
var parts: [String] = []
|
||||
for index in 0 ..< depth {
|
||||
if index < depth - 1 {
|
||||
parts.append("<xs:complexType name=\"B\(index)\"><xs:complexContent>"
|
||||
+ "<xs:extension base=\"B\(index + 1)\"><xs:sequence>"
|
||||
+ "<xs:element name=\"e\(index)\" type=\"xs:string\"/></xs:sequence>"
|
||||
+ "</xs:extension></xs:complexContent></xs:complexType>")
|
||||
} else {
|
||||
parts.append("<xs:complexType name=\"B\(index)\"><xs:sequence>"
|
||||
+ "<xs:element name=\"leaf\" type=\"xs:string\"/></xs:sequence></xs:complexType>")
|
||||
}
|
||||
}
|
||||
return "<xs:schema xmlns:xs=\"http://www.w3.org/2001/XMLSchema\">\(parts.joined())"
|
||||
+ "<xs:element name=\"doc\" type=\"B0\"/></xs:schema>"
|
||||
}
|
||||
|
||||
private struct Outcome {
|
||||
var compiled: Bool
|
||||
var findings: [PureXML.Validation.ValidationError]
|
||||
}
|
||||
|
||||
/// Compiles `schema` on a native-sized stack and reports whether it compiled and,
|
||||
/// if not, the located findings.
|
||||
private func compile(_ schema: String) -> Outcome {
|
||||
let result = ResultBox()
|
||||
runOnNativeStack {
|
||||
do {
|
||||
_ = try PureXML.Schema.Document(schema)
|
||||
result.outcome = Outcome(compiled: true, findings: [])
|
||||
} catch let error as PureXML.Schema.SchemaError {
|
||||
if case let .inconsistent(findings) = error {
|
||||
result.outcome = Outcome(compiled: false, findings: findings)
|
||||
} else {
|
||||
result.outcome = Outcome(compiled: false, findings: [])
|
||||
}
|
||||
} catch {
|
||||
result.outcome = Outcome(compiled: false, findings: [])
|
||||
}
|
||||
}
|
||||
return result.outcome
|
||||
}
|
||||
|
||||
private final class ResultBox: @unchecked Sendable {
|
||||
var outcome = Outcome(compiled: false, findings: [])
|
||||
}
|
||||
|
||||
/// Runs `work` on a thread with the native main-thread stack budget and waits. The
|
||||
/// semaphore orders the worker's write before the read, so the box is safe.
|
||||
private func runOnNativeStack(_ work: @escaping @Sendable () -> Void) {
|
||||
let done = DispatchSemaphore(value: 0)
|
||||
let thread = Thread {
|
||||
work()
|
||||
done.signal()
|
||||
}
|
||||
thread.stackSize = 8 << 20
|
||||
thread.start()
|
||||
done.wait()
|
||||
}
|
||||
}
|
||||
#endif
|
||||
@ -1,63 +0,0 @@
|
||||
#if !os(WASI)
|
||||
import Foundation
|
||||
import Testing
|
||||
@testable import PureXML
|
||||
|
||||
/// A `<xs:group>` reference chain (`g0 = (e0, g1)`, `g1 = (e1, g2)`, ...) is FLAT in
|
||||
/// the schema document: the group definitions are siblings, not nested, so the chain
|
||||
/// is NOT bounded by the parser's `maxDepth`, and compiling it recurses once per
|
||||
/// link. The Unique Particle Attribution determinism check summarised groups lazily
|
||||
/// by recursing through each reference, so a long chain overflowed the stack while
|
||||
/// compiling a legal (if hostile) schema. It now summarises every reachable group
|
||||
/// once in dependency order, so the walk's group lookup is a cache hit and recursion
|
||||
/// is bounded by a single group's own model nesting.
|
||||
///
|
||||
/// Runs on an 8 MB stack, the native main-thread budget a real caller has: swift
|
||||
/// testing executes a test on a much smaller Task stack, where the separately
|
||||
/// tracked particle-compiler group expansion (which still recurses on the chain)
|
||||
/// would overflow first and mask the determinism fix. WASI cannot size a thread
|
||||
/// stack, so the case is gated off there; its budget is a link-time setting.
|
||||
@Suite("Schema group-chain depth safety")
|
||||
struct SchemaGroupChainDepthSafetyTests {
|
||||
@Test("a deep group-reference chain compiles without overflowing the stack")
|
||||
func test_deepGroupChainCompiles() {
|
||||
let depth = 1500
|
||||
var groups = ""
|
||||
for index in 0 ..< depth {
|
||||
let next = index < depth - 1 ? "<xs:group ref=\"g\(index + 1)\"/>" : ""
|
||||
groups += "<xs:group name=\"g\(index)\"><xs:sequence>"
|
||||
+ "<xs:element name=\"e\(index)\" type=\"xs:string\"/>\(next)</xs:sequence></xs:group>"
|
||||
}
|
||||
let schema = "<xs:schema xmlns:xs=\"http://www.w3.org/2001/XMLSchema\">\(groups)"
|
||||
+ "<xs:element name=\"doc\"><xs:complexType><xs:sequence>"
|
||||
+ "<xs:group ref=\"g0\"/></xs:sequence></xs:complexType></xs:element></xs:schema>"
|
||||
|
||||
// The semaphore makes the worker's write happen-before the read, so the box's
|
||||
// unchecked-Sendable mutation is safe (the closure must be Sendable for
|
||||
// Foundation's Thread on Linux).
|
||||
let result = ResultBox()
|
||||
runOnNativeStack { result.compiled = (try? PureXML.Schema.Document(schema)) != nil }
|
||||
|
||||
// The chain is deterministic, so it must compile (not be rejected) and, above
|
||||
// all, must not crash: before the dependency-order precompute the determinism
|
||||
// check overflowed the stack on this shape well before this depth.
|
||||
#expect(result.compiled)
|
||||
}
|
||||
|
||||
private final class ResultBox: @unchecked Sendable {
|
||||
var compiled = false
|
||||
}
|
||||
|
||||
/// Runs `work` on a thread with the native main-thread stack budget and waits.
|
||||
private func runOnNativeStack(_ work: @escaping @Sendable () -> Void) {
|
||||
let done = DispatchSemaphore(value: 0)
|
||||
let thread = Thread {
|
||||
work()
|
||||
done.signal()
|
||||
}
|
||||
thread.stackSize = 8 << 20
|
||||
thread.start()
|
||||
done.wait()
|
||||
}
|
||||
}
|
||||
#endif
|
||||
Loading…
Reference in New Issue
Block a user