From 90ae16cba1a41ed7c50436d368b1035be0cad136 Mon Sep 17 00:00:00 2001 From: Mihaela Mihaljevic Date: Sun, 14 Jun 2026 09:35:33 +0200 Subject: [PATCH] feat(schema): enforce Unique Particle Attribution (cos-nonambig) Reject ambiguous content models at compile time via a Glushkov position automaton over a literal, QName-resolved view of the raw schema tree, built just for this rule and independent of the instance-validation content model. The literal view resolves a ref's QName through the schema's prefix bindings (not the blanket target namespace) and does not expand substitution groups, removing the two false-overlap sources a check over the compiled model produced. Determinism is per particle: a particle competing with a repetition of itself is never a violation, so a fixed count {n,n} unrolls into copies in sequence while a true range or unbounded repeat uses a self-loop, and only two distinct source particles with overlapping labels conflict. The overlap test is QName-only, an under-rejection that can never over-reject. A position cap bounds pathological nesting (skipping the check, never over-rejecting). The element-declarations-consistent test loses its incidentally-ambiguous (a?, a) case in favor of a deterministic (a, b, a). XSTS: invalid schemas accepted 884 to 864, valid schemas rejected held at 72, instance buckets unchanged. --- CHANGELOG.md | 1 + Sources/Schema/ContentModelDeterminism.swift | 330 +++++++++++++++++++ Sources/Schema/XSDParser.swift | 4 +- Tests/ContentModelDeterminismTests.swift | 114 +++++++ Tests/SchemaStructureTests.swift | 5 +- Tests/XSTSSuiteTests.swift | 2 +- docs/schema-validity-burndown.md | 2 +- 7 files changed, 453 insertions(+), 5 deletions(-) create mode 100644 Sources/Schema/ContentModelDeterminism.swift create mode 100644 Tests/ContentModelDeterminismTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 0bf1057..70ef85f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- Unique Particle Attribution (`cos-nonambig`, the content-model determinism constraint) is now enforced at compile time (#156, #157), the keystone of the remaining schema-validity tail. A content model must be deterministic so each element in an instance is attributable to a single particle without lookahead; an ambiguous model (`` of two same-name elements, an optional or repeating element followed by a distinct same-name element, two same-name members of an `all`, an element competing with a wildcard that admits it) was accepted. The check runs a Glushkov position automaton over a literal, QName-resolved view of the raw schema tree built specifically for this rule, independent of the instance-validation content model: a `ref` is one position carrying its prefix-resolved QName (not the blanket target namespace), and substitution groups are not expanded, which removes the two false-overlap sources (cross-namespace refs, expanded substitution members) that a check over the compiled model produced. Determinism is per particle, the W3C rule's defining property: a particle competing with a repetition of itself is never a violation, so a fixed count `{n,n}` unrolls into copies in sequence (`b{2,2}, b` stays the deterministic `b, b, b`) while a true range or unbounded repeat uses a self-loop, and only two distinct source particles with overlapping labels conflict. The overlap test is QName-only (not substitution-group-aware), an accepted under-rejection that can never over-reject. XSTS: invalid schemas accepted 884 to 864, no other bucket raised (valid schemas rejected held at 72). - A `keyref`'s `refer` must name a `key` or `unique` constraint (#147). A `refer` pointing at a non-existent (or non-key/unique) constraint was accepted; it is now resolved against the document's `key`/`unique` names, with a forward `refer` (textually before the key) handled by collecting all names first. The check is skipped when the document pulls in external definitions through `import`/`include`/`redefine`, which the default compile does not load. XSTS: invalid schemas accepted 891 to 884, no other bucket raised. - Component-name uniqueness is now enforced (#147). Within a schema, global type names (`simpleType` and `complexType` share one symbol space), global element names, global attribute names, named model-group names, and named attribute-group names must each be unique, and identity-constraint names must be unique across the document; a duplicate was accepted, with the later definition silently overwriting the earlier. Only the document's own globals are examined (an `xs:redefine`'s same-named type is not a clash), and the symbol spaces are kept separate (a type and an element may share a name). XSTS: invalid schemas accepted 909 to 891, no other bucket raised. - Element Declarations Consistent (`cos-element-consistent`) is now enforced (#147). Within one complex type's content model, two element declarations with the same name must have the same type; two same-name elements with different types (e.g. `a` as `xs:string` and as `xs:integer`) were accepted. The check walks the content model (through model groups and content-derivation wrappers) but not into an element's own nested type, so a reused name in a nested anonymous type, a separate content model, is not flagged. Modeled on Xerces (libxml2 only partially enforces this); substitution-group expansion is not folded in (an accepted under-rejection). XSTS: invalid schemas accepted 925 to 909, no other bucket raised. diff --git a/Sources/Schema/ContentModelDeterminism.swift b/Sources/Schema/ContentModelDeterminism.swift new file mode 100644 index 0000000..ea06021 --- /dev/null +++ b/Sources/Schema/ContentModelDeterminism.swift @@ -0,0 +1,330 @@ +public extension PureXML.Schema { + /// Unique Particle Attribution (cos-nonambig, XSD 1.0 3.8.6): a content model + /// must be deterministic, so each element in an instance is attributed to one + /// particle without lookahead. Checked on the Glushkov position automaton: the + /// model is ambiguous iff some decision set, the start `firstpos` or any + /// `followpos(p)`, holds two distinct positions whose labels can match the same + /// element. No DFA subset construction is needed. + /// + /// This works on a LITERAL view of the raw schema tree, distinct from the + /// compiled (instance) content model: a `ref` is one position carrying its + /// resolved QName (prefix bindings, not the blanket target namespace), and + /// substitution groups are NOT expanded. The overlap test is QName-only (not + /// substitution-group-aware), so it can only under-report substitution-group + /// ambiguities, never over-report. Those two choices remove the false overlaps + /// (cross-namespace refs; expanded substitution members) that a check over the + /// compiled model produced. + enum ContentModelDeterminism { + private static let modelGroupNames: Set = ["sequence", "choice", "all"] + private static let particleNames: Set = ["element", "sequence", "choice", "all", "group", "any"] + + /// Every UPA-violation message across the schema's complex-type content + /// models, deduplicated in first-seen order. + static func violations(in schema: XSDTree, context: XSDContext) -> [String] { + let bindings = namespaceBindings(schema) + var messages: [String] = [] + var seen: Set = [] + forEachComplexType(schema) { complexType in + guard let model = contentModelNode(complexType), + let conflict = violation(model, bindings, context) + else { return } + if seen.insert(conflict).inserted { messages.append(conflict) } + } + return messages + } + + /// The prefix-to-namespace bindings declared on the schema element (its + /// `xmlns:prefix` attributes; the default namespace under the empty key). + private static func namespaceBindings(_ schema: XSDTree) -> [String: String] { + var bindings: [String: String] = [:] + for attribute in schema.attributes { + if attribute.name.prefix == "xmlns" { + bindings[attribute.name.localName] = attribute.value + } else if attribute.name.prefix == nil, attribute.name.localName == "xmlns" { + bindings[""] = attribute.value + } + } + return bindings + } + + /// Visits every `complexType` element in the schema document, skipping + /// foreign annotation content. + private static func forEachComplexType(_ node: XSDTree, _ visit: (XSDTree) -> Void) { + let local = PureXML.Schema.XSDNode.localName(node) + if local == "appinfo" || local == "documentation" { return } + if local == "complexType" { visit(node) } + for child in PureXML.Schema.XSDNode.elementChildren(node) { + forEachComplexType(child, visit) + } + } + + /// The model-group node (`sequence`/`choice`/`all`/`group`) that holds a + /// complex type's content model, reached through `complexContent` and its + /// `restriction`/`extension`, or nil for simple or empty content. + private static func contentModelNode(_ complexType: XSDTree) -> XSDTree? { + if let direct = modelGroupChild(complexType) { return direct } + guard let complexContent = PureXML.Schema.XSDNode.firstChild(complexType, named: "complexContent") else { return nil } + for derivation in PureXML.Schema.XSDNode.elementChildren(complexContent) { + if let model = modelGroupChild(derivation) { return model } + } + return nil + } + + private static func modelGroupChild(_ node: XSDTree) -> XSDTree? { + PureXML.Schema.XSDNode.elementChildren(node).first { + ["sequence", "choice", "all", "group"].contains(PureXML.Schema.XSDNode.localName($0) ?? "") + } + } + + /// A UPA-violation message for the content model rooted at `model`, or nil. + private static func violation(_ model: XSDTree, _ bindings: [String: String], _ context: XSDContext) -> String? { + var automaton = PureXML.Schema.PositionAutomaton() + let root = build(model, &automaton, bindings, context, []) + // A model that hit the position cap was only partially built; skip it + // rather than risk a result from an incomplete automaton. + if automaton.labels.count > positionCap { return nil } + var decisionSets = automaton.followpos + decisionSets.append(root.first) + for set in decisionSets { + if let conflict = overlap(in: set, automaton: automaton) { + return "ambiguous content model: '\(conflict)' can be matched by more than one particle" + } + } + return nil + } + + /// A ceiling on positions per content model. A pathological schema is bounded + /// here and its UPA check skipped (an under-rejection, never an + /// over-rejection). Real content models are far below this. + private static let positionCap = 4096 + + private static func build( + _ node: XSDTree, + _ automaton: inout PureXML.Schema.PositionAutomaton, + _ bindings: [String: String], + _ context: XSDContext, + _ visiting: Set, + ) -> PureXML.Schema.Positions { + if automaton.labels.count > positionCap { return .empty } + let (minimum, maximum) = PureXML.Schema.XSDNode.occurrence(node) + // A `maxOccurs="0"` particle can never occur, and a negative maxOccurs is + // malformed (flagged by occurrence-order validation): either contributes + // nothing here, and this guards the copy count below against a bad bound. + if let maximum, maximum <= 0 { return .empty } + // A fixed count {n,n} is n copies in sequence with no repetition loop, so + // `b{2,2}, b` stays the deterministic `b, b, b`. The exact count above one + // does not change the determinism question, so it is clamped to two. + if let maximum, minimum == maximum { + let copies = (0 ..< Swift.min(maximum, 2)).map { _ in buildInner(node, &automaton, bindings, context, visiting) } + return sequence(copies, &automaton) + } + // Here min < max. Build one copy. A self-loop (last -> first) is added + // only when the particle can truly repeat (max unbounded or above one); + // unrolling a repetition into separate copies would put one element in two + // positions of a decision set and read as ambiguity, when a repetition of + // a deterministic body is itself deterministic. A {0,1} is merely optional, + // not a repetition, so it gets no loop. The floor sets nullability. + let inner = buildInner(node, &automaton, bindings, context, visiting) + if maximum == nil || (maximum ?? 0) > 1 { + for position in inner.last { + automaton.followpos[position].formUnion(inner.first) + } + } + return PureXML.Schema.Positions(nullable: minimum == 0 || inner.nullable, first: inner.first, last: inner.last) + } + + private static func buildInner( + _ node: XSDTree, + _ automaton: inout PureXML.Schema.PositionAutomaton, + _ bindings: [String: String], + _ context: XSDContext, + _ visiting: Set, + ) -> PureXML.Schema.Positions { + switch PureXML.Schema.XSDNode.localName(node) { + case "element": + let position = automaton.position(.name(elementQName(node, bindings, context)), node) + return PureXML.Schema.Positions(nullable: false, first: [position], last: [position]) + case "any": + let position = automaton.position(.wildcard(PureXML.Schema.XSDParser.wildcard(node, context)), node) + return PureXML.Schema.Positions(nullable: false, first: [position], last: [position]) + case "sequence": + return sequence(childSets(node, &automaton, bindings, context, visiting), &automaton) + case "choice": + return choice(childSets(node, &automaton, bindings, context, visiting)) + case "all": + return all(childSets(node, &automaton, bindings, context, visiting), &automaton) + case "group": + return groupReference(node, &automaton, bindings, context, visiting) + default: + return .empty + } + } + + private static func childSets( + _ node: XSDTree, + _ automaton: inout PureXML.Schema.PositionAutomaton, + _ bindings: [String: String], + _ context: XSDContext, + _ visiting: Set, + ) -> [PureXML.Schema.Positions] { + PureXML.Schema.XSDNode.elementChildren(node) + .filter { particleNames.contains(PureXML.Schema.XSDNode.localName($0) ?? "") } + .map { build($0, &automaton, bindings, context, visiting) } + } + + /// Inlines a `group` reference's model group (its sole `all`/`choice`/ + /// `sequence`), guarding against a cyclic reference. + private static func groupReference( + _ node: XSDTree, + _ automaton: inout PureXML.Schema.PositionAutomaton, + _ bindings: [String: String], + _ context: XSDContext, + _ visiting: Set, + ) -> PureXML.Schema.Positions { + guard let ref = PureXML.Schema.XSDNode.attribute(node, "ref") else { return .empty } + let name = PureXML.Schema.XSDNode.stripPrefix(ref) + guard !visiting.contains(name), let definition = context.groups[name], + let model = PureXML.Schema.XSDNode.elementChildren(definition).first(where: { modelGroupNames.contains(PureXML.Schema.XSDNode.localName($0) ?? "") }) + else { return .empty } + return build(model, &automaton, bindings, context, visiting.union([name])) + } + + private static func elementQName(_ node: XSDTree, _ bindings: [String: String], _ context: XSDContext) -> PureXML.Model.QualifiedName { + if let ref = PureXML.Schema.XSDNode.attribute(node, "ref") { + return resolveQName(ref, bindings, context) + } + let name = PureXML.Schema.XSDNode.attribute(node, "name") ?? "" + let form = PureXML.Schema.XSDNode.attribute(node, "form") + let qualified = form == "qualified" || (form == nil && context.elementFormQualified) + return PureXML.Model.QualifiedName(localName: name, namespaceURI: qualified ? context.targetNamespace : nil) + } + + /// Resolves a QName reference to (localName, namespaceURI): a prefix maps + /// through the schema's bindings (or stands for itself if unbound, keeping + /// distinct prefixes distinct); an unprefixed name takes the default + /// namespace binding. + private static func resolveQName(_ qname: String, _ bindings: [String: String], _: XSDContext) -> PureXML.Model.QualifiedName { + let parts = qname.split(separator: ":", maxSplits: 1).map(String.init) + if parts.count == 2 { + return PureXML.Model.QualifiedName(localName: parts[1], namespaceURI: bindings[parts[0]] ?? parts[0]) + } + return PureXML.Model.QualifiedName(localName: qname, namespaceURI: bindings[""]) + } + + private static func sequence(_ children: [PureXML.Schema.Positions], _ automaton: inout PureXML.Schema.PositionAutomaton) -> PureXML.Schema.Positions { + var first: Set = [] + var prefixNullable = true + var preceding: Set = [] + for child in children { + if prefixNullable { first.formUnion(child.first) } + for position in preceding { + automaton.followpos[position].formUnion(child.first) + } + preceding = child.nullable ? preceding.union(child.last) : child.last + if !child.nullable { prefixNullable = false } + } + var last: Set = [] + var suffixNullable = true + for child in children.reversed() { + if suffixNullable { last.formUnion(child.last) } + if !child.nullable { suffixNullable = false } + } + return PureXML.Schema.Positions(nullable: children.allSatisfy(\.nullable), first: first, last: last) + } + + private static func choice(_ children: [PureXML.Schema.Positions]) -> PureXML.Schema.Positions { + PureXML.Schema.Positions( + nullable: children.isEmpty || children.contains(where: \.nullable), + first: children.reduce(into: Set()) { $0.formUnion($1.first) }, + last: children.reduce(into: Set()) { $0.formUnion($1.last) }, + ) + } + + /// `all`: members are order-independent, so each may follow every other and + /// the members form one decision set; two same-named members then overlap. + private static func all(_ children: [PureXML.Schema.Positions], _ automaton: inout PureXML.Schema.PositionAutomaton) -> PureXML.Schema.Positions { + for source in children { + for target in children where target.first != source.first { + for position in source.last { + automaton.followpos[position].formUnion(target.first) + } + } + } + return choice(children) + } + + private static func overlap(in set: Set, automaton: PureXML.Schema.PositionAutomaton) -> String? { + let positions = set.sorted() + for outer in 0 ..< positions.count { + let outerPosition = positions[outer] + for inner in (outer + 1) ..< positions.count { + let innerPosition = positions[inner] + // Two positions conflict only if they are distinct particles whose + // labels can match the same element; one particle's own repetition + // (same source) is deterministic by construction. + guard automaton.particles[outerPosition] != automaton.particles[innerPosition], + labelsOverlap(automaton.labels[outerPosition], automaton.labels[innerPosition]) + else { continue } + return describe(automaton.labels[outerPosition]) + } + } + return nil + } + + private static func labelsOverlap(_ lhs: TermLabel, _ rhs: TermLabel) -> Bool { + switch (lhs, rhs) { + case let (.name(lhsName), .name(rhsName)): + lhsName.localName == rhsName.localName && lhsName.namespaceURI == rhsName.namespaceURI + case let (.name(name), .wildcard(wildcard)), let (.wildcard(wildcard), .name(name)): + wildcard.admits(name) + case let (.wildcard(lhsCard), .wildcard(rhsCard)): + wildcardsOverlap(lhsCard, rhsCard) + } + } + + private static func wildcardsOverlap(_ lhs: Wildcard, _ rhs: Wildcard) -> Bool { + switch (lhs.namespace, rhs.namespace) { + case (.any, _), (_, .any): + true + case let (.enumerated(lhsSet), .enumerated(rhsSet)): + !lhsSet.isDisjoint(with: rhsSet) + default: + false + } + } + + private static func describe(_ label: TermLabel) -> String { + switch label { + case let .name(name): name.localName + case .wildcard: "any" + } + } + } + + /// The firstpos/lastpos/nullable summary of a sub-model, used while building the + /// position automaton. + private struct Positions { + let nullable: Bool + let first: Set + let last: Set + + static let empty = Positions(nullable: true, first: [], last: []) + } + + /// The Glushkov position automaton under construction: one entry per position + /// for its transition label, its `followpos` set, and the source particle it + /// came from (Unique Particle Attribution is per particle, so a particle never + /// conflicts with a repetition of itself). + private struct PositionAutomaton { + var labels: [TermLabel] = [] + var followpos: [Set] = [] + var particles: [ObjectIdentifier] = [] + + mutating func position(_ label: TermLabel, _ source: XSDTree) -> Int { + labels.append(label) + followpos.append([]) + particles.append(ObjectIdentifier(source)) + return labels.count - 1 + } + } +} diff --git a/Sources/Schema/XSDParser.swift b/Sources/Schema/XSDParser.swift index faa2a30..ee3f345 100644 --- a/Sources/Schema/XSDParser.swift +++ b/Sources/Schema/XSDParser.swift @@ -33,7 +33,9 @@ extension PureXML.Schema { context.attributeFormQualified = XSDNode.attribute(schema, "attributeFormDefault") == "qualified" context.complexTypeNodes = indexByName(allChildren(containers, named: "complexType")) context.globalAttributes = indexByName(allChildren(containers, named: "attribute")) - for error in idAttributeErrors(schema) + structureErrors(schema) + componentNameErrors(schema) { + let consistencyErrors = idAttributeErrors(schema) + structureErrors(schema) + + componentNameErrors(schema) + ContentModelDeterminism.violations(in: schema, context: context) + for error in consistencyErrors { context.diagnostics.report(error) } var types = namedTypes(containers, into: &context) diff --git a/Tests/ContentModelDeterminismTests.swift b/Tests/ContentModelDeterminismTests.swift new file mode 100644 index 0000000..6c61607 --- /dev/null +++ b/Tests/ContentModelDeterminismTests.swift @@ -0,0 +1,114 @@ +@testable import PureXML +import Testing + +/// Unique Particle Attribution (cos-nonambig, XSD 1.0 3.8.6): a content model +/// must be deterministic, so each element in an instance is attributable to a +/// single particle without lookahead. The check runs on a literal, QName-resolved +/// view of the raw schema tree (refs are not substitution-group expanded, the +/// overlap test is QName-only), and is per particle: a particle competing with a +/// repetition of itself is never a violation. These cases pin both the rejections +/// and the deterministic shapes that must keep compiling (XSTS valid mgZ005, +/// ctZ008, particlesHa142, groupH001). +@Suite("Content model determinism (UPA)") +struct ContentModelDeterminismTests { + private func compile(_ body: String) throws { + _ = try PureXML.Schema.Document(""" + + \(body) + + """) + } + + private func rejects(_ body: String) -> Bool { + do { try compile(body) + return false + } catch { return true } + } + + private func complexType(_ model: String) -> String { + #"\#(model)"# + } + + // MARK: - Ambiguous models are rejected + + @Test("a choice of two same-name elements is ambiguous") + func test_choiceOfSameName() { + #expect(rejects(complexType(#""#))) + } + + @Test("an optional element followed by a same-name element is ambiguous") + func test_optionalThenSameName() { + #expect(rejects(complexType(#""#))) + } + + @Test("a repeating element followed by a same-name element is ambiguous") + func test_repeatThenSameName() { + #expect(rejects(complexType(#""#))) + } + + @Test("two same-name members of an all group are ambiguous") + func test_allWithSameName() { + #expect(rejects(complexType(#""#))) + } + + @Test("an element competing with a wildcard that admits it is ambiguous") + func test_elementAndWildcard() { + #expect(rejects(complexType(#""#))) + } + + // MARK: - Deterministic models keep compiling + + @Test("distinct elements in sequence and choice are deterministic") + func test_distinctNames() throws { + try compile(complexType(#""#)) + try compile(complexType(#""#)) + try compile(complexType(#""#)) + } + + @Test("a fixed-count element followed by a distinct same-name element is deterministic (mgZ005)") + func test_fixedCountThenSameName() throws { + try compile(complexType(#""" + + + + + + """#)) + } + + @Test("a fixed-count group wrapping a repeating element is deterministic (ctZ008)") + func test_fixedCountGroupWithRepeat() throws { + try compile(complexType(#""" + + + + + """#)) + } + + @Test("an element followed by its own optional repeat is deterministic (particlesHa142)") + func test_elementThenOptionalSameName() throws { + try compile(complexType(#""" + + + + + + + """#)) + } + + @Test("a bounded-repeat choice of group references is deterministic (groupH001)") + func test_repeatedChoiceOfGroups() throws { + try compile(#""" + + + + + + + + + """#) + } +} diff --git a/Tests/SchemaStructureTests.swift b/Tests/SchemaStructureTests.swift index 7233ebe..86747fc 100644 --- a/Tests/SchemaStructureTests.swift +++ b/Tests/SchemaStructureTests.swift @@ -204,10 +204,11 @@ struct SchemaStructureTests { """#)) - // Same name, same type is fine. + // Same name, same type is consistent (separated so it stays unambiguous). try compile(#""" - + + """#) diff --git a/Tests/XSTSSuiteTests.swift b/Tests/XSTSSuiteTests.swift index b88b27c..4d6d85a 100644 --- a/Tests/XSTSSuiteTests.swift +++ b/Tests/XSTSSuiteTests.swift @@ -29,7 +29,7 @@ struct XSTSSuiteTests { /// XSTS` (debug is far slower and the corpus is large). Per-case deviations /// are written to /tmp/xsts-failures.txt for the burn-down. private let knownSchemaValidRejected = 72 - private let knownSchemaInvalidAccepted = 884 + private let knownSchemaInvalidAccepted = 864 private let knownInstanceValidRejected = 214 private let knownInstanceInvalidAccepted = 165 diff --git a/docs/schema-validity-burndown.md b/docs/schema-validity-burndown.md index d45d44d..4d5e6a1 100644 --- a/docs/schema-validity-burndown.md +++ b/docs/schema-validity-burndown.md @@ -10,7 +10,7 @@ irreducible tail. Do not deviate to new features while this is open. | Bucket | Count | Reading | |---|---|---| | valid-schemas-rejected | 72 | we rarely reject a good schema | -| **invalid-schemas-accepted** | **2461 -> 884** | we rarely catch a bad one (facet, id, and structural content-model validity landed) | +| **invalid-schemas-accepted** | **2461 -> 864** | we rarely catch a bad one (facet, id, structural content-model, and UPA determinism validity landed) | | valid-instances-rejected | 233 | (instance side, separately tracked) | | invalid-instances-accepted | 165 | |