From cbeea0e3c24bfc5f56a4313f2b76c12ad0b38410 Mon Sep 17 00:00:00 2001 From: Mihaela Mihaljevic Date: Sun, 14 Jun 2026 00:17:29 +0200 Subject: [PATCH] feat(schema): reject unresolvable references at compile time Every QName a schema names must resolve to a declared component or a built-in: type/base/itemType/memberTypes to a type, element/attribute/group/attributeGroup ref to the matching global component, and element substitutionGroup to a global element. An undeclared reference was previously accepted at compile time and only surfaced lazily during instance validation, if at all. References are matched by local name, with the value trimmed (a QName attribute is whiteSpace=collapse). The whole check is skipped when the document declares an import/include/redefine, since the default compile does not load externals and the pools would be incomplete; that guard is gated on the XSD namespace so a foreign element named "import" does not trip it. XSTS: invalid schemas accepted 1092 to 1051 (-41); valid schemas rejected held at 72 (an interim version regressed one valid schema on a whitespace-padded base=" xs:string ", fixed by trimming); instance buckets unchanged. --- CHANGELOG.md | 1 + Sources/Schema/SchemaReferences.swift | 95 +++++++++++++++++++++++++++ Sources/Schema/SchemaStructure.swift | 2 +- Sources/Schema/XSDParser.swift | 24 +++++-- Tests/SchemaReferenceTests.swift | 56 ++++++++++++++++ Tests/XSTSSuiteTests.swift | 2 +- docs/schema-validity-burndown.md | 14 ++-- docs/xsts-deviations.md | 4 +- 8 files changed, 181 insertions(+), 17 deletions(-) create mode 100644 Sources/Schema/SchemaReferences.swift create mode 100644 Tests/SchemaReferenceTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index f6ca59f..b435b2d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- Unresolvable references are now rejected at compile time (#147). Every QName a schema names (a `type`/`base`/`itemType`/`memberTypes` type, an `element`/`attribute`/`group`/`attributeGroup` `ref`, an element `substitutionGroup`) must resolve to a declared component or a built-in; an undeclared reference was previously accepted and only surfaced (if at all) lazily during instance validation. The reference is matched by local name with whitespace trimmed, as the rest of the compiler resolves it. The check is skipped when the document pulls in external definitions through `import`/`include`/`redefine`, which the default compile does not load, so a reference into them is never flagged. XSTS: invalid schemas accepted 1092 to 1051, no other bucket raised. - `pattern` facet regular expressions are now validated at compile time (#147). Each `pattern` value is compiled when the schema is; a structurally malformed one (unbalanced parentheses `((a)`, a quantifier with nothing to repeat `?a`) is rejected, where before it was only compiled lazily at instance time (`try?`) and so silently ignored. The check is conservative: a construct the engine merely does not support on an otherwise-valid pattern (the empty pattern, an untabulated `\p{Is...}` block such as a surrogate or historic block, the lenient `{,m}` quantifier) is tolerated, so no valid schema is rejected. XSTS: invalid schemas accepted 1129 to 1092, no other bucket raised. - Schema-vocabulary name and reference attribute values are now checked for lexical validity at compile time (#147). A component's `name` must be a valid NCName (`name=""`, `name="123"`, `name="a:b"` are rejected), and the reference attributes `type`/`base`/`ref`/`itemType`/`refer`/`substitutionGroup` must be lexical QNames (`type=":_"`, `type="a:b:c"` rejected), with `memberTypes` a list of QNames; these were unchecked and the schema accepted. Only the lexical form is checked (a QName's prefix is resolved elsewhere), so no valid schema is rejected. XSTS: invalid schemas accepted 1190 to 1129, no other bucket raised. - Schema-vocabulary attribute values are now validated at compile time (#147), the attribute-model sibling of the structural content-model check. Enumerated attributes (`form`/`elementFormDefault`/`attributeFormDefault` to `qualified`/`unqualified`, `use` to `optional`/`prohibited`/`required`, `processContents` to `skip`/`lax`/`strict`, `mixed`/`abstract`/`nillable` to a boolean) and `minOccurs`/`maxOccurs` (a `nonNegativeInteger`, `maxOccurs` also `unbounded`) are checked against their fixed value space; an out-of-space value (`use="foo"`, `mixed="yes"`, `minOccurs="-1"`) was previously ignored and the schema accepted. Only unprefixed attributes on XSD-namespace elements are checked, so a foreign attribute is never flagged. XSTS: invalid schemas accepted 1313 to 1190, no other bucket raised. diff --git a/Sources/Schema/SchemaReferences.swift b/Sources/Schema/SchemaReferences.swift new file mode 100644 index 0000000..b32dd3f --- /dev/null +++ b/Sources/Schema/SchemaReferences.swift @@ -0,0 +1,95 @@ +extension PureXML.Schema.XSDParser { + /// The built-in type names a reference may name without a declaration: the + /// XSD Part 2 datatypes, the built-in list types, and the ur-types. + private static let referenceBuiltins: Set = { + var names = Set(PureXML.Schema.BuiltinType.allCases.map(\.rawValue)) + names.formUnion(["anyType", "anySimpleType", "anyAtomicType", "NOTATION", "IDREFS", "ENTITIES", "NMTOKENS"]) + return names + }() + + /// Schema-validity findings for unresolvable references: every QName a schema + /// names (a `type`/`base`/`itemType`/`memberTypes` type, an `element`/ + /// `attribute`/`group`/`attributeGroup` `ref`, an element `substitutionGroup`) + /// must resolve to a declared component or a built-in. The reference is matched + /// by local name, as the rest of the compiler resolves it. + /// + /// Skipped when the document pulls in external definitions through + /// `import`/`include`/`redefine`: the default compile does not load them, so + /// the pools would be incomplete and a reference into them must not be flagged. + static func referenceErrors(_ schema: XSDTree, in context: PureXML.Schema.XSDContext, elements: [String: PureXML.Schema.ElementType]) -> [String] { + if hasExternalReference(schema) { return [] } + let types = referenceBuiltins.union(context.simpleTypes.keys).union(context.complexTypeNodes.keys) + let pools: [String: Set] = [ + "element": Set(elements.keys), + "attribute": Set(context.globalAttributes.keys), + "group": Set(context.groups.keys), + "attributeGroup": Set(context.attributeGroups.keys), + ] + var errors: [String] = [] + collectReferenceErrors(schema, types: types, pools: pools, into: &errors) + return errors + } + + /// Whether the document declares an `import`, `include`, or `redefine`, so its + /// component pools may be completed by an external document the default compile + /// does not load. + private static func hasExternalReference(_ schema: XSDTree) -> Bool { + var found = false + func walk(_ node: XSDTree) { + if node.name?.namespaceURI == xsdNamespace, let local = PureXML.Schema.XSDNode.localName(node), + ["import", "include", "redefine"].contains(local) { + found = true + return + } + for child in PureXML.Schema.XSDNode.elementChildren(node) where !found { + walk(child) + } + } + walk(schema) + return found + } + + private static func collectReferenceErrors( + _ node: XSDTree, + types: Set, + pools: [String: Set], + into errors: inout [String], + ) { + let local = PureXML.Schema.XSDNode.localName(node) + if local == "appinfo" || local == "documentation" { return } + if node.name?.namespaceURI == xsdNamespace, let local { + errors += referenceErrors(at: node, local: local, types: types, pools: pools) + } + for child in PureXML.Schema.XSDNode.elementChildren(node) { + collectReferenceErrors(child, types: types, pools: pools, into: &errors) + } + } + + private static func referenceErrors(at node: XSDTree, local: String, types: Set, pools: [String: Set]) -> [String] { + var errors: [String] = [] + for attribute in ["type", "base", "itemType"] { + if let qname = PureXML.Schema.XSDNode.attribute(node, attribute), !types.contains(localName(qname)) { + errors.append("\(attribute) references undeclared type '\(qname)'") + } + } + if let members = PureXML.Schema.XSDNode.attribute(node, "memberTypes") { + for token in members.split(whereSeparator: \.isWhitespace) where !types.contains(localName(String(token))) { + errors.append("memberTypes references undeclared type '\(token)'") + } + } + if let head = PureXML.Schema.XSDNode.attribute(node, "substitutionGroup"), pools["element"]?.contains(localName(head)) != true { + errors.append("substitutionGroup references undeclared element '\(head)'") + } + if let reference = PureXML.Schema.XSDNode.attribute(node, "ref"), let pool = pools[local], !pool.contains(localName(reference)) { + errors.append("\(local) ref references undeclared '\(reference)'") + } + return errors + } + + /// The local part of a QName reference, after the whitespace normalization a + /// `whiteSpace="collapse"` QName attribute receives (a value may be written + /// with surrounding or, harmlessly, interior whitespace). + private static func localName(_ qname: String) -> String { + PureXML.Schema.XSDNode.stripPrefix(qname.trimmingXMLWhitespace()) + } +} diff --git a/Sources/Schema/SchemaStructure.swift b/Sources/Schema/SchemaStructure.swift index db1593d..18ad8a4 100644 --- a/Sources/Schema/SchemaStructure.swift +++ b/Sources/Schema/SchemaStructure.swift @@ -1,7 +1,7 @@ extension PureXML.Schema.XSDParser { /// The XML Schema namespace; only elements in it are schema vocabulary, so /// foreign elements (and annotation content) are not structurally checked. - private static let xsdNamespace = "http://www.w3.org/2001/XMLSchema" + static let xsdNamespace = "http://www.w3.org/2001/XMLSchema" /// The schema-for-schemas child content model (XSD 1.0 Structures): the /// XSD-namespace element local names each component admits as children. The diff --git a/Sources/Schema/XSDParser.swift b/Sources/Schema/XSDParser.swift index 271fd62..64853bd 100644 --- a/Sources/Schema/XSDParser.swift +++ b/Sources/Schema/XSDParser.swift @@ -36,14 +36,11 @@ extension PureXML.Schema { for error in idAttributeErrors(schema) + structureErrors(schema) { context.diagnostics.report(error) } - resolveSimpleTypes(allChildren(containers, named: "simpleType"), into: &context) - var types: [String: ElementType] = context.simpleTypes.mapValues(ElementType.simple) - for node in allChildren(containers, named: "complexType") { - if let name = XSDNode.attribute(node, "name") { - types[name] = .complex(complexType(node, context)) - } - } + var types = namedTypes(containers, into: &context) let elements = globalElements(containers, context, into: &types) + for error in referenceErrors(schema, in: context, elements: elements) { + context.diagnostics.report(error) + } let (nillable, elementConstraints) = elementMetadata(containers) return XSDCompiled( elements: elements, @@ -62,6 +59,19 @@ extension PureXML.Schema { ) } + /// The named-type table: the global simple types resolved into `context`, + /// then each global complex type compiled and added by name. + private static func namedTypes(_ containers: [XSDTree], into context: inout XSDContext) -> [String: ElementType] { + resolveSimpleTypes(allChildren(containers, named: "simpleType"), into: &context) + var types: [String: ElementType] = context.simpleTypes.mapValues(ElementType.simple) + for node in allChildren(containers, named: "complexType") { + if let name = XSDNode.attribute(node, "name") { + types[name] = .complex(complexType(node, context)) + } + } + return types + } + /// The global element declarations, also registering each under a reserved /// `element:name` key in `types` so an `xs:element ref="name"` resolves to /// this declaration's type (a colon keeps it out of the NCName-only type diff --git a/Tests/SchemaReferenceTests.swift b/Tests/SchemaReferenceTests.swift new file mode 100644 index 0000000..f2e19a0 --- /dev/null +++ b/Tests/SchemaReferenceTests.swift @@ -0,0 +1,56 @@ +@testable import PureXML +import Testing + +/// Every QName a schema names (a `type`/`base`/`itemType`/`memberTypes` type, an +/// element/attribute/group/attributeGroup `ref`, an element `substitutionGroup`) +/// must resolve to a declared component or a built-in. An undeclared reference +/// was previously accepted at compile time and only surfaced (if at all) lazily +/// during instance validation. The check is skipped when the document pulls in +/// external definitions through `import`/`include`/`redefine`, which the default +/// compile does not load. +@Suite("Schema reference resolution") +struct SchemaReferenceTests { + private func compile(_ body: String) throws { + _ = try PureXML.Schema.Document(""" + + \(body) + + """) + } + + private func rejects(_ body: String) -> Bool { + do { try compile(body) + return false + } catch { return true } + } + + @Test("a reference to a declared component or a built-in resolves") + func test_resolves() throws { + try compile(#""#) + try compile(#""#) + try compile(#""#) + } + + @Test("an undeclared reference is rejected") + func test_undeclared() { + #expect(rejects(#""#)) + #expect(rejects(#""#)) + #expect(rejects(#""#)) + #expect(rejects(#""#)) + } + + @Test("a reference value tolerates surrounding whitespace (whiteSpace collapse)") + func test_whitespacePadded() throws { + try compile("") + } + + @Test("references are not checked when an import may supply them") + func test_importSkipsCheck() throws { + // A dangling reference into an imported namespace is tolerated: the default + // compile does not load the import, so the reference cannot be verified. + try compile(#""" + + + """#) + } +} diff --git a/Tests/XSTSSuiteTests.swift b/Tests/XSTSSuiteTests.swift index 1688a93..8b36d6c 100644 --- a/Tests/XSTSSuiteTests.swift +++ b/Tests/XSTSSuiteTests.swift @@ -29,7 +29,7 @@ struct XSTSSuiteTests { /// XSTS` (debug is far slower and the corpus is large). Per-case deviations /// are written to /tmp/xsts-failures.txt for the burn-down. private let knownSchemaValidRejected = 72 - private let knownSchemaInvalidAccepted = 1092 + private let knownSchemaInvalidAccepted = 1051 private let knownInstanceValidRejected = 214 private let knownInstanceInvalidAccepted = 165 diff --git a/docs/schema-validity-burndown.md b/docs/schema-validity-burndown.md index 7ca31d3..75f1aa8 100644 --- a/docs/schema-validity-burndown.md +++ b/docs/schema-validity-burndown.md @@ -10,7 +10,7 @@ irreducible tail. Do not deviate to new features while this is open. | Bucket | Count | Reading | |---|---|---| | valid-schemas-rejected | 72 | we rarely reject a good schema | -| **invalid-schemas-accepted** | **2461 -> 1092** | we rarely catch a bad one (facet, id, and structural content-model validity landed) | +| **invalid-schemas-accepted** | **2461 -> 1051** | we rarely catch a bad one (facet, id, and structural content-model validity landed) | | valid-instances-rejected | 233 | (instance side, separately tracked) | | invalid-instances-accepted | 165 | | @@ -61,11 +61,13 @@ Each iteration targets one self-contained rule family with a clean root cause. type / element / attribute names unique per namespace; no duplicate attribute uses on a type. Cheap graph checks (~12 identity-constraint-name cases, plus type/element/attribute name collisions). -4. **Resolvable references.** Every QName a schema names (`type`, element / - attribute / group `ref`, restriction / extension `base`, `itemType`, - `memberTypes`, `substitutionGroup`, keyref `refer`) must resolve to a - declaration at compile time. Machinery exists (`resolveReference`); not run - exhaustively at compile time. The `xsd0NN.e` "undeclared X" family. +4. **Resolvable references.** _Done (1092 -> 1051)._ `type`/`base`/`itemType`/ + `memberTypes` types, `element`/`attribute`/`group`/`attributeGroup` `ref`, and + element `substitutionGroup` must resolve to a declared component or a built-in + (`SchemaReferences.swift`), matched by local name, skipped when + `import`/`include`/`redefine` may supply externals. **Still open:** keyref + `refer` resolution; the context-sensitive applicability rules (`form` on a + global decl, `ref` excluding `name`/`type`). 5. **Facet applicability per base** (the open facet-family tail): which facets a base admits (`fractionDigits` only on decimal-derived, `length` only on string/binary/list, etc.), and a fixed facet may not be changed. diff --git a/docs/xsts-deviations.md b/docs/xsts-deviations.md index e6f92cd..2f32fc6 100644 --- a/docs/xsts-deviations.md +++ b/docs/xsts-deviations.md @@ -47,7 +47,7 @@ and to document the remainder as spec-justified, deliberate exclusions. ## Category A: genuine correctness bugs -### Fixed (twenty-one root causes, ~2043 deviations cleared) +### Fixed (twenty-two root causes, ~2084 deviations cleared) list-datatype length facets (#146) · union pattern/enumeration facets · duration partial-order facets · QName length non-constraining · the full XSD `\p{Is...}` @@ -65,7 +65,7 @@ validity (`min`/`maxInclusive`, `min`/`maxExclusive`, `enumeration` values valid in the base space; inclusive/exclusive exclusions; bound ordering) · `gMonth` `--MM--` lexical form · schema `id` attribute validity (`xs:ID`: NCName and unique within the document). The schema-validity work brought invalid schemas -accepted 2461 to 1092 (also: schema-document structural validity, content model + attribute values + name/reference + pattern-regex lexical validity, against the schema-for-schemas); see `schema-validity-burndown.md`. +accepted 2461 to 1051 (schema-document structural validity: content model, attribute values, name/reference + pattern-regex lexical validity, and reference resolution, against the schema-for-schemas); see `schema-validity-burndown.md`. Plus the measurement fix (exclude W3C-disputed `status="queried"` entries). ### Remaining, root-caused (the hard tail)