feat(schema): structural validity against the schema-for-schemas content model

The general mechanism for the structural tail of the schema-validity gap, in
place of hand-coding a rule per component. A data-driven table encodes the XSD
1.0 schema-for-schemas child content model; at compile time it rejects any
component child the model does not admit, more than one annotation (or an
annotation that is not first, except on schema/redefine, which allow several),
and an identity constraint missing its selector or field.

The walk visits only XSD-namespace elements and skips appinfo/documentation
foreign content, so foreign children are never flagged; restriction/extension
use the permissive union of their simple/complex/simpleContent contexts, so a
child valid in some context is never rejected. One table covers every component.

XSTS: invalid schemas accepted 1688 to 1313 (-375); valid schemas rejected held
at 72 (no over-rejection); instance buckets unchanged.

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- Schema documents are now structurally validated against the schema-for-schemas content model (#147), the general mechanism for the structural tail of the schema-validity gap. A data-driven table of the XSD 1.0 child content model rejects, at compile time, any component child the model does not admit, more than one `annotation` (or an `annotation` that is not first, except on `schema`/`redefine`), and an identity constraint missing its selector or field. The check walks only XSD-namespace elements and skips `xs:appinfo`/`xs:documentation` foreign content, so no valid schema is rejected; `restriction`/`extension` use the permissive union of their contexts (never over-rejecting). One table covers every component rather than a rule hand-coded per element. XSTS: invalid schemas accepted 1688 to 1313 (-375), no other bucket raised.
 - The `id` attribute on XSD components is now validated at compile time (#147). It is of type `xs:ID`, so each value must be a valid NCName and unique within the schema document; the values were never checked, so a malformed (`id=""`, `id="123"`) or duplicated `id` left the schema wrongly accepted. The schema-document tree is walked, skipping `xs:appinfo`/`xs:documentation` foreign content (whose `id` attributes are not `xs:ID`) and treating only the unprefixed `id` (not a foreign `xml:id`/`pre:id`) as the component identifier, so no valid schema is rejected. XSTS: invalid schemas accepted 1789 to 1688, no other bucket raised.
 - Value-bound facet definitions are now validated at compile time (#147), continuing the facet-validity work. Per XSD Part 2 Datatypes 4.3, each `minInclusive`/`maxInclusive`/`minExclusive`/`maxExclusive` and `enumeration` value must be a valid value of the base type; `minInclusive` excludes `minExclusive` and `maxInclusive` excludes `maxExclusive`; and the lower bound may not exceed the upper (nor equal it when either side is exclusive, an empty value space), compared in the base primitive's value space. XSTS: invalid schemas accepted 2082 to 1789, no other bucket raised.
 - `xs:gMonth` now accepts the XSD 1.0 lexical form `--MM--` (the trailing `--` is optional, the later `--MM` erratum form still accepted), with an optional timezone (#147). Only `--MM` was recognized, so valid `gMonth` values and the schemas whose facets used them were wrongly rejected. Surfaced by the value-bound facet check above (which validates a bound value against its base type). XSTS: valid instances rejected 233 to 214; it also removed 26 false schema rejections the bound-facet check would otherwise have caused.

Sources/Schema/SchemaStructure.swift

@@ -0,0 +1,93 @@
+extension PureXML.Schema.XSDParser {
+    /// The XML Schema namespace; only elements in it are schema vocabulary, so
+    /// foreign elements (and annotation content) are not structurally checked.
+    private static let xsdNamespace = "http://www.w3.org/2001/XMLSchema"
+
+    /// The schema-for-schemas child content model (XSD 1.0 Structures): the
+    /// XSD-namespace element local names each component admits as children. The
+    /// `restriction`/`extension` entries are the permissive union across their
+    /// simple/complex/simpleContent contexts, so this never rejects a child that
+    /// is valid in some context; order beyond the leading annotation is not
+    /// modelled here. Leaf components and facets admit only `annotation`.
+    private static let allowedChildren: [String: Set<String>] = {
+        let facets: Set = [
+            "minExclusive", "minInclusive", "maxExclusive", "maxInclusive", "totalDigits",
+            "fractionDigits", "length", "minLength", "maxLength", "enumeration", "whiteSpace", "pattern",
+        ]
+        var table: [String: Set<String>] = [
+            "schema": ["include", "import", "redefine", "annotation", "simpleType", "complexType", "group", "attributeGroup", "element", "attribute", "notation"],
+            "complexType": ["annotation", "simpleContent", "complexContent", "group", "all", "choice", "sequence", "attribute", "attributeGroup", "anyAttribute"],
+            "simpleContent": ["annotation", "restriction", "extension"],
+            "complexContent": ["annotation", "restriction", "extension"],
+            "simpleType": ["annotation", "restriction", "list", "union"],
+            "element": ["annotation", "simpleType", "complexType", "unique", "key", "keyref"],
+            "attribute": ["annotation", "simpleType"],
+            "attributeGroup": ["annotation", "attribute", "attributeGroup", "anyAttribute"],
+            "group": ["annotation", "all", "choice", "sequence"],
+            "all": ["annotation", "element"],
+            "choice": ["annotation", "element", "group", "choice", "sequence", "any"],
+            "sequence": ["annotation", "element", "group", "choice", "sequence", "any"],
+            "restriction": Set(["annotation", "simpleType", "group", "all", "choice", "sequence", "attribute", "attributeGroup", "anyAttribute"]).union(facets),
+            "extension": ["annotation", "group", "all", "choice", "sequence", "attribute", "attributeGroup", "anyAttribute"],
+            "unique": ["annotation", "selector", "field"],
+            "key": ["annotation", "selector", "field"],
+            "keyref": ["annotation", "selector", "field"],
+            "annotation": ["appinfo", "documentation"],
+            "list": ["annotation", "simpleType"],
+            "union": ["annotation", "simpleType"],
+            "redefine": ["annotation", "simpleType", "complexType", "group", "attributeGroup"],
+        ]
+        for leaf in ["selector", "field", "any", "anyAttribute", "notation", "import", "include"] + Array(facets) {
+            table[leaf] = ["annotation"]
+        }
+        return table
+    }()
+
+    /// Components whose annotation may be repeated and interspersed (XSD 1.0:
+    /// `schema` and `redefine`); everywhere else `annotation` is `annotation?`
+    /// and must be the first child.
+    private static let multipleAnnotation: Set<String> = ["schema", "redefine"]
+
+    /// Schema-validity findings for the structural content of a schema document:
+    /// each XSD-namespace element's XSD-namespace children must be admitted by the
+    /// schema-for-schemas content model, the single `annotation` (where allowed
+    /// once) must be first, and an identity constraint needs a selector and field.
+    /// Walks the one document rooted at `schema`, skipping foreign content.
+    static func structureErrors(_ schema: XSDTree) -> [String] {
+        var errors: [String] = []
+        collectStructure(schema, into: &errors)
+        return errors
+    }
+
+    private static func collectStructure(_ node: XSDTree, into errors: inout [String]) {
+        let local = PureXML.Schema.XSDNode.localName(node)
+        if local == "appinfo" || local == "documentation" { return }
+        let children = PureXML.Schema.XSDNode.elementChildren(node)
+        if node.name?.namespaceURI == xsdNamespace, let local, let allowed = allowedChildren[local] {
+            let names = children.filter { $0.name?.namespaceURI == xsdNamespace }.compactMap(PureXML.Schema.XSDNode.localName)
+            errors += childErrors(local: local, children: names, allowed: allowed)
+        }
+        for child in children {
+            collectStructure(child, into: &errors)
+        }
+    }
+
+    private static func childErrors(local: String, children: [String], allowed: Set<String>) -> [String] {
+        var errors: [String] = []
+        for name in children where !allowed.contains(name) {
+            errors.append("element '\(name)' is not allowed in '\(local)'")
+        }
+        if !multipleAnnotation.contains(local) {
+            let annotations = children.count(where: { $0 == "annotation" })
+            if annotations > 1 {
+                errors.append("'\(local)' may have at most one annotation")
+            } else if annotations == 1, children.first != "annotation" {
+                errors.append("the annotation in '\(local)' must be the first child")
+            }
+        }
+        if ["unique", "key", "keyref"].contains(local), !(children.contains("selector") && children.contains("field")) {
+            errors.append("identity constraint '\(local)' requires a selector and at least one field")
+        }
+        return errors
+    }
+}

Sources/Schema/XSDParser.swift

@@ -33,7 +33,7 @@ extension PureXML.Schema {
             context.attributeFormQualified = XSDNode.attribute(schema, "attributeFormDefault") == "qualified"
             context.complexTypeNodes = indexByName(allChildren(containers, named: "complexType"))
             context.globalAttributes = indexByName(allChildren(containers, named: "attribute"))
-            for error in idAttributeErrors(schema) {
+            for error in idAttributeErrors(schema) + structureErrors(schema) {
                 context.diagnostics.report(error)
             }
             resolveSimpleTypes(allChildren(containers, named: "simpleType"), into: &context)

Tests/SchemaStructureTests.swift

@@ -0,0 +1,87 @@
+@testable import PureXML
+import Testing
+
+/// Structural validity of a schema document against the schema-for-schemas
+/// content model (XSD 1.0 Structures): each component's children must be
+/// admitted by the model, an `annotation` (where allowed once) must be first,
+/// and an identity constraint needs a selector and field. Such a schema is
+/// invalid and must be rejected at compile time; the children were previously
+/// unchecked and the schema accepted (XSTS invalid-schema ctB/ctG/ctH, etc.).
+@Suite("Schema structural validity")
+struct SchemaStructureTests {
+    private func compile(_ body: String) throws {
+        _ = try PureXML.Schema.Document("""
+        <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
+        \(body)
+        </xs:schema>
+        """)
+    }
+
+    private func rejects(_ body: String) -> Bool {
+        do { try compile(body)
+            return false
+        } catch { return true }
+    }
+
+    @Test("well-formed components compile")
+    func test_valid() throws {
+        try compile(#"""
+        <xs:complexType name="t">
+          <xs:annotation><xs:documentation>doc</xs:documentation></xs:annotation>
+          <xs:sequence><xs:element name="a" type="xs:string"/></xs:sequence>
+          <xs:attribute name="x" type="xs:string"/>
+        </xs:complexType>
+        """#)
+        try compile(#"""
+        <xs:element name="root">
+          <xs:complexType><xs:sequence><xs:element ref="a"/></xs:sequence></xs:complexType>
+          <xs:key name="k"><xs:selector xpath="a"/><xs:field xpath="@id"/></xs:key>
+        </xs:element>
+        <xs:element name="a"><xs:complexType><xs:attribute name="id" type="xs:string"/></xs:complexType></xs:element>
+        """#)
+    }
+
+    @Test("a child not admitted by the content model is rejected")
+    func test_disallowedChild() {
+        // element is not a direct child of complexType (it must sit in a group).
+        #expect(rejects(#"<xs:complexType name="t"><xs:element name="a" type="xs:string"/></xs:complexType>"#))
+        // attribute admits only annotation and simpleType, not sequence.
+        #expect(rejects(#"<xs:attribute name="a"><xs:sequence/></xs:attribute>"#))
+    }
+
+    @Test("at most one annotation, and it must be first")
+    func test_annotationPlacement() {
+        #expect(rejects(#"""
+        <xs:complexType name="t">
+          <xs:annotation><xs:documentation>one</xs:documentation></xs:annotation>
+          <xs:annotation><xs:documentation>two</xs:documentation></xs:annotation>
+        </xs:complexType>
+        """#))
+        #expect(rejects(#"""
+        <xs:complexType name="t">
+          <xs:sequence><xs:element name="a" type="xs:string"/></xs:sequence>
+          <xs:annotation><xs:documentation>late</xs:documentation></xs:annotation>
+        </xs:complexType>
+        """#))
+    }
+
+    @Test("schema and redefine may carry several annotations")
+    func test_multipleAnnotationOnSchema() throws {
+        try compile(#"""
+        <xs:annotation><xs:documentation>one</xs:documentation></xs:annotation>
+        <xs:element name="a" type="xs:string"/>
+        <xs:annotation><xs:documentation>two</xs:documentation></xs:annotation>
+        """#)
+    }
+
+    @Test("an identity constraint requires a selector and field")
+    func test_identityConstraintRequiresParts() {
+        #expect(rejects(#"""
+        <xs:element name="root">
+          <xs:complexType><xs:sequence><xs:element ref="a"/></xs:sequence></xs:complexType>
+          <xs:key name="k"><xs:selector xpath="a"/></xs:key>
+        </xs:element>
+        <xs:element name="a"><xs:complexType><xs:attribute name="id" type="xs:string"/></xs:complexType></xs:element>
+        """#))
+    }
+}

Tests/XSTSSuiteTests.swift

@@ -29,7 +29,7 @@ struct XSTSSuiteTests {
     /// XSTS` (debug is far slower and the corpus is large). Per-case deviations
     /// are written to /tmp/xsts-failures.txt for the burn-down.
     private let knownSchemaValidRejected = 72
-    private let knownSchemaInvalidAccepted = 1688
+    private let knownSchemaInvalidAccepted = 1313
     private let knownInstanceValidRejected = 214
     private let knownInstanceInvalidAccepted = 165
 

docs/schema-validity-burndown.md

@@ -10,7 +10,7 @@ irreducible tail. Do not deviate to new features while this is open.
 | Bucket | Count | Reading |
 |---|---|---|
 | valid-schemas-rejected | 72 | we rarely reject a good schema |
-| **invalid-schemas-accepted** | **2461 -> 1688** | we rarely catch a bad one (facet-definition validity landed) |
+| **invalid-schemas-accepted** | **2461 -> 1313** | we rarely catch a bad one (facet, id, and structural content-model validity landed) |
 | valid-instances-rejected | 233 | (instance side, separately tracked) |
 | invalid-instances-accepted | 165 | |
 
@@ -69,11 +69,15 @@ Each iteration targets one self-contained rule family with a clean root cause.
 5. **Facet applicability per base** (the open facet-family tail): which facets
    a base admits (`fractionDigits` only on decimal-derived, `length` only on
    string/binary/list, etc.), and a fixed facet may not be changed.
-6. **Particle / model-group structural rules** and **schema-for-schemas
-   structural validity** (child order, misplaced annotation, illegal content):
-   the largest, most tedious tail. Altitude decision: validate the schema
-   *document* against the normative schema-for-schemas (one general mechanism)
-   rather than hand-coding each rule. Defer until the above are done; re-measure.
+6. **Schema-for-schemas structural validity.** _Content-model pass done
+   (1688 -> 1313, -375)._ A data-driven `allowedChildren` table (the schema-for-
+   schemas child content model) rejects disallowed children, multi/misplaced
+   `annotation`, and identity constraints missing selector/field, via one general
+   mechanism (`SchemaStructure.swift`). **Still open:** the **attribute model**
+   (which attributes each component admits, and their value types) is the obvious
+   next sibling table and the largest remaining lever; child *order* beyond the
+   leading annotation; and particle / model-group semantic rules (Unique Particle
+   Attribution, Particle-Valid-Restriction completeness).
 
 ## Per-iteration protocol (the PR-critic-loop)
 

docs/xsts-deviations.md

@@ -47,7 +47,7 @@ and to document the remainder as spec-justified, deliberate exclusions.
 
 ## Category A: genuine correctness bugs
 
-### Fixed (seventeen root causes, ~1447 deviations cleared)
+### Fixed (eighteen root causes, ~1822 deviations cleared)
 
 list-datatype length facets (#146) · union pattern/enumeration facets · duration
 partial-order facets · QName length non-constraining · the full XSD `\p{Is...}`
@@ -65,7 +65,7 @@ validity (`min`/`maxInclusive`, `min`/`maxExclusive`, `enumeration` values valid
 in the base space; inclusive/exclusive exclusions; bound ordering) · `gMonth`
 `--MM--` lexical form · schema `id` attribute validity (`xs:ID`: NCName and
 unique within the document). The schema-validity work brought invalid schemas
-accepted 2461 to 1688; see `schema-validity-burndown.md`.
+accepted 2461 to 1313 (also: schema-document structural validity against the schema-for-schemas content model); see `schema-validity-burndown.md`.
 Plus the measurement fix (exclude W3C-disputed `status="queried"` entries).
 
 ### Remaining, root-caused (the hard tail)
@@ -106,12 +106,14 @@ schema-for-schemas. Examples found:
 | `annotation/annotB` | ~62 |
 | `addB`, `stF`, and the long tail | remainder of 2467 |
 
-**Decision:** enforcing this is a large, low-real-world-value effort (people do
-not author structurally broken schemas, and a too-lenient *compiler* does not
-corrupt valid-document validation). It would require a schema-document
-validation pass against the schema-for-schemas. Treated as a **documented,
-deliberate exclusion** unless a concrete need arises. If implemented, it is its
-own epic, not part of the instance-correctness burn-down.
+**Status (revised):** now being implemented as the general schema-for-schemas
+mechanism, not a per-rule pile. The content-model pass has landed (`id` validity,
+facet-definition validity, and the `allowedChildren` structural table), driving
+invalid-schemas-accepted from 2461 to 1313. The remaining structural work (the
+component **attribute model**, child order, particle semantics) continues under
+`schema-validity-burndown.md`. A too-lenient compiler does not corrupt
+valid-document validation, but rubber-stamping invalid schemas is the largest
+conformance gap, so it is being closed rather than excluded.
 
 ## Category C: arguable / needs investigation