fix(schema): enforce substitution group exclusions based on final

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- Enforce substitution group exclusions based on element final and schema finalDefault attributes (XSD 1.0 Structures §3.3.6 / Schema Component Constraint: Substitution Group OK (Transitive)). The entire derivation path from the member type to the head type is checked; if any step contains a derivation method prohibited by the head element's final or the schema's finalDefault, the substitution is rejected. XSTS: invalid schemas accepted 323 to 318, no other bucket raised.
 - Enforce targetNamespace scope matching on local schema references (XSD 1.0 Structures §3.14). If a schema has a targetNamespace declared, any local type, element, attribute, group, or attributeGroup reference without a namespace prefix (resolving to no namespace) is rejected as undeclared unless a default namespace matching the targetNamespace is declared on the element, since all global components defined in the document belong to the targetNamespace. XSTS: invalid schemas accepted 325 to 323, no other bucket raised.
 - Enforce variety constraints on list itemType and union memberTypes (XSD 1.0 Datatypes §3.3.1 / §3.4.1). A list's itemType must be atomic or union of atomic types, and cannot be a list or a union containing a list, and cannot resolve to 'anySimpleType'. Resolves namespace prefixes dynamically in scope to handle local prefix declarations without false positives. XSTS: invalid schemas accepted 327 to 325, no other bucket raised.
 - Implement namespace-aware schema reference resolution checks, validating type references, element/attribute/group references, and substitutionGroup headers using XML namespace prefix bindings (XSD 1.0 Structures §3.14). If a prefix is bound to a namespace that is neither the targetNamespace nor the XMLSchema namespace (when no external schemas are loaded), the reference is flagged as undeclared. Bypasses built-in validation checks on references in the XMLSchema namespace when the targetNamespace is the XMLSchema namespace itself. Validate simpleContent base restrictions namespace-aware. XSTS: invalid schemas accepted 333 to 327, no other bucket raised.

Sources/Schema/SchemaSubstitutionType.swift

@@ -50,11 +50,59 @@ extension PureXML.Schema.XSDParser {
             guard !isListOrUnion(memberType, types), !isListOrUnion(headType, types) else { continue }
             if !PureXML.Schema.ParticleRestriction.typeDerivesOrEqual(memberType, headType, tables.typeDerivation, types) {
                 errors.append("element '\(member)' may not be in the substitution group of '\(head)': its type '\(memberType)' is not derived from '\(headType)'")
+            } else {
+                errors += checkExclusions(
+                    member: member,
+                    memberType: memberType,
+                    head: head,
+                    headType: headType,
+                    tables: tables,
+                )
             }
         }
         return errors
     }
 
+    private static func checkExclusions(
+        member: String,
+        memberType: String,
+        head: String,
+        headType: String,
+        tables: DerivationTables,
+    ) -> [String] {
+        guard let exclusions = tables.elementFinal[head] else { return [] }
+        var errors: [String] = []
+        var currentType = memberType
+        var pathMethods: Set<PureXML.Schema.DerivationMethod> = []
+        while currentType != headType {
+            guard let derivation = tables.typeDerivation[currentType] else {
+                break
+            }
+            pathMethods.insert(derivation.method)
+            currentType = derivation.base
+        }
+        if currentType != headType {
+            let member = PureXML.Schema.BuiltinType(rawValue: currentType)
+            let head = PureXML.Schema.BuiltinType(rawValue: headType)
+            if let member, let head, member.derives(from: head) {
+                pathMethods.insert(.restriction)
+            }
+        }
+        if exclusions.contains(.extension), pathMethods.contains(.extension) {
+            errors.append(
+                "element '\(member)' may not be in the substitution group of '\(head)': "
+                    + "its type '\(memberType)' is derived by extension from '\(headType)' which is excluded by the head element",
+            )
+        }
+        if exclusions.contains(.restriction), pathMethods.contains(.restriction) {
+            errors.append(
+                "element '\(member)' may not be in the substitution group of '\(head)': "
+                    + "its type '\(memberType)' is derived by restriction from '\(headType)' which is excluded by the head element",
+            )
+        }
+        return errors
+    }
+
     /// Whether the named type is a list- or union-variety simple type, whose
     /// derivation rules `typeDerivesOrEqual` does not model. A built-in or unknown
     /// name is treated as atomic.

Sources/Schema/XSDDerivation.swift

@@ -19,6 +19,7 @@ extension PureXML.Schema.XSDParser {
         var typeFinal: [String: Set<PureXML.Schema.DerivationMethod>] = [:]
         var abstractElements: Set<String> = []
         var elementBlock: [String: Set<PureXML.Schema.DerivationMethod>] = [:]
+        var elementFinal: [String: Set<PureXML.Schema.DerivationMethod>] = [:]
         var elementTypeNames: [String: String] = [:]
     }
 
@@ -67,6 +68,32 @@ extension PureXML.Schema.XSDParser {
         if XSDDerivNode.attribute(element, "abstract") == "true" { tables.abstractElements.insert(name) }
         let block = methodSet(XSDDerivNode.attribute(element, "block"))
         if !block.isEmpty { tables.elementBlock[name] = block }
+
+        let isGlobal = element.parent.map {
+            $0.name?.namespaceURI == xsdNamespace && XSDDerivNode.localName($0) == "schema"
+        } ?? false
+        if isGlobal {
+            var schemaFinalDefault: String?
+            var current: XSDTree? = element
+            while let parentSchema = current {
+                if parentSchema.name?.namespaceURI == xsdNamespace, XSDDerivNode.localName(parentSchema) == "schema" {
+                    schemaFinalDefault = XSDDerivNode.attribute(parentSchema, "finalDefault")
+                    break
+                }
+                current = parentSchema.parent
+            }
+
+            let finalAttr = XSDDerivNode.attribute(element, "final")
+            let finalSet: Set<PureXML.Schema.DerivationMethod> = if let finalAttr {
+                methodSet(finalAttr).intersection([.extension, .restriction])
+            } else if let schemaFinalDefault {
+                methodSet(schemaFinalDefault).intersection([.extension, .restriction])
+            } else {
+                []
+            }
+            if !finalSet.isEmpty { tables.elementFinal[name] = finalSet }
+        }
+
         if let type = XSDDerivNode.attribute(element, "type") {
             tables.elementTypeNames[name] = XSDDerivNode.stripPrefix(type)
         }
@@ -335,14 +362,6 @@ extension PureXML.Schema.XSDParser {
         return result
     }
 
-    private static func methodName(_ method: PureXML.Schema.DerivationMethod) -> String {
-        switch method {
-        case .extension: "extension"
-        case .restriction: "restriction"
-        case .substitution: "substitution"
-        }
-    }
-
     /// Throws when any `xs:all` group violates its XSD 1.0 constraints: its members
     /// must all be elements, each occurring at most once, and the group itself may
     /// occur at most once. Nested model groups inside an `all` are not allowed.

Tests/SchemaSubstitutionTypeTests.swift

@@ -85,4 +85,77 @@ struct SchemaSubstitutionTypeTests {
             + "<xs:element name=\"member\" substitutionGroup=\"imp:head\" type=\"xs:int\"/></xs:schema>"
         #expect((try? PureXML.Schema.Document(main, schemaLoader: { $0 == "imp.xsd" ? imported : nil })) != nil)
     }
+
+    @Test("Substitution group member is blocked if type derivation method is final on the head element")
+    func test_finalSubstitutionExclusion() {
+        // Direct extension derivation rejected when final="extension"
+        #expect(!compiles(
+            "<xs:complexType name=\"Base\"><xs:sequence/></xs:complexType>"
+                + "<xs:complexType name=\"Ext\"><xs:complexContent><xs:extension base=\"Base\"><xs:sequence/></xs:extension></xs:complexContent></xs:complexType>"
+                + "<xs:element name=\"head\" type=\"Base\" final=\"extension\"/>"
+                + "<xs:element name=\"member\" substitutionGroup=\"head\" type=\"Ext\"/>",
+        ))
+
+        // Direct restriction derivation accepted when final="extension"
+        #expect(compiles(
+            "<xs:complexType name=\"Base\"><xs:sequence/></xs:complexType>"
+                + "<xs:complexType name=\"Res\"><xs:complexContent><xs:restriction base=\"Base\"><xs:sequence/></xs:restriction></xs:complexContent></xs:complexType>"
+                + "<xs:element name=\"head\" type=\"Base\" final=\"extension\"/>"
+                + "<xs:element name=\"member\" substitutionGroup=\"head\" type=\"Res\"/>",
+        ))
+
+        // Indirect extension derivation rejected when final="extension"
+        #expect(!compiles(
+            "<xs:complexType name=\"Base\"><xs:sequence/></xs:complexType>"
+                + "<xs:complexType name=\"Mid\"><xs:complexContent><xs:extension base=\"Base\"><xs:sequence/></xs:extension></xs:complexContent></xs:complexType>"
+                + "<xs:complexType name=\"Derived\"><xs:complexContent><xs:restriction base=\"Mid\"><xs:sequence/></xs:restriction></xs:complexContent></xs:complexType>"
+                + "<xs:element name=\"head\" type=\"Base\" final=\"extension\"/>"
+                + "<xs:element name=\"member\" substitutionGroup=\"head\" type=\"Derived\"/>",
+        ))
+
+        // Schema finalDefault="extension" rejects extension derivation
+        #expect((try? PureXML.Schema.Document("""
+        <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" finalDefault="extension">
+            <xs:complexType name="Base"><xs:sequence/></xs:complexType>
+            <xs:complexType name="Ext"><xs:complexContent><xs:extension base="Base"><xs:sequence/></xs:extension></xs:complexContent></xs:complexType>
+            <xs:element name="head" type="Base"/>
+            <xs:element name="member" substitutionGroup="head" type="Ext"/>
+        </xs:schema>
+        """)) == nil)
+
+        // final="#all" rejects both extension and restriction
+        #expect(!compiles(
+            "<xs:complexType name=\"Base\"><xs:sequence/></xs:complexType>"
+                + "<xs:complexType name=\"Ext\"><xs:complexContent><xs:extension base=\"Base\"><xs:sequence/></xs:extension></xs:complexContent></xs:complexType>"
+                + "<xs:element name=\"head\" type=\"Base\" final=\"#all\"/>"
+                + "<xs:element name=\"member\" substitutionGroup=\"head\" type=\"Ext\"/>",
+        ))
+    }
+
+    @Test("Substitution group member final exclusions: local name collisions and built-ins")
+    func test_finalSubstitutionExclusionEdgeCases() {
+        // Local element name collision: local element does not overwrite global final setting
+        #expect((try? PureXML.Schema.Document("""
+        <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" finalDefault="extension">
+            <xs:element name="head" type="Base" final="restriction"/>
+            <xs:element name="member" substitutionGroup="head" type="Ext"/>
+            <xs:complexType name="Base"><xs:sequence/></xs:complexType>
+            <xs:complexType name="Ext">
+                <xs:complexContent>
+                    <xs:extension base="Base">
+                        <xs:sequence>
+                            <xs:element name="head" type="xs:string"/>
+                        </xs:sequence>
+                    </xs:extension>
+                </xs:complexContent>
+            </xs:complexType>
+        </xs:schema>
+        """)) != nil)
+
+        // Built-in simple type restriction rejected when final="restriction"
+        #expect(!compiles(
+            "<xs:element name=\"head\" type=\"xs:decimal\" final=\"restriction\"/>"
+                + "<xs:element name=\"member\" substitutionGroup=\"head\" type=\"xs:int\"/>",
+        ))
+    }
 }

Tests/XSTSSuiteTests.swift

@@ -29,7 +29,7 @@ struct XSTSSuiteTests {
     /// XSTS` (debug is far slower and the corpus is large). Per-case deviations
     /// are written to /tmp/xsts-failures.txt for the burn-down.
     private let knownSchemaValidRejected = 1
-    private let knownSchemaInvalidAccepted = 325
+    private let knownSchemaInvalidAccepted = 318
     private let knownInstanceValidRejected = 171
     private let knownInstanceInvalidAccepted = 155
 

docs/schema-validity-burndown.md

@@ -10,7 +10,7 @@ irreducible tail. Do not deviate to new features while this is open.
 | Bucket | Count | Reading |
 |---|---|---|
 | valid-schemas-rejected | **72 -> 4** | production target is 0: rejecting a correct schema is the most user-hostile defect (#148). Particle-restriction over-rejection fixed: pointless `maxOccurs=0` particles removed, content-free restriction of an EMPTY base, empty-derived emptiable-base, and (#163) the pairwise check over the normalized content model with occurrence subsumption applied only to same-kind pairings (cross-kind element/group, group/element, group/wildcard match occurrence inside, by effective total range / member). #164 resolved element-ref namespaces by prefix binding (the `particlesJj/Jk/Jm/Jn/Js`, `particlesQ/R` cluster, -28). Content-free (`maxOccurs=0`) derived groups and substitution-group declaration order closed `particlesW006`/`elemZ027_a`; the W3C MapAndSum rule (Sequence:Choice, #165) closed `particlesV003`. Remaining 1 (`particlesZ001`): a documented spec ambiguity (RecurseAsIfGroup; the reference rejects it), not a defect. |
-| **invalid-schemas-accepted** | **2461 -> 323** | we rarely catch a bad one (facet, default/fixed value, particle-restriction name+type, id, structural content-model, UPA determinism, content order, identity-constraint XPath-subset, wildcard-namespace, final/block, simpleType content, group/wildcard and group/element cardinality, element-ref namespace, sequence/choice MapAndSum, circular type derivation, circular group/attributeGroup/substitution references, attribute-use uniqueness and single-ID, type-excludes-inline-type, ID-typed value-constraint exclusion, all-group reference placement, substitution-member type derivation, default/fixed mutual exclusion, elementTypeRestrictionOK NameAndTypeOK union checks, prohibited attribute type-check bypass, keyref arity verification, top-level group/attributeGroup local-attribute checks, simpleType restriction structure and prefix binding checks, namespace-aware reference resolution checks, simpleType variety constraints on list itemType and union memberTypes, targetNamespace scope matching on local schema references landed). |
+| **invalid-schemas-accepted** | **2461 -> 318** | we rarely catch a bad one (facet, default/fixed value, particle-restriction name+type, id, structural content-model, UPA determinism, content order, identity-constraint XPath-subset, wildcard-namespace, final/block, simpleType content, group/wildcard and group/element cardinality, element-ref namespace, sequence/choice MapAndSum, circular type derivation, circular group/attributeGroup/substitution references, attribute-use uniqueness and single-ID, type-excludes-inline-type, ID-typed value-constraint exclusion, all-group reference placement, substitution-member type derivation, default/fixed mutual exclusion, elementTypeRestrictionOK NameAndTypeOK union checks, prohibited attribute type-check bypass, keyref arity verification, top-level group/attributeGroup local-attribute checks, simpleType restriction structure and prefix binding checks, namespace-aware reference resolution checks, simpleType variety constraints on list itemType and union memberTypes, targetNamespace scope matching on local schema references, substitution group exclusions based on element final and schema finalDefault attributes landed). |
 | valid-instances-rejected | **233 -> 171** | (instance side; #164's element-ref namespace fix is shared with instance validation, complexContent restriction attributes inheritance and namespace fixes) |
 | invalid-instances-accepted | **160 -> 155** | (instance side; resolving inline `<simpleType>` restriction base, complexContent restriction attributes prohibition) |
 

docs/xsts-deviations.md

@@ -34,7 +34,7 @@ entries removed about 70 apparent deviations that were never bugs.
 | Kind | First measurement | Now |
 |---|---|---|
 | valid schema rejected | 75 | 1 |
-| invalid schema accepted | 2467 | 323 |
+| invalid schema accepted | 2467 | 318 |
 | valid instance rejected | 604 | 171 |
 | invalid instance accepted | 582 | 155 |
 
@@ -65,7 +65,7 @@ validity (`min`/`maxInclusive`, `min`/`maxExclusive`, `enumeration` values valid
 in the base space; inclusive/exclusive exclusions; bound ordering) · `gMonth`
 `--MM--` lexical form · schema `id` attribute validity (`xs:ID`: NCName and
 unique within the document). The schema-validity work brought invalid schemas
-accepted 2461 to 884 (schema-document structural validity: content model + complexType content shape, attribute values, name/reference + pattern-regex lexical validity, and reference resolution, against the schema-for-schemas, plus simpleType variety constraints on list itemType and union memberTypes, plus targetNamespace scope matching on local schema references); see `schema-validity-burndown.md`.
+accepted 2461 to 318 (schema-document structural validity: content model + complexType content shape, attribute values, name/reference + pattern-regex lexical validity, and reference resolution, against the schema-for-schemas, plus simpleType variety constraints on list itemType and union memberTypes, plus targetNamespace scope matching on local schema references, plus substitution group exclusions based on element final and schema finalDefault attributes); see `schema-validity-burndown.md`.
 Plus the measurement fix (exclude W3C-disputed `status="queried"` entries).
 
 ### Remaining, root-caused (the hard tail)