From ed38a76acd4df60f8ce39bead658453693d2f949 Mon Sep 17 00:00:00 2001 From: Mihaela Mihaljevic Date: Thu, 18 Jun 2026 22:01:01 +0200 Subject: [PATCH] Reject invalid simpleContent inline restrictions --- CHANGELOG.md | 2 + Sources/Schema/SchemaCompileValidation.swift | 1 + Sources/Schema/SchemaExtensionAllGroup.swift | 33 ++ Sources/Schema/XSDParser.swift | 31 -- Sources/Schema/XSDSimpleContentParser.swift | 66 ++++ ...pleContentRestrictionInlineTypeTests.swift | 61 +++ Tests/XSTSDifferentialTests.swift | 34 +- Tests/XSTSSuiteTests.swift | 2 +- docs/plans/2026-06-18-false-negative-tail.md | 347 ++++++++++++++++++ docs/release-hurdles.md | 8 +- 10 files changed, 536 insertions(+), 49 deletions(-) create mode 100644 Sources/Schema/XSDSimpleContentParser.swift create mode 100644 Tests/SchemaSimpleContentRestrictionInlineTypeTests.swift create mode 100644 docs/plans/2026-06-18-false-negative-tail.md diff --git a/CHANGELOG.md b/CHANGELOG.md index edee316..55d090c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -21,6 +21,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- XSTS invalid-schemas-accepted (#145): a `simpleContent` restriction that supplies an inline `simpleType` must use a type that is itself a valid restriction of the base complex type's simple content. A list of `xs:int` is not a restriction of an atomic `xs:decimal` content type, so particlesZ018 is now rejected. Valid inline atomic restrictions become the effective simpleContent type for instance validation, including facets declared beside the inline type. invalid-schemas-accepted 48 → 47, no false positive (valid-schemas-rejected and valid-instances-rejected held at 0), no other bucket moved. + - XSTS invalid-schemas-accepted (#145): a constraining facet may not be applied where a `simpleContent` restriction's base chain resolves to `xs:anySimpleType`, the ur simple type, which carries no constraining facets (XSD Part 2 §3.2.1); stZ010 (a complex type extending `xs:anySimpleType`, restricted by another that adds `minLength`) is now rejected. The base chain is followed through local complex types' own `simpleContent` bases; a base that bottoms at any other built-in, a simple type, or an external/unresolved type is left alone, and a restriction supplying its own nested `simpleType` base is skipped (the facet applies to that), so a facet on a string-, int-, or other real-typed simpleContent stays accepted. invalid-schemas-accepted 49 → 48, no false positive (valid-schemas-rejected and valid-instances-rejected held at 0), no other bucket moved. - XSTS invalid-schemas-accepted (#145, #161): a redefinition of an attribute group may not ADD an attribute the original does not declare (a superset is not a restriction, src-redefine / cos-ct-restricts); schM3 (redeclaring the original's attributes plus a new `b4`) is now rejected. The base is resolved by namespaced identity with a `{}name` chameleon fallback (a no-targetNamespace redefined schema is absorbed into the redefining namespace). To stay false-positive-free the added-attribute check runs only when the original's attribute set is fully known from its direct declarations (the original has no nested `attributeGroup` reference) AND the redefinition itself carries no `attributeGroup` reference: a self-reference brings in the original's attributes, so declaring further ones alongside it is a valid redefine (W3C attgC007/attgC038/groupB018, and test75083/xsd003b), not an addition. invalid-schemas-accepted 51 → 49, no false positive (valid-schemas-rejected and valid-instances-rejected held at 0; five self-reference over-rejections surfaced by the gate were fixed before commit), no other bucket moved. diff --git a/Sources/Schema/SchemaCompileValidation.swift b/Sources/Schema/SchemaCompileValidation.swift index ecde1f4..404a995 100644 --- a/Sources/Schema/SchemaCompileValidation.swift +++ b/Sources/Schema/SchemaCompileValidation.swift @@ -376,6 +376,7 @@ extension PureXML.Validation.SchemaCompile { PureXML.Schema.XSDParser.simpleContentExtensionBaseErrors(document.schema, document.context, namedTypes) + PureXML.Schema.XSDParser.extensionMixedAgreementErrors(document.schema, document.context, namedTypes) + PureXML.Schema.XSDParser.simpleContentRestrictionBaseErrors(document.schema, document.context, namedTypes) + + PureXML.Schema.XSDParser.simpleContentRestrictionTypeErrors(document.schema, document.context, namedTypes) + PureXML.Schema.XSDParser.complexContentBaseKindErrors(document.schema, document.context, namedTypes) + PureXML.Schema.XSDParser.simpleContentExtensionBaseKindErrors(document.schema, document.context, namedTypes) + PureXML.Schema.XSDParser.elementValueConstraintContentErrors(document.schema, document.context, namedTypes), diff --git a/Sources/Schema/SchemaExtensionAllGroup.swift b/Sources/Schema/SchemaExtensionAllGroup.swift index fad60bf..c8dc4b6 100644 --- a/Sources/Schema/SchemaExtensionAllGroup.swift +++ b/Sources/Schema/SchemaExtensionAllGroup.swift @@ -138,6 +138,39 @@ extension PureXML.Schema.XSDParser { return errors } + /// XSD 1.0 Derivation Valid (Restriction, Simple): when a `simpleContent` + /// restriction supplies an inline `simpleType`, that inline type must itself be + /// a valid restriction of the base complex type's simple content type. A list + /// of `xs:int`, for example, is not a restriction of `xs:decimal` because it + /// changes the variety from atomic to list (W3C msData particlesZ018). + /// + /// Only local, resolved complex bases with simple content are checked. Built-in, + /// foreign, unresolved, and composed bases are left to the surrounding base-kind + /// and cross-document rules so this cannot reject a schema whose base type is + /// unknown to this compile pass. + static func simpleContentRestrictionTypeErrors(_ schema: XSDTree, _ context: PureXML.Schema.XSDContext, _ types: [String: PureXML.Schema.ElementType]) -> [String] { + guard !skipsCrossDocumentRules(schema, compositionLoaded: context.compositionLoaded) else { return [] } + var errors: [String] = [] + for content in descendants(schema, named: "simpleContent") { + guard let restriction = ExtAllNode.firstChild(content, named: "restriction"), + let inline = ExtAllNode.firstChild(restriction, named: "simpleType"), + let base = ExtAllNode.attribute(restriction, "base") + else { continue } + let bindings = namespaceBindingsInScope(of: restriction, defaultBindings: context.namespaceBindings) + let namespace = ExtAllNode.referenceNamespace(base, bindings) + let baseName = ExtAllNode.stripPrefix(base) + guard namespace == context.targetNamespace, + case let .complex(complex)? = types[baseName], + case let .simpleContent(baseSimpleType) = complex.content + else { continue } + let derivedSimpleType = scopedSimpleType(inline, context) + if !isSimpleTypeRestrictionOK(derived: derivedSimpleType, base: baseSimpleType) { + errors.append("the inline simpleType in a simpleContent restriction is not a valid restriction of base type '\(baseName)'") + } + } + return errors + } + /// XSD 1.0 `src-ct.1`: the base of a `complexContent` derivation (restriction or /// extension) must be a COMPLEX type. The ur-type `xs:anyType` is complex and is /// the legitimate base of most complex types; every other XSD built-in, and a diff --git a/Sources/Schema/XSDParser.swift b/Sources/Schema/XSDParser.swift index 5399026..231ebae 100644 --- a/Sources/Schema/XSDParser.swift +++ b/Sources/Schema/XSDParser.swift @@ -366,35 +366,4 @@ extension PureXML.Schema.XSDParser { } return ComplexType(attributes: attributes, attributeWildcard: wildcard, content: mixed ? .mixed(particle) : .elementOnly(particle)) } - - private static func simpleContentType(_ node: XSDTree, _ context: XSDContext, visited: Set = []) -> SimpleType { - guard let inner = derivation(node) else { return SimpleType(base: .string) } - let rawBase = XSDNode.attribute(inner, "base") ?? "string" - let baseName = XSDNode.stripPrefix(rawBase) - let bindings = PureXML.Schema.XSDParser.namespaceBindingsInScope(of: inner, defaultBindings: context.namespaceBindings) - let uri = XSDNode.referenceNamespace(rawBase, bindings) - - let baseType = simpleContentBaseType(baseName, uri: uri, context, visited: visited) - var facets = baseType.facets - XSDSimpleParser.applyFacets(inner, into: &facets) - return SimpleType(base: baseType.base, facets: facets, variety: baseType.variety, isBuiltinList: baseType.isBuiltinList) - } - - /// The simple type a `simpleContent` derivation derives from: an XSD built-in, a - /// named simple type, or, when the base is another complex type with simpleContent, - /// that type's own resolved simple type (so `Test` restricting `Test2` extending - /// `xs:int` resolves to `int`). A cycle in the base chain falls back to `string`. - private static func simpleContentBaseType(_ baseName: String, uri: String?, _ context: XSDContext, visited: Set) -> SimpleType { - if uri == PureXML.Schema.XSDParser.xsdNamespace { - return SimpleType(base: BuiltinType(rawValue: baseName) ?? .string) - } - if let simple = context.simpleTypes[baseName] { - return simple - } - guard !visited.contains(baseName), - let complexNode = context.complexTypeNodes[baseName], - let baseSimpleContent = XSDNode.firstChild(complexNode, named: "simpleContent") - else { return SimpleType(base: .string) } - return simpleContentType(baseSimpleContent, context, visited: visited.union([baseName])) - } } diff --git a/Sources/Schema/XSDSimpleContentParser.swift b/Sources/Schema/XSDSimpleContentParser.swift new file mode 100644 index 0000000..0c6140f --- /dev/null +++ b/Sources/Schema/XSDSimpleContentParser.swift @@ -0,0 +1,66 @@ +private typealias SimpleContentNode = PureXML.Schema.XSDNode +private typealias SimpleContentType = PureXML.Schema.SimpleType +private typealias SimpleContentContext = PureXML.Schema.XSDContext +private typealias SimpleContentBuiltinType = PureXML.Schema.BuiltinType + +extension PureXML.Schema.XSDParser { + static func simpleContentType( + _ node: XSDTree, + _ context: PureXML.Schema.XSDContext, + visited: Set = [], + ) -> PureXML.Schema.SimpleType { + guard let inner = derivation(node) else { return SimpleContentType(base: .string) } + let rawBase = SimpleContentNode.attribute(inner, "base") ?? "string" + let baseName = SimpleContentNode.stripPrefix(rawBase) + let bindings = namespaceBindingsInScope(of: inner, defaultBindings: context.namespaceBindings) + let uri = SimpleContentNode.referenceNamespace(rawBase, bindings) + + let baseType = simpleContentBaseType(baseName, uri: uri, context, visited: visited) + let effectiveType: SimpleContentType + let inlineType = SimpleContentNode.localName(inner) == "restriction" + ? SimpleContentNode.firstChild(inner, named: "simpleType") + : nil + if let inline = inlineType { + effectiveType = scopedSimpleType(inline, context) + } else { + effectiveType = baseType + } + var facets = effectiveType.facets + let declaresEnumeration = SimpleContentNode.elementChildren(inner) + .contains { SimpleContentNode.localName($0) == "enumeration" } + if SimpleContentNode.localName(inner) == "restriction", declaresEnumeration { + facets.enumeration = nil + } + PureXML.Schema.XSDSimpleParser.applyFacets(inner, into: &facets) + return SimpleContentType(base: effectiveType.base, facets: facets, variety: effectiveType.variety, isBuiltinList: effectiveType.isBuiltinList) + } + + static func scopedSimpleType(_ node: XSDTree, _ context: PureXML.Schema.XSDContext) -> PureXML.Schema.SimpleType { + var scoped = context + scoped.namespaceBindings = namespaceBindingsInScope(of: node, defaultBindings: context.namespaceBindings) + return PureXML.Schema.XSDSimpleParser.simpleType(node, scoped) + } + + /// The simple type a `simpleContent` derivation derives from: an XSD built-in, a + /// named simple type, or, when the base is another complex type with + /// simpleContent, that type's own resolved simple type. A cycle in the base + /// chain falls back to `string`. + private static func simpleContentBaseType( + _ baseName: String, + uri: String?, + _ context: SimpleContentContext, + visited: Set, + ) -> SimpleContentType { + if uri == PureXML.Schema.XSDParser.xsdNamespace { + return SimpleContentType(base: SimpleContentBuiltinType(rawValue: baseName) ?? .string) + } + if let simple = context.simpleTypes[baseName] { + return simple + } + guard !visited.contains(baseName), + let complexNode = context.complexTypeNodes[baseName], + let baseSimpleContent = SimpleContentNode.firstChild(complexNode, named: "simpleContent") + else { return SimpleContentType(base: .string) } + return simpleContentType(baseSimpleContent, context, visited: visited.union([baseName])) + } +} diff --git a/Tests/SchemaSimpleContentRestrictionInlineTypeTests.swift b/Tests/SchemaSimpleContentRestrictionInlineTypeTests.swift new file mode 100644 index 0000000..945ad9e --- /dev/null +++ b/Tests/SchemaSimpleContentRestrictionInlineTypeTests.swift @@ -0,0 +1,61 @@ +import Testing +@testable import PureXML + +@Suite("simpleContent restriction inline simpleType") +struct SimpleContentInlineTypeTests { + @Test("an inline list simpleType is not a restriction of an atomic decimal simpleContent base") + func test_listInlineTypeOverDecimalBaseRejected() { + let schema = """ + + + + + + + + + + + + + + + + + + + """ + + #expect((try? PureXML.Schema.Document(schema)) == nil) + } + + @Test("a valid inline atomic restriction becomes the effective simpleContent type") + func test_validInlineTypeRestrictsInstanceValues() throws { + let schema = try PureXML.Schema.Document(""" + + + + + + + + + + + + + + + + + + + + + """) + + #expect(try schema.validate(#"5"#).isEmpty) + #expect(try !schema.validate(#"1.5"#).isEmpty) + #expect(try !schema.validate(#"11"#).isEmpty) + } +} diff --git a/Tests/XSTSDifferentialTests.swift b/Tests/XSTSDifferentialTests.swift index d19e16a..47f29f4 100644 --- a/Tests/XSTSDifferentialTests.swift +++ b/Tests/XSTSDifferentialTests.swift @@ -77,7 +77,11 @@ struct XSTSDifferentialTests { // MARK: - libxml2 oracle private func xmllintAvailable() -> Bool { - FileManager.default.isExecutableFile(atPath: "/usr/bin/xmllint") + #if os(WASI) + return false + #else + FileManager.default.isExecutableFile(atPath: "/usr/bin/xmllint") + #endif } /// libxml2's schema-validity verdict for one `.xsd`, or nil if it could not be @@ -86,18 +90,22 @@ struct XSTSDifferentialTests { /// empty) always fails to parse, which is ignored -- only schema-compile errors /// count toward schema validity. private func xmllintSchemaValid(_ path: String) -> Bool? { - let process = Process() - process.executableURL = URL(fileURLWithPath: "/usr/bin/xmllint") - process.arguments = ["--noout", "--schema", path, "/dev/null"] - let pipe = Pipe() - process.standardError = pipe - process.standardOutput = Pipe() - do { try process.run() } catch { return nil } - let data = pipe.fileHandleForReading.readDataToEndOfFile() - process.waitUntilExit() - let output = String(decoding: data, as: UTF8.self) - if output.contains("Schemas parser error") || output.contains("failed to compile") { return false } - return true + #if os(WASI) + return nil + #else + let process = Process() + process.executableURL = URL(fileURLWithPath: "/usr/bin/xmllint") + process.arguments = ["--noout", "--schema", path, "/dev/null"] + let pipe = Pipe() + process.standardError = pipe + process.standardOutput = Pipe() + do { try process.run() } catch { return nil } + let data = pipe.fileHandleForReading.readDataToEndOfFile() + process.waitUntilExit() + let output = String(decoding: data, as: UTF8.self) + if output.contains("Schemas parser error") || output.contains("failed to compile") { return false } + return true + #endif } private func xsdFiles(under directory: String) -> [String] { diff --git a/Tests/XSTSSuiteTests.swift b/Tests/XSTSSuiteTests.swift index 4228bda..6bbfe3d 100644 --- a/Tests/XSTSSuiteTests.swift +++ b/Tests/XSTSSuiteTests.swift @@ -148,7 +148,7 @@ struct XSTSSuiteTests { /// and the cross-document composition family (src-resolve, src-import.3, /// src-redefine.5/6.1/7.2.1, redefine cross-namespace, required schemaLocation). private let knownSchemaValidRejected = 0 - private let knownSchemaInvalidAccepted = 48 + private let knownSchemaInvalidAccepted = 47 private let knownInstanceValidRejected = 0 /// Instance bucket (133 -> 31), per-step deltas in CHANGELOG.md: xsi:type must derive from the /// declared type (complex, then atomic and list/union, resolved through the element-ref chain); diff --git a/docs/plans/2026-06-18-false-negative-tail.md b/docs/plans/2026-06-18-false-negative-tail.md new file mode 100644 index 0000000..e3c1eb6 --- /dev/null +++ b/docs/plans/2026-06-18-false-negative-tail.md @@ -0,0 +1,347 @@ +# XSTS false-negative tail: clause-by-clause closure + +**Goal:** Drive the remaining false-negative buckets to zero without ever +raising either false-positive bucket. +**Approved design:** Current-session design: freeze the 79-case tail, split it by +owning oracle, add adversarial valid fixtures before tightening each rule, and +ratchet one clause at a time. + +## Current measurement + +Measured from `/tmp/xsts-failures.txt`, last written 2026-06-18 21:31 local time +by `XSTSSuiteTests`. + +```sh +awk -F': ' '/invalid schema accepted/ {schema++} /invalid instance accepted/ {inst++} /valid schema rejected/ {vs++} /valid instance rejected/ {vi++} END {print "valid schema rejected", vs+0; print "invalid schema accepted", schema+0; print "valid instance rejected", vi+0; print "invalid instance accepted", inst+0}' /tmp/xsts-failures.txt +``` + +Result: + +| Bucket | Count | +|---|---:| +| valid schema rejected | 0 | +| invalid schema accepted | 47 | +| valid instance rejected | 0 | +| invalid instance accepted | 31 | + +The false-positive buckets are already zero. That is the non-negotiable gate: +every task below must keep them at zero. + +Progress since the starting 48/31 snapshot: `particlesZ018` is closed by the +simpleContent inline type-restriction rule, dropping the schema false-negative +bucket by one with both false-positive buckets still at zero. + +## File map + +| Path | Change | Notes | +|---|---|---| +| `docs/plans/2026-06-18-false-negative-tail.md` | new | This plan and the seed manifest. | +| `Tests/XSTSSuiteTests.swift` | edit per fix | Ratchet only after a full XSTS run proves a count fell and no other bucket rose. | +| `CHANGELOG.md` | edit per fix | Record the exact before/after bucket movement and false-positive result. | +| `docs/release-hurdles.md` | edit per phase | Keep the release blocker summary current. | +| `docs/xsts-deviations.md` | edit per phase | Move cases between fixable, disclosed, and closed categories. | +| `docs/schema-validity-burndown.md` | edit per schema fix | Preserve the historical burndown and rule-family status. | +| `Sources/Schema/ParticleRestriction.swift` | likely edit | High-fan-in particle restriction oracle. | +| `Sources/Schema/ParticleRestrictionElement.swift` | likely edit | NameAndTypeOK type-derivation and union-member clauses. | +| `Sources/Schema/ParticleRestrictionHelpers.swift` | likely edit | Fixed value, wildcard strength, range helpers. | +| `Sources/Schema/ContentModelDeterminism.swift` | likely edit | UPA, especially wildcard overlap. | +| `Sources/Schema/SchemaRedefineSelfReference.swift` | likely edit | Redefine self-reference and cross-container restriction rules. | +| `Sources/Schema/SchemaReferences.swift` | likely edit | Include/import/redefine reference and composition validity. | +| `Sources/Schema/SchemaSubstitutionType.swift` | likely edit | Substitution-group member derivation. | +| `Sources/Schema/XSDSubstitutionMembers.swift` | likely edit | Instance substitution-group filtering. | +| `Sources/Schema/IdentityValidator*.swift` | likely edit | Remaining identity-constraint instance false negatives. | +| `Sources/Schema/SimpleType.swift` | likely edit | Simple-type derivation, regex, and value-space edges. | +| `Sources/Regex/*` | likely edit | Regex cases that are genuine errors, not disclosed engine gaps. | +| `Tests/XSDParticleRestrictionTests.swift` | edit per particle fix | Add invalid and adversarial valid fixtures. | +| `Tests/ContentModelDeterminismTests.swift` | edit per UPA fix | Add wildcard ambiguity and valid deterministic controls. | +| Existing `Tests/Schema*Tests.swift` suites | edit per fix | Add focused tests near the owning rule. | + +## Seed manifest + +Every row is a current accepted-invalid case. The lane is a starting hypothesis, +not a proof. Before code, read the XSTS fixture, the matching PureXMLResearch +reference, and the XSD 1.0 rule text; update the lane if the evidence says this +classification is wrong. + +### Invalid schemas accepted: 47 + +| Lane | Cases | First owning surface | +|---|---|---| +| Schema-for-schemas / XSD self-schema edge | `xsd013`, `xsd014` | `SchemaStructure`, `SchemaReferences` | +| Additional-suite element/substitution triage | `addB009`, `addB177` | `SchemaSubstitutionType`, `SchemaReferences` | +| Attribute group restriction triage | `attgC028` | `SchemaAttributeRestriction`, redefine machinery | +| Attribute type/use triage | `attKa015`, `attKb018a` | `SchemaAttributeApplicability`, `SchemaAttributeRestriction` | +| Attribute identity/composition triage | `attQ011`, `attQ016`, `attQ017`, `attQ018` | Attribute restrictions and per-document composition | +| Complex/simple derivation edge | `ctZ010d` | `SchemaStructureDerivation`, `SimpleType` | +| Substitution-group head/member modeling | `elemZ026`, `elemZ027_c`, `elemZ028e` | `SchemaSubstitutionType`, `XSDSubstitutionMembers` | +| ParticleRestriction exactness | `particlesEa025`, `particlesFb003`, `particlesHa161`, `particlesHb011`, `particlesIg004`, `particlesIj008`, `particlesIk011`, `particlesIk025`, `particlesIk027`, `particlesK006`, `particlesM034`, `particlesV020` | `ParticleRestriction*` | +| UPA / wildcard determinism | `particlesZ022`, `particlesZ030_d`, `particlesZ039` | `ContentModelDeterminism` | +| Regex disclosed-vs-fixable triage | `reK88`, `RegexTest_993`, `RegexTest_1477` | `Sources/Regex/*`, `SimpleType` | +| Redefine / composition | `schG10`, `schM4`, `schM8`, `schN10`, `schN12`, `schZ011_a`, `schZ011_b`, `schZ011_c`, `schZ011_d` | `SchemaReferences`, `SchemaRedefineSelfReference`, `ParticleRestriction` | +| Simple-type validity | `stZ048` | `SchemaSimpleTypeValidity`, `SimpleType` | +| Wildcard UPA | `wildI009`, `wildI013`, `wildI014`, `wildZ013` | `ContentModelDeterminism` | + +### Invalid instances accepted: 31 + +| Lane | Cases | First owning surface | +|---|---|---| +| Target namespace / form matching | `targetns00101m/targetNS00101m1_n` | `ComplexValidator*`, reference resolution | +| Disallowed substitution | `disallowedsubst00105m/disallowedSubst00105m1_n`, `disallowedsubst00106m2/Negative` | `XSDSubstitutionMembers`, `ComplexValidatorXSIType` | +| Nillability | `nillable00201m/nillable00201m2_n` | `XSDParserHelpers`, instance validator | +| Type derivation | `typedef00204m/typeDef00204m1_n` | `ComplexValidatorXSIType`, derivation tables | +| Additional-suite instance triage | `addB065/addB065.i` | to classify from fixture | +| Datatype value-space edges | `anyURI_a004_1339/anyURI_a004_1339.i`, `dateTime011_2008/dateTime011_2008.i` | `SimpleType` | +| Element constraints | `elemO011/elemO011.i`, `elemT074/elemT074.i` | instance validator, element declarations | +| Identity constraints | `idG006/idG006.i`, `idK012/idK012.i`, `idZ010/idZ010.i`, `idZ012/idZ012.i` | `IdentityValidator*` | +| Model group instance matching | `mgO013/mgO013.i`, `mgZ001/mgZ001.i` | `ContentMatcher`, `ParticleRestriction` fallout | +| Known particle under-rejection fallout | `particlesZ001/particlesZ001.i` | `ParticleRestriction` | +| Regex instance edges | `reT17/reT17.i`, `reT38/reT38.i`, `RegexTest_422/RegexTest_422.i`, `RegexTest_430/RegexTest_430.i`, `reZ006i/reZ006i.i` | `Sources/Regex/*`, `SimpleType` | +| Schema composition instance fallout | `schA2/schA2.i`, `schA5/schA5.i`, `schA7/schA7.i`, `schU3/schU3.i`, `schU4/schU4.i`, `schU5/schU5.i` | schema composition and instance validator | +| Simple-type instance edge | `stZ056/stZ056.i` | `SimpleType` | +| Wildcard instance matching | `wildZ013a/wildZ013a.i`, `wildZ013d/wildZ013d.i` | `ContentMatcher`, wildcard declaration lookup | + +## Tasks (ordered) + +### T1. Promote the manifest into the working loop + +**Files:** `docs/plans/2026-06-18-false-negative-tail.md`, +`docs/xsts-deviations.md` + +**Does:** Keep the 79-case seed manifest current as work lands. Every case is +classified exactly once as open-fixable, open-disclosed, closed, or reclassified +with reason. If a new XSTS run changes the failure list, update the manifest in +the same change that ratchets the baseline. + +**Verifies:** + +```sh +awk -F': ' '/invalid schema accepted/ {schema++} /invalid instance accepted/ {inst++} /valid schema rejected/ {vs++} /valid instance rejected/ {vi++} END {print vs+0, schema+0, vi+0, inst+0}' /tmp/xsts-failures.txt +``` + +**Commit:** `docs(xsts): freeze false-negative tail manifest` + +### T2. Close low-fan-in non-particle cases first + +**Files:** likely `Sources/Schema/SimpleType.swift`, `Sources/Regex/*`, +`Sources/Schema/SchemaSimpleTypeValidity.swift`, focused `Tests/Schema*Tests.swift` + +**Does:** Start with cases that do not route through `ParticleRestriction`: +`stZ048`, `ctZ010d`, datatype instance cases, and regex cases. `particlesZ018` +is closed by the first pass. +For regex, first decide whether each case is a genuine unsupported-pattern +under-rejection or a spec-backed error. Disclosed cases must be documented, not +silently left in the bucket. + +**Verifies:** + +```sh +swift test --filter Schema +XSTS_ROOT=/private/tmp/xsts/xmlschema2006-11-06 swift test -c release --filter XSTS +``` + +**Commit:** `fix(schema): close simple-type false-negative tail` + +### T3. Add wildcard UPA without changing particle restriction + +**Files:** `Sources/Schema/ContentModelDeterminism.swift`, +`Tests/ContentModelDeterminismTests.swift` + +**Does:** Handle wildcard overlap for `wildI*`, `wildZ013`, `particlesZ022`, +`particlesZ030_d`, and `particlesZ039`. Add adversarial valid content models: +disjoint wildcards, element plus non-overlapping wildcard, same-source wildcard +cases that remain deterministic, and namespace-qualified variants. + +**Verifies:** + +```sh +swift test --filter ContentModelDeterminismTests +XSTS_ROOT=/private/tmp/xsts/xmlschema2006-11-06 swift test -c release --filter XSTS +``` + +**Commit:** `fix(schema): detect wildcard UPA ambiguity` + +### T4. Separate substitution-group modeling from particle exactness + +**Files:** `Sources/Schema/SchemaSubstitutionType.swift`, +`Sources/Schema/XSDSubstitutionMembers.swift`, +`Sources/Schema/ParticleRestrictionElement.swift`, +`Tests/SchemaSubstitutionTypeTests.swift`, +`Tests/SchemaSubstitutionBlockTests.swift` + +**Does:** Model the `elemZ026`/`elemZ027_c`/`elemZ028e` schema failures and +the `disallowedsubst*` instance failures without changing the general +MapAndSum/RecurseAsIfGroup logic. The key safety check is that a substitution +head/member graph must be namespace-exact and must not conflate same-local names. + +**Verifies:** + +```sh +swift test --filter SchemaSubstitution +XSTS_ROOT=/private/tmp/xsts/xmlschema2006-11-06 swift test -c release --filter XSTS +``` + +**Commit:** `fix(schema): model substitution group restriction edges` + +### T5. Tighten NameAndTypeOK union and type clauses + +**Files:** `Sources/Schema/ParticleRestrictionElement.swift`, +`Sources/Schema/ParticleRestrictionHelpers.swift`, +`Tests/XSDParticleRestrictionTests.swift` + +**Does:** Target `particlesIg004`, `particlesIj008`, and +`particlesIk011`/`025`/`027`. Before code, write valid controls where a +restriction type derives from a valid union member and must still compile. +Do not replace union membership with a simple chain walk; that is the known +false-positive trap. + +**Verifies:** + +```sh +swift test --filter XSDParticleRestrictionTests +XSTS_ROOT=/private/tmp/xsts/xmlschema2006-11-06 swift test -c release --filter XSTS +``` + +**Commit:** `fix(schema): enforce NameAndTypeOK union restriction` + +### T6. Make one ParticleRestriction clause exact at a time + +**Files:** `Sources/Schema/ParticleRestriction.swift`, +`Sources/Schema/ParticleRestrictionHelpers.swift`, +`Tests/XSDParticleRestrictionTests.swift` + +**Does:** Work in this order, stopping after each clause for a full XSTS gate: + +1. `MapAndSum` Sequence:Choice: `particlesV020`. +2. `RecurseAsIfGroup` Elt:All/Elt:Sequence: `particlesK006`, `particlesM034`. +3. Remaining element/group and group/wildcard edge cases: + `particlesEa025`, `particlesFb003`, `particlesHa161`, `particlesHb011`. + +Each clause needs both invalid fixtures and valid near-misses. A valid near-miss +is mandatory because this is the highest-fan-in false-positive surface in the +repo. + +**Verifies:** + +```sh +swift test --filter XSDParticleRestrictionTests +XSTS_ROOT=/private/tmp/xsts/xmlschema2006-11-06 swift test -c release --filter XSTS +``` + +**Commit:** `fix(schema): make particle restriction clause exact` + +### T7. Close redefine and composition fallout + +**Files:** `Sources/Schema/SchemaReferences.swift`, +`Sources/Schema/SchemaRedefineSelfReference.swift`, +`Sources/Schema/SchemaCompositionValidations.swift`, +`Tests/SchemaRedefineAttributeGroupTests.swift`, +`Tests/SchemaGroupRedefineRestrictionTests.swift`, +`Tests/SchemaReferenceTests.swift` + +**Does:** Target `schG10`, `schM4`, `schM8`, `schN10`, `schN12`, +`schZ011_a` through `schZ011_d`, and the related `schA*`/`schU*` instance +fallout. Every name lookup must be namespace-gated and bounded to the owning +schema container. Self-reference handling must be tested with nested local +definitions so the walk cannot overreach. + +**Verifies:** + +```sh +swift test --filter SchemaRedefine +swift test --filter SchemaReference +XSTS_ROOT=/private/tmp/xsts/xmlschema2006-11-06 swift test -c release --filter XSTS +``` + +**Commit:** `fix(schema): close redefine false-negative tail` + +### T8. Close identity-constraint instance tail + +**Files:** `Sources/Schema/IdentityValidator*.swift`, +`Sources/Schema/XSDParserHelpers.swift`, +`Tests/XSDIdentityTests.swift`, +`Tests/SchemaIdentity*Tests.swift` + +**Does:** Target `idG006`, `idK012`, `idZ010`, and `idZ012`. Classify whether +each is selector/field XPath, default/fixed value, typed-value equality, or +keyref closure. Add a valid paired case for every invalid fixture. + +**Verifies:** + +```sh +swift test --filter Identity +XSTS_ROOT=/private/tmp/xsts/xmlschema2006-11-06 swift test -c release --filter XSTS +``` + +**Commit:** `fix(schema): close identity constraint instance tail` + +### T9. Design the counting automaton separately + +**Files:** new `docs/design/` or `docs/plans/` design artifact, +`Sources/Schema/ContentMatcher.swift`, +`Sources/Schema/ContentModelDeterminism.swift`, +`Tests/ContentMatcherDifferentialTests.swift`, +`Tests/ContentNFAStateBudgetTests.swift` + +**Does:** Do not mix resource-bound architecture with conformance fixes. The +acceptance criteria for this later arc are: delete `occursUnrollCap`, +`totalStateCap`, and `positionCap`; remove silent "degrade to star" behavior; +state a worst-case time and memory bound; prove high numeric occurrence bounds +do not create proportional states. + +**Verifies:** + +```sh +swift test --filter ContentMatcher +swift test --filter ContentModelDeterminism +swift test --filter ContentNFAStateBudgetTests +``` + +**Commit:** `docs(schema): design counted content automaton` + +## Constraints + +- False positives stay at zero. A rise in `valid-schemas-rejected` or + `valid-instances-rejected` blocks the change. +- Before changing parser, schema, validation, or typed-conversion behavior, read + the matching PureXMLResearch fixture/reference and the XSD rule text. +- External validators are witnesses, not ground truth. If libxml2/Xerces and the + spec disagree, document the disagreement and choose the defensible spec reading. +- No new SwiftPM dependencies, C targets, generated parser runtime, Foundation-only + workaround, or JavaScript tooling. +- One rule family per implementation commit. Do not combine a particle oracle + change with identity, regex, or composition work. +- Every under-rejection that remains by design must be named, bounded, and visible + in `docs/xsts-deviations.md`; no silent acceptances. + +## Test plan + +Every implementation task runs, at minimum: + +```sh +bash scripts/check-style.sh +bash scripts/check-namespacing.sh +bash scripts/check-forbidden-patterns.sh +swiftformat . --config .swiftformat --lint +swiftlint --config .swiftlint.yml --strict +swift build +swift test +XSTS_ROOT=/private/tmp/xsts/xmlschema2006-11-06 swift test -c release --filter XSTS +bash scripts/check-wasm.sh +``` + +Use targeted Swift tests before the full gate, but never claim a conformance +fix complete without the full XSTS ratchet. + +## Done definition + +- `Tests/XSTSSuiteTests.swift` reports: + - `valid-schemas-rejected = 0` + - `invalid-schemas-accepted = 0` + - `valid-instances-rejected = 0` + - `invalid-instances-accepted = 0` +- `docs/xsts-deviations.md` has no unclassified false negative. +- `docs/release-hurdles.md` is updated to move the false-negative tail from + "biggest hurdle" to closed, leaving only the counted-automaton resource-bound + work if that has not landed. +- All relevant local gates pass, including WASI. +- CHANGELOG records the final XSTS zeroing and any named spec divergences. diff --git a/docs/release-hurdles.md b/docs/release-hurdles.md index d3219c2..998fadc 100644 --- a/docs/release-hurdles.md +++ b/docs/release-hurdles.md @@ -6,7 +6,7 @@ evidence, what actually stands between the current engine and a 1.0 fit to put i front of a developer as the authority on XML/XSD correctness. Snapshot at time of writing: v0.2.0 released; full 2006-11-06 XSTS archive (14383 -groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 48`, +groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 47`, `valid-instances-rejected = 0`, `invalid-instances-accepted = 31`. ## Status against the four stoppers @@ -16,7 +16,7 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 48`, libxml2 differential over 9618 schemas. **Met.** This is the doc's bar to be "an authority"; it is cleared on the measured corpora. 2. **Silently accepting faulty input (false negatives).** `invalid-schemas-accepted - = 48`, `invalid-instances-accepted = 31`. **Not at 0.** This is the bulk of named + = 47`, `invalid-instances-accepted = 31`. **Not at 0.** This is the bulk of named milestone work (M2, M3) and the subject of Part 1 below. 3. **Uncharacterized correctness.** Differential harness vs libxml2 and a fuzz suite exist (M1, ~done). Gap: real-world corpora beyond XSTS, and the silent @@ -28,7 +28,7 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 48`, ## The biggest hurdle, stated plainly -Driving the false-negative buckets (`invalid-schemas-accepted = 48`, +Driving the false-negative buckets (`invalid-schemas-accepted = 47`, `invalid-instances-accepted = 31`) to zero is the largest remaining body of named work, and it is hard for a specific reason: **the ~95% already closed were the tractable cases; every case left fights the cardinal rule.** The remainder sits in @@ -57,7 +57,7 @@ chose to under-approximate to stay false-positive-free. | particlesK006, M034 | RecurseAsIfGroup (Elt:All, Elt:Sequence) | "a documented, bounded under-rejection… over-rejection is the non-starter" (ParticleRestriction.swift:298) | | particlesEa025, Fb003, Ha161, Hb011 | pointless-particle / extension-from-`any` / forbidden sequence:elt | partial | | elemZ026/027/028 | element-decl restriction where base is a substitution-group head | not modelled | -| particlesZ018 | list-of-int ⊄ decimal (simple-type derivation, a *different* oracle) | separate | +| particlesZ018 | list-of-int ⊄ decimal (simple-type derivation, a *different* oracle) | closed by inline simpleContent type restriction check | | particlesZ022/030 | non-deterministic wildcards (UPA, not restriction) | determinism check is deliberately QName-only | **Why this is the risk, by fan-in.** `ParticleRestriction.violation` is called from