PureXML/Tests/XSTSSuiteTests.swift
Mihaela Mihaljevic 0f90b3a4d3 fix(schema): generalize redefine self-reference to attributeGroups + bound the walk; invalid-schemas-accepted 75 to 74 (#161)
Extends the redefine self-reference rule (redefineSelfReferenceErrors, moved to
SchemaRedefineSelfReference.swift) to attribute groups: a group or attributeGroup
inside xs:redefine may reference itself at most once (src-redefine.6.1.1 /
7.2.1), and a group self-reference must occur exactly once. Catches a redefined
attributeGroup with two self-references (attgC044).

Fixes a latent false positive an adversarial review found in the group case: the
search now uses a BOUNDED walk (boundedSelfReferenceNodes) that stops at nested
element/complexType scopes, matching the established cycle-rule boundary. The
prior unbounded walk would miscount a recursive group reference inside a nested
element's content (a data-structure recursion, legal) as a redefinition
self-reference. attributeGroup content has no such nesting, so it is unaffected.

XSTS: invalid-schemas-accepted 75 to 74 (attgC044), valid-schemas-rejected held
at 0, no other bucket moved. The validation description broadened accordingly
(configuration pin updated).
2026-06-18 01:15:16 +02:00

400 lines
22 KiB
Swift

import Foundation
@testable import PureXML
import Testing
/// Opt-in runner for the W3C XML Schema Test Suite (XSTS, the 2006-11-06
/// release; W3C Document License, never vendored). Point `XSTS_ROOT` at the
/// directory containing `suite.xml`. The suite nests testSet files holding
/// testGroups: a schemaTest's documents must compile when expected valid and
/// be rejected when expected invalid; each instanceTest validates against
/// the group's schema with the expected outcome. Hrefs resolve against the
/// referencing file; include/import locations resolve from disk relative to
/// the schema. Counts are asserted exactly so progress and regressions both
/// show; current failures are written to /tmp/xsts-failures.txt for the
/// burn-down. Foundation is used for file access only.
@Suite("W3C XSTS (opt-in via XSTS_ROOT)")
struct XSTSSuiteTests {
private var root: String? {
ProcessInfo.processInfo.environment["XSTS_ROOT"]
}
/// Exact-count baselines against the 2006-11-06 archive (first measured
/// 2026-06-13, over 14383 groups), counting only tests whose expected outcome
/// is settled: entries marked `<current status="queried">` (disputed in W3C
/// Bugzilla, with the expected validity itself contested) are excluded, since
/// agreeing or disagreeing with a contested expectation is not meaningful.
/// These ratchet down as conformance work closes deviations (#145-#148); a
/// count rising is a regression. The suite is opt-in, so CI and plain
/// `swift test` are unaffected; run it with `swift test -c release --filter
/// XSTS` (debug is far slower and the corpus is large). Per-case deviations
/// are written to /tmp/xsts-failures.txt for the burn-down.
///
/// Re-measured 2026-06-16 after the harness was corrected to compile and merge
/// ALL of a schemaTest's schemaDocuments (it previously loaded only the first,
/// under-loading any multi-document schema). That removed a spurious valid-
/// instance rejection (wildZ003, whose root element is declared in the second
/// document) and exposed one previously-masked invalid-instance acceptance
/// (schA5, a schema-collection test the engine wrongly accepts once the full
/// collection is loaded): instances rejected 8 → 7, invalid accepted 146 → 147.
///
/// Pinning the regex repertoire to Unicode 3.2 (the version the corpus is
/// consistent at) then cleared reZ003v, whose `\w` instance is all code points
/// assigned by 3.2: instances rejected 7 → 6, no other bucket moved.
///
/// Driving per-child assessment from the matched content-model particle rather
/// than a by-name lookup (#180) then cleared isDefault079 (two same-named
/// particles with different `fixed` values) and QFE1700c2 (an element matching
/// a `skip` wildcard must not be validated against its global declaration):
/// instances rejected 6 → 4, no other bucket moved.
///
/// Compiling an import chain of schema documents as a union rather than merging
/// separately compiled documents (#161) then cleared ctZ007, a cross-document
/// substitution group: instances rejected 4 → 3, no other bucket moved. The
/// remaining three are the Ethiopic-digit `\d` cases (reS17, reS38, reZ004v),
/// which contradict `\d` = `\p{Nd}` and are tracked as suspect tests.
///
/// Enforcing identity-constraint field cardinality (a field must select at most
/// one node; cvc-identity-constraint.3) then caught four invalid instances that
/// were accepted (idF005, idG005, and kin): invalid accepted 147 → 143, no
/// other bucket moved.
///
/// Resolving identity-constraint selector/field XPath prefixes against the
/// schema's namespace context (where the constraint is declared) rather than
/// the instance document then caught ten more (the idc004/5/6 cluster and
/// namespace-resolution cases): invalid accepted 143 → 133, no other bucket
/// moved.
///
/// Resolving an identity-constraint attribute field's type through the
/// selector's target global element (not just the host's descendants) lets a
/// `unique`/`key` compare values in their value space (3.0 and 3 are the same
/// xsd:decimal key): invalid accepted 133 → 132, no other bucket moved.
/// The invalid-schemas-accepted bucket has since been driven 262 → 181 across
/// many schema-validity rules (regex `pattern` validity, schema-for-schemas
/// structure, derivation/`final`/`finalDefault`, notation, attribute and
/// attributeGroup constraints, value constraints, and `redefine` resolution);
/// each rule and its exact bucket delta is recorded per-entry in CHANGELOG.md.
/// The remaining residue is the cos-particle-restrict, redefine/composition, and
/// wildcard-restriction tail (see schema-validity-burndown notes). Recent steps:
/// Requiring each component a `redefine` names to exist (same kind, same name) in
/// the redefined schema once that schema is loaded (src-redefine.6/7.2.1) then
/// caught one more invalid schema: invalid-schemas accepted 183 → 182, no other
/// bucket moved.
/// Rejecting a `redefine` that redefines the same component twice (src-redefine.7.2.2)
/// then caught one more invalid schema: invalid-schemas accepted 182 → 181, no
/// other bucket moved.
/// Checking the RecurseAsIfGroup occurrence in Particle-Valid-Restriction (an
/// element restricting a base group is a synthetic {1,1} group, so a base group
/// that must occur 2+ times cannot be restricted to one element; §3.9.6) then
/// caught two more invalid schemas: invalid-schemas accepted 181 → 179, no other
/// bucket moved.
/// Enforcing the NameAndTypeOK block-superset clause (a restricting element's
/// `{disallowed substitutions}` must be a superset of the base element's, with
/// `block`/`blockDefault` carried on the particle) then caught sixteen more
/// invalid schemas: invalid-schemas accepted 179 → 163, no other bucket moved.
/// Subset-checking ANONYMOUS complex-type restrictions (inline `<complexType>`
/// declarations, which the named-type rule never saw) with the same
/// Particle-Valid-Restriction algorithm then caught fifteen more invalid schemas:
/// invalid-schemas accepted 163 → 148, no other bucket moved.
/// Enforcing the NameAndTypeOK fixed-value clause (a restriction of a `fixed`
/// base element must be fixed to the same value, compared in the base type's
/// value space) then caught four more invalid schemas: invalid-schemas accepted
/// 148 → 144, no other bucket moved.
/// Enforcing the NameAndTypeOK nillable clause (a restriction may not be nillable
/// unless the base element is) then caught two more invalid schemas:
/// invalid-schemas accepted 144 → 142, no other bucket moved.
/// Requiring a `simpleContent` restriction's base to be a complex type, not a
/// built-in or user simple type (src-ct.2), then caught five more invalid
/// schemas: invalid-schemas accepted 142 → 137, no other bucket moved.
/// Requiring a restricting wildcard's `processContents` to be at least as strong
/// as the base's (strict > lax > skip; Wildcard Subset §3.10.6) then caught three
/// more invalid schemas: invalid-schemas accepted 137 → 134, no other bucket moved.
/// Rejecting an empty-string namespace value (`targetNamespace=""`, `import
/// namespace=""`) then caught two more invalid schemas: invalid-schemas accepted
/// 134 → 132, no other bucket moved.
/// Requiring an attribute a restriction adds (with no matching base attribute use)
/// to be admitted by the base's attribute wildcard (cos-ct-restricts.3) then
/// caught one more invalid schema: invalid-schemas accepted 132 → 131, no other
/// bucket moved.
/// Requiring a restriction's own attribute wildcard to be a subset of the base's
/// (cos-ct-restricts.4: a base without one admits none; `##any` over `##other` is
/// not a subset) then caught three more invalid schemas: invalid-schemas accepted
/// 131 → 128, no other bucket moved.
/// Instance tests whose expected validity contradicts the normative XSD
/// definition of `\d`, excluded as a named, bounded spec divergence (not a
/// validator defect). XSD 1.0 Datatypes Appendix F defines `\d` as `\p{Nd}`.
/// reS17 (U+1369), reS38 (U+1371), and reZ004v use Ethiopic digits, which are
/// general category `No` (Other Number), not `Nd`, so `\d` does not match them
/// and the instance is invalid; the corpus marks these `valid`. Verified that
/// our `\d` matches `Nd` (incl. Khmer U+17E0) and rejects Ethiopic `No`, which is
/// what every conformant processor (e.g. Xerces) does. Faking `\d` to match `No`
/// would corrupt the definition and is refused; the tests are the outliers.
private static let specDivergentInstances: Set<String> = ["reS17.v", "reS38.v", "reZ004v.v"]
/// FALSE POSITIVES ARE 0 (production stopper #1). The last over-rejection,
/// particlesZ001 (`element{0,∞}` restricting `choice{0,∞}` over that element;
/// the test's own doc calls the RecurseAsIfGroup rule "ambiguous"), is fixed by
/// reading element-vs-choice restriction via effective total range. That reading
/// is a bounded, NAMED under-rejection: element-vs-choice no longer models exact
/// in-branch sequencing, so invalid-schemas-accepted rose 128 → 130 (two
/// element-vs-choice restriction cases) and, with particlesZ001 now correctly
/// compiling, its instance particlesZ001.i (a previously-masked invalid-instance
/// gap) is accepted, 132 → 133. Per the production standard over-rejection is the
/// non-starter and under-rejection is recoverable; this debt is recovered by the
/// full effective-total-range Particle-Valid-Restriction (tracked).
/// Further rules (each with valid-schemas-rejected held at 0; see CHANGELOG for
/// detail) drove invalid-schemas-accepted 130 -> 79: model-group occurrence ranges
/// (p-props-correct.1, cos-all-limited.2), identity-constraint content model,
/// derivation base-type kind (src-ct.1/.2), element value-constraint content type
/// (cos-valid-default), attribute type must be simple (au-props-correct), resolved
/// schemaLocation must be a schema (src-resolve), import namespace matches the
/// imported target (src-import.3), redefined type derives from itself
/// (src-redefine.5), local complexType is anonymous, a local element has a name
/// or ref, a redefined group's self-reference occurs exactly once
/// (src-redefine.6.1), an identity-constraint xpath uses no undeclared namespace
/// prefix, and a redefined group/attributeGroup has at most one self-reference.
private let knownSchemaValidRejected = 0
private let knownSchemaInvalidAccepted = 74
private let knownInstanceValidRejected = 0
private let knownInstanceInvalidAccepted = 133
@Test("Every XSTS case behaves: compile, reject, validate, invalidate")
func test_suite() throws {
guard let root else {
return // Opt-in: suite not present.
}
var counters = Counters()
let suite = try parsed(at: root + "/suite.xml")
for setHref in references(in: suite, named: "testSetRef") {
let setPath = root + "/" + setHref
let setDirectory = directory(of: setPath)
FileHandle.standardError.write(Data("xsts: \(setHref)\n".utf8))
guard let testSet = try? parsed(at: setPath) else {
counters.note("unreadable testSet \(setHref)")
continue
}
for group in elements(named: "testGroup", under: testSet) {
runGroup(group, directory: setDirectory, into: &counters)
}
}
try? counters.failures.joined(separator: "\n").write(toFile: "/tmp/xsts-failures.txt", atomically: true, encoding: .utf8)
let report = "groups \(counters.groups), schema ok \(counters.schemaAgreed), instance ok \(counters.instanceAgreed)"
#expect(counters.schemaValidRejected == knownSchemaValidRejected, "\(counters.schemaValidRejected) valid schemas rejected (\(report))")
#expect(counters.schemaInvalidAccepted == knownSchemaInvalidAccepted, "\(counters.schemaInvalidAccepted) invalid schemas accepted (\(report))")
#expect(counters.instanceValidRejected == knownInstanceValidRejected, "\(counters.instanceValidRejected) valid instances rejected (\(report))")
#expect(counters.instanceInvalidAccepted == knownInstanceInvalidAccepted, "\(counters.instanceInvalidAccepted) invalid instances accepted (\(report))")
}
private struct Counters {
var groups = 0
var schemaAgreed = 0
var schemaValidRejected = 0
var schemaInvalidAccepted = 0
var instanceAgreed = 0
var instanceValidRejected = 0
var instanceInvalidAccepted = 0
var failures: [String] = []
mutating func note(_ failure: String) {
failures.append(failure)
}
}
private func runGroup(_ group: PureXML.Model.Element, directory: String, into counters: inout Counters) {
counters.groups += 1
let name = attribute("name", of: group) ?? "?"
if ProcessInfo.processInfo.environment["XSTS_TRACE"] != nil {
FileHandle.standardError.write(Data("group: \(name)\n".utf8))
}
guard let (schema, expectedValid) = runSchemaTests(group, name: name, directory: directory, into: &counters),
expectedValid, let compiled = schema
else { return }
runInstanceTests(group, name: name, directory: directory, schema: compiled, into: &counters)
}
/// Compiles the group's schema tests, recording disagreements; returns
/// the last compiled schema and whether it was expected valid.
private func runSchemaTests(
_ group: PureXML.Model.Element,
name: String,
directory: String,
into counters: inout Counters,
) -> (PureXML.Schema.Document?, Bool)? {
var schema: PureXML.Schema.Document?
var schemaExpectedValid = true
for schemaTest in elements(named: "schemaTest", under: .element(group)) {
guard isAccepted(schemaTest) else { continue }
let hrefs = references(in: .element(schemaTest), named: "schemaDocument")
guard !hrefs.isEmpty else { continue }
schemaExpectedValid = expectedValidity(schemaTest) != "invalid"
let compiled = compileSchemaDocuments(hrefs, directory: directory, name: name, into: &counters)
switch (schemaExpectedValid, compiled == nil) {
case (true, true):
counters.schemaValidRejected += 1
counters.note("\(name): valid schema rejected")
case (false, false):
counters.schemaInvalidAccepted += 1
counters.note("\(name): invalid schema accepted")
default:
counters.schemaAgreed += 1
}
schema = compiled
}
return (schema, schemaExpectedValid)
}
/// Compiles a schemaTest's documents into one schema. When one document
/// imports another listed document they form an import chain and are compiled
/// as a union so cross-document substitution groups resolve (#161); otherwise
/// they are independent roots, composed through the lenient per-document merge
/// (a union of independent roots wrongly trips cross-document checks).
private func compileSchemaDocuments(
_ hrefs: [String],
directory: String,
name: String,
into counters: inout Counters,
) -> PureXML.Schema.Document? {
let schemaDirectory = self.directory(of: resolve(hrefs[0], against: directory))
let loader: (String) -> String? = { location in
(try? String(contentsOfFile: resolve(location, against: schemaDirectory), encoding: .utf8))
}
let basenames = hrefs.map { String($0.split(separator: "/").last ?? "") }
var sources: [(name: String, text: String)] = []
for (index, href) in hrefs.enumerated() {
guard let source = try? String(contentsOfFile: resolve(href, against: directory), encoding: .utf8) else {
counters.note("\(name): unreadable schema \(href)")
return nil
}
sources.append((basenames[index], source))
}
let isImportChain = sources.contains { source in
importedBasenames(of: source.text).contains { $0 != source.name && basenames.contains($0) }
}
if isImportChain {
return try? PureXML.Schema.Document(composing: sources.map(\.text), schemaLoader: loader)
}
var merged: PureXML.Schema.Document?
for source in sources {
guard let document = try? PureXML.Schema.Document(source.text, schemaLoader: loader) else { return nil }
merged = merged.map { $0.merged(with: document) } ?? document
}
return merged
}
/// The basenames a schema source imports, includes, or redefines, used to tell
/// an import chain (compile as a union) from independent roots (merge). Matches
/// the `schemaLocation` attribute, not a raw substring, so a name appearing in
/// documentation or as a suffix of another name does not misclassify.
private func importedBasenames(of source: String) -> Set<String> {
guard let node = try? PureXML.parse(source, limits: .init(allowDoctype: true)) else { return [] }
var result: Set<String> = []
for kind in ["import", "include", "redefine"] {
for element in elements(named: kind, under: node) {
guard let location = element.attributes.first(where: { $0.name.localName == "schemaLocation" })?.value else { continue }
result.insert(String(location.split(separator: "/").last ?? ""))
}
}
return result
}
private func runInstanceTests(
_ group: PureXML.Model.Element,
name: String,
directory: String,
schema: PureXML.Schema.Document,
into counters: inout Counters,
) {
for instanceTest in elements(named: "instanceTest", under: .element(group)) {
guard isAccepted(instanceTest) else { continue }
guard !Self.specDivergentInstances.contains(attribute("name", of: instanceTest) ?? "") else { continue }
let expected = expectedValidity(instanceTest)
guard expected == "valid" || expected == "invalid",
let href = references(in: .element(instanceTest), named: "instanceDocument").first
else { continue }
let instancePath = resolve(href, against: directory)
guard let xml = try? String(contentsOfFile: instancePath, encoding: .utf8) else { continue }
// Resolve the instance's xsi:schemaLocation hints relative to the
// instance file, so a strict wildcard can find declarations in them.
let instanceDirectory = self.directory(of: instancePath)
let loader: (String) -> String? = { location in
(try? String(contentsOfFile: resolve(location, against: instanceDirectory), encoding: .utf8))
}
let isValid = ((try? schema.validate(xml, schemaLoader: loader))?.isEmpty) ?? false
if expected == "valid", !isValid {
counters.instanceValidRejected += 1
counters.note("\(name)/\(attribute("name", of: instanceTest) ?? "?"): valid instance rejected")
} else if expected == "invalid", isValid {
counters.instanceInvalidAccepted += 1
counters.note("\(name)/\(attribute("name", of: instanceTest) ?? "?"): invalid instance accepted")
} else {
counters.instanceAgreed += 1
}
}
}
// MARK: Manifest helpers
private func parsed(at path: String) throws -> PureXML.Model.Node {
try PureXML.parse(String(contentsOfFile: path, encoding: .utf8), limits: .init(allowDoctype: true))
}
private func elements(named name: String, under node: PureXML.Model.Node) -> [PureXML.Model.Element] {
var found: [PureXML.Model.Element] = []
switch node {
case let .document(children):
for child in children {
found += elements(named: name, under: child)
}
case let .element(element):
if element.name.localName == name { found.append(element) }
for child in element.children {
found += elements(named: name, under: child)
}
default:
break
}
return found
}
private func references(in node: PureXML.Model.Node, named name: String) -> [String] {
elements(named: name, under: node).compactMap { element in
element.attributes.first { $0.name.localName == "href" }?.value
}
}
private func attribute(_ name: String, of element: PureXML.Model.Element) -> String? {
element.attributes.first { $0.name.localName == name }?.value
}
/// Whether a schemaTest/instanceTest's expected outcome is settled. The suite
/// marks disputed entries `<current status="queried">` (often with a W3C
/// Bugzilla reference) where the expected validity is itself contested; those
/// are not authoritative, so they are not counted as agreements or deviations.
private func isAccepted(_ test: PureXML.Model.Element) -> Bool {
guard let current = elements(named: "current", under: .element(test)).first else { return true }
return (attribute("status", of: current) ?? "accepted") == "accepted"
}
private func expectedValidity(_ element: PureXML.Model.Element) -> String {
elements(named: "expected", under: .element(element)).first
.flatMap { attribute("validity", of: $0) } ?? "notKnown"
}
private func directory(of path: String) -> String {
guard let slash = path.lastIndex(of: "/") else { return "." }
return String(path[..<slash])
}
/// Resolves an href against a base directory, folding `..` segments.
private func resolve(_ href: String, against base: String) -> String {
var parts = base.split(separator: "/", omittingEmptySubsequences: false).map(String.init)
for segment in href.split(separator: "/") {
if segment == ".." {
if !parts.isEmpty { parts.removeLast() }
} else if segment != "." {
parts.append(String(segment))
}
}
return parts.joined(separator: "/")
}
}