PureXML/Tests/XSTSSuiteTests.swift
Mihaela Mihaljevic d43d1bb1bb fix(schema): subset-check anonymous complex-type restrictions (#145)
Particle Valid (Restriction) was keyed by type name, so an inline complexType
deriving by complexContent restriction was never checked against its base. Such
anonymous restrictions are now compiled and run through the same subset
algorithm as named types.

XSTS invalid-schemas-accepted 163 to 148, no other bucket moved.
2026-06-17 17:02:20 +02:00

342 lines
18 KiB
Swift

import Foundation
@testable import PureXML
import Testing
/// Opt-in runner for the W3C XML Schema Test Suite (XSTS, the 2006-11-06
/// release; W3C Document License, never vendored). Point `XSTS_ROOT` at the
/// directory containing `suite.xml`. The suite nests testSet files holding
/// testGroups: a schemaTest's documents must compile when expected valid and
/// be rejected when expected invalid; each instanceTest validates against
/// the group's schema with the expected outcome. Hrefs resolve against the
/// referencing file; include/import locations resolve from disk relative to
/// the schema. Counts are asserted exactly so progress and regressions both
/// show; current failures are written to /tmp/xsts-failures.txt for the
/// burn-down. Foundation is used for file access only.
@Suite("W3C XSTS (opt-in via XSTS_ROOT)")
struct XSTSSuiteTests {
private var root: String? {
ProcessInfo.processInfo.environment["XSTS_ROOT"]
}
/// Exact-count baselines against the 2006-11-06 archive (first measured
/// 2026-06-13, over 14383 groups), counting only tests whose expected outcome
/// is settled: entries marked `<current status="queried">` (disputed in W3C
/// Bugzilla, with the expected validity itself contested) are excluded, since
/// agreeing or disagreeing with a contested expectation is not meaningful.
/// These ratchet down as conformance work closes deviations (#145-#148); a
/// count rising is a regression. The suite is opt-in, so CI and plain
/// `swift test` are unaffected; run it with `swift test -c release --filter
/// XSTS` (debug is far slower and the corpus is large). Per-case deviations
/// are written to /tmp/xsts-failures.txt for the burn-down.
///
/// Re-measured 2026-06-16 after the harness was corrected to compile and merge
/// ALL of a schemaTest's schemaDocuments (it previously loaded only the first,
/// under-loading any multi-document schema). That removed a spurious valid-
/// instance rejection (wildZ003, whose root element is declared in the second
/// document) and exposed one previously-masked invalid-instance acceptance
/// (schA5, a schema-collection test the engine wrongly accepts once the full
/// collection is loaded): instances rejected 8 → 7, invalid accepted 146 → 147.
///
/// Pinning the regex repertoire to Unicode 3.2 (the version the corpus is
/// consistent at) then cleared reZ003v, whose `\w` instance is all code points
/// assigned by 3.2: instances rejected 7 → 6, no other bucket moved.
///
/// Driving per-child assessment from the matched content-model particle rather
/// than a by-name lookup (#180) then cleared isDefault079 (two same-named
/// particles with different `fixed` values) and QFE1700c2 (an element matching
/// a `skip` wildcard must not be validated against its global declaration):
/// instances rejected 6 → 4, no other bucket moved.
///
/// Compiling an import chain of schema documents as a union rather than merging
/// separately compiled documents (#161) then cleared ctZ007, a cross-document
/// substitution group: instances rejected 4 → 3, no other bucket moved. The
/// remaining three are the Ethiopic-digit `\d` cases (reS17, reS38, reZ004v),
/// which contradict `\d` = `\p{Nd}` and are tracked as suspect tests.
///
/// Enforcing identity-constraint field cardinality (a field must select at most
/// one node; cvc-identity-constraint.3) then caught four invalid instances that
/// were accepted (idF005, idG005, and kin): invalid accepted 147 → 143, no
/// other bucket moved.
///
/// Resolving identity-constraint selector/field XPath prefixes against the
/// schema's namespace context (where the constraint is declared) rather than
/// the instance document then caught ten more (the idc004/5/6 cluster and
/// namespace-resolution cases): invalid accepted 143 → 133, no other bucket
/// moved.
///
/// Resolving an identity-constraint attribute field's type through the
/// selector's target global element (not just the host's descendants) lets a
/// `unique`/`key` compare values in their value space (3.0 and 3 are the same
/// xsd:decimal key): invalid accepted 133 → 132, no other bucket moved.
/// The invalid-schemas-accepted bucket has since been driven 262 → 181 across
/// many schema-validity rules (regex `pattern` validity, schema-for-schemas
/// structure, derivation/`final`/`finalDefault`, notation, attribute and
/// attributeGroup constraints, value constraints, and `redefine` resolution);
/// each rule and its exact bucket delta is recorded per-entry in CHANGELOG.md.
/// The remaining residue is the cos-particle-restrict, redefine/composition, and
/// wildcard-restriction tail (see schema-validity-burndown notes). Recent steps:
/// Requiring each component a `redefine` names to exist (same kind, same name) in
/// the redefined schema once that schema is loaded (src-redefine.6/7.2.1) then
/// caught one more invalid schema: invalid-schemas accepted 183 → 182, no other
/// bucket moved.
/// Rejecting a `redefine` that redefines the same component twice (src-redefine.7.2.2)
/// then caught one more invalid schema: invalid-schemas accepted 182 → 181, no
/// other bucket moved.
/// Checking the RecurseAsIfGroup occurrence in Particle-Valid-Restriction (an
/// element restricting a base group is a synthetic {1,1} group, so a base group
/// that must occur 2+ times cannot be restricted to one element; §3.9.6) then
/// caught two more invalid schemas: invalid-schemas accepted 181 → 179, no other
/// bucket moved.
/// Enforcing the NameAndTypeOK block-superset clause (a restricting element's
/// `{disallowed substitutions}` must be a superset of the base element's, with
/// `block`/`blockDefault` carried on the particle) then caught sixteen more
/// invalid schemas: invalid-schemas accepted 179 → 163, no other bucket moved.
/// Subset-checking ANONYMOUS complex-type restrictions (inline `<complexType>`
/// declarations, which the named-type rule never saw) with the same
/// Particle-Valid-Restriction algorithm then caught fifteen more invalid schemas:
/// invalid-schemas accepted 163 → 148, no other bucket moved.
private let knownSchemaValidRejected = 1
private let knownSchemaInvalidAccepted = 148
private let knownInstanceValidRejected = 3
private let knownInstanceInvalidAccepted = 132
@Test("Every XSTS case behaves: compile, reject, validate, invalidate")
func test_suite() throws {
guard let root else {
return // Opt-in: suite not present.
}
var counters = Counters()
let suite = try parsed(at: root + "/suite.xml")
for setHref in references(in: suite, named: "testSetRef") {
let setPath = root + "/" + setHref
let setDirectory = directory(of: setPath)
FileHandle.standardError.write(Data("xsts: \(setHref)\n".utf8))
guard let testSet = try? parsed(at: setPath) else {
counters.note("unreadable testSet \(setHref)")
continue
}
for group in elements(named: "testGroup", under: testSet) {
runGroup(group, directory: setDirectory, into: &counters)
}
}
try? counters.failures.joined(separator: "\n").write(toFile: "/tmp/xsts-failures.txt", atomically: true, encoding: .utf8)
let report = "groups \(counters.groups), schema ok \(counters.schemaAgreed), instance ok \(counters.instanceAgreed)"
#expect(counters.schemaValidRejected == knownSchemaValidRejected, "\(counters.schemaValidRejected) valid schemas rejected (\(report))")
#expect(counters.schemaInvalidAccepted == knownSchemaInvalidAccepted, "\(counters.schemaInvalidAccepted) invalid schemas accepted (\(report))")
#expect(counters.instanceValidRejected == knownInstanceValidRejected, "\(counters.instanceValidRejected) valid instances rejected (\(report))")
#expect(counters.instanceInvalidAccepted == knownInstanceInvalidAccepted, "\(counters.instanceInvalidAccepted) invalid instances accepted (\(report))")
}
private struct Counters {
var groups = 0
var schemaAgreed = 0
var schemaValidRejected = 0
var schemaInvalidAccepted = 0
var instanceAgreed = 0
var instanceValidRejected = 0
var instanceInvalidAccepted = 0
var failures: [String] = []
mutating func note(_ failure: String) {
failures.append(failure)
}
}
private func runGroup(_ group: PureXML.Model.Element, directory: String, into counters: inout Counters) {
counters.groups += 1
let name = attribute("name", of: group) ?? "?"
if ProcessInfo.processInfo.environment["XSTS_TRACE"] != nil {
FileHandle.standardError.write(Data("group: \(name)\n".utf8))
}
guard let (schema, expectedValid) = runSchemaTests(group, name: name, directory: directory, into: &counters),
expectedValid, let compiled = schema
else { return }
runInstanceTests(group, name: name, directory: directory, schema: compiled, into: &counters)
}
/// Compiles the group's schema tests, recording disagreements; returns
/// the last compiled schema and whether it was expected valid.
private func runSchemaTests(
_ group: PureXML.Model.Element,
name: String,
directory: String,
into counters: inout Counters,
) -> (PureXML.Schema.Document?, Bool)? {
var schema: PureXML.Schema.Document?
var schemaExpectedValid = true
for schemaTest in elements(named: "schemaTest", under: .element(group)) {
guard isAccepted(schemaTest) else { continue }
let hrefs = references(in: .element(schemaTest), named: "schemaDocument")
guard !hrefs.isEmpty else { continue }
schemaExpectedValid = expectedValidity(schemaTest) != "invalid"
let compiled = compileSchemaDocuments(hrefs, directory: directory, name: name, into: &counters)
switch (schemaExpectedValid, compiled == nil) {
case (true, true):
counters.schemaValidRejected += 1
counters.note("\(name): valid schema rejected")
case (false, false):
counters.schemaInvalidAccepted += 1
counters.note("\(name): invalid schema accepted")
default:
counters.schemaAgreed += 1
}
schema = compiled
}
return (schema, schemaExpectedValid)
}
/// Compiles a schemaTest's documents into one schema. When one document
/// imports another listed document they form an import chain and are compiled
/// as a union so cross-document substitution groups resolve (#161); otherwise
/// they are independent roots, composed through the lenient per-document merge
/// (a union of independent roots wrongly trips cross-document checks).
private func compileSchemaDocuments(
_ hrefs: [String],
directory: String,
name: String,
into counters: inout Counters,
) -> PureXML.Schema.Document? {
let schemaDirectory = self.directory(of: resolve(hrefs[0], against: directory))
let loader: (String) -> String? = { location in
(try? String(contentsOfFile: resolve(location, against: schemaDirectory), encoding: .utf8))
}
let basenames = hrefs.map { String($0.split(separator: "/").last ?? "") }
var sources: [(name: String, text: String)] = []
for (index, href) in hrefs.enumerated() {
guard let source = try? String(contentsOfFile: resolve(href, against: directory), encoding: .utf8) else {
counters.note("\(name): unreadable schema \(href)")
return nil
}
sources.append((basenames[index], source))
}
let isImportChain = sources.contains { source in
importedBasenames(of: source.text).contains { $0 != source.name && basenames.contains($0) }
}
if isImportChain {
return try? PureXML.Schema.Document(composing: sources.map(\.text), schemaLoader: loader)
}
var merged: PureXML.Schema.Document?
for source in sources {
guard let document = try? PureXML.Schema.Document(source.text, schemaLoader: loader) else { return nil }
merged = merged.map { $0.merged(with: document) } ?? document
}
return merged
}
/// The basenames a schema source imports, includes, or redefines, used to tell
/// an import chain (compile as a union) from independent roots (merge). Matches
/// the `schemaLocation` attribute, not a raw substring, so a name appearing in
/// documentation or as a suffix of another name does not misclassify.
private func importedBasenames(of source: String) -> Set<String> {
guard let node = try? PureXML.parse(source, limits: .init(allowDoctype: true)) else { return [] }
var result: Set<String> = []
for kind in ["import", "include", "redefine"] {
for element in elements(named: kind, under: node) {
guard let location = element.attributes.first(where: { $0.name.localName == "schemaLocation" })?.value else { continue }
result.insert(String(location.split(separator: "/").last ?? ""))
}
}
return result
}
private func runInstanceTests(
_ group: PureXML.Model.Element,
name: String,
directory: String,
schema: PureXML.Schema.Document,
into counters: inout Counters,
) {
for instanceTest in elements(named: "instanceTest", under: .element(group)) {
guard isAccepted(instanceTest) else { continue }
let expected = expectedValidity(instanceTest)
guard expected == "valid" || expected == "invalid",
let href = references(in: .element(instanceTest), named: "instanceDocument").first
else { continue }
let instancePath = resolve(href, against: directory)
guard let xml = try? String(contentsOfFile: instancePath, encoding: .utf8) else { continue }
// Resolve the instance's xsi:schemaLocation hints relative to the
// instance file, so a strict wildcard can find declarations in them.
let instanceDirectory = self.directory(of: instancePath)
let loader: (String) -> String? = { location in
(try? String(contentsOfFile: resolve(location, against: instanceDirectory), encoding: .utf8))
}
let isValid = ((try? schema.validate(xml, schemaLoader: loader))?.isEmpty) ?? false
if expected == "valid", !isValid {
counters.instanceValidRejected += 1
counters.note("\(name)/\(attribute("name", of: instanceTest) ?? "?"): valid instance rejected")
} else if expected == "invalid", isValid {
counters.instanceInvalidAccepted += 1
counters.note("\(name)/\(attribute("name", of: instanceTest) ?? "?"): invalid instance accepted")
} else {
counters.instanceAgreed += 1
}
}
}
// MARK: Manifest helpers
private func parsed(at path: String) throws -> PureXML.Model.Node {
try PureXML.parse(String(contentsOfFile: path, encoding: .utf8), limits: .init(allowDoctype: true))
}
private func elements(named name: String, under node: PureXML.Model.Node) -> [PureXML.Model.Element] {
var found: [PureXML.Model.Element] = []
switch node {
case let .document(children):
for child in children {
found += elements(named: name, under: child)
}
case let .element(element):
if element.name.localName == name { found.append(element) }
for child in element.children {
found += elements(named: name, under: child)
}
default:
break
}
return found
}
private func references(in node: PureXML.Model.Node, named name: String) -> [String] {
elements(named: name, under: node).compactMap { element in
element.attributes.first { $0.name.localName == "href" }?.value
}
}
private func attribute(_ name: String, of element: PureXML.Model.Element) -> String? {
element.attributes.first { $0.name.localName == name }?.value
}
/// Whether a schemaTest/instanceTest's expected outcome is settled. The suite
/// marks disputed entries `<current status="queried">` (often with a W3C
/// Bugzilla reference) where the expected validity is itself contested; those
/// are not authoritative, so they are not counted as agreements or deviations.
private func isAccepted(_ test: PureXML.Model.Element) -> Bool {
guard let current = elements(named: "current", under: .element(test)).first else { return true }
return (attribute("status", of: current) ?? "accepted") == "accepted"
}
private func expectedValidity(_ element: PureXML.Model.Element) -> String {
elements(named: "expected", under: .element(element)).first
.flatMap { attribute("validity", of: $0) } ?? "notKnown"
}
private func directory(of path: String) -> String {
guard let slash = path.lastIndex(of: "/") else { return "." }
return String(path[..<slash])
}
/// Resolves an href against a base directory, folding `..` segments.
private func resolve(_ href: String, against base: String) -> String {
var parts = base.split(separator: "/", omittingEmptySubsequences: false).map(String.init)
for segment in href.split(separator: "/") {
if segment == ".." {
if !parts.isEmpty { parts.removeLast() }
} else if segment != "." {
parts.append(String(segment))
}
}
return parts.joined(separator: "/")
}
}