test(conformance): RELAX NG spec suite runner with exact baselines

James Clark's spectest.xml runs opt-in via RNG_TS_ROOT: nested
testSuite scopes thread resource/dir definitions into the compiler
loader, and each case asserts schema compile/reject plus per-instance
accept/reject. First measurement carried as exact baselines: all
correct schemas compile, valid 233/289, invalid 277/291; schema
correctness is not yet validated (all 213 incorrect schemas compile,
count asserted). Burn-down classes recorded: grammar-level schema
validation with the 4.16-4.18 restrictions, foreign-namespace
stripping, xml:base-aware resource resolution, and the section 6
compatibility datatypes.
This commit is contained in:
Mihaela Mihaljevic 2026-06-10 09:10:13 +02:00
parent ddfe14370c
commit b6981baf19
3 changed files with 215 additions and 2 deletions

View File

@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- RELAX NG spec test suite runner (#122, #131): James Clark's spectest.xml (385 cases, distributed with jing-trang) runs via an opt-in RNG_TS_ROOT runner, never vendored; the manifest is parsed with PureXML itself, nested testSuite scopes thread their resource/dir definitions into the compiler's loader, and each case asserts compile/reject for the schema plus accept/reject per instance. First measurement, carried as exact baselines: every correct schema compiles and most instances behave (valid 233/289 accepted, invalid 277/291 rejected), while schema-correctness validation is absent (all 213 incorrect schemas compile, count asserted exactly). The burn-down classes are recorded in the runner: schema validation against the RELAX NG grammar and the 4.16-4.18 restrictions, foreign-namespace stripping (4.1), xml:base-aware include/externalRef resolution (4.5), and the section 6 compatibility datatypes.
- Canonical XML 1.0 spec-example vectors (#122, #132): the worked examples of the C14N 1.0 Recommendation's section 3 run as input/expected-output tests (cited per example; used under the W3C Software and Document License), all six passing: top-level PIs/comments with and without comments mode, whitespace preservation, start/end tags with attribute sorting and DTD defaults, character modifications and references, entity expansion, and UTF-8 output from legacy-encoded input. The vectors exposed three real conformance fixes: top-level siblings in the canonical form are now separated by single line feeds; an empty default-namespace declaration renders only when it undeclares a rendered non-empty default (superfluous `xmlns=""` is dropped, inclusive mode); and the text/attribute escape loops iterate Unicode scalars, so a carriage return that grapheme-clusters with a following line feed still escapes to `
` (the #135 class, surfacing in output rather than lexing).
- Scalar-level lexing (#135): the reader now buffers exactly one Unicode scalar per element (multi-scalar graphemes from any source are split, string input iterates scalars directly), so a combining mark directly after an ASCII delimiter can no longer merge with it into one Swift grapheme cluster and XML's scalar-defined name productions classify correctly: U+06D6 lexes as a Fifth Edition PI-target NameStartChar, ZWNJ/ZWJ attribute names parse, and U+309A markup from entity expansion forms a real element. The replacement-text, content-model, and ATTLIST sub-scanners and the entity-literal validator go scalar-level the same way, while grapheme clusters reassemble naturally wherever scanned text is appended to a String (asserted by test). This clears the last eduni baseline (now empty), grows the IBM pre-5e class from 279 to 300 exactly as the Fifth Edition prescribes, and, combined with the EDITION="1 2 3 4" exclusions the xmltest manifest itself carries for cases 140/141, empties the xmltest deviation baseline entirely: every xmlconf case applicable to a Fifth Edition, namespace-aware processor now passes.

View File

@ -101,7 +101,7 @@ flowchart TB
Epic122["#122 official spec suites"]:::epic
XSTS["#129 W3C XSD test suite (XSTS)"]:::todo
XSLTTS["#130 OASIS XSLT 1.0 suite (+XPath corpus)"]:::todo
RNGTS["#131 RELAX NG spec test suite"]:::todo
RNGTS["#131 RELAX NG spec suite: runner + baselines in, burning down"]:::review
C14NTS["#132 C14N spec vectors: complete"]:::done
Epic105 --> Epic121
Epic105 --> Epic122
@ -125,7 +125,7 @@ flowchart TB
PureXML is a working, dependency-free XML library today: parse, emit, validate,
query, and stream documents on macOS, Linux, Windows, and WASM. The test suite
currently runs **931 tests in 159 suites** (`swift test`).
currently runs **932 tests in 160 suites** (`swift test`).
### Shipped (libxml2-aligned surface)

View File

@ -0,0 +1,212 @@
import Foundation
@testable import PureXML
import Testing
/// Opt-in runner for James Clark's RELAX NG spec test suite (`spectest.xml`,
/// distributed with jing-trang under its BSD license). Point `RNG_TS_ROOT` at
/// the directory containing `spectest.xml` to run; without the variable the
/// suite is skipped and it is never vendored into this repository. The
/// manifest nests testSuite/testCase elements: an `incorrect` schema must
/// fail to compile, a `correct` schema must compile and then `valid`
/// instances validate while `invalid` instances do not. `resource` and `dir`
/// elements model the include/externalRef files a case references, served to
/// the compiler through its loader keyed by relative path. Foundation is used
/// for file access only.
@Suite("RELAX NG spec suite (opt-in via RNG_TS_ROOT)")
struct RelaxNGSpecSuiteTests {
private var root: String? {
ProcessInfo.processInfo.environment["RNG_TS_ROOT"]
}
/// The burn-down frontier, by case index in document order (exact, so
/// progress and regressions both show). Three classes:
/// 1. Schema correctness is not validated: every `incorrect` schema
/// compiles (the count is asserted exactly below). Closing it means
/// validating schemas against the RELAX NG grammar and the section
/// 4.16-4.18 restrictions before pattern interpretation.
/// 2. Simplification gaps reject valid instances: foreign-namespace
/// element/attribute stripping (4.1: cases 90-98), xml:base-aware
/// include/externalRef resolution (4.5: 100-101), include/div/override
/// depth (4.6-4.12), the 4.17-4.18 normalization corners, and the
/// section 6 compatibility datatypes (ID/IDREF, 6.2.x).
/// 3. The same classes accept some `invalid` instances.
private let knownIncorrectCompiledCount = 213
private let knownCorrectRejected: Set<Int> = []
private let knownValidRejected: Set<Int> = [
90, 92, 93, 94, 95, 96, 98, 100, 101, 104, 108, 109, 110, 111, 115,
119, 120, 124, 125, 126, 127, 128, 130, 131, 132, 133, 142, 190, 191,
194, 195, 208, 209, 210, 236, 256, 258, 264, 265, 266, 268, 269, 271,
328, 372, 374, 378, 379, 380,
]
private let knownInvalidAccepted: Set<Int> = [
101, 104, 124, 125, 126, 127, 128, 142, 264, 271,
]
private struct SpecCase {
let index: Int
let section: String
let schema: String
let mustCompile: Bool
let validInstances: [String]
let invalidInstances: [String]
let resources: [String: String]
}
private final class Collector {
var cases: [SpecCase] = []
var counter = 0
}
private func loadCases(from path: String) throws -> [SpecCase] {
let source = try String(contentsOfFile: path, encoding: .utf8)
let document = try PureXML.parse(source, limits: .init(allowDoctype: true))
guard case let .document(children) = document,
let suite = children.compactMap(\.element).first
else { return [] }
let collector = Collector()
collect(suite, resources: [:], collector: collector)
return collector.cases
}
/// Walks nested testSuite elements, accumulating the `resource`/`dir`
/// definitions in scope, and turns each testCase into a SpecCase.
private func collect(_ element: PureXML.Model.Element, resources: [String: String], collector: Collector) {
var scope = resources
for child in element.children.compactMap(\.element) {
switch child.name.localName {
case "resource", "dir":
merge(child, prefix: "", into: &scope)
default:
break
}
}
for child in element.children.compactMap(\.element) {
switch child.name.localName {
case "testSuite":
collect(child, resources: scope, collector: collector)
case "testCase":
appendCase(child, resources: scope, collector: collector)
default:
break
}
}
}
/// Files a `resource` (name -> serialized content) or recurses into a
/// `dir`, prefixing names with the directory path.
private func merge(_ element: PureXML.Model.Element, prefix: String, into scope: inout [String: String]) {
let name = element.attributes.first { $0.name.description == "name" }?.value ?? ""
if element.name.localName == "resource" {
if let content = firstElementChild(element) {
scope[prefix + name] = PureXML.serialize(.element(content))
}
return
}
for child in element.children.compactMap(\.element) {
merge(child, prefix: prefix + name + "/", into: &scope)
}
}
private func firstElementChild(_ element: PureXML.Model.Element) -> PureXML.Model.Element? {
element.children.compactMap(\.element).first
}
/// The wrapper's first element child, serialized back to markup.
private func serializedChild(_ element: PureXML.Model.Element) -> String? {
firstElementChild(element).map { PureXML.serialize(.element($0)) }
}
private func appendCase(_ testCase: PureXML.Model.Element, resources: [String: String], collector: Collector) {
var scope = resources
var section = ""
var schema: String?
var mustCompile = true
var valid: [String] = []
var invalid: [String] = []
for child in testCase.children.compactMap(\.element) {
switch child.name.localName {
case "section":
section = child.children.compactMap { if case let .text(value) = $0 { value } else { nil } }.joined()
case "resource", "dir":
merge(child, prefix: "", into: &scope)
case "correct", "incorrect":
mustCompile = child.name.localName == "correct"
schema = serializedChild(child)
case "valid":
serializedChild(child).map { valid.append($0) }
case "invalid":
serializedChild(child).map { invalid.append($0) }
default:
break
}
}
guard let schema else { return }
collector.counter += 1
collector.cases.append(SpecCase(
index: collector.counter,
section: section,
schema: schema,
mustCompile: mustCompile,
validInstances: valid,
invalidInstances: invalid,
resources: scope,
))
}
@Test("Every spec case behaves: compile, reject, validate, invalidate")
func test_specSuite() throws {
guard let root else {
return // Opt-in: suite not present.
}
let cases = try loadCases(from: root + "/spectest.xml")
var incorrectAccepted: [Int] = []
var correctRejected: [String: String] = [:]
var validRejected: [String] = []
var invalidAccepted: [String] = []
for specCase in cases {
let resources = specCase.resources
let loader: (String) -> String? = { resources[$0] }
let compiled = try? PureXML.Schema.RelaxNG(specCase.schema, schemaLoader: loader)
guard specCase.mustCompile else {
if compiled != nil {
incorrectAccepted.append(specCase.index)
}
continue
}
guard let compiled else {
if !knownCorrectRejected.contains(specCase.index) {
correctRejected["#\(specCase.index) §\(specCase.section)"] = compileError(specCase, loader: loader)
}
continue
}
for (offset, instance) in specCase.validInstances.enumerated() {
let accepted = (try? compiled.validate(instance)) == true
if !accepted, !knownValidRejected.contains(specCase.index) {
validRejected.append("#\(specCase.index).\(offset) §\(specCase.section)")
}
}
for (offset, instance) in specCase.invalidInstances.enumerated() {
let accepted = (try? compiled.validate(instance)) == true
if accepted, !knownInvalidAccepted.contains(specCase.index) {
invalidAccepted.append("#\(specCase.index).\(offset) §\(specCase.section)")
}
}
}
#expect(
incorrectAccepted.count == knownIncorrectCompiledCount,
"incorrect-schema class drifted: \(incorrectAccepted.count) (baseline \(knownIncorrectCompiledCount))",
)
#expect(correctRejected.isEmpty, "rejected \(correctRejected.count) correct schemas: \(correctRejected)")
#expect(validRejected.isEmpty, "rejected \(validRejected.count) valid instances: \(validRejected)")
#expect(invalidAccepted.isEmpty, "accepted \(invalidAccepted.count) invalid instances: \(invalidAccepted)")
}
private func compileError(_ specCase: SpecCase, loader: @escaping (String) -> String?) -> String {
do {
_ = try PureXML.Schema.RelaxNG(specCase.schema, schemaLoader: loader)
return "compiled on retry"
} catch {
return String(describing: error)
}
}
}