test(conformance): RELAX NG spec suite runner with exact baselines
James Clark's spectest.xml runs opt-in via RNG_TS_ROOT: nested testSuite scopes thread resource/dir definitions into the compiler loader, and each case asserts schema compile/reject plus per-instance accept/reject. First measurement carried as exact baselines: all correct schemas compile, valid 233/289, invalid 277/291; schema correctness is not yet validated (all 213 incorrect schemas compile, count asserted). Burn-down classes recorded: grammar-level schema validation with the 4.16-4.18 restrictions, foreign-namespace stripping, xml:base-aware resource resolution, and the section 6 compatibility datatypes.
This commit is contained in:
parent
ddfe14370c
commit
b6981baf19
@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Added
|
||||
|
||||
- RELAX NG spec test suite runner (#122, #131): James Clark's spectest.xml (385 cases, distributed with jing-trang) runs via an opt-in RNG_TS_ROOT runner, never vendored; the manifest is parsed with PureXML itself, nested testSuite scopes thread their resource/dir definitions into the compiler's loader, and each case asserts compile/reject for the schema plus accept/reject per instance. First measurement, carried as exact baselines: every correct schema compiles and most instances behave (valid 233/289 accepted, invalid 277/291 rejected), while schema-correctness validation is absent (all 213 incorrect schemas compile, count asserted exactly). The burn-down classes are recorded in the runner: schema validation against the RELAX NG grammar and the 4.16-4.18 restrictions, foreign-namespace stripping (4.1), xml:base-aware include/externalRef resolution (4.5), and the section 6 compatibility datatypes.
|
||||
- Canonical XML 1.0 spec-example vectors (#122, #132): the worked examples of the C14N 1.0 Recommendation's section 3 run as input/expected-output tests (cited per example; used under the W3C Software and Document License), all six passing: top-level PIs/comments with and without comments mode, whitespace preservation, start/end tags with attribute sorting and DTD defaults, character modifications and references, entity expansion, and UTF-8 output from legacy-encoded input. The vectors exposed three real conformance fixes: top-level siblings in the canonical form are now separated by single line feeds; an empty default-namespace declaration renders only when it undeclares a rendered non-empty default (superfluous `xmlns=""` is dropped, inclusive mode); and the text/attribute escape loops iterate Unicode scalars, so a carriage return that grapheme-clusters with a following line feed still escapes to `
` (the #135 class, surfacing in output rather than lexing).
|
||||
- Scalar-level lexing (#135): the reader now buffers exactly one Unicode scalar per element (multi-scalar graphemes from any source are split, string input iterates scalars directly), so a combining mark directly after an ASCII delimiter can no longer merge with it into one Swift grapheme cluster and XML's scalar-defined name productions classify correctly: U+06D6 lexes as a Fifth Edition PI-target NameStartChar, ZWNJ/ZWJ attribute names parse, and U+309A markup from entity expansion forms a real element. The replacement-text, content-model, and ATTLIST sub-scanners and the entity-literal validator go scalar-level the same way, while grapheme clusters reassemble naturally wherever scanned text is appended to a String (asserted by test). This clears the last eduni baseline (now empty), grows the IBM pre-5e class from 279 to 300 exactly as the Fifth Edition prescribes, and, combined with the EDITION="1 2 3 4" exclusions the xmltest manifest itself carries for cases 140/141, empties the xmltest deviation baseline entirely: every xmlconf case applicable to a Fifth Edition, namespace-aware processor now passes.
|
||||
|
||||
|
||||
@ -101,7 +101,7 @@ flowchart TB
|
||||
Epic122["#122 official spec suites"]:::epic
|
||||
XSTS["#129 W3C XSD test suite (XSTS)"]:::todo
|
||||
XSLTTS["#130 OASIS XSLT 1.0 suite (+XPath corpus)"]:::todo
|
||||
RNGTS["#131 RELAX NG spec test suite"]:::todo
|
||||
RNGTS["#131 RELAX NG spec suite: runner + baselines in, burning down"]:::review
|
||||
C14NTS["#132 C14N spec vectors: complete"]:::done
|
||||
Epic105 --> Epic121
|
||||
Epic105 --> Epic122
|
||||
@ -125,7 +125,7 @@ flowchart TB
|
||||
|
||||
PureXML is a working, dependency-free XML library today: parse, emit, validate,
|
||||
query, and stream documents on macOS, Linux, Windows, and WASM. The test suite
|
||||
currently runs **931 tests in 159 suites** (`swift test`).
|
||||
currently runs **932 tests in 160 suites** (`swift test`).
|
||||
|
||||
### Shipped (libxml2-aligned surface)
|
||||
|
||||
|
||||
212
Tests/RelaxNGSpecSuiteTests.swift
Normal file
212
Tests/RelaxNGSpecSuiteTests.swift
Normal file
@ -0,0 +1,212 @@
|
||||
import Foundation
|
||||
@testable import PureXML
|
||||
import Testing
|
||||
|
||||
/// Opt-in runner for James Clark's RELAX NG spec test suite (`spectest.xml`,
|
||||
/// distributed with jing-trang under its BSD license). Point `RNG_TS_ROOT` at
|
||||
/// the directory containing `spectest.xml` to run; without the variable the
|
||||
/// suite is skipped and it is never vendored into this repository. The
|
||||
/// manifest nests testSuite/testCase elements: an `incorrect` schema must
|
||||
/// fail to compile, a `correct` schema must compile and then `valid`
|
||||
/// instances validate while `invalid` instances do not. `resource` and `dir`
|
||||
/// elements model the include/externalRef files a case references, served to
|
||||
/// the compiler through its loader keyed by relative path. Foundation is used
|
||||
/// for file access only.
|
||||
@Suite("RELAX NG spec suite (opt-in via RNG_TS_ROOT)")
|
||||
struct RelaxNGSpecSuiteTests {
|
||||
private var root: String? {
|
||||
ProcessInfo.processInfo.environment["RNG_TS_ROOT"]
|
||||
}
|
||||
|
||||
/// The burn-down frontier, by case index in document order (exact, so
|
||||
/// progress and regressions both show). Three classes:
|
||||
/// 1. Schema correctness is not validated: every `incorrect` schema
|
||||
/// compiles (the count is asserted exactly below). Closing it means
|
||||
/// validating schemas against the RELAX NG grammar and the section
|
||||
/// 4.16-4.18 restrictions before pattern interpretation.
|
||||
/// 2. Simplification gaps reject valid instances: foreign-namespace
|
||||
/// element/attribute stripping (4.1: cases 90-98), xml:base-aware
|
||||
/// include/externalRef resolution (4.5: 100-101), include/div/override
|
||||
/// depth (4.6-4.12), the 4.17-4.18 normalization corners, and the
|
||||
/// section 6 compatibility datatypes (ID/IDREF, 6.2.x).
|
||||
/// 3. The same classes accept some `invalid` instances.
|
||||
private let knownIncorrectCompiledCount = 213
|
||||
private let knownCorrectRejected: Set<Int> = []
|
||||
private let knownValidRejected: Set<Int> = [
|
||||
90, 92, 93, 94, 95, 96, 98, 100, 101, 104, 108, 109, 110, 111, 115,
|
||||
119, 120, 124, 125, 126, 127, 128, 130, 131, 132, 133, 142, 190, 191,
|
||||
194, 195, 208, 209, 210, 236, 256, 258, 264, 265, 266, 268, 269, 271,
|
||||
328, 372, 374, 378, 379, 380,
|
||||
]
|
||||
private let knownInvalidAccepted: Set<Int> = [
|
||||
101, 104, 124, 125, 126, 127, 128, 142, 264, 271,
|
||||
]
|
||||
|
||||
private struct SpecCase {
|
||||
let index: Int
|
||||
let section: String
|
||||
let schema: String
|
||||
let mustCompile: Bool
|
||||
let validInstances: [String]
|
||||
let invalidInstances: [String]
|
||||
let resources: [String: String]
|
||||
}
|
||||
|
||||
private final class Collector {
|
||||
var cases: [SpecCase] = []
|
||||
var counter = 0
|
||||
}
|
||||
|
||||
private func loadCases(from path: String) throws -> [SpecCase] {
|
||||
let source = try String(contentsOfFile: path, encoding: .utf8)
|
||||
let document = try PureXML.parse(source, limits: .init(allowDoctype: true))
|
||||
guard case let .document(children) = document,
|
||||
let suite = children.compactMap(\.element).first
|
||||
else { return [] }
|
||||
let collector = Collector()
|
||||
collect(suite, resources: [:], collector: collector)
|
||||
return collector.cases
|
||||
}
|
||||
|
||||
/// Walks nested testSuite elements, accumulating the `resource`/`dir`
|
||||
/// definitions in scope, and turns each testCase into a SpecCase.
|
||||
private func collect(_ element: PureXML.Model.Element, resources: [String: String], collector: Collector) {
|
||||
var scope = resources
|
||||
for child in element.children.compactMap(\.element) {
|
||||
switch child.name.localName {
|
||||
case "resource", "dir":
|
||||
merge(child, prefix: "", into: &scope)
|
||||
default:
|
||||
break
|
||||
}
|
||||
}
|
||||
for child in element.children.compactMap(\.element) {
|
||||
switch child.name.localName {
|
||||
case "testSuite":
|
||||
collect(child, resources: scope, collector: collector)
|
||||
case "testCase":
|
||||
appendCase(child, resources: scope, collector: collector)
|
||||
default:
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Files a `resource` (name -> serialized content) or recurses into a
|
||||
/// `dir`, prefixing names with the directory path.
|
||||
private func merge(_ element: PureXML.Model.Element, prefix: String, into scope: inout [String: String]) {
|
||||
let name = element.attributes.first { $0.name.description == "name" }?.value ?? ""
|
||||
if element.name.localName == "resource" {
|
||||
if let content = firstElementChild(element) {
|
||||
scope[prefix + name] = PureXML.serialize(.element(content))
|
||||
}
|
||||
return
|
||||
}
|
||||
for child in element.children.compactMap(\.element) {
|
||||
merge(child, prefix: prefix + name + "/", into: &scope)
|
||||
}
|
||||
}
|
||||
|
||||
private func firstElementChild(_ element: PureXML.Model.Element) -> PureXML.Model.Element? {
|
||||
element.children.compactMap(\.element).first
|
||||
}
|
||||
|
||||
/// The wrapper's first element child, serialized back to markup.
|
||||
private func serializedChild(_ element: PureXML.Model.Element) -> String? {
|
||||
firstElementChild(element).map { PureXML.serialize(.element($0)) }
|
||||
}
|
||||
|
||||
private func appendCase(_ testCase: PureXML.Model.Element, resources: [String: String], collector: Collector) {
|
||||
var scope = resources
|
||||
var section = ""
|
||||
var schema: String?
|
||||
var mustCompile = true
|
||||
var valid: [String] = []
|
||||
var invalid: [String] = []
|
||||
for child in testCase.children.compactMap(\.element) {
|
||||
switch child.name.localName {
|
||||
case "section":
|
||||
section = child.children.compactMap { if case let .text(value) = $0 { value } else { nil } }.joined()
|
||||
case "resource", "dir":
|
||||
merge(child, prefix: "", into: &scope)
|
||||
case "correct", "incorrect":
|
||||
mustCompile = child.name.localName == "correct"
|
||||
schema = serializedChild(child)
|
||||
case "valid":
|
||||
serializedChild(child).map { valid.append($0) }
|
||||
case "invalid":
|
||||
serializedChild(child).map { invalid.append($0) }
|
||||
default:
|
||||
break
|
||||
}
|
||||
}
|
||||
guard let schema else { return }
|
||||
collector.counter += 1
|
||||
collector.cases.append(SpecCase(
|
||||
index: collector.counter,
|
||||
section: section,
|
||||
schema: schema,
|
||||
mustCompile: mustCompile,
|
||||
validInstances: valid,
|
||||
invalidInstances: invalid,
|
||||
resources: scope,
|
||||
))
|
||||
}
|
||||
|
||||
@Test("Every spec case behaves: compile, reject, validate, invalidate")
|
||||
func test_specSuite() throws {
|
||||
guard let root else {
|
||||
return // Opt-in: suite not present.
|
||||
}
|
||||
let cases = try loadCases(from: root + "/spectest.xml")
|
||||
var incorrectAccepted: [Int] = []
|
||||
var correctRejected: [String: String] = [:]
|
||||
var validRejected: [String] = []
|
||||
var invalidAccepted: [String] = []
|
||||
for specCase in cases {
|
||||
let resources = specCase.resources
|
||||
let loader: (String) -> String? = { resources[$0] }
|
||||
let compiled = try? PureXML.Schema.RelaxNG(specCase.schema, schemaLoader: loader)
|
||||
guard specCase.mustCompile else {
|
||||
if compiled != nil {
|
||||
incorrectAccepted.append(specCase.index)
|
||||
}
|
||||
continue
|
||||
}
|
||||
guard let compiled else {
|
||||
if !knownCorrectRejected.contains(specCase.index) {
|
||||
correctRejected["#\(specCase.index) §\(specCase.section)"] = compileError(specCase, loader: loader)
|
||||
}
|
||||
continue
|
||||
}
|
||||
for (offset, instance) in specCase.validInstances.enumerated() {
|
||||
let accepted = (try? compiled.validate(instance)) == true
|
||||
if !accepted, !knownValidRejected.contains(specCase.index) {
|
||||
validRejected.append("#\(specCase.index).\(offset) §\(specCase.section)")
|
||||
}
|
||||
}
|
||||
for (offset, instance) in specCase.invalidInstances.enumerated() {
|
||||
let accepted = (try? compiled.validate(instance)) == true
|
||||
if accepted, !knownInvalidAccepted.contains(specCase.index) {
|
||||
invalidAccepted.append("#\(specCase.index).\(offset) §\(specCase.section)")
|
||||
}
|
||||
}
|
||||
}
|
||||
#expect(
|
||||
incorrectAccepted.count == knownIncorrectCompiledCount,
|
||||
"incorrect-schema class drifted: \(incorrectAccepted.count) (baseline \(knownIncorrectCompiledCount))",
|
||||
)
|
||||
#expect(correctRejected.isEmpty, "rejected \(correctRejected.count) correct schemas: \(correctRejected)")
|
||||
#expect(validRejected.isEmpty, "rejected \(validRejected.count) valid instances: \(validRejected)")
|
||||
#expect(invalidAccepted.isEmpty, "accepted \(invalidAccepted.count) invalid instances: \(invalidAccepted)")
|
||||
}
|
||||
|
||||
private func compileError(_ specCase: SpecCase, loader: @escaping (String) -> String?) -> String {
|
||||
do {
|
||||
_ = try PureXML.Schema.RelaxNG(specCase.schema, schemaLoader: loader)
|
||||
return "compiled on retry"
|
||||
} catch {
|
||||
return String(describing: error)
|
||||
}
|
||||
}
|
||||
}
|
||||
Loading…
Reference in New Issue
Block a user