feat(parsing): first W3C-suite strictness pass; opt-in xmltest runner
Add five well-formedness checks the W3C xmltest suite surfaced: '--' inside comments (also rejecting the ---> ending), raw '<' in attribute values, the literal ]]> sequence in character data, attributes not separated by whitespace, and characters outside the XML Char production in content, each a located ParseError. The attribute scanners move to EventReaderAttributes to stay under the type-body cap. Add the opt-in W3C xmltest runner (XMLTS_ROOT-gated; the suite is never vendored, James Clark's license permits redistribution only as the unmodified archive; Foundation is used in this one test file for file access only). Baseline: valid/sa 118/119, not-wf/sa 93/186 rejected (up from 80), the remaining 93 carried as an exact known-deviations baseline so progress and regressions are both visible. The two remaining leniency classes (internal- subset DTD grammar, entity replacement-text well-formedness) are #120. Part of #105 (Tier 2)
This commit is contained in:
parent
953c688f62
commit
4ed8e9518a
@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Added
|
||||
|
||||
- Parser strictness, first W3C-suite pass (#105 Tier 2, #120): the strict parser now rejects `--` inside comments (including the `--->` ending), a raw `<` in attribute values, the literal `]]>` sequence in character data, attributes not separated by whitespace, and characters outside the XML `Char` production in content, five new located `ParseError` cases. Driven by a new opt-in runner for the W3C XML Conformance Test Suite's xmltest section (`XMLTS_ROOT`-gated; the suite is never vendored because its license permits redistribution only as the unmodified archive): valid/sa 118/119, not-wf/sa rejection improved from 80 to 93 of 186, with the exact remaining baseline carried in the test and the two remaining leniency classes (internal-subset DTD grammar, entity replacement-text well-formedness) tracked as #120.
|
||||
- XSD Particle Valid (Restriction) (#117): a complex type derived by restriction is now checked at schema compile to accept a structural subset of its base, covering element name matching, element-vs-wildcard admission, wildcard narrowing, occurrence-range subsumption, and the group recursions (sequence/all order-preserving with emptiable skips, choice mapping, sequence-into-choice, sequence-into-all, RecurseAsIfGroup). A structurally unfaithful restriction (widened occurrence, new element name, reordered sequence, content added to an EMPTY or required-into-empty base, an element outside a base wildcard's namespace) throws `SchemaError.invalidRestriction` with the type, base, and reason. The spec's effective-total-range arithmetic for MapAndSum is approximated by per-particle checks, a documented simplification.
|
||||
- Conformance corpus harness (#105, Tier 2): the start of conformance-suite checking, expressed through the validation framework (the OpenAPIKit idiom). A `PureXML.Validation.ConformanceCase` is a `Validatable` named expectation (actual vs spec-authoritative expected output), and `Conformance.matchesExpected` is a `Validation<ConformanceCase, Void>` rule that emits one located `ValidationError` per diverging case, so a whole suite reports every failure at once with the case name as its coding path. Seeded with a Canonical XML 1.0 corpus (attribute ordering, empty-element expansion, comment removal, CDATA-to-escaped-text, processing instructions, namespace rendering) an XPath 1.0 core-function corpus, and an XSD datatype/facet corpus (length, range, pattern, enumeration, lexical space), a RELAX NG pattern corpus (text, optional, repetition, choice, ordered group, attribute, interleave, empty), an XSLT 1.0 transformation corpus (value-of, for-each, if, choose, sort, variable, format-number, apply-templates, call-template), an HTML5 tree-construction corpus (void elements, implied tag closing, case normalization, the adoption agency, entities), a Schematron corpus (assert, report, rule context), a C14N exclusive/2.0 corpus (unused-namespace dropping, render-at-use, whitespace trimming), an XPath axes/predicates corpus (descendant, ancestor, sibling axes, positional and attribute predicates, union), a DTD content-model corpus (sequence, choice, occurrence indicators, mixed, EMPTY, ANY), an XInclude/XPointer corpus (substitution, text inclusion with escaping, fallback, relative href resolution; shorthand ids, element() navigation, xpointer() including the per-step predicate semantic), an XSD regex corpus (implicit whole-string anchoring, the character-class escapes, Unicode categories, quantifier ranges, alternation, class negation and subtraction), and an XML Catalog corpus (system/public/uri entries, rewrite prefixes, unmatched identifiers). The HTML corpus surfaced its first divergence (the fragment parser does not reconstruct active formatting elements like the document parser, tracked as #109). Growing the corpora against the official W3C/OASIS suites is the ongoing Tier-2 work.
|
||||
- EXSLT extension functions (#105, Tier 3): the `common`, `math`, and `sets` EXSLT modules, dispatched by resolving a function's prefix to its namespace (so any prefix bound to an EXSLT namespace works). `math:min`, `math:max`, `math:highest`, `math:lowest`, `math:abs`, `math:sqrt` (with the EXSLT NaN rule for non-numeric nodes); `set:distinct`, `set:difference`, `set:intersection`, `set:has-same-node`, `set:leading`, `set:trailing` (by node identity and document order); `exsl:object-type`. `math:power` and the other transcendentals are omitted because they need a C math library, which the pure-Swift, Foundation-free target excludes; `exsl:node-set` awaits result-tree-fragment support.
|
||||
|
||||
@ -99,7 +99,7 @@ flowchart TB
|
||||
|
||||
PureXML is a working, dependency-free XML library today: parse, emit, validate,
|
||||
query, and stream documents on macOS, Linux, Windows, and WASM. The test suite
|
||||
currently runs **844 tests in 140 suites** (`swift test`).
|
||||
currently runs **846 tests in 141 suites** (`swift test`).
|
||||
|
||||
### Shipped (libxml2-aligned surface)
|
||||
|
||||
|
||||
@ -37,7 +37,10 @@ public extension PureXML.Parsing.ParseError {
|
||||
case let .unexpectedEndOfInput(mark), let .unterminatedTag(mark), let .unterminatedComment(mark),
|
||||
let .unterminatedCDATA(mark), let .expectedName(mark), let .expectedEquals(mark),
|
||||
let .unquotedAttributeValue(mark), let .amplificationLimitExceeded(mark),
|
||||
let .junkAfterDocumentElement(mark), let .unsupportedDoctype(mark), let .malformedDeclaration(mark):
|
||||
let .junkAfterDocumentElement(mark), let .unsupportedDoctype(mark), let .malformedDeclaration(mark),
|
||||
let .doubleHyphenInComment(mark), let .rawLessThanInAttribute(mark),
|
||||
let .cdataCloseInContent(mark), let .missingSpaceBeforeAttribute(mark),
|
||||
let .invalidCharacter(mark):
|
||||
mark
|
||||
case let .unexpectedCharacter(_, mark), let .mismatchedEndTag(_, _, mark),
|
||||
let .duplicateAttribute(_, mark), let .invalidReference(_, mark),
|
||||
|
||||
@ -23,14 +23,14 @@ public extension PureXML.Parsing {
|
||||
/// The parsed XML declaration `<?xml ... ?>`, when the document opens with
|
||||
/// one; nil otherwise.
|
||||
var xmlDeclaration: XMLDeclaration?
|
||||
private var entityBudget: Int
|
||||
var entityBudget: Int
|
||||
|
||||
var pending: [Event] = []
|
||||
private var sawRoot = false
|
||||
private var primed = false
|
||||
/// When true, ``next()`` repairs malformed input in place instead of
|
||||
/// throwing: it records a ``Diagnostic`` and emits best-effort events.
|
||||
private let recovering: Bool
|
||||
let recovering: Bool
|
||||
var reachedEnd = false
|
||||
/// The problems found so far, when reading in recovering mode.
|
||||
var diagnostics: [Diagnostic] = []
|
||||
@ -202,6 +202,12 @@ public extension PureXML.Parsing {
|
||||
while let character = reader.peek(), character != "<" {
|
||||
length += 1
|
||||
try checkContent(length, mark)
|
||||
if character == ">", raw.hasSuffix("]]") {
|
||||
throw ParseError.cdataCloseInContent(reader.mark)
|
||||
}
|
||||
guard character.unicodeScalars.allSatisfy(XMLCharacter.isChar) else {
|
||||
throw ParseError.invalidCharacter(reader.mark)
|
||||
}
|
||||
raw.append(character)
|
||||
reader.advance()
|
||||
}
|
||||
@ -227,56 +233,7 @@ public extension PureXML.Parsing {
|
||||
return (target.description, data)
|
||||
}
|
||||
|
||||
private mutating func scanAttributes() throws -> [PureXML.Model.Attribute] {
|
||||
var attributes: [PureXML.Model.Attribute] = []
|
||||
var seen: Set<String> = []
|
||||
while true {
|
||||
reader.skipSpace()
|
||||
guard let next = reader.peek(), next != ">", next != "/" else {
|
||||
return attributes
|
||||
}
|
||||
let mark = reader.mark
|
||||
let name = try scanName()
|
||||
reader.skipSpace()
|
||||
guard reader.consume("=") else {
|
||||
throw ParseError.expectedEquals(reader.mark)
|
||||
}
|
||||
reader.skipSpace()
|
||||
let value = try scanAttributeValue()
|
||||
guard seen.insert(name.description).inserted else {
|
||||
if recovering {
|
||||
// Keep the element: record the duplicate and drop the later
|
||||
// occurrence rather than failing the whole start tag.
|
||||
diagnostics.append(Diagnostic(.duplicateAttribute(name: name.description, mark)))
|
||||
continue
|
||||
}
|
||||
throw ParseError.duplicateAttribute(name: name.description, mark)
|
||||
}
|
||||
attributes.append(PureXML.Model.Attribute(name: name, value: value))
|
||||
}
|
||||
}
|
||||
|
||||
private mutating func scanAttributeValue() throws -> String {
|
||||
let mark = reader.mark
|
||||
guard let quote = reader.peek(), quote == "\"" || quote == "'" else {
|
||||
throw ParseError.unquotedAttributeValue(reader.mark)
|
||||
}
|
||||
reader.advance()
|
||||
var raw = ""
|
||||
var length = 0
|
||||
while let character = reader.peek(), character != quote {
|
||||
length += 1
|
||||
try checkContent(length, mark)
|
||||
raw.append(character)
|
||||
reader.advance()
|
||||
}
|
||||
guard reader.consume(String(quote)) else {
|
||||
throw ParseError.unexpectedEndOfInput(reader.mark)
|
||||
}
|
||||
return try EntityDecoder.decode(raw, entities: documentType.entities, budget: &entityBudget, at: mark)
|
||||
}
|
||||
|
||||
private mutating func scanName() throws -> PureXML.Model.QualifiedName {
|
||||
mutating func scanName() throws -> PureXML.Model.QualifiedName {
|
||||
let mark = reader.mark
|
||||
guard let first = reader.peek(), first.isXMLNameStart else {
|
||||
throw ParseError.expectedName(reader.mark)
|
||||
|
||||
59
Sources/Parsing/EventReaderAttributes.swift
Normal file
59
Sources/Parsing/EventReaderAttributes.swift
Normal file
@ -0,0 +1,59 @@
|
||||
/// The start-tag attribute scanners, in their own file to keep the reader's
|
||||
/// type body under the length cap. Behavior is part of EventReader proper.
|
||||
extension PureXML.Parsing.EventReader {
|
||||
mutating func scanAttributes() throws -> [PureXML.Model.Attribute] {
|
||||
var attributes: [PureXML.Model.Attribute] = []
|
||||
var seen: Set<String> = []
|
||||
while true {
|
||||
reader.skipSpace()
|
||||
guard let next = reader.peek(), next != ">", next != "/" else {
|
||||
return attributes
|
||||
}
|
||||
let mark = reader.mark
|
||||
let name = try scanName()
|
||||
reader.skipSpace()
|
||||
guard reader.consume("=") else {
|
||||
throw PureXML.Parsing.ParseError.expectedEquals(reader.mark)
|
||||
}
|
||||
reader.skipSpace()
|
||||
let value = try scanAttributeValue()
|
||||
guard seen.insert(name.description).inserted else {
|
||||
if recovering {
|
||||
// Keep the element: record the duplicate and drop the later
|
||||
// occurrence rather than failing the whole start tag.
|
||||
diagnostics.append(PureXML.Parsing.Diagnostic(PureXML.Parsing.ParseError.duplicateAttribute(name: name.description, mark)))
|
||||
continue
|
||||
}
|
||||
throw PureXML.Parsing.ParseError.duplicateAttribute(name: name.description, mark)
|
||||
}
|
||||
attributes.append(PureXML.Model.Attribute(name: name, value: value))
|
||||
// A following attribute needs whitespace before it.
|
||||
if let next = reader.peek(), next != ">", next != "/", next != "?", !next.isWhitespace {
|
||||
throw PureXML.Parsing.ParseError.missingSpaceBeforeAttribute(reader.mark)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
mutating func scanAttributeValue() throws -> String {
|
||||
let mark = reader.mark
|
||||
guard let quote = reader.peek(), quote == "\"" || quote == "'" else {
|
||||
throw PureXML.Parsing.ParseError.unquotedAttributeValue(reader.mark)
|
||||
}
|
||||
reader.advance()
|
||||
var raw = ""
|
||||
var length = 0
|
||||
while let character = reader.peek(), character != quote {
|
||||
length += 1
|
||||
try checkContent(length, mark)
|
||||
if character == "<" {
|
||||
throw PureXML.Parsing.ParseError.rawLessThanInAttribute(reader.mark)
|
||||
}
|
||||
raw.append(character)
|
||||
reader.advance()
|
||||
}
|
||||
guard reader.consume(String(quote)) else {
|
||||
throw PureXML.Parsing.ParseError.unexpectedEndOfInput(reader.mark)
|
||||
}
|
||||
return try PureXML.Parsing.EntityDecoder.decode(raw, entities: documentType.entities, budget: &entityBudget, at: mark)
|
||||
}
|
||||
}
|
||||
@ -55,6 +55,11 @@ extension PureXML.Parsing.EventReader {
|
||||
}
|
||||
length += 1
|
||||
try checkContent(length, mark)
|
||||
// XML 1.0: '--' may not appear inside a comment (this also rejects
|
||||
// the '--->' ending, since its first two hyphens hit this check).
|
||||
if character == "-", reader.peek() == "-" {
|
||||
throw PureXML.Parsing.ParseError.doubleHyphenInComment(mark)
|
||||
}
|
||||
content.append(character)
|
||||
}
|
||||
reader.consume("-->")
|
||||
|
||||
@ -34,6 +34,17 @@ public extension PureXML.Parsing {
|
||||
/// The XML declaration `<?xml ... ?>` is malformed: its pseudo-attributes
|
||||
/// are out of order, unknown, or carry an illegal value.
|
||||
case malformedDeclaration(Mark)
|
||||
/// A comment contains `--`, which XML 1.0 forbids inside comments.
|
||||
case doubleHyphenInComment(Mark)
|
||||
/// An attribute value contains a raw `<`, which must be escaped.
|
||||
case rawLessThanInAttribute(Mark)
|
||||
/// Character data contains the literal `]]>` sequence, which must be
|
||||
/// escaped outside a CDATA section.
|
||||
case cdataCloseInContent(Mark)
|
||||
/// Two attributes are not separated by whitespace.
|
||||
case missingSpaceBeforeAttribute(Mark)
|
||||
/// A character outside the XML `Char` production appears in content.
|
||||
case invalidCharacter(Mark)
|
||||
|
||||
public var description: String {
|
||||
switch self {
|
||||
@ -85,6 +96,16 @@ public extension PureXML.Parsing {
|
||||
"input bytes are malformed for the detected encoding"
|
||||
case let .notImplemented(detail):
|
||||
"not implemented: \(detail)"
|
||||
case let .doubleHyphenInComment(mark):
|
||||
"'--' is not allowed inside a comment at \(mark)"
|
||||
case let .rawLessThanInAttribute(mark):
|
||||
"raw '<' is not allowed in an attribute value at \(mark)"
|
||||
case let .cdataCloseInContent(mark):
|
||||
"']]>' must be escaped in character data at \(mark)"
|
||||
case let .missingSpaceBeforeAttribute(mark):
|
||||
"attributes must be separated by whitespace at \(mark)"
|
||||
case let .invalidCharacter(mark):
|
||||
"character is not allowed in XML content at \(mark)"
|
||||
case let .malformedDeclaration(mark):
|
||||
"malformed XML declaration at \(mark)"
|
||||
}
|
||||
|
||||
109
Tests/W3CXMLSuiteTests.swift
Normal file
109
Tests/W3CXMLSuiteTests.swift
Normal file
@ -0,0 +1,109 @@
|
||||
import Foundation
|
||||
@testable import PureXML
|
||||
import Testing
|
||||
|
||||
/// Opt-in runner for the W3C XML Conformance Test Suite's xmltest section
|
||||
/// (James Clark's cases). The suite's license permits redistribution only as
|
||||
/// the unmodified archive, so it is never vendored into this repository:
|
||||
/// download `xmlts` from w3.org, extract it, and point `XMLTS_ROOT` at the
|
||||
/// `xmlconf/xmltest` directory to run. Without the variable the suite is
|
||||
/// skipped. Foundation is used here for file access only; this is a test-only
|
||||
/// dependency and the library target remains Foundation-free.
|
||||
@Suite("W3C xmltest conformance (opt-in via XMLTS_ROOT)")
|
||||
struct W3CXMLSuiteTests {
|
||||
private var root: String? {
|
||||
ProcessInfo.processInfo.environment["XMLTS_ROOT"]
|
||||
}
|
||||
|
||||
/// Cases this implementation knowingly accepts although the suite marks
|
||||
/// them not-well-formed. Two classes, tracked for later hardening: the
|
||||
/// internal-subset scanner is deliberately lenient about DTD grammar
|
||||
/// details (declaration syntax, conditional sections, parameter-entity
|
||||
/// placement), and entity replacement text is not yet checked for
|
||||
/// standalone well-formedness (tags spanning entity boundaries, partial
|
||||
/// markup in values). The baseline below is exact, so any regression in
|
||||
/// the cases that DO pass is still caught.
|
||||
private let knownNotWFDeviations: Set<String> = [
|
||||
"031.xml", "032.xml", "054.xml", "057.xml", "058.xml", "059.xml", "060.xml", "061.xml",
|
||||
"062.xml", "063.xml", "064.xml", "065.xml", "066.xml", "067.xml", "068.xml", "069.xml",
|
||||
"074.xml", "078.xml", "079.xml", "080.xml", "082.xml", "084.xml", "085.xml", "086.xml",
|
||||
"087.xml", "089.xml", "090.xml", "091.xml", "092.xml", "093.xml", "096.xml", "101.xml",
|
||||
"102.xml", "103.xml", "107.xml", "113.xml", "114.xml", "115.xml", "116.xml", "117.xml",
|
||||
"119.xml", "120.xml", "121.xml", "122.xml", "123.xml", "124.xml", "125.xml", "126.xml",
|
||||
"127.xml", "128.xml", "129.xml", "130.xml", "131.xml", "132.xml", "133.xml", "134.xml",
|
||||
"135.xml", "136.xml", "137.xml", "138.xml", "139.xml", "140.xml", "141.xml", "142.xml",
|
||||
"143.xml", "144.xml", "147.xml", "148.xml", "149.xml", "150.xml", "151.xml", "153.xml",
|
||||
"154.xml", "155.xml", "156.xml", "157.xml", "158.xml", "160.xml", "161.xml", "162.xml",
|
||||
"165.xml", "168.xml", "169.xml", "170.xml", "171.xml", "172.xml", "173.xml", "174.xml",
|
||||
"175.xml", "180.xml", "182.xml", "183.xml", "184.xml",
|
||||
]
|
||||
|
||||
/// valid/sa/114.xml: a CDATA section inside an entity value must protect
|
||||
/// the &foo; reference from expansion; the entity decoder expands it
|
||||
/// before the CDATA section is recognized. Same hardening track.
|
||||
private let knownValidDeviations: Set<String> = ["114.xml"]
|
||||
|
||||
private func files(in directory: String) throws -> [String] {
|
||||
try FileManager.default.contentsOfDirectory(atPath: directory)
|
||||
.filter { $0.hasSuffix(".xml") }
|
||||
.sorted()
|
||||
}
|
||||
|
||||
private func limits() -> PureXML.Parsing.Limits {
|
||||
PureXML.Parsing.Limits(allowDoctype: true)
|
||||
}
|
||||
|
||||
/// Resolves relative SYSTEM identifiers against the case's directory, so the
|
||||
/// suite's external-entity cases load their sibling `.ent` files.
|
||||
private func resolver(directory: String) -> PureXML.Parsing.EntityResolver {
|
||||
PureXML.Parsing.EntityResolver(
|
||||
resolveEntity: { _, id in
|
||||
(try? String(contentsOfFile: directory + "/" + id.systemID, encoding: .utf8))
|
||||
},
|
||||
resolveExternalSubset: { id in
|
||||
(try? String(contentsOfFile: directory + "/" + id.systemID, encoding: .utf8))
|
||||
},
|
||||
)
|
||||
}
|
||||
|
||||
/// Reads a case's raw bytes and decodes through PureXML's own byte decoder,
|
||||
/// since the suite deliberately includes UTF-16 documents and broken
|
||||
/// encodings (a decode failure counts as rejection for not-wf cases).
|
||||
private func parse(file: String, in directory: String) throws -> PureXML.Model.Node {
|
||||
let data = try Data(contentsOf: URL(fileURLWithPath: directory + "/" + file))
|
||||
let source = try PureXML.Parsing.ByteDecoder.decode([UInt8](data))
|
||||
return try PureXML.parse(source, limits: limits(), resolver: resolver(directory: directory))
|
||||
}
|
||||
|
||||
@Test("Every not-wf/sa case is rejected by the strict parser")
|
||||
func test_notWellFormedRejected() throws {
|
||||
guard let root else {
|
||||
return // Opt-in: suite not present.
|
||||
}
|
||||
let directory = root + "/not-wf/sa"
|
||||
var unexpectedlyAccepted: [String] = []
|
||||
for file in try files(in: directory) where !knownNotWFDeviations.contains(file) {
|
||||
if (try? parse(file: file, in: directory)) != nil {
|
||||
unexpectedlyAccepted.append(file)
|
||||
}
|
||||
}
|
||||
#expect(unexpectedlyAccepted.isEmpty, "accepted \(unexpectedlyAccepted.count) not-wf cases: \(unexpectedlyAccepted)")
|
||||
}
|
||||
|
||||
@Test("Every valid/sa case parses")
|
||||
func test_validParses() throws {
|
||||
guard let root else {
|
||||
return // Opt-in: suite not present.
|
||||
}
|
||||
let directory = root + "/valid/sa"
|
||||
var failures: [String: String] = [:]
|
||||
for file in try files(in: directory) where !knownValidDeviations.contains(file) {
|
||||
do {
|
||||
_ = try parse(file: file, in: directory)
|
||||
} catch {
|
||||
failures[file] = String(describing: error)
|
||||
}
|
||||
}
|
||||
#expect(failures.isEmpty, "\(failures.count) valid cases failed: \(failures)")
|
||||
}
|
||||
}
|
||||
Loading…
Reference in New Issue
Block a user