diff --git a/CHANGELOG.md b/CHANGELOG.md index ebea33e..04efbc8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- W3C xmlconf IBM section, clean to two documented classes (#121, #125): the deepest block, 1094 cases. Valid 220/220 (parse + strict DTD validation), invalid 64/64 reporting, not-wf 529/810 rejected with the 281 accepted cases being exactly two documented classes: 279 cases from productions 85-89, the 1998 character-class appendices deleted by XML 1.0 Fifth Edition (whose count is asserted exactly so drift in either direction is caught), and 2 parameter-entity-in-internal-subset feature cases (the xmltest 160-162 class, headed for #128's strict profile). Closing the section's real gaps: a document without a root element is rejected (production 1); an entity reference in an entity literal must be a lexical Name or character reference (production 68); WFC: No Recursion is enforced at declaration over the combined general/parameter reference graph (CDATA-shielded), so a recursive pair is rejected without being referenced; an undeclared parameter-entity reference is a well-formedness error when the document is standalone or has neither an external subset nor parameter entities, and a deferred validity finding otherwise (the production-68 WFC/VC split, also applied to entity references in attribute defaults); a declared external PE the resolver refused is neither; and a content-model group opening in one PE replacement and closing in another is reported (VC: Proper Group/PE Nesting). - Standalone validity constraints (#134): a document declaring `standalone='yes'` is now checked against its external-subset dependencies (2.9). The DTD scanner records declaration provenance (which entities, element models, and attribute lists came from the internal subset); the validator reports an externally-declared attribute default that would be supplied, an attribute value an externally-declared non-CDATA type would normalize, and whitespace inside externally-declared element content. The parser enforces the matching well-formedness side: a standalone document referencing an entity declared outside the internal subset reports it as undeclared (WFC: Entity Declared). This empties the entire Sun baseline: the section is fully clean at not-wf 56/56, valid 28/28, invalid 74/74, zero deviations, closing #124. - DTD validity constraints, second Sun burn-down (#121, #124): the validator now enforces VC Root Element Type (the root element must match the DOCTYPE name, recorded by the scanner), VC Unique Element Type Declaration (duplicate `` reported, first stays in effect), VC No Duplicate Types in mixed content, VC One ID per Element Type, VC Notation Attributes (listed names must be declared) and VC Notation Declared (an NDATA entity's notation must exist), VC Attribute Default Legal across the tokenized types plus VC ID Attribute Default (an ID attribute must be #IMPLIED or #REQUIRED), lexical Name checks for ID/IDREF/IDREFS values, and CDATA sections as character data in element content even when empty. Strict mode additionally requires every attribute to be declared (VC Attribute Value Type; namespace declarations exempt). Declaration-level findings are computed once when the schema is built and reported at the document root. Sun invalid coverage moves from 43/74 to 62/74 reporting; the twelve remaining silent cases are one family, the standalone='yes' VCs, which need declaration-provenance tracking. - Entity content splicing (#133): a general entity whose replacement text contains markup, directly or through entities it references, is now reparsed as content at the point of reference (4.4.2 Included), so `

text

` inside an entity becomes a `p` element in the tree and in the pull-event stream, in both strict and recovering modes, instead of being included as escaped character data. The replacement is validated (the reference-time balanced-content WFC), checked for reference cycles, and budgeted against the amplification cap before being spliced into the reader's stream. Character references in internal entity literals now expand when the declaration is parsed for general entities too (4.4.5, previously parameter entities only), so `<foo>` stores real markup that splices, and the Appendix D double escape `&#38;` stores `&` and renders as a literal ampersand; the reference-time WFC accordingly validates remaining references instead of re-expanding them. Text-only entities keep the single-pass expansion path and its event coalescing. Closes the last two Sun valid deviations: the section is now 28/28 valid. diff --git a/README.md b/README.md index a943ab9..7f53e6a 100644 --- a/README.md +++ b/README.md @@ -90,7 +90,7 @@ flowchart TB Epic121["#121 xmlconf full archive"]:::epic Oasis["#123 OASIS/NIST section: complete, zero deviations"]:::done Sun["#124 Sun section: complete, zero deviations"]:::done - IBM["#125 IBM section: per-production validity depth"]:::todo + IBM["#125 IBM section: clean to two documented classes"]:::done Japanese["#126 japanese section: encoding declarations"]:::todo Eduni["#127 eduni: errata + namespaces"]:::todo StrictSubset["#128 strict internal-subset profile: xmltest 185/186"]:::todo @@ -117,7 +117,7 @@ flowchart TB PureXML is a working, dependency-free XML library today: parse, emit, validate, query, and stream documents on macOS, Linux, Windows, and WASM. The test suite -currently runs **890 tests in 148 suites** (`swift test`). +currently runs **898 tests in 150 suites** (`swift test`). ### Shipped (libxml2-aligned surface) diff --git a/Sources/Parsing/DoctypeScanner.swift b/Sources/Parsing/DoctypeScanner.swift index c695a9a..225fdbb 100644 --- a/Sources/Parsing/DoctypeScanner.swift +++ b/Sources/Parsing/DoctypeScanner.swift @@ -14,12 +14,13 @@ extension PureXML.Parsing { _ reader: inout Reader, limits: Limits, resolver: EntityResolver = .refusing, + standalone: Bool = false, ) throws -> DocumentType { let mark = reader.mark guard limits.allowDoctype else { throw ParseError.unsupportedDoctype(mark) } - var scanner = DTDScanner(limits: limits, resolver: resolver) + var scanner = DTDScanner(limits: limits, resolver: resolver, standalone: standalone) return try scanner.scan(&reader, at: mark) } } @@ -37,21 +38,39 @@ struct DTDScanner { let limits: PureXML.Parsing.Limits let resolver: PureXML.Parsing.EntityResolver + /// Whether the document declared standalone='yes', which makes an + /// undeclared-entity reference a well-formedness error rather than a + /// validity finding (WFC vs VC: Entity Declared). + let standalone: Bool var doctype = PureXML.Parsing.DocumentType() /// True while scanning the external subset, so declarations record their /// origin (the standalone validity constraints depend on it). var inExternalContext = false + /// External parameter entities that were declared but not loaded (the + /// resolver refused them): a reference to one is not an undeclared-entity + /// error, the processor simply did not read it. + var unresolvedParameterEntities: Set = [] var parameterBudget: Int /// Bounds parameter-entity injection recursion (modularized DTDs nest, but /// only a little); deeper references are ignored rather than trapping. let maxDepth = 40 - init(limits: PureXML.Parsing.Limits, resolver: PureXML.Parsing.EntityResolver) { + init(limits: PureXML.Parsing.Limits, resolver: PureXML.Parsing.EntityResolver, standalone: Bool = false) { self.limits = limits self.resolver = resolver + self.standalone = standalone parameterBudget = limits.maxEntityExpansion } + /// Entity Declared is a WFC (throw) when the document is standalone, or + /// has neither an external subset nor parameter entities; otherwise it is + /// a VC (deferred finding), since the unread external declarations might + /// have declared the entity (production 68). + var entityDeclaredIsWellFormedness: Bool { + if standalone { return true } + return doctype.externalSubset == nil && doctype.parameterEntities.isEmpty && unresolvedParameterEntities.isEmpty + } + mutating func scan(_ reader: inout Reader, at mark: Mark) throws -> PureXML.Parsing.DocumentType { reader.consume("= replacement.count else { - return - } - parameterBudget -= replacement.count - var sub = Reader(replacement) - try scanDeclarations(&sub, depth: depth + 1, terminatedByBracket: false, at: mark) - } - - /// Files an external entity declaration by kind: an `NDATA` entity is unparsed - /// (recorded with its notation); an external general entity records its - /// identifier for the resolver; an external parameter entity is loaded through - /// the resolver so its replacement text is available for `%name;` expansion - /// (the default refusing resolver loads nothing, keeping XXE closed). - mutating func storeExternalEntity(name: String, id: ExternalID, isParameter: Bool, notation: String?) { - if let notation, !notation.isEmpty, !isParameter { - if doctype.unparsedEntities[name] == nil { - doctype.unparsedEntities[name] = PureXML.Parsing.UnparsedEntity(id: id, notation: notation) - } - } else if isParameter { - if doctype.parameterEntities[name] == nil, let text = resolver.resolveExternalSubset(id) { - doctype.parameterEntities[name] = expandParameterReferences(strippingTextDeclaration(text)) - } - } else if doctype.externalEntities[name] == nil { - doctype.externalEntities[name] = id - } - } - - /// `'` (production 82): - /// the identifier is required and nothing may follow it but whitespace. private mutating func scanNotationDeclaration(_ reader: inout Reader) throws { let mark = reader.mark reader.consume(" String { - guard raw.contains("%") else { return raw } - var result = "" - var index = raw.startIndex - while index < raw.endIndex { - let character = raw[index] - guard character == "%", let semicolon = raw[index...].firstIndex(of: ";") else { - result.append(character) - index = raw.index(after: index) - continue - } - let name = String(raw[raw.index(after: index) ..< semicolon]) - result += doctype.parameterEntities[name] ?? "%\(name);" - index = raw.index(after: semicolon) - } - return result - } - private func parseExternalID(_ reader: inout Reader) -> ExternalID? { if reader.consume("SYSTEM") { reader.skipSpace() diff --git a/Sources/Parsing/DoctypeScannerConditional.swift b/Sources/Parsing/DoctypeScannerConditional.swift index c84ce15..677ea99 100644 --- a/Sources/Parsing/DoctypeScannerConditional.swift +++ b/Sources/Parsing/DoctypeScannerConditional.swift @@ -54,4 +54,89 @@ extension DTDScanner { } return nil } + + /// VC: Proper Group/PE Nesting. A parameter entity used in a content model + /// must contain balanced parentheses, so a group cannot open in one + /// replacement text and close in another. Recorded as a validity finding. + mutating func checkGroupNesting(_ rawModel: String, element: String) { + guard rawModel.contains("%") else { return } + var index = rawModel.startIndex + while index < rawModel.endIndex, let percent = rawModel[index...].firstIndex(of: "%") { + guard let semicolon = rawModel[percent...].firstIndex(of: ";") else { return } + let name = String(rawModel[rawModel.index(after: percent) ..< semicolon]) + if let replacement = doctype.parameterEntities[name] { + let opens = replacement.count(where: { $0 == "(" }) + let closes = replacement.count(where: { $0 == ")" }) + if opens != closes { + doctype.validityFindings.append( + "the content model of '\(element)' uses parameter entity '%\(name);' with improper group/PE nesting", + ) + } + } + index = rawModel.index(after: semicolon) + } + } + + /// WFC: No Recursion, checked at declaration: no parsed entity (general or + /// parameter) may reference itself directly or indirectly through the + /// stored replacement texts. + func checkEntityRecursion(at mark: Mark) throws { + var visiting: Set = [] + var cleared: Set = [] + for key in doctype.entities.keys.sorted() { + try walkEntityGraph("ge:" + key, visiting: &visiting, cleared: &cleared, at: mark) + } + for key in doctype.parameterEntities.keys.sorted() { + try walkEntityGraph("pe:" + key, visiting: &visiting, cleared: &cleared, at: mark) + } + } + + private func walkEntityGraph(_ node: String, visiting: inout Set, cleared: inout Set, at mark: Mark) throws { + guard !cleared.contains(node) else { return } + guard visiting.insert(node).inserted else { + throw ParseError.recursiveEntity(name: String(node.dropFirst(3)), mark) + } + defer { + visiting.remove(node) + cleared.insert(node) + } + let name = String(node.dropFirst(3)) + let value = node.hasPrefix("ge:") ? doctype.entities[name] : doctype.parameterEntities[name] + guard let value else { return } + for reference in entityReferences(in: value) { + try walkEntityGraph(reference, visiting: &visiting, cleared: &cleared, at: mark) + } + } + + /// The `ge:`/`pe:`-tagged references in a replacement text, with CDATA + /// sections shielded and predefined entities skipped. + private func entityReferences(in text: String) -> [String] { + let predefined: Set = ["amp", "lt", "gt", "quot", "apos"] + var references: [String] = [] + var index = text.startIndex + while index < text.endIndex { + if text[index...].hasPrefix("") { + cursor = text.index(after: cursor) + } + index = cursor < text.endIndex ? text.index(cursor, offsetBy: 3) : text.endIndex + continue + } + let character = text[index] + guard character == "&" || character == "%" else { + index = text.index(after: index) + continue + } + guard let semicolon = text[index...].firstIndex(of: ";") else { return references } + let body = String(text[text.index(after: index) ..< semicolon]) + if character == "&", !body.hasPrefix("#"), !predefined.contains(body) { + references.append("ge:" + body) + } else if character == "%", !body.isEmpty { + references.append("pe:" + body) + } + index = text.index(after: semicolon) + } + return references + } } diff --git a/Sources/Parsing/DoctypeScannerEntities.swift b/Sources/Parsing/DoctypeScannerEntities.swift new file mode 100644 index 0000000..dbff933 --- /dev/null +++ b/Sources/Parsing/DoctypeScannerEntities.swift @@ -0,0 +1,101 @@ +/// The entity side of the DTD scan: storing declarations, resolving external +/// entities through the injected resolver, expanding parameter-entity +/// references, and the bare `%name;` declaration-stream injection. Split from +/// the scanner body to keep it under the length caps. +extension DTDScanner { + /// Folds declared external general entities into the general-entity table by + /// asking the resolver for their replacement text. A refused (nil) entity + /// stays undeclared, so a reference to it errors and the default refusing + /// resolver keeps XXE closed. + mutating func resolveExternalEntities() { + for (name, id) in doctype.externalEntities where doctype.entities[name] == nil { + if let text = resolver.resolveEntity(name, id) { + // An external parsed entity may begin with a text declaration, + // which is not part of its replacement text (4.3.1). + doctype.entities[name] = strippingTextDeclaration(text) + } + } + } + + /// Handles a bare `%name;` between declarations by injecting the parameter + /// entity's replacement text and scanning its declarations. Bounded by depth + /// and the expansion budget. + mutating func scanParameterReference(_ reader: inout Reader, depth: Int, at mark: Mark) throws { + let refMark = reader.mark + reader.consume("%") + let name = scanName(&reader) + // PEReference is '%' Name ';' exactly: no space after '%' (an empty + // name) and the semicolon must follow the name directly. + guard !name.isEmpty, reader.consume(";") else { + throw ParseError.invalidReference("%\(name)", refMark) + } + guard let replacement = doctype.parameterEntities[name] else { + // A declared external PE the resolver refused is not undeclared, + // the processor just did not read it. + guard !unresolvedParameterEntities.contains(name) else { return } + // VC/WFC: Entity Declared (also covers a reference before the + // declaration: declarations bind in document order). + if entityDeclaredIsWellFormedness { + throw ParseError.undefinedEntity(name: name, refMark) + } + doctype.validityFindings.append("parameter entity '%\(name);' is referenced but not declared") + return + } + guard depth < maxDepth else { + return + } + guard parameterBudget >= replacement.count else { + return + } + parameterBudget -= replacement.count + var sub = Reader(replacement) + try scanDeclarations(&sub, depth: depth + 1, terminatedByBracket: false, at: mark) + } + + /// Files an external entity declaration by kind: an `NDATA` entity is unparsed + /// (recorded with its notation); an external general entity records its + /// identifier for the resolver; an external parameter entity is loaded through + /// the resolver so its replacement text is available for `%name;` expansion + /// (the default refusing resolver loads nothing, keeping XXE closed). + mutating func storeExternalEntity(name: String, id: ExternalID, isParameter: Bool, notation: String?) { + if let notation, !notation.isEmpty, !isParameter { + if doctype.unparsedEntities[name] == nil { + doctype.unparsedEntities[name] = PureXML.Parsing.UnparsedEntity(id: id, notation: notation) + } + } else if isParameter { + if doctype.parameterEntities[name] == nil { + if let text = resolver.resolveExternalSubset(id) { + doctype.parameterEntities[name] = expandParameterReferences(strippingTextDeclaration(text)) + } else { + unresolvedParameterEntities.insert(name) + } + } + } else if doctype.externalEntities[name] == nil { + doctype.externalEntities[name] = id + } + } + + /// `'` (production 82): + /// the identifier is required and nothing may follow it but whitespace. + /// Replaces `%name;` references with their (already-expanded) parameter-entity + /// values. Undefined references are left literal. A single forward pass is + /// enough because each value was expanded against the entities defined before + /// it, so no reference can reach a later definition. + func expandParameterReferences(_ raw: String) -> String { + guard raw.contains("%") else { return raw } + var result = "" + var index = raw.startIndex + while index < raw.endIndex { + let character = raw[index] + guard character == "%", let semicolon = raw[index...].firstIndex(of: ";") else { + result.append(character) + index = raw.index(after: index) + continue + } + let name = String(raw[raw.index(after: index) ..< semicolon]) + result += doctype.parameterEntities[name] ?? "%\(name);" + index = raw.index(after: semicolon) + } + return result + } +} diff --git a/Sources/Parsing/DoctypeScannerStrict.swift b/Sources/Parsing/DoctypeScannerStrict.swift index ea0a8d4..2b417c7 100644 --- a/Sources/Parsing/DoctypeScannerStrict.swift +++ b/Sources/Parsing/DoctypeScannerStrict.swift @@ -93,7 +93,8 @@ extension DTDScanner { } if character == "&" || character == "%" { var probe = index + 1 - if character == "&", probe < characters.count, characters[probe] == "#" { probe += 1 } + let isCharacterReference = character == "&" && probe < characters.count && characters[probe] == "#" + if isCharacterReference { probe += 1 } let bodyStart = probe let stops: Set = [";", "&", "%"] while probe < characters.count, !stops.contains(characters[probe]), !characters[probe].isWhitespace { @@ -102,6 +103,14 @@ extension DTDScanner { guard probe > bodyStart, probe < characters.count, characters[probe] == ";" else { throw ParseError.invalidReference(String(character), mark) } + // An entity reference's body must be a Name: '&49;' is neither + // a character reference nor a legal entity name (production 68). + if !isCharacterReference { + let body = String(characters[bodyStart ..< probe]) + guard PureXML.Parsing.XMLCharacter.isValidName(body) else { + throw ParseError.invalidReference(String(character) + body, mark) + } + } index = probe } index += 1 @@ -234,11 +243,11 @@ extension DTDScanner { /// already declared and internal: an undeclared, forward-declared, external, /// or unparsed entity in a default is not well-formed, and the referenced /// chain must not recurse. - func validateDefaultValueReferences(_ value: String, at mark: Mark) throws { + mutating func validateDefaultValueReferences(_ value: String, at mark: Mark) throws { try walkDefaultReferences(value, visiting: [], at: mark) } - func walkDefaultReferences(_ value: String, visiting: Set, at mark: Mark) throws { + mutating func walkDefaultReferences(_ value: String, visiting: Set, at mark: Mark) throws { let predefined: Set = ["amp", "lt", "gt", "quot", "apos"] let characters = Array(value) var index = 0 @@ -269,7 +278,12 @@ extension DTDScanner { throw ParseError.invalidAttributeListDeclaration(mark) } guard let replacement = doctype.entities[name] else { - throw ParseError.undefinedEntity(name: name, mark) + // VC/WFC: Entity Declared, same split as parameter references. + if entityDeclaredIsWellFormedness { + throw ParseError.undefinedEntity(name: name, mark) + } + doctype.validityFindings.append("entity '&\(name);' in an attribute default is referenced but not declared") + return } try walkDefaultReferences(replacement, visiting: visiting.union([name]), at: mark) } diff --git a/Sources/Parsing/DocumentType.swift b/Sources/Parsing/DocumentType.swift index 38c6ad1..cb85780 100644 --- a/Sources/Parsing/DocumentType.swift +++ b/Sources/Parsing/DocumentType.swift @@ -38,6 +38,12 @@ public extension PureXML.Parsing { /// The `` bodies declared in the internal subset only, keyed /// by element, so externally-declared attributes are distinguishable. public var internalAttributeLists: [String: String] = [:] + /// Validity (not well-formedness) findings discovered while scanning + /// the DTD: an undeclared entity referenced where the external subset + /// might have declared it (VC: Entity Declared) or a content-model + /// group split across parameter entities (VC: Proper Group/PE + /// Nesting). Reported by the validator at the document root. + public var validityFindings: [String] = [] public init( entities: [String: String] = [:], diff --git a/Sources/Parsing/EventReader.swift b/Sources/Parsing/EventReader.swift index 645df85..ca11043 100644 --- a/Sources/Parsing/EventReader.swift +++ b/Sources/Parsing/EventReader.swift @@ -125,7 +125,7 @@ public extension PureXML.Parsing { return .processingInstruction(target: instruction.target, data: instruction.data) } if reader.matches(" PureXML.Parsing.Limits { + PureXML.Parsing.Limits(allowDoctype: true) + } + + @Test("A document without a root element is rejected") + func test_missingRoot() { + #expect(throws: PureXML.Parsing.ParseError.self) { + _ = try PureXML.parse("\n]>\n", limits: limits()) + } + #expect(throws: PureXML.Parsing.ParseError.self) { + _ = try PureXML.parse("") + } + } + + @Test("An entity reference in a literal must be a Name or character reference") + func test_referenceNameInLiteral() { + let xml = "]>\n" + #expect(throws: PureXML.Parsing.ParseError.self) { + _ = try PureXML.parse(xml, limits: limits()) + } + let legal = "]>\n" + #expect((try? PureXML.parse(legal, limits: limits())) != nil) + } + + @Test("Recursive entity declarations are rejected without being referenced") + func test_declarationRecursion() { + let direct = """ + + + + ]> + + """ + #expect(throws: PureXML.Parsing.ParseError.self) { + _ = try PureXML.parse(direct, limits: limits()) + } + let indirect = """ + + + + + ]> + + """ + #expect(throws: PureXML.Parsing.ParseError.self) { + _ = try PureXML.parse(indirect, limits: limits()) + } + } + + @Test("An undeclared PE reference is a WFC when standalone, a VC otherwise") + func test_undeclaredParameterEntity() throws { + let standalone = """ + + + %paaa; + "> + ]> + + """ + #expect(throws: PureXML.Parsing.ParseError.self) { + _ = try PureXML.parse(standalone, limits: limits()) + } + // With an external PE in play the same reference is a validity finding. + let loose = """ + + + %outside; + %undeclared; + ]> + + """ + let resolver = PureXML.Parsing.EntityResolver( + resolveEntity: { _, _ in nil }, + resolveExternalSubset: { _ in "" }, + ) + let errors = try PureXML.validateAgainstInternalDTD(loose, resolver: resolver) + #expect(errors.contains { $0.reason.contains("'%undeclared;' is referenced but not declared") }) + } + + @Test("A group split across parameter entities is a validity finding") + func test_properGroupNesting() throws { + let dtd = """ + + + + + """ + let xml = "]>\n" + let resolver = PureXML.Parsing.EntityResolver( + resolveEntity: { _, _ in nil }, + resolveExternalSubset: { _ in dtd }, + ) + let errors = try PureXML.validateAgainstInternalDTD(xml, resolver: resolver) + #expect(errors.contains { $0.reason.contains("improper group/PE nesting") }) + } +} diff --git a/Tests/W3CIBMSuiteTests.swift b/Tests/W3CIBMSuiteTests.swift new file mode 100644 index 0000000..b7ae533 --- /dev/null +++ b/Tests/W3CIBMSuiteTests.swift @@ -0,0 +1,175 @@ +import Foundation +@testable import PureXML +import Testing + +/// Opt-in runner for the W3C XML Conformance Test Suite's IBM section, the +/// deepest block: per-production directories covering valid, invalid, and +/// not-wf cases with heavy DTD usage. Classification follows the Sun model: +/// `valid` must parse and DTD-validate cleanly (strict mode), `invalid` must +/// parse but report at least one validity error, `not-wf` must be rejected. +/// Point `XMLCONF_ROOT` at the extracted `xmlconf` directory to run (the +/// suite is never vendored; its license permits redistribution only as the +/// unmodified archive). The `xml-1.1` subdirectory is out of scope: PureXML +/// targets XML 1.0 Fifth Edition. Foundation is used for file access only. +@Suite("W3C ibm conformance (opt-in via XMLCONF_ROOT)") +struct W3CIBMSuiteTests { + private var root: String? { + ProcessInfo.processInfo.environment["XMLCONF_ROOT"] + } + + /// not-wf cases this implementation knowingly accepts: both are + /// parameter-entity references inside internal-subset markup declarations, + /// which this package supports as a feature (the xmltest 160-162 class; + /// the strict profile of #128 will reject them). + private let knownNotWFAccepted: Set = [ + "not-wf/P29/ibm29n03.xml", "not-wf/P29/ibm29n04.xml", + ] + + /// Productions 85-89 are the 1998 character-class appendices (BaseChar, + /// Ideographic, CombiningChar, Digit, Extender), deleted by XML 1.0 Fifth + /// Edition, whose Name productions this package implements; the characters + /// these cases reject became legal (the xmltest 141 class). The count is + /// asserted exactly so a regression in either direction is caught. + private let preFifthEditionCaseCount = 279 + + private func isPreFifthEditionCharacterClass(_ uri: String) -> Bool { + (85 ... 89).contains { uri.hasPrefix("not-wf/P\($0)/") } + } + + /// valid cases this implementation knowingly fails (parse or validation). + private let knownValidFailures: Set = [] + + /// invalid cases where this implementation knowingly reports no validity + /// error. + private let knownInvalidAccepted: Set = [] + + private struct ManifestCase { + let id: String + let uri: String + } + + /// Reads one manifest (a fragment with multiple TESTCASES roots), wrapped + /// before parsing with PureXML itself. + private func manifest(_ file: String, in directory: String) throws -> [ManifestCase] { + let source = try String(contentsOfFile: directory + "/" + file, encoding: .utf8) + let document = try PureXML.parse("\(source)") + return try PureXML.xpath("//TEST", over: document).compactMap { selection in + guard let element = selection.element else { return nil } + func attribute(_ name: String) -> String? { + element.attributes.first { $0.name.description == name }?.value + } + guard let id = attribute("ID"), let uri = attribute("URI") else { return nil } + return ManifestCase(id: id, uri: uri) + } + } + + private func limits() -> PureXML.Parsing.Limits { + PureXML.Parsing.Limits(allowDoctype: true) + } + + /// Resolves relative SYSTEM identifiers against the case's own directory, + /// byte-decoding through PureXML's own decoder. + private func resolver(directory: String) -> PureXML.Parsing.EntityResolver { + @Sendable func load(_ id: PureXML.Parsing.ExternalID) -> String? { + guard let data = try? Data(contentsOf: URL(fileURLWithPath: directory + "/" + id.systemID)) else { + return nil + } + return try? PureXML.Parsing.ByteDecoder.decode([UInt8](data)) + } + return PureXML.Parsing.EntityResolver( + resolveEntity: { _, id in load(id) }, + resolveExternalSubset: { id in load(id) }, + ) + } + + private func caseDirectory(_ section: String, uri: String) -> String { + guard let slash = uri.lastIndex(of: "/") else { return section } + return section + "/" + String(uri.prefix(upTo: slash)) + } + + private func source(of uri: String, in section: String) throws -> String { + let data = try Data(contentsOf: URL(fileURLWithPath: section + "/" + uri)) + return try PureXML.Parsing.ByteDecoder.decode([UInt8](data)) + } + + @Test("Every ibm not-wf case is rejected by the strict parser") + func test_notWellFormedRejected() throws { + guard let root else { + return // Opt-in: suite not present. + } + let section = root + "/ibm" + var unexpectedlyAccepted: [String] = [] + var fifthEditionAccepted = 0 + for testCase in try manifest("ibm_oasis_not-wf.xml", in: section) where !knownNotWFAccepted.contains(testCase.uri) { + let directory = caseDirectory(section, uri: testCase.uri) + let accepted = (try? PureXML.parse( + source(of: testCase.uri, in: section), + limits: limits(), + resolver: resolver(directory: directory), + )) != nil + if accepted { + if isPreFifthEditionCharacterClass(testCase.uri) { + fifthEditionAccepted += 1 + } else { + unexpectedlyAccepted.append(testCase.uri) + } + } + } + #expect(unexpectedlyAccepted.isEmpty, "accepted \(unexpectedlyAccepted.count) not-wf cases: \(unexpectedlyAccepted)") + #expect(fifthEditionAccepted == preFifthEditionCaseCount, "pre-5e class drifted: \(fifthEditionAccepted)") + } + + @Test("Every ibm valid case parses and validates cleanly against its DTD") + func test_validCleans() throws { + guard let root else { + return // Opt-in: suite not present. + } + let section = root + "/ibm" + var failures: [String: String] = [:] + for testCase in try manifest("ibm_oasis_valid.xml", in: section) where !knownValidFailures.contains(testCase.uri) { + let directory = caseDirectory(section, uri: testCase.uri) + do { + let errors = try PureXML.validateAgainstInternalDTD( + source(of: testCase.uri, in: section), + limits: limits(), + strict: true, + resolver: resolver(directory: directory), + ) + if let first = errors.first { + failures[testCase.uri] = String(describing: first) + } + } catch { + failures[testCase.uri] = String(describing: error) + } + } + #expect(failures.isEmpty, "\(failures.count) valid cases failed: \(failures)") + } + + @Test("Every ibm invalid case parses but reports a validity error") + func test_invalidReports() throws { + guard let root else { + return // Opt-in: suite not present. + } + let section = root + "/ibm" + var silentlyClean: [String] = [] + var parseFailures: [String: String] = [:] + for testCase in try manifest("ibm_oasis_invalid.xml", in: section) where !knownInvalidAccepted.contains(testCase.uri) { + let directory = caseDirectory(section, uri: testCase.uri) + do { + let errors = try PureXML.validateAgainstInternalDTD( + source(of: testCase.uri, in: section), + limits: limits(), + strict: true, + resolver: resolver(directory: directory), + ) + if errors.isEmpty { + silentlyClean.append(testCase.uri) + } + } catch { + parseFailures[testCase.uri] = String(describing: error) + } + } + #expect(parseFailures.isEmpty, "\(parseFailures.count) invalid (well-formed) cases failed to parse: \(parseFailures)") + #expect(silentlyClean.isEmpty, "\(silentlyClean.count) invalid cases validated clean: \(silentlyClean)") + } +}