diff --git a/CHANGELOG.md b/CHANGELOG.md index a7cace4..e4d0f93 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- W3C xmlconf eduni sections (#121, #127): the Edinburgh errata suites (2e, 3e, and the 5th-edition 4e), Namespaces 1.0 with its errata, and the misc cases run via the XMLCONF_ROOT runner; XML 1.1 and Namespaces 1.1 are explicitly out of scope and EDITION-gated cases skip when they do not apply to the Fifth Edition. The burn-down fixed a real spec bug and several errata behaviors: attribute values now get the 3.3.3 CDATA normalization (literal whitespace becomes a space before reference decoding, so character-referenced whitespace survives, fixing errata E36 and E20 semantics) and tokenized normalization collapses only SPACE characters; an undeclared general entity in content or an attribute value is a deferred validity finding instead of a fatal error when unread external declarations might supply it (production 68; the reference stays literal, nothing is fetched), matching the scanner-side split; the errata E15 family is enforced (any reference, comment, PI, or whitespace inside an element declared EMPTY; direct or double-escaped character-reference whitespace as character data in element content, while entity-supplied whitespace stays legal); errata E2 reports duplicate tokens in enumerated and NOTATION declarations; errata E14's declaration-completed-inside-a-PE-replacement parses and reports VC: Proper Declaration/PE Nesting; the ID family must be NCNames under namespace validation; and DTD-strict mode requires namespace declarations to be declared like any attribute. Remaining deviations are exact and classified: 24+1 grapheme-cluster lexing cases (a combining mark after an ASCII delimiter merges into one Swift Character), 18 namespace-constraint cases, 4 encoding-declaration mismatches, and E18 (per-entity base-URI tracking), each filed as its own issue. - W3C xmlconf japanese section, fully clean (#121, #126): all 13 cases (the same prose and parameter-entity DTDs encoded as UTF-8, UTF-16 in both byte orders, Shift_JIS, EUC-JP, and ISO-2022-JP) decode through PureXML's own byte decoder, parse, and DTD-validate cleanly with zero deviations; the optional encodings the suite classifies as TYPE='error' are all supported, so they are held to the must-parse bar. - W3C xmlconf IBM section, clean to two documented classes (#121, #125): the deepest block, 1094 cases. Valid 220/220 (parse + strict DTD validation), invalid 64/64 reporting, not-wf 529/810 rejected with the 281 accepted cases being exactly two documented classes: 279 cases from productions 85-89, the 1998 character-class appendices deleted by XML 1.0 Fifth Edition (whose count is asserted exactly so drift in either direction is caught), and 2 parameter-entity-in-internal-subset feature cases (the xmltest 160-162 class, headed for #128's strict profile). Closing the section's real gaps: a document without a root element is rejected (production 1); an entity reference in an entity literal must be a lexical Name or character reference (production 68); WFC: No Recursion is enforced at declaration over the combined general/parameter reference graph (CDATA-shielded), so a recursive pair is rejected without being referenced; an undeclared parameter-entity reference is a well-formedness error when the document is standalone or has neither an external subset nor parameter entities, and a deferred validity finding otherwise (the production-68 WFC/VC split, also applied to entity references in attribute defaults); a declared external PE the resolver refused is neither; and a content-model group opening in one PE replacement and closing in another is reported (VC: Proper Group/PE Nesting). - Standalone validity constraints (#134): a document declaring `standalone='yes'` is now checked against its external-subset dependencies (2.9). The DTD scanner records declaration provenance (which entities, element models, and attribute lists came from the internal subset); the validator reports an externally-declared attribute default that would be supplied, an attribute value an externally-declared non-CDATA type would normalize, and whitespace inside externally-declared element content. The parser enforces the matching well-formedness side: a standalone document referencing an entity declared outside the internal subset reports it as undeclared (WFC: Entity Declared). This empties the entire Sun baseline: the section is fully clean at not-wf 56/56, valid 28/28, invalid 74/74, zero deviations, closing #124. diff --git a/README.md b/README.md index 4b2c4f5..e535231 100644 --- a/README.md +++ b/README.md @@ -92,7 +92,7 @@ flowchart TB Sun["#124 Sun section: complete, zero deviations"]:::done IBM["#125 IBM section: clean to two documented classes"]:::done Japanese["#126 japanese section: complete, zero deviations"]:::done - Eduni["#127 eduni: errata + namespaces"]:::todo + Eduni["#127 eduni: runner + exact baselines, four classified deviations"]:::done StrictSubset["#128 strict internal-subset profile: xmltest 185/186"]:::todo Epic122["#122 official spec suites"]:::epic XSTS["#129 W3C XSD test suite (XSTS)"]:::todo @@ -117,7 +117,7 @@ flowchart TB PureXML is a working, dependency-free XML library today: parse, emit, validate, query, and stream documents on macOS, Linux, Windows, and WASM. The test suite -currently runs **899 tests in 151 suites** (`swift test`). +currently runs **908 tests in 153 suites** (`swift test`). ### Shipped (libxml2-aligned surface) diff --git a/Sources/Parsing/DoctypeScanner.swift b/Sources/Parsing/DoctypeScanner.swift index 225fdbb..7b68298 100644 --- a/Sources/Parsing/DoctypeScanner.swift +++ b/Sources/Parsing/DoctypeScanner.swift @@ -220,7 +220,7 @@ struct DTDScanner { throw PureXML.Parsing.ParseError.invalidContentModel(element: name, mark) } reader.skipSpace() - let raw = readUntilClose(&reader) + let raw = recoveringDeclarationBody(readUntilClose(&reader), kind: "", name: name) checkGroupNesting(raw, element: name) let model = expandParameterReferences(raw).trimmingXMLWhitespace() guard PureXML.Parsing.DTDContentModelGrammar.isValid(model) else { @@ -248,7 +248,7 @@ struct DTDScanner { guard !name.isEmpty else { throw ParseError.invalidAttributeListDeclaration(mark) } - let body = expandParameterReferences(readUntilClose(&reader)) + let body = expandParameterReferences(recoveringDeclarationBody(readUntilClose(&reader), kind: "", name: name)) guard let defaults = PureXML.Parsing.DTDAttListGrammar.defaultValues(body) else { throw ParseError.invalidAttributeListDeclaration(mark) } @@ -300,14 +300,29 @@ extension DTDScanner { return value } - private func readUntilClose(_ reader: inout Reader) -> String { + /// Reads to the declaration's closing `>`. `closed` is false when the + /// input ended first, which can still be legal if a parameter-entity + /// replacement supplies the `>` (VC: Proper Declaration/PE Nesting). + private func readUntilClose(_ reader: inout Reader) -> (text: String, closed: Bool) { var text = "" while let character = reader.peek(), character != ">" { text.append(character) reader.advance() } - reader.consume(">") - return text + return (text, reader.consume(">")) + } + + /// When a declaration's `>` came from a parameter-entity replacement (the + /// source ended first), records the VC: Proper Declaration/PE Nesting + /// finding and truncates the expanded body at that `>`. + mutating func recoveringDeclarationBody(_ read: (text: String, closed: Bool), kind: String, name: String) -> String { + guard !read.closed else { return read.text } + let expanded = expandParameterReferences(read.text) + guard let close = expanded.firstIndex(of: ">") else { return read.text } + doctype.validityFindings.append( + "the \(kind) declaration for '\(name)' is completed inside a parameter-entity replacement (VC: Proper Declaration/PE Nesting)", + ) + return String(expanded[.. String { diff --git a/Sources/Parsing/EntityDecoder.swift b/Sources/Parsing/EntityDecoder.swift index 88190b6..756e722 100644 --- a/Sources/Parsing/EntityDecoder.swift +++ b/Sources/Parsing/EntityDecoder.swift @@ -24,6 +24,26 @@ extension PureXML.Parsing { budget = expander.budget return expander.result } + + /// Decodes like ``decode(_:entities:budget:at:)`` but treats an + /// undeclared entity as a validity finding instead of a fatal error + /// (production 68: with an external subset or parameter entities in + /// play, Entity Declared is a VC): the reference is kept literally and + /// its name appended to `undeclared`. + static func decodeLenient( + _ raw: String, + entities: [String: String], + budget: inout Int, + at mark: Mark, + undeclared: inout [String], + ) throws -> String { + guard raw.contains("&") else { return raw } + var expander = EntityExpander(entities: entities, mark: mark, budget: budget, undeclared: []) + try expander.expand(raw, visiting: [], counts: false) + budget = expander.budget + undeclared += expander.undeclared ?? [] + return expander.result + } } } @@ -154,6 +174,9 @@ private struct EntityExpander { let entities: [String: String] let mark: PureXML.Parsing.Mark var budget: Int + /// When non-nil, an undeclared entity is recorded here and kept literal + /// instead of throwing. + var undeclared: [String]? var result = "" mutating func expand(_ raw: String, visiting: Set, counts: Bool) throws { @@ -203,6 +226,11 @@ private struct EntityExpander { return } guard let replacement = entities[body] else { + if undeclared != nil { + undeclared?.append(body) + result += "&\(body);" + return + } throw PureXML.Parsing.ParseError.undefinedEntity(name: body, mark) } guard !visiting.contains(body) else { diff --git a/Sources/Parsing/EventReader.swift b/Sources/Parsing/EventReader.swift index ca11043..01bfabd 100644 --- a/Sources/Parsing/EventReader.swift +++ b/Sources/Parsing/EventReader.swift @@ -19,7 +19,7 @@ public extension PureXML.Parsing { var namespaces = NamespaceContext() /// The DTD information extracted so far (entities, element models, /// attribute lists, parameter entities, and external identifiers). - private(set) var documentType = DocumentType() + var documentType = DocumentType() /// The parsed XML declaration ``, when the document opens with /// one; nil otherwise. var xmlDeclaration: XMLDeclaration? @@ -91,15 +91,6 @@ public extension PureXML.Parsing { } } - /// The entity table visible to references. A standalone document must - /// not reference an entity declared outside its internal subset (WFC: - /// Entity Declared, 2.9), so external declarations are hidden and such - /// a reference reports as undeclared. - var referencableEntities: [String: String] { - guard xmlDeclaration?.standalone == true else { return documentType.entities } - return documentType.entities.filter { documentType.internalEntities.contains($0.key) } - } - private mutating func nextAtTopLevel() throws -> Event? { while true { reader.skipSpace() @@ -153,10 +144,14 @@ public extension PureXML.Parsing { } eventStart = reader.mark if reader.matches(" (target: String, data: String) { diff --git a/Sources/Parsing/EventReaderAttributes.swift b/Sources/Parsing/EventReaderAttributes.swift index f994277..202e4b9 100644 --- a/Sources/Parsing/EventReaderAttributes.swift +++ b/Sources/Parsing/EventReaderAttributes.swift @@ -51,12 +51,15 @@ extension PureXML.Parsing.EventReader { guard character.unicodeScalars.allSatisfy(PureXML.Parsing.XMLCharacter.isChar) else { throw PureXML.Parsing.ParseError.invalidCharacter(reader.mark) } - raw.append(character) + // 3.3.3: a literal whitespace character in an attribute value + // normalizes to a space; this happens before reference decoding, + // so whitespace that arrives via a character reference survives. + raw.append(character.isXMLWhitespace ? " " : character) reader.advance() } guard reader.consume(String(quote)) else { throw PureXML.Parsing.ParseError.unexpectedEndOfInput(reader.mark) } - return try PureXML.Parsing.EntityDecoder.decode(raw, entities: referencableEntities, budget: &entityBudget, at: mark) + return try decodeReferences(raw, at: mark) } } diff --git a/Sources/Parsing/EventReaderValidity.swift b/Sources/Parsing/EventReaderValidity.swift new file mode 100644 index 0000000..550e316 --- /dev/null +++ b/Sources/Parsing/EventReaderValidity.swift @@ -0,0 +1,106 @@ +/// The validity-aware side of reference handling: which entities a reference +/// may see (the standalone WFC), whether an undeclared reference is fatal +/// (production 68), and the errata E15 content findings. Split from the +/// reader body to keep it under the length caps. +extension PureXML.Parsing.EventReader { + typealias EntityDecoder = PureXML.Parsing.EntityDecoder + typealias Mark = PureXML.Parsing.Mark + + /// The entity table visible to references. A standalone document must + /// not reference an entity declared outside its internal subset (WFC: + /// Entity Declared, 2.9), so external declarations are hidden and such + /// a reference reports as undeclared. + var referencableEntities: [String: String] { + guard xmlDeclaration?.standalone == true else { return documentType.entities } + return documentType.entities.filter { documentType.internalEntities.contains($0.key) } + } + + /// Whether an undeclared entity reference is fatal (production 68): + /// it is a WFC when the document is standalone or could not have + /// external declarations, and a validity finding otherwise. + var undeclaredEntityIsFatal: Bool { + if xmlDeclaration?.standalone == true { return true } + return documentType.externalSubset == nil && documentType.parameterEntities.isEmpty + } + + /// Decodes references in text or an attribute value, with undeclared + /// entities routed per `undeclaredEntityIsFatal`. + mutating func decodeReferences(_ raw: String, at mark: Mark) throws -> String { + guard undeclaredEntityIsFatal else { + var undeclared: [String] = [] + let decoded = try EntityDecoder.decodeLenient( + raw, + entities: referencableEntities, + budget: &entityBudget, + at: mark, + undeclared: &undeclared, + ) + for name in undeclared { + documentType.validityFindings.append("general entity '&\(name);' is referenced but not declared") + } + return decoded + } + return try EntityDecoder.decode(raw, entities: referencableEntities, budget: &entityBudget, at: mark) + } + + /// Whether the run contains a direct character reference, or references + /// an entity whose stored replacement does (the errata E15h shape: the + /// double-escaped reference surfaces when the replacement is reparsed). + mutating func referencesCharacter(_ raw: String) -> Bool { + if containsCharacterReference(raw) { return true } + let entities = referencableEntities + var index = raw.startIndex + while index < raw.endIndex, let amp = raw[index...].firstIndex(of: "&") { + guard let semicolon = raw[amp...].firstIndex(of: ";") else { return false } + let name = String(raw[raw.index(after: amp) ..< semicolon]) + if let replacement = entities[name], containsCharacterReference(replacement) { + return true + } + index = raw.index(after: semicolon) + } + return false + } + + /// Whether the text contains a direct character reference (Foundation- + /// free substring test). + func containsCharacterReference(_ raw: String) -> Bool { + var sawAmp = false + for character in raw { + if sawAmp, character == "#" { return true } + sawAmp = character == "&" + } + return false + } + + /// Records the errata E15 finding for non-element content inside an + /// element declared EMPTY. + mutating func recordEmptyElementContent(_ what: String) { + guard let current = open.last, + let model = documentType.elementModels[current.localName] ?? documentType.elementModels[current.description], + model == "EMPTY" + else { return } + documentType.validityFindings.append("\(what) appears in the EMPTY element <\(current.description)>") + } + + /// The errata E15 validity findings a text run can carry: any reference + /// inside an element declared EMPTY, and reference-derived whitespace + /// (which is character data, not ignorable whitespace) inside element + /// content. + mutating func recordReferenceContentFindings(raw: String, decoded: String) { + guard raw.contains("&"), let current = open.last, + let model = documentType.elementModels[current.localName] ?? documentType.elementModels[current.description] + else { return } + if model == "EMPTY" { + documentType.validityFindings.append("a reference appears in the EMPTY element <\(current.description)>") + return + } + // Errata E15: a direct character reference is character data even + // when it names whitespace; an entity reference whose replacement is + // whitespace is permitted in element content. + let isElementContent = model.hasPrefix("(") && !model.hasPrefix("(#PCDATA") + let isReferenceWhitespace = !decoded.isEmpty && decoded.allSatisfy(\.isWhitespace) + if isElementContent, isReferenceWhitespace, referencesCharacter(raw) { + documentType.validityFindings.append("character-reference whitespace in the element content of <\(current.description)>") + } + } +} diff --git a/Sources/Validation/DTDDeclarationValidity.swift b/Sources/Validation/DTDDeclarationValidity.swift index 0492a84..7b4efb8 100644 --- a/Sources/Validation/DTDDeclarationValidity.swift +++ b/Sources/Validation/DTDDeclarationValidity.swift @@ -46,6 +46,11 @@ extension PureXML.Validation.DTDSchema { findings.append("NOTATION attribute '\(declaration.name)' on '\(element)' lists undeclared notation '\(name)'") } } + // Errata E2: tokens in an enumerated or NOTATION list must be + // distinct. + if let duplicate = duplicateToken(declaration.type) { + findings.append("attribute '\(declaration.name)' on '\(element)' repeats the token '\(duplicate)'") + } if let problem = defaultProblem(declaration) { findings.append("attribute '\(declaration.name)' on '\(element)' \(problem)") } @@ -53,6 +58,20 @@ extension PureXML.Validation.DTDSchema { return findings } + /// The first repeated token of an enumerated or NOTATION type, or nil. + private static func duplicateToken(_ type: PureXML.Validation.AttributeType) -> String? { + let tokens: [String] + switch type { + case let .enumeration(allowed), let .notation(allowed): tokens = allowed + default: return nil + } + var seen: Set = [] + for token in tokens where !seen.insert(token).inserted { + return token + } + return nil + } + /// The first repeated name in a `(#PCDATA|a|b)` mixed model, or nil. private static func duplicateMixedName(_ model: String) -> String? { let text = model.trimmingXMLWhitespace() diff --git a/Sources/Validation/DTDValidationHelpers.swift b/Sources/Validation/DTDValidationHelpers.swift index 44b14bb..ad354af 100644 --- a/Sources/Validation/DTDValidationHelpers.swift +++ b/Sources/Validation/DTDValidationHelpers.swift @@ -6,7 +6,8 @@ extension PureXML.Validation.DTD { let content = childContent(of: element) switch model { case .empty: - return content.names.isEmpty && !content.hasText && !content.hasCDATA + // EMPTY admits nothing, not even whitespace (errata E15). + return content.names.isEmpty && !content.hasText && !content.hasCDATA && !content.hasWhitespaceText ? [] : [DTDFailure(reason: "element <\(name)> is declared EMPTY but has content", at: path)] case .any: @@ -63,6 +64,9 @@ extension PureXML.Validation.DTD { struct ChildContent { var names: [String] = [] var hasText = false + /// Whitespace-only text, which is ignorable in element content but + /// still content for an EMPTY element. + var hasWhitespaceText = false /// Present CDATA sections, which count as character data in element /// content even when empty or whitespace-only. var hasCDATA = false @@ -75,7 +79,11 @@ extension PureXML.Validation.DTD { case let .element(inner): content.names.append(inner.name.description) case let .text(value): - if value.contains(where: { !$0.isWhitespace }) { content.hasText = true } + if value.contains(where: { !$0.isWhitespace }) { + content.hasText = true + } else if !value.isEmpty { + content.hasWhitespaceText = true + } case .cdata: content.hasCDATA = true default: @@ -99,10 +107,13 @@ extension PureXML.Validation.DTD { return raw.map { normalize($0, for: declaration.type) } } - /// The 3.3.3 tokenized normalization. CDATA values pass through untouched. + /// The 3.3.3 tokenized normalization: sequences of SPACE characters + /// collapse and outer spaces strip. Only #x20 counts; whitespace that + /// arrived through a character reference is data, not separation. CDATA + /// values pass through untouched. static func normalize(_ value: String, for type: PureXML.Validation.AttributeType) -> String { if case .cdata = type { return value } - return value.split(whereSeparator: \.isWhitespace).joined(separator: " ") + return value.split(separator: " ").joined(separator: " ") } /// The error when a `#REQUIRED` attribute is absent. @@ -161,17 +172,18 @@ extension PureXML.Validation.DTD { let isNmtoken = PureXML.Parsing.XMLCharacter.isNmtoken let isName = PureXML.Parsing.XMLCharacter.isValidName let isEntity = { (token: String) in isName(token) && entities.contains(token) } - let tokens = value.split(whereSeparator: \.isWhitespace).map(String.init) + let tokens = value.split(separator: " ").map(String.init) let valid: Bool switch declaration.type { case .nmToken: valid = isNmtoken(value) case .nmTokens: valid = !tokens.isEmpty && tokens.allSatisfy(isNmtoken) case .entity: valid = isEntity(value) case .entities: valid = !tokens.isEmpty && tokens.allSatisfy(isEntity) - // The ID family must be lexical Names (VC: ID, IDREF); uniqueness and - // resolution are checked by the whole-tree integrity rule. - case .id, .idReference: valid = isName(value) - case .idReferences: valid = !tokens.isEmpty && tokens.allSatisfy(isName) + // The ID family must be lexical Names (VC: ID, IDREF), and NCNames + // under namespace validation (no colon); uniqueness and resolution + // are checked by the whole-tree integrity rule. + case .id, .idReference: valid = isName(value) && !value.contains(":") + case .idReferences: valid = !tokens.isEmpty && tokens.allSatisfy { isName($0) && !$0.contains(":") } case .cdata, .enumeration, .notation: return nil } guard !valid else { return nil } @@ -245,7 +257,7 @@ extension PureXML.Validation.DTD { case .idReference: references.append((value, name)) case .idReferences: - for token in value.split(whereSeparator: { $0.isWhitespace }) { + for token in value.split(separator: " ") { references.append((String(token), name)) } case .cdata, .enumeration, .notation, .nmToken, .nmTokens, .entity, .entities: diff --git a/Sources/Validation/DTDValidations.swift b/Sources/Validation/DTDValidations.swift index 5f4679f..453eaec 100644 --- a/Sources/Validation/DTDValidations.swift +++ b/Sources/Validation/DTDValidations.swift @@ -102,8 +102,8 @@ public extension PureXML.Validation { } /// In strict mode, every attribute is declared (VC: Attribute Value - /// Type). Namespace declarations are exempt: they are bindings, not - /// attributes, in the namespace-aware model. + /// Type), including namespace declarations: a DTD-valid namespaced + /// document declares its xmlns attributes. static var undeclaredAttributes: Validation { .init(description: "Every attribute is declared in the DTD") { context in let element = context.subject @@ -111,7 +111,6 @@ public extension PureXML.Validation { return element.attributes .filter { attribute in let name = attribute.name.description - guard name != "xmlns", !name.hasPrefix("xmlns:") else { return false } return !declarations.contains { $0.name == name || $0.name == attribute.name.localName } } .map { DTDFailure(reason: "attribute '\($0.name.description)' on <\(element.name.description)> is not declared", at: context.codingPath) } diff --git a/Tests/CatalogTests.swift b/Tests/CatalogTests.swift index aaf1e6f..f213dd9 100644 --- a/Tests/CatalogTests.swift +++ b/Tests/CatalogTests.swift @@ -67,9 +67,11 @@ struct CatalogTests { func test_noLoaderRefuses() throws { let resolver = try resolver().entityResolver { _ in nil } let xml = "&greeting;" - #expect(throws: PureXML.Parsing.ParseError.self) { - _ = try PureXML.parse(xml, limits: .init(allowDoctype: true), resolver: resolver) - } + // Nothing is fetched; the reference the unread DTD might have + // declared stays literal (production 68 makes it a validity issue, + // not a well-formedness error). + let node = try PureXML.parse(xml, limits: .init(allowDoctype: true), resolver: resolver) + #expect(stringValue(node) == "&greeting;") } private func stringValue(_ node: PureXML.Model.Node) -> String { diff --git a/Tests/EntityResolverTests.swift b/Tests/EntityResolverTests.swift index e6ff495..51cb8b4 100644 --- a/Tests/EntityResolverTests.swift +++ b/Tests/EntityResolverTests.swift @@ -127,11 +127,14 @@ struct EntityResolverTests { } @Test("The external subset is not fetched without a resolver") - func test_externalSubsetNotFetchedByDefault() { + func test_externalSubsetNotFetchedByDefault() throws { let xml = "&fromdtd;" - #expect(throws: PureXML.Parsing.ParseError.self) { - _ = try PureXML.parse(xml, limits: dtdAllowed) - } + // Nothing external is fetched. Per production 68 the undeclared + // reference is a validity (not well-formedness) issue when an unread + // external subset might have declared it, so the reference stays + // literal in the text. + let node = try PureXML.parse(xml, limits: dtdAllowed) + #expect(text(of: node) == "&fromdtd;") } // MARK: Helpers diff --git a/Tests/ErrataValidityTests.swift b/Tests/ErrataValidityTests.swift new file mode 100644 index 0000000..fddaad8 --- /dev/null +++ b/Tests/ErrataValidityTests.swift @@ -0,0 +1,80 @@ +@testable import PureXML +import Testing + +/// The XML 1.0 errata behaviors from the eduni burn-down: 3.3.3 attribute- +/// value normalization, the lenient undeclared-entity path (production 68), +/// the E15 family (references and comments in EMPTY elements, character- +/// reference whitespace in element content), E2 duplicate enumeration +/// tokens, and E14 declarations completed inside PE replacements. +@Suite("XML errata validity") +struct ErrataValidityTests { + private func limits() -> PureXML.Parsing.Limits { + PureXML.Parsing.Limits(allowDoctype: true) + } + + private func errors(_ xml: String, strict: Bool = false) throws -> [String] { + try PureXML.validateAgainstInternalDTD(xml, strict: strict).map(\.reason) + } + + @Test("Literal whitespace in attribute values normalizes to spaces; character references survive") + func test_attributeValueNormalization() throws { + let node = try PureXML.parse("") + guard case let .document(children) = node, case let .element(element)? = children.first else { + Issue.record("no element") + return + } + #expect(element.attributes[0].value == "1 2 3") + #expect(element.attributes[1].value == "x\ty") + } + + @Test("With an unread external subset, an undeclared entity is a finding, not a fatal error") + func test_lenientUndeclaredEntity() throws { + let xml = "]>\n&mystery;" + let found = try errors(xml) + #expect(found.contains { $0.contains("'&mystery;' is referenced but not declared") }) + // Without any external declarations possible it stays fatal. + #expect(throws: PureXML.Parsing.ParseError.self) { + _ = try PureXML.parse("]>\n&mystery;", limits: limits()) + } + } + + @Test("E15: references, comments, and whitespace are content for EMPTY elements") + func test_emptyElementContent() throws { + let reference = "]>\n∅" + #expect(try !errors(reference).isEmpty) + let comment = "]>\n" + #expect(try !errors(comment).isEmpty) + let whitespace = "]>\n " + #expect(try !errors(whitespace).isEmpty) + let clean = "]>\n" + #expect(try errors(clean).isEmpty) + } + + @Test("E15: character-reference whitespace is data in element content; entity whitespace is not") + func test_elementContentWhitespace() throws { + let charRef = "]>\n " + #expect(try !errors(charRef).isEmpty) + let viaEntity = "]>\n&s;" + #expect(try errors(viaEntity).isEmpty) + let doubleEscape = "]>\n&s;" + #expect(try !errors(doubleEscape).isEmpty) + } + + @Test("E2: duplicate tokens in enumerated and NOTATION lists are reported") + func test_duplicateEnumerationTokens() throws { + let xml = "]>\n" + #expect(try errors(xml).contains { $0.contains("repeats the token 'one'") }) + } + + @Test("E14: a declaration completed inside a PE replacement parses and reports the VC") + func test_declarationPENesting() throws { + let dtd = "\n\">\n\n" + let resolver = PureXML.Parsing.EntityResolver( + resolveEntity: { _, _ in nil }, + resolveExternalSubset: { _ in dtd }, + ) + let found = try PureXML.validateAgainstInternalDTD(xml, resolver: resolver).map(\.reason) + #expect(found.contains { $0.contains("Proper Declaration/PE Nesting") }) + } +} diff --git a/Tests/W3CEduniSuiteTests.swift b/Tests/W3CEduniSuiteTests.swift new file mode 100644 index 0000000..9ad499c --- /dev/null +++ b/Tests/W3CEduniSuiteTests.swift @@ -0,0 +1,203 @@ +import Foundation +@testable import PureXML +import Testing + +/// Opt-in runner for the W3C XML Conformance Test Suite's eduni (Edinburgh) +/// sections: the XML 1.0 errata suites (2e, 3e, and the 5th-edition 4e), the +/// Namespaces 1.0 suite with its errata, and the misc cases. The XML 1.1 and +/// Namespaces 1.1 sections are out of scope: PureXML targets XML 1.0 Fifth +/// Edition. Cases marked EDITION without a 5 apply only to earlier editions +/// and are skipped; TYPE='error' cases are optional behavior by definition. +/// Point `XMLCONF_ROOT` at the extracted `xmlconf` directory to run (the +/// suite is never vendored). Foundation is used for file access only. +@Suite("W3C eduni conformance (opt-in via XMLCONF_ROOT)") +struct W3CEduniSuiteTests { + private var root: String? { + ProcessInfo.processInfo.environment["XMLCONF_ROOT"] + } + + /// Sections in scope: manifest file and its base directory under eduni/. + private let sections: [(manifest: String, directory: String)] = [ + ("errata-2e/errata2e.xml", "errata-2e"), + ("errata-3e/errata3e.xml", "errata-3e"), + ("errata-4e/errata4e.xml", "errata-4e"), + ("namespaces/1.0/rmt-ns10.xml", "namespaces/1.0"), + ("namespaces/errata-1e/errata1e.xml", "namespaces/errata-1e"), + ("misc/ht-bh.xml", "misc"), + ] + + /// not-wf cases this implementation knowingly accepts, two classes: + /// the namespace constraints not yet enforced (reserved prefixes and + /// namespace names, duplicate expanded-name attributes, NCName rules) + /// and encoding-declaration mismatches the byte decoder does not yet + /// reject (a declared encoding contradicting the actual bytes or BOM). + /// Both classes are tracked as their own issues under #121. + private let knownNotWFAccepted: Set = [ + // Namespace constraints. + "namespaces/1.0/009.xml", "namespaces/1.0/010.xml", "namespaces/1.0/011.xml", + "namespaces/1.0/012.xml", "namespaces/1.0/014.xml", "namespaces/1.0/015.xml", + "namespaces/1.0/016.xml", "namespaces/1.0/029.xml", "namespaces/1.0/030.xml", + "namespaces/1.0/031.xml", "namespaces/1.0/032.xml", "namespaces/1.0/033.xml", + "namespaces/1.0/036.xml", "namespaces/1.0/042.xml", "namespaces/1.0/043.xml", + "namespaces/1.0/044.xml", "namespaces/errata-1e/NE13a.xml", "namespaces/errata-1e/NE13b.xml", + // Encoding-declaration mismatches. + "errata-2e/E38.xml", "errata-2e/E61.xml", "misc/007.xml", "misc/008.xml", + ] + + /// valid cases this implementation knowingly fails, two classes: the + /// grapheme-cluster lexing limitation (a combining mark directly after an + /// ASCII delimiter merges into one Swift Character, so scalar-level 5e + /// name characters are mis-lexed; its own issue under #121) and E18, + /// which needs per-entity base-URI tracking for nested external entities. + private let knownValidFailures: Set = [ + "errata-2e/E18.xml", + "errata-4e/ibm04v01.xml", "errata-4e/ibm05v03.xml", "errata-4e/ibm07v01.xml", + "errata-4e/ibm85n107.xml", "errata-4e/ibm85n114.xml", "errata-4e/ibm85n119.xml", + "errata-4e/ibm85n121.xml", "errata-4e/ibm85n122.xml", "errata-4e/ibm85n136.xml", + "errata-4e/ibm85n137.xml", "errata-4e/ibm85n45.xml", "errata-4e/ibm85n50.xml", + "errata-4e/ibm85n51.xml", "errata-4e/ibm85n52.xml", "errata-4e/ibm85n53.xml", + "errata-4e/ibm85n54.xml", "errata-4e/ibm85n63.xml", "errata-4e/ibm85n73.xml", + "errata-4e/ibm85n74.xml", "errata-4e/ibm85n76.xml", "errata-4e/ibm85n89.xml", + "errata-4e/ibm85n91.xml", "errata-4e/ibm86n04.xml", "errata-4e/ibm87n60.xml", + ] + + /// invalid cases where this implementation knowingly reports no validity + /// error (none); 140.xml is carried here because it cannot parse yet (the + /// grapheme-cluster lexing class: its element name is a combining mark). + private let knownInvalidAccepted: Set = [ + "errata-4e/140.xml", + ] + + private struct ManifestCase { + let id: String + let type: String + let uri: String + let editions: String? + } + + private func manifest(_ file: String, in eduni: String) throws -> [ManifestCase] { + var source = try String(contentsOfFile: eduni + "/" + file, encoding: .utf8) + if source.hasPrefix("") { + source = String(source[declarationEnd.upperBound...]) + } + let document = try PureXML.parse("\(source)") + return try PureXML.xpath("//TEST", over: document).compactMap { selection in + guard let element = selection.element else { return nil } + func attribute(_ name: String) -> String? { + element.attributes.first { $0.name.description == name }?.value + } + guard let id = attribute("ID"), let type = attribute("TYPE"), let uri = attribute("URI") else { + return nil + } + return ManifestCase(id: id, type: type, uri: uri, editions: attribute("EDITION")) + } + } + + /// Whether a case applies to the Fifth Edition this package implements. + private func appliesToFifthEdition(_ testCase: ManifestCase) -> Bool { + guard let editions = testCase.editions else { return true } + return editions.split(separator: " ").contains("5") + } + + private func resolver(directory: String) -> PureXML.Parsing.EntityResolver { + @Sendable func load(_ id: PureXML.Parsing.ExternalID) -> String? { + guard let data = try? Data(contentsOf: URL(fileURLWithPath: directory + "/" + id.systemID)) else { + return nil + } + return try? PureXML.Parsing.ByteDecoder.decode([UInt8](data)) + } + return PureXML.Parsing.EntityResolver( + resolveEntity: { _, id in load(id) }, + resolveExternalSubset: { id in load(id) }, + ) + } + + private func caseDirectory(_ base: String, uri: String) -> String { + guard let slash = uri.lastIndex(of: "/") else { return base } + return base + "/" + String(uri.prefix(upTo: slash)) + } + + private func validate(_ uri: String, in base: String) throws -> [PureXML.Validation.ValidationError] { + let data = try Data(contentsOf: URL(fileURLWithPath: base + "/" + uri)) + let source = try PureXML.Parsing.ByteDecoder.decode([UInt8](data)) + return try PureXML.validateAgainstInternalDTD( + source, + limits: .init(allowDoctype: true), + strict: true, + resolver: resolver(directory: caseDirectory(base, uri: uri)), + ) + } + + @Test("Every eduni not-wf case is rejected by the strict parser") + func test_notWellFormedRejected() throws { + guard let root else { + return // Opt-in: suite not present. + } + let eduni = root + "/eduni" + var unexpectedlyAccepted: [String] = [] + for section in sections { + let base = eduni + "/" + section.directory + for testCase in try manifest(section.manifest, in: eduni) { + guard testCase.type == "not-wf", appliesToFifthEdition(testCase) else { continue } + let key = section.directory + "/" + testCase.uri + guard !knownNotWFAccepted.contains(key) else { continue } + if (try? validate(testCase.uri, in: base)) != nil { + unexpectedlyAccepted.append(key) + } + } + } + #expect(unexpectedlyAccepted.isEmpty, "accepted \(unexpectedlyAccepted.count) not-wf cases: \(unexpectedlyAccepted)") + } + + @Test("Every eduni valid case parses and validates cleanly") + func test_validCleans() throws { + guard let root else { + return // Opt-in: suite not present. + } + let eduni = root + "/eduni" + var failures: [String: String] = [:] + for section in sections { + let base = eduni + "/" + section.directory + for testCase in try manifest(section.manifest, in: eduni) { + guard testCase.type == "valid", appliesToFifthEdition(testCase) else { continue } + let key = section.directory + "/" + testCase.uri + guard !knownValidFailures.contains(key) else { continue } + do { + if let first = try validate(testCase.uri, in: base).first { + failures[key] = String(describing: first) + } + } catch { + failures[key] = String(describing: error) + } + } + } + #expect(failures.isEmpty, "\(failures.count) valid cases failed: \(failures)") + } + + @Test("Every eduni invalid case parses but reports a validity error") + func test_invalidReports() throws { + guard let root else { + return // Opt-in: suite not present. + } + let eduni = root + "/eduni" + var silentlyClean: [String] = [] + var parseFailures: [String: String] = [:] + for section in sections { + let base = eduni + "/" + section.directory + for testCase in try manifest(section.manifest, in: eduni) { + guard testCase.type == "invalid", appliesToFifthEdition(testCase) else { continue } + let key = section.directory + "/" + testCase.uri + guard !knownInvalidAccepted.contains(key) else { continue } + do { + if try validate(testCase.uri, in: base).isEmpty { + silentlyClean.append(key) + } + } catch { + parseFailures[key] = String(describing: error) + } + } + } + #expect(parseFailures.isEmpty, "\(parseFailures.count) invalid cases failed to parse: \(parseFailures)") + #expect(silentlyClean.isEmpty, "\(silentlyClean.count) invalid cases validated clean: \(silentlyClean)") + } +}