diff --git a/CHANGELOG.md b/CHANGELOG.md index 5390acd..7d4204a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Encoding-declaration contradictions are fatal (#137, 4.3.3): a document whose XML declaration names a 16- or 32-bit encoding over BOM-less 8-bit bytes is rejected; a BOM followed by a declaration outside its family (a UTF-8 BOM with `encoding='iso-8859-1'`, a UTF-16 BOM with an 8-bit name) is rejected, with the declaration read through the BOM's own byte pattern; and an external entity whose text declaration names a higher XML version than the document's is refused (errata E38), in both the external-subset and external-entity paths. The eduni not-wf baseline is now completely empty. - Namespaces 1.0 constraints (#136): the namespace layer now enforces the binding rules as located parse errors: malformed qualified names (`foo:`, `:foo`, `a:b:c`, `xmlns:`) are rejected; the `xmlns` prefix may never be declared; `xml` may be bound only to its own namespace name, which no other prefix (nor the default) may take, likewise the xmlns namespace name; a prefix may not be undeclared (Namespaces 1.0); attributes must be distinct by expanded name (the same URI+local name through two prefixes is a duplicate); PI targets and entity/notation names are NCNames; and a namespace declaration whose attribute is DTD-declared with a tokenized type binds its normalized value (so ` urn:x ` and `urn:x` are one namespace name, eduni ns 012). Clears all 18 namespace cases from the eduni baseline, leaving only the four #137 encoding-mismatch cases; xmltest valid/sa/012 (an attribute literally named `:`) joins the documented deviations per the suite's own NAMESPACE='no' flag, mirrored by the eduni runner's new NAMESPACE filter. - W3C xmlconf eduni sections (#121, #127): the Edinburgh errata suites (2e, 3e, and the 5th-edition 4e), Namespaces 1.0 with its errata, and the misc cases run via the XMLCONF_ROOT runner; XML 1.1 and Namespaces 1.1 are explicitly out of scope and EDITION-gated cases skip when they do not apply to the Fifth Edition. The burn-down fixed a real spec bug and several errata behaviors: attribute values now get the 3.3.3 CDATA normalization (literal whitespace becomes a space before reference decoding, so character-referenced whitespace survives, fixing errata E36 and E20 semantics) and tokenized normalization collapses only SPACE characters; an undeclared general entity in content or an attribute value is a deferred validity finding instead of a fatal error when unread external declarations might supply it (production 68; the reference stays literal, nothing is fetched), matching the scanner-side split; the errata E15 family is enforced (any reference, comment, PI, or whitespace inside an element declared EMPTY; direct or double-escaped character-reference whitespace as character data in element content, while entity-supplied whitespace stays legal); errata E2 reports duplicate tokens in enumerated and NOTATION declarations; errata E14's declaration-completed-inside-a-PE-replacement parses and reports VC: Proper Declaration/PE Nesting; the ID family must be NCNames under namespace validation; and DTD-strict mode requires namespace declarations to be declared like any attribute. Remaining deviations are exact and classified: 24+1 grapheme-cluster lexing cases (a combining mark after an ASCII delimiter merges into one Swift Character), 18 namespace-constraint cases, 4 encoding-declaration mismatches, and E18 (per-entity base-URI tracking), each filed as its own issue. - W3C xmlconf japanese section, fully clean (#121, #126): all 13 cases (the same prose and parameter-entity DTDs encoded as UTF-8, UTF-16 in both byte orders, Shift_JIS, EUC-JP, and ISO-2022-JP) decode through PureXML's own byte decoder, parse, and DTD-validate cleanly with zero deviations; the optional encodings the suite classifies as TYPE='error' are all supported, so they are held to the must-parse bar. diff --git a/README.md b/README.md index 8dbe794..7c7cb4e 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,7 @@ flowchart TB StrictSubset["#128 strict internal-subset profile: xmltest 185/186"]:::todo ScalarLex["#135 scalar-level lexing (grapheme clusters)"]:::todo NSConstraints["#136 namespace constraints: complete"]:::done - EncodingMismatch["#137 encoding-declaration mismatches"]:::todo + EncodingMismatch["#137 encoding-declaration mismatches: complete"]:::done BaseURI["#138 per-entity base URIs"]:::todo Epic122["#122 official spec suites"]:::epic XSTS["#129 W3C XSD test suite (XSTS)"]:::todo @@ -125,7 +125,7 @@ flowchart TB PureXML is a working, dependency-free XML library today: parse, emit, validate, query, and stream documents on macOS, Linux, Windows, and WASM. The test suite -currently runs **914 tests in 154 suites** (`swift test`). +currently runs **917 tests in 155 suites** (`swift test`). ### Shipped (libxml2-aligned surface) diff --git a/Sources/Parsing/ByteDecoder.swift b/Sources/Parsing/ByteDecoder.swift index 2ef40cf..8e8e04f 100644 --- a/Sources/Parsing/ByteDecoder.swift +++ b/Sources/Parsing/ByteDecoder.swift @@ -7,11 +7,11 @@ extension PureXML.Parsing { static func decode(_ bytes: [UInt8]) throws -> String { let (sniffed, bomLength) = InputEncoding.detectWithBOM(bytes) // With no BOM the byte sniff defaults to UTF-8; honor a declared - // single-byte encoding if the XML declaration names one. - var encoding = sniffed - if sniffed == .utf8, bomLength == 0, let declared = declaredEncoding(in: bytes) { - encoding = declared - } + // single-byte encoding if the XML declaration names one. A declared + // encoding that contradicts the actual bytes or the BOM is a fatal + // error (4.3.3): 16/32-bit names on 8-bit content, or any name + // outside the BOM's family. + let encoding = try declarationConsistentEncoding(sniffed: sniffed, bomLength: bomLength, bytes: bytes) let body = bytes.dropFirst(bomLength) if let cjk = multiByteDecode(encoding, body) { return cjk @@ -130,6 +130,76 @@ extension PureXML.Parsing { return String(scalars) } + /// Applies the declared encoding when the sniff allows it, throwing on + /// a contradiction (4.3.3): a 16/32-bit name over 8-bit bytes, or any + /// name outside the BOM's family. + private static func declarationConsistentEncoding( + sniffed: InputEncoding, + bomLength: Int, + bytes: [UInt8], + ) throws -> InputEncoding { + if bomLength == 0 { + guard sniffed == .utf8, let declared = declaredEncoding(in: bytes) else { return sniffed } + guard !isWideUnicode(declared) else { + throw PureXML.Parsing.ParseError.malformedEncoding + } + return declared + } + let declared = declaredEncoding(after: sniffed, in: bytes, bomLength: bomLength) + if let declared, !sameFamily(declared, as: sniffed) { + throw PureXML.Parsing.ParseError.malformedEncoding + } + return sniffed + } + + /// Whether the encoding is a 16- or 32-bit Unicode form, which cannot + /// describe content whose declaration was readable as 8-bit bytes. + private static func isWideUnicode(_ encoding: InputEncoding) -> Bool { + switch encoding { + case .utf16BigEndian, .utf16LittleEndian, .utf32BigEndian, .utf32LittleEndian: true + default: false + } + } + + /// Whether a declared encoding is consistent with the BOM-sniffed one: + /// the same encoding, or any 8-bit ASCII-compatible name under a UTF-8 + /// BOM is still a contradiction, only utf-8 itself agrees; the generic + /// utf-16/utf-32 names agree with either byte order. + private static func sameFamily(_ declared: InputEncoding, as sniffed: InputEncoding) -> Bool { + if declared == sniffed { return true } + switch (sniffed, declared) { + case (.utf16BigEndian, .utf16LittleEndian), (.utf16LittleEndian, .utf16BigEndian), + (.utf32BigEndian, .utf32LittleEndian), (.utf32LittleEndian, .utf32BigEndian): + // The generic names map to one byte order; a BOM of the other + // order still names the same encoding. + return true + default: + return false + } + } + + /// Reads the `encoding` pseudo-attribute of a BOM-prefixed document by + /// reducing the header to its ASCII bytes per the sniffed family. + private static func declaredEncoding(after sniffed: InputEncoding, in bytes: [UInt8], bomLength: Int) -> InputEncoding? { + switch sniffed { + case .utf8: + return declaredEncoding(in: Array(bytes.dropFirst(bomLength))) + case .utf16BigEndian, .utf16LittleEndian: + let body = bytes.dropFirst(bomLength).prefix(512) + var ascii: [UInt8] = [] + var index = body.startIndex + while index < body.endIndex, body.index(after: index) < body.endIndex { + let high = sniffed == .utf16BigEndian ? body[index] : body[body.index(after: index)] + let low = sniffed == .utf16BigEndian ? body[body.index(after: index)] : body[index] + if high == 0 { ascii.append(low) } + index = body.index(index, offsetBy: 2) + } + return declaredEncoding(in: ascii) + default: + return nil + } + } + /// Reads the `encoding` pseudo-attribute from an XML declaration that is /// ASCII-compatible at the front (UTF-8 or a single-byte encoding). static func declaredEncoding(in bytes: [UInt8]) -> InputEncoding? { @@ -156,6 +226,10 @@ extension PureXML.Parsing { /// The declared encoding names PureXML recognizes, mapped to their encoding. private static let encodingByName: [String: InputEncoding] = [ "utf-8": .utf8, "utf8": .utf8, "us-ascii": .utf8, "ascii": .utf8, + "utf-16": .utf16BigEndian, "utf16": .utf16BigEndian, + "utf-16be": .utf16BigEndian, "utf-16le": .utf16LittleEndian, + "utf-32": .utf32BigEndian, "utf32": .utf32BigEndian, + "utf-32be": .utf32BigEndian, "utf-32le": .utf32LittleEndian, "iso-8859-1": .latin1, "latin1": .latin1, "latin-1": .latin1, "l1": .latin1, "windows-1252": .windows1252, "cp1252": .windows1252, "cp-1252": .windows1252, "windows-1254": .windows1254, "cp1254": .windows1254, diff --git a/Sources/Parsing/DoctypeScanner.swift b/Sources/Parsing/DoctypeScanner.swift index 2c721e9..4072103 100644 --- a/Sources/Parsing/DoctypeScanner.swift +++ b/Sources/Parsing/DoctypeScanner.swift @@ -15,12 +15,13 @@ extension PureXML.Parsing { limits: Limits, resolver: EntityResolver = .refusing, standalone: Bool = false, + documentVersion: String? = nil, ) throws -> DocumentType { let mark = reader.mark guard limits.allowDoctype else { throw ParseError.unsupportedDoctype(mark) } - var scanner = DTDScanner(limits: limits, resolver: resolver, standalone: standalone) + var scanner = DTDScanner(limits: limits, resolver: resolver, standalone: standalone, documentVersion: documentVersion) return try scanner.scan(&reader, at: mark) } } @@ -42,6 +43,9 @@ struct DTDScanner { /// undeclared-entity reference a well-formedness error rather than a /// validity finding (WFC vs VC: Entity Declared). let standalone: Bool + /// The document's XML version: an external entity may not declare a + /// higher version in its text declaration (errata E38). + let documentVersion: String var doctype = PureXML.Parsing.DocumentType() /// True while scanning the external subset, so declarations record their /// origin (the standalone validity constraints depend on it). @@ -55,10 +59,16 @@ struct DTDScanner { /// only a little); deeper references are ignored rather than trapping. let maxDepth = 40 - init(limits: PureXML.Parsing.Limits, resolver: PureXML.Parsing.EntityResolver, standalone: Bool = false) { + init( + limits: PureXML.Parsing.Limits, + resolver: PureXML.Parsing.EntityResolver, + standalone: Bool = false, + documentVersion: String? = nil, + ) { self.limits = limits self.resolver = resolver self.standalone = standalone + self.documentVersion = documentVersion ?? "1.0" parameterBudget = limits.maxEntityExpansion } diff --git a/Sources/Parsing/DoctypeScannerEntities.swift b/Sources/Parsing/DoctypeScannerEntities.swift index dbff933..aacb5cc 100644 --- a/Sources/Parsing/DoctypeScannerEntities.swift +++ b/Sources/Parsing/DoctypeScannerEntities.swift @@ -12,7 +12,7 @@ extension DTDScanner { if let text = resolver.resolveEntity(name, id) { // An external parsed entity may begin with a text declaration, // which is not part of its replacement text (4.3.1). - doctype.entities[name] = strippingTextDeclaration(text) + doctype.entities[name] = try? strippingTextDeclaration(text) } } } @@ -65,7 +65,11 @@ extension DTDScanner { } else if isParameter { if doctype.parameterEntities[name] == nil { if let text = resolver.resolveExternalSubset(id) { - doctype.parameterEntities[name] = expandParameterReferences(strippingTextDeclaration(text)) + if let stripped = try? strippingTextDeclaration(text) { + doctype.parameterEntities[name] = expandParameterReferences(stripped) + } else { + unresolvedParameterEntities.insert(name) + } } else { unresolvedParameterEntities.insert(name) } diff --git a/Sources/Parsing/DoctypeScannerStrict.swift b/Sources/Parsing/DoctypeScannerStrict.swift index dd729cc..9861dac 100644 --- a/Sources/Parsing/DoctypeScannerStrict.swift +++ b/Sources/Parsing/DoctypeScannerStrict.swift @@ -136,7 +136,14 @@ extension DTDScanner { /// text (4.3.1): the declaration is not part of the replacement text. Text /// without one passes through unchanged, including text whose ` String { + /// Errata E38: an external entity may not declare a version the document + /// does not at least match ("1.1" inside a "1.0" document is an error). + func textDeclarationVersionAllowed(_ declaration: PureXML.Parsing.XMLDeclaration) -> Bool { + guard let declared = declaration.version else { return true } + return declared <= documentVersion + } + + func strippingTextDeclaration(_ text: String) throws -> String { guard text.hasPrefix("'. var close = text.index(text.startIndex, offsetBy: 5) @@ -147,10 +154,13 @@ extension DTDScanner { close = text.index(after: close) } guard close < text.endIndex else { return text } - let declaration = String(text[text.index(text.startIndex, offsetBy: 5) ..< close]) - guard declaration.first?.isWhitespace == true, - PureXML.Parsing.XMLDeclaration.parseTextDeclaration(declaration) != nil + let body = String(text[text.index(text.startIndex, offsetBy: 5) ..< close]) + guard body.first?.isWhitespace == true, + let declaration = PureXML.Parsing.XMLDeclaration.parseTextDeclaration(body) else { return text } + guard textDeclarationVersionAllowed(declaration) else { + throw ParseError.malformedDeclaration(.start) + } return String(text[text.index(close, offsetBy: 2)...]) } diff --git a/Sources/Parsing/EventReader.swift b/Sources/Parsing/EventReader.swift index 6d18ccc..8e1bc6a 100644 --- a/Sources/Parsing/EventReader.swift +++ b/Sources/Parsing/EventReader.swift @@ -116,7 +116,13 @@ public extension PureXML.Parsing { return .processingInstruction(target: instruction.target, data: instruction.data) } if reader.matches(" Event { - let mark = reader.mark - reader.consume("") else { - throw ParseError.unterminatedTag(reader.mark) - } - guard let top = open.last else { - throw ParseError.unexpectedEndTag(name: name.description, mark) - } - // Tag matching is lexical (the qualified-name text must match); the - // open name additionally carries its resolved namespace URI. - guard top.description == name.description else { - throw ParseError.mismatchedEndTag(expected: top.description, found: name.description, mark) - } - open.removeLast() - namespaces.leaveElement() - return .endElement(name: top) - } - /// Returns the next text event, or nil after splicing: a reference to /// an entity whose replacement (directly or transitively) contains /// markup is validated, budgeted, and pushed back into the reader so diff --git a/Sources/Parsing/EventReaderEndTag.swift b/Sources/Parsing/EventReaderEndTag.swift new file mode 100644 index 0000000..caa8981 --- /dev/null +++ b/Sources/Parsing/EventReaderEndTag.swift @@ -0,0 +1,26 @@ +/// The end-tag scanner, split from the reader body to keep it under the +/// length caps. Behavior is part of EventReader proper. +extension PureXML.Parsing.EventReader { + typealias Event = PureXML.Parsing.Event + typealias ParseError = PureXML.Parsing.ParseError + mutating func scanEndTag() throws -> Event { + let mark = reader.mark + reader.consume("") else { + throw ParseError.unterminatedTag(reader.mark) + } + guard let top = open.last else { + throw ParseError.unexpectedEndTag(name: name.description, mark) + } + // Tag matching is lexical (the qualified-name text must match); the + // open name additionally carries its resolved namespace URI. + guard top.description == name.description else { + throw ParseError.mismatchedEndTag(expected: top.description, found: name.description, mark) + } + open.removeLast() + namespaces.leaveElement() + return .endElement(name: top) + } +} diff --git a/Tests/EncodingMismatchTests.swift b/Tests/EncodingMismatchTests.swift new file mode 100644 index 0000000..7514037 --- /dev/null +++ b/Tests/EncodingMismatchTests.swift @@ -0,0 +1,55 @@ +@testable import PureXML +import Testing + +/// Encoding-declaration contradictions are fatal (#137, 4.3.3): a declared +/// 16/32-bit encoding over 8-bit bytes, a declaration outside the BOM's +/// family, and an external entity declaring a version above the document's +/// (errata E38). +@Suite("Encoding declaration mismatches") +struct EncodingMismatchTests { + @Test("UTF-16 declared over 8-bit bytes is rejected") + func test_wideOverNarrow() { + let bytes = Array("\n\n".utf8) + #expect(throws: PureXML.Parsing.ParseError.self) { + _ = try PureXML.Parsing.ByteDecoder.decode(bytes) + } + } + + @Test("A declaration outside the BOM's family is rejected") + func test_bomContradiction() { + // UTF-8 BOM + iso-8859-1 declaration. + let utf8: [UInt8] = [0xEF, 0xBB, 0xBF] + Array("".utf8) + #expect(throws: PureXML.Parsing.ParseError.self) { + _ = try PureXML.Parsing.ByteDecoder.decode(utf8) + } + // UTF-16BE BOM + iso-8859-1 declaration. + var utf16: [UInt8] = [0xFE, 0xFF] + for scalar in "".unicodeScalars { + utf16.append(0) + utf16.append(UInt8(scalar.value)) + } + #expect(throws: PureXML.Parsing.ParseError.self) { + _ = try PureXML.Parsing.ByteDecoder.decode(utf16) + } + // Matching declarations stay accepted. + let consistent: [UInt8] = [0xEF, 0xBB, 0xBF] + Array("".utf8) + #expect((try? PureXML.Parsing.ByteDecoder.decode(consistent)) != nil) + } + + @Test("An external entity declaring a higher version is rejected (E38)") + func test_entityVersionAboveDocument() { + let xml = "]>\n&e;" + let resolver = PureXML.Parsing.EntityResolver( + resolveEntity: { _, _ in "ok" }, + resolveExternalSubset: { _ in nil }, + ) + #expect(throws: PureXML.Parsing.ParseError.self) { + _ = try PureXML.parse(xml, limits: .init(allowDoctype: true), resolver: resolver) + } + let sameVersion = PureXML.Parsing.EntityResolver( + resolveEntity: { _, _ in "ok" }, + resolveExternalSubset: { _ in nil }, + ) + #expect((try? PureXML.parse(xml, limits: .init(allowDoctype: true), resolver: sameVersion)) != nil) + } +} diff --git a/Tests/W3CEduniSuiteTests.swift b/Tests/W3CEduniSuiteTests.swift index 234eb2f..49677a2 100644 --- a/Tests/W3CEduniSuiteTests.swift +++ b/Tests/W3CEduniSuiteTests.swift @@ -26,13 +26,8 @@ struct W3CEduniSuiteTests { ("misc/ht-bh.xml", "misc"), ] - /// not-wf cases this implementation knowingly accepts: the encoding- - /// declaration mismatches the byte decoder does not yet reject (a - /// declared encoding contradicting the actual bytes or BOM), tracked - /// as #137. - private let knownNotWFAccepted: Set = [ - "errata-2e/E38.xml", "errata-2e/E61.xml", "misc/007.xml", "misc/008.xml", - ] + /// not-wf cases this implementation knowingly accepts (none). + private let knownNotWFAccepted: Set = [] /// valid cases this implementation knowingly fails, two classes: the /// grapheme-cluster lexing limitation (a combining mark directly after an