NodetteHQ/PureXML
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat(parsing): fatal encoding-declaration contradictions, E38 version rule

Browse Source 
A declared 16/32-bit encoding over BOM-less 8-bit bytes is rejected; a
BOM followed by a declaration outside its family is rejected, reading
the declaration through the BOM's own byte pattern; and an external
entity whose text declaration names a higher XML version than the
document is refused (errata E38). The eduni not-wf baseline is empty.
Reader and decoder helpers split for the length caps.

Closes #137
This commit is contained in:
Mihaela Mihaljevic 2026-06-10 08:16:08 +02:00
parent 4b03c45434
commit 2c375ac74b
10 changed files with 204 additions and 44 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
CHANGELOG.md
View File

@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- Encoding-declaration contradictions are fatal (#137, 4.3.3): a document whose XML declaration names a 16- or 32-bit encoding over BOM-less 8-bit bytes is rejected; a BOM followed by a declaration outside its family (a UTF-8 BOM with `encoding='iso-8859-1'`, a UTF-16 BOM with an 8-bit name) is rejected, with the declaration read through the BOM's own byte pattern; and an external entity whose text declaration names a higher XML version than the document's is refused (errata E38), in both the external-subset and external-entity paths. The eduni not-wf baseline is now completely empty.
- Namespaces 1.0 constraints (#136): the namespace layer now enforces the binding rules as located parse errors: malformed qualified names (`foo:`, `:foo`, `a:b:c`, `xmlns:`) are rejected; the `xmlns` prefix may never be declared; `xml` may be bound only to its own namespace name, which no other prefix (nor the default) may take, likewise the xmlns namespace name; a prefix may not be undeclared (Namespaces 1.0); attributes must be distinct by expanded name (the same URI+local name through two prefixes is a duplicate); PI targets and entity/notation names are NCNames; and a namespace declaration whose attribute is DTD-declared with a tokenized type binds its normalized value (so ` urn:x ` and `urn:x` are one namespace name, eduni ns 012). Clears all 18 namespace cases from the eduni baseline, leaving only the four #137 encoding-mismatch cases; xmltest valid/sa/012 (an attribute literally named `:`) joins the documented deviations per the suite's own NAMESPACE='no' flag, mirrored by the eduni runner's new NAMESPACE filter.
- W3C xmlconf eduni sections (#121, #127): the Edinburgh errata suites (2e, 3e, and the 5th-edition 4e), Namespaces 1.0 with its errata, and the misc cases run via the XMLCONF_ROOT runner; XML 1.1 and Namespaces 1.1 are explicitly out of scope and EDITION-gated cases skip when they do not apply to the Fifth Edition. The burn-down fixed a real spec bug and several errata behaviors: attribute values now get the 3.3.3 CDATA normalization (literal whitespace becomes a space before reference decoding, so character-referenced whitespace survives, fixing errata E36 and E20 semantics) and tokenized normalization collapses only SPACE characters; an undeclared general entity in content or an attribute value is a deferred validity finding instead of a fatal error when unread external declarations might supply it (production 68; the reference stays literal, nothing is fetched), matching the scanner-side split; the errata E15 family is enforced (any reference, comment, PI, or whitespace inside an element declared EMPTY; direct or double-escaped character-reference whitespace as character data in element content, while entity-supplied whitespace stays legal); errata E2 reports duplicate tokens in enumerated and NOTATION declarations; errata E14's declaration-completed-inside-a-PE-replacement parses and reports VC: Proper Declaration/PE Nesting; the ID family must be NCNames under namespace validation; and DTD-strict mode requires namespace declarations to be declared like any attribute. Remaining deviations are exact and classified: 24+1 grapheme-cluster lexing cases (a combining mark after an ASCII delimiter merges into one Swift Character), 18 namespace-constraint cases, 4 encoding-declaration mismatches, and E18 (per-entity base-URI tracking), each filed as its own issue.
- W3C xmlconf japanese section, fully clean (#121, #126): all 13 cases (the same prose and parameter-entity DTDs encoded as UTF-8, UTF-16 in both byte orders, Shift_JIS, EUC-JP, and ISO-2022-JP) decode through PureXML's own byte decoder, parse, and DTD-validate cleanly with zero deviations; the optional encodings the suite classifies as TYPE='error' are all supported, so they are held to the must-parse bar.

4
README.md
View File

@ -96,7 +96,7 @@ flowchart TB
StrictSubset["#128 strict internal-subset profile: xmltest 185/186"]:::todo
ScalarLex["#135 scalar-level lexing (grapheme clusters)"]:::todo
NSConstraints["#136 namespace constraints: complete"]:::done
EncodingMismatch["#137 encoding-declaration mismatches"]:::todo
EncodingMismatch["#137 encoding-declaration mismatches: complete"]:::done
BaseURI["#138 per-entity base URIs"]:::todo
Epic122["#122 official spec suites"]:::epic
XSTS["#129 W3C XSD test suite (XSTS)"]:::todo
@ -125,7 +125,7 @@ flowchart TB
PureXML is a working, dependency-free XML library today: parse, emit, validate,
query, and stream documents on macOS, Linux, Windows, and WASM. The test suite
currently runs **914 tests in 154 suites** (`swift test`).
currently runs **917 tests in 155 suites** (`swift test`).
### Shipped (libxml2-aligned surface)

84
Sources/Parsing/ByteDecoder.swift
View File

@ -7,11 +7,11 @@ extension PureXML.Parsing {
static func decode(_ bytes: [UInt8]) throws -> String {
let (sniffed, bomLength) = InputEncoding.detectWithBOM(bytes)
// With no BOM the byte sniff defaults to UTF-8; honor a declared
// single-byte encoding if the XML declaration names one.
var encoding = sniffed
if sniffed == .utf8, bomLength == 0, let declared = declaredEncoding(in: bytes) {
encoding = declared
}
// single-byte encoding if the XML declaration names one. A declared
// encoding that contradicts the actual bytes or the BOM is a fatal
// error (4.3.3): 16/32-bit names on 8-bit content, or any name
// outside the BOM's family.
let encoding = try declarationConsistentEncoding(sniffed: sniffed, bomLength: bomLength, bytes: bytes)
let body = bytes.dropFirst(bomLength)
if let cjk = multiByteDecode(encoding, body) {
return cjk
@ -130,6 +130,76 @@ extension PureXML.Parsing {
return String(scalars)
}
/// Applies the declared encoding when the sniff allows it, throwing on
/// a contradiction (4.3.3): a 16/32-bit name over 8-bit bytes, or any
/// name outside the BOM's family.
private static func declarationConsistentEncoding(
sniffed: InputEncoding,
bomLength: Int,
bytes: [UInt8],
) throws -> InputEncoding {
if bomLength == 0 {
guard sniffed == .utf8, let declared = declaredEncoding(in: bytes) else { return sniffed }
guard !isWideUnicode(declared) else {
throw PureXML.Parsing.ParseError.malformedEncoding
}
return declared
}
let declared = declaredEncoding(after: sniffed, in: bytes, bomLength: bomLength)
if let declared, !sameFamily(declared, as: sniffed) {
throw PureXML.Parsing.ParseError.malformedEncoding
}
return sniffed
}
/// Whether the encoding is a 16- or 32-bit Unicode form, which cannot
/// describe content whose declaration was readable as 8-bit bytes.
private static func isWideUnicode(_ encoding: InputEncoding) -> Bool {
switch encoding {
case .utf16BigEndian, .utf16LittleEndian, .utf32BigEndian, .utf32LittleEndian: true
default: false
}
}
/// Whether a declared encoding is consistent with the BOM-sniffed one:
/// the same encoding, or any 8-bit ASCII-compatible name under a UTF-8
/// BOM is still a contradiction, only utf-8 itself agrees; the generic
/// utf-16/utf-32 names agree with either byte order.
private static func sameFamily(_ declared: InputEncoding, as sniffed: InputEncoding) -> Bool {
if declared == sniffed { return true }
switch (sniffed, declared) {
case (.utf16BigEndian, .utf16LittleEndian), (.utf16LittleEndian, .utf16BigEndian),
(.utf32BigEndian, .utf32LittleEndian), (.utf32LittleEndian, .utf32BigEndian):
// The generic names map to one byte order; a BOM of the other
// order still names the same encoding.
return true
default:
return false
}
}
/// Reads the `encoding` pseudo-attribute of a BOM-prefixed document by
/// reducing the header to its ASCII bytes per the sniffed family.
private static func declaredEncoding(after sniffed: InputEncoding, in bytes: [UInt8], bomLength: Int) -> InputEncoding? {
switch sniffed {
case .utf8:
return declaredEncoding(in: Array(bytes.dropFirst(bomLength)))
case .utf16BigEndian, .utf16LittleEndian:
let body = bytes.dropFirst(bomLength).prefix(512)
var ascii: [UInt8] = []
var index = body.startIndex
while index < body.endIndex, body.index(after: index) < body.endIndex {
let high = sniffed == .utf16BigEndian ? body[index] : body[body.index(after: index)]
let low = sniffed == .utf16BigEndian ? body[body.index(after: index)] : body[index]
if high == 0 { ascii.append(low) }
index = body.index(index, offsetBy: 2)
}
return declaredEncoding(in: ascii)
default:
return nil
}
}
/// Reads the `encoding` pseudo-attribute from an XML declaration that is
/// ASCII-compatible at the front (UTF-8 or a single-byte encoding).
static func declaredEncoding(in bytes: [UInt8]) -> InputEncoding? {
@ -156,6 +226,10 @@ extension PureXML.Parsing {
/// The declared encoding names PureXML recognizes, mapped to their encoding.
private static let encodingByName: [String: InputEncoding] = [
"utf-8": .utf8, "utf8": .utf8, "us-ascii": .utf8, "ascii": .utf8,
"utf-16": .utf16BigEndian, "utf16": .utf16BigEndian,
"utf-16be": .utf16BigEndian, "utf-16le": .utf16LittleEndian,
"utf-32": .utf32BigEndian, "utf32": .utf32BigEndian,
"utf-32be": .utf32BigEndian, "utf-32le": .utf32LittleEndian,
"iso-8859-1": .latin1, "latin1": .latin1, "latin-1": .latin1, "l1": .latin1,
"windows-1252": .windows1252, "cp1252": .windows1252, "cp-1252": .windows1252,
"windows-1254": .windows1254, "cp1254": .windows1254,

14
Sources/Parsing/DoctypeScanner.swift
View File

@ -15,12 +15,13 @@ extension PureXML.Parsing {
limits: Limits,
resolver: EntityResolver = .refusing,
standalone: Bool = false,
documentVersion: String? = nil,
) throws -> DocumentType {
let mark = reader.mark
guard limits.allowDoctype else {
throw ParseError.unsupportedDoctype(mark)
}
var scanner = DTDScanner(limits: limits, resolver: resolver, standalone: standalone)
var scanner = DTDScanner(limits: limits, resolver: resolver, standalone: standalone, documentVersion: documentVersion)
return try scanner.scan(&reader, at: mark)
}
}
@ -42,6 +43,9 @@ struct DTDScanner {
/// undeclared-entity reference a well-formedness error rather than a
/// validity finding (WFC vs VC: Entity Declared).
let standalone: Bool
/// The document's XML version: an external entity may not declare a
/// higher version in its text declaration (errata E38).
let documentVersion: String
var doctype = PureXML.Parsing.DocumentType()
/// True while scanning the external subset, so declarations record their
/// origin (the standalone validity constraints depend on it).
@ -55,10 +59,16 @@ struct DTDScanner {
/// only a little); deeper references are ignored rather than trapping.
let maxDepth = 40
init(limits: PureXML.Parsing.Limits, resolver: PureXML.Parsing.EntityResolver, standalone: Bool = false) {
init(
limits: PureXML.Parsing.Limits,
resolver: PureXML.Parsing.EntityResolver,
standalone: Bool = false,
documentVersion: String? = nil,
) {
self.limits = limits
self.resolver = resolver
self.standalone = standalone
self.documentVersion = documentVersion ?? "1.0"
parameterBudget = limits.maxEntityExpansion
}

8
Sources/Parsing/DoctypeScannerEntities.swift
View File

@ -12,7 +12,7 @@ extension DTDScanner {
if let text = resolver.resolveEntity(name, id) {
// An external parsed entity may begin with a text declaration,
// which is not part of its replacement text (4.3.1).
doctype.entities[name] = strippingTextDeclaration(text)
doctype.entities[name] = try? strippingTextDeclaration(text)
}
}
}
@ -65,7 +65,11 @@ extension DTDScanner {
} else if isParameter {
if doctype.parameterEntities[name] == nil {
if let text = resolver.resolveExternalSubset(id) {
doctype.parameterEntities[name] = expandParameterReferences(strippingTextDeclaration(text))
if let stripped = try? strippingTextDeclaration(text) {
doctype.parameterEntities[name] = expandParameterReferences(stripped)
} else {
unresolvedParameterEntities.insert(name)
}
} else {
unresolvedParameterEntities.insert(name)
}

18
Sources/Parsing/DoctypeScannerStrict.swift
View File

@ -136,7 +136,14 @@ extension DTDScanner {
/// text (4.3.1): the declaration is not part of the replacement text. Text
/// without one passes through unchanged, including text whose `<?xml` is
/// not actually a text declaration.
func strippingTextDeclaration(_ text: String) -> String {
/// Errata E38: an external entity may not declare a version the document
/// does not at least match ("1.1" inside a "1.0" document is an error).
func textDeclarationVersionAllowed(_ declaration: PureXML.Parsing.XMLDeclaration) -> Bool {
guard let declared = declaration.version else { return true }
return declared <= documentVersion
}
func strippingTextDeclaration(_ text: String) throws -> String {
guard text.hasPrefix("<?xml") else { return text }
// Foundation-free search for the closing '?>'.
var close = text.index(text.startIndex, offsetBy: 5)
@ -147,10 +154,13 @@ extension DTDScanner {
close = text.index(after: close)
}
guard close < text.endIndex else { return text }
let declaration = String(text[text.index(text.startIndex, offsetBy: 5) ..< close])
guard declaration.first?.isWhitespace == true,
PureXML.Parsing.XMLDeclaration.parseTextDeclaration(declaration) != nil
let body = String(text[text.index(text.startIndex, offsetBy: 5) ..< close])
guard body.first?.isWhitespace == true,
let declaration = PureXML.Parsing.XMLDeclaration.parseTextDeclaration(body)
else { return text }
guard textDeclarationVersionAllowed(declaration) else {
throw ParseError.malformedDeclaration(.start)
}
return String(text[text.index(close, offsetBy: 2)...])
}

29
Sources/Parsing/EventReader.swift
View File

@ -116,7 +116,13 @@ public extension PureXML.Parsing {
return .processingInstruction(target: instruction.target, data: instruction.data)
}
if reader.matches("<!DOCTYPE") {
documentType = try DoctypeScanner.scan(&reader, limits: limits, resolver: resolver, standalone: xmlDeclaration?.standalone == true)
documentType = try DoctypeScanner.scan(
&reader,
limits: limits,
resolver: resolver,
standalone: xmlDeclaration?.standalone == true,
documentVersion: xmlDeclaration?.version,
)
continue
}
if reader.matches("</") || reader.matches("<![CDATA[") {
@ -192,27 +198,6 @@ public extension PureXML.Parsing {
return .startElement(name: resolved.name, attributes: resolved.attributes)
}
private mutating func scanEndTag() throws -> Event {
let mark = reader.mark
reader.consume("</")
let name = try scanName()
reader.skipSpace()
guard reader.consume(">") else {
throw ParseError.unterminatedTag(reader.mark)
}
guard let top = open.last else {
throw ParseError.unexpectedEndTag(name: name.description, mark)
}
// Tag matching is lexical (the qualified-name text must match); the
// open name additionally carries its resolved namespace URI.
guard top.description == name.description else {
throw ParseError.mismatchedEndTag(expected: top.description, found: name.description, mark)
}
open.removeLast()
namespaces.leaveElement()
return .endElement(name: top)
}
/// Returns the next text event, or nil after splicing: a reference to
/// an entity whose replacement (directly or transitively) contains
/// markup is validated, budgeted, and pushed back into the reader so

26
Sources/Parsing/EventReaderEndTag.swift Normal file
View File

@ -0,0 +1,26 @@
/// The end-tag scanner, split from the reader body to keep it under the
/// length caps. Behavior is part of EventReader proper.
extension PureXML.Parsing.EventReader {
typealias Event = PureXML.Parsing.Event
typealias ParseError = PureXML.Parsing.ParseError
mutating func scanEndTag() throws -> Event {
let mark = reader.mark
reader.consume("</")
let name = try scanName()
reader.skipSpace()
guard reader.consume(">") else {
throw ParseError.unterminatedTag(reader.mark)
}
guard let top = open.last else {
throw ParseError.unexpectedEndTag(name: name.description, mark)
}
// Tag matching is lexical (the qualified-name text must match); the
// open name additionally carries its resolved namespace URI.
guard top.description == name.description else {
throw ParseError.mismatchedEndTag(expected: top.description, found: name.description, mark)
}
open.removeLast()
namespaces.leaveElement()
return .endElement(name: top)
}
}

55
Tests/EncodingMismatchTests.swift Normal file
View File

@ -0,0 +1,55 @@
@testable import PureXML
import Testing
/// Encoding-declaration contradictions are fatal (#137, 4.3.3): a declared
/// 16/32-bit encoding over 8-bit bytes, a declaration outside the BOM's
/// family, and an external entity declaring a version above the document's
/// (errata E38).
@Suite("Encoding declaration mismatches")
struct EncodingMismatchTests {
@Test("UTF-16 declared over 8-bit bytes is rejected")
func test_wideOverNarrow() {
let bytes = Array("<?xml version=\"1.0\" encoding=\"UTF-16\"?>\n<root/>\n".utf8)
#expect(throws: PureXML.Parsing.ParseError.self) {
_ = try PureXML.Parsing.ByteDecoder.decode(bytes)
}
}
@Test("A declaration outside the BOM's family is rejected")
func test_bomContradiction() {
// UTF-8 BOM + iso-8859-1 declaration.
let utf8: [UInt8] = [0xEF, 0xBB, 0xBF] + Array("<?xml version='1.0' encoding='iso-8859-1'?><x/>".utf8)
#expect(throws: PureXML.Parsing.ParseError.self) {
_ = try PureXML.Parsing.ByteDecoder.decode(utf8)
}
// UTF-16BE BOM + iso-8859-1 declaration.
var utf16: [UInt8] = [0xFE, 0xFF]
for scalar in "<?xml version='1.0' encoding='iso-8859-1'?><x/>".unicodeScalars {
utf16.append(0)
utf16.append(UInt8(scalar.value))
}
#expect(throws: PureXML.Parsing.ParseError.self) {
_ = try PureXML.Parsing.ByteDecoder.decode(utf16)
}
// Matching declarations stay accepted.
let consistent: [UInt8] = [0xEF, 0xBB, 0xBF] + Array("<?xml version='1.0' encoding='UTF-8'?><x/>".utf8)
#expect((try? PureXML.Parsing.ByteDecoder.decode(consistent)) != nil)
}
@Test("An external entity declaring a higher version is rejected (E38)")
func test_entityVersionAboveDocument() {
let xml = "<!DOCTYPE foo [<!ELEMENT foo ANY><!ENTITY e SYSTEM \"e.ent\">]>\n<foo>&e;</foo>"
let resolver = PureXML.Parsing.EntityResolver(
resolveEntity: { _, _ in "<?xml version=\"1.1\" encoding=\"utf-8\"?>ok" },
resolveExternalSubset: { _ in nil },
)
#expect(throws: PureXML.Parsing.ParseError.self) {
_ = try PureXML.parse(xml, limits: .init(allowDoctype: true), resolver: resolver)
}
let sameVersion = PureXML.Parsing.EntityResolver(
resolveEntity: { _, _ in "<?xml version=\"1.0\" encoding=\"utf-8\"?>ok" },
resolveExternalSubset: { _ in nil },
)
#expect((try? PureXML.parse(xml, limits: .init(allowDoctype: true), resolver: sameVersion)) != nil)
}
}

9
Tests/W3CEduniSuiteTests.swift
View File

@ -26,13 +26,8 @@ struct W3CEduniSuiteTests {
("misc/ht-bh.xml", "misc"),
]
/// not-wf cases this implementation knowingly accepts: the encoding-
/// declaration mismatches the byte decoder does not yet reject (a
/// declared encoding contradicting the actual bytes or BOM), tracked
/// as #137.
private let knownNotWFAccepted: Set<String> = [
"errata-2e/E38.xml", "errata-2e/E61.xml", "misc/007.xml", "misc/008.xml",
]
/// not-wf cases this implementation knowingly accepts (none).
private let knownNotWFAccepted: Set<String> = []
/// valid cases this implementation knowingly fails, two classes: the
/// grapheme-cluster lexing limitation (a combining mark directly after an