From 2f1b936e36ff32eba8172da110b951969c440934 Mon Sep 17 00:00:00 2001 From: Mihaela Mihaljevic Date: Wed, 10 Jun 2026 08:39:43 +0200 Subject: [PATCH] feat(parsing): scalar-level lexing, xmlconf archive fully conformant The reader buffers one Unicode scalar per element, splitting multi-scalar graphemes from any source, so combining marks cannot merge with delimiters and the Fifth Edition's scalar-defined name productions classify correctly; the replacement-text, content-model, and ATTLIST sub-scanners and the entity-literal validator follow, while grapheme clusters reassemble wherever scanned text accumulates. Clears the last eduni baseline, grows the asserted IBM pre-5e class to 300, and with the xmltest manifest's own EDITION exclusions for 140/141 empties the xmltest baseline: every xmlconf case applicable to a Fifth Edition, namespace-aware processor passes. Closes #135 --- CHANGELOG.md | 2 + README.md | 4 +- Sources/Parsing/DTDAttListGrammar.swift | 3 +- Sources/Parsing/DTDContentModelGrammar.swift | 3 +- .../Parsing/EntityReplacementGrammar.swift | 3 +- Sources/Parsing/Reader.swift | 27 +++++++-- Tests/ScalarLexingTests.swift | 55 +++++++++++++++++++ Tests/W3CEduniSuiteTests.swift | 23 ++------ Tests/W3CIBMSuiteTests.swift | 6 +- Tests/W3CXMLSuiteTests.swift | 22 ++++---- 10 files changed, 106 insertions(+), 42 deletions(-) create mode 100644 Tests/ScalarLexingTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index aa9d394..0c9db13 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Scalar-level lexing (#135): the reader now buffers exactly one Unicode scalar per element (multi-scalar graphemes from any source are split, string input iterates scalars directly), so a combining mark directly after an ASCII delimiter can no longer merge with it into one Swift grapheme cluster and XML's scalar-defined name productions classify correctly: U+06D6 lexes as a Fifth Edition PI-target NameStartChar, ZWNJ/ZWJ attribute names parse, and U+309A markup from entity expansion forms a real element. The replacement-text, content-model, and ATTLIST sub-scanners and the entity-literal validator go scalar-level the same way, while grapheme clusters reassemble naturally wherever scanned text is appended to a String (asserted by test). This clears the last eduni baseline (now empty), grows the IBM pre-5e class from 279 to 300 exactly as the Fifth Edition prescribes, and, combined with the EDITION="1 2 3 4" exclusions the xmltest manifest itself carries for cases 140/141, empties the xmltest deviation baseline entirely: every xmlconf case applicable to a Fifth Edition, namespace-aware processor now passes. + - Strict internal-subset profile (#128): `Limits(strictInternalSubset: true)` holds the internal DTD subset to the letter of XML 1.0, rejecting conditional sections and parameter-entity references inside markup declarations (entity values, content models, ATTLIST bodies), both of which PureXML otherwise supports as features; DeclSep references between declarations and `%name;` inside quoted attribute defaults (AttValue, where `%` is literal) stay legal, and defaults are unchanged. The xmltest runner opts in, taking its baseline from 181/186 to 185/186 with one remaining documented deviation: case 141, which expects pre-Fifth-Edition name-character classes. - Per-entity base-URI tracking (#138): a relative system identifier declared inside an external entity now resolves against that entity's own URI (RFC 3986, reusing the C14N resolver), so nested external entities find their siblings; `ExternalID` gains an optional `base` and a `resolvedSystemID`, the DTD scanner threads the current entity's URI through external-subset and parameter-entity scans, and resolvers receive the already-merged identifier (raw identifiers unchanged when no base applies). Clears eduni errata-2e/E18; the eduni baseline now holds only the #135 grapheme-lexing class. - Encoding-declaration contradictions are fatal (#137, 4.3.3): a document whose XML declaration names a 16- or 32-bit encoding over BOM-less 8-bit bytes is rejected; a BOM followed by a declaration outside its family (a UTF-8 BOM with `encoding='iso-8859-1'`, a UTF-16 BOM with an 8-bit name) is rejected, with the declaration read through the BOM's own byte pattern; and an external entity whose text declaration names a higher XML version than the document's is refused (errata E38), in both the external-subset and external-entity paths. The eduni not-wf baseline is now completely empty. diff --git a/README.md b/README.md index 3d7cbc7..8c670b6 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ flowchart TB Japanese["#126 japanese section: complete, zero deviations"]:::done Eduni["#127 eduni: runner + exact baselines, four classified deviations"]:::done StrictSubset["#128 strict internal-subset profile: complete, xmltest 185/186"]:::done - ScalarLex["#135 scalar-level lexing (grapheme clusters)"]:::todo + ScalarLex["#135 scalar-level lexing: complete"]:::done NSConstraints["#136 namespace constraints: complete"]:::done EncodingMismatch["#137 encoding-declaration mismatches: complete"]:::done BaseURI["#138 per-entity base URIs: complete"]:::done @@ -125,7 +125,7 @@ flowchart TB PureXML is a working, dependency-free XML library today: parse, emit, validate, query, and stream documents on macOS, Linux, Windows, and WASM. The test suite -currently runs **922 tests in 157 suites** (`swift test`). +currently runs **925 tests in 158 suites** (`swift test`). ### Shipped (libxml2-aligned surface) diff --git a/Sources/Parsing/DTDAttListGrammar.swift b/Sources/Parsing/DTDAttListGrammar.swift index 87f0cf3..6658750 100644 --- a/Sources/Parsing/DTDAttListGrammar.swift +++ b/Sources/Parsing/DTDAttListGrammar.swift @@ -70,7 +70,8 @@ private struct AttListScanner { private var index = 0 init(_ text: String) { - characters = Array(text) + // Scalar-level: a combining mark must not merge with a delimiter. + characters = text.unicodeScalars.map(Character.init) } var isAtEnd: Bool { diff --git a/Sources/Parsing/DTDContentModelGrammar.swift b/Sources/Parsing/DTDContentModelGrammar.swift index 85e7548..be22f9e 100644 --- a/Sources/Parsing/DTDContentModelGrammar.swift +++ b/Sources/Parsing/DTDContentModelGrammar.swift @@ -26,7 +26,8 @@ private struct ModelScanner { private var index = 0 init(_ text: String) { - characters = Array(text) + // Scalar-level: a combining mark must not merge with a delimiter. + characters = text.unicodeScalars.map(Character.init) } var isAtEnd: Bool { diff --git a/Sources/Parsing/EntityReplacementGrammar.swift b/Sources/Parsing/EntityReplacementGrammar.swift index 06a8f00..a19436c 100644 --- a/Sources/Parsing/EntityReplacementGrammar.swift +++ b/Sources/Parsing/EntityReplacementGrammar.swift @@ -75,7 +75,8 @@ private struct ReplacementScanner { private var index = 0 init(_ text: String) { - characters = Array(text) + // Scalar-level: a combining mark must not merge with a delimiter. + characters = text.unicodeScalars.map(Character.init) } mutating func advance() -> Character? { diff --git a/Sources/Parsing/Reader.swift b/Sources/Parsing/Reader.swift index 2dae389..40012e3 100644 --- a/Sources/Parsing/Reader.swift +++ b/Sources/Parsing/Reader.swift @@ -3,6 +3,13 @@ extension PureXML.Parsing { /// one at a time, keeping only a small lookahead buffer, so it never holds /// the whole input in memory. Tracks line, column, and offset for /// diagnostics. Internal to the parser; not part of the API. + /// + /// Lexing is scalar-level: every buffered `Character` holds exactly one + /// Unicode scalar (multi-scalar graphemes from the source are split), so a + /// combining mark directly after an ASCII delimiter cannot merge with it + /// and XML's scalar-defined productions classify correctly. Grapheme + /// clusters reassemble naturally wherever scanned text is appended to a + /// `String`. struct Reader { private let pull: () -> Character? private var buffer: [Character] = [] @@ -19,12 +26,24 @@ extension PureXML.Parsing { private var pendingRaw: Character? init(pulling pull: @escaping () -> Character?) { - self.pull = pull + // Split multi-scalar graphemes so the buffer is scalar-level. + var queued: [Character] = [] + self.pull = { + if !queued.isEmpty { return queued.removeFirst() } + guard let next = pull() else { return nil } + let scalars = next.unicodeScalars + guard scalars.count > 1 else { return next } + var parts = scalars.map(Character.init) + let first = parts.removeFirst() + queued = parts + return first + } } init(_ string: String) { - var iterator = string.makeIterator() - self.init(pulling: { iterator.next() }) + // Iterate scalars directly: each yielded Character is one scalar. + var iterator = string.unicodeScalars.makeIterator() + self.init(pulling: { iterator.next().map(Character.init) }) } /// Ensures the lookahead buffer holds at least `count` characters (or the @@ -123,7 +142,7 @@ extension PureXML.Parsing { /// (4.4.2 Included). Position tracking keeps reporting the source /// document's marks, the libxml2 model for entity boundaries. mutating func inject(_ text: String) { - buffer.insert(contentsOf: text, at: 0) + buffer.insert(contentsOf: text.unicodeScalars.map(Character.init), at: 0) } } } diff --git a/Tests/ScalarLexingTests.swift b/Tests/ScalarLexingTests.swift new file mode 100644 index 0000000..1701c22 --- /dev/null +++ b/Tests/ScalarLexingTests.swift @@ -0,0 +1,55 @@ +@testable import PureXML +import Testing + +/// Scalar-level lexing (#135): a combining mark directly after an ASCII +/// delimiter must not merge with it into one grapheme cluster, so Fifth +/// Edition name characters lex correctly, while grapheme clusters still +/// reassemble in scanned text content. +@Suite("Scalar-level lexing") +struct ScalarLexingTests { + private func limits() -> PureXML.Parsing.Limits { + PureXML.Parsing.Limits(allowDoctype: true) + } + + @Test("A combining-mark name character after a delimiter lexes correctly") + func test_combiningMarkNames() throws { + // U+06D6 clusters with the preceding '?' as a Swift grapheme; per 5e + // it is a NameStartChar and a legal PI target. + let instruction = "]>\n" + #expect((try? PureXML.parse(instruction, limits: limits())) != nil) + // U+309A as an element name via declaration-expanded markup. + let element = "\">]>\n&e;" + let node = try PureXML.parse(element, limits: limits()) + guard case let .document(children) = node, let doc = children.compactMap(\.element).first else { + Issue.record("no root") + return + } + #expect(doc.children.compactMap(\.element).first?.name.localName == "\u{309A}") + } + + @Test("ZWNJ and ZWJ are name characters in attribute names") + func test_zeroWidthJoiners() throws { + let xml = "" + let node = try PureXML.parse(xml) + guard case let .document(children) = node, case let .element(doc)? = children.first else { + Issue.record("no root") + return + } + #expect(doc.attributes.first?.name.localName == "\u{200C}\u{200D}attr") + } + + @Test("Grapheme clusters reassemble in text content") + func test_textReassembly() throws { + // e + combining acute arrive as separate scalars and recombine. + let xml = "e\u{0301}x" + let node = try PureXML.parse(xml) + guard case let .document(children) = node, case let .element(doc)? = children.first, + case let .text(value)? = doc.children.first + else { + Issue.record("no text") + return + } + #expect(value == "e\u{0301}x") + #expect(Array(value).count == 2) + } +} diff --git a/Tests/W3CEduniSuiteTests.swift b/Tests/W3CEduniSuiteTests.swift index 4dbc2d8..75fc853 100644 --- a/Tests/W3CEduniSuiteTests.swift +++ b/Tests/W3CEduniSuiteTests.swift @@ -29,27 +29,12 @@ struct W3CEduniSuiteTests { /// not-wf cases this implementation knowingly accepts (none). private let knownNotWFAccepted: Set = [] - /// valid cases this implementation knowingly fails, one class: the - /// grapheme-cluster lexing limitation (a combining mark directly after an - /// ASCII delimiter merges into one Swift Character, so scalar-level 5e - /// name characters are mis-lexed; #135). - private let knownValidFailures: Set = [ - "errata-4e/ibm04v01.xml", "errata-4e/ibm05v03.xml", "errata-4e/ibm07v01.xml", - "errata-4e/ibm85n107.xml", "errata-4e/ibm85n114.xml", "errata-4e/ibm85n119.xml", - "errata-4e/ibm85n121.xml", "errata-4e/ibm85n122.xml", "errata-4e/ibm85n136.xml", - "errata-4e/ibm85n137.xml", "errata-4e/ibm85n45.xml", "errata-4e/ibm85n50.xml", - "errata-4e/ibm85n51.xml", "errata-4e/ibm85n52.xml", "errata-4e/ibm85n53.xml", - "errata-4e/ibm85n54.xml", "errata-4e/ibm85n63.xml", "errata-4e/ibm85n73.xml", - "errata-4e/ibm85n74.xml", "errata-4e/ibm85n76.xml", "errata-4e/ibm85n89.xml", - "errata-4e/ibm85n91.xml", "errata-4e/ibm86n04.xml", "errata-4e/ibm87n60.xml", - ] + /// valid cases this implementation knowingly fails (none). + private let knownValidFailures: Set = [] /// invalid cases where this implementation knowingly reports no validity - /// error (none); 140.xml is carried here because it cannot parse yet (the - /// grapheme-cluster lexing class: its element name is a combining mark). - private let knownInvalidAccepted: Set = [ - "errata-4e/140.xml", - ] + /// error (none). + private let knownInvalidAccepted: Set = [] private struct ManifestCase { let id: String diff --git a/Tests/W3CIBMSuiteTests.swift b/Tests/W3CIBMSuiteTests.swift index b7ae533..2954990 100644 --- a/Tests/W3CIBMSuiteTests.swift +++ b/Tests/W3CIBMSuiteTests.swift @@ -29,8 +29,10 @@ struct W3CIBMSuiteTests { /// Ideographic, CombiningChar, Digit, Extender), deleted by XML 1.0 Fifth /// Edition, whose Name productions this package implements; the characters /// these cases reject became legal (the xmltest 141 class). The count is - /// asserted exactly so a regression in either direction is caught. - private let preFifthEditionCaseCount = 279 + /// asserted exactly so a regression in either direction is caught (it grew + /// from 279 to 300 when scalar-level lexing let combining-mark names parse + /// as the Fifth Edition says they should). + private let preFifthEditionCaseCount = 300 private func isPreFifthEditionCharacterClass(_ uri: String) -> Bool { (85 ... 89).contains { uri.hasPrefix("not-wf/P\($0)/") } diff --git a/Tests/W3CXMLSuiteTests.swift b/Tests/W3CXMLSuiteTests.swift index 5bee8f3..ec58bc3 100644 --- a/Tests/W3CXMLSuiteTests.swift +++ b/Tests/W3CXMLSuiteTests.swift @@ -15,17 +15,15 @@ struct W3CXMLSuiteTests { ProcessInfo.processInfo.environment["XMLTS_ROOT"] } - /// The one case this implementation knowingly accepts although the suite - /// marks it not-well-formed: 141 expects the 1998 suite's name-character - /// classes where this package implements XML 1.0 Fifth Edition's (U+0E5C - /// is a name character there). The runner opts into the strict - /// internal-subset profile, so the conditional-section and PE-reference - /// feature cases (063, 160-162) are rejected as the suite expects. The - /// baseline is exact, so any regression in the 185 cases that DO pass is - /// still caught. - private let knownNotWFDeviations: Set = [ - "141.xml", - ] + /// Cases the suite's own manifest excludes from a Fifth Edition, + /// namespace-aware processor: 140 and 141 are EDITION="1 2 3 4" (their + /// rejected characters became legal name characters in 5e). With the + /// strict internal-subset profile opted in, every applicable not-wf case + /// is rejected: the deviation baseline is empty. + private let editionExcluded: Set = ["140.xml", "141.xml"] + + /// not-wf cases this implementation knowingly accepts (none). + private let knownNotWFDeviations: Set = [] /// valid/sa/012.xml is flagged NAMESPACE='no' by the suite's own /// manifest: it declares an attribute named ':', a pre-namespace XML 1.0 @@ -71,7 +69,7 @@ struct W3CXMLSuiteTests { } let directory = root + "/not-wf/sa" var unexpectedlyAccepted: [String] = [] - for file in try files(in: directory) where !knownNotWFDeviations.contains(file) { + for file in try files(in: directory) where !knownNotWFDeviations.contains(file) && !editionExcluded.contains(file) { if (try? parse(file: file, in: directory)) != nil { unexpectedlyAccepted.append(file) }