feat(parsing): scalar-level lexing, xmlconf archive fully conformant
The reader buffers one Unicode scalar per element, splitting multi-scalar graphemes from any source, so combining marks cannot merge with delimiters and the Fifth Edition's scalar-defined name productions classify correctly; the replacement-text, content-model, and ATTLIST sub-scanners and the entity-literal validator follow, while grapheme clusters reassemble wherever scanned text accumulates. Clears the last eduni baseline, grows the asserted IBM pre-5e class to 300, and with the xmltest manifest's own EDITION exclusions for 140/141 empties the xmltest baseline: every xmlconf case applicable to a Fifth Edition, namespace-aware processor passes. Closes #135
This commit is contained in:
parent
98792a1715
commit
2f1b936e36
@ -9,6 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Added
|
||||
|
||||
- Scalar-level lexing (#135): the reader now buffers exactly one Unicode scalar per element (multi-scalar graphemes from any source are split, string input iterates scalars directly), so a combining mark directly after an ASCII delimiter can no longer merge with it into one Swift grapheme cluster and XML's scalar-defined name productions classify correctly: U+06D6 lexes as a Fifth Edition PI-target NameStartChar, ZWNJ/ZWJ attribute names parse, and U+309A markup from entity expansion forms a real element. The replacement-text, content-model, and ATTLIST sub-scanners and the entity-literal validator go scalar-level the same way, while grapheme clusters reassemble naturally wherever scanned text is appended to a String (asserted by test). This clears the last eduni baseline (now empty), grows the IBM pre-5e class from 279 to 300 exactly as the Fifth Edition prescribes, and, combined with the EDITION="1 2 3 4" exclusions the xmltest manifest itself carries for cases 140/141, empties the xmltest deviation baseline entirely: every xmlconf case applicable to a Fifth Edition, namespace-aware processor now passes.
|
||||
|
||||
- Strict internal-subset profile (#128): `Limits(strictInternalSubset: true)` holds the internal DTD subset to the letter of XML 1.0, rejecting conditional sections and parameter-entity references inside markup declarations (entity values, content models, ATTLIST bodies), both of which PureXML otherwise supports as features; DeclSep references between declarations and `%name;` inside quoted attribute defaults (AttValue, where `%` is literal) stay legal, and defaults are unchanged. The xmltest runner opts in, taking its baseline from 181/186 to 185/186 with one remaining documented deviation: case 141, which expects pre-Fifth-Edition name-character classes.
|
||||
- Per-entity base-URI tracking (#138): a relative system identifier declared inside an external entity now resolves against that entity's own URI (RFC 3986, reusing the C14N resolver), so nested external entities find their siblings; `ExternalID` gains an optional `base` and a `resolvedSystemID`, the DTD scanner threads the current entity's URI through external-subset and parameter-entity scans, and resolvers receive the already-merged identifier (raw identifiers unchanged when no base applies). Clears eduni errata-2e/E18; the eduni baseline now holds only the #135 grapheme-lexing class.
|
||||
- Encoding-declaration contradictions are fatal (#137, 4.3.3): a document whose XML declaration names a 16- or 32-bit encoding over BOM-less 8-bit bytes is rejected; a BOM followed by a declaration outside its family (a UTF-8 BOM with `encoding='iso-8859-1'`, a UTF-16 BOM with an 8-bit name) is rejected, with the declaration read through the BOM's own byte pattern; and an external entity whose text declaration names a higher XML version than the document's is refused (errata E38), in both the external-subset and external-entity paths. The eduni not-wf baseline is now completely empty.
|
||||
|
||||
@ -94,7 +94,7 @@ flowchart TB
|
||||
Japanese["#126 japanese section: complete, zero deviations"]:::done
|
||||
Eduni["#127 eduni: runner + exact baselines, four classified deviations"]:::done
|
||||
StrictSubset["#128 strict internal-subset profile: complete, xmltest 185/186"]:::done
|
||||
ScalarLex["#135 scalar-level lexing (grapheme clusters)"]:::todo
|
||||
ScalarLex["#135 scalar-level lexing: complete"]:::done
|
||||
NSConstraints["#136 namespace constraints: complete"]:::done
|
||||
EncodingMismatch["#137 encoding-declaration mismatches: complete"]:::done
|
||||
BaseURI["#138 per-entity base URIs: complete"]:::done
|
||||
@ -125,7 +125,7 @@ flowchart TB
|
||||
|
||||
PureXML is a working, dependency-free XML library today: parse, emit, validate,
|
||||
query, and stream documents on macOS, Linux, Windows, and WASM. The test suite
|
||||
currently runs **922 tests in 157 suites** (`swift test`).
|
||||
currently runs **925 tests in 158 suites** (`swift test`).
|
||||
|
||||
### Shipped (libxml2-aligned surface)
|
||||
|
||||
|
||||
@ -70,7 +70,8 @@ private struct AttListScanner {
|
||||
private var index = 0
|
||||
|
||||
init(_ text: String) {
|
||||
characters = Array(text)
|
||||
// Scalar-level: a combining mark must not merge with a delimiter.
|
||||
characters = text.unicodeScalars.map(Character.init)
|
||||
}
|
||||
|
||||
var isAtEnd: Bool {
|
||||
|
||||
@ -26,7 +26,8 @@ private struct ModelScanner {
|
||||
private var index = 0
|
||||
|
||||
init(_ text: String) {
|
||||
characters = Array(text)
|
||||
// Scalar-level: a combining mark must not merge with a delimiter.
|
||||
characters = text.unicodeScalars.map(Character.init)
|
||||
}
|
||||
|
||||
var isAtEnd: Bool {
|
||||
|
||||
@ -75,7 +75,8 @@ private struct ReplacementScanner {
|
||||
private var index = 0
|
||||
|
||||
init(_ text: String) {
|
||||
characters = Array(text)
|
||||
// Scalar-level: a combining mark must not merge with a delimiter.
|
||||
characters = text.unicodeScalars.map(Character.init)
|
||||
}
|
||||
|
||||
mutating func advance() -> Character? {
|
||||
|
||||
@ -3,6 +3,13 @@ extension PureXML.Parsing {
|
||||
/// one at a time, keeping only a small lookahead buffer, so it never holds
|
||||
/// the whole input in memory. Tracks line, column, and offset for
|
||||
/// diagnostics. Internal to the parser; not part of the API.
|
||||
///
|
||||
/// Lexing is scalar-level: every buffered `Character` holds exactly one
|
||||
/// Unicode scalar (multi-scalar graphemes from the source are split), so a
|
||||
/// combining mark directly after an ASCII delimiter cannot merge with it
|
||||
/// and XML's scalar-defined productions classify correctly. Grapheme
|
||||
/// clusters reassemble naturally wherever scanned text is appended to a
|
||||
/// `String`.
|
||||
struct Reader {
|
||||
private let pull: () -> Character?
|
||||
private var buffer: [Character] = []
|
||||
@ -19,12 +26,24 @@ extension PureXML.Parsing {
|
||||
private var pendingRaw: Character?
|
||||
|
||||
init(pulling pull: @escaping () -> Character?) {
|
||||
self.pull = pull
|
||||
// Split multi-scalar graphemes so the buffer is scalar-level.
|
||||
var queued: [Character] = []
|
||||
self.pull = {
|
||||
if !queued.isEmpty { return queued.removeFirst() }
|
||||
guard let next = pull() else { return nil }
|
||||
let scalars = next.unicodeScalars
|
||||
guard scalars.count > 1 else { return next }
|
||||
var parts = scalars.map(Character.init)
|
||||
let first = parts.removeFirst()
|
||||
queued = parts
|
||||
return first
|
||||
}
|
||||
}
|
||||
|
||||
init(_ string: String) {
|
||||
var iterator = string.makeIterator()
|
||||
self.init(pulling: { iterator.next() })
|
||||
// Iterate scalars directly: each yielded Character is one scalar.
|
||||
var iterator = string.unicodeScalars.makeIterator()
|
||||
self.init(pulling: { iterator.next().map(Character.init) })
|
||||
}
|
||||
|
||||
/// Ensures the lookahead buffer holds at least `count` characters (or the
|
||||
@ -123,7 +142,7 @@ extension PureXML.Parsing {
|
||||
/// (4.4.2 Included). Position tracking keeps reporting the source
|
||||
/// document's marks, the libxml2 model for entity boundaries.
|
||||
mutating func inject(_ text: String) {
|
||||
buffer.insert(contentsOf: text, at: 0)
|
||||
buffer.insert(contentsOf: text.unicodeScalars.map(Character.init), at: 0)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
55
Tests/ScalarLexingTests.swift
Normal file
55
Tests/ScalarLexingTests.swift
Normal file
@ -0,0 +1,55 @@
|
||||
@testable import PureXML
|
||||
import Testing
|
||||
|
||||
/// Scalar-level lexing (#135): a combining mark directly after an ASCII
|
||||
/// delimiter must not merge with it into one grapheme cluster, so Fifth
|
||||
/// Edition name characters lex correctly, while grapheme clusters still
|
||||
/// reassemble in scanned text content.
|
||||
@Suite("Scalar-level lexing")
|
||||
struct ScalarLexingTests {
|
||||
private func limits() -> PureXML.Parsing.Limits {
|
||||
PureXML.Parsing.Limits(allowDoctype: true)
|
||||
}
|
||||
|
||||
@Test("A combining-mark name character after a delimiter lexes correctly")
|
||||
func test_combiningMarkNames() throws {
|
||||
// U+06D6 clusters with the preceding '?' as a Swift grapheme; per 5e
|
||||
// it is a NameStartChar and a legal PI target.
|
||||
let instruction = "<!DOCTYPE doc [<!ELEMENT doc ANY><?\u{06D6} data ?>]>\n<doc/>"
|
||||
#expect((try? PureXML.parse(instruction, limits: limits())) != nil)
|
||||
// U+309A as an element name via declaration-expanded markup.
|
||||
let element = "<!DOCTYPE doc [<!ENTITY e \"<\u{309A}></\u{309A}>\">]>\n<doc>&e;</doc>"
|
||||
let node = try PureXML.parse(element, limits: limits())
|
||||
guard case let .document(children) = node, let doc = children.compactMap(\.element).first else {
|
||||
Issue.record("no root")
|
||||
return
|
||||
}
|
||||
#expect(doc.children.compactMap(\.element).first?.name.localName == "\u{309A}")
|
||||
}
|
||||
|
||||
@Test("ZWNJ and ZWJ are name characters in attribute names")
|
||||
func test_zeroWidthJoiners() throws {
|
||||
let xml = "<doc \u{200C}\u{200D}attr=\"v\"/>"
|
||||
let node = try PureXML.parse(xml)
|
||||
guard case let .document(children) = node, case let .element(doc)? = children.first else {
|
||||
Issue.record("no root")
|
||||
return
|
||||
}
|
||||
#expect(doc.attributes.first?.name.localName == "\u{200C}\u{200D}attr")
|
||||
}
|
||||
|
||||
@Test("Grapheme clusters reassemble in text content")
|
||||
func test_textReassembly() throws {
|
||||
// e + combining acute arrive as separate scalars and recombine.
|
||||
let xml = "<doc>e\u{0301}x</doc>"
|
||||
let node = try PureXML.parse(xml)
|
||||
guard case let .document(children) = node, case let .element(doc)? = children.first,
|
||||
case let .text(value)? = doc.children.first
|
||||
else {
|
||||
Issue.record("no text")
|
||||
return
|
||||
}
|
||||
#expect(value == "e\u{0301}x")
|
||||
#expect(Array(value).count == 2)
|
||||
}
|
||||
}
|
||||
@ -29,27 +29,12 @@ struct W3CEduniSuiteTests {
|
||||
/// not-wf cases this implementation knowingly accepts (none).
|
||||
private let knownNotWFAccepted: Set<String> = []
|
||||
|
||||
/// valid cases this implementation knowingly fails, one class: the
|
||||
/// grapheme-cluster lexing limitation (a combining mark directly after an
|
||||
/// ASCII delimiter merges into one Swift Character, so scalar-level 5e
|
||||
/// name characters are mis-lexed; #135).
|
||||
private let knownValidFailures: Set<String> = [
|
||||
"errata-4e/ibm04v01.xml", "errata-4e/ibm05v03.xml", "errata-4e/ibm07v01.xml",
|
||||
"errata-4e/ibm85n107.xml", "errata-4e/ibm85n114.xml", "errata-4e/ibm85n119.xml",
|
||||
"errata-4e/ibm85n121.xml", "errata-4e/ibm85n122.xml", "errata-4e/ibm85n136.xml",
|
||||
"errata-4e/ibm85n137.xml", "errata-4e/ibm85n45.xml", "errata-4e/ibm85n50.xml",
|
||||
"errata-4e/ibm85n51.xml", "errata-4e/ibm85n52.xml", "errata-4e/ibm85n53.xml",
|
||||
"errata-4e/ibm85n54.xml", "errata-4e/ibm85n63.xml", "errata-4e/ibm85n73.xml",
|
||||
"errata-4e/ibm85n74.xml", "errata-4e/ibm85n76.xml", "errata-4e/ibm85n89.xml",
|
||||
"errata-4e/ibm85n91.xml", "errata-4e/ibm86n04.xml", "errata-4e/ibm87n60.xml",
|
||||
]
|
||||
/// valid cases this implementation knowingly fails (none).
|
||||
private let knownValidFailures: Set<String> = []
|
||||
|
||||
/// invalid cases where this implementation knowingly reports no validity
|
||||
/// error (none); 140.xml is carried here because it cannot parse yet (the
|
||||
/// grapheme-cluster lexing class: its element name is a combining mark).
|
||||
private let knownInvalidAccepted: Set<String> = [
|
||||
"errata-4e/140.xml",
|
||||
]
|
||||
/// error (none).
|
||||
private let knownInvalidAccepted: Set<String> = []
|
||||
|
||||
private struct ManifestCase {
|
||||
let id: String
|
||||
|
||||
@ -29,8 +29,10 @@ struct W3CIBMSuiteTests {
|
||||
/// Ideographic, CombiningChar, Digit, Extender), deleted by XML 1.0 Fifth
|
||||
/// Edition, whose Name productions this package implements; the characters
|
||||
/// these cases reject became legal (the xmltest 141 class). The count is
|
||||
/// asserted exactly so a regression in either direction is caught.
|
||||
private let preFifthEditionCaseCount = 279
|
||||
/// asserted exactly so a regression in either direction is caught (it grew
|
||||
/// from 279 to 300 when scalar-level lexing let combining-mark names parse
|
||||
/// as the Fifth Edition says they should).
|
||||
private let preFifthEditionCaseCount = 300
|
||||
|
||||
private func isPreFifthEditionCharacterClass(_ uri: String) -> Bool {
|
||||
(85 ... 89).contains { uri.hasPrefix("not-wf/P\($0)/") }
|
||||
|
||||
@ -15,17 +15,15 @@ struct W3CXMLSuiteTests {
|
||||
ProcessInfo.processInfo.environment["XMLTS_ROOT"]
|
||||
}
|
||||
|
||||
/// The one case this implementation knowingly accepts although the suite
|
||||
/// marks it not-well-formed: 141 expects the 1998 suite's name-character
|
||||
/// classes where this package implements XML 1.0 Fifth Edition's (U+0E5C
|
||||
/// is a name character there). The runner opts into the strict
|
||||
/// internal-subset profile, so the conditional-section and PE-reference
|
||||
/// feature cases (063, 160-162) are rejected as the suite expects. The
|
||||
/// baseline is exact, so any regression in the 185 cases that DO pass is
|
||||
/// still caught.
|
||||
private let knownNotWFDeviations: Set<String> = [
|
||||
"141.xml",
|
||||
]
|
||||
/// Cases the suite's own manifest excludes from a Fifth Edition,
|
||||
/// namespace-aware processor: 140 and 141 are EDITION="1 2 3 4" (their
|
||||
/// rejected characters became legal name characters in 5e). With the
|
||||
/// strict internal-subset profile opted in, every applicable not-wf case
|
||||
/// is rejected: the deviation baseline is empty.
|
||||
private let editionExcluded: Set<String> = ["140.xml", "141.xml"]
|
||||
|
||||
/// not-wf cases this implementation knowingly accepts (none).
|
||||
private let knownNotWFDeviations: Set<String> = []
|
||||
|
||||
/// valid/sa/012.xml is flagged NAMESPACE='no' by the suite's own
|
||||
/// manifest: it declares an attribute named ':', a pre-namespace XML 1.0
|
||||
@ -71,7 +69,7 @@ struct W3CXMLSuiteTests {
|
||||
}
|
||||
let directory = root + "/not-wf/sa"
|
||||
var unexpectedlyAccepted: [String] = []
|
||||
for file in try files(in: directory) where !knownNotWFDeviations.contains(file) {
|
||||
for file in try files(in: directory) where !knownNotWFDeviations.contains(file) && !editionExcluded.contains(file) {
|
||||
if (try? parse(file: file, in: directory)) != nil {
|
||||
unexpectedlyAccepted.append(file)
|
||||
}
|
||||
|
||||
Loading…
Reference in New Issue
Block a user