feat(parsing): splice entity replacement markup into content

A general entity whose replacement text contains markup (directly or
transitively) is reparsed as content at the reference (4.4.2 Included):
the replacement is WFC-validated, cycle-checked, and budgeted, then
pushed back into the reader stream, so its elements become elements in
the tree and the pull-event stream rather than escaped character data.
Character references in internal entity literals now expand at
declaration time for general entities too (4.4.5), making char-ref
markup splice and the Appendix D double escape render as a literal
ampersand; the reference-time WFC validates remaining references
instead of re-expanding. Text-only entities keep the coalescing
single-pass path. Sun valid is 28/28.

Closes #133

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Entity content splicing (#133): a general entity whose replacement text contains markup, directly or through entities it references, is now reparsed as content at the point of reference (4.4.2 Included), so `<p>text</p>` inside an entity becomes a `p` element in the tree and in the pull-event stream, in both strict and recovering modes, instead of being included as escaped character data. The replacement is validated (the reference-time balanced-content WFC), checked for reference cycles, and budgeted against the amplification cap before being spliced into the reader's stream. Character references in internal entity literals now expand when the declaration is parsed for general entities too (4.4.5, previously parameter entities only), so `&#60;foo>` stores real markup that splices, and the Appendix D double escape `&#38;#38;` stores `&#38;` and renders as a literal ampersand; the reference-time WFC accordingly validates remaining references instead of re-expanding them. Text-only entities keep the single-pass expansion path and its event coalescing. Closes the last two Sun valid deviations: the section is now 28/28 valid.
 - Validity-layer conformance, first Sun burn-down (#121, #124): tokenized attribute values are normalized per 3.3.3 before validity checks (whitespace runs collapse, outer whitespace strips, CDATA untouched), so ` nonce ` satisfies an enumeration and an entity-expanded IDREF resolves; an external parsed entity's leading text declaration is stripped per 4.3.1 (and validated: version optional, encoding required, standalone forbidden) instead of being included in the replacement text; character references in parameter-entity literals are expanded at declaration time per 4.4.5, making the spec's Appendix D `&#37;zz;` example parse; and a PI target inside the DTD subset must be separated from its data (production 16). Sun baselines tighten to not-wf 55/56, valid 26/28, invalid 43 reporting; the two remaining valid failures share one cause, filed separately: general-entity replacement text containing markup is included as character data rather than reparsed into elements.
 - W3C xmlconf Sun section runner (#121, #124): the first validity-layer runner. Sun's manifests classify cases three ways and the runner checks all three: not-wf rejected by the strict parser (54/56), valid documents parse AND DTD-validate cleanly (19/28), invalid documents parse but must report at least one validity error (43/74). The exact deviation baselines are carried in the test with their causes: tokenized attribute-value normalization (3.3.3), external-entity text declarations (4.3.1), declaration-time character-reference expansion in PE literals (4.4.5), the standalone VCs, ID/IDREF corners, #REQUIRED enforcement, per-type attribute VCs, and the root/DOCTYPE name match, the work list this issue burns down.
 - W3C xmlconf OASIS/NIST section, fully clean (#121, #123): a second opt-in runner (XMLCONF_ROOT-gated, suite never vendored) drives the oasis section from its own manifest, parsed with PureXML itself; all 247 not-wf cases are rejected and all 100 well-formed (valid + invalid) cases parse, with zero deviations (the one excluded case is flagged NAMESPACE='no' by the manifest itself: pre-namespace colons in names, which a namespace-aware parser correctly refuses). Closing the section's last 24 gaps made the DTD subset grammar strict: unknown, lowercase, or space-separated markup-declaration keywords are rejected; a parameter-entity reference must be '%' Name ';' exactly; entity, parameter-entity, and notation names must start with a NameStartChar; NDATA requires whitespace and a strict name; a notation declaration requires its identifier and a clean tail; entity value literals admit no bare '%' (production 9); conditional-section keywords are case-sensitive INCLUDE/IGNORE and sections must balance to their ']]>'; junk between declarations is not well-formed; a grammar violation in the external subset now rejects the document (previously swallowed); the external subset may open with a text declaration (version optional, encoding required, standalone forbidden, production 77); and a processing-instruction target must be separated from its data by whitespace (production 16). The new strictness also tightened the xmltest baseline from 180/186 to 181/186: a CDATA section in the internal subset (case 107) is now correctly rejected by the conditional-keyword check, leaving five documented deviations.

README.md

@@ -117,7 +117,7 @@ flowchart TB
 
 PureXML is a working, dependency-free XML library today: parse, emit, validate,
 query, and stream documents on macOS, Linux, Windows, and WASM. The test suite
-currently runs **869 tests in 145 suites** (`swift test`).
+currently runs **876 tests in 146 suites** (`swift test`).
 
 ### Shipped (libxml2-aligned surface)
 

Sources/Parsing/DoctypeScannerStrict.swift

@@ -37,19 +37,15 @@ extension DTDScanner {
             throw ParseError.invalidEntityDeclaration(mark)
         }
         skip(&reader, until: ">")
-        // Character references in a parameter-entity literal are expanded when
-        // the declaration is parsed (4.4.5; the Appendix D `%zz;` case), so
-        // a stored PE value can itself carry a usable `%name;`. General-entity
-        // values keep their character references; the reference-time decoder
-        // expands them. The replacement-text well-formedness constraint binds
-        // at reference time (an unreferenced entity may carry a bad value), so
-        // the reparse happens in EntityDecoder when the entity is included.
-        var replacement = value
-        if isParameter {
-            guard let expanded = PureXML.Parsing.EntityReplacementGrammar.expandCharacterReferences(value) else {
-                throw ParseError.invalidEntityDeclaration(mark)
-            }
-            replacement = expanded
+        // Character references in an entity literal are expanded when the
+        // declaration is parsed (4.4.5): `%zz;` stores a usable `%zz;`,
+        // `<foo>` stores markup that content splicing reparses, and the
+        // Appendix D double escape `&` stores `&`, which the
+        // content reparse turns into a literal ampersand. The replacement-text
+        // well-formedness constraint still binds at reference time (an
+        // unreferenced entity may carry a bad value).
+        guard let replacement = PureXML.Parsing.EntityReplacementGrammar.expandCharacterReferences(value) else {
+            throw ParseError.invalidEntityDeclaration(mark)
         }
         storeEntity(name: name, value: expandParameterReferences(replacement), isParameter: isParameter)
     }

Sources/Parsing/EntityDecoder.swift

@@ -27,6 +27,122 @@ extension PureXML.Parsing {
     }
 }
 
+/// Content splicing (4.4.2 Included): a declared general entity whose
+/// replacement text, directly or through entities it references, contains
+/// markup must be reparsed as content, not included as character data. The
+/// event reader uses these helpers to find such references in a text run, to
+/// validate and budget the inclusion, and to splice the replacement back into
+/// its character stream.
+extension PureXML.Parsing.EntityDecoder {
+    private static let predefined: Set<String> = ["amp", "lt", "gt", "quot", "apos"]
+
+    /// A text run split at a markup-producing entity reference: the text
+    /// before the reference, the entity name, and the unconsumed text after.
+    struct MarkupSplit {
+        let prefix: String
+        let name: String
+        let remainder: String
+    }
+
+    /// Splits `raw` at the first reference to a declared entity whose
+    /// replacement produces markup. Nil when no such reference occurs and
+    /// plain decoding suffices.
+    static func splitAtMarkupEntity(_ raw: String, entities: [String: String]) -> MarkupSplit? {
+        guard !entities.isEmpty else { return nil }
+        var index = raw.startIndex
+        while index < raw.endIndex, let amp = raw[index...].firstIndex(of: "&") {
+            guard let semicolon = raw[amp...].firstIndex(of: ";") else { return nil }
+            let name = String(raw[raw.index(after: amp) ..< semicolon])
+            if !name.hasPrefix("#"), !predefined.contains(name), producesMarkup(name, entities: entities, visiting: []) {
+                return MarkupSplit(
+                    prefix: String(raw[..<amp]),
+                    name: name,
+                    remainder: String(raw[raw.index(after: semicolon)...]),
+                )
+            }
+            index = raw.index(after: semicolon)
+        }
+        return nil
+    }
+
+    /// Validates and budgets one content inclusion of `name`, returning its
+    /// replacement text for splicing: the replacement must reparse as balanced
+    /// content (the reference-time WFC), the reference chain must not recurse,
+    /// and the replacement's size counts against the amplification budget.
+    static func includeForContent(
+        _ name: String,
+        entities: [String: String],
+        budget: inout Int,
+        at mark: PureXML.Parsing.Mark,
+    ) throws -> String {
+        guard let replacement = entities[name] else {
+            throw PureXML.Parsing.ParseError.undefinedEntity(name: name, mark)
+        }
+        if PureXML.Parsing.EntityReplacementGrammar.violation(inValue: replacement) != nil {
+            throw PureXML.Parsing.ParseError.invalidReference("&\(name);", mark)
+        }
+        try checkRecursion(name, entities: entities, visiting: [name], at: mark)
+        guard budget >= replacement.count else {
+            throw PureXML.Parsing.ParseError.amplificationLimitExceeded(mark)
+        }
+        budget -= replacement.count
+        return replacement
+    }
+
+    /// Whether `name`'s replacement contains markup, directly or transitively.
+    private static func producesMarkup(_ name: String, entities: [String: String], visiting: Set<String>) -> Bool {
+        guard !visiting.contains(name), let replacement = entities[name] else { return false }
+        if replacement.contains("<") { return true }
+        return declaredReferences(in: replacement).contains {
+            producesMarkup($0, entities: entities, visiting: visiting.union([name]))
+        }
+    }
+
+    private static func checkRecursion(
+        _ name: String,
+        entities: [String: String],
+        visiting: Set<String>,
+        at mark: PureXML.Parsing.Mark,
+    ) throws {
+        guard let replacement = entities[name] else { return }
+        for reference in declaredReferences(in: replacement) {
+            guard !visiting.contains(reference) else {
+                throw PureXML.Parsing.ParseError.recursiveEntity(name: reference, mark)
+            }
+            try checkRecursion(reference, entities: entities, visiting: visiting.union([reference]), at: mark)
+        }
+    }
+
+    /// The declared-entity names referenced in `text`: `&name;` forms that are
+    /// neither predefined nor character references. References inside CDATA
+    /// sections are shielded and not counted.
+    private static func declaredReferences(in text: String) -> [String] {
+        var names: [String] = []
+        var index = text.startIndex
+        while index < text.endIndex {
+            if text[index...].hasPrefix("<![CDATA[") {
+                var cursor = text.index(index, offsetBy: 9)
+                while cursor < text.endIndex, !text[cursor...].hasPrefix("]]>") {
+                    cursor = text.index(after: cursor)
+                }
+                index = cursor < text.endIndex ? text.index(cursor, offsetBy: 3) : text.endIndex
+                continue
+            }
+            guard text[index] == "&" else {
+                index = text.index(after: index)
+                continue
+            }
+            guard let semicolon = text[index...].firstIndex(of: ";") else { return names }
+            let name = String(text[text.index(after: index) ..< semicolon])
+            if !name.hasPrefix("#"), !predefined.contains(name) {
+                names.append(name)
+            }
+            index = text.index(after: semicolon)
+        }
+        return names
+    }
+}
+
 /// Carries the mutable state of one decode pass (output and remaining budget) so
 /// the recursive expansion stays a few small methods. File-scope and private.
 ///

Sources/Parsing/EntityReplacementGrammar.swift

@@ -8,14 +8,13 @@ extension PureXML.Parsing {
     /// incomplete reference), or `<` followed by a non-name character is
     /// rejected at declaration.
     enum EntityReplacementGrammar {
-        /// Expands character references the way entity declaration does, then
-        /// checks the replacement parses as balanced content. Returns nil when
-        /// valid, else a short reason.
+        /// Checks stored replacement text parses as balanced content. Character
+        /// references were already expanded when the declaration was parsed
+        /// (4.4.5), so references remaining in the text (the Appendix D double
+        /// escape) are validated as references, not expanded again. Returns nil
+        /// when valid, else a short reason.
         static func violation(inValue value: String) -> String? {
-            guard let replacement = expandCharacterReferences(value) else {
-                return "incomplete character reference"
-            }
-            return contentViolation(replacement)
+            contentViolation(value)
         }
 
         /// One declaration-time pass over the entity value: `&#...;` becomes its

Sources/Parsing/EventReader.swift

@@ -135,26 +135,31 @@ public extension PureXML.Parsing {
         }
 
         private mutating func nextInContent() throws -> Event? {
-            guard let lead = reader.peek() else {
-                throw ParseError.unexpectedEndOfInput(reader.mark)
-            }
-            eventStart = reader.mark
-            if reader.matches("</") { return try scanEndTag() }
-            if reader.matches("<!--") { return try scanComment() }
-            if reader.matches("<![CDATA[") { return try scanCDATA() }
-            if reader.matches("<?") {
-                let mark = reader.mark
-                let instruction = try scanProcessingInstruction()
-                if instruction.target.lowercased() == "xml" {
-                    throw ParseError.reservedProcessingInstructionTarget(mark)
+            // Loops because a text scan can end by splicing an entity's
+            // replacement into the reader with no text before it: the next
+            // construct is then read from the spliced replacement.
+            while true {
+                guard let lead = reader.peek() else {
+                    throw ParseError.unexpectedEndOfInput(reader.mark)
                 }
-                return .processingInstruction(target: instruction.target, data: instruction.data)
+                eventStart = reader.mark
+                if reader.matches("</") { return try scanEndTag() }
+                if reader.matches("<!--") { return try scanComment() }
+                if reader.matches("<![CDATA[") { return try scanCDATA() }
+                if reader.matches("<?") {
+                    let mark = reader.mark
+                    let instruction = try scanProcessingInstruction()
+                    if instruction.target.lowercased() == "xml" {
+                        throw ParseError.reservedProcessingInstructionTarget(mark)
+                    }
+                    return .processingInstruction(target: instruction.target, data: instruction.data)
+                }
+                if reader.matches("<!DOCTYPE") {
+                    throw ParseError.unsupportedDoctype(reader.mark)
+                }
+                if lead == "<" { return try scanStartTag() }
+                if let text = try scanText() { return text }
             }
-            if reader.matches("<!DOCTYPE") {
-                throw ParseError.unsupportedDoctype(reader.mark)
-            }
-            if lead == "<" { return try scanStartTag() }
-            return try scanText()
         }
 
         private mutating func scanStartTag() throws -> Event {
@@ -204,7 +209,12 @@ public extension PureXML.Parsing {
             return .endElement(name: top)
         }
 
-        private mutating func scanText() throws -> Event {
+        /// Returns the next text event, or nil after splicing: a reference to
+        /// an entity whose replacement (directly or transitively) contains
+        /// markup is validated, budgeted, and pushed back into the reader so
+        /// it is reparsed as content (4.4.2 Included), elements in replacement
+        /// text become elements in the stream, not character data.
+        private mutating func scanText() throws -> Event? {
             let mark = reader.mark
             var raw = ""
             var length = 0
@@ -220,6 +230,18 @@ public extension PureXML.Parsing {
                 raw.append(character)
                 reader.advance()
             }
+            let split = raw.contains("&") ? EntityDecoder.splitAtMarkupEntity(raw, entities: documentType.entities) : nil
+            if let split {
+                let replacement = try EntityDecoder.includeForContent(
+                    split.name,
+                    entities: documentType.entities,
+                    budget: &entityBudget,
+                    at: mark,
+                )
+                reader.inject(replacement + split.remainder)
+                let prefix = try EntityDecoder.decode(split.prefix, entities: documentType.entities, budget: &entityBudget, at: mark)
+                return prefix.isEmpty ? nil : .characters(prefix)
+            }
             return try .characters(EntityDecoder.decode(raw, entities: documentType.entities, budget: &entityBudget, at: mark))
         }
 

Sources/Parsing/Reader.swift

@@ -117,6 +117,14 @@ extension PureXML.Parsing {
                 advance()
             }
         }
+
+        /// Prepends `text` to the unread stream: used to splice a general
+        /// entity's replacement text into content so it is reparsed as markup
+        /// (4.4.2 Included). Position tracking keeps reporting the source
+        /// document's marks, the libxml2 model for entity boundaries.
+        mutating func inject(_ text: String) {
+            buffer.insert(contentsOf: text, at: 0)
+        }
     }
 }
 

Tests/EntitySplicingTests.swift

@@ -0,0 +1,130 @@
+@testable import PureXML
+import Testing
+
+/// Content splicing (4.4.2 Included, #133): a general entity whose replacement
+/// text contains markup is reparsed as content, so elements inside an entity
+/// become elements in the tree and in the pull-event stream, with character
+/// references expanded at declaration time (4.4.5).
+@Suite("Entity replacement splicing")
+struct EntitySplicingTests {
+    private func limits() -> PureXML.Parsing.Limits {
+        PureXML.Parsing.Limits(allowDoctype: true)
+    }
+
+    private let document = """
+    <!DOCTYPE root [
+    <!ELEMENT root (p)>
+    <!ELEMENT p (#PCDATA)>
+    <!ENTITY example "<p>text</p>">
+    ]>
+    <root>&example;</root>
+    """
+
+    @Test("An entity's markup becomes elements in the tree")
+    func test_treeShape() throws {
+        let node = try PureXML.parse(document, limits: limits())
+        guard case let .document(children) = node, let root = children.compactMap(\.element).first else {
+            Issue.record("no root")
+            return
+        }
+        #expect(root.children.count == 1)
+        guard case let .element(paragraph)? = root.children.first else {
+            Issue.record("expected a p element, got \(String(describing: root.children.first))")
+            return
+        }
+        #expect(paragraph.name.localName == "p")
+        #expect(paragraph.children.count == 1)
+        if case let .text(value)? = paragraph.children.first {
+            #expect(value == "text")
+        } else {
+            Issue.record("expected text inside p")
+        }
+    }
+
+    @Test("The pull-event stream sees the spliced elements")
+    func test_pullEvents() throws {
+        var reader = PureXML.Parsing.EventReader(document, limits: limits())
+        var names: [String] = []
+        while let event = try reader.next() {
+            switch event {
+            case let .startElement(name, _): names.append("+\(name.localName)")
+            case let .endElement(name): names.append("-\(name.localName)")
+            case let .characters(text) where !text.isEmpty: names.append("t:\(text)")
+            default: break
+            }
+        }
+        #expect(names == ["+root", "+p", "t:text", "-p", "-root"])
+    }
+
+    @Test("The spliced document validates against its DTD content model")
+    func test_dtdValidates() throws {
+        let errors = try PureXML.validateAgainstInternalDTD(document, limits: limits())
+        #expect(errors.isEmpty, "\(errors)")
+    }
+
+    @Test("Character references expand at declaration, so markup from them splices")
+    func test_characterReferenceMarkup() throws {
+        // xmltest valid/sa/024 and 087 shapes.
+        let viaCharRef = """
+        <!DOCTYPE doc [
+        <!ELEMENT doc (foo)>
+        <!ELEMENT foo (#PCDATA)>
+        <!ENTITY e "&#60;foo></foo>">
+        ]>
+        <doc>&e;</doc>
+        """
+        let node = try PureXML.parse(viaCharRef, limits: limits())
+        guard case let .document(children) = node, let doc = children.compactMap(\.element).first else {
+            Issue.record("no root")
+            return
+        }
+        #expect(doc.children.compactMap(\.element).first?.name.localName == "foo")
+    }
+
+    @Test("The Appendix D double escape yields a literal ampersand in content")
+    func test_doubleEscape() throws {
+        let xml = """
+        <!DOCTYPE doc [
+        <!ELEMENT doc (#PCDATA)>
+        <!ENTITY e "a &#38;#38; b">
+        ]>
+        <doc>&e;</doc>
+        """
+        let node = try PureXML.parse(xml, limits: limits())
+        guard case let .document(children) = node, let doc = children.compactMap(\.element).first,
+              case let .text(value)? = doc.children.first
+        else {
+            Issue.record("no text")
+            return
+        }
+        #expect(value == "a & b")
+    }
+
+    @Test("A recursive markup entity is rejected, not looped")
+    func test_recursionRejected() {
+        let xml = """
+        <!DOCTYPE doc [
+        <!ELEMENT doc ANY>
+        <!ENTITY a "<p>&a;</p>">
+        ]>
+        <doc>&a;</doc>
+        """
+        #expect(throws: PureXML.Parsing.ParseError.self) {
+            _ = try PureXML.parse(xml, limits: limits())
+        }
+    }
+
+    @Test("Tags spanning the entity boundary stay rejected")
+    func test_boundarySpanRejected() {
+        let xml = """
+        <!DOCTYPE doc [
+        <!ELEMENT doc ANY>
+        <!ENTITY e "</doc><doc>">
+        ]>
+        <doc>&e;</doc>
+        """
+        #expect(throws: PureXML.Parsing.ParseError.self) {
+            _ = try PureXML.parse(xml, limits: limits())
+        }
+    }
+}

Tests/W3CSunSuiteTests.swift

@@ -25,12 +25,8 @@ struct W3CSunSuiteTests {
         "not-wf/not-sa03.xml",
     ]
 
-    /// valid cases this implementation knowingly fails: both need general-
-    /// entity replacement text containing markup to be reparsed into elements
-    /// rather than included as character data (tracked as its own issue).
-    private let knownValidFailures: Set<String> = [
-        "valid/pe03.xml", "valid/ext02.xml",
-    ]
+    /// valid cases this implementation knowingly fails.
+    private let knownValidFailures: Set<String> = []
 
     /// invalid cases where this implementation knowingly reports no validity
     /// error yet, the validity-layer frontier this issue works down: the