feat(parsing): IBM section clean, entity-declared WFC/VC split, declaration recursion

The IBM section (1094 cases) runs clean to two documented classes:
279 pre-Fifth-Edition character-class cases (P85-89, count asserted
exactly) and 2 PE-in-internal-subset feature cases. Real gaps closed:
a document needs a root element (production 1); entity references in
literals must be Names or character references; WFC: No Recursion
checked at declaration over the general/parameter reference graph;
undeclared PE references split into WFC (standalone, or no external
declarations possible) versus deferred validity finding, also for
attribute-default references; refused external PEs are neither; and
groups split across PE replacements report VC: Proper Group/PE
Nesting. Entity machinery split to DoctypeScannerEntities for caps.

Closes #125
This commit is contained in:
Mihaela Mihaljevic 2026-06-10 07:26:50 +02:00
parent b27b015f78
commit c084d01d7c
12 changed files with 530 additions and 92 deletions

View File

@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- W3C xmlconf IBM section, clean to two documented classes (#121, #125): the deepest block, 1094 cases. Valid 220/220 (parse + strict DTD validation), invalid 64/64 reporting, not-wf 529/810 rejected with the 281 accepted cases being exactly two documented classes: 279 cases from productions 85-89, the 1998 character-class appendices deleted by XML 1.0 Fifth Edition (whose count is asserted exactly so drift in either direction is caught), and 2 parameter-entity-in-internal-subset feature cases (the xmltest 160-162 class, headed for #128's strict profile). Closing the section's real gaps: a document without a root element is rejected (production 1); an entity reference in an entity literal must be a lexical Name or character reference (production 68); WFC: No Recursion is enforced at declaration over the combined general/parameter reference graph (CDATA-shielded), so a recursive pair is rejected without being referenced; an undeclared parameter-entity reference is a well-formedness error when the document is standalone or has neither an external subset nor parameter entities, and a deferred validity finding otherwise (the production-68 WFC/VC split, also applied to entity references in attribute defaults); a declared external PE the resolver refused is neither; and a content-model group opening in one PE replacement and closing in another is reported (VC: Proper Group/PE Nesting).
- Standalone validity constraints (#134): a document declaring `standalone='yes'` is now checked against its external-subset dependencies (2.9). The DTD scanner records declaration provenance (which entities, element models, and attribute lists came from the internal subset); the validator reports an externally-declared attribute default that would be supplied, an attribute value an externally-declared non-CDATA type would normalize, and whitespace inside externally-declared element content. The parser enforces the matching well-formedness side: a standalone document referencing an entity declared outside the internal subset reports it as undeclared (WFC: Entity Declared). This empties the entire Sun baseline: the section is fully clean at not-wf 56/56, valid 28/28, invalid 74/74, zero deviations, closing #124.
- DTD validity constraints, second Sun burn-down (#121, #124): the validator now enforces VC Root Element Type (the root element must match the DOCTYPE name, recorded by the scanner), VC Unique Element Type Declaration (duplicate `<!ELEMENT>` reported, first stays in effect), VC No Duplicate Types in mixed content, VC One ID per Element Type, VC Notation Attributes (listed names must be declared) and VC Notation Declared (an NDATA entity's notation must exist), VC Attribute Default Legal across the tokenized types plus VC ID Attribute Default (an ID attribute must be #IMPLIED or #REQUIRED), lexical Name checks for ID/IDREF/IDREFS values, and CDATA sections as character data in element content even when empty. Strict mode additionally requires every attribute to be declared (VC Attribute Value Type; namespace declarations exempt). Declaration-level findings are computed once when the schema is built and reported at the document root. Sun invalid coverage moves from 43/74 to 62/74 reporting; the twelve remaining silent cases are one family, the standalone='yes' VCs, which need declaration-provenance tracking.
- Entity content splicing (#133): a general entity whose replacement text contains markup, directly or through entities it references, is now reparsed as content at the point of reference (4.4.2 Included), so `<p>text</p>` inside an entity becomes a `p` element in the tree and in the pull-event stream, in both strict and recovering modes, instead of being included as escaped character data. The replacement is validated (the reference-time balanced-content WFC), checked for reference cycles, and budgeted against the amplification cap before being spliced into the reader's stream. Character references in internal entity literals now expand when the declaration is parsed for general entities too (4.4.5, previously parameter entities only), so `&#60;foo>` stores real markup that splices, and the Appendix D double escape `&#38;#38;` stores `&#38;` and renders as a literal ampersand; the reference-time WFC accordingly validates remaining references instead of re-expanding them. Text-only entities keep the single-pass expansion path and its event coalescing. Closes the last two Sun valid deviations: the section is now 28/28 valid.

View File

@ -90,7 +90,7 @@ flowchart TB
Epic121["#121 xmlconf full archive"]:::epic
Oasis["#123 OASIS/NIST section: complete, zero deviations"]:::done
Sun["#124 Sun section: complete, zero deviations"]:::done
IBM["#125 IBM section: per-production validity depth"]:::todo
IBM["#125 IBM section: clean to two documented classes"]:::done
Japanese["#126 japanese section: encoding declarations"]:::todo
Eduni["#127 eduni: errata + namespaces"]:::todo
StrictSubset["#128 strict internal-subset profile: xmltest 185/186"]:::todo
@ -117,7 +117,7 @@ flowchart TB
PureXML is a working, dependency-free XML library today: parse, emit, validate,
query, and stream documents on macOS, Linux, Windows, and WASM. The test suite
currently runs **890 tests in 148 suites** (`swift test`).
currently runs **898 tests in 150 suites** (`swift test`).
### Shipped (libxml2-aligned surface)

View File

@ -14,12 +14,13 @@ extension PureXML.Parsing {
_ reader: inout Reader,
limits: Limits,
resolver: EntityResolver = .refusing,
standalone: Bool = false,
) throws -> DocumentType {
let mark = reader.mark
guard limits.allowDoctype else {
throw ParseError.unsupportedDoctype(mark)
}
var scanner = DTDScanner(limits: limits, resolver: resolver)
var scanner = DTDScanner(limits: limits, resolver: resolver, standalone: standalone)
return try scanner.scan(&reader, at: mark)
}
}
@ -37,21 +38,39 @@ struct DTDScanner {
let limits: PureXML.Parsing.Limits
let resolver: PureXML.Parsing.EntityResolver
/// Whether the document declared standalone='yes', which makes an
/// undeclared-entity reference a well-formedness error rather than a
/// validity finding (WFC vs VC: Entity Declared).
let standalone: Bool
var doctype = PureXML.Parsing.DocumentType()
/// True while scanning the external subset, so declarations record their
/// origin (the standalone validity constraints depend on it).
var inExternalContext = false
/// External parameter entities that were declared but not loaded (the
/// resolver refused them): a reference to one is not an undeclared-entity
/// error, the processor simply did not read it.
var unresolvedParameterEntities: Set<String> = []
var parameterBudget: Int
/// Bounds parameter-entity injection recursion (modularized DTDs nest, but
/// only a little); deeper references are ignored rather than trapping.
let maxDepth = 40
init(limits: PureXML.Parsing.Limits, resolver: PureXML.Parsing.EntityResolver) {
init(limits: PureXML.Parsing.Limits, resolver: PureXML.Parsing.EntityResolver, standalone: Bool = false) {
self.limits = limits
self.resolver = resolver
self.standalone = standalone
parameterBudget = limits.maxEntityExpansion
}
/// Entity Declared is a WFC (throw) when the document is standalone, or
/// has neither an external subset nor parameter entities; otherwise it is
/// a VC (deferred finding), since the unread external declarations might
/// have declared the entity (production 68).
var entityDeclaredIsWellFormedness: Bool {
if standalone { return true }
return doctype.externalSubset == nil && doctype.parameterEntities.isEmpty && unresolvedParameterEntities.isEmpty
}
mutating func scan(_ reader: inout Reader, at mark: Mark) throws -> PureXML.Parsing.DocumentType {
reader.consume("<!DOCTYPE")
reader.skipSpace()
@ -68,23 +87,10 @@ struct DTDScanner {
}
try loadExternalSubset(at: mark)
resolveExternalEntities()
try checkEntityRecursion(at: mark)
return doctype
}
/// Folds declared external general entities into the general-entity table by
/// asking the resolver for their replacement text. A refused (nil) entity
/// stays undeclared, so a reference to it errors and the default refusing
/// resolver keeps XXE closed.
private mutating func resolveExternalEntities() {
for (name, id) in doctype.externalEntities where doctype.entities[name] == nil {
if let text = resolver.resolveEntity(name, id) {
// An external parsed entity may begin with a text declaration,
// which is not part of its replacement text (4.3.1).
doctype.entities[name] = strippingTextDeclaration(text)
}
}
}
/// Loads the external subset through the resolver, if one is configured and
/// returns text. External declarations never override internal ones (the
/// internal subset is scanned first and wins). A grammar violation in the
@ -164,50 +170,6 @@ struct DTDScanner {
}
}
/// Handles a bare `%name;` between declarations by injecting the parameter
/// entity's replacement text and scanning its declarations. Bounded by depth
/// and the expansion budget.
private mutating func scanParameterReference(_ reader: inout Reader, depth: Int, at mark: Mark) throws {
let refMark = reader.mark
reader.consume("%")
let name = scanName(&reader)
// PEReference is '%' Name ';' exactly: no space after '%' (an empty
// name) and the semicolon must follow the name directly.
guard !name.isEmpty, reader.consume(";") else {
throw ParseError.invalidReference("%\(name)", refMark)
}
guard depth < maxDepth, let replacement = doctype.parameterEntities[name] else {
return
}
guard parameterBudget >= replacement.count else {
return
}
parameterBudget -= replacement.count
var sub = Reader(replacement)
try scanDeclarations(&sub, depth: depth + 1, terminatedByBracket: false, at: mark)
}
/// Files an external entity declaration by kind: an `NDATA` entity is unparsed
/// (recorded with its notation); an external general entity records its
/// identifier for the resolver; an external parameter entity is loaded through
/// the resolver so its replacement text is available for `%name;` expansion
/// (the default refusing resolver loads nothing, keeping XXE closed).
mutating func storeExternalEntity(name: String, id: ExternalID, isParameter: Bool, notation: String?) {
if let notation, !notation.isEmpty, !isParameter {
if doctype.unparsedEntities[name] == nil {
doctype.unparsedEntities[name] = PureXML.Parsing.UnparsedEntity(id: id, notation: notation)
}
} else if isParameter {
if doctype.parameterEntities[name] == nil, let text = resolver.resolveExternalSubset(id) {
doctype.parameterEntities[name] = expandParameterReferences(strippingTextDeclaration(text))
}
} else if doctype.externalEntities[name] == nil {
doctype.externalEntities[name] = id
}
}
/// `<!NOTATION S Name S (ExternalID | PublicID) S? '>'` (production 82):
/// the identifier is required and nothing may follow it but whitespace.
private mutating func scanNotationDeclaration(_ reader: inout Reader) throws {
let mark = reader.mark
reader.consume("<!NOTATION")
@ -258,7 +220,9 @@ struct DTDScanner {
throw PureXML.Parsing.ParseError.invalidContentModel(element: name, mark)
}
reader.skipSpace()
let model = expandParameterReferences(readUntilClose(&reader)).trimmingXMLWhitespace()
let raw = readUntilClose(&reader)
checkGroupNesting(raw, element: name)
let model = expandParameterReferences(raw).trimmingXMLWhitespace()
guard PureXML.Parsing.DTDContentModelGrammar.isValid(model) else {
throw PureXML.Parsing.ParseError.invalidContentModel(element: name, mark)
}
@ -308,28 +272,6 @@ struct DTDScanner {
}
extension DTDScanner {
/// Replaces `%name;` references with their (already-expanded) parameter-entity
/// values. Undefined references are left literal. A single forward pass is
/// enough because each value was expanded against the entities defined before
/// it, so no reference can reach a later definition.
func expandParameterReferences(_ raw: String) -> String {
guard raw.contains("%") else { return raw }
var result = ""
var index = raw.startIndex
while index < raw.endIndex {
let character = raw[index]
guard character == "%", let semicolon = raw[index...].firstIndex(of: ";") else {
result.append(character)
index = raw.index(after: index)
continue
}
let name = String(raw[raw.index(after: index) ..< semicolon])
result += doctype.parameterEntities[name] ?? "%\(name);"
index = raw.index(after: semicolon)
}
return result
}
private func parseExternalID(_ reader: inout Reader) -> ExternalID? {
if reader.consume("SYSTEM") {
reader.skipSpace()

View File

@ -54,4 +54,89 @@ extension DTDScanner {
}
return nil
}
/// VC: Proper Group/PE Nesting. A parameter entity used in a content model
/// must contain balanced parentheses, so a group cannot open in one
/// replacement text and close in another. Recorded as a validity finding.
mutating func checkGroupNesting(_ rawModel: String, element: String) {
guard rawModel.contains("%") else { return }
var index = rawModel.startIndex
while index < rawModel.endIndex, let percent = rawModel[index...].firstIndex(of: "%") {
guard let semicolon = rawModel[percent...].firstIndex(of: ";") else { return }
let name = String(rawModel[rawModel.index(after: percent) ..< semicolon])
if let replacement = doctype.parameterEntities[name] {
let opens = replacement.count(where: { $0 == "(" })
let closes = replacement.count(where: { $0 == ")" })
if opens != closes {
doctype.validityFindings.append(
"the content model of '\(element)' uses parameter entity '%\(name);' with improper group/PE nesting",
)
}
}
index = rawModel.index(after: semicolon)
}
}
/// WFC: No Recursion, checked at declaration: no parsed entity (general or
/// parameter) may reference itself directly or indirectly through the
/// stored replacement texts.
func checkEntityRecursion(at mark: Mark) throws {
var visiting: Set<String> = []
var cleared: Set<String> = []
for key in doctype.entities.keys.sorted() {
try walkEntityGraph("ge:" + key, visiting: &visiting, cleared: &cleared, at: mark)
}
for key in doctype.parameterEntities.keys.sorted() {
try walkEntityGraph("pe:" + key, visiting: &visiting, cleared: &cleared, at: mark)
}
}
private func walkEntityGraph(_ node: String, visiting: inout Set<String>, cleared: inout Set<String>, at mark: Mark) throws {
guard !cleared.contains(node) else { return }
guard visiting.insert(node).inserted else {
throw ParseError.recursiveEntity(name: String(node.dropFirst(3)), mark)
}
defer {
visiting.remove(node)
cleared.insert(node)
}
let name = String(node.dropFirst(3))
let value = node.hasPrefix("ge:") ? doctype.entities[name] : doctype.parameterEntities[name]
guard let value else { return }
for reference in entityReferences(in: value) {
try walkEntityGraph(reference, visiting: &visiting, cleared: &cleared, at: mark)
}
}
/// The `ge:`/`pe:`-tagged references in a replacement text, with CDATA
/// sections shielded and predefined entities skipped.
private func entityReferences(in text: String) -> [String] {
let predefined: Set = ["amp", "lt", "gt", "quot", "apos"]
var references: [String] = []
var index = text.startIndex
while index < text.endIndex {
if text[index...].hasPrefix("<![CDATA[") {
var cursor = text.index(index, offsetBy: 9)
while cursor < text.endIndex, !text[cursor...].hasPrefix("]]>") {
cursor = text.index(after: cursor)
}
index = cursor < text.endIndex ? text.index(cursor, offsetBy: 3) : text.endIndex
continue
}
let character = text[index]
guard character == "&" || character == "%" else {
index = text.index(after: index)
continue
}
guard let semicolon = text[index...].firstIndex(of: ";") else { return references }
let body = String(text[text.index(after: index) ..< semicolon])
if character == "&", !body.hasPrefix("#"), !predefined.contains(body) {
references.append("ge:" + body)
} else if character == "%", !body.isEmpty {
references.append("pe:" + body)
}
index = text.index(after: semicolon)
}
return references
}
}

View File

@ -0,0 +1,101 @@
/// The entity side of the DTD scan: storing declarations, resolving external
/// entities through the injected resolver, expanding parameter-entity
/// references, and the bare `%name;` declaration-stream injection. Split from
/// the scanner body to keep it under the length caps.
extension DTDScanner {
/// Folds declared external general entities into the general-entity table by
/// asking the resolver for their replacement text. A refused (nil) entity
/// stays undeclared, so a reference to it errors and the default refusing
/// resolver keeps XXE closed.
mutating func resolveExternalEntities() {
for (name, id) in doctype.externalEntities where doctype.entities[name] == nil {
if let text = resolver.resolveEntity(name, id) {
// An external parsed entity may begin with a text declaration,
// which is not part of its replacement text (4.3.1).
doctype.entities[name] = strippingTextDeclaration(text)
}
}
}
/// Handles a bare `%name;` between declarations by injecting the parameter
/// entity's replacement text and scanning its declarations. Bounded by depth
/// and the expansion budget.
mutating func scanParameterReference(_ reader: inout Reader, depth: Int, at mark: Mark) throws {
let refMark = reader.mark
reader.consume("%")
let name = scanName(&reader)
// PEReference is '%' Name ';' exactly: no space after '%' (an empty
// name) and the semicolon must follow the name directly.
guard !name.isEmpty, reader.consume(";") else {
throw ParseError.invalidReference("%\(name)", refMark)
}
guard let replacement = doctype.parameterEntities[name] else {
// A declared external PE the resolver refused is not undeclared,
// the processor just did not read it.
guard !unresolvedParameterEntities.contains(name) else { return }
// VC/WFC: Entity Declared (also covers a reference before the
// declaration: declarations bind in document order).
if entityDeclaredIsWellFormedness {
throw ParseError.undefinedEntity(name: name, refMark)
}
doctype.validityFindings.append("parameter entity '%\(name);' is referenced but not declared")
return
}
guard depth < maxDepth else {
return
}
guard parameterBudget >= replacement.count else {
return
}
parameterBudget -= replacement.count
var sub = Reader(replacement)
try scanDeclarations(&sub, depth: depth + 1, terminatedByBracket: false, at: mark)
}
/// Files an external entity declaration by kind: an `NDATA` entity is unparsed
/// (recorded with its notation); an external general entity records its
/// identifier for the resolver; an external parameter entity is loaded through
/// the resolver so its replacement text is available for `%name;` expansion
/// (the default refusing resolver loads nothing, keeping XXE closed).
mutating func storeExternalEntity(name: String, id: ExternalID, isParameter: Bool, notation: String?) {
if let notation, !notation.isEmpty, !isParameter {
if doctype.unparsedEntities[name] == nil {
doctype.unparsedEntities[name] = PureXML.Parsing.UnparsedEntity(id: id, notation: notation)
}
} else if isParameter {
if doctype.parameterEntities[name] == nil {
if let text = resolver.resolveExternalSubset(id) {
doctype.parameterEntities[name] = expandParameterReferences(strippingTextDeclaration(text))
} else {
unresolvedParameterEntities.insert(name)
}
}
} else if doctype.externalEntities[name] == nil {
doctype.externalEntities[name] = id
}
}
/// `<!NOTATION S Name S (ExternalID | PublicID) S? '>'` (production 82):
/// the identifier is required and nothing may follow it but whitespace.
/// Replaces `%name;` references with their (already-expanded) parameter-entity
/// values. Undefined references are left literal. A single forward pass is
/// enough because each value was expanded against the entities defined before
/// it, so no reference can reach a later definition.
func expandParameterReferences(_ raw: String) -> String {
guard raw.contains("%") else { return raw }
var result = ""
var index = raw.startIndex
while index < raw.endIndex {
let character = raw[index]
guard character == "%", let semicolon = raw[index...].firstIndex(of: ";") else {
result.append(character)
index = raw.index(after: index)
continue
}
let name = String(raw[raw.index(after: index) ..< semicolon])
result += doctype.parameterEntities[name] ?? "%\(name);"
index = raw.index(after: semicolon)
}
return result
}
}

View File

@ -93,7 +93,8 @@ extension DTDScanner {
}
if character == "&" || character == "%" {
var probe = index + 1
if character == "&", probe < characters.count, characters[probe] == "#" { probe += 1 }
let isCharacterReference = character == "&" && probe < characters.count && characters[probe] == "#"
if isCharacterReference { probe += 1 }
let bodyStart = probe
let stops: Set<Character> = [";", "&", "%"]
while probe < characters.count, !stops.contains(characters[probe]), !characters[probe].isWhitespace {
@ -102,6 +103,14 @@ extension DTDScanner {
guard probe > bodyStart, probe < characters.count, characters[probe] == ";" else {
throw ParseError.invalidReference(String(character), mark)
}
// An entity reference's body must be a Name: '&49;' is neither
// a character reference nor a legal entity name (production 68).
if !isCharacterReference {
let body = String(characters[bodyStart ..< probe])
guard PureXML.Parsing.XMLCharacter.isValidName(body) else {
throw ParseError.invalidReference(String(character) + body, mark)
}
}
index = probe
}
index += 1
@ -234,11 +243,11 @@ extension DTDScanner {
/// already declared and internal: an undeclared, forward-declared, external,
/// or unparsed entity in a default is not well-formed, and the referenced
/// chain must not recurse.
func validateDefaultValueReferences(_ value: String, at mark: Mark) throws {
mutating func validateDefaultValueReferences(_ value: String, at mark: Mark) throws {
try walkDefaultReferences(value, visiting: [], at: mark)
}
func walkDefaultReferences(_ value: String, visiting: Set<String>, at mark: Mark) throws {
mutating func walkDefaultReferences(_ value: String, visiting: Set<String>, at mark: Mark) throws {
let predefined: Set = ["amp", "lt", "gt", "quot", "apos"]
let characters = Array(value)
var index = 0
@ -269,7 +278,12 @@ extension DTDScanner {
throw ParseError.invalidAttributeListDeclaration(mark)
}
guard let replacement = doctype.entities[name] else {
throw ParseError.undefinedEntity(name: name, mark)
// VC/WFC: Entity Declared, same split as parameter references.
if entityDeclaredIsWellFormedness {
throw ParseError.undefinedEntity(name: name, mark)
}
doctype.validityFindings.append("entity '&\(name);' in an attribute default is referenced but not declared")
return
}
try walkDefaultReferences(replacement, visiting: visiting.union([name]), at: mark)
}

View File

@ -38,6 +38,12 @@ public extension PureXML.Parsing {
/// The `<!ATTLIST>` bodies declared in the internal subset only, keyed
/// by element, so externally-declared attributes are distinguishable.
public var internalAttributeLists: [String: String] = [:]
/// Validity (not well-formedness) findings discovered while scanning
/// the DTD: an undeclared entity referenced where the external subset
/// might have declared it (VC: Entity Declared) or a content-model
/// group split across parameter entities (VC: Proper Group/PE
/// Nesting). Reported by the validator at the document root.
public var validityFindings: [String] = []
public init(
entities: [String: String] = [:],

View File

@ -125,7 +125,7 @@ public extension PureXML.Parsing {
return .processingInstruction(target: instruction.target, data: instruction.data)
}
if reader.matches("<!DOCTYPE") {
documentType = try DoctypeScanner.scan(&reader, limits: limits, resolver: resolver)
documentType = try DoctypeScanner.scan(&reader, limits: limits, resolver: resolver, standalone: xmlDeclaration?.standalone == true)
continue
}
if reader.matches("</") || reader.matches("<![CDATA[") {

View File

@ -164,7 +164,9 @@ public extension PureXML.Parsing {
}
}
guard produced else {
// Production 1: a document is prolog, ELEMENT, Misc*; comments and
// PIs alone (or a DOCTYPE with no element) are not a document.
guard produced, roots.contains(where: { $0.element != nil }) else {
throw ParseError.emptyDocument
}
return ParsedDocument(node: .document(roots), documentType: reader.documentType, declaration: reader.xmlDeclaration)

View File

@ -43,7 +43,7 @@ public extension PureXML.Validation {
parsedAttributes[name] = AttributeListParser.parse(body)
}
attributes = parsedAttributes
declarationErrors = Self.declarationFindings(documentType, attributes: parsedAttributes, notations: notations)
declarationErrors = documentType.validityFindings + Self.declarationFindings(documentType, attributes: parsedAttributes, notations: notations)
self.standalone = standalone
externalElementModels = Set(documentType.elementModels.keys).subtracting(documentType.internalElementModels)
var external: [String: [AttributeDeclaration]] = [:]

View File

@ -0,0 +1,112 @@
@testable import PureXML
import Testing
/// The IBM-section burn-down behaviors: a document needs a root element
/// (production 1), entity references in literals must be lexical Names (P66),
/// WFC: No Recursion is checked at declaration (P69), an undeclared
/// parameter-entity reference is a WFC or a VC depending on standalone and
/// external declarations (production 68), and a content-model group split
/// across parameter entities is a validity finding (VC: Proper Group/PE
/// Nesting, P49).
@Suite("Entity declared and recursion constraints")
struct DTDEntityDeclaredTests {
private func limits() -> PureXML.Parsing.Limits {
PureXML.Parsing.Limits(allowDoctype: true)
}
@Test("A document without a root element is rejected")
func test_missingRoot() {
#expect(throws: PureXML.Parsing.ParseError.self) {
_ = try PureXML.parse("<?xml version=\"1.0\"?>\n<!DOCTYPE book [<!ELEMENT book ANY>]>\n<!-- element is missing -->", limits: limits())
}
#expect(throws: PureXML.Parsing.ParseError.self) {
_ = try PureXML.parse("<!-- only a comment -->")
}
}
@Test("An entity reference in a literal must be a Name or character reference")
func test_referenceNameInLiteral() {
let xml = "<!DOCTYPE r [<!ELEMENT r EMPTY><!ENTITY aaa \"wrong: &49;\">]>\n<r/>"
#expect(throws: PureXML.Parsing.ParseError.self) {
_ = try PureXML.parse(xml, limits: limits())
}
let legal = "<!DOCTYPE r [<!ELEMENT r EMPTY><!ENTITY aaa \"fine: &#49;\">]>\n<r/>"
#expect((try? PureXML.parse(legal, limits: limits())) != nil)
}
@Test("Recursive entity declarations are rejected without being referenced")
func test_declarationRecursion() {
let direct = """
<!DOCTYPE root [
<!ELEMENT root (#PCDATA)>
<!ENTITY % paaa "&bbb;">
<!ENTITY bbb "%paaa;">
]>
<root/>
"""
#expect(throws: PureXML.Parsing.ParseError.self) {
_ = try PureXML.parse(direct, limits: limits())
}
let indirect = """
<!DOCTYPE root [
<!ELEMENT root (#PCDATA)>
<!ENTITY % paaa "&bbb;">
<!ENTITY bbb "&ccc;">
<!ENTITY ccc "%paaa;">
]>
<root/>
"""
#expect(throws: PureXML.Parsing.ParseError.self) {
_ = try PureXML.parse(indirect, limits: limits())
}
}
@Test("An undeclared PE reference is a WFC when standalone, a VC otherwise")
func test_undeclaredParameterEntity() throws {
let standalone = """
<?xml version="1.0" standalone="yes"?>
<!DOCTYPE root [
<!ELEMENT root (#PCDATA)>
%paaa;
<!ENTITY % paaa "<!ATTLIST root att CDATA #IMPLIED>">
]>
<root/>
"""
#expect(throws: PureXML.Parsing.ParseError.self) {
_ = try PureXML.parse(standalone, limits: limits())
}
// With an external PE in play the same reference is a validity finding.
let loose = """
<!DOCTYPE root [
<!ELEMENT root (#PCDATA)>
<!ENTITY % outside SYSTEM "x.ent">
%outside;
%undeclared;
]>
<root/>
"""
let resolver = PureXML.Parsing.EntityResolver(
resolveEntity: { _, _ in nil },
resolveExternalSubset: { _ in "<!ATTLIST root att CDATA #IMPLIED>" },
)
let errors = try PureXML.validateAgainstInternalDTD(loose, resolver: resolver)
#expect(errors.contains { $0.reason.contains("'%undeclared;' is referenced but not declared") })
}
@Test("A group split across parameter entities is a validity finding")
func test_properGroupNesting() throws {
let dtd = """
<!ENTITY % choice1 "(a|b">
<!ENTITY % choice2 "|c)">
<!ELEMENT child1 %choice1;%choice2; >
<!ELEMENT a EMPTY><!ELEMENT b EMPTY><!ELEMENT c EMPTY>
"""
let xml = "<!DOCTYPE root SYSTEM \"x.dtd\" [<!ELEMENT root ANY>]>\n<root/>"
let resolver = PureXML.Parsing.EntityResolver(
resolveEntity: { _, _ in nil },
resolveExternalSubset: { _ in dtd },
)
let errors = try PureXML.validateAgainstInternalDTD(xml, resolver: resolver)
#expect(errors.contains { $0.reason.contains("improper group/PE nesting") })
}
}

View File

@ -0,0 +1,175 @@
import Foundation
@testable import PureXML
import Testing
/// Opt-in runner for the W3C XML Conformance Test Suite's IBM section, the
/// deepest block: per-production directories covering valid, invalid, and
/// not-wf cases with heavy DTD usage. Classification follows the Sun model:
/// `valid` must parse and DTD-validate cleanly (strict mode), `invalid` must
/// parse but report at least one validity error, `not-wf` must be rejected.
/// Point `XMLCONF_ROOT` at the extracted `xmlconf` directory to run (the
/// suite is never vendored; its license permits redistribution only as the
/// unmodified archive). The `xml-1.1` subdirectory is out of scope: PureXML
/// targets XML 1.0 Fifth Edition. Foundation is used for file access only.
@Suite("W3C ibm conformance (opt-in via XMLCONF_ROOT)")
struct W3CIBMSuiteTests {
private var root: String? {
ProcessInfo.processInfo.environment["XMLCONF_ROOT"]
}
/// not-wf cases this implementation knowingly accepts: both are
/// parameter-entity references inside internal-subset markup declarations,
/// which this package supports as a feature (the xmltest 160-162 class;
/// the strict profile of #128 will reject them).
private let knownNotWFAccepted: Set<String> = [
"not-wf/P29/ibm29n03.xml", "not-wf/P29/ibm29n04.xml",
]
/// Productions 85-89 are the 1998 character-class appendices (BaseChar,
/// Ideographic, CombiningChar, Digit, Extender), deleted by XML 1.0 Fifth
/// Edition, whose Name productions this package implements; the characters
/// these cases reject became legal (the xmltest 141 class). The count is
/// asserted exactly so a regression in either direction is caught.
private let preFifthEditionCaseCount = 279
private func isPreFifthEditionCharacterClass(_ uri: String) -> Bool {
(85 ... 89).contains { uri.hasPrefix("not-wf/P\($0)/") }
}
/// valid cases this implementation knowingly fails (parse or validation).
private let knownValidFailures: Set<String> = []
/// invalid cases where this implementation knowingly reports no validity
/// error.
private let knownInvalidAccepted: Set<String> = []
private struct ManifestCase {
let id: String
let uri: String
}
/// Reads one manifest (a fragment with multiple TESTCASES roots), wrapped
/// before parsing with PureXML itself.
private func manifest(_ file: String, in directory: String) throws -> [ManifestCase] {
let source = try String(contentsOfFile: directory + "/" + file, encoding: .utf8)
let document = try PureXML.parse("<TESTS>\(source)</TESTS>")
return try PureXML.xpath("//TEST", over: document).compactMap { selection in
guard let element = selection.element else { return nil }
func attribute(_ name: String) -> String? {
element.attributes.first { $0.name.description == name }?.value
}
guard let id = attribute("ID"), let uri = attribute("URI") else { return nil }
return ManifestCase(id: id, uri: uri)
}
}
private func limits() -> PureXML.Parsing.Limits {
PureXML.Parsing.Limits(allowDoctype: true)
}
/// Resolves relative SYSTEM identifiers against the case's own directory,
/// byte-decoding through PureXML's own decoder.
private func resolver(directory: String) -> PureXML.Parsing.EntityResolver {
@Sendable func load(_ id: PureXML.Parsing.ExternalID) -> String? {
guard let data = try? Data(contentsOf: URL(fileURLWithPath: directory + "/" + id.systemID)) else {
return nil
}
return try? PureXML.Parsing.ByteDecoder.decode([UInt8](data))
}
return PureXML.Parsing.EntityResolver(
resolveEntity: { _, id in load(id) },
resolveExternalSubset: { id in load(id) },
)
}
private func caseDirectory(_ section: String, uri: String) -> String {
guard let slash = uri.lastIndex(of: "/") else { return section }
return section + "/" + String(uri.prefix(upTo: slash))
}
private func source(of uri: String, in section: String) throws -> String {
let data = try Data(contentsOf: URL(fileURLWithPath: section + "/" + uri))
return try PureXML.Parsing.ByteDecoder.decode([UInt8](data))
}
@Test("Every ibm not-wf case is rejected by the strict parser")
func test_notWellFormedRejected() throws {
guard let root else {
return // Opt-in: suite not present.
}
let section = root + "/ibm"
var unexpectedlyAccepted: [String] = []
var fifthEditionAccepted = 0
for testCase in try manifest("ibm_oasis_not-wf.xml", in: section) where !knownNotWFAccepted.contains(testCase.uri) {
let directory = caseDirectory(section, uri: testCase.uri)
let accepted = (try? PureXML.parse(
source(of: testCase.uri, in: section),
limits: limits(),
resolver: resolver(directory: directory),
)) != nil
if accepted {
if isPreFifthEditionCharacterClass(testCase.uri) {
fifthEditionAccepted += 1
} else {
unexpectedlyAccepted.append(testCase.uri)
}
}
}
#expect(unexpectedlyAccepted.isEmpty, "accepted \(unexpectedlyAccepted.count) not-wf cases: \(unexpectedlyAccepted)")
#expect(fifthEditionAccepted == preFifthEditionCaseCount, "pre-5e class drifted: \(fifthEditionAccepted)")
}
@Test("Every ibm valid case parses and validates cleanly against its DTD")
func test_validCleans() throws {
guard let root else {
return // Opt-in: suite not present.
}
let section = root + "/ibm"
var failures: [String: String] = [:]
for testCase in try manifest("ibm_oasis_valid.xml", in: section) where !knownValidFailures.contains(testCase.uri) {
let directory = caseDirectory(section, uri: testCase.uri)
do {
let errors = try PureXML.validateAgainstInternalDTD(
source(of: testCase.uri, in: section),
limits: limits(),
strict: true,
resolver: resolver(directory: directory),
)
if let first = errors.first {
failures[testCase.uri] = String(describing: first)
}
} catch {
failures[testCase.uri] = String(describing: error)
}
}
#expect(failures.isEmpty, "\(failures.count) valid cases failed: \(failures)")
}
@Test("Every ibm invalid case parses but reports a validity error")
func test_invalidReports() throws {
guard let root else {
return // Opt-in: suite not present.
}
let section = root + "/ibm"
var silentlyClean: [String] = []
var parseFailures: [String: String] = [:]
for testCase in try manifest("ibm_oasis_invalid.xml", in: section) where !knownInvalidAccepted.contains(testCase.uri) {
let directory = caseDirectory(section, uri: testCase.uri)
do {
let errors = try PureXML.validateAgainstInternalDTD(
source(of: testCase.uri, in: section),
limits: limits(),
strict: true,
resolver: resolver(directory: directory),
)
if errors.isEmpty {
silentlyClean.append(testCase.uri)
}
} catch {
parseFailures[testCase.uri] = String(describing: error)
}
}
#expect(parseFailures.isEmpty, "\(parseFailures.count) invalid (well-formed) cases failed to parse: \(parseFailures)")
#expect(silentlyClean.isEmpty, "\(silentlyClean.count) invalid cases validated clean: \(silentlyClean)")
}
}