diff --git a/CHANGELOG.md b/CHANGELOG.md index 81a3df0..8af21d9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- DTD validity constraints, second Sun burn-down (#121, #124): the validator now enforces VC Root Element Type (the root element must match the DOCTYPE name, recorded by the scanner), VC Unique Element Type Declaration (duplicate `` reported, first stays in effect), VC No Duplicate Types in mixed content, VC One ID per Element Type, VC Notation Attributes (listed names must be declared) and VC Notation Declared (an NDATA entity's notation must exist), VC Attribute Default Legal across the tokenized types plus VC ID Attribute Default (an ID attribute must be #IMPLIED or #REQUIRED), lexical Name checks for ID/IDREF/IDREFS values, and CDATA sections as character data in element content even when empty. Strict mode additionally requires every attribute to be declared (VC Attribute Value Type; namespace declarations exempt). Declaration-level findings are computed once when the schema is built and reported at the document root. Sun invalid coverage moves from 43/74 to 62/74 reporting; the twelve remaining silent cases are one family, the standalone='yes' VCs, which need declaration-provenance tracking. - Entity content splicing (#133): a general entity whose replacement text contains markup, directly or through entities it references, is now reparsed as content at the point of reference (4.4.2 Included), so `

text

` inside an entity becomes a `p` element in the tree and in the pull-event stream, in both strict and recovering modes, instead of being included as escaped character data. The replacement is validated (the reference-time balanced-content WFC), checked for reference cycles, and budgeted against the amplification cap before being spliced into the reader's stream. Character references in internal entity literals now expand when the declaration is parsed for general entities too (4.4.5, previously parameter entities only), so `<foo>` stores real markup that splices, and the Appendix D double escape `&#38;` stores `&` and renders as a literal ampersand; the reference-time WFC accordingly validates remaining references instead of re-expanding them. Text-only entities keep the single-pass expansion path and its event coalescing. Closes the last two Sun valid deviations: the section is now 28/28 valid. - Validity-layer conformance, first Sun burn-down (#121, #124): tokenized attribute values are normalized per 3.3.3 before validity checks (whitespace runs collapse, outer whitespace strips, CDATA untouched), so ` nonce ` satisfies an enumeration and an entity-expanded IDREF resolves; an external parsed entity's leading text declaration is stripped per 4.3.1 (and validated: version optional, encoding required, standalone forbidden) instead of being included in the replacement text; character references in parameter-entity literals are expanded at declaration time per 4.4.5, making the spec's Appendix D `%zz;` example parse; and a PI target inside the DTD subset must be separated from its data (production 16). Sun baselines tighten to not-wf 55/56, valid 26/28, invalid 43 reporting; the two remaining valid failures share one cause, filed separately: general-entity replacement text containing markup is included as character data rather than reparsed into elements. - W3C xmlconf Sun section runner (#121, #124): the first validity-layer runner. Sun's manifests classify cases three ways and the runner checks all three: not-wf rejected by the strict parser (54/56), valid documents parse AND DTD-validate cleanly (19/28), invalid documents parse but must report at least one validity error (43/74). The exact deviation baselines are carried in the test with their causes: tokenized attribute-value normalization (3.3.3), external-entity text declarations (4.3.1), declaration-time character-reference expansion in PE literals (4.4.5), the standalone VCs, ID/IDREF corners, #REQUIRED enforcement, per-type attribute VCs, and the root/DOCTYPE name match, the work list this issue burns down. diff --git a/README.md b/README.md index ad1a673..9297997 100644 --- a/README.md +++ b/README.md @@ -117,7 +117,7 @@ flowchart TB PureXML is a working, dependency-free XML library today: parse, emit, validate, query, and stream documents on macOS, Linux, Windows, and WASM. The test suite -currently runs **876 tests in 146 suites** (`swift test`). +currently runs **885 tests in 147 suites** (`swift test`). ### Shipped (libxml2-aligned surface) diff --git a/Sources/Parsing/DoctypeScanner.swift b/Sources/Parsing/DoctypeScanner.swift index 049259e..9717103 100644 --- a/Sources/Parsing/DoctypeScanner.swift +++ b/Sources/Parsing/DoctypeScanner.swift @@ -52,7 +52,7 @@ struct DTDScanner { mutating func scan(_ reader: inout Reader, at mark: Mark) throws -> PureXML.Parsing.DocumentType { reader.consume("`) keyed by /// name. Their content is never parsed; the notation names how to handle it. public var unparsedEntities: [String: UnparsedEntity] + /// The name in ``: the root element's required type + /// (VC: Root Element Type). + public var name: String? + /// Element types declared by more than one `` (VC: Unique + /// Element Type Declaration); the first declaration stays in effect. + public var duplicateElements: Set = [] public init( entities: [String: String] = [:], diff --git a/Sources/Validation/DTDDeclarationValidity.swift b/Sources/Validation/DTDDeclarationValidity.swift new file mode 100644 index 0000000..0492a84 --- /dev/null +++ b/Sources/Validation/DTDDeclarationValidity.swift @@ -0,0 +1,114 @@ +extension PureXML.Validation.DTDSchema { + /// The declaration-level validity constraints: violations that exist in the + /// DTD itself, independent of any instance content. Computed once when the + /// schema is built and reported at the document root. + static func declarationFindings( + _ documentType: PureXML.Parsing.DocumentType, + attributes: [String: [PureXML.Validation.AttributeDeclaration]], + notations: Set, + ) -> [String] { + var findings: [String] = [] + // VC: Unique Element Type Declaration. + for name in documentType.duplicateElements.sorted() { + findings.append("element type '\(name)' is declared more than once") + } + // VC: Notation Declared, an unparsed entity's notation must exist. + for (entity, unparsed) in documentType.unparsedEntities.sorted(by: { $0.key < $1.key }) { + guard !notations.contains(unparsed.notation) else { continue } + findings.append("unparsed entity '\(entity)' names undeclared notation '\(unparsed.notation)'") + } + // VC: No Duplicate Types in mixed content. + for (name, model) in documentType.elementModels.sorted(by: { $0.key < $1.key }) { + if let duplicate = duplicateMixedName(model) { + findings.append("mixed content of '\(name)' repeats '\(duplicate)'") + } + } + for (element, declarations) in attributes.sorted(by: { $0.key < $1.key }) { + findings += attributeFindings(element: element, declarations: declarations, notations: notations) + } + return findings + } + + /// One element's attribute-declaration findings: at most one ID attribute, + /// NOTATION lists name declared notations, and defaults are legal. + private static func attributeFindings( + element: String, + declarations: [PureXML.Validation.AttributeDeclaration], + notations: Set, + ) -> [String] { + var findings: [String] = [] + if declarations.count(where: { $0.type == .id }) > 1 { + findings.append("element '\(element)' declares more than one ID attribute") + } + for declaration in declarations { + if case let .notation(allowed) = declaration.type { + for name in allowed.sorted() where !notations.contains(name) { + findings.append("NOTATION attribute '\(declaration.name)' on '\(element)' lists undeclared notation '\(name)'") + } + } + if let problem = defaultProblem(declaration) { + findings.append("attribute '\(declaration.name)' on '\(element)' \(problem)") + } + } + return findings + } + + /// The first repeated name in a `(#PCDATA|a|b)` mixed model, or nil. + private static func duplicateMixedName(_ model: String) -> String? { + let text = model.trimmingXMLWhitespace() + guard text.hasPrefix("(#PCDATA") else { return nil } + let inner = text.dropFirst().prefix { $0 != ")" } + var seen: Set = [] + for token in inner.split(separator: "|").dropFirst() { + let name = token.trimmingXMLWhitespace() + if !seen.insert(name).inserted { + return name + } + } + return nil + } + + /// Why a declaration's default value is illegal for its type, or nil. + private static func defaultProblem(_ declaration: PureXML.Validation.AttributeDeclaration) -> String? { + let defaultValue: String? = switch declaration.defaultDecl { + case let .fixed(value), let .value(value): value + case .required, .implied: nil + } + // VC: ID Attribute Default, an ID attribute must be #IMPLIED or #REQUIRED. + if declaration.type == .id { + return defaultValue == nil ? nil : "is an ID and must be #IMPLIED or #REQUIRED" + } + guard let value = defaultValue else { return nil } + switch declaration.type { + case let .enumeration(allowed) where !allowed.contains(value): + return "has a default outside its enumeration" + case let .notation(allowed) where !allowed.contains(value): + return "has a default outside its NOTATION list" + default: + return tokenizedDefaultProblem(declaration.type, value: value) + } + } + + /// The lexical-form problems of a tokenized type's default value. + private static func tokenizedDefaultProblem(_ type: PureXML.Validation.AttributeType, value: String) -> String? { + let isName = PureXML.Parsing.XMLCharacter.isValidName + let isNmtoken = PureXML.Parsing.XMLCharacter.isNmtoken + let tokens = value.split(whereSeparator: \.isWhitespace).map(String.init) + switch type { + case .idReference where !isName(value): + return "has an IDREF default that is not a Name" + case .idReferences where tokens.isEmpty || !tokens.allSatisfy(isName): + return "has an IDREFS default that is not Names" + case .nmToken where !isNmtoken(value): + return "has an NMTOKEN default that is not a name token" + case .nmTokens where tokens.isEmpty || !tokens.allSatisfy(isNmtoken): + return "has an NMTOKENS default that is not name tokens" + case .entity where !isName(value): + return "has an ENTITY default that is not a Name" + case .entities where tokens.isEmpty || !tokens.allSatisfy(isName): + return "has an ENTITIES default that is not Names" + default: + return nil + } + } +} diff --git a/Sources/Validation/DTDSchema.swift b/Sources/Validation/DTDSchema.swift index 1253389..caa795f 100644 --- a/Sources/Validation/DTDSchema.swift +++ b/Sources/Validation/DTDSchema.swift @@ -11,10 +11,19 @@ public extension PureXML.Validation { /// The names of declared unparsed (`NDATA`) entities, against which an /// `ENTITY`/`ENTITIES` attribute value is checked. let unparsedEntities: Set + /// The `` name the root element must match + /// (VC: Root Element Type). + let doctypeName: String? + /// Validity findings about the declarations themselves (duplicate + /// element types, repeated mixed-content names, multiple ID attributes, + /// undeclared notations in NOTATION lists, illegal attribute defaults), + /// reported once per validation at the document root. + let declarationErrors: [String] init(_ documentType: PureXML.Parsing.DocumentType) { notations = Set(documentType.notations.keys) unparsedEntities = Set(documentType.unparsedEntities.keys) + doctypeName = documentType.name var parsedModels: [String: ContentModel] = [:] for (name, model) in documentType.elementModels { parsedModels[name] = ContentModelParser.parse(model) @@ -26,6 +35,7 @@ public extension PureXML.Validation { parsedAttributes[name] = AttributeListParser.parse(body) } attributes = parsedAttributes + declarationErrors = Self.declarationFindings(documentType, attributes: parsedAttributes, notations: notations) } /// Whether the schema declares any elements or attributes. diff --git a/Sources/Validation/DTDValidationHelpers.swift b/Sources/Validation/DTDValidationHelpers.swift index 6cab6f4..44b14bb 100644 --- a/Sources/Validation/DTDValidationHelpers.swift +++ b/Sources/Validation/DTDValidationHelpers.swift @@ -6,7 +6,7 @@ extension PureXML.Validation.DTD { let content = childContent(of: element) switch model { case .empty: - return content.names.isEmpty && !content.hasText + return content.names.isEmpty && !content.hasText && !content.hasCDATA ? [] : [DTDFailure(reason: "element <\(name)> is declared EMPTY but has content", at: path)] case .any: @@ -27,11 +27,13 @@ extension PureXML.Validation.DTD { private static func childrenViolations( name: String, particle: PureXML.Validation.Particle, - content: (names: [String], hasText: Bool), + content: ChildContent, at path: DTDPath, ) -> [DTDFailure] { var result: [DTDFailure] = [] - if content.hasText { + if content.hasText || content.hasCDATA { + // A CDATA section is character data in element content even when + // empty or whitespace-only. result.append(DTDFailure(reason: "element <\(name)> has element content but contains character data", at: path)) } guard !PureXML.Validation.ContentModelMatcher.matchesChildren(particle, content.names) else { return result } @@ -57,20 +59,30 @@ extension PureXML.Validation.DTD { return names.filter { seen.insert($0).inserted } } - private static func childContent(of element: DTDElement) -> (names: [String], hasText: Bool) { + /// One element's immediate content, summarized for model matching. + struct ChildContent { var names: [String] = [] var hasText = false + /// Present CDATA sections, which count as character data in element + /// content even when empty or whitespace-only. + var hasCDATA = false + } + + private static func childContent(of element: DTDElement) -> ChildContent { + var content = ChildContent() for child in element.children { switch child { case let .element(inner): - names.append(inner.name.description) - case let .text(value), let .cdata(value): - if value.contains(where: { !$0.isWhitespace }) { hasText = true } + content.names.append(inner.name.description) + case let .text(value): + if value.contains(where: { !$0.isWhitespace }) { content.hasText = true } + case .cdata: + content.hasCDATA = true default: break } } - return (names, hasText) + return content } // MARK: Attributes @@ -147,7 +159,8 @@ extension PureXML.Validation.DTD { at path: DTDPath, ) -> DTDFailure? { let isNmtoken = PureXML.Parsing.XMLCharacter.isNmtoken - let isEntity = { (token: String) in PureXML.Parsing.XMLCharacter.isValidName(token) && entities.contains(token) } + let isName = PureXML.Parsing.XMLCharacter.isValidName + let isEntity = { (token: String) in isName(token) && entities.contains(token) } let tokens = value.split(whereSeparator: \.isWhitespace).map(String.init) let valid: Bool switch declaration.type { @@ -155,7 +168,11 @@ extension PureXML.Validation.DTD { case .nmTokens: valid = !tokens.isEmpty && tokens.allSatisfy(isNmtoken) case .entity: valid = isEntity(value) case .entities: valid = !tokens.isEmpty && tokens.allSatisfy(isEntity) - case .cdata, .enumeration, .notation, .id, .idReference, .idReferences: return nil + // The ID family must be lexical Names (VC: ID, IDREF); uniqueness and + // resolution are checked by the whole-tree integrity rule. + case .id, .idReference: valid = isName(value) + case .idReferences: valid = !tokens.isEmpty && tokens.allSatisfy(isName) + case .cdata, .enumeration, .notation: return nil } guard !valid else { return nil } return DTDFailure(reason: "attribute '\(declaration.name)' on <\(element)> is not a valid \(typeName(declaration.type))", at: path) @@ -167,6 +184,9 @@ extension PureXML.Validation.DTD { case .nmTokens: "NMTOKENS" case .entity: "ENTITY" case .entities: "ENTITIES" + case .id: "ID" + case .idReference: "IDREF" + case .idReferences: "IDREFS" default: "value" } } diff --git a/Sources/Validation/DTDValidations.swift b/Sources/Validation/DTDValidations.swift index 708a730..fd3bc09 100644 --- a/Sources/Validation/DTDValidations.swift +++ b/Sources/Validation/DTDValidations.swift @@ -18,7 +18,8 @@ public extension PureXML.Validation { .validating(requiredAttributes, fixedAttributeValues, enumeratedAttributeValues) .validating(tokenizedAttributeTypes, notationAttributes) .validating(identifierIntegrity) - if strict { validator = validator.validating(undeclaredElement) } + .validating(declarationValidity, rootElementType) + if strict { validator = validator.validating(undeclaredElement, undeclaredAttributes) } return validator } @@ -99,6 +100,53 @@ public extension PureXML.Validation { } } + /// In strict mode, every attribute is declared (VC: Attribute Value + /// Type). Namespace declarations are exempt: they are bindings, not + /// attributes, in the namespace-aware model. + static var undeclaredAttributes: Validation { + .init(description: "Every attribute is declared in the DTD") { context in + let element = context.subject + let declarations = context.document.attributes[element.name.description] ?? [] + return element.attributes + .filter { attribute in + let name = attribute.name.description + guard name != "xmlns", !name.hasPrefix("xmlns:") else { return false } + return !declarations.contains { $0.name == name || $0.name == attribute.name.localName } + } + .map { DTDFailure(reason: "attribute '\($0.name.description)' on <\(element.name.description)> is not declared", at: context.codingPath) } + } + } + + /// The DTD's own declarations are valid (duplicate element types, + /// repeated mixed names, multiple IDs, undeclared notations in lists, + /// illegal defaults). Reported once, at the document root. + static var declarationValidity: Validation { + .init( + description: "The DTD declarations satisfy their validity constraints", + check: { context in + context.document.declarationErrors.map { DTDFailure(reason: $0, at: context.codingPath) } + }, + when: { $0.codingPath.isEmpty }, + ) + } + + /// The root element's type matches the DOCTYPE name (VC: Root Element + /// Type). Reported once, at the document root. + static var rootElementType: Validation { + .init( + description: "The root element matches the DOCTYPE name", + check: { context in + guard let expected = context.document.doctypeName, + case let .document(children) = context.subject, + let root = children.compactMap(\.element).first, + root.name.description != expected + else { return [] } + return [DTDFailure(reason: "root element <\(root.name.description)> does not match the DOCTYPE name '\(expected)'", at: context.codingPath)] + }, + when: { $0.codingPath.isEmpty }, + ) + } + /// ID values are unique and every IDREF resolves. Runs once over the whole /// tree (a reference may point forward), so it is gated to the root. static var identifierIntegrity: Validation { diff --git a/Tests/DTDValidityConstraintTests.swift b/Tests/DTDValidityConstraintTests.swift new file mode 100644 index 0000000..3684019 --- /dev/null +++ b/Tests/DTDValidityConstraintTests.swift @@ -0,0 +1,84 @@ +@testable import PureXML +import Testing + +/// The validity constraints added with the Sun-section burn-down: declaration- +/// level checks (reported at the document root), the root/DOCTYPE name match, +/// lexical checks on the ID family, CDATA as character data in element +/// content, and the strict-mode every-attribute-declared rule. +@Suite("DTD validity constraints") +struct DTDValidityConstraintTests { + private func errors(_ xml: String, strict: Bool = false) throws -> [String] { + try PureXML.validateAgainstInternalDTD(xml, strict: strict).map(\.reason) + } + + @Test("The root element must match the DOCTYPE name") + func test_rootElementType() throws { + let xml = "]>\n" + #expect(try errors(xml).contains { $0.contains("does not match the DOCTYPE name") }) + let matching = "]>\n" + #expect(try errors(matching).isEmpty) + } + + @Test("An element type declared twice is reported") + func test_uniqueElementDeclaration() throws { + let xml = "]>\n" + #expect(try errors(xml).contains { $0.contains("declared more than once") }) + } + + @Test("Mixed content may not repeat a name") + func test_mixedDuplicates() throws { + let xml = "]>\n" + #expect(try errors(xml).contains { $0.contains("repeats 'x'") }) + } + + @Test("An element type may declare at most one ID attribute") + func test_oneIDPerElement() throws { + let xml = """ + + ]> + + """ + #expect(try errors(xml).contains { $0.contains("more than one ID attribute") }) + } + + @Test("NOTATION lists and NDATA entities must name declared notations") + func test_notationsDeclared() throws { + let list = "]>\n" + #expect(try errors(list).contains { $0.contains("lists undeclared notation 'b'") }) + let ndata = "]>\n" + #expect(try errors(ndata).contains { $0.contains("names undeclared notation 'missing'") }) + } + + @Test("Attribute defaults must be legal for their type, and IDs default-free") + func test_attributeDefaultLegal() throws { + let id = "]>\n" + #expect(try errors(id).contains { $0.contains("must be #IMPLIED or #REQUIRED") }) + let idref = "]>\n" + #expect(try errors(idref).contains { $0.contains("IDREF default") }) + let nmtokens = "]>\n" + #expect(try errors(nmtokens).contains { $0.contains("NMTOKENS default") }) + let entities = "]>\n" + #expect(try errors(entities).contains { $0.contains("ENTITIES default") }) + let enumerated = "]>\n" + #expect(try errors(enumerated).contains { $0.contains("outside its enumeration") }) + } + + @Test("ID and IDREF values must be lexical Names") + func test_idLexicalForm() throws { + let xml = "]>\n" + #expect(try errors(xml).contains { $0.contains("not a valid ID") }) + } + + @Test("A CDATA section is character data in element content") + func test_cdataInElementContent() throws { + let xml = "]>\n" + #expect(try errors(xml).contains { $0.contains("contains character data") }) + } + + @Test("Strict mode requires every attribute to be declared") + func test_undeclaredAttribute() throws { + let xml = "]>\n" + #expect(try errors(xml, strict: true).contains { $0.contains("is not declared") }) + #expect(try errors(xml).isEmpty) + } +} diff --git a/Tests/W3CSunSuiteTests.swift b/Tests/W3CSunSuiteTests.swift index 2b2e859..509d8b3 100644 --- a/Tests/W3CSunSuiteTests.swift +++ b/Tests/W3CSunSuiteTests.swift @@ -29,24 +29,15 @@ struct W3CSunSuiteTests { private let knownValidFailures: Set = [] /// invalid cases where this implementation knowingly reports no validity - /// error yet, the validity-layer frontier this issue works down: the - /// standalone VCs (not-sa*), ID/IDREF corners, #REQUIRED enforcement, - /// per-type attribute VCs (attr*), the root-element/DOCTYPE name match, - /// EMPTY content, and UTF-16 cases. + /// error yet: one family, the standalone='yes' validity constraints + /// (externally-declared defaults/entities/normalization effects must not + /// change the standalone document's infoset), which needs standalone and + /// declaration-provenance tracking. private let knownInvalidAccepted: Set = [ - "invalid/dtd01.xml", "invalid/dtd02.xml", "invalid/el01.xml", - "invalid/el04.xml", "invalid/el05.xml", "invalid/id01.xml", - "invalid/id03.xml", "invalid/id04.xml", "invalid/id05.xml", "invalid/not-sa01.xml", "invalid/not-sa02.xml", "invalid/not-sa04.xml", "invalid/not-sa05.xml", "invalid/not-sa06.xml", "invalid/not-sa07.xml", "invalid/not-sa08.xml", "invalid/not-sa09.xml", "invalid/not-sa10.xml", "invalid/not-sa11.xml", "invalid/not-sa12.xml", "invalid/not-sa13.xml", - "invalid/not-sa14.xml", "invalid/required01.xml", "invalid/required02.xml", - "invalid/root.xml", "invalid/attr04.xml", "invalid/attr09.xml", - "invalid/attr10.xml", "invalid/attr11.xml", "invalid/attr12.xml", - "invalid/attr13.xml", "invalid/attr14.xml", "invalid/attr15.xml", - "invalid/attr16.xml", "invalid/utf16b.xml", "invalid/utf16l.xml", - "invalid/empty.xml", ] private struct ManifestCase { @@ -135,6 +126,7 @@ struct W3CSunSuiteTests { let errors = try PureXML.validateAgainstInternalDTD( source(of: testCase.uri, in: section), limits: limits(), + strict: true, resolver: resolver(directory: directory), ) if let first = errors.first { @@ -161,6 +153,7 @@ struct W3CSunSuiteTests { let errors = try PureXML.validateAgainstInternalDTD( source(of: testCase.uri, in: section), limits: limits(), + strict: true, resolver: resolver(directory: directory), ) if errors.isEmpty {