From 805993e6563939944fae484ed3e4d4245ea455b1 Mon Sep 17 00:00:00 2001 From: Mihaela Mihaljevic Date: Wed, 10 Jun 2026 05:20:33 +0200 Subject: [PATCH] feat(parsing): strict content-model grammar Third W3C-suite strictness pass: DTDContentModelGrammar validates each element declaration's content specification against the XML 1.0 grammar (productions 46-51) at DTD scan time, EMPTY/ANY, Mixed (with #PCDATA first, names only, the trailing * required with alternatives, no nesting), and children (one connector per group, quantifiers bound directly to their particle, balanced parentheses, no empty groups, no SGML leftovers like '- -', 'O O', '&' connectors, or inclusion/exclusion suffixes). The element declaration also requires whitespace after ` ending), a raw `<` in attribute values, the literal `]]>` sequence in character data, attributes not separated by whitespace, and characters outside the XML `Char` production in content, five new located `ParseError` cases. Driven by a new opt-in runner for the W3C XML Conformance Test Suite's xmltest section (`XMLTS_ROOT`-gated; the suite is never vendored because its license permits redistribution only as the unmodified archive): valid/sa 118/119, not-wf/sa rejection improved from 80 to 109 of 186 across the first two strictness passes (the second adds character-reference validation, lowercase-x-only hex references and the referenced code point must be a valid XML Char; the reserved 'xml' PI target, legal only as the declaration at the document's very first bytes; and XML Char validation in comments and PI data), with the exact remaining baseline carried in the test and the two remaining leniency classes (internal-subset DTD grammar, entity replacement-text well-formedness) tracked as #120. +- Parser strictness, first W3C-suite pass (#105 Tier 2, #120): the strict parser now rejects `--` inside comments (including the `--->` ending), a raw `<` in attribute values, the literal `]]>` sequence in character data, attributes not separated by whitespace, and characters outside the XML `Char` production in content, five new located `ParseError` cases. Driven by a new opt-in runner for the W3C XML Conformance Test Suite's xmltest section (`XMLTS_ROOT`-gated; the suite is never vendored because its license permits redistribution only as the unmodified archive): valid/sa 118/119, not-wf/sa rejection improved from 80 to 129 of 186 across the first three strictness passes (the second adds character-reference validation, lowercase-x-only hex references and the referenced code point must be a valid XML Char; the reserved 'xml' PI target, legal only as the declaration at the document's very first bytes; and XML Char validation in comments and PI data; the third validates content models against the XML 1.0 grammar at DTD scan time, rejecting unbalanced parentheses, #PCDATA misplacement and nesting, mixed connectors, whitespace-detached or doubled quantifiers, empty groups, and SGML leftovers), with the exact remaining baseline carried in the test and the two remaining leniency classes (internal-subset DTD grammar, entity replacement-text well-formedness) tracked as #120. - XSD Particle Valid (Restriction) (#117): a complex type derived by restriction is now checked at schema compile to accept a structural subset of its base, covering element name matching, element-vs-wildcard admission, wildcard narrowing, occurrence-range subsumption, and the group recursions (sequence/all order-preserving with emptiable skips, choice mapping, sequence-into-choice, sequence-into-all, RecurseAsIfGroup). A structurally unfaithful restriction (widened occurrence, new element name, reordered sequence, content added to an EMPTY or required-into-empty base, an element outside a base wildcard's namespace) throws `SchemaError.invalidRestriction` with the type, base, and reason. The spec's effective-total-range arithmetic for MapAndSum is approximated by per-particle checks, a documented simplification. - Conformance corpus harness (#105, Tier 2): the start of conformance-suite checking, expressed through the validation framework (the OpenAPIKit idiom). A `PureXML.Validation.ConformanceCase` is a `Validatable` named expectation (actual vs spec-authoritative expected output), and `Conformance.matchesExpected` is a `Validation` rule that emits one located `ValidationError` per diverging case, so a whole suite reports every failure at once with the case name as its coding path. Seeded with a Canonical XML 1.0 corpus (attribute ordering, empty-element expansion, comment removal, CDATA-to-escaped-text, processing instructions, namespace rendering) an XPath 1.0 core-function corpus, and an XSD datatype/facet corpus (length, range, pattern, enumeration, lexical space), a RELAX NG pattern corpus (text, optional, repetition, choice, ordered group, attribute, interleave, empty), an XSLT 1.0 transformation corpus (value-of, for-each, if, choose, sort, variable, format-number, apply-templates, call-template), an HTML5 tree-construction corpus (void elements, implied tag closing, case normalization, the adoption agency, entities), a Schematron corpus (assert, report, rule context), a C14N exclusive/2.0 corpus (unused-namespace dropping, render-at-use, whitespace trimming), an XPath axes/predicates corpus (descendant, ancestor, sibling axes, positional and attribute predicates, union), a DTD content-model corpus (sequence, choice, occurrence indicators, mixed, EMPTY, ANY), an XInclude/XPointer corpus (substitution, text inclusion with escaping, fallback, relative href resolution; shorthand ids, element() navigation, xpointer() including the per-step predicate semantic), an XSD regex corpus (implicit whole-string anchoring, the character-class escapes, Unicode categories, quantifier ranges, alternation, class negation and subtraction), and an XML Catalog corpus (system/public/uri entries, rewrite prefixes, unmatched identifiers). The HTML corpus surfaced its first divergence (the fragment parser does not reconstruct active formatting elements like the document parser, tracked as #109). Growing the corpora against the official W3C/OASIS suites is the ongoing Tier-2 work. - EXSLT extension functions (#105, Tier 3): the `common`, `math`, and `sets` EXSLT modules, dispatched by resolving a function's prefix to its namespace (so any prefix bound to an EXSLT namespace works). `math:min`, `math:max`, `math:highest`, `math:lowest`, `math:abs`, `math:sqrt` (with the EXSLT NaN rule for non-numeric nodes); `set:distinct`, `set:difference`, `set:intersection`, `set:has-same-node`, `set:leading`, `set:trailing` (by node identity and document order); `exsl:object-type`. `math:power` and the other transcendentals are omitted because they need a C math library, which the pure-Swift, Foundation-free target excludes; `exsl:node-set` awaits result-tree-fragment support. diff --git a/Sources/Parsing/DTDContentModelGrammar.swift b/Sources/Parsing/DTDContentModelGrammar.swift new file mode 100644 index 0000000..85e7548 --- /dev/null +++ b/Sources/Parsing/DTDContentModelGrammar.swift @@ -0,0 +1,152 @@ +extension PureXML.Parsing { + /// Validates an `` content specification against the XML 1.0 + /// grammar (productions 46-51): `EMPTY`, `ANY`, `Mixed`, or `children`. + /// The lenient `ContentModelParser` still drives validation behavior; this + /// strict check runs at DTD scan time so a malformed model (unbalanced + /// parentheses, `#PCDATA` misplacement, mixed connectors, a quantifier + /// detached by whitespace, SGML leftovers) is rejected as not well-formed. + enum DTDContentModelGrammar { + static func isValid(_ model: String) -> Bool { + let text = model.trimmingXMLWhitespace() + if text == "EMPTY" || text == "ANY" { return true } + var scanner = ModelScanner(text) + guard scanner.peek() == "(" else { return false } + // Mixed and children both start with '('; Mixed is identified by + // '#PCDATA' as the first token inside. + let isMixed = scanner.looksMixed() + let parsed = isMixed ? scanner.mixed() : scanner.children() + return parsed && scanner.isAtEnd + } + } +} + +/// A tiny recursive-descent scanner over one content-model expression. +private struct ModelScanner { + private let characters: [Character] + private var index = 0 + + init(_ text: String) { + characters = Array(text) + } + + var isAtEnd: Bool { + index >= characters.count + } + + func peek(_ ahead: Int = 0) -> Character? { + index + ahead < characters.count ? characters[index + ahead] : nil + } + + private mutating func advance() -> Character? { + guard index < characters.count else { return nil } + defer { index += 1 } + return characters[index] + } + + private mutating func skipSpace() { + while let character = peek(), character.isWhitespace { + index += 1 + } + } + + private mutating func consume(_ character: Character) -> Bool { + if peek() == character { + index += 1 + return true + } + return false + } + + /// Whether the expression opens as `( S? #PCDATA`. + func looksMixed() -> Bool { + var probe = self + guard probe.consume("(") else { return false } + probe.skipSpace() + return probe.peek() == "#" + } + + /// `Mixed ::= '(' S? '#PCDATA' (S? '|' S? Name)* S? ')*' | '(' S? '#PCDATA' S? ')'` + mutating func mixed() -> Bool { + guard consume("(") else { return false } + skipSpace() + guard consumeLiteral("#PCDATA") else { return false } + var sawAlternative = false + while true { + skipSpace() + if consume(")") { break } + guard consume("|") else { return false } + sawAlternative = true + skipSpace() + guard name() else { return false } + } + // With alternatives the trailing '*' is required; without, optional none. + if sawAlternative { + return consume("*") + } + return peek() != "*" || consume("*") + } + + /// `children ::= (choice | seq) ('?' | '*' | '+')?` with + /// `cp ::= (Name | choice | seq) ('?' | '*' | '+')?`. A group commits to one + /// connector (`|` or `,`); a quantifier must follow its particle directly. + mutating func children() -> Bool { + guard group() else { return false } + _ = quantifier() + return true + } + + private mutating func group() -> Bool { + guard consume("(") else { return false } + skipSpace() + guard particle() else { return false } + skipSpace() + var connector: Character? + while !consume(")") { + guard let next = advance(), next == "|" || next == "," else { return false } + if let connector, connector != next { return false } + connector = next + skipSpace() + guard particle() else { return false } + skipSpace() + } + return true + } + + private mutating func particle() -> Bool { + if peek() == "(" { + guard group() else { return false } + } else { + guard name() else { return false } + } + _ = quantifier() + return true + } + + /// A quantifier binds directly to its particle: no whitespace before it, + /// and at most one. + private mutating func quantifier() -> Bool { + guard let next = peek(), next == "?" || next == "*" || next == "+" else { return false } + index += 1 + return true + } + + private mutating func name() -> Bool { + guard let first = peek(), first.unicodeScalars.allSatisfy(PureXML.Parsing.XMLCharacter.isNameStart) else { + return false + } + index += 1 + while let next = peek(), next.unicodeScalars.allSatisfy(PureXML.Parsing.XMLCharacter.isNameChar) { + index += 1 + } + return true + } + + private mutating func consumeLiteral(_ literal: String) -> Bool { + let count = literal.count + guard index + count <= characters.count, + String(characters[index ..< index + count]) == literal + else { return false } + index += count + return true + } +} diff --git a/Sources/Parsing/Diagnostic.swift b/Sources/Parsing/Diagnostic.swift index 924fbb8..2d8d785 100644 --- a/Sources/Parsing/Diagnostic.swift +++ b/Sources/Parsing/Diagnostic.swift @@ -46,7 +46,7 @@ public extension PureXML.Parsing.ParseError { let .duplicateAttribute(_, mark), let .invalidReference(_, mark), let .undefinedEntity(_, mark), let .recursiveEntity(_, mark), let .unexpectedEndTag(_, mark), let .undefinedNamespacePrefix(_, mark), - let .nestingTooDeep(_, mark), let .nameTooLong(_, mark), let .contentTooLong(_, mark): + let .nestingTooDeep(_, mark), let .nameTooLong(_, mark), let .contentTooLong(_, mark), let .invalidContentModel(_, mark): mark } } diff --git a/Sources/Parsing/DoctypeScanner.swift b/Sources/Parsing/DoctypeScanner.swift index b0ed181..603c4d8 100644 --- a/Sources/Parsing/DoctypeScanner.swift +++ b/Sources/Parsing/DoctypeScanner.swift @@ -112,7 +112,7 @@ private struct DTDScanner { } else if reader.matches("` content model does not match the XML 1.0 grammar. + case invalidContentModel(element: String, Mark) public var description: String { switch self { @@ -111,6 +113,8 @@ public extension PureXML.Parsing { "character is not allowed in XML content at \(mark)" case let .reservedProcessingInstructionTarget(mark): "the processing-instruction target 'xml' is reserved at \(mark)" + case let .invalidContentModel(element, mark): + "the content model of is malformed at \(mark)" case let .malformedDeclaration(mark): "malformed XML declaration at \(mark)" } diff --git a/Tests/W3CXMLSuiteTests.swift b/Tests/W3CXMLSuiteTests.swift index 9a4fc4c..73ac2f7 100644 --- a/Tests/W3CXMLSuiteTests.swift +++ b/Tests/W3CXMLSuiteTests.swift @@ -29,11 +29,9 @@ struct W3CXMLSuiteTests { "079.xml", "080.xml", "082.xml", "084.xml", "085.xml", "086.xml", "087.xml", "089.xml", "090.xml", "091.xml", "092.xml", "096.xml", "101.xml", "102.xml", "103.xml", "107.xml", "113.xml", "114.xml", "115.xml", "116.xml", "117.xml", "119.xml", "120.xml", "121.xml", - "122.xml", "123.xml", "124.xml", "125.xml", "126.xml", "127.xml", "128.xml", "129.xml", - "130.xml", "131.xml", "132.xml", "133.xml", "134.xml", "135.xml", "136.xml", "137.xml", - "138.xml", "139.xml", "140.xml", "141.xml", "149.xml", "153.xml", "158.xml", "160.xml", - "161.xml", "162.xml", "165.xml", "168.xml", "169.xml", "170.xml", "173.xml", "174.xml", - "175.xml", "180.xml", "182.xml", "183.xml", "184.xml", + "140.xml", "141.xml", "149.xml", "153.xml", "158.xml", "160.xml", "161.xml", "162.xml", + "165.xml", "168.xml", "169.xml", "170.xml", "173.xml", "174.xml", "175.xml", "180.xml", + "182.xml", ] /// valid/sa/114.xml: a CDATA section inside an entity value must protect