diff --git a/CHANGELOG.md b/CHANGELOG.md index f086a9c..0b991c7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Parser strictness, first W3C-suite pass (#105 Tier 2, #120): the strict parser now rejects `--` inside comments (including the `--->` ending), a raw `<` in attribute values, the literal `]]>` sequence in character data, attributes not separated by whitespace, and characters outside the XML `Char` production in content, five new located `ParseError` cases. Driven by a new opt-in runner for the W3C XML Conformance Test Suite's xmltest section (`XMLTS_ROOT`-gated; the suite is never vendored because its license permits redistribution only as the unmodified archive): valid/sa 118/119, not-wf/sa rejection improved from 80 to 93 of 186, with the exact remaining baseline carried in the test and the two remaining leniency classes (internal-subset DTD grammar, entity replacement-text well-formedness) tracked as #120. - XSD Particle Valid (Restriction) (#117): a complex type derived by restriction is now checked at schema compile to accept a structural subset of its base, covering element name matching, element-vs-wildcard admission, wildcard narrowing, occurrence-range subsumption, and the group recursions (sequence/all order-preserving with emptiable skips, choice mapping, sequence-into-choice, sequence-into-all, RecurseAsIfGroup). A structurally unfaithful restriction (widened occurrence, new element name, reordered sequence, content added to an EMPTY or required-into-empty base, an element outside a base wildcard's namespace) throws `SchemaError.invalidRestriction` with the type, base, and reason. The spec's effective-total-range arithmetic for MapAndSum is approximated by per-particle checks, a documented simplification. - Conformance corpus harness (#105, Tier 2): the start of conformance-suite checking, expressed through the validation framework (the OpenAPIKit idiom). A `PureXML.Validation.ConformanceCase` is a `Validatable` named expectation (actual vs spec-authoritative expected output), and `Conformance.matchesExpected` is a `Validation` rule that emits one located `ValidationError` per diverging case, so a whole suite reports every failure at once with the case name as its coding path. Seeded with a Canonical XML 1.0 corpus (attribute ordering, empty-element expansion, comment removal, CDATA-to-escaped-text, processing instructions, namespace rendering) an XPath 1.0 core-function corpus, and an XSD datatype/facet corpus (length, range, pattern, enumeration, lexical space), a RELAX NG pattern corpus (text, optional, repetition, choice, ordered group, attribute, interleave, empty), an XSLT 1.0 transformation corpus (value-of, for-each, if, choose, sort, variable, format-number, apply-templates, call-template), an HTML5 tree-construction corpus (void elements, implied tag closing, case normalization, the adoption agency, entities), a Schematron corpus (assert, report, rule context), a C14N exclusive/2.0 corpus (unused-namespace dropping, render-at-use, whitespace trimming), an XPath axes/predicates corpus (descendant, ancestor, sibling axes, positional and attribute predicates, union), a DTD content-model corpus (sequence, choice, occurrence indicators, mixed, EMPTY, ANY), an XInclude/XPointer corpus (substitution, text inclusion with escaping, fallback, relative href resolution; shorthand ids, element() navigation, xpointer() including the per-step predicate semantic), an XSD regex corpus (implicit whole-string anchoring, the character-class escapes, Unicode categories, quantifier ranges, alternation, class negation and subtraction), and an XML Catalog corpus (system/public/uri entries, rewrite prefixes, unmatched identifiers). The HTML corpus surfaced its first divergence (the fragment parser does not reconstruct active formatting elements like the document parser, tracked as #109). Growing the corpora against the official W3C/OASIS suites is the ongoing Tier-2 work. - EXSLT extension functions (#105, Tier 3): the `common`, `math`, and `sets` EXSLT modules, dispatched by resolving a function's prefix to its namespace (so any prefix bound to an EXSLT namespace works). `math:min`, `math:max`, `math:highest`, `math:lowest`, `math:abs`, `math:sqrt` (with the EXSLT NaN rule for non-numeric nodes); `set:distinct`, `set:difference`, `set:intersection`, `set:has-same-node`, `set:leading`, `set:trailing` (by node identity and document order); `exsl:object-type`. `math:power` and the other transcendentals are omitted because they need a C math library, which the pure-Swift, Foundation-free target excludes; `exsl:node-set` awaits result-tree-fragment support. diff --git a/README.md b/README.md index 4d022bb..43d15be 100644 --- a/README.md +++ b/README.md @@ -99,7 +99,7 @@ flowchart TB PureXML is a working, dependency-free XML library today: parse, emit, validate, query, and stream documents on macOS, Linux, Windows, and WASM. The test suite -currently runs **844 tests in 140 suites** (`swift test`). +currently runs **846 tests in 141 suites** (`swift test`). ### Shipped (libxml2-aligned surface) diff --git a/Sources/Parsing/Diagnostic.swift b/Sources/Parsing/Diagnostic.swift index 7ffc387..903dc6d 100644 --- a/Sources/Parsing/Diagnostic.swift +++ b/Sources/Parsing/Diagnostic.swift @@ -37,7 +37,10 @@ public extension PureXML.Parsing.ParseError { case let .unexpectedEndOfInput(mark), let .unterminatedTag(mark), let .unterminatedComment(mark), let .unterminatedCDATA(mark), let .expectedName(mark), let .expectedEquals(mark), let .unquotedAttributeValue(mark), let .amplificationLimitExceeded(mark), - let .junkAfterDocumentElement(mark), let .unsupportedDoctype(mark), let .malformedDeclaration(mark): + let .junkAfterDocumentElement(mark), let .unsupportedDoctype(mark), let .malformedDeclaration(mark), + let .doubleHyphenInComment(mark), let .rawLessThanInAttribute(mark), + let .cdataCloseInContent(mark), let .missingSpaceBeforeAttribute(mark), + let .invalidCharacter(mark): mark case let .unexpectedCharacter(_, mark), let .mismatchedEndTag(_, _, mark), let .duplicateAttribute(_, mark), let .invalidReference(_, mark), diff --git a/Sources/Parsing/EventReader.swift b/Sources/Parsing/EventReader.swift index 62fd106..12cf3db 100644 --- a/Sources/Parsing/EventReader.swift +++ b/Sources/Parsing/EventReader.swift @@ -23,14 +23,14 @@ public extension PureXML.Parsing { /// The parsed XML declaration ``, when the document opens with /// one; nil otherwise. var xmlDeclaration: XMLDeclaration? - private var entityBudget: Int + var entityBudget: Int var pending: [Event] = [] private var sawRoot = false private var primed = false /// When true, ``next()`` repairs malformed input in place instead of /// throwing: it records a ``Diagnostic`` and emits best-effort events. - private let recovering: Bool + let recovering: Bool var reachedEnd = false /// The problems found so far, when reading in recovering mode. var diagnostics: [Diagnostic] = [] @@ -202,6 +202,12 @@ public extension PureXML.Parsing { while let character = reader.peek(), character != "<" { length += 1 try checkContent(length, mark) + if character == ">", raw.hasSuffix("]]") { + throw ParseError.cdataCloseInContent(reader.mark) + } + guard character.unicodeScalars.allSatisfy(XMLCharacter.isChar) else { + throw ParseError.invalidCharacter(reader.mark) + } raw.append(character) reader.advance() } @@ -227,56 +233,7 @@ public extension PureXML.Parsing { return (target.description, data) } - private mutating func scanAttributes() throws -> [PureXML.Model.Attribute] { - var attributes: [PureXML.Model.Attribute] = [] - var seen: Set = [] - while true { - reader.skipSpace() - guard let next = reader.peek(), next != ">", next != "/" else { - return attributes - } - let mark = reader.mark - let name = try scanName() - reader.skipSpace() - guard reader.consume("=") else { - throw ParseError.expectedEquals(reader.mark) - } - reader.skipSpace() - let value = try scanAttributeValue() - guard seen.insert(name.description).inserted else { - if recovering { - // Keep the element: record the duplicate and drop the later - // occurrence rather than failing the whole start tag. - diagnostics.append(Diagnostic(.duplicateAttribute(name: name.description, mark))) - continue - } - throw ParseError.duplicateAttribute(name: name.description, mark) - } - attributes.append(PureXML.Model.Attribute(name: name, value: value)) - } - } - - private mutating func scanAttributeValue() throws -> String { - let mark = reader.mark - guard let quote = reader.peek(), quote == "\"" || quote == "'" else { - throw ParseError.unquotedAttributeValue(reader.mark) - } - reader.advance() - var raw = "" - var length = 0 - while let character = reader.peek(), character != quote { - length += 1 - try checkContent(length, mark) - raw.append(character) - reader.advance() - } - guard reader.consume(String(quote)) else { - throw ParseError.unexpectedEndOfInput(reader.mark) - } - return try EntityDecoder.decode(raw, entities: documentType.entities, budget: &entityBudget, at: mark) - } - - private mutating func scanName() throws -> PureXML.Model.QualifiedName { + mutating func scanName() throws -> PureXML.Model.QualifiedName { let mark = reader.mark guard let first = reader.peek(), first.isXMLNameStart else { throw ParseError.expectedName(reader.mark) diff --git a/Sources/Parsing/EventReaderAttributes.swift b/Sources/Parsing/EventReaderAttributes.swift new file mode 100644 index 0000000..72112b1 --- /dev/null +++ b/Sources/Parsing/EventReaderAttributes.swift @@ -0,0 +1,59 @@ +/// The start-tag attribute scanners, in their own file to keep the reader's +/// type body under the length cap. Behavior is part of EventReader proper. +extension PureXML.Parsing.EventReader { + mutating func scanAttributes() throws -> [PureXML.Model.Attribute] { + var attributes: [PureXML.Model.Attribute] = [] + var seen: Set = [] + while true { + reader.skipSpace() + guard let next = reader.peek(), next != ">", next != "/" else { + return attributes + } + let mark = reader.mark + let name = try scanName() + reader.skipSpace() + guard reader.consume("=") else { + throw PureXML.Parsing.ParseError.expectedEquals(reader.mark) + } + reader.skipSpace() + let value = try scanAttributeValue() + guard seen.insert(name.description).inserted else { + if recovering { + // Keep the element: record the duplicate and drop the later + // occurrence rather than failing the whole start tag. + diagnostics.append(PureXML.Parsing.Diagnostic(PureXML.Parsing.ParseError.duplicateAttribute(name: name.description, mark))) + continue + } + throw PureXML.Parsing.ParseError.duplicateAttribute(name: name.description, mark) + } + attributes.append(PureXML.Model.Attribute(name: name, value: value)) + // A following attribute needs whitespace before it. + if let next = reader.peek(), next != ">", next != "/", next != "?", !next.isWhitespace { + throw PureXML.Parsing.ParseError.missingSpaceBeforeAttribute(reader.mark) + } + } + } + + mutating func scanAttributeValue() throws -> String { + let mark = reader.mark + guard let quote = reader.peek(), quote == "\"" || quote == "'" else { + throw PureXML.Parsing.ParseError.unquotedAttributeValue(reader.mark) + } + reader.advance() + var raw = "" + var length = 0 + while let character = reader.peek(), character != quote { + length += 1 + try checkContent(length, mark) + if character == "<" { + throw PureXML.Parsing.ParseError.rawLessThanInAttribute(reader.mark) + } + raw.append(character) + reader.advance() + } + guard reader.consume(String(quote)) else { + throw PureXML.Parsing.ParseError.unexpectedEndOfInput(reader.mark) + } + return try PureXML.Parsing.EntityDecoder.decode(raw, entities: documentType.entities, budget: &entityBudget, at: mark) + } +} diff --git a/Sources/Parsing/EventReaderRecovery.swift b/Sources/Parsing/EventReaderRecovery.swift index ce097eb..74a1128 100644 --- a/Sources/Parsing/EventReaderRecovery.swift +++ b/Sources/Parsing/EventReaderRecovery.swift @@ -55,6 +55,11 @@ extension PureXML.Parsing.EventReader { } length += 1 try checkContent(length, mark) + // XML 1.0: '--' may not appear inside a comment (this also rejects + // the '--->' ending, since its first two hyphens hit this check). + if character == "-", reader.peek() == "-" { + throw PureXML.Parsing.ParseError.doubleHyphenInComment(mark) + } content.append(character) } reader.consume("-->") diff --git a/Sources/Parsing/ParseError.swift b/Sources/Parsing/ParseError.swift index 7463861..ea2948b 100644 --- a/Sources/Parsing/ParseError.swift +++ b/Sources/Parsing/ParseError.swift @@ -34,6 +34,17 @@ public extension PureXML.Parsing { /// The XML declaration `` is malformed: its pseudo-attributes /// are out of order, unknown, or carry an illegal value. case malformedDeclaration(Mark) + /// A comment contains `--`, which XML 1.0 forbids inside comments. + case doubleHyphenInComment(Mark) + /// An attribute value contains a raw `<`, which must be escaped. + case rawLessThanInAttribute(Mark) + /// Character data contains the literal `]]>` sequence, which must be + /// escaped outside a CDATA section. + case cdataCloseInContent(Mark) + /// Two attributes are not separated by whitespace. + case missingSpaceBeforeAttribute(Mark) + /// A character outside the XML `Char` production appears in content. + case invalidCharacter(Mark) public var description: String { switch self { @@ -85,6 +96,16 @@ public extension PureXML.Parsing { "input bytes are malformed for the detected encoding" case let .notImplemented(detail): "not implemented: \(detail)" + case let .doubleHyphenInComment(mark): + "'--' is not allowed inside a comment at \(mark)" + case let .rawLessThanInAttribute(mark): + "raw '<' is not allowed in an attribute value at \(mark)" + case let .cdataCloseInContent(mark): + "']]>' must be escaped in character data at \(mark)" + case let .missingSpaceBeforeAttribute(mark): + "attributes must be separated by whitespace at \(mark)" + case let .invalidCharacter(mark): + "character is not allowed in XML content at \(mark)" case let .malformedDeclaration(mark): "malformed XML declaration at \(mark)" } diff --git a/Tests/W3CXMLSuiteTests.swift b/Tests/W3CXMLSuiteTests.swift new file mode 100644 index 0000000..32cca59 --- /dev/null +++ b/Tests/W3CXMLSuiteTests.swift @@ -0,0 +1,109 @@ +import Foundation +@testable import PureXML +import Testing + +/// Opt-in runner for the W3C XML Conformance Test Suite's xmltest section +/// (James Clark's cases). The suite's license permits redistribution only as +/// the unmodified archive, so it is never vendored into this repository: +/// download `xmlts` from w3.org, extract it, and point `XMLTS_ROOT` at the +/// `xmlconf/xmltest` directory to run. Without the variable the suite is +/// skipped. Foundation is used here for file access only; this is a test-only +/// dependency and the library target remains Foundation-free. +@Suite("W3C xmltest conformance (opt-in via XMLTS_ROOT)") +struct W3CXMLSuiteTests { + private var root: String? { + ProcessInfo.processInfo.environment["XMLTS_ROOT"] + } + + /// Cases this implementation knowingly accepts although the suite marks + /// them not-well-formed. Two classes, tracked for later hardening: the + /// internal-subset scanner is deliberately lenient about DTD grammar + /// details (declaration syntax, conditional sections, parameter-entity + /// placement), and entity replacement text is not yet checked for + /// standalone well-formedness (tags spanning entity boundaries, partial + /// markup in values). The baseline below is exact, so any regression in + /// the cases that DO pass is still caught. + private let knownNotWFDeviations: Set = [ + "031.xml", "032.xml", "054.xml", "057.xml", "058.xml", "059.xml", "060.xml", "061.xml", + "062.xml", "063.xml", "064.xml", "065.xml", "066.xml", "067.xml", "068.xml", "069.xml", + "074.xml", "078.xml", "079.xml", "080.xml", "082.xml", "084.xml", "085.xml", "086.xml", + "087.xml", "089.xml", "090.xml", "091.xml", "092.xml", "093.xml", "096.xml", "101.xml", + "102.xml", "103.xml", "107.xml", "113.xml", "114.xml", "115.xml", "116.xml", "117.xml", + "119.xml", "120.xml", "121.xml", "122.xml", "123.xml", "124.xml", "125.xml", "126.xml", + "127.xml", "128.xml", "129.xml", "130.xml", "131.xml", "132.xml", "133.xml", "134.xml", + "135.xml", "136.xml", "137.xml", "138.xml", "139.xml", "140.xml", "141.xml", "142.xml", + "143.xml", "144.xml", "147.xml", "148.xml", "149.xml", "150.xml", "151.xml", "153.xml", + "154.xml", "155.xml", "156.xml", "157.xml", "158.xml", "160.xml", "161.xml", "162.xml", + "165.xml", "168.xml", "169.xml", "170.xml", "171.xml", "172.xml", "173.xml", "174.xml", + "175.xml", "180.xml", "182.xml", "183.xml", "184.xml", + ] + + /// valid/sa/114.xml: a CDATA section inside an entity value must protect + /// the &foo; reference from expansion; the entity decoder expands it + /// before the CDATA section is recognized. Same hardening track. + private let knownValidDeviations: Set = ["114.xml"] + + private func files(in directory: String) throws -> [String] { + try FileManager.default.contentsOfDirectory(atPath: directory) + .filter { $0.hasSuffix(".xml") } + .sorted() + } + + private func limits() -> PureXML.Parsing.Limits { + PureXML.Parsing.Limits(allowDoctype: true) + } + + /// Resolves relative SYSTEM identifiers against the case's directory, so the + /// suite's external-entity cases load their sibling `.ent` files. + private func resolver(directory: String) -> PureXML.Parsing.EntityResolver { + PureXML.Parsing.EntityResolver( + resolveEntity: { _, id in + (try? String(contentsOfFile: directory + "/" + id.systemID, encoding: .utf8)) + }, + resolveExternalSubset: { id in + (try? String(contentsOfFile: directory + "/" + id.systemID, encoding: .utf8)) + }, + ) + } + + /// Reads a case's raw bytes and decodes through PureXML's own byte decoder, + /// since the suite deliberately includes UTF-16 documents and broken + /// encodings (a decode failure counts as rejection for not-wf cases). + private func parse(file: String, in directory: String) throws -> PureXML.Model.Node { + let data = try Data(contentsOf: URL(fileURLWithPath: directory + "/" + file)) + let source = try PureXML.Parsing.ByteDecoder.decode([UInt8](data)) + return try PureXML.parse(source, limits: limits(), resolver: resolver(directory: directory)) + } + + @Test("Every not-wf/sa case is rejected by the strict parser") + func test_notWellFormedRejected() throws { + guard let root else { + return // Opt-in: suite not present. + } + let directory = root + "/not-wf/sa" + var unexpectedlyAccepted: [String] = [] + for file in try files(in: directory) where !knownNotWFDeviations.contains(file) { + if (try? parse(file: file, in: directory)) != nil { + unexpectedlyAccepted.append(file) + } + } + #expect(unexpectedlyAccepted.isEmpty, "accepted \(unexpectedlyAccepted.count) not-wf cases: \(unexpectedlyAccepted)") + } + + @Test("Every valid/sa case parses") + func test_validParses() throws { + guard let root else { + return // Opt-in: suite not present. + } + let directory = root + "/valid/sa" + var failures: [String: String] = [:] + for file in try files(in: directory) where !knownValidDeviations.contains(file) { + do { + _ = try parse(file: file, in: directory) + } catch { + failures[file] = String(describing: error) + } + } + #expect(failures.isEmpty, "\(failures.count) valid cases failed: \(failures)") + } +}