test(conformance): japanese section runner, fully clean

All 13 cases of the xmlconf japanese section decode through the
package byte decoder (UTF-8, UTF-16 both orders, Shift_JIS, EUC-JP,
ISO-2022-JP), parse, and DTD-validate cleanly with zero deviations.
The optional encodings the suite marks TYPE='error' are all supported,
so every case is held to the must-parse bar.

Closes #126

CHANGELOG.md

@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- W3C xmlconf japanese section, fully clean (#121, #126): all 13 cases (the same prose and parameter-entity DTDs encoded as UTF-8, UTF-16 in both byte orders, Shift_JIS, EUC-JP, and ISO-2022-JP) decode through PureXML's own byte decoder, parse, and DTD-validate cleanly with zero deviations; the optional encodings the suite classifies as TYPE='error' are all supported, so they are held to the must-parse bar.
 - W3C xmlconf IBM section, clean to two documented classes (#121, #125): the deepest block, 1094 cases. Valid 220/220 (parse + strict DTD validation), invalid 64/64 reporting, not-wf 529/810 rejected with the 281 accepted cases being exactly two documented classes: 279 cases from productions 85-89, the 1998 character-class appendices deleted by XML 1.0 Fifth Edition (whose count is asserted exactly so drift in either direction is caught), and 2 parameter-entity-in-internal-subset feature cases (the xmltest 160-162 class, headed for #128's strict profile). Closing the section's real gaps: a document without a root element is rejected (production 1); an entity reference in an entity literal must be a lexical Name or character reference (production 68); WFC: No Recursion is enforced at declaration over the combined general/parameter reference graph (CDATA-shielded), so a recursive pair is rejected without being referenced; an undeclared parameter-entity reference is a well-formedness error when the document is standalone or has neither an external subset nor parameter entities, and a deferred validity finding otherwise (the production-68 WFC/VC split, also applied to entity references in attribute defaults); a declared external PE the resolver refused is neither; and a content-model group opening in one PE replacement and closing in another is reported (VC: Proper Group/PE Nesting).
 - Standalone validity constraints (#134): a document declaring `standalone='yes'` is now checked against its external-subset dependencies (2.9). The DTD scanner records declaration provenance (which entities, element models, and attribute lists came from the internal subset); the validator reports an externally-declared attribute default that would be supplied, an attribute value an externally-declared non-CDATA type would normalize, and whitespace inside externally-declared element content. The parser enforces the matching well-formedness side: a standalone document referencing an entity declared outside the internal subset reports it as undeclared (WFC: Entity Declared). This empties the entire Sun baseline: the section is fully clean at not-wf 56/56, valid 28/28, invalid 74/74, zero deviations, closing #124.
 - DTD validity constraints, second Sun burn-down (#121, #124): the validator now enforces VC Root Element Type (the root element must match the DOCTYPE name, recorded by the scanner), VC Unique Element Type Declaration (duplicate `<!ELEMENT>` reported, first stays in effect), VC No Duplicate Types in mixed content, VC One ID per Element Type, VC Notation Attributes (listed names must be declared) and VC Notation Declared (an NDATA entity's notation must exist), VC Attribute Default Legal across the tokenized types plus VC ID Attribute Default (an ID attribute must be #IMPLIED or #REQUIRED), lexical Name checks for ID/IDREF/IDREFS values, and CDATA sections as character data in element content even when empty. Strict mode additionally requires every attribute to be declared (VC Attribute Value Type; namespace declarations exempt). Declaration-level findings are computed once when the schema is built and reported at the document root. Sun invalid coverage moves from 43/74 to 62/74 reporting; the twelve remaining silent cases are one family, the standalone='yes' VCs, which need declaration-provenance tracking.

README.md

@@ -91,7 +91,7 @@ flowchart TB
   Oasis["#123 OASIS/NIST section: complete, zero deviations"]:::done
   Sun["#124 Sun section: complete, zero deviations"]:::done
   IBM["#125 IBM section: clean to two documented classes"]:::done
-  Japanese["#126 japanese section: encoding declarations"]:::todo
+  Japanese["#126 japanese section: complete, zero deviations"]:::done
   Eduni["#127 eduni: errata + namespaces"]:::todo
   StrictSubset["#128 strict internal-subset profile: xmltest 185/186"]:::todo
   Epic122["#122 official spec suites"]:::epic
@@ -117,7 +117,7 @@ flowchart TB
 
 PureXML is a working, dependency-free XML library today: parse, emit, validate,
 query, and stream documents on macOS, Linux, Windows, and WASM. The test suite
-currently runs **898 tests in 150 suites** (`swift test`).
+currently runs **899 tests in 151 suites** (`swift test`).
 
 ### Shipped (libxml2-aligned surface)
 

Tests/W3CJapaneseSuiteTests.swift

@@ -0,0 +1,73 @@
+import Foundation
+@testable import PureXML
+import Testing
+
+/// Opt-in runner for the W3C XML Conformance Test Suite's japanese section:
+/// the same prose encoded as UTF-8, UTF-16 (both orders), Shift_JIS, EUC-JP,
+/// and ISO-2022-JP, with moderately complex parameter-entity DTDs. The
+/// optional encodings are classified TYPE='error' (a processor either
+/// supports them or reports a fatal error); PureXML ships all of them, so
+/// every case must parse and DTD-validate cleanly. Point `XMLCONF_ROOT` at
+/// the extracted `xmlconf` directory to run (the suite is never vendored).
+/// Foundation is used for file access only.
+@Suite("W3C japanese conformance (opt-in via XMLCONF_ROOT)")
+struct W3CJapaneseSuiteTests {
+    private var root: String? {
+        ProcessInfo.processInfo.environment["XMLCONF_ROOT"]
+    }
+
+    /// Cases this implementation knowingly fails.
+    private let knownFailures: Set<String> = []
+
+    private func manifest(in directory: String) throws -> [String] {
+        var source = try String(contentsOfFile: directory + "/japanese.xml", encoding: .utf8)
+        if source.hasPrefix("<?xml"), let declarationEnd = source.range(of: "?>") {
+            source = String(source[declarationEnd.upperBound...])
+        }
+        let document = try PureXML.parse("<TESTS>\(source)</TESTS>")
+        return try PureXML.xpath("//TEST", over: document).compactMap { selection in
+            selection.element?.attributes.first { $0.name.description == "URI" }?.value
+        }
+    }
+
+    /// Resolves sibling DTD files through PureXML's own byte decoder, since
+    /// the DTDs themselves are in the document's encoding.
+    private func resolver(directory: String) -> PureXML.Parsing.EntityResolver {
+        @Sendable func load(_ id: PureXML.Parsing.ExternalID) -> String? {
+            guard let data = try? Data(contentsOf: URL(fileURLWithPath: directory + "/" + id.systemID)) else {
+                return nil
+            }
+            return try? PureXML.Parsing.ByteDecoder.decode([UInt8](data))
+        }
+        return PureXML.Parsing.EntityResolver(
+            resolveEntity: { _, id in load(id) },
+            resolveExternalSubset: { id in load(id) },
+        )
+    }
+
+    @Test("Every japanese case decodes, parses, and validates cleanly")
+    func test_allCasesParse() throws {
+        guard let root else {
+            return // Opt-in: suite not present.
+        }
+        let directory = root + "/japanese"
+        var failures: [String: String] = [:]
+        for uri in try manifest(in: directory) where !knownFailures.contains(uri) {
+            do {
+                let data = try Data(contentsOf: URL(fileURLWithPath: directory + "/" + uri))
+                let source = try PureXML.Parsing.ByteDecoder.decode([UInt8](data))
+                let errors = try PureXML.validateAgainstInternalDTD(
+                    source,
+                    limits: .init(allowDoctype: true),
+                    resolver: resolver(directory: directory),
+                )
+                if let first = errors.first {
+                    failures[uri] = String(describing: first)
+                }
+            } catch {
+                failures[uri] = String(describing: error)
+            }
+        }
+        #expect(failures.isEmpty, "\(failures.count) japanese cases failed: \(failures)")
+    }
+}