feat(emitting): CJK output encodings, closing output-encoding parity

Extend ByteEncoder to the multi-byte CJK encodings (Shift-JIS, EUC-JP,
EUC-KR, GBK, GB18030, Big5). Each inverse is built by running the existing,
tested whole-buffer decoder over every valid byte sequence and recording
the shortest sequence per scalar, so the encoder is by construction the
exact inverse of the decoder, with no new data or separate encode tables.
GB18030's astral range uses the four-byte pointer formula, so every scalar
encodes. Round-trip tests cover Japanese, Korean, Chinese, and a GB18030
astral character.

Closes #106

CHANGELOG.md

@@ -23,7 +23,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
-- Output encoding (#106, in progress): `Serializer.serialize(_:encoding:)` and `PureXML.serialize(_:encoding:)` now emit bytes in a chosen encoding (the libxml2 `xmlSaveToFilename(..., encoding)` model), not only UTF-8. A new `PureXML.Parsing.ByteEncoder` inverts the existing vendored to-Unicode tables (no new data) to cover the Unicode transformation formats and the single-byte legacy families (ISO-8859, Windows code pages, KOI8). The declaration carries the encoding's canonical name, a UTF-16/32 stream is preceded by a byte-order mark, and any scalar the target encoding cannot represent is written as a decimal numeric character reference, so the output always round-trips. CJK output encodings are tracked separately.
+- Output encoding (#106): `Serializer.serialize(_:encoding:)` and `PureXML.serialize(_:encoding:)` now emit bytes in a chosen encoding (the libxml2 `xmlSaveToFilename(..., encoding)` model), not only UTF-8. A new `PureXML.Parsing.ByteEncoder` covers the Unicode transformation formats, the single-byte legacy families (ISO-8859, Windows code pages, KOI8), and the multi-byte CJK encodings (Shift-JIS, EUC-JP, EUC-KR, GBK, GB18030, Big5). The CJK and single-byte inverses are built by running the existing, tested decoders over every valid byte sequence and recording the shortest sequence per scalar, so each encoder is by construction the exact inverse of its decoder (no new data, no separately maintained encode tables); GB18030's astral range uses the four-byte pointer formula. The declaration carries the encoding's canonical name, a UTF-16/32 stream is preceded by a byte-order mark, and any scalar the target encoding cannot represent is written as a decimal numeric character reference, so output always round-trips.
 - Big5 (#103): the byte decoder gains the WHATWG `big5` algorithm, for both the whole-buffer and streaming decoders, selected by the `big5` declaration name (and the `big5-hkscs` / `cn-big5` aliases). A lead in `0x81`-`0xFE` followed by a trail in `0x40`-`0x7E` or `0xA1`-`0xFE` resolves through the index, covering Traditional Chinese and the HKSCS extensions including the astral CJK ideographs (Extension B and the Compatibility Supplement). Four pointers decode to a base letter plus a combining mark (a two-scalar sequence). The 18,590-entry index is vendored verbatim from the WHATWG `index-big5.txt`. Big5 completes the major CJK families libxml2 reaches through iconv (Japanese, Korean, Simplified and Traditional Chinese).
 - Full GB18030 (#102): the byte decoder gains the GB18030 four-byte sequences on top of the GBK two-byte subset, for both the whole-buffer and streaming decoders, selected by the `gb18030` declaration name. A lead in `0x81`-`0xFE` followed by a digit opens a four-byte form whose pointer is mapped through the WHATWG `index-gb18030-ranges` linear ranges, reaching the supplementary planes (the `189000` range maps to `U+10000` and beyond). The 207 ranges are vendored verbatim from the WHATWG `index-gb18030-ranges.txt`. GB18030 is China's mandatory standard, supported in libxml2 through iconv.
 - CJK multi-byte encodings (#99, in progress): the byte decoder gains a multi-byte path and the WHATWG **Shift-JIS** and **EUC-JP** algorithms, for both the whole-buffer and streaming decoders, selected by the XML declaration's `encoding` name. Shift-JIS covers ASCII, half-width katakana, and the JIS X 0208 plane; EUC-JP adds the `0x8E` half-width katakana and `0x8F` JIS X 0212 leads. EUC-KR (CP949/UHC) decodes Hangul and Hanja through the WHATWG `index-euc-kr`, and GBK (which also covers GB2312) decodes Chinese through the two-byte range of `index-gb18030` with `0x80` as the euro sign. The JIS X 0208/0212, CP949, and GBK indexes are all vendored verbatim from the corresponding WHATWG `index-*.txt` files. This completes the CJK multi-byte encodings.

README.md

@@ -99,7 +99,7 @@ flowchart TB
 
 PureXML is a working, dependency-free XML library today: parse, emit, validate,
 query, and stream documents on macOS, Linux, Windows, and WASM. The test suite
-currently runs **773 tests in 118 suites** (`swift test`).
+currently runs **777 tests in 118 suites** (`swift test`).
 
 ### Shipped (libxml2-aligned surface)
 

Sources/Parsing/ByteEncoder.swift

@@ -12,7 +12,7 @@ public extension PureXML.Parsing {
             case .utf8, .utf16BigEndian, .utf16LittleEndian, .utf32BigEndian, .utf32LittleEndian:
                 true
             default:
-                ByteDecoder.singleByteMap(encoding) != nil
+                supportsCJK(encoding) || ByteDecoder.singleByteMap(encoding) != nil
             }
         }
 
@@ -26,7 +26,7 @@ public extension PureXML.Parsing {
             case .utf16LittleEndian: utf16(string, bigEndian: false)
             case .utf32BigEndian: utf32(string, bigEndian: true)
             case .utf32LittleEndian: utf32(string, bigEndian: false)
-            default: singleByte(string, encoding: encoding)
+            default: cjkEncode(string, as: encoding) ?? singleByte(string, encoding: encoding)
             }
         }
 
@@ -116,6 +116,8 @@ public extension PureXML.Parsing {
             .windows1253: "windows-1253", .windows1254: "windows-1254", .windows1255: "windows-1255",
             .windows1256: "windows-1256", .windows1257: "windows-1257", .windows1258: "windows-1258",
             .koi8r: "KOI8-R", .koi8u: "KOI8-U",
+            .shiftJIS: "Shift_JIS", .eucJP: "EUC-JP", .eucKR: "EUC-KR",
+            .gbk: "GBK", .gb18030: "GB18030", .big5: "Big5",
         ]
     }
 }

Sources/Parsing/CJKByteEncoder.swift

@@ -0,0 +1,126 @@
+extension PureXML.Parsing.ByteEncoder {
+    private typealias Decoder = PureXML.Parsing.ByteDecoder
+
+    /// The bytes of `string` in a multi-byte CJK encoding, or nil for a
+    /// non-CJK encoding. Each inverse is built by running the existing,
+    /// tested whole-buffer decoder over every valid byte sequence and
+    /// recording the shortest sequence per scalar, so the encoder is by
+    /// construction the exact inverse of the decoder.
+    static func cjkEncode(_ string: String, as encoding: PureXML.Parsing.InputEncoding) -> [UInt8]? {
+        let inverse: [UInt32: [UInt8]]
+        switch encoding {
+        case .shiftJIS: inverse = shiftJISInverse
+        case .eucJP: inverse = eucJPInverse
+        case .eucKR: inverse = eucKRInverse
+        case .gbk: inverse = gbkInverse
+        case .big5: inverse = big5Inverse
+        case .gb18030: return gb18030Encode(string)
+        default: return nil
+        }
+        return mapped(string, through: inverse)
+    }
+
+    static func supportsCJK(_ encoding: PureXML.Parsing.InputEncoding) -> Bool {
+        switch encoding {
+        case .shiftJIS, .eucJP, .eucKR, .gbk, .big5, .gb18030: true
+        default: false
+        }
+    }
+
+    private static func mapped(_ string: String, through inverse: [UInt32: [UInt8]]) -> [UInt8] {
+        var bytes: [UInt8] = []
+        for scalar in string.unicodeScalars {
+            if scalar.value < 0x80 {
+                bytes.append(UInt8(scalar.value))
+            } else if let sequence = inverse[scalar.value] {
+                bytes.append(contentsOf: sequence)
+            } else {
+                bytes.append(contentsOf: Array("&#\(scalar.value);".utf8))
+            }
+        }
+        return bytes
+    }
+
+    // MARK: GB18030 (two-byte plus the four-byte ranges, a full Unicode encoding)
+
+    private static func gb18030Encode(_ string: String) -> [UInt8] {
+        var bytes: [UInt8] = []
+        for scalar in string.unicodeScalars {
+            if scalar.value < 0x80 {
+                bytes.append(UInt8(scalar.value))
+            } else if let sequence = gbkInverse[scalar.value] {
+                bytes.append(contentsOf: sequence)
+            } else {
+                bytes.append(contentsOf: gb18030FourBytes(scalar.value))
+            }
+        }
+        return bytes
+    }
+
+    /// The four-byte GB18030 sequence for a scalar, the inverse of the decoder's
+    /// pointer-to-scalar range mapping.
+    private static func gb18030FourBytes(_ codepoint: UInt32) -> [UInt8] {
+        let pointers = Decoder.gb18030RangePointers
+        let codepoints = Decoder.gb18030RangeCodePoints
+        var low = 0
+        var high = codepoints.count - 1
+        var found = 0
+        while low <= high {
+            let mid = (low + high) / 2
+            if codepoints[mid] <= codepoint {
+                found = mid
+                low = mid + 1
+            } else {
+                high = mid - 1
+            }
+        }
+        let pointer = Int(pointers[found]) + Int(codepoint - codepoints[found])
+        let byte1 = pointer / 12600
+        var rest = pointer % 12600
+        let byte2 = rest / 1260
+        rest %= 1260
+        let byte3 = rest / 10
+        let byte4 = rest % 10
+        return [UInt8(byte1 + 0x81), UInt8(byte2 + 0x30), UInt8(byte3 + 0x81), UInt8(byte4 + 0x30)]
+    }
+
+    // MARK: Inverse tables, built once from the decoders
+
+    private static let shiftJISInverse = buildInverse(Decoder.ShiftJIS.decode, leads: 0x81 ... 0xFE)
+    private static let eucKRInverse = buildInverse(Decoder.EUCKR.decode, leads: 0x81 ... 0xFE)
+    private static let gbkInverse = buildInverse(Decoder.GBK.decode, leads: 0x81 ... 0xFE)
+    private static let big5Inverse = buildInverse(Decoder.Big5.decode, leads: 0x81 ... 0xFE)
+    private static let eucJPInverse = buildInverse(Decoder.EUCJP.decode, leads: 0x8E ... 0xFE, threeByteLead: 0x8F)
+
+    /// Enumerates every one-, two-, and (for EUC-JP) three-byte sequence the
+    /// decoder accepts and records the scalar it yields. The first (shortest,
+    /// lowest) sequence wins a scalar so the encoder stays deterministic.
+    private static func buildInverse(
+        _ decode: (ArraySlice<UInt8>) -> String,
+        leads: ClosedRange<Int>,
+        threeByteLead: UInt8? = nil,
+    ) -> [UInt32: [UInt8]] {
+        var map: [UInt32: [UInt8]] = [:]
+        func record(_ bytes: [UInt8]) {
+            let scalars = Array(decode(bytes[...]).unicodeScalars)
+            guard scalars.count == 1, scalars[0].value != 0xFFFD, scalars[0].value >= 0x80 else { return }
+            if map[scalars[0].value] == nil { map[scalars[0].value] = bytes }
+        }
+        for byte in 0x80 ... 0xFF {
+            record([UInt8(byte)])
+        }
+        for lead in leads {
+            for trail in 0x40 ... 0xFE {
+                record([UInt8(lead), UInt8(trail)])
+            }
+        }
+        if let threeByteLead {
+            for middle in 0xA1 ... 0xFE {
+                for trail in 0xA1 ... 0xFE {
+                    record([threeByteLead, UInt8(middle), UInt8(trail)])
+                }
+            }
+        }
+        return map
+    }
+}

Tests/OutputEncodingTests.swift

@@ -51,4 +51,38 @@ struct OutputEncodingTests {
         #expect(Array(bytes.prefix(2)) == [0xFE, 0xFF]) // BOM
         try #expect(rootText(PureXML.parse(bytes: bytes)) == "hi\u{4E2D}")
     }
+
+    /// Encodes the content, asserts it was truly encoded (no character-reference
+    /// fallback), then re-parses it back to the original text.
+    private func roundTrip(_ content: String, _ encoding: PureXML.Parsing.InputEncoding) throws {
+        let bytes = try serialized("<r>\(content)</r>", encoding)
+        #expect(!String(decoding: bytes, as: UTF8.self).contains("&#"))
+        try #expect(rootText(PureXML.parse(bytes: bytes)) == content)
+    }
+
+    @Test("Shift-JIS and EUC-JP round-trip Japanese")
+    func test_japanese() throws {
+        try roundTrip("\u{65E5}\u{672C}\u{FF71}", .shiftJIS) // 日本 + half-width katakana ｱ
+        try roundTrip("\u{65E5}\u{672C}", .eucJP)
+    }
+
+    @Test("EUC-KR round-trips Hangul")
+    func test_korean() throws {
+        try roundTrip("\u{AC00}\u{D55C}", .eucKR) // 가한
+    }
+
+    @Test("GBK and Big5 round-trip Chinese")
+    func test_chinese() throws {
+        try roundTrip("\u{4E2D}\u{6587}", .gbk) // 中文
+        try roundTrip("\u{4E2D}\u{6587}", .big5)
+    }
+
+    @Test("GB18030 round-trips a two-byte BMP char and a four-byte astral char")
+    func test_gb18030() throws {
+        try roundTrip("\u{4E2D}", .gb18030) // 中, two-byte
+        try roundTrip("\u{20000}", .gb18030) // 𠀀, four-byte astral
+        // The astral char must use the four-byte form, not a character reference.
+        let bytes = try serialized("<r>\u{20000}</r>", .gb18030)
+        try #expect(rootText(PureXML.parse(bytes: bytes)) == "\u{20000}")
+    }
 }