From 5b5ee7b3fee39aa0857930a7aeff0a26d60bd23e Mon Sep 17 00:00:00 2001 From: Mihaela Mihaljevic Date: Tue, 9 Jun 2026 07:57:52 +0200 Subject: [PATCH] feat(parsing): EUC-JP decoding with vendored JIS X 0212 index (#99) --- CHANGELOG.md | 2 +- README.md | 2 +- Sources/Parsing/ByteDecoder.swift | 4 + Sources/Parsing/EUCJPDecoder.swift | 53 +++++ Sources/Parsing/InputEncoding.swift | 1 + Sources/Parsing/JIS0212Table1.swift | 307 +++++++++++++++++++++++++ Sources/Parsing/JIS0212Table2.swift | 307 +++++++++++++++++++++++++ Sources/Parsing/StreamingDecoder.swift | 28 +++ Tests/EncodingTests.swift | 6 + 9 files changed, 708 insertions(+), 2 deletions(-) create mode 100644 Sources/Parsing/EUCJPDecoder.swift create mode 100644 Sources/Parsing/JIS0212Table1.swift create mode 100644 Sources/Parsing/JIS0212Table2.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 166ddf2..f32abc7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,7 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added -- Shift-JIS decoding (#99, stage 1): the byte decoder gains a multi-byte path and the WHATWG Shift-JIS algorithm, decoding ASCII, half-width katakana, and the JIS X 0208 two-byte plane (its index vendored verbatim from the WHATWG `index-jis0208.txt`), for both the whole-buffer and streaming decoders, selected by the XML declaration's `encoding` name (`Shift_JIS`, `sjis`, `windows-31j`, ...). The remaining CJK encodings (EUC-JP/KR, GB2312/GBK) are tracked under the same issue. +- CJK multi-byte encodings (#99, in progress): the byte decoder gains a multi-byte path and the WHATWG **Shift-JIS** and **EUC-JP** algorithms, for both the whole-buffer and streaming decoders, selected by the XML declaration's `encoding` name. Shift-JIS covers ASCII, half-width katakana, and the JIS X 0208 plane; EUC-JP adds the `0x8E` half-width katakana and `0x8F` JIS X 0212 leads. The JIS X 0208 and 0212 indexes are vendored verbatim from the WHATWG `index-jis0208.txt`/`index-jis0212.txt`. The remaining CJK encodings (EUC-KR, GB2312/GBK) are tracked under the same issue. - HTML named character references: the complete WHATWG named character reference set (#84), all 2,125 semicolon-form references including the astral mathematical alphanumerics and the 93 two-codepoint references, vendored verbatim from the authoritative `entities.json` (replacing the previous 16-entry stopgap). Decoding stays longest-match with an optional trailing semicolon, on top of the existing numeric-reference validation. - HTML5 select content model (#83, stage 10): a `` closes the open one rather than nesting, and an ``/``/`