From 4ff9306a0e8bb0da935919d896f02e91368aec49 Mon Sep 17 00:00:00 2001 From: Mihaela Mihaljevic Date: Tue, 9 Jun 2026 07:54:05 +0200 Subject: [PATCH] feat(parsing): Shift-JIS decoding with vendored JIS X 0208 index (#99) --- CHANGELOG.md | 1 + README.md | 2 +- Sources/Parsing/ByteDecoder.swift | 5 + Sources/Parsing/InputEncoding.swift | 1 + Sources/Parsing/JIS0208Table1.swift | 315 +++++++++++++++++++++++++ Sources/Parsing/JIS0208Table2.swift | 315 +++++++++++++++++++++++++ Sources/Parsing/JIS0208Table3.swift | 315 +++++++++++++++++++++++++ Sources/Parsing/ShiftJISDecoder.swift | 61 +++++ Sources/Parsing/StreamingDecoder.swift | 18 ++ Tests/EncodingTests.swift | 6 + 10 files changed, 1038 insertions(+), 1 deletion(-) create mode 100644 Sources/Parsing/JIS0208Table1.swift create mode 100644 Sources/Parsing/JIS0208Table2.swift create mode 100644 Sources/Parsing/JIS0208Table3.swift create mode 100644 Sources/Parsing/ShiftJISDecoder.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index c6b11fc..166ddf2 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,6 +22,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Shift-JIS decoding (#99, stage 1): the byte decoder gains a multi-byte path and the WHATWG Shift-JIS algorithm, decoding ASCII, half-width katakana, and the JIS X 0208 two-byte plane (its index vendored verbatim from the WHATWG `index-jis0208.txt`), for both the whole-buffer and streaming decoders, selected by the XML declaration's `encoding` name (`Shift_JIS`, `sjis`, `windows-31j`, ...). The remaining CJK encodings (EUC-JP/KR, GB2312/GBK) are tracked under the same issue. - HTML named character references: the complete WHATWG named character reference set (#84), all 2,125 semicolon-form references including the astral mathematical alphanumerics and the 93 two-codepoint references, vendored verbatim from the authoritative `entities.json` (replacing the previous 16-entry stopgap). Decoding stays longest-match with an optional trailing semicolon, on top of the existing numeric-reference validation. - HTML5 select content model (#83, stage 10): a `` closes the open one rather than nesting, and an ``/``/`