Commit Graph

73 Commits

Author SHA1 Message Date
7d22194614
perf(parse): build the TreeNode tree directly from the event stream (#295)
parseTree ran TreeNode(parse(...)), which allocates two full trees: the parser
builds an immutable value Node tree, then TreeNode.init(_:) walks it to build
the mutable reference tree. The parsed tree is the deliverable, so building the
value tree first is pure overhead.

Parser.buildTree is the direct counterpart to build: the same iterative event
loop with the same element stack and the same well-formedness guards (an
unmatched end tag throws unexpectedEndOfInput; a document with no element throws
emptyDocument), but it accumulates TreeNode children per frame and adopts them
when an element closes, so the tree is allocated once. The produced tree is
identical: same kinds, names, attributes, values, child order, and parent links.

Measured parse ~8% faster on the 20k-item generated corpus via a same-state A/B
(6.1x to ~5.5x slower than libxml2). Verified by ParseTreeDirectBuildTests (a
structural differential against the convert-from-Node path, plus throw-parity
and a serialized round-trip), the full suite and the W3C XML conformance,
RELAX NG, and Xalan corpora (which all build trees through parseTree), the WASM
build, and an adversarial equivalence review.
2026-06-25 14:15:57 +02:00
6be96e51ce
perf(parse): scan for entity-decoder markers at the byte level (#294)
EntityExpander.expand located the next '&', '<', and ';' with String.firstIndex
over graphemes, so every literal byte of a run was decoded into a grapheme just
to look for a one-byte marker. All three markers are ASCII, so a UTF-8 index is
a valid string index: scan over raw.utf8 instead, and the found index still
subscripts the string directly (the marker byte is never a continuation byte,
so its index lands on a scalar and grapheme boundary).

The decoded text is unchanged, including runs where the marker follows
multibyte UTF-8: a precomposed letter, a combining sequence, or an astral
character all leave the marker on a grapheme boundary.

Measured parse ~5.5% faster on the 20k-item generated corpus via a same-state
A/B (6.5x to ~6.1x slower than libxml2). Verified by EntityDecodeBulkTests
(extended with four multibyte-boundary cases), the W3C XML conformance corpora,
the Xalan XSLT gold-output corpus, the full suite, and the WASM build.
2026-06-25 13:37:27 +02:00
7a5761b64d
perf(parse): split the name prefix from a scanned colon offset (#292)
QualifiedName(_:) ran firstIndex(of: ":") over the whole name, grapheme by
grapheme, on every element and attribute. Most names have no prefix, so the
common case scanned the entire name to find no colon at all.

The byte scanner already visits every byte of an all-ASCII name, so have
takeASCIIName record the offset of the first colon (one comparison per byte),
and split at that known offset in QualifiedName(ascii:colonOffset:). The
unprefixed name (no colon) is then built with no scan; a prefixed name splits
at the recorded offset. The names are identical to the firstIndex path for
every colon placement: none, leading (:local), trailing (prefix:), middle,
doubled (::), and multiple (first colon wins). The hint is confined to the
all-ASCII fast path; non-ASCII names and every other construction site keep
QualifiedName(_:).

Measured parse ~12% faster on the 20k-item generated corpus via a same-state
A/B (8.5x to ~7.7x slower than libxml2). Verified by QualifiedNameColonHintTests
(colon-placement differential), the W3C XML conformance corpora, the Xalan XSLT
gold-output corpus, the full suite, the WASM build, and an adversarial
equivalence review.
2026-06-25 12:23:53 +02:00
2cca4912ff
perf(parse): bulk-copy literal runs in the entity decoder (#291)
EntityExpander.expand rebuilt every reference-bearing text run one character at
a time: result.append(character) per character, with a per-character charge
against the amplification budget. A text node holding a single &amp; was copied
grapheme by grapheme, which is the bulk of the per-text-node cost in a document
whose content uses entities.

Scan to the next '&' or '<' and append the literal run in one operation,
charging the budget by the run length. The decoded text, the CDATA-verbatim
shielding, and the billion-laughs budget enforcement are unchanged: the same
overrun throws at the same input, since charging a run of length L against
budget B fails exactly when L > B, the same condition the per-character guard
enforced.

Also return early from recordReferenceContentFindings when there are no DTD
content models (the common no-DTD case): it otherwise scanned each text run for
'&' and rendered the enclosing element's qualified name before the model lookup
that would fail anyway.

Measured parse ~5% faster on the 20k-item generated corpus via a same-state A/B
(9.06x to ~8.5x slower than libxml2). Verified by the W3C XML conformance
corpora, the new EntityDecodeBulkTests six-case decode differential, the full
suite, the Xalan XSLT gold-output corpus, and the WASM build, with an
adversarial equivalence review of the rewrite.
2026-06-25 11:45:02 +02:00
38caec7098
perf(parsing): skip reference decoding for ampersand-free attribute values (#278)
The attribute-value scanner ran the entity decoder on every value, and the
decoder re-scanned the whole value for '&' before doing nothing when none
was present. The byte-run scanner already visits every value byte, so it
now notes any '&' and an ampersand-free value (the common case) returns its
normalized run directly. Measured against a same-machine-state baseline,
every run came in below every baseline run, ~4% faster parse on the
20k-item generated corpus.

Move the bulk byte-run scanners (contentRunBytes, attributeRunBytes) into
ReaderByteRuns.swift and centralize their cursor mutation through a single
advanceByteRun method, so Reader's position fields stay write-encapsulated
in Reader.swift while both files stay under the length cap. The byte and
Character paths are held to identical decoded values by the differential
oracle, extended with ampersand-free, entity, and character-reference
attribute cases. No spec behavior changed.
2026-06-25 00:17:31 +02:00
ead838f83c
perf(parsing): dispatch element-content markup on the byte after '<' (#276)
The element-content markup dispatch ran up to five literal string
comparisons (</, <!--, <![CDATA[, <?, <!DOCTYPE) before the start-tag
branch won, a per-element cost the parse profile indicted as
Reader.matchesBytes. Branch instead on the single byte after '<' on the
byte fast path: '/' end tag, '?' processing instruction, '!' a
declaration, otherwise a start tag.

The Character path keeps the equivalent matches()-based dispatch for a
streaming source or spliced entity text, and a '<!' that matches none of
comment/CDATA/DOCTYPE still falls through to the start-tag scanner with
the same expectedName error. Measured parse ~9.5x -> ~9.0x slower than
libxml2 on the 20k-item corpus; the byte and Character paths are held
identical by ParserPathDifferentialTests over its adversarial corpus,
and the Xalan, W3C, and RELAX NG suites stay green. No spec behavior
changed.
2026-06-24 19:25:10 +02:00
2f8ca9b289
perf(parsing): byte-level scanning fast paths halve parse time vs libxml2 (#275)
Keep the reader on its byte fast path through dispatch and name/attribute
scanning so plain-ASCII documents avoid the per-Character String machinery
(String.Iterator.next, string comparison, small-string materialization,
bridge-object ARC) that a profile indicted as the parse bottleneck.

The root cause was that a single buffered lookahead Character defeated every
existing byte path, each of which requires an empty buffer, and the content,
markup, and attribute dispatch each peeked a Character before scanning. Add a
non-buffering peekByte for dispatch, a byte-mode skipSpace, an all-ASCII
takeASCIIName, and an attributeRunBytes bulk scanner (with the 3.3.3
whitespace-to-space transform, deferring any carriage return to the Character
path so CRLF is not double-counted). Detect a text run's closing '<' via
peekByte instead of buffering it, which had cascaded the next element's whole
dispatch and name scan onto the slow path. Skip reference decoding for
ampersand-free text. The Character path remains the exact fallback for a
streaming source, spliced entity text, and any non-ASCII byte; the two paths
share scanContentMarkup so they cannot diverge.

Measured parse 20.1x -> 9.5x slower than libxml2 on the 20k-item generated
corpus, roughly a 53 percent wall-clock reduction. A differential oracle pins
the byte and Character paths to identical event streams and error marks over
an adversarial corpus, alongside boundary tests. Verified against the full
local suite, the Apache Xalan XSLT gold-output corpus, the W3C XML conformance
corpora, the RELAX NG spec suite, and the WASM build. No spec behavior changed.
2026-06-24 18:46:58 +02:00
8d30010e0d docs: align stale figures with code and the enforced XSTS gate
- README, roadmap: XSTS status quad and gates table now read the enforced
  0/43/0/31 (Tests/XSTSSuiteTests.swift); test count 986/169 to 1448/258;
  encoding list and the remaining-work section reflect shipped reality;
  check-all expansion lists the validation and changelog gates
- GEMINI.md, onboarding-antigravity.md: valid-schemas-rejected 1 to 0 and the
  pinned baselines updated to current
- SECURITY.md: complete the reporting address
- Parsing.InputEncoding: doc comment lists the full shipped encoding set
- Schema.XSDParserUnion: guard the localName interpolation with ?? "" to match
  the same line and silence an optional-interpolation warning (internal dedup
  key, behavior preserving)
- CHANGELOG: Unreleased entries for the above
2026-06-19 23:32:04 +02:00
9e345ea0fe feat(xpath,parsing): public streaming surface, linear document-order at scale
EventReader.next(), both initializers, and the Event alias are public
(an internal typealias in a split file had been shadowing the public
Parsing.Event); a new PureXMLPublicAPITests target compiles without
testable import so facade gaps fail CI. The large-corpus benchmarks
exposed document-order sorting as quadratic on flat fan-outs; the fix's
first attempt was itself worse (an eager sibling table built per
predicate evaluation), caught by sampling the live run, and replaced
with a lazy table: single-node sets never touch order keys and a
parent's table builds on its second lookup. XPath at 42 MB drops
27.1 s to 1.57 s (190x to 11.7x); at 210 MB PureXML completes in 9.4 s
the query libxml2 refuses at its 10M node-set cap (#143 tracks our
configurable budget). Benchmark harness: size series to 210 MB, CORPUS
mode for real files (NASA, SwissProt measured), SAX stage,
refusal-aware table, NULL-safe C driver. All conformance suites,
Linux, and WASM verified. Closes #142.
2026-06-11 06:06:31 +02:00
72eaeeb439 perf(parsing): byte-backed Reader and bulk content runs
String input copies its UTF-8 once into owned storage (an unsafe
pointer encapsulated in a class lifetime, never escaping the parser)
and scalars decode straight off the pointer, eliminating the
per-character closure chain; ASCII literals match and consume at the
byte level; and plain-ASCII character data scans as a single byte run
with one String built per run. Parse drops 32 percent cumulative
(0.49 s to 0.34 s on the 4.2 MB corpus, libxml2 ratio 30x to 20x).
The W3C suite caught one boundary (a run starting with '>' could
swallow the ]]> check after the character path consumed ]]); the run
scanner leaves a leading '>' to the character path, pinned as a unit
regression. All conformance suites, Linux, and WASM verified.
2026-06-10 18:48:57 +02:00
0ce59c826b perf(parsing): allocation-free Reader.matches(), first measured pass
matches() ran on every scan decision and materialized an Array of the
literal per call, the hottest comparison in the parse profile. It now
walks the literal directly. Parse time on the 4.2 MB benchmark corpus
drops 16 percent (0.49 s to 0.41 s, libxml2 ratio 30x to 26x); profile
findings and the optimization log recorded in docs/benchmarks.md.
2026-06-10 18:21:08 +02:00
d0175d5ca0 refactor(validation): located structured validity findings
DocumentType.validityFindings carries ValidityFinding values (reason
plus subject) instead of strings, DTDSchema.declarationErrors holds
ready ValidationErrors, and every declaration-level finding renders at
its declaration's coding path (x, r/@a) instead of at the document
root, matching the XSD consistency-errors precedent. Texts unchanged.
2026-06-10 10:45:17 +02:00
2f1b936e36 feat(parsing): scalar-level lexing, xmlconf archive fully conformant
The reader buffers one Unicode scalar per element, splitting
multi-scalar graphemes from any source, so combining marks cannot merge
with delimiters and the Fifth Edition's scalar-defined name productions
classify correctly; the replacement-text, content-model, and ATTLIST
sub-scanners and the entity-literal validator follow, while grapheme
clusters reassemble wherever scanned text accumulates. Clears the last
eduni baseline, grows the asserted IBM pre-5e class to 300, and with
the xmltest manifest's own EDITION exclusions for 140/141 empties the
xmltest baseline: every xmlconf case applicable to a Fifth Edition,
namespace-aware processor passes.

Closes #135
2026-06-10 08:40:26 +02:00
98792a1715 feat(parsing): strict internal-subset profile, xmltest 185 of 186
Limits(strictInternalSubset: true) rejects conditional sections and
parameter-entity references inside markup declarations in the internal
subset, the two feature classes XML 1.0 reserves for the external
subset. DeclSep references and quoted attribute defaults (AttValue)
stay legal; defaults unchanged. The xmltest runner opts in, leaving
case 141 (pre-Fifth-Edition name classes) as its only deviation.

Closes #128
2026-06-10 08:29:54 +02:00
3d56d19cbd feat(parsing): per-entity base-URI tracking for nested external entities
A relative system identifier declared inside an external entity
resolves against that entity's own URI (RFC 3986, the C14N resolver):
ExternalID gains base and resolvedSystemID, the scanner threads the
current entity's URI through external-subset and PE scans, and
resolvers receive the merged identifier. Clears eduni E18; the eduni
baseline now holds only the grapheme-lexing class (#135).

Closes #138
2026-06-10 08:24:17 +02:00
2c375ac74b feat(parsing): fatal encoding-declaration contradictions, E38 version rule
A declared 16/32-bit encoding over BOM-less 8-bit bytes is rejected; a
BOM followed by a declaration outside its family is rejected, reading
the declaration through the BOM's own byte pattern; and an external
entity whose text declaration names a higher XML version than the
document is refused (errata E38). The eduni not-wf baseline is empty.
Reader and decoder helpers split for the length caps.

Closes #137
2026-06-10 08:16:08 +02:00
4b03c45434 feat(parsing): Namespaces 1.0 constraint enforcement
The namespace layer rejects malformed qualified names, declarations of
the xmlns prefix, misbindings of the xml prefix and either reserved
namespace name (including as the default namespace), and prefix
undeclaration; attributes must be distinct by expanded name; PI targets
and entity/notation names are NCNames; and DTD-typed xmlns values bind
their normalized form. Clears all 18 namespace cases from the eduni
baseline; xmltest valid/sa/012 joins the documented deviations per the
suite's own NAMESPACE='no' flag.

Closes #136
2026-06-10 08:06:45 +02:00
6861cd5d62 feat(validation): eduni errata behaviors, 3.3.3 attribute normalization
The eduni sections run with exact classified baselines. Fixes: 3.3.3
CDATA attribute-value normalization (literal whitespace to space before
reference decoding; char-ref whitespace survives; tokenized collapse on
SPACE only); production-68 lenient undeclared entities in content and
attribute values (validity finding, reference kept literal, nothing
fetched); errata E15 (references/comments/PIs/whitespace in EMPTY,
char-ref whitespace as data in element content, double-escape shape
included); errata E2 duplicate enumeration tokens; errata E14
declarations completed inside PE replacements (VC: Proper
Declaration/PE Nesting); NCName ID values; namespace declarations
subject to the every-attribute-declared rule. Reader validity machinery
split to EventReaderValidity for caps.

Closes #127
2026-06-10 07:50:47 +02:00
c084d01d7c feat(parsing): IBM section clean, entity-declared WFC/VC split, declaration recursion
The IBM section (1094 cases) runs clean to two documented classes:
279 pre-Fifth-Edition character-class cases (P85-89, count asserted
exactly) and 2 PE-in-internal-subset feature cases. Real gaps closed:
a document needs a root element (production 1); entity references in
literals must be Names or character references; WFC: No Recursion
checked at declaration over the general/parameter reference graph;
undeclared PE references split into WFC (standalone, or no external
declarations possible) versus deferred validity finding, also for
attribute-default references; refused external PEs are neither; and
groups split across PE replacements report VC: Proper Group/PE
Nesting. Entity machinery split to DoctypeScannerEntities for caps.

Closes #125
2026-06-10 07:26:50 +02:00
b27b015f78 feat(validation): standalone validity constraints and entity WFC
The scanner records declaration provenance (internal-subset entities,
element models, attribute lists); the validator reports the three
standalone='yes' dependencies on the external subset: a default that
would be supplied, a non-CDATA normalization that would change a value,
and whitespace in externally-declared element content. The parser hides
externally-declared entities from standalone documents, so referencing
one reports as undeclared (WFC: Entity Declared, 2.9). The Sun section
is fully clean: not-wf 56/56, valid 28/28, invalid 74/74, no baseline.

Closes #134
Closes #124
2026-06-10 07:13:30 +02:00
89cb299201 feat(validation): DTD declaration-level validity constraints
VC Root Element Type (root matches the DOCTYPE name), VC Unique
Element Type Declaration, VC No Duplicate Types in mixed content, VC
One ID per Element Type, VC Notation Attributes and Notation Declared,
VC Attribute Default Legal plus ID Attribute Default, lexical Name
checks for the ID family, and CDATA as character data in element
content. Strict mode adds VC Attribute Value Type (every attribute
declared; namespace declarations exempt). Declaration findings are
computed once at schema build and reported at the document root. Sun
invalid coverage moves from 43/74 to 62/74; the twelve remaining are
all the standalone='yes' family.
2026-06-10 07:05:14 +02:00
497e2460a3 feat(parsing): splice entity replacement markup into content
A general entity whose replacement text contains markup (directly or
transitively) is reparsed as content at the reference (4.4.2 Included):
the replacement is WFC-validated, cycle-checked, and budgeted, then
pushed back into the reader stream, so its elements become elements in
the tree and the pull-event stream rather than escaped character data.
Character references in internal entity literals now expand at
declaration time for general entities too (4.4.5), making char-ref
markup splice and the Appendix D double escape render as a literal
ampersand; the reference-time WFC validates remaining references
instead of re-expanding. Text-only entities keep the coalescing
single-pass path. Sun valid is 28/28.

Closes #133
2026-06-10 06:55:40 +02:00
82e70679ef feat(validation): 3.3.3 attribute normalization, 4.3.1 text declarations, 4.4.5 PE char-refs
Tokenized (non-CDATA) attribute values normalize before validity
checks, so enumerations, NOTATION lists, and the ID/IDREF table see
collapsed values. External parsed entities lose their leading text
declaration (validated: version optional, encoding required, never
standalone) instead of carrying it into replacement text. Character
references in PE literals expand at declaration time, making the
Appendix D '&#37;zz;' example parse. A PI target inside the DTD subset
must be separated from its data. Sun baselines tighten to not-wf 55/56
and valid 26/28; the two remaining valid failures share one cause
(replacement text with markup is included as character data, filed
separately).
2026-06-10 06:43:25 +02:00
5671fb3ce1 feat(parsing): strict DTD subset grammar, oasis section of xmlconf fully clean
The OASIS/NIST section of the W3C conformance suite runs via a new
XMLCONF_ROOT-gated runner driven by the section's own manifest (parsed
with PureXML): 247/247 not-wf rejected, all 100 well-formed cases parse,
zero deviations. Closing its 24 gaps makes the DTD subset grammar
strict: exact PEReference shape, strict declaration names and keywords,
NDATA and notation tails, no bare '%' in entity values, case-sensitive
balanced conditional sections, junk between declarations rejected,
external-subset violations now propagate (with text-declaration
support, production 77), and a PI target must be separated from its
data. The xmltest baseline tightens to 181/186: CDATA in the internal
subset (107) is now rejected, five documented deviations remain.

Closes #123
2026-06-10 06:29:13 +02:00
32d42410b9 feat(parsing): entity replacement-text well-formedness, fifth suite pass
Implement the replacement-text constraint at the point of reference: when a
general entity is included, its replacement (character references expanded
as at declaration time) must reparse as balanced content in isolation, via a
new EntityReplacementGrammar mini-parser. This rejects a bare or incomplete
reference ('&', '&#9' unterminated), tags spanning the entity boundary
('</foo><foo>'), a raw '<' or incomplete reference inside an embedded
attribute value, '<' followed by a non-name character, and a reserved-target
PI in the replacement, exactly the suite's semantics, including that an
UNREFERENCED entity may carry any value and that only the first declaration
of a name binds. Attribute-list defaults must reference already-declared
internal (not external/unparsed/forward-declared) entities with a
non-recursive chain. A CDATA section inside replacement text now shields its
content from reference expansion, fixing the last valid-side case.

W3C xmltest final standing: valid/sa 119/119, not-wf/sa 180/186, with the
six remaining deviations all deliberate and documented in the baseline:
internal-subset conditional/CDATA sections (063, 107) and parameter-entity
references (160-162) are features this package supports, and 141 expects the
1998 suite's name-character classes where this package implements XML 1.0
Fifth Edition's. Zero false positives across all 846 tests at every step.

Closes #120
2026-06-10 05:47:42 +02:00
dd0ff5d086 feat(parsing): fourth W3C-suite strictness pass, declaration grammar
Make the DTD declaration grammar strict: <!ENTITY> requires its whitespace
(after the keyword, around %, after the name), a real name (not #DEFAULT),
nothing but '>' after the value, NDATA preceded by whitespace and forbidden
on parameter entities; entity-value literals are validated (XML Chars only,
every & a complete reference); SYSTEM/PUBLIC external identifiers parse
strictly (PUBLIC needs both literals for entities and the DOCTYPE, public-only
stays legal for notations) with PubidChar validation; <!ATTLIST> bodies are
validated against XML 1.0 productions 52-60 via a new DTDAttListGrammar
(spacing, attribute types, enumerations, quoted defaults); the reserved xml
PI target is rejected inside the subset. Also: strict UTF-8 decoding (no
silent replacement of malformed sequences), XML Char checks in attribute
values and CDATA, the declaration's version/encoding value grammar, and
required whitespace between pseudo-attributes. The strict machinery lives in
DoctypeScannerStrict/DTDAttListGrammar to stay under the length caps.

W3C xmltest: not-wf/sa 162/186 rejected (from 129; 80 at first contact),
valid/sa 118/119, zero false positives across all 846 tests at every step.
The remaining 24-case baseline is exactly: the entity replacement-text class
(19) and the deliberate internal-subset extensions this package supports as
features (conditional sections, PEs in the internal subset: 5).

Part of #120, #105
2026-06-10 05:34:11 +02:00
805993e656 feat(parsing): strict <!ELEMENT> content-model grammar
Third W3C-suite strictness pass: DTDContentModelGrammar validates each
element declaration's content specification against the XML 1.0 grammar
(productions 46-51) at DTD scan time, EMPTY/ANY, Mixed (with #PCDATA first,
names only, the trailing * required with alternatives, no nesting), and
children (one connector per group, quantifiers bound directly to their
particle, balanced parentheses, no empty groups, no SGML leftovers like
'- -', 'O O', '&' connectors, or inclusion/exclusion suffixes). The element
declaration also requires whitespace after <!ELEMENT and after the name. The
lenient ContentModelParser still drives validation behavior; this is a
well-formedness gate. not-wf/sa: 129/186 rejected (from 109); valid/sa
unchanged; zero false positives across all 846 suite tests; baseline rebased
to the exact 57 remaining (declaration-grammar details and the entity
replacement-text class).

Part of #120, #105
2026-06-10 05:20:33 +02:00
848c7002c7 feat(parsing): second W3C-suite strictness pass
Character references now require lowercase 'x' for hex and a referenced code
point inside the XML Char production (rejecting &#0;, &#31;, &#xFFFF;,
surrogates, and &#X58;). The reserved 'xml' processing-instruction target
(any case) is legal only as the declaration at the document's very first
bytes: whitespace or a comment before <?xml ...?>, a declaration inside
content or after the root, and case variants like <?XML?> are all rejected.
Comments and PI data are checked for the XML Char production like content.
not-wf/sa: 109/186 rejected (from 93); valid/sa unchanged at 118/119; all
846 suite tests green, baseline rebased to the exact 77 remaining (the
internal-subset grammar and entity replacement-text classes on #120).

Part of #120, #105
2026-06-10 05:16:05 +02:00
4ed8e9518a feat(parsing): first W3C-suite strictness pass; opt-in xmltest runner
Add five well-formedness checks the W3C xmltest suite surfaced: '--' inside
comments (also rejecting the ---> ending), raw '<' in attribute values, the
literal ]]> sequence in character data, attributes not separated by
whitespace, and characters outside the XML Char production in content, each
a located ParseError. The attribute scanners move to EventReaderAttributes
to stay under the type-body cap.

Add the opt-in W3C xmltest runner (XMLTS_ROOT-gated; the suite is never
vendored, James Clark's license permits redistribution only as the unmodified
archive; Foundation is used in this one test file for file access only).
Baseline: valid/sa 118/119, not-wf/sa 93/186 rejected (up from 80), the
remaining 93 carried as an exact known-deviations baseline so progress and
regressions are both visible. The two remaining leniency classes (internal-
subset DTD grammar, entity replacement-text well-formedness) are #120.

Part of #105 (Tier 2)
2026-06-10 04:59:39 +02:00
4130d905c8 refactor(parsing): drop slice copies; document the expansion bounds
The whole-buffer ISO-2022-JP, GB18030, and EUC-TW decoders index the input
ArraySlice directly instead of materializing an Array first (one avoidable
O(n) allocation per decode each); declaredEncoding's subsequence scan uses
elementsEqual instead of allocating an Array per candidate offset. Document
the two analysis-flagged bounds: entity expansion is doubly bounded (the
character budget caps output; the visiting set caps recursion depth at the
number of declared entities), and the RELAX NG derivative engine carries no
memoization by design (the simplifying constructors keep real grammars
small; adversarial nested-choice grammars can grow the residual, so treat
untrusted schemas accordingly). Behavior unchanged, all suites green.

Closes #118
2026-06-10 04:01:16 +02:00
56e889f790 feat(parsing): EUC-TW (CNS 11643) decoding, closing #108
Add EUC-TW for the whole-buffer and streaming decoders: ASCII, a two-byte
plane-1 pair, and a 0x8E four-byte plane selector plus a pair. CNS planes 1
and 2, which carry essentially all real-world EUC-TW text, are vendored from
the ICU euc-tw-2014 mapping (private-use fallbacks dropped); planes 3-15 are a
documented follow-up. Tested for plane 1 (two-byte and four-byte forms), plane
2, and streaming. The streaming method moved to its own file to stay under the
type-body cap.

Closes #108
2026-06-09 17:44:22 +02:00
191d2245de feat(parsing): streaming honors the declared encoding; streaming ISO-2022-JP
The streaming decoder previously selected an encoding only from a byte-order
mark or the UTF sniff, so its single-byte and CJK decode paths were latent. It
now reads the encoding from the XML declaration (the libxml2 behavior) when
there is no BOM, activating those paths over an incremental byte source. Add
the streaming form of ISO-2022-JP, carrying the stateful mode across reads
(extracted to its own file to stay under the type-body cap). A test asserts
streaming agrees with whole-buffer decode for a single-byte (ISO-8859-1), a
CJK (Shift_JIS), and an ISO-2022-JP document.

Part of #108, #105
2026-06-09 17:33:52 +02:00
1e815d6309 feat(parsing): ISO-2022-JP decoding
Add the WHATWG iso-2022-jp decoder, the first stateful encoding: escape
sequences switch between ASCII, JIS X 0201 Roman (yen, overline), JIS X 0201
katakana, and the JIS X 0208 plane, resolved through the shared jis0208 table
(no new data). Tested across all four modes, with the JIS X 0208 kanji
cross-checked against the EUC-JP decoder (same table, same pointer). EUC-TW and
the streaming form of ISO-2022-JP remain.

Part of #108, #105
2026-06-09 10:55:34 +02:00
b98c2ff41f feat(emitting): CJK output encodings, closing output-encoding parity
Extend ByteEncoder to the multi-byte CJK encodings (Shift-JIS, EUC-JP,
EUC-KR, GBK, GB18030, Big5). Each inverse is built by running the existing,
tested whole-buffer decoder over every valid byte sequence and recording
the shortest sequence per scalar, so the encoder is by construction the
exact inverse of the decoder, with no new data or separate encode tables.
GB18030's astral range uses the four-byte pointer formula, so every scalar
encodes. Round-trip tests cover Japanese, Korean, Chinese, and a GB18030
astral character.

Closes #106
2026-06-09 09:41:28 +02:00
e62995a7a8 feat(emitting): output encoding for UTF and single-byte families
Add PureXML.Parsing.ByteEncoder, the inverse of ByteDecoder, and a
Serializer.serialize(_:encoding:) / PureXML.serialize(_:encoding:) that
emits bytes in a chosen encoding (the libxml2 save-with-encoding model).
Covers the Unicode transformation formats and the single-byte legacy
families, built by inverting the existing vendored to-Unicode tables, so
no new data is introduced. The declaration carries the canonical encoding
name, a UTF-16/32 stream is preceded by a byte-order mark, and any
unrepresentable scalar becomes a decimal numeric character reference, so
output round-trips. CJK output encodings remain (tracked on #106).

Part of #106, #105
2026-06-09 09:32:39 +02:00
74acad4628 feat(parsing): Big5 (Traditional Chinese / HKSCS)
Add the WHATWG big5 decoder over the vendored index-big5 (18,590 entries),
for both the whole-buffer and streaming decoders. A lead in 0x81-0xFE plus
a trail in 0x40-0x7E or 0xA1-0xFE resolves through the index, covering
Traditional Chinese and the HKSCS extensions including the astral CJK
ideographs. Four pointers decode to a base letter plus a combining mark.
Selected by the big5 / big5-hkscs declaration names.

Closes #103
2026-06-09 08:30:57 +02:00
2b2c1fb692 feat(parsing): full GB18030 four-byte sequences
Extend the GBK two-byte subset to full GB18030: a lead byte followed by
a digit opens a four-byte form whose pointer is mapped through the WHATWG
index-gb18030-ranges linear ranges, reaching the supplementary planes.
The 207 ranges are vendored verbatim. Wired into the whole-buffer and
streaming decoders and the encoding-name table (gb18030/gb-18030).

Closes #102
2026-06-09 08:24:00 +02:00
b688952cef feat(parsing): GBK/GB2312 decoding completes the CJK encodings (#99) 2026-06-09 08:08:52 +02:00
7f2991a90b Merge #99 (stage 3): EUC-KR 2026-06-09 08:03:02 +02:00
d8b2db4943 feat(parsing): EUC-KR (CP949) decoding with vendored index (#99) 2026-06-09 08:02:14 +02:00
5b5ee7b3fe feat(parsing): EUC-JP decoding with vendored JIS X 0212 index (#99) 2026-06-09 07:57:52 +02:00
4ff9306a0e feat(parsing): Shift-JIS decoding with vendored JIS X 0208 index (#99) 2026-06-09 07:54:05 +02:00
3f45dacf62 feat(parsing): Windows-1255/1256/1258, KOI8-U complete the single-byte set (#97) 2026-06-09 07:29:25 +02:00
5646e81b94 feat(parsing): Windows-1250/1251/1253/1257, KOI8-R, ISO-8859-11 (#97) 2026-06-09 07:25:18 +02:00
3f293ec1e7 feat(parsing): ISO-8859-6/8/10/14/16 vendored from Unicode mappings (#97) 2026-06-09 07:19:50 +02:00
43dc7b00c4 feat(parsing): ISO-8859-3/4/7/13 vendored from Unicode mappings (#97) 2026-06-09 07:15:35 +02:00
5f35621ae6 feat(parsing): ISO-8859-2 (Latin-2) vendored from Unicode mapping (#97) 2026-06-09 07:08:19 +02:00
a005def333 feat(parsing): Windows-1254 (Turkish) encoding (#97) 2026-06-09 07:02:13 +02:00
94890172f2 feat(parsing): single-byte encoding framework + ISO-8859-5/9/15 (#97) 2026-06-09 06:57:21 +02:00
5351a5c777 feat(parsing): line-ending normalization and XML 1.1 line terminators
Normalize line endings in the Reader as the spec requires, which was not
done before:

- CR and CRLF fold to a single line feed (XML 1.0), so every downstream
  scan sees line feeds only.
- When the declaration names version 1.1, NEL (U+0085) and LINE SEPARATOR
  (U+2028), and a CR followed by NEL, also fold to a line feed; in 1.0
  those characters are preserved as data.

The XML 1.1 name-character productions are identical to XML 1.0 Fifth
Edition, which the character classifier already implements, so names need
no change. The 1.1 character range (C0 controls permitted only as
references) is not yet enforced.

Closes #96
2026-06-08 15:07:21 +02:00