* perf(parsing): memoize the markup-producing entity check (billion-laughs)
splitAtMarkupEntity / producesMarkup recursed into every reference of
every replacement with no memoization, so a nested entity chain was
traversed O(references^depth) times: a classic eight-level billion-laughs
took ~18s of CPU to reject. The check is now memoized per entity name (the
property depends only on the fixed entity graph), so it is O(entities). The
same document now rejects in ~0.6s, bounded by the expansion budget. No
change to which documents are accepted; cyclic entities still rejected.
Found by adversarial probing; closes the no-markup case of #339.
* perf(parsing): three-colour DFS for the No-Recursion check
checkRecursion validated the entity reference graph with only an
in-progress (grey) set and no record of fully-verified entities, so a
billion-laughs graph was re-walked O(references^depth) times: a
markup-producing eight-level document took ~31s to reject. It is now a
three-colour DFS: an entity is added to a black set only after its whole
reachable subgraph returns acyclic, so it can never reach a chain node
that reaches it, and skipping it on revisit is sound. The walk becomes one
pass over the distinct edges (~31s -> ~11s for that document). Cyclic
entities are still rejected; no accepted document changes. Part of #339.
EntityExpander.expand located the next '&', '<', and ';' with String.firstIndex
over graphemes, so every literal byte of a run was decoded into a grapheme just
to look for a one-byte marker. All three markers are ASCII, so a UTF-8 index is
a valid string index: scan over raw.utf8 instead, and the found index still
subscripts the string directly (the marker byte is never a continuation byte,
so its index lands on a scalar and grapheme boundary).
The decoded text is unchanged, including runs where the marker follows
multibyte UTF-8: a precomposed letter, a combining sequence, or an astral
character all leave the marker on a grapheme boundary.
Measured parse ~5.5% faster on the 20k-item generated corpus via a same-state
A/B (6.5x to ~6.1x slower than libxml2). Verified by EntityDecodeBulkTests
(extended with four multibyte-boundary cases), the W3C XML conformance corpora,
the Xalan XSLT gold-output corpus, the full suite, and the WASM build.
EntityExpander.expand rebuilt every reference-bearing text run one character at
a time: result.append(character) per character, with a per-character charge
against the amplification budget. A text node holding a single & was copied
grapheme by grapheme, which is the bulk of the per-text-node cost in a document
whose content uses entities.
Scan to the next '&' or '<' and append the literal run in one operation,
charging the budget by the run length. The decoded text, the CDATA-verbatim
shielding, and the billion-laughs budget enforcement are unchanged: the same
overrun throws at the same input, since charging a run of length L against
budget B fails exactly when L > B, the same condition the per-character guard
enforced.
Also return early from recordReferenceContentFindings when there are no DTD
content models (the common no-DTD case): it otherwise scanned each text run for
'&' and rendered the enclosing element's qualified name before the model lookup
that would fail anyway.
Measured parse ~5% faster on the 20k-item generated corpus via a same-state A/B
(9.06x to ~8.5x slower than libxml2). Verified by the W3C XML conformance
corpora, the new EntityDecodeBulkTests six-case decode differential, the full
suite, the Xalan XSLT gold-output corpus, and the WASM build, with an
adversarial equivalence review of the rewrite.
The eduni sections run with exact classified baselines. Fixes: 3.3.3
CDATA attribute-value normalization (literal whitespace to space before
reference decoding; char-ref whitespace survives; tokenized collapse on
SPACE only); production-68 lenient undeclared entities in content and
attribute values (validity finding, reference kept literal, nothing
fetched); errata E15 (references/comments/PIs/whitespace in EMPTY,
char-ref whitespace as data in element content, double-escape shape
included); errata E2 duplicate enumeration tokens; errata E14
declarations completed inside PE replacements (VC: Proper
Declaration/PE Nesting); NCName ID values; namespace declarations
subject to the every-attribute-declared rule. Reader validity machinery
split to EventReaderValidity for caps.
Closes#127
A general entity whose replacement text contains markup (directly or
transitively) is reparsed as content at the reference (4.4.2 Included):
the replacement is WFC-validated, cycle-checked, and budgeted, then
pushed back into the reader stream, so its elements become elements in
the tree and the pull-event stream rather than escaped character data.
Character references in internal entity literals now expand at
declaration time for general entities too (4.4.5), making char-ref
markup splice and the Appendix D double escape render as a literal
ampersand; the reference-time WFC validates remaining references
instead of re-expanding. Text-only entities keep the coalescing
single-pass path. Sun valid is 28/28.
Closes#133
Implement the replacement-text constraint at the point of reference: when a
general entity is included, its replacement (character references expanded
as at declaration time) must reparse as balanced content in isolation, via a
new EntityReplacementGrammar mini-parser. This rejects a bare or incomplete
reference ('&', '	' unterminated), tags spanning the entity boundary
('</foo><foo>'), a raw '<' or incomplete reference inside an embedded
attribute value, '<' followed by a non-name character, and a reserved-target
PI in the replacement, exactly the suite's semantics, including that an
UNREFERENCED entity may carry any value and that only the first declaration
of a name binds. Attribute-list defaults must reference already-declared
internal (not external/unparsed/forward-declared) entities with a
non-recursive chain. A CDATA section inside replacement text now shields its
content from reference expansion, fixing the last valid-side case.
W3C xmltest final standing: valid/sa 119/119, not-wf/sa 180/186, with the
six remaining deviations all deliberate and documented in the baseline:
internal-subset conditional/CDATA sections (063, 107) and parameter-entity
references (160-162) are features this package supports, and 141 expects the
1998 suite's name-character classes where this package implements XML 1.0
Fifth Edition's. Zero false positives across all 846 tests at every step.
Closes#120
Character references now require lowercase 'x' for hex and a referenced code
point inside the XML Char production (rejecting �, , ,
surrogates, and X). The reserved 'xml' processing-instruction target
(any case) is legal only as the declaration at the document's very first
bytes: whitespace or a comment before <?xml ...?>, a declaration inside
content or after the root, and case variants like <?XML?> are all rejected.
Comments and PI data are checked for the XML Char production like content.
not-wf/sa: 109/186 rejected (from 93); valid/sa unchanged at 118/119; all
846 suite tests green, baseline rebased to the exact 77 remaining (the
internal-subset grammar and entity replacement-text classes on #120).
Part of #120, #105
The whole-buffer ISO-2022-JP, GB18030, and EUC-TW decoders index the input
ArraySlice directly instead of materializing an Array first (one avoidable
O(n) allocation per decode each); declaredEncoding's subsequence scan uses
elementsEqual instead of allocating an Array per candidate offset. Document
the two analysis-flagged bounds: entity expansion is doubly bounded (the
character budget caps output; the visiting set caps recursion depth at the
number of declared entities), and the RELAX NG derivative engine carries no
memoization by design (the simplifying constructors keep real grammars
small; adversarial nested-choice grammars can grow the residual, so treat
untrusted schemas accordingly). Behavior unchanged, all suites green.
Closes#118
Add opt-in DTD processing behind Limits(allowDoctype:), off by default. When on,
DoctypeScanner reads the <!DOCTYPE> internal subset and honors internal
<!ENTITY name "value"> general entities, which EntityDecoder expands recursively
in text and attribute values.
Safety is mandatory and reimplemented from the Expat amplification-accounting
model (no upstream code copied):
- only entity-expanded characters count against a shared budget
(Limits.maxEntityExpansion); overflow raises amplificationLimitExceeded, which
defeats billion-laughs.
- a visiting set detects recursive entities (recursiveEntity).
- undeclared references raise undefinedEntity.
- external entities are never stored or loaded, so XXE stays closed.
The default still rejects <!DOCTYPE>. 8 DTD tests (incl. billion-laughs and the
external-entity refusal); 49 total.
Replace the notImplemented stub with a real parser built on a pull-based,
streaming event core that never requires the whole document in memory.
- Parsing.EventReader: pulls characters through a source closure with a bounded
lookahead buffer and emits one Parsing.Event at a time, holding only an
open-element stack. Drives chunked/streamed input.
- Parsing.Parser: builds a Model.Node tree iteratively over the event core
(explicit stack, no recursion).
- Reader: streaming character cursor (closure-backed, bounded buffer) replacing
the previous whole-string Array.
- EntityDecoder: the five predefined entities and decimal/hex char references.
- Handles elements, attributes, text, comments, CDATA, processing instructions,
and an XML declaration in the prolog. Rejects <!DOCTYPE> by default (no
DTD/XXE/entity-expansion exposure).
- Top-level API: parse(_:), parse(pulling:), events(_:), events(pulling:).
- 21 tests including event-stream and chunked-source coverage.
