DocumentType.validityFindings carries ValidityFinding values (reason
plus subject) instead of strings, DTDSchema.declarationErrors holds
ready ValidationErrors, and every declaration-level finding renders at
its declaration's coding path (x, r/@a) instead of at the document
root, matching the XSD consistency-errors precedent. Texts unchanged.
The reader buffers one Unicode scalar per element, splitting
multi-scalar graphemes from any source, so combining marks cannot merge
with delimiters and the Fifth Edition's scalar-defined name productions
classify correctly; the replacement-text, content-model, and ATTLIST
sub-scanners and the entity-literal validator follow, while grapheme
clusters reassemble wherever scanned text accumulates. Clears the last
eduni baseline, grows the asserted IBM pre-5e class to 300, and with
the xmltest manifest's own EDITION exclusions for 140/141 empties the
xmltest baseline: every xmlconf case applicable to a Fifth Edition,
namespace-aware processor passes.
Closes#135
Limits(strictInternalSubset: true) rejects conditional sections and
parameter-entity references inside markup declarations in the internal
subset, the two feature classes XML 1.0 reserves for the external
subset. DeclSep references and quoted attribute defaults (AttValue)
stay legal; defaults unchanged. The xmltest runner opts in, leaving
case 141 (pre-Fifth-Edition name classes) as its only deviation.
Closes#128
A relative system identifier declared inside an external entity
resolves against that entity's own URI (RFC 3986, the C14N resolver):
ExternalID gains base and resolvedSystemID, the scanner threads the
current entity's URI through external-subset and PE scans, and
resolvers receive the merged identifier. Clears eduni E18; the eduni
baseline now holds only the grapheme-lexing class (#135).
Closes#138
A declared 16/32-bit encoding over BOM-less 8-bit bytes is rejected; a
BOM followed by a declaration outside its family is rejected, reading
the declaration through the BOM's own byte pattern; and an external
entity whose text declaration names a higher XML version than the
document is refused (errata E38). The eduni not-wf baseline is empty.
Reader and decoder helpers split for the length caps.
Closes#137
The namespace layer rejects malformed qualified names, declarations of
the xmlns prefix, misbindings of the xml prefix and either reserved
namespace name (including as the default namespace), and prefix
undeclaration; attributes must be distinct by expanded name; PI targets
and entity/notation names are NCNames; and DTD-typed xmlns values bind
their normalized form. Clears all 18 namespace cases from the eduni
baseline; xmltest valid/sa/012 joins the documented deviations per the
suite's own NAMESPACE='no' flag.
Closes#136
The eduni sections run with exact classified baselines. Fixes: 3.3.3
CDATA attribute-value normalization (literal whitespace to space before
reference decoding; char-ref whitespace survives; tokenized collapse on
SPACE only); production-68 lenient undeclared entities in content and
attribute values (validity finding, reference kept literal, nothing
fetched); errata E15 (references/comments/PIs/whitespace in EMPTY,
char-ref whitespace as data in element content, double-escape shape
included); errata E2 duplicate enumeration tokens; errata E14
declarations completed inside PE replacements (VC: Proper
Declaration/PE Nesting); NCName ID values; namespace declarations
subject to the every-attribute-declared rule. Reader validity machinery
split to EventReaderValidity for caps.
Closes#127
The IBM section (1094 cases) runs clean to two documented classes:
279 pre-Fifth-Edition character-class cases (P85-89, count asserted
exactly) and 2 PE-in-internal-subset feature cases. Real gaps closed:
a document needs a root element (production 1); entity references in
literals must be Names or character references; WFC: No Recursion
checked at declaration over the general/parameter reference graph;
undeclared PE references split into WFC (standalone, or no external
declarations possible) versus deferred validity finding, also for
attribute-default references; refused external PEs are neither; and
groups split across PE replacements report VC: Proper Group/PE
Nesting. Entity machinery split to DoctypeScannerEntities for caps.
Closes#125
The scanner records declaration provenance (internal-subset entities,
element models, attribute lists); the validator reports the three
standalone='yes' dependencies on the external subset: a default that
would be supplied, a non-CDATA normalization that would change a value,
and whitespace in externally-declared element content. The parser hides
externally-declared entities from standalone documents, so referencing
one reports as undeclared (WFC: Entity Declared, 2.9). The Sun section
is fully clean: not-wf 56/56, valid 28/28, invalid 74/74, no baseline.
Closes#134Closes#124
VC Root Element Type (root matches the DOCTYPE name), VC Unique
Element Type Declaration, VC No Duplicate Types in mixed content, VC
One ID per Element Type, VC Notation Attributes and Notation Declared,
VC Attribute Default Legal plus ID Attribute Default, lexical Name
checks for the ID family, and CDATA as character data in element
content. Strict mode adds VC Attribute Value Type (every attribute
declared; namespace declarations exempt). Declaration findings are
computed once at schema build and reported at the document root. Sun
invalid coverage moves from 43/74 to 62/74; the twelve remaining are
all the standalone='yes' family.
A general entity whose replacement text contains markup (directly or
transitively) is reparsed as content at the reference (4.4.2 Included):
the replacement is WFC-validated, cycle-checked, and budgeted, then
pushed back into the reader stream, so its elements become elements in
the tree and the pull-event stream rather than escaped character data.
Character references in internal entity literals now expand at
declaration time for general entities too (4.4.5), making char-ref
markup splice and the Appendix D double escape render as a literal
ampersand; the reference-time WFC validates remaining references
instead of re-expanding. Text-only entities keep the coalescing
single-pass path. Sun valid is 28/28.
Closes#133
Tokenized (non-CDATA) attribute values normalize before validity
checks, so enumerations, NOTATION lists, and the ID/IDREF table see
collapsed values. External parsed entities lose their leading text
declaration (validated: version optional, encoding required, never
standalone) instead of carrying it into replacement text. Character
references in PE literals expand at declaration time, making the
Appendix D '%zz;' example parse. A PI target inside the DTD subset
must be separated from its data. Sun baselines tighten to not-wf 55/56
and valid 26/28; the two remaining valid failures share one cause
(replacement text with markup is included as character data, filed
separately).
The OASIS/NIST section of the W3C conformance suite runs via a new
XMLCONF_ROOT-gated runner driven by the section's own manifest (parsed
with PureXML): 247/247 not-wf rejected, all 100 well-formed cases parse,
zero deviations. Closing its 24 gaps makes the DTD subset grammar
strict: exact PEReference shape, strict declaration names and keywords,
NDATA and notation tails, no bare '%' in entity values, case-sensitive
balanced conditional sections, junk between declarations rejected,
external-subset violations now propagate (with text-declaration
support, production 77), and a PI target must be separated from its
data. The xmltest baseline tightens to 181/186: CDATA in the internal
subset (107) is now rejected, five documented deviations remain.
Closes#123
Implement the replacement-text constraint at the point of reference: when a
general entity is included, its replacement (character references expanded
as at declaration time) must reparse as balanced content in isolation, via a
new EntityReplacementGrammar mini-parser. This rejects a bare or incomplete
reference ('&', '	' unterminated), tags spanning the entity boundary
('</foo><foo>'), a raw '<' or incomplete reference inside an embedded
attribute value, '<' followed by a non-name character, and a reserved-target
PI in the replacement, exactly the suite's semantics, including that an
UNREFERENCED entity may carry any value and that only the first declaration
of a name binds. Attribute-list defaults must reference already-declared
internal (not external/unparsed/forward-declared) entities with a
non-recursive chain. A CDATA section inside replacement text now shields its
content from reference expansion, fixing the last valid-side case.
W3C xmltest final standing: valid/sa 119/119, not-wf/sa 180/186, with the
six remaining deviations all deliberate and documented in the baseline:
internal-subset conditional/CDATA sections (063, 107) and parameter-entity
references (160-162) are features this package supports, and 141 expects the
1998 suite's name-character classes where this package implements XML 1.0
Fifth Edition's. Zero false positives across all 846 tests at every step.
Closes#120
Make the DTD declaration grammar strict: <!ENTITY> requires its whitespace
(after the keyword, around %, after the name), a real name (not #DEFAULT),
nothing but '>' after the value, NDATA preceded by whitespace and forbidden
on parameter entities; entity-value literals are validated (XML Chars only,
every & a complete reference); SYSTEM/PUBLIC external identifiers parse
strictly (PUBLIC needs both literals for entities and the DOCTYPE, public-only
stays legal for notations) with PubidChar validation; <!ATTLIST> bodies are
validated against XML 1.0 productions 52-60 via a new DTDAttListGrammar
(spacing, attribute types, enumerations, quoted defaults); the reserved xml
PI target is rejected inside the subset. Also: strict UTF-8 decoding (no
silent replacement of malformed sequences), XML Char checks in attribute
values and CDATA, the declaration's version/encoding value grammar, and
required whitespace between pseudo-attributes. The strict machinery lives in
DoctypeScannerStrict/DTDAttListGrammar to stay under the length caps.
W3C xmltest: not-wf/sa 162/186 rejected (from 129; 80 at first contact),
valid/sa 118/119, zero false positives across all 846 tests at every step.
The remaining 24-case baseline is exactly: the entity replacement-text class
(19) and the deliberate internal-subset extensions this package supports as
features (conditional sections, PEs in the internal subset: 5).
Part of #120, #105
Third W3C-suite strictness pass: DTDContentModelGrammar validates each
element declaration's content specification against the XML 1.0 grammar
(productions 46-51) at DTD scan time, EMPTY/ANY, Mixed (with #PCDATA first,
names only, the trailing * required with alternatives, no nesting), and
children (one connector per group, quantifiers bound directly to their
particle, balanced parentheses, no empty groups, no SGML leftovers like
'- -', 'O O', '&' connectors, or inclusion/exclusion suffixes). The element
declaration also requires whitespace after <!ELEMENT and after the name. The
lenient ContentModelParser still drives validation behavior; this is a
well-formedness gate. not-wf/sa: 129/186 rejected (from 109); valid/sa
unchanged; zero false positives across all 846 suite tests; baseline rebased
to the exact 57 remaining (declaration-grammar details and the entity
replacement-text class).
Part of #120, #105
Character references now require lowercase 'x' for hex and a referenced code
point inside the XML Char production (rejecting �, , ,
surrogates, and X). The reserved 'xml' processing-instruction target
(any case) is legal only as the declaration at the document's very first
bytes: whitespace or a comment before <?xml ...?>, a declaration inside
content or after the root, and case variants like <?XML?> are all rejected.
Comments and PI data are checked for the XML Char production like content.
not-wf/sa: 109/186 rejected (from 93); valid/sa unchanged at 118/119; all
846 suite tests green, baseline rebased to the exact 77 remaining (the
internal-subset grammar and entity replacement-text classes on #120).
Part of #120, #105
Add five well-formedness checks the W3C xmltest suite surfaced: '--' inside
comments (also rejecting the ---> ending), raw '<' in attribute values, the
literal ]]> sequence in character data, attributes not separated by
whitespace, and characters outside the XML Char production in content, each
a located ParseError. The attribute scanners move to EventReaderAttributes
to stay under the type-body cap.
Add the opt-in W3C xmltest runner (XMLTS_ROOT-gated; the suite is never
vendored, James Clark's license permits redistribution only as the unmodified
archive; Foundation is used in this one test file for file access only).
Baseline: valid/sa 118/119, not-wf/sa 93/186 rejected (up from 80), the
remaining 93 carried as an exact known-deviations baseline so progress and
regressions are both visible. The two remaining leniency classes (internal-
subset DTD grammar, entity replacement-text well-formedness) are #120.
Part of #105 (Tier 2)
The whole-buffer ISO-2022-JP, GB18030, and EUC-TW decoders index the input
ArraySlice directly instead of materializing an Array first (one avoidable
O(n) allocation per decode each); declaredEncoding's subsequence scan uses
elementsEqual instead of allocating an Array per candidate offset. Document
the two analysis-flagged bounds: entity expansion is doubly bounded (the
character budget caps output; the visiting set caps recursion depth at the
number of declared entities), and the RELAX NG derivative engine carries no
memoization by design (the simplifying constructors keep real grammars
small; adversarial nested-choice grammars can grow the residual, so treat
untrusted schemas accordingly). Behavior unchanged, all suites green.
Closes#118
Add EUC-TW for the whole-buffer and streaming decoders: ASCII, a two-byte
plane-1 pair, and a 0x8E four-byte plane selector plus a pair. CNS planes 1
and 2, which carry essentially all real-world EUC-TW text, are vendored from
the ICU euc-tw-2014 mapping (private-use fallbacks dropped); planes 3-15 are a
documented follow-up. Tested for plane 1 (two-byte and four-byte forms), plane
2, and streaming. The streaming method moved to its own file to stay under the
type-body cap.
Closes#108
The streaming decoder previously selected an encoding only from a byte-order
mark or the UTF sniff, so its single-byte and CJK decode paths were latent. It
now reads the encoding from the XML declaration (the libxml2 behavior) when
there is no BOM, activating those paths over an incremental byte source. Add
the streaming form of ISO-2022-JP, carrying the stateful mode across reads
(extracted to its own file to stay under the type-body cap). A test asserts
streaming agrees with whole-buffer decode for a single-byte (ISO-8859-1), a
CJK (Shift_JIS), and an ISO-2022-JP document.
Part of #108, #105
Add the WHATWG iso-2022-jp decoder, the first stateful encoding: escape
sequences switch between ASCII, JIS X 0201 Roman (yen, overline), JIS X 0201
katakana, and the JIS X 0208 plane, resolved through the shared jis0208 table
(no new data). Tested across all four modes, with the JIS X 0208 kanji
cross-checked against the EUC-JP decoder (same table, same pointer). EUC-TW and
the streaming form of ISO-2022-JP remain.
Part of #108, #105
Extend ByteEncoder to the multi-byte CJK encodings (Shift-JIS, EUC-JP,
EUC-KR, GBK, GB18030, Big5). Each inverse is built by running the existing,
tested whole-buffer decoder over every valid byte sequence and recording
the shortest sequence per scalar, so the encoder is by construction the
exact inverse of the decoder, with no new data or separate encode tables.
GB18030's astral range uses the four-byte pointer formula, so every scalar
encodes. Round-trip tests cover Japanese, Korean, Chinese, and a GB18030
astral character.
Closes#106
Add PureXML.Parsing.ByteEncoder, the inverse of ByteDecoder, and a
Serializer.serialize(_:encoding:) / PureXML.serialize(_:encoding:) that
emits bytes in a chosen encoding (the libxml2 save-with-encoding model).
Covers the Unicode transformation formats and the single-byte legacy
families, built by inverting the existing vendored to-Unicode tables, so
no new data is introduced. The declaration carries the canonical encoding
name, a UTF-16/32 stream is preceded by a byte-order mark, and any
unrepresentable scalar becomes a decimal numeric character reference, so
output round-trips. CJK output encodings remain (tracked on #106).
Part of #106, #105
Add the WHATWG big5 decoder over the vendored index-big5 (18,590 entries),
for both the whole-buffer and streaming decoders. A lead in 0x81-0xFE plus
a trail in 0x40-0x7E or 0xA1-0xFE resolves through the index, covering
Traditional Chinese and the HKSCS extensions including the astral CJK
ideographs. Four pointers decode to a base letter plus a combining mark.
Selected by the big5 / big5-hkscs declaration names.
Closes#103
Extend the GBK two-byte subset to full GB18030: a lead byte followed by
a digit opens a four-byte form whose pointer is mapped through the WHATWG
index-gb18030-ranges linear ranges, reaching the supplementary planes.
The 207 ranges are vendored verbatim. Wired into the whole-buffer and
streaming decoders and the encoding-name table (gb18030/gb-18030).
Closes#102
Normalize line endings in the Reader as the spec requires, which was not
done before:
- CR and CRLF fold to a single line feed (XML 1.0), so every downstream
scan sees line feeds only.
- When the declaration names version 1.1, NEL (U+0085) and LINE SEPARATOR
(U+2028), and a CR followed by NEL, also fold to a line feed; in 1.0
those characters are preserved as data.
The XML 1.1 name-character productions are identical to XML 1.0 Fifth
Edition, which the character classifier already implements, so names need
no change. The 1.1 character range (C0 controls permitted only as
references) is not yet enforced.
Closes#96
Broaden the TextReader pull cursor toward libxml2's xmlTextReader:
- Attribute nodes: moveToFirstAttribute/moveToNextAttribute/
moveToAttribute(_:)/moveToElement reposition the cursor onto each
attribute as an .attribute node and back.
- Namespace accessors: namespaceURI, localName, prefix from the current
element's or attribute's resolved name.
- xml:lang: inherited down the tree, overridable per element, reset when
it goes out of scope.
- Position: lineNumber/columnNumber report where each node begins, so a
streaming consumer can locate findings in the source. EventReader now
records each node's start mark.
DOCTYPE and entity-reference nodes are not surfaced (the DTD is exposed
through documentType, and entities are expanded inline); whitespace is
not split into significant/insignificant (needs the content model);
baseURI needs a document URI. The accurate positioning path for editor
findings remains readTree with source ranges.
Closes#98
Parse the XML declaration's pseudo-attributes instead of skipping the
<?xml?> processing instruction:
- version, encoding, and standalone are surfaced on a new XMLDeclaration,
returned from parseWithDocumentType and via PureXML.xmlDeclaration(_:).
- The pseudo-attributes are validated: fixed order (version, encoding,
standalone), known names only, and standalone must be yes or no. A
malformed declaration is rejected.
- The declared version is now readable, a prerequisite for XML 1.1.
parseWithDocumentType returns a ParsedDocument struct (node,
documentType, declaration) rather than a 2-tuple; the member names are
unchanged so existing call sites are unaffected.
Closes#95
Deepen DTD processing in the doctype scanner:
- Conditional sections are processed, not skipped: INCLUDE sections have
their declarations scanned, IGNORE sections are discarded, the keyword
may be a parameter-entity reference, and nested sections are handled.
- <!NOTATION> declarations are parsed and stored on DocumentType.
- Unparsed entities (<!ENTITY ... NDATA n>) are parsed and stored with
their notation, distinct from general entities.
- External parameter entities are loaded through the injected resolver so
their declarations expand; the default refusing resolver loads nothing,
keeping XXE closed.
- NOTATION attributes are a distinct type, validated against the declared
notations (value must be listed and the notation declared).
Tracking unparsed entities also lets ENTITY/ENTITIES attribute values be
validated against the declared NDATA entities, closing the gap left when
#73 could only check name syntax.
Closes#72
Complete the DTD attribute handling found in the audit:
- Default and #FIXED attribute values are applied to the parsed tree.
parseApplyingInternalDTDDefaults returns a tree where every element
carries the attributes its <!ATTLIST> defaults, so an editor sees the
same effective attributes a validating processor would.
- NMTOKEN, NMTOKENS, ENTITY, ENTITIES are validated: NMTOKEN(S) must be
XML name tokens, ENTITY/ENTITIES must be XML names. Previously all fell
through to unconstrained CDATA.
- DTD default values are normalized per XML 1.0: character and predefined
entity references are expanded, literal whitespace becomes a space, and
non-CDATA values are collapsed and trimmed.
ENTITY/ENTITIES check name syntax only; that the named entity is a
declared unparsed entity is not verified, since NDATA entity tracking is
not yet modeled.
Closes#73
Slice 4 of #94 (completes the epic). Quick-fixes are derived from the structured
schema completions, never by parsing a diagnostic message:
- TreeNode gains contentRange (the span between an element's start and end tags),
captured by the ranged reader, so an inserted attribute or child is placed
exactly.
- QuickFix and TextEdit model an editor edit (a replacement over a source range;
an insertion is a zero-width range).
- QuickFixEngine turns a Completions into fixes: a required-but-absent attribute
becomes an insertion before the start tag's >, and a still-expected required
child becomes an insertion before the end tag. Schema.Document.quickFixes(at:in:)
is the entry over a ranged tree.
Tests apply a fix and re-validate: adding the required id, or inserting the
required qty, yields XML the schema then accepts; a complete element offers none.
#94 (editor integration) is complete: ranged recovering read, unified lint,
schema completions, and quick-fixes.
Slice 1 of #94 (editor integration). The keystone: a located validation finding
can now be mapped back to a source range.
- TreeNode gains a sourceRange (start/end Mark); SourceRange is a half-open span.
- PureXML.readTree builds a mutable, parent-aware, best-effort tree whose every
node carries its span (an element from the < of its start tag to past its end
tag; leaves over the text they consumed). Never throws, recovers in place,
returns located diagnostics.
- TreeNode.node(at:) navigates a validation coding path to its node using the
same canonical convention the validators build paths with (a sibling index
only when a name repeats), and sourceRange(at:) returns the span. So
Document.validate's ValidationError.codingPath resolves to characters in the
document.
Tests prove the spans, the navigation, the end-to-end bridge (an XSD finding to
a source range), and that ranged reading of invalid input never crashes.
Closes#93. Recovery moves into a recovering mode on EventReader, so malformed
input is repaired in place rather than dropped to the next markup boundary:
- A duplicate attribute is recorded and the later occurrence dropped, keeping the
element. <a id='1' id='2'>y</a> now reads as <a id="1">y</a> with one
diagnostic (was an empty tree with three).
- A mismatched or unexpected end tag pops the open stack to the matching ancestor,
implicitly closing the elements nested inside it. <a><b><c>x</b></a> reads as
<a><b><c>x</c></b></a> with a single diagnostic.
- Pre-root junk reports "unexpected character" instead of the misleading
"content after the root element".
In recovering mode next() no longer throws: it records a Diagnostic and emits
best-effort events, popping to a match on a mismatch and stopping on truncation
so the builder closes what is open. The strict path is unchanged. readRecovering
becomes a plain non-throwing drain that takes the reader's diagnostics. Recovery
stays deterministic; new tests pin the kept-element and pop-to-match results.
Slice 2 of #92. Add PureXML.read, a deterministic recovering reader: it never
throws and never traps, returning the maximal best-effort tree plus one located
Diagnostic per problem.
- ReadResult { node, diagnostics } is the Equatable result type, so a
deterministic read can be compared whole.
- Diagnostic carries a message and a structured source Mark; a thrown ParseError
maps to one directly (ParseError.mark exposes the position each case carries).
- The reader resynchronizes after a failure (EventReader.recover(from:)):
progress-aware, so a failure that already consumed the bad construct retries
from there, while a stuck failure skips to the next markup start; it always
advances when input remains, guaranteeing termination.
- The tree builder assembles leniently: elements left open by truncation are
closed, an end tag with no open element is dropped. Well-formed input yields
exactly parse(_:)'s tree with no diagnostics.
Tests pin exact trees and diagnostics for truncation, mismatched tags, and stray
junk, prove determinism (same bytes give an identical ReadResult), and fuzz 400
seeded inputs (arbitrary plus truncated valid docs) asserting no crash.
Recovery here is resync-based; finer in-reader recovery (keep the element on a
duplicate attribute, pop to a matching open tag) is the next slice.
Add a resumable XML scanner that reports a token only once its terminator is
fully buffered and otherwise signals need-more (the Expat XML_TOK_PARTIAL
model), and PushParser with feed(_:)/finish() driving a SAXHandler while
retaining only the current incomplete token plus the open-element stack.
events(feeding:) exposes an AsyncThrowingStream of events over any async
sequence of text chunks. Splitting a document at any byte boundary, down to
one character per feed, yields identical events. Collapse the completed
roadmap into the legend summary.
Closes#4Closes#5Closes#6Closes#7Closes#1
Add PureXML.Parsing.TextReader (via PureXML.reader), a pull cursor over the
streaming core. read() advances node by node, exposing nodeKind, name,
value, depth, attributes, and isEmptyElement. A childless element is
reported once as an empty element, normalizing <a/> and <a></a>. The
documentType accessor surfaces the DTD read so far as the validation hook.
Closes#33
Add internal parameter entities (parsed, stored, and expanded within the
DTD, including bare %name; references that inject markup declarations,
bounded by depth and the expansion budget). Record external general
entities and the external subset identifier, and load them only through an
injected EntityResolver (a struct of closures). The default refusing
resolver fetches nothing, so XXE stays closed: an external entity fails as
undefined and the external subset is left unread. Internal declarations win
over the external subset.
Closes#29
