Tokenized (non-CDATA) attribute values normalize before validity
checks, so enumerations, NOTATION lists, and the ID/IDREF table see
collapsed values. External parsed entities lose their leading text
declaration (validated: version optional, encoding required, never
standalone) instead of carrying it into replacement text. Character
references in PE literals expand at declaration time, making the
Appendix D '%zz;' example parse. A PI target inside the DTD subset
must be separated from its data. Sun baselines tighten to not-wf 55/56
and valid 26/28; the two remaining valid failures share one cause
(replacement text with markup is included as character data, filed
separately).
The first runner to exercise the validity layer: not-wf must be
rejected (54/56), valid must parse and DTD-validate clean (19/28),
invalid must parse but report at least one validity error (43/74).
Deviation baselines are exact and annotated with their causes, the
burn-down list for #124: tokenized attribute-value normalization,
external-entity text declarations, declaration-time char-ref expansion
in PE literals, standalone VCs, ID/IDREF corners, REQUIRED enforcement,
per-type attribute VCs, and the root/DOCTYPE name match.
The OASIS/NIST section of the W3C conformance suite runs via a new
XMLCONF_ROOT-gated runner driven by the section's own manifest (parsed
with PureXML): 247/247 not-wf rejected, all 100 well-formed cases parse,
zero deviations. Closing its 24 gaps makes the DTD subset grammar
strict: exact PEReference shape, strict declaration names and keywords,
NDATA and notation tails, no bare '%' in entity values, case-sensitive
balanced conditional sections, junk between declarations rejected,
external-subset violations now propagate (with text-declaration
support, production 77), and a PI target must be separated from its
data. The xmltest baseline tightens to 181/186: CDATA in the internal
subset (107) is now rejected, five documented deviations remain.
Closes#123
The derivation final check and Particle Valid (Restriction) move from
fail-fast throws at schema compile into composable Validation rules
(finalRespected, restrictionsAreSubsets) over per-type SchemaTypeFact
subjects, so a schema with several problems reports every finding at
once. SchemaError gains .inconsistent([String]) carrying all findings
(texts unchanged) and drops .finalViolation/.invalidRestriction.
checkRedefine, checkAllGroups, and notASchema stay throws: they examine
the raw schema source, the compilation analog of well-formedness.
Implement the replacement-text constraint at the point of reference: when a
general entity is included, its replacement (character references expanded
as at declaration time) must reparse as balanced content in isolation, via a
new EntityReplacementGrammar mini-parser. This rejects a bare or incomplete
reference ('&', '	' unterminated), tags spanning the entity boundary
('</foo><foo>'), a raw '<' or incomplete reference inside an embedded
attribute value, '<' followed by a non-name character, and a reserved-target
PI in the replacement, exactly the suite's semantics, including that an
UNREFERENCED entity may carry any value and that only the first declaration
of a name binds. Attribute-list defaults must reference already-declared
internal (not external/unparsed/forward-declared) entities with a
non-recursive chain. A CDATA section inside replacement text now shields its
content from reference expansion, fixing the last valid-side case.
W3C xmltest final standing: valid/sa 119/119, not-wf/sa 180/186, with the
six remaining deviations all deliberate and documented in the baseline:
internal-subset conditional/CDATA sections (063, 107) and parameter-entity
references (160-162) are features this package supports, and 141 expects the
1998 suite's name-character classes where this package implements XML 1.0
Fifth Edition's. Zero false positives across all 846 tests at every step.
Closes#120
Make the DTD declaration grammar strict: <!ENTITY> requires its whitespace
(after the keyword, around %, after the name), a real name (not #DEFAULT),
nothing but '>' after the value, NDATA preceded by whitespace and forbidden
on parameter entities; entity-value literals are validated (XML Chars only,
every & a complete reference); SYSTEM/PUBLIC external identifiers parse
strictly (PUBLIC needs both literals for entities and the DOCTYPE, public-only
stays legal for notations) with PubidChar validation; <!ATTLIST> bodies are
validated against XML 1.0 productions 52-60 via a new DTDAttListGrammar
(spacing, attribute types, enumerations, quoted defaults); the reserved xml
PI target is rejected inside the subset. Also: strict UTF-8 decoding (no
silent replacement of malformed sequences), XML Char checks in attribute
values and CDATA, the declaration's version/encoding value grammar, and
required whitespace between pseudo-attributes. The strict machinery lives in
DoctypeScannerStrict/DTDAttListGrammar to stay under the length caps.
W3C xmltest: not-wf/sa 162/186 rejected (from 129; 80 at first contact),
valid/sa 118/119, zero false positives across all 846 tests at every step.
The remaining 24-case baseline is exactly: the entity replacement-text class
(19) and the deliberate internal-subset extensions this package supports as
features (conditional sections, PEs in the internal subset: 5).
Part of #120, #105
Third W3C-suite strictness pass: DTDContentModelGrammar validates each
element declaration's content specification against the XML 1.0 grammar
(productions 46-51) at DTD scan time, EMPTY/ANY, Mixed (with #PCDATA first,
names only, the trailing * required with alternatives, no nesting), and
children (one connector per group, quantifiers bound directly to their
particle, balanced parentheses, no empty groups, no SGML leftovers like
'- -', 'O O', '&' connectors, or inclusion/exclusion suffixes). The element
declaration also requires whitespace after <!ELEMENT and after the name. The
lenient ContentModelParser still drives validation behavior; this is a
well-formedness gate. not-wf/sa: 129/186 rejected (from 109); valid/sa
unchanged; zero false positives across all 846 suite tests; baseline rebased
to the exact 57 remaining (declaration-grammar details and the entity
replacement-text class).
Part of #120, #105
Character references now require lowercase 'x' for hex and a referenced code
point inside the XML Char production (rejecting �, , ,
surrogates, and X). The reserved 'xml' processing-instruction target
(any case) is legal only as the declaration at the document's very first
bytes: whitespace or a comment before <?xml ...?>, a declaration inside
content or after the root, and case variants like <?XML?> are all rejected.
Comments and PI data are checked for the XML Char production like content.
not-wf/sa: 109/186 rejected (from 93); valid/sa unchanged at 118/119; all
846 suite tests green, baseline rebased to the exact 77 remaining (the
internal-subset grammar and entity replacement-text classes on #120).
Part of #120, #105
Add five well-formedness checks the W3C xmltest suite surfaced: '--' inside
comments (also rejecting the ---> ending), raw '<' in attribute values, the
literal ]]> sequence in character data, attributes not separated by
whitespace, and characters outside the XML Char production in content, each
a located ParseError. The attribute scanners move to EventReaderAttributes
to stay under the type-body cap.
Add the opt-in W3C xmltest runner (XMLTS_ROOT-gated; the suite is never
vendored, James Clark's license permits redistribution only as the unmodified
archive; Foundation is used in this one test file for file access only).
Baseline: valid/sa 118/119, not-wf/sa 93/186 rejected (up from 80), the
remaining 93 carried as an exact known-deviations baseline so progress and
regressions are both visible. The two remaining leniency classes (internal-
subset DTD grammar, entity replacement-text well-formedness) are #120.
Part of #105 (Tier 2)
Grow the Tier-2 conformance corpus with 7 OASIS catalog cases through the
harness: exact system/public/uri entries, rewriteSystem and rewriteURI prefix
matching, and the unresolved outcomes for unmatched identifiers. All conform.
Part of #105 (Tier 2)
Grow the Tier-2 conformance corpus with 17 XSD regex cases through the
pattern-facet path: implicit whole-string anchoring, the character-class
escapes (backslash-d/w/s), Unicode category and negated-category escapes,
quantifier ranges (exact, bounded, open), alternation, grouping, negated
character classes, and class subtraction. All conform.
Part of #105 (Tier 2)
Grow the Tier-2 conformance corpus with 10 cases through the harness:
XInclude substitution, parse=text inclusion with escaping, xi:fallback on a
missing href, and relative href resolution against the base; XPointer
shorthand ids, element() navigation from an id and from the root, and
xpointer() including the classic per-step predicate semantic (//para[1]
selects every parent-first para; (//para)[1] the document-first), which the
corpus now locks explicitly.
Part of #105 (Tier 2)
A complex type derived by restriction is checked at schema compile to accept
a structural subset of its base, closing the declared XSD 1.0 spec gap.
ParticleRestriction covers element name matching, element-vs-wildcard
admission, wildcard narrowing, occurrence-range subsumption, and the group
recursions: sequence/all order-preserving with emptiable skips (Recurse),
choice mapping (RecurseLax), sequence-into-choice (MapAndSum, approximated by
per-particle checks, a documented simplification), sequence-into-all
(RecurseUnordered), and RecurseAsIfGroup for a lone element against a group.
An unfaithful restriction throws SchemaError.invalidRestriction(type:base:
reason:). All 834 pre-existing tests pass unchanged (no false positives on
the existing restriction-using schemas); 7 new tests cover the accept and
reject paths plus a conformance-harness pair.
Closes#117
Extract PathKey.steps(forChildNames:) as the single path construction behind
the validator walk, the XSD content validator, and the identity validator,
replacing three identical implementations, so located errors can never
disagree on a path. Add two differentials locking the duplicated engines
together: equivalent DTD and XSD content models (sequence, choice, *, +, ?,
nested choice) must agree on every child sequence across the positional
matcher and the Thompson NFA; and the pull (EventReader) and push
(PushScanner) tokenizers must yield the same event stream for the same
document, even with the push side fed one character at a time (adjacent text
events coalesced, since chunk boundaries legitimately split text). Both
pairs agree on everything today; the differentials guard them.
Closes#115
The whole-buffer ISO-2022-JP, GB18030, and EUC-TW decoders index the input
ArraySlice directly instead of materializing an Array first (one avoidable
O(n) allocation per decode each); declaredEncoding's subsequence scan uses
elementsEqual instead of allocating an Array per candidate offset. Document
the two analysis-flagged bounds: entity expansion is doubly bounded (the
character budget caps output; the visiting set caps recursion depth at the
number of declared entities), and the RELAX NG derivative engine carries no
memoization by design (the simplifying constructors keep real grammars
small; adversarial nested-choice grammars can grow the residual, so treat
untrusted schemas accordingly). Behavior unchanged, all suites green.
Closes#118
Deepen the two thin coverage zones (#119) through the validation-driven
harness. Serializer options matrix (10 cases): self-close on/off, CDATA kept
vs as-text, asciiOnly hex character references, round-trip carriage-return
escaping, declaration with standalone, exact pretty-print shape, attribute
escapes, and the Latin-1 output-encoding declaration via the byte path. HTML
depth (8 cases): the adoption agency's furthest-block and deep-misnesting
paths, foster-parented table text (with the implied tbody per the HTML5
algorithm), template content nesting through the document parser, and
raw-text script, implied dt/dd and table-cell closes, and comments through
the fragment parser. All conform; the one initial mismatch was a wrong
expectation in the test (the implied tbody), not a code divergence.
Closes#119
Lock the strict and recovering parse paths together: recovery synthesizes
events through machinery the strict path never exercises, so a 21-document
corpus of well-formed input (the C14N conformance inputs plus nesting,
mixed content, attribute quoting/escapes, namespaces and redeclaration,
entities, character references, prolog, comments, CDATA, xml:space) asserts
that PureXML.read produces the identical Node tree as PureXML.parse with no
diagnostics, and that readTree's editor tree projects to the same node. Both
paths agree on every document; the differential now guards them.
Closes#116
Document the weak-parent ownership trap at the public entry points: parseTree
and readTree now state that a tree (or ownerDocument) must be held while using
any node inside it, that TreeNode has no internal synchronization (confine to
one thread or actor), and that the immutable Sendable Node projection is the
way across concurrency boundaries; the TreeNode class doc carries the same
contract. QualifiedName's malformed-colon fallback (':local', 'prefix:', '::'
stay whole and unprefixed), already implemented, is now documented on init
and locked by a test.
Closes#114
Node-set ordering previously recomputed both nodes' root paths on every
precedes comparison (treePath is O(depth x siblings), so sorts were
O(n log n x depth)). Add decorate-sort-undecorate helpers,
sortedByDocumentOrder() and firstInDocumentOrder(), that compute each node's
order key once, and switch the evaluator's order-and-dedup, the EXSLT set
functions, string-value coercion, and NodeFunctions' target lookup to them.
set:leading/trailing also decorate once, reusing the keys for the pivot
filter and the final sort. precedes stays for pairwise use, sharing the
key comparator (Node.ordered). Behavior unchanged, all suites green.
Closes#113
A selector or field XPath that fails to compile is now reported once up
front as a located error (identity constraint 'k': invalid selector/field
XPath '...'), instead of try? silently turning a schema author's typo into
a disabled constraint. The per-run query cache from #112 is the choke point:
compileErrors() probes every constraint's selector and fields in
deterministic order, the failures cache as nil, and the walk afterwards
treats those constraints as matching nothing, so the rest of the document
still validates. Tests cover a malformed field and a malformed selector.
Closes#111
XSLT: template selection resolves each match pattern once per transform via
a per-run MatchCache (the pattern's node set is computed once over the source
tree; selection becomes an identity lookup), replacing per-(node, template)
recompilation and full-tree re-walks. XSD identity validation: a per-run query
cache compiles each selector/field XPath once instead of at every visited
element. Schematron was verified to already store compiled queries. Behavior
unchanged (all suites green); a multi-template shape test over 300 nodes
guards the optimized path. MatchCache lives in its own file to keep the
transformer under the length cap.
Closes#112
An xsi:type naming an undeclared type is now a located error in both the
tree validator (validateChild) and the streaming driver, instead of a silent
fallback to the declared type. typeReference chains resolve through one
shared resolver (resolveReference, with a visited set), replacing the inline
recursion, the 64-hop cap, and the completions guard: a circular chain now
reports 'circular type reference' instead of silently truncating, and the
streaming shallow check no longer recurses without bound on a cyclic table
(a latent crash). Tests cover unknown xsi:type (tree + streaming agree),
the circular chain (tree + shallow), the resolver's three outcomes, and a
conformance-harness case pair.
Closes#110
Grow the Tier-2 conformance corpus with 14 DTD content-model cases (sequence
order/completeness, choice exclusivity, the *, +, ? occurrence indicators,
#PCDATA, mixed content, EMPTY, and ANY) validated against the internal subset
and checked through the validation-driven harness. Own suite.
Part of #105 (Tier 2)
Grow the Tier-2 conformance corpus with 13 XPath navigation cases (descendant
and all-element counts, positional and last() predicates, ancestor and
ancestor-or-self axes, following/preceding-sibling, the attribute axis, an
attribute predicate, union, the parent abbreviation, and position()) over a
fixed tree, checked through the validation-driven harness. Own suite.
Part of #105 (Tier 2)
Grow the Tier-2 conformance corpus with 4 Canonical XML mode cases: exclusive
drops an unused namespace and renders one at its point of use; Canonical XML
2.0 drops whitespace-only text nodes and trims surrounding whitespace. Its own
suite (ConformanceCanonicalModesTests).
Part of #105 (Tier 2)
Grow the Tier-2 conformance corpus with 7 Schematron cases (assert satisfied/
violated, an inert non-matching context, numeric comparison, and report fired/
not-fired) checked through the validation-driven harness. Moved to its own
suite (ConformanceSchematronTests) so the main corpus stays under the
type-body cap.
Part of #105 (Tier 2)
Rule-canon audit (validation-rules.md / the OpenAPIKit idiom): the streaming
XSD validator was the one surface applying a bare method (validateShallow)
rather than a Validation<> value. Wrap it: ComplexValidator.shallowValidity is
a Validation<ResolvedElement, ComplexValidator> whose subject carries the
element with its already-resolved effective type (the contextual typing a
node-tree walk cannot express), and the streaming driver applies it via
apply(to:at:in:). Now every validation surface, DTD, XSD, RELAX NG, Schematron,
HTML, structural, conformance, and both streaming validators, routes through a
Validation value. Adds a direct unit test of the rule.
The fragment TreeBuilder gains the HTML5 active-formatting-list and reconstruct
step that the document builder already had: when a formatting element (b, i,
em, ...) is closed out of order, it is remembered and re-opened for the
following content. So HTML.parse now agrees with HTML.parseDocument and the
spec, e.g. <b><i></b>X</i> -> <b><i></i></b><i>X</i> and
<p>1<b>2<i>3</b>4</i>5</p> -> <p>1<b>2<i>3</i></b><i>4</i>5</p>. The conformance
corpus that surfaced this now checks the fragment path directly.
Closes#109
Grow the Tier-2 conformance corpus with 8 HTML5 cases (void elements, implied
li/p closing, tag-case normalization, the adoption agency with and without
trailing text, entity decoding, stray-end-tag) checked through the
validation-driven harness against the HTML5 parsing algorithm.
The corpus surfaced its first real divergence: HTML.parse (fragment) does not
reconstruct active formatting elements the way HTML.parseDocument (document,
spec-correct) does, so <b><i></b>X</i> yields <b><i></i></b>X instead of
<b><i></i></b><i>X</i>. The corpus uses the document path (spec-correct) and
the divergence is filed as #109.
Grow the Tier-2 conformance corpus with 10 XSLT cases (value-of, for-each,
if, choose/otherwise, sort, variable, format-number, concat, apply-templates,
and a named template with a parameter) transformed through the public API and
checked through the validation-driven harness against the spec output.
Part of #105 (Tier 2)
Grow the Tier-2 conformance corpus with 12 RELAX NG pattern cases (text,
optional, oneOrMore, choice, ordered group, required attribute, interleave,
empty) validated through the public compact-syntax API and checked through
the validation-driven harness against the pattern semantics.
Part of #105 (Tier 2)
Grow the Tier-2 conformance corpus with 13 XSD datatype and facet cases
(integer lexical space, minInclusive/maxInclusive range, length, pattern,
enumeration, boolean) validated end-to-end through the public schema API and
checked through the validation-driven harness against the spec verdict.
Part of #105 (Tier 2)
Grow the Tier-2 conformance corpus with 16 XPath core-function cases (concat,
substring, string-length, normalize-space, translate, substring-before/after,
contains, starts-with, round/floor/ceiling, count, and the Infinity/NaN
division cases) checked through the same validation-driven harness against
their spec-mandated string results.
Part of #105 (Tier 2)
Begin Tier-2 conformance checking, expressed through the OpenAPIKit validation
idiom rather than an ad-hoc loop. ConformanceCase is a Validatable named
expectation; Conformance.matchesExpected is a Validation<ConformanceCase, Void>
that emits one located ValidationError per diverging case, so a suite reports
every failure at once with the case name as its coding path. Seeded with a
Canonical XML 1.0 corpus (attribute ordering, empty-element expansion, comment
removal, CDATA-to-escaped-text, PIs, namespace rendering) and a test proving a
divergence is reported as a located failure. Growing the corpora against the
official suites is the ongoing Tier-2 work.
Part of #105 (Tier 2)
Extend an identity error's coding path past the target element to the single
field the constraint names: an attribute (list/item[2]/@id) or a child element
(.../code), resolved from the field XPath. Multi-field or complex-path
constraints stay on the target. Also add a cross-scope keyref test confirming
a keyref resolves against a key declared on an ancestor element. Updates the
identity-path tests to the more precise locations.
Closes#104
Add depth tests confirming the existing DTD machinery handles parameter
entities within element-model and attribute-list declarations, deep (5-level)
PE composition, and an external subset that itself loads a further external
parameter entity through the resolver. The Tier-3 DTD PE / external-subset
depth item was already implemented; these lock it to full depth. (The XML
1.0 PE-reference spacing rule, leading/trailing space on DTD-content
inclusion, remains a known minor non-conformance.)
Part of #105 (Tier 3)
Add the EXSLT common/math/sets modules, dispatched by resolving a function's
prefix to its namespace (FunctionTable.call falls back to namespace lookup),
so any prefix bound to an EXSLT namespace works. math:min/max/highest/lowest/
abs/sqrt (with the NaN rule), set:distinct/difference/intersection/
has-same-node/leading/trailing (by node identity and document order), and
exsl:object-type. math:power and the transcendentals are omitted (they need a
C math library the Foundation-free target excludes); exsl:node-set awaits
result-tree-fragment support.
Part of #105 (Tier 3)
Add a canonical11 preset / mergeInheritedBase option. On a document subset,
the omitted-ancestor xml:base chain is merged into the apex by RFC 3986
reference resolution (vs 1.0 nearest-wins), the apex's own relative xml:base
is resolved against it, and xml:id is no longer inherited; xml:lang/xml:space
still inherit nearest. A self-contained RFC 3986 resolver (parse, path merge,
dot-segment removal) backs the merge, with tests covering the 1.1-vs-1.0
difference, dot segments, xml:id, and lang/space.
Part of #105 (Tier 3)
Add EUC-TW for the whole-buffer and streaming decoders: ASCII, a two-byte
plane-1 pair, and a 0x8E four-byte plane selector plus a pair. CNS planes 1
and 2, which carry essentially all real-world EUC-TW text, are vendored from
the ICU euc-tw-2014 mapping (private-use fallbacks dropped); planes 3-15 are a
documented follow-up. Tested for plane 1 (two-byte and four-byte forms), plane
2, and streaming. The streaming method moved to its own file to stay under the
type-body cap.
Closes#108
The streaming decoder previously selected an encoding only from a byte-order
mark or the UTF sniff, so its single-byte and CJK decode paths were latent. It
now reads the encoding from the XML declaration (the libxml2 behavior) when
there is no BOM, activating those paths over an incremental byte source. Add
the streaming form of ISO-2022-JP, carrying the stateful mode across reads
(extracted to its own file to stay under the type-body cap). A test asserts
streaming agrees with whole-buffer decode for a single-byte (ISO-8859-1), a
CJK (Shift_JIS), and an ISO-2022-JP document.
Part of #108, #105
Add Schema.Document.validate(streaming:) and ComplexValidator.validateShallow:
resolve each element's type from its parent's content model (and xsi:type /
typeReference) as it opens, then check its attributes and content-model
structure shallowly as it closes, against a shallow element synthesized from
the streamed child names. A child's deeper validity is checked when it closes,
so only the open-element stack is held. A differential test asserts streaming
agrees with the tree validator across valid and invalid documents (missing
attributes, bad simple values, missing/extra/misordered children). Wildcard
processContents and substitution-group resolution in the streaming form stay
on the tree validator. The streaming methods moved to ComplexValidatorStreaming
to keep ComplexValidator under the file/type-body caps.
Closes#107
Add the WHATWG iso-2022-jp decoder, the first stateful encoding: escape
sequences switch between ASCII, JIS X 0201 Roman (yen, overline), JIS X 0201
katakana, and the JIS X 0208 plane, resolved through the shared jis0208 table
(no new data). Tested across all four modes, with the JIS X 0208 kanji
cross-checked against the EUC-JP decoder (same table, same pointer). EUC-TW and
the streaming form of ISO-2022-JP remain.
Part of #108, #105
Add RelaxNG.validate(streaming:), driving the Brzozowski derivative directly
from the parse-event stream (the libxml2 xmlTextReader model). The derivative
is already incremental: the after-threaded pattern chain is the validation
stack, so only the residual pattern and a light per-element frame (for the
childrenDeriv ignorable-whitespace rule) are retained, never the tree. A
differential test asserts streaming agrees with the trusted tree engine across
valid, invalid, whitespace, and mixed-content documents. XSD streaming remains.
Part of #107, #105
Complete the DTD streaming validator with document-scoped ID/IDREF checks,
held in bounded sets (the identifier values, not the tree): a duplicate ID
is reported at its repeat occurrence, and forward IDREFs resolve at the end
of the stream via finish(). The streaming-vs-tree agreement test now covers
a duplicate ID as well. DTD streaming validation (content, attributes,
ID/IDREF) is now complete; RELAX NG and XSD streaming remain.
Part of #107, #105
Add PureXML.Validation.StreamingDTDValidator and PureXML.validate(streaming:
dtd:), validating a document against a DTD while it is pulled event by event
(the libxml2 xmlTextReader model), holding only the open-element stack rather
than the whole tree. Each element's content model and attributes are checked
as it closes by applying the exact per-element DTD rules to a shallow element
synthesized from the streamed child names, so streaming and tree validation
agree (a test asserts matching reasons). Errors carry an xmlGetNodePath-style
coding path. RELAX NG/XSD streaming and ID/IDREF streaming remain on #107.
Part of #107, #105
Extend ByteEncoder to the multi-byte CJK encodings (Shift-JIS, EUC-JP,
EUC-KR, GBK, GB18030, Big5). Each inverse is built by running the existing,
tested whole-buffer decoder over every valid byte sequence and recording
the shortest sequence per scalar, so the encoder is by construction the
exact inverse of the decoder, with no new data or separate encode tables.
GB18030's astral range uses the four-byte pointer formula, so every scalar
encodes. Round-trip tests cover Japanese, Korean, Chinese, and a GB18030
astral character.
Closes#106
