Fifth serialize-path pass (#139, docs/benchmarks.md pass 31). textNeedsEscaping,
attributeNeedsEscaping, and the comment/processingInstruction existence checks
run unconditionally on every text node, attribute value, comment, and PI in a
document, scanning through UTF8View's default Collection iteration. Profiling
a new deeply-nested, attribute-dense corpus (the standard flat catalog corpus
has no density to indict this) put the two needsEscaping functions as the
largest PureXML-specific frames in the sample. A standalone micro-benchmark
confirmed withContiguousStorageIfAvailable's raw buffer scan measures ~2-2.75x
faster than the Collection-based scan, with a Collection fallback kept for a
non-contiguous (foreign/bridged) string.
Two abstraction attempts were tried and rejected before landing this shape: a
shared generic helper taking a predicate closure recovered most but not all of
the win, and calling a separate small function from inside the buffer loop (to
dodge a cyclomatic-complexity violation) erased nearly the entire win even
though it isn't a closure value. The violation is instead resolved by splitting
each function into a contiguous/collection pair, moving the call boundary
outside the loop rather than inside it.
Output is byte-identical by construction; full local suite (1925 tests) and
full lint unchanged. Measured with interleaved same-state A/B under real
concurrent machine load: clean with zero overlap across 5 pairs at the nested
corpus (~9%), noisier at the standard corpus but the means still separate
clearly (~4%).
Two profile-guided serialize passes (#139, docs/benchmarks.md passes 29-30).
The attribute writer now precomputes the quote character as a String once
per element instead of interpolating a Character into every attribute,
avoiding the grapheme-breadcrumb machinery that repeats twice per attribute
for a value that never changes within an element. Escaping.comment and
processingInstruction's escaping fallback (taken whenever a comment has a
hyphen or PI data has a '?', a case pass 8 could skip past but not speed up)
is rewritten in the same byte-scan bulk-copy shape as passes 23-24, so a
value with one rare trigger no longer pays a full character-by-character
rebuild.
Both are same-observable-behavior changes. The quote change is byte-
identical by construction. The comment/PI rewrite is verified against a new
differential suite that reimplements the original character-loop as an
independent oracle and checks adjacency/trailing/combining-mark cases plus
two 20000-case seeded fuzzes, since a byte match inside a combining-mark
cluster is not, on its own, the bare trigger character. Full local suite
(1925 tests) unchanged.
Measured with interleaved same-state A/B: pass 29 ~1.5% at a 100k-item
corpus; pass 30 ~2x on a dedicated comment/PI micro-benchmark (the standard
corpus has no comment/PI content, so real documents with copyright banners
or hyphenated comment prose see this, not the headline ratio). A same-
session multi-append restructuring attempt measured a clear ~6% regression
and was reverted; a suspected hasInlineContent double-scan was profiled at
~0.2% of samples and not worth a pass. Both outcomes are recorded in
docs/benchmarks.md rather than dropped.
The two monolithic XSD content/identity rules only satisfied the OpenAPIKit
validation idiom at the wrong granularity: no per-constraint removability,
no isolable failing/near-miss tests. Split them into 32 named rules, each
finding tagged at its origin and cached per XSDContext so the walk still
runs once per document (the #101 decision against duplicating XSD's
recursive type resolution still holds). Wired the streaming path's ID/IDREF
and identity checks through the same framework instead of calling the
validators imperatively, and added the unwrapAndLookup combinator the rule
already named as required.
Also includes the mandated repo-wide swiftformat/swiftlint pass.
The byte-path close-quote consume candidate measured NEUTRAL in an
interleaved A/B despite the sampler attributing ~4% of parse to the
advance() chain it replaced: decoding one ASCII scalar into a
small-string-backed Character costs about the same as consume's
literal-matching path. Recorded under Known structure of the gap so no
future round redoes it, with the meta-lesson that sample attribution
overstates single-character round trips and only the interleaved A/B
verdict counts.
Also records the #8 decision: the TreeNode weak-parent side-table cost
(~9% of parse) is accepted for now. unowned is rejected outright since a
trap after release contradicts the never-crash posture and the documented
nil-after-release behavior; the arena/index-based tree remains the
designed future path, sequenced after the scanning-architecture work,
which carries ~47% of parse to this finding's ~9%.
Fourth profile-guided pass (#139, docs/benchmarks.md pass 28). scanAttributes
allocated a fresh Set<String>'s native storage on every element with
attributes, just to check a handful of names for duplicates; the set is now a
stored EventReader scratch property cleared with removeAll(keepingCapacity:)
per element, so each element's check reuses the previous element's storage.
Safe because scanAttributes cannot re-enter itself (nothing inside an
attribute scan starts another start-tag scan), so the shared storage never
sees two interleaved scans; duplicate detection, recovering-mode behavior,
and thrown errors are unchanged.
Verified in an isolated worktree at the previous commit: full local suite
(1858 tests) and full W3C XSTS suite (14383 groups) pass unchanged. Measured
with an interleaved same-state A/B, baseline/patched/baseline/patched each
best-of-5, because the machine carried concurrent load this session: both
patched runs (0.1073s, 0.1067s) beat both baseline runs (0.1091s, 0.1117s),
~2-4% on parse.
Recorded alongside, per the benchmarks log's discipline: a sibling candidate
(reserving each tree frame's children capacity on first append) measured
neutral under the same protocol and was reverted rather than landed, and the
TreeNode weak-parent side-table cost (~9% of parse, swift_weakAssign per
child plus a runtime side table per parent) is documented under Known
structure of the gap as needing a design decision, since parent's
nil-after-release semantics are documented public API. Touched lines carry
repo-standard swiftformat rewraps applied by a concurrent formatter run.
Third profile-guided pass (#139, docs/benchmarks.md pass 27), targeting
scanContentMarkup(), which inlines the start-tag/attribute/namespace scanners
and carries ~46% of parse time.
scanAttributeValue's opening and closing quote checks ran on the Character
path (peek + advance, buffering the quote as a Character only to compare and
discard it) on every attribute value in the document; both now check the raw
byte first, reaching the byte fast path with no Character buffering, with the
Character path kept as the off-fast-path fallback for streaming sources. The
quote bytes are ASCII and non-CR/LF, so the raw byte and the normalized
stream agree, the same reasoning peekByte's other callers already rely on.
NamespaceContext.checkExpandedNames allocated a Set's native storage and
built a concatenated URI+local key string per element even when at most one
attribute carries a namespace URI, where no duplicate is possible (the
default namespace never applies to attributes, and xmlns declarations carry
no URI). Counting URI-bearing attributes first, with no allocation, and
returning early below two skips all of it on the overwhelming majority of
elements; for xml:* attributes specifically the key exceeded the small-string
limit, so this also removes a real per-element heap allocation.
EntityDecoder.decode/decodeLenient guarded with raw.contains("&"), a
grapheme-by-grapheme walk to find one ASCII byte, once per ampersand-bearing
text node or attribute value; raw.utf8.contains(0x26) finds the same byte
without decoding graphemes, the same shape as checkNameShape's colon check.
The quote-handling extraction into two helpers also resolves the cyclomatic
complexity the inline version tripped. Six new regression tests pin the
quote edge cases (empty value, single-quoted, a quote kind nested inside the
other, unterminated at end of input) plus byte/Character-path agreement for
each in the parser path differential.
Verified: full local suite (1858 tests) and full W3C XSTS suite (14383
groups) pass unchanged. Parse ~6.00x -> ~4.75x versus libxml2 at the
20000-item corpus, best of 5.
Two profile-guided passes (#139), found by sample-profiling a 42MB generated
corpus after re-measuring the parse/libxml2 ratio fresh:
scanAttributeValue's closing-quote check re-derived an already-known answer
through Reader.consume's general String-iteration path, which allocates a
fresh String(quote) and iterates it via Character machinery twice, on every
single attribute value in the document. The content loop's own break
condition already guarantees the buffered peek is either the quote or end of
input, so checking it directly and advancing once is the same behavior for
less work.
NamespaceContext.enterElement, called once per element: checkNameShape's
name.localName.contains(":") resolved to Sequence.contains(_:), walking
extended grapheme clusters to find one ASCII byte; utf8.contains(0x3A) finds
the same colon via the UTF-8 byte view. Separately, attributes.map{
resolveAttribute } allocated a fresh backing array on every element even
though resolveAttribute returns its argument unchanged whenever the attribute
has no prefix or the xmlns prefix (the common case); checking first with no
allocation and reusing the input array when nothing would change gives the
identical result without it.
Both are same-observable-behavior changes, verified against the full local
suite (1852 tests) and the full W3C XSTS conformance suite (14383 groups),
both unchanged. Measured parse ~6.59x -> ~6.00x versus libxml2 at the
20000-item/4.2MB corpus, best of 5; see docs/benchmarks.md passes 25-26.
An element restricting a base choice was always checked by the
effective-total-range approximation, which exists for one reason: the
literal section 3.9.6 pairing has no allowance for occurrence counts
accumulated across the choice's own repetitions and over-rejects
W3C-read-valid schemas (particlesZ001, whose manifest calls the rule
ambiguous). That ambiguity is impossible when the choice cannot repeat:
at {max occurs} = 1 every accepted child comes from one traversal of one
branch, so the literal rule governs. elementRestrictsChoice now applies
the literal pairing (the derived element, with its own occurrence range,
must validly restrict some non-pointless branch) when base maxOccurs is
1, and keeps the effective-range reading only for repeating choices.
Closes XSTS particlesHa161 (a derived a{0,1} is not a pointwise
NameAndTypeOK restriction of a required branch a{1,1}) and particlesM034
(a derived d1+ cannot map into the branch (d1*, d2+): the required d2 is
unmapped and not emptiable). The settled-valid neighbors that pin the
boundary from the other side all still pass: Ha160, M002, M003, M035,
and the Z001 shape, each now a regression test in
XSDParticleChoiceRestrictionTests. invalid-schemas-accepted 6 -> 4.
particlesIg004 is reclassified as spec-divergent (4 -> 3): section 3.3.2
maps block="#all" and the explicit substitution/extension/restriction
list to the SAME {disallowed substitutions} set, so NameAndTypeOK clause
5's superset requirement holds and the invalid expectation has no spec
path; it is jointly unsatisfiable with the settled Ig001/002/003 (equal
sets, valid) and Ig005 (#all covers the triple, valid).
The remaining three schema deviations are documented divergences, not
open bugs: Ik025/Ik027 (cos-st-derived-ok 2.2.4 reading) and schN12,
newly adjudicated in docs/xsts-deviations.md: its chameleon redefine
fails src-redefine 6.2.2 on NameAndTypeOK clause 1 (no-namespace locals
vs qualified locals), but 6.2.2 cannot be enforced uniformly because
mgO013's settled-VALID schema also violates it; any uniform enforcement
trades a false negative for a valid-schema-rejected false positive.
Also corrects the stale deviations-doc row that listed the already-closed
particlesIk011 and omitted schN12; the failures file is the authority.
Gates: full XSTS release suite green at 0/3/0/4 (valid buckets unchanged
at zero), libxml2 differential green at 244 pinned entries (Ha161 and
M034 newly pinned, both compile under libxml2), full local gate green.
NameAndTypeOK clause 1 (XSD 1.0 section 3.9.6, Elt:Elt) pairs the declared
element declarations and requires the same {name} and {target namespace};
no clause grants substitution an allowance. The substitution-group
expansion rewrites the compiled term (an abstract head is dropped, members
join a synthetic choice), so the checker previously compared the expanded
names and a head-for-member swap looked name-identical.
Particle.declaredReference now preserves the pre-expansion identity of an
element-ref particle, and the restriction check pairs the declared
references before the structural pairing table (extracted as
structuralValid). Closes XSTS elemZ026 and particlesV020:
invalid-schemas-accepted 8 -> 6, valid buckets unchanged at zero.
Stricter than libxml2, which does not implement this edge; elemZ026 is
pinned in the differential baseline accordingly.
particlesZ008, byte-for-byte the same construct as elemZ026 but carrying
a settled valid expectation contradicting elemZ026's settled invalid, is
reclassified as a spec-divergent schema test with the adjudication
recorded in the suite (exclusion tables split into
Tests/XSTSSuiteExclusions.swift for file length).
Separately attributed: the differential baseline gains ten further pinned
entries (addB070a, addB108, addB177, attO025, elemZ027_c, errC008,
groupH021v, notatB005, notatE002, particlesEb040) that reproduce with
this change stashed. They are pre-existing drift from earlier main-line
fixes, not effects of this commit. Baseline 231 -> 242, differential
green.
The roadmap and README cited issue numbers from the deleted GitHub
tracker and carried stale XSTS figures (0/39/0/26 and 0/43/0/31 against
the enforced 0/8/0/4). Recreate the load-bearing issues on Codeberg
(#3 production-readiness epic, #4 and #5 gate trackers, #6 performance,
#7 Xalan burn-down), cite them, correct every baseline figure to the
test ratchet, mark shipped work (located diagnostics, the differential
harness) as done, and note that pre-migration numbers in the historical
narrative refer to the lost tracker.
Closes#2
Every unique commit on the branch (the changelog stamp, the Codeberg
repoint, and the rules-baseline stamps) was re-applied on main before
tagging v0.4.8, so the merge records the supersession without content
changes.
The earlier release/0.4.8 branch stamped the changelog but the tag was
never cut, and main moved 60 commits past it. Fold the accumulated
unreleased entries into the still-unpublished 0.4.8 section, redate it,
repoint every remaining github.com/mihaelamj reference (badges, support
links, roadmap, the XSLT vendor-url system property), and track the
shared-rules baseline v1.1.2.
Adopt the rules-swift pin-and-codify strategy in snapshot mode (third_party/rules-swift
@97dc70f + sha256), add codified gates, a conformance audit ledger, and
.github/workflows/rules-swift.yml (corpus-pin integrity required, full gate report mode).
Remove the docs/rules mirror it supersedes (repo-specific files kept under docs/). Style
and leak gates exclude the vendored corpus.
The rules-swift concurrency rule's acceptance check requires every `@unchecked
Sendable` and `nonisolated(unsafe)` to carry an adjacent comment explaining why
the race is acceptable; an unjustified one is a violation. An audit found six
escape hatches whose comment explained the type but not the race:
- The four XPath `FunctionTable` statics (`CoreFunctions`, `StringFunctions`,
`NodeFunctions`, `Evaluator.library`): each is an immutable, build-once lookup
table never mutated after initialization, so concurrent reads race on nothing;
`FunctionTable` is un-`Sendable` only because it stores closures.
- `Validation.Validator`: immutable after `init` (every stored validation list is
a `let`); the erased `AnyValidation` closures are simply not provably `Sendable`.
- The test `ResultBox`: written once by the worker thread before its semaphore
signals and read only after the wait, so the write happens-before the read.
Comments only; no behaviour change. `LoaderBox` and `ElementStorage` were already
justified. swiftformat/swiftlint --strict and the full suite stay green.
The instance-validation differential treated "PureXML rejects, libxml2 accepts"
as a PureXML false positive. That is unsound: libxml2's own content validation is
lenient and accepts some INVALID instances. A DIFF_SEED sweep surfaced exactly
such a case (seed 13371337): the model ( (a*|c*){1,3}, b? ){1,3} and the instance
<doc><b/><b/><a/><b/><c/><a/></doc>. libxml2 accepts it, but it is genuinely
INVALID (the three b's anchor three (choice,b?) segments, leaving the trailing
'ca' with no fourth segment), so PureXML's rejection is correct, not a false
positive. Verified against an external regex engine and exhaustive enumeration.
Add ContentModelOracle, an independent brute-force membership matcher over element
names, implemented separately from PureXML's matcher. The generated instances are
empty xs:string elements, so an instance is valid iff its element-name sequence is
in the model's language, which this oracle decides exactly. The generator now
emits the structured ContentModel alongside the schema string (group references
inlined), and the differential classifies a false positive / false negative
against the ORACLE as ground truth; libxml2 is kept as a secondary cross-check
whose disagreements with the oracle are counted (its leniency) but never charged
to PureXML.
The oracle is unit-tested (ContentModelOracleTests), including the libxml2-leniency
case it exists to catch. Over a sweep of seeds the differential now reports zero
oracle-adjudicated false positives and zero false negatives, so PureXML's content
matcher agrees with the independent oracle on every generated case; the default
seed is unchanged. Test-only; no library behaviour change.
The three generated-vs-libxml2 differentials (instance, facet, determinism) each
ran a single hardcoded seed, so they characterised one fixed slice of the open
input space rather than sampling it. Each now reads an optional DIFF_SEED
environment override (defaulting to its existing seed, so the default run is
unchanged), turning the fixed smoke test into an open-space fuzzer: a sweep over
many seeds explores far more of the unbounded schema/instance space than the
finite XSTS corpus, which is the production-readiness "uncharacterized
correctness" stopper.
A 12-seed sweep of the instance differential immediately surfaced a genuine
content-matcher false positive (cardinal-rule violation) at DIFF_SEED=13371337:
the model `( (choice(a*|(c*)+)){1,3}, b? ){1,3}` rejects `<doc><b/><b/><a/><b/>
<c/><a/></doc>` ("element 'c' is not allowed here") though libxml2 accepts it. A
nested nullable-repetition edge in the graph-structured-stack matcher, captured
for a focused fix; the default seed and every other swept seed hold false
positives at 0.
particlesZ001 declares Derived as a complexContent restriction of Base and an
instance whose <elem> (type Derived) content is annotation + four elements. The
suite pairs a valid schemaTest with an invalid instanceTest, but the two are
mutually exclusive for a consistent conformant processor, and the manifest's own
documentation labels the governing rule, Particle Derivation OK (RecurseAsIfGroup),
ambiguous.
cvc-complex-type (§3.4.4) validates an instance against the element's own
{content type}: for <elem> that is Derived = sequence(annotation?, element*),
which the instance satisfies, so the instance is valid. Particle Valid
(Restriction) / RecurseAsIfGroup (§3.9.6) is a schema-assembly constraint that is
never re-run during instance validation, so there is no path from an accepted
schema to an invalid instance. The only way to reach the suite's invalid is to
reject the schema as an invalid restriction, which is a valid-schema-rejected
false positive and would still fail the suite's own valid schemaTest. libxml2
accepts both schema and instance, agreeing with PureXML.
Exclude particlesZ001.i via specDivergentInstances with the spec-grounded
justification in code and docs/xsts-deviations.md. invalid-instances-accepted
5 -> 4; full XSTS in release now 0/8/0/4, both false-positive buckets at 0. No
library behaviour change. Confirmed by an adversarial analysis that found no
spec-coherent reading yielding valid-schema + invalid-instance.
An xsi:type value is a QName: a prefix binds through that prefix, an unprefixed
name through the in-scope default namespace, and with no default namespace an
unprefixed name is in no namespace. PureXML stores each named type under both its
namespaced key and a bare local-name alias (for lenient lookup), so an unprefixed
xsi:type="Test" matched a {targetNS}Test through the alias even when the instance
had no default namespace, wrongly typing an otherwise-undeclared root and
accepting the document (W3C XSTS targetNS00101m; libxml2 rejects it identically).
xsiTypeReference now keys an unprefixed xsi:type to the in-scope default namespace
(none here) rather than returning the bare local name, so xsi:type="Test" names
no-namespace Test, which does not exist; the root then has neither a declaration
nor a usable type and the instance is invalid. A prefixed xsi:type and an
unprefixed one under a default namespace equal to the target namespace are
unaffected, and the child-element builtin-substitution path keeps its bare-name
fallback (xsiTypeName), so an unprefixed builtin still substitutes where valid.
Verified over the full W3C XSTS corpus in release: invalid-instances-accepted
6 -> 5, valid-instances-rejected held at 0 across all 14383 groups, full suite
(1840 tests) and wasm green. An adversarial critic traced every resolveNamedType
branch, including the unprefixed-builtin-via-key case, and confirmed the only
path that no longer resolves corresponds to a spec-invalid reference, so no valid
instance is newly rejected.
mgO013 redefines a model group `group` to `all{e1(1,1)}`, redefining a base
`all{e1(0,0)}` (e1 forbidden) without referencing itself. The suite pairs a
`valid` schemaTest with an `invalid` instanceTest, but the two are mutually
exclusive for a consistent conformant processor.
src-redefine clause 6.2.2 makes a no-self-reference group redefine a restriction
obligation: the redefined group must be a valid restriction of the original.
`all{e1(1,1)}` is not a valid restriction of `all{e1(0,0)}` (Particle Valid
(Restriction), Occurrence Range OK: derived max 1 > base max 0), so enforcing
6.2.2 rejects the schema, a valid-schema-rejected false positive against the
schemaTest's own `valid` verdict. Yet §4.2.2 makes the referencing type's group
reference resolve to the redefined `all{e1(1,1)}`, against which the single-e1
instance is valid. There is no third path: reaching instance-invalid requires
rejecting the schema. libxml2 accepts both, agreeing with PureXML and the
schemaTest. An adversarial analysis confirmed no reading yields valid-schema +
invalid-instance.
Exclude mgO013.i via specDivergentInstances with the spec-grounded justification
in code and docs/xsts-deviations.md. invalid-instances-accepted 7 -> 6; full XSTS
in release now 0/8/0/6, both false-positive buckets at 0. No library behaviour
change: matching the suite would introduce the false positive the production bar
forbids.
XSD 1.0 §3.11.4 (cvc-identity-constraint.3) requires each key/keyref/unique
field to identify a node of simple type. A field that reaches an element
declared with no type (the ur-type anyType) or an inline complex type without
simple content does not, so the instance is invalid; the reference (libxml2)
rejects it identically. PureXML accepted such instances because its identity
validator could resolve a field target's type only for a global element reached
by a direct-child name, not for a local element reached through a path (the W3C
XSTS case idZ010 nests anyType fields several levels deep, e.g. a key field
./b/c where c has no type).
A compile-time pass (XSDParser.identityNonSimpleFieldKeys) resolves each field's
target element through its selector and field paths, by element-name steps, and
records the fields whose declared target is provably non-simple. The validator
rejects those at runtime unless the instance gives the node a simple type
through xsi:type, since cvc reads the node's governing type (a text-only anyType
element with xsi:type="xsd:int" is a valid field). The resolution is
conservative: any wildcard, descendant axis, predicate, ref, or named
non-builtin type leaves the field unflagged, so a valid field is never rejected.
The one residual divergence path, two same-name child elements of differing
simplicity in one inline model, is an Element-Declarations-Consistent violation
the compiler already rejects.
Verified over the full W3C XSTS corpus in release: invalid-instances-accepted
8 -> 7 (idZ010 now rejects), valid-instances-rejected held at 0 across all 14383
groups (no false positive), full suite (1837 tests), wasm, and lint all green.
An adversarial critic that read every file and probed for a false positive could
not construct one; the failure mode is under-rejection, the safe direction.
The sch* schema-location cluster, dissected per case rather than swept. An
adversarial analysis separated the contested tests from the genuine deviations.
schA2.i and schA5.i are contested and now excluded (invalid-instances-accepted
10 -> 8). Each group's own documentation states the inline xsi:schemaLocation
hint "is ignored" and the instance "must conform to" the collection schema, but
under exactly that prescribed behavior the instances are valid: libxml2 validates
schA2.xml against its a.xsd and schA5.xml against its a.xsd+b.xsd collection. So
the suite's invalid expectation contradicts its own documentation. Reaching
invalid requires letting the instance's hint override a schema already loaded for
that namespace, which XSD 1.0 §4.3.2 makes optional (strategy 1: do nothing when a
schema for the namespace is already known) and which would be a direct false
positive for an IDE: an instance file forcing rejection against the project's own
schema. PureXML's policy (honor hints but keep the collection's declarations on
conflict) is conformant.
schA7.i is NOT excluded. The adversarial pass showed its invalid IS spec-grounded:
with hints ignored, the instance's b:e1 (namespace ns-b) has no declaration in the
a.xsd-only collection and is demanded by root's default-strict ##any wildcard, so
§3.10.4 makes it invalid. PureXML accepts it only by honoring the ns-b hint to
gap-fill, the other conformant §4.3.2 strategy. A genuine policy-dependent
deviation, left in the count and documented.
schU3.i/schU4.i/schU5.i are also left in the count, documented as a deferred
under-rejection in an underspecified area: circular redefine chains where an
attribute group is redefined to conflicting types across an include/redefine
cycle. The suite expects the int reading (so c1="a" is invalid); PureXML resolves
the laxer type and accepts. libxml2 does not support the construct at all (rejects
the schema: multiple redefinition not supported), so matching the suite means
out-resolving the reference on a construct whose cyclic composition order the spec
does not pin down.
Full XSTS in release now 0/8/0/8, both false-positive buckets at 0. No library
behavior change; reasoning recorded in code and docs/xsts-deviations.md.
The W3C XSTS group anyURI_a004_1339 pairs a `valid` schemaTest with an `invalid`
instanceTest, but the two expectations are mutually inconsistent, so PureXML's
acceptance of the instance is spec-correct rather than an under-rejection.
The schema's include, import, and redefine carry unresolvable schemaLocations
(ftp://, gopher://, mailto:). XSD 1.0 §4.2.1 makes an unresolvable include "not
an error" (no inclusion performed), the import likewise, and the empty redefine
imposes no resolution requirement (src-redefine clause 1 gates that on
non-annotation children), so the schema is conformantly valid, agreeing with the
schemaTest. The only components the instance exercises (root, bar, st, att) are
fully locally defined, and every instance value is byte-identical to one of the
type's six anyURI enumeration members, so a strict assessment of the document
element is valid. There is no spec path to valid-schema + invalid-instance.
libxml2 is no authority here: it non-conformantly fails to compile the schema on
the unresolvable include, so it disagrees with the suite's valid-schema verdict
too. Matching the suite's invalid-instance expectation would mean rejecting a
valid instance whenever an include cannot be resolved, a false positive an IDE
editing loop hits constantly, which the production bar forbids.
Exclude it from the counts via specDivergentInstances, mirroring how the suite's
own status="queried" entries are skipped, with the spec-grounded justification
recorded in code and docs/xsts-deviations.md. invalid-instances-accepted 11 -> 10;
full XSTS in release now 0/8/0/10, both false-positive buckets at 0. Two
independent adversarial analyses failed to refute the instance's validity.
XSD 1.0 (Datatypes Appendix F) defines the regex escape \i as the characters
that may begin an XML Name (Letter | '_' | ':') and \c as the characters that
may appear in one (NameChar), citing the XML 1.0 Second Edition productions:
the enumerated, Unicode-2.0-era character classes of XML 1.0 Appendix B
(BaseChar, Ideographic, CombiningChar, Digit, Extender). \i and \c were wired
to PureXML.Parsing.XMLCharacter, which correctly implements the Fifth Edition
Name production used to parse live XML, but the two genuinely differ: the Fifth
Edition admits whole blocks where the Second Edition has gaps, so \c
over-accepted, e.g. the combining mark U+0346 (COMBINING BRIDGE ABOVE), which
Appendix B's CombiningChar excludes (its first range ends at U+0345).
Add XSDNameCharacter, transcribed clean-room from the Appendix B grammar
(identical across editions one through four) and stored as sorted, non-
overlapping code-point ranges with binary-search membership; the parser's own
name check is untouched. The two tables reproduce the W3C XSTS \i/\c
enumerations (reZ005, reZ006: 34516 and 35122 code points) exactly and in both
directions, so the more restrictive definition rejects no spec-valid input:
there is no false-positive surface.
Verified over the full W3C XSTS corpus in release: invalid-instances-accepted
12 -> 11 (reZ006i, whose value is U+0346, now rejects), both false-positive
buckets at 0. Full suite (1834 tests) green; an adversarial critic brute-forced
the binary search over the whole code-point space (0 mismatches), confirmed the
tables are sorted and non-overlapping and equal the oracle sets, and that the
parser's Fifth Edition path is left intact. Found by triaging the invalid-
instance tail to a single boundary code point (U+0345 valid vs U+0346 invalid)
and reading the XML 1.0 Appendix B grammar.
XSD 1.0 fixes the regular-expression repertoire at Unicode 3.2 (Datatypes
Appendix F), where U+1369..U+1371 (ETHIOPIC DIGIT ONE..NINE) are general
category Nd, each carrying a decimal value; a later Unicode reclassified them
to No. XSDCategory derives \d (== \p{Nd}) from the running platform's category,
pinned to 3.2 by an assigned-in-3.2 gate plus a per-code-point override table,
but the table omitted these nine, so \d used the modern No: a valid \d-constrained
Ethiopic digit was wrongly rejected (cardinal-rule false positive) and a
\D-constrained one wrongly accepted.
Pin the nine to Nd. This is the complete set of code points that were Nd at 3.2
and are not today; the reverse direction is empty (every later-Nd point was
unassigned at 3.2 and already reports Cn), verified by diffing the 3.2 and current
Unicode Nd sets. \w is unaffected: an Ethiopic numeral is category N either way,
neither punctuation, separator, nor other.
Verified over the full W3C XSTS corpus in release: invalid-instances-accepted
14 -> 12 (reT17, reT38, the \D cases, now reject), and the matching \d-valid cases
reS17/reS38/reZ004 no longer need a suspect-test exclusion and pass as ordinary
agreements, with both false-positive buckets at 0. Full suite (1830 tests) green;
an adversarial over-rejection critic re-derived the Nd delta bidirectionally and
confirmed no false-positive surface. A regression-pinning test exercises the
production regex path, including the boundary that U+1372 (No even at 3.2) is
not \d.
OrderedValue.== compared doubles with IEEE equality (NaN != NaN), so a NaN value never matched a NaN enumeration member and was rejected: <v>NaN</v> against xs:double restricted by <xs:enumeration value="NaN"/> is valid, and PureXML rejected it (a cardinal-rule false positive). NaN is now treated as identical to itself in value-space equality, the identity the reference (libxml2) and XSD 1.1 use throughout; an adversarial critic flagged the shared comparison (also used by key/unique and fixed), so the key/fixed behaviour was verified against xmllint: libxml2 collides two NaN unique keys and matches fixed="NaN", i.e. the same identity, so one identity == matches the reference everywhere. Ordering (<) keeps NaN incomparable, so range facets are unaffected. Found by a new generated-value facet differential (FacetValidationDifferentialTests, opt-in via FACET_DIFF) over ~10000 datatype/facet values: one FP found, zero after; the four W3C XSTS buckets held, full suite passes, SchemaDoubleEnumerationTests locks all three relations.
A long <xs:attributeGroup ref> chain (ag0 refs ag1 refs ag2 ...) is flat in the document, so flattening a complex type's attributes through it recursed once per link: the stack overflowed past ~6000 links and it was near-cubic before that (~17s at 3000). attributeUses already threads a visited cycle-guard set whose count is the chain depth, so bound it against maximumContentModelExpansionDepth (512): a chain past the bound is rejected with a located error rather than crashed. Behaviour-preserving within the bound, which no realistic schema or the XSTS corpus approaches; verified conformance-neutral over the full W3C XSTS corpus and the full suite. Residual: many distinct over-bound chains still reject in O(n^2) per group (attribute flattening re-done per group); a memo is the linear follow-up.
A long <xs:complexType> simpleContent extension chain (T0 extends T1 extends T2 ...) is flat in the document, so each of the N complex types re-walked its whole base chain to resolve the simple type its simpleContent derives from (an O(n^3) compile; a 400-link chain took ~23s). The resolved simple type is now memoized by complex-type name (SimpleContentMemo, a reference type surviving the value-type context copies, mirroring GroupParticleMemo); the same chain compiles in ~1.2s, and past the 512 expansion-depth bound it is rejected with a located error as before. Behaviour-preserving (an acyclic resolution is intrinsic to the name; a derivation cycle is rejected elsewhere), verified conformance-neutral over the full W3C XSTS corpus, the determinism differential, and the full suite. The attribute flattening along the chain is inherently quadratic in the bounded length and is tracked.
A long <xs:simpleType> restriction chain is flat in the document, so the same base was resolved repeatedly: the named simple types were resolved by an N-pass fixpoint, and the facet-applicability variety check re-scanned the schema's children to find each base at every link of every type's chain (compiling a 600-link chain took ~4.4s). Named simple types are now resolved once in dependency order (XSDSimpleTypeOrder), and the variety check uses an O(1) name map and memoizes each type's variety (SchemaSimpleTypeVariety); the same chain now compiles in ~30ms. Behaviour-preserving, verified conformance-neutral over the full W3C XSTS corpus, the determinism differential, and the full suite.
Two constructs are flat in the schema document, so they escape the parser's element-nesting maxDepth (256) yet make a content model arbitrarily deep once compiled: a <xs:group ref> chain and a complexContent derivation chain (the definitions are siblings). The particle compiler inlines each group reference and composes each derivation once per link, overflowing the stack on a long chain. The expanded depth is now bounded by one counter (XSDContext.expansionDepth, incremented at each compositor level, group-reference expansion, and derivation step) against maximumContentModelExpansionDepth (512); a model past the bound is rejected with a located error, not crashed. The counter lives in a value-type context so it tracks depth not width (no width false positive), and the rejection is harvested into the schema's errors (no silent truncation). Verified conformance-neutral over the full W3C XSTS corpus (all four buckets unchanged), the libxml2 determinism differential unchanged, and the full suite; an adversarial critic found the derivation-chain gap, which this fixes with the same counter.
A flat chain of <xs:group> references is not bounded by the parser's maxDepth (the definitions are siblings, not nested), so the compositional UPA determinism check's lazy per-reference recursion overflowed the native 8 MB stack while compiling a legal schema. Group summaries form an acyclic graph, so they are now precomputed bottom-up in dependency order (GroupDependencyOrder): the walk's group lookup is a cache hit and recursion is bounded by a single group's own model nesting. The precomputed reachable set is identical to the lazy walk's, so the verdict is unchanged. Verified verdict-equivalent over the full W3C XSTS corpus, the libxml2 determinism differential (0 FP, same 3 FN), the full suite, and an adversarial critic. Residual, tracked: the particle compiler that inlines a group reference still overflows on a far longer chain; closing it needs a content-model nesting-depth limit with a located error.
The schema-compile determinism check has a generated-schema differential; this adds the
counterpart for the instance-validation path, exercising the per-instance content matcher
(the graph-structured-stack ContentRTNMatcher now in production) over the open input space.
For every generated, both-engines-compile schema it validates many generated <doc>
instances with PureXML and with libxml2 and compares the verdicts, characterising the
cardinal rule's instance half beyond the finite XSTS corpus.
It asserts the cardinal rule (PureXML never rejects an instance libxml2 accepts) and
reports false negatives for triage. Over 6568 generated validations (1357 both-valid, 5211
both-invalid) across sequences, choices, all groups, group references, occurrences, and
wildcards it found zero false positives and zero false negatives: independent confirmation
that the matcher agrees with libxml2. Opt-in via INSTANCE_DIFF and xmllint, a no-op in
plain swift test and CI; the libxml2 subprocess is guarded for WASI. No behaviour change.
The interactive-safety bar wants a proven worst-case bound, not a cap or a timeout. The
graph-structured-stack matcher shares stack suffixes, so on a model with S program states
and an input of length N the live-configuration set and the GSS node count stay O(S*N)
where an explicit return stack reaches O(distinct stacks): a nullable doubling chain that
gives an explicit stack ~3^depth configurations gives the GSS O(depth).
ContentRTNMatcher now exposes peakLiveConfigurations and gssNodeCount, and the differential
asserts they stay linear in the chain depth (measured gss = K + 2, peak configs = 3K + 3),
by count rather than a machine-dependent wall-clock timeout, so a regression to super-linear
fails the test. Native-only (the depth-80 RTN build recurses on nesting depth and overflows
WASI's small stack; that builder depth-safety is tracked separately). No behaviour change.
A repeating model whose optional trailing particle and leading particle share a name,
(c, b?, c?)*, is ambiguous, as libxml2/Xerces reject it: after b the next c could be the
trailing c? or the looped leading c, two distinct particles. The compositional determinism
check formed the internal sequence gap before the enclosing repetition's loop edge was
added, so the combined {trailing, leading} followpos was never checked, an invalid-schemas-
accepted false negative the finite XSTS corpus did not contain.
The sequence now carries each trailing end-item's internal suffix in its lastFollow, so a
parent gap or the loop edge (withRepeatEdge) forms the end-item's full followpos. The
carried suffix is always a subset of the true followpos, so the check only catches more
genuine conflicts and never over-rejects: the non-repeated (c, b?, c?) and disjoint shapes
like (a, b?)* stay valid.
Found by the new generated-schema determinism differential against libxml2. Verified over
the full W3C XSTS corpus in release (valid-schemas-rejected/valid-instances-rejected stay
0, invalid-*-accepted unchanged at 8/14), full suite on macOS/Linux/WASI, and an
adversarial over-rejection critic finding no valid schema newly rejected over 47 shapes.
The related ambiguity through two references to one group in a repeating sequence stays an
under-rejection (the cached group identity cannot distinguish a looped from a trailing
occurrence without per-occurrence tracking), named in DeterminismDifferentialTests.
The XSTS corpus is finite and did not exercise the multiply-referenced-group overlap that
was a real UPA gap (it was found by reasoning, not the corpus). This adds the systematic
characterisation the production-readiness bar calls for: a generated-schema differential
that fuzzes content models (elements, group references, choices, sequences, all groups,
occurrences, wildcards) and compares PureXML's Unique Particle Attribution accept/reject
verdict to libxml2's. Opt-in via DETERMINISM_DIFF and xmllint, isolated to the determinism
dimension (non-determinism rejections are skipped).
It asserts the cardinal rule, zero schemas PureXML rejects as ambiguous that libxml2
accepts, and reports false negatives for triage rather than leaving them silent. Over this
generator it found zero false positives and one genuine, named under-rejection: a repeating
model whose optional trailing particle and leading particle share a name, (c, b?, c?)*,
where the compositional check forms the internal sequence gap before the enclosing
repetition's loop edge is added, so the combined decision set is never checked. The other
disagreements are contested (unused-group checking; (a?)+ epsilon loop) with PureXML the
more-correct lenient party. The genuine gap is tracked as a follow-up, its own FP-critical
fix.
Two references to the same <xs:group> are two expanded occurrences for Unique Particle
Attribution, so when they overlap, (g?, g?), (g | g), (g*, g), the model is ambiguous,
exactly as inline (x?, x?) / (x | x) is, and as libxml2/Xerces reject them. The
compositional determinism check summarises each group once and reused the group's node
identity across references, so the distinct-identity overlap test could not see the
conflict: PureXML accepted these (an invalid-schemas-accepted false negative). It also let
the per-instance content matcher run on a non-UPA model, where a multiply-referenced
nullable group is exponential, so a depth-18 optional doubling chain was accepted and then
hung instance validation: a keystroke-time hang on a schema PureXML itself accepted.
The fix dedupes each decision set (first set, sequence gap, all cross-follow) by particle
identity and treats a same-identity collision as the conflict; the repetition self-loop
(a*, (a*)*) keeps a non-flagging dedupe so one particle looping is not misread as two
occurrences. This also bounds the check, the deduped first/follow sets stay O(labels) not
2^K, so the depth-40 chain is now rejected at compile in 11 ms and no instance is matched
against it. A redefine self-extension lists a type's own content twice through
effectiveModelNodes; that artifact is deduped before the walk so it is not a false
conflict (addB094, schZ007).
The repeat-edge fold is kept in a separate method, not inline in summarize: summarize
recurses on nesting depth, so a local it holds multiplies the per-level stack frame and a
deep group chain overflows WASI's small stack (the recursive check's deeper WASI depth-
safety is tracked separately).
Verified over the full W3C XSTS corpus in release: valid-schemas-rejected and
valid-instances-rejected stay 0, invalid-*-accepted unchanged at 8 and 14; an adversarial
over-rejection critic found no valid schema newly rejected; the full suite passes on
macOS, Linux, and WASI (run locally under wasmtime).
When swiftly is absent, check-wasm.sh fell back to the active `swift`. The Xcode-bundled
Apple toolchain has no WebAssembly LLVM codegen backend, so a wasm cross-compile through
it does not fail cleanly: it aborts deep inside LLVM (clang module build, "No available
targets are compatible with triple wasm32-...") with a multi-megabyte stack dump that
reads like a code bug. The SDK-presence check passed (the SDK is installed) and the build
crashed anyway, so the real cause, the wrong host toolchain, was invisible.
Detect it up front and exit 2 with an actionable message. Discriminate by the compiler's
self-reported build tag, not its path: both toolchains print "Apple Swift version X", but
the Xcode/Apple build tags it "(swiftlang-...)" while a swift.org toolchain tags it
"(swift-...-RELEASE/SNAPSHOT)". The self-report is authoritative however `swift` is
symlinked or shimmed, which a path-string match would miss. `swift -print-target-info
-target wasm32-...` is not a usable probe: it succeeds even without the backend, because
the frontend knows the triple and only codegen is missing.
* perf(schema): graph-structured-stack content matcher, sharing each xs:group (alongside)
The per-instance content-model build inlines every xs:group reference, so a
multiply-referenced nested group (g0=(g1,g1), g1=(g2,g2), ...) explodes to 2^K
states and reset actions: a depth-16 doubling chain already takes seconds, depth
20+ hangs. This compiles the same particle to a counted Recursive Transition
Network instead, where each definitionID group is compiled once and reached by a
call, and simulates it with a graph-structured stack.
ContentRTNBuilder mirrors ContentNFABuilder exactly except that a group with a
definitionID becomes a call into its shared, compiled-once fragment (memoized by
id; the fragment's accept is a return). The callee's counters are deliberately not
propagated to the caller: every counter is reset on its own particle's entry edge,
which is on the path from the callee's start, so a shared fragment resets itself on
every re-entry without the caller re-listing it. Re-listing was the residual
blow-up: it nested the reset-action lists to 2^K even when the states stayed O(K).
ContentRTNMatcher runs configurations (state, counters, gssNode): a call saves the
caller's counters on a GSS node interned by (returnState, savedCounters, position)
and a return restores them, so the shared callee never mixes counter state across
its callers; parents are a set and a popped node replays its pop to a parent added
later (the GLL completeness detail). The build drops to O(groups): a depth-24
doubling chain (2^24 inlined states) compiles to under 480 states in under a
millisecond.
It is gated by a differential (ContentRTNDifferentialTests) against the trusted
inlining matcher ContentNFA as the oracle: the two agree on the accept verdict and
the matched particle over the hand model cases, the named-over-wildcard tie-break,
the shared-group doubling chains that exercise call and return, and a 4000-case
fuzz of nested models with occurrences, wildcards, and shared groups, plus a bound
assertion. Built alongside the inlining matcher and not yet wired into validation,
so no behaviour change and the XSTS buckets are unchanged; the switch-over is gated
on this differential.
* perf(schema): add the streaming/located surface to the GSS matcher, lock it differentially
The content-model validator reads two things off the inlining matcher: matchedParticles
for per-child attribution, and the streaming startStates/step/admissible/isComplete for
located diagnostics (which child breaks the model, and the "expected a, b" hint). The GSS
matcher only had matchesWhole/matchedParticles, so it could not yet replace the inlining
matcher. This adds the streaming surface: admissible works because the closure already
expands call states into the callee's label states, so the follow set has the same shape
as ContentNFA's.
The differential gate now locks the whole surface, not just the accept verdict: driving
both matchers child by child, the first rejected child (step is nil) agrees, the
admissible expectation set agrees at the start and after every accepted child, and
isComplete agrees once the sequence is consumed. The fuzz now stamps group references the
way the parser does, a unique definitionID per distinct group (so a singly-referenced
named group also takes the call path, not only doubled groups), a shared id only for
genuinely shared structure, and some anonymous inline groups, over occurrences and
wildcards. Still alongside, not wired; no behaviour change, XSTS buckets unchanged.
* perf(schema): complete the GSS matcher surface with follow(after:), verified
follow(after:) is the last ContentNFA method the GSS matcher lacked: the completion
path (XSDCompletions) reads it for the acceptable-next labels and whether the content
may end. It composes the streaming surface, and the differential now locks its allowed
set and complete flag to ContentNFA too. The RTN matcher is now a complete drop-in for
the ContentNFA surface the validator and completions read, so the switch-over is a
mechanical builder swap gated on XSTS neutrality. Still alongside; no behaviour change.
* docs(design): record GSS Steps 1-3 done, the switch as a corpus-gated mechanical swap
Updates the counted-content-automaton Step 4 to the current state: the GSS matcher and
its full differential gate are built and green in-repo, and the production switch is now
a three-site builder swap blocked only on the XSTS neutrality measurement, which must run
on a machine that has the corpus. Records the exact swap sites and the gate so the corpus
machine can execute it mechanically.
* perf(schema): validate and complete on the GSS matcher, killing the last 2^K content-model hang
The three call sites that built the inlining content automaton once per validated
element, ComplexValidatorContentModel.sequenceStructureErrors (located structure
errors), ComplexValidator.validateParticle (per-child particle attribution), and
XSDCompletions (acceptable-next labels), now build ContentRTNBuilder and run
ContentRTNMatcher. A schema whose nested <xs:group> references inline to 2^K positions
(the g{i}=(g{i+1},g{i+1}) doubling shape) no longer makes each keystroke-time validation
of an element exponential in the nesting depth: the automaton is built in O(groups).
Validating a short instance against a depth-40 doubling schema (2^40 inlined positions),
the shape an editor sees mid-typing, now completes in milliseconds with a located error
instead of hanging; a new end-to-end test pins it.
Verified conformance-neutral on the full W3C XSTS corpus in release: valid-schemas-rejected
and valid-instances-rejected stay 0, invalid-schemas-accepted and invalid-instances-accepted
unchanged at 8 and 14. The located diagnostics, per-child attribution, and completions are
behaviour-identical by the surface differential (ContentRTNDifferentialTests), which locks
the verdict, matched particle, rejection index, admissible set, isComplete, and follow to
the inlining ContentNFA. ContentNFABuilder/ContentNFA are retained as that oracle.
* docs(design): mark GSS content-matcher Step 4 done, XSTS-neutral in production
Records that the three production sites are switched to the graph-structured-stack
matcher and the switch was measured conformance-neutral on the full W3C XSTS corpus in
release (all four buckets held). Notes the local corpus path and that ContentNFABuilder
is kept as the differential oracle rather than deleted.
* test(schema): compare full matched-particle attribution; bound a cyclic group id
From the adversarial review of the GSS matcher:
- The differential compared only the matched-particle kind (element/wildcard), so it
could not observe the type/value metadata matchedParticles carries. Every fuzz and
hand element now carries a per-name fixed value constraint and the comparison projects
it, so the gate verifies the metadata is carried identically through call/return, not
only that the kind agrees. A name maps to one constraint, so the comparison does not
depend on the lowest-state-index tie-break (which only distinguishes same-name
particles, a UPA violation the compiler rejects before instance validation); the
matchedParticles doc now records that caveat.
- ContentRTNBuilder.groupFragment memoizes after compiling, so a (malformed) cyclic
definitionID would recur without bound, the interactive-safety hazard this matcher
removes. definitionID groups are acyclic by construction (the parser breaks cycles at
element boundaries, the inlining builder relies on the same), but bound it anyway:
a cycle degrades to a terminating empty fragment instead of recurring.
No production behaviour change; the differential, full suite, and the full XSTS corpus
in release (16/16, all four buckets held) stay green.
Step 1 of the GSS content-matcher build (docs/design/counted-content-automaton.md).
A multiply-referenced `<xs:group>` expands to one copy-on-write-shared `Particle`
(GroupParticleMemo), but `Group`/`Term` are value types with no exposed identity,
so a content-model compiler cannot tell two references apart and inlines the group
per reference (the 2^K build the GSS matcher will close).
Group now carries a `definitionID: Int?`, stamped from a per-compile counter on
GroupParticleMemo when an expansion is first compiled, so every reference sharing
the memo key reuses the same stamped term and distinct definitions differ. It is
metadata only: Group is not Equatable/Hashable, so it perturbs no determinism or
restriction comparison, and it is nil for an inline group.
FP-neutral by construction (no consumer yet). Full in-repo suite green; the XSTS
buckets are unchanged (valid-schemas-rejected 0, invalid-schemas-accepted 8,
valid-instances-rejected 0, invalid-instances-accepted 14); swiftformat/swiftlint
clean. Step 2 (a call/return content-model build keyed on this id, alongside the
inlining oracle) consumes it.
The previous section settled the algorithm and the bound proof; this makes the
build turnkey rather than exploratory. It is the full scope of the reverted RTN
attempt (whose group sharing was correct) with the explicit return stack replaced
by the graph-structured stack:
- Step 1: a group-definition identity (`Group.definitionID`, stamped from the
existing `GroupParticleMemoKey` in `XSDParserParticles`), FP-neutral on its own.
- Step 2: a `ContentRTNBuilder` alongside the trusted inlining `ContentNFABuilder`,
compiling each definition once via `call`/`return` epsilon edges, counters still
per-configuration and reset on entry so a shared fragment does not alias.
- Step 3: the GSS simulation: a node `(returnState, parents-set)` interned per
`(returnState, inputPosition)`, configuration `(state, counters, gssNode)`, with
call interning and pop fan-out, deduplicated on the configuration key.
- Step 4: the differential gate against the inlining oracle (XSTS + a
nullable-nested-group fuzz, verdict AND matched-particle equal), then the switch
and a perf assertion on GSS/configuration COUNTS (not a wall-clock timeout, the
reverted attempt's lesson).
Each step is its own PR; steps 1-2 keep the oracle authoritative, step 4 is the
only behaviour change and is gated to equivalence before it lands. Design only.
Closes the proof gap that sank the reverted RTN matcher. The counted automaton
removed the occurrence-magnitude blow-up and the compositional determinism check
removed the determinism group-sharing blow-up, but the per-instance matcher build
(ContentNFABuilder.group) still expands a multiply-referenced group once per
reference, so a doubling chain g0=(g1,g1)... builds 2^K states and hangs on the
first validate. That is the last place stopper 4 is a hang, not a proven bound.
The reverted pushdown rewrite compiled the group DAG to call/return edges
(O(groups) program) but kept EXPLICIT return stacks, so a nullable doubling chain
made the start epsilon-closure reach ~3^depth distinct stacks: the exponential
moved from the build to the per-keystroke closure. It was verdict-correct yet
bounded nothing, and was reverted.
This section derives the fix: a graph-structured stack (the GLL/Earley device).
A GSS node (returnState, parents-set) shares stack suffixes, so configurations are
keyed on (state, counters, gssNode) and the live set is bounded by O(states), not
by the number of distinct stacks. The nullable doubling chain that produced
3^depth explicit stacks produces O(K) GSS nodes. The section states the size and
termination bounds (counters clamped to N+1, non-recursive group DAG), the
deterministic-model O(N*S*D) match, and the same alongside-then-differential-gated
rollout that made the compositional-determinism change FP-safe, with a perf
ASSERTION on configuration/GSS counts rather than a wall-clock limit (the reverted
attempt's lesson). Design only; no code change. Status: DESIGNED, not implemented.
The schema-validity differential harness (stopper 3) reported each disagreeing
schema by bare basename, but the same name recurs across subdirectories, so an
entry hid which file actually disagreed: Facets/Schemas/gMonth_maxInclusive001
(value="", correctly rejected) and Facets/gMonth/gMonth_maxInclusive001 (the
contested 2001-REC --MM-- form) collapse to one ambiguous line. The report now
uses the path relative to the corpus root.
The header records the triage of the "libxml2 stricter" queue (~66 over msData):
it is dominated by libxml2 over-strictness (a maxOccurs like 999999999999999 is
a valid unbounded-precision nonNegativeInteger libxml2 rejects on its machine-int
limit; an unresolvable include/import) and contested datatype-version readings
(gMonth --MM--, which PureXML accepts to keep valid XSTS instances; anyURI
strictness; a regex {,5} tolerated per iter 7), with a small FP-delicate residue
(base64Binary canonical padding, keyref no-namespace refer). So the queue is
characterised and named, not a backlog of bugs. Characterisation only; the
shipped validator is unchanged, and the false-positive gate stays the XSTS
suite's valid-schemas-rejected == 0.
A particle restriction whose restricting element has a union-typed base element
was checked by structural member subsetting: PureXML accepted a separately
declared union whenever its members were a subset of the base union's. That is
not the spec's derivation relation. The NameAndTypeOK type clause (XSD 1.0
section 3.9.6) requires the restricting element's type to be validly derived
from the base element's type per Type Derivation OK (Simple), cos-st-derived-ok.
The restricting type is now valid over a union base only if it is the base
type, derives from it through a declared restriction chain, or (clause 2.2.4)
validly derives from one of the union's member types. Scoped to the
element-restriction path, so no attribute or simple-type derivation check moved.
Catches particlesIk011 (a fresh union(integer) is not a restriction of
union(integer, string)): XSTS invalid-schemas-accepted 9 to 8, valid-schemas-
rejected held at 0 over the full corpus (the cardinal no-over-rejection check).
Disclosed: particlesIk025/Ik027, where the restricting type derives from a
union member, are accepted per cos-st-derived-ok 2.2.4, a contested reading
their XSTS invalid expectation does not share. An in-repo test that asserted
the old structural-subset behavior is corrected.
Move the [Unreleased] section to [0.4.8] - 2026-06-27. No source changes.
0.4.8 collects the interactive-safety depth-recursion program: every walk over
the value-Node/TreeNode model, the parser and XPath result ordering, the
canonical, XInclude, HTML, and XSLT serialization paths, and the XSLT
transformation engine itself are now iterative, so parsing, validating,
querying, transforming, and serializing adversarial or deeply-nested input no
longer overflows the stack. Also includes the xpointer string-range() and XSLT
id() quadratic fixes.
The XSLT engine was mutually recursive on template-application and result-tree
nesting (instantiate, evaluate, applyTemplates), so a recursive named template
whose depth comes from source data could overflow the native stack (SIGBUS) on
attacker-controllable input, and an identity transform of a deep source
overflowed on a small (Swift-concurrency Task) stack.
The engine is now an explicit work-stack driver (XSLTDriver). Each instruction
appends its produced items to a sink or pushes sub-work, with the rest of the
body pushed first so result-tree document order is preserved; template
application, the built-in rule's source descent, and result-element nesting all
run on the heap. Only stylesheet-bounded helper evaluation (a variable's or
parameter's result-tree fragment, an attribute set, an xsl:message body)
re-enters through a nested driver call, whose native depth is bounded by the
stylesheet's own nesting.
The result-tree serialization walks the engine now exposes at depth were made
iterative too: XSLTNamespaceFixup (an explicit pre-order visit that keeps the
ns0/ns1 counter in document order), the text and string-value extractions
(iterative accumulators), and the content-type-meta and CDATA-section rewrites
(the shared Node.rebuildingBottomUp spine).
Deep recursion now succeeds on any stack: a 5000-level recursive template builds
and serializes on the test Task stack. maxTemplateDepth is retained as runaway
protection (an unbounded result tree would exhaust memory), defaulting to 40000
and throwing XSLTError.recursionLimitExceeded only past that.
Behavior is otherwise unchanged: the full Apache Xalan XSLT 1.0 conformance
corpus (1086 cases) and the in-repo XSLT/XPath suites pass identically; one
previously-failing Xalan case (mdocs07) now passes and was de-baselined. Full
suite green. Closes#356.
* fix(xslt): bound template recursion so it fails gracefully, not with a crash
A recursive named template whose depth is driven by source data (e.g.
`<xsl:with-param name="n" select="count(//x)"/>`) could recurse without bound
and overflow the native stack (SIGBUS) on attacker-controllable input. Every
XSLT processor guards this (libxslt's xsltMaxDepth); PureXML did not.
Measurement first: source *nesting* depth is not the exposure. `transform`
parses the source at the default maxDepth (256) and there is no XSLT API that
accepts a pre-built tree, so the engine never sees a source deeper than ~255;
an identity transform of a 254-deep source completes normally on a normal
stack. The reachable crash is data-driven template recursion, which overflows
an 8 MB stack between ~400 and 600 levels.
Template-instantiation nesting (instantiateTemplate and builtInRule) is now
counted and capped at maxTemplateDepth; past it the guard latches, the
instantiation loops unwind, and `transform` throws
XSLTError.recursionLimitExceeded instead of crashing. The default (300, exposed
as PureXML.XSLT.defaultMaxTemplateDepth, overridable via a new maxTemplateDepth
parameter on transform) clears the parser's max source depth so a deep-source
identity transform still runs, while staying within an 8 MB main-thread stack
with margin. A stylesheet with deeper legitimate recursion raises the limit on
a correspondingly larger stack.
copyOf moved to XSLTSorting.swift to keep XSLTTransformer.swift within the file
length limit; it is unchanged (a whole-subtree copy that does not recurse on
depth). Guarded by XSLTRecursionLimitTests (run on a large-stack thread, since
the limit is above what swift-testing's Task stack holds). Full suite 1802 pass.
Part of #356; making the engine fully iterative (deep recursion succeeds, and
is safe on a small Task stack) remains tracked there.
* test(xslt): exclude the large-stack recursion tests on WASI
WASI is single-threaded and has no Foundation Thread/DispatchSemaphore, which
the recursion-limit tests need to exercise the guard on a stack large enough for
the limit (300) to fire before the stack overflows. Guard the file with
#if !os(WASI); the recursion guard itself is platform-independent pure Swift,
covered by the macOS and Linux runs.
Two Canonical XML transforms still recursed one native frame per element level,
so a deeply-nested document overflowed the stack.
The C14N 2.0 sequential prefix rewrite walks the tree twice. Its assign pass
(which records one canonical prefix per namespace in document order) is now an
iterative pre-order walk; its rebuild pass is now a child-gathering accumulator
driven by an explicit work stack, threading the in-scope declarations and the
root flag down per element. The document-order prefix assignment and the
root-hoisted declarations are unchanged.
The node-subset serializer (`canonicalize(_:including:)`) now drives the same
deferred-close work stack as the main serializer, threading each element's
in-scope and rendered namespace context; an omitted element still passes its
declarations to descendants without rendering or a close tag.
The canonical bytes are unchanged: the C14N exclusive, 2.0, and node-subset
corpora pass (1794 tests). A 50000-deep document now rewrites and
subset-canonicalizes without crashing, guarded by `CanonicalDepthSafetyTests`.
Part of #350.
`PureXML.XInclude.process` rebuilt the tree recursively (`process →
processElement → process`), so a deeply-nested document overflowed the stack
one native frame per element level, even with no includes present. It now
drives an explicit work stack: each branch gathers its processed children into
a reference accumulator, then a deferred build step assembles the node. The
`xml:base`, inclusion-chain depth (bounded by `maxDepth`), and cycle-detection
visited set thread down per context exactly as before; an include splices its
(or its fallback's) content in place by pushing the loaded nodes back onto the
stack.
Behavior is unchanged: the XInclude/XPointer conformance corpus and the
fallback, xpointer, cycle, and xml:base cases all pass. A 50000-deep document,
with or without an include at its base, now processes without crashing, guarded
by `XIncludeDepthSafetyTests`. Part of #350.
`PureXML.HTML.serialize` walked an element's children recursively, so a
deeply-nested tree overflowed the stack (as shallow as a couple of thousand
deep on a Swift Task stack). The serializer now drives a deferred-close work
stack, the same shape as the XML serializer and `Canonicalizer.emit`: an
element pushes a close step then its children reversed, and a raw-text element
(`script`, `style`) flags its direct text children to emit verbatim, so nested
elements still serialize by their own name.
The output is byte-identical. Void-element, raw-text, comment, and escaping
semantics are unchanged (a new micro-test pins them) and the HTML conformance
corpus passes. A 50000-deep tree now serializes without crashing, guarded by
`HTMLSerializeDepthSafetyTests`. Part of #350.
Computing a node's string-value walked its subtree recursively, so it
overflowed the stack on a deeply-nested node (around 2000 deep on a Swift
Task stack): `XPath.Query("/a/a").strings(over:)` over a 4000-deep chain
crashed. `TreeNode.textContent` and the value-Node string-value behind
`Selection.stringValue` now walk an explicit stack, pushing children reversed
to concatenate in document order.
The value-Node path also wrongly folded descendant comment and
processing-instruction data into an element's string-value, disagreeing with
the tree-node path and XPath 1.0 (5.1/5.5: only text and CDATA contribute to
an element's string-value; a comment or PI node's own string-value, 5.6/5.7,
is still its data). It now matches. No test depended on the old behaviour:
1789 pass, including the XPath and XSLT value corpora.
Closes the string-value sites of #350; the transform walks (canonicalize,
DTD defaults, XSLT result tree) remain tracked there.
* perf(model): rebuild DTD-default application iteratively, not recursively
applyingDefaults rebuilt the tree with a per-node recursive walk, so applying
DTD attribute defaults to a deeply-nested document overflowed the stack (around
2000 deep on a Swift Task stack). It now rebuilds bottom-up through a new shared
iterative spine, Node.rebuildingBottomUp, which transforms each node once its
children are rebuilt via an explicit work stack.
The same primitive will carry the other tree-to-tree transforms (canonical
prefix rewrite, XSLT namespace fixup) in follow-ups. Defaults are applied
identically: 1788 tests pass. Part of #350.
* perf(canonical): serialize C14N iteratively, not recursively
Canonicalizer.emit recursed on element children, so canonicalizing a
deeply-nested document overflowed the stack (crashed around 5000 deep).
The element serializer now drives a deferred-close work stack (the same
shape as the main serializer): each step carries the in-scope and
already-rendered namespace context, children push reversed to emit in
document order, and a close step emits the end tag. The shallow document
case and all namespace-rendering logic are unchanged.
The canonical bytes are byte-identical: the C14N 1.0, 1.1, and exclusive
conformance corpora pass, full suite 1789. A 50000-deep document now
canonicalizes without crashing. The node-subset helpers moved to their
own file to keep within the file-length limit. Part of #350.
Mapping an ordered node-set to selections rebuilt each result as an
independent value Node through TreeNode.node, projecting that node's whole
subtree. For a result set whose nodes nest within one another (`//a` over a
deep chain, where every `a` contains all deeper ones) this materialized the
sum of their subtrees: quadratic. `//a` over an 8000-deep chain took ~19s.
The projection now threads a memo keyed by tree-node identity across one
result set. A tree node is projected once; a result that nests a
deeper one reuses its already-built value Node by reference, which the
copy-on-write Element storage (from the value-Node depth-safety work) makes
a shared, not copied, subtree. The set is built in one pass over the
distinct nodes. The same query now returns in ~0.04s and scales linearly.
The non-memoized `node` projection (serialization, value comparison) is
unchanged. Results are identical: 1787 tests pass, including the XPath and
XSLT value corpora. Closes#348.
Sorting a node-set into document order keyed each node by its root path, an
array as long as the node's depth, computed with a per-node ancestor walk.
Ordering the descendants of a deeply-nested document was therefore
O(n log n x depth): `//a` over an 8000-deep document took ~19s through the
reference query path.
The key is now a single O(1) pre-order index from a per-root numbering,
built once per document and shared through the evaluation's navigation
cache, with a band ordering an element before its namespace nodes, its
attribute nodes, and its children. A node-set spanning several documents
(an XSLT document() result) is grouped by root, ranked by first appearance,
so only a foreign node walks its ancestors. `appendSubtree` (the numbering
walk) is now iterative, so it no longer recurses on depth either.
Document order is unchanged: 1786 tests pass, including the XPath, XSLT,
XPointer, and Schematron conformance corpora. `Query.nodes` over a
50000-deep document now orders in a fraction of a second.
Each open element resolved its name and attributes by looking up namespace
bindings through NamespaceContext.lookup, which walked a stack of scopes as
deep as the document. With one lookup per element over an O(depth) stack,
parsing a deeply-nested document was O(depth^2): a 40000-deep document took
~39s.
The context now maintains the current in-scope bindings incrementally: a
single prefix-to-URI map, plus a per-element undo list recording the
bindings each element overwrote so closing it restores the enclosing state.
A lookup is O(1) and entering/leaving an element is O(its xmlns
declarations), so parsing is O(n). The same document now parses in ~0.2s.
Namespace semantics are unchanged: the nearest in-scope binding still wins,
an empty default binding still undeclares, and 81 namespace tests plus the
full suite (1784) pass. Closes#345.
Element stored its children inline and Node was `indirect`, so a Node's
recursive layout made building, comparing, hashing, and releasing a deeply
nested tree recurse one native frame per level and overflow the stack
(signal 10) -- on a constrained stack such as a Swift Task, as shallow as a
few thousand deep.
Element now holds its children behind a copy-on-write reference
(ElementStorage), so Node is a fixed-size value (no longer indirect) and:
- `==` and `hashValue` walk the tree iteratively through explicit work
stacks (the synthesized conformances recursed); equal trees still hash
equally.
- ElementStorage's `deinit` flattens the subtree into a heap worklist
before releasing, draining only storage it solely owns so a copy-on-write
sibling keeps its subtree.
- `children` is a computed copy-on-write view with a `_modify` accessor, so
in-place mutation (the parser's append loop) stays amortized O(1).
Value semantics are unchanged; Element/Node remain Equatable, Hashable,
Sendable. A 50000-deep tree builds, compares, hashes, and releases in a
fraction of a second; 1782 tests pass. Completes the value-Node half of #341.
Building a TreeNode from a value Node, projecting it back via `node`, and
releasing it each recursed one native frame per tree level, so a deeply
nested document overflowed the call stack (signal 10 around 30-60k deep,
and far shallower on a constrained stack such as a Swift Task). None of
these is fixed by the serializer, which was already iterative.
- `TreeNode(_:)` and `node`/`projectedChildren` now drive explicit
post-order work stacks (entity-reference splicing, doctype/namespace
dropping, and the root special-cases are unchanged).
- `deinit` flattens the subtree into a heap worklist before releasing, so
each node frees with empty children and the teardown stays flat. Only a
node the teardown solely owns (isKnownUniquelyReferenced) is flattened,
so a child still referenced elsewhere keeps its subtree.
- The validator walk and the XPath descendant axis were made iterative for
the same reason.
A 60000-deep tree now releases in well under 0.1s; 1778 tests pass and no
accepted output changes. Part of #341; the value Node's own recursive
release and XPath result ordering/dedup remain depth-bounded and tracked
there.
* perf(parsing): memoize the markup-producing entity check (billion-laughs)
splitAtMarkupEntity / producesMarkup recursed into every reference of
every replacement with no memoization, so a nested entity chain was
traversed O(references^depth) times: a classic eight-level billion-laughs
took ~18s of CPU to reject. The check is now memoized per entity name (the
property depends only on the fixed entity graph), so it is O(entities). The
same document now rejects in ~0.6s, bounded by the expansion budget. No
change to which documents are accepted; cyclic entities still rejected.
Found by adversarial probing; closes the no-markup case of #339.
* perf(parsing): three-colour DFS for the No-Recursion check
checkRecursion validated the entity reference graph with only an
in-progress (grey) set and no record of fully-verified entities, so a
billion-laughs graph was re-walked O(references^depth) times: a
markup-producing eight-level document took ~31s to reject. It is now a
three-colour DFS: an entity is added to a black set only after its whole
reachable subgraph returns acyclic, so it can never reach a chain node
that reaches it, and skipping it on revisit is sound. The walk becomes one
pass over the distinct edges (~31s -> ~11s for that document). Cyclic
entities are still rejected; no accepted document changes. Part of #339.
The lookahead buffer consumed from the front with Array.removeFirst() and
re-injected entity replacement text with insert(at: 0), both O(buffer), so
a single large entity (or a long content run pushed back by reference
inclusion) was consumed in O(n^2). The buffer now advances a head index
(front-consume O(1)) and re-injection reuses the freed prefix space, making
a single large expansion linear. An 800k-character internal text entity
drops from quadratic time to about 0.6 s. Same characters, same limits;
1775 tests unchanged. Found by adversarial robustness probing.
Repo-wide reference sweep: XSLTTransform.transformToNode (a transform-to-
tree variant nothing calls; the live string-returning transform is
independent), and ComplexValidatorHelpers.matchesAll plus its orphaned
helper label(of:). Zero references; build and 1775 tests confirm no
behavior change.
A private-scope sweep (a private/fileprivate function with zero in-file
callers is provably dead, with no name-conflation risk) found three:
selectNodes in XSLTTransformer (selectXPathNodes is the live sibling),
processingInstructionMatches in EvaluatorSteps, and parseExternalID in
DoctypeScanner. A doc comment referencing selectNodes was reworded.
Build and 1775 tests confirm no behavior change.
Windows runs the full build and test suite and is the slowest CI job
(about 22 minutes), so on every push and PR it adds the most wait for the
least marginal signal on pure-Swift changes. It now triggers only on a v*
tag push (cutting a release) or via workflow_dispatch, keeping per-PR CI
fast. The badge reflects the last release's Windows run. Deviates from the
uniform-trigger convention deliberately, for CI speed.
Triaging the 24 open XSTS failures against libxml2 (the stated reference)
separates genuine reference-aligned bugs from stricter-than-libxml2 scope
decisions. On the schema side, 9 of 10 are accepted by libxml2 too (Xerces-
level strictness, false-positive-prone); only particlesZ030_d was a real
gap, now fixed. The instance tail's reference-aligned cases are intricate
and need per-case analysis. Guides the burndown toward bugs, not non-bugs.
derivation-ok-restriction.2 (a base required attribute may not become
optional in a restriction) ran only over complexContent restrictions, so
a simpleContent restriction that relaxed a required attribute was accepted.
The attribute-use restriction check now runs over both derivation roots.
Closes XSTS particlesZ030_d, which libxml2 also rejects, so this aligns
with the reference rather than exceeding it. XSTS invalid-schemas-accepted
10 to 9 with valid-schemas-rejected held at 0 (full archive). Guarded by
SimpleContentAttrRestrictionTests; the XSTSSuiteTests baseline is updated.
The schema differential ran xmllint and then read its stderr and waited
for exit with no timeout. xmllint infinite-loops on a few pathological
schemas (particlesZ012), so a hang blocked the harness forever and left
the child as a zombie. A bounded wait now terminates and reaps the child
at a deadline and skips the case (it cannot be compared against a hung
reference). A non-gated test proves the mechanism by terminating a hung
sleep at its deadline instead of waiting it out. Addresses the leak noted
in #194.
API.swift's parse(_:sax:), parse(pullingBytes:), the three events(...)
overloads, and validateHTML were reached only through subsystem types,
not the public wrappers. Adds direct tests: SAX callbacks in document
order, an incremental byte source yielding the same tree as the string
parse, all three events overloads agreeing on event count, and
validateHTML flagging a duplicate id.
The schema/instance validity baselines (Tests/XSTSSuiteTests.swift) were
only checked opt-in. Now that the unmodified corpus archive is vendored,
a dedicated workflow extracts it (no network) and runs the ratchet in
release on every push and PR, so a conformance regression fails CI. Adds
the matching README badge.
Saves the W3C XML Schema Test Suite (2006-11-06) in the repo as the
complete, unmodified xsts-2007-06-20.tar.gz (4.4 MB), the only form the
W3C Document License permits redistributing. The extracted 39,399 files
are still never committed (that would be a modified redistribution).
A README records the provenance, SHA-256, and license. scripts/fetch-xsts.sh
now extracts this vendored archive when present (no network) and only
downloads as a fallback. The archive is excluded from the test bundle in
Package.swift since the runner reads it off disk via XSTS_ROOT, not from
resources.
The W3C XML Schema Test Suite (2006-11-06) drives the opt-in conformance
runner but is never vendored: the W3C Document License permits
redistribution only as the complete unmodified archive, and the corpus
is about 200 MB. This script downloads the official archive on demand,
verifies its SHA-256, and extracts it to the path XSTS_ROOT expects.
Idempotent and portable (shasum or sha256sum). Points the onboarding doc
at it.
ByteEncoder (the public output encoder) was reached only through the
serializer's UTF-8 default, leaving its transformation-format, single-
byte, BOM, and naming paths untested. Adds direct assertions of the
spec-defined bytes: UTF-8/16/32 (incl. an astral surrogate pair),
ISO-8859-1 with a decimal character-reference fallback for an
unrepresentable scalar, byte-order marks, supports, and canonical names.
Adds XSLTNamespaceFixupTests for the previously-untested live paths in
XSLTNamespaceFixup.apply: generated ns0/ns1 prefixes for namespaced
attributes with no carried prefix, and xmlns="" undeclaration of an
inherited default namespace on an unqualified element (XSLT 1.0 7.1).
Exact serialized-output assertions, matching the existing XSLT test idiom.
XSLTNamespaceFixup.fixAttributeName and resolvedPrefix had zero callers.
The result-tree attribute fixup was inlined into fix(_ element:) and
evolved past them (the inlined path also shadows a carried prefix via
localPrefixes, which the helpers lacked), so the helpers were dead,
diverged duplicates. No behavior change; build green, 1759 tests pass.
* refactor(schema): remove unused [String] consistency wrappers
Drop four internal schema-consistency wrappers that had no callers
(idAttributeErrors, postNamedTypeErrors, structureErrors,
includeCompositionErrors). The located-diagnostic *Findings API
superseded them and the live call sites project reasons with
map(reason) directly, so these were dead code. No behavior change;
the *Findings functions are unchanged. Also correct a stale
doc-comment reference to the renamed userTypeValueConstraintFindings.
* critic-fix(schema): drop the dead consistencyErrors chain the first pass missed
The name-only dead-code sweep read XSDParser.consistencyErrors as used,
but its callers were the distinct Validation.XSDSchema.consistencyErrors
(a different signature returning ValidationError). The XSDParser overload
had zero callers. Removing it, plus the postCompileErrors(namedTypes:)
overload and the SchemaCompileNamedTypes struct that only the already-
removed postNamedTypeErrors reached, completes the dead chain. Build
green, 1759 tests pass, no behavior change.
id() filtered the entire document ID index by tokens.contains on every
call, O(index) per call, so a stylesheet calling id() across a wide
document was quadratic. Look each requested token up directly in the
index dictionary instead, O(tokens). The result is unchanged: the same
elements, in document order (the ID values are the keys, each mapping to
one element). key() already used direct lookups.
Resolving 4000 id() references against a 4000-entry index drops from
about 0.34 s to about 0.03 s (~13x), now linear.
xpointer(string-range(...)) matched the search string at every position
of each node's value and re-allocated the whole needle window per
position, an O(text * needle) search and a denial-of-service vector on a
long node value with a near-matching needle. Replace it with a
linear-time Knuth-Morris-Pratt search (failure function built once, since
the needle is fixed across nodes). Matches are unchanged: same
non-overlapping, left-to-right occurrences, same offset/length.
A near-matching needle over a 16000-character node drops from about
0.18 s to about 0.002 s (~100x), now linear.
xs:unique/xs:key/xs:keyref were quadratic over a wide list of targets.
The duplicate check compared each value tuple against every tuple already
seen (and a keyref against every referenced key), and the error-location
helper rescanned the parent's whole child list to position each target.
Key tuples in a Set by their raw string when field equality provably
reduces to a raw string comparison (whitespace-preserving lexical types,
no colon, which is most ids); value-space types (numeric, boolean, date),
whitespace-collapsing types, and QName-bearing values keep the exact
pairwise comparison, so which documents pass is unchanged. Position each
target through a per-parent, per-name index built once.
A 16000-item xs:unique document drops from about 4.0 s to about 0.12 s
(~34x), and the cost is now linear. The full local suite and the W3C
XSTS datatype/identity conformance corpus are unchanged.
Stamp the XPath string-function and xsl:number quadratic fixes as 0.4.6:
translate() and the substring search (contains/substring-before/after) are now
linear (the substring search via Knuth-Morris-Pratt, closing a DoS vector), and
xsl:number is linear for level single/multiple and quadratic (not cubic) for
level any. An 8000-item default xsl:number drops ~3.5s to ~0.04s.
siblingPosition ranked a numbered node among its parent's matching children by
scanning the child list up to the node, counting matches. Run for every node of
a wide list, numbering was O(n^2): the default xsl:number over 8000 list items
took ~3.5s.
The count test is the same for every candidate of one xsl:number call, so cache
per parent and count pattern the one-based rank of each matching child, built
once. Each node's rank is then an O(1) lookup. The transformer holds the cache
across the whole transform; the count test itself is already O(1) through the
match cache, so building a parent's ranks is linear in its children. The numbers
are identical (a node's rank among matching siblings is 1 + the matching
siblings before it).
The 8000-item numbering drops ~3.5s to ~0.04s (~90x), quadratic to linear. The
full suite, the W3C XSLT and Apache Xalan corpora, and the WASM build stay
green; XSLTNumberSingleTests pins single, partial-match, and multiple-level.
walkPreceding visits a node and everything before it in document order. It found
each preceding sibling with precedingSibling(of:), which rescans the parent's
whole child list, so visiting k preceding siblings was O(k^2) and numbering n
nodes under a wide parent was O(n^3): 1000 items took ~4.7s and 4000 did not
finish in minutes.
Find the candidate's index in the child list once, then walk the children before
it directly (parent.children[..<index].reversed()), visiting each preceding
subtree in reverse. The walk is now linear in the number of preceding nodes, so
the numbering is quadratic (the residual is inherent to level="any", which
counts the matching preceding nodes per numbered node). The numbers are
identical.
Numbering 1000 items drops ~4.7s to ~0.7s (~6x); cubic becomes quadratic. The
full suite, the W3C XSLT and Apache Xalan corpora, and the WASM build stay
green; XSLTNumberAnyTests pins document-order numbering across nesting.
translate(string, from, to) ran from.firstIndex(of: character) for every
character of the source, so a large from/to map made it O(string * from). And
contains/substring-before/substring-after matched the needle at every position
of the haystack (Array.starts(with:) per index), O(n*m) and quadratic on a
repetitive string, which is a denial-of-service vector on untrusted input.
translate() now builds a from-character to replacement map (and a delete set)
once, first occurrence winning, and translates each source character with an
O(1) lookup. The substring search is Knuth-Morris-Pratt, O(n+m), which never
rescans. The results are unchanged.
A 4000-character large-alphabet translate drops ~0.13s to ~0.002s; a 16000-
character repetitive contains drops ~0.054s to ~0.0002s. The full suite, the
W3C XPath/XSLT and Apache Xalan corpora, and the WASM build stay green;
TranslateFunctionTests and SubstringSearchTests pin the semantics.
Stamp the attribute-index and sibling-axis quadratic fixes as 0.4.5: sorting an
element's attributes into document order was quadratic in the attribute count,
and the following-sibling/preceding-sibling axes scanned a wide parent's child
list per context node. Both now use the shared sibling-index cache (and the
sibling axes fuse the node test). Also correct the changelog: these two entries
were mistakenly grouped under the released 0.4.4 heading.
siblingPosition(of:) located the context node with
parent.children.firstIndex { $0 === tree } on every call: a linear scan of the
parent's child list, independent of the result. Over a wide parent evaluated
from many context siblings (//x/following-sibling::y) that scan was quadratic,
and the axes then materialized every sibling before the node test filtered it.
Look the index up through the shared per-evaluation SiblingIndexCache (O(1)
after the per-parent table is built), and fuse the step's node test into the
walk so a rejected sibling is never wrapped (the same fusion the descendant and
following/preceding axes use). The selected siblings and their order are
unchanged.
The gratuitous per-context scan and the non-match wrapping are gone (the
remaining per-context iteration over the siblings is inherent to the axis); a
wide-parent following-sibling query is ~1.2-1.6x faster. The full suite, the
W3C XPath/XSLT and Apache Xalan corpora, and the WASM build stay green;
SiblingAxisTests pins the fused node test and document order.
An attribute node's document-order key bands by the attribute's index among its
owner's attributes, found with owner.attributes.firstIndex { $0.name == name }.
That linear scan ran for every attribute node, so sorting an element's K
attributes into document order was O(K^2): an 8000-attribute element took ~0.18s
and grew quadratically.
Cache a per-owner attribute-name-to-index map in SiblingIndexCache, built once
on the second lookup (a single lookup still scans), exactly as the sibling-index
cache already does for child positions. The index is the same first-occurrence
position; only its cost changes.
The 8000-attribute sort drops to ~0.01s (~16x) and is now linear. The full
suite, the W3C XPath and Apache Xalan corpora, the C14N suite (which orders
attributes byte-exactly), and the WASM build stay green; AttributeDocumentOrderTests
pins declaration order through the cached sort.
Stamp the following/preceding axis quadratic fix as 0.4.4: those axes shared no
state across context nodes, rebuilding and linearly searching the whole document
node list on each, and materialized the entire span before the node test. They
now share a per-evaluation document-order cache, skip the context subtree by
index arithmetic, and fuse the node test into the walk. A 200-context following::
query over a 20k-element document drops ~16s to ~6s (~2.6x); results identical.
After #302 cached the document node list, the following and preceding axes still
materialized the entire following/preceding span (Array(slice), one Node copy
with a slow side-table retain per node) before the step's node test filtered it.
For a rare-target query that copy is almost entirely waste.
Fuse the node test into the walk: apply the test to the cached document-order
slice directly, so a node the test rejects is never copied out of the shared
list, only the matches are materialized. The selected nodes and their order are
unchanged (the test is the same compiled node test the general axis path used).
The 200-context following:: query over a 20k-element document drops from ~9s
(post-#302) to ~6s, ~2.6x below the original ~16s. The full suite, the W3C
XPath/XSLT and Apache Xalan corpora, and the WASM build stay green;
FollowingPrecedingAxisTests gains a fused-name-test case.
following(of:) and preceding(of:) rebuilt the entire document node list and ran
a linear firstIndex search for the context node on every call. Evaluated over
many context nodes (//x/following::y), that rebuild-and-search was quadratic: a
200-context following:: query over a 20k-element document took ~16s.
Share a per-evaluation DocumentNavigationCache: each document's ordered node
list and a node-to-index map are built once (the document is immutable during
evaluation) and reused, so the position lookup is O(1). following:: also skips
the context's contiguous subtree by index arithmetic (start = index + 1 +
descendant count) instead of scanning the tail and filtering every descendant
out. The selected nodes and their document order are unchanged.
The 200-context query drops ~16s to ~9s; the residual is the produced
node-set's own size (materialization), which is inherent to the axis. The full
suite, the W3C XPath/XSLT and Apache Xalan corpora, and the WASM build stay
green; FollowingPrecedingAxisTests pins the subtree skip, ancestor exclusion,
and the shared-cache multi-context path.
Stamp the XSLT/XPath quadratic fixes as 0.4.3: single-node string extraction
(string(@x), value-of) and the key(), id(), and EXSLT set:leading/set:trailing
functions no longer rescan a wide parent's children per node, so they are linear
rather than quadratic over a flat fan-out. A representative transform over a
20k-element document drops ~2.3s to ~0.65s (~3.5x); key()-driven ~1.15s to
~0.43s. Output identical (Xalan gold-output byte-for-byte).
key(), id(), and EXSLT set:leading/set:trailing sorted their result with
sorted(by: Node.precedes). precedes recomputes both nodes' document-order keys
on every comparison, and each key walks the node to the root taking its index
among each parent's children, a linear scan of a wide parent. Over a wide flat
fan-out the sort was therefore quadratic. key() also deduplicated its matches
with matched.contains(where: ===), a linear membership scan, O(m^2) in the
match count. The EXSLT relative() used the cache-less documentOrder, rescanning
the parent per node.
Sort through sortedByDocumentOrder() (each key computed once behind a shared
sibling-index cache), deduplicate key()'s matches through a Set, and give
relative() a shared cache. The selected nodes, their document order, and the
deduplication are unchanged.
A key()-driven transform over a 20k-element document drops from ~1.15s to
~0.43s (~2.7x). The full suite, the Apache Xalan XSLT gold-output corpus, the
W3C XPath/XSLT/EXSLT tests, and the WASM build stay green; XSLTKeyIdOrderTests
pins key() dedup-and-order and id() document order.
firstInDocumentOrder() computed a document-order key even for a zero- or
one-node set. The key walks a node up to the root, taking its index among each
parent's children, which is a linear scan of a wide parent. string(@x) and
value-of take the string-value of a (usually single) node-set through this
path, so an XSLT transform over a wide flat fan-out rescanned the wide parent
once per extraction, turning the transform quadratic.
For a set of zero or one node the document order is irrelevant: the node is
trivially first. Return it directly without computing the key, the same
count <= 1 early-out sortedByDocumentOrder() already has.
A transform reading a couple of attributes and child values per element over a
20k-element document drops from ~2.3s to ~0.65s (~3.5x). The output is
unchanged: the full suite, the Apache Xalan XSLT gold-output corpus, the W3C
XPath/XSLT/EXSLT tests, and the WASM build stay green, and FirstInDocumentOrderTests
pins the single-node fast path and the multi-node document-first selection.
Stamp the post-0.4.1 parse and serialize performance work as 0.4.2: the entity
decoder's bulk literal copy and byte-level marker scan, the scanned name-prefix
colon offset, the direct TreeNode adoption and direct-from-events tree build,
and the bulk byte-level output escaping. parse goes ~9x to ~5.8x and serialize
~3x to ~2.3x slower than libxml2 across these passes, with identical results
(every change differential-tested and adversarially reviewed).
The bulk-copy escape loop still iterated Characters and compared each to '&',
'<', '>' (a grapheme comparison per character), the dominant remaining serialize
cost for content that needs escaping. Every escaping trigger is an ASCII marker,
or in ASCII-only mode a non-ASCII lead byte, so find the next one with
value.utf8.firstIndex(where:) and decode no grapheme for the run between markers.
The marker byte always lands on a grapheme boundary: an ASCII byte is its own
scalar and forces a cluster break (so '&', '<', '>', the quote, and whitespace
are never mid-cluster), and after a grapheme-aligned runStart the first byte at
or above 0x80 is a lead byte. So value[marker] and value[runStart ..< marker]
stay valid, and a multi-scalar cluster escapes as a unit (Swift rounds a
mid-grapheme index to its boundary, so the base letter is neither dropped nor
duplicated). The output is byte-identical.
Measured serialize ~23% faster on the 20k-item generated corpus via a same-state
A/B (~2.8x to ~2.2x slower than libxml2). Verified by EscapingBulkCopyTests
(the oracle differential extended with combining-mark, ZWJ, and flag clusters),
the Apache Xalan gold-output and C14N suites, the full suite, the WASM build,
and an adversarial grapheme-alignment review.
Escaping.text and Escaping.attribute rebuilt a value that needed escaping one
character at a time: result += String(character) for every unescaped character,
plus the escaped form for each marker. Re-escaping a text node that holds a
single '&' rebuilt the whole value grapheme by grapheme, and the per-character
String construction and append dominated the serialize path for entity-bearing
content.
Copy each run between escapable characters in a single append and rewrite only
the markers. The no-escaping-needed fast path (return the value unchanged) is
untouched. The output is byte-identical: matching is per-Character so multibyte
graphemes copy whole, and the escaped forms are the same.
Measured serialize ~7% faster on the 20k-item generated corpus via a same-state
A/B (~3.0x to ~2.8x slower than libxml2). Verified by EscapingBulkCopyTests (a
per-character-oracle differential with multibyte cases), the Apache Xalan
gold-output and C14N canonicalization suites, the full suite, the WASM build,
and an adversarial review that included a 400k-input fuzz.
parseTree ran TreeNode(parse(...)), which allocates two full trees: the parser
builds an immutable value Node tree, then TreeNode.init(_:) walks it to build
the mutable reference tree. The parsed tree is the deliverable, so building the
value tree first is pure overhead.
Parser.buildTree is the direct counterpart to build: the same iterative event
loop with the same element stack and the same well-formedness guards (an
unmatched end tag throws unexpectedEndOfInput; a document with no element throws
emptyDocument), but it accumulates TreeNode children per frame and adopts them
when an element closes, so the tree is allocated once. The produced tree is
identical: same kinds, names, attributes, values, child order, and parent links.
Measured parse ~8% faster on the 20k-item generated corpus via a same-state A/B
(6.1x to ~5.5x slower than libxml2). Verified by ParseTreeDirectBuildTests (a
structural differential against the convert-from-Node path, plus throw-parity
and a serialized round-trip), the full suite and the W3C XML conformance,
RELAX NG, and Xalan corpora (which all build trees through parseTree), the WASM
build, and an adversarial equivalence review.
EntityExpander.expand located the next '&', '<', and ';' with String.firstIndex
over graphemes, so every literal byte of a run was decoded into a grapheme just
to look for a one-byte marker. All three markers are ASCII, so a UTF-8 index is
a valid string index: scan over raw.utf8 instead, and the found index still
subscripts the string directly (the marker byte is never a continuation byte,
so its index lands on a scalar and grapheme boundary).
The decoded text is unchanged, including runs where the marker follows
multibyte UTF-8: a precomposed letter, a combining sequence, or an astral
character all leave the marker on a grapheme boundary.
Measured parse ~5.5% faster on the 20k-item generated corpus via a same-state
A/B (6.5x to ~6.1x slower than libxml2). Verified by EntityDecodeBulkTests
(extended with four multibyte-boundary cases), the W3C XML conformance corpora,
the Xalan XSLT gold-output corpus, the full suite, and the WASM build.
Building the mutable TreeNode tree from a parsed value Node ran append for every
child. append -> insert detaches the child from any previous parent, rejects a
self/ancestor cycle, and inserts into a children array that grows element by
element. The children the conversion passes come from element.children.map and
are freshly built and parentless, so all three guards are no-ops and the
incremental growth rebuilds an array that was already prepared.
TreeNode.init(adopting:) adopts the prepared array directly and wires each
child's parent. The tree is identical: the same children in the same order with
the same parent links. The adopting initializer is used only by the value-Node
conversion; every public factory keeps the detaching, cycle-checking append path
for externally supplied children that may already live in a tree.
Measured parse ~12% faster on the 20k-item generated corpus via a same-state
A/B (7.7x to ~6.5x slower than libxml2). Verified by TreeNodeAdoptingInitTests
(parent-wiring and post-adoption mutability), the W3C XML conformance corpora,
the Xalan XSLT gold-output corpus, the full suite, the WASM build, and an
adversarial equivalence review.
QualifiedName(_:) ran firstIndex(of: ":") over the whole name, grapheme by
grapheme, on every element and attribute. Most names have no prefix, so the
common case scanned the entire name to find no colon at all.
The byte scanner already visits every byte of an all-ASCII name, so have
takeASCIIName record the offset of the first colon (one comparison per byte),
and split at that known offset in QualifiedName(ascii:colonOffset:). The
unprefixed name (no colon) is then built with no scan; a prefixed name splits
at the recorded offset. The names are identical to the firstIndex path for
every colon placement: none, leading (:local), trailing (prefix:), middle,
doubled (::), and multiple (first colon wins). The hint is confined to the
all-ASCII fast path; non-ASCII names and every other construction site keep
QualifiedName(_:).
Measured parse ~12% faster on the 20k-item generated corpus via a same-state
A/B (8.5x to ~7.7x slower than libxml2). Verified by QualifiedNameColonHintTests
(colon-placement differential), the W3C XML conformance corpora, the Xalan XSLT
gold-output corpus, the full suite, the WASM build, and an adversarial
equivalence review.
EntityExpander.expand rebuilt every reference-bearing text run one character at
a time: result.append(character) per character, with a per-character charge
against the amplification budget. A text node holding a single & was copied
grapheme by grapheme, which is the bulk of the per-text-node cost in a document
whose content uses entities.
Scan to the next '&' or '<' and append the literal run in one operation,
charging the budget by the run length. The decoded text, the CDATA-verbatim
shielding, and the billion-laughs budget enforcement are unchanged: the same
overrun throws at the same input, since charging a run of length L against
budget B fails exactly when L > B, the same condition the per-character guard
enforced.
Also return early from recordReferenceContentFindings when there are no DTD
content models (the common no-DTD case): it otherwise scanned each text run for
'&' and rendered the enclosing element's qualified name before the model lookup
that would fail anyway.
Measured parse ~5% faster on the 20k-item generated corpus via a same-state A/B
(9.06x to ~8.5x slower than libxml2). Verified by the W3C XML conformance
corpora, the new EntityDecodeBulkTests six-case decode differential, the full
suite, the Xalan XSLT gold-output corpus, and the WASM build, with an
adversarial equivalence review of the rewrite.
Stamp the post-0.4.0 XPath performance work as 0.4.1: the descendant
single-accumulator traversal, the descendant and attribute node-test fusions,
the shared evaluation Environment, and the compiled per-query node test. xpath
goes from ~5.5x to ~1.3x slower than libxml2 across these passes, with identical
results (each correctness-sensitive change differential-tested and adversarially
reviewed).
The descendant and attribute walks tested every candidate node through a
closure that called matchesTree / matchesAttribute with the step's NodeTest and
the evaluation namespaces dictionary:
descendants(of: tree) { matchesTree($0, step.test, on: step.axis, context.namespaces) }
That passes the namespaces dictionary (and the NodeTest's associated string) by
value on every call, so a //name-style query over a wide document carried a
dictionary retain/release on each of tens of thousands of nodes, on the
traversal that dominates evaluation.
compiledTreeTest, compiledAttributeTest, and compiledNodeTest resolve the parts
that do not change as the walk moves (the axis principal kind, whether eval-time
bindings are in effect, the prefix-wildcard shape of the name) exactly once and
return a tight per-node closure capturing only the local name. The selected
nodes are identical to the unfused matchesTree / matchesAttribute / matches.
Measured xpath ~35% faster on the 20k-item generated corpus via a same-state
A/B, from ~2.0x to ~1.3x slower than libxml2. The full suite, the Apache Xalan
XSLT gold-output corpus, and the WASM build stay green. XPathCompiledNodeTestTests
is an exhaustive differential asserting each compiled closure equals the
function it specializes across every node, test, axis, and binding.
An EvaluationContext carried the variable bindings, the function library, the
namespace bindings, and the budget by value. Re-focusing the context onto each
node of a node-set (focused(on:position:size:), called once per node when a
predicate is applied) copied all three dictionaries every time, even though
none of them changes as the focus moves.
Bundle the focus-independent state in an Environment reference type. The
context now holds the node, position, size, and an Environment reference;
re-focusing copies the three integers and shares the environment by reference
instead of copying and reference-counting three dictionaries. The values and
behavior are identical.
Measured xpath ~16% faster on the 20k-item generated corpus via a same-state
A/B (count unchanged at 10000). The full XPath suite, the Apache Xalan XSLT
gold-output corpus, and the WASM build stay green.
The attribute axis built an XPath node for every attribute of the element
and then filtered by the step's node test. A predicate like [@name='value']
runs that per matched element, so for a wide document it wrapped many
attributes only to discard all but one, and each wrap retains the owner
element and copies the attribute's qualified name and value.
Apply the same fusion the descendant axes use: test the raw attribute during
the walk and wrap only the ones that pass. matches delegates, for an
attribute node, to a new matchesAttribute (the attribute axis always carries
the attribute principal kind, so the reduction is exact), and the attribute
walk applies it as a keep filter. The node-set, declaration order, and
uniqueness are identical to build-then-filter.
Measured xpath ~20% faster on the 20k-item generated corpus via a same-state
A/B (count unchanged at 10000); xpath is now ~2x libxml2, down from ~13x at
the start of this work. XPathFusedAttributeTests covers every node-test kind
through the fused path, and an adversarial equivalence review found no query
whose result changes. Full suite, Apache Xalan XSLT gold output, and the WASM
build stay green.
The descendant and descendant-or-self axes built the full subtree node list
and then filtered it by the step's node test. For a //name query over a wide
document most of the wrapped nodes were discarded immediately, and each wrap
retains a tree node whose weak parent forces the slow reference-counting path.
Run the node test against the raw tree node during the walk, so a rejected
node is never wrapped. matches(_:_:on:namespaces:) now delegates, for a tree
node, to a new tree-node matchesTree (one source of truth, exercised by the
full XPath suite), and the descendant walk applies it as a keep filter. The
node-set, document order, and uniqueness are identical to build-then-filter;
only non-matching nodes are no longer materialized.
Measured xpath ~11% faster on the 20k-item generated corpus via a same-state
A/B (count unchanged at 10000). XPathFusedDescendantTests covers every
node-test kind through the fused path, and an adversarial equivalence review
found no query whose result changes. Full suite, Apache Xalan XSLT gold
output, and the WASM build stay green.
`descendants(of:)` recursed on wrapped XPath nodes and, at every node, built
a fresh array for that node's subtree and concatenated it
(`result.append(contentsOf: descendants(of: child))`). That is an array
allocation and copy per node, roughly O(n x depth) work over the subtree, and
the parse profile showed it dominating XPath evaluation for a `//`-style query
over a wide document.
Recurse on the raw tree node instead, appending into one shared inout
accumulator. The result is the identical node-set in the identical document
order; only the way the array is built changes, from O(n x depth) to O(n).
Measured xpath ~46% faster on the 20k-item generated corpus via a same-state
A/B (count unchanged at 10000). Full suite, the Apache Xalan XSLT gold-output
corpus, and the WASM build stay green.
Performance across all three benchmarked operations plus one XSD conformance
fix, all backward compatible. Versus libxml2 on the 20k-item generated corpus:
parse ~30x -> ~9x, serialize ~5.5x -> ~2.7x, xpath ~13x -> ~5.5x. The fix
rejects an xsi:type that reaches a union declared type through a blocked
member (XSTS invalid-instances-accepted 15 -> 14, valid-rejected held at 0).
Verified against the full local suite (1695 tests), the Apache Xalan XSLT
gold-output corpus, the W3C XML conformance corpora, the RELAX NG spec suite,
the full W3C XSTS archive, and the WASM build.
A path that is a single forward-axis step from one context node returns its
nodes already in document order with no duplicates, so the document-order
sort is redundant. AxisNavigation produces the structural forward axes
(child, descendant, descendant-or-self, self, following, following-sibling)
in document order, deriving sibling order from the same children array the
order key compares, so the two can never disagree. This is the common shape
after //x compiles and fuses to a single descendant::x step.
The attribute axis (whose order interleaves with namespace declarations), the
namespace axis (produced in dictionary order), and every reverse axis
(nearest-first) are excluded and still sorted. Measured xpath ~7% faster on
the 20k-item generated corpus via a same-state A/B, identical results.
XPathSingleStepOrderTests pins document order for each included axis, that the
reverse axes still normalize, and that multi-step paths still sort. An
adversarial order-correctness review found no query whose order or uniqueness
the change breaks.
A path expression sorted its result into document order through orderUnique,
which first ran a Set-based de-duplication. But evaluateSteps already returns
a duplicate-free node-set: each step de-duplicates across its context nodes,
or uses an axis whose per-context results cannot overlap. So that Set copied
every node for nothing, and each copy retains the wrapped tree node, whose
weak parent reference forces a side table and the slow reference-counting
path.
Path and trailing-path evaluation now sort the steps' result directly.
Unions and filter primaries, which genuinely can carry duplicates, keep the
de-duplicating pass. Measured xpath ~9% faster on the 20k-item generated
corpus via a same-state A/B, identical results. The existing node-set
de-duplication and union tests cover that a path still yields each node once.
`//X` compiles to `descendant-or-self::node()/child::X`, which materializes
every node of the subtree as an intermediate context before selecting the X
children of each. Fuse that pair into a single `descendant::X` step at compile
time: the two select the same node-set, so evaluation skips the intermediate.
The fusion is transparent except for a positional predicate: `//x[1]` is the
first x child of EACH parent, not the first x in document order, so it must
not become `descendant::x[1]`. Apply the rewrite only when every predicate is
provably non-positional: its value is a boolean, node-set, or string (never a
number, which XPath reads as position() = value) and it nowhere references
position() or last(). Anything not provably both is left unfused, so a result
can never change.
Measured xpath ~10x to ~7x slower than libxml2 on the 20k-item generated
corpus via a same-state A/B, identical results. XPathDescendantFusionTests
pins the positional forms keeping their per-parent meaning, the non-positional
forms selecting across the subtree, and composition with trailing and nested
steps. An adversarial differential review (fused versus a spec-literal unfused
reference over nine attack suites) found no query whose result changes. The
boolean-function allowlist carries a documented invariant: every entry must be
position- and size-independent, since the position scan is syntactic.
Each location step accumulated its context nodes' axis results into a
Set<Node> to drop nodes reachable from more than one context, hashing and
copying every node. The wrapped TreeNode is reference-counted, so each copy
is ARC traffic, and the profile showed the Set insert, hashing, and Node
copies dominating evaluation.
De-duplication is needed only when two distinct context nodes can reach the
same node. Skip it on a single context and on the axes whose results are
disjoint per context (child, attribute, namespace, self): those produce the
identical node sequence the Set would, since every insertion succeeds, with
no hashing. Keep the Set on the axes that genuinely overlap (descendant,
ancestor, parent, the siblings, following/preceding). Predicates stay applied
per context and the path result still passes through the final
order-and-unique pass, so results are identical.
Measured xpath ~13x to ~10x slower than libxml2 on the 20k-item generated
corpus via a same-state A/B, ~28% faster, same result. XPathNodeSetDedupTests
pins that a node shared by many contexts through ancestor/parent/descendant
is still counted once, the disjoint axes return every distinct result, and an
overlapping union still de-duplicates. The full XPath suite and the Apache
Xalan XSLT gold-output corpus stay green.
The serializer's escaping functions rebuilt every text, attribute, comment,
and processing-instruction value character by character, even when the value
contained nothing to escape, which is the common case. Each now scans the
value's UTF-8 bytes for the bytes it would escape (the markup characters, the
active attribute quote, the whitespace characters, a comment hyphen, a PI
question mark, and any non-ASCII byte in ASCII-only mode) and returns the
value unchanged when none is present, taking the original character path only
when something must be escaped.
Measured serialize roughly halved, ~5.5x to ~2.7x slower than libxml2 on the
20k-item generated corpus, with byte-identical output. EscapingFastPathTests
pins both the unchanged fast path and the escaped slow path for each function,
and the Apache Xalan XSLT gold-output corpus (a serialize-heavy differential)
stays green. No output changed.
The attribute-value scanner ran the entity decoder on every value, and the
decoder re-scanned the whole value for '&' before doing nothing when none
was present. The byte-run scanner already visits every value byte, so it
now notes any '&' and an ampersand-free value (the common case) returns its
normalized run directly. Measured against a same-machine-state baseline,
every run came in below every baseline run, ~4% faster parse on the
20k-item generated corpus.
Move the bulk byte-run scanners (contentRunBytes, attributeRunBytes) into
ReaderByteRuns.swift and centralize their cursor mutation through a single
advanceByteRun method, so Reader's position fields stay write-encapsulated
in Reader.swift while both files stay under the length cap. The byte and
Character paths are held to identical decoded values by the differential
oracle, extended with ampersand-free, entity, and character-reference
attribute cases. No spec behavior changed.
cvc-elt.4.3 defers to cos-st-derived-OK, whose clause 2.2.4 makes a type
validly derived from a union when it is validly derived from one of the
union's member types. The block check followed only the named base chain,
so when the declared type was a union and the substitute reached it through
a member by a blocked derivation method, the check found no path and stayed
silent, wrongly accepting the substitution (W3C XSTS elemT074: block on the
element is restriction and the xsi:type restricts a union member).
Record a union's direct member types during schema compilation and follow a
derivation through a member when the base chain does not reach the union.
The member clause is existential: bar the substitution only when every
confirmed derivation path to a member is blocked, so a substitute that is
itself a member (its self-membership path uses no method) or reaches one
member by an unblocked method is accepted whatever other members it reaches
by a blocked method. The check adds a rejection only on a positively
confirmed path, so an unresolved, inline-anonymous, or nested-union member
leaves it silent: a bounded, disclosed under-rejection, never a false
positive.
Drops XSTS invalid-instances-accepted 15 to 14 with valid-schemas-rejected
and valid-instances-rejected held at 0 over the full archive. An adversarial
over-rejection review caught a first cut that order-shadowed the
self-membership path; the existential form rules it out and the case is a
regression test.
The element-content markup dispatch ran up to five literal string
comparisons (</, <!--, <![CDATA[, <?, <!DOCTYPE) before the start-tag
branch won, a per-element cost the parse profile indicted as
Reader.matchesBytes. Branch instead on the single byte after '<' on the
byte fast path: '/' end tag, '?' processing instruction, '!' a
declaration, otherwise a start tag.
The Character path keeps the equivalent matches()-based dispatch for a
streaming source or spliced entity text, and a '<!' that matches none of
comment/CDATA/DOCTYPE still falls through to the start-tag scanner with
the same expectedName error. Measured parse ~9.5x -> ~9.0x slower than
libxml2 on the 20k-item corpus; the byte and Character paths are held
identical by ParserPathDifferentialTests over its adversarial corpus,
and the Xalan, W3C, and RELAX NG suites stay green. No spec behavior
changed.
Keep the reader on its byte fast path through dispatch and name/attribute
scanning so plain-ASCII documents avoid the per-Character String machinery
(String.Iterator.next, string comparison, small-string materialization,
bridge-object ARC) that a profile indicted as the parse bottleneck.
The root cause was that a single buffered lookahead Character defeated every
existing byte path, each of which requires an empty buffer, and the content,
markup, and attribute dispatch each peeked a Character before scanning. Add a
non-buffering peekByte for dispatch, a byte-mode skipSpace, an all-ASCII
takeASCIIName, and an attributeRunBytes bulk scanner (with the 3.3.3
whitespace-to-space transform, deferring any carriage return to the Character
path so CRLF is not double-counted). Detect a text run's closing '<' via
peekByte instead of buffering it, which had cascaded the next element's whole
dispatch and name scan onto the slow path. Skip reference decoding for
ampersand-free text. The Character path remains the exact fallback for a
streaming source, spliced entity text, and any non-ASCII byte; the two paths
share scanContentMarkup so they cannot diverge.
Measured parse 20.1x -> 9.5x slower than libxml2 on the 20k-item generated
corpus, roughly a 53 percent wall-clock reduction. A differential oracle pins
the byte and Character paths to identical event streams and error marks over
an adversarial corpus, alongside boundary tests. Verified against the full
local suite, the Apache Xalan XSLT gold-output corpus, the W3C XML conformance
corpora, the RELAX NG spec suite, and the WASM build. No spec behavior changed.
* fix(xslt): keep the literal prefix for xsl:namespace-alias, remap only the URI
XSLT 1.0 7.1.1: an aliased result element keeps the prefix used in the
stylesheet; only its namespace URI is replaced by the result namespace. The
serializer adopted the result-prefix as the output prefix, so axsl:foo aliased
to the XSLT namespace came out as xsl:foo. The result-prefix now only selects
the replacement namespace; the element name, attributes, and copied namespace
declarations all keep their literal prefix.
Closes Apache Xalan conformance cases namespace19, namespace23, namespace24, namespace35, namespace113.
* refactor(xslt): drop the now-unused NamespaceAlias prefix field
The namespace-alias output keeps the literal prefix, so the result-prefix is no
longer stored; only the result namespace URI is needed. Removes the dead field
and corrects the addNamespaceAlias doc comment.
The xsl:key match was turned into a path by prepending //, but for a union
match like "Level1 | Level2" the // applied only to the first branch
(//Level1 | Level2), so the second branch matched relative to the root and
found nothing, dropping those nodes from the key. Distribute // over each
top-level union branch (// inside a predicate or function call is left alone).
Closes Apache Xalan conformance cases idkey45, idkey46, idkey47, idkey48.
XSLT 1.0 3.4: a whitespace-only text node in the stylesheet is preserved when
the nearest ancestor-or-self with an xml:space attribute has value preserve.
The body parser dropped every whitespace-only run unconditionally, so a literal
result element with xml:space="preserve" lost its whitespace content. Consult
the in-scope xml:space before stripping.
Closes Apache Xalan conformance case whitespace20.
XSLT 1.0 7.7: for level=single the count node may be the from-boundary node
itself when that node matches both the count and from patterns. The walk
stopped at the nearest from-matching ancestor before testing it against count,
so count="a|b" from="a" gave an empty number for a node inside an a. Test the
count pattern before the boundary stops the walk.
Closes Apache Xalan conformance case numbering63.
XSLT 1.0 7.7: the format attribute of xsl:number is an attribute value
template. It was stored and used as a literal string, so format="{$f}" was
never evaluated and a passed-in or computed format was ignored. Parse it as a
ValueTemplate and evaluate it per instantiation.
Closes Apache Xalan conformance case namedtemplate12.
XPath 1.0 4.2: an integer-valued number converts to a string with no decimal
point. The formatter only took the integer branch for a magnitude below 1e15,
so a value like 1234567890123456 (exact in a double, which holds integers
exactly to 2^53) printed as 1234567890123456.0. Raise the bound to 2^63, the
range where the Int64 conversion is safe; larger magnitudes still expand
through the exponent path.
Closes Apache Xalan conformance case string132.
The XPath grammar makes the relative part after / optional, but the compiler
only treated / as the root location path when it was at the end of the
expression (isAtEnd). When a / was followed by ), ], |, or an operator (as in
count(/), generate-id(/), or / = .), it fell through and tried to parse a step
from the terminator, yielding an empty node-set. Recognize / as the root
whenever no location step follows.
Closes Apache Xalan conformance cases position106 and mdocs15.
XSLT 1.0 5.5: the default priority of a single-step match pattern is set by its
node test, regardless of an explicit child:: or attribute:: axis. defaultPriority
matched the whole pattern string, so attribute::node() fell through to 0 (a name
test) instead of -0.5 (a node() node test), wrongly outranking @*. Strip a
leading child::/attribute:: axis before classifying the node test.
Closes Apache Xalan conformance cases conflictres29 and conflictres30.
XSLT 1.0 3.4: an xsl:strip-space/xsl:preserve-space elements name test is a
namespace-qualified NameTest, matched by namespace, not by prefix. A specifier
prefix:* or prefix:local now resolves its prefix to a URI at parse time, and a
source element is matched on its expanded name by specificity (exact {uri}local
over namespace wildcard {uri}* over *). The tests compared raw prefixed strings,
so ns1:* did not strip a source element bound to the same namespace under a
different prefix.
Closes Apache Xalan conformance cases whitespace06 and whitespace07.
XSLT 1.0 16.4: disabling output escaping for a value used as something other
than a text node (an attribute, comment, or processing-instruction string
value) is an error; the recovery is to ignore it. The attribute value built
from such content kept its raw-output markers, so the serializer emitted < and
> unescaped in the attribute. Strip the markers for these three node kinds, via
a shared helper, gated on the stylesheet actually using the feature so a
private-use character in ordinary data is never removed.
Closes Apache Xalan conformance case output75.
* fix(html): percent-escape non-ASCII characters in URI-valued attributes
XSLT 1.0 16.2 / HTML 4.01 B.2.1: the html output method percent-escapes
non-ASCII and control characters in attributes of type %URI (href, src, cite,
action, and the rest), writing each as the uppercase hex of its UTF-8 bytes,
while keeping " and & as entities. Other attributes are unaffected. The
serializer left such characters unescaped. Grounded in the libxml2
htmlSerializeUri study; reimplemented in pure Swift, no Foundation.
Closes Apache Xalan conformance case output32.
* docs(html): disclose URI-attribute escaping round-trip normalization
Address the URI-escaping critics: the html serializer follows libxml2's HTML
output model where URI-valued attributes are always percent-escaped, so a
parse-then-serialize round-trip normalizes a space or non-ASCII byte to %HH.
Document that this is idempotent (a literal % is left as is) and that every
space is escaped (leading ones included, where libxml2 skips them). Add an
idempotency test.
PI data may not contain ?>, which would terminate the instruction early. The
serializer emitted it raw, producing malformed output. Insert a space between
an offending ? and > (XSLT 1.0 7.3 recovery: a processor recovers from a
generated ?> by inserting a space after the ?), via a shared Escaping helper
applied at both XML PI serialization sites. A trailing ? is left alone.
Closes Apache Xalan conformance case output72.
The Xalan harness read source and stylesheet files as UTF-8, so a UTF-16
source (output80) or other non-UTF-8 input (attribvaltemplate08) was refused
as unreadable before the transform ran. Read them through PureXML's own
ByteDecoder (BOM or XML-declaration detection, UTF-8 fallback), as the harness
already does for gold files. The library round-trips them correctly, so both
cases now pass.
Closes Apache Xalan conformance cases output80 and attribvaltemplate08.
* fix(xslt): serialize a namespaced element as XML under the html output method
XSLT 1.0 16.2: the html output method outputs an element whose expanded-name
has a non-null namespace URI as XML, not HTML. A namespaced document element's
tree was serialized as HTML, leaving empty namespaced elements unclosed (e.g.
a void name like br in a namespace) and repeating xmlns on every element, which
is not well-formed XML. Such a tree now serializes through the XML path (empty
elements self-close, the namespace is declared once and inherited); the html
method still omits the XML declaration. Null-namespace trees are unchanged.
Grounded in XSLT 1.0 16.2 and the libxml2 HTML serializer study; reimplemented
in Swift. Apache Xalan output63 exercises this rule but stays baselined because
its gold differs further only in non-normative attribute order.
* docs(xslt): disclose the document-element granularity of the 16.2 namespace rule
Make the whole-tree approximation explicit in the serializedBody comment and
the changelog: the html-outputs-namespaced-elements-as-XML rule is applied at
document-element granularity, not per element, so a mixed-namespace tree
follows its document element's path.
* fix(xslt): name the html output method doctype HTML, not the element name
XSLT 1.0 16.2: for the html output method the name following <!DOCTYPE should
be HTML or html, not the document element name (which is the xml method's rule,
16.1). The serializer used the document element name for both methods, so
method=html with a non-html root element (e.g. <root>) emitted
<!DOCTYPE root PUBLIC ...> instead of <!DOCTYPE HTML PUBLIC ...>.
Closes Apache Xalan conformance cases output40, output48, output60.
* fix(xslt): emit no html doctype when the result has no element
Addresses the doctype critic finding: a doctype precedes the first element
(16.2), so a text-only result must emit none even for the html method. The
fixed HTML name made the existing no-element guard unreachable; restore it by
guarding on the document element before choosing the name.
XSLT 1.0 16.2: the html output method escapes & and " in attribute values
but, unlike the xml method, leaves < (and >) literal. The serializer was
escaping < to < in attribute values. Removing that case lets it pass
through; the xml method is unchanged and still escapes both.
Closes Apache Xalan conformance cases output49 and output74.
* fix(xslt): resolve string-form document() against the stylesheet element's base URI
A string-form document() reference resolves against the base URI of the
stylesheet element holding it (XSLT 1.0 12.1). For a declaration in an
included or imported file that is the included file's URI, not the top
stylesheet's, so document('data.xml') (or document('')) in an included file
must resolve against the included file.
The parser already tracked each top-level declaration's source-file base but
discarded it after collection. That base is now carried onto the compiled
templates and global variables, into the runtime context, and through to the
function table, so the document() string form resolves against it. Templates
and globals both carry it, so the rule holds wherever the call appears.
Closes Apache Xalan conformance cases mdocs12 and mdocs13.
* fix(xslt): carry base URI into for-each and sort contexts; keep document('') self-referential
Follow-up to the string-form document() base URI work. Three coverage gaps:
- xsl:for-each and xsl:sort evaluate their body and key expressions in a
context derived from, not equal to, the enclosing template's, so a
document() call there resolved against the empty base when the template came
from an included file. Both now inherit the enclosing context's base URI.
- A non-empty base URI made document('') in the top stylesheet resolve to that
base and route through the loader, rather than returning the stylesheet's own
parsed document as before. The parsed stylesheet is now registered under its
base URI, so document('') reaches it without the loader; document('') in an
included file (a different base) still loads that file.
* fix(xslt): resolve document() references against the correct base URI
The document() function resolved every relative URI reference against the
single global base, so a reference relative to a document that document()
itself had loaded from a subdirectory could not be found. XSLT 1.0 12.1
defines the base per reference: the two-argument form against the second
argument's first node, the node-set form against each reference node's own
document.
Each loaded document now records the URI it came from as its base, and a
relative reference resolves against that base before the loader applies its
own source-document base. The common one-argument string form is unchanged
(its base is empty, deferring to the loader as before).
Closes Apache Xalan conformance cases reluri09, reluri10, reluri11.
* fix(xslt): record base URI for document() fragment selections
A fragment selection (document('x#frag')) is returned as a detached subtree
whose root identity differs from the cached whole-document tree, so its base
URI was not recorded. Using such a node as the base for a relative document()
reference then resolved against the empty base instead of the fragment's
document. Record the base on each fragment subtree's own root through a shared
helper, so baseURI(of:) finds it whichever node the reference is taken from.
XSLT 1.0 names every QName-keyed declaration by expanded name, so two prefixes
bound to the same namespace name the same thing. mode and variable names were
already resolved this way; named templates, xsl:key, xsl:attribute-set, and
xsl:decimal-format names were still keyed and compared as raw prefix strings.
A declaration name now resolves to {uri}local at parse time, and the references
resolve their prefix against the in-scope namespaces the same way: call-template
and use-attribute-sets at parse time, and the key() name argument and
format-number's format name at evaluation (a shared Library.expandedName helper
covers the two runtime sites). The parse-time expansion reuses a factored
expandedQName helper shared with mode and variable.
Closes Apache Xalan conformance cases namedtemplate16, attribset48, idkey53.
Full suite 1632 tests green in debug and release; the Xalan suite passes.
* fix(xslt): keep namespace nodes on an included literal result element
An xsl:include'd stylesheet's declarations are absorbed after collectUnit
returns, and each literal result element resolves its 7.1.1 namespace nodes
by walking the weak parent chain back to its own included stylesheet element.
Those elements lived only in the write-only retainedRoots array, so an
optimized build released them before the deferred resolution ran and the
in-scope xmlns declarations vanished, dropping the namespace nodes the
included element should carry. The output was correct in debug and wrong in
release. Pin the retained roots across the resolution with
withExtendedLifetime. Imports were unaffected because they compile eagerly
while their tree is still a live local.
Catches Apache Xalan conformance case lre/lre05, surfaced by running the
apache/xalan-test corpus.
* fix(xslt): select a called template by import precedence, not document order
xsl:call-template looked up the target with the first match by position in the
flattened template list. Imports fold in before the including unit's own and
included templates, so when an import and an include both define a template
with the same name, the imported one was wrongly chosen. An included template
carries the including unit's higher import precedence and must win (XSLT 1.0
section 6). The lookup now picks the highest-precedence definition among the
same-name templates.
Closes Apache Xalan conformance cases namedtemplate17, namedtemplate18,
namedtemplate19, and namespace13 (removed from the xalan-test baseline). Full
suite 1608 tests green in debug and release; the Xalan suite passes.
* fix(xpath): string-value of a comment or PI node is its own data
The XPath string-value of a comment node is its character data and of a
processing-instruction node is its data (XPath 1.0 5.6, 5.7), but TreeNode
returned the empty string for both, so value-of select="comment()", and any
matching, sorting, or comparison over comment/PI nodes, produced nothing.
stringValue was conflated with the text-only concatenation that forms an
element's or root's string-value, where comment and PI descendants correctly
contribute nothing (5.1, 5.5). That concatenation now uses a separate
textContent, so an element's string-value still excludes comment and PI text
while a comment or PI node reports its own data.
Closes 24 Apache Xalan conformance cases across node, axes, position, select,
modes, conflictres, and variable (removed from the baseline). Full suite 1609
tests green in debug and release; the Xalan suite passes.
* fix(xslt): resolve top-level variables in any order, not document order
Top-level variables and parameters may reference one another regardless of
declaration order (XSLT 1.0 section 11.4), but they were evaluated once
top-to-bottom, so a forward reference (a global declared before the one it
uses) saw an undefined value and resolved to empty. Globals now evaluate to a
fixpoint: each pass re-evaluates the unsettled ones with the latest bindings,
so a dependency chain in any order settles in at most N passes, a pass that
changes nothing stops early, and a circular reference stops at the cap.
Extracted the evaluation into Transformer.evaluatedGlobals (XSLTGlobals.swift).
Closes Apache Xalan conformance cases variable33, variable34, variable35. Full
suite 1611 tests green in debug and release; the Xalan suite passes.
* fix(xslt): an empty-content variable with no select is the empty string
XSLT 1.0 section 11.2 makes <xsl:variable name="x"/> (empty content, no select)
equivalent to select="", so boolean() of it is false. It was always built as a
result-tree-fragment document node, which boolean() reports as true. A variable
with no select and empty content now yields the empty string; only non-empty
content remains a result tree fragment.
Closes Apache Xalan conformance cases boolean43 and boolean87. Full suite 1612
tests green in debug and release; the Xalan suite passes.
* fix(xpath): allow whitespace around the :: axis separator
XPath 1.0 allows whitespace between tokens, so `child :: sub` is the child axis
with node test `sub`. The parser only matched `::` immediately after the axis
name, so a spaced separator fell back to the child axis with the axis name read
as an element test, selecting nothing. parseAxis now skips whitespace after the
axis name and after `::`; on no match the reset to before the name is unchanged,
so a bare name step is unaffected.
Closes Apache Xalan conformance cases select16, select27, select28. Full suite
1613 tests green in debug and release; the Xalan suite passes.
* feat(xslt): document('') returns the stylesheet's own document
XSLT 1.0 section 12.1 makes an empty URI reference the document containing the
expression, so a stylesheet can read its own xsl:* nodes (embedded lookup
tables, self-inspection). The reference went straight to the document loader,
which has no entry for it, so document('') returned an empty node-set. The
transformer now keeps its parsed stylesheet tree and the document function
returns it for an empty reference, cached like any loaded document; a #fragment
still applies.
Closes Apache Xalan conformance cases select67, select68, idkey50, mdocs17,
namespace20. Full suite 1614 tests green in debug and release; Xalan passes.
* fix(xslt): implement unparsed-entity-uri from the source DTD
unparsed-entity-uri was a stub returning the empty string. It now reads the
source document type's unparsed (NDATA) entities, threaded to the function table
alongside the document loader, and returns the named entity's system URI; an
unknown name still returns empty.
Closes Apache Xalan conformance case expression02. Full suite 1615 tests green
in debug and release; the Xalan suite passes.
* fix(xpath): substring-after(s, '') returns s
Per the XPath 1.0 errata, an empty second argument matches at the start of the
string, so substring-after returns the whole string. The function bailed to the
empty string on an empty separator. substring-before(s, '') stays empty, as the
spec requires.
Closes Apache Xalan conformance case string143. Full suite 1615 tests green in
debug and release; the Xalan suite passes.
* fix(xslt): match a template mode by expanded name, not its prefix
XSLT 1.0 section 5.7 compares a mode by expanded name, so two prefixes bound to
the same namespace name the same mode. The prefixed QName string was stored and
compared verbatim, so apply-templates mode="foo:m" missed a template with
mode="moo:m" even when foo and moo share a namespace. Mode names on xsl:template
and xsl:apply-templates now resolve to {uri}local at parse time.
Moved the parser's declaration helpers to XSLTParserDeclarations.swift to keep
the file within the length budget.
Closes Apache Xalan conformance case modes16. Full suite 1616 tests green in
debug and release; the Xalan suite passes.
* fix(xslt): name a variable by expanded QName, not its prefix
XSLT 1.0 section 11.1 names a variable by expanded name, so two prefixes bound
to the same namespace name the same variable. The declared name and the
$prefix:local reference were compared as raw strings, so $new:x did not see a
variable declared txt:x even when new and txt share a namespace. A prefixed
declared name now resolves to {uri}local at parse time, and the XPath evaluator
falls back to the namespace-resolved name when the raw key is unbound, so a
caller-supplied raw binding still works and unprefixed names are unchanged.
Closes Apache Xalan conformance case variable55. Full suite 1617 tests green in
debug and release; the Xalan suite passes.
* fix(xslt): a match pattern predicate sees the global variables
XSLT 1.0 section 5.2 evaluates a pattern predicate with the top-level (global)
variables in scope, but pattern matching ran with no variables bound, so a
template like match="foo[. > $screen]" never matched ($screen was undefined).
The transformer supplies the global bindings to the match cache; they are
constant per transform, so they do not enter the cache key.
Closes Apache Xalan conformance cases match14 and idkey43. Full suite 1618 tests
green in debug and release; the Xalan suite passes.
* fix(emit): make a comment ending in a hyphen or containing -- well-formed
XML forbids `--` inside a comment and a `-` immediately before `-->`, so a
comment whose text ends with `-` was emitted as <!--text---> (malformed). The
serializer now inserts a space after each offending hyphen (the XSLT 1.0 16
recovery), so `text-` becomes `text- `. Applied in both the Serializer and the
streaming Writer via a shared Escaping.comment helper.
Closes Apache Xalan conformance cases output89 and output90. Full suite 1619
tests green in debug and release; the Xalan suite passes.
* fix(xslt): preserve whitespace adjacent to a CDATA section in a template
Text and CDATA are one text node in the data model, so <out> <![CDATA[test]]>
</out> is the node " test ", whose surrounding spaces must survive whitespace
stripping (XSLT 1.0 3.4). The template body dropped each whitespace-only text
node independently, so the spaces around the CDATA vanished. Adjacent text and
CDATA children now coalesce into one run, dropped only when the whole run is
whitespace; a purely whitespace run between elements is still removed.
Closes Apache Xalan conformance case whitespace13. Full suite 1620 tests green
in debug and release; the Xalan suite passes.
* fix(xslt): xsl:processing-instruction/comment keep only text-node content
XSLT 1.0 sections 7.4 and 7.6 make it an error for xsl:comment or
xsl:processing-instruction content to create a non-text node; the recovery is
to ignore the offending node together with its content. PureXML took the
string-value of all instantiated content, so a copied element leaked its text
into the PI or comment. The content now contributes only its text and CDATA
node items.
Closes Apache Xalan conformance case copy60. Full suite 1621 tests green in
debug and release; the Xalan suite passes.
* fix(xslt): xsl:attribute keeps only text-node content (errata E27)
XSLT 1.0 errata E27 makes it an error for xsl:attribute content to create a
non-text node, recovering by ignoring the offending node together with its
content (the same rule as xsl:comment/processing-instruction). PureXML took the
string-value of all instantiated content, so a copied element leaked its text
into the attribute value. The content now contributes only its text and CDATA
node items, reusing textNodesOnly.
Closes Apache Xalan conformance cases copy56 and copy58. Full suite 1622 tests
green in debug and release; the Xalan suite passes.
* fix(xslt): xsl:copy-of carries an element's inherited namespace nodes
XSLT 1.0 11.3 copies a node together with its namespace nodes, so copying an
element must reproduce its full in-scope set, including a declaration inherited
from an ancestor. copy-of reproduced only the element's own xmlns attributes,
dropping inherited ones. The inherited declarations are now added to the copied
element and the serializer's namespace fixup drops any already in the result
scope.
Closes Apache Xalan conformance cases copy09, copy27, copy48, mdocs09. Full
suite 1623 tests green in debug and release; the Xalan suite passes.
* fix(xslt): xsl:element with an unusable name passes its content through
XSLT 1.0 7.1.2 makes a name that is not a QName an error; the recovery (the
pinned element-name-not-QName "pass-through") emits the element's content
without the wrapper. The check rejected only an empty or multi-colon name, so
an undeclared prefix (none:foo) or a name that is not an NCName (this is a
string) still produced a malformed element. The name is now also rejected when
a part is not an NCName or its prefix is unbound.
Closes Apache Xalan conformance cases namespace40, namespace42, namespace43,
namespace60, namespace73, namespace79. Full suite 1624 tests green in debug and
release; the Xalan suite passes.
* fix(xslt): xsl:copy of the root node honors use-attribute-sets
Copying the root node produces no element of its own, so the use-attribute-sets
attributes have nowhere to attach on the copy and instead join the enclosing
result element (XSLT 1.0 7.5). copyInstruction dropped them for the root case;
they are now emitted ahead of the copied content so the enclosing element picks
them up.
Closes Apache Xalan conformance cases attribset27 and attribset29. Full suite
1625 tests green in debug and release; the Xalan suite passes.
* refactor(xslt): address review (canonical name/whitespace checks, html comment safety)
Review follow-ups on the conformance series:
- xsl:element name validation now uses the canonical Parsing.XMLCharacter
.isValidName instead of a hand-rolled grapheme check, which had narrowed valid
XML names (it dropped a name using a non-ASCII NameChar such as U+00B7).
- The template-body whitespace check reuses Parsing.XMLCharacter.isWhitespace
instead of a third local copy of the same predicate.
- The HTML serializer applies the section-16 comment recovery too, so a
method="html" comment ending in a hyphen (or containing two) is well-formed;
the C14N canonicalizer keeps comments verbatim as its contract requires.
Adds regression tests for a non-ASCII NameChar element name and for html
comment hyphen-safety. Full suite 1626 tests green in debug and release; the
Xalan suite passes.
A redefined model group whose body references itself expands, in the compiled
content model, to the original group it redefines. Two gaps let an ambiguity
created that way slip through the Unique Particle Attribution check (cos-nonambig).
The compositional determinism engine treated the self-reference as contributing
an empty decision set, so the original group's particles were invisible to the
overlap test. It now expands one level to the redefine base, exactly as the
instance content-model path does, so a particle the original contributes is seen.
The expansion only surfaces genuinely reachable items, so the decision sets stay
a subset of the true model: the check can catch a real ambiguity but never invent
one, holding the false-positive direction at zero.
Separately, the UPA check walked only the root document, so an ambiguity a root
redefine creates in a complex type that lives in an included document was never
evaluated. It now runs over every schema root in the composition under the shared
redefine-effective context, mirroring the existing cross-document top-level
declaration check.
Closes XSTS schN10 (a three-level redefine chain). XSTS invalid-schemas-accepted
11 -> 10, valid-schemas-rejected held at 0. Full suite 1609 tests green in debug
and release.
* docs(design): derive the exact Particle Valid (Restriction) procedure
The false-negative tail (particles*/elemZ) routes through one approximate
oracle, ParticleRestriction.violation. Rather than patch cases, derive the
exact XSD 1.0 3.9.6 decision procedure once, clean-room from the spec and the
Xerces reference algorithm, with the soundness (no-false-positive) argument for
each clause: normalization, substitution-group expansion, effective total
range, the compositor dispatch table, and the rcases.
Identifies the closeable gaps (subst-group expansion for elemZ026/V020,
MapAndSum effective-total-range, NameAndTypeOK union-member derivation) and
names the genuine spec limitations (Ha161/Z001 contradiction, mgO013 contested,
union-member by-name derivation) as bounded, not bugs. Sets the implementation
order, each clause gated by the proof plus the XSTS valid-rejected=0 gate and an
adversarial over-rejection critic. No behaviour change.
* docs(design): record the empirical finding -- particle-restriction tail is contested
Probing the actual cases overturned the first hypothesis. elemZ026 and
particlesV020 are the same pattern (a derived ref to a substitution-group
MEMBER restricting a base ref to the HEAD), and PureXML already accepts them
because its content model expands a head ref to admit members (for instance
validation), which the restriction oracle inherits -- the Xerces-lenient
reading. They are CONTESTED (Xerces/PureXML accept, XSTS rejects under strict
NameAndTypeOK), so the keystone (adding expansion) would entrench the
acceptance, not close them; closing them risks a false positive. Every
remaining particle case is contested or FP-delicate, so the exact-oracle
keystone yields no clean conformance win. The remaining lever toward the full
bar is the proven resource bound (pushdown matcher), not this oracle. Proof
first earned its keep: it prevented shipping against contested targets.
Generalise the wildcard {namespace constraint} model to the four XSD 1.0 (with
Errata E1-10) expressible forms -- any; a set (with absent); a pair of not and
a namespace name; a pair of not and absent -- replacing the any/##other/set
model that could not represent the union/intersection results. Both not forms
exclude absent, so a union admitting "every namespace except a name, plus
absent" (E1-10 rule 5.1.3) has no expressible form.
A new SchemaWildcardExpressibility compile rule rejects a complexContent
extension whose attribute-wildcard union with its base is not expressible
(src-ct.5), closing XSTS wildZ013 (test328873i). The expressible unions are
computed exactly in XSDComplexContent, so a derived type's wildcard admits the
correct attributes at instance time, closing wildZ013a and wildZ013d.
invalid-schemas-accepted 12 -> 11, invalid-instances-accepted 17 -> 15, with
valid-schemas-rejected and valid-instances-rejected held at 0 (full XSTS
archive). The model is additive for element wildcards (which never produce the
new forms); admits/narrows/UPA-overlap updated and verified unchanged for the
##other cases.
The intersection of two not forms negating different namespace names is not
expressible (src-ct.4), a deferred rule: rather than drop the wildcard and
over-reject valid cross-namespace attributes on an accepted schema, the
intersection stays the less-restrictive not(lhs) over-approximation (FP-safe,
matching prior behaviour). Union/intersection algebra taken clean-room from
Xerces-J XSWildcardDecl. Guarded by SchemaWildcardUnionExpressibilityTests.
#247 rejects an unescaped [ inside a regex character class, which closes the
W3C-settled invalid cases (RegexTest_993/1477) but also rejects siblings the MS
suite marks valid (([[:]+), ([[=]+), ([[.]+), RegexTest_989/990/991/1473-1475,
and the space-bearing subtraction [a - c - [ b ] ]+, 479/480). Those entries
are W3C-queried (status="queried", Bugzilla #4118) because the "valid" marking
contradicts the Appendix F grammar (SingleCharNoEsc excludes [); the suite's
settled position (993/1477) agrees with rejecting, so PureXML follows the
grammar. The queried siblings are excluded from the XSTS conformance gate, so
this is not a counted false positive.
This was shipped in #247 without surfacing the contested nature. Document the
reasoning at the error case and in the changelog so the spec-strict choice is
explicit, not silent. No behaviour change.
XSD charClassSub ::= (posCharGroup | negCharGroup) '-' charClassExpr is the set
difference of the (possibly negated) group and the subtrahend, so for
[^cde-[ag]] negation applies to the base first and the subtrahend is removed
from that result: the class admits everything except {a, c, d, e, g}.
CharClass.matches applied the subtraction to the positive membership and
negated afterwards, so [^cde-[ag]] excluded only {c, d, e} and wrongly matched
a/g, accepting instance values a correct engine rejects. Negation now applies
to the base before subtraction (new result !B && !S is a strict subset of the
old !B || S, matching the Xerces-J reference exactly), so only negated classes
that also carry a subtraction change behaviour.
Closes XSTS RegexTest_430 and RegexTest_422: invalid-instances-accepted 19 ->
17, valid-instances-rejected and valid-schemas-rejected held at 0 (full XSTS
archive). Guarded by RegexCategoryTests.test_negatedSubtraction.
XSD 1.0 charProp is IsCategory | IsBlock, an exhaustive set: every Unicode
general category plus a fixed Unicode-3.1 block list. The block table was
missing the nine supplementary-plane blocks (Old Italic, Gothic, Deseret, the
Byzantine/Musical/Mathematical symbol blocks, CJK Unified Ideographs Extension
B, CJK Compatibility Ideographs Supplement, Tags) and the three surrogate
blocks, so parseCategory could not tell a real-but-untabulated block from a
bogus name and stayed lenient (RegexError.unsupported), accepting the pattern
\p{IsaA0-a9} and the schema declaring it (XSTS reK88).
Complete the table to the full XSD set: supplementary blocks with their ranges,
and surrogate blocks as valid names whose ranges hold no scalar (so they match
nothing). The recognised set is now authoritative, so parseCategory throws the
rejecting RegexError.invalidProperty for any unrecognised name. The now-dead
RegexError.unsupported case and the isMalformedProperty heuristic are removed.
Closes XSTS reK88: invalid-schemas-accepted 13 -> 12, valid-schemas-rejected
held at 0, instance buckets unchanged (full XSTS archive). Side effect: a
pattern naming a supplementary-plane block (e.g. \p{IsMusicalSymbols}) now
matches its characters instead of failing to compile, removing a latent
instance false positive. Block names/ranges taken from the XSD-supported list
in Xerces-J Token.java (clean-room: factual data only). The list is
spec-frozen at Unicode 3.1, so later blocks are rejected by design.
XSD 1.0 Datatypes Appendix F defines SingleCharNoEsc as [^\[\]] and admits a
nested class only as a -[...] subtraction, so a `[` inside a character class
that is not introduced by `-` is a syntax error. RegexParser treated such a
`[` as a literal scalar, so `[a[:xyz:]` compiled and the schema declaring it
was wrongly accepted (XSTS RegexTest_993, RegexTest_1477).
nextClassMember now throws a dedicated RegexError.unescapedClassBracket, kept
distinct from the generic, deliberately-lenient badClass so the change cannot
regress a valid-but-unsupported pattern. XSDSimpleParser.rejectsPattern treats
it as a genuine syntax error, so the declaring schema is invalid.
Closes RegexTest_993 and RegexTest_1477: invalid-schemas-accepted 15 -> 13,
valid-schemas-rejected held at 0, instance buckets unchanged (full XSTS
archive). Escaped [a\[c] and subtraction [a-z-[aeiou]] stay valid; the
unterminated case [a[:xyz: now faults on the inner [.
The streaming-vs-tree differential previously exercised group references only
through the shallow g0/g1 mutual cycle, too weak to gate a future pushdown
content-model matcher whose call/return correctness hinges on deep group
nesting. Add a deep LINEAR group-reference chain (c0 -> ... -> c{chainDepth},
single ref per level so its inlined size is O(depth), not 2^depth) reachable
from any leaf, so the two engines are compared over deep call/return without the
exponential build a doubling chain would trigger. Run divergence-free over the
standing 2000-seed gate and a 60000-seed campaign.
Reconcile docs/release-hurdles.md Part 2 with the deleted positionCap and record
the one remaining structural residual, distinct from occurrence unrolling: the
per-instance NFA build still inlines group references, so a 2^K-inlining schema
makes automaton construction exponential on first instance validation. Document
the pushdown/RTN matcher as the general fix, gated by this hardened differential
(FP-critical: naive sub-model sharing without a stack is unsound).
Compiling a schema with deeply nested, multiply-referenced model groups was
exponential: groupReferenceParticle inlined each <xs:group ref> into the Particle
content model, so g0=(g1,g1), g1=(g2,g2), ... built 2^K particles (depth 14 =
0.21s, depth 20 ~13-18s). The expansion is now memoized per (group definition,
cycle-visiting set) in a per-compile GroupParticleMemo shared across the value
copies XSDContext.scoped/visiting make, so sibling references share it and the
build drops to O(groups) (depth 40 now ~0.016s; previously infeasible).
The shared content term is reused by value (copy-on-write), so the compiled model
is identical and validation is unchanged: the XSTS gate holds (invalid-schemas-
accepted 15, valid-schemas-rejected 0, instance buckets unchanged) and the
streaming-vs-tree fuzz differential still passes. This closes the schema-compile
half of the interactive-safety group blowup (companion to the positionCap
determinism fix); a depth-40 compile test guards against regression. Full suite,
style, namespacing, and WASM all pass.
The Unique Particle Attribution check no longer inlines group references or caps.
A new CompositionalDeterminism engine computes first/followlast over particle
identities and summarizes each xs:group once, so a multiply-referenced nested
group whose inlined Glushkov automaton would be 2^K positions is checked in
O(nodes * particles^2) time. ContentModelDeterminism now delegates to it and the
inlining position automaton plus positionCap are deleted, closing the last named
interactive-safety resource-bound blocker (production-readiness stopper #4) and
the silent under-rejection the cap caused (stopper #3).
The compositional engine was proven verdict-equivalent to the prior inlining
automaton by a differential over the entire XSTS corpus (zero divergences) before
replacing it. The followlast detail was essential: a first attempt that emitted a
repeating end-item's loop-back and the suffix-first as separate decision sets
under-detected 13 XSTS content models; the differential caught every one.
Verification: XSTS gate unchanged (invalid-schemas-accepted 15, valid-schemas-
rejected 0, instance buckets held); SchemaDeterminismGroupBlowupTests compiles a
2^20-position model without hanging and rejects an ambiguous one the old cap would
have skipped (WASI uses a smaller depth, native time limits gated off WASI); full
suite, style, namespacing, and WASM all pass. The per-particle model and QName-only
overlap are preserved, so the false-positive direction stays at zero.
Turns the 'blocked design frontier' for ContentModelDeterminism.positionCap into
a ready-to-implement, FP-safe plan derived from the current engine. Documents the
root cause (group-ref inlining gives 2^K positions), why naive position
dedup/sharing is unsound (cross-context followpos merge = false positive), the
sound compositional algorithm (per-group memoized nullable/firstParticles/
lastParticles/followlast over particle identities, cross-boundary conflicts
computed per reference-site by the parent, the star condition for repetition,
bounded O(nodes x particles^2)), and the differential-gated rollout (build
alongside the inlining oracle, prove verdict-equality on XSTS + a group-ref fuzz,
only then switch and delete the cap) with acceptance criteria.
Production-readiness stopper #3 asks for correctness bounded, not sampled. The
standing fuzz gate keeps a fast, hang-guarded 2000/250/600-iteration sample per
CI run; this adds test_deepFuzzCampaign, which runs the same streaming-vs-tree
differential over PUREXML_FUZZ_ITERATIONS seeds when that variable is set
(skipped otherwise), so a long campaign is reproducible without editing the
bounds and any divergence reproduces from its seed. It carries no .timeLimit (a
60k-seed run legitimately exceeds a minute); termination stays guaranteed by the
engine's own occurrence and state caps. A 60,000-seed campaign (plus 20,000
schema/instance and 80,000 parser-byte iterations) found zero divergences,
crashes, or hangs.
cos-ct-restricts 5.2: a complexContent restriction's content type is element-
only/mixed/empty (from its model group), which can never validly restrict a
simple content type. complexContentBaseKindFindings only flagged a simple-type
base; a complex type whose content is simpleContent slipped through. The check
is added for the restriction direction only (the extension direction has a valid
attribute-only idiom, handled by simpleContentExtensionBaseFindings). Catches
particlesZ039. XSTS invalid-schemas-accepted 16 to 15, valid-schemas-rejected
held at 0.
XSD 1.0 §3.9.6 Particle Valid (Restriction) defines a case for an element
restricting a base group (RecurseAsIfGroup) but none for the reverse, so a
derived sequence/choice/all that can contribute a child never validly restricts
a single base element. The oracle had a deliberate non-spec 'mirror' extension
that bounded the derived group by the element's effective occurrence range and
accepted it; that masked a real violation inside a choice-to-choice restriction,
where a derived sequence branch was allowed to map onto a base element branch it
cannot legally restrict. The (.group, .element) case now returns invalid; a
content-free derived group is still a valid restriction of any emptiable base
(handled before this case). Catches particlesHb011. XSTS invalid-schemas-
accepted 17 to 16, valid-schemas-rejected held at 0 (differential agree-valid
unchanged).
cvc-id: when an instance omits an attribute, its default/fixed value applies, so
a defaulted or fixed xs:IDREF/xs:IDREFS must still resolve to a matching xs:ID.
The reference was recorded only for attributes physically present, so a dangling
defaulted IDREF was wrongly accepted. The supplied value is now recorded for
document-scoped ID/IDREF checking exactly as a present value would be (shared by
the tree and streaming validators, which both call validateAttributes). A
required absent attribute is still reported as missing, recordIDs is a no-op for
non-ID types, and a defaulted IDREF that does resolve stays valid. Catches
idZ012. XSTS invalid-instances-accepted 20 to 19, valid-instances-rejected held
at 0.
An xs:all group's own minOccurs="0" makes the GROUP optional (it occurs zero
or one times), not each member optional. The parser forced every member's
minOccurs to 0 whenever the all-group was optional, so a non-empty but
incomplete optional all-group (a required member missing) was wrongly accepted,
and a complexContent restriction that relaxed such a required member also slipped
past the particle-restriction check (the zeroed minOccurs made it look already
optional). Members now keep their declared occurrence; the absent-group case is
handled at validation time by skipping the required-member check only when the
all-group contributes no children at all. Applied identically on the tree and
streaming paths (both call allStructureErrors). A required all-group is
unchanged. Catches mgZ001 (instance) and particlesK006 (schema). XSTS invalid-
instances-accepted 21 to 20 and invalid-schemas-accepted 18 to 17, both valid-
rejected buckets held at 0.
cvc-elt.3.2.2: an element nilled by xsi:nil="true" may not have a fixed
{value constraint} (a nilled element has no value, so it cannot take a fixed
one). A nillable element MAY declare a fixed value, so the schema stays valid;
the conflict is an instance error only when such an element is actually nilled.
The check rides the shared nilErrors chokepoint, so the tree and streaming
validators enforce it identically; it prefers the matched particle's constraint
and falls back to the by-local-name map for particle-less paths (the unique-
named root). A nillable element that supplies its fixed value, a nilled element
with no fixed constraint, and a nilled element with a default (not fixed) value
are all unaffected. Catches addB065. XSTS invalid-instances-accepted 22 to 21,
valid-instances-rejected held at 0.
The top-level declaration constraints (a global element/attribute may not carry
the local-only form/use/ref/minOccurs/maxOccurs; a global definition needs a
name) apply to every document in a schema composition, but the existing check
ran only over the document being compiled. A violation inside an xs:include,
xs:import, or xs:redefine target was therefore missed and the composed schema
wrongly accepted. The same global-component check now runs over the other
schema-document roots in the composition (the main document is excluded as
already covered; redefine entries are skipped because each schema root walks its
own redefine children). The constraint is absolute and position-independent, so
no valid schema is newly rejected; libxml2 rejects both cases for the same
reason. Catches attQ016 (import) and attQ018 (include). XSTS invalid-schemas-
accepted 20 to 18, valid-schemas-rejected held at 0.
Exercise the byte-at-a-time ISO-2022-JP mode escapes (ASCII, JIS-Roman,
half-width katakana, JIS X 0208) and EUC-TW plane-1 plus the 0x8E four-byte
selector. Raises streaming ISO-2022-JP coverage from 54% to 92% and EUC-TW
from 52% to 76%. Output is compared by Unicode scalar to avoid a grapheme
cluster swallowing the trailing sound mark.
The coverage analysis noted there is no mechanism that measures test completeness,
so a source file silently sitting at 0% (untested or dead) is invisible.
Add scripts/check-coverage.sh: it runs the instrumented suite, prints overall
line/region/function coverage and the lowest-covered files, and FAILS if any source
file is at 0% line coverage (the exact signal that found the earlier gaps). It is
opt-in (instrumented build + llvm-cov are slow and macOS-only) and skips cleanly
where xcrun is unavailable; an optional COVERAGE_REGION_FLOOR enforces a floor.
Running it immediately surfaced two more 0% value types, now closed by
CoverageGateValueTypeTests: the HTML tokenizer's `Token` (a custom Equatable) and
the public `Stream.Document` (built by API consumers, never internally). Overall
line coverage is now 92.66% with no source file at 0%, so the gate passes.
Tooling + tests only; +2 tests; full suite green (1570).
A code-coverage pass found two source files at 0% in the standing suite (covered
only by opt-in gates) and one large under-covered file:
- XSDParserUnion.swift (Document(composing:), the multi-document union, #161) was
exercised only by the opt-in XSTS harness, so plain `swift test` never ran
multi-document composition. SchemaComposingTests now covers it via a
cross-document substitution group (0% to 92% line).
- XSLTCollation.swift (language-tailored xsl:sort orders) was unexercised by any
standing test. XSLTCollationTests covers the Polish/Russian tables and the
fallback (0% to 88% line).
- StreamingDecoder.swift (byte-at-a-time CJK/UTF decode) had the streaming/split
path under-tested. StreamingDecoderTests feeds bytes one at a time through GBK,
Big5, Shift_JIS, EUC-KR, EUC-JP, GB18030, the UTF-16/32 BOM paths, and a
malformed sequence (48% to 80% line, 37% to 72% region). Expected scalars were
taken from the decoder and pinned.
Test-only; +10 tests; full suite green (1568).
SchemaStructure.swift was the last source file at the 400-line cap. The
cos-element-consistent check (a complex type's content model may not declare two
same-named elements with different types) is a self-contained constraint with its
own helpers (collectElementDecls, elementTypeKey) and the contentModelContainers
set used only by it.
Move that cluster into SchemaElementDeclarationsConsistent.swift;
elementDeclsConsistentErrors becomes internal (its sole caller is in
SchemaStructure). Pure code move, no behaviour change: SchemaStructure 400 -> 342,
new file 59. With this, no source file remains at the 400-line cap. Full suite and
the XSTS gate pass unchanged.
SchemaRedefineSelfReference.swift had reached the 400-line limit, blocking further
redefine-restriction work without a squeeze. It also mixed three cohesive concerns:
redefine self-reference well-formedness (src-redefine.6/7.2.1), attribute-group
restriction (cos-ct-restricts over the redefinition), and model-group restriction.
Move the attribute-group restriction machinery (redefineAttributeGroupRestrictionFindings
and its helpers) into SchemaRedefineAttributeGroupRestriction.swift, following the
established one-concern-per-file pattern (IdentityFieldType.swift). The shared base-
component lookups (baseComponents/resolveBaseComponent) become internal so both files
use them. Pure code move, no behaviour change: 171 + 232 lines (was 400); full suite
and the XSTS gate pass unchanged (20/0/22/0 buckets, 1558 unit tests).
The XSTSSuiteTests doc comment had grown a ~120-line per-step burn-down narrative
that duplicated CHANGELOG.md (every entry already said "per-step deltas in
CHANGELOG.md") and kept the file pinned at the 400-line limit, so each conformance
iteration had to hand-squeeze it to stay under the cap.
Replace the narrative with a concise pointer to CHANGELOG.md and
docs/xsts-deviations.md (their canonical home); keep the baseline explanation, the
run instructions, the pinned-count constants, and the specDivergentInstances
exclusion unchanged. File 400 -> 286 lines; the gate's behaviour and pinned counts
are unchanged (verified: XSTSSuiteTests still passes at 20/0/0/22).
cvc-identity-constraint.3 / c-fields-xpaths require an identity-constraint field to
identify a node with a simple type. The prior fix rejected a field whose selected
instance node carries element children (structural). This adds the case where the
field's DECLARED type is a complex type with non-simple content (empty or
element-only/mixed): an attributes-only element resolves to an empty-content complex
type, so its instance node is childless yet not simple-typed.
The field's target type is resolved through the schema type model (the selector
target's declared type, then the field's same-named child element in its content
model) and classified. A simple or simpleContent target, a type reached only
through a named reference, a complex field path, or an unresolvable target is left
alone, so no valid field is rejected. The resolution helpers live in a separate
extension file to keep the validator within the type-body limit.
This catches W3C XSTS idG006. XSTS invalid-instances-accepted 23 -> 22,
valid-instances-rejected held at 0.
The substitution-group filter drops a member when the head element's
{disallowed substitutions} (or the head type's {prohibited substitutions}) blocks
the method by which the member's type derives from the head's type. It looked the
member's type up in the named derivation backbone, so a member with an INLINE
(anonymous) complex type had no entry and was never dropped, even when its inline
type extended the head's type and the head blocked extension.
Record an inline-typed element's derivation under a synthetic key (the element key
plus a `#` marker no QName can contain, so it cannot collide with a real type and
is unreachable as any other type's derivation base) so the filter can see the
member's method. A member reaching the head's type by an unblocked method, a plain
inline type with no derivation, and same-local-name elements in different
namespaces are all unaffected.
This catches W3C XSTS disallowedSubst00105m (and a sibling). XSTS
invalid-instances-accepted 25 -> 23, valid-instances-rejected held at 0.
Per cvc-identity-constraint.3 / c-fields-xpaths an identity-constraint
(key/unique/keyref) field must evaluate to a node with a simple type: a
simple-typed element, an element with simple content, or an attribute. The
field-cardinality clause (at most one node) was enforced, but the simple-content
clause was not, so a field selecting a complex-content element was wrongly
accepted at instance-validation time.
Add a structural, false-positive-free check: a field whose selected node is an
element carrying element children has complex (element-only or mixed) content and
can never be simple-typed, so it is flagged. A simple-typed or simple-content
element (text only), an empty or text-only node, and an attribute are left alone,
so no valid field is rejected.
This catches W3C XSTS idK012. XSTS invalid-instances-accepted 26 -> 25,
valid-instances-rejected held at 0.
The attribute-use uniqueness check (ag-props-correct.2 / ct-props-correct.4)
flattens a complex type's or attribute group's uses through attributeGroup
references. A reference to a redefined group whose redefinition self-references its
former self stopped at the cycle guard, so the original's attributes were never
reached and a duplicate they cause was missed.
Expand a redefinition's self-reference to the group's pre-redefine base definition
(already retained in the context for this purpose) so the original's attributes
join the set; a genuine non-redefine cycle still terminates without contributing.
Separately, exclude use="prohibited" attributes from the duplicate-name and
single-ID counts: a prohibited use is a removal marker, not a member of
{attribute uses} (XSD 3.4.2 / 3.6.2). This is the spec-correct reading and keeps a
self-reference that re-imports an attribute and then prohibits it from being
counted as a duplicate.
This catches W3C XSTS attQ011 and attQ017. XSTS schema invalid-accepted 22 -> 20,
valid-rejected held at 0.
A redefinition of an attributeGroup that self-references the original re-imports
all of its attribute uses, so directly re-declaring one of those attributes
produces two attribute uses of the same name, a duplicate forbidden by
ag-props-correct.2.
Flag a non-prohibited direct re-declaration of an attribute the original directly
declares, when the redefinition has a self-reference. To stay false-positive-free
the check fires only when both the original and the re-declaration land in no
namespace (a qualified attribute's target namespace depends on cross-document
form/chameleon resolution, so it is not compared); a self-referencing redefinition
that adds genuinely new attributes (attgC007/attgC038/groupB018) stays valid.
Also extracts the use-transition restriction errors into a helper to keep the
function within the complexity limit.
This catches W3C XSTS schM8. XSTS schema invalid-accepted 23 -> 22, valid-rejected
held at 0.
A redefinition of an attributeGroup is a restriction of the original
(src-redefine / cos-ct-restricts), so a re-declared attribute's type must derive
by restriction from the original's. The attribute-restriction checks covered
drop-required, re-introduce-prohibited, add-attribute, and relax-fixed, but not a
type change that widens rather than restricts.
Add a built-in-only, namespace-gated check: it fires only when both the original
and the re-declared attribute name a built-in type resolving to the XSD namespace
and the re-declared built-in does not derive from the original's in the XSD
derivation lattice (int does not derive from boolean). A user, inline, or absent
type on either side is not compared, so the check declines and stays lenient: an
absent original type is anySimpleType (which any type restricts), a user/inline
type's derivation is not resolvable here, and a user type whose local name
coincides with a built-in resolves to a different namespace and is not mistaken
for the built-in.
This catches W3C XSTS schM4 (a redefinition re-typing an attribute from boolean
to int). XSTS schema invalid-accepted 24 -> 23, valid-rejected held at 0.
A redefinition of an attributeGroup is a restriction of the original
(src-redefine / cos-ct-restricts), so it may not introduce an attribute the
original lacks unless the original carries an attribute wildcard. The existing
add-attribute check resolved only the redefinition's DIRECT declarations and was
suppressed whenever the redefinition carried any attributeGroup reference, which
conflated a self-reference (re-importing the original's own attributes, valid)
with a non-self reference (injecting foreign attributes, invalid).
Resolve the attribute names a non-self reference contributes and key the
suppression on a genuine self-reference (matched by namespace and local name).
The resolution declines, staying lenient, whenever the contributed set is not
fully knowable: an unresolved group, an attribute-by-ref, an attribute wildcard,
or a further nested reference. This catches W3C XSTS attgC028 (a redefinition
whose ref="car" replaces the original's attributes with foreign ones) while
keeping attgC007/attgC038/groupB018 (added attributes alongside a self-reference)
valid.
XSTS schema invalid-accepted 25 -> 24, valid-rejected held at 0.
isUndeclaredReferenceType matched a no-namespace type/base/itemType/memberTypes
reference by local name, ignoring that an unprefixed QName with no default
xmlns resolves to the absent namespace: type="t" is {}t, not {urn:x}t where the
type is declared, so the reference does not resolve (XSD 1.0 §3.15.3). It now
mirrors isUndeclaredReferenceRef's nil-namespace branch: a no-namespace
reference is undeclared when the schema has a non-empty target namespace; a
chameleon-included global and an imported no-namespace type (via the foreign
pools, checked first) remain accepted.
This corrects an asymmetry where the ref side already rejected the pattern but
the type side did not. The existing SchemaXSIAttributeTests case that asserted
this very pattern compiles used an invalid schema (the same shape as XSTS
addB009/test66059); its type reference is now qualified (tns:attRef), which
preserves the test's xsi:Not-Allowed intent while being valid.
Verified against the full W3C XSTS 2006-11-06 suite: invalid-schemas-accepted
27 -> 25 (addB009/test66059 and one more), valid-schemas-rejected held at 0,
no other bucket moved. Adds a reference test for the type-side rule.
isUndeclaredReferenceRef fell through to a lenient accept when a ref or
substitutionGroup resolved to the XSD namespace but the schema did not target
it. The XSD namespace declares no referenceable user component (element,
group, attribute, attribute group), so such a reference is undeclared, the
asymmetry isUndeclaredReferenceType already handled for type/base.
This is reached when an unprefixed reference picks up a default xmlns of the
XSD namespace, so it is not in the target namespace where the schema's own
components live: xmlns="http://www.w3.org/2001/XMLSchema" targetNamespace="foo"
with <element ref="root"/> resolves to {XSD}root, not {foo}root (XSTS
xsd013/xsd014). A schema that itself targets the XSD namespace is the
inTargetNamespace case, and a genuinely loaded XSD-namespace document is the
foreignPools case before this branch, so both are unaffected.
Verified against the full W3C XSTS 2006-11-06 suite: invalid-schemas-accepted
29 -> 27, valid-schemas-rejected held at 0, no other bucket moved. Adds
reference tests for both reject cases and a qualified-ref guard.
Unique Particle Attribution checked only a complexContent extension's own
particles, missing the cross-boundary non-determinism with its base: the
effective content of an extension is the base type's content followed by the
extension's, so a base particle that can recur or end optionally competes
with the extension's leading particles. An element matching either was a UPA
violation the single-node check did not see, and the schema was wrongly
accepted (XSTS particlesZ022).
Build the content model over the effective sequence (base content prepended,
resolved recursively). The base is followed only for a local
(target-namespace), non-cyclic base chain; a foreign, built-in (xs:anyType),
unresolved, or cyclic base leaves the check on the extension's own content, so
no model that cannot be fully assembled is rejected (an under-rejection, never
a false positive). For a non-extension the effective list is a single node, so
sequence([x]) reproduces the prior automaton exactly.
Verified against the full W3C XSTS 2006-11-06 suite: invalid-schemas-accepted
31 -> 29 (particlesZ022 and one more), valid-schemas-rejected held at 0, no
other bucket moved. Adds UPA tests for an ambiguous extension and a
deterministic one (the over-rejection guard); moves the content-model-node and
effective-model helpers to a new file to stay within the type-body and
file-length budgets.
Unique Particle Attribution missed overlapping xs:any wildcards involving
##other: wildcardsOverlap returned false for any pair touching .other, so two
competing wildcards whose namespaces actually intersect were not flagged as
non-deterministic and the schema was wrongly accepted.
Compute the overlap correctly: ##other against ##other always overlaps (both
admit every namespace but their target and the absent one, infinite sets that
always intersect), and ##other against an enumerated set overlaps iff the set
names a namespace that is neither absent (##local) nor the target. The new
predicate matches Wildcard.admits exactly, so disjoint cases (##other vs
##local, or vs the target namespace) stay deterministic and no valid schema
is newly rejected.
Verified against the full W3C XSTS 2006-11-06 suite: invalid-schemas-accepted
35 -> 31 (wildI009/wildI013/wildI014 and one more), valid-schemas-rejected
held at 0 (no other bucket moved). Adds UPA unit tests for both rejection
shapes and a ##other-vs-##local determinism guard; updates the pinned XSTS
count 35 -> 31.
Convert the five composition/redefine compile checkers to located findings,
the last of the #169 burndown (#210-#217): a resolved schemaLocation whose
content is not a schema underlines the include/import/redefine reference, a
redefined type that does not restrict or extend itself underlines the
redefined type, a redefined group or attribute group with a malformed
self-reference underlines the offending reference or redefinition, and a
redefined attribute group or model group that does not restrict its original
underlines the redefinition.
To locate the referenced-schema rule, failedSchemaReferences carries the
referencing node, captured at the load-failure site in XSDContext. Only a
top-level reference's node belongs to the main schema tree the diagnostic
resolves against; a failure nested in a loaded sub-document is captured from
a throwaway parse, so it stays unlocated (the prior schema-root behavior)
rather than mis-resolve, a strict improvement.
Every predicate, aggregation order, and reason string is unchanged; only the
location each finding carries differs, verified conformance-neutral by the
full XSTS suite (14383 groups, pinned counts unchanged). The redefine
checkers had exactly one caller each, so each is renamed in place and the
.unlocated wrap dropped. Adds a located-diagnostics test per node-locatable
checker; the attribute-group and model-group restriction redefine checkers
share the same redefinition-node idiom and are exercised by the XSTS suite.
Convert the three names/UPA/notation-pass compile checkers to located
findings so each error attaches to its declaring node instead of the schema
root (#169): component-name uniqueness underlines the second duplicate global,
identity-constraint name uniqueness underlines the second constraint, a
keyref whose refer names no key/unique (or whose arity differs) underlines
the keyref (KeyrefInfo gains the keyref node), a Unique Particle Attribution
violation underlines the ambiguous content-model group, and notation
validity underlines the notation or enumeration.
Every predicate, aggregation order, reason string, and the message-dedup of
the UPA pass are unchanged; only the location each finding carries differs,
verified conformance-neutral by the full XSTS suite (14383 groups, pinned
counts unchanged). The findUniqueIndices circular-import (rootIndex)
workaround is left byte-identical, since it governs which containers are
examined, not where findings locate. The three checkers had exactly one
caller each (their dispatch), so each is renamed in place and the .unlocated
wrap dropped. Adds a located-diagnostics test per sub-check.
Convert the three extension/substitution-pass compile checkers to located
findings so each error attaches to its declaring node instead of the schema
root (#169): an xs:all group reached through an extension over a base that
already has content underlines the extension (extensionAllGroupFindings); an
anonymous complex-type restriction that is not a subset of its base
underlines the restriction (anonymousRestrictionFindings); and a
substitution-group member whose type is not validly derived from its head,
including the head's final-exclusion violations, underlines the member
element (substitutionTypeFindings).
Every predicate, iteration order, and reason string is unchanged; only the
location each finding carries differs, verified conformance-neutral by the
full XSTS suite (14383 groups, pinned counts unchanged). The
substitution checker's checkExclusions helper stays [String] and is wrapped
onto the member element at the call site. The three checkers had exactly one
caller each (their dispatch), so each is renamed in place and the .unlocated
wrap dropped. Adds a located-diagnostics test per checker.
Close the last superficially-located checker (#169): referenceFindings was
dispatched as located but its body wrapped the whole aggregate in
.unlocated, so every reference-resolution error still collapsed onto the
schema root. Convert all six sub-collectors to located findings, each
attaching to the node in scope: the core type/base/itemType/memberTypes/ref/
substitutionGroup walk and the xs:-namespace builtin check locate on the
referencing element; the simpleContent base-kind check on the restriction;
the un-imported-namespace check on the referencing element; the
redefine-component-existence check on the redefine's child component; and
the import-namespace-agreement check on the import element.
Every predicate, aggregation order, and reason string is unchanged; only the
location each finding carries differs, verified conformance-neutral by the
full XSTS suite (14383 groups, pinned counts unchanged). Removes two dead
helpers exposed along the way: the unused referenceErrors [String] wrapper
and the unused locatedFindings helper in SchemaCompileValidation. The outer
aggregation is rewritten from a single + chain to incremental appends to keep
the type-checker within budget. Adds a located-diagnostics test per
node-locatable sub-collector; the loader-dependent redefine/import families
are exercised by the XSTS suite.
Convert the seven checkers the complexExtensionBaseValid compile rule
dispatches (the cos-ct-extends/src-ct content-derivation family) to located
findings so each error attaches to its derivation node instead of the schema
root (#169): a complexContent extension adding element content over a
simpleContent base, a simpleContent restriction whose base is a built-in or
user simple type, an unfaithful inline type in a simpleContent restriction, a
complexContent base that is a simple type, a simpleContent extension whose
base is xs:anyType, and a mismatched mixed setting on a complexContent
extension each underline their extension/restriction node; an element
carrying a default/fixed value over element-only or empty content underlines
the element.
Each predicate, iteration order, and reason string is unchanged; only the
location each finding carries differs. The seven checkers had exactly one
caller each (the dispatch), so each is renamed in place and the .unlocated
wrap dropped. To keep the file under the 400-line limit, the seven functions
and their two private helpers move to a new SchemaContentDerivation.swift;
the shared extensionAddsElementContent helper, still used by the all-group
checker that stays behind, becomes module-internal. Adds a located-
diagnostics test per checker.
Convert the three derivation/cycle-pass compile checkers to located findings
so each error attaches to its declaring node instead of the schema root
(#169): an all-group referenced inside a sequence/choice/all locates on the
offending group reference (allGroupReferencePlacementFindings); a type
derived from itself locates on its complexType/simpleType
(derivationCycleFindings); and an element that is a member of its own
substitution group, or a model group or attribute group that references
itself, locates on its declaration (circularReferenceFindings).
The cycle checkers report a cycle by component name over a name-keyed graph,
so each now records the declaring node in a parallel nodeOf map built in the
same loop as the graph; a name that yields a finding always has a node, so a
mutual cycle underlines each member on its own declaration. Every predicate,
key ordering, and reason string is unchanged; only the location each finding
carries differs. The three checkers had exactly one caller each, so each is
renamed in place and the .unlocated wrap dropped. Adds a located-diagnostics
test per checker family, covering both substitution and group-reference
cycles.
Convert the three attribute-pass compile checkers to located findings so
each error attaches to its declaring node instead of the schema root (#169):
an attribute declaration in the XSI namespace locates on the attribute
(xsiNamespaceAttributeFindings); a duplicate attribute use or a second
ID-typed attribute locates on its declaring complexType or attributeGroup,
and an attribute reference contradicting a global fixed value locates on the
reference (attributeUseFindings); and a complex-type restriction that
relaxes a base attribute use or widens the attribute wildcard locates on the
restriction (attributeRestrictionFindings).
Each predicate, traversal order, and reason string is unchanged; only the
location each finding carries differs. The set-level uniqueness/single-ID
findings reuse the existing attributeSetErrors helper, mapping each onto its
declaring container node. An attribute restriction violation has no source
node of its own (the attribute use is a resolved model value), so it locates
on the restriction performing the illegal narrowing. The three checkers had
exactly one caller each, so each is renamed in place and the .unlocated wrap
dropped. Adds a located-diagnostics test per checker family.
Convert the four simple-type-pass compile checkers to located findings so
each error attaches to its own declaring node instead of the schema root
(#169): a list whose item type is itself a list or a union containing a
list and a list declaring both or neither of itemType/inline simpleType
(simpleTypeVarietyFindings), an empty union, a simpleType restriction whose
base resolves to a complex type (simpleTypeBaseNotComplexFindings), an
itemType/memberType referencing a type final for list/union derivation
(simpleTypeFinalFindings), and an attribute typed by a complex type
(attributeTypeMustBeSimpleFindings).
Each predicate, traversal order, and reason string is unchanged; only the
location each finding carries differs, from the schema root to the
offending list, union, restriction, or attribute node. The four checkers
had exactly one caller each (their compile-rule dispatch), so each is
renamed in place and the .unlocated wrap dropped, leaving no dead wrapper.
Adds a located-diagnostics test per checker in a new suite kept within the
type-body budget.
Convert the seven structure-pass compile checkers to located findings so
each error attaches to its own declaring node instead of the schema root
(#169): a facet inapplicable to a list/union variety, a restriction of
xs:anySimpleType and a constraining facet on anySimpleType-based simple
content, a top-level definition missing its name or carrying a local-only
attribute, a nested named definition, an empty targetNamespace or import
namespace, and an all-group reference with an illegal minOccurs/maxOccurs.
The rewire in structureFindings preserves the prior finding order and
count; each error predicate and reason string is unchanged, only the
location each finding carries differs. Adds a located-diagnostics test
per checker family in a new suite kept within the type-body budget.
Convert the three value-constraint checkers (built-in, named user type,
and inline type) to located findings so a default/fixed value that is
invalid for its declared type, or an illegal combination of value
constraints, is attached to its declaring element or attribute instead
of the schema root. An editor can underline the offending declaration
by line and column.
Error predicates and reason strings are unchanged; only the location
each finding carries differs. Adds located-diagnostics tests for the
built-in, user-type, and inline-type paths.
Convert idValueConstraintErrors to idValueConstraintFindings so each
a-props-correct.3 / e-props-correct.5 violation is attached to its
declaring element or attribute node, letting an editor underline the
offending declaration by line and column instead of the schema root.
The error predicate and reason strings are unchanged: only where each
error points differs. Adds a located-diagnostics test asserting the
finding resolves to the declaring component's source range.
The counted content-model automaton represents its active set as
(state, counter-vector) configurations bounded by states times the product of
the per-counter saturation caps. An unbounded counter already saturates at its
minimum (every higher count is matching-equivalent). A finite maximum the input
cannot reach is equivalent: the active set increments a counter at most to its
cap, the cap is at most inputLength+1, so when maximum > inputLength+1 the value
can never reach maximum, belowMaximum is always satisfied, and every value at or
above the minimum is indistinguishable. Saturate such a counter at its minimum
too, so a huge but finite maxOccurs on a nested counter no longer multiplies the
active set by an inputLength factor per nesting level.
A document that did not terminate within a minute against the Z035 worst case
(nested sequences/choices with maxOccurs up to 1e11) now validates in linear
time. No accept/reject decision changes: the full XSTS suite holds at
0/35/0/26, and an adversarial no-caps reference-matcher differential found no
false positive or false negative, including the nullable-body case (where the
builder makes canExit unconditional, so the counter value is matching-irrelevant)
and the occurrence boundary. Guarded by ContentMatcherWorstCaseTests.
Partial: a model with several moderate (input-reachable) maxOccurs nested deeply
still enumerates a large active set; the general polynomial bound (a position-
indexed DP over the counting automaton) remains under #192.
The streaming XSD validator is a separate implementation of the same semantics
as the tree validator (the conformance oracle), and three false positives in
it were found and fixed this cycle (#186, #200, #202) only by adversarial
hunting, because the XSTS gate exercises the tree path, not validate(streaming:).
A snapshot is not an invariant.
Add test_streamingMatchesTreeDifferential to the generative fuzz suite: 2000
deterministic schema/instance pairs (15 on WASI), asserting that streaming and
the tree validator agree on whether each document is valid (so streaming never
rejects a valid document and never accepts an invalid one the tree rejects),
and that the tree's errors are a subset of streaming's (streaming may add
errors on an already-invalid document, #206, but may never miss a real error
or invent a rejection). A failure prints the seed, schema, instance, and both
error sets, so it reproduces exactly.
This runs in the standard suite on all five platforms, so a future streaming
false positive fails CI rather than reaching users.
XSD 1.0 cos-all-limited: a reference to a model group whose content is an
`all` group is itself an all-group particle, so its maxOccurs must be 1 and
its minOccurs 0 or 1. The direct `<all>` element was already checked by
allGroupLimitedErrors, but a `<group ref>` resolving to an all group was not,
so corpus particlesEa025 (`<group ref="x:G" maxOccurs="2">` to an all group)
was accepted.
Add allGroupReferenceMaxOccursErrors: collect the names of group definitions
whose content is an all group, then flag a group reference to one whose
maxOccurs is not 1 or whose minOccurs is not 0/1. The check runs over a single
schema document tree, where group names are unique, so the by-local-name match
cannot collide; a cross-document reference lives in a separate container and is
under-rejected (never falsely rejected).
XSTS invalid-schemas-accepted 36 -> 35; valid-schemas-rejected held at 0.
XSD 1.0 cos-particle-restrict NameAndTypeOK.2.2: when a complexContent
restriction narrows an element, the element's type must be validly derived
from the base element's type EXCLUDING extension (and list/union), i.e. by
restriction only. The element-type check followed the derivation chain
regardless of method, so a type derived from the base by extension was wrongly
accepted as a valid restriction (corpus particlesIj008).
Add typeDerivesByRestriction (as typeDerivesOrEqual, but the user-declared
chain is followed across restriction steps only; an extension step breaks the
walk) and use it on the non-union fall-through of elementTypeRestrictionOK.
The shared typeDerivesOrEqual is untouched, so substitution-group validity
(where extension is legal) is unaffected. Built-in lattice and union-base
derivation are unchanged, so valid restrictions, including the union-member
restriction addB150, still compile.
XSTS invalid-schemas-accepted 37 -> 36; valid-schemas-rejected held at 0
across all 14383 groups. Union-base (Ik011/Ik025/Ik027) and substitution-group
(V020) cases need the by-name union-member and ref-resolution clauses and stay
deferred.
A fixed value on an element whose type is a complex type with simpleContent
was compared as a raw string, so value-space-equal content was wrongly
rejected (e.g. "05" against an xs:int fixed "5"). The complex-element fixed
check on both the tree path (validate(_:against:)) and the streaming path
passed no value type to elementFixedErrors, falling through to a string
compare; both now pass the simpleContent simple type as the value type, in
lockstep, so the comparison is in value space (cvc-elt.5.2.2.1) as the
simple-typed-element path already does.
XSTS gate held at 0/37/0/26 across all 14383 groups.
The streaming XSD validator ignored an element's default/fixed value constraint
in simple-content validation, so an empty element with a value constraint was
validated against the empty string (a false positive), and a present fixed
value was compared as a raw string rather than in the type's value space.
Mirror the tree path's validateChild for the simple-typed element:
- an empty element takes its constraint value via a shared effectiveSimpleText
helper both paths use; the constraint comes from the matched particle (keyed
by namespaced identity), not the flat by-local-name elementConstraints map,
so same-local-name elements in different types do not collide.
- a present fixed value is compared in the type's value space and against the
matched particle's fixed value (elementFixedErrors with valueType and
particleFixed), not a raw string compare ("05" equals int fixed "5").
The particle constraint is looked up from the parent's content model at
element-close time. Streaming now agrees with the tree-path oracle on value
constraints, verified by an adversarial differential (13 configurations).
No XSTS bucket changes; gate held at 0/37/0/26.
The streaming XSD validator resolved element and xsi:type references by bare
local name, while its type tables are keyed by namespaced identity
({ns}local). A prefixed xsi:type in an imported namespace missed (a false
"unknown xsi:type") or hit a same-local-name type in the wrong namespace, and
two global elements sharing a local name across namespaces collided.
Resolve through in-scope prefix bindings accumulated down the element stack,
keyed by namespaced identity, mirroring the tree path:
- the xsi:type override type, the undeclared-type check, and root/child
element declarations are namespaced; root resolution mirrors the tree's
globalElementDeclaration (bare local-name map only for a no-namespace root).
- run the full cvc-elt.4 substitution gate the tree path runs (blocked /
abstract / not-validly-derived / list-or-union / anyType-for-anySimpleType
override, and an abstract declared type carrying no xsi:type).
- resolve a type from an element's own xsi:type where there is no governing
declaration: an undeclared root (the Sun target-namespace fallback) and a
strict-wildcard child, both via a shared xsiTypeNamedType helper.
Streaming now agrees with the tree-path oracle across these, verified by an
adversarial streaming-vs-tree differential. No XSTS bucket changes (the tree
path is the conformance oracle); gate held at 0/37/0/26.
XSD cos-ct-restricts restriction mixedness is one-directional: a restriction
may keep mixedness or narrow mixed to element-only, but must not add mixed
content to an element-only base. The two #184 targets are already rejected
(ctZ010d by extensionMixedAgreementErrors, ctZ010e by ParticleRestriction's
mixed-vs-element-only case) and the four guard schemas already compile, so no
behavior change is needed. This adds the isolated regression pin the
restriction half lacked, guarding against a bidirectional "derived mixed must
equal base mixed" rule, which regressed four valid schemas when last tried.
XSD 1.0 Schema Component Constraint "xsi: Not Allowed" (§3.2.6): an attribute
declaration's {target namespace} must not be the XSI namespace. A schema may
target XSI and declare unqualified attributes (they land in no namespace),
but a top-level attribute, or a local one made qualified by form="qualified"
or attributeFormDefault="qualified", lands in XSI and is forbidden.
Add xsiNamespaceAttributeErrors, scoped to schemas whose target namespace is
XSI, registered as the pre-compile rule attributeDeclarationsNotInXSI. It
rejects corpus attKb018a (qualified attribute-group attributes) and attKa015
(top-level attributes), while attKb018 and attKc018 (local unqualified) stay
valid.
XSTS invalid-schemas-accepted 39 -> 37; valid-schemas-rejected held at 0; no
other bucket moved.
Add the #195 xsi:type/substitution derivation epic to the README roadmap
diagram (the epic-coverage gate requires every epic issue to appear), and
update the settled XSTS counts to 0/39/0/26.
On the cross-document skip path (a schema declares import/include/redefine
but no external document was loaded), the reference check stood down whole,
so a prefixed reference whose namespace is never imported was accepted even
though it can resolve to no component under any loading (corpus schZ011's
d:d, where namespaces a/b/c are imported but d is not).
Add undeclaredNamespaceReferenceErrors, run only on that skip path, to flag
a reference whose namespace is neither the target, the XSD or XML namespace,
nor named by any xs:import. To stay sound when the import closure is unknown,
it stands down entirely the moment any external carries a schemaLocation:
that document may transitively import the referenced namespace (corpus
schZ004 imports only a, yet a's document imports b, so b:b resolves). It
applies only when every external is location-less and the closure is
provably complete.
XSTS invalid-schemas-accepted 43 -> 39 (schZ011_a-d); valid-schemas-rejected
held at 0; no other bucket moved.
cvc-elt.3.1: when an element's declaration is not nillable, the xsi:nil attribute
is forbidden entirely, whatever its value. PureXML only rejected xsi:nil="true" on
a non-nillable element, so xsi:nil="false" slipped through. ComplexValidator.nilErrors
now flags the presence of any xsi:nil on a non-nillable element via the new
nilAttributeValue helper, while a nillable element still accepts xsi:nil="false"
(explicitly not nilled, valid empty-or-content per its type).
XSTS invalid-instances-accepted 28 -> 26 (both xsi:nil-on-non-nillable cases), no
other bucket moved (valid-schemas-rejected and valid-instances-rejected held at 0
across all 14383 groups). 1452 unit tests pass.
XSD 1.0 3.3.2: an element with neither a `type` attribute nor an inline type, but
a `substitutionGroup`, takes the {type definition} of its head. PureXML defaulted
such an element to `anyType`, so an instance of it accepted any content.
`XSDParser.elementType` now resolves the head declaration through
`context.globalElements` (transitively, with a cycle guard) and uses its type.
XSTS invalid-instances-accepted 29 -> 28 (typeDef00204m: <root>Yes</root> is no
longer accepted where root substitutes a boolean head), no other bucket moved
(valid-schemas-rejected and valid-instances-rejected held at 0 across all 14383
groups). 1451 unit tests pass.
cvc-elt.4.3.2.1 requires an xsi:type to name a type DERIVED from the element's
declared type. anyType is the ur-supertype of every type, so it is never derived
from anySimpleType (its own subtype); an anyType override on an anySimpleType
element is therefore invalid. The general not-derived check stays silent on
ur-type declared types to avoid false positives, so this adds the narrow,
provably-invalid anyType-for-anySimpleType pairing only, leaving anyType valid on
an anyType-declared element (which resolves to a complex ur-type and is excluded).
XSTS invalid-instances-accepted 30 -> 29 (stZ056), no other bucket moved
(valid-schemas-rejected and valid-instances-rejected held at 0 across all 14383
groups). 1450 unit tests pass.
XSD 1.0 has no year zero (the year before 0001 is -0001), so 0000-01-01T00:00:00
and the other 0000 forms are not valid values. DateTimeParser.parseYearOnly now
rejects a zero magnitude for both signs, so negative years like -0001 still parse.
Closes the XSTS dateTime011 false negative: invalid-instances-accepted 31 -> 30,
no other bucket moved (valid-schemas-rejected and valid-instances-rejected held at
0). 1449 unit tests pass; the full XSTS suite settles at 0/43/0/30.
The Antigravity onboarding definition-of-done checklist still read
'valid-schemas-rejected still 1'; the enforced baseline is 0
(Tests/XSTSSuiteTests.swift), matching the rest of this file and GEMINI.md.
Caught by the PR review pass.
containerLocationMap stored `loc ?? rootLocation` into a [ObjectIdentifier:
String?]. Because the value type is itself optional the coalescing bound at the
outer optional and rootLocation was never used (the compiler flagged it), so a
container with no location of its own (the compile root) mapped to nil even when
a circular import had loaded the root under a schemaLocation. An explicit String?
annotation keeps the coalescing at the String level so the root inherits
rootLocation.
Conformance is unchanged: 1448 unit tests pass and the full XSTS suite still
settles at 0/43/0/31. The SchemaNameUniqueness root-index dedup case is kept, not
removed: an experiment removing it on the theory it was now redundant regressed
three valid schemas to rejected, so it does real work beyond the fixed fallback.
The counted content automaton clamped every occurrence counter to a single
inputLength + 1 ceiling, so a content model with nested occurrence bounds let
the epsilon-closure grow toward inputLength raised to the counter count. The
XSTS particlesQ005 instance (about 1000 elements against
sequence{1,1000}(e1{1,unbounded})) drove that to roughly 16M configurations per
closure and hundreds of millions of hash operations, so validation did not
terminate in reasonable time.
Each counter now saturates at min(maximum, inputLength + 1), and an unbounded
counter at its minimum: once the minimum is reached every higher count is
matching-equivalent, because canExit only tests the minimum and belowMaximum is
vacuous, so collapsing them is exact. The active-configuration set is then
bounded by the content model rather than by the input length.
particlesQ005 goes from a multi-minute near-hang to about two seconds with
identical results. The 1448 unit tests pass and the full XSTS suite still
settles at 0/43/0/31, and now completes instead of hanging.
- README, roadmap: XSTS status quad and gates table now read the enforced
0/43/0/31 (Tests/XSTSSuiteTests.swift); test count 986/169 to 1448/258;
encoding list and the remaining-work section reflect shipped reality;
check-all expansion lists the validation and changelog gates
- GEMINI.md, onboarding-antigravity.md: valid-schemas-rejected 1 to 0 and the
pinned baselines updated to current
- SECURITY.md: complete the reporting address
- Parsing.InputEncoding: doc comment lists the full shipped encoding set
- Schema.XSDParserUnion: guard the localName interpolation with ?? "" to match
the same line and silence an optional-interpolation warning (internal dedup
key, behavior preserving)
- CHANGELOG: Unreleased entries for the above
Companion to production-readiness.md and roadmap.md. Characterizes, with code
evidence, the two biggest hurdles. (1) The invalid-accepted tail concentrates in
ParticleRestriction.violation's deliberately under-approximated clauses (MapAndSum,
RecurseAsIfGroup, NameAndTypeOK): a single high-fan-in oracle where tightening
risks a false positive. (2) ContentMatcher unrolls occurrence values into NFA
states, so the occursUnrollCap/totalStateCap/positionCap caps are band-aids that
silently degrade to star; a counting automaton is what turns the cap into a proven
bound.
Group @testable imports at the bottom, per public/rules-swift's swiftformat. A
behavior-preserving import-ordering reformat across 252 files (mostly tests).
--trailingclosures was deliberately not adopted (cosmetic, and swiftformat flags
it as caution-worthy).
Make public/rules-swift the canonical source (per project direction) and adopt
its opt-in lint rules that PureXML lacked: empty_count, explicit_init, first_where,
sorted_first_last, unused_import, plus its todo disable. All pass with zero
violations on the existing sources.
PureXML keeps its coherent, stricter tuning where the canon is a self-described
'starting point': force_unwrapping/force_try/force_cast = error (safety), and
trailing commas mandated in both swiftformat (--commas always) and swiftlint
(mandatory_comma) to resolve the canon's formatter-adds vs linter-flags mismatch.
docs/rules/README.md now points at public/rules-swift and documents the tuning.
The attribute-group and model-group redefine-restriction rules built the
namespaced base-component pool and did the namespaced-key-plus-chameleon-fallback
lookup with duplicated code and near-identical comments. Extract baseComponents(in:
named:) and resolveBaseComponent(_:name:owner:) so the recurring namespace-keying
and chameleon-fallback pattern lives in one place. Behavior-preserving: all redefine
unit tests pass and the XSTS buckets are unchanged (0/48/0/31).
A constraining facet may not be applied where a simpleContent restriction's base
chain resolves to xs:anySimpleType, the ur simple type, which carries no
constraining facets (XSD Part 2 3.2.1); stZ010 (a complex type extending
xs:anySimpleType, restricted by another that adds minLength) is now rejected.
The base chain is followed through local complex types' own simpleContent bases;
it stops at any other built-in, a simple type, or an external/unresolved base, and
a restriction supplying its own nested simpleType base is skipped (the facet
applies to that), so a facet on a string-, int-, or other real-typed simpleContent
stays accepted.
XSTS invalid-schemas-accepted 49 -> 48, no false positive (valid-schemas-rejected
and valid-instances-rejected held at 0; an adversarial critic probed 12 valid
schemas across every resolver branch and confirmed FP-safety), no other bucket
moved.
A redefinition of an attribute group may not ADD an attribute the original does
not declare (a superset is not a restriction, src-redefine / cos-ct-restricts);
schM3 (redeclaring the original's attributes plus a new b4) is now rejected. The
base is resolved by namespaced identity with a {}name chameleon fallback for a
no-targetNamespace redefined schema.
False-positive-free: the added-attribute check runs only when the original's
attribute set is fully known from its direct declarations and the redefinition
carries no attributeGroup reference (a self-reference brings in the original's
attributes, so declaring more alongside it is a valid redefine, not an addition,
W3C attgC007/attgC038/groupB018, test75083/xsd003b), and a use=prohibited
attribute the base lacks forbids rather than permits a name, so it does not widen
the usable set and is left alone.
XSTS invalid-schemas-accepted 51 -> 49, no false positive (valid-schemas-rejected
and valid-instances-rejected held at 0; five self-reference over-rejections were
caught by the gate and one prohibited-attribute over-rejection by an adversarial
critic, all fixed before commit), no other bucket moved.
A non-self-referencing model group redefined inside xs:redefine must be a valid
restriction of the group it redefines (cos-group-restrict / src-redefine 6.1.2),
so its content model may not widen the original's language. The check reuses the
particle-restriction oracle used for complex-type restriction, resolves the
original group by namespaced identity with a {}name chameleon fallback, and runs
post-compile where element type compatibility is available.
Now rejected: adding required content to an empty base (schZ006), dropping a
required all member (schL1), reordering an all (schL5/L6, the Recurse rule is
order-preserving even for all), growing to a superset (schL8), and the
multi-document/chained redefines schN4/5/6/7 and schL10. To stay
false-positive-free the check skips a redefinition carrying a self-reference and
declines to judge a base with a maxOccurs=0 pointless particle (the contested
case W3C accepts, mgO013); genuine restrictions (dropping an optional member, an
unchanged redefinition) are accepted.
XSTS invalid-schemas-accepted 64 -> 51, no false positive (valid-schemas-rejected
and valid-instances-rejected held at 0; an over-rejection of mgO013 surfaced by
the gate was fixed before commit, and an adversarial critic confirmed FP-safety),
no other bucket moved. Disclosed under-rejections: self-referencing redefinitions
(schN10) and maxOccurs=0-base redefines.
A redefinition of an attribute group restricts the original (src-redefine /
cos-ct-restricts), so it may not relax a fixed value the original sets. schM10
(changing a fixed attribute to a default one) is now rejected. The check fires
only when the redefinition declares the matching attribute as non-prohibited
with a differing fixed value; keeping the same fixed value, prohibiting it, or
omitting it is left alone, and it rides the existing namespaced base lookup, so
no valid redefinition is rejected.
XSTS invalid-schemas-accepted 65 -> 64, no false positive (valid-schemas-rejected
and valid-instances-rejected held at 0), no other bucket moved.
A redefinition of an attribute group restricts the original (src-redefine /
cos-ct-restricts), so it may neither eliminate a required attribute the original
declares (attgZ002) nor re-introduce a prohibited one as usable (attgZ003).
redefineAttributeGroupRestrictionErrors compares the redefinition to the original
attribute group and reports either violation.
False positives the verification caught and the rule guards against: re-declaring
a base prohibited attribute AS prohibited keeps it prohibited and is valid (the
check requires a non-prohibited use); the drop-required check is skipped when the
redefinition pulls attributes back via a nested attributeGroup reference; and the
base is matched by NAMESPACED identity ({target}name), so a same-local-name
attribute group imported from a different namespace is never mistaken for the base
(the same namespace guard the redefine self-reference rule uses).
invalid-schemas-accepted 67 to 65, no false positive (valid-schemas-rejected and
valid-instances-rejected held at 0). The gate caught a re-prohibition false
positive and an adversarial review caught a cross-namespace collision false
positive; both are fixed and covered by tests.
A complex type whose simpleContent derives from a list or union simple type now
keeps that variety, so its element content is validated as a list or union rather
than collapsing to an atomic value space. simpleContentType built the content type
with only the resolved base and facets, dropping the variety, so a list-typed
simpleContent accepted any whitespace-separated text and an invalid item (e.g. a
non-int in a list of int) was not caught.
A valid list/union value is unaffected; the change only tightens validation of
content that was wrongly accepted. The W3C XSTS corpus does not exercise this
case, so all four bucket counts are unchanged (vsr=0, isa=67, vir=0, ivi=31); the
gap was surfaced by the libxml2 differential and is covered by a unit test.
A substitution-group member whose type is a list or union derives only from
xsd:anySimpleType (XSD Part 2), so it may validly affiliate only to a head typed
anySimpleType, to the same type, or as a recorded restriction of the head's type
(cos-equiv-derived-OK). substitutionTypeErrors skipped the derivation check
entirely when either the member or the head type was a list/union, so a
list/union-typed member affiliated to an unrelated head type (addB141: a union
member of a list-typed head) was accepted.
checkSingleElement now rejects a list/union member type that is not a recorded
derivation of a non-ur head type, before the blanket list/union skip. A list/union
member of an anySimpleType (or untyped, hence anyType) head, the same type, and a
restriction of the head's list/union type are all left valid (isDerived covers the
restriction chains via typeDerivesOrEqual).
invalid-schemas-accepted 69 to 67, no false positive (valid-schemas-rejected and
valid-instances-rejected held at 0). An adversarial review verified untyped and
anySimpleType heads, multi-level restriction-of-list/union members, same-type and
cross-namespace affiliations all still compile.
The namespaced derivation backbone keyed every element by the container target
namespace, but a local element is in the target namespace only when qualified (its
own form, else the schema's elementFormDefault). An unqualified local element's
block was keyed {targetNS}name while the instance element is {}name, so
blockedSubstitutionError's lookup missed and a blocked xsi:type substitution on it
was accepted (typeDef00801: a local Element with block="extension" did not bar an
extension substitution).
gatherElement now keys a local element by localElementNamespace (target namespace
when qualified, none otherwise); global elements stay keyed by the target
namespace, and the bare-keyed tables are unchanged.
invalid-instances-accepted 32 to 31, no false positive (valid-instances-rejected
and valid-schemas-rejected held at 0). An adversarial review confirmed qualified
local elements are unchanged, global and same-named local element keys stay
distinct, and a non-blocked substitution on a local element is still accepted.
An abstract type cannot be the type of an instance element (cvc-elt.4.3.2), so an
instance xsi:type naming an abstract type is invalid as a substitution even when
the element's declared type would otherwise admit it (typeDef01101, xsi:type on an
anyType-declared element). The validator only checked for an abstract DECLARED
type with no xsi:type; abstractXsiTypeError now also rejects an abstract xsi:type
substitute (matched against the namespaced abstract-type set). A concrete type is
unaffected.
invalid-instances-accepted 34 to 32, no false positive (valid-instances-rejected
and valid-schemas-rejected held at 0; a valid instance never names an abstract
type via xsi:type), no other bucket moved.
An element with a fixed value constraint carries its fixed character value and no
element children, even under a mixed content type (cvc-elt.5.2.2.2.1). The fixed
check compared only the concatenated text, so a mixed fixed element whose text
matched the fixed value but which also contained child elements (ctZ013c/d) was
accepted. elementFixedErrors now rejects a fixed element that has element
children. An empty element (which takes the fixed value) and text equal to the
fixed value remain valid.
invalid-instances-accepted 37 to 34, no false positive (valid-instances-rejected
and valid-schemas-rejected held at 0; a fixed element never validly carries
element children), no other bucket moved.
A complex type's simpleContent derivation can name another complex type with
simpleContent as its base. simpleContentType resolved a non-XSD, non-simpleType
base via context.simpleTypes only, so a complex-type base (absent from that map)
fell back to xs:string and left the content unconstrained: text invalid for the
real base type was accepted (baseTD00101, where Test restricts Test2 which extends
xs:int, yet "b" was accepted).
simpleContentBaseType now resolves the base to an XSD built-in, a named simple
type, or, when the base is a complex type with simpleContent, that type's own
resolved simple type by recursing through complexTypeNodes (cycle-guarded by a
visited set); the deriving node's facets are then applied over the resolved base's
facets. So Test resolves to int and non-int text is rejected, while a valid int
and the accumulated facets along the chain are honoured.
invalid-instances-accepted 39 to 37, no false positive (valid-instances-rejected
and valid-schemas-rejected held at 0), no other bucket moved. An adversarial
review exercised facet accumulation, named-simpleType bases in the chain,
multi-level and extension chains, and cycles, and found no over-rejection.
use="prohibited" removes an attribute from a complex type's effective {attribute
uses} (XSD 1.0 §3.4.2): the attribute is then undeclared, so its presence is
rejected when no attribute wildcard admits it (attJ), but is still admitted (and
validated per the wildcard's processContents) when one does (attZ002). The use
had been parsed as merely optional, so a prohibited attribute was always allowed.
AttributeUse gains a `prohibited` flag, set for both a local declaration and an
<attribute ref> use site. The declaration is kept in the type's attribute list so
schema-validity checks still see it, while instance validation excludes prohibited
uses: the matching loop and the wildcard's declared-set skip them, and the ID-type
count ignores them. There is no explicit "prohibited attribute present" error, so
a wildcard that admits the attribute name still governs it (an earlier attempt to
reject outright wrongly overrode the wildcard).
invalid-instances-accepted 43 to 39, no false positive (valid-instances-rejected
and valid-schemas-rejected held at 0, invalid-schemas-accepted unchanged at 69).
An adversarial review confirmed prohibited attributes admitted by a wildcard
(every namespace/processContents variation, local and ref), restriction removal,
and the ID-count interaction are all handled without over-rejection.
XSD 1.0 §3.4.6 (cvc-complex-type) allows an element at most one attribute whose
type is xs:ID. A complex type that DECLARES two ID-typed attribute uses is already
a compile-time schema error, but two ID-typed attributes can also reach an element
through an xs:anyAttribute wildcard that admits two ID-typed global attributes
(attZ014), which was accepted.
validateAttributes now counts the present attributes whose effective type is
xs:ID, taking the type from the declared use an attribute matches or the global
declaration a matching wildcard admits (so a restriction of xs:ID counts), and
reports an error when more than one is present. xs:IDREF/xs:ENTITY and a single
ID attribute are unaffected, and a valid instance never carries two, so the check
cannot over-reject.
invalid-instances-accepted 45 to 43, no false positive (valid-instances-rejected
and valid-schemas-rejected held at 0), no other bucket moved.
An identity-constraint (unique/key/keyref) field that selects an absent attribute
or an empty element whose declaration carries a default or fixed value now takes
that value as its identity-tuple component, so two members can collide on the
defaulted value (idG011/idG012 default, idF016/idF017 fixed) where the duplicate
was previously missed.
The default/fixed value constraints on identity-field targets are gathered at
compile time (identityFieldConstraints, mirroring identityFieldTypes) and threaded
through to the identity validator. The substitution is exact about absent versus
present-empty: an @attr field is defaulted only when the attribute is ABSENT (an
empty node-set), never when present with value ""; a "." element field only when
the selected element has no character content. Applying a default only adds an
identity value where the schema declares one, so it can create a genuine
duplicate-key detection but never reject a valid instance.
To keep within the file/type-size and one-type-per-file limits, FieldValue,
XPathQueryCache, and the schema-location hints were extracted to their own files
(behavior-preserving moves).
invalid-instances-accepted 50 to 45, no false positive (valid-instances-rejected
and valid-schemas-rejected held at 0), no other bucket moved. An adversarial
review probed present-empty attributes, element content winning over a default,
fixed values, keyref matching, multi-field keys, and merged-schema defaults, and
found no over-rejection.
A block on a substitution-group head's TYPE (its {prohibited substitutions}), not
only on the head element, bars a member whose type reaches the head's type by the
blocked derivation method. filterSubstitutions consulted only the head element's
own block, so block="restriction" on the head's complex type let a
restriction-derived member substitute.
The substitution-group filter now drops a member when its derivation method to the
head's type is in the union of the head element's {disallowed substitutions} and
the head type's {prohibited substitutions}. The element-level block="substitution"
outright bar stays element-only (a type's prohibited-substitutions never carries
substitution), and a member derived by a method the block does not list still
substitutes.
invalid-instances-accepted 53 to 50, no false positive (valid-instances-rejected
and valid-schemas-rejected held at 0). An adversarial review confirmed members
derived by an allowed method, the positive disallowedSubst instances, transitive
chains, and blockDefault interactions all still substitute. (A separate
cross-document cross-namespace substitution-group rejection was confirmed
pre-existing on HEAD, not introduced here.)
A complex type that records no base of its own derives only from xsd:anyType, so
it cannot validly substitute (cvc-elt.4.3.2.1) for an element whose declared type
is a more specific complex type. The not-derived check required the substitute to
be present in the derivation backbone, so naming the declared type's own complex
ancestor, or an unrelated baseless complex type, via xsi:type was accepted.
notDerivedSubstitutionError now treats a complex substitute as in scope even when
it is absent from the backbone: its non-derivation from a non-ur declared type is
certain (a baseless complex type derives only from anyType, and any real
extension/restriction chain is recorded in the backbone, where derivationMethods
finds it). An atomic substitute must still be in the backbone, so a built-in
stays silent. The declared guard excludes the ur-types, so a baseless complex
substitute under an anyType declaration is admitted; a genuine extension or
restriction substitute is accepted.
invalid-instances-accepted 60 to 53, no false positive (valid-instances-rejected
and valid-schemas-rejected held at 0), no other bucket moved. An adversarial
review exercised deep, cross-namespace, redefine, and element-ref-path derivation
chains plus anyType-declared baseless substitutes, and found no over-rejection.
cvc-elt.4.3.2.1 requires an xsi:type to name a type derived from the element's
declared type. The not-derived check covered only complex substitutes, so an
xsi:type naming an atomic type unrelated to the declared atomic type, such as a
sibling restriction or the declared type's own ancestor, was accepted.
notDerivedSubstitutionError now admits atomic simple substitutes too (via
isAtomicOrComplex, which excludes list/union, handled separately). It still fires
only when both the substitute and the declared type are recorded in the
namespaced derivation backbone and derivationMethods finds no path between them.
Built-in types and the ur-types are absent from the backbone, so a built-in
substitute or declared type leaves the check silent: a valid built-in derivation
such as xsi:type="xs:int" on an xs:decimal element is never flagged, which was
the source of an earlier false-positive regression. A genuine user restriction
chain stays inside the backbone the whole way down, so a validly derived
substitute is accepted.
invalid-instances-accepted 66 to 60, no false positive (valid-instances-rejected
and valid-schemas-rejected held at 0), no other bucket moved. An adversarial
review probed built-in derivation in both directions, deep and cross-namespace
restriction chains, and sibling/ancestor substitutes, and found no over-rejection.
The list/union xsi:type derivation check decided whether the declared type admits
a list or union substitute from a single type-table lookup, so it never reached
the concrete type of an element declared by reference (`<xs:element ref="x">`):
such an element's list/union xsi:type was wrongly accepted.
declaredAdmitsAnySimpleType now resolves the declared type through the full
element-ref / type-reference chain (resolvedDeclaredType, the type-returning
companion to resolvedDeclaredTypeName) to the concrete .simple/.complex before
deciding. A list or union xsi:type on a ref'd element whose declaration resolves
to a more specific atomic type is now rejected; the chain is cycle-bounded and
falls through to suppress (under-reject) when it cannot resolve, so a ur-typed,
list/union-typed, or unresolvable declared type still admits the substitute.
invalid-instances-accepted 78 to 66, no false positive (valid-instances-rejected
and valid-schemas-rejected held at 0), no other bucket moved. An adversarial
review exercised ref chains to anySimpleType/anyType/untyped declarations, named
and aliased list/union declarations, a deep alias chain, cross-namespace imported
refs, substitution-group heads, atomic restriction substitutes, and a self-
referential alias cycle, and found no over-rejection.
A list or union simple type is derived only from xsd:anySimpleType (XSD Part 2),
so an xsi:type naming a list or union can validly stand in (cvc-elt.4.3.2.1) only
for an element whose declared type is a ur-type (anySimpleType/anyType). Naming
one on an element of a more specific atomic or complex type, such as
xsi:type="myList" on an xs:int element, is not a valid derivation and is now
rejected.
listUnionSubstitutionError flags the case; declaredAdmitsAnySimpleType suppresses
it when the declared type is a ur-type, is itself a list or union (whose inline
identity cannot be matched), or is an unresolvable reference, so the check biases
to under-reject and never over-rejects. A genuinely derived restriction
substitute, a list/union under an anySimpleType declaration, and a restriction of
a list (which keeps list variety) are all left accepted.
This widening became safe once the derivation backbone was keyed by namespaced
identity (#186); the conservative atomic and built-in not-derived cases remain
under-rejected pending the full type-identity relation.
invalid-instances-accepted 81 to 78, no false positive (valid-instances-rejected
and valid-schemas-rejected held at 0), no other bucket moved. An adversarial
review exercised a list-restriction substitute, a union declared type, ur-type
and typeReference-chain declarations, and cross-namespace list types, and found
no over-rejection.
The instance-validity derivation and substitution tables were keyed by bare
local name, so two same-local-name types or elements in different namespaces
collided. In multi-document validation this let one schema's block/derivation
facts shadow another's, and it blocked honouring blockDefault (whose re-land
widened that collision into an over-rejection).
Key the instance subsystem by namespaced identity {ns}local instead. The shared
DerivationTables keeps its bare-keyed fields for the schema-validity machinery
(Particle-Valid-Restriction and the consistency rules, which resolve QNames to
bare names and walk the same backbone) and gains parallel ns* fields built in
the same pass, resolving each container's components through its target
namespace and prefix bindings. Only the instance consumers
(blockedSubstitutionError, notDerivedSubstitutionError, abstractTypeError, the
substitution-group closure, and referenceParticle) read the ns* tables, via a
derivationKey(fromReference:) that reduces a type:{ns}local reference, a bare
name, or an element ref to a {ns}local backbone key. A key miss stays permissive
(no constraint, substitution allowed), so the re-keying cannot over-reject.
With the namespaced backbone in place, blockDefault is re-landed FP-free: a
schema's blockDefault supplies the {disallowed substitutions} of every element
and complex type without an explicit block (own block overrides; block="" clears;
intersected with the methods each component admits), so blockDefault="extension"
bars an extension substitution and "#all" bars all, without one namespace's
defaults shadowing another's.
The pure re-keying preserves single-document behaviour exactly (the corpus is
overwhelmingly single-namespace, where {ns}local and bare coincide): all four
XSTS buckets held at vsr=0 / isa=69 / vir=0 / ivi=89 after re-keying alone.
blockDefault then drove invalid-instances-accepted 89 to 81, no false positive
(valid-instances-rejected and valid-schemas-rejected held at 0), no other bucket
moved. An adversarial review exercised same- and cross-namespace substitutions,
no-targetNamespace keys, simple-type restriction, transitive substitution groups,
and blockDefault override/over-block cases, and the prior multi-document
cross-namespace blockDefault over-rejection is now accepted while a same-namespace
blocked substitution is still rejected (precise, not merely permissive).
A complex type's effective {attribute wildcard} is the intersection of the
anyAttribute wildcards it draws from its own declaration and each referenced
attribute group (XSD 1.0 cos-aw-intersect), not their union. An attribute is
admitted only if every source admits it, so two attribute groups constraining
disjoint namespaces admit nothing and an attribute in either is rejected.
attributeWildcard(under:_:visited:) combined the sources with Wildcard.union,
which let through an attribute that only one source allowed. It now uses a new
Wildcard.intersection covering the namespace cases (##any identity, enumerated
set intersection, enumerated-with-##other exclusion via exclude(_:otherThan:)).
A type with a single anyAttribute source is unchanged (intersection with the
nil identity is the lone wildcard), and a complexContent extension's union with
its base wildcard (XSDComplexContent) is untouched. The inexpressible
##other/##other-different-target and mismatched processContents cases resolve to
the less restrictive result, so the change never over-rejects.
XSTS invalid-instances-accepted 94 to 89, no false positive (valid-instances-
rejected and valid-schemas-rejected held at 0 across the whole corpus), no other
bucket moved. An adversarial review exercised nested groups, ##other/##local and
cross-import namespace cases, processContents, extension-vs-restriction, and
single-source invariance, and found no new over-rejection.
An identity constraint (unique/key/keyref) compares its field values in their
value space. Two fields typed by different members of the decimal-derived
numeric family denote the same value when their numbers are equal, so a
decimal "1" and an unsignedByte "1" (supplied here via xsi:type) collide for a
unique. FieldValue equality previously required the two declared base types to
match exactly and reported them unequal when they differed, so the duplicate
went undetected.
Add BuiltinType.isDecimalDerived (decimal plus the integer family, which share
one value space) and compare those values as DecimalValue. The unification is
confined to that family: float and double are distinct primitive spaces and a
number is never equal to a string, so no valid instance is newly rejected.
XSTS invalid-instances-accepted 95 to 94, no false positive (valid-instances-
rejected and valid-schemas-rejected held at 0 across the whole corpus), no
other bucket moved.
The ur-type xsd:anyType modeled its element and attribute wildcards with
processContents="skip", so a child element or attribute of an untyped element
(or an element typed xs:anyType) was never validated. XSD 1.0 §3.4.7 specifies
those wildcards as "lax": content that has a global declaration is validated
against it, while undeclared content is admitted. This is why an invalid
declared value reached through a wildcard (e.g. a duration-typed attribute with
value "P" matched by <xsd:anyAttribute/>) was accepted.
Set both wildcards to .lax at all three ur-type construction sites and loosen
isUrType to recognise the lax (or skip) shape.
The lax change required making the wildcard lookup of a global element
declaration namespace-exact. A global element is always in the schema target
namespace, so an unqualified instance element must match only a no-namespace
global, which the qualified key element:{}local already covers. The previous
bare-local-name fallback conflated namespaces and would have validated an
unqualified child against a same-named namespaced declaration; it is removed in
the document path and both streaming sites.
XSTS invalid-instances-accepted 101 to 95, no false positive (valid-instances-
rejected and valid-schemas-rejected held at 0; the spec's own errC007 anyType-
lax case validates correctly), no other bucket moved. An adversarial review
re-ran the full suite and found no new over-rejection.
block="substitution" (or a block set containing it, including #all) on a
substitution-group head forbids every member from substituting, regardless of
the head's or members' types. filterSubstitutions gated the whole filter --
including this type-independent substitution ban -- on the head having a type,
so an abstract untyped head (the particlesDc cluster) skipped filtering and its
members still substituted. The substitution ban now applies first and
type-independently; the type-derivation block (extension/restriction) still
needs the head's type, unchanged.
An adversarial review confirmed no false positive: the head's own appearance is
computed independently of the member list (a concrete blocked head still
validates), and the typed-and-substitution-blocked case is provably equivalent
to the prior per-member rejection. Disclosed pre-existing under-rejection (not
introduced here): blockDefault="substitution" is not honored, only an explicit
block on the head.
XSTS: invalid-instances-accepted 107 to 101 (particlesDc004/005/006/008/009 and
a sibling), valid-instances-rejected and valid-schemas-rejected held at 0.
First instance-bucket reduction of the campaign. An instance xsi:type naming a
complex type must be derived from the element's declared type. notDerivedSubstitutionError
flags a complex substitute on a different branch of the hierarchy; previously
only a derived-but-blocked substitution was caught (blockedSubstitutionError
returns nil when there is no derivation path), so a non-derived xsi:type was
silently accepted (the test006 cluster, 32 instances, and more).
Conservatively scoped after a broader first cut took 20 false positives (now
reverted): it fires only when the substitute resolves to a COMPLEX type and both
substitute and declared names are in the derivation backbone with no path between
them. Simple/list/union substitutes, ur-types, and unresolvable references are
left alone -- under-reject, never over-reject. An adversarial review confirmed
the conservative guards eliminate the prior false positives; the one residual is
the pre-existing bare-local-name keying of the derivation backbone (shared with
blockedSubstitutionError), disclosed in the code, corpus-absent.
The three xsi:type override checks (blocked / not-derived / abstract) are folded
into one xsiTypeOverrideError helper to keep validateChild under the complexity limit.
XSTS: invalid-instances-accepted 133 to 107 (26 caught), valid-instances-rejected
and valid-schemas-rejected both held at 0, no other bucket moved.
xs:include and xs:redefine require a schemaLocation attribute (only xs:import's
is optional, being a resolution hint). An include or redefine declared without
it is now rejected in the structure walk.
A required-attribute check with no name resolution or reference walking; it
cannot over-reject (schemaLocation is mandatory on these per the
schema-for-schemas), so the full critic loop is not needed here -- reserved for
the false-positive-prone resolution rules. XSTS: invalid-schemas-accepted 71 to
69 (schB3 and a sibling), valid-schemas-rejected held at 0, no other bucket moved.
A schema pulled in by xs:redefine must have the same targetNamespace as the
redefining schema, or no targetNamespace (chameleon) -- the same constraint
xs:include already enforced. includeCompositionFindings now also walks
xs:redefine, selecting the loaded redefined schema document (guarding the
container lookup with localName == "schema" so the redefine wrapper, which
carries no targetNamespace, is not matched) and comparing its target namespace
to the redefiner's effective (chameleon-resolved) namespace.
An adversarial review confirmed no false positive: same-namespace, chameleon
(no targetNamespace on the redefined schema), and both-no-namespace redefines
are never flagged; only a present-and-different targetNamespace fires.
XSTS: invalid-schemas-accepted 74 to 71 (schI2/schI5 and a schN cross-namespace
case), valid-schemas-rejected held at 0, no other bucket moved.
Adds a dated accounting to the XSTS deviation worklist: the labelled-suite
ratchet stands at 0/74/0/133 with false positives held at 0 across the campaign.
Partitions the remaining 74 into fixable-but-intricate debt (redefine
restriction, NameAndTypeOK union membership, wildcard UPA, chameleon/cross-ns
composition, heterogeneous particle edges) and irreducible-by-design or disclosed
items (RecurseAsIfGroup spec ambiguities, the deliberate regex-tolerance), with
the root cause and false-positive risk for each so the burndown is driven by
understanding. Records that the adversarial critic loop caught three genuine
false positives this session that the suite and differential both missed.
Extends the redefine self-reference rule (redefineSelfReferenceErrors, moved to
SchemaRedefineSelfReference.swift) to attribute groups: a group or attributeGroup
inside xs:redefine may reference itself at most once (src-redefine.6.1.1 /
7.2.1), and a group self-reference must occur exactly once. Catches a redefined
attributeGroup with two self-references (attgC044).
Fixes a latent false positive an adversarial review found in the group case: the
search now uses a BOUNDED walk (boundedSelfReferenceNodes) that stops at nested
element/complexType scopes, matching the established cycle-rule boundary. The
prior unbounded walk would miscount a recursive group reference inside a nested
element's content (a data-structure recursion, legal) as a redefinition
self-reference. attributeGroup content has no such nesting, so it is unaffected.
XSTS: invalid-schemas-accepted 75 to 74 (attgC044), valid-schemas-rejected held
at 0, no other bucket moved. The validation description broadened accordingly
(configuration pin updated).
A selector/field xpath in an identity constraint may name elements/attributes
with namespace-qualified name tests; every prefix used must be a declared
xmlns: prefix in scope (the implicit xml prefix excepted). identityXPathErrors,
after the existing syntax check, now resolves each name-test prefix against the
in-scope bindings (namespaceBindingsInScope, which captures the schema root and
every ancestor) and rejects an undeclared one.
An adversarial review confirmed no false positive: a prefix declared on the
schema root, any intermediate ancestor, or the selector/field element resolves
correctly; xml, the child::/attribute:: axis keywords (colon-free), prefix:*,
and inter-token whitespace are all handled.
XSTS: invalid-schemas-accepted 77 to 75 (idI010, idJ011), valid-schemas-rejected
held at 0, no other bucket moved. (Two existing xpath-syntax fixtures used an
undeclared 'ns' prefix to test name-test syntax; the test schema now declares
xmlns:ns so they exercise syntax with a bound prefix.)
A group inside xs:redefine may reference itself (the group being redefined) at
most once, and that self-reference must have minOccurs = maxOccurs = 1. A
self-reference with minOccurs="0", maxOccurs="unbounded", or a count above 1 is
invalid. redefineGroupSelfReferenceErrors flags it, surfaced by the pre-compile
Validation redefineGroupSelfReferencesAreUnit.
The self-reference is matched by NAMESPACE (referenceNamespace == the redefining
schema's targetNamespace), not local name alone, so a redefined group that
references an imported group merely sharing its local name is correctly left
unconstrained -- the false-positive boundary an adversarial review flagged
before merge, mirroring the namespace guard the sibling redefine/cycle rules use.
XSTS: invalid-schemas-accepted 79 to 77 (schR3, schR4), valid-schemas-rejected
held at 0, no other bucket moved.
A local element declaration (one not a child of schema) must identify itself
with a name or a ref; an xs:element with neither is invalid. localElementErrors
now flags it, alongside the existing local-element forbidden-attribute checks.
XSTS: invalid-schemas-accepted 80 to 79 (elemP005), valid-schemas-rejected held
at 0, no other bucket moved.
A complexType nested in an element is a local (anonymous) type definition and
may not carry a name attribute; only a top-level complexType (a child of schema,
or a redefine redefinition) is named. localComplexTypeNameErrors flags a named
complexType whose parent is an element, wired into the structure-walk validation.
XSTS: invalid-schemas-accepted 81 to 80 (ctA042), valid-schemas-rejected held
at 0, no other bucket moved.
The convergence audit for "validators everywhere" found the one remaining XSD
schema-validity verdict reported outside the Validator framework: facet
definition validity and pattern syntax validity were reported straight to the
diagnostics collector during simple-type compilation (validation folded into
parsing, which validation-rules.md forbids).
These checks genuinely need the resolved base type, computed during compilation,
so the finding is gathered into a context finding sink (XSDContext
.facetFindingSink) during type resolution and surfaced by the named post-compile
Validation simpleTypeFacetsAreValid -- located findings preserved, now pinned in
the configuration test and removable by identity like every other rule.
Re-sweep confirms the XSD path has converged: the only remaining throws are
notASchema (a not-a-schema precondition) and inconsistent (the framework's own
surfacing throw); no validity verdict bypasses the framework. No bucket change
(invalid-schemas-accepted 81, valid-schemas-rejected 0); full suite + fuzz pass.
Continues incorporating every schema-validity check into the Validator framework
(Rules/universal/validation-rules.md) rather than ad-hoc parse-pipeline throws.
- checkRedefine (src-redefine.5: a redefined type restricts or extends itself)
becomes the pre-compile Validation redefinitionsDeriveFromThemselves;
redefineDerivationErrors returns findings instead of throwing.
- checkAllGroups is removed entirely: every check it made is already produced by
Validations -- "a member must be an element" by schemaStructureValid
(allowedChildren[all]), and the occurrence limits by allGroupLimitedErrors
(cos-all-limited.2). Verified redundant and not load-bearing: an empirical
probe with the calls removed left invalid-schemas-accepted at 81 unchanged,
the fuzz suite's "compile and validation never crash or hang" still passes
(the NFA builder's own guards hold), and false positives stayed 0.
compileRule is now internal so composition Validations can live in their own
file (SchemaCompositionValidations.swift). Both new pre-compile rules are pinned
in ValidationConfigurationTests. The pre-existing public SchemaError.invalidAllGroup
case is kept (released API) though no longer thrown. No bucket change
(invalid-schemas-accepted 81, valid-schemas-rejected 0); full suite + fuzz pass.
Per the OpenAPIKit validation idiom (Rules/universal/validation-rules.md), every
schema-validity check should run through the Validator framework as a described,
composable Validation rather than an ad-hoc throw. The src-resolve check (a
resolved include/import/redefine schemaLocation must be a well-formed schema)
was a parse-pipeline throw; it is now the pre-compile Validation
referencedSchemasResolveToSchemas.
failedSchemaReferences (formerly the throwing checkSchemaLocations) returns the
resolved-but-not-a-schema locations; the pipeline records them on XSDContext and
the validation reports one finding each. Same rejection behaviour (the finding
makes the schema invalid), now located in the framework, pinned in the
configuration test. Removed the now-unused SchemaError.invalidSchemaReference
case (added earlier this branch, never released).
No bucket change (invalid-schemas-accepted stays 81, valid-schemas-rejected 0);
full suite and fuzz pass (report-and-continue does not crash downstream).
A type inside xs:redefine must restrict or extend the type it redefines, which
lives in the redefining schema's own target namespace. checkRedefine compared
local names only (derivationInfo strips the prefix), so a base like foo:b-ct
-- an imported type sharing the redefined type's local name -- was treated as
self and accepted. Now the base's namespace is resolved: a base bound to an
explicit namespace other than the redefining schema's target namespace is not
self and is rejected (src-redefine.5).
redefineBaseIsForeign (SchemaImportNamespace.swift) leaves a base with no
resolvable namespace alone, so an unprefixed or own-namespace self-reference
never over-rejects -- the false-positive boundary, the same namespace-aware
fix applied across the campaign.
XSTS: invalid-schemas-accepted 84 to 81 (schQ2/schQ4/schP3),
valid-schemas-rejected held at 0, no other bucket moved. Differential:
agree-valid unchanged at 3161, PureXML-stricter unchanged at 87 over 9618
schemas. Third step of the cross-document composition campaign (#161).
src-import.3.1/3.2: an xs:import's namespace attribute must equal the
targetNamespace of the schema document it resolves to (both present and equal,
or both absent). Importing a no-namespace document under a declared namespace,
or one whose target namespace differs, is now rejected.
importNamespaceErrors (new SchemaImportNamespace.swift) builds a resolved
schemaLocation -> targetNamespace map from the loaded containers and compares
each import node's namespace to it. Checked only when the schemaLocation
resolved to a loaded schema (an unresolved import is the processor's choice,
not an error) -- the false-positive boundary; imports keep their own target
namespace so no chameleon adjustment applies.
XSTS: invalid-schemas-accepted 87 to 84 (schF6/schG10/schG13),
valid-schemas-rejected held at 0, no other bucket moved. Differential:
agree-valid unchanged at 3161, PureXML-stricter unchanged at 87 over 9618
schemas. Second step of the cross-document composition campaign (#161).
When an xs:include / xs:import / xs:redefine schemaLocation IS resolved (the
schema loader returns content for it), that content must be a well-formed
schema document. Not-well-formed XML, or well-formed XML whose root is not
xs:schema, now makes the composing schema invalid (SchemaError
.invalidSchemaReference). collectContainers previously loaded externals with
`try? parseTree` and silently dropped a malformed one, accepting the schema.
A location the loader does NOT resolve (returns nil) is not an error, since
resolution is the processor's choice (src-resolve) -- so a self-contained
schema, or one whose externals are simply unavailable, is unaffected. This is
the false-positive boundary: fire only when the loader returned content that
is not a schema.
XSTS: invalid-schemas-accepted 95 to 87 (eight caught), valid-schemas-rejected
held at 0, no other bucket moved. Differential: agree-valid unchanged at 3161,
PureXML-stricter unchanged at 87 over 9618 schemas (zero new over-rejection).
First step of the cross-document composition campaign (#161).
The {type definition} of an attribute is a simple type (au-props-correct /
src-attribute). An attribute whose type resolves to a complex type, or to the
complex ur-type xs:anyType, is now rejected; a built-in or user simple type,
an inline simpleType, or no type (anySimpleType) stays valid. A locally-
declared type takes precedence over the built-in reading, consistent with the
src-ct rules.
Strictly zero false positive: an attribute can never carry a complex type, so
no valid schema is affected. XSTS: invalid-schemas-accepted 96 to 95 (attD002),
valid-schemas-rejected held at 0, no other bucket moved. Differential:
PureXML-stricter unchanged at 87 over 9618 schemas.
An element with a default or fixed value constraint must have a content type
that admits character data: a simple type, or mixed content (cos-valid-default
/ e-props-correct). An element whose type is, or whose inline complex type has,
element-only or empty content may not carry a value constraint.
elementValueConstraintContentErrors flags only a content type that is certainly
element-only or empty; a simple type, mixed content, xs:anyType (mixed), an
untyped element (anyType), a complexContent-derived type (mixedness may be
inherited), and a foreign or unresolved type are all left alone, so it cannot
over-reject.
XSTS: invalid-schemas-accepted 99 to 96 (elemG003/elemG004/elemD004),
valid-schemas-rejected held at 0, no other bucket moved. Differential:
PureXML-stricter unchanged at 87 over 9618 schemas.
A content-model derivation's base must match its content-model keyword
(XSD 1.0 Structures 3.4.6):
- src-ct.1: a complexContent restriction/extension base must be a complex
type. A simpleType or a built-in simple type (xs:string) is rejected; the
ur-type xs:anyType is complex and remains a valid base.
- src-ct.2 (extension): a simpleContent extension base must be a simple type
or a complex type with simple content. A complex type with element-only,
mixed, or empty content, including xs:anyType, is rejected.
A locally-declared type takes precedence over the built-in reading, so a
schema that targets the XSD namespace and extends its own complex types
(the schema-for-schemas, e.g. xs:openAttrs) is not rejected. This caught a
self-inflicted false positive during development; the fix restores it.
XSTS: invalid-schemas-accepted 104 to 99 (ctE003/ctE004/ctK002/ctJ002/
ctJ003), valid-schemas-rejected held at 0, no other bucket moved. The
differential corroborates: PureXML-stricter unchanged at 87 over 9618
schemas (zero new over-rejection).
An xs:unique / xs:key / xs:keyref content model is (annotation?, selector,
field+): exactly one selector, then one or more fields, in that order. The
structure walk previously checked only that the children were drawn from
{annotation, selector, field}, not their cardinality or order, so a field
before the selector, a second selector, or a missing field was accepted.
identityConstraintContentErrors enforces it, wired into the per-component
structural dispatch. XSTS: invalid-schemas-accepted 110 to 104 (six caught:
idA044/046, idB044/046, idC047/049), valid-schemas-rejected held at 0, no
other bucket moved.
Enforce the two occurrence-range constraints on model groups that PureXML
was not checking:
- p-props-correct.1: a particle's minOccurs may not exceed its maxOccurs,
both defaulting to 1 when the attribute is absent. occurrenceOrderErrors
previously fired only when BOTH attributes were present, so it missed
minOccurs="2" with a defaulted maxOccurs (2 > 1) and maxOccurs="0" with a
defaulted minOccurs (1 > 0). It now orders against the effective defaults.
- cos-all-limited.2: an all group's maxOccurs must be 1 and its minOccurs
0 or 1, and every particle it contains is limited to maxOccurs 0 or 1.
Both are zero-false-positive rules (no valid schema violates them). XSTS:
invalid-schemas-accepted 130 to 110 (20 caught), valid-schemas-rejected
held at 0, no other bucket moved. The differential harness corroborates:
agree-invalid 1751 to 1767, PureXML-stricter unchanged at 87 (no new
over-rejection across 9618 schemas).
Production-readiness stopper #3: characterise correctness against an independent
reference. For each .xsd under XSTS_ROOT, compares PureXML's compile verdict with
libxml2 (xmllint --schema) and writes disagreements to /tmp/xsts-differential.txt.
Over 9618 schemas: agree on 4912 (3161 valid, 1751 invalid). The 87 where PureXML
is stricter are triaged as PureXML being more correct (libxml2 omits facet
consistency, EDC, whiteSpace restriction, au-props, notation), independent
corroboration that PureXML does not over-reject. The 117 where libxml2 is stricter
are a false-negative triage queue (stopper #2). Opt-in; xmllint and the corpus are
test-harness-only, not package dependencies.
Read element-vs-choice Particle Valid (Restriction) through effective total
range: an element validly restricts a base choice when its leaf is a
NameAndTypeOK restriction of an element the choice admits and its effective
occurrence range is within the choice's. This fixes particlesZ001 (the
RecurseAsIfGroup rule its own test calls ambiguous; W3C reads it valid) while
keeping particlesM002/003 valid and particlesL rejected.
Named, bounded under-rejection (exact in-branch sequencing unmodelled), the
production-sanctioned direction: invalid-schemas-accepted 128->130,
invalid-instances-accepted 132->133 (particlesZ001.i, a masked gap now exposed).
XSTS valid-schemas-rejected 1->0. With valid-instances-rejected already 0,
false positives are now 0 in both buckets (production stopper #1).
reS17/reS38/reZ004v assert \d matches Ethiopic digits (category No), contrary to
the normative \d = \p{Nd} (XSD Datatypes Appendix F). Verified our \d matches Nd
(incl. Khmer U+17E0) and rejects Ethiopic No, as conformant processors do.
Excluded as a named, bounded spec divergence rather than corrupting \d.
XSTS valid-instances-rejected 3 to 0; the false-positive metric now reflects
spec correctness on the instance side (one schema, particlesZ001, remains).
cos-ct-restricts.4: a base type with no attribute wildcard admits none (so a
restriction may not introduce one), and a wider namespace constraint such as
##any over ##other, or a weakened processContents, is not a subset. Reuses the
wildcard subset check (ParticleRestriction.narrows, now internal).
XSTS invalid-schemas-accepted 131 to 128, no other bucket moved.
An attribute a complexContent restriction declares with no matching base
attribute use must be admitted by the base type's anyAttribute wildcard
(cos-ct-restricts.3); e.g. a no-namespace attribute is not admitted by a
##other wildcard. A prohibited declaration still validly removes an attribute.
XSTS invalid-schemas-accepted 132 to 131, no other bucket moved.
A namespace name must be a non-empty anyURI, so <schema targetNamespace=""> and
<import namespace=""> are invalid. Omitting the attribute (no target namespace,
or a no-namespace import) stays valid.
XSTS invalid-schemas-accepted 134 to 132, no other bucket moved.
A restricting xs:any must have processContents at least as strong as the base
wildcard's (strict > lax > skip; Wildcard Subset, XSD 1.0 sec 3.10.6), in
addition to narrowing the namespace constraint.
XSTS invalid-schemas-accepted 137 to 134, no other bucket moved.
Restricting a built-in or user simple type through <simpleContent><restriction>
is invalid (cos-ct-restricts.1 / src-ct.2): a complex type with simple content
is built by extending a simple type, not by restricting one. A unit-test
fixture that restricted xs:string directly was spec-invalid and now restricts a
complexType with simple content instead.
XSTS invalid-schemas-accepted 142 to 137, no other bucket moved.
A restricting element may not be nillable unless the base element it restricts
is also nillable. The element particle (Term.element) now carries a nillable
flag, populated from the element's nillable attribute.
XSTS invalid-schemas-accepted 144 to 142, no other bucket moved.
When the base element has a fixed value constraint, a restricting element must
also be fixed to the same value, compared in the base type's value space so a
value-equal pair written in different lexical forms is not wrongly flagged.
XSTS invalid-schemas-accepted 148 to 144, no other bucket moved.
Particle Valid (Restriction) was keyed by type name, so an inline complexType
deriving by complexContent restriction was never checked against its base. Such
anonymous restrictions are now compiled and run through the same subset
algorithm as named types.
XSTS invalid-schemas-accepted 163 to 148, no other bucket moved.
Particle Valid (Restriction) now requires a restricting element's {disallowed
substitutions} to be a superset of the base element's: a restriction may block
more substitutions, never fewer. The element particle (Term.element) carries a
block set, populated from the element's block attribute or the schema's
blockDefault.
XSTS invalid-schemas-accepted 179 to 163, no other bucket moved.
An element restricting a base model group is treated as a synthetic group with
minOccurs=maxOccurs=1 holding the element (XSD 1.0 sec 3.9.6), so its {1,1}
occurrence must be within the base group's range: a base group that must occur
two or more times cannot be restricted to a single element. The element's own
occurrence is still matched against the base branch it maps to, so an optional
element restricting a once-or-more base group stays valid.
XSTS invalid-schemas-accepted 181 to 179, no other bucket moved.
The per-iteration invalid-schemas rationale had grown to the 400-line file
limit and blocked further baseline updates. Each rule's exact bucket delta is
recorded per-entry in CHANGELOG.md, so the log is condensed to a pointer plus
the recent steps and the residual-tail note. No behavior change.
A redefine may redefine a given component at most once (src-redefine.7.2.2);
two redefinitions of the same kind and name in one redefine are invalid.
XSTS invalid-schemas-accepted 182 to 181, no other bucket moved.
Once the redefined document is loaded through a schemaLoader, every component a
redefine names (complexType, simpleType, group, attributeGroup) must have a
same-kind component of the same name in the redefined schema (src-redefine.6 /
7.2.1). A redefinition of a component that does not exist there is rejected; a
stand-alone schema (target not loaded) is unaffected.
XSTS invalid-schemas-accepted 183 to 182, no other bucket moved.
An element or attribute default/fixed value is now validated against an inline
type (an inline simpleType, or an inline complex type with simpleContent) the
same way it already was against a named type reference. Validated against the
simpleContent base's value space, so only a clear value-space mismatch (a
non-numeric default on a decimal-based type) is flagged.
XSTS invalid-schemas-accepted 185 to 183, no other bucket moved.
A complex type without its own final attribute now inherits the schema's
finalDefault as its {final}, so finalDefault=#all or restriction/extension
blocks a complexContent restriction or extension derived from it. An explicit
final= overrides the default to empty. Complex types previously honoured only
an explicit final (elements already inherited finalDefault).
XSTS invalid-schemas-accepted 189 to 185, no other bucket moved.
A local element declaration (an element particle, not a direct child of schema)
may not carry abstract, final, or substitutionGroup, which belong to the
top-level element form only. Their presence is invalid at any value, so
abstract=false on a local element is rejected as well as abstract=true.
XSTS invalid-schemas-accepted 194 to 189, no other bucket moved.
A complexContent extension that explicitly states mixed must match its base
type's mixedness (cos-ct-extends.1.4.3.2.2.1): a mixed extension extends only a
mixed base, an element-only extension only an element-only base. An extension
that omits mixed inherits the base's setting and stays valid, so only an
explicit conflict is rejected (ctZ012b, which inherits, is not flagged).
XSTS invalid-schemas-accepted 197 to 194, no other bucket moved.
A complexContent or simpleContent's content model requires exactly one
restriction or extension. An empty <complexContent/> declares no derivation
and is now rejected as a structural error.
XSTS invalid-schemas-accepted 200 to 197, no other bucket moved.
A \p{...}/\P{...} property name that can be no XSD charProp under any
repertoire (empty, the bare prefix Is, a non-block name carrying a non-letter,
or Is followed by a non-block character) is now a schema-validity error,
distinct from a well-formed but unrecognised category or block name, which
stays a lenient engine limitation.
XSTS invalid-schemas-accepted 203 to 200, no other bucket moved.
An attribute ref is a use of a global declaration, so it may not also carry
form (src-attribute.3.2). And no attribute declaration may be named xmlns,
which is reserved for namespace declarations (no-xmlns).
XSTS invalid-schemas-accepted 207 to 203, no other bucket moved.
A referencing attributeGroup (one with a ref) names a definition elsewhere; its
own content model is (annotation?), so it may not also declare attributes,
nested attributeGroup references, or an anyAttribute. Only a defining
attributeGroup (with a name) carries those.
XSTS invalid-schemas-accepted 210 to 207, no other bucket moved.
A notation declaration occupies its own symbol space, so two notations sharing
a name and target namespace are a duplicate just as two global types or
elements would be. Notations were omitted from the uniqueness check.
XSTS invalid-schemas-accepted 211 to 210, no other bucket moved.
A notation declaration's content model is (annotation?): it may carry only an
optional annotation child. Any other element child, or any non-whitespace
character content, is now rejected.
XSTS invalid-schemas-accepted 214 to 211, no other bucket moved.
A list simple type must draw its item type from exactly one of an itemType
attribute or an inline simpleType child; neither, or both, is invalid. A union
must declare at least one member type through a memberTypes attribute or an
inline simpleType child; an empty union declares none (src-simple-type.2/.3).
XSTS invalid-schemas-accepted 215 to 214, no other bucket moved.
A simple type that declares no `final` of its own takes the schema's
`finalDefault` as its `{final}` property (XSD 1.0 Schemas section 3.14.2). The
list and union derivation checks read the `final` attribute directly but
ignored `finalDefault`, so `finalDefault="list"` or `"union"` did not block a
list item type or union member type and the schema was wrongly accepted.
The final map now falls back to `finalDefault` for a type without its own
`final`; an explicit `final=""` still overrides the default back to no
restriction, and a built-in item or member type stays unaffected (it is matched
only when the reference resolves to the schema's own namespace).
Caught two invalid schemas: XSTS invalid-schemas-accepted 217 -> 215, no other
bucket moved.
A `pattern` facet's value must be a valid XSD regular expression. Two
truncation cases were compiled with a non-rejecting error and the schema was
wrongly accepted:
- a trailing `\` with no character to escape (`a\`), and
- a `[` character class the expression ends before closing with `]`
(`a[`, `[a[:xyz:`).
Both are unambiguously invalid, never an engine limitation on an otherwise
valid pattern, so they now raise the dedicated `RegexError.incompleteEscape`
and `RegexError.unterminatedClass` and `rejectsPattern` treats them as
schema-validity errors. A completed escape (`\[`, `\\`), a closed class, and a
class containing an escaped `]` or a subtraction stay valid.
Caught thirteen invalid schemas: XSTS invalid-schemas-accepted 230 -> 217, no
other bucket moved.
An attribute use that references a global attribute carrying a `fixed` value
constraint may not contradict it (au-props-correct.2): a `fixed` on the
reference whose value differs from the global's, or a `default` on a reference
to a fixed attribute, is invalid. A reference that omits its own value
constraint inherits the global's and stays valid.
The two `fixed` values are compared in the attribute type's value space, so a
value-equal pair written in different lexical forms (`01` for `1`) is not
flagged. The global is resolved by local name and only checked once its own
namespace matches the reference's resolved namespace, so a same-local-name
attribute imported from another namespace cannot surface as the wrong global
and contradict a valid reference.
Caught two invalid schemas: XSTS invalid-schemas-accepted 232 -> 230, no other
bucket moved.
Extending a complexType whose whole content is a non-empty xs:all group, via
complexContent extension that adds element content, nests that all in the
sequence joining base and extension content, which cos-all-limited forbids
(an all group must be the whole content model). Such schemas were accepted.
This is the symmetric companion to the existing rule (an all group in the
extension over a base with content); both now live in one pass. An empty-all
base contributes no content, and an attribute-only extension adds none, so
both stay valid.
XSTS invalid-schemas-accepted 233 to 232, no other bucket moved.
A complexContent extension that adds element content (a non-empty model
group) produces an element-only/mixed content type, so its base must have
complex or empty content, not simpleContent (cos-ct-extends.1.4.2.2). Such
schemas were accepted.
The check requires the extension to add actual element content (an element or
wildcard, not an empty group), so an attribute-only complexContent extension
of a simpleContent base stays valid (the complexContent-around-simpleContent
idiom, e.g. particlesZ031).
XSTS invalid-schemas-accepted 235 to 233, no other bucket moved.
Two schema-for-schemas structural constraints were not enforced, so the
malformed schemas were accepted:
- The compositor of a top-level named xs:group (all/choice/sequence) may not
carry minOccurs/maxOccurs; those belong on a group reference, not the
definition.
- An attribute in the XML Schema namespace (e.g. xsd:type) is never valid on a
schema element, since the vocabulary defines no namespaced attributes. A
foreign attribute in any other namespace stays the author's own.
XSTS invalid-schemas-accepted 244 to 235, no other bucket moved. (A third
candidate, form on a top-level declaration, was already enforced by the
existing top-level attribute rule, so no new check was needed.)
A pattern facet with an unambiguously malformed regular expression was
compiled leniently and the schema accepted. Three such constructs are now
rejected: a reversed character-class range ([b-a]), an empty class ([]),
and a reversed bounded quantifier ({37,17}). The parser raises distinct
RegexError cases (reversedRange, emptyClass, reversedQuantifier) that the
schema-validity check treats as invalid.
Constructs the engine merely does not support (an untabulated \p{Is...}
block, the lenient {,m} form) stay non-rejecting, so a valid pattern is
never refused. XSTS invalid-schemas-accepted 262 to 244, no other bucket
moved; the 27 regex unit tests still pass.
An identity-constraint attribute field's type was resolved only among the
constraint host's descendant elements, so when the selector targets a global
element referenced from the host (e.g. unique over `.//uid` with field
`@val`), the field type was unknown and values compared lexically. Two keys
equal in value space but not lexically (3.0 and 3 for xsd:decimal) were then
treated as distinct and the duplicate was accepted.
The attribute type now resolves through the selector's target global element,
so values compare in their value space. XSTS invalid-instances-accepted
133 to 132 (idF010), no other bucket moved.
Identity-constraint selector and field XPath prefixes were resolved against
the instance document's namespace bindings, so a constraint declared with
xmlns:v="..." and a selector like .//v:vehicle matched nothing when the
instance used a default namespace instead of the v prefix. The selector then
selected no nodes, no key tuples were collected, and duplicate-key and
keyref violations went undetected.
Prefixes now resolve against the namespace bindings in scope where the
constraint is declared in the schema (XSD Part 1, Identity-constraint
Definition), captured at parse time onto IdentityConstraint. A QName field
value still compares using the instance's bindings, so QName keys are
unaffected.
XSTS invalid-instances-accepted 143 to 133: the idc004/5/6 namespaced
key/keyref cluster and namespace-resolution cases (idZ010 and kin), no other
bucket moved.
An identity-constraint field must evaluate to a node-set with at most one
member (cvc-identity-constraint.3). The validator built a value tuple per
target without checking how many nodes each field selected, so an instance
whose unique/key/keyref field matched several nodes was accepted.
fieldCardinalityError now rejects a target whose field selects more than one
node, in both the key/unique and keyref paths. XSTS invalid-instances-accepted
4 → fewer (147 to 143), no other bucket moved.
Schema documents that form an import chain are now compiled as a single
union (Document.init(composing:)) rather than as separately compiled
documents merged after the fact. Merging froze each document's content
models with only the substitution-group members visible in its own import
closure, so a substitution member declared in an importing document never
reached the head's content model in the imported document.
This clears XSTS ctZ007, a cross-document substitution group, taking
valid-instances-rejected (#146) from 4 to 3 with no other bucket moving
(now 1/262/3/147). The union pools each document's container closure into
one context, deduplicating a document supplied directly and also reached
through another's import by target namespace and component-name set.
The XSTS runner uses the union only for import chains, detected by matching
import/include/redefine schemaLocation basenames against the listed
documents; independent roots stay on the lenient per-document merge, which a
union would wrongly trip with cross-document checks.
The remaining three valid-instance rejections are the Ethiopic-digit \d
cases (reS17, reS38, reZ004v), which contradict \d = \p{Nd}.
Consolidates the in-progress #146 work; valid-instances-rejected reaches 4
on the 2006-11-06 XSTS archive (buckets now 1/262/4/147).
Regex: match Unicode scalars rather than grapheme clusters; the empty
pattern matches only the empty string; pin the category repertoire to
Unicode 3.2, the version the corpus is consistent at, so code points added
in 3.2 (Tagalog, Cyrillic Komi) are word characters while later additions
(the Tamil digit zero) stay unassigned and match \D. New XSDCategory and a
3.2 assignment table derived from DerivedAge, replacing the old Unicode31
files.
Content model: assess each child against the particle the content automaton
matched, not a by-name lookup (#180). Two same-named particles keep their
own fixed values, and an element matching a skip wildcard is no longer
validated against its global declaration. Selection is made reproducible
when an ambiguous schema slips past the UPA check.
XSTS runner: compile and merge every schemaDocument in a schemaTest, not
just the first; this also surfaced one previously-masked invalid-instance
acceptance (recorded in the baselines).
Also folds in attribute-group anyAttribute inheritance, extension wildcard
union, namespace-qualified type tables for multi-document
xsi:schemaLocation, per-container elementFormDefault, strict and lax
attribute wildcard validation, built-in type resolution, and xs:redefine
simpleType overrides.
Schema compile findings now carry ValidationError coding paths that resolve to source spans on a ranged tree, enabling IDE underlines for id, structure, include, and facet violations.
Ship the OpenAPIKit-style validation layer (builtin rules, warnings channel,
Validator tiers, coverage gates) and wire XSD compile checks through it. Advance
#161 with chameleon include validation, multi-container reference resolution,
and include targetNamespace composition rules so merged schema sets compile
correctly when a schemaLoader is provided.
Document the restructured GitHub issue hierarchy (epic #167, gates,
milestones M1-M5, task labels) and point README Status at the current
production-readiness focus and v0.2.0 XSTS baselines.
Cut the second pre-release: schema-validity burn-down to XSTS baselines
1/266/171/155, component child-content shape checks, and synced conformance
docs. CHANGELOG updated for v0.2.0.
[](https://github.com/mihaelamj/PureXML/actions/workflows/style.yml)
[](https://git.aleahim.com/NodetteHQ/PureXML/actions?workflow=style.yml)
BaseURI["#138 per-entity base URIs: complete"]:::done
Epic122["#122 official spec suites"]:::epic
XSTS["#129 W3C XSD test suite (XSTS): first measurement 2026-06-13 baselined (deviations 75/2467/604/582); occurrence-explosion and reversed-range crashes fixed, suite green in 8s"]:::review
XSLTTS["#130 XSLT 1.0 suite (xalan-test): runner + baselines in, burning down (125)"]:::review
Epic122["#122 conformance suite maintenance: complete (#130 survivor)"]:::epic
XSTS["#129 XSTS runner shipped; gates #145-#148 at 0/39/0/26"]:::review
XSLTTS["#7 XSLT 1.0 suite (xalan-test): 26 pinned baseline failures"]:::review
Epic195["#195 xsi:type and substitution-group derivation correctness"]:::epic
Epic105 --> Epic121
Epic105 --> Epic122
Epic121 --> Oasis
@ -124,13 +135,21 @@ flowchart TB
Epic122 --> XSLTTS
Epic122 --> RNGTS
Epic122 --> C14NTS
Epic122 --> Epic167
Epic167 --> Oracle171
Epic167 --> CrossDoc161
Epic167 --> Gates145
Epic167 --> Loc169
Epic167 --> Epic195
```
## Status
**Current focus:** production readiness for IDE use ([#3](https://git.aleahim.com/NodetteHQ/PureXML/issues/3)). XSTS settled baselines (valid schemas rejected / invalid schemas accepted / valid instances rejected / invalid instances accepted), enforced in `Tests/XSTSSuiteTests.swift`: **0 / 8 / 0 / 4**. See [`docs/roadmap.md`](docs/roadmap.md) for milestones and open tasks.
PureXML is a working, dependency-free XML library today: parse, emit, validate,
query, and stream documents on macOS, Linux, Windows, and WASM. The test suite
currently runs **986 tests in 169 suites** (`swift test`).
currently runs **1448 tests in 258 suites** (`swift test`).
### Shipped (libxml2-aligned surface)
@ -143,8 +162,10 @@ currently runs **986 tests in 169 suites** (`swift test`).
(libxml2 `xmlTextReader`). Handles elements, attributes, text, predefined and
numeric entities, comments, CDATA, and processing instructions. Namespace
bindings (`xmlns`) are resolved to URIs on parse.
- **Encoding**: UTF-8/16/32 (with BOM sniffing), ISO-8859-1, and Windows-1252
from raw bytes or streaming byte input.
- **Encoding**: UTF-8/16/32 (with BOM sniffing) from raw bytes or streaming byte
input, plus the ISO-8859 single-byte family, the Windows code pages
(1250-1258), KOI8-R/U, and the CJK multi-byte encodings (Shift-JIS,
EUC-JP/KR/TW, GBK, GB18030, Big5, ISO-2022-JP).
- **DTD (opt-in)**: `<!DOCTYPE>` is **off by default** (`Limits.allowDoctype:
false`) to keep XXE and entity-expansion closed. When enabled, the internal
subset is parsed; internal general entities expand with a bounded cap; external
@ -193,25 +214,23 @@ subset validation or internal entities, and wire a resolver only for trusted,
in-memory replacements. Network fetching (`nanohttp`/`nanoftp`) and libxml2's
threading/memory infrastructure are deliberate non-goals.
### Remaining toward full libxml2 feature parity
### Remaining work
The original parity epics (the core I/O stack, query, transform, the schema
languages, the validation framework, and editor integration) are all shipped. The
remaining work is the depth gaps a read-only conformance audit surfaced, tracked
under epic **#71** (and its parser-breadth sub-epic **#74**):
Feature parity with libxml2's documented surface is **complete**: the core I/O
stack, the query and transform layers, all four schema languages, the validation
framework, editor integration, and the encoding set (the ISO-8859 family, the
Windows code pages, and the CJK multi-byte encodings) are shipped. The
beyond-parity audit epics (**#71**, its parser-breadth sub-epic **#74**) and the
parity-frontier epic (**#105**) are closed.
| Open work | Issues | What it adds |
|---|---|---|
| XSLT 1.0 breadth | #81#82 | Missing top-level elements and functions, html output |
| HTML5 | #83#84 | Full tree construction, tokenizer states, entity table |
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
