PureXML/docs/xsts-deviations.md

XSTS deviation worklist

This partitions the W3C XML Schema Test Suite (XSTS, 2006-11-06 archive) deviations by root cause and kind, so the burn-down is driven by understanding each rule rather than by chasing the count. First measured 2026-06-13; baselined and ratcheted in Tests/XSTSSuiteTests.swift.

What the oracle measures

The runner compares PureXML's verdict to the suite's expected validity. A schema is "accepted" iff Schema.Document(source) does not throw; an instance is "accepted" iff validate returns no errors. The four counts:

Kind	Meaning	Direction
valid schema rejected	legal XSD that PureXML refused to compile	too strict
invalid schema accepted	illegal XSD that PureXML compiled anyway	too lenient
valid instance rejected	valid document PureXML flagged	too strict
invalid instance accepted	invalid document PureXML passed	too lenient

The oracle is binary and coarse: "invalid schema accepted" conflates we have a bug with we do not implement this validity constraint by design. The point of this document is to separate those.

Only tests with a settled expectation are counted. The suite marks contested entries <current status="queried"> (disputed in W3C Bugzilla, the expected validity itself in question, e.g. \p{Lu}* over two uppercase letters one of which is an astral math capital); agreeing or disagreeing with a contested expectation is not meaningful, so the runner skips them. Excluding the queried entries removed about 70 apparent deviations that were never bugs.

Baseline (settled expectations; synced 2026-06-16)

Kind	First measurement	Now
valid schema rejected	75	1
invalid schema accepted	2467	264
valid instance rejected	604	173
invalid instance accepted	582	155

Counts are against settled W3C expectations (queried/disputed entries excluded).

Current ratchet (2026-06-19): valid schema rejected = 0, invalid schema accepted = 43, valid instance rejected = 0, invalid instance accepted = 31. The historical table above is retained as the published baseline snapshot; the active burn-down is in docs/plans/2026-06-18-false-negative-tail.md.

XSTS is a deliberately adversarial, exhaustive corpus; mature validators (Xerces, libxml2) do not pass it fully. The goal is not a green number. It is to fix the deviations that are genuine correctness bugs affecting real documents, and to document the remainder as spec-justified, deliberate exclusions.

libxml2 differential triage (2026-06-26)

With the corpus runnable, the open tail (10 invalid-schemas-accepted, 14 invalid-instances-accepted) was triaged against libxml2 (xmllint --schema), the package's stated reference. A case libxml2 also accepts is one where matching the suite means being stricter than libxml2 (Xerces/.NET territory): a deliberate scope choice with false-positive risk. A case libxml2 rejects is a genuine gap whose fix aligns PureXML with its reference rather than exceeding it.

Schema side (10 cases):

Case	libxml2	Reading
particlesZ030_d	rejects	Reference-aligned bug, now fixed: a simpleContent restriction relaxing a base required attribute to optional (derivation-ok-restriction.2). invalid-schemas-accepted 10 -> 9.
elemZ026, particlesHa161, particlesIg004, particlesIk011/025/027, particlesM034, particlesV020	accept too	Stricter-than-libxml2: NameAndTypeOK / particle-restriction edges (union membership, element-block superset) that libxml2 does not implement. Pursue only if Xerces-level strictness is wanted.
schN12	(multi-document)	Cross-namespace composition; not triaged in isolation.

Instance side (sampled): idZ010, mgO013, particlesZ001, reT17, reT38 are rejected by libxml2 (reference-aligned). reT17/reT38 are now fixed: they are the \D half of the Ethiopic-digit \d bug. XSD pins the regex repertoire at Unicode 3.2, where U+1369..U+1371 are Nd (reclassified to No later), but XSDCategory's per-code-point override table omitted them, so \d used the modern No. Pinning the nine to Nd makes reT17/reT38 reject (invalid-instances-accepted 14 -> 12) and the matching \d-valid cases reS17/reS38/reZ004 pass without their former suspect-test exclusion; this is the complete Nd 3.2-vs-now delta, and \w is unaffected. reZ006i (the \c case) is also now fixed: \i/\c were wired to the parser's XML 1.0 Fifth Edition name check, but XSD defines them against the Second Edition (Appendix B) classes; the new XSDNameCharacter uses the Appendix B ranges, so the Fifth-Edition-only combining mark U+0346 no longer matches \c (invalid-instances-accepted 12 -> 11). The remaining intricate cases: particlesZ001's rule is marked ambiguous by the suite itself (RecurseAsIfGroup); mgO013 is an xs:all under redefine; idZ010 turns on keyref namespace resolution with non-absolute namespace URIs. The multi-document sch* / targetns cases need per-schema pairing before a verdict.

anyURI_a004_1339.i is reclassified as a contested test, not a deviation (invalid-instances-accepted 11 -> 10, now excluded like the suite's queried entries). The suite pairs a valid schemaTest with an invalid instanceTest, but the two are mutually inconsistent: the schema's include/import/redefine have unresolvable schemaLocations, which XSD 1.0 §4.2.1 makes "not an error" (no inclusion performed; the empty redefine imposes no resolution requirement per src-redefine clause 1), so the schema is conformantly valid, and the only components the instance exercises (root/bar/st/att) are fully locally defined with every value byte-identical to one of the type's six anyURI enumeration members, so the instance is valid. There is no spec path to valid-schema + invalid-instance. Two independent adversarial analyses failed to refute this; libxml2 is no authority (it non-conformantly fails to compile the schema on the unresolvable include). Matching the suite would reject a valid instance whenever an include cannot be resolved, a false positive an IDE hits constantly. Recorded in XSTSSuiteTests.specDivergentInstances.

The sch* schema-location cluster, dissected per case (an adversarial analysis separated the contested from the genuine):

• schA2.i, schA5.i: contested, reclassified (invalid-instances-accepted 10 -> 8). Each group's own documentation says the inline xsi:schemaLocation hint "is ignored" and the instance "must conform to" the collection schema; under exactly that behavior libxml2 validates the instance (schA2 against a.xsd, schA5 against a.xsd+b.xsd), so the invalid expectation contradicts the group's own documentation. Reaching invalid needs the instance's hint to override a schema already loaded for that namespace, which XSD 1.0 §4.3.2 makes optional (strategy 1) and which would be a direct IDE false positive. Excluded in specDivergentInstances.
• schA7.i: genuine, policy-dependent deviation, left in the count. Here the invalid IS spec-grounded: with hints ignored, the instance's b:e1 (namespace ns-b) has no declaration in the a.xsd-only collection and is demanded by root's default-strict ##any wildcard, so §3.10.4 makes it invalid. PureXML accepts it only by honoring the ns-b hint to gap-fill the wildcard, the other conformant §4.3.2 strategy. A real divergence in schema-location policy, not a suite bug; not excluded.
• schU3.i, schU4.i, schU5.i: genuine under-rejection in an underspecified area, deferred. These are circular redefine chains where one attribute group (c-g1) is redefined to conflicting types (int via one document, string via another) across an include/redefine cycle. The suite expects the int reading, so the instance's c1="a" is invalid; PureXML resolves to the laxer type and accepts. libxml2 does not support this at all (it rejects the schema: "Multiple redefinition of the same component is not supported"), so matching the suite means out-resolving the reference on a construct whose composition order in a cycle the spec does not pin down. Tracked as deferred scope, not excluded.

Takeaway: on the schema side, libxml2 parity is reached except the one case now fixed; the remaining nine are a deliberate choice to exceed the reference. On the instance side the Ethiopic-digit \d/\D and \i/\c name-class cases are now fixed (reference-aligned) and anyURI_a004_1339 is shown contested; the remaining reference-aligned cases are real but each needs its own analysis, not a sweep.

Category A: genuine correctness bugs

Fixed (thirty-four root causes, ~2258 deviations cleared)

list-datatype length facets (#146) · union pattern/enumeration facets · duration partial-order facets · QName length non-constraining · the full XSD \p{Is...} Unicode block set · element-level block against xsi:type · empty-element default/fixed values + enumeration-restriction-replaces · block="substitution" on substitution-group heads · substitution-group closure stops chaining through members whose own block contains substitution · \w/\W XSD definition · document-scoped xs:ID/xs:IDREF uniqueness and resolution · <xs:attribute ref> resolution · mixed-content type with no element model · built-in xsi:type resolution gated by substitution validity (the XSD Part 2 derivation lattice), which also closed the two gaps it exposed (a simple-typed element rejecting stray attributes; identity fields comparing in value space when the node carries an xsi:type) · compile-time constraining-facet definition validity (length-family facets are nonNegativeInteger, co-occurrence and range order) · value-bound facet validity (min/maxInclusive, min/maxExclusive, enumeration values valid in the base space; inclusive/exclusive exclusions; bound ordering) · substitution-group heads typed xs:anySimpleType only admitting simple and simpleContent members · global element refs carrying declaration metadata into Particle-Valid-Restriction · gMonth --MM-- lexical form · schema id attribute validity (xs:ID: NCName and unique within the document). The schema-validity work brought invalid schemas accepted 2461 to 43 (schema-document structural validity: content model + complexType content shape, attribute values, name/reference + pattern-regex lexical validity, and reference resolution, against the schema-for-schemas, plus simpleType variety constraints on list itemType and union memberTypes, plus targetNamespace scope matching on local schema references, plus substitution group exclusions based on element final and schema finalDefault attributes, duplicate schema container validation across separate files with identical content, schema element order, import constraints validation, and the later tail fixes); see schema-validity-burndown.md. Plus the measurement fix (exclude W3C-disputed status="queried" entries).

Remaining, root-caused (the hard tail)

Each is a substantial, self-contained study, not a one-locus cluster fix:

Cluster(s)	~Count	Root cause (diagnosed)	Difficulty
particlesOb/Je/Jf, related	~30	<xs:any> defaults to processContents="strict"; the matched element's declaration lives in a separate schema document referenced only by the instance xsi:schemaLocation (no <import>). Needs multi-document schema loading + xsi:schemaLocation honoring, plus strict-wildcard semantics and Particle-Valid-Restriction.	large (new capability)
reI	~10	pattern values containing literal tab/newline: XML attribute-value normalization (tab/newline to space) versus the value's whiteSpace facet. The "right" pattern text depends on that interaction.	subtle
test, elemT	~65	heterogeneous grab-bags (MS "Additional"/element tests): many distinct minor causes, not one root cause. Each case needs individual triage.	per-case grind
idF, idc remainder	~14	identity-constraint selector/field XPath edge cases beyond the implemented core. The value-space comparison via instance xsi:type and built-in xsi:type resolution now land; what remains is the declared-type-driven value-space comparison (needs PSVI type assignment, since the field's declared simple type is not threaded into identity validation) and selector/field XPath edges.	moderate
wildG, stZ, addB	~25	wildcard-in-group composition, simple-type edge cases, attribute-default edges.	mixed

The clean, single-root-cause, high-leverage fixes are exhausted. Reaching zero on the instance buckets requires the multi-document-schema capability (the largest single lever, ~30 cases) and per-case work on the grab-bags; neither is a quick fix, and rushing them would violate the project's no-shortcuts ethic.

Category B: schema-for-schemas validity (unimplemented by design)

The bulk of the 2467 "invalid schema accepted". PureXML compiles schemas permissively: it reads the components it understands and ignores structurally illegal content rather than validating the schema document against the schema-for-schemas. Examples found:

• ctB: <complexType>Annotation information</complexType> (bare text content) is accepted.
• notatF: <notation> placed inside <restriction> (illegal position) is accepted.
• idA/idB/idC: malformed identity-constraint definitions accepted.
• annotation/annotB: illegal annotation placement accepted.
• addB, stF, ctG, ctH: assorted illegal component structures accepted.

Cluster(s)	Count
ctB/ctG/ctH (complexType structure)	~183
idA/idB/idC (identity constraints)	~118
notatF (notation placement)	~66
annotation/annotB	~62
addB, stF, and the long tail	remainder of 2467

Status (revised): now being implemented as the general schema-for-schemas mechanism, not a per-rule pile. The content-model pass has landed (id validity, facet-definition validity, and the allowedChildren structural table), driving invalid-schemas-accepted from 2461 to 1313. The remaining structural work (the component attribute model, child order, particle semantics) continues under schema-validity-burndown.md. A too-lenient compiler does not corrupt valid-document validation, but rubber-stamping invalid schemas is the largest conformance gap, so it is being closed rather than excluded.

Category C: arguable / needs investigation

• test, elemT (invalid instance accepted, ~86): not yet sampled; classify before fixing.

Category D: deliberate interpretations of spec-ambiguous points

These are resolved, not open: the spec underspecifies the point and the suite's own contributors disagree, so a single behavior cannot satisfy all of them. We pick the defensible reading and accept the residual deviation.

• QName length facets are non-constraining. XSD 1.0 Datatypes 4.3.1 defines the unit of length for string (characters), anyURI (characters), binary (octets), and list (items), but not for QName, so it is implementation-defined. NIST (36+ cases) and Xerces treat length/minLength/maxLength on QName as always satisfied; Microsoft (4 cases: QName_length001/003, QName_minLength003, QName_maxLength001) treats them as character counts and expects rejection. These are directly contradictory (NIST marks a 15-character value valid under length=1; MS marks foofo invalid under length=4). We chose non-constraining, matching Xerces and the larger NIST set, because a QName's value is an abstract (namespace, local-name) pair with no defined character length. Cost: the 4 MS cases remain in "invalid instances accepted". Fixing the 36+ NIST false rejections (valid documents wrongly rejected) is the more important direction.

• RecurseAsIfGroup for a repeating element against a repeating choice (particlesZ001). A derived element{0,unbounded} restricting a base choice{0,unbounded}(element, any). The Microsoft test marks it valid, and its own documentation states the reason: "Particle Derivation OK (Elt:Choice -- RecurseAsIfGroup) rule is ambiguous." Under the literal §3.9.6 rule (which Xerces follows) the element is wrapped as a one-member {1,1} synthetic group whose member keeps the element's {0,unbounded} occurrence, and that member must be a valid restriction of a {1,1} choice branch; {0,unbounded} does not fit {1,1}, so the strict reading rejects it. We follow the strict reading (matching the reference implementation), accepting this one residual "valid schema rejected". Adopting the lenient reading would require a special case that cannot be expressed without over-accepting genuinely invalid restrictions on PureXML's content-model representation (an attempt to do so regressed roughly thirty other cases). Tracked on #165. The sibling particlesV003 (Sequence:Choice MapAndSum) was a genuine bug, since fixed.

How a fix lands

Pick a Category A cluster, read 2-3 cases (schema + instance + expected) against the spec text, fix the owning code, add a focused regression test, re-run swift test -c release --filter XSTS, confirm the relevant count drops, and ratchet the baseline in Tests/XSTSSuiteTests.swift down to the new value. Tracking issues: #145 (invalid schema accepted), #146 (valid instance rejected), #147 (invalid instance accepted), #148 (valid schema rejected).

Historical autopilot accounting (2026-06-18): labelled-suite invalid-schemas-accepted = 74

At this earlier snapshot, the XSTSSuiteTests ratchet (settled-expectation labelled subset) stood at 0 valid-schemas-rejected, 74 invalid-schemas-accepted, 0 valid-instances-rejected, 133 invalid-instances-accepted. False positives have been held at 0 across the whole campaign (the hard gate). The remaining 74 were the hard tail at that snapshot. They partitioned as:

Fixable debt (genuine bugs, but intricate and false-positive-prone)

Cluster	~Count	Root cause	Why deferred
schL, schM, attgZ	~12	redefine RESTRICTION: a redefined group/attributeGroup must be a valid restriction of the original (src-redefine.6.1.2/7.2.2); a redefined attributeGroup may not drop or widen an attribute.	reuses ParticleRestriction + attribute-restriction (the most false-positive-prone machinery); needs the original resolved across containers. Fresh focus.
particlesIk (011/025/027)	~3	NameAndTypeOK union membership: a restricting element's type must derive from a member of the base element's union type.	deliberately bypassed earlier to avoid a false positive (string is a valid member of union(decimal|string)); chain-walk cannot model union membership.
wildI, wildZ	~4	Unique Particle Attribution for WILDCARDS: two overlapping <any> (or any vs element) make a non-deterministic content model.	ContentModelDeterminism is the most false-positive-prone area; wildcard overlap must be added carefully.
schN, schI, schZ011, schO, schZ006, schN10	~12	chameleon / cross-namespace include-import-redefine, self-pointing redefine, duplicate-in-redefine.	cross-document namespace semantics; intricate.
particlesEa/Fb/Ha/Hb/K/M/V, elemZ, ctZ, stZ, addB, test, xsd*.e	~28	heterogeneous particle-restriction edges, element derivation/substitution, anySimpleType edges, adhoc grab-bags.	per-case triage; several need PSVI/model enrichment.

Irreducible by design / disclosed (document, do not "fix")

Cluster	~Count	Reason
particlesZ (RecurseAsIfGroup / MapAndSum ambiguities, e.g. Z001)	a few	the test's own documentation calls the rule ambiguous; matches the Xerces strict reading (Category D).
reK, RegexTest_*	~3	the pattern engine deliberately tolerates .unsupported/.badEscape categories to avoid rejecting valid-but-unsupported patterns (a disclosed under-rejection, not a defect).

Process note

Three autopilot rules this session (redefine self-reference, identity-constraint xpath prefix, redefined attributeGroup self-reference count) each passed an adversarial critic pass before merge; the critic caught three genuine false positives the W3C suite and the differential both missed (two namespace-collision, one nesting-boundary), each fixed before merge. Any rule that resolves names or walks references MUST namespace-gate and bound its walk, and MUST go through the critic loop. The intricate clusters above are best executed supervised, one at a time, with that loop.