PureXML/docs/xsts-deviations.md

XSTS deviation worklist

This partitions the W3C XML Schema Test Suite (XSTS, 2006-11-06 archive) deviations by root cause and kind, so the burn-down is driven by understanding each rule rather than by chasing the count. First measured 2026-06-13; baselined and ratcheted in Tests/XSTSSuiteTests.swift.

What the oracle measures

The runner compares PureXML's verdict to the suite's expected validity. A schema is "accepted" iff Schema.Document(source) does not throw; an instance is "accepted" iff validate returns no errors. The four counts:

Kind	Meaning	Direction
valid schema rejected	legal XSD that PureXML refused to compile	too strict
invalid schema accepted	illegal XSD that PureXML compiled anyway	too lenient
valid instance rejected	valid document PureXML flagged	too strict
invalid instance accepted	invalid document PureXML passed	too lenient

The oracle is binary and coarse: "invalid schema accepted" conflates we have a bug with we do not implement this validity constraint by design. The point of this document is to separate those.

Only tests with a settled expectation are counted. The suite marks contested entries <current status="queried"> (disputed in W3C Bugzilla, the expected validity itself in question, e.g. \p{Lu}* over two uppercase letters one of which is an astral math capital); agreeing or disagreeing with a contested expectation is not meaningful, so the runner skips them. Excluding the queried entries removed about 70 apparent deviations that were never bugs.

Baseline (post #146 list-facet fix)

Kind	Count
valid schema rejected	75
invalid schema accepted	2467
valid instance rejected	547
invalid instance accepted	555

XSTS is a deliberately adversarial, exhaustive corpus; mature validators (Xerces, libxml2) do not pass it fully. The goal is not a green number. It is to fix the deviations that are genuine correctness bugs affecting real documents, and to document the remainder as spec-justified, deliberate exclusions.

Category A: genuine correctness bugs (fix; real-data impact)

These are cases where PureXML mishandles a feature it does support. Each is a real bug a user could hit with a well-formed schema and document.

Cluster(s)	Count	Root cause	Locus	Status
list-NMTOKENS/IDREFS/ENTITIES length facets	~84	length on a built-in list counted characters, not items	XSDSimpleParser.simpleType	fixed (#146)
union-*-pattern, union-*-enumeration	~200	a union restricted by enumeration/pattern does not enforce the facet (value outside the enumeration is accepted)	SimpleType.validateUnion	open, one fix, high value
atomic-duration-minInclusive/maxInclusive	~50	duration ordering bounds not enforced	duration comparison	open
reL, reI, RegexTest (instance)	~155	missing Unicode block escapes (\p{IsArmenian} etc.) and category coverage in the regex engine	Sources/Regex	open, feature completion
atomic-QName-length, -minLength	~36	character-length facet wrongly applied to QName; the spec/NIST treat QName length as non-constraining (Xerces agrees)	length facet on QName	open, over-strict
valueconstraint, isDefault	~26	default/fixed value handling	element/attr defaults	open, needs sampling
particlesJk/Jj/Ha/Q/R (schema), particlesOb/Je (instance)	~110	content-model construction/validation edge cases	content model	open, bounded
int_min/maxInclusive, int_min/maxExclusive (schema)	~16	integer bound facets rejecting legal schemas	numeric facets	open, bounded

Recommended order: union facets (~200, one fix) → QName length (~36) → regex Unicode blocks (~155) → duration ordering (~50) → particles/int facets.

Category B: schema-for-schemas validity (unimplemented by design)

The bulk of the 2467 "invalid schema accepted". PureXML compiles schemas permissively: it reads the components it understands and ignores structurally illegal content rather than validating the schema document against the schema-for-schemas. Examples found:

• ctB: <complexType>Annotation information</complexType> (bare text content) is accepted.
• notatF: <notation> placed inside <restriction> (illegal position) is accepted.
• idA/idB/idC: malformed identity-constraint definitions accepted.
• annotation/annotB: illegal annotation placement accepted.
• addB, stF, ctG, ctH: assorted illegal component structures accepted.

Cluster(s)	Count
ctB/ctG/ctH (complexType structure)	~183
idA/idB/idC (identity constraints)	~118
notatF (notation placement)	~66
annotation/annotB	~62
addB, stF, and the long tail	remainder of 2467

Decision: enforcing this is a large, low-real-world-value effort (people do not author structurally broken schemas, and a too-lenient compiler does not corrupt valid-document validation). It would require a schema-document validation pass against the schema-for-schemas. Treated as a documented, deliberate exclusion unless a concrete need arises. If implemented, it is its own epic, not part of the instance-correctness burn-down.

Category C: arguable / needs investigation

• test, elemT (invalid instance accepted, ~86): not yet sampled; classify before fixing.

Category D: deliberate interpretations of spec-ambiguous points

These are resolved, not open: the spec underspecifies the point and the suite's own contributors disagree, so a single behavior cannot satisfy all of them. We pick the defensible reading and accept the residual deviation.

• QName length facets are non-constraining. XSD 1.0 Datatypes 4.3.1 defines the unit of length for string (characters), anyURI (characters), binary (octets), and list (items), but not for QName, so it is implementation-defined. NIST (36+ cases) and Xerces treat length/minLength/maxLength on QName as always satisfied; Microsoft (4 cases: QName_length001/003, QName_minLength003, QName_maxLength001) treats them as character counts and expects rejection. These are directly contradictory (NIST marks a 15-character value valid under length=1; MS marks foofo invalid under length=4). We chose non-constraining, matching Xerces and the larger NIST set, because a QName's value is an abstract (namespace, local-name) pair with no defined character length. Cost: the 4 MS cases remain in "invalid instances accepted". Fixing the 36+ NIST false rejections (valid documents wrongly rejected) is the more important direction.

How a fix lands

Pick a Category A cluster, read 2-3 cases (schema + instance + expected) against the spec text, fix the owning code, add a focused regression test, re-run swift test -c release --filter XSTS, confirm the relevant count drops, and ratchet the baseline in Tests/XSTSSuiteTests.swift down to the new value. Tracking issues: #145 (invalid schema accepted), #146 (valid instance rejected), #147 (invalid instance accepted), #148 (valid schema rejected).