Mihaela Mihaljevic
aa7cbb71f8
Continue the facet-validity work. Per XSD Part 2 Datatypes 4.3, each minInclusive/maxInclusive/minExclusive/maxExclusive and enumeration value must be a valid value of the base type; minInclusive excludes minExclusive and maxInclusive excludes maxExclusive; and the lower bound may not exceed the upper bound (nor equal it when either side is exclusive, an empty value space), compared in the base primitive's value space. Bound and enumeration values were previously unchecked, so a malformed or out-of-space value (maxInclusive="", enumeration="" on float) was accepted. The bound-value check (which validates a bound against its base type) exposed a gMonth lexical gap: only --MM was recognized, not the XSD 1.0 form --MM-- used by valid schemas across the corpus. parseMonthOnly now accepts an optional trailing -- (the --MM erratum form still works), with an optional timezone, which removes 26 false schema rejections the bound check would otherwise have caused and also fixes instance-side gMonth validation. XSTS: invalid schemas accepted 2082 to 1789; valid instances rejected 233 to 214 (the gMonth fix); valid schemas rejected held at 72; invalid instances accepted held at 165.
8.6 KiB
XSTS deviation worklist
This partitions the W3C XML Schema Test Suite (XSTS, 2006-11-06 archive)
deviations by root cause and kind, so the burn-down is driven by
understanding each rule rather than by chasing the count. First measured
2026-06-13; baselined and ratcheted in Tests/XSTSSuiteTests.swift.
What the oracle measures
The runner compares PureXML's verdict to the suite's expected validity. A
schema is "accepted" iff Schema.Document(source) does not throw; an instance
is "accepted" iff validate returns no errors. The four counts:
| Kind | Meaning | Direction |
|---|---|---|
| valid schema rejected | legal XSD that PureXML refused to compile | too strict |
| invalid schema accepted | illegal XSD that PureXML compiled anyway | too lenient |
| valid instance rejected | valid document PureXML flagged | too strict |
| invalid instance accepted | invalid document PureXML passed | too lenient |
The oracle is binary and coarse: "invalid schema accepted" conflates we have a bug with we do not implement this validity constraint by design. The point of this document is to separate those.
Only tests with a settled expectation are counted. The suite marks contested
entries <current status="queried"> (disputed in W3C Bugzilla, the expected
validity itself in question, e.g. \p{Lu}* over two uppercase letters one of
which is an astral math capital); agreeing or disagreeing with a contested
expectation is not meaningful, so the runner skips them. Excluding the queried
entries removed about 70 apparent deviations that were never bugs.
Baseline (settled expectations; after twelve fixes)
| Kind | First measurement | Now |
|---|---|---|
| valid schema rejected | 75 | 72 |
| invalid schema accepted | 2467 | 2461 |
| valid instance rejected | 604 | 308 |
| invalid instance accepted | 582 | 165 |
Counts are against settled W3C expectations (queried/disputed entries excluded).
XSTS is a deliberately adversarial, exhaustive corpus; mature validators (Xerces, libxml2) do not pass it fully. The goal is not a green number. It is to fix the deviations that are genuine correctness bugs affecting real documents, and to document the remainder as spec-justified, deliberate exclusions.
Category A: genuine correctness bugs
Fixed (sixteen root causes, ~1346 deviations cleared)
list-datatype length facets (#146) · union pattern/enumeration facets · duration
partial-order facets · QName length non-constraining · the full XSD \p{Is...}
Unicode block set · element-level block against xsi:type · empty-element
default/fixed values + enumeration-restriction-replaces · block="substitution"
on substitution-group heads · \w/\W XSD definition · document-scoped
xs:ID/xs:IDREF uniqueness and resolution · <xs:attribute ref> resolution ·
mixed-content type with no element model · built-in xsi:type resolution gated
by substitution validity (the XSD Part 2 derivation lattice), which also closed
the two gaps it exposed (a simple-typed element rejecting stray attributes;
identity fields comparing in value space when the node carries an xsi:type) ·
compile-time constraining-facet definition validity (length-family facets are
nonNegativeInteger, co-occurrence and range order) · value-bound facet
validity (min/maxInclusive, min/maxExclusive, enumeration values valid
in the base space; inclusive/exclusive exclusions; bound ordering) · gMonth
--MM-- lexical form. The schema-validity work brought invalid schemas accepted
2461 to 1789; see schema-validity-burndown.md.
Plus the measurement fix (exclude W3C-disputed status="queried" entries).
Remaining, root-caused (the hard tail)
Each is a substantial, self-contained study, not a one-locus cluster fix:
| Cluster(s) | ~Count | Root cause (diagnosed) | Difficulty |
|---|---|---|---|
particlesOb/Je/Jf, related |
~30 | <xs:any> defaults to processContents="strict"; the matched element's declaration lives in a separate schema document referenced only by the instance xsi:schemaLocation (no <import>). Needs multi-document schema loading + xsi:schemaLocation honoring, plus strict-wildcard semantics and Particle-Valid-Restriction. |
large (new capability) |
reI |
~10 | pattern values containing literal tab/newline: XML attribute-value normalization (tab/newline to space) versus the value's whiteSpace facet. The "right" pattern text depends on that interaction. |
subtle |
test, elemT |
~65 | heterogeneous grab-bags (MS "Additional"/element tests): many distinct minor causes, not one root cause. Each case needs individual triage. | per-case grind |
idF, idc remainder |
~14 | identity-constraint selector/field XPath edge cases beyond the implemented core. The value-space comparison via instance xsi:type and built-in xsi:type resolution now land; what remains is the declared-type-driven value-space comparison (needs PSVI type assignment, since the field's declared simple type is not threaded into identity validation) and selector/field XPath edges. |
moderate |
wildG, stZ, addB |
~25 | wildcard-in-group composition, simple-type edge cases, attribute-default edges. | mixed |
The clean, single-root-cause, high-leverage fixes are exhausted. Reaching zero on the instance buckets requires the multi-document-schema capability (the largest single lever, ~30 cases) and per-case work on the grab-bags; neither is a quick fix, and rushing them would violate the project's no-shortcuts ethic.
Category B: schema-for-schemas validity (unimplemented by design)
The bulk of the 2467 "invalid schema accepted". PureXML compiles schemas permissively: it reads the components it understands and ignores structurally illegal content rather than validating the schema document against the schema-for-schemas. Examples found:
ctB:<complexType>Annotation information</complexType>(bare text content) is accepted.notatF:<notation>placed inside<restriction>(illegal position) is accepted.idA/idB/idC: malformed identity-constraint definitions accepted.annotation/annotB: illegal annotation placement accepted.addB,stF,ctG,ctH: assorted illegal component structures accepted.
| Cluster(s) | Count |
|---|---|
ctB/ctG/ctH (complexType structure) |
~183 |
idA/idB/idC (identity constraints) |
~118 |
notatF (notation placement) |
~66 |
annotation/annotB |
~62 |
addB, stF, and the long tail |
remainder of 2467 |
Decision: enforcing this is a large, low-real-world-value effort (people do not author structurally broken schemas, and a too-lenient compiler does not corrupt valid-document validation). It would require a schema-document validation pass against the schema-for-schemas. Treated as a documented, deliberate exclusion unless a concrete need arises. If implemented, it is its own epic, not part of the instance-correctness burn-down.
Category C: arguable / needs investigation
test,elemT(invalid instance accepted, ~86): not yet sampled; classify before fixing.
Category D: deliberate interpretations of spec-ambiguous points
These are resolved, not open: the spec underspecifies the point and the suite's own contributors disagree, so a single behavior cannot satisfy all of them. We pick the defensible reading and accept the residual deviation.
- QName length facets are non-constraining. XSD 1.0 Datatypes 4.3.1 defines
the unit of length for string (characters), anyURI (characters), binary
(octets), and list (items), but not for QName, so it is implementation-defined.
NIST (36+ cases) and Xerces treat
length/minLength/maxLengthon QName as always satisfied; Microsoft (4 cases:QName_length001/003,QName_minLength003,QName_maxLength001) treats them as character counts and expects rejection. These are directly contradictory (NIST marks a 15-character value valid underlength=1; MS marksfoofoinvalid underlength=4). We chose non-constraining, matching Xerces and the larger NIST set, because a QName's value is an abstract (namespace, local-name) pair with no defined character length. Cost: the 4 MS cases remain in "invalid instances accepted". Fixing the 36+ NIST false rejections (valid documents wrongly rejected) is the more important direction.
How a fix lands
Pick a Category A cluster, read 2-3 cases (schema + instance + expected) against
the spec text, fix the owning code, add a focused regression test, re-run
swift test -c release --filter XSTS, confirm the relevant count drops, and
ratchet the baseline in Tests/XSTSSuiteTests.swift down to the new value.
Tracking issues: #145 (invalid schema accepted), #146 (valid instance
rejected), #147 (invalid instance accepted), #148 (valid schema rejected).