Mihaela Mihaljevic
d3daea5b91
First cut at the schema-validity gap: the validator ran instance-validity rules but not schema-validity rules, so it silently accepted invalid schemas. Per XSD Part 2 Datatypes 4.3, a length-family facet value (length, minLength, maxLength, totalDigits, fractionDigits) must be a valid nonNegativeInteger (totalDigits a positiveInteger), length may not co-occur with minLength/maxLength, and the integer ranges must be ordered. Malformed values (length="", "-1", "a", "1e2") were silently dropped by numeric facet parsing and the schema accepted. The definition is now read raw from the restriction and validated at compile time. Well-formedness is checked independently of machine-integer range (via a canonical digit string), so a huge but valid value is not misreported as malformed, and ordering compares by length then lexically. Findings travel through a shared reference-type diagnostics collector on the value-type parsing context (surviving its copies) and join the model-level consistency findings thrown from Schema.Document.init. Limited to facets declared on the local restriction, so an inherited facet is never wrongly flagged. The min/maxInclusive value-space rules, facet applicability per base, and inherited co-occurrence remain for later iterations (see docs/schema-validity-burndown.md). XSTS: invalid schemas accepted 2461 to 2082, no change to any other bucket.
8.4 KiB
XSTS deviation worklist
This partitions the W3C XML Schema Test Suite (XSTS, 2006-11-06 archive)
deviations by root cause and kind, so the burn-down is driven by
understanding each rule rather than by chasing the count. First measured
2026-06-13; baselined and ratcheted in Tests/XSTSSuiteTests.swift.
What the oracle measures
The runner compares PureXML's verdict to the suite's expected validity. A
schema is "accepted" iff Schema.Document(source) does not throw; an instance
is "accepted" iff validate returns no errors. The four counts:
| Kind | Meaning | Direction |
|---|---|---|
| valid schema rejected | legal XSD that PureXML refused to compile | too strict |
| invalid schema accepted | illegal XSD that PureXML compiled anyway | too lenient |
| valid instance rejected | valid document PureXML flagged | too strict |
| invalid instance accepted | invalid document PureXML passed | too lenient |
The oracle is binary and coarse: "invalid schema accepted" conflates we have a bug with we do not implement this validity constraint by design. The point of this document is to separate those.
Only tests with a settled expectation are counted. The suite marks contested
entries <current status="queried"> (disputed in W3C Bugzilla, the expected
validity itself in question, e.g. \p{Lu}* over two uppercase letters one of
which is an astral math capital); agreeing or disagreeing with a contested
expectation is not meaningful, so the runner skips them. Excluding the queried
entries removed about 70 apparent deviations that were never bugs.
Baseline (settled expectations; after twelve fixes)
| Kind | First measurement | Now |
|---|---|---|
| valid schema rejected | 75 | 72 |
| invalid schema accepted | 2467 | 2461 |
| valid instance rejected | 604 | 308 |
| invalid instance accepted | 582 | 165 |
Counts are against settled W3C expectations (queried/disputed entries excluded).
XSTS is a deliberately adversarial, exhaustive corpus; mature validators (Xerces, libxml2) do not pass it fully. The goal is not a green number. It is to fix the deviations that are genuine correctness bugs affecting real documents, and to document the remainder as spec-justified, deliberate exclusions.
Category A: genuine correctness bugs
Fixed (fourteen root causes, ~1034 deviations cleared)
list-datatype length facets (#146) · union pattern/enumeration facets · duration
partial-order facets · QName length non-constraining · the full XSD \p{Is...}
Unicode block set · element-level block against xsi:type · empty-element
default/fixed values + enumeration-restriction-replaces · block="substitution"
on substitution-group heads · \w/\W XSD definition · document-scoped
xs:ID/xs:IDREF uniqueness and resolution · <xs:attribute ref> resolution ·
mixed-content type with no element model · built-in xsi:type resolution gated
by substitution validity (the XSD Part 2 derivation lattice), which also closed
the two gaps it exposed (a simple-typed element rejecting stray attributes;
identity fields comparing in value space when the node carries an xsi:type) ·
compile-time constraining-facet definition validity (length-family facets are
nonNegativeInteger, co-occurrence and range order), the first cut at the
schema-validity gap (invalid schemas accepted 2461 to 2082; see
schema-validity-burndown.md).
Plus the measurement fix (exclude W3C-disputed status="queried" entries).
Remaining, root-caused (the hard tail)
Each is a substantial, self-contained study, not a one-locus cluster fix:
| Cluster(s) | ~Count | Root cause (diagnosed) | Difficulty |
|---|---|---|---|
particlesOb/Je/Jf, related |
~30 | <xs:any> defaults to processContents="strict"; the matched element's declaration lives in a separate schema document referenced only by the instance xsi:schemaLocation (no <import>). Needs multi-document schema loading + xsi:schemaLocation honoring, plus strict-wildcard semantics and Particle-Valid-Restriction. |
large (new capability) |
reI |
~10 | pattern values containing literal tab/newline: XML attribute-value normalization (tab/newline to space) versus the value's whiteSpace facet. The "right" pattern text depends on that interaction. |
subtle |
test, elemT |
~65 | heterogeneous grab-bags (MS "Additional"/element tests): many distinct minor causes, not one root cause. Each case needs individual triage. | per-case grind |
idF, idc remainder |
~14 | identity-constraint selector/field XPath edge cases beyond the implemented core. The value-space comparison via instance xsi:type and built-in xsi:type resolution now land; what remains is the declared-type-driven value-space comparison (needs PSVI type assignment, since the field's declared simple type is not threaded into identity validation) and selector/field XPath edges. |
moderate |
wildG, stZ, addB |
~25 | wildcard-in-group composition, simple-type edge cases, attribute-default edges. | mixed |
The clean, single-root-cause, high-leverage fixes are exhausted. Reaching zero on the instance buckets requires the multi-document-schema capability (the largest single lever, ~30 cases) and per-case work on the grab-bags; neither is a quick fix, and rushing them would violate the project's no-shortcuts ethic.
Category B: schema-for-schemas validity (unimplemented by design)
The bulk of the 2467 "invalid schema accepted". PureXML compiles schemas permissively: it reads the components it understands and ignores structurally illegal content rather than validating the schema document against the schema-for-schemas. Examples found:
ctB:<complexType>Annotation information</complexType>(bare text content) is accepted.notatF:<notation>placed inside<restriction>(illegal position) is accepted.idA/idB/idC: malformed identity-constraint definitions accepted.annotation/annotB: illegal annotation placement accepted.addB,stF,ctG,ctH: assorted illegal component structures accepted.
| Cluster(s) | Count |
|---|---|
ctB/ctG/ctH (complexType structure) |
~183 |
idA/idB/idC (identity constraints) |
~118 |
notatF (notation placement) |
~66 |
annotation/annotB |
~62 |
addB, stF, and the long tail |
remainder of 2467 |
Decision: enforcing this is a large, low-real-world-value effort (people do not author structurally broken schemas, and a too-lenient compiler does not corrupt valid-document validation). It would require a schema-document validation pass against the schema-for-schemas. Treated as a documented, deliberate exclusion unless a concrete need arises. If implemented, it is its own epic, not part of the instance-correctness burn-down.
Category C: arguable / needs investigation
test,elemT(invalid instance accepted, ~86): not yet sampled; classify before fixing.
Category D: deliberate interpretations of spec-ambiguous points
These are resolved, not open: the spec underspecifies the point and the suite's own contributors disagree, so a single behavior cannot satisfy all of them. We pick the defensible reading and accept the residual deviation.
- QName length facets are non-constraining. XSD 1.0 Datatypes 4.3.1 defines
the unit of length for string (characters), anyURI (characters), binary
(octets), and list (items), but not for QName, so it is implementation-defined.
NIST (36+ cases) and Xerces treat
length/minLength/maxLengthon QName as always satisfied; Microsoft (4 cases:QName_length001/003,QName_minLength003,QName_maxLength001) treats them as character counts and expects rejection. These are directly contradictory (NIST marks a 15-character value valid underlength=1; MS marksfoofoinvalid underlength=4). We chose non-constraining, matching Xerces and the larger NIST set, because a QName's value is an abstract (namespace, local-name) pair with no defined character length. Cost: the 4 MS cases remain in "invalid instances accepted". Fixing the 36+ NIST false rejections (valid documents wrongly rejected) is the more important direction.
How a fix lands
Pick a Category A cluster, read 2-3 cases (schema + instance + expected) against
the spec text, fix the owning code, add a focused regression test, re-run
swift test -c release --filter XSTS, confirm the relevant count drops, and
ratchet the baseline in Tests/XSTSSuiteTests.swift down to the new value.
Tracking issues: #145 (invalid schema accepted), #146 (valid instance
rejected), #147 (invalid instance accepted), #148 (valid schema rejected).