Mihaela Mihaljevic
6fbc92593e
XSD 1.0 allows a complexType's content to be simpleContent, or complexContent, or a model group followed by attributes, never a mix. The allowedChildren set-membership check permitted illegal combinations, so a complexType with two simpleContent, simpleContent beside a sequence, or two model groups was accepted. complexTypeContentErrors now enforces, over the complexType's direct children: at most one of simpleContent/complexContent; when one is present, no direct model group and no direct attribute (those belong inside the extension/restriction); and at most one model group. Only direct children are examined, so attributes nested inside a derivation are not flagged. XSTS: invalid schemas accepted 1051 to 997 (-54); valid schemas rejected held at 72; instance buckets unchanged. Also folds in a behavior-identical lint cleanup of the reference-walk external-definition check.
9.1 KiB
XSTS deviation worklist
This partitions the W3C XML Schema Test Suite (XSTS, 2006-11-06 archive)
deviations by root cause and kind, so the burn-down is driven by
understanding each rule rather than by chasing the count. First measured
2026-06-13; baselined and ratcheted in Tests/XSTSSuiteTests.swift.
What the oracle measures
The runner compares PureXML's verdict to the suite's expected validity. A
schema is "accepted" iff Schema.Document(source) does not throw; an instance
is "accepted" iff validate returns no errors. The four counts:
| Kind | Meaning | Direction |
|---|---|---|
| valid schema rejected | legal XSD that PureXML refused to compile | too strict |
| invalid schema accepted | illegal XSD that PureXML compiled anyway | too lenient |
| valid instance rejected | valid document PureXML flagged | too strict |
| invalid instance accepted | invalid document PureXML passed | too lenient |
The oracle is binary and coarse: "invalid schema accepted" conflates we have a bug with we do not implement this validity constraint by design. The point of this document is to separate those.
Only tests with a settled expectation are counted. The suite marks contested
entries <current status="queried"> (disputed in W3C Bugzilla, the expected
validity itself in question, e.g. \p{Lu}* over two uppercase letters one of
which is an astral math capital); agreeing or disagreeing with a contested
expectation is not meaningful, so the runner skips them. Excluding the queried
entries removed about 70 apparent deviations that were never bugs.
Baseline (settled expectations; after twelve fixes)
| Kind | First measurement | Now |
|---|---|---|
| valid schema rejected | 75 | 72 |
| invalid schema accepted | 2467 | 2461 |
| valid instance rejected | 604 | 308 |
| invalid instance accepted | 582 | 165 |
Counts are against settled W3C expectations (queried/disputed entries excluded).
XSTS is a deliberately adversarial, exhaustive corpus; mature validators (Xerces, libxml2) do not pass it fully. The goal is not a green number. It is to fix the deviations that are genuine correctness bugs affecting real documents, and to document the remainder as spec-justified, deliberate exclusions.
Category A: genuine correctness bugs
Fixed (twenty-three root causes, ~2138 deviations cleared)
list-datatype length facets (#146) · union pattern/enumeration facets · duration
partial-order facets · QName length non-constraining · the full XSD \p{Is...}
Unicode block set · element-level block against xsi:type · empty-element
default/fixed values + enumeration-restriction-replaces · block="substitution"
on substitution-group heads · \w/\W XSD definition · document-scoped
xs:ID/xs:IDREF uniqueness and resolution · <xs:attribute ref> resolution ·
mixed-content type with no element model · built-in xsi:type resolution gated
by substitution validity (the XSD Part 2 derivation lattice), which also closed
the two gaps it exposed (a simple-typed element rejecting stray attributes;
identity fields comparing in value space when the node carries an xsi:type) ·
compile-time constraining-facet definition validity (length-family facets are
nonNegativeInteger, co-occurrence and range order) · value-bound facet
validity (min/maxInclusive, min/maxExclusive, enumeration values valid
in the base space; inclusive/exclusive exclusions; bound ordering) · gMonth
--MM-- lexical form · schema id attribute validity (xs:ID: NCName and
unique within the document). The schema-validity work brought invalid schemas
accepted 2461 to 997 (schema-document structural validity: content model + complexType content shape, attribute values, name/reference + pattern-regex lexical validity, and reference resolution, against the schema-for-schemas); see schema-validity-burndown.md.
Plus the measurement fix (exclude W3C-disputed status="queried" entries).
Remaining, root-caused (the hard tail)
Each is a substantial, self-contained study, not a one-locus cluster fix:
| Cluster(s) | ~Count | Root cause (diagnosed) | Difficulty |
|---|---|---|---|
particlesOb/Je/Jf, related |
~30 | <xs:any> defaults to processContents="strict"; the matched element's declaration lives in a separate schema document referenced only by the instance xsi:schemaLocation (no <import>). Needs multi-document schema loading + xsi:schemaLocation honoring, plus strict-wildcard semantics and Particle-Valid-Restriction. |
large (new capability) |
reI |
~10 | pattern values containing literal tab/newline: XML attribute-value normalization (tab/newline to space) versus the value's whiteSpace facet. The "right" pattern text depends on that interaction. |
subtle |
test, elemT |
~65 | heterogeneous grab-bags (MS "Additional"/element tests): many distinct minor causes, not one root cause. Each case needs individual triage. | per-case grind |
idF, idc remainder |
~14 | identity-constraint selector/field XPath edge cases beyond the implemented core. The value-space comparison via instance xsi:type and built-in xsi:type resolution now land; what remains is the declared-type-driven value-space comparison (needs PSVI type assignment, since the field's declared simple type is not threaded into identity validation) and selector/field XPath edges. |
moderate |
wildG, stZ, addB |
~25 | wildcard-in-group composition, simple-type edge cases, attribute-default edges. | mixed |
The clean, single-root-cause, high-leverage fixes are exhausted. Reaching zero on the instance buckets requires the multi-document-schema capability (the largest single lever, ~30 cases) and per-case work on the grab-bags; neither is a quick fix, and rushing them would violate the project's no-shortcuts ethic.
Category B: schema-for-schemas validity (unimplemented by design)
The bulk of the 2467 "invalid schema accepted". PureXML compiles schemas permissively: it reads the components it understands and ignores structurally illegal content rather than validating the schema document against the schema-for-schemas. Examples found:
ctB:<complexType>Annotation information</complexType>(bare text content) is accepted.notatF:<notation>placed inside<restriction>(illegal position) is accepted.idA/idB/idC: malformed identity-constraint definitions accepted.annotation/annotB: illegal annotation placement accepted.addB,stF,ctG,ctH: assorted illegal component structures accepted.
| Cluster(s) | Count |
|---|---|
ctB/ctG/ctH (complexType structure) |
~183 |
idA/idB/idC (identity constraints) |
~118 |
notatF (notation placement) |
~66 |
annotation/annotB |
~62 |
addB, stF, and the long tail |
remainder of 2467 |
Status (revised): now being implemented as the general schema-for-schemas
mechanism, not a per-rule pile. The content-model pass has landed (id validity,
facet-definition validity, and the allowedChildren structural table), driving
invalid-schemas-accepted from 2461 to 1313. The remaining structural work (the
component attribute model, child order, particle semantics) continues under
schema-validity-burndown.md. A too-lenient compiler does not corrupt
valid-document validation, but rubber-stamping invalid schemas is the largest
conformance gap, so it is being closed rather than excluded.
Category C: arguable / needs investigation
test,elemT(invalid instance accepted, ~86): not yet sampled; classify before fixing.
Category D: deliberate interpretations of spec-ambiguous points
These are resolved, not open: the spec underspecifies the point and the suite's own contributors disagree, so a single behavior cannot satisfy all of them. We pick the defensible reading and accept the residual deviation.
- QName length facets are non-constraining. XSD 1.0 Datatypes 4.3.1 defines
the unit of length for string (characters), anyURI (characters), binary
(octets), and list (items), but not for QName, so it is implementation-defined.
NIST (36+ cases) and Xerces treat
length/minLength/maxLengthon QName as always satisfied; Microsoft (4 cases:QName_length001/003,QName_minLength003,QName_maxLength001) treats them as character counts and expects rejection. These are directly contradictory (NIST marks a 15-character value valid underlength=1; MS marksfoofoinvalid underlength=4). We chose non-constraining, matching Xerces and the larger NIST set, because a QName's value is an abstract (namespace, local-name) pair with no defined character length. Cost: the 4 MS cases remain in "invalid instances accepted". Fixing the 36+ NIST false rejections (valid documents wrongly rejected) is the more important direction.
How a fix lands
Pick a Category A cluster, read 2-3 cases (schema + instance + expected) against
the spec text, fix the owning code, add a focused regression test, re-run
swift test -c release --filter XSTS, confirm the relevant count drops, and
ratchet the baseline in Tests/XSTSSuiteTests.swift down to the new value.
Tracking issues: #145 (invalid schema accepted), #146 (valid instance
rejected), #147 (invalid instance accepted), #148 (valid schema rejected).