Mihaela Mihaljevic
c6305c9d93
XSD 1.0 Datatypes 4.3.1 defines the unit of length only for string, anyURI, binary, and list; for QName it is implementation-defined. We had counted characters, rejecting valid QName values. Like Xerces and the XSTS NIST set we now treat length/minLength/maxLength on QName as always satisfied. Lexical QName validity is still enforced. The suite's contributors disagree here (NIST: non-constraining; Microsoft: a character count), which the spec does not resolve, so a single behavior cannot satisfy both. Net XSTS: valid instances rejected 547 to 502; 4 Microsoft cases move to invalid instances accepted (257 to 261), recorded as a deliberate interpretation in docs/xsts-deviations.md. Refs #147, #129.
118 lines
6.6 KiB
Markdown
118 lines
6.6 KiB
Markdown
# XSTS deviation worklist
|
|
|
|
This partitions the W3C XML Schema Test Suite (XSTS, 2006-11-06 archive)
|
|
deviations by **root cause and kind**, so the burn-down is driven by
|
|
understanding each rule rather than by chasing the count. First measured
|
|
2026-06-13; baselined and ratcheted in `Tests/XSTSSuiteTests.swift`.
|
|
|
|
## What the oracle measures
|
|
|
|
The runner compares PureXML's verdict to the suite's expected validity. A
|
|
schema is "accepted" iff `Schema.Document(source)` does not throw; an instance
|
|
is "accepted" iff `validate` returns no errors. The four counts:
|
|
|
|
| Kind | Meaning | Direction |
|
|
|---|---|---|
|
|
| valid schema rejected | legal XSD that PureXML refused to compile | too strict |
|
|
| invalid schema accepted | illegal XSD that PureXML compiled anyway | too lenient |
|
|
| valid instance rejected | valid document PureXML flagged | too strict |
|
|
| invalid instance accepted | invalid document PureXML passed | too lenient |
|
|
|
|
The oracle is binary and coarse: "invalid schema accepted" conflates *we have a
|
|
bug* with *we do not implement this validity constraint by design*. The point of
|
|
this document is to separate those.
|
|
|
|
## Baseline (post #146 list-facet fix)
|
|
|
|
| Kind | Count |
|
|
|---|---|
|
|
| valid schema rejected | 75 |
|
|
| invalid schema accepted | 2467 |
|
|
| valid instance rejected | 547 |
|
|
| invalid instance accepted | 555 |
|
|
|
|
XSTS is a deliberately adversarial, exhaustive corpus; mature validators
|
|
(Xerces, libxml2) do not pass it fully. The goal is not a green number. It is to
|
|
fix the deviations that are genuine correctness bugs affecting real documents,
|
|
and to document the remainder as spec-justified, deliberate exclusions.
|
|
|
|
## Category A: genuine correctness bugs (fix; real-data impact)
|
|
|
|
These are cases where PureXML mishandles a feature it does support. Each is a
|
|
real bug a user could hit with a well-formed schema and document.
|
|
|
|
| Cluster(s) | Count | Root cause | Locus | Status |
|
|
|---|---|---|---|---|
|
|
| `list-NMTOKENS/IDREFS/ENTITIES` length facets | ~84 | length on a built-in list counted characters, not items | `XSDSimpleParser.simpleType` | **fixed (#146)** |
|
|
| `union-*-pattern`, `union-*-enumeration` | ~200 | a union restricted by `enumeration`/`pattern` does not enforce the facet (value outside the enumeration is accepted) | `SimpleType.validateUnion` | open, one fix, high value |
|
|
| `atomic-duration-minInclusive/maxInclusive` | ~50 | duration ordering bounds not enforced | duration comparison | open |
|
|
| `reL`, `reI`, `RegexTest` (instance) | ~155 | missing Unicode block escapes (`\p{IsArmenian}` etc.) and category coverage in the regex engine | `Sources/Regex` | open, feature completion |
|
|
| `atomic-QName-length`, `-minLength` | ~36 | character-length facet wrongly applied to `QName`; the spec/NIST treat QName length as non-constraining (Xerces agrees) | length facet on QName | open, over-strict |
|
|
| `valueconstraint`, `isDefault` | ~26 | default/fixed value handling | element/attr defaults | open, needs sampling |
|
|
| `particlesJk/Jj/Ha/Q/R` (schema), `particlesOb/Je` (instance) | ~110 | content-model construction/validation edge cases | content model | open, bounded |
|
|
| `int_min/maxInclusive`, `int_min/maxExclusive` (schema) | ~16 | integer bound facets rejecting legal schemas | numeric facets | open, bounded |
|
|
|
|
Recommended order: union facets (~200, one fix) → QName length (~36) → regex
|
|
Unicode blocks (~155) → duration ordering (~50) → particles/int facets.
|
|
|
|
## Category B: schema-for-schemas validity (unimplemented by design)
|
|
|
|
The bulk of the 2467 "invalid schema accepted". PureXML compiles schemas
|
|
permissively: it reads the components it understands and ignores structurally
|
|
illegal content rather than validating the schema document against the
|
|
schema-for-schemas. Examples found:
|
|
|
|
- `ctB`: `<complexType>Annotation information</complexType>` (bare text content) is accepted.
|
|
- `notatF`: `<notation>` placed inside `<restriction>` (illegal position) is accepted.
|
|
- `idA/idB/idC`: malformed identity-constraint definitions accepted.
|
|
- `annotation/annotB`: illegal annotation placement accepted.
|
|
- `addB`, `stF`, `ctG`, `ctH`: assorted illegal component structures accepted.
|
|
|
|
| Cluster(s) | Count |
|
|
|---|---|
|
|
| `ctB/ctG/ctH` (complexType structure) | ~183 |
|
|
| `idA/idB/idC` (identity constraints) | ~118 |
|
|
| `notatF` (notation placement) | ~66 |
|
|
| `annotation/annotB` | ~62 |
|
|
| `addB`, `stF`, and the long tail | remainder of 2467 |
|
|
|
|
**Decision:** enforcing this is a large, low-real-world-value effort (people do
|
|
not author structurally broken schemas, and a too-lenient *compiler* does not
|
|
corrupt valid-document validation). It would require a schema-document
|
|
validation pass against the schema-for-schemas. Treated as a **documented,
|
|
deliberate exclusion** unless a concrete need arises. If implemented, it is its
|
|
own epic, not part of the instance-correctness burn-down.
|
|
|
|
## Category C: arguable / needs investigation
|
|
|
|
- `test`, `elemT` (invalid instance accepted, ~86): not yet sampled; classify before fixing.
|
|
|
|
## Category D: deliberate interpretations of spec-ambiguous points
|
|
|
|
These are resolved, not open: the spec underspecifies the point and the suite's
|
|
own contributors disagree, so a single behavior cannot satisfy all of them. We
|
|
pick the defensible reading and accept the residual deviation.
|
|
|
|
- **QName length facets are non-constraining.** XSD 1.0 Datatypes 4.3.1 defines
|
|
the unit of length for string (characters), anyURI (characters), binary
|
|
(octets), and list (items), but not for QName, so it is implementation-defined.
|
|
NIST (36+ cases) and Xerces treat `length`/`minLength`/`maxLength` on QName as
|
|
always satisfied; Microsoft (4 cases: `QName_length001/003`, `QName_minLength003`,
|
|
`QName_maxLength001`) treats them as character counts and expects rejection.
|
|
These are directly contradictory (NIST marks a 15-character value valid under
|
|
`length=1`; MS marks `foofo` invalid under `length=4`). We chose
|
|
non-constraining, matching Xerces and the larger NIST set, because a QName's
|
|
value is an abstract (namespace, local-name) pair with no defined character
|
|
length. Cost: the 4 MS cases remain in "invalid instances accepted". Fixing
|
|
the 36+ NIST false rejections (valid documents wrongly rejected) is the more
|
|
important direction.
|
|
|
|
## How a fix lands
|
|
|
|
Pick a Category A cluster, read 2-3 cases (schema + instance + expected) against
|
|
the spec text, fix the owning code, add a focused regression test, re-run
|
|
`swift test -c release --filter XSTS`, confirm the relevant count drops, and
|
|
ratchet the baseline in `Tests/XSTSSuiteTests.swift` down to the new value.
|
|
Tracking issues: #145 (invalid schema accepted), #146 (valid instance
|
|
rejected), #147 (invalid instance accepted), #148 (valid schema rejected).
|