Consolidates the in-progress #146 work; valid-instances-rejected reaches 4
on the 2006-11-06 XSTS archive (buckets now 1/262/4/147).
Regex: match Unicode scalars rather than grapheme clusters; the empty
pattern matches only the empty string; pin the category repertoire to
Unicode 3.2, the version the corpus is consistent at, so code points added
in 3.2 (Tagalog, Cyrillic Komi) are word characters while later additions
(the Tamil digit zero) stay unassigned and match \D. New XSDCategory and a
3.2 assignment table derived from DerivedAge, replacing the old Unicode31
files.
Content model: assess each child against the particle the content automaton
matched, not a by-name lookup (#180). Two same-named particles keep their
own fixed values, and an element matching a skip wildcard is no longer
validated against its global declaration. Selection is made reproducible
when an ambiguous schema slips past the UPA check.
XSTS runner: compile and merge every schemaDocument in a schemaTest, not
just the first; this also surfaced one previously-masked invalid-instance
acceptance (recorded in the baselines).
Also folds in attribute-group anyAttribute inheritance, extension wildcard
union, namespace-qualified type tables for multi-document
xsi:schemaLocation, per-container elementFormDefault, strict and lax
attribute wildcard validation, built-in type resolution, and xs:redefine
simpleType overrides.
An xs:all group in a complexContent extension of a base that already has element
content was accepted. The effective content model of such an extension is
sequence(base content, extension content), which nests the all and violates the
rule that an all group must be the whole content model. The base type is read
from the compiled model: rejected only when the base resolves, in the schema's
own target namespace, to a complex type with element-only or mixed content. An
explicitly-empty base (an empty model group with mixed=false) has effectively
empty content and may still be extended by an all (the all becomes the whole
content); a simple-content, empty, foreign, or unresolved base is left alone.
Checked for a self-contained schema.
XSTS: invalid schemas accepted 411 to 403, no other bucket raised.
An element declaring substitutionGroup="head" whose own type is unrelated to
the head's type was accepted; XSD 1.0 e-props-correct.4 requires the member's
type be validly derived from the head's. The check runs after named types
resolve and reuses the restriction/extension + built-in-lattice derivation
oracle (an ur-type head admits any member).
It is conservative so it never rejects a valid member: it runs only for a
self-contained schema (no import/include/redefine, where the derivation table
is complete and single-namespace names are unambiguous), reads the member's
type from its own type attribute and the head's from the top-level element
declarations only (so a local element of the same name supplies neither), and
stands down when either type is a list or union (cos-st-derived-ok clauses
2.3/2.4 union-member derivation is not modelled).
The post-namedTypes consistency calls move into a postNamedTypeErrors
aggregator alongside consistencyErrors. XSTS: invalid schemas accepted
480 to 479, no other bucket raised.
The W3C "Particle Valid (Restriction)" MapAndSum rule (XSD 1.0 §3.9.6): a derived
sequence restricting a base choice is valid iff each derived particle restricts
some base branch AND the derived sequence's occurrence count times its number of
particles lies within the base choice's occurrence range.
PureXML applied the same-compositor occurrence subsumption (the sequence's own
range against the choice's), which both rejected a valid restriction
(sequence{2,4}(e1,e2) of choice{3,9}(e1,e2): 4..8 elements fit 3..9) and accepted
an invalid one (a two-member sequence restricting a single-occurrence choice,
count-product 2 > 1). The count-product is the spec's measure, matched to the
reference implementation (checkMapAndSum/checkOccurrenceRange).
The (sequence, choice) branch in groupValid is retained: it is reached by the
(element, group) RecurseAsIfGroup path, which wraps an element as a one-particle
sequence against a base choice.
XSTS: valid schemas rejected 2 to 1, invalid schemas accepted 520 to 519, no
other bucket raised. The remaining valid-rejected case (particlesZ001) is a
documented spec ambiguity (RecurseAsIfGroup), recorded in docs/xsts-deviations.md.
Two more Particle Valid (Restriction) over-rejections.
A derived particle that can never contribute a child (its own maxOccurs="0", or
a group whose every member is content-free) accepts only the empty sequence, so
it validly restricts any emptiable base. Checked before the structural cases,
which previously recursed into the members of a maxOccurs="0" group and rejected
it for their own occurrences (particlesW006).
Substitution-group expansion now lists members in declaration order: the closure
walk used a popLast() stack that reversed them, so a reference to a head expanded
to choice(head, m2, m1) and an order-preserving RecurseLax restriction by
choice(m1, m2) could not map (elemZ027a). The choice/choice restriction stays
order-preserving, as the W3C suite requires (choice(b|a) is not a valid
restriction of choice(a|b), particlesT002); only the expansion order is fixed.
The expansion is shared with instance validation and the determinism check, both
order-neutral.
XSTS: valid schemas rejected 4 to 2, no other bucket raised. The pinned XSTS
baselines (never asserted under a plain swift test, the suite being opt-in) are
corrected to the measured 2/520/180/160.
Particle Valid (Restriction) derives its pairwise check over the spec's
normalized content model: pointless particles are eliminated (maxOccurs=0
members dropped, a {1,1} single-member group unwrapped, a nested
same-compositor {1,1} group spliced), all language-preserving. The outer
occurrence-range check was wrongly applied to the cross-kind pairings, where
it compares a derived particle's own range against a base group's or
wildcard's. It is now applied only to the same-kind pairings (element/element,
wildcard/wildcard, group/group) and element/wildcard; element/group
(RecurseAsIfGroup), group/element, and group/wildcard match occurrence inside
the case, against a member or as an effective total range.
A group restricting a wildcard (NSRecurseCheckCardinality) is valid when every
leaf is admitted and the effective total occurrence range is within the
wildcard's; a group restricting a single element is bounded the same way, by
that element's name, type derivation, and occurrence range.
XSTS: valid schemas rejected 45 to 32, invalid schemas accepted 528 to 526,
no other bucket raised.
The restriction subset check rejected 72 valid schemas (#148). Three
language-preserving corrections:
- Remove pointless particles (maxOccurs=0) from both the derived and base
content before mapping, so they no longer consume a base particle and starve
the particles after them (e.g. (e1 maxOccurs=0, e2) restricting (any)).
- A content-free restriction (an empty sequence, or a group whose members all
cannot occur) is a valid restriction of an EMPTY base: both accept only the
empty sequence.
- A derived group that reduces to empty after that removal validly restricts its
base only when the base is itself emptiable (emptiable(base) accounts for the
base's own occurrence), closing the vacuous-acceptance gap the removal would
otherwise open in the MapAndSum and choice paths.
XSTS: valid schemas rejected 72 to 45. invalid schemas accepted 527 to 528 from
addB32, whose only invalidity is a malformed imported document not yet loaded
(#161); the restriction itself is valid, so accepting it is correct.
Complete the element-vs-element type clause of Particle Valid (Restriction):
the restricting element's type must be, or derive by restriction from, the base
element's type. Previously only names were compared, so an unrelated, wider, or
ur-type element type was accepted under restriction.
Enrich the content-model particle: Term.element carries the element type's
derivation identity (the resolved type name, anyType for an absent type, nil for
an inline anonymous type). Record named simpleType restriction derivations in the
derivation table alongside complex ones. The element-vs-element check walks the
user restriction chain and then the built-in lattice to decide derivation by
name; an ur-type base, a union base (membership is not a restriction chain), and
an inline anonymous type are permitted, so no valid restriction is rejected.
XSTS: invalid schemas accepted 616 to 594, invalid instances accepted 165 to 160
(more invalid schemas rejected up front), valid schemas rejected held at 72,
valid instances rejected held at 214.
Extend NameAndTypeOK: restricting a concrete-typed element to an ur-type
(xs:anyType / xs:anySimpleType / xs:anyAtomicType) is widening, hence an invalid
restriction. The element-vs-element check now rejects it before the atomic
derivation check.
The ur-type is recognised by a type reference whose key is anyType /
anySimpleType / anyAtomicType AND which is absent from the named-type table, so
a user type that merely shares the local name (in another namespace) is not
mistaken for the ur-type. The reverse (an ur-type base narrowed by the
restriction) stays valid.
XSTS: invalid schemas accepted 620 to 616, valid schemas rejected held at 72,
instance buckets unchanged.
When a restriction maps a same-named element onto a base element, the
restricting type must derive from the base's (NameAndTypeOK). Renaming an
xs:string element to xs:int under restriction was accepted because the
element-vs-element check compared only names.
ParticleRestriction now threads the compiled type table through its recursion;
the element-vs-element case resolves both element types (following a type
reference through the table) and, when both are atomic simple types, requires
the restricting type's built-in base to derive from the base's. A complex,
list, union, untyped, or unresolvable type is left alone, so no valid
restriction is rejected.
XSTS: invalid schemas accepted 621 to 620, valid schemas rejected held at 72,
instance buckets unchanged.
A complex type derived by restriction is checked at schema compile to accept
a structural subset of its base, closing the declared XSD 1.0 spec gap.
ParticleRestriction covers element name matching, element-vs-wildcard
admission, wildcard narrowing, occurrence-range subsumption, and the group
recursions: sequence/all order-preserving with emptiable skips (Recurse),
choice mapping (RecurseLax), sequence-into-choice (MapAndSum, approximated by
per-particle checks, a documented simplification), sequence-into-all
(RecurseUnordered), and RecurseAsIfGroup for a lone element against a group.
An unfaithful restriction throws SchemaError.invalidRestriction(type:base:
reason:). All 834 pre-existing tests pass unchanged (no false positives on
the existing restriction-using schemas); 7 new tests cover the accept and
reject paths plus a conformance-harness pair.
Closes#117
