The two monolithic XSD content/identity rules only satisfied the OpenAPIKit
validation idiom at the wrong granularity: no per-constraint removability,
no isolable failing/near-miss tests. Split them into 32 named rules, each
finding tagged at its origin and cached per XSDContext so the walk still
runs once per document (the #101 decision against duplicating XSD's
recursive type resolution still holds). Wired the streaming path's ID/IDREF
and identity checks through the same framework instead of calling the
validators imperatively, and added the unwrapAndLookup combinator the rule
already named as required.
Also includes the mandated repo-wide swiftformat/swiftlint pass.
An element restricting a base choice was always checked by the
effective-total-range approximation, which exists for one reason: the
literal section 3.9.6 pairing has no allowance for occurrence counts
accumulated across the choice's own repetitions and over-rejects
W3C-read-valid schemas (particlesZ001, whose manifest calls the rule
ambiguous). That ambiguity is impossible when the choice cannot repeat:
at {max occurs} = 1 every accepted child comes from one traversal of one
branch, so the literal rule governs. elementRestrictsChoice now applies
the literal pairing (the derived element, with its own occurrence range,
must validly restrict some non-pointless branch) when base maxOccurs is
1, and keeps the effective-range reading only for repeating choices.
Closes XSTS particlesHa161 (a derived a{0,1} is not a pointwise
NameAndTypeOK restriction of a required branch a{1,1}) and particlesM034
(a derived d1+ cannot map into the branch (d1*, d2+): the required d2 is
unmapped and not emptiable). The settled-valid neighbors that pin the
boundary from the other side all still pass: Ha160, M002, M003, M035,
and the Z001 shape, each now a regression test in
XSDParticleChoiceRestrictionTests. invalid-schemas-accepted 6 -> 4.
particlesIg004 is reclassified as spec-divergent (4 -> 3): section 3.3.2
maps block="#all" and the explicit substitution/extension/restriction
list to the SAME {disallowed substitutions} set, so NameAndTypeOK clause
5's superset requirement holds and the invalid expectation has no spec
path; it is jointly unsatisfiable with the settled Ig001/002/003 (equal
sets, valid) and Ig005 (#all covers the triple, valid).
The remaining three schema deviations are documented divergences, not
open bugs: Ik025/Ik027 (cos-st-derived-ok 2.2.4 reading) and schN12,
newly adjudicated in docs/xsts-deviations.md: its chameleon redefine
fails src-redefine 6.2.2 on NameAndTypeOK clause 1 (no-namespace locals
vs qualified locals), but 6.2.2 cannot be enforced uniformly because
mgO013's settled-VALID schema also violates it; any uniform enforcement
trades a false negative for a valid-schema-rejected false positive.
Also corrects the stale deviations-doc row that listed the already-closed
particlesIk011 and omitted schN12; the failures file is the authority.
Gates: full XSTS release suite green at 0/3/0/4 (valid buckets unchanged
at zero), libxml2 differential green at 244 pinned entries (Ha161 and
M034 newly pinned, both compile under libxml2), full local gate green.
NameAndTypeOK clause 1 (XSD 1.0 section 3.9.6, Elt:Elt) pairs the declared
element declarations and requires the same {name} and {target namespace};
no clause grants substitution an allowance. The substitution-group
expansion rewrites the compiled term (an abstract head is dropped, members
join a synthetic choice), so the checker previously compared the expanded
names and a head-for-member swap looked name-identical.
Particle.declaredReference now preserves the pre-expansion identity of an
element-ref particle, and the restriction check pairs the declared
references before the structural pairing table (extracted as
structuralValid). Closes XSTS elemZ026 and particlesV020:
invalid-schemas-accepted 8 -> 6, valid buckets unchanged at zero.
Stricter than libxml2, which does not implement this edge; elemZ026 is
pinned in the differential baseline accordingly.
particlesZ008, byte-for-byte the same construct as elemZ026 but carrying
a settled valid expectation contradicting elemZ026's settled invalid, is
reclassified as a spec-divergent schema test with the adjudication
recorded in the suite (exclusion tables split into
Tests/XSTSSuiteExclusions.swift for file length).
Separately attributed: the differential baseline gains ten further pinned
entries (addB070a, addB108, addB177, attO025, elemZ027_c, errC008,
groupH021v, notatB005, notatE002, particlesEb040) that reproduce with
this change stashed. They are pre-existing drift from earlier main-line
fixes, not effects of this commit. Baseline 231 -> 242, differential
green.
Generalise the wildcard {namespace constraint} model to the four XSD 1.0 (with
Errata E1-10) expressible forms -- any; a set (with absent); a pair of not and
a namespace name; a pair of not and absent -- replacing the any/##other/set
model that could not represent the union/intersection results. Both not forms
exclude absent, so a union admitting "every namespace except a name, plus
absent" (E1-10 rule 5.1.3) has no expressible form.
A new SchemaWildcardExpressibility compile rule rejects a complexContent
extension whose attribute-wildcard union with its base is not expressible
(src-ct.5), closing XSTS wildZ013 (test328873i). The expressible unions are
computed exactly in XSDComplexContent, so a derived type's wildcard admits the
correct attributes at instance time, closing wildZ013a and wildZ013d.
invalid-schemas-accepted 12 -> 11, invalid-instances-accepted 17 -> 15, with
valid-schemas-rejected and valid-instances-rejected held at 0 (full XSTS
archive). The model is additive for element wildcards (which never produce the
new forms); admits/narrows/UPA-overlap updated and verified unchanged for the
##other cases.
The intersection of two not forms negating different namespace names is not
expressible (src-ct.4), a deferred rule: rather than drop the wildcard and
over-reject valid cross-namespace attributes on an accepted schema, the
intersection stays the less-restrictive not(lhs) over-approximation (FP-safe,
matching prior behaviour). Union/intersection algebra taken clean-room from
Xerces-J XSWildcardDecl. Guarded by SchemaWildcardUnionExpressibilityTests.
XSD 1.0 §3.9.6 Particle Valid (Restriction) defines a case for an element
restricting a base group (RecurseAsIfGroup) but none for the reverse, so a
derived sequence/choice/all that can contribute a child never validly restricts
a single base element. The oracle had a deliberate non-spec 'mirror' extension
that bounded the derived group by the element's effective occurrence range and
accepted it; that masked a real violation inside a choice-to-choice restriction,
where a derived sequence branch was allowed to map onto a base element branch it
cannot legally restrict. The (.group, .element) case now returns invalid; a
content-free derived group is still a valid restriction of any emptiable base
(handled before this case). Catches particlesHb011. XSTS invalid-schemas-
accepted 17 to 16, valid-schemas-rejected held at 0 (differential agree-valid
unchanged).
Read element-vs-choice Particle Valid (Restriction) through effective total
range: an element validly restricts a base choice when its leaf is a
NameAndTypeOK restriction of an element the choice admits and its effective
occurrence range is within the choice's. This fixes particlesZ001 (the
RecurseAsIfGroup rule its own test calls ambiguous; W3C reads it valid) while
keeping particlesM002/003 valid and particlesL rejected.
Named, bounded under-rejection (exact in-branch sequencing unmodelled), the
production-sanctioned direction: invalid-schemas-accepted 128->130,
invalid-instances-accepted 132->133 (particlesZ001.i, a masked gap now exposed).
XSTS valid-schemas-rejected 1->0. With valid-instances-rejected already 0,
false positives are now 0 in both buckets (production stopper #1).
cos-ct-restricts.4: a base type with no attribute wildcard admits none (so a
restriction may not introduce one), and a wider namespace constraint such as
##any over ##other, or a weakened processContents, is not a subset. Reuses the
wildcard subset check (ParticleRestriction.narrows, now internal).
XSTS invalid-schemas-accepted 131 to 128, no other bucket moved.
A restricting xs:any must have processContents at least as strong as the base
wildcard's (strict > lax > skip; Wildcard Subset, XSD 1.0 sec 3.10.6), in
addition to narrowing the namespace constraint.
XSTS invalid-schemas-accepted 137 to 134, no other bucket moved.
A restricting element may not be nillable unless the base element it restricts
is also nillable. The element particle (Term.element) now carries a nillable
flag, populated from the element's nillable attribute.
XSTS invalid-schemas-accepted 144 to 142, no other bucket moved.
When the base element has a fixed value constraint, a restricting element must
also be fixed to the same value, compared in the base type's value space so a
value-equal pair written in different lexical forms is not wrongly flagged.
XSTS invalid-schemas-accepted 148 to 144, no other bucket moved.
Particle Valid (Restriction) now requires a restricting element's {disallowed
substitutions} to be a superset of the base element's: a restriction may block
more substitutions, never fewer. The element particle (Term.element) carries a
block set, populated from the element's block attribute or the schema's
blockDefault.
XSTS invalid-schemas-accepted 179 to 163, no other bucket moved.
An element restricting a base model group is treated as a synthetic group with
minOccurs=maxOccurs=1 holding the element (XSD 1.0 sec 3.9.6), so its {1,1}
occurrence must be within the base group's range: a base group that must occur
two or more times cannot be restricted to a single element. The element's own
occurrence is still matched against the base branch it maps to, so an optional
element restricting a once-or-more base group stays valid.
XSTS invalid-schemas-accepted 181 to 179, no other bucket moved.
Consolidates the in-progress #146 work; valid-instances-rejected reaches 4
on the 2006-11-06 XSTS archive (buckets now 1/262/4/147).
Regex: match Unicode scalars rather than grapheme clusters; the empty
pattern matches only the empty string; pin the category repertoire to
Unicode 3.2, the version the corpus is consistent at, so code points added
in 3.2 (Tagalog, Cyrillic Komi) are word characters while later additions
(the Tamil digit zero) stay unassigned and match \D. New XSDCategory and a
3.2 assignment table derived from DerivedAge, replacing the old Unicode31
files.
Content model: assess each child against the particle the content automaton
matched, not a by-name lookup (#180). Two same-named particles keep their
own fixed values, and an element matching a skip wildcard is no longer
validated against its global declaration. Selection is made reproducible
when an ambiguous schema slips past the UPA check.
XSTS runner: compile and merge every schemaDocument in a schemaTest, not
just the first; this also surfaced one previously-masked invalid-instance
acceptance (recorded in the baselines).
Also folds in attribute-group anyAttribute inheritance, extension wildcard
union, namespace-qualified type tables for multi-document
xsi:schemaLocation, per-container elementFormDefault, strict and lax
attribute wildcard validation, built-in type resolution, and xs:redefine
simpleType overrides.
An xs:all group in a complexContent extension of a base that already has element
content was accepted. The effective content model of such an extension is
sequence(base content, extension content), which nests the all and violates the
rule that an all group must be the whole content model. The base type is read
from the compiled model: rejected only when the base resolves, in the schema's
own target namespace, to a complex type with element-only or mixed content. An
explicitly-empty base (an empty model group with mixed=false) has effectively
empty content and may still be extended by an all (the all becomes the whole
content); a simple-content, empty, foreign, or unresolved base is left alone.
Checked for a self-contained schema.
XSTS: invalid schemas accepted 411 to 403, no other bucket raised.
An element declaring substitutionGroup="head" whose own type is unrelated to
the head's type was accepted; XSD 1.0 e-props-correct.4 requires the member's
type be validly derived from the head's. The check runs after named types
resolve and reuses the restriction/extension + built-in-lattice derivation
oracle (an ur-type head admits any member).
It is conservative so it never rejects a valid member: it runs only for a
self-contained schema (no import/include/redefine, where the derivation table
is complete and single-namespace names are unambiguous), reads the member's
type from its own type attribute and the head's from the top-level element
declarations only (so a local element of the same name supplies neither), and
stands down when either type is a list or union (cos-st-derived-ok clauses
2.3/2.4 union-member derivation is not modelled).
The post-namedTypes consistency calls move into a postNamedTypeErrors
aggregator alongside consistencyErrors. XSTS: invalid schemas accepted
480 to 479, no other bucket raised.
The W3C "Particle Valid (Restriction)" MapAndSum rule (XSD 1.0 §3.9.6): a derived
sequence restricting a base choice is valid iff each derived particle restricts
some base branch AND the derived sequence's occurrence count times its number of
particles lies within the base choice's occurrence range.
PureXML applied the same-compositor occurrence subsumption (the sequence's own
range against the choice's), which both rejected a valid restriction
(sequence{2,4}(e1,e2) of choice{3,9}(e1,e2): 4..8 elements fit 3..9) and accepted
an invalid one (a two-member sequence restricting a single-occurrence choice,
count-product 2 > 1). The count-product is the spec's measure, matched to the
reference implementation (checkMapAndSum/checkOccurrenceRange).
The (sequence, choice) branch in groupValid is retained: it is reached by the
(element, group) RecurseAsIfGroup path, which wraps an element as a one-particle
sequence against a base choice.
XSTS: valid schemas rejected 2 to 1, invalid schemas accepted 520 to 519, no
other bucket raised. The remaining valid-rejected case (particlesZ001) is a
documented spec ambiguity (RecurseAsIfGroup), recorded in docs/xsts-deviations.md.
Two more Particle Valid (Restriction) over-rejections.
A derived particle that can never contribute a child (its own maxOccurs="0", or
a group whose every member is content-free) accepts only the empty sequence, so
it validly restricts any emptiable base. Checked before the structural cases,
which previously recursed into the members of a maxOccurs="0" group and rejected
it for their own occurrences (particlesW006).
Substitution-group expansion now lists members in declaration order: the closure
walk used a popLast() stack that reversed them, so a reference to a head expanded
to choice(head, m2, m1) and an order-preserving RecurseLax restriction by
choice(m1, m2) could not map (elemZ027a). The choice/choice restriction stays
order-preserving, as the W3C suite requires (choice(b|a) is not a valid
restriction of choice(a|b), particlesT002); only the expansion order is fixed.
The expansion is shared with instance validation and the determinism check, both
order-neutral.
XSTS: valid schemas rejected 4 to 2, no other bucket raised. The pinned XSTS
baselines (never asserted under a plain swift test, the suite being opt-in) are
corrected to the measured 2/520/180/160.
Particle Valid (Restriction) derives its pairwise check over the spec's
normalized content model: pointless particles are eliminated (maxOccurs=0
members dropped, a {1,1} single-member group unwrapped, a nested
same-compositor {1,1} group spliced), all language-preserving. The outer
occurrence-range check was wrongly applied to the cross-kind pairings, where
it compares a derived particle's own range against a base group's or
wildcard's. It is now applied only to the same-kind pairings (element/element,
wildcard/wildcard, group/group) and element/wildcard; element/group
(RecurseAsIfGroup), group/element, and group/wildcard match occurrence inside
the case, against a member or as an effective total range.
A group restricting a wildcard (NSRecurseCheckCardinality) is valid when every
leaf is admitted and the effective total occurrence range is within the
wildcard's; a group restricting a single element is bounded the same way, by
that element's name, type derivation, and occurrence range.
XSTS: valid schemas rejected 45 to 32, invalid schemas accepted 528 to 526,
no other bucket raised.
The restriction subset check rejected 72 valid schemas (#148). Three
language-preserving corrections:
- Remove pointless particles (maxOccurs=0) from both the derived and base
content before mapping, so they no longer consume a base particle and starve
the particles after them (e.g. (e1 maxOccurs=0, e2) restricting (any)).
- A content-free restriction (an empty sequence, or a group whose members all
cannot occur) is a valid restriction of an EMPTY base: both accept only the
empty sequence.
- A derived group that reduces to empty after that removal validly restricts its
base only when the base is itself emptiable (emptiable(base) accounts for the
base's own occurrence), closing the vacuous-acceptance gap the removal would
otherwise open in the MapAndSum and choice paths.
XSTS: valid schemas rejected 72 to 45. invalid schemas accepted 527 to 528 from
addB32, whose only invalidity is a malformed imported document not yet loaded
(#161); the restriction itself is valid, so accepting it is correct.
Complete the element-vs-element type clause of Particle Valid (Restriction):
the restricting element's type must be, or derive by restriction from, the base
element's type. Previously only names were compared, so an unrelated, wider, or
ur-type element type was accepted under restriction.
Enrich the content-model particle: Term.element carries the element type's
derivation identity (the resolved type name, anyType for an absent type, nil for
an inline anonymous type). Record named simpleType restriction derivations in the
derivation table alongside complex ones. The element-vs-element check walks the
user restriction chain and then the built-in lattice to decide derivation by
name; an ur-type base, a union base (membership is not a restriction chain), and
an inline anonymous type are permitted, so no valid restriction is rejected.
XSTS: invalid schemas accepted 616 to 594, invalid instances accepted 165 to 160
(more invalid schemas rejected up front), valid schemas rejected held at 72,
valid instances rejected held at 214.
Extend NameAndTypeOK: restricting a concrete-typed element to an ur-type
(xs:anyType / xs:anySimpleType / xs:anyAtomicType) is widening, hence an invalid
restriction. The element-vs-element check now rejects it before the atomic
derivation check.
The ur-type is recognised by a type reference whose key is anyType /
anySimpleType / anyAtomicType AND which is absent from the named-type table, so
a user type that merely shares the local name (in another namespace) is not
mistaken for the ur-type. The reverse (an ur-type base narrowed by the
restriction) stays valid.
XSTS: invalid schemas accepted 620 to 616, valid schemas rejected held at 72,
instance buckets unchanged.
When a restriction maps a same-named element onto a base element, the
restricting type must derive from the base's (NameAndTypeOK). Renaming an
xs:string element to xs:int under restriction was accepted because the
element-vs-element check compared only names.
ParticleRestriction now threads the compiled type table through its recursion;
the element-vs-element case resolves both element types (following a type
reference through the table) and, when both are atomic simple types, requires
the restricting type's built-in base to derive from the base's. A complex,
list, union, untyped, or unresolvable type is left alone, so no valid
restriction is rejected.
XSTS: invalid schemas accepted 621 to 620, valid schemas rejected held at 72,
instance buckets unchanged.
A complex type derived by restriction is checked at schema compile to accept
a structural subset of its base, closing the declared XSD 1.0 spec gap.
ParticleRestriction covers element name matching, element-vs-wildcard
admission, wildcard narrowing, occurrence-range subsumption, and the group
recursions: sequence/all order-preserving with emptiable skips (Recurse),
choice mapping (RecurseLax), sequence-into-choice (MapAndSum, approximated by
per-particle checks, a documented simplification), sequence-into-all
(RecurseUnordered), and RecurseAsIfGroup for a lone element against a group.
An unfaithful restriction throws SchemaError.invalidRestriction(type:base:
reason:). All 834 pre-existing tests pass unchanged (no false positives on
the existing restriction-using schemas); 7 new tests cover the accept and
reject paths plus a conformance-harness pair.
Closes#117
