Compiling a schema with deeply nested, multiply-referenced model groups was
exponential: groupReferenceParticle inlined each <xs:group ref> into the Particle
content model, so g0=(g1,g1), g1=(g2,g2), ... built 2^K particles (depth 14 =
0.21s, depth 20 ~13-18s). The expansion is now memoized per (group definition,
cycle-visiting set) in a per-compile GroupParticleMemo shared across the value
copies XSDContext.scoped/visiting make, so sibling references share it and the
build drops to O(groups) (depth 40 now ~0.016s; previously infeasible).
The shared content term is reused by value (copy-on-write), so the compiled model
is identical and validation is unchanged: the XSTS gate holds (invalid-schemas-
accepted 15, valid-schemas-rejected 0, instance buckets unchanged) and the
streaming-vs-tree fuzz differential still passes. This closes the schema-compile
half of the interactive-safety group blowup (companion to the positionCap
determinism fix); a depth-40 compile test guards against regression. Full suite,
style, namespacing, and WASM all pass.
Convert the five composition/redefine compile checkers to located findings,
the last of the #169 burndown (#210-#217): a resolved schemaLocation whose
content is not a schema underlines the include/import/redefine reference, a
redefined type that does not restrict or extend itself underlines the
redefined type, a redefined group or attribute group with a malformed
self-reference underlines the offending reference or redefinition, and a
redefined attribute group or model group that does not restrict its original
underlines the redefinition.
To locate the referenced-schema rule, failedSchemaReferences carries the
referencing node, captured at the load-failure site in XSDContext. Only a
top-level reference's node belongs to the main schema tree the diagnostic
resolves against; a failure nested in a loaded sub-document is captured from
a throwaway parse, so it stays unlocated (the prior schema-root behavior)
rather than mis-resolve, a strict improvement.
Every predicate, aggregation order, and reason string is unchanged; only the
location each finding carries differs, verified conformance-neutral by the
full XSTS suite (14383 groups, pinned counts unchanged). The redefine
checkers had exactly one caller each, so each is renamed in place and the
.unlocated wrap dropped. Adds a located-diagnostics test per node-locatable
checker; the attribute-group and model-group restriction redefine checkers
share the same redefinition-node idiom and are exercised by the XSTS suite.
An identity-constraint (unique/key/keyref) field that selects an absent attribute
or an empty element whose declaration carries a default or fixed value now takes
that value as its identity-tuple component, so two members can collide on the
defaulted value (idG011/idG012 default, idF016/idF017 fixed) where the duplicate
was previously missed.
The default/fixed value constraints on identity-field targets are gathered at
compile time (identityFieldConstraints, mirroring identityFieldTypes) and threaded
through to the identity validator. The substitution is exact about absent versus
present-empty: an @attr field is defaulted only when the attribute is ABSENT (an
empty node-set), never when present with value ""; a "." element field only when
the selected element has no character content. Applying a default only adds an
identity value where the schema declares one, so it can create a genuine
duplicate-key detection but never reject a valid instance.
To keep within the file/type-size and one-type-per-file limits, FieldValue,
XPathQueryCache, and the schema-location hints were extracted to their own files
(behavior-preserving moves).
invalid-instances-accepted 50 to 45, no false positive (valid-instances-rejected
and valid-schemas-rejected held at 0), no other bucket moved. An adversarial
review probed present-empty attributes, element content winning over a default,
fixed values, keyref matching, multi-field keys, and merged-schema defaults, and
found no over-rejection.
The instance-validity derivation and substitution tables were keyed by bare
local name, so two same-local-name types or elements in different namespaces
collided. In multi-document validation this let one schema's block/derivation
facts shadow another's, and it blocked honouring blockDefault (whose re-land
widened that collision into an over-rejection).
Key the instance subsystem by namespaced identity {ns}local instead. The shared
DerivationTables keeps its bare-keyed fields for the schema-validity machinery
(Particle-Valid-Restriction and the consistency rules, which resolve QNames to
bare names and walk the same backbone) and gains parallel ns* fields built in
the same pass, resolving each container's components through its target
namespace and prefix bindings. Only the instance consumers
(blockedSubstitutionError, notDerivedSubstitutionError, abstractTypeError, the
substitution-group closure, and referenceParticle) read the ns* tables, via a
derivationKey(fromReference:) that reduces a type:{ns}local reference, a bare
name, or an element ref to a {ns}local backbone key. A key miss stays permissive
(no constraint, substitution allowed), so the re-keying cannot over-reject.
With the namespaced backbone in place, blockDefault is re-landed FP-free: a
schema's blockDefault supplies the {disallowed substitutions} of every element
and complex type without an explicit block (own block overrides; block="" clears;
intersected with the methods each component admits), so blockDefault="extension"
bars an extension substitution and "#all" bars all, without one namespace's
defaults shadowing another's.
The pure re-keying preserves single-document behaviour exactly (the corpus is
overwhelmingly single-namespace, where {ns}local and bare coincide): all four
XSTS buckets held at vsr=0 / isa=69 / vir=0 / ivi=89 after re-keying alone.
blockDefault then drove invalid-instances-accepted 89 to 81, no false positive
(valid-instances-rejected and valid-schemas-rejected held at 0), no other bucket
moved. An adversarial review exercised same- and cross-namespace substitutions,
no-targetNamespace keys, simple-type restriction, transitive substitution groups,
and blockDefault override/over-block cases, and the prior multi-document
cross-namespace blockDefault over-rejection is now accepted while a same-namespace
blocked substitution is still rejected (precise, not merely permissive).
The convergence audit for "validators everywhere" found the one remaining XSD
schema-validity verdict reported outside the Validator framework: facet
definition validity and pattern syntax validity were reported straight to the
diagnostics collector during simple-type compilation (validation folded into
parsing, which validation-rules.md forbids).
These checks genuinely need the resolved base type, computed during compilation,
so the finding is gathered into a context finding sink (XSDContext
.facetFindingSink) during type resolution and surfaced by the named post-compile
Validation simpleTypeFacetsAreValid -- located findings preserved, now pinned in
the configuration test and removable by identity like every other rule.
Re-sweep confirms the XSD path has converged: the only remaining throws are
notASchema (a not-a-schema precondition) and inconsistent (the framework's own
surfacing throw); no validity verdict bypasses the framework. No bucket change
(invalid-schemas-accepted 81, valid-schemas-rejected 0); full suite + fuzz pass.
Per the OpenAPIKit validation idiom (Rules/universal/validation-rules.md), every
schema-validity check should run through the Validator framework as a described,
composable Validation rather than an ad-hoc throw. The src-resolve check (a
resolved include/import/redefine schemaLocation must be a well-formed schema)
was a parse-pipeline throw; it is now the pre-compile Validation
referencedSchemasResolveToSchemas.
failedSchemaReferences (formerly the throwing checkSchemaLocations) returns the
resolved-but-not-a-schema locations; the pipeline records them on XSDContext and
the validation reports one finding each. Same rejection behaviour (the finding
makes the schema invalid), now located in the framework, pinned in the
configuration test. Removed the now-unused SchemaError.invalidSchemaReference
case (added earlier this branch, never released).
No bucket change (invalid-schemas-accepted stays 81, valid-schemas-rejected 0);
full suite and fuzz pass (report-and-continue does not crash downstream).
When an xs:include / xs:import / xs:redefine schemaLocation IS resolved (the
schema loader returns content for it), that content must be a well-formed
schema document. Not-well-formed XML, or well-formed XML whose root is not
xs:schema, now makes the composing schema invalid (SchemaError
.invalidSchemaReference). collectContainers previously loaded externals with
`try? parseTree` and silently dropped a malformed one, accepting the schema.
A location the loader does NOT resolve (returns nil) is not an error, since
resolution is the processor's choice (src-resolve) -- so a self-contained
schema, or one whose externals are simply unavailable, is unaffected. This is
the false-positive boundary: fire only when the loader returned content that
is not a schema.
XSTS: invalid-schemas-accepted 95 to 87 (eight caught), valid-schemas-rejected
held at 0, no other bucket moved. Differential: agree-valid unchanged at 3161,
PureXML-stricter unchanged at 87 over 9618 schemas (zero new over-rejection).
First step of the cross-document composition campaign (#161).
Particle Valid (Restriction) now requires a restricting element's {disallowed
substitutions} to be a superset of the base element's: a restriction may block
more substitutions, never fewer. The element particle (Term.element) carries a
block set, populated from the element's block attribute or the schema's
blockDefault.
XSTS invalid-schemas-accepted 179 to 163, no other bucket moved.
Consolidates the in-progress #146 work; valid-instances-rejected reaches 4
on the 2006-11-06 XSTS archive (buckets now 1/262/4/147).
Regex: match Unicode scalars rather than grapheme clusters; the empty
pattern matches only the empty string; pin the category repertoire to
Unicode 3.2, the version the corpus is consistent at, so code points added
in 3.2 (Tagalog, Cyrillic Komi) are word characters while later additions
(the Tamil digit zero) stay unassigned and match \D. New XSDCategory and a
3.2 assignment table derived from DerivedAge, replacing the old Unicode31
files.
Content model: assess each child against the particle the content automaton
matched, not a by-name lookup (#180). Two same-named particles keep their
own fixed values, and an element matching a skip wildcard is no longer
validated against its global declaration. Selection is made reproducible
when an ambiguous schema slips past the UPA check.
XSTS runner: compile and merge every schemaDocument in a schemaTest, not
just the first; this also surfaced one previously-masked invalid-instance
acceptance (recorded in the baselines).
Also folds in attribute-group anyAttribute inheritance, extension wildcard
union, namespace-qualified type tables for multi-document
xsi:schemaLocation, per-container elementFormDefault, strict and lax
attribute wildcard validation, built-in type resolution, and xs:redefine
simpleType overrides.
Schema compile findings now carry ValidationError coding paths that resolve to source spans on a ranged tree, enabling IDE underlines for id, structure, include, and facet violations.
Ship the OpenAPIKit-style validation layer (builtin rules, warnings channel,
Validator tiers, coverage gates) and wire XSD compile checks through it. Advance
#161 with chameleon include validation, multi-container reference resolution,
and include targetNamespace composition rules so merged schema sets compile
correctly when a schemaLoader is provided.
Two more Particle Valid (Restriction) over-rejections.
A derived particle that can never contribute a child (its own maxOccurs="0", or
a group whose every member is content-free) accepts only the empty sequence, so
it validly restricts any emptiable base. Checked before the structural cases,
which previously recursed into the members of a maxOccurs="0" group and rejected
it for their own occurrences (particlesW006).
Substitution-group expansion now lists members in declaration order: the closure
walk used a popLast() stack that reversed them, so a reference to a head expanded
to choice(head, m2, m1) and an order-preserving RecurseLax restriction by
choice(m1, m2) could not map (elemZ027a). The choice/choice restriction stays
order-preserving, as the W3C suite requires (choice(b|a) is not a valid
restriction of choice(a|b), particlesT002); only the expansion order is fixed.
The expansion is shared with instance validation and the determinism check, both
order-neutral.
XSTS: valid schemas rejected 4 to 2, no other bucket raised. The pinned XSTS
baselines (never asserted under a plain swift test, the suite being opt-in) are
corrected to the measured 2/520/180/160.
An `<xs:element ref="p:foo">` had its prefix stripped and the element assumed to
be in the schema's target namespace. That mis-qualified every reference to an
imported element: a restriction of an `##other` wildcard by such a reference was
rejected (the element looked like the target namespace, which `##other` forbids),
and the same mis-qualification rejected valid instances whose elements were
correctly in the imported namespace.
The reference's QName is now resolved by standard XML namespace rules against the
schema root's xmlns bindings: a bound prefix gives its namespace (an imported
element lands in the imported namespace), an unprefixed name the default
namespace, or no namespace when none is declared. Substitution-group members
stay in the target namespace (this schema's own globals).
The content model is shared between schema-restriction checks and instance
validation, so both improve. XSTS: valid schemas rejected 32 to 4, invalid
schemas accepted 526 to 520, valid instances rejected 214 to 180, invalid
instances accepted held at 160.
Disclosed limitations: a prefix re-bound on an inner element is not honored,
cross-namespace substitution-group members are not modeled, and a dangling
reference is not yet flagged.
First cut at the schema-validity gap: the validator ran instance-validity rules
but not schema-validity rules, so it silently accepted invalid schemas. Per XSD
Part 2 Datatypes 4.3, a length-family facet value (length, minLength, maxLength,
totalDigits, fractionDigits) must be a valid nonNegativeInteger (totalDigits a
positiveInteger), length may not co-occur with minLength/maxLength, and the
integer ranges must be ordered. Malformed values (length="", "-1", "a", "1e2")
were silently dropped by numeric facet parsing and the schema accepted.
The definition is now read raw from the restriction and validated at compile
time. Well-formedness is checked independently of machine-integer range (via a
canonical digit string), so a huge but valid value is not misreported as
malformed, and ordering compares by length then lexically. Findings travel
through a shared reference-type diagnostics collector on the value-type parsing
context (surviving its copies) and join the model-level consistency findings
thrown from Schema.Document.init.
Limited to facets declared on the local restriction, so an inherited facet is
never wrongly flagged. The min/maxInclusive value-space rules, facet
applicability per base, and inherited co-occurrence remain for later iterations
(see docs/schema-validity-burndown.md).
XSTS: invalid schemas accepted 2461 to 2082, no change to any other bucket.
An <xs:attribute ref="..."> reference has no name attribute, so attributeUse
returned nil and the referenced attribute was dropped, making it look undeclared
on the instance (nested attributeGroup refs were already flattened, but the
per-attribute reference was not). References now resolve through a new global
attribute index to the declaration's type, with use/default/fixed taken from the
reference site and the name qualified to the target namespace.
XSTS: valid instances rejected 338 to 311. Baseline updated; regression tests
cover a referenced attribute admitted, its type enforced, and an attribute
outside the group still rejected. The attribute-parsing functions move to
XSDAttributeParser.swift to keep XSDParser within the type-body length limit.
Refs #147, #129.
The block on an element declaration (distinct from block on its type) was
parsed into the derivation tables but never threaded to the instance validator,
so an xsi:type that substitutes a type derived by a blocked method was accepted.
elementBlock now flows through XSDCompiled, SchemaDocument, the validation
context, and both ComplexValidator construction sites to blockedSubstitutionError,
which checks the element's own block alongside the type's. It also resolves the
element-ref type indirection (a ref'd child carries an `element:foo` key that
resolves through `types` to the concrete declared type) so the derivation chain
and block checks fire for referenced children, not only the document root.
XSTS: invalid instances accepted 261 to 240. Baseline updated; regression tests
cover element block rejecting a restriction xsi:type, the no-block control, and
an unrelated block not over-rejecting.
Refs #147, #129.
The derivation final check and Particle Valid (Restriction) move from
fail-fast throws at schema compile into composable Validation rules
(finalRespected, restrictionsAreSubsets) over per-type SchemaTypeFact
subjects, so a schema with several problems reports every finding at
once. SchemaError gains .inconsistent([String]) carrying all findings
(texts unchanged) and drops .finalViolation/.invalidRestriction.
checkRedefine, checkAllGroups, and notASchema stay throws: they examine
the raw schema source, the compilation analog of well-formedness.
Make XSD validation namespace-aware and enforce the form defaults and
xs:all structural constraints that were previously ignored:
- targetNamespace is tracked: a schema's global elements are in its
target namespace, and an instance's root element must be in it.
- elementFormDefault / attributeFormDefault (and per-declaration form)
qualify local element and attribute declarations. The content-model
matcher already compares namespace URIs, so qualified declarations
now match only same-namespace instance nodes; attribute matching
became namespace-aware too.
- xs:all groups are checked at compile time: every member must be an
element occurring at most once, the group may occur at most once, and
nested model groups inside an all are rejected (a SchemaError).
When a schema has no target namespace the stamping is a no-op, so
unqualified schemas validate exactly as before.
Cross-namespace type references through xs:import resolve by local name
only; full multi-namespace type tables are not modeled.
Closes#78
Read and enforce the XSD 1.0 derivation controls that were previously
ignored:
- abstract elements may not appear in an instance directly, only through
a substitution member; an element of an abstract complex type must
supply an xsi:type naming a concrete derived type.
- block on a type forbids the disallowed xsi:type substitutions; block on
an element drops substitution-group members that derive by a blocked
method.
- final on a type rejects, at compile time, any type that derives from it
by a final method.
- xs:redefine requires the redefined type to derive from itself.
Track each named complex type's base and derivation method as the
backbone these checks walk. Also compose complexContent extension and
restriction content models, which were previously dropped: an extension
now inherits its base's attributes and content (base first, then the
extension, in sequence).
Complex-type restriction is not verified to be a structurally valid
subset of its base (Particle Valid (Restriction)); that limit is
documented at its sites.
Closes#76
Add xs:any and xs:anyAttribute wildcards with namespace and
processContents constraints (skip/lax/strict), and resolve the
list built-in datatypes (IDREFS, ENTITIES, NMTOKENS) to their
item types so each list item validates.
WildcardNamespace models ##any/##other/enumerated URI sets;
Wildcard.admits gates undeclared attributes and wildcard-matched
children. ContentNFA.follow exposes the follow set for completions.
Closes#77
Closes#75, on the validation framework. XSD instance-level features:
- nillable elements: an xsi:nil="true" element is accepted (its content model is
not checked) only when the element is declared nillable and carries no content;
xsi:nil on a non-nillable element, or a nilled element with content, is a
located error.
- fixed value constraints: a present fixed attribute must equal its fixed value,
and a fixed element's text must match.
- default/fixed are parsed onto AttributeUse (per use) and collected per element
name (nillable set + value constraints), threaded through XSDContext to the
ComplexValidator.
All findings are located ValidationErrors. Collection and helper methods move to
same-file extensions to stay within type-body limits.
Validate XSD identity constraints over the instance document in a pass that
walks the tree alongside type validation:
- xs:unique requires the selected nodes' field tuples to be distinct, excluding
any node that is missing a field.
- xs:key additionally requires every field to be present.
- xs:keyref requires each tuple to equal a tuple of the named key or unique
visible in an enclosing scope, resolved through a stack of scope frames so a
keyref and the key it refers to may sit on the same element.
The selector and field XPaths run on the existing XPath engine, relative to the
declaring element and each selected node. Constraints are gathered for every
element declaration at any depth, keyed by element name.
Closes#67
Compose schemas across documents and resolve types at the instance:
- xs:include, xs:import, and xs:redefine pull a transitive closure of schema
documents through an injected loader (resolved by schemaLocation), base
schemas first so the including schema wins; a redefine's nested definitions
override the schema it includes.
- substitution groups: an xs:element ref to a head also admits every element
that substitutes for it, expanded at parse time into a choice over the head
and its transitive members.
- xsi:type overrides an element's declared type with the named type at the
instance; XML Schema instance attributes are exempt from attribute checks.
The schema parser splits across XSDContext (shared tree helpers, document
collection, parsing context), XSDSimpleParser (simple types and facets), and
XSDParser (elements, complex types, model groups) to keep each within bounds.
Closes#66