diff --git a/CHANGELOG.md b/CHANGELOG.md index 74d9761..62d5a7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- Particle Valid (Restriction) now derives its pairwise check over the normalized content model, and applies occurrence subsumption only where the two particles' own ranges are comparable (#163). The W3C restriction algorithm is defined over a content model with pointless particles eliminated, so both the derived and base particles are first normalized (`maxOccurs="0"` members dropped, a `{1,1}` single-member group unwrapped to its member, a nested same-compositor `{1,1}` group spliced into its parent), all language-preserving. The outer occurrence-range check (`rangeSubsumed`) was being applied to every pairing, including the cross-kind ones where it compares mismatched ranges: a derived element's range against a base group's, or a derived group's against a base wildcard's. It is now applied only to the same-kind pairings (element/element, wildcard/wildcard, group/group) and element/wildcard; for element/group (RecurseAsIfGroup), group/element, and group/wildcard the occurrence is matched inside the case, against a member or as an effective total range. This unblocks a single-member derived sequence narrowing an occurrence (`sequence(e1{2,3})` restricting `sequence(e1{1,3}, ...)`, the `e1{2,3}` no longer compared to the base sequence's own `{1,1}`). A group restricting a wildcard (NSRecurseCheckCardinality) is now valid when every leaf it can contain is admitted by the wildcard and the group's effective total occurrence range lies within the wildcard's; a group restricting a single element is bounded the same way, by that element's name, type derivation, and occurrence range. XSTS: valid schemas rejected 45 to 32 (13 correct schemas no longer wrongly rejected, the `particlesHa` family of element-restricts-wildcard cardinality among them), invalid schemas accepted 528 to 526 (the cardinality bound also catches two over-wide restrictions), no other bucket raised. - Particle Valid (Restriction) no longer over-rejects valid restrictions (#148). Three corrections, all language-preserving: a particle that can never occur (`maxOccurs="0"`, a "pointless particle") is removed from both the derived and base content before mapping, so it no longer consumes a base particle and starves the ones after it (e.g. `(e1 maxOccurs="0", e2)` restricting `(any)` is now valid); a content-free restriction (an empty ``, or a group whose members all cannot occur) is a valid restriction of an EMPTY base, since both accept only the empty sequence; and a derived group that reduces to empty after that removal is a valid restriction of its base only when the base is itself emptiable (by structure or its own occurrence), closing a vacuous-acceptance gap the removal would otherwise open. XSTS: valid schemas rejected 72 to 45 (27 correct schemas no longer wrongly rejected), with one schema (`addB32`) moving to invalid-accepted because its only invalidity is a malformed imported document PureXML does not yet load (tracked by #161), not the restriction. - A `simpleType` must contain exactly one of `restriction`, `list`, or `union` (#158). Per XSD 1.0 Datatypes its content is `(annotation?, (restriction | list | union))`, but the membership table admitted each name without the cardinality, so an empty `simpleType` or one combining two derivations (two `union`s, a `union` and a `list`, two `restriction`s) was accepted (the stB family). The exactly-one rule is now enforced. XSTS: invalid schemas accepted 537 to 527, no other bucket raised (valid schemas rejected held at 72). - The `final` and `block` derivation-control attributes are now checked against their value space (#158). Each is `#all` or a whitespace list of method tokens, but the admitted tokens depend on the component: an element's `final` is `extension`/`restriction` and its `block` adds `substitution`; a complexType's `final`/`block` are `extension`/`restriction`; a simpleType's `final` is `list`/`union`/`restriction`. The tokens were parsed leniently (unknown tokens ignored, `substitution` admitted everywhere), so `final="substitution"` on an element, a misspelled or miscased token, or `#all` mixed with other tokens was accepted (the elemF/st_final families). The list is now validated against the component's exact set; `finalDefault`/`blockDefault` (the wider schema-level value space) and foreign attributes are left alone. XSTS: invalid schemas accepted 580 to 537, no other bucket raised (valid schemas rejected held at 72). diff --git a/Sources/Schema/ParticleNormalization.swift b/Sources/Schema/ParticleNormalization.swift new file mode 100644 index 0000000..c33bf9f --- /dev/null +++ b/Sources/Schema/ParticleNormalization.swift @@ -0,0 +1,55 @@ +extension PureXML.Schema.Particle { + /// Eliminates pointless particles so the restriction algorithm sees the spec's + /// normalized content model: a `maxOccurs=0` member contributes nothing and is + /// dropped; a nested same-compositor `{1,1}` group is spliced into its parent; + /// a `{1,1}` group with a single member is unwrapped to that member. + /// Language-preserving. + func normalized() -> Self { + guard case let .group(group) = term else { return self } + var members: [Self] = [] + for child in group.particles { + let normalized = child.normalized() + if normalized.maxOccurs == 0 { continue } + let splice = normalized.minOccurs == 1 && normalized.maxOccurs == 1 + if case let .group(inner) = normalized.term, inner.compositor == group.compositor, splice { + members.append(contentsOf: inner.particles) + } else { + members.append(normalized) + } + } + if members.count == 1, minOccurs == 1, maxOccurs == 1 { + return members[0] + } + return Self(minOccurs: minOccurs, maxOccurs: maxOccurs, term: .group(.init(compositor: group.compositor, particles: members))) + } + + /// The minimum number of elements this particle can contribute (effective total + /// range, lower bound): a leaf is its `minOccurs`; a sequence/all sums its + /// members, a choice takes the smallest; times the particle's own `minOccurs`. + func effectiveOccurrenceMin() -> Int { + switch term { + case .element, .wildcard: + return minOccurs + case let .group(group): + let inner: Int = switch group.compositor { + case .choice: group.particles.map { $0.effectiveOccurrenceMin() }.min() ?? 0 + case .sequence, .all: group.particles.map { $0.effectiveOccurrenceMin() }.reduce(0, +) + } + return minOccurs * inner + } + } + + /// The maximum number of elements this particle can contribute (nil = unbounded). + func effectiveOccurrenceMax() -> Int? { + switch term { + case .element, .wildcard: + return maxOccurs + case let .group(group): + let childMaxes = group.particles.map { $0.effectiveOccurrenceMax() } + guard let particleMax = maxOccurs, !childMaxes.contains(where: { $0 == nil }) else { return nil } + let maxes = childMaxes.compactMap(\.self) + let inner = group.compositor == .choice ? (maxes.max() ?? 0) : maxes.reduce(0, +) + return particleMax * inner + } + } +} diff --git a/Sources/Schema/ParticleRestriction.swift b/Sources/Schema/ParticleRestriction.swift index 3db3779..89ecf7c 100644 --- a/Sources/Schema/ParticleRestriction.swift +++ b/Sources/Schema/ParticleRestriction.swift @@ -31,26 +31,63 @@ extension PureXML.Schema { case let (.elementOnly(restrictedParticle), .elementOnly(baseParticle)), let (.elementOnly(restrictedParticle), .mixed(baseParticle)), let (.mixed(restrictedParticle), .mixed(baseParticle)): - valid(restrictedParticle, baseParticle, types, derivation) + // The W3C restriction algorithm is defined over the normalized content + // model (pointless particles eliminated): a `maxOccurs=0` member + // removed, a `{1,1}` single-member group unwrapped, a nested + // same-compositor `{1,1}` group spliced. Without this, e.g. a base + // `sequence(any{2,3})` is not seen as the wildcard it equals. + valid(restrictedParticle.normalized(), baseParticle.normalized(), types, derivation) ? nil : "the restricted content model is not a subset of the base's" } } /// Whether `restricted` accepts a subset of `base` (the pairwise check). + /// + /// Occurrence subsumption (`rangeSubsumed`) is applied only to the + /// same-kind pairings (element/element, wildcard/wildcard, group/group) and + /// element/wildcard, where the two particles' own occurrence ranges are + /// directly comparable. For the cross-kind pairings (element/group and + /// group/element via RecurseAsIfGroup; group/wildcard via + /// NSRecurseCheckCardinality) the occurrence is matched *inside* the case, + /// against a member or as an effective total range, never the derived + /// particle's own occurrence against the base group/wildcard's. Applying the + /// outer check there over-rejects: e.g. `sequence(e1{2,3})` restricting + /// `sequence(e1{1,3},...)` once the pointless single-member group is + /// normalized to the bare `e1{2,3}` (`3 > 1` against the base sequence). static func valid(_ restricted: Particle, _ base: Particle, _ types: [String: ElementType], _ derivation: [String: TypeDerivation]) -> Bool { - guard rangeSubsumed(restricted, base) else { return false } switch (restricted.term, base.term) { case let (.element(restrictedName, _, restrictedTypeName), .element(baseName, _, baseTypeName)): - return sameName(restrictedName, baseName) && elementTypeRestrictionOK(restrictedTypeName, baseTypeName, types, derivation) + return rangeSubsumed(restricted, base) + && sameName(restrictedName, baseName) + && elementTypeRestrictionOK(restrictedTypeName, baseTypeName, types, derivation) case let (.element(name, _, _), .wildcard(wildcard)): - return wildcard.admits(name) + return rangeSubsumed(restricted, base) && wildcard.admits(name) case let (.wildcard(restrictedWildcard), .wildcard(baseWildcard)): - return narrows(restrictedWildcard, baseWildcard) + return rangeSubsumed(restricted, base) && narrows(restrictedWildcard, baseWildcard) case let (.element, .group(baseGroup)): // RecurseAsIfGroup: the lone element, as a one-particle sequence, - // against the base group. - return groupValid(Group(compositor: .sequence, particles: [restricted.withUnitRange()]), baseGroup, types, derivation) + // against the base group. The element's occurrence is matched against + // a base member inside `groupValid`, not against the base group's own + // occurrence, so no outer `rangeSubsumed` here. + return groupValid(Group(compositor: .sequence, particles: [restricted]), baseGroup, types, derivation) + case let (.group, .element(baseName, _, baseTypeName)): + // The mirror of RecurseAsIfGroup, by the same cardinality argument as + // group/wildcard: a base element accepts only its own name, its own + // occurrence count of times. The derived group is a valid restriction + // iff every leaf it can contain is that element (same name, and a type + // that derives from the base's) AND the group's effective total + // occurrence range is within the base element's. Comparing the derived + // group's *own* occurrence to the element's would be wrong (a derived + // `sequence(a,a)` emits two a's at occurrence {1,1}); the effective + // range is the count that must be bounded. + return leavesRestrict(restricted, toElement: baseName, baseTypeName, types, derivation) + && rangeWithinWildcard( + derivedMin: restricted.effectiveOccurrenceMin(), + derivedMax: restricted.effectiveOccurrenceMax(), + wildcardMin: base.minOccurs, + wildcardMax: base.maxOccurs, + ) case let (.group(restrictedGroup), .group(baseGroup)): // If every member of the derived group is a pointless (maxOccurs=0) // particle, the group accepts only the empty sequence; that is a valid @@ -60,11 +97,19 @@ extension PureXML.Schema { if restrictedGroup.particles.allSatisfy({ $0.maxOccurs == 0 }) { return emptiable(base) } - return groupValid(restrictedGroup, baseGroup, types, derivation) - case let (.group(restrictedGroup), .wildcard(wildcard)): - // NSRecurseCheckCardinality, namespace part: every leaf element - // the group can contain must be admitted by the wildcard. - return leafNames(of: restrictedGroup).allSatisfy { wildcard.admits($0) } + return rangeSubsumed(restricted, base) && groupValid(restrictedGroup, baseGroup, types, derivation) + case let (.group, .wildcard(wildcard)): + // NSRecurseCheckCardinality: every leaf the group can contain is + // admitted by the wildcard (an element by name, a nested wildcard by + // narrowing), AND the group's effective total occurrence range is + // within the wildcard's occurrence range. + return leavesAdmitted(restricted, by: wildcard) + && rangeWithinWildcard( + derivedMin: restricted.effectiveOccurrenceMin(), + derivedMax: restricted.effectiveOccurrenceMax(), + wildcardMin: base.minOccurs, + wildcardMax: base.maxOccurs, + ) default: return false } @@ -271,22 +316,44 @@ extension PureXML.Schema { lhs.localName == rhs.localName && (lhs.namespaceURI ?? "") == (rhs.namespaceURI ?? "") } - /// Every element name a group can contain (its alphabet). - private static func leafNames(of group: Group) -> [PureXML.Model.QualifiedName] { - group.particles.flatMap { particle -> [PureXML.Model.QualifiedName] in - switch particle.term { - case let .element(name, _, _): [name] - case let .group(inner): leafNames(of: inner) - case .wildcard: [] - } + /// Whether every leaf a particle can contain is admitted by `wildcard`: an + /// element by name, a nested wildcard by narrowing. + private static func leavesAdmitted(_ particle: Particle, by wildcard: Wildcard) -> Bool { + switch particle.term { + case let .element(name, _, _): wildcard.admits(name) + case let .wildcard(inner): narrows(inner, wildcard) + case let .group(group): group.particles.allSatisfy { leavesAdmitted($0, by: wildcard) } } } + + /// Whether every leaf a particle can contain is the named base element (same + /// name, and a type that derives from the base element's). A wildcard leaf is + /// rejected: it can match names other than the base element's, so it is not a + /// subset. Used when the base content model is a single element. + private static func leavesRestrict( + _ particle: Particle, + toElement baseName: PureXML.Model.QualifiedName, + _ baseTypeName: String?, + _ types: [String: ElementType], + _ derivation: [String: TypeDerivation], + ) -> Bool { + switch particle.term { + case let .element(name, _, typeName): + sameName(name, baseName) && elementTypeRestrictionOK(typeName, baseTypeName, types, derivation) + case .wildcard: + false + case let .group(group): + group.particles.allSatisfy { leavesRestrict($0, toElement: baseName, baseTypeName, types, derivation) } + } + } + + /// Whether a derived effective total range fits within a base wildcard's + /// occurrence range: at least the wildcard's minimum, at most its maximum. + private static func rangeWithinWildcard(derivedMin: Int, derivedMax: Int?, wildcardMin: Int, wildcardMax: Int?) -> Bool { + guard derivedMin >= wildcardMin else { return false } + guard let wildcardMax else { return true } + guard let derivedMax else { return false } + return derivedMax <= wildcardMax + } } } - -private extension PureXML.Schema.Particle { - /// A copy occurring exactly once, for RecurseAsIfGroup's wrapping. - func withUnitRange() -> PureXML.Schema.Particle { - .init(minOccurs: minOccurs, maxOccurs: maxOccurs, term: term) - } -} diff --git a/Tests/XSDParticleRestrictionTests.swift b/Tests/XSDParticleRestrictionTests.swift index 6454704..0d30967 100644 --- a/Tests/XSDParticleRestrictionTests.swift +++ b/Tests/XSDParticleRestrictionTests.swift @@ -136,6 +136,76 @@ struct XSDParticleRestrictionTests { #expect(error?.contains("not a valid restriction") == true) } + @Test("A single-member derived sequence narrowing an occurrence restricts a wider base sequence") + func test_singleMemberSequenceNarrowsOccurrence() { + // particlesW008: the derived sequence has one member, so it normalizes to the + // bare element e1{2,3}; that must still be matched as a member of the base + // sequence (RecurseAsIfGroup), not have its {2,3} compared to the base + // sequence's own {1,1} occurrence. + #expect(compiles( + base: "" + + "" + + "", + restriction: "", + )) + // particlesM002: a derived single-branch choice normalizes to the bare element + // and still restricts a choice of sequences whose first branch begins with it. + #expect(compiles( + base: "" + + "" + + "", + restriction: "", + )) + } + + @Test("A group restricting a wildcard is valid only when its effective count fits the wildcard's range") + func test_groupRestrictsWildcardCardinality() { + // particlesHa: a sequence of named elements restricts a wildcard when every leaf + // is admitted AND the effective total count lies within the wildcard's range. + #expect(compiles( + base: "", + restriction: "", + )) + // Too many leaves: three elements exceed the wildcard's maxOccurs of 2. + #expect(restrictionError( + base: "", + restriction: "", + )?.contains("not a valid restriction") == true) + // Too few leaves: one element is below the wildcard's minOccurs of 2. + #expect(restrictionError( + base: "", + restriction: "", + )?.contains("not a valid restriction") == true) + } + + @Test("A group restricting a base that is a single element is bounded by that element's range") + func test_groupRestrictsSingleElement() { + // mgE014: base sequence(title?) normalizes to the bare element title{0,1}; + // the derived sequence(title{0,0}) normalizes to the empty group, which + // accepts only the empty sequence, valid against an emptiable element. + #expect(compiles( + base: "", + restriction: "", + )) + // A derived choice repeated within the element's range is valid (the base + // element, normalized from a single-member sequence, is compositor-agnostic). + #expect(compiles( + base: "", + restriction: "", + )) + // A derived group whose effective count exceeds the base element's maxOccurs is + // rejected: sequence(a){1,10} emits up to ten a's, the base allows three. + #expect(restrictionError( + base: "", + restriction: "", + )?.contains("not a valid restriction") == true) + // A derived leaf with a different name than the base element is rejected. + #expect(restrictionError( + base: "", + restriction: "", + )?.contains("not a valid restriction") == true) + } + @Test("Conformance: the restriction corpus passes through the harness") func test_conformanceCases() { let valid = compiles( diff --git a/Tests/XSTSSuiteTests.swift b/Tests/XSTSSuiteTests.swift index f2d9783..4905d89 100644 --- a/Tests/XSTSSuiteTests.swift +++ b/Tests/XSTSSuiteTests.swift @@ -28,8 +28,8 @@ struct XSTSSuiteTests { /// `swift test` are unaffected; run it with `swift test -c release --filter /// XSTS` (debug is far slower and the corpus is large). Per-case deviations /// are written to /tmp/xsts-failures.txt for the burn-down. - private let knownSchemaValidRejected = 45 - private let knownSchemaInvalidAccepted = 528 + private let knownSchemaValidRejected = 32 + private let knownSchemaInvalidAccepted = 526 private let knownInstanceValidRejected = 214 private let knownInstanceInvalidAccepted = 160 diff --git a/docs/schema-validity-burndown.md b/docs/schema-validity-burndown.md index e425bd8..be60caf 100644 --- a/docs/schema-validity-burndown.md +++ b/docs/schema-validity-burndown.md @@ -9,8 +9,8 @@ irreducible tail. Do not deviate to new features while this is open. | Bucket | Count | Reading | |---|---|---| -| valid-schemas-rejected | **72 -> 45** | production target is 0: rejecting a correct schema is the most user-hostile defect (#148). Particle-restriction over-rejection fixed: pointless `maxOccurs=0` particles removed, content-free restriction of an EMPTY base, empty-derived emptiable-base. | -| **invalid-schemas-accepted** | **2461 -> 528** | we rarely catch a bad one (facet, default/fixed value, particle-restriction name+type, id, structural content-model, UPA determinism, content order, identity-constraint XPath-subset, wildcard-namespace, final/block, simpleType content landed). The 527->528 step is `addB32`, invalid only via a malformed import (#161), unmasked by removing a wrong restriction rejection. | +| valid-schemas-rejected | **72 -> 32** | production target is 0: rejecting a correct schema is the most user-hostile defect (#148). Particle-restriction over-rejection fixed: pointless `maxOccurs=0` particles removed, content-free restriction of an EMPTY base, empty-derived emptiable-base, and (#163) the pairwise check over the normalized content model with occurrence subsumption applied only to same-kind pairings (cross-kind element/group, group/element, group/wildcard match occurrence inside, by effective total range / member). Remaining 32 are mostly element-ref namespace resolution (`particlesJj/Jk/Jm`, #164). | +| **invalid-schemas-accepted** | **2461 -> 526** | we rarely catch a bad one (facet, default/fixed value, particle-restriction name+type, id, structural content-model, UPA determinism, content order, identity-constraint XPath-subset, wildcard-namespace, final/block, simpleType content, group/wildcard and group/element cardinality landed). | | valid-instances-rejected | 233 | (instance side, separately tracked) | | invalid-instances-accepted | 165 | |