diff --git a/CHANGELOG.md b/CHANGELOG.md index 9a8fc4e..f6ca59f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- `pattern` facet regular expressions are now validated at compile time (#147). Each `pattern` value is compiled when the schema is; a structurally malformed one (unbalanced parentheses `((a)`, a quantifier with nothing to repeat `?a`) is rejected, where before it was only compiled lazily at instance time (`try?`) and so silently ignored. The check is conservative: a construct the engine merely does not support on an otherwise-valid pattern (the empty pattern, an untabulated `\p{Is...}` block such as a surrogate or historic block, the lenient `{,m}` quantifier) is tolerated, so no valid schema is rejected. XSTS: invalid schemas accepted 1129 to 1092, no other bucket raised. - Schema-vocabulary name and reference attribute values are now checked for lexical validity at compile time (#147). A component's `name` must be a valid NCName (`name=""`, `name="123"`, `name="a:b"` are rejected), and the reference attributes `type`/`base`/`ref`/`itemType`/`refer`/`substitutionGroup` must be lexical QNames (`type=":_"`, `type="a:b:c"` rejected), with `memberTypes` a list of QNames; these were unchecked and the schema accepted. Only the lexical form is checked (a QName's prefix is resolved elsewhere), so no valid schema is rejected. XSTS: invalid schemas accepted 1190 to 1129, no other bucket raised. - Schema-vocabulary attribute values are now validated at compile time (#147), the attribute-model sibling of the structural content-model check. Enumerated attributes (`form`/`elementFormDefault`/`attributeFormDefault` to `qualified`/`unqualified`, `use` to `optional`/`prohibited`/`required`, `processContents` to `skip`/`lax`/`strict`, `mixed`/`abstract`/`nillable` to a boolean) and `minOccurs`/`maxOccurs` (a `nonNegativeInteger`, `maxOccurs` also `unbounded`) are checked against their fixed value space; an out-of-space value (`use="foo"`, `mixed="yes"`, `minOccurs="-1"`) was previously ignored and the schema accepted. Only unprefixed attributes on XSD-namespace elements are checked, so a foreign attribute is never flagged. XSTS: invalid schemas accepted 1313 to 1190, no other bucket raised. - Schema documents are now structurally validated against the schema-for-schemas content model (#147), the general mechanism for the structural tail of the schema-validity gap. A data-driven table of the XSD 1.0 child content model rejects, at compile time, any component child the model does not admit, more than one `annotation` (or an `annotation` that is not first, except on `schema`/`redefine`), and an identity constraint missing its selector or field. The check walks only XSD-namespace elements and skips `xs:appinfo`/`xs:documentation` foreign content, so no valid schema is rejected; `restriction`/`extension` use the permissive union of their contexts (never over-rejecting). One table covers every component rather than a rule hand-coded per element. XSTS: invalid schemas accepted 1688 to 1313 (-375), no other bucket raised. diff --git a/Sources/Schema/XSDSimpleParser.swift b/Sources/Schema/XSDSimpleParser.swift index 98abc35..4e1af9c 100644 --- a/Sources/Schema/XSDSimpleParser.swift +++ b/Sources/Schema/XSDSimpleParser.swift @@ -27,12 +27,48 @@ extension PureXML.Schema { facets.enumeration = nil } applyFacets(restriction, into: &facets) - for error in facetDefinitionErrors(restriction, base: baseType) { + for error in facetDefinitionErrors(restriction, base: baseType) + patternErrors(restriction) { context.diagnostics.report(error) } return SimpleType(base: baseType.base, facets: facets, variety: baseType.variety) } + /// Schema-validity findings for the `pattern` facets declared on one + /// `restriction`: each `pattern` value must be a valid XSD regular + /// expression. The pattern was only compiled lazily at instance time + /// (`try?`), so an unparseable one was silently ignored and the schema + /// accepted. Only the unambiguous structural errors reject (a quantifier + /// with nothing to repeat, `?a`; unbalanced parentheses, `((a)`); see + /// `rejectsPattern`. + static func patternErrors(_ restriction: XSDTree) -> [String] { + var errors: [String] = [] + for facet in XSDNode.elementChildren(restriction) where XSDNode.localName(facet) == "pattern" { + let value = XSDNode.attribute(facet, "value") ?? "" + do { + _ = try PureXML.Regex.Pattern(value) + } catch let error as PureXML.Regex.RegexError where rejectsPattern(error) { + errors.append("pattern '\(value)' is not a valid regular expression: \(error)") + } catch { + // A construct the engine does not support (`.empty`, `.unsupported` + // for an untabulated `\p{Is...}` block, `.badQuantifier` for the + // lenient `{,m}` form) is a valid XSD pattern, not a schema error. + } + } + return errors + } + + /// Whether a regex-compile failure is a genuine syntax error (so the schema + /// is invalid), as opposed to an engine limitation on an otherwise-valid XSD + /// pattern. Conservative: only the unambiguous structural errors reject. + private static func rejectsPattern(_ error: PureXML.Regex.RegexError) -> Bool { + switch error { + case .unbalanced, .danglingQuantifier: + true + case .empty, .unsupported, .badQuantifier, .badEscape, .badClass: + false + } + } + /// Schema-validity findings for the constraining facets declared on one /// `restriction` (XSD Part 2 Datatypes 4.3): the length-family rules plus /// the value-bound rules, both limited to the facets declared on this step diff --git a/Tests/SchemaPatternFacetTests.swift b/Tests/SchemaPatternFacetTests.swift new file mode 100644 index 0000000..17fd209 --- /dev/null +++ b/Tests/SchemaPatternFacetTests.swift @@ -0,0 +1,55 @@ +@testable import PureXML +import Testing + +/// A `pattern` facet value must be a valid XSD regular expression. The pattern +/// was only compiled lazily at instance time (`try?`), so an unparseable one was +/// silently ignored and the schema accepted. Compile-time validation rejects the +/// unambiguous structural errors (unbalanced parentheses, a quantifier with +/// nothing to repeat) while tolerating constructs the engine merely does not +/// support on an otherwise-valid pattern (an untabulated `\p{Is...}` block, the +/// empty pattern, the lenient `{,m}` quantifier), so no valid schema is rejected. +@Suite("Pattern facet regex validity") +struct SchemaPatternFacetTests { + private func compile(_ pattern: String) throws { + _ = try PureXML.Schema.Document(""" + + + + + + + + """) + } + + private func rejects(_ pattern: String) -> Bool { + do { try compile(pattern) + return false + } catch { return true } + } + + @Test("a structurally malformed pattern is rejected") + func test_malformed() { + #expect(rejects("((a)")) + #expect(rejects("(a))")) + #expect(rejects("?a")) + #expect(rejects("*a")) + #expect(rejects("+a")) + } + + @Test("a valid pattern compiles") + func test_valid() throws { + try compile("[a-z]+") + try compile("[0-9]{2,4}") + try compile("(ab)*c?") + } + + @Test("a construct the engine does not support is not treated as a schema error") + func test_engineGapTolerated() throws { + // Empty pattern (valid: matches the empty string), an untabulated Unicode + // block escape, and the lenient `{,m}` quantifier are all accepted. + try compile("") + try compile(#"\p{IsHighSurrogates}"#) + try compile("[0-9]{,5}") + } +} diff --git a/Tests/XSTSSuiteTests.swift b/Tests/XSTSSuiteTests.swift index 317a9c9..1688a93 100644 --- a/Tests/XSTSSuiteTests.swift +++ b/Tests/XSTSSuiteTests.swift @@ -29,7 +29,7 @@ struct XSTSSuiteTests { /// XSTS` (debug is far slower and the corpus is large). Per-case deviations /// are written to /tmp/xsts-failures.txt for the burn-down. private let knownSchemaValidRejected = 72 - private let knownSchemaInvalidAccepted = 1129 + private let knownSchemaInvalidAccepted = 1092 private let knownInstanceValidRejected = 214 private let knownInstanceInvalidAccepted = 165 diff --git a/docs/schema-validity-burndown.md b/docs/schema-validity-burndown.md index 1e788a8..7ca31d3 100644 --- a/docs/schema-validity-burndown.md +++ b/docs/schema-validity-burndown.md @@ -10,7 +10,7 @@ irreducible tail. Do not deviate to new features while this is open. | Bucket | Count | Reading | |---|---|---| | valid-schemas-rejected | 72 | we rarely reject a good schema | -| **invalid-schemas-accepted** | **2461 -> 1129** | we rarely catch a bad one (facet, id, and structural content-model validity landed) | +| **invalid-schemas-accepted** | **2461 -> 1092** | we rarely catch a bad one (facet, id, and structural content-model validity landed) | | valid-instances-rejected | 233 | (instance side, separately tracked) | | invalid-instances-accepted | 165 | | @@ -73,7 +73,7 @@ Each iteration targets one self-contained rule family with a clean root cause. (1688 -> 1313, -375)._ A data-driven `allowedChildren` table (the schema-for- schemas child content model) rejects disallowed children, multi/misplaced `annotation`, and identity constraints missing selector/field, via one general - mechanism (`SchemaStructure.swift`). Attribute **value** validity also landed (enumerated attributes, `minOccurs`/`maxOccurs`; 1313 -> 1190), plus **name/reference lexical validity** (`name` an NCName; `type`/`base`/`ref`/`itemType`/`refer`/`substitutionGroup` QNames; 1190 -> 1129). **Still open:** attribute **applicability** (which attributes each component admits, e.g. `form` is prohibited on a global declaration, `ref` excludes `name`); child *order* beyond the leading annotation; and particle / model-group semantic rules (Unique Particle Attribution, Particle-Valid-Restriction completeness). + mechanism (`SchemaStructure.swift`). Attribute **value** validity also landed (enumerated attributes, `minOccurs`/`maxOccurs`; 1313 -> 1190), plus **name/reference lexical validity** (`name` an NCName; `type`/`base`/`ref`/`itemType`/`refer`/`substitutionGroup` QNames; 1190 -> 1129), plus **`pattern` regex validity** (compile each pattern; reject only unambiguous syntax errors, tolerating engine gaps; 1129 -> 1092). **Still open:** attribute **applicability** (which attributes each component admits, e.g. `form` is prohibited on a global declaration, `ref` excludes `name`); child *order* beyond the leading annotation; and particle / model-group semantic rules (Unique Particle Attribution, Particle-Valid-Restriction completeness). ## Per-iteration protocol (the PR-critic-loop) diff --git a/docs/xsts-deviations.md b/docs/xsts-deviations.md index 3965400..e6f92cd 100644 --- a/docs/xsts-deviations.md +++ b/docs/xsts-deviations.md @@ -47,7 +47,7 @@ and to document the remainder as spec-justified, deliberate exclusions. ## Category A: genuine correctness bugs -### Fixed (twenty root causes, ~2006 deviations cleared) +### Fixed (twenty-one root causes, ~2043 deviations cleared) list-datatype length facets (#146) · union pattern/enumeration facets · duration partial-order facets · QName length non-constraining · the full XSD `\p{Is...}` @@ -65,7 +65,7 @@ validity (`min`/`maxInclusive`, `min`/`maxExclusive`, `enumeration` values valid in the base space; inclusive/exclusive exclusions; bound ordering) · `gMonth` `--MM--` lexical form · schema `id` attribute validity (`xs:ID`: NCName and unique within the document). The schema-validity work brought invalid schemas -accepted 2461 to 1129 (also: schema-document structural validity, content model + attribute values + name/reference lexical validity, against the schema-for-schemas); see `schema-validity-burndown.md`. +accepted 2461 to 1092 (also: schema-document structural validity, content model + attribute values + name/reference + pattern-regex lexical validity, against the schema-for-schemas); see `schema-validity-burndown.md`. Plus the measurement fix (exclude W3C-disputed `status="queried"` entries). ### Remaining, root-caused (the hard tail)