From ba934bbb585bd445410062342abf77849038f4b9 Mon Sep 17 00:00:00 2001 From: Mihaela Mihaljevic Date: Tue, 23 Jun 2026 01:19:43 +0200 Subject: [PATCH] fix(schema): reject unescaped '[' inside a regex character class (#247) XSD 1.0 Datatypes Appendix F defines SingleCharNoEsc as [^\[\]] and admits a nested class only as a -[...] subtraction, so a `[` inside a character class that is not introduced by `-` is a syntax error. RegexParser treated such a `[` as a literal scalar, so `[a[:xyz:]` compiled and the schema declaring it was wrongly accepted (XSTS RegexTest_993, RegexTest_1477). nextClassMember now throws a dedicated RegexError.unescapedClassBracket, kept distinct from the generic, deliberately-lenient badClass so the change cannot regress a valid-but-unsupported pattern. XSDSimpleParser.rejectsPattern treats it as a genuine syntax error, so the declaring schema is invalid. Closes RegexTest_993 and RegexTest_1477: invalid-schemas-accepted 15 -> 13, valid-schemas-rejected held at 0, instance buckets unchanged (full XSTS archive). Escaped [a\[c] and subtraction [a-z-[aeiou]] stay valid; the unterminated case [a[:xyz: now faults on the inner [. --- CHANGELOG.md | 2 ++ Sources/Regex/RegexParser.swift | 8 +++++++- Sources/Regex/RegexSyntax.swift | 8 +++++++- Sources/Schema/XSDSimpleParser.swift | 2 +- Tests/RegexTests.swift | 19 ++++++++++++++++++- Tests/SchemaPatternFacetTests.swift | 11 +++++++++++ Tests/XSTSSuiteTests.swift | 5 ++++- docs/release-hurdles.md | 10 +++++----- 8 files changed, 55 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index f474b48..c4b3eeb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- A `pattern` facet whose value contains an unescaped `[` inside a character class is now rejected at schema-compile time (false-negative burndown, stopper #2). XSD 1.0 Appendix F defines `SingleCharNoEsc` as `[^\[\]]` and admits a nested class only as a `-[...]` subtraction, so a `[` not introduced by `-` is a syntax error; `RegexParser` was treating it as a literal scalar, so `[a[:xyz:]` compiled and the declaring schema was accepted. The parser now throws a dedicated `RegexError.unescapedClassBracket` (kept distinct from the generic, deliberately-lenient `badClass` so the change cannot regress a valid-but-unsupported pattern), which `XSDSimpleParser.rejectsPattern` treats as a genuine syntax error. This closes XSTS RegexTest_993 and RegexTest_1477, dropping `invalid-schemas-accepted` from 15 to 13 with `valid-schemas-rejected` held at 0 (full XSTS archive) and the instance buckets unchanged (`invalid-instances-accepted` 19). The escaped form `[a\[c]` and the `-[...]` subtraction `[a-z-[aeiou]]` stay valid; the unterminated case `[a[:xyz:` now faults on the inner `[` (more precise than the prior unterminated-class diagnosis). Regressions are guarded by `RegexTests.test_unescapedClassBracket` and `SchemaPatternFacetTests`. + - Compiling a schema with deeply nested, multiply-referenced model groups is no longer exponential. `XSDParserParticles.groupReferenceParticle` inlined each `` into the compiled `Particle` content model, so `g0=(g1,g1), g1=(g2,g2), …` built `2^K` particles (measured: depth 14 = 0.21s, depth 20 ~13-18s, growing ~4x per two levels). The expansion is now memoized per (group definition, cycle-visiting set) in a per-compile `GroupParticleMemo`, so sibling references share it and the build drops to `O(groups)` (depth 40 now compiles in ~0.016s; previously infeasible). The shared content term is reused by value (copy-on-write), so the compiled model is byte-for-byte the same and validation is unchanged: the XSTS gate is unchanged (invalid-schemas-accepted 15, valid-schemas-rejected 0, instance buckets held) and the streaming-vs-tree fuzz differential still passes. This closes the schema-compile half of the interactive-safety group-blowup (the companion to the `positionCap` determinism fix); a regression is guarded by a depth-40 compile test. - The Unique Particle Attribution (UPA / cos-nonambig) determinism check is now compositional and no longer caps or skips. `ContentModelDeterminism` delegates to a new `CompositionalDeterminism` engine that computes `first`/`followlast` over PARTICLE identities and summarizes each `` once, so a multiply-referenced nested group whose inlined Glushkov automaton would be `2^K` positions is checked in `O(nodes * particles^2)` time instead of being silently skipped. This DELETES `ContentModelDeterminism.positionCap` (the last named interactive-safety resource-bound blocker, production-readiness stopper #4) and the silent under-rejection it caused (stopper #3). The compositional engine was proven verdict-equivalent to the prior inlining position automaton by a differential over the entire XSTS corpus (zero divergences) before replacing it; the XSTS conformance gate is unchanged (invalid-schemas-accepted 15, valid-schemas-rejected 0), and a new test compiles a 2^20-position nested-group schema without hanging and rejects an ambiguous one the old cap would have skipped. The per-particle model (a particle never conflicts with a repetition of itself) and the QName-only overlap (no substitution-group expansion) are preserved, so the false-positive direction stays at zero. diff --git a/Sources/Regex/RegexParser.swift b/Sources/Regex/RegexParser.swift index 74a305e..879f180 100644 --- a/Sources/Regex/RegexParser.swift +++ b/Sources/Regex/RegexParser.swift @@ -243,7 +243,13 @@ extension PureXML.Regex { private mutating func nextClassMember() throws -> ClassMember { guard let character = peek() else { throw RegexError.unterminatedClass } advance() - guard character == "\\" else { return .scalar(scalar(of: character)) } + guard character == "\\" else { + // A nested `[` is legal only as a `-[...]` subtraction, routed by + // parseClass/parseClassMember before reaching here; any other `[` + // is an unescaped class bracket, invalid per Appendix F. + if character == "[" { throw RegexError.unescapedClassBracket } + return .scalar(scalar(of: character)) + } guard let escaped = peek() else { throw RegexError.incompleteEscape } advance() if let single = Self.singleEscape(escaped) { return .scalar(single) } diff --git a/Sources/Regex/RegexSyntax.swift b/Sources/Regex/RegexSyntax.swift index 4725a78..951b6ed 100644 --- a/Sources/Regex/RegexSyntax.swift +++ b/Sources/Regex/RegexSyntax.swift @@ -18,8 +18,13 @@ extension PureXML.Regex { /// expression ended. Unambiguously invalid, never an engine limitation. case incompleteEscape /// A character class opened with `[` that the expression ends before - /// closing with `]` (`a[`, `[a[:xyz:`). Unambiguously invalid. + /// closing with `]` (`a[`). Unambiguously invalid. case unterminatedClass + /// An unescaped `[` inside a character class (`[a[bc]`, `[a[:xyz:]`): XSD + /// Appendix F `SingleCharNoEsc` excludes `[`, and a nested class is legal + /// only as a `-[...]` subtraction, so a `[` not introduced by `-` is a + /// syntax error. Unambiguously invalid, never an engine limitation. + case unescapedClassBracket /// The `{...}` of a `\p`/`\P` escape whose content can be no XSD charProp: /// empty, the bare prefix `Is`, a non-block name carrying a character that /// is not a letter, or `Is` followed by a non-block-name character @@ -40,6 +45,7 @@ extension PureXML.Regex { case .reversedQuantifier: "quantifier minimum exceeds maximum" case .incompleteEscape: "an escape '\\' has no following character" case .unterminatedClass: "a character class is not closed with ']'" + case .unescapedClassBracket: "an unescaped '[' inside a character class" case .invalidProperty: "a \\p{...} property name is malformed" } } diff --git a/Sources/Schema/XSDSimpleParser.swift b/Sources/Schema/XSDSimpleParser.swift index 9e4fa12..eb41d01 100644 --- a/Sources/Schema/XSDSimpleParser.swift +++ b/Sources/Schema/XSDSimpleParser.swift @@ -80,7 +80,7 @@ extension PureXML.Schema { private static func rejectsPattern(_ error: PureXML.Regex.RegexError) -> Bool { switch error { case .unbalanced, .danglingQuantifier, .reversedRange, .emptyClass, .reversedQuantifier, - .incompleteEscape, .unterminatedClass, .invalidProperty: + .incompleteEscape, .unterminatedClass, .unescapedClassBracket, .invalidProperty: true case .unsupported, .badQuantifier, .badEscape, .badClass: false diff --git a/Tests/RegexTests.swift b/Tests/RegexTests.swift index 9c521ee..7a38415 100644 --- a/Tests/RegexTests.swift +++ b/Tests/RegexTests.swift @@ -125,13 +125,30 @@ struct RegexTests { func test_unterminatedClass() throws { // A `[` the expression ends before closing with `]` is invalid. #expect(throws: PureXML.Regex.RegexError.unterminatedClass) { _ = try PureXML.Regex.Pattern("a[") } - #expect(throws: PureXML.Regex.RegexError.unterminatedClass) { _ = try PureXML.Regex.Pattern("[a[:xyz:") } #expect(throws: PureXML.Regex.RegexError.unterminatedClass) { _ = try PureXML.Regex.Pattern("[a-") } // A closed class, and a class whose only `]` is escaped, stay valid. #expect(try matches("[abc]", "b")) #expect(try matches("[a\\]]", "]")) } + @Test("An unescaped '[' inside a character class is rejected") + func test_unescapedClassBracket() throws { + // Per XSD Appendix F a literal `[` inside a class must be escaped (`\[`); + // a nested class is legal only as a `-[...]` subtraction. The XSTS cases + // RegexTest_993 / RegexTest_1477 declare the pattern `[a[:xyz:]`, an + // unescaped inner `[`, so the schema declaring them is invalid. + #expect(throws: PureXML.Regex.RegexError.unescapedClassBracket) { _ = try PureXML.Regex.Pattern("[a[:xyz:]") } + #expect(throws: PureXML.Regex.RegexError.unescapedClassBracket) { _ = try PureXML.Regex.Pattern("[a[bc]") } + #expect(throws: PureXML.Regex.RegexError.unescapedClassBracket) { _ = try PureXML.Regex.Pattern("[[abc]") } + // With no closing `]`, the inner `[` still faults first (more precise than + // the unterminated-class diagnosis the swallowed `[` used to produce). + #expect(throws: PureXML.Regex.RegexError.unescapedClassBracket) { _ = try PureXML.Regex.Pattern("[a[:xyz:") } + // An escaped `[` and a `-[...]` subtraction stay valid. + #expect(try matches("[a\\[c]", "[")) + #expect(try matches("[a-z-[aeiou]]", "b")) + #expect(try !matches("[a-z-[aeiou]]", "e")) + } + @Test("The empty pattern is valid and matches only the empty string") func test_emptyPattern() throws { // XSD grammar (Datatypes Appendix F): branch ::= piece*, so a branch diff --git a/Tests/SchemaPatternFacetTests.swift b/Tests/SchemaPatternFacetTests.swift index 49348b3..dde9fde 100644 --- a/Tests/SchemaPatternFacetTests.swift +++ b/Tests/SchemaPatternFacetTests.swift @@ -44,6 +44,17 @@ struct SchemaPatternFacetTests { try compile("(ab)*c?") } + @Test("an unescaped '[' inside a character class is rejected (XSTS RegexTest_993/1477)") + func test_unescapedClassBracket() { + // `[a[:xyz:]` (XSTS RegexTest_993 and RegexTest_1477) has an unescaped + // inner `[`, invalid per XSD Appendix F, so the schema is invalid. + #expect(rejects("[a[:xyz:]")) + #expect(rejects("[a[bc]")) + // The escaped form and a `-[...]` subtraction stay valid patterns. + #expect(!rejects("[a\\[c]")) + #expect(!rejects("[a-z-[aeiou]]")) + } + @Test("a construct the engine does not support is not treated as a schema error") func test_engineGapTolerated() throws { // Empty pattern (valid: matches the empty string), an untabulated Unicode diff --git a/Tests/XSTSSuiteTests.swift b/Tests/XSTSSuiteTests.swift index 3d80869..ca31dc5 100644 --- a/Tests/XSTSSuiteTests.swift +++ b/Tests/XSTSSuiteTests.swift @@ -31,7 +31,10 @@ struct XSTSSuiteTests { /// history of how each bucket reached its current value (every rule and its exact /// delta) lives in CHANGELOG.md and docs/xsts-deviations.md, not inline here. private let knownSchemaValidRejected = 0 - private let knownSchemaInvalidAccepted = 15 + /// Schema bucket, per-step deltas in CHANGELOG.md. Latest (15 -> 13): an + /// unescaped `[` inside a pattern-facet character class is a syntax error per + /// XSD Appendix F, so the schema is invalid (RegexTest_993, RegexTest_1477). + private let knownSchemaInvalidAccepted = 13 private let knownInstanceValidRejected = 0 /// Instance bucket (133 -> 22), per-step deltas in CHANGELOG.md: xsi:type must derive from the declared /// type; anyType cannot stand in for anySimpleType; an untyped substitutionGroup member inherits its head's diff --git a/docs/release-hurdles.md b/docs/release-hurdles.md index 99e8bc6..7350229 100644 --- a/docs/release-hurdles.md +++ b/docs/release-hurdles.md @@ -6,8 +6,8 @@ evidence, what actually stands between the current engine and a 1.0 fit to put i front of a developer as the authority on XML/XSD correctness. Snapshot at time of writing: v0.2.0 released; full 2006-11-06 XSTS archive (14383 -groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 43`, -`valid-instances-rejected = 0`, `invalid-instances-accepted = 31`. +groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 13`, +`valid-instances-rejected = 0`, `invalid-instances-accepted = 19`. ## Status against the four stoppers @@ -16,7 +16,7 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 43`, libxml2 differential over 9618 schemas. **Met.** This is the doc's bar to be "an authority"; it is cleared on the measured corpora. 2. **Silently accepting faulty input (false negatives).** `invalid-schemas-accepted - = 43`, `invalid-instances-accepted = 31`. **Not at 0.** This is the bulk of named + = 13`, `invalid-instances-accepted = 19`. **Not at 0.** This is the bulk of named milestone work (M2, M3) and the subject of Part 1 below. 3. **Uncharacterized correctness.** Differential harness vs libxml2 and a fuzz suite exist (M1, ~done). Gap: real-world corpora beyond XSTS, and the silent @@ -28,8 +28,8 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 43`, ## The biggest hurdle, stated plainly -Driving the false-negative buckets (`invalid-schemas-accepted = 43`, -`invalid-instances-accepted = 31`) to zero is the largest remaining body of named +Driving the false-negative buckets (`invalid-schemas-accepted = 13`, +`invalid-instances-accepted = 19`) to zero is the largest remaining body of named work, and it is hard for a specific reason: **the ~95% already closed were the tractable cases; every case left fights the cardinal rule.** The remainder sits in shared-oracle / contested-spec / multi-document-composition territory where the fix