fix(schema): reject unescaped '[' inside a regex character class (#247)

XSD 1.0 Datatypes Appendix F defines SingleCharNoEsc as [^\[\]] and admits a
nested class only as a -[...] subtraction, so a `[` inside a character class
that is not introduced by `-` is a syntax error. RegexParser treated such a
`[` as a literal scalar, so `[a[:xyz:]` compiled and the schema declaring it
was wrongly accepted (XSTS RegexTest_993, RegexTest_1477).

nextClassMember now throws a dedicated RegexError.unescapedClassBracket, kept
distinct from the generic, deliberately-lenient badClass so the change cannot
regress a valid-but-unsupported pattern. XSDSimpleParser.rejectsPattern treats
it as a genuine syntax error, so the declaring schema is invalid.

Closes RegexTest_993 and RegexTest_1477: invalid-schemas-accepted 15 -> 13,
valid-schemas-rejected held at 0, instance buckets unchanged (full XSTS
archive). Escaped [a\[c] and subtraction [a-z-[aeiou]] stay valid; the
unterminated case [a[:xyz: now faults on the inner [.
This commit is contained in:
Mihaela Mihaljevic 2026-06-23 01:19:43 +02:00 committed by GitHub
parent bb094aa13c
commit ba934bbb58
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
8 changed files with 55 additions and 10 deletions

View File

@ -34,6 +34,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- A `pattern` facet whose value contains an unescaped `[` inside a character class is now rejected at schema-compile time (false-negative burndown, stopper #2). XSD 1.0 Appendix F defines `SingleCharNoEsc` as `[^\[\]]` and admits a nested class only as a `-[...]` subtraction, so a `[` not introduced by `-` is a syntax error; `RegexParser` was treating it as a literal scalar, so `[a[:xyz:]` compiled and the declaring schema was accepted. The parser now throws a dedicated `RegexError.unescapedClassBracket` (kept distinct from the generic, deliberately-lenient `badClass` so the change cannot regress a valid-but-unsupported pattern), which `XSDSimpleParser.rejectsPattern` treats as a genuine syntax error. This closes XSTS RegexTest_993 and RegexTest_1477, dropping `invalid-schemas-accepted` from 15 to 13 with `valid-schemas-rejected` held at 0 (full XSTS archive) and the instance buckets unchanged (`invalid-instances-accepted` 19). The escaped form `[a\[c]` and the `-[...]` subtraction `[a-z-[aeiou]]` stay valid; the unterminated case `[a[:xyz:` now faults on the inner `[` (more precise than the prior unterminated-class diagnosis). Regressions are guarded by `RegexTests.test_unescapedClassBracket` and `SchemaPatternFacetTests`.
- Compiling a schema with deeply nested, multiply-referenced model groups is no longer exponential. `XSDParserParticles.groupReferenceParticle` inlined each `<xs:group ref>` into the compiled `Particle` content model, so `g0=(g1,g1), g1=(g2,g2), …` built `2^K` particles (measured: depth 14 = 0.21s, depth 20 ~13-18s, growing ~4x per two levels). The expansion is now memoized per (group definition, cycle-visiting set) in a per-compile `GroupParticleMemo`, so sibling references share it and the build drops to `O(groups)` (depth 40 now compiles in ~0.016s; previously infeasible). The shared content term is reused by value (copy-on-write), so the compiled model is byte-for-byte the same and validation is unchanged: the XSTS gate is unchanged (invalid-schemas-accepted 15, valid-schemas-rejected 0, instance buckets held) and the streaming-vs-tree fuzz differential still passes. This closes the schema-compile half of the interactive-safety group-blowup (the companion to the `positionCap` determinism fix); a regression is guarded by a depth-40 compile test.
- The Unique Particle Attribution (UPA / cos-nonambig) determinism check is now compositional and no longer caps or skips. `ContentModelDeterminism` delegates to a new `CompositionalDeterminism` engine that computes `first`/`followlast` over PARTICLE identities and summarizes each `<xs:group>` once, so a multiply-referenced nested group whose inlined Glushkov automaton would be `2^K` positions is checked in `O(nodes * particles^2)` time instead of being silently skipped. This DELETES `ContentModelDeterminism.positionCap` (the last named interactive-safety resource-bound blocker, production-readiness stopper #4) and the silent under-rejection it caused (stopper #3). The compositional engine was proven verdict-equivalent to the prior inlining position automaton by a differential over the entire XSTS corpus (zero divergences) before replacing it; the XSTS conformance gate is unchanged (invalid-schemas-accepted 15, valid-schemas-rejected 0), and a new test compiles a 2^20-position nested-group schema without hanging and rejects an ambiguous one the old cap would have skipped. The per-particle model (a particle never conflicts with a repetition of itself) and the QName-only overlap (no substitution-group expansion) are preserved, so the false-positive direction stays at zero.

View File

@ -243,7 +243,13 @@ extension PureXML.Regex {
private mutating func nextClassMember() throws -> ClassMember {
guard let character = peek() else { throw RegexError.unterminatedClass }
advance()
guard character == "\\" else { return .scalar(scalar(of: character)) }
guard character == "\\" else {
// A nested `[` is legal only as a `-[...]` subtraction, routed by
// parseClass/parseClassMember before reaching here; any other `[`
// is an unescaped class bracket, invalid per Appendix F.
if character == "[" { throw RegexError.unescapedClassBracket }
return .scalar(scalar(of: character))
}
guard let escaped = peek() else { throw RegexError.incompleteEscape }
advance()
if let single = Self.singleEscape(escaped) { return .scalar(single) }

View File

@ -18,8 +18,13 @@ extension PureXML.Regex {
/// expression ended. Unambiguously invalid, never an engine limitation.
case incompleteEscape
/// A character class opened with `[` that the expression ends before
/// closing with `]` (`a[`, `[a[:xyz:`). Unambiguously invalid.
/// closing with `]` (`a[`). Unambiguously invalid.
case unterminatedClass
/// An unescaped `[` inside a character class (`[a[bc]`, `[a[:xyz:]`): XSD
/// Appendix F `SingleCharNoEsc` excludes `[`, and a nested class is legal
/// only as a `-[...]` subtraction, so a `[` not introduced by `-` is a
/// syntax error. Unambiguously invalid, never an engine limitation.
case unescapedClassBracket
/// The `{...}` of a `\p`/`\P` escape whose content can be no XSD charProp:
/// empty, the bare prefix `Is`, a non-block name carrying a character that
/// is not a letter, or `Is` followed by a non-block-name character
@ -40,6 +45,7 @@ extension PureXML.Regex {
case .reversedQuantifier: "quantifier minimum exceeds maximum"
case .incompleteEscape: "an escape '\\' has no following character"
case .unterminatedClass: "a character class is not closed with ']'"
case .unescapedClassBracket: "an unescaped '[' inside a character class"
case .invalidProperty: "a \\p{...} property name is malformed"
}
}

View File

@ -80,7 +80,7 @@ extension PureXML.Schema {
private static func rejectsPattern(_ error: PureXML.Regex.RegexError) -> Bool {
switch error {
case .unbalanced, .danglingQuantifier, .reversedRange, .emptyClass, .reversedQuantifier,
.incompleteEscape, .unterminatedClass, .invalidProperty:
.incompleteEscape, .unterminatedClass, .unescapedClassBracket, .invalidProperty:
true
case .unsupported, .badQuantifier, .badEscape, .badClass:
false

View File

@ -125,13 +125,30 @@ struct RegexTests {
func test_unterminatedClass() throws {
// A `[` the expression ends before closing with `]` is invalid.
#expect(throws: PureXML.Regex.RegexError.unterminatedClass) { _ = try PureXML.Regex.Pattern("a[") }
#expect(throws: PureXML.Regex.RegexError.unterminatedClass) { _ = try PureXML.Regex.Pattern("[a[:xyz:") }
#expect(throws: PureXML.Regex.RegexError.unterminatedClass) { _ = try PureXML.Regex.Pattern("[a-") }
// A closed class, and a class whose only `]` is escaped, stay valid.
#expect(try matches("[abc]", "b"))
#expect(try matches("[a\\]]", "]"))
}
@Test("An unescaped '[' inside a character class is rejected")
func test_unescapedClassBracket() throws {
// Per XSD Appendix F a literal `[` inside a class must be escaped (`\[`);
// a nested class is legal only as a `-[...]` subtraction. The XSTS cases
// RegexTest_993 / RegexTest_1477 declare the pattern `[a[:xyz:]`, an
// unescaped inner `[`, so the schema declaring them is invalid.
#expect(throws: PureXML.Regex.RegexError.unescapedClassBracket) { _ = try PureXML.Regex.Pattern("[a[:xyz:]") }
#expect(throws: PureXML.Regex.RegexError.unescapedClassBracket) { _ = try PureXML.Regex.Pattern("[a[bc]") }
#expect(throws: PureXML.Regex.RegexError.unescapedClassBracket) { _ = try PureXML.Regex.Pattern("[[abc]") }
// With no closing `]`, the inner `[` still faults first (more precise than
// the unterminated-class diagnosis the swallowed `[` used to produce).
#expect(throws: PureXML.Regex.RegexError.unescapedClassBracket) { _ = try PureXML.Regex.Pattern("[a[:xyz:") }
// An escaped `[` and a `-[...]` subtraction stay valid.
#expect(try matches("[a\\[c]", "["))
#expect(try matches("[a-z-[aeiou]]", "b"))
#expect(try !matches("[a-z-[aeiou]]", "e"))
}
@Test("The empty pattern is valid and matches only the empty string")
func test_emptyPattern() throws {
// XSD grammar (Datatypes Appendix F): branch ::= piece*, so a branch

View File

@ -44,6 +44,17 @@ struct SchemaPatternFacetTests {
try compile("(ab)*c?")
}
@Test("an unescaped '[' inside a character class is rejected (XSTS RegexTest_993/1477)")
func test_unescapedClassBracket() {
// `[a[:xyz:]` (XSTS RegexTest_993 and RegexTest_1477) has an unescaped
// inner `[`, invalid per XSD Appendix F, so the schema is invalid.
#expect(rejects("[a[:xyz:]"))
#expect(rejects("[a[bc]"))
// The escaped form and a `-[...]` subtraction stay valid patterns.
#expect(!rejects("[a\\[c]"))
#expect(!rejects("[a-z-[aeiou]]"))
}
@Test("a construct the engine does not support is not treated as a schema error")
func test_engineGapTolerated() throws {
// Empty pattern (valid: matches the empty string), an untabulated Unicode

View File

@ -31,7 +31,10 @@ struct XSTSSuiteTests {
/// history of how each bucket reached its current value (every rule and its exact
/// delta) lives in CHANGELOG.md and docs/xsts-deviations.md, not inline here.
private let knownSchemaValidRejected = 0
private let knownSchemaInvalidAccepted = 15
/// Schema bucket, per-step deltas in CHANGELOG.md. Latest (15 -> 13): an
/// unescaped `[` inside a pattern-facet character class is a syntax error per
/// XSD Appendix F, so the schema is invalid (RegexTest_993, RegexTest_1477).
private let knownSchemaInvalidAccepted = 13
private let knownInstanceValidRejected = 0
/// Instance bucket (133 -> 22), per-step deltas in CHANGELOG.md: xsi:type must derive from the declared
/// type; anyType cannot stand in for anySimpleType; an untyped substitutionGroup member inherits its head's

View File

@ -6,8 +6,8 @@ evidence, what actually stands between the current engine and a 1.0 fit to put i
front of a developer as the authority on XML/XSD correctness.
Snapshot at time of writing: v0.2.0 released; full 2006-11-06 XSTS archive (14383
groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 43`,
`valid-instances-rejected = 0`, `invalid-instances-accepted = 31`.
groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 13`,
`valid-instances-rejected = 0`, `invalid-instances-accepted = 19`.
## Status against the four stoppers
@ -16,7 +16,7 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 43`,
libxml2 differential over 9618 schemas. **Met.** This is the doc's bar to be "an
authority"; it is cleared on the measured corpora.
2. **Silently accepting faulty input (false negatives).** `invalid-schemas-accepted
= 43`, `invalid-instances-accepted = 31`. **Not at 0.** This is the bulk of named
= 13`, `invalid-instances-accepted = 19`. **Not at 0.** This is the bulk of named
milestone work (M2, M3) and the subject of Part 1 below.
3. **Uncharacterized correctness.** Differential harness vs libxml2 and a fuzz
suite exist (M1, ~done). Gap: real-world corpora beyond XSTS, and the silent
@ -28,8 +28,8 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 43`,
## The biggest hurdle, stated plainly
Driving the false-negative buckets (`invalid-schemas-accepted = 43`,
`invalid-instances-accepted = 31`) to zero is the largest remaining body of named
Driving the false-negative buckets (`invalid-schemas-accepted = 13`,
`invalid-instances-accepted = 19`) to zero is the largest remaining body of named
work, and it is hard for a specific reason: **the ~95% already closed were the
tractable cases; every case left fights the cardinal rule.** The remainder sits in
shared-oracle / contested-spec / multi-document-composition territory where the fix