fix(schema): reject unescaped '[' inside a regex character class (#247)
XSD 1.0 Datatypes Appendix F defines SingleCharNoEsc as [^\[\]] and admits a nested class only as a -[...] subtraction, so a `[` inside a character class that is not introduced by `-` is a syntax error. RegexParser treated such a `[` as a literal scalar, so `[a[:xyz:]` compiled and the schema declaring it was wrongly accepted (XSTS RegexTest_993, RegexTest_1477). nextClassMember now throws a dedicated RegexError.unescapedClassBracket, kept distinct from the generic, deliberately-lenient badClass so the change cannot regress a valid-but-unsupported pattern. XSDSimpleParser.rejectsPattern treats it as a genuine syntax error, so the declaring schema is invalid. Closes RegexTest_993 and RegexTest_1477: invalid-schemas-accepted 15 -> 13, valid-schemas-rejected held at 0, instance buckets unchanged (full XSTS archive). Escaped [a\[c] and subtraction [a-z-[aeiou]] stay valid; the unterminated case [a[:xyz: now faults on the inner [.
This commit is contained in:
parent
bb094aa13c
commit
ba934bbb58
@ -34,6 +34,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Changed
|
||||
|
||||
- A `pattern` facet whose value contains an unescaped `[` inside a character class is now rejected at schema-compile time (false-negative burndown, stopper #2). XSD 1.0 Appendix F defines `SingleCharNoEsc` as `[^\[\]]` and admits a nested class only as a `-[...]` subtraction, so a `[` not introduced by `-` is a syntax error; `RegexParser` was treating it as a literal scalar, so `[a[:xyz:]` compiled and the declaring schema was accepted. The parser now throws a dedicated `RegexError.unescapedClassBracket` (kept distinct from the generic, deliberately-lenient `badClass` so the change cannot regress a valid-but-unsupported pattern), which `XSDSimpleParser.rejectsPattern` treats as a genuine syntax error. This closes XSTS RegexTest_993 and RegexTest_1477, dropping `invalid-schemas-accepted` from 15 to 13 with `valid-schemas-rejected` held at 0 (full XSTS archive) and the instance buckets unchanged (`invalid-instances-accepted` 19). The escaped form `[a\[c]` and the `-[...]` subtraction `[a-z-[aeiou]]` stay valid; the unterminated case `[a[:xyz:` now faults on the inner `[` (more precise than the prior unterminated-class diagnosis). Regressions are guarded by `RegexTests.test_unescapedClassBracket` and `SchemaPatternFacetTests`.
|
||||
|
||||
- Compiling a schema with deeply nested, multiply-referenced model groups is no longer exponential. `XSDParserParticles.groupReferenceParticle` inlined each `<xs:group ref>` into the compiled `Particle` content model, so `g0=(g1,g1), g1=(g2,g2), …` built `2^K` particles (measured: depth 14 = 0.21s, depth 20 ~13-18s, growing ~4x per two levels). The expansion is now memoized per (group definition, cycle-visiting set) in a per-compile `GroupParticleMemo`, so sibling references share it and the build drops to `O(groups)` (depth 40 now compiles in ~0.016s; previously infeasible). The shared content term is reused by value (copy-on-write), so the compiled model is byte-for-byte the same and validation is unchanged: the XSTS gate is unchanged (invalid-schemas-accepted 15, valid-schemas-rejected 0, instance buckets held) and the streaming-vs-tree fuzz differential still passes. This closes the schema-compile half of the interactive-safety group-blowup (the companion to the `positionCap` determinism fix); a regression is guarded by a depth-40 compile test.
|
||||
|
||||
- The Unique Particle Attribution (UPA / cos-nonambig) determinism check is now compositional and no longer caps or skips. `ContentModelDeterminism` delegates to a new `CompositionalDeterminism` engine that computes `first`/`followlast` over PARTICLE identities and summarizes each `<xs:group>` once, so a multiply-referenced nested group whose inlined Glushkov automaton would be `2^K` positions is checked in `O(nodes * particles^2)` time instead of being silently skipped. This DELETES `ContentModelDeterminism.positionCap` (the last named interactive-safety resource-bound blocker, production-readiness stopper #4) and the silent under-rejection it caused (stopper #3). The compositional engine was proven verdict-equivalent to the prior inlining position automaton by a differential over the entire XSTS corpus (zero divergences) before replacing it; the XSTS conformance gate is unchanged (invalid-schemas-accepted 15, valid-schemas-rejected 0), and a new test compiles a 2^20-position nested-group schema without hanging and rejects an ambiguous one the old cap would have skipped. The per-particle model (a particle never conflicts with a repetition of itself) and the QName-only overlap (no substitution-group expansion) are preserved, so the false-positive direction stays at zero.
|
||||
|
||||
@ -243,7 +243,13 @@ extension PureXML.Regex {
|
||||
private mutating func nextClassMember() throws -> ClassMember {
|
||||
guard let character = peek() else { throw RegexError.unterminatedClass }
|
||||
advance()
|
||||
guard character == "\\" else { return .scalar(scalar(of: character)) }
|
||||
guard character == "\\" else {
|
||||
// A nested `[` is legal only as a `-[...]` subtraction, routed by
|
||||
// parseClass/parseClassMember before reaching here; any other `[`
|
||||
// is an unescaped class bracket, invalid per Appendix F.
|
||||
if character == "[" { throw RegexError.unescapedClassBracket }
|
||||
return .scalar(scalar(of: character))
|
||||
}
|
||||
guard let escaped = peek() else { throw RegexError.incompleteEscape }
|
||||
advance()
|
||||
if let single = Self.singleEscape(escaped) { return .scalar(single) }
|
||||
|
||||
@ -18,8 +18,13 @@ extension PureXML.Regex {
|
||||
/// expression ended. Unambiguously invalid, never an engine limitation.
|
||||
case incompleteEscape
|
||||
/// A character class opened with `[` that the expression ends before
|
||||
/// closing with `]` (`a[`, `[a[:xyz:`). Unambiguously invalid.
|
||||
/// closing with `]` (`a[`). Unambiguously invalid.
|
||||
case unterminatedClass
|
||||
/// An unescaped `[` inside a character class (`[a[bc]`, `[a[:xyz:]`): XSD
|
||||
/// Appendix F `SingleCharNoEsc` excludes `[`, and a nested class is legal
|
||||
/// only as a `-[...]` subtraction, so a `[` not introduced by `-` is a
|
||||
/// syntax error. Unambiguously invalid, never an engine limitation.
|
||||
case unescapedClassBracket
|
||||
/// The `{...}` of a `\p`/`\P` escape whose content can be no XSD charProp:
|
||||
/// empty, the bare prefix `Is`, a non-block name carrying a character that
|
||||
/// is not a letter, or `Is` followed by a non-block-name character
|
||||
@ -40,6 +45,7 @@ extension PureXML.Regex {
|
||||
case .reversedQuantifier: "quantifier minimum exceeds maximum"
|
||||
case .incompleteEscape: "an escape '\\' has no following character"
|
||||
case .unterminatedClass: "a character class is not closed with ']'"
|
||||
case .unescapedClassBracket: "an unescaped '[' inside a character class"
|
||||
case .invalidProperty: "a \\p{...} property name is malformed"
|
||||
}
|
||||
}
|
||||
|
||||
@ -80,7 +80,7 @@ extension PureXML.Schema {
|
||||
private static func rejectsPattern(_ error: PureXML.Regex.RegexError) -> Bool {
|
||||
switch error {
|
||||
case .unbalanced, .danglingQuantifier, .reversedRange, .emptyClass, .reversedQuantifier,
|
||||
.incompleteEscape, .unterminatedClass, .invalidProperty:
|
||||
.incompleteEscape, .unterminatedClass, .unescapedClassBracket, .invalidProperty:
|
||||
true
|
||||
case .unsupported, .badQuantifier, .badEscape, .badClass:
|
||||
false
|
||||
|
||||
@ -125,13 +125,30 @@ struct RegexTests {
|
||||
func test_unterminatedClass() throws {
|
||||
// A `[` the expression ends before closing with `]` is invalid.
|
||||
#expect(throws: PureXML.Regex.RegexError.unterminatedClass) { _ = try PureXML.Regex.Pattern("a[") }
|
||||
#expect(throws: PureXML.Regex.RegexError.unterminatedClass) { _ = try PureXML.Regex.Pattern("[a[:xyz:") }
|
||||
#expect(throws: PureXML.Regex.RegexError.unterminatedClass) { _ = try PureXML.Regex.Pattern("[a-") }
|
||||
// A closed class, and a class whose only `]` is escaped, stay valid.
|
||||
#expect(try matches("[abc]", "b"))
|
||||
#expect(try matches("[a\\]]", "]"))
|
||||
}
|
||||
|
||||
@Test("An unescaped '[' inside a character class is rejected")
|
||||
func test_unescapedClassBracket() throws {
|
||||
// Per XSD Appendix F a literal `[` inside a class must be escaped (`\[`);
|
||||
// a nested class is legal only as a `-[...]` subtraction. The XSTS cases
|
||||
// RegexTest_993 / RegexTest_1477 declare the pattern `[a[:xyz:]`, an
|
||||
// unescaped inner `[`, so the schema declaring them is invalid.
|
||||
#expect(throws: PureXML.Regex.RegexError.unescapedClassBracket) { _ = try PureXML.Regex.Pattern("[a[:xyz:]") }
|
||||
#expect(throws: PureXML.Regex.RegexError.unescapedClassBracket) { _ = try PureXML.Regex.Pattern("[a[bc]") }
|
||||
#expect(throws: PureXML.Regex.RegexError.unescapedClassBracket) { _ = try PureXML.Regex.Pattern("[[abc]") }
|
||||
// With no closing `]`, the inner `[` still faults first (more precise than
|
||||
// the unterminated-class diagnosis the swallowed `[` used to produce).
|
||||
#expect(throws: PureXML.Regex.RegexError.unescapedClassBracket) { _ = try PureXML.Regex.Pattern("[a[:xyz:") }
|
||||
// An escaped `[` and a `-[...]` subtraction stay valid.
|
||||
#expect(try matches("[a\\[c]", "["))
|
||||
#expect(try matches("[a-z-[aeiou]]", "b"))
|
||||
#expect(try !matches("[a-z-[aeiou]]", "e"))
|
||||
}
|
||||
|
||||
@Test("The empty pattern is valid and matches only the empty string")
|
||||
func test_emptyPattern() throws {
|
||||
// XSD grammar (Datatypes Appendix F): branch ::= piece*, so a branch
|
||||
|
||||
@ -44,6 +44,17 @@ struct SchemaPatternFacetTests {
|
||||
try compile("(ab)*c?")
|
||||
}
|
||||
|
||||
@Test("an unescaped '[' inside a character class is rejected (XSTS RegexTest_993/1477)")
|
||||
func test_unescapedClassBracket() {
|
||||
// `[a[:xyz:]` (XSTS RegexTest_993 and RegexTest_1477) has an unescaped
|
||||
// inner `[`, invalid per XSD Appendix F, so the schema is invalid.
|
||||
#expect(rejects("[a[:xyz:]"))
|
||||
#expect(rejects("[a[bc]"))
|
||||
// The escaped form and a `-[...]` subtraction stay valid patterns.
|
||||
#expect(!rejects("[a\\[c]"))
|
||||
#expect(!rejects("[a-z-[aeiou]]"))
|
||||
}
|
||||
|
||||
@Test("a construct the engine does not support is not treated as a schema error")
|
||||
func test_engineGapTolerated() throws {
|
||||
// Empty pattern (valid: matches the empty string), an untabulated Unicode
|
||||
|
||||
@ -31,7 +31,10 @@ struct XSTSSuiteTests {
|
||||
/// history of how each bucket reached its current value (every rule and its exact
|
||||
/// delta) lives in CHANGELOG.md and docs/xsts-deviations.md, not inline here.
|
||||
private let knownSchemaValidRejected = 0
|
||||
private let knownSchemaInvalidAccepted = 15
|
||||
/// Schema bucket, per-step deltas in CHANGELOG.md. Latest (15 -> 13): an
|
||||
/// unescaped `[` inside a pattern-facet character class is a syntax error per
|
||||
/// XSD Appendix F, so the schema is invalid (RegexTest_993, RegexTest_1477).
|
||||
private let knownSchemaInvalidAccepted = 13
|
||||
private let knownInstanceValidRejected = 0
|
||||
/// Instance bucket (133 -> 22), per-step deltas in CHANGELOG.md: xsi:type must derive from the declared
|
||||
/// type; anyType cannot stand in for anySimpleType; an untyped substitutionGroup member inherits its head's
|
||||
|
||||
@ -6,8 +6,8 @@ evidence, what actually stands between the current engine and a 1.0 fit to put i
|
||||
front of a developer as the authority on XML/XSD correctness.
|
||||
|
||||
Snapshot at time of writing: v0.2.0 released; full 2006-11-06 XSTS archive (14383
|
||||
groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 43`,
|
||||
`valid-instances-rejected = 0`, `invalid-instances-accepted = 31`.
|
||||
groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 13`,
|
||||
`valid-instances-rejected = 0`, `invalid-instances-accepted = 19`.
|
||||
|
||||
## Status against the four stoppers
|
||||
|
||||
@ -16,7 +16,7 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 43`,
|
||||
libxml2 differential over 9618 schemas. **Met.** This is the doc's bar to be "an
|
||||
authority"; it is cleared on the measured corpora.
|
||||
2. **Silently accepting faulty input (false negatives).** `invalid-schemas-accepted
|
||||
= 43`, `invalid-instances-accepted = 31`. **Not at 0.** This is the bulk of named
|
||||
= 13`, `invalid-instances-accepted = 19`. **Not at 0.** This is the bulk of named
|
||||
milestone work (M2, M3) and the subject of Part 1 below.
|
||||
3. **Uncharacterized correctness.** Differential harness vs libxml2 and a fuzz
|
||||
suite exist (M1, ~done). Gap: real-world corpora beyond XSTS, and the silent
|
||||
@ -28,8 +28,8 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 43`,
|
||||
|
||||
## The biggest hurdle, stated plainly
|
||||
|
||||
Driving the false-negative buckets (`invalid-schemas-accepted = 43`,
|
||||
`invalid-instances-accepted = 31`) to zero is the largest remaining body of named
|
||||
Driving the false-negative buckets (`invalid-schemas-accepted = 13`,
|
||||
`invalid-instances-accepted = 19`) to zero is the largest remaining body of named
|
||||
work, and it is hard for a specific reason: **the ~95% already closed were the
|
||||
tractable cases; every case left fights the cardinal rule.** The remainder sits in
|
||||
shared-oracle / contested-spec / multi-document-composition territory where the fix
|
||||
|
||||
Loading…
Reference in New Issue
Block a user