From 118beeb61aa7ec3c3695366a3b22a79d596bfe46 Mon Sep 17 00:00:00 2001 From: Mihaela Mihaljevic Date: Tue, 23 Jun 2026 03:10:45 +0200 Subject: [PATCH] fix(schema): negated character-class subtraction excludes the subtrahend (#249) XSD charClassSub ::= (posCharGroup | negCharGroup) '-' charClassExpr is the set difference of the (possibly negated) group and the subtrahend, so for [^cde-[ag]] negation applies to the base first and the subtrahend is removed from that result: the class admits everything except {a, c, d, e, g}. CharClass.matches applied the subtraction to the positive membership and negated afterwards, so [^cde-[ag]] excluded only {c, d, e} and wrongly matched a/g, accepting instance values a correct engine rejects. Negation now applies to the base before subtraction (new result !B && !S is a strict subset of the old !B || S, matching the Xerces-J reference exactly), so only negated classes that also carry a subtraction change behaviour. Closes XSTS RegexTest_430 and RegexTest_422: invalid-instances-accepted 19 -> 17, valid-instances-rejected and valid-schemas-rejected held at 0 (full XSTS archive). Guarded by RegexCategoryTests.test_negatedSubtraction. --- CHANGELOG.md | 2 ++ Sources/Regex/RegexSyntax.swift | 10 ++++++++-- Tests/RegexCategoryTests.swift | 13 +++++++++++++ Tests/XSTSSuiteTests.swift | 4 +++- docs/release-hurdles.md | 6 +++--- 5 files changed, 29 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 33c8ba5..4d89c02 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- A negated character-class subtraction now excludes both the negated base and the subtrahend (false-negative burndown, stopper #2). XSD `charClassSub ::= (posCharGroup | negCharGroup) '-' charClassExpr` is the set difference of the (possibly negated) group and the subtrahend, so for `[^cde-[ag]]` negation applies to the base first and the subtrahend is removed from that result: the class admits everything except `{a, c, d, e, g}`. `CharClass.matches` applied the subtraction to the positive membership and negated afterwards, so it excluded only `{c, d, e}` and wrongly matched `a`/`g`, accepting instances a correct engine rejects. Negation now applies to the base before subtraction. Only negated classes that also carry a subtraction change behaviour. This closes XSTS RegexTest_430 and RegexTest_422, dropping `invalid-instances-accepted` from 19 to 17 with `valid-instances-rejected` held at 0 (full XSTS archive). A regression is guarded by `RegexCategoryTests.test_negatedSubtraction`. + - An unknown `\p{...}` category or block name in a `pattern` facet is now rejected at schema-compile time (false-negative burndown, stopper #2). XSD 1.0 `charProp` is `IsCategory | IsBlock`, an exhaustive set: every Unicode general category plus a fixed Unicode-3.1 block list. The block table (`CategoryMatcher.blocks`) was missing the nine supplementary-plane blocks (Old Italic, Gothic, Deseret, the Byzantine/Musical/Mathematical symbol blocks, CJK Unified Ideographs Extension B, CJK Compatibility Ideographs Supplement, Tags) and the three surrogate blocks, so `parseCategory` could not tell a real-but-untabulated block from a bogus name and stayed lenient (`RegexError.unsupported`), accepting `\p{IsaA0-a9}` and the schema declaring it. The table is now the complete XSD set (supplementary blocks added with their ranges; surrogate blocks added as valid names whose ranges hold no scalar, so they match nothing), making the recognised set authoritative, so an unrecognised name throws the rejecting `RegexError.invalidProperty`. The now-dead `RegexError.unsupported` case and the `isMalformedProperty` heuristic are removed. This closes XSTS reK88, dropping `invalid-schemas-accepted` from 13 to 12 with `valid-schemas-rejected` held at 0 (full XSTS archive) and the instance buckets unchanged (`invalid-instances-accepted` 19). As a side effect, a pattern naming a supplementary-plane block (e.g. `\p{IsMusicalSymbols}`) now matches its characters instead of failing to compile. Block names/ranges were taken from the XSD-supported list in Xerces-J `Token.java` (clean-room: factual data only). Regressions are guarded by `RegexCategoryTests` (supplementary/surrogate matching, unknown-name rejection) and `SchemaPatternFacetTests`. - A `pattern` facet whose value contains an unescaped `[` inside a character class is now rejected at schema-compile time (false-negative burndown, stopper #2). XSD 1.0 Appendix F defines `SingleCharNoEsc` as `[^\[\]]` and admits a nested class only as a `-[...]` subtraction, so a `[` not introduced by `-` is a syntax error; `RegexParser` was treating it as a literal scalar, so `[a[:xyz:]` compiled and the declaring schema was accepted. The parser now throws a dedicated `RegexError.unescapedClassBracket` (kept distinct from the generic, deliberately-lenient `badClass` so the change cannot regress a valid-but-unsupported pattern), which `XSDSimpleParser.rejectsPattern` treats as a genuine syntax error. This closes XSTS RegexTest_993 and RegexTest_1477, dropping `invalid-schemas-accepted` from 15 to 13 with `valid-schemas-rejected` held at 0 (full XSTS archive) and the instance buckets unchanged (`invalid-instances-accepted` 19). The escaped form `[a\[c]` and the `-[...]` subtraction `[a-z-[aeiou]]` stay valid; the unterminated case `[a[:xyz:` now faults on the inner `[` (more precise than the prior unterminated-class diagnosis). Regressions are guarded by `RegexTests.test_unescapedClassBracket` and `SchemaPatternFacetTests`. diff --git a/Sources/Regex/RegexSyntax.swift b/Sources/Regex/RegexSyntax.swift index b4920df..da0ee93 100644 --- a/Sources/Regex/RegexSyntax.swift +++ b/Sources/Regex/RegexSyntax.swift @@ -95,12 +95,18 @@ extension PureXML.Regex { } func matches(_ scalar: Unicode.Scalar) -> Bool { - var hit = ranges.contains { $0.contains(scalar) } + let inBase = ranges.contains { $0.contains(scalar) } || named.contains { $0.contains(scalar) } || negatedNamed.contains { !$0.contains(scalar) } || categories.contains { $0.contains(scalar) } + // XSD `charClassSub ::= (posCharGroup | negCharGroup) '-' charClassExpr` + // is the set difference of the (possibly NEGATED) group and the + // subtrahend, so negation applies to the base FIRST, then subtraction + // removes from that result: `[^cde-[ag]]` is (not c/d/e) minus a/g, + // i.e. excludes {a,c,d,e,g}, not just {c,d,e} (XSTS RegexTest_430). + var hit = negated ? !inBase : inBase if hit, subtraction.contains(where: { $0.matches(scalar) }) { hit = false } - return negated ? !hit : hit + return hit } static func single(_ scalar: Unicode.Scalar) -> CharClass { diff --git a/Tests/RegexCategoryTests.swift b/Tests/RegexCategoryTests.swift index fdecdcb..ed6f2ca 100644 --- a/Tests/RegexCategoryTests.swift +++ b/Tests/RegexCategoryTests.swift @@ -88,6 +88,19 @@ struct RegexCategoryTests { #expect(try !matches("[a-z-[aeiou]]+", "bce")) } + @Test("A negated class with subtraction excludes both the base and the subtrahend") + func test_negatedSubtraction() throws { + // `[^cde-[ag]]` is (not c/d/e) minus a/g, i.e. excludes {a,c,d,e,g} + // (XSD charClassSub: the difference of the NEGATED group and the + // subtrahend). XSTS RegexTest_430: `agbfxyzga` contains a and g, so it + // must NOT match `[^cde-[ag]]+`. + #expect(try !matches("[^cde-[ag]]+", "agbfxyzga")) + #expect(try matches("[^cde-[ag]]+", "bfxyz")) // none of a,c,d,e,g + #expect(try !matches("[^cde-[ag]]+", "a")) // a is subtracted out + #expect(try !matches("[^cde-[ag]]+", "c")) // c is in the negated base + #expect(try matches("[^cde-[ag]]+", "b")) // b is admitted + } + @Test("Subtraction nests") func test_nestedSubtraction() throws { // a-z, minus (b-d minus c): so b and d are removed, c stays. diff --git a/Tests/XSTSSuiteTests.swift b/Tests/XSTSSuiteTests.swift index fd18d51..5085087 100644 --- a/Tests/XSTSSuiteTests.swift +++ b/Tests/XSTSSuiteTests.swift @@ -47,7 +47,9 @@ struct XSTSSuiteTests { /// a nilled element may not have a fixed value constraint, cvc-elt.3.2.2 (addB065); /// an optional xs:all group present in the instance still requires its members (mgZ001); /// a defaulted/fixed IDREF/IDREFS must resolve to a matching ID, cvc-id (idZ012). - private let knownInstanceInvalidAccepted = 19 + /// Latest (19 -> 17): a negated character-class subtraction excludes both the + /// negated base and the subtrahend (`[^cde-[ag]]`), RegexTest_430 and _422. + private let knownInstanceInvalidAccepted = 17 /// Suspect instance tests excluded from the counts: the Ethiopic-digit `\d` /// cases contradict `\d` = `\p{Nd}` at Unicode 3.2 and are tracked as disputed. private static let specDivergentInstances: Set = ["reS17.v", "reS38.v", "reZ004v.v"] diff --git a/docs/release-hurdles.md b/docs/release-hurdles.md index b318f80..f735533 100644 --- a/docs/release-hurdles.md +++ b/docs/release-hurdles.md @@ -7,7 +7,7 @@ front of a developer as the authority on XML/XSD correctness. Snapshot at time of writing: v0.2.0 released; full 2006-11-06 XSTS archive (14383 groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 12`, -`valid-instances-rejected = 0`, `invalid-instances-accepted = 19`. +`valid-instances-rejected = 0`, `invalid-instances-accepted = 17`. ## Status against the four stoppers @@ -16,7 +16,7 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 12`, libxml2 differential over 9618 schemas. **Met.** This is the doc's bar to be "an authority"; it is cleared on the measured corpora. 2. **Silently accepting faulty input (false negatives).** `invalid-schemas-accepted - = 12`, `invalid-instances-accepted = 19`. **Not at 0.** This is the bulk of named + = 12`, `invalid-instances-accepted = 17`. **Not at 0.** This is the bulk of named milestone work (M2, M3) and the subject of Part 1 below. 3. **Uncharacterized correctness.** Differential harness vs libxml2 and a fuzz suite exist (M1, ~done). Gap: real-world corpora beyond XSTS, and the silent @@ -29,7 +29,7 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 12`, ## The biggest hurdle, stated plainly Driving the false-negative buckets (`invalid-schemas-accepted = 12`, -`invalid-instances-accepted = 19`) to zero is the largest remaining body of named +`invalid-instances-accepted = 17`) to zero is the largest remaining body of named work, and it is hard for a specific reason: **the ~95% already closed were the tractable cases; every case left fights the cardinal rule.** The remainder sits in shared-oracle / contested-spec / multi-document-composition territory where the fix