NodetteHQ/PureXML
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(schema): negated character-class subtraction excludes the subtrahend (#249)

Browse Source 
XSD charClassSub ::= (posCharGroup | negCharGroup) '-' charClassExpr is the set
difference of the (possibly negated) group and the subtrahend, so for
[^cde-[ag]] negation applies to the base first and the subtrahend is removed
from that result: the class admits everything except {a, c, d, e, g}.

CharClass.matches applied the subtraction to the positive membership and
negated afterwards, so [^cde-[ag]] excluded only {c, d, e} and wrongly matched
a/g, accepting instance values a correct engine rejects. Negation now applies
to the base before subtraction (new result !B && !S is a strict subset of the
old !B || S, matching the Xerces-J reference exactly), so only negated classes
that also carry a subtraction change behaviour.

Closes XSTS RegexTest_430 and RegexTest_422: invalid-instances-accepted 19 ->
17, valid-instances-rejected and valid-schemas-rejected held at 0 (full XSTS
archive). Guarded by RegexCategoryTests.test_negatedSubtraction.
This commit is contained in:
Mihaela Mihaljevic 2026-06-23 03:10:45 +02:00 committed by GitHub
parent bf37375600
commit 118beeb61a
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
5 changed files with 29 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
CHANGELOG.md
View File

@ -34,6 +34,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Changed
- A negated character-class subtraction now excludes both the negated base and the subtrahend (false-negative burndown, stopper #2). XSD `charClassSub ::= (posCharGroup | negCharGroup) '-' charClassExpr` is the set difference of the (possibly negated) group and the subtrahend, so for `[^cde-[ag]]` negation applies to the base first and the subtrahend is removed from that result: the class admits everything except `{a, c, d, e, g}`. `CharClass.matches` applied the subtraction to the positive membership and negated afterwards, so it excluded only `{c, d, e}` and wrongly matched `a`/`g`, accepting instances a correct engine rejects. Negation now applies to the base before subtraction. Only negated classes that also carry a subtraction change behaviour. This closes XSTS RegexTest_430 and RegexTest_422, dropping `invalid-instances-accepted` from 19 to 17 with `valid-instances-rejected` held at 0 (full XSTS archive). A regression is guarded by `RegexCategoryTests.test_negatedSubtraction`.
- An unknown `\p{...}` category or block name in a `pattern` facet is now rejected at schema-compile time (false-negative burndown, stopper #2). XSD 1.0 `charProp` is `IsCategory | IsBlock`, an exhaustive set: every Unicode general category plus a fixed Unicode-3.1 block list. The block table (`CategoryMatcher.blocks`) was missing the nine supplementary-plane blocks (Old Italic, Gothic, Deseret, the Byzantine/Musical/Mathematical symbol blocks, CJK Unified Ideographs Extension B, CJK Compatibility Ideographs Supplement, Tags) and the three surrogate blocks, so `parseCategory` could not tell a real-but-untabulated block from a bogus name and stayed lenient (`RegexError.unsupported`), accepting `\p{IsaA0-a9}` and the schema declaring it. The table is now the complete XSD set (supplementary blocks added with their ranges; surrogate blocks added as valid names whose ranges hold no scalar, so they match nothing), making the recognised set authoritative, so an unrecognised name throws the rejecting `RegexError.invalidProperty`. The now-dead `RegexError.unsupported` case and the `isMalformedProperty` heuristic are removed. This closes XSTS reK88, dropping `invalid-schemas-accepted` from 13 to 12 with `valid-schemas-rejected` held at 0 (full XSTS archive) and the instance buckets unchanged (`invalid-instances-accepted` 19). As a side effect, a pattern naming a supplementary-plane block (e.g. `\p{IsMusicalSymbols}`) now matches its characters instead of failing to compile. Block names/ranges were taken from the XSD-supported list in Xerces-J `Token.java` (clean-room: factual data only). Regressions are guarded by `RegexCategoryTests` (supplementary/surrogate matching, unknown-name rejection) and `SchemaPatternFacetTests`.
- A `pattern` facet whose value contains an unescaped `[` inside a character class is now rejected at schema-compile time (false-negative burndown, stopper #2). XSD 1.0 Appendix F defines `SingleCharNoEsc` as `[^\[\]]` and admits a nested class only as a `-[...]` subtraction, so a `[` not introduced by `-` is a syntax error; `RegexParser` was treating it as a literal scalar, so `[a[:xyz:]` compiled and the declaring schema was accepted. The parser now throws a dedicated `RegexError.unescapedClassBracket` (kept distinct from the generic, deliberately-lenient `badClass` so the change cannot regress a valid-but-unsupported pattern), which `XSDSimpleParser.rejectsPattern` treats as a genuine syntax error. This closes XSTS RegexTest_993 and RegexTest_1477, dropping `invalid-schemas-accepted` from 15 to 13 with `valid-schemas-rejected` held at 0 (full XSTS archive) and the instance buckets unchanged (`invalid-instances-accepted` 19). The escaped form `[a\[c]` and the `-[...]` subtraction `[a-z-[aeiou]]` stay valid; the unterminated case `[a[:xyz:` now faults on the inner `[` (more precise than the prior unterminated-class diagnosis). Regressions are guarded by `RegexTests.test_unescapedClassBracket` and `SchemaPatternFacetTests`.

10
Sources/Regex/RegexSyntax.swift
View File

@ -95,12 +95,18 @@ extension PureXML.Regex {
}
func matches(_ scalar: Unicode.Scalar) -> Bool {
var hit = ranges.contains { $0.contains(scalar) }
let inBase = ranges.contains { $0.contains(scalar) }
|| named.contains { $0.contains(scalar) }
|| negatedNamed.contains { !$0.contains(scalar) }
|| categories.contains { $0.contains(scalar) }
// XSD `charClassSub ::= (posCharGroup | negCharGroup) '-' charClassExpr`
// is the set difference of the (possibly NEGATED) group and the
// subtrahend, so negation applies to the base FIRST, then subtraction
// removes from that result: `[^cde-[ag]]` is (not c/d/e) minus a/g,
// i.e. excludes {a,c,d,e,g}, not just {c,d,e} (XSTS RegexTest_430).
var hit = negated ? !inBase : inBase
if hit, subtraction.contains(where: { $0.matches(scalar) }) { hit = false }
return negated ? !hit : hit
return hit
}
static func single(_ scalar: Unicode.Scalar) -> CharClass {

13
Tests/RegexCategoryTests.swift
View File

@ -88,6 +88,19 @@ struct RegexCategoryTests {
#expect(try !matches("[a-z-[aeiou]]+", "bce"))
}
@Test("A negated class with subtraction excludes both the base and the subtrahend")
func test_negatedSubtraction() throws {
// `[^cde-[ag]]` is (not c/d/e) minus a/g, i.e. excludes {a,c,d,e,g}
// (XSD charClassSub: the difference of the NEGATED group and the
// subtrahend). XSTS RegexTest_430: `agbfxyzga` contains a and g, so it
// must NOT match `[^cde-[ag]]+`.
#expect(try !matches("[^cde-[ag]]+", "agbfxyzga"))
#expect(try matches("[^cde-[ag]]+", "bfxyz")) // none of a,c,d,e,g
#expect(try !matches("[^cde-[ag]]+", "a")) // a is subtracted out
#expect(try !matches("[^cde-[ag]]+", "c")) // c is in the negated base
#expect(try matches("[^cde-[ag]]+", "b")) // b is admitted
}
@Test("Subtraction nests")
func test_nestedSubtraction() throws {
// a-z, minus (b-d minus c): so b and d are removed, c stays.

4
Tests/XSTSSuiteTests.swift
View File

@ -47,7 +47,9 @@ struct XSTSSuiteTests {
/// a nilled element may not have a fixed value constraint, cvc-elt.3.2.2 (addB065);
/// an optional xs:all group present in the instance still requires its members (mgZ001);
/// a defaulted/fixed IDREF/IDREFS must resolve to a matching ID, cvc-id (idZ012).
private let knownInstanceInvalidAccepted = 19
/// Latest (19 -> 17): a negated character-class subtraction excludes both the
/// negated base and the subtrahend (`[^cde-[ag]]`), RegexTest_430 and _422.
private let knownInstanceInvalidAccepted = 17
/// Suspect instance tests excluded from the counts: the Ethiopic-digit `\d`
/// cases contradict `\d` = `\p{Nd}` at Unicode 3.2 and are tracked as disputed.
private static let specDivergentInstances: Set<String> = ["reS17.v", "reS38.v", "reZ004v.v"]

6
docs/release-hurdles.md
View File

@ -7,7 +7,7 @@ front of a developer as the authority on XML/XSD correctness.
Snapshot at time of writing: v0.2.0 released; full 2006-11-06 XSTS archive (14383
groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 12`,
`valid-instances-rejected = 0`, `invalid-instances-accepted = 19`.
`valid-instances-rejected = 0`, `invalid-instances-accepted = 17`.
## Status against the four stoppers
@ -16,7 +16,7 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 12`,
libxml2 differential over 9618 schemas. **Met.** This is the doc's bar to be "an
authority"; it is cleared on the measured corpora.
2. **Silently accepting faulty input (false negatives).** `invalid-schemas-accepted
= 12`, `invalid-instances-accepted = 19`. **Not at 0.** This is the bulk of named
= 12`, `invalid-instances-accepted = 17`. **Not at 0.** This is the bulk of named
milestone work (M2, M3) and the subject of Part 1 below.
3. **Uncharacterized correctness.** Differential harness vs libxml2 and a fuzz
suite exist (M1, ~done). Gap: real-world corpora beyond XSTS, and the silent
@ -29,7 +29,7 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 12`,
## The biggest hurdle, stated plainly
Driving the false-negative buckets (`invalid-schemas-accepted = 12`,
`invalid-instances-accepted = 19`) to zero is the largest remaining body of named
`invalid-instances-accepted = 17`) to zero is the largest remaining body of named
work, and it is hard for a specific reason: **the ~95% already closed were the
tractable cases; every case left fights the cardinal rule.** The remainder sits in
shared-oracle / contested-spec / multi-document-composition territory where the fix