From bf373756003f7d8af2563c87d56990fa16aee61b Mon Sep 17 00:00:00 2001 From: Mihaela Mihaljevic Date: Tue, 23 Jun 2026 02:15:26 +0200 Subject: [PATCH] fix(schema): reject an unknown \p{...} category or block name (#248) XSD 1.0 charProp is IsCategory | IsBlock, an exhaustive set: every Unicode general category plus a fixed Unicode-3.1 block list. The block table was missing the nine supplementary-plane blocks (Old Italic, Gothic, Deseret, the Byzantine/Musical/Mathematical symbol blocks, CJK Unified Ideographs Extension B, CJK Compatibility Ideographs Supplement, Tags) and the three surrogate blocks, so parseCategory could not tell a real-but-untabulated block from a bogus name and stayed lenient (RegexError.unsupported), accepting the pattern \p{IsaA0-a9} and the schema declaring it (XSTS reK88). Complete the table to the full XSD set: supplementary blocks with their ranges, and surrogate blocks as valid names whose ranges hold no scalar (so they match nothing). The recognised set is now authoritative, so parseCategory throws the rejecting RegexError.invalidProperty for any unrecognised name. The now-dead RegexError.unsupported case and the isMalformedProperty heuristic are removed. Closes XSTS reK88: invalid-schemas-accepted 13 -> 12, valid-schemas-rejected held at 0, instance buckets unchanged (full XSTS archive). Side effect: a pattern naming a supplementary-plane block (e.g. \p{IsMusicalSymbols}) now matches its characters instead of failing to compile, removing a latent instance false positive. Block names/ranges taken from the XSD-supported list in Xerces-J Token.java (clean-room: factual data only). The list is spec-frozen at Unicode 3.1, so later blocks are rejected by design. --- CHANGELOG.md | 2 ++ Sources/Regex/RegexCategory.swift | 37 ++++++++++++++++++++++------ Sources/Regex/RegexParser.swift | 25 ++++++------------- Sources/Regex/RegexSyntax.swift | 2 -- Sources/Schema/XSDSimpleParser.swift | 9 ++++--- Tests/RegexCategoryTests.swift | 23 ++++++++++++++++- Tests/RegexTests.swift | 5 ++-- Tests/SchemaPatternFacetTests.swift | 30 ++++++++++++++++++---- Tests/XSTSSuiteTests.swift | 8 +++--- docs/release-hurdles.md | 6 ++--- 10 files changed, 102 insertions(+), 45 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index c4b3eeb..33c8ba5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- An unknown `\p{...}` category or block name in a `pattern` facet is now rejected at schema-compile time (false-negative burndown, stopper #2). XSD 1.0 `charProp` is `IsCategory | IsBlock`, an exhaustive set: every Unicode general category plus a fixed Unicode-3.1 block list. The block table (`CategoryMatcher.blocks`) was missing the nine supplementary-plane blocks (Old Italic, Gothic, Deseret, the Byzantine/Musical/Mathematical symbol blocks, CJK Unified Ideographs Extension B, CJK Compatibility Ideographs Supplement, Tags) and the three surrogate blocks, so `parseCategory` could not tell a real-but-untabulated block from a bogus name and stayed lenient (`RegexError.unsupported`), accepting `\p{IsaA0-a9}` and the schema declaring it. The table is now the complete XSD set (supplementary blocks added with their ranges; surrogate blocks added as valid names whose ranges hold no scalar, so they match nothing), making the recognised set authoritative, so an unrecognised name throws the rejecting `RegexError.invalidProperty`. The now-dead `RegexError.unsupported` case and the `isMalformedProperty` heuristic are removed. This closes XSTS reK88, dropping `invalid-schemas-accepted` from 13 to 12 with `valid-schemas-rejected` held at 0 (full XSTS archive) and the instance buckets unchanged (`invalid-instances-accepted` 19). As a side effect, a pattern naming a supplementary-plane block (e.g. `\p{IsMusicalSymbols}`) now matches its characters instead of failing to compile. Block names/ranges were taken from the XSD-supported list in Xerces-J `Token.java` (clean-room: factual data only). Regressions are guarded by `RegexCategoryTests` (supplementary/surrogate matching, unknown-name rejection) and `SchemaPatternFacetTests`. + - A `pattern` facet whose value contains an unescaped `[` inside a character class is now rejected at schema-compile time (false-negative burndown, stopper #2). XSD 1.0 Appendix F defines `SingleCharNoEsc` as `[^\[\]]` and admits a nested class only as a `-[...]` subtraction, so a `[` not introduced by `-` is a syntax error; `RegexParser` was treating it as a literal scalar, so `[a[:xyz:]` compiled and the declaring schema was accepted. The parser now throws a dedicated `RegexError.unescapedClassBracket` (kept distinct from the generic, deliberately-lenient `badClass` so the change cannot regress a valid-but-unsupported pattern), which `XSDSimpleParser.rejectsPattern` treats as a genuine syntax error. This closes XSTS RegexTest_993 and RegexTest_1477, dropping `invalid-schemas-accepted` from 15 to 13 with `valid-schemas-rejected` held at 0 (full XSTS archive) and the instance buckets unchanged (`invalid-instances-accepted` 19). The escaped form `[a\[c]` and the `-[...]` subtraction `[a-z-[aeiou]]` stay valid; the unterminated case `[a[:xyz:` now faults on the inner `[` (more precise than the prior unterminated-class diagnosis). Regressions are guarded by `RegexTests.test_unescapedClassBracket` and `SchemaPatternFacetTests`. - Compiling a schema with deeply nested, multiply-referenced model groups is no longer exponential. `XSDParserParticles.groupReferenceParticle` inlined each `` into the compiled `Particle` content model, so `g0=(g1,g1), g1=(g2,g2), …` built `2^K` particles (measured: depth 14 = 0.21s, depth 20 ~13-18s, growing ~4x per two levels). The expansion is now memoized per (group definition, cycle-visiting set) in a per-compile `GroupParticleMemo`, so sibling references share it and the build drops to `O(groups)` (depth 40 now compiles in ~0.016s; previously infeasible). The shared content term is reused by value (copy-on-write), so the compiled model is byte-for-byte the same and validation is unchanged: the XSTS gate is unchanged (invalid-schemas-accepted 15, valid-schemas-rejected 0, instance buckets held) and the streaming-vs-tree fuzz differential still passes. This closes the schema-compile half of the interactive-safety group-blowup (the companion to the `positionCap` determinism fix); a regression is guarded by a depth-40 compile test. diff --git a/Sources/Regex/RegexCategory.swift b/Sources/Regex/RegexCategory.swift index 45cac0f..1700f5c 100644 --- a/Sources/Regex/RegexCategory.swift +++ b/Sources/Regex/RegexCategory.swift @@ -52,13 +52,21 @@ extension PureXML.Regex { .control: "Cc", .format: "Cf", .surrogate: "Cs", .privateUse: "Co", .unassigned: "Cn", ] - /// The Unicode block names XSD 1.0 admits in `\p{Is}` (the - /// Unicode 3.1 blocks the regular-expression appendix fixes), each mapped - /// to its code-point range. An `Is` name outside this set is - /// rejected rather than guessed, so a pattern never matches against a - /// fabricated range. Surrogate blocks are omitted: surrogates are not - /// scalar values. `Specials` uses the main `FFF0..FFFD` range (the spec's - /// separate `FEFF` half is the byte-order mark, not matched here). + /// The complete set of Unicode block names XSD 1.0 admits in + /// `\p{Is}` (the Unicode 3.1 blocks the regular-expression appendix + /// fixes), each mapped to its code-point range. The set is exhaustive: an + /// `Is` name outside it is no XSD block and is rejected at compile + /// time (`isMalformedProperty` is gone; the predicate's presence here is the + /// sole authority), so a pattern never matches against a fabricated range. + /// XSD 1.0 freezes this list at Unicode 3.1, so a block introduced later + /// (e.g. `CyrillicSupplement`) is by design not an XSD 1.0 block and is + /// rejected, even though some processors accept any `Is[A-Za-z0-9-]+` + /// leniently. This is a deliberate spec-strict choice, not an oversight. + /// The three surrogate blocks are listed so a pattern naming them still + /// compiles, but their ranges hold no scalar value, so they match nothing. + /// `Specials` uses the main `FFF0..FFFD` range (the spec's separate `FEFF` + /// half is the byte-order mark, not matched here). `GreekandCoptic` is the + /// later Unicode rename of `Greek`, kept as a lenient alias. private static let blocks: [String: ClosedRange] = [ "BasicLatin": 0x0000 ... 0x007F, "Latin-1Supplement": 0x0080 ... 0x00FF, @@ -145,6 +153,21 @@ extension PureXML.Regex { "ArabicPresentationForms-B": 0xFE70 ... 0xFEFF, "HalfwidthandFullwidthForms": 0xFF00 ... 0xFFEF, "Specials": 0xFFF0 ... 0xFFFD, + // Surrogate blocks: valid XSD block names whose ranges hold no scalar + // value (lone surrogates are not Unicode scalars), so they match nothing. + "HighSurrogates": 0xD800 ... 0xDB7F, + "HighPrivateUseSurrogates": 0xDB80 ... 0xDBFF, + "LowSurrogates": 0xDC00 ... 0xDFFF, + // Supplementary-plane blocks (Unicode 3.1, beyond the BMP). + "OldItalic": 0x10300 ... 0x1032F, + "Gothic": 0x10330 ... 0x1034F, + "Deseret": 0x10400 ... 0x1044F, + "ByzantineMusicalSymbols": 0x1D000 ... 0x1D0FF, + "MusicalSymbols": 0x1D100 ... 0x1D1FF, + "MathematicalAlphanumericSymbols": 0x1D400 ... 0x1D7FF, + "CJKUnifiedIdeographsExtensionB": 0x20000 ... 0x2A6D6, + "CJKCompatibilityIdeographsSupplement": 0x2F800 ... 0x2FA1F, + "Tags": 0xE0000 ... 0xE007F, ] } } diff --git a/Sources/Regex/RegexParser.swift b/Sources/Regex/RegexParser.swift index 879f180..2f68f89 100644 --- a/Sources/Regex/RegexParser.swift +++ b/Sources/Regex/RegexParser.swift @@ -133,28 +133,17 @@ extension PureXML.Regex { guard peek() == "}" else { throw RegexError.badClass } advance() guard let predicate = CategoryPredicate(name: name, negated: negated) else { - // An unknown name is by default a limit of this engine, not an error - // (stay lenient). But a name that can be NO XSD charProp at all is - // unambiguously invalid and rejects, independent of which categories - // or blocks this engine happens to recognise. - throw Self.isMalformedProperty(name) ? RegexError.invalidProperty : RegexError.unsupported("\\p{\(name)}") + // The recognised set (every Unicode general category plus the + // complete XSD 1.0 `Is` list, see CategoryMatcher) is the + // exhaustive set of XSD `charProp` names, so any name outside it is + // no XSD category or block and is a genuine syntax error, not an + // engine limitation. Rejecting it (XSTS reK88, `\p{IsaA0-a9}`) + // cannot reject a valid pattern because the set is complete. + throw RegexError.invalidProperty } return predicate } - /// Whether `name` can be no XSD `charProp` (`IsCategory | IsBlock`) under any - /// repertoire: empty, the bare `Is` prefix, a non-block name (no `Is` prefix) - /// carrying a non-letter, or `Is` followed by a non-`[A-Za-z0-9-]` character. - /// A well-formed but unrecognised name (an unknown block such as `IsaA0-a9`) - /// is NOT malformed and stays a lenient `unsupported` limit. - private static func isMalformedProperty(_ name: String) -> Bool { - if name.isEmpty || name == "Is" { return true } - guard name.hasPrefix("Is") else { - return name.contains { !$0.isLetter } - } - return name.dropFirst(2).contains { !($0.isLetter || $0.isNumber || $0 == "-") } - } - private func classFrom(_ member: ClassMember) -> CharClass { switch member { case let .scalar(value): .single(value) diff --git a/Sources/Regex/RegexSyntax.swift b/Sources/Regex/RegexSyntax.swift index 951b6ed..b4920df 100644 --- a/Sources/Regex/RegexSyntax.swift +++ b/Sources/Regex/RegexSyntax.swift @@ -6,7 +6,6 @@ extension PureXML.Regex { case badEscape(String) case badClass case badQuantifier - case unsupported(String) /// A character-class range whose first endpoint exceeds its second /// (`[b-a]`): unambiguously invalid, never an engine limitation. case reversedRange @@ -39,7 +38,6 @@ extension PureXML.Regex { case let .badEscape(detail): "invalid escape '\\\(detail)'" case .badClass: "malformed character class" case .badQuantifier: "malformed quantifier" - case let .unsupported(detail): "unsupported construct: \(detail)" case .reversedRange: "character-class range is reversed" case .emptyClass: "empty character class" case .reversedQuantifier: "quantifier minimum exceeds maximum" diff --git a/Sources/Schema/XSDSimpleParser.swift b/Sources/Schema/XSDSimpleParser.swift index eb41d01..8f4d5f0 100644 --- a/Sources/Schema/XSDSimpleParser.swift +++ b/Sources/Schema/XSDSimpleParser.swift @@ -66,9 +66,10 @@ extension PureXML.Schema { } catch let error as PureXML.Regex.RegexError where rejectsPattern(error) { errors.append("pattern '\(value)' is not a valid regular expression: \(error)") } catch { - // A construct the engine does not support (`.empty`, `.unsupported` - // for an untabulated `\p{Is...}` block, `.badQuantifier` for the - // lenient `{,m}` form) is a valid XSD pattern, not a schema error. + // A construct the engine merely under-supports on an otherwise- + // valid pattern (`.badQuantifier` for the lenient `{,m}` form, + // `.badEscape`/`.badClass`) is a valid XSD pattern, not a schema + // error. The empty pattern is valid and never throws. } } return errors @@ -82,7 +83,7 @@ extension PureXML.Schema { case .unbalanced, .danglingQuantifier, .reversedRange, .emptyClass, .reversedQuantifier, .incompleteEscape, .unterminatedClass, .unescapedClassBracket, .invalidProperty: true - case .unsupported, .badQuantifier, .badEscape, .badClass: + case .badQuantifier, .badEscape, .badClass: false } } diff --git a/Tests/RegexCategoryTests.swift b/Tests/RegexCategoryTests.swift index 4b5fd0e..fdecdcb 100644 --- a/Tests/RegexCategoryTests.swift +++ b/Tests/RegexCategoryTests.swift @@ -56,9 +56,30 @@ struct RegexCategoryTests { @Test("An unknown category name is a compile error") func test_unknownCategory() { - #expect(throws: PureXML.Regex.RegexError.self) { + #expect(throws: PureXML.Regex.RegexError.invalidProperty) { _ = try PureXML.Regex.Pattern("\\p{Zz}") } + // An unknown Is name is rejected too (the block set is complete). + #expect(throws: PureXML.Regex.RegexError.invalidProperty) { + _ = try PureXML.Regex.Pattern("\\p{IsaA0-a9}") + } + } + + @Test("Supplementary-plane Is names match beyond the BMP") + func test_supplementaryBlock() throws { + // U+1D11E (musical symbol G clef) is in MusicalSymbols (1D100..1D1FF). + #expect(try matches("\\p{IsMusicalSymbols}+", "\u{1D11E}")) + #expect(try !matches("\\p{IsMusicalSymbols}+", "a")) + // U+10300 (Old Italic letter A) is in OldItalic (10300..1032F). + #expect(try matches("\\p{IsOldItalic}", "\u{10300}")) + } + + @Test("Surrogate Is names compile but match no scalar") + func test_surrogateBlockMatchesNothing() throws { + // Lone surrogates are not Unicode scalars, so the block matches nothing, + // but the name is valid XSD so the pattern still compiles (no false reject). + #expect(try !matches("\\p{IsHighSurrogates}", "a")) + #expect(try !matches("\\p{IsLowSurrogates}+", "\u{1D11E}")) } @Test("Character-class subtraction removes the subtrahend") diff --git a/Tests/RegexTests.swift b/Tests/RegexTests.swift index 7a38415..2919df6 100644 --- a/Tests/RegexTests.swift +++ b/Tests/RegexTests.swift @@ -178,7 +178,8 @@ struct RegexTests { #expect(try matches(#"\p{IsTamil}+"#, "\u{0B85}")) #expect(try matches(#"\p{IsHangulJamo}+"#, "\u{1100}")) #expect(try !matches(#"\p{IsTamil}+"#, "\u{0531}")) // Armenian is not Tamil - // An unknown block name is still a compile error. - #expect(throws: PureXML.Regex.RegexError.self) { _ = try PureXML.Regex.Pattern(#"\p{IsNotARealBlock}"#) } + // An unknown block name is a compile error (the block set is complete, so + // an unrecognised name is a genuine syntax error, not an engine limitation). + #expect(throws: PureXML.Regex.RegexError.invalidProperty) { _ = try PureXML.Regex.Pattern(#"\p{IsNotARealBlock}"#) } } } diff --git a/Tests/SchemaPatternFacetTests.swift b/Tests/SchemaPatternFacetTests.swift index dde9fde..e54e2f1 100644 --- a/Tests/SchemaPatternFacetTests.swift +++ b/Tests/SchemaPatternFacetTests.swift @@ -5,8 +5,8 @@ import Testing /// was only compiled lazily at instance time (`try?`), so an unparseable one was /// silently ignored and the schema accepted. Compile-time validation rejects the /// unambiguous structural errors (unbalanced parentheses, a quantifier with -/// nothing to repeat) while tolerating constructs the engine merely does not -/// support on an otherwise-valid pattern (an untabulated `\p{Is...}` block, the +/// nothing to repeat, an unknown `\p{...}` category or block name) while tolerating +/// constructs the engine merely under-supports on an otherwise-valid pattern (the /// empty pattern, the lenient `{,m}` quantifier), so no valid schema is rejected. @Suite("Pattern facet regex validity") struct SchemaPatternFacetTests { @@ -57,10 +57,30 @@ struct SchemaPatternFacetTests { @Test("a construct the engine does not support is not treated as a schema error") func test_engineGapTolerated() throws { - // Empty pattern (valid: matches the empty string), an untabulated Unicode - // block escape, and the lenient `{,m}` quantifier are all accepted. + // The empty pattern (valid: matches the empty string) and the lenient + // `{,m}` quantifier are accepted. try compile("") - try compile(#"\p{IsHighSurrogates}"#) try compile("[0-9]{,5}") } + + @Test("an unknown \\p{...} category or block name is rejected (XSTS reK88)") + func test_unknownPropertyRejected() { + // `\p{IsaA0-a9}` (XSTS reK88) names no Unicode block; the recognised set is + // the complete XSD 1.0 charProp list, so an unknown name is a syntax error. + #expect(rejects(#"\p{IsaA0-a9}"#)) + #expect(rejects(#"\p{IsNotARealBlock}"#)) + #expect(rejects(#"\p{Zz}"#)) + } + + @Test("the complete XSD 1.0 block set compiles, including supplementary and surrogate blocks") + func test_fullBlockSetCompiles() throws { + // Supplementary-plane blocks (Unicode 3.1) are valid XSD block names. + try compile(#"\p{IsMusicalSymbols}"#) + try compile(#"\p{IsOldItalic}"#) + try compile(#"\p{IsCJKUnifiedIdeographsExtensionB}"#) + try compile(#"\p{IsTags}"#) + // Surrogate blocks are valid names whose ranges hold no scalar (match nothing). + try compile(#"\p{IsHighSurrogates}"#) + try compile(#"\p{IsLowSurrogates}"#) + } } diff --git a/Tests/XSTSSuiteTests.swift b/Tests/XSTSSuiteTests.swift index ca31dc5..fd18d51 100644 --- a/Tests/XSTSSuiteTests.swift +++ b/Tests/XSTSSuiteTests.swift @@ -31,10 +31,12 @@ struct XSTSSuiteTests { /// history of how each bucket reached its current value (every rule and its exact /// delta) lives in CHANGELOG.md and docs/xsts-deviations.md, not inline here. private let knownSchemaValidRejected = 0 - /// Schema bucket, per-step deltas in CHANGELOG.md. Latest (15 -> 13): an + /// Schema bucket, per-step deltas in CHANGELOG.md. Recent: 15 -> 13, an /// unescaped `[` inside a pattern-facet character class is a syntax error per - /// XSD Appendix F, so the schema is invalid (RegexTest_993, RegexTest_1477). - private let knownSchemaInvalidAccepted = 13 + /// XSD Appendix F (RegexTest_993, RegexTest_1477); 13 -> 12, an unknown + /// `\p{...}` category or block name is rejected now the block set is complete + /// (reK88, `\p{IsaA0-a9}`). + private let knownSchemaInvalidAccepted = 12 private let knownInstanceValidRejected = 0 /// Instance bucket (133 -> 22), per-step deltas in CHANGELOG.md: xsi:type must derive from the declared /// type; anyType cannot stand in for anySimpleType; an untyped substitutionGroup member inherits its head's diff --git a/docs/release-hurdles.md b/docs/release-hurdles.md index 7350229..b318f80 100644 --- a/docs/release-hurdles.md +++ b/docs/release-hurdles.md @@ -6,7 +6,7 @@ evidence, what actually stands between the current engine and a 1.0 fit to put i front of a developer as the authority on XML/XSD correctness. Snapshot at time of writing: v0.2.0 released; full 2006-11-06 XSTS archive (14383 -groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 13`, +groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 12`, `valid-instances-rejected = 0`, `invalid-instances-accepted = 19`. ## Status against the four stoppers @@ -16,7 +16,7 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 13`, libxml2 differential over 9618 schemas. **Met.** This is the doc's bar to be "an authority"; it is cleared on the measured corpora. 2. **Silently accepting faulty input (false negatives).** `invalid-schemas-accepted - = 13`, `invalid-instances-accepted = 19`. **Not at 0.** This is the bulk of named + = 12`, `invalid-instances-accepted = 19`. **Not at 0.** This is the bulk of named milestone work (M2, M3) and the subject of Part 1 below. 3. **Uncharacterized correctness.** Differential harness vs libxml2 and a fuzz suite exist (M1, ~done). Gap: real-world corpora beyond XSTS, and the silent @@ -28,7 +28,7 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 13`, ## The biggest hurdle, stated plainly -Driving the false-negative buckets (`invalid-schemas-accepted = 13`, +Driving the false-negative buckets (`invalid-schemas-accepted = 12`, `invalid-instances-accepted = 19`) to zero is the largest remaining body of named work, and it is hard for a specific reason: **the ~95% already closed were the tractable cases; every case left fights the cardinal rule.** The remainder sits in