diff --git a/CHANGELOG.md b/CHANGELOG.md index c4b3eeb..33c8ba5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -34,6 +34,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- An unknown `\p{...}` category or block name in a `pattern` facet is now rejected at schema-compile time (false-negative burndown, stopper #2). XSD 1.0 `charProp` is `IsCategory | IsBlock`, an exhaustive set: every Unicode general category plus a fixed Unicode-3.1 block list. The block table (`CategoryMatcher.blocks`) was missing the nine supplementary-plane blocks (Old Italic, Gothic, Deseret, the Byzantine/Musical/Mathematical symbol blocks, CJK Unified Ideographs Extension B, CJK Compatibility Ideographs Supplement, Tags) and the three surrogate blocks, so `parseCategory` could not tell a real-but-untabulated block from a bogus name and stayed lenient (`RegexError.unsupported`), accepting `\p{IsaA0-a9}` and the schema declaring it. The table is now the complete XSD set (supplementary blocks added with their ranges; surrogate blocks added as valid names whose ranges hold no scalar, so they match nothing), making the recognised set authoritative, so an unrecognised name throws the rejecting `RegexError.invalidProperty`. The now-dead `RegexError.unsupported` case and the `isMalformedProperty` heuristic are removed. This closes XSTS reK88, dropping `invalid-schemas-accepted` from 13 to 12 with `valid-schemas-rejected` held at 0 (full XSTS archive) and the instance buckets unchanged (`invalid-instances-accepted` 19). As a side effect, a pattern naming a supplementary-plane block (e.g. `\p{IsMusicalSymbols}`) now matches its characters instead of failing to compile. Block names/ranges were taken from the XSD-supported list in Xerces-J `Token.java` (clean-room: factual data only). Regressions are guarded by `RegexCategoryTests` (supplementary/surrogate matching, unknown-name rejection) and `SchemaPatternFacetTests`. + - A `pattern` facet whose value contains an unescaped `[` inside a character class is now rejected at schema-compile time (false-negative burndown, stopper #2). XSD 1.0 Appendix F defines `SingleCharNoEsc` as `[^\[\]]` and admits a nested class only as a `-[...]` subtraction, so a `[` not introduced by `-` is a syntax error; `RegexParser` was treating it as a literal scalar, so `[a[:xyz:]` compiled and the declaring schema was accepted. The parser now throws a dedicated `RegexError.unescapedClassBracket` (kept distinct from the generic, deliberately-lenient `badClass` so the change cannot regress a valid-but-unsupported pattern), which `XSDSimpleParser.rejectsPattern` treats as a genuine syntax error. This closes XSTS RegexTest_993 and RegexTest_1477, dropping `invalid-schemas-accepted` from 15 to 13 with `valid-schemas-rejected` held at 0 (full XSTS archive) and the instance buckets unchanged (`invalid-instances-accepted` 19). The escaped form `[a\[c]` and the `-[...]` subtraction `[a-z-[aeiou]]` stay valid; the unterminated case `[a[:xyz:` now faults on the inner `[` (more precise than the prior unterminated-class diagnosis). Regressions are guarded by `RegexTests.test_unescapedClassBracket` and `SchemaPatternFacetTests`. - Compiling a schema with deeply nested, multiply-referenced model groups is no longer exponential. `XSDParserParticles.groupReferenceParticle` inlined each `` into the compiled `Particle` content model, so `g0=(g1,g1), g1=(g2,g2), …` built `2^K` particles (measured: depth 14 = 0.21s, depth 20 ~13-18s, growing ~4x per two levels). The expansion is now memoized per (group definition, cycle-visiting set) in a per-compile `GroupParticleMemo`, so sibling references share it and the build drops to `O(groups)` (depth 40 now compiles in ~0.016s; previously infeasible). The shared content term is reused by value (copy-on-write), so the compiled model is byte-for-byte the same and validation is unchanged: the XSTS gate is unchanged (invalid-schemas-accepted 15, valid-schemas-rejected 0, instance buckets held) and the streaming-vs-tree fuzz differential still passes. This closes the schema-compile half of the interactive-safety group-blowup (the companion to the `positionCap` determinism fix); a regression is guarded by a depth-40 compile test. diff --git a/Sources/Regex/RegexCategory.swift b/Sources/Regex/RegexCategory.swift index 45cac0f..1700f5c 100644 --- a/Sources/Regex/RegexCategory.swift +++ b/Sources/Regex/RegexCategory.swift @@ -52,13 +52,21 @@ extension PureXML.Regex { .control: "Cc", .format: "Cf", .surrogate: "Cs", .privateUse: "Co", .unassigned: "Cn", ] - /// The Unicode block names XSD 1.0 admits in `\p{Is}` (the - /// Unicode 3.1 blocks the regular-expression appendix fixes), each mapped - /// to its code-point range. An `Is` name outside this set is - /// rejected rather than guessed, so a pattern never matches against a - /// fabricated range. Surrogate blocks are omitted: surrogates are not - /// scalar values. `Specials` uses the main `FFF0..FFFD` range (the spec's - /// separate `FEFF` half is the byte-order mark, not matched here). + /// The complete set of Unicode block names XSD 1.0 admits in + /// `\p{Is}` (the Unicode 3.1 blocks the regular-expression appendix + /// fixes), each mapped to its code-point range. The set is exhaustive: an + /// `Is` name outside it is no XSD block and is rejected at compile + /// time (`isMalformedProperty` is gone; the predicate's presence here is the + /// sole authority), so a pattern never matches against a fabricated range. + /// XSD 1.0 freezes this list at Unicode 3.1, so a block introduced later + /// (e.g. `CyrillicSupplement`) is by design not an XSD 1.0 block and is + /// rejected, even though some processors accept any `Is[A-Za-z0-9-]+` + /// leniently. This is a deliberate spec-strict choice, not an oversight. + /// The three surrogate blocks are listed so a pattern naming them still + /// compiles, but their ranges hold no scalar value, so they match nothing. + /// `Specials` uses the main `FFF0..FFFD` range (the spec's separate `FEFF` + /// half is the byte-order mark, not matched here). `GreekandCoptic` is the + /// later Unicode rename of `Greek`, kept as a lenient alias. private static let blocks: [String: ClosedRange] = [ "BasicLatin": 0x0000 ... 0x007F, "Latin-1Supplement": 0x0080 ... 0x00FF, @@ -145,6 +153,21 @@ extension PureXML.Regex { "ArabicPresentationForms-B": 0xFE70 ... 0xFEFF, "HalfwidthandFullwidthForms": 0xFF00 ... 0xFFEF, "Specials": 0xFFF0 ... 0xFFFD, + // Surrogate blocks: valid XSD block names whose ranges hold no scalar + // value (lone surrogates are not Unicode scalars), so they match nothing. + "HighSurrogates": 0xD800 ... 0xDB7F, + "HighPrivateUseSurrogates": 0xDB80 ... 0xDBFF, + "LowSurrogates": 0xDC00 ... 0xDFFF, + // Supplementary-plane blocks (Unicode 3.1, beyond the BMP). + "OldItalic": 0x10300 ... 0x1032F, + "Gothic": 0x10330 ... 0x1034F, + "Deseret": 0x10400 ... 0x1044F, + "ByzantineMusicalSymbols": 0x1D000 ... 0x1D0FF, + "MusicalSymbols": 0x1D100 ... 0x1D1FF, + "MathematicalAlphanumericSymbols": 0x1D400 ... 0x1D7FF, + "CJKUnifiedIdeographsExtensionB": 0x20000 ... 0x2A6D6, + "CJKCompatibilityIdeographsSupplement": 0x2F800 ... 0x2FA1F, + "Tags": 0xE0000 ... 0xE007F, ] } } diff --git a/Sources/Regex/RegexParser.swift b/Sources/Regex/RegexParser.swift index 879f180..2f68f89 100644 --- a/Sources/Regex/RegexParser.swift +++ b/Sources/Regex/RegexParser.swift @@ -133,28 +133,17 @@ extension PureXML.Regex { guard peek() == "}" else { throw RegexError.badClass } advance() guard let predicate = CategoryPredicate(name: name, negated: negated) else { - // An unknown name is by default a limit of this engine, not an error - // (stay lenient). But a name that can be NO XSD charProp at all is - // unambiguously invalid and rejects, independent of which categories - // or blocks this engine happens to recognise. - throw Self.isMalformedProperty(name) ? RegexError.invalidProperty : RegexError.unsupported("\\p{\(name)}") + // The recognised set (every Unicode general category plus the + // complete XSD 1.0 `Is` list, see CategoryMatcher) is the + // exhaustive set of XSD `charProp` names, so any name outside it is + // no XSD category or block and is a genuine syntax error, not an + // engine limitation. Rejecting it (XSTS reK88, `\p{IsaA0-a9}`) + // cannot reject a valid pattern because the set is complete. + throw RegexError.invalidProperty } return predicate } - /// Whether `name` can be no XSD `charProp` (`IsCategory | IsBlock`) under any - /// repertoire: empty, the bare `Is` prefix, a non-block name (no `Is` prefix) - /// carrying a non-letter, or `Is` followed by a non-`[A-Za-z0-9-]` character. - /// A well-formed but unrecognised name (an unknown block such as `IsaA0-a9`) - /// is NOT malformed and stays a lenient `unsupported` limit. - private static func isMalformedProperty(_ name: String) -> Bool { - if name.isEmpty || name == "Is" { return true } - guard name.hasPrefix("Is") else { - return name.contains { !$0.isLetter } - } - return name.dropFirst(2).contains { !($0.isLetter || $0.isNumber || $0 == "-") } - } - private func classFrom(_ member: ClassMember) -> CharClass { switch member { case let .scalar(value): .single(value) diff --git a/Sources/Regex/RegexSyntax.swift b/Sources/Regex/RegexSyntax.swift index 951b6ed..b4920df 100644 --- a/Sources/Regex/RegexSyntax.swift +++ b/Sources/Regex/RegexSyntax.swift @@ -6,7 +6,6 @@ extension PureXML.Regex { case badEscape(String) case badClass case badQuantifier - case unsupported(String) /// A character-class range whose first endpoint exceeds its second /// (`[b-a]`): unambiguously invalid, never an engine limitation. case reversedRange @@ -39,7 +38,6 @@ extension PureXML.Regex { case let .badEscape(detail): "invalid escape '\\\(detail)'" case .badClass: "malformed character class" case .badQuantifier: "malformed quantifier" - case let .unsupported(detail): "unsupported construct: \(detail)" case .reversedRange: "character-class range is reversed" case .emptyClass: "empty character class" case .reversedQuantifier: "quantifier minimum exceeds maximum" diff --git a/Sources/Schema/XSDSimpleParser.swift b/Sources/Schema/XSDSimpleParser.swift index eb41d01..8f4d5f0 100644 --- a/Sources/Schema/XSDSimpleParser.swift +++ b/Sources/Schema/XSDSimpleParser.swift @@ -66,9 +66,10 @@ extension PureXML.Schema { } catch let error as PureXML.Regex.RegexError where rejectsPattern(error) { errors.append("pattern '\(value)' is not a valid regular expression: \(error)") } catch { - // A construct the engine does not support (`.empty`, `.unsupported` - // for an untabulated `\p{Is...}` block, `.badQuantifier` for the - // lenient `{,m}` form) is a valid XSD pattern, not a schema error. + // A construct the engine merely under-supports on an otherwise- + // valid pattern (`.badQuantifier` for the lenient `{,m}` form, + // `.badEscape`/`.badClass`) is a valid XSD pattern, not a schema + // error. The empty pattern is valid and never throws. } } return errors @@ -82,7 +83,7 @@ extension PureXML.Schema { case .unbalanced, .danglingQuantifier, .reversedRange, .emptyClass, .reversedQuantifier, .incompleteEscape, .unterminatedClass, .unescapedClassBracket, .invalidProperty: true - case .unsupported, .badQuantifier, .badEscape, .badClass: + case .badQuantifier, .badEscape, .badClass: false } } diff --git a/Tests/RegexCategoryTests.swift b/Tests/RegexCategoryTests.swift index 4b5fd0e..fdecdcb 100644 --- a/Tests/RegexCategoryTests.swift +++ b/Tests/RegexCategoryTests.swift @@ -56,9 +56,30 @@ struct RegexCategoryTests { @Test("An unknown category name is a compile error") func test_unknownCategory() { - #expect(throws: PureXML.Regex.RegexError.self) { + #expect(throws: PureXML.Regex.RegexError.invalidProperty) { _ = try PureXML.Regex.Pattern("\\p{Zz}") } + // An unknown Is name is rejected too (the block set is complete). + #expect(throws: PureXML.Regex.RegexError.invalidProperty) { + _ = try PureXML.Regex.Pattern("\\p{IsaA0-a9}") + } + } + + @Test("Supplementary-plane Is names match beyond the BMP") + func test_supplementaryBlock() throws { + // U+1D11E (musical symbol G clef) is in MusicalSymbols (1D100..1D1FF). + #expect(try matches("\\p{IsMusicalSymbols}+", "\u{1D11E}")) + #expect(try !matches("\\p{IsMusicalSymbols}+", "a")) + // U+10300 (Old Italic letter A) is in OldItalic (10300..1032F). + #expect(try matches("\\p{IsOldItalic}", "\u{10300}")) + } + + @Test("Surrogate Is names compile but match no scalar") + func test_surrogateBlockMatchesNothing() throws { + // Lone surrogates are not Unicode scalars, so the block matches nothing, + // but the name is valid XSD so the pattern still compiles (no false reject). + #expect(try !matches("\\p{IsHighSurrogates}", "a")) + #expect(try !matches("\\p{IsLowSurrogates}+", "\u{1D11E}")) } @Test("Character-class subtraction removes the subtrahend") diff --git a/Tests/RegexTests.swift b/Tests/RegexTests.swift index 7a38415..2919df6 100644 --- a/Tests/RegexTests.swift +++ b/Tests/RegexTests.swift @@ -178,7 +178,8 @@ struct RegexTests { #expect(try matches(#"\p{IsTamil}+"#, "\u{0B85}")) #expect(try matches(#"\p{IsHangulJamo}+"#, "\u{1100}")) #expect(try !matches(#"\p{IsTamil}+"#, "\u{0531}")) // Armenian is not Tamil - // An unknown block name is still a compile error. - #expect(throws: PureXML.Regex.RegexError.self) { _ = try PureXML.Regex.Pattern(#"\p{IsNotARealBlock}"#) } + // An unknown block name is a compile error (the block set is complete, so + // an unrecognised name is a genuine syntax error, not an engine limitation). + #expect(throws: PureXML.Regex.RegexError.invalidProperty) { _ = try PureXML.Regex.Pattern(#"\p{IsNotARealBlock}"#) } } } diff --git a/Tests/SchemaPatternFacetTests.swift b/Tests/SchemaPatternFacetTests.swift index dde9fde..e54e2f1 100644 --- a/Tests/SchemaPatternFacetTests.swift +++ b/Tests/SchemaPatternFacetTests.swift @@ -5,8 +5,8 @@ import Testing /// was only compiled lazily at instance time (`try?`), so an unparseable one was /// silently ignored and the schema accepted. Compile-time validation rejects the /// unambiguous structural errors (unbalanced parentheses, a quantifier with -/// nothing to repeat) while tolerating constructs the engine merely does not -/// support on an otherwise-valid pattern (an untabulated `\p{Is...}` block, the +/// nothing to repeat, an unknown `\p{...}` category or block name) while tolerating +/// constructs the engine merely under-supports on an otherwise-valid pattern (the /// empty pattern, the lenient `{,m}` quantifier), so no valid schema is rejected. @Suite("Pattern facet regex validity") struct SchemaPatternFacetTests { @@ -57,10 +57,30 @@ struct SchemaPatternFacetTests { @Test("a construct the engine does not support is not treated as a schema error") func test_engineGapTolerated() throws { - // Empty pattern (valid: matches the empty string), an untabulated Unicode - // block escape, and the lenient `{,m}` quantifier are all accepted. + // The empty pattern (valid: matches the empty string) and the lenient + // `{,m}` quantifier are accepted. try compile("") - try compile(#"\p{IsHighSurrogates}"#) try compile("[0-9]{,5}") } + + @Test("an unknown \\p{...} category or block name is rejected (XSTS reK88)") + func test_unknownPropertyRejected() { + // `\p{IsaA0-a9}` (XSTS reK88) names no Unicode block; the recognised set is + // the complete XSD 1.0 charProp list, so an unknown name is a syntax error. + #expect(rejects(#"\p{IsaA0-a9}"#)) + #expect(rejects(#"\p{IsNotARealBlock}"#)) + #expect(rejects(#"\p{Zz}"#)) + } + + @Test("the complete XSD 1.0 block set compiles, including supplementary and surrogate blocks") + func test_fullBlockSetCompiles() throws { + // Supplementary-plane blocks (Unicode 3.1) are valid XSD block names. + try compile(#"\p{IsMusicalSymbols}"#) + try compile(#"\p{IsOldItalic}"#) + try compile(#"\p{IsCJKUnifiedIdeographsExtensionB}"#) + try compile(#"\p{IsTags}"#) + // Surrogate blocks are valid names whose ranges hold no scalar (match nothing). + try compile(#"\p{IsHighSurrogates}"#) + try compile(#"\p{IsLowSurrogates}"#) + } } diff --git a/Tests/XSTSSuiteTests.swift b/Tests/XSTSSuiteTests.swift index ca31dc5..fd18d51 100644 --- a/Tests/XSTSSuiteTests.swift +++ b/Tests/XSTSSuiteTests.swift @@ -31,10 +31,12 @@ struct XSTSSuiteTests { /// history of how each bucket reached its current value (every rule and its exact /// delta) lives in CHANGELOG.md and docs/xsts-deviations.md, not inline here. private let knownSchemaValidRejected = 0 - /// Schema bucket, per-step deltas in CHANGELOG.md. Latest (15 -> 13): an + /// Schema bucket, per-step deltas in CHANGELOG.md. Recent: 15 -> 13, an /// unescaped `[` inside a pattern-facet character class is a syntax error per - /// XSD Appendix F, so the schema is invalid (RegexTest_993, RegexTest_1477). - private let knownSchemaInvalidAccepted = 13 + /// XSD Appendix F (RegexTest_993, RegexTest_1477); 13 -> 12, an unknown + /// `\p{...}` category or block name is rejected now the block set is complete + /// (reK88, `\p{IsaA0-a9}`). + private let knownSchemaInvalidAccepted = 12 private let knownInstanceValidRejected = 0 /// Instance bucket (133 -> 22), per-step deltas in CHANGELOG.md: xsi:type must derive from the declared /// type; anyType cannot stand in for anySimpleType; an untyped substitutionGroup member inherits its head's diff --git a/docs/release-hurdles.md b/docs/release-hurdles.md index 7350229..b318f80 100644 --- a/docs/release-hurdles.md +++ b/docs/release-hurdles.md @@ -6,7 +6,7 @@ evidence, what actually stands between the current engine and a 1.0 fit to put i front of a developer as the authority on XML/XSD correctness. Snapshot at time of writing: v0.2.0 released; full 2006-11-06 XSTS archive (14383 -groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 13`, +groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 12`, `valid-instances-rejected = 0`, `invalid-instances-accepted = 19`. ## Status against the four stoppers @@ -16,7 +16,7 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 13`, libxml2 differential over 9618 schemas. **Met.** This is the doc's bar to be "an authority"; it is cleared on the measured corpora. 2. **Silently accepting faulty input (false negatives).** `invalid-schemas-accepted - = 13`, `invalid-instances-accepted = 19`. **Not at 0.** This is the bulk of named + = 12`, `invalid-instances-accepted = 19`. **Not at 0.** This is the bulk of named milestone work (M2, M3) and the subject of Part 1 below. 3. **Uncharacterized correctness.** Differential harness vs libxml2 and a fuzz suite exist (M1, ~done). Gap: real-world corpora beyond XSTS, and the silent @@ -28,7 +28,7 @@ groups) with `valid-schemas-rejected = 0`, `invalid-schemas-accepted = 13`, ## The biggest hurdle, stated plainly -Driving the false-negative buckets (`invalid-schemas-accepted = 13`, +Driving the false-negative buckets (`invalid-schemas-accepted = 12`, `invalid-instances-accepted = 19`) to zero is the largest remaining body of named work, and it is hard for a specific reason: **the ~95% already closed were the tractable cases; every case left fights the cardinal rule.** The remainder sits in