From 2eb9f83958c32670a0b3d8c4db484a26e68d58e5 Mon Sep 17 00:00:00 2001 From: Mihaela Mihaljevic Date: Mon, 29 Jun 2026 07:05:27 +0200 Subject: [PATCH] fix(schema): back XSD \i and \c with the XML 1.0 Appendix B name classes (#378) XSD 1.0 (Datatypes Appendix F) defines the regex escape \i as the characters that may begin an XML Name (Letter | '_' | ':') and \c as the characters that may appear in one (NameChar), citing the XML 1.0 Second Edition productions: the enumerated, Unicode-2.0-era character classes of XML 1.0 Appendix B (BaseChar, Ideographic, CombiningChar, Digit, Extender). \i and \c were wired to PureXML.Parsing.XMLCharacter, which correctly implements the Fifth Edition Name production used to parse live XML, but the two genuinely differ: the Fifth Edition admits whole blocks where the Second Edition has gaps, so \c over-accepted, e.g. the combining mark U+0346 (COMBINING BRIDGE ABOVE), which Appendix B's CombiningChar excludes (its first range ends at U+0345). Add XSDNameCharacter, transcribed clean-room from the Appendix B grammar (identical across editions one through four) and stored as sorted, non- overlapping code-point ranges with binary-search membership; the parser's own name check is untouched. The two tables reproduce the W3C XSTS \i/\c enumerations (reZ005, reZ006: 34516 and 35122 code points) exactly and in both directions, so the more restrictive definition rejects no spec-valid input: there is no false-positive surface. Verified over the full W3C XSTS corpus in release: invalid-instances-accepted 12 -> 11 (reZ006i, whose value is U+0346, now rejects), both false-positive buckets at 0. Full suite (1834 tests) green; an adversarial critic brute-forced the binary search over the whole code-point space (0 mismatches), confirmed the tables are sorted and non-overlapping and equal the oracle sets, and that the parser's Fifth Edition path is left intact. Found by triaging the invalid- instance tail to a single boundary code point (U+0345 valid vs U+0346 invalid) and reading the XML 1.0 Appendix B grammar. --- CHANGELOG.md | 2 + Sources/Regex/RegexSyntax.swift | 7 +- Sources/Regex/XSDNameCharacter.swift | 147 +++++++++++++++++++++++++++ Tests/XSDNameCharacterTests.swift | 65 ++++++++++++ Tests/XSTSSuiteTests.swift | 6 +- docs/xsts-deviations.md | 20 ++-- 6 files changed, 236 insertions(+), 11 deletions(-) create mode 100644 Sources/Regex/XSDNameCharacter.swift create mode 100644 Tests/XSDNameCharacterTests.swift diff --git a/CHANGELOG.md b/CHANGELOG.md index 9acc47a..833495b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -31,6 +31,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- The XSD regex escapes `\i` and `\c` now use the XML 1.0 Second Edition name classes (Appendix B), not the parser's Fifth Edition Name production. XSD 1.0 (Datatypes Appendix F) defines `\i` as the characters that may begin an XML Name (`Letter | '_' | ':'`) and `\c` as the characters that may appear in one (`NameChar`), citing the XML 1.0 Second Edition productions, which are the enumerated Unicode-2.0-era character classes of XML 1.0 Appendix B. `\i`/`\c` were wired to `PureXML.Parsing.XMLCharacter`, which correctly implements the Fifth Edition production used to parse live XML, but the two genuinely differ: the Fifth Edition admits whole blocks where the Second Edition has gaps, so `\c` over-accepted, for example the combining mark `U+0346` (`COMBINING BRIDGE ABOVE`), which Appendix B's `CombiningChar` excludes (its first range ends at `U+0345`). A new `XSDNameCharacter`, transcribed clean-room from the Appendix B grammar (identical across editions one through four) and stored as sorted code-point ranges with binary-search membership, now backs both escapes; the parser's own name check is untouched. The two range tables are verified to reproduce the W3C XSTS `\i`/`\c` enumerations (`reZ005`, `reZ006`, 34516 and 35122 code points) exactly and in both directions, so there is no false-positive surface. Verified over the full W3C XSTS corpus in release: `invalid-instances-accepted` 12 -> 11 (`reZ006i`, whose value is `U+0346`, now rejects), with both false-positive buckets at 0. Found by triaging the invalid-instance tail to a single boundary code point (`U+0345` valid vs `U+0346` invalid) and reading the XML 1.0 Appendix B grammar. + - The Ethiopic digits `U+1369`..`U+1371` (`ETHIOPIC DIGIT ONE`..`NINE`) now match `\d` in an XSD pattern and not `\D`. XSD 1.0 fixes the regular-expression repertoire at Unicode 3.2 (Datatypes Appendix F), where these nine code points are general category `Nd` and each carries a decimal value; a later Unicode reclassified them to `No`. `XSDCategory` derives `\d` (`== \p{Nd}`) from the running platform's category, pinned to 3.2 by an assigned-in-3.2 gate plus a per-code-point override table, but the override table was missing these nine, so `\d` used the modern `No` and rejected them. This was a two-directional bug: a valid instance whose `\d`-constrained value was an Ethiopic digit was wrongly rejected (the cardinal rule: never reject a valid value), and a `\D`-constrained value of one was wrongly accepted. They are now pinned to `Nd`, which is the complete set of code points that were `Nd` at 3.2 and are not today (the reverse direction is empty: every code point that became `Nd` later was unassigned at 3.2 and so already reports `Cn`, verified by diffing the 3.2 and current Unicode `Nd` sets), so `\w` is unaffected (an Ethiopic numeral is category `N` either way, neither punctuation, separator, nor other). Verified over the full W3C XSTS corpus in release: `invalid-instances-accepted` 14 -> 12 (`reT17`, `reT38`, the `\D` cases, now reject), the matching `\d` valid cases (`reS17`, `reS38`, `reZ004`) no longer need their suspect-test exclusion and pass as ordinary agreements, and the two false-positive buckets stay 0. Found while triaging the invalid-instance tail against libxml2 (which rejects `reT17`/`reT38`, the reference-aligned reading) back to the authoritative `UnicodeData-3.2.0`; a prior note that had treated these as a disputed spec-divergence was the wrong call and is corrected. - A value of `NaN` is no longer wrongly rejected by an `xs:double` or `xs:float` `enumeration` that lists `NaN`. Value-space equality (`OrderedValue.==`) compared with IEEE equality, under which `NaN != NaN`, so a `NaN` value never matched a `NaN` enumeration member and was rejected, a false positive (the cardinal rule: never reject a valid value). The comparison now treats `NaN` as identical to itself, the value-space identity the reference (libxml2) and XSD 1.1 use throughout, so `NaN` validates against a type restricting `xs:double` by ``; `INF`/`-INF` already matched. This equality also backs the `key`/`unique` identity constraints and `fixed`, where the same identity holds (verified against `xmllint`): two `NaN`-valued unique fields collide as duplicates, and `fixed="NaN"` matches `NaN`. The ordering facets (`minInclusive` and friends) use the order relation, in which `NaN` stays incomparable, and are unaffected. Found by the new generated-value facet differential against libxml2, with the key/fixed behaviour confirmed after an adversarial critic flagged the shared comparison; the four W3C XSTS buckets held and the full suite passes, with regressions in `SchemaDoubleEnumerationTests`. diff --git a/Sources/Regex/RegexSyntax.swift b/Sources/Regex/RegexSyntax.swift index d8d2349..7d22896 100644 --- a/Sources/Regex/RegexSyntax.swift +++ b/Sources/Regex/RegexSyntax.swift @@ -81,8 +81,11 @@ extension PureXML.Regex { case .word: let code = XSDCategory.generalCategoryCode(scalar) return !(code.hasPrefix("P") || code.hasPrefix("Z") || code.hasPrefix("C")) - case .nameStart: return PureXML.Parsing.XMLCharacter.isNameStart(scalar) - case .nameChar: return PureXML.Parsing.XMLCharacter.isNameChar(scalar) + // \i and \c are the XML 1.0 Second Edition name classes (Appendix B), + // a fixed Unicode-2.0 repertoire, NOT the parser's Fifth Edition Name + // production: the two genuinely differ (XSTS reZ005/reZ006). + case .nameStart: return XSDNameCharacter.isNameStart(scalar) + case .nameChar: return XSDNameCharacter.isNameChar(scalar) case .anyButLineBreak: return scalar != "\n" && scalar != "\r" } } diff --git a/Sources/Regex/XSDNameCharacter.swift b/Sources/Regex/XSDNameCharacter.swift new file mode 100644 index 0000000..87a01be --- /dev/null +++ b/Sources/Regex/XSDNameCharacter.swift @@ -0,0 +1,147 @@ +extension PureXML.Regex { + /// The XML 1.0 name-character classes that back the XSD regular-expression + /// escapes `\i` and `\c` (XSD 1.0 Datatypes Appendix F). XSD 1.0 defines + /// `\i` as the characters that may begin an XML Name (`Letter | '_' | ':'`) + /// and `\c` as the characters that may appear in one (`NameChar`), citing the + /// productions of XML 1.0 Second Edition. Those are the enumerated character + /// classes of XML 1.0 Appendix B (BaseChar, Ideographic, CombiningChar, Digit, + /// Extender), a fixed Unicode-2.0-era repertoire, not the broad ranges XML 1.0 + /// Fifth Edition later substituted. + /// + /// This is deliberately distinct from ``PureXML/Parsing/XMLCharacter``, which + /// implements the Fifth Edition Name production used to parse live XML: the two + /// definitions genuinely differ (Fifth Edition admits whole blocks where Second + /// Edition has gaps, e.g. the combining range is `[#x0300-#x0345]` here but the + /// entire `0x300...0x36F` block in the parser). Using the parser's check for + /// `\c` over-accepted, e.g. the combining mark U+0346 (XSTS reZ006i). + /// + /// The ranges are transcribed clean-room from the XML 1.0 Appendix B grammar + /// (identical across editions one through four; only the fifth replaced it) and + /// stored sorted and non-overlapping so membership is a binary search. They are + /// verified to reproduce the W3C XSTS `\i`/`\c` enumerations (reZ005, reZ006) + /// exactly, in both directions, by ``XSDNameCharacterTests``. + enum XSDNameCharacter { + /// Whether `scalar` may begin an XML Name per XSD `\i` (`Letter | '_' | ':'`). + static func isNameStart(_ scalar: Unicode.Scalar) -> Bool { + contains(scalar.value, in: nameStartRanges) + } + + /// Whether `scalar` may appear in an XML Name per XSD `\c` (`NameChar`). + static func isNameChar(_ scalar: Unicode.Scalar) -> Bool { + contains(scalar.value, in: nameCharRanges) + } + + /// Binary search over a sorted, non-overlapping `[lo, hi, lo, hi, ...]` + /// table; each step compares one range's inclusive bounds. + private static func contains(_ value: UInt32, in ranges: [UInt32]) -> Bool { + var low = 0 + var high = ranges.count / 2 + while low < high { + let mid = (low + high) / 2 + if value < ranges[mid * 2] { + high = mid + } else if value > ranges[mid * 2 + 1] { + low = mid + 1 + } else { + return true + } + } + return false + } + } +} + +private extension PureXML.Regex.XSDNameCharacter { + /// Inclusive `[lo, hi]` bounds of the XSD `\i` set: XML 1.0 Appendix B + /// `Letter` (`BaseChar | Ideographic`) plus `'_'` and `':'`. + static let nameStartRanges: [UInt32] = [ + 0x3A, 0x3A, 0x41, 0x5A, 0x5F, 0x5F, 0x61, 0x7A, 0xC0, 0xD6, 0xD8, 0xF6, + 0xF8, 0x131, 0x134, 0x13E, 0x141, 0x148, 0x14A, 0x17E, 0x180, 0x1C3, 0x1CD, 0x1F0, + 0x1F4, 0x1F5, 0x1FA, 0x217, 0x250, 0x2A8, 0x2BB, 0x2C1, 0x386, 0x386, 0x388, 0x38A, + 0x38C, 0x38C, 0x38E, 0x3A1, 0x3A3, 0x3CE, 0x3D0, 0x3D6, 0x3DA, 0x3DA, 0x3DC, 0x3DC, + 0x3DE, 0x3DE, 0x3E0, 0x3E0, 0x3E2, 0x3F3, 0x401, 0x40C, 0x40E, 0x44F, 0x451, 0x45C, + 0x45E, 0x481, 0x490, 0x4C4, 0x4C7, 0x4C8, 0x4CB, 0x4CC, 0x4D0, 0x4EB, 0x4EE, 0x4F5, + 0x4F8, 0x4F9, 0x531, 0x556, 0x559, 0x559, 0x561, 0x586, 0x5D0, 0x5EA, 0x5F0, 0x5F2, + 0x621, 0x63A, 0x641, 0x64A, 0x671, 0x6B7, 0x6BA, 0x6BE, 0x6C0, 0x6CE, 0x6D0, 0x6D3, + 0x6D5, 0x6D5, 0x6E5, 0x6E6, 0x905, 0x939, 0x93D, 0x93D, 0x958, 0x961, 0x985, 0x98C, + 0x98F, 0x990, 0x993, 0x9A8, 0x9AA, 0x9B0, 0x9B2, 0x9B2, 0x9B6, 0x9B9, 0x9DC, 0x9DD, + 0x9DF, 0x9E1, 0x9F0, 0x9F1, 0xA05, 0xA0A, 0xA0F, 0xA10, 0xA13, 0xA28, 0xA2A, 0xA30, + 0xA32, 0xA33, 0xA35, 0xA36, 0xA38, 0xA39, 0xA59, 0xA5C, 0xA5E, 0xA5E, 0xA72, 0xA74, + 0xA85, 0xA8B, 0xA8D, 0xA8D, 0xA8F, 0xA91, 0xA93, 0xAA8, 0xAAA, 0xAB0, 0xAB2, 0xAB3, + 0xAB5, 0xAB9, 0xABD, 0xABD, 0xAE0, 0xAE0, 0xB05, 0xB0C, 0xB0F, 0xB10, 0xB13, 0xB28, + 0xB2A, 0xB30, 0xB32, 0xB33, 0xB36, 0xB39, 0xB3D, 0xB3D, 0xB5C, 0xB5D, 0xB5F, 0xB61, + 0xB85, 0xB8A, 0xB8E, 0xB90, 0xB92, 0xB95, 0xB99, 0xB9A, 0xB9C, 0xB9C, 0xB9E, 0xB9F, + 0xBA3, 0xBA4, 0xBA8, 0xBAA, 0xBAE, 0xBB5, 0xBB7, 0xBB9, 0xC05, 0xC0C, 0xC0E, 0xC10, + 0xC12, 0xC28, 0xC2A, 0xC33, 0xC35, 0xC39, 0xC60, 0xC61, 0xC85, 0xC8C, 0xC8E, 0xC90, + 0xC92, 0xCA8, 0xCAA, 0xCB3, 0xCB5, 0xCB9, 0xCDE, 0xCDE, 0xCE0, 0xCE1, 0xD05, 0xD0C, + 0xD0E, 0xD10, 0xD12, 0xD28, 0xD2A, 0xD39, 0xD60, 0xD61, 0xE01, 0xE2E, 0xE30, 0xE30, + 0xE32, 0xE33, 0xE40, 0xE45, 0xE81, 0xE82, 0xE84, 0xE84, 0xE87, 0xE88, 0xE8A, 0xE8A, + 0xE8D, 0xE8D, 0xE94, 0xE97, 0xE99, 0xE9F, 0xEA1, 0xEA3, 0xEA5, 0xEA5, 0xEA7, 0xEA7, + 0xEAA, 0xEAB, 0xEAD, 0xEAE, 0xEB0, 0xEB0, 0xEB2, 0xEB3, 0xEBD, 0xEBD, 0xEC0, 0xEC4, + 0xF40, 0xF47, 0xF49, 0xF69, 0x10A0, 0x10C5, 0x10D0, 0x10F6, 0x1100, 0x1100, 0x1102, 0x1103, + 0x1105, 0x1107, 0x1109, 0x1109, 0x110B, 0x110C, 0x110E, 0x1112, 0x113C, 0x113C, 0x113E, 0x113E, + 0x1140, 0x1140, 0x114C, 0x114C, 0x114E, 0x114E, 0x1150, 0x1150, 0x1154, 0x1155, 0x1159, 0x1159, + 0x115F, 0x1161, 0x1163, 0x1163, 0x1165, 0x1165, 0x1167, 0x1167, 0x1169, 0x1169, 0x116D, 0x116E, + 0x1172, 0x1173, 0x1175, 0x1175, 0x119E, 0x119E, 0x11A8, 0x11A8, 0x11AB, 0x11AB, 0x11AE, 0x11AF, + 0x11B7, 0x11B8, 0x11BA, 0x11BA, 0x11BC, 0x11C2, 0x11EB, 0x11EB, 0x11F0, 0x11F0, 0x11F9, 0x11F9, + 0x1E00, 0x1E9B, 0x1EA0, 0x1EF9, 0x1F00, 0x1F15, 0x1F18, 0x1F1D, 0x1F20, 0x1F45, 0x1F48, 0x1F4D, + 0x1F50, 0x1F57, 0x1F59, 0x1F59, 0x1F5B, 0x1F5B, 0x1F5D, 0x1F5D, 0x1F5F, 0x1F7D, 0x1F80, 0x1FB4, + 0x1FB6, 0x1FBC, 0x1FBE, 0x1FBE, 0x1FC2, 0x1FC4, 0x1FC6, 0x1FCC, 0x1FD0, 0x1FD3, 0x1FD6, 0x1FDB, + 0x1FE0, 0x1FEC, 0x1FF2, 0x1FF4, 0x1FF6, 0x1FFC, 0x2126, 0x2126, 0x212A, 0x212B, 0x212E, 0x212E, + 0x2180, 0x2182, 0x3007, 0x3007, 0x3021, 0x3029, 0x3041, 0x3094, 0x30A1, 0x30FA, 0x3105, 0x312C, + 0x4E00, 0x9FA5, 0xAC00, 0xD7A3, + ] + + /// Inclusive `[lo, hi]` bounds of the XSD `\c` set: XML 1.0 Appendix B + /// `NameChar` (`Letter | Digit | '.' | '-' | '_' | ':' | CombiningChar | Extender`). + static let nameCharRanges: [UInt32] = [ + 0x2D, 0x2E, 0x30, 0x3A, 0x41, 0x5A, 0x5F, 0x5F, 0x61, 0x7A, 0xB7, 0xB7, + 0xC0, 0xD6, 0xD8, 0xF6, 0xF8, 0x131, 0x134, 0x13E, 0x141, 0x148, 0x14A, 0x17E, + 0x180, 0x1C3, 0x1CD, 0x1F0, 0x1F4, 0x1F5, 0x1FA, 0x217, 0x250, 0x2A8, 0x2BB, 0x2C1, + 0x2D0, 0x2D1, 0x300, 0x345, 0x360, 0x361, 0x386, 0x38A, 0x38C, 0x38C, 0x38E, 0x3A1, + 0x3A3, 0x3CE, 0x3D0, 0x3D6, 0x3DA, 0x3DA, 0x3DC, 0x3DC, 0x3DE, 0x3DE, 0x3E0, 0x3E0, + 0x3E2, 0x3F3, 0x401, 0x40C, 0x40E, 0x44F, 0x451, 0x45C, 0x45E, 0x481, 0x483, 0x486, + 0x490, 0x4C4, 0x4C7, 0x4C8, 0x4CB, 0x4CC, 0x4D0, 0x4EB, 0x4EE, 0x4F5, 0x4F8, 0x4F9, + 0x531, 0x556, 0x559, 0x559, 0x561, 0x586, 0x591, 0x5A1, 0x5A3, 0x5B9, 0x5BB, 0x5BD, + 0x5BF, 0x5BF, 0x5C1, 0x5C2, 0x5C4, 0x5C4, 0x5D0, 0x5EA, 0x5F0, 0x5F2, 0x621, 0x63A, + 0x640, 0x652, 0x660, 0x669, 0x670, 0x6B7, 0x6BA, 0x6BE, 0x6C0, 0x6CE, 0x6D0, 0x6D3, + 0x6D5, 0x6E8, 0x6EA, 0x6ED, 0x6F0, 0x6F9, 0x901, 0x903, 0x905, 0x939, 0x93C, 0x94D, + 0x951, 0x954, 0x958, 0x963, 0x966, 0x96F, 0x981, 0x983, 0x985, 0x98C, 0x98F, 0x990, + 0x993, 0x9A8, 0x9AA, 0x9B0, 0x9B2, 0x9B2, 0x9B6, 0x9B9, 0x9BC, 0x9BC, 0x9BE, 0x9C4, + 0x9C7, 0x9C8, 0x9CB, 0x9CD, 0x9D7, 0x9D7, 0x9DC, 0x9DD, 0x9DF, 0x9E3, 0x9E6, 0x9F1, + 0xA02, 0xA02, 0xA05, 0xA0A, 0xA0F, 0xA10, 0xA13, 0xA28, 0xA2A, 0xA30, 0xA32, 0xA33, + 0xA35, 0xA36, 0xA38, 0xA39, 0xA3C, 0xA3C, 0xA3E, 0xA42, 0xA47, 0xA48, 0xA4B, 0xA4D, + 0xA59, 0xA5C, 0xA5E, 0xA5E, 0xA66, 0xA74, 0xA81, 0xA83, 0xA85, 0xA8B, 0xA8D, 0xA8D, + 0xA8F, 0xA91, 0xA93, 0xAA8, 0xAAA, 0xAB0, 0xAB2, 0xAB3, 0xAB5, 0xAB9, 0xABC, 0xAC5, + 0xAC7, 0xAC9, 0xACB, 0xACD, 0xAE0, 0xAE0, 0xAE6, 0xAEF, 0xB01, 0xB03, 0xB05, 0xB0C, + 0xB0F, 0xB10, 0xB13, 0xB28, 0xB2A, 0xB30, 0xB32, 0xB33, 0xB36, 0xB39, 0xB3C, 0xB43, + 0xB47, 0xB48, 0xB4B, 0xB4D, 0xB56, 0xB57, 0xB5C, 0xB5D, 0xB5F, 0xB61, 0xB66, 0xB6F, + 0xB82, 0xB83, 0xB85, 0xB8A, 0xB8E, 0xB90, 0xB92, 0xB95, 0xB99, 0xB9A, 0xB9C, 0xB9C, + 0xB9E, 0xB9F, 0xBA3, 0xBA4, 0xBA8, 0xBAA, 0xBAE, 0xBB5, 0xBB7, 0xBB9, 0xBBE, 0xBC2, + 0xBC6, 0xBC8, 0xBCA, 0xBCD, 0xBD7, 0xBD7, 0xBE7, 0xBEF, 0xC01, 0xC03, 0xC05, 0xC0C, + 0xC0E, 0xC10, 0xC12, 0xC28, 0xC2A, 0xC33, 0xC35, 0xC39, 0xC3E, 0xC44, 0xC46, 0xC48, + 0xC4A, 0xC4D, 0xC55, 0xC56, 0xC60, 0xC61, 0xC66, 0xC6F, 0xC82, 0xC83, 0xC85, 0xC8C, + 0xC8E, 0xC90, 0xC92, 0xCA8, 0xCAA, 0xCB3, 0xCB5, 0xCB9, 0xCBE, 0xCC4, 0xCC6, 0xCC8, + 0xCCA, 0xCCD, 0xCD5, 0xCD6, 0xCDE, 0xCDE, 0xCE0, 0xCE1, 0xCE6, 0xCEF, 0xD02, 0xD03, + 0xD05, 0xD0C, 0xD0E, 0xD10, 0xD12, 0xD28, 0xD2A, 0xD39, 0xD3E, 0xD43, 0xD46, 0xD48, + 0xD4A, 0xD4D, 0xD57, 0xD57, 0xD60, 0xD61, 0xD66, 0xD6F, 0xE01, 0xE2E, 0xE30, 0xE3A, + 0xE40, 0xE4E, 0xE50, 0xE59, 0xE81, 0xE82, 0xE84, 0xE84, 0xE87, 0xE88, 0xE8A, 0xE8A, + 0xE8D, 0xE8D, 0xE94, 0xE97, 0xE99, 0xE9F, 0xEA1, 0xEA3, 0xEA5, 0xEA5, 0xEA7, 0xEA7, + 0xEAA, 0xEAB, 0xEAD, 0xEAE, 0xEB0, 0xEB9, 0xEBB, 0xEBD, 0xEC0, 0xEC4, 0xEC6, 0xEC6, + 0xEC8, 0xECD, 0xED0, 0xED9, 0xF18, 0xF19, 0xF20, 0xF29, 0xF35, 0xF35, 0xF37, 0xF37, + 0xF39, 0xF39, 0xF3E, 0xF47, 0xF49, 0xF69, 0xF71, 0xF84, 0xF86, 0xF8B, 0xF90, 0xF95, + 0xF97, 0xF97, 0xF99, 0xFAD, 0xFB1, 0xFB7, 0xFB9, 0xFB9, 0x10A0, 0x10C5, 0x10D0, 0x10F6, + 0x1100, 0x1100, 0x1102, 0x1103, 0x1105, 0x1107, 0x1109, 0x1109, 0x110B, 0x110C, 0x110E, 0x1112, + 0x113C, 0x113C, 0x113E, 0x113E, 0x1140, 0x1140, 0x114C, 0x114C, 0x114E, 0x114E, 0x1150, 0x1150, + 0x1154, 0x1155, 0x1159, 0x1159, 0x115F, 0x1161, 0x1163, 0x1163, 0x1165, 0x1165, 0x1167, 0x1167, + 0x1169, 0x1169, 0x116D, 0x116E, 0x1172, 0x1173, 0x1175, 0x1175, 0x119E, 0x119E, 0x11A8, 0x11A8, + 0x11AB, 0x11AB, 0x11AE, 0x11AF, 0x11B7, 0x11B8, 0x11BA, 0x11BA, 0x11BC, 0x11C2, 0x11EB, 0x11EB, + 0x11F0, 0x11F0, 0x11F9, 0x11F9, 0x1E00, 0x1E9B, 0x1EA0, 0x1EF9, 0x1F00, 0x1F15, 0x1F18, 0x1F1D, + 0x1F20, 0x1F45, 0x1F48, 0x1F4D, 0x1F50, 0x1F57, 0x1F59, 0x1F59, 0x1F5B, 0x1F5B, 0x1F5D, 0x1F5D, + 0x1F5F, 0x1F7D, 0x1F80, 0x1FB4, 0x1FB6, 0x1FBC, 0x1FBE, 0x1FBE, 0x1FC2, 0x1FC4, 0x1FC6, 0x1FCC, + 0x1FD0, 0x1FD3, 0x1FD6, 0x1FDB, 0x1FE0, 0x1FEC, 0x1FF2, 0x1FF4, 0x1FF6, 0x1FFC, 0x20D0, 0x20DC, + 0x20E1, 0x20E1, 0x2126, 0x2126, 0x212A, 0x212B, 0x212E, 0x212E, 0x2180, 0x2182, 0x3005, 0x3005, + 0x3007, 0x3007, 0x3021, 0x302F, 0x3031, 0x3035, 0x3041, 0x3094, 0x3099, 0x309A, 0x309D, 0x309E, + 0x30A1, 0x30FA, 0x30FC, 0x30FE, 0x3105, 0x312C, 0x4E00, 0x9FA5, 0xAC00, 0xD7A3, + ] +} diff --git a/Tests/XSDNameCharacterTests.swift b/Tests/XSDNameCharacterTests.swift new file mode 100644 index 0000000..34c39d5 --- /dev/null +++ b/Tests/XSDNameCharacterTests.swift @@ -0,0 +1,65 @@ +import Testing +@testable import PureXML + +/// The XSD `\i`/`\c` regex escapes use the XML 1.0 Second Edition name classes +/// (Appendix B), a fixed Unicode-2.0 repertoire, distinct from the parser's +/// Fifth Edition Name production. These exercise the boundary cases the W3C XSTS +/// `\i`/`\c` tests (reZ005, reZ006) probe and the deliberate divergence from +/// `PureXML.Parsing.XMLCharacter`. +@Suite("XSD \\i / \\c name-character classes (XML 1.0 Appendix B)") +struct XSDNameCharacterTests { + private typealias NameChar = PureXML.Regex.XSDNameCharacter + + private func matches(_ pattern: String, _ scalar: UInt32) throws -> Bool { + try PureXML.Regex.matches(pattern, String(#require(Unicode.Scalar(scalar)))) + } + + @Test("\\c admits the Appendix B CombiningChar range but not the gap above it") + func test_combiningCharBoundary() throws { + // Appendix B CombiningChar begins [#x0300-#x0345]; U+0346 starts the gap. + // XSTS reZ006: U+0345 is a valid \c, U+0346 is not (the failing case). + #expect(NameChar.isNameChar("\u{0345}")) + #expect(!NameChar.isNameChar("\u{0346}")) + #expect(try matches("\\c", 0x0345)) + #expect(try !matches("\\c", 0x0346)) + // \C is the negation. + #expect(try !matches("\\C", 0x0345)) + #expect(try matches("\\C", 0x0346)) + } + + @Test("\\c and \\i deliberately differ from the parser's Fifth Edition name check") + func test_divergesFromParser() { + // U+0346 is a Fifth Edition name character (the parser accepts it), but not + // an XML 1.0 Second Edition one, so XSD \c must reject it. This is the bug + // that wiring \c to the parser caused. + #expect(PureXML.Parsing.XMLCharacter.isNameChar("\u{0346}")) + #expect(!NameChar.isNameChar("\u{0346}")) + } + + @Test("\\i is the name-start class: letters, '_' and ':', not digits or combining marks") + func test_nameStartClass() throws { + #expect(NameChar.isNameStart("a")) + #expect(NameChar.isNameStart("_")) + #expect(NameChar.isNameStart(":")) + // A digit may appear in a Name but may not start one. + #expect(!NameChar.isNameStart("0")) + #expect(NameChar.isNameChar("0")) + // A combining mark is a NameChar but not a name-start. + #expect(!NameChar.isNameStart("\u{0345}")) + #expect(NameChar.isNameChar("\u{0345}")) + // XSTS reZ005 boundary: '[' (U+005B) is neither. + #expect(try !matches("\\i", 0x005B)) + #expect(try matches("\\i", 0x0061)) + } + + @Test("\\c covers the ASCII name punctuation and excludes a clear non-name char") + func test_asciiAndNegative() { + let nameChars: [Unicode.Scalar] = ["A", "z", "0", "9", "-", ".", "_", ":"] + for scalar in nameChars { + #expect(NameChar.isNameChar(scalar), "U+\(String(scalar.value, radix: 16)) is a NameChar") + } + // '+' (U+002B) and ' ' (space) are not NameChars in any edition. + #expect(!NameChar.isNameChar("+")) + #expect(!NameChar.isNameChar(" ")) + } +} diff --git a/Tests/XSTSSuiteTests.swift b/Tests/XSTSSuiteTests.swift index 24e593e..643a481 100644 --- a/Tests/XSTSSuiteTests.swift +++ b/Tests/XSTSSuiteTests.swift @@ -74,7 +74,11 @@ struct XSTSSuiteTests { /// repertoire (reclassified to No later), so `\d` matches them and `\D` does not /// (reT17/reT38 now reject as expected; the matching valid cases reS17/reS38/reZ004 /// no longer need exclusion). See ``XSDCategory`` categoryOverrides. - private let knownInstanceInvalidAccepted = 12 + /// 12 -> 11: `\i`/`\c` use the XML 1.0 Second Edition name classes (Appendix B), + /// not the parser's Fifth Edition production, so a Fifth-Edition-only name char + /// (the combining mark U+0346) no longer matches `\c` (reZ006i now rejects). See + /// ``XSDNameCharacter``. + private let knownInstanceInvalidAccepted = 11 /// Instance tests excluded from the counts. Empty: the Ethiopic-digit `\d` cases /// (reS17.v, reS38.v, reZ004v.v) were not divergent but a 3.2-repertoire bug, now /// fixed, so they are counted as ordinary agreements rather than excluded. diff --git a/docs/xsts-deviations.md b/docs/xsts-deviations.md index d625063..49b54df 100644 --- a/docs/xsts-deviations.md +++ b/docs/xsts-deviations.md @@ -76,17 +76,21 @@ Unicode 3.2, where `U+1369`..`U+1371` are `Nd` (reclassified to `No` later), but `No`. Pinning the nine to `Nd` makes `reT17`/`reT38` reject (`invalid-instances-accepted` 14 -> 12) and the matching `\d`-valid cases `reS17`/`reS38`/`reZ004` pass without their former suspect-test exclusion; this is the complete `Nd` 3.2-vs-now delta, and -`\w` is unaffected. The remaining intricate cases: `particlesZ001`'s rule is marked -ambiguous by the suite itself (RecurseAsIfGroup); `mgO013` is an `xs:all` under -`redefine`; `idZ010` turns on keyref namespace resolution with non-absolute namespace -URIs. The multi-document `sch*` / `targetns` cases need per-schema pairing before a -verdict. +`\w` is unaffected. `reZ006i` (the `\c` case) is also now fixed: `\i`/`\c` were wired +to the parser's XML 1.0 Fifth Edition name check, but XSD defines them against the +Second Edition (Appendix B) classes; the new `XSDNameCharacter` uses the Appendix B +ranges, so the Fifth-Edition-only combining mark `U+0346` no longer matches `\c` +(`invalid-instances-accepted` 12 -> 11). The remaining intricate cases: `particlesZ001`'s +rule is marked ambiguous by the suite itself (RecurseAsIfGroup); `mgO013` is an +`xs:all` under `redefine`; `idZ010` turns on keyref namespace resolution with +non-absolute namespace URIs. The multi-document `sch*` / `targetns` cases need +per-schema pairing before a verdict. Takeaway: on the schema side, libxml2 parity is reached except the one case now fixed; the remaining nine are a deliberate choice to exceed the reference. On the -instance side the Ethiopic-digit `\d`/`\D` cases are now fixed (reference-aligned); -the remaining reference-aligned cases are real but each needs its own analysis, not -a sweep. +instance side the Ethiopic-digit `\d`/`\D` and `\i`/`\c` name-class cases are now +fixed (reference-aligned); the remaining reference-aligned cases are real but each +needs its own analysis, not a sweep. ## Category A: genuine correctness bugs