NodetteHQ/PureXML
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases 1 Wiki Activity

fix(schema): match Ethiopic digits against \d at the pinned Unicode 3.2 repertoire (#377)

Browse Source 
XSD 1.0 fixes the regular-expression repertoire at Unicode 3.2 (Datatypes
Appendix F), where U+1369..U+1371 (ETHIOPIC DIGIT ONE..NINE) are general
category Nd, each carrying a decimal value; a later Unicode reclassified them
to No. XSDCategory derives \d (== \p{Nd}) from the running platform's category,
pinned to 3.2 by an assigned-in-3.2 gate plus a per-code-point override table,
but the table omitted these nine, so \d used the modern No: a valid \d-constrained
Ethiopic digit was wrongly rejected (cardinal-rule false positive) and a
\D-constrained one wrongly accepted.

Pin the nine to Nd. This is the complete set of code points that were Nd at 3.2
and are not today; the reverse direction is empty (every later-Nd point was
unassigned at 3.2 and already reports Cn), verified by diffing the 3.2 and current
Unicode Nd sets. \w is unaffected: an Ethiopic numeral is category N either way,
neither punctuation, separator, nor other.

Verified over the full W3C XSTS corpus in release: invalid-instances-accepted
14 -> 12 (reT17, reT38, the \D cases, now reject), and the matching \d-valid cases
reS17/reS38/reZ004 no longer need a suspect-test exclusion and pass as ordinary
agreements, with both false-positive buckets at 0. Full suite (1830 tests) green;
an adversarial over-rejection critic re-derived the Nd delta bidirectionally and
confirmed no false-positive surface. A regression-pinning test exercises the
production regex path, including the boundary that U+1372 (No even at 3.2) is
not \d.
This commit is contained in:
Mihaela Mihaljevic 2026-06-29 06:27:27 +02:00 committed by GitHub
parent 7be7fa25f3
commit 1919b92956
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
5 changed files with 58 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
CHANGELOG.md
View File

@ -31,6 +31,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- The Ethiopic digits `U+1369`..`U+1371` (`ETHIOPIC DIGIT ONE`..`NINE`) now match `\d` in an XSD pattern and not `\D`. XSD 1.0 fixes the regular-expression repertoire at Unicode 3.2 (Datatypes Appendix F), where these nine code points are general category `Nd` and each carries a decimal value; a later Unicode reclassified them to `No`. `XSDCategory` derives `\d` (`== \p{Nd}`) from the running platform's category, pinned to 3.2 by an assigned-in-3.2 gate plus a per-code-point override table, but the override table was missing these nine, so `\d` used the modern `No` and rejected them. This was a two-directional bug: a valid instance whose `\d`-constrained value was an Ethiopic digit was wrongly rejected (the cardinal rule: never reject a valid value), and a `\D`-constrained value of one was wrongly accepted. They are now pinned to `Nd`, which is the complete set of code points that were `Nd` at 3.2 and are not today (the reverse direction is empty: every code point that became `Nd` later was unassigned at 3.2 and so already reports `Cn`, verified by diffing the 3.2 and current Unicode `Nd` sets), so `\w` is unaffected (an Ethiopic numeral is category `N` either way, neither punctuation, separator, nor other). Verified over the full W3C XSTS corpus in release: `invalid-instances-accepted` 14 -> 12 (`reT17`, `reT38`, the `\D` cases, now reject), the matching `\d` valid cases (`reS17`, `reS38`, `reZ004`) no longer need their suspect-test exclusion and pass as ordinary agreements, and the two false-positive buckets stay 0. Found while triaging the invalid-instance tail against libxml2 (which rejects `reT17`/`reT38`, the reference-aligned reading) back to the authoritative `UnicodeData-3.2.0`; a prior note that had treated these as a disputed spec-divergence was the wrong call and is corrected.
- A value of `NaN` is no longer wrongly rejected by an `xs:double` or `xs:float` `enumeration` that lists `NaN`. Value-space equality (`OrderedValue.==`) compared with IEEE equality, under which `NaN != NaN`, so a `NaN` value never matched a `NaN` enumeration member and was rejected, a false positive (the cardinal rule: never reject a valid value). The comparison now treats `NaN` as identical to itself, the value-space identity the reference (libxml2) and XSD 1.1 use throughout, so `<v>NaN</v>` validates against a type restricting `xs:double` by `<xs:enumeration value="NaN"/>`; `INF`/`-INF` already matched. This equality also backs the `key`/`unique` identity constraints and `fixed`, where the same identity holds (verified against `xmllint`): two `NaN`-valued unique fields collide as duplicates, and `fixed="NaN"` matches `NaN`. The ordering facets (`minInclusive` and friends) use the order relation, in which `NaN` stays incomparable, and are unaffected. Found by the new generated-value facet differential against libxml2, with the key/fixed behaviour confirmed after an adversarial critic flagged the shared comparison; the four W3C XSTS buckets held and the full suite passes, with regressions in `SchemaDoubleEnumerationTests`.
- A long `<xs:attributeGroup ref>` chain (`ag0` referencing `ag1` referencing `ag2`, and so on) no longer overflows the stack or hangs the compiler. Such a chain is flat in the schema document, so flattening a complex type's attributes through it recursed once per link, overflowing the stack past about six thousand links and running near-cubic before that (about 17 seconds at three thousand links). The attribute-group reference depth is now bounded by `maximumContentModelExpansionDepth` (512, the same bound the content-model expansion uses), since the cycle-guard `visited` set already counts the chain depth: a chain past the bound is rejected with a located error (`attribute group reference nesting exceeds the maximum depth of 512`) rather than crashed. Behaviour-preserving for every schema within the bound, which no realistic schema or the XSTS corpus approaches; verified conformance-neutral over the full W3C XSTS corpus in release (all four buckets unchanged) and by the full suite. Disclosed residual: a schema with very many distinct over-bound attribute-group chains still rejects in time quadratic in the bounded depth per group, because the attribute flattening is re-done per group; memoizing it, as the `simpleContent` resolution now is, is the linear follow-up.

7
Sources/Regex/XSDCategory.swift
View File

@ -20,6 +20,13 @@ extension PureXML.Regex {
0x0F14: "So",
0x166D: "Po", 0x17D7: "Po", 0x17DC: "Po",
0x180B: "Cf", 0x180C: "Cf", 0x180D: "Cf",
// Ethiopic digits ONE..NINE were Nd at Unicode 3.2 (each carried a
// decimal value); a later Unicode reclassified them to No. XSD `\d`
// is `\p{Nd}` over the pinned 3.2 repertoire, so they must match `\d`
// (and not `\D`). This is the complete set of code points that were
// Nd at 3.2 and are not Nd today (XSTS reS17/reS38/reT17/reT38/reZ004).
0x1369: "Nd", 0x136A: "Nd", 0x136B: "Nd", 0x136C: "Nd", 0x136D: "Nd",
0x136E: "Nd", 0x136F: "Nd", 0x1370: "Nd", 0x1371: "Nd",
0x2308: "Sm", 0x2309: "Sm", 0x230A: "Sm", 0x230B: "Sm",
]

24
Tests/RegexCategoryTests.swift
View File

@ -25,6 +25,30 @@ struct RegexCategoryTests {
#expect(try !matches("\\p{Nd}+", "12a"))
}
@Test("Ethiopic digits are Nd at the pinned Unicode 3.2 repertoire")
func test_ethiopicDigitsAreNd() throws {
// U+1369..U+1371 (ETHIOPIC DIGIT ONE..NINE) were Nd at Unicode 3.2 (each
// had a decimal value) and were reclassified to No later. XSD pins the
// repertoire at 3.2, so `\d` (== \p{Nd}) must match them and `\D` must not
// (XSTS reS17/reS38 valid, reT17/reT38 invalid; reZ004 the full \d set).
for value: UInt32 in 0x1369 ... 0x1371 {
let text = try String(#require(Unicode.Scalar(value)))
let matchesDigit = try matches("\\d", text)
let matchesNd = try matches("\\p{Nd}", text)
let matchesNonDigit = try matches("\\D", text)
let matchesNo = try matches("\\p{No}", text)
#expect(matchesDigit, "U+\(String(value, radix: 16)) must match \\d")
#expect(matchesNd)
#expect(!matchesNonDigit)
#expect(!matchesNo)
}
// The override is scoped to the unit digits: U+1372 (ETHIOPIC NUMBER TEN)
// was No even at 3.2, so it is not a `\d` and is a `\p{No}`.
let ten = "\u{1372}"
#expect(try !matches("\\d", ten))
#expect(try matches("\\p{No}", ten))
}
@Test("\\P{...} negates a category")
func test_negatedCategory() throws {
#expect(try matches("\\P{L}+", "123 !"))

13
Tests/XSTSSuiteTests.swift
View File

@ -70,10 +70,15 @@ struct XSTSSuiteTests {
/// one of its members by a blocked derivation method is rejected (cvc-elt.4.3
/// / cos-st-derived-OK 2.2.4, elemT074: `block="restriction"` on the element,
/// the substitute restricting a union member).
private let knownInstanceInvalidAccepted = 14
/// Suspect instance tests excluded from the counts: the Ethiopic-digit `\d`
/// cases contradict `\d` = `\p{Nd}` at Unicode 3.2 and are tracked as disputed.
private static let specDivergentInstances: Set<String> = ["reS17.v", "reS38.v", "reZ004v.v"]
/// 14 -> 12: the Ethiopic digits U+1369..U+1371 are Nd at the pinned Unicode 3.2
/// repertoire (reclassified to No later), so `\d` matches them and `\D` does not
/// (reT17/reT38 now reject as expected; the matching valid cases reS17/reS38/reZ004
/// no longer need exclusion). See ``XSDCategory`` categoryOverrides.
private let knownInstanceInvalidAccepted = 12
/// Instance tests excluded from the counts. Empty: the Ethiopic-digit `\d` cases
/// (reS17.v, reS38.v, reZ004v.v) were not divergent but a 3.2-repertoire bug, now
/// fixed, so they are counted as ordinary agreements rather than excluded.
private static let specDivergentInstances: Set<String> = []
@Test("Every XSTS case behaves: compile, reject, validate, invalidate")
func test_suite() throws {

24
docs/xsts-deviations.md
View File

@ -69,16 +69,24 @@ Schema side (10 cases):
| `schN12` | (multi-document) | Cross-namespace composition; not triaged in isolation. |
Instance side (sampled): `idZ010`, `mgO013`, `particlesZ001`, `reT17`, `reT38`
are rejected by libxml2 (reference-aligned), but each is individually intricate:
`particlesZ001`'s rule is marked ambiguous by the suite itself (RecurseAsIfGroup);
`mgO013` is an `xs:all` under `redefine`; `idZ010` turns on keyref namespace
resolution with non-absolute namespace URIs. The multi-document `sch*` / `targetns`
cases need per-schema pairing before a verdict.
are rejected by libxml2 (reference-aligned). `reT17`/`reT38` are now fixed: they
are the `\D` half of the Ethiopic-digit `\d` bug. XSD pins the regex repertoire at
Unicode 3.2, where `U+1369`..`U+1371` are `Nd` (reclassified to `No` later), but
`XSDCategory`'s per-code-point override table omitted them, so `\d` used the modern
`No`. Pinning the nine to `Nd` makes `reT17`/`reT38` reject (`invalid-instances-accepted`
14 -> 12) and the matching `\d`-valid cases `reS17`/`reS38`/`reZ004` pass without
their former suspect-test exclusion; this is the complete `Nd` 3.2-vs-now delta, and
`\w` is unaffected. The remaining intricate cases: `particlesZ001`'s rule is marked
ambiguous by the suite itself (RecurseAsIfGroup); `mgO013` is an `xs:all` under
`redefine`; `idZ010` turns on keyref namespace resolution with non-absolute namespace
URIs. The multi-document `sch*` / `targetns` cases need per-schema pairing before a
verdict.
Takeaway: on the schema side, libxml2 parity is reached except the one case now
fixed; the remaining nine are a deliberate choice to exceed the reference. The
instance tail's reference-aligned cases are real but each needs its own analysis,
not a sweep.
fixed; the remaining nine are a deliberate choice to exceed the reference. On the
instance side the Ethiopic-digit `\d`/`\D` cases are now fixed (reference-aligned);
the remaining reference-aligned cases are real but each needs its own analysis, not
a sweep.
## Category A: genuine correctness bugs