diff --git a/CHANGELOG.md b/CHANGELOG.md index 3d958ee..1caf78b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Changed + +- Refresh `docs/YAML_TEST_SUITE_CONFORMANCE.md` to match the manifest: the + headline is 402 passing, 0 gap; the "remaining gaps" narrative is retired; a + closing section records the libyaml-port cutover and the final fix series, + and the manifest is named as the source of truth over the prose. + ### Added - The located parse now reports an exact end position for scalars, not just a start: the diff --git a/docs/YAML_TEST_SUITE_CONFORMANCE.md b/docs/YAML_TEST_SUITE_CONFORMANCE.md index 7188a76..b56787f 100644 --- a/docs/YAML_TEST_SUITE_CONFORMANCE.md +++ b/docs/YAML_TEST_SUITE_CONFORMANCE.md @@ -1,10 +1,15 @@ # YAML Test Suite Conformance This document tracks PureYAML against the official -[YAML test suite](https://github.com/yaml/yaml-test-suite) and lays out the -ordered plan for closing the gaps. It is a living document: buckets 1-8 have -landed, and the conformance gate now guards the **entire** suite (all 402 cases), -not just the open gaps, so a fixed case can no longer silently regress. +[YAML test suite](https://github.com/yaml/yaml-test-suite) and chronicles the +ordered plan that closed the gaps. The gap-closing work is complete: **all 402 +cases pass**, and the conformance gate guards the **entire** suite, not just the +formerly open gaps, so a fixed case can no longer silently regress. + +The machine-authoritative record is +`Tests/Fixtures/yaml-test-suite-manifest.yaml`. If this prose and the manifest +ever disagree, the manifest wins; update this document to match it, never the +other way around. ## How conformance is measured @@ -43,8 +48,8 @@ update the manifest). A regression in any passing case now fails the sweep. ### Result at the pinned commit -402 cases (including numbered sub-cases): **329 passing, 73 gap.** Run the opt-in -sweep with: +402 cases (including numbered sub-cases): **402 passing, 0 gap**, per the +manifest header (`total: 402, passing: 402, gap: 0`). Run the opt-in sweep with: ```sh bash scripts/check-yaml-test-suite.sh @@ -82,9 +87,10 @@ bucket and then silently regressed. All were repaired and set back to `passing`: `mappingKey` now uses the same key parser as the implicit form, so a bare tag or anchor before the `:` composes an empty tagged scalar key. -## Failure categories (remaining) +## Failure categories (historical) -The 73 remaining gaps are tracked individually in the manifest. They cluster as: +While gaps were open they were tracked individually in the manifest and +clustered as: | Category | Meaning | |---|---| @@ -92,13 +98,15 @@ The 73 remaining gaps are tracked individually in the manifest. They cluster as: | json-mismatch | Parses, but produces a value that disagrees with the expected JSON. | | accepted-invalid | Malformed YAML that PureYAML wrongly accepts (it should reject). | -Notable still-open gaps include flow-context plain scalars that start with an +The last gaps to close included flow-context plain scalars that start with an indicator (`[:x]` HM87/00, `[?x]` HM87/01, `{x: :x}` 58MP, `{?foo: bar}` 652Z), non-ASCII and special-character anchor/alias names (`&😁` 8XYN, `&an:chor` Y2GN, -W5VH), multi-line flow keys (4MUZ/02, VJP3/01), and a few inputs that should be -rejected but are accepted (DK4H, ZXT5, Y79Y/003, VJP3/00). These were verified to -be genuine never-supported gaps, not regressions, and are tracked with -`status: gap`. +W5VH), multi-line flow keys (4MUZ/02, VJP3/01), and a few inputs that had to be +rejected but were accepted (DK4H, ZXT5, Y79Y/003, VJP3/00). All of these are +`status: passing` in the manifest today; the closing fix series is itemized in +`CHANGELOG.md`, and several of the final rejections (`QB6E`, `DK95/01`, +`Y79Y/003`, `S98Z`, `ZXT5`, `DK4H`) surpass libyaml, which mis-handles those +inputs. ## Root-cause buckets and fix order @@ -259,7 +267,7 @@ gaps), so they remain recorded. `%TAG` shorthand *resolution* (rejecting a handle used in a later document where it is undefined, e.g. `QLJ7`) is left to the tag-resolution work and those cases stay recorded as accepted-invalid. -### 6. Stricter rejection of invalid input (PARTIAL) +### 6. Stricter rejection of invalid input (DONE) Inputs that must be rejected were silently accepted: tabs used as indentation, document markers inside quoted scalars or flow collections, an anchor on an @@ -294,8 +302,8 @@ boundary fix also flipped `6BCT` and `Y79Y/010` to passing and moved `DC7X` and `J3BT` from `false-reject` to `json-mismatch` (they now parse but still need trailing-tab scalar trimming and tab-in-quoted-scalar handling to match exactly). -Still tracked as `accepted-invalid` (deferred to avoid regressing valid input or -the default suite): +Deferred at this stage (to avoid regressing valid input or the default suite), +and closed later by the final series (section 9): - **`---`-line content** is dropped by the scanner (`skipLine`), so cases whose invalidity lives in content after a `---`/`...` on the same line are not yet @@ -337,10 +345,11 @@ Fixed false-rejects (now parse with matching values): `35KP`, `4CQQ`, `4Q9F`, `ZVH3`, `9MQT/01`. Headline 305 -> 338 pass; categories drop to false-reject 18, json-mismatch 10, accepted-invalid 36. -Two cases now parse but still mismatch and stay tracked as `json-mismatch`: -`M7A3` (a bare document followed by `...`-delimited empty documents splits into one -extra document) and `NB6Z` (tabs on the empty lines of a multi-line plain scalar -must fold to newlines rather than survive as literal tabs). +Two cases parsed but still mismatched at this stage and stayed `json-mismatch` +until the final series (section 9): `M7A3` (a bare document followed by +`...`-delimited empty documents splits into one extra document) and `NB6Z` (tabs +on the empty lines of a multi-line plain scalar must fold to newlines rather +than survive as literal tabs). - **Reference:** `yaml_parser_scan_plain_scalar` and the `parser->indent` roll/unroll logic (`yaml_parser_roll_indent`/`yaml_parser_unroll_indent`), plus @@ -370,17 +379,39 @@ Fixed false-rejects (now parse, values match where an oracle exists): `4FJ6`, matching value. Headline 338 -> 348 pass; categories drop to false-reject 9, json-mismatch 9, accepted-invalid 36. -Still tracked as `false-reject`: `LX3P` (`[flow]: block`) and `6BFJ` -(`&key [ ... ]: value`) use a flow collection as a *block* mapping key, which -needs the simple-key splice in block context, a separate change with its own -regression surface. The flow-internal `7ZZ5` mismatch is a block-sequence -nesting issue (`- - - []`), not flow, and remains `json-mismatch`. +Deferred at this stage and closed later by the final series (section 9): +`LX3P` (`[flow]: block`) and `6BFJ` (`&key [ ... ]: value`) use a flow +collection as a *block* mapping key, which needed the simple-key splice in +block context, a separate change with its own regression surface. The +flow-internal `7ZZ5` mismatch was a block-sequence nesting issue (`- - - []`), +not flow, and stayed `json-mismatch` until then. - **Reference:** `yaml_parser_save_simple_key`, `yaml_parser_fetch_value` (KEY-token splice), `yaml_parser_fetch_key`, the flow-collection start/end and flow-entry fetches in `scanner.c`, and the `yaml_parser_parse_flow_*` productions in `parser.c`. +### 9. The libyaml port and the closing series: 348 -> 402 (DONE) + +The buckets above were fixed on the original hand-rolled scanner and token +parser. That engine was then replaced wholesale: a faithful Swift port of +libyaml's scanner and parser landed under `PureYAML.Parsing.LibYAML` +(`docs/LIBYAML_PORT.md`), was verified against libyaml built from source, and +became the sole parsing engine; the hand-rolled `Scanner.*` and +`TokenEventParser.*` files named in buckets 1-8 no longer exist, so those +references are historical. + +On the ported engine, a final fix series closed the remaining 54 cases, +itemized in `CHANGELOG.md`: stricter rejection of under-indented or +tab-indented continuations in quoted scalars, tab-indented flow content, +mis-indented leading empty lines in block scalars, multi-line implicit keys in +flow sequences, block sequences opening mid-line after another node, full +YAML 1.2 anchor and alias names, and the remaining flow-key and `---`-line +content gaps. Several of these rejections +(`QB6E`, `DK95/01`, `Y79Y/003`, `S98Z`, `ZXT5`, `DK4H`) surpass libyaml itself, +which accepts the malformed input. The result is the manifest's current state: +**402 passing, 0 gap**. + ## Updating this document When a fix lands: