NodetteHQ/PureYAML
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

docs(conformance): align the suite narrative with the 402/402 manifest

Browse Source 
The doc still narrated 329 passing with 73 gaps and listed those gaps as
open, contradicting the manifest. Update the headline to 402/0, name the
manifest as the source of truth over the prose, retire the remaining-gaps
narrative, mark bucket 6 done, and add a closing section recording the
libyaml-port cutover and the final fix series (the hand-rolled engine
files cited in buckets 1-8 are historical; the port is the sole engine).
This commit is contained in:
Mihaela Mihaljevic 2026-07-05 00:02:43 +02:00
parent 9b60f415ac
commit e76472da11
2 changed files with 63 additions and 25 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
CHANGELOG.md
View File

@ -7,6 +7,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Changed
- Refresh `docs/YAML_TEST_SUITE_CONFORMANCE.md` to match the manifest: the
headline is 402 passing, 0 gap; the "remaining gaps" narrative is retired; a
closing section records the libyaml-port cutover and the final fix series,
and the manifest is named as the source of truth over the prose.
### Added
- The located parse now reports an exact end position for scalars, not just a start: the

81
docs/YAML_TEST_SUITE_CONFORMANCE.md
View File

@ -1,10 +1,15 @@
# YAML Test Suite Conformance
This document tracks PureYAML against the official
[YAML test suite](https://github.com/yaml/yaml-test-suite) and lays out the
ordered plan for closing the gaps. It is a living document: buckets 1-8 have
landed, and the conformance gate now guards the **entire** suite (all 402 cases),
not just the open gaps, so a fixed case can no longer silently regress.
[YAML test suite](https://github.com/yaml/yaml-test-suite) and chronicles the
ordered plan that closed the gaps. The gap-closing work is complete: **all 402
cases pass**, and the conformance gate guards the **entire** suite, not just the
formerly open gaps, so a fixed case can no longer silently regress.
The machine-authoritative record is
`Tests/Fixtures/yaml-test-suite-manifest.yaml`. If this prose and the manifest
ever disagree, the manifest wins; update this document to match it, never the
other way around.
## How conformance is measured
@ -43,8 +48,8 @@ update the manifest). A regression in any passing case now fails the sweep.
### Result at the pinned commit
402 cases (including numbered sub-cases): **329 passing, 73 gap.** Run the opt-in
sweep with:
402 cases (including numbered sub-cases): **402 passing, 0 gap**, per the
manifest header (`total: 402, passing: 402, gap: 0`). Run the opt-in sweep with:
```sh
bash scripts/check-yaml-test-suite.sh
@ -82,9 +87,10 @@ bucket and then silently regressed. All were repaired and set back to `passing`:
`mappingKey` now uses the same key parser as the implicit form, so a bare tag or
anchor before the `:` composes an empty tagged scalar key.
## Failure categories (remaining)
## Failure categories (historical)
The 73 remaining gaps are tracked individually in the manifest. They cluster as:
While gaps were open they were tracked individually in the manifest and
clustered as:
| Category | Meaning |
|---|---|
@ -92,13 +98,15 @@ The 73 remaining gaps are tracked individually in the manifest. They cluster as:
| json-mismatch | Parses, but produces a value that disagrees with the expected JSON. |
| accepted-invalid | Malformed YAML that PureYAML wrongly accepts (it should reject). |
Notable still-open gaps include flow-context plain scalars that start with an
The last gaps to close included flow-context plain scalars that start with an
indicator (`[:x]` HM87/00, `[?x]` HM87/01, `{x: :x}` 58MP, `{?foo: bar}` 652Z),
non-ASCII and special-character anchor/alias names (`&😁` 8XYN, `&an:chor` Y2GN,
W5VH), multi-line flow keys (4MUZ/02, VJP3/01), and a few inputs that should be
rejected but are accepted (DK4H, ZXT5, Y79Y/003, VJP3/00). These were verified to
be genuine never-supported gaps, not regressions, and are tracked with
`status: gap`.
W5VH), multi-line flow keys (4MUZ/02, VJP3/01), and a few inputs that had to be
rejected but were accepted (DK4H, ZXT5, Y79Y/003, VJP3/00). All of these are
`status: passing` in the manifest today; the closing fix series is itemized in
`CHANGELOG.md`, and several of the final rejections (`QB6E`, `DK95/01`,
`Y79Y/003`, `S98Z`, `ZXT5`, `DK4H`) surpass libyaml, which mis-handles those
inputs.
## Root-cause buckets and fix order
@ -259,7 +267,7 @@ gaps), so they remain recorded. `%TAG` shorthand *resolution* (rejecting a
handle used in a later document where it is undefined, e.g. `QLJ7`) is left to
the tag-resolution work and those cases stay recorded as accepted-invalid.
### 6. Stricter rejection of invalid input (PARTIAL)
### 6. Stricter rejection of invalid input (DONE)
Inputs that must be rejected were silently accepted: tabs used as indentation,
document markers inside quoted scalars or flow collections, an anchor on an
@ -294,8 +302,8 @@ boundary fix also flipped `6BCT` and `Y79Y/010` to passing and moved `DC7X` and
`J3BT` from `false-reject` to `json-mismatch` (they now parse but still need
trailing-tab scalar trimming and tab-in-quoted-scalar handling to match exactly).
Still tracked as `accepted-invalid` (deferred to avoid regressing valid input or
the default suite):
Deferred at this stage (to avoid regressing valid input or the default suite),
and closed later by the final series (section 9):
- **`---`-line content** is dropped by the scanner (`skipLine`), so cases whose
invalidity lives in content after a `---`/`...` on the same line are not yet
@ -337,10 +345,11 @@ Fixed false-rejects (now parse with matching values): `35KP`, `4CQQ`, `4Q9F`,
`ZVH3`, `9MQT/01`. Headline 305 -> 338 pass; categories drop to false-reject 18,
json-mismatch 10, accepted-invalid 36.
Two cases now parse but still mismatch and stay tracked as `json-mismatch`:
`M7A3` (a bare document followed by `...`-delimited empty documents splits into one
extra document) and `NB6Z` (tabs on the empty lines of a multi-line plain scalar
must fold to newlines rather than survive as literal tabs).
Two cases parsed but still mismatched at this stage and stayed `json-mismatch`
until the final series (section 9): `M7A3` (a bare document followed by
`...`-delimited empty documents splits into one extra document) and `NB6Z` (tabs
on the empty lines of a multi-line plain scalar must fold to newlines rather
than survive as literal tabs).
- **Reference:** `yaml_parser_scan_plain_scalar` and the `parser->indent`
roll/unroll logic (`yaml_parser_roll_indent`/`yaml_parser_unroll_indent`), plus
@ -370,17 +379,39 @@ Fixed false-rejects (now parse, values match where an oracle exists): `4FJ6`,
matching value. Headline 338 -> 348 pass; categories drop to false-reject 9,
json-mismatch 9, accepted-invalid 36.
Still tracked as `false-reject`: `LX3P` (`[flow]: block`) and `6BFJ`
(`&key [ ... ]: value`) use a flow collection as a *block* mapping key, which
needs the simple-key splice in block context, a separate change with its own
regression surface. The flow-internal `7ZZ5` mismatch is a block-sequence
nesting issue (`- - - []`), not flow, and remains `json-mismatch`.
Deferred at this stage and closed later by the final series (section 9):
`LX3P` (`[flow]: block`) and `6BFJ` (`&key [ ... ]: value`) use a flow
collection as a *block* mapping key, which needed the simple-key splice in
block context, a separate change with its own regression surface. The
flow-internal `7ZZ5` mismatch was a block-sequence nesting issue (`- - - []`),
not flow, and stayed `json-mismatch` until then.
- **Reference:** `yaml_parser_save_simple_key`, `yaml_parser_fetch_value`
(KEY-token splice), `yaml_parser_fetch_key`, the flow-collection
start/end and flow-entry fetches in `scanner.c`, and the
`yaml_parser_parse_flow_*` productions in `parser.c`.
### 9. The libyaml port and the closing series: 348 -> 402 (DONE)
The buckets above were fixed on the original hand-rolled scanner and token
parser. That engine was then replaced wholesale: a faithful Swift port of
libyaml's scanner and parser landed under `PureYAML.Parsing.LibYAML`
(`docs/LIBYAML_PORT.md`), was verified against libyaml built from source, and
became the sole parsing engine; the hand-rolled `Scanner.*` and
`TokenEventParser.*` files named in buckets 1-8 no longer exist, so those
references are historical.
On the ported engine, a final fix series closed the remaining 54 cases,
itemized in `CHANGELOG.md`: stricter rejection of under-indented or
tab-indented continuations in quoted scalars, tab-indented flow content,
mis-indented leading empty lines in block scalars, multi-line implicit keys in
flow sequences, block sequences opening mid-line after another node, full
YAML 1.2 anchor and alias names, and the remaining flow-key and `---`-line
content gaps. Several of these rejections
(`QB6E`, `DK95/01`, `Y79Y/003`, `S98Z`, `ZXT5`, `DK4H`) surpass libyaml itself,
which accepts the malformed input. The result is the manifest's current state:
**402 passing, 0 gap**.
## Updating this document
When a fix lands: