Three root causes: (1) a '%' begins a directive only in directive position (before document content, outside flow); everywhere else it is plain content, fixing misfires inside plain scalars, flow mappings, and block scalars. (2) %YAML accepts any 1.x (a higher minor is treated as 1.2 without error), rejects major != 1 and duplicate %YAML, and requires a following '---'; reserved and %TAG directives are accepted and ignored. A directive not followed by a document start is rejected with the new directiveWithoutDocument error. (3) An explicit '...' end marker closes the current document so following content begins a fresh document in parseStream; single-document parse still rejects multi-document input at compose time. Tests/UnsupportedYAMLGapsFixtures.swift, StreamParsingTests, ScannerTests, and ParsingErrorTests were updated in lockstep to assert the now-correct behavior (percent-after-content is plain text; '...' then content is two documents; %YAML 1.3 is valid while 2.0 and duplicate %YAML are rejected); no test was weakened and multi-document rejection for single-document parse is still covered. Flips 15 official YAML test suite cases (295/402); SF5V, 9MMA, B63P, H7TQ now correctly rejected. 6LVF and W4TN are reclassified to json-mismatch (blocked by other scalar gaps). Updates the manifest, fixtures, counts, and roadmap.
13 KiB
YAML Test Suite Conformance
This document tracks PureYAML against the official YAML test suite and lays out the ordered plan for closing the gaps. It is a living document: bucket 1 (block scalars), bucket 2 (quoted-scalar folding), bucket 3 (empty nodes and empty keys), and bucket 4 (anchors, tags, and aliases as mapping keys) have landed; buckets 5-6 remain.
How conformance is measured
The suite's data branch (commit 6ad3d2c62885d82fc349026c136ef560838fdf3d) was
run against PureYAML.parseStream + PureYAML.validate, with each case judged by
its own oracle:
- error cases (an
errormarker): a conformant parser must reject the input. - data cases (an
in.json): the parsed value must match the expected JSON projection (core-schema resolved), compared structurally. - plain valid cases (neither): the input must at least parse without error.
Result at the pinned commit
402 cases (including numbered sub-cases). First measurement was 196 pass / 206
fail; after bucket 1 (block scalars, +24), bucket 2 (quoted-scalar folding, +30),
bucket 3 (empty nodes and empty keys, +20), bucket 4 (anchors, tags, and
aliases as mapping keys, +10), and bucket 5 (directives and document boundaries,
+15) it is now 295 pass, 107 fail.
The 107 remaining failing cases are vendored under
Tests/Fixtures/yaml-test-suite/ and recorded in
Tests/Fixtures/yaml-test-suite-gaps.yaml. Run the opt-in sweep with:
bash scripts/check-yaml-test-suite.sh
The default swift test only checks that the manifest still matches the vendored
files; the behavioral sweep is gated behind PUREYAML_RUN_YAML_TEST_SUITE=1, the
same way the full real-world corpus is gated. The sweep asserts that each recorded
gap still reproduces, so fixing a case fails the sweep until the manifest is
updated, the deliberate forcing function for keeping this document honest.
Failure categories (remaining)
| Category | Count | Meaning |
|---|---|---|
false-reject |
40 | Well-formed YAML that PureYAML wrongly rejects at parse time. |
json-mismatch |
20 | Parses, but produces a value that disagrees with the expected JSON. |
accepted-invalid |
47 | Malformed YAML that PureYAML wrongly accepts. |
Root-cause buckets and fix order
Per AGENTS.md, every fix below must start from the matching behavior in the
private PureYAMLResearch repository (the libyaml-derived Sources/CYaml/src/
and the Yams Swift layer), not from re-derived intuition. The order is chosen so
that structural prerequisites land before the correctness work that depends on
them.
1. Block scalar folding and chomping (DONE)
Literal (|) and folded (>) scalar content now follows YAML 1.2 §8.1: interior
blank lines are preserved, §8.1.3 folding is applied (single break between equally
indented non-empty lines folds to a space; blank lines and more-indented lines keep
their breaks), tabs in content are preserved, and the + keep chomping indicator is
supported alongside clip and strip. The block scalar is scanned as a single unit
with indentation auto-detection, mirroring scanner.c
yaml_parser_scan_block_scalar. Implemented in Sources/Parsing/Scanner.BlockScalar.swift.
Flipped 24 cases to passing: 4QFQ, 5LLU, 6VJK, 7T8X, 96NN/00, 96NN/01,
DWX9, F6MC, F8F9, H2RW, JEF9/00, JEF9/01, K527, K858, L24T/00,
L24T/01, M29M, M9B4, MJS9, P2AD, R4YG, TS54, W9L4, Y79Y/001.
(T5N4 looks like a block-scalar case but is blocked by document-start handling,
bucket 5.)
2. Quoted-scalar line breaks and empty-line folding (DONE)
Single (') and double (") quoted multi-line scalars now apply YAML 1.2 flow
folding (§6.5, §7.3.1, §7.3.2): trailing whitespace before a break is stripped,
leading whitespace on a continuation line is stripped, a single break between
non-empty lines folds to one space, and N consecutive breaks yield N-1 literal
line feeds (the first and last lines keep their own leading and trailing spaces as
content). A double quoted backslash at end of line suppresses the break entirely.
The double quoted escape set is completed with \xXX and \UXXXXXXXX, plus the
previously missing \0 \a \v \e, literal \<space>, and named breaks
\N \_ \L \P (the existing \uXXXX surrogate-pair handling is kept). The folding
state machine mirrors scanner.c yaml_parser_scan_flow_scalar; implemented in
Sources/Parsing/Scalar.swift (foldFlowScalar and the escape decoder).
Flipped 30 cases to passing: 3RLN/00, 3RLN/01, 3RLN/02, 3RLN/03,
3RLN/04, 3RLN/05, 4ZYM, 565N, 5GBF, 6WPF, 7A4E, 8UDB, 9SA2,
9TFX, DE56/00, DE56/01, DE56/02, DE56/03, DE56/04, DE56/05,
DK95/02, DK95/08, G4RS, NAT4, NP9H, PRH3, Q8AD, T4YY, TL85,
XV9V. (DK95/01 looks similar but is blocked by tab-as-indentation handling,
bucket 6.)
3. Empty nodes and empty keys (DONE)
Wherever a node is grammatically required but absent, the parser now composes an
empty (null) plain scalar instead of raising "expected a YAML node", mirroring
parser.c's yaml_parser_process_empty_scalar fallback. The single production
covers every missing-node position: after a tag or anchor property with no
following node, an absent mapping value, an absent mapping key (both the plain
: value form and the explicit ? form), and the empty key and value forms
inside flow collections ({ : }, { , : v }, { k : !!str , }). An empty node
that carries an anchor or tag is explicit; a bare absent node with no property in
a grammatically-required position is still a parse error, matching the reference
parser's "did not find expected node content". The empty scalar resolves through
the normal scalar resolver against an empty value, so !!str yields "",
!!null and a bare empty yield null, and an empty key resolves like any other
empty scalar. Implemented in Sources/Parsing/TokenEventParser.EmptyNode.swift
with call sites in TokenEventParser.swift, TokenEventParser.Block.swift,
TokenEventParser.Flow.swift, and TokenEventParser.Helpers.swift.
Flipped 20 cases to passing: 2JQS, 2XXW, 5WE3, 6M2F, 7W2P, BU8L,
CFD4, JTV5, KK5P, LE5A, M2N8/00, NHX8, NKF9, PW8X, RZP5, S3PD,
SM9W/01, UKK6/02, WZ62, XW4D. (CT4Q no longer false-rejects but its
flow multi-line plain-scalar key still folds incorrectly, so it moves from
false-reject to json-mismatch rather than passing; that fold is a separate
flow-scanning gap.)
4. Anchors, tags, and aliases as mapping keys (DONE)
A mapping key is a full node: it may carry &anchor and/or !tag properties and
may itself be an *alias. These were rejected with "expected document boundary"
/ "expected mapping key" / "undefined alias". Four root causes were fixed:
- Anchor/alias name termination. The scanner consumed the name up to a
whitespace or flow indicator but not
:, so*alias:read the alias name asalias:. It now terminates the name at the first character outside libyaml'sIS_ALPHAset (ASCII letters, digits,-,_), mirroringyaml_parser_scan_anchor, so the trailing:is a mapping indicator. - Properties/alias in key position. The block-mapping parser only began a
pair at the key indicator, but the scanner emits a key's anchor/tag before
that indicator (and an alias key has no indicator at all). The parser now
begins a pair at leading properties or an alias, and properties on the same
line as the first key attach to that key node, not to the mapping (
&a a: banchors the keya;&aalone on its own line anchors the map). - Anchor on an empty value. An anchor or tag at value position with no node
following on the same line now composes an empty (null) value that carries the
property, so
a: &anchorregistersanchoras null and a later*anchorresolves to null instead of raising "undefined alias". - Resolved-value registration. An anchored scalar registers its resolved
value (an empty anchored scalar registers null, not
""). The implicit-key scanner also now recognizes a key with trailing spaces before its:.
- Reference: the block + flow mapping productions in
parser.c,yaml_parser_scan_anchorfor name termination, and Yams'Constructoranchor table for resolution semantics.
Flipped 10 cases to passing: 26DV, 6KGN, 74H7, 87E4, 9KAX, E76Z,
G9HC, HMQ5, LQZ7, ZWK4. (G9HC is an invalid-anchor-in-zero-indented-
sequence case that is now correctly rejected.) 2SXE no longer false-rejects
but its compact block-mapping structure still disagrees with the expected JSON,
so it moved from false-reject to json-mismatch rather than passing; that
structure is a separate compact-mapping gap.
5. Directives and document boundaries (DONE)
Directives are now handled and the ... document-end marker may be followed by a
new document. Three root causes were fixed:
%position detection. A%begins a directive only in directive position: before any document content and before the---that starts a document (which includes the position immediately after a...end marker). After content, after---, or in any flow context,%is ordinary plain text. The scanner previously treated%at column 1 as a directive unconditionally and rejected it once content had begun, mis-firing onXLQ9,UT92,W4TN,6ZKB,9DXL. This mirrorsscanner.c, which only entersyaml_parser_scan_directivefrom the start-of-line directive context.- Lenient directive parsing.
%YAMLaccepts a1.xversion (a higher minor is treated as 1.2 with a warning, not an error, so%YAML 1.3parses) while rejecting an unsupported major version, a duplicate%YAMLin one document, or extra non-comment fields. Reserved and unknown directives, and%TAGhandle definitions, are accepted and ignored. A directive must be followed by a---document start. This followsparser.cprocess_directives(duplicate and version checks) but relaxes the unknown-directive and minor-version errors to the spec's reserved-directive warning behavior. - Document end then new document. A
...end marker closes the current document; content that follows (with or without an explicit---) starts a fresh document in the multi-document stream parser. The scanner no longer rejects this as an unsupported multi-document stream; the single-documentparsestill rejects multi-document input at compose time.
Implemented in Sources/Parsing/Scanner.Directives.swift (directive scanning,
% position gate, ... boundary), Sources/Parsing/TokenEventParser.swift
(a ... end marker returns to start a new document), and
Sources/Parsing/TokenCursor.swift (the streaming scanner rejects a pending
directive at end of stream). The new directiveWithoutDocument ParseError
covers a directive with no following ---.
- Reference:
scanner.cyaml_parser_scan_directive/yaml_parser_scan_version_directive_value+parser.cyaml_parser_process_directivesand the document-start state machine.
Flipped 11 cases to passing with matching values: 2LFX, 5TYM, 6WLZ,
6ZKB, 7Z25, 9DXL, 9WXW, BEC7, MUS6/03, MUS6/05, MUS6/06.
Correctly rejected four cases that were previously accepted-invalid: 9MMA
and B63P (directive without a document), H7TQ (extra words on %YAML), and
SF5V (duplicate %YAML). 6LVF and W4TN no longer parse-error but their
values still disagree with the expected JSON, so they moved from false-reject
to json-mismatch: 6LVF needs same-line content after --- (the KSS4
"scalars on --- line" gap) and W4TN needs literal block-scalar clip chomping
to keep the trailing newline (the T5N4/4WA9 literal-scalar gap). XLQ9 and
UT92 no longer mis-fire on % but still parse-error on top-level and flow
multi-line plain-scalar folding respectively (the 9YRD/EX5H and NJ66
gaps), so they remain recorded. %TAG shorthand resolution (rejecting a
handle used in a later document where it is undefined, e.g. QLJ7) is left to
the tag-resolution work and those cases stay recorded as accepted-invalid.
6. Stricter rejection of invalid input (47 cases, accepted-invalid)
Inputs that must be rejected are silently accepted: tabs used as indentation,
bad document-end markers inside quoted scalars, malformed flow collections, and
%TAG shorthands used in a document that does not define them. Examples:
QB6E, RXY3, ZVH3, QLJ7, the Y79Y tab cases. (The directive-shape
rejections, duplicate %YAML SF5V and extra-words H7TQ, landed with bucket 5.)
- Reference: the corresponding error productions in
scanner.c/parser.c. - Why last among the parse work: several of these only become reachable once buckets 1-5 stop short-circuiting; tightening errors earlier would mask the positive-case work.
Updating this document
When a bucket lands:
- Re-run
bash scripts/check-yaml-test-suite.sh; it will fail on the now-fixed cases. - For each newly-passing case, assert the round-trip law (parse → dump → parse is
stable) in addition to the value/JSON match, per
core/round-trip-transformation.md; the parser fix is only proven once the emitter agrees on the same input. - Remove those cases from
Tests/Fixtures/yaml-test-suite-gaps.yamland delete their vendored fixtures (or, forjson-mismatchcases that now parse but are still being verified, updatecurrent). - Update the counts in this document and in
CHANGELOG.md.