PureYAML/docs/YAML_TEST_SUITE_CONFORMANCE.md
Mihaela Mihaljevic 16da7248c7 fix(parsing): complete flow-collection key and empty-node handling
Implement libyaml's simple-key mechanism for flow context: the scanner records a potential key position before a plain or quoted scalar, an anchor/tag, or a [/{ opener, and a later ':' splices a mappingKey token retroactively in front of that node. This lets a quoted scalar ("k":v), a nested collection ({k}:v), or a property-carrying node (&c c: d) become a flow mapping key. In flow context ':' is always a value indicator and '?' always an explicit-key indicator regardless of the next character (the block-only following-blank lookahead is disabled), so adjacent forms and trailing '?' parse, and multi-line plain keys fold before the ':' is read. The flow parser is rebuilt around the flow_sequence_entry / flow_mapping_key / flow_mapping_value productions, composing empty scalars for absent keys and values. The streaming token cursor now buffers an open flow collection's tokens until it closes so the retroactive key splice stays valid.

Flips 10 official YAML test suite cases (348/402): 4FJ6, 8KB6, 9BXH, 9MMW, CN3R, DFF7, FRK4, NJ66, UT92, CT4Q. Flow collections used as block mapping keys (LX3P, 6BFJ) need the splice in block context and stay tracked. Updates manifest, fixtures, counts, and roadmap.
2026-06-23 23:18:59 +02:00

20 KiB

YAML Test Suite Conformance

This document tracks PureYAML against the official YAML test suite and lays out the ordered plan for closing the gaps. It is a living document: bucket 1 (block scalars), bucket 2 (quoted-scalar folding), bucket 3 (empty nodes and empty keys), and bucket 4 (anchors, tags, and aliases as mapping keys) have landed; buckets 5-6 remain.

How conformance is measured

The suite's data branch (commit 6ad3d2c62885d82fc349026c136ef560838fdf3d) was run against PureYAML.parseStream + PureYAML.validate, with each case judged by its own oracle:

  • error cases (an error marker): a conformant parser must reject the input.
  • data cases (an in.json): the parsed value must match the expected JSON projection (core-schema resolved), compared structurally.
  • plain valid cases (neither): the input must at least parse without error.

Result at the pinned commit

402 cases (including numbered sub-cases). First measurement was 196 pass / 206 fail; after bucket 1 (block scalars, +24), bucket 2 (quoted-scalar folding, +30), bucket 3 (empty nodes and empty keys, +20), bucket 4 (anchors, tags, and aliases as mapping keys, +10), bucket 5 (directives and document boundaries, +15), bucket 6 (stricter rejection of invalid input, +15), bucket 7 (document-root multi-line plain and block scalars, +33), and bucket 8 (flow-collection completeness, +10) it is now 348 pass, 54 fail. The 54 remaining failing cases are vendored under Tests/Fixtures/yaml-test-suite/ and recorded in Tests/Fixtures/yaml-test-suite-gaps.yaml. Run the opt-in sweep with:

bash scripts/check-yaml-test-suite.sh

The default swift test only checks that the manifest still matches the vendored files; the behavioral sweep is gated behind PUREYAML_RUN_YAML_TEST_SUITE=1, the same way the full real-world corpus is gated. The sweep asserts that each recorded gap still reproduces, so fixing a case fails the sweep until the manifest is updated, the deliberate forcing function for keeping this document honest.

Failure categories (remaining)

Category Count Meaning
false-reject 9 Well-formed YAML that PureYAML wrongly rejects at parse time.
json-mismatch 9 Parses, but produces a value that disagrees with the expected JSON.
accepted-invalid 36 Malformed YAML that PureYAML wrongly accepts.

Root-cause buckets and fix order

Per AGENTS.md, every fix below must start from the matching behavior in the private PureYAMLResearch repository (the libyaml-derived Sources/CYaml/src/ and the Yams Swift layer), not from re-derived intuition. The order is chosen so that structural prerequisites land before the correctness work that depends on them.

1. Block scalar folding and chomping (DONE)

Literal (|) and folded (>) scalar content now follows YAML 1.2 §8.1: interior blank lines are preserved, §8.1.3 folding is applied (single break between equally indented non-empty lines folds to a space; blank lines and more-indented lines keep their breaks), tabs in content are preserved, and the + keep chomping indicator is supported alongside clip and strip. The block scalar is scanned as a single unit with indentation auto-detection, mirroring scanner.c yaml_parser_scan_block_scalar. Implemented in Sources/Parsing/Scanner.BlockScalar.swift.

Flipped 24 cases to passing: 4QFQ, 5LLU, 6VJK, 7T8X, 96NN/00, 96NN/01, DWX9, F6MC, F8F9, H2RW, JEF9/00, JEF9/01, K527, K858, L24T/00, L24T/01, M29M, M9B4, MJS9, P2AD, R4YG, TS54, W9L4, Y79Y/001. (T5N4 looks like a block-scalar case but is blocked by document-start handling, bucket 5.)

2. Quoted-scalar line breaks and empty-line folding (DONE)

Single (') and double (") quoted multi-line scalars now apply YAML 1.2 flow folding (§6.5, §7.3.1, §7.3.2): trailing whitespace before a break is stripped, leading whitespace on a continuation line is stripped, a single break between non-empty lines folds to one space, and N consecutive breaks yield N-1 literal line feeds (the first and last lines keep their own leading and trailing spaces as content). A double quoted backslash at end of line suppresses the break entirely. The double quoted escape set is completed with \xXX and \UXXXXXXXX, plus the previously missing \0 \a \v \e, literal \<space>, and named breaks \N \_ \L \P (the existing \uXXXX surrogate-pair handling is kept). The folding state machine mirrors scanner.c yaml_parser_scan_flow_scalar; implemented in Sources/Parsing/Scalar.swift (foldFlowScalar and the escape decoder).

Flipped 30 cases to passing: 3RLN/00, 3RLN/01, 3RLN/02, 3RLN/03, 3RLN/04, 3RLN/05, 4ZYM, 565N, 5GBF, 6WPF, 7A4E, 8UDB, 9SA2, 9TFX, DE56/00, DE56/01, DE56/02, DE56/03, DE56/04, DE56/05, DK95/02, DK95/08, G4RS, NAT4, NP9H, PRH3, Q8AD, T4YY, TL85, XV9V. (DK95/01 looks similar but is blocked by tab-as-indentation handling, bucket 6.)

3. Empty nodes and empty keys (DONE)

Wherever a node is grammatically required but absent, the parser now composes an empty (null) plain scalar instead of raising "expected a YAML node", mirroring parser.c's yaml_parser_process_empty_scalar fallback. The single production covers every missing-node position: after a tag or anchor property with no following node, an absent mapping value, an absent mapping key (both the plain : value form and the explicit ? form), and the empty key and value forms inside flow collections ({ : }, { , : v }, { k : !!str , }). An empty node that carries an anchor or tag is explicit; a bare absent node with no property in a grammatically-required position is still a parse error, matching the reference parser's "did not find expected node content". The empty scalar resolves through the normal scalar resolver against an empty value, so !!str yields "", !!null and a bare empty yield null, and an empty key resolves like any other empty scalar. Implemented in Sources/Parsing/TokenEventParser.EmptyNode.swift with call sites in TokenEventParser.swift, TokenEventParser.Block.swift, TokenEventParser.Flow.swift, and TokenEventParser.Helpers.swift.

Flipped 20 cases to passing: 2JQS, 2XXW, 5WE3, 6M2F, 7W2P, BU8L, CFD4, JTV5, KK5P, LE5A, M2N8/00, NHX8, NKF9, PW8X, RZP5, S3PD, SM9W/01, UKK6/02, WZ62, XW4D. (CT4Q no longer false-rejects but its flow multi-line plain-scalar key still folds incorrectly, so it moves from false-reject to json-mismatch rather than passing; that fold is a separate flow-scanning gap.)

4. Anchors, tags, and aliases as mapping keys (DONE)

A mapping key is a full node: it may carry &anchor and/or !tag properties and may itself be an *alias. These were rejected with "expected document boundary" / "expected mapping key" / "undefined alias". Four root causes were fixed:

  1. Anchor/alias name termination. The scanner consumed the name up to a whitespace or flow indicator but not :, so *alias: read the alias name as alias:. It now terminates the name at the first character outside libyaml's IS_ALPHA set (ASCII letters, digits, -, _), mirroring yaml_parser_scan_anchor, so the trailing : is a mapping indicator.
  2. Properties/alias in key position. The block-mapping parser only began a pair at the key indicator, but the scanner emits a key's anchor/tag before that indicator (and an alias key has no indicator at all). The parser now begins a pair at leading properties or an alias, and properties on the same line as the first key attach to that key node, not to the mapping (&a a: b anchors the key a; &a alone on its own line anchors the map).
  3. Anchor on an empty value. An anchor or tag at value position with no node following on the same line now composes an empty (null) value that carries the property, so a: &anchor registers anchor as null and a later *anchor resolves to null instead of raising "undefined alias".
  4. Resolved-value registration. An anchored scalar registers its resolved value (an empty anchored scalar registers null, not ""). The implicit-key scanner also now recognizes a key with trailing spaces before its :.
  • Reference: the block + flow mapping productions in parser.c, yaml_parser_scan_anchor for name termination, and Yams' Constructor anchor table for resolution semantics.

Flipped 10 cases to passing: 26DV, 6KGN, 74H7, 87E4, 9KAX, E76Z, G9HC, HMQ5, LQZ7, ZWK4. (G9HC is an invalid-anchor-in-zero-indented- sequence case that is now correctly rejected.) 2SXE no longer false-rejects but its compact block-mapping structure still disagrees with the expected JSON, so it moved from false-reject to json-mismatch rather than passing; that structure is a separate compact-mapping gap.

5. Directives and document boundaries (DONE)

Directives are now handled and the ... document-end marker may be followed by a new document. Three root causes were fixed:

  1. % position detection. A % begins a directive only in directive position: before any document content and before the --- that starts a document (which includes the position immediately after a ... end marker). After content, after ---, or in any flow context, % is ordinary plain text. The scanner previously treated % at column 1 as a directive unconditionally and rejected it once content had begun, mis-firing on XLQ9, UT92, W4TN, 6ZKB, 9DXL. This mirrors scanner.c, which only enters yaml_parser_scan_directive from the start-of-line directive context.
  2. Lenient directive parsing. %YAML accepts a 1.x version (a higher minor is treated as 1.2 with a warning, not an error, so %YAML 1.3 parses) while rejecting an unsupported major version, a duplicate %YAML in one document, or extra non-comment fields. Reserved and unknown directives, and %TAG handle definitions, are accepted and ignored. A directive must be followed by a --- document start. This follows parser.c process_directives (duplicate and version checks) but relaxes the unknown-directive and minor-version errors to the spec's reserved-directive warning behavior.
  3. Document end then new document. A ... end marker closes the current document; content that follows (with or without an explicit ---) starts a fresh document in the multi-document stream parser. The scanner no longer rejects this as an unsupported multi-document stream; the single-document parse still rejects multi-document input at compose time.

Implemented in Sources/Parsing/Scanner.Directives.swift (directive scanning, % position gate, ... boundary), Sources/Parsing/TokenEventParser.swift (a ... end marker returns to start a new document), and Sources/Parsing/TokenCursor.swift (the streaming scanner rejects a pending directive at end of stream). The new directiveWithoutDocument ParseError covers a directive with no following ---.

  • Reference: scanner.c yaml_parser_scan_directive / yaml_parser_scan_version_directive_value + parser.c yaml_parser_process_directives and the document-start state machine.

Flipped 11 cases to passing with matching values: 2LFX, 5TYM, 6WLZ, 6ZKB, 7Z25, 9DXL, 9WXW, BEC7, MUS6/03, MUS6/05, MUS6/06. Correctly rejected four cases that were previously accepted-invalid: 9MMA and B63P (directive without a document), H7TQ (extra words on %YAML), and SF5V (duplicate %YAML). 6LVF and W4TN no longer parse-error but their values still disagree with the expected JSON, so they moved from false-reject to json-mismatch: 6LVF needs same-line content after --- (the KSS4 "scalars on --- line" gap) and W4TN needs literal block-scalar clip chomping to keep the trailing newline (the T5N4/4WA9 literal-scalar gap). XLQ9 and UT92 no longer mis-fire on % but still parse-error on top-level and flow multi-line plain-scalar folding respectively (the 9YRD/EX5H and NJ66 gaps), so they remain recorded. %TAG shorthand resolution (rejecting a handle used in a later document where it is undefined, e.g. QLJ7) is left to the tag-resolution work and those cases stay recorded as accepted-invalid.

6. Stricter rejection of invalid input (PARTIAL)

Inputs that must be rejected were silently accepted: tabs used as indentation, document markers inside quoted scalars or flow collections, an anchor on an alias, and overlapping directive sections. Grounded in the error productions in scanner.c/parser.c, this bucket tightened the scanner and event parser:

  • Tabs as indentation. A tab may never supply block-structure indentation (YAML 1.2 s-indent is spaces only). Sources/Parsing/Scanner.Indentation.swift rejects a tab at the start of a block line; Scanner.BlockScalar.swift rejects a tab where a block-scalar indentation space is expected (scanner.c:2993); and Scanner.Tabs.swift rejects a tab in separation position only when a nested block node follows (a - entry, a ? key, or a compact mapping key), while leaving a tab that merely separates an indicator from a scalar value valid. The IS_BLANKZ boundary fix (scanner.c:1948) also lets :/-/? be recognized when followed by a tab.
  • Document markers in scalars/flow. A ---/... marker at column 0 inside an open quoted scalar (scanner.c:3058) or an open flow collection is rejected in Scanner.swift/Scanner.Directives.swift rather than silently ending the document.
  • Overlapping directive sections. A %YAML/%TAG line that opens a second explicit document with no intervening ... suffix is rejected (Scanner.Directives.swift), matching the grammar's l-document-suffix rule.
  • Anchor on an alias. An alias cannot carry an anchor or tag, and an anchor-plus-alias on one node is rejected (TokenEventParser.swift), mirroring the alias-first branch of yaml_parser_parse_node (parser.c:563).

Rejected (removed from the gap manifest): 4EJS, 5TRB, MUS6/01, N782, RXY3, SR86, SU74, Y79Y/000, Y79Y/004, Y79Y/005, Y79Y/006, Y79Y/008, Y79Y/009. Four of these (9MQT/01 aside) were the bucket-3..5 regressions; MUS6/01, N782, SU74, and Y79Y/009 are repaired here. The tab boundary fix also flipped 6BCT and Y79Y/010 to passing and moved DC7X and J3BT from false-reject to json-mismatch (they now parse but still need trailing-tab scalar trimming and tab-in-quoted-scalar handling to match exactly).

Still tracked as accepted-invalid (deferred to avoid regressing valid input or the default suite):

  • ----line content is dropped by the scanner (skipLine), so cases whose invalidity lives in content after a ---/... on the same line are not yet reachable: 9MQT/01, 2G84/00, 2G84/01, CXX2, and QLJ7 (per-document %TAG handle scoping). Fixing these means scanning ----line content, which touches the document-content path used by many valid cases.

  • Full simple-key / multi-colon tracking is needed for ZCZ6, ZL4Z, 9KBC, and the other "two colons on one line" / wrong-indent mapping cases.

  • Flow trailing-junk (9JBA, CVW2, 62EZ, P2EQ) and assorted scalar edge cases (SU5Z, X4QW, JY7Z, HU3P, DK95/01, Y79Y/003, ...) remain.

  • Reference: the corresponding error productions in scanner.c/parser.c.

  • Why last among the parse work: several of these only become reachable once buckets 1-5 stop short-circuiting; tightening errors earlier would mask the positive-case work.

7. Document-root multi-line plain and block scalars (DONE)

A multi-line plain scalar folded correctly as a mapping value or sequence item but not as the document's own root node, and a block scalar header on the --- line (--- |, --- >) was discarded. Both stemmed from the scanner not modelling libyaml's parser->indent (the enclosing block-collection column, -1 at the bare root). The scanner now tracks that block indent separately from per-line indentation: it rolls on a block sequence entry, an implicit/explicit mapping key, and unrolls at each block-line start and at every document boundary. Plain-scalar continuation folding compares against this block indent, so a root scalar folds its column-0 continuations while a collection member still ends at a sibling line, and a mapping value whose continuation sits at its own column (4CQQ) folds. A --- start marker no longer skips the rest of its line; its trailing content is scanned as the document's root node (--- foo, --- [1, 2], --- |, --- >), and a block scalar whose body sits at the root's column 0 captures that body and ends at the next ---/... marker.

Fixed false-rejects (now parse with matching values): 35KP, 4CQQ, 4Q9F, 6JQW, 82AN, 93WF, 96L6, 9MQT/00, 9YRD, B3HG, DK3J, EX5H, EXG3, FP8R, RZT7, T26H, UGM3, XLQ9. Corrected previously mismatched values: 27NA, 2G84/02, 2G84/03, 4WA9, 6FWR, 6LVF, A2M4, K54U, KSS4, L383, T5N4, W4TN. Now correctly rejected (were accepted-invalid): 9KBC, ZVH3, 9MQT/01. Headline 305 -> 338 pass; categories drop to false-reject 18, json-mismatch 10, accepted-invalid 36.

Two cases now parse but still mismatch and stay tracked as json-mismatch: M7A3 (a bare document followed by ...-delimited empty documents splits into one extra document) and NB6Z (tabs on the empty lines of a multi-line plain scalar must fold to newlines rather than survive as literal tabs).

  • Reference: yaml_parser_scan_plain_scalar and the parser->indent roll/unroll logic (yaml_parser_roll_indent/yaml_parser_unroll_indent), plus block-scalar indent detection in scanner.c.

8. Flow-collection completeness (DONE)

Flow mappings and sequences now accept keys of any node kind, empty keys and values, and multi-line scalar keys, matching libyaml's flow productions. The scanner implements libyaml's simple-key mechanism for flow context: it records a potential key position before scanning a plain/quoted scalar, an anchor/tag, or a [/{ opener (when a simple key is allowed there), and a later : value indicator splices a mappingKey token in front of that node. In flow context a : is always a value indicator and a ? always an explicit key indicator regardless of the following character (libyaml's flow_level || IS_BLANKZ_AT), so adjacent forms ("k":v, {k}:v) and a trailing ? parse; the block-only implicit-key forward lookahead no longer fires in flow. A multi-line plain key folds in full before its : is read. The flow parser was rebuilt around flow_sequence_entry/flow_mapping_key/flow_mapping_value, composing empty scalars where a key or value is absent. The streaming token cursor keeps an open flow collection's tokens buffered until it closes so the retroactive key splice stays valid.

Fixed false-rejects (now parse, values match where an oracle exists): 4FJ6, 8KB6, 9BXH, 9MMW, CN3R, DFF7, FRK4, NJ66, UT92. Corrected the json-mismatch CT4Q (spec example 7.20, single-pair explicit entry) to a matching value. Headline 338 -> 348 pass; categories drop to false-reject 9, json-mismatch 9, accepted-invalid 36.

Still tracked as false-reject: LX3P ([flow]: block) and 6BFJ (&key [ ... ]: value) use a flow collection as a block mapping key, which needs the simple-key splice in block context, a separate change with its own regression surface. The flow-internal 7ZZ5 mismatch is a block-sequence nesting issue (- - - []), not flow, and remains json-mismatch.

  • Reference: yaml_parser_save_simple_key, yaml_parser_fetch_value (KEY-token splice), yaml_parser_fetch_key, the flow-collection start/end and flow-entry fetches in scanner.c, and the yaml_parser_parse_flow_* productions in parser.c.

Updating this document

When a bucket lands:

  1. Re-run bash scripts/check-yaml-test-suite.sh; it will fail on the now-fixed cases.
  2. For each newly-passing case, assert the round-trip law (parse → dump → parse is stable) in addition to the value/JSON match, per core/round-trip-transformation.md; the parser fix is only proven once the emitter agrees on the same input.
  3. Remove those cases from Tests/Fixtures/yaml-test-suite-gaps.yaml and delete their vendored fixtures (or, for json-mismatch cases that now parse but are still being verified, update current).
  4. Update the counts in this document and in CHANGELOG.md.