PureYAML/docs/YAML_TEST_SUITE_CONFORMANCE.md

YAML Test Suite Conformance

This document tracks PureYAML against the official YAML test suite and chronicles the ordered plan that closed the gaps. The gap-closing work is complete: all 402 cases pass, and the conformance gate guards the entire suite, not just the formerly open gaps, so a fixed case can no longer silently regress.

The machine-authoritative record is Tests/Fixtures/yaml-test-suite-manifest.yaml. If this prose and the manifest ever disagree, the manifest wins; update this document to match it, never the other way around.

How conformance is measured

The suite's data branch (commit 6ad3d2c62885d82fc349026c136ef560838fdf3d) is run against PureYAML.parseStream, with each case judged by its own oracle:

• error cases (an error marker): a conformant parser must reject the input.
• data cases (an in.json): the parsed value must match the expected JSON projection (core-schema resolved), compared structurally. An empty in.json means the conformant output is an empty stream (no document).
• plain valid cases (neither): the input must parse without error.

Conformance is decided by parsing alone. Strict validation (for example duplicate-key detection in PureYAML.validate) is a separate, orthogonal concern and is deliberately not part of the conformance decision, so a parseable document that strict mode would reject is still counted as a passing parse.

The full-suite gate

Earlier revisions tracked only the failing cases in a gap-only manifest and deleted a case from it once fixed. That let later work silently regress an already-fixed case, because nothing guarded it any longer. The gate is now comprehensive: Tests/Fixtures/yaml-test-suite-manifest.yaml records all 402 cases (numbered sub-cases counted), each with its id, name, localPath, the conformant expected behavior (parse-ok | reject), its oracle (json | none), the input byteCount and lineCount, and a status:

• passing: PureYAML handles the case conformantly today.
• gap: PureYAML does not yet handle it conformantly.

Every case's files (in.yaml, plus in.json and/or error) are vendored under Tests/Fixtures/yaml-test-suite/<id>/. The gated behavioral sweep asserts, for every case, that a passing case still passes (the regression guard) and a gap case still reproduces its non-conformant behavior (so a fix is forced to update the manifest). A regression in any passing case now fails the sweep.

Result at the pinned commit

402 cases (including numbered sub-cases): 402 passing, 0 gap, per the manifest header (total: 402, passing: 402, gap: 0). Run the opt-in sweep with:

bash scripts/check-yaml-test-suite.sh

The default swift test only checks manifest/fixture/byte/line integrity and that the recorded counts are consistent; the behavioral sweep is gated behind PUREYAML_RUN_YAML_TEST_SUITE=1, the same way the full real-world corpus is gated.

Regressions repaired by the full-suite gate

Making the gate comprehensive surfaced cases that had been fixed in an earlier bucket and then silently regressed. All were repaired and set back to passing:

• Tab over-rejection. A tab was being rejected wherever it appeared at a block line start, over-rejecting valid forms. Per scanner.c (scan_to_next_token, scan_block_scalar_breaks, scan_plain_scalar), a tab is illegal only as block-structure indentation. Block indentation is now spaces-only; a tab is left for the token scanner, which allows it (a) inside block-scalar content once the content column is established by spaces (96NN/00, 96NN/01, R4YG, Y79Y/001), (b) as separation before a scalar value (DK95/00) or a flow node (6CA3, Q5MG, J3BT), and (c) on a whitespace-only blank line (DK95/03, DK95/07), while still rejecting genuine tab indentation (4EJS, Y79Y/000, Y79Y/004-/008, DK95/06). The batch scanner and the incremental token cursor were unified onto a single scanStep, since the divergence between them is what let the streaming parser miss a tab error the batch scanner caught.
• 26DV (anchors/aliases as mapping keys). An anchor on a key: value line followed by an indented *alias : key was throwing, because the anchor was being pushed onto the alias key (which cannot carry properties) instead of the nested mapping. The alias-as-key path now attaches a property scanned on a prior line to the enclosing mapping, and only rejects an alias that carries a property on its own line (&b *alias, still correctly rejected by SU74).
• WZ62 (flow rework). { foo : !!str, !!str : bar, } mis-parsed the second entry's bare-tag key into a mapping-as-key. A flow key introduced by a spliced mappingKey now uses the same key parser as the implicit form, so a bare tag or anchor before the : composes an empty tagged scalar key.

Failure categories (historical)

While gaps were open they were tracked individually in the manifest and clustered as:

Category	Meaning
false-reject	Well-formed YAML that PureYAML wrongly rejects at parse time.
json-mismatch	Parses, but produces a value that disagrees with the expected JSON.
accepted-invalid	Malformed YAML that PureYAML wrongly accepts (it should reject).

The last gaps to close included flow-context plain scalars that start with an indicator ([:x] HM87/00, [?x] HM87/01, {x: :x} 58MP, {?foo: bar} 652Z), non-ASCII and special-character anchor/alias names (&😁 8XYN, &an:chor Y2GN, W5VH), multi-line flow keys (4MUZ/02, VJP3/01), and a few inputs that had to be rejected but were accepted (DK4H, ZXT5, Y79Y/003, VJP3/00). All of these are status: passing in the manifest today; the closing fix series is itemized in CHANGELOG.md, and several of the final rejections (QB6E, DK95/01, Y79Y/003, S98Z, ZXT5, DK4H) surpass libyaml, which mis-handles those inputs.

Root-cause buckets and fix order

Per AGENTS.md, every fix below must start from the matching behavior in the private PureYAMLResearch repository (the libyaml-derived Sources/CYaml/src/ and the Yams Swift layer), not from re-derived intuition. The order is chosen so that structural prerequisites land before the correctness work that depends on them.

1. Block scalar folding and chomping (DONE)

Literal (|) and folded (>) scalar content now follows YAML 1.2 §8.1: interior blank lines are preserved, §8.1.3 folding is applied (single break between equally indented non-empty lines folds to a space; blank lines and more-indented lines keep their breaks), tabs in content are preserved, and the + keep chomping indicator is supported alongside clip and strip. The block scalar is scanned as a single unit with indentation auto-detection, mirroring scanner.c yaml_parser_scan_block_scalar. Implemented in Sources/Parsing/Scanner.BlockScalar.swift.

Flipped 24 cases to passing: 4QFQ, 5LLU, 6VJK, 7T8X, 96NN/00, 96NN/01, DWX9, F6MC, F8F9, H2RW, JEF9/00, JEF9/01, K527, K858, L24T/00, L24T/01, M29M, M9B4, MJS9, P2AD, R4YG, TS54, W9L4, Y79Y/001. (T5N4 looks like a block-scalar case but is blocked by document-start handling, bucket 5.)

2. Quoted-scalar line breaks and empty-line folding (DONE)

Single (') and double (") quoted multi-line scalars now apply YAML 1.2 flow folding (§6.5, §7.3.1, §7.3.2): trailing whitespace before a break is stripped, leading whitespace on a continuation line is stripped, a single break between non-empty lines folds to one space, and N consecutive breaks yield N-1 literal line feeds (the first and last lines keep their own leading and trailing spaces as content). A double quoted backslash at end of line suppresses the break entirely. The double quoted escape set is completed with \xXX and \UXXXXXXXX, plus the previously missing \0 \a \v \e, literal \<space>, and named breaks \N \_ \L \P (the existing \uXXXX surrogate-pair handling is kept). The folding state machine mirrors scanner.c yaml_parser_scan_flow_scalar; implemented in Sources/Parsing/Scalar.swift (foldFlowScalar and the escape decoder).

Flipped 30 cases to passing: 3RLN/00, 3RLN/01, 3RLN/02, 3RLN/03, 3RLN/04, 3RLN/05, 4ZYM, 565N, 5GBF, 6WPF, 7A4E, 8UDB, 9SA2, 9TFX, DE56/00, DE56/01, DE56/02, DE56/03, DE56/04, DE56/05, DK95/02, DK95/08, G4RS, NAT4, NP9H, PRH3, Q8AD, T4YY, TL85, XV9V. (DK95/01 looks similar but is blocked by tab-as-indentation handling, bucket 6.)

3. Empty nodes and empty keys (DONE)

Wherever a node is grammatically required but absent, the parser now composes an empty (null) plain scalar instead of raising "expected a YAML node", mirroring parser.c's yaml_parser_process_empty_scalar fallback. The single production covers every missing-node position: after a tag or anchor property with no following node, an absent mapping value, an absent mapping key (both the plain : value form and the explicit ? form), and the empty key and value forms inside flow collections ({ : }, { , : v }, { k : !!str , }). An empty node that carries an anchor or tag is explicit; a bare absent node with no property in a grammatically-required position is still a parse error, matching the reference parser's "did not find expected node content". The empty scalar resolves through the normal scalar resolver against an empty value, so !!str yields "", !!null and a bare empty yield null, and an empty key resolves like any other empty scalar. Implemented in Sources/Parsing/TokenEventParser.EmptyNode.swift with call sites in TokenEventParser.swift, TokenEventParser.Block.swift, TokenEventParser.Flow.swift, and TokenEventParser.Helpers.swift.

Flipped 20 cases to passing: 2JQS, 2XXW, 5WE3, 6M2F, 7W2P, BU8L, CFD4, JTV5, KK5P, LE5A, M2N8/00, NHX8, NKF9, PW8X, RZP5, S3PD, SM9W/01, UKK6/02, WZ62, XW4D. (CT4Q no longer false-rejects but its flow multi-line plain-scalar key still folds incorrectly, so it moves from false-reject to json-mismatch rather than passing; that fold is a separate flow-scanning gap.)

4. Anchors, tags, and aliases as mapping keys (DONE)

A mapping key is a full node: it may carry &anchor and/or !tag properties and may itself be an *alias. These were rejected with "expected document boundary" / "expected mapping key" / "undefined alias". Four root causes were fixed:

1. Anchor/alias name termination. The scanner consumed the name up to a whitespace or flow indicator but not :, so *alias: read the alias name as alias:. It now terminates the name at the first character outside libyaml's IS_ALPHA set (ASCII letters, digits, -, _), mirroring yaml_parser_scan_anchor, so the trailing : is a mapping indicator.
2. Properties/alias in key position. The block-mapping parser only began a pair at the key indicator, but the scanner emits a key's anchor/tag before that indicator (and an alias key has no indicator at all). The parser now begins a pair at leading properties or an alias, and properties on the same line as the first key attach to that key node, not to the mapping (&a a: b anchors the key a; &a alone on its own line anchors the map).
3. Anchor on an empty value. An anchor or tag at value position with no node following on the same line now composes an empty (null) value that carries the property, so a: &anchor registers anchor as null and a later *anchor resolves to null instead of raising "undefined alias".
4. Resolved-value registration. An anchored scalar registers its resolved value (an empty anchored scalar registers null, not ""). The implicit-key scanner also now recognizes a key with trailing spaces before its :.

• Reference: the block + flow mapping productions in parser.c, yaml_parser_scan_anchor for name termination, and Yams' Constructor anchor table for resolution semantics.

Flipped 10 cases to passing: 26DV, 6KGN, 74H7, 87E4, 9KAX, E76Z, G9HC, HMQ5, LQZ7, ZWK4. (G9HC is an invalid-anchor-in-zero-indented- sequence case that is now correctly rejected.) 2SXE no longer false-rejects but its compact block-mapping structure still disagrees with the expected JSON, so it moved from false-reject to json-mismatch rather than passing; that structure is a separate compact-mapping gap.

5. Directives and document boundaries (DONE)

Directives are now handled and the ... document-end marker may be followed by a new document. Three root causes were fixed:

1. % position detection. A % begins a directive only in directive position: before any document content and before the --- that starts a document (which includes the position immediately after a ... end marker). After content, after ---, or in any flow context, % is ordinary plain text. The scanner previously treated % at column 1 as a directive unconditionally and rejected it once content had begun, mis-firing on XLQ9, UT92, W4TN, 6ZKB, 9DXL. This mirrors scanner.c, which only enters yaml_parser_scan_directive from the start-of-line directive context.
2. Lenient directive parsing. %YAML accepts a 1.x version (a higher minor is treated as 1.2 with a warning, not an error, so %YAML 1.3 parses) while rejecting an unsupported major version, a duplicate %YAML in one document, or extra non-comment fields. Reserved and unknown directives, and %TAG handle definitions, are accepted and ignored. A directive must be followed by a --- document start. This follows parser.c process_directives (duplicate and version checks) but relaxes the unknown-directive and minor-version errors to the spec's reserved-directive warning behavior.
3. Document end then new document. A ... end marker closes the current document; content that follows (with or without an explicit ---) starts a fresh document in the multi-document stream parser. The scanner no longer rejects this as an unsupported multi-document stream; the single-document parse still rejects multi-document input at compose time.

Implemented in Sources/Parsing/Scanner.Directives.swift (directive scanning, % position gate, ... boundary), Sources/Parsing/TokenEventParser.swift (a ... end marker returns to start a new document), and Sources/Parsing/TokenCursor.swift (the streaming scanner rejects a pending directive at end of stream). The new directiveWithoutDocument ParseError covers a directive with no following ---.

• Reference: scanner.c yaml_parser_scan_directive / yaml_parser_scan_version_directive_value + parser.c yaml_parser_process_directives and the document-start state machine.

Flipped 11 cases to passing with matching values: 2LFX, 5TYM, 6WLZ, 6ZKB, 7Z25, 9DXL, 9WXW, BEC7, MUS6/03, MUS6/05, MUS6/06. Correctly rejected four cases that were previously accepted-invalid: 9MMA and B63P (directive without a document), H7TQ (extra words on %YAML), and SF5V (duplicate %YAML). 6LVF and W4TN no longer parse-error but their values still disagree with the expected JSON, so they moved from false-reject to json-mismatch: 6LVF needs same-line content after --- (the KSS4 "scalars on --- line" gap) and W4TN needs literal block-scalar clip chomping to keep the trailing newline (the T5N4/4WA9 literal-scalar gap). XLQ9 and UT92 no longer mis-fire on % but still parse-error on top-level and flow multi-line plain-scalar folding respectively (the 9YRD/EX5H and NJ66 gaps), so they remain recorded. %TAG shorthand resolution (rejecting a handle used in a later document where it is undefined, e.g. QLJ7) is left to the tag-resolution work and those cases stay recorded as accepted-invalid.

6. Stricter rejection of invalid input (DONE)

Inputs that must be rejected were silently accepted: tabs used as indentation, document markers inside quoted scalars or flow collections, an anchor on an alias, and overlapping directive sections. Grounded in the error productions in scanner.c/parser.c, this bucket tightened the scanner and event parser:

• Tabs as indentation. A tab may never supply block-structure indentation (YAML 1.2 s-indent is spaces only). Sources/Parsing/Scanner.Indentation.swift rejects a tab at the start of a block line; Scanner.BlockScalar.swift rejects a tab where a block-scalar indentation space is expected (scanner.c:2993); and Scanner.Tabs.swift rejects a tab in separation position only when a nested block node follows (a - entry, a ? key, or a compact mapping key), while leaving a tab that merely separates an indicator from a scalar value valid. The IS_BLANKZ boundary fix (scanner.c:1948) also lets :/-/? be recognized when followed by a tab.
• Document markers in scalars/flow. A ---/... marker at column 0 inside an open quoted scalar (scanner.c:3058) or an open flow collection is rejected in Scanner.swift/Scanner.Directives.swift rather than silently ending the document.
• Overlapping directive sections. A %YAML/%TAG line that opens a second explicit document with no intervening ... suffix is rejected (Scanner.Directives.swift), matching the grammar's l-document-suffix rule.
• Anchor on an alias. An alias cannot carry an anchor or tag, and an anchor-plus-alias on one node is rejected (TokenEventParser.swift), mirroring the alias-first branch of yaml_parser_parse_node (parser.c:563).

Rejected (removed from the gap manifest): 4EJS, 5TRB, MUS6/01, N782, RXY3, SR86, SU74, Y79Y/000, Y79Y/004, Y79Y/005, Y79Y/006, Y79Y/008, Y79Y/009. Four of these (9MQT/01 aside) were the bucket-3..5 regressions; MUS6/01, N782, SU74, and Y79Y/009 are repaired here. The tab boundary fix also flipped 6BCT and Y79Y/010 to passing and moved DC7X and J3BT from false-reject to json-mismatch (they now parse but still need trailing-tab scalar trimming and tab-in-quoted-scalar handling to match exactly).

Deferred at this stage (to avoid regressing valid input or the default suite), and closed later by the final series (section 9):

• ----line content is dropped by the scanner (skipLine), so cases whose invalidity lives in content after a ---/... on the same line are not yet reachable: 9MQT/01, 2G84/00, 2G84/01, CXX2, and QLJ7 (per-document %TAG handle scoping). Fixing these means scanning ----line content, which touches the document-content path used by many valid cases.

• Full simple-key / multi-colon tracking is needed for ZCZ6, ZL4Z, 9KBC, and the other "two colons on one line" / wrong-indent mapping cases.

• Flow trailing-junk (9JBA, CVW2, 62EZ, P2EQ) and assorted scalar edge cases (SU5Z, X4QW, JY7Z, HU3P, DK95/01, Y79Y/003, ...) remain.

• Reference: the corresponding error productions in scanner.c/parser.c.

• Why last among the parse work: several of these only become reachable once buckets 1-5 stop short-circuiting; tightening errors earlier would mask the positive-case work.

7. Document-root multi-line plain and block scalars (DONE)

A multi-line plain scalar folded correctly as a mapping value or sequence item but not as the document's own root node, and a block scalar header on the --- line (--- |, --- >) was discarded. Both stemmed from the scanner not modelling libyaml's parser->indent (the enclosing block-collection column, -1 at the bare root). The scanner now tracks that block indent separately from per-line indentation: it rolls on a block sequence entry, an implicit/explicit mapping key, and unrolls at each block-line start and at every document boundary. Plain-scalar continuation folding compares against this block indent, so a root scalar folds its column-0 continuations while a collection member still ends at a sibling line, and a mapping value whose continuation sits at its own column (4CQQ) folds. A --- start marker no longer skips the rest of its line; its trailing content is scanned as the document's root node (--- foo, --- [1, 2], --- |, --- >), and a block scalar whose body sits at the root's column 0 captures that body and ends at the next ---/... marker.

Fixed false-rejects (now parse with matching values): 35KP, 4CQQ, 4Q9F, 6JQW, 82AN, 93WF, 96L6, 9MQT/00, 9YRD, B3HG, DK3J, EX5H, EXG3, FP8R, RZT7, T26H, UGM3, XLQ9. Corrected previously mismatched values: 27NA, 2G84/02, 2G84/03, 4WA9, 6FWR, 6LVF, A2M4, K54U, KSS4, L383, T5N4, W4TN. Now correctly rejected (were accepted-invalid): 9KBC, ZVH3, 9MQT/01. Headline 305 -> 338 pass; categories drop to false-reject 18, json-mismatch 10, accepted-invalid 36.

Two cases parsed but still mismatched at this stage and stayed json-mismatch until the final series (section 9): M7A3 (a bare document followed by ...-delimited empty documents splits into one extra document) and NB6Z (tabs on the empty lines of a multi-line plain scalar must fold to newlines rather than survive as literal tabs).

• Reference: yaml_parser_scan_plain_scalar and the parser->indent roll/unroll logic (yaml_parser_roll_indent/yaml_parser_unroll_indent), plus block-scalar indent detection in scanner.c.

8. Flow-collection completeness (DONE)

Flow mappings and sequences now accept keys of any node kind, empty keys and values, and multi-line scalar keys, matching libyaml's flow productions. The scanner implements libyaml's simple-key mechanism for flow context: it records a potential key position before scanning a plain/quoted scalar, an anchor/tag, or a [/{ opener (when a simple key is allowed there), and a later : value indicator splices a mappingKey token in front of that node. In flow context a : is always a value indicator and a ? always an explicit key indicator regardless of the following character (libyaml's flow_level || IS_BLANKZ_AT), so adjacent forms ("k":v, {k}:v) and a trailing ? parse; the block-only implicit-key forward lookahead no longer fires in flow. A multi-line plain key folds in full before its : is read. The flow parser was rebuilt around flow_sequence_entry/flow_mapping_key/flow_mapping_value, composing empty scalars where a key or value is absent. The streaming token cursor keeps an open flow collection's tokens buffered until it closes so the retroactive key splice stays valid.

Fixed false-rejects (now parse, values match where an oracle exists): 4FJ6, 8KB6, 9BXH, 9MMW, CN3R, DFF7, FRK4, NJ66, UT92. Corrected the json-mismatch CT4Q (spec example 7.20, single-pair explicit entry) to a matching value. Headline 338 -> 348 pass; categories drop to false-reject 9, json-mismatch 9, accepted-invalid 36.

Deferred at this stage and closed later by the final series (section 9): LX3P ([flow]: block) and 6BFJ (&key [ ... ]: value) use a flow collection as a block mapping key, which needed the simple-key splice in block context, a separate change with its own regression surface. The flow-internal 7ZZ5 mismatch was a block-sequence nesting issue (- - - []), not flow, and stayed json-mismatch until then.

• Reference: yaml_parser_save_simple_key, yaml_parser_fetch_value (KEY-token splice), yaml_parser_fetch_key, the flow-collection start/end and flow-entry fetches in scanner.c, and the yaml_parser_parse_flow_* productions in parser.c.

9. The libyaml port and the closing series: 348 -> 402 (DONE)

The buckets above were fixed on the original hand-rolled scanner and token parser. That engine was then replaced wholesale: a faithful Swift port of libyaml's scanner and parser landed under PureYAML.Parsing.LibYAML (docs/LIBYAML_PORT.md), was verified against libyaml built from source, and became the sole parsing engine; the hand-rolled Scanner.* and TokenEventParser.* files named in buckets 1-8 no longer exist, so those references are historical.

On the ported engine, a final fix series closed the remaining 54 cases, itemized in CHANGELOG.md: stricter rejection of under-indented or tab-indented continuations in quoted scalars, tab-indented flow content, mis-indented leading empty lines in block scalars, multi-line implicit keys in flow sequences, block sequences opening mid-line after another node, full YAML 1.2 anchor and alias names, and the remaining flow-key and ----line content gaps. Several of these rejections (QB6E, DK95/01, Y79Y/003, S98Z, ZXT5, DK4H) surpass libyaml itself, which accepts the malformed input. The result is the manifest's current state: 402 passing, 0 gap.

Updating this document

When a fix lands:

1. Re-run bash scripts/check-yaml-test-suite.sh; it will fail on the now-fixed cases (a gap whose behavior changed) and on any case the change regressed (a passing case that now fails).
2. For each newly-passing case, assert the round-trip law (parse → dump → parse is stable) in addition to the value/JSON match, per core/round-trip-transformation.md; the parser fix is only proven once the emitter agrees on the same input.
3. Flip those cases' status to passing in Tests/Fixtures/yaml-test-suite-manifest.yaml. Do not delete a case or its vendored fixtures: the whole suite stays vendored and guarded so a future change cannot silently regress a fixed case.
4. Update the passing / gap counts in the manifest header and in this document and CHANGELOG.md.