PureYAML/docs/YAML_TEST_SUITE_CONFORMANCE.md
Mihaela Mihaljevic cd05976fac feat(parsing): handle directives and document-end boundaries
Three root causes: (1) a '%' begins a directive only in directive position (before document content, outside flow); everywhere else it is plain content, fixing misfires inside plain scalars, flow mappings, and block scalars. (2) %YAML accepts any 1.x (a higher minor is treated as 1.2 without error), rejects major != 1 and duplicate %YAML, and requires a following '---'; reserved and %TAG directives are accepted and ignored. A directive not followed by a document start is rejected with the new directiveWithoutDocument error. (3) An explicit '...' end marker closes the current document so following content begins a fresh document in parseStream; single-document parse still rejects multi-document input at compose time.

Tests/UnsupportedYAMLGapsFixtures.swift, StreamParsingTests, ScannerTests, and ParsingErrorTests were updated in lockstep to assert the now-correct behavior (percent-after-content is plain text; '...' then content is two documents; %YAML 1.3 is valid while 2.0 and duplicate %YAML are rejected); no test was weakened and multi-document rejection for single-document parse is still covered.

Flips 15 official YAML test suite cases (295/402); SF5V, 9MMA, B63P, H7TQ now correctly rejected. 6LVF and W4TN are reclassified to json-mismatch (blocked by other scalar gaps). Updates the manifest, fixtures, counts, and roadmap.
2026-06-23 21:31:58 +02:00

13 KiB

YAML Test Suite Conformance

This document tracks PureYAML against the official YAML test suite and lays out the ordered plan for closing the gaps. It is a living document: bucket 1 (block scalars), bucket 2 (quoted-scalar folding), bucket 3 (empty nodes and empty keys), and bucket 4 (anchors, tags, and aliases as mapping keys) have landed; buckets 5-6 remain.

How conformance is measured

The suite's data branch (commit 6ad3d2c62885d82fc349026c136ef560838fdf3d) was run against PureYAML.parseStream + PureYAML.validate, with each case judged by its own oracle:

  • error cases (an error marker): a conformant parser must reject the input.
  • data cases (an in.json): the parsed value must match the expected JSON projection (core-schema resolved), compared structurally.
  • plain valid cases (neither): the input must at least parse without error.

Result at the pinned commit

402 cases (including numbered sub-cases). First measurement was 196 pass / 206 fail; after bucket 1 (block scalars, +24), bucket 2 (quoted-scalar folding, +30), bucket 3 (empty nodes and empty keys, +20), bucket 4 (anchors, tags, and aliases as mapping keys, +10), and bucket 5 (directives and document boundaries, +15) it is now 295 pass, 107 fail. The 107 remaining failing cases are vendored under Tests/Fixtures/yaml-test-suite/ and recorded in Tests/Fixtures/yaml-test-suite-gaps.yaml. Run the opt-in sweep with:

bash scripts/check-yaml-test-suite.sh

The default swift test only checks that the manifest still matches the vendored files; the behavioral sweep is gated behind PUREYAML_RUN_YAML_TEST_SUITE=1, the same way the full real-world corpus is gated. The sweep asserts that each recorded gap still reproduces, so fixing a case fails the sweep until the manifest is updated, the deliberate forcing function for keeping this document honest.

Failure categories (remaining)

Category Count Meaning
false-reject 40 Well-formed YAML that PureYAML wrongly rejects at parse time.
json-mismatch 20 Parses, but produces a value that disagrees with the expected JSON.
accepted-invalid 47 Malformed YAML that PureYAML wrongly accepts.

Root-cause buckets and fix order

Per AGENTS.md, every fix below must start from the matching behavior in the private PureYAMLResearch repository (the libyaml-derived Sources/CYaml/src/ and the Yams Swift layer), not from re-derived intuition. The order is chosen so that structural prerequisites land before the correctness work that depends on them.

1. Block scalar folding and chomping (DONE)

Literal (|) and folded (>) scalar content now follows YAML 1.2 §8.1: interior blank lines are preserved, §8.1.3 folding is applied (single break between equally indented non-empty lines folds to a space; blank lines and more-indented lines keep their breaks), tabs in content are preserved, and the + keep chomping indicator is supported alongside clip and strip. The block scalar is scanned as a single unit with indentation auto-detection, mirroring scanner.c yaml_parser_scan_block_scalar. Implemented in Sources/Parsing/Scanner.BlockScalar.swift.

Flipped 24 cases to passing: 4QFQ, 5LLU, 6VJK, 7T8X, 96NN/00, 96NN/01, DWX9, F6MC, F8F9, H2RW, JEF9/00, JEF9/01, K527, K858, L24T/00, L24T/01, M29M, M9B4, MJS9, P2AD, R4YG, TS54, W9L4, Y79Y/001. (T5N4 looks like a block-scalar case but is blocked by document-start handling, bucket 5.)

2. Quoted-scalar line breaks and empty-line folding (DONE)

Single (') and double (") quoted multi-line scalars now apply YAML 1.2 flow folding (§6.5, §7.3.1, §7.3.2): trailing whitespace before a break is stripped, leading whitespace on a continuation line is stripped, a single break between non-empty lines folds to one space, and N consecutive breaks yield N-1 literal line feeds (the first and last lines keep their own leading and trailing spaces as content). A double quoted backslash at end of line suppresses the break entirely. The double quoted escape set is completed with \xXX and \UXXXXXXXX, plus the previously missing \0 \a \v \e, literal \<space>, and named breaks \N \_ \L \P (the existing \uXXXX surrogate-pair handling is kept). The folding state machine mirrors scanner.c yaml_parser_scan_flow_scalar; implemented in Sources/Parsing/Scalar.swift (foldFlowScalar and the escape decoder).

Flipped 30 cases to passing: 3RLN/00, 3RLN/01, 3RLN/02, 3RLN/03, 3RLN/04, 3RLN/05, 4ZYM, 565N, 5GBF, 6WPF, 7A4E, 8UDB, 9SA2, 9TFX, DE56/00, DE56/01, DE56/02, DE56/03, DE56/04, DE56/05, DK95/02, DK95/08, G4RS, NAT4, NP9H, PRH3, Q8AD, T4YY, TL85, XV9V. (DK95/01 looks similar but is blocked by tab-as-indentation handling, bucket 6.)

3. Empty nodes and empty keys (DONE)

Wherever a node is grammatically required but absent, the parser now composes an empty (null) plain scalar instead of raising "expected a YAML node", mirroring parser.c's yaml_parser_process_empty_scalar fallback. The single production covers every missing-node position: after a tag or anchor property with no following node, an absent mapping value, an absent mapping key (both the plain : value form and the explicit ? form), and the empty key and value forms inside flow collections ({ : }, { , : v }, { k : !!str , }). An empty node that carries an anchor or tag is explicit; a bare absent node with no property in a grammatically-required position is still a parse error, matching the reference parser's "did not find expected node content". The empty scalar resolves through the normal scalar resolver against an empty value, so !!str yields "", !!null and a bare empty yield null, and an empty key resolves like any other empty scalar. Implemented in Sources/Parsing/TokenEventParser.EmptyNode.swift with call sites in TokenEventParser.swift, TokenEventParser.Block.swift, TokenEventParser.Flow.swift, and TokenEventParser.Helpers.swift.

Flipped 20 cases to passing: 2JQS, 2XXW, 5WE3, 6M2F, 7W2P, BU8L, CFD4, JTV5, KK5P, LE5A, M2N8/00, NHX8, NKF9, PW8X, RZP5, S3PD, SM9W/01, UKK6/02, WZ62, XW4D. (CT4Q no longer false-rejects but its flow multi-line plain-scalar key still folds incorrectly, so it moves from false-reject to json-mismatch rather than passing; that fold is a separate flow-scanning gap.)

4. Anchors, tags, and aliases as mapping keys (DONE)

A mapping key is a full node: it may carry &anchor and/or !tag properties and may itself be an *alias. These were rejected with "expected document boundary" / "expected mapping key" / "undefined alias". Four root causes were fixed:

  1. Anchor/alias name termination. The scanner consumed the name up to a whitespace or flow indicator but not :, so *alias: read the alias name as alias:. It now terminates the name at the first character outside libyaml's IS_ALPHA set (ASCII letters, digits, -, _), mirroring yaml_parser_scan_anchor, so the trailing : is a mapping indicator.
  2. Properties/alias in key position. The block-mapping parser only began a pair at the key indicator, but the scanner emits a key's anchor/tag before that indicator (and an alias key has no indicator at all). The parser now begins a pair at leading properties or an alias, and properties on the same line as the first key attach to that key node, not to the mapping (&a a: b anchors the key a; &a alone on its own line anchors the map).
  3. Anchor on an empty value. An anchor or tag at value position with no node following on the same line now composes an empty (null) value that carries the property, so a: &anchor registers anchor as null and a later *anchor resolves to null instead of raising "undefined alias".
  4. Resolved-value registration. An anchored scalar registers its resolved value (an empty anchored scalar registers null, not ""). The implicit-key scanner also now recognizes a key with trailing spaces before its :.
  • Reference: the block + flow mapping productions in parser.c, yaml_parser_scan_anchor for name termination, and Yams' Constructor anchor table for resolution semantics.

Flipped 10 cases to passing: 26DV, 6KGN, 74H7, 87E4, 9KAX, E76Z, G9HC, HMQ5, LQZ7, ZWK4. (G9HC is an invalid-anchor-in-zero-indented- sequence case that is now correctly rejected.) 2SXE no longer false-rejects but its compact block-mapping structure still disagrees with the expected JSON, so it moved from false-reject to json-mismatch rather than passing; that structure is a separate compact-mapping gap.

5. Directives and document boundaries (DONE)

Directives are now handled and the ... document-end marker may be followed by a new document. Three root causes were fixed:

  1. % position detection. A % begins a directive only in directive position: before any document content and before the --- that starts a document (which includes the position immediately after a ... end marker). After content, after ---, or in any flow context, % is ordinary plain text. The scanner previously treated % at column 1 as a directive unconditionally and rejected it once content had begun, mis-firing on XLQ9, UT92, W4TN, 6ZKB, 9DXL. This mirrors scanner.c, which only enters yaml_parser_scan_directive from the start-of-line directive context.
  2. Lenient directive parsing. %YAML accepts a 1.x version (a higher minor is treated as 1.2 with a warning, not an error, so %YAML 1.3 parses) while rejecting an unsupported major version, a duplicate %YAML in one document, or extra non-comment fields. Reserved and unknown directives, and %TAG handle definitions, are accepted and ignored. A directive must be followed by a --- document start. This follows parser.c process_directives (duplicate and version checks) but relaxes the unknown-directive and minor-version errors to the spec's reserved-directive warning behavior.
  3. Document end then new document. A ... end marker closes the current document; content that follows (with or without an explicit ---) starts a fresh document in the multi-document stream parser. The scanner no longer rejects this as an unsupported multi-document stream; the single-document parse still rejects multi-document input at compose time.

Implemented in Sources/Parsing/Scanner.Directives.swift (directive scanning, % position gate, ... boundary), Sources/Parsing/TokenEventParser.swift (a ... end marker returns to start a new document), and Sources/Parsing/TokenCursor.swift (the streaming scanner rejects a pending directive at end of stream). The new directiveWithoutDocument ParseError covers a directive with no following ---.

  • Reference: scanner.c yaml_parser_scan_directive / yaml_parser_scan_version_directive_value + parser.c yaml_parser_process_directives and the document-start state machine.

Flipped 11 cases to passing with matching values: 2LFX, 5TYM, 6WLZ, 6ZKB, 7Z25, 9DXL, 9WXW, BEC7, MUS6/03, MUS6/05, MUS6/06. Correctly rejected four cases that were previously accepted-invalid: 9MMA and B63P (directive without a document), H7TQ (extra words on %YAML), and SF5V (duplicate %YAML). 6LVF and W4TN no longer parse-error but their values still disagree with the expected JSON, so they moved from false-reject to json-mismatch: 6LVF needs same-line content after --- (the KSS4 "scalars on --- line" gap) and W4TN needs literal block-scalar clip chomping to keep the trailing newline (the T5N4/4WA9 literal-scalar gap). XLQ9 and UT92 no longer mis-fire on % but still parse-error on top-level and flow multi-line plain-scalar folding respectively (the 9YRD/EX5H and NJ66 gaps), so they remain recorded. %TAG shorthand resolution (rejecting a handle used in a later document where it is undefined, e.g. QLJ7) is left to the tag-resolution work and those cases stay recorded as accepted-invalid.

6. Stricter rejection of invalid input (47 cases, accepted-invalid)

Inputs that must be rejected are silently accepted: tabs used as indentation, bad document-end markers inside quoted scalars, malformed flow collections, and %TAG shorthands used in a document that does not define them. Examples: QB6E, RXY3, ZVH3, QLJ7, the Y79Y tab cases. (The directive-shape rejections, duplicate %YAML SF5V and extra-words H7TQ, landed with bucket 5.)

  • Reference: the corresponding error productions in scanner.c/parser.c.
  • Why last among the parse work: several of these only become reachable once buckets 1-5 stop short-circuiting; tightening errors earlier would mask the positive-case work.

Updating this document

When a bucket lands:

  1. Re-run bash scripts/check-yaml-test-suite.sh; it will fail on the now-fixed cases.
  2. For each newly-passing case, assert the round-trip law (parse → dump → parse is stable) in addition to the value/JSON match, per core/round-trip-transformation.md; the parser fix is only proven once the emitter agrees on the same input.
  3. Remove those cases from Tests/Fixtures/yaml-test-suite-gaps.yaml and delete their vendored fixtures (or, for json-mismatch cases that now parse but are still being verified, update current).
  4. Update the counts in this document and in CHANGELOG.md.