PureYAML/docs/LIBYAML_PORT.md

Faithful libyaml port

Status: complete and integrated

The port is finished and is now PureYAML's only parsing engine. Every step below has landed, the hand-rolled approximation has been deleted, and parseEvents, parseStreaming, and parseStreamStreaming all run on the port.

• Faithfulness: verified byte-for-byte against the libyaml reference event dumper across all 308 vendored valid cases. The port matches libyaml on 307; the single difference (L24T/01) is the port being more correct than libyaml.
• Conformance (canonical test.event gate, of 308): libyaml itself scores 252, the old hand-rolled parser scored 267, and the port scores 275 after closing real YAML 1.2 gaps libyaml lacks (reserved directives ignored with a warning, newer %YAML minor versions, multi-line flow keys, bare documents after ..., tab-only blank lines, JSON UTF-16 surrogate-pair \u escapes, full ns-anchor-char anchor/alias names).
• Integration boundary (Sources/Parsing/Parser.Events.swift): converts libyaml's 0-based marks to PureYAML's 1-based, translates scanner/parser errors to the diagnostic-aware ParseError, and enforces a 64-level nesting bound so deep input throws nestingTooDeep rather than overflowing the recursive composer.

Remaining open gaps and their clusters are tracked by knownEventGaps in Tests/CanonicalEventConformanceTests.swift.

Why

PureYAML's original scanner is a hand-rolled approximation of libyaml's algorithm. Its bugs (block-mapping indentation, simple keys, empty values) are the gap between that approximation and the proven reference. Rather than keep patching the approximation, the scanner is being re-derived as a faithful, function-by-function Swift port of libyaml's scanner.c, with the parser to follow from parser.c. Correctness is then by construction: each Swift function mirrors its C counterpart and is verified against libyaml directly.

The constraint that PureYAML ships no C or generated runtime is preserved: this is a translation of the algorithm into pure Swift, not a binding to libyaml.

Reference and verification

• Source of truth: PureYAMLResearch/Sources/CYaml/src/scanner.c and parser.c (libyaml 0.2.5).
• Oracle: the libyaml differential harness (scratchpad/run_events.c compiled against the same CYaml source emits the canonical yeast event stream). Each ported stage is checked token-for-token / event-for-event against it, in addition to the vendored test.event suite ground truth.
• The port lands as a parallel implementation under PureYAML.Parsing.LibYAML and is only swapped in for the current scanner once it matches the reference on the full suite and the real-world corpus, so the working parser is never regressed mid-port.

Order (each step verified before the next)

1. Indent engine (DONE): yaml_parser_roll_indent / yaml_parser_unroll_indent and the indents stack. A block collection opens only when content is strictly more indented than the current level, which fixes the - a: / same-column b: bug by construction. Implemented in Sources/Parsing/LibYAML.IndentEngine.swift, unit-tested in Tests/LibYAMLIndentEngineTests.swift.
2. Simple-key mechanism (DONE): save_simple_key, remove_simple_key, stale_simple_keys, the simple_keys stack and simple_key_allowed flag.
3. Token-fetch core (DONE): fetch_more_tokens, fetch_next_token, scan_to_next_token, and the fetch_* dispatch (fetch_key, fetch_value with its empty-scalar and roll-indent logic, fetch_block_entry, fetch_flow_*, document indicators, stream start/end).
4. Scalar scanners (DONE): scan_plain_scalar, scan_flow_scalar, scan_block_scalar (+ breaks), scan_anchor, scan_tag, scan_directive.
5. Reader (DONE): scalar-based Reader with peek/skip/skipLine and marks.
6. Parser (DONE): parser.c ported to compose events from the token stream.
7. Integration (DONE): parseEvents / parseStream run on the port and the approximation has been deleted; the canonical event gate and libyaml differential are green with zero regressions.