Mihaela Mihaljevic
884ff7d9a1
Marks every step of the port plan done and adds a status section: the port is integrated as PureYAML's only parser, verified byte-for-byte against libyaml on 307/308 cases, and scores 275/308 on the canonical event gate, ahead of libyaml's own 252 and the prior hand-rolled parser's 267.
72 lines
4.1 KiB
Markdown
72 lines
4.1 KiB
Markdown
# Faithful libyaml port
|
|
|
|
## Status: complete and integrated
|
|
|
|
The port is finished and is now PureYAML's **only** parsing engine. Every step
|
|
below has landed, the hand-rolled approximation has been deleted, and `parseEvents`,
|
|
`parseStreaming`, and `parseStreamStreaming` all run on the port.
|
|
|
|
- **Faithfulness**: verified byte-for-byte against the libyaml reference event
|
|
dumper across all 308 vendored valid cases. The port matches libyaml on 307; the
|
|
single difference (`L24T/01`) is the port being more correct than libyaml.
|
|
- **Conformance** (canonical `test.event` gate, of 308): libyaml itself scores
|
|
252, the old hand-rolled parser scored 267, and the port scores **275** after
|
|
closing real YAML 1.2 gaps libyaml lacks (reserved directives ignored with a
|
|
warning, newer `%YAML` minor versions, multi-line flow keys, bare documents
|
|
after `...`, tab-only blank lines, JSON UTF-16 surrogate-pair `\u` escapes, full
|
|
`ns-anchor-char` anchor/alias names).
|
|
- **Integration boundary** (`Sources/Parsing/Parser.Events.swift`): converts
|
|
libyaml's 0-based marks to PureYAML's 1-based, translates scanner/parser errors
|
|
to the diagnostic-aware `ParseError`, and enforces a 64-level nesting bound so
|
|
deep input throws `nestingTooDeep` rather than overflowing the recursive composer.
|
|
|
|
Remaining open gaps and their clusters are tracked by `knownEventGaps` in
|
|
`Tests/CanonicalEventConformanceTests.swift`.
|
|
|
|
## Why
|
|
|
|
PureYAML's original scanner is a hand-rolled approximation of libyaml's
|
|
algorithm. Its bugs (block-mapping indentation, simple keys, empty values) are
|
|
the gap between that approximation and the proven reference. Rather than keep
|
|
patching the approximation, the scanner is being re-derived as a faithful,
|
|
function-by-function Swift port of libyaml's `scanner.c`, with the parser to
|
|
follow from `parser.c`. Correctness is then by construction: each Swift function
|
|
mirrors its C counterpart and is verified against libyaml directly.
|
|
|
|
The constraint that PureYAML ships no C or generated runtime is preserved: this
|
|
is a translation of the algorithm into pure Swift, not a binding to libyaml.
|
|
|
|
## Reference and verification
|
|
|
|
- Source of truth: `PureYAMLResearch/Sources/CYaml/src/scanner.c` and `parser.c`
|
|
(libyaml 0.2.5).
|
|
- Oracle: the libyaml differential harness (`scratchpad/run_events.c` compiled
|
|
against the same `CYaml` source emits the canonical yeast event stream). Each
|
|
ported stage is checked token-for-token / event-for-event against it, in
|
|
addition to the vendored `test.event` suite ground truth.
|
|
- The port lands as a parallel implementation under `PureYAML.Parsing.LibYAML`
|
|
and is only swapped in for the current scanner once it matches the reference on
|
|
the full suite and the real-world corpus, so the working parser is never
|
|
regressed mid-port.
|
|
|
|
## Order (each step verified before the next)
|
|
|
|
1. **Indent engine** (DONE): `yaml_parser_roll_indent` / `yaml_parser_unroll_indent`
|
|
and the `indents` stack. A block collection opens only when content is strictly
|
|
more indented than the current level, which fixes the `- a:` / same-column `b:`
|
|
bug by construction. Implemented in `Sources/Parsing/LibYAML.IndentEngine.swift`,
|
|
unit-tested in `Tests/LibYAMLIndentEngineTests.swift`.
|
|
2. **Simple-key mechanism** (DONE): `save_simple_key`, `remove_simple_key`,
|
|
`stale_simple_keys`, the `simple_keys` stack and `simple_key_allowed` flag.
|
|
3. **Token-fetch core** (DONE): `fetch_more_tokens`, `fetch_next_token`,
|
|
`scan_to_next_token`, and the `fetch_*` dispatch (`fetch_key`, `fetch_value`
|
|
with its empty-scalar and roll-indent logic, `fetch_block_entry`,
|
|
`fetch_flow_*`, document indicators, stream start/end).
|
|
4. **Scalar scanners** (DONE): `scan_plain_scalar`, `scan_flow_scalar`,
|
|
`scan_block_scalar` (+ breaks), `scan_anchor`, `scan_tag`, `scan_directive`.
|
|
5. **Reader** (DONE): scalar-based `Reader` with `peek`/`skip`/`skipLine` and marks.
|
|
6. **Parser** (DONE): `parser.c` ported to compose events from the token stream.
|
|
7. **Integration** (DONE): `parseEvents` / `parseStream` run on the port and the
|
|
approximation has been deleted; the canonical event gate and libyaml
|
|
differential are green with zero regressions.
|