diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a926c8..4f65558 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,19 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added + +- Begin a faithful Swift port of libyaml's scanner under + `PureYAML.Parsing.LibYAML`, to replace the hand-rolled approximation whose + indentation and simple-key bugs the test suite surfaced. The first piece is the + indentation engine, a line-for-line translation of `yaml_parser_roll_indent` and + `yaml_parser_unroll_indent`: a block collection opens only when content is + strictly more indented than the current level, which fixes the `- a:` followed + by a same-column `b:` mis-nesting by construction. The port is verified against + libyaml (built from the `CYaml` source) and lands as a parallel implementation, + swapped in only when it matches the reference with no regression. See + `docs/LIBYAML_PORT.md`. No public behavior changed yet. + ### Fixed - Emit no document for a stream whose only content is comments or whitespace. A diff --git a/Sources/Parsing/LibYAML.IndentEngine.swift b/Sources/Parsing/LibYAML.IndentEngine.swift new file mode 100644 index 0000000..cb574ee --- /dev/null +++ b/Sources/Parsing/LibYAML.IndentEngine.swift @@ -0,0 +1,74 @@ +extension PureYAML.Parsing.LibYAML { + /// Port of the indentation stack from `yaml_parser_t` plus + /// `yaml_parser_roll_indent` and `yaml_parser_unroll_indent`. libyaml tracks a + /// single `indent` column (the current block collection's indentation, -1 at + /// the document root) and a stack of the enclosing levels. A block collection + /// is opened only when content is strictly more indented than the current + /// level, which is exactly why `- a:` followed by a same-column `b:` keeps `b` + /// a sibling and leaves `a` an empty value. + struct IndentEngine { + /// `parser->indent`. -1 at the document root, matching libyaml. + private(set) var indent = -1 + /// `parser->indents`, the stack of enclosing indentation levels. + private(set) var indents: [Int] = [] + /// `parser->flow_level`; indentation is inert inside flow collections. + var flowLevel = 0 + /// The token queue (`parser->tokens`) the engine appends to or, for + /// `roll_indent`, inserts a start token into at a recorded position. + private(set) var tokens: [Token] = [] + /// `parser->tokens_parsed`, the count already dequeued, used to turn an + /// absolute token number into an index into the live queue. + var tokensParsed = 0 + + /// Port of `yaml_parser_roll_indent`. In flow context it does nothing. + /// When the column is greater than the current indent, it pushes the + /// current indent, adopts the new column, and emits the block start token. + /// `number == -1` appends; otherwise the token is inserted at the recorded + /// queue position so a `BLOCK-MAPPING-START` precedes a simple key that was + /// already queued. + mutating func rollIndent( + column: Int, + number: Int, + type: TokenType, + mark: PureYAML.Parsing.Mark, + ) { + if flowLevel != 0 { + return + } + if indent < column { + indents.append(indent) + indent = column + let token = Token(type: type, startMark: mark, endMark: mark) + if number == -1 { + tokens.append(token) + } else { + tokens.insert(token, at: number - tokensParsed) + } + } + } + + /// Port of `yaml_parser_unroll_indent`. In flow context it does nothing. + /// While the current indent is deeper than the column, it emits a + /// `BLOCK-END` and pops the enclosing level, closing every block collection + /// that the dedent ends. + mutating func unrollIndent(column: Int, mark: PureYAML.Parsing.Mark) { + if flowLevel != 0 { + return + } + while indent > column { + tokens.append(Token(type: .blockEnd, startMark: mark, endMark: mark)) + indent = indents.removeLast() + } + } + + /// Dequeue the front token, advancing `tokensParsed` like libyaml's + /// `yaml_parser_scan` does when it hands a token to the parser. + mutating func dequeue() -> Token? { + guard !tokens.isEmpty else { + return nil + } + tokensParsed += 1 + return tokens.removeFirst() + } + } +} diff --git a/Sources/Parsing/LibYAML.Token.swift b/Sources/Parsing/LibYAML.Token.swift new file mode 100644 index 0000000..fe6078a --- /dev/null +++ b/Sources/Parsing/LibYAML.Token.swift @@ -0,0 +1,10 @@ +extension PureYAML.Parsing.LibYAML { + /// A scanner token, mirroring `yaml_token_t`: a kind plus its start and end + /// marks. Token-specific payloads (scalar value, anchor name, tag) are added + /// as their fetchers are ported. + struct Token: Equatable { + var type: TokenType + var startMark: PureYAML.Parsing.Mark + var endMark: PureYAML.Parsing.Mark + } +} diff --git a/Sources/Parsing/LibYAML.TokenType.swift b/Sources/Parsing/LibYAML.TokenType.swift new file mode 100644 index 0000000..2752582 --- /dev/null +++ b/Sources/Parsing/LibYAML.TokenType.swift @@ -0,0 +1,10 @@ +extension PureYAML.Parsing.LibYAML { + /// A scanner token kind, mirroring `yaml_token_type_t`. Only the structural + /// kinds the indentation engine produces are modelled here; scalar, anchor, + /// tag, and document kinds are added as their fetchers are ported. + enum TokenType: Equatable { + case blockSequenceStart + case blockMappingStart + case blockEnd + } +} diff --git a/Sources/Parsing/LibYAML.swift b/Sources/Parsing/LibYAML.swift new file mode 100644 index 0000000..162a852 --- /dev/null +++ b/Sources/Parsing/LibYAML.swift @@ -0,0 +1,10 @@ +extension PureYAML.Parsing { + /// Faithful Swift port of libyaml's scanner (`CYaml/src/scanner.c`). It is + /// built as a parallel implementation, verified token-for-token against + /// libyaml through the differential harness, to replace the hand-rolled + /// approximation whose indentation and simple-key handling the official test + /// suite surfaced as incorrect. The port proceeds function by function; each + /// piece mirrors its C counterpart so correctness is by construction rather + /// than re-derived. Members live under this namespace in their own files. + enum LibYAML {} +} diff --git a/Tests/LibYAMLIndentEngineTests.swift b/Tests/LibYAMLIndentEngineTests.swift new file mode 100644 index 0000000..024bf6e --- /dev/null +++ b/Tests/LibYAMLIndentEngineTests.swift @@ -0,0 +1,58 @@ +@testable import PureYAML +import Testing + +/// Verifies the ported indentation engine reproduces libyaml's `roll_indent` / +/// `unroll_indent` behavior, in particular the rule that fixes the block-mapping +/// bug: a block collection opens only when content is strictly more indented than +/// the current level. +@Suite("LibYAML Indent Engine") +struct LibYAMLIndentEngineTests { + private let mark = PureYAML.Parsing.Mark.start + + @Test("a same-column key after an empty value is a sibling, not nested") + func test_sameColumnKeyIsSibling() { + // `- a:` then ` b: c`: the `-` opens a sequence at column 0; the key `a` + // opens a mapping at column 2; `a` has no value; `b` arrives at column 2. + var engine = PureYAML.Parsing.LibYAML.IndentEngine() + engine.rollIndent(column: 0, number: -1, type: .blockSequenceStart, mark: mark) + engine.rollIndent(column: 2, number: -1, type: .blockMappingStart, mark: mark) + engine.unrollIndent(column: 2, mark: mark) + engine.rollIndent(column: 2, number: -1, type: .blockMappingStart, mark: mark) + engine.unrollIndent(column: -1, mark: mark) + + // One sequence, one mapping (b is a sibling of a, not nested under it). + #expect(engine.tokens.map(\.type) == [ + .blockSequenceStart, + .blockMappingStart, + .blockEnd, + .blockEnd, + ]) + } + + @Test("a more-indented value opens a nested mapping") + func test_moreIndentedValueNests() { + // `a:` then ` b: c`: the value is more indented, so it nests. + var engine = PureYAML.Parsing.LibYAML.IndentEngine() + engine.rollIndent(column: 0, number: -1, type: .blockMappingStart, mark: mark) + engine.unrollIndent(column: 2, mark: mark) + engine.rollIndent(column: 2, number: -1, type: .blockMappingStart, mark: mark) + engine.unrollIndent(column: -1, mark: mark) + + #expect(engine.tokens.map(\.type) == [ + .blockMappingStart, + .blockMappingStart, + .blockEnd, + .blockEnd, + ]) + } + + @Test("flow level makes indentation inert") + func test_flowLevelIsInert() { + var engine = PureYAML.Parsing.LibYAML.IndentEngine() + engine.flowLevel = 1 + engine.rollIndent(column: 4, number: -1, type: .blockMappingStart, mark: mark) + engine.unrollIndent(column: 0, mark: mark) + #expect(engine.tokens.isEmpty) + #expect(engine.indent == -1) + } +} diff --git a/docs/LIBYAML_PORT.md b/docs/LIBYAML_PORT.md new file mode 100644 index 0000000..d637f52 --- /dev/null +++ b/docs/LIBYAML_PORT.md @@ -0,0 +1,49 @@ +# Faithful libyaml port + +## Why + +PureYAML's original scanner is a hand-rolled approximation of libyaml's +algorithm. Its bugs (block-mapping indentation, simple keys, empty values) are +the gap between that approximation and the proven reference. Rather than keep +patching the approximation, the scanner is being re-derived as a faithful, +function-by-function Swift port of libyaml's `scanner.c`, with the parser to +follow from `parser.c`. Correctness is then by construction: each Swift function +mirrors its C counterpart and is verified against libyaml directly. + +The constraint that PureYAML ships no C or generated runtime is preserved: this +is a translation of the algorithm into pure Swift, not a binding to libyaml. + +## Reference and verification + +- Source of truth: `PureYAMLResearch/Sources/CYaml/src/scanner.c` and `parser.c` + (libyaml 0.2.5). +- Oracle: the libyaml differential harness (`scratchpad/run_events.c` compiled + against the same `CYaml` source emits the canonical yeast event stream). Each + ported stage is checked token-for-token / event-for-event against it, in + addition to the vendored `test.event` suite ground truth. +- The port lands as a parallel implementation under `PureYAML.Parsing.LibYAML` + and is only swapped in for the current scanner once it matches the reference on + the full suite and the real-world corpus, so the working parser is never + regressed mid-port. + +## Order (each step verified before the next) + +1. **Indent engine** (DONE): `yaml_parser_roll_indent` / `yaml_parser_unroll_indent` + and the `indents` stack. A block collection opens only when content is strictly + more indented than the current level, which fixes the `- a:` / same-column `b:` + bug by construction. Implemented in `Sources/Parsing/LibYAML.IndentEngine.swift`, + unit-tested in `Tests/LibYAMLIndentEngineTests.swift`. +2. **Simple-key mechanism**: `save_simple_key`, `remove_simple_key`, + `stale_simple_keys`, the `simple_keys` stack and `simple_key_allowed` flag. +3. **Token-fetch core**: `fetch_more_tokens`, `fetch_next_token`, + `scan_to_next_token`, and the `fetch_*` dispatch (`fetch_key`, `fetch_value` + with its empty-scalar and roll-indent logic, `fetch_block_entry`, + `fetch_flow_*`, document indicators, stream start/end). +4. **Scalar scanners**: `scan_plain_scalar`, `scan_flow_scalar`, + `scan_block_scalar` (+ breaks), `scan_anchor`, `scan_tag`, `scan_directive`. +5. **Reader**: confirm the existing `Reader` covers `CACHE`/`SKIP`/`READ` and + UTF-8 marks, or port the needed pieces. +6. **Parser**: port `parser.c` to compose events from the token stream. +7. **Integration**: route `parseEvents` / `parseStream` through the port; delete + the approximation once the canonical event gate and libyaml differential are + green with zero regressions.