Rewire ResumableDocumentScanner onto PureYAML.Streaming.Scanner #3

New Issue

mihaela · 2026-11-04T19:25:27Z

mihaela commented

2026-11-04 19:25:27 +00:00

Copied from https://codeberg.org/NodetteHQ/PureYAMLStreaming/issues/3. Opened by Mihaela on 2026-07-10.

Status (2026-07-10)

Open. ResumableDocumentScanner (Sources/PureYAMLStreaming/ResumableDocumentScanner.swift) is a naive line-based heuristic: it looks for --- at the start of a line to split documents, with no understanding of YAML grammar (block scalars, flow context, or quoting can all contain a line starting with --- that is not a real document boundary, and it would misfire).

PureYAML now ships a real, proven, resumable token-level scanner as public API: PureYAML.Streaming.Scanner (init(), feed(_:), finish(), next() throws -> Outcome), added in NodetteHQ/PureYAML commit 41ca600. It is a thin wrapper around the internal libyaml-ported scanner's scanStep(), which is proven byte-for-byte identical to batch scanning across the whole yaml-test-suite corpus at multiple chunk sizes, and the public wrapper has its own separate differential proof (PublicStreamingScannerTests.swift) covering the translation layer itself.

This is the "resumable token scanner" NodetteHQ/PureYAML#4 named as this package's actual intended consumer.

Work

Bump the PureYAML dependency pin to the commit (or next tagged release) that includes PureYAML.Streaming.Scanner.
Replace ResumableDocumentScanner's line-based consume(_:)/consumeLine(_:documents:) with a real token-driven document-boundary detector built on PureYAML.Streaming.Scanner.next(): watch for .documentStart/.documentEnd/.streamEnd token kinds instead of scanning raw lines for ---.
Decide what Parser.swift/StreamingToken.swift should look like once real tokens are available: keep the current documentSource(String)-only StreamingToken (feeding whole document text to PureYAML.parse() per document, just with a correct boundary detector), or start exposing token-level events directly per the "feed event-level consumers directly" goal in NodetteHQ/PureYAML#4.
Re-run this package's own correctness suite (and, if practical, a differential check against the current line-based scanner's output on the existing multi-document fixtures) to confirm no regression in document splitting.
Update README.md's "Status" section, which currently states "It is not yet a token-level YAML scanner," once this lands.

Not in scope here

Any change to PureYAML itself; the public API this depends on already exists and is proven.
The WASM benchmark site or its build scripts, unless the rewiring changes what PureYAMLStreamingWasmSmoke needs to link against.

Related: NodetteHQ/PureYAML#4.

_Copied from https://codeberg.org/NodetteHQ/PureYAMLStreaming/issues/3. Opened by Mihaela on 2026-07-10._ ## Status (2026-07-10) Open. `ResumableDocumentScanner` (`Sources/PureYAMLStreaming/ResumableDocumentScanner.swift`) is a naive line-based heuristic: it looks for `---` at the start of a line to split documents, with no understanding of YAML grammar (block scalars, flow context, or quoting can all contain a line starting with `---` that is not a real document boundary, and it would misfire). PureYAML now ships a real, proven, resumable token-level scanner as public API: `PureYAML.Streaming.Scanner` (`init()`, `feed(_:)`, `finish()`, `next() throws -> Outcome`), added in NodetteHQ/PureYAML commit `41ca600`. It is a thin wrapper around the internal libyaml-ported scanner's `scanStep()`, which is proven byte-for-byte identical to batch scanning across the whole yaml-test-suite corpus at multiple chunk sizes, and the public wrapper has its own separate differential proof (`PublicStreamingScannerTests.swift`) covering the translation layer itself. This is the "resumable token scanner" NodetteHQ/PureYAML#4 named as this package's actual intended consumer. ## Work - Bump the `PureYAML` dependency pin to the commit (or next tagged release) that includes `PureYAML.Streaming.Scanner`. - Replace `ResumableDocumentScanner`'s line-based `consume(_:)`/`consumeLine(_:documents:)` with a real token-driven document-boundary detector built on `PureYAML.Streaming.Scanner.next()`: watch for `.documentStart`/`.documentEnd`/`.streamEnd` token kinds instead of scanning raw lines for `---`. - Decide what `Parser.swift`/`StreamingToken.swift` should look like once real tokens are available: keep the current `documentSource(String)`-only `StreamingToken` (feeding whole document text to `PureYAML.parse()` per document, just with a correct boundary detector), or start exposing token-level events directly per the "feed event-level consumers directly" goal in NodetteHQ/PureYAML#4. - Re-run this package's own correctness suite (and, if practical, a differential check against the current line-based scanner's output on the existing multi-document fixtures) to confirm no regression in document splitting. - Update `README.md`'s "Status" section, which currently states "It is not yet a token-level YAML scanner," once this lands. ## Not in scope here - Any change to `PureYAML` itself; the public API this depends on already exists and is proven. - The WASM benchmark site or its build scripts, unless the rewiring changes what `PureYAMLStreamingWasmSmoke` needs to link against. Related: NodetteHQ/PureYAML#4.

Sign in to join this conversation.

No Label

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: NodetteHQ/PureYAMLStreaming#3