True chunked input streaming #4

New Issue

2026-07-09T21:03:51Z

Mihaela commented

2026-07-09 21:03:51 +00:00

(Migrated from codeberg.org)

Tracks true chunked/streaming input parsing in PureYAML's core engine, distinct from the document-boundary chunking already prototyped in PureYAMLStreaming.

Current state (per PureYAMLStreaming's README): the incubator streams YAML input in chunks and emits one parsed PureYAML.Stream.Document at a time, bounded by the largest single document plus parser state. It is not yet a token-level scanner.

Remaining work:

Replace the document-boundary scanner with a resumable token scanner that can feed event-level consumers directly, so memory is bounded by tokens in flight rather than a whole document.
Prove the resumable scanner matches the batch scanner's conformance (whole-suite parity), the same discipline applied to the libyaml-port merge in PureYAML 0.2.0.
Decide what (if anything) moves back into the core PureYAML package once the streaming API and memory behavior are proven, per PureYAMLStreaming's README "Relationship To PureYAML" section.

See NodetteHQ/PureYAMLStreaming for the incubator implementation.

Reconstructed from PureYAML's README roadmap, which referenced this as issue #43 before the GitHub to Codeberg migration reset issue numbering on this repo.

Part of #8.

Tracks true chunked/streaming input parsing in PureYAML's core engine, distinct from the document-boundary chunking already prototyped in PureYAMLStreaming. Current state (per PureYAMLStreaming's README): the incubator streams YAML input in chunks and emits one parsed `PureYAML.Stream.Document` at a time, bounded by the largest single document plus parser state. It is not yet a token-level scanner. Remaining work: - Replace the document-boundary scanner with a resumable token scanner that can feed event-level consumers directly, so memory is bounded by tokens in flight rather than a whole document. - Prove the resumable scanner matches the batch scanner's conformance (whole-suite parity), the same discipline applied to the libyaml-port merge in PureYAML 0.2.0. - Decide what (if anything) moves back into the core `PureYAML` package once the streaming API and memory behavior are proven, per PureYAMLStreaming's README "Relationship To PureYAML" section. See NodetteHQ/PureYAMLStreaming for the incubator implementation. Reconstructed from PureYAML's README roadmap, which referenced this as issue #43 before the GitHub to Codeberg migration reset issue numbering on this repo. Part of #8.

Mihaela commented

2026-07-10 15:10:24 +00:00

(Migrated from codeberg.org)

Closing: all three "remaining work" items are done.

Resumable token scanner, feeding event-level consumers directly: done in PureYAML core. scanStep() (LibYAML.Scanner.Streaming.swift) resumes token-by-token across chunk boundaries, including persisted mid-token suspension for plain scalars (no checkpoint-and-rescan), and is now exposed as public API: PureYAML.Streaming.Scanner (init()/feed(_:)/finish()/next()), Streaming.Token/Token.Kind/Token.ScalarStyle.
Proven against whole-suite conformance: StreamingScannerDifferentialTests (the internal scanStep()) and PublicStreamingScannerTests (the public wrapper's own translation layer, checked separately since a wrong case mapping there wouldn't be caught by the internal proof) both run the full yaml-test-suite corpus at multiple chunk sizes and match batch parsing exactly.
Decision on what moves into core PureYAML: made and executed — the resumable scanner lives in core now, not just PureYAMLStreaming's incubator.

What's left is wiring PureYAMLStreaming's own ResumableDocumentScanner (still a naive line-based --- heuristic) onto this new public API, which is separate, cross-repo work with its own build/test/CI cycle: tracked as NodetteHQ/PureYAMLStreaming#3.

Closing: all three "remaining work" items are done. - **Resumable token scanner, feeding event-level consumers directly**: done in PureYAML core. `scanStep()` (`LibYAML.Scanner.Streaming.swift`) resumes token-by-token across chunk boundaries, including persisted mid-token suspension for plain scalars (no checkpoint-and-rescan), and is now exposed as public API: `PureYAML.Streaming.Scanner` (`init()`/`feed(_:)`/`finish()`/`next()`), `Streaming.Token`/`Token.Kind`/`Token.ScalarStyle`. - **Proven against whole-suite conformance**: `StreamingScannerDifferentialTests` (the internal `scanStep()`) and `PublicStreamingScannerTests` (the public wrapper's own translation layer, checked separately since a wrong case mapping there wouldn't be caught by the internal proof) both run the full yaml-test-suite corpus at multiple chunk sizes and match batch parsing exactly. - **Decision on what moves into core `PureYAML`**: made and executed — the resumable scanner lives in core now, not just PureYAMLStreaming's incubator. What's left is wiring `PureYAMLStreaming`'s own `ResumableDocumentScanner` (still a naive line-based `---` heuristic) onto this new public API, which is separate, cross-repo work with its own build/test/CI cycle: tracked as NodetteHQ/PureYAMLStreaming#3.

Sign in to join this conversation.

No Label

No Milestone

No project

No Assignees

1 Participants

Notifications

Due Date

The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: NodetteHQ/PureYAML#4