PureYAMLStreaming/docs/ARCHITECTURE.md

Streaming Architecture

PureYAMLStreaming separates three concerns:

1. ChunkedUTF8Reader pulls bounded byte chunks from a file or data source and tracks byte offsets plus line and column marks.
2. ResumableDocumentScanner consumes chunks, tokenizes them with PureYAML.Streaming.Scanner, and emits one document source at a time, cut immediately before each documentStart token or before the first directive token that follows an explicit ... document end.
3. Parser parses each emitted document through PureYAML and hands completed documents to the caller.

flowchart TD
    fileSource["File or Data"] --> chunkedReader["Chunked UTF8 Reader"]
    chunkedReader --> documentScanner["Resumable Document Scanner"]
    documentScanner --> pureParser["PureYAML Parser"]
    pureParser --> documentSink["Document Sink"]

Token-Driven Boundary Detection

PureYAML ships a resumable token scanner as public API (PureYAML.Streaming.Scanner), proven byte-for-byte identical to batch scanning across the yaml-test-suite corpus at multiple chunk sizes. ResumableDocumentScanner drives it token by token and cuts the buffered source text at grammar-confirmed boundaries, so a --- inside a quoted scalar can never misfire as a document split and directives stay attached to the document they introduce. Chunk boundaries may fall anywhere, including mid-token; the token scanner suspends and resumes across them.

Two implementation details keep the contract tight:

• After each cut the token scanner is restarted on the remaining buffer. A document boundary fully resets scanner state, and the restart bounds retained state by the current document rather than the whole stream (the scanner's reader keeps everything it was ever fed).
• A trailing CR is withheld from the token scanner while more input may still arrive, so it never sees a CR LF pair split at a suspension point. Split exactly there, the persisted plain-scalar suspension counts the pair as two line breaks and every later token mark drifts down a line (observed at PureYAML commit 2fcf546). Withholding the CR keeps the token line numbers the cuts rely on identical to batch scanning.

Parsing stays document-at-a-time: each emitted segment goes through the stable batch PureYAML entry point, which keeps the parser semantics identical to non-streaming use.

Future Event Streaming

The next milestone can feed token-level events to a public event sink directly instead of emitting whole document sources.

The intended final shape is:

flowchart TD
    bytes["Async Bytes"] --> utf8Reader["Chunked UTF8 Reader"]
    utf8Reader --> tokenScanner["Resumable Token Scanner"]
    tokenScanner --> eventParser["Token Event Parser"]
    eventParser --> eventSink["Event Sink"]
    eventParser --> documentComposer["Optional Document Composer"]