# Streaming Architecture

PureYAMLStreaming separates three concerns:

1. `ChunkedUTF8Reader` pulls bounded byte chunks from a file or data source and
   tracks byte offsets plus line and column marks.
2. `ResumableDocumentScanner` consumes chunks, tokenizes them with
   `PureYAML.Streaming.Scanner`, and emits one document source at a time, cut
   immediately before each `documentStart` token or before the first directive
   token that follows an explicit `...` document end.
3. `Parser` parses each emitted document through PureYAML and hands completed
   documents to the caller.

```mermaid
flowchart TD
    fileSource["File or Data"] --> chunkedReader["Chunked UTF8 Reader"]
    chunkedReader --> documentScanner["Resumable Document Scanner"]
    documentScanner --> pureParser["PureYAML Parser"]
    pureParser --> documentSink["Document Sink"]
```

## Token-Driven Boundary Detection

PureYAML ships a resumable token scanner as public API
(`PureYAML.Streaming.Scanner`), proven byte-for-byte identical to batch
scanning across the yaml-test-suite corpus at multiple chunk sizes.
`ResumableDocumentScanner` drives it token by token and cuts the buffered
source text at grammar-confirmed boundaries, so a `---` inside a quoted
scalar can never misfire as a document split and directives stay attached to
the document they introduce. Chunk boundaries may fall anywhere, including
mid-token; the token scanner suspends and resumes across them.

Two implementation details keep the contract tight:

- After each cut the token scanner is restarted on the remaining buffer. A
  document boundary fully resets scanner state, and the restart bounds
  retained state by the current document rather than the whole stream (the
  scanner's reader keeps everything it was ever fed).
- A trailing CR is withheld from the token scanner while more input may still
  arrive, so it never sees a CR LF pair split at a suspension point. Split
  exactly there, the persisted plain-scalar suspension counts the pair as two
  line breaks and every later token mark drifts down a line (observed at
  PureYAML commit `2fcf546`). Withholding the CR keeps the token line numbers
  the cuts rely on identical to batch scanning.

Parsing stays document-at-a-time: each emitted segment goes through the
stable batch `PureYAML` entry point, which keeps the parser semantics
identical to non-streaming use.

## Future Event Streaming

The next milestone can feed token-level events to a public event sink
directly instead of emitting whole document sources.

The intended final shape is:

```mermaid
flowchart TD
    bytes["Async Bytes"] --> utf8Reader["Chunked UTF8 Reader"]
    utf8Reader --> tokenScanner["Resumable Token Scanner"]
    tokenScanner --> eventParser["Token Event Parser"]
    eventParser --> eventSink["Event Sink"]
    eventParser --> documentComposer["Optional Document Composer"]
```