Mihaela Mihaljevic
b4253fd289
Replace the line-based --- heuristic with a scan over PureYAML's resumable token scanner, so a --- inside a quoted scalar can no longer misfire as a document boundary and directives stay attached to the document they introduce. Add a differential suite that checks the new scanner against the old line-based reference across generated streams and the full Geekbench corpus.
65 lines
2.8 KiB
Markdown
65 lines
2.8 KiB
Markdown
# Streaming Architecture
|
|
|
|
PureYAMLStreaming separates three concerns:
|
|
|
|
1. `ChunkedUTF8Reader` pulls bounded byte chunks from a file or data source and
|
|
tracks byte offsets plus line and column marks.
|
|
2. `ResumableDocumentScanner` consumes chunks, tokenizes them with
|
|
`PureYAML.Streaming.Scanner`, and emits one document source at a time, cut
|
|
immediately before each `documentStart` token or before the first directive
|
|
token that follows an explicit `...` document end.
|
|
3. `Parser` parses each emitted document through PureYAML and hands completed
|
|
documents to the caller.
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
fileSource["File or Data"] --> chunkedReader["Chunked UTF8 Reader"]
|
|
chunkedReader --> documentScanner["Resumable Document Scanner"]
|
|
documentScanner --> pureParser["PureYAML Parser"]
|
|
pureParser --> documentSink["Document Sink"]
|
|
```
|
|
|
|
## Token-Driven Boundary Detection
|
|
|
|
PureYAML ships a resumable token scanner as public API
|
|
(`PureYAML.Streaming.Scanner`), proven byte-for-byte identical to batch
|
|
scanning across the yaml-test-suite corpus at multiple chunk sizes.
|
|
`ResumableDocumentScanner` drives it token by token and cuts the buffered
|
|
source text at grammar-confirmed boundaries, so a `---` inside a quoted
|
|
scalar can never misfire as a document split and directives stay attached to
|
|
the document they introduce. Chunk boundaries may fall anywhere, including
|
|
mid-token; the token scanner suspends and resumes across them.
|
|
|
|
Two implementation details keep the contract tight:
|
|
|
|
- After each cut the token scanner is restarted on the remaining buffer. A
|
|
document boundary fully resets scanner state, and the restart bounds
|
|
retained state by the current document rather than the whole stream (the
|
|
scanner's reader keeps everything it was ever fed).
|
|
- A trailing CR is withheld from the token scanner while more input may still
|
|
arrive, so it never sees a CR LF pair split at a suspension point. Split
|
|
exactly there, the persisted plain-scalar suspension counts the pair as two
|
|
line breaks and every later token mark drifts down a line (observed at
|
|
PureYAML commit `2fcf546`). Withholding the CR keeps the token line numbers
|
|
the cuts rely on identical to batch scanning.
|
|
|
|
Parsing stays document-at-a-time: each emitted segment goes through the
|
|
stable batch `PureYAML` entry point, which keeps the parser semantics
|
|
identical to non-streaming use.
|
|
|
|
## Future Event Streaming
|
|
|
|
The next milestone can feed token-level events to a public event sink
|
|
directly instead of emitting whole document sources.
|
|
|
|
The intended final shape is:
|
|
|
|
```mermaid
|
|
flowchart TD
|
|
bytes["Async Bytes"] --> utf8Reader["Chunked UTF8 Reader"]
|
|
utf8Reader --> tokenScanner["Resumable Token Scanner"]
|
|
tokenScanner --> eventParser["Token Event Parser"]
|
|
eventParser --> eventSink["Event Sink"]
|
|
eventParser --> documentComposer["Optional Document Composer"]
|
|
```
|