Add Scanner.scanStep(), the first increment of true chunked input streaming
(#4): Reader gains isFinal/append(_:)/markFinal(), and peek(_:) flags ranDry
instead of asserting end of input when the buffer runs out mid-fetch and more
chunks may still arrive. scanStep() snapshots the scanner before each
fetchNextToken() and restores it whenever a fetch runs dry, so a construct
spanning a chunk boundary is retried after more input lands rather than
misread as an early end of input. Every existing call site keeps the default
isFinal: true, so scanAll() and batch parsing are unchanged; ranDry can never
be set when isFinal is true, by construction. A token already queued is not
always safe to release, since libyaml's simple-key mechanism can still splice
a KEY token in front of it, so scanStep() ports libyaml's
yaml_parser_fetch_more_tokens release condition.
Proof: StreamingScannerDifferentialTests.swift drives scanStep() at several
chunk sizes over representative cases always, and over the entire
yaml-test-suite manifest (402 cases) behind PUREYAML_RUN_YAML_TEST_SUITE=1,
asserting the token sequence matches scanAll() exactly. That proof caught a
real bug: lastTokenForbidsInlineSequence() read indentEngine.tokens.last,
which goes empty once tokens are released incrementally, silently breaking
the key: - a rejection. Fixed by tracking IndentEngine.lastToken explicitly,
kept in sync with tokens.last so batch behavior (which never dequeues) is
unchanged.
Not yet wired into PureYAMLStreaming; scanStep() and its supporting types stay
internal until that integration is designed.
Also broaden the real-seed mutation property suite (#6) with the three
mutation classes it named but didn't yet cover: invalidControlCharacter,
mixedTabSpaceIndentation, and truncatedFlowCollection. The test now asserts
diagnostic quality, not just pass/fail: line must fall within the mutated
document, and column, when present, must be 1-based (some ParseError cases,
like tabIndentation, are line-only by design).
A quoted scalar's continuation line must be indented past the parent
block, so '"a' / 'b"' (continuation in column 0) and '"bar' / '\tbaz'
(tab indentation) are malformed; libyaml accepts both.
Record the indentation error during the continuation scan but defer
raising it until the closing quote is found. An unterminated quote reaches
end of input first and still reports the missing quote (the root cause)
rather than the indentation symptom, preserving the multi-error diagnostic.
Surpasses libyaml on QB6E and DK95/01. Whole-suite conformance is now
402/402 with zero recorded gaps; canonical events stay 308/308.
Routes the value and stream APIs through the ported scanner+parser (the
event API already was), so PureYAML now parses entirely on the libyaml
port. Adds the features needed to match and exceed the previous engine:
JSON UTF-16 surrogate-pair \u escapes, bare documents after an explicit
`...`, reserved-directive handling that still requires a following `---`,
and a 64-level nesting bound enforced on the event stream so deeply nested
input fails with nestingTooDeep instead of overflowing a recursive
consumer. Parser errors are translated to the diagnostic-aware ParseError,
recovering precise codes where identifiable and a readable malformed
fallback otherwise; error-reporting fixtures are rebased onto the port.
Canonical event conformance: 268 -> 271 of 308 (7Z25, HWV9, QT73).
393 tests pass.
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_flow_scalar and scan_flow_scalar: single- and double-quoted scalars with line-break folding (a single break folds to a space, blank lines to newlines), the single-quote '' escape, double-quoted backslash escapes (the full \0 \a \b \t \n \v \f \r \e space \" / \ \N \_ \L \P table plus \xXX/\uXXXX/\UXXXXXXXX hex), and the escaped end-of-line continuation that suppresses the break. Escapes are decoded to Unicode scalars directly rather than UTF-8 bytes. Unit-tested for the escaped single quote, the escape table including a hex unicode escape, and multi-line folding. The remaining unported fetchers are block scalars, anchors, tags, and directives. 310 tests, no regression.