Add Scanner.scanStep(), the first increment of true chunked input streaming
(#4): Reader gains isFinal/append(_:)/markFinal(), and peek(_:) flags ranDry
instead of asserting end of input when the buffer runs out mid-fetch and more
chunks may still arrive. scanStep() snapshots the scanner before each
fetchNextToken() and restores it whenever a fetch runs dry, so a construct
spanning a chunk boundary is retried after more input lands rather than
misread as an early end of input. Every existing call site keeps the default
isFinal: true, so scanAll() and batch parsing are unchanged; ranDry can never
be set when isFinal is true, by construction. A token already queued is not
always safe to release, since libyaml's simple-key mechanism can still splice
a KEY token in front of it, so scanStep() ports libyaml's
yaml_parser_fetch_more_tokens release condition.
Proof: StreamingScannerDifferentialTests.swift drives scanStep() at several
chunk sizes over representative cases always, and over the entire
yaml-test-suite manifest (402 cases) behind PUREYAML_RUN_YAML_TEST_SUITE=1,
asserting the token sequence matches scanAll() exactly. That proof caught a
real bug: lastTokenForbidsInlineSequence() read indentEngine.tokens.last,
which goes empty once tokens are released incrementally, silently breaking
the key: - a rejection. Fixed by tracking IndentEngine.lastToken explicitly,
kept in sync with tokens.last so batch behavior (which never dequeues) is
unchanged.
Not yet wired into PureYAMLStreaming; scanStep() and its supporting types stay
internal until that integration is designed.
Also broaden the real-seed mutation property suite (#6) with the three
mutation classes it named but didn't yet cover: invalidControlCharacter,
mixedTabSpaceIndentation, and truncatedFlowCollection. The test now asserts
diagnostic quality, not just pass/fail: line must fall within the mutated
document, and column, when present, must be 1-based (some ParseError cases,
like tabIndentation, are line-only by design).
A leading empty line in a block scalar may not be more indented than the
first content line. libyaml auto-detects the deeper indentation (from the
empty lines) and silently drops the content. The auto-detection now errors
when the content line is less indented than an earlier empty line on the
same scan.
Surpasses libyaml on S98Z; whole-suite conformance 399/402, canonical
events 308/308.
A '#' starts a comment only after a space or tab, or at the start of a
line. A '#' adjacent to the previous token is not a comment: 'key: "v"#c',
'[ a ]#c', '[ a,#c', and a '>#c' block-scalar header are malformed, but
the port (like libyaml) accepted them.
Validate the character immediately before the '#' (a plain scalar already
consumes its own trailing blanks, so an adjacent '#' the next-token scan
sees genuinely had no separator). Applied in scan_to_next_token and the
block-scalar header tail.
Surpasses libyaml on 9JBA, CVW2, SU5Z, X4QW; whole-suite conformance
388/402. Canonical event stream stays 308/308.
A block scalar whose last line is whitespace only and ends at end of
input without a break still owns one line break, which keep (+) chomping
preserves. libyaml records no break because there is no break byte, so
'- |+' over a spaces-only unterminated line yields '' instead of '\n'.
In scan_block_scalar_breaks, when the break loop stops at end of input
after consuming whitespace on that line, append one implicit '\n' to the
trailing breaks (used only by keep chomping). Content lines are
unaffected because they do not skip whitespace in this loop.
With this the ported engine emits the exact canonical event stream for
every valid case in the vendored YAML test suite: 308/308, ahead of
libyaml (252) and the old hand-rolled parser (267). Port score 307 -> 308.
Now that a root-level block scalar can auto-detect indentation 0, its
content loop reaches lines in column 0, so a '---' or '...' document
marker there must end the scalar. Without the guard a '--- |' literal
folded the following '...', the next '%YAML' directive, and every later
document into its value.
Break out of the block-scalar content loop on a column-0 document marker.
The guard only fires at indent 0 (column == indent == 0), so nested block
scalars are unaffected. Surpasses libyaml on W4TN. Port score 306 -> 307.
The block-scalar break scanner raised 'found a tab character where an
indentation space is expected' for every tab seen while auto-detecting
the content indentation. But a tab at or beyond the content column is
content, not indentation: only a tab below the content column is illegal.
Gate the tab check on a content-column floor (the resolved indent, or the
parent indent + 1 while auto-detecting) instead of firing on every tab in
auto-detect mode. A tab-only line shallower than the content is still
rejected, so tab-as-indentation errors (Y79Y/000 and friends) are
preserved. Surpasses libyaml on 96NN/00, 96NN/01, R4YG, Y79Y/001.
Port score 286 -> 290.
The block-scalar scanner used 0 as the 'indentation not yet detected'
sentinel and floored the auto-detected indent at 1, mirroring libyaml's
scan_block_scalar_breaks. A root-level block scalar (parent indent -1)
whose content starts in column 0 therefore detected no content, and the
first content line was reparsed as a fresh document, raising 'did not
find expected document start'.
Switch the sentinel to -1 so 0 is a representable detected indent, and
floor the auto-detected indent at parent indent + 1 (0 at the document
root) instead of 1. Nested block scalars keep a parent indent >= 0, so
their floor stays >= 1 and they are unaffected.
Surpasses libyaml on DK3J, FP8R, and M7A3. Port score 283 -> 286.
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_block_scalar, scan_block_scalar, and scan_block_scalar_breaks: literal (|) and folded (>) block scalars with the chomping (+/-) and indentation indicators in either order, indentation auto-detection from the first non-empty line (clamped to parser indent + 1), folded joining of equally-indented non-empty lines into a space, and clip/strip/keep chomping of the trailing breaks. Unit-tested for literal clip, folded join, and strip chomping. Only tags and directives remain unported in the scanner. 314 tests, no regression.
