Plain scalars now suspend and resume mid-token instead of rolling back to a
checkpoint and rescanning from the token's start on every chunk boundary
inside them: scanPlainScalar's loop is an explicit PlainScalarPhase state
machine that throws PlainScalarSuspended the moment a lookahead runs dry, and
scanStep() resumes it directly from Scanner.suspendedPlainScalar. Getting
this right took four rounds of bugs, each caught by the existing whole-suite
streaming differential test: checking ranDry only once at the loop's end
instead of after every peek-dependent decision, resuming a dry-interrupted
content or break run through a pre-check meant only for a fresh entry point,
trusting dispatchToken's own dry-fooled document-marker check, and a nested
blankLineAhead() lookahead independently going dry before an outer check
could catch it.
A synthetic long-scalar test (needed since the real-world benchmark fixture's
scalars are too short to exercise this) then caught a fifth, genuine bug: the
first working resumePlainScalar took the whole suspended value by parameter,
which kept a reference to its accumulated string alive for the entire resume
call and defeated copy-on-write, copying the whole scalar on every resume.
That is O(N^2) in the scalar's length, and measured 29x slower than not
suspending at all at small chunk sizes. Passing state inout directly, with
the alias broken at the call site before the call, fixed it to 3-4x, verified
genuinely linear (not just a smaller quadratic constant) by a permanent test
scaling scalar length 16x at a fixed chunk size.
Also exposes this scanner as public API for the first time: scanStep() and
everything around it were internal-only, so no consumer outside this module,
including the separate PureYAMLStreaming package that actually wants a
resumable token scanner, could call it. PureYAML.Streaming.Scanner wraps
scanStep() directly with no independent logic, translating its tokens into
public Streaming.Token/Kind/ScalarStyle types that don't leak the libyaml
port's internal naming. Proven separately from the internal scanner with its
own differential test, since a wrong case mapping in the translation layer
wouldn't be caught by a proof that never looks at the public types.
Stops short of rewiring PureYAMLStreaming's own document-boundary scanner
onto this; that is separate, cross-repo work with its own build and CI cycle.
Add Scanner.scanStep(), the first increment of true chunked input streaming
(#4): Reader gains isFinal/append(_:)/markFinal(), and peek(_:) flags ranDry
instead of asserting end of input when the buffer runs out mid-fetch and more
chunks may still arrive. scanStep() snapshots the scanner before each
fetchNextToken() and restores it whenever a fetch runs dry, so a construct
spanning a chunk boundary is retried after more input lands rather than
misread as an early end of input. Every existing call site keeps the default
isFinal: true, so scanAll() and batch parsing are unchanged; ranDry can never
be set when isFinal is true, by construction. A token already queued is not
always safe to release, since libyaml's simple-key mechanism can still splice
a KEY token in front of it, so scanStep() ports libyaml's
yaml_parser_fetch_more_tokens release condition.
Proof: StreamingScannerDifferentialTests.swift drives scanStep() at several
chunk sizes over representative cases always, and over the entire
yaml-test-suite manifest (402 cases) behind PUREYAML_RUN_YAML_TEST_SUITE=1,
asserting the token sequence matches scanAll() exactly. That proof caught a
real bug: lastTokenForbidsInlineSequence() read indentEngine.tokens.last,
which goes empty once tokens are released incrementally, silently breaking
the key: - a rejection. Fixed by tracking IndentEngine.lastToken explicitly,
kept in sync with tokens.last so batch behavior (which never dequeues) is
unchanged.
Not yet wired into PureYAMLStreaming; scanStep() and its supporting types stay
internal until that integration is designed.
Also broaden the real-seed mutation property suite (#6) with the three
mutation classes it named but didn't yet cover: invalidControlCharacter,
mixedTabSpaceIndentation, and truncatedFlowCollection. The test now asserts
diagnostic quality, not just pass/fail: line must fall within the mutated
document, and column, when present, must be 1-based (some ParseError cases,
like tabIndentation, are line-only by design).
A tab is never indentation, so flow-collection content reached through a
tab below the continuation indent ('- [' / '\tfoo') is malformed even
though the tab inflates the column. A tab at or past the indent is
separation, and a tab-only blank line stays blank, so both remain valid.
Track a tab in a flow continuation's leading whitespace below the
parent + 1 indent and reject content (but not a closing bracket) reached
through it. Surpasses libyaml on Y79Y/003; whole-suite conformance
400/402, canonical events 308/308.
A flow-sequence single-pair implicit key is single-line
(c-s-implicit-json-key uses s-separate-in-line), so its ':' may not cross
a line break: '[ "key"\n :value ]' is malformed. A flow MAPPING adjacent
value may still span lines ('{ "foo"\n :bar }' stays valid), which is why
the check is scoped to flow levels that are sequences.
Track per-flow-level whether it is a sequence ('[' vs '{') in a
flowIsSequence stack pushed/popped with the flow level, and in fetchValue
reject a pending simple key whose ':' crossed a line break when the
current flow level is a sequence.
Surpasses libyaml on ZXT5 (5MUD, the flow-mapping analogue, stays valid);
whole-suite conformance 397/402, canonical events 308/308.
A block sequence must begin its own line. A '-' entry mid-line after an
inline mapping value ('key: - a') or a node property ('&anchor - x') is
invalid, but libyaml accepts both. A '-' after another '-' is still a
valid compact sequence ('- - a'), and an explicit key whose ':' is on a
later line may still open a sequence ('? k' / ': - a').
In fetchBlockEntry, when no line break was crossed, reject if the last
token is an anchor/tag, or a value that closed an inline implicit key
(told apart by a pending simple key: an explicit key's simple key has
gone stale across its line break). Extract the scanner guard helpers into
LibYAML.Scanner.Guards to stay within the file-length limit.
Surpasses libyaml on 5U3A, SY6V; whole-suite conformance 396/402,
canonical events 308/308.
YAML requires multi-line flow content to be indented past the enclosing
block context (s-flow-line-prefix). libyaml does not enforce this, so
'flow: [a,' / 'b]' with 'b' in column 0, and the equivalent multi-line
flow mapping, wrongly parsed.
Track whether reaching a token crossed a line break, and in flow context
reject a token at or below the parent block indent. A closing ']' / '}'
or a leading ',' may sit at any indentation, so those are exempt (a flow
mapping's closing brace in column 0 stays valid).
Surpasses libyaml on 9C9N, VJP3/00; whole-suite conformance 394/402,
canonical events 308/308.
libyaml refuses to skip a tab in the inter-token whitespace whenever a
simple key is allowed, so a tab used as valid separation (after a '-',
'?', or ':' indicator, or before a flow collection) is reported as a
character that cannot start a token. But a tab is legal in-line
separation; it is only illegal as indentation.
The scanner now skips tabs in scan_to_next_token and records in
tabbedSeparation whether one preceded the pending token. A block entry,
an explicit key, or an implicit mapping key promoted by ':' is rejected
when reached through a tab (there the tab would be indentation), while a
scalar value or a flow collection is accepted. Tab-indented keys and
'-\t-' nested entries still reject.
Surpasses libyaml on 6BCT, 6CA3, A2M4, DK95/00, Q5MG, Y79Y/010.
Port score 299 -> 305.
YAML 1.2 lets a flow-context key be separated from its ':' value indicator
by line breaks, so a flow-level simple key must survive a new line (it is
still bounded to 1024 characters). libyaml invalidates every simple key on
a new line, which wrongly rejects keys like `{"foo"\n: "bar"}` and
multi-line quoted keys.
Canonical event conformance: 259 -> 268 of 308 (4MUZ/00-02, 5MUD, 9SA2,
K3WX, NJ66, UT92, VJP3/01). This passes the hand-rolled parser's 267 and
exceeds libyaml's own 252. 333 tests pass.
Stage 3 of the faithful libyaml scanner port: the token producer. Translates scan_to_next_token, fetch_more_tokens/fetch_next_token dispatch, and the fetchers for stream start/end, document indicators, block entry, key, value (with the simple-key splice and roll-indent), plus scan_plain_scalar with its line-break folding. Reuses the ported indent engine, simple-key mechanism, character classes, and reader. Constructs not yet ported (flow collections, quoted scalars, anchors, tags, directives, block scalars) throw an explicit 'not yet ported' error rather than mis-scan; the port is not yet wired to the public API.
Milestone: the ported scanner tokenizes '- a:' followed by a same-column 'b:' as siblings (one BLOCK-MAPPING-START), matching libyaml, fixing the original block-mapping bug by construction; a more-indented value still nests. Unit-tested for the sibling case, the nested case, a simple pair, and multi-line plain-scalar folding. 305 tests, no regression.
Stage 2 of the faithful libyaml scanner port. Translates yaml_parser_save_simple_key, yaml_parser_remove_simple_key, yaml_parser_stale_simple_keys, and the flow-level push/pop (yaml_parser_increase_flow_level / decrease) onto a new PureYAML.Parsing.LibYAML.Scanner aggregate that embeds the indent engine. The simple-key mechanism is what lets a quoted scalar, a nested collection, or a property-carrying node become a mapping key when a ':' follows, and is required exactly at the block indent. Modelled faithfully: a potential key goes stale once it spans a line break or exceeds 1024 characters, and a required key that is removed or goes stale is the 'could not find expected colon' error. Unit-tested for the required/not-required, removal-error, staleness, and flow-level cases. No public behavior changed yet.
