PureYAML.parse() silently accepted raw C0 control characters anywhere in
the input; Yams's underlying libyaml rejects them unconditionally at the
reader level. Found by comparing parse() directly against Yams on a
real-world OpenAPI corpus. The printable-character predicate this needed
already existed (CharacterClass.isPrintable, matching libyaml's
c-printable set) but was only ever wired into anchor-name validation, and
the separate PreflightScanner control-character check is opt-in via
validate(), not part of what parse() itself runs.
Reader now records the index of the first non-printable scalar as it
buffers input (covering both batch and streaming), and fetchNextToken()
throws once scanning has actually reached that position, not merely once
one exists, so an earlier real syntax error is not hidden behind a later
control character.
Verified with 5 new tests and a full re-run of the official yaml-test-suite
conformance (333/333 parse-level, 208/208 value-level, unchanged) and the
whole-suite streaming differential (still clean), confirming no regression.
Plain scalars now suspend and resume mid-token instead of rolling back to a
checkpoint and rescanning from the token's start on every chunk boundary
inside them: scanPlainScalar's loop is an explicit PlainScalarPhase state
machine that throws PlainScalarSuspended the moment a lookahead runs dry, and
scanStep() resumes it directly from Scanner.suspendedPlainScalar. Getting
this right took four rounds of bugs, each caught by the existing whole-suite
streaming differential test: checking ranDry only once at the loop's end
instead of after every peek-dependent decision, resuming a dry-interrupted
content or break run through a pre-check meant only for a fresh entry point,
trusting dispatchToken's own dry-fooled document-marker check, and a nested
blankLineAhead() lookahead independently going dry before an outer check
could catch it.
A synthetic long-scalar test (needed since the real-world benchmark fixture's
scalars are too short to exercise this) then caught a fifth, genuine bug: the
first working resumePlainScalar took the whole suspended value by parameter,
which kept a reference to its accumulated string alive for the entire resume
call and defeated copy-on-write, copying the whole scalar on every resume.
That is O(N^2) in the scalar's length, and measured 29x slower than not
suspending at all at small chunk sizes. Passing state inout directly, with
the alias broken at the call site before the call, fixed it to 3-4x, verified
genuinely linear (not just a smaller quadratic constant) by a permanent test
scaling scalar length 16x at a fixed chunk size.
Also exposes this scanner as public API for the first time: scanStep() and
everything around it were internal-only, so no consumer outside this module,
including the separate PureYAMLStreaming package that actually wants a
resumable token scanner, could call it. PureYAML.Streaming.Scanner wraps
scanStep() directly with no independent logic, translating its tokens into
public Streaming.Token/Kind/ScalarStyle types that don't leak the libyaml
port's internal naming. Proven separately from the internal scanner with its
own differential test, since a wrong case mapping in the translation layer
wouldn't be caught by a proof that never looks at the public types.
Stops short of rewiring PureYAMLStreaming's own document-boundary scanner
onto this; that is separate, cross-repo work with its own build and CI cycle.
Add Scanner.scanStep(), the first increment of true chunked input streaming
(#4): Reader gains isFinal/append(_:)/markFinal(), and peek(_:) flags ranDry
instead of asserting end of input when the buffer runs out mid-fetch and more
chunks may still arrive. scanStep() snapshots the scanner before each
fetchNextToken() and restores it whenever a fetch runs dry, so a construct
spanning a chunk boundary is retried after more input lands rather than
misread as an early end of input. Every existing call site keeps the default
isFinal: true, so scanAll() and batch parsing are unchanged; ranDry can never
be set when isFinal is true, by construction. A token already queued is not
always safe to release, since libyaml's simple-key mechanism can still splice
a KEY token in front of it, so scanStep() ports libyaml's
yaml_parser_fetch_more_tokens release condition.
Proof: StreamingScannerDifferentialTests.swift drives scanStep() at several
chunk sizes over representative cases always, and over the entire
yaml-test-suite manifest (402 cases) behind PUREYAML_RUN_YAML_TEST_SUITE=1,
asserting the token sequence matches scanAll() exactly. That proof caught a
real bug: lastTokenForbidsInlineSequence() read indentEngine.tokens.last,
which goes empty once tokens are released incrementally, silently breaking
the key: - a rejection. Fixed by tracking IndentEngine.lastToken explicitly,
kept in sync with tokens.last so batch behavior (which never dequeues) is
unchanged.
Not yet wired into PureYAMLStreaming; scanStep() and its supporting types stay
internal until that integration is designed.
Also broaden the real-seed mutation property suite (#6) with the three
mutation classes it named but didn't yet cover: invalidControlCharacter,
mixedTabSpaceIndentation, and truncatedFlowCollection. The test now asserts
diagnostic quality, not just pass/fail: line must fall within the mutated
document, and column, when present, must be 1-based (some ParseError cases,
like tabIndentation, are line-only by design).
Stage 3 foundation of the faithful libyaml scanner port. Translates libyaml's input buffer and position tracking (the buffer/mark fields and the PEEK/SKIP/SKIP_LINE macros) into PureYAML.Parsing.LibYAML.Reader. It iterates Unicode scalars rather than grapheme clusters so a CR and an LF are distinct, matching libyaml's byte-oriented scanning (a CR+LF pair is still one line break), and tracks the mark in libyaml's 0-based line and column convention. Unit-tested for peek/skip column tracking and LF and CRLF line breaks. This is the input layer the token fetchers consume.
