PureYAML.parse() silently accepted raw C0 control characters anywhere in
the input; Yams's underlying libyaml rejects them unconditionally at the
reader level. Found by comparing parse() directly against Yams on a
real-world OpenAPI corpus. The printable-character predicate this needed
already existed (CharacterClass.isPrintable, matching libyaml's
c-printable set) but was only ever wired into anchor-name validation, and
the separate PreflightScanner control-character check is opt-in via
validate(), not part of what parse() itself runs.
Reader now records the index of the first non-printable scalar as it
buffers input (covering both batch and streaming), and fetchNextToken()
throws once scanning has actually reached that position, not merely once
one exists, so an earlier real syntax error is not hidden behind a later
control character.
Verified with 5 new tests and a full re-run of the official yaml-test-suite
conformance (333/333 parse-level, 208/208 value-level, unchanged) and the
whole-suite streaming differential (still clean), confirming no regression.
Add Scanner.scanStep(), the first increment of true chunked input streaming
(#4): Reader gains isFinal/append(_:)/markFinal(), and peek(_:) flags ranDry
instead of asserting end of input when the buffer runs out mid-fetch and more
chunks may still arrive. scanStep() snapshots the scanner before each
fetchNextToken() and restores it whenever a fetch runs dry, so a construct
spanning a chunk boundary is retried after more input lands rather than
misread as an early end of input. Every existing call site keeps the default
isFinal: true, so scanAll() and batch parsing are unchanged; ranDry can never
be set when isFinal is true, by construction. A token already queued is not
always safe to release, since libyaml's simple-key mechanism can still splice
a KEY token in front of it, so scanStep() ports libyaml's
yaml_parser_fetch_more_tokens release condition.
Proof: StreamingScannerDifferentialTests.swift drives scanStep() at several
chunk sizes over representative cases always, and over the entire
yaml-test-suite manifest (402 cases) behind PUREYAML_RUN_YAML_TEST_SUITE=1,
asserting the token sequence matches scanAll() exactly. That proof caught a
real bug: lastTokenForbidsInlineSequence() read indentEngine.tokens.last,
which goes empty once tokens are released incrementally, silently breaking
the key: - a rejection. Fixed by tracking IndentEngine.lastToken explicitly,
kept in sync with tokens.last so batch behavior (which never dequeues) is
unchanged.
Not yet wired into PureYAMLStreaming; scanStep() and its supporting types stay
internal until that integration is designed.
Also broaden the real-seed mutation property suite (#6) with the three
mutation classes it named but didn't yet cover: invalidControlCharacter,
mixedTabSpaceIndentation, and truncatedFlowCollection. The test now asserts
diagnostic quality, not just pass/fail: line must fall within the mutated
document, and column, when present, must be 1-based (some ParseError cases,
like tabIndentation, are line-only by design).
A flow-sequence single-pair implicit key must be single-line, so its ':'
must sit on the key's line. The quoted form ('[ "key"\n :value ]') was
already rejected via a crossed line break, but a plain key folds its line
break, hiding it. Compare the simple key's mark line with the ':' line
instead, which catches both forms and still exempts flow mappings.
Surpasses libyaml on DK4H (and keeps ZXT5); whole-suite conformance
398/402, canonical events 308/308.
A flow-sequence single-pair implicit key is single-line
(c-s-implicit-json-key uses s-separate-in-line), so its ':' may not cross
a line break: '[ "key"\n :value ]' is malformed. A flow MAPPING adjacent
value may still span lines ('{ "foo"\n :bar }' stays valid), which is why
the check is scoped to flow levels that are sequences.
Track per-flow-level whether it is a sequence ('[' vs '{') in a
flowIsSequence stack pushed/popped with the flow level, and in fetchValue
reject a pending simple key whose ':' crossed a line break when the
current flow level is a sequence.
Surpasses libyaml on ZXT5 (5MUD, the flow-mapping analogue, stays valid);
whole-suite conformance 397/402, canonical events 308/308.
A block sequence must begin its own line. A '-' entry mid-line after an
inline mapping value ('key: - a') or a node property ('&anchor - x') is
invalid, but libyaml accepts both. A '-' after another '-' is still a
valid compact sequence ('- - a'), and an explicit key whose ':' is on a
later line may still open a sequence ('? k' / ': - a').
In fetchBlockEntry, when no line break was crossed, reject if the last
token is an anchor/tag, or a value that closed an inline implicit key
(told apart by a pending simple key: an explicit key's simple key has
gone stale across its line break). Extract the scanner guard helpers into
LibYAML.Scanner.Guards to stay within the file-length limit.
Surpasses libyaml on 5U3A, SY6V; whole-suite conformance 396/402,
canonical events 308/308.
YAML requires multi-line flow content to be indented past the enclosing
block context (s-flow-line-prefix). libyaml does not enforce this, so
'flow: [a,' / 'b]' with 'b' in column 0, and the equivalent multi-line
flow mapping, wrongly parsed.
Track whether reaching a token crossed a line break, and in flow context
reject a token at or below the parent block indent. A closing ']' / '}'
or a leading ',' may sit at any indentation, so those are exempt (a flow
mapping's closing brace in column 0 stays valid).
Surpasses libyaml on 9C9N, VJP3/00; whole-suite conformance 394/402,
canonical events 308/308.
A '...' may be followed only by whitespace and a comment on its line.
Unlike '--- x', which carries inline document content, '... x' is
malformed, but the port (like libyaml) parsed the trailing text as a new
bare document.
After emitting the document-end token, require the rest of the line to be
blank or a comment. Surpasses libyaml on 3HFZ; whole-suite conformance
392/402, canonical events 308/308.
Per ns-plain-first a '-' starts a plain scalar in flow only when a
flow-safe character follows. A '-' before a flow indicator ('[-]',
'[-, -]') cannot start a node, but the port (like libyaml) read it as the
scalar '-'.
Reject that form in the indicator dispatch. Extract scan_to_next_token,
the comment validator, and blankLineAhead into LibYAML.Scanner.SkipToToken
to keep the fetch file within the length limit.
Surpasses libyaml on G5U8, YJV2; whole-suite conformance 390/402,
canonical events 308/308.
A '#' starts a comment only after a space or tab, or at the start of a
line. A '#' adjacent to the previous token is not a comment: 'key: "v"#c',
'[ a ]#c', '[ a,#c', and a '>#c' block-scalar header are malformed, but
the port (like libyaml) accepted them.
Validate the character immediately before the '#' (a plain scalar already
consumes its own trailing blanks, so an adjacent '#' the next-token scan
sees genuinely had no separator). Applied in scan_to_next_token and the
block-scalar header tail.
Surpasses libyaml on 9JBA, CVW2, SU5Z, X4QW; whole-suite conformance
388/402. Canonical event stream stays 308/308.
A plain scalar may span blank lines, and a blank line that contains a tab
is still blank, so the tab is not indentation. libyaml rejects any tab
below the scalar indentation with 'found a tab character that violates
indentation', which wrongly rejects 'foo: 1' / tab-only line / 'bar: 2'.
Guard the tab check with blankLineAhead so it fires only when the tab
indents real content. Surpasses libyaml on DK95/04. Port score 305 -> 306,
leaving zero false-rejects.
libyaml refuses to skip a tab in the inter-token whitespace whenever a
simple key is allowed, so a tab used as valid separation (after a '-',
'?', or ':' indicator, or before a flow collection) is reported as a
character that cannot start a token. But a tab is legal in-line
separation; it is only illegal as indentation.
The scanner now skips tabs in scan_to_next_token and records in
tabbedSeparation whether one preceded the pending token. A block entry,
an explicit key, or an implicit mapping key promoted by ':' is rejected
when reached through a tab (there the tab would be indentation), while a
scalar value or a flow collection is accepted. Tab-indented keys and
'-\t-' nested entries still reject.
Surpasses libyaml on 6BCT, 6CA3, A2M4, DK95/00, Q5MG, Y79Y/010.
Port score 299 -> 305.
libyaml makes every ':' in a flow collection a value indicator, so it
mis-scans ':x', 'http://foo.com', '::vector', and 'omitted value:,', and
drops the value after an adjacent quoted or collection key. Per the spec a
':' only ends a node when followed by whitespace, end of input, or a flow
indicator, except an adjacent ':' after a JSON key (a quoted flow scalar
or a flow collection used as the key).
Gate the ':' value-indicator dispatch on that rule, tracking on the
pending simple key whether it is a JSON node (set for quoted flow scalars
and flow-collection starts). End the plain scalar at a ':' before a flow
indicator instead of rejecting it as unexpected.
Surpasses libyaml on 4ABK, 58MP, 5T43, DBG4, FRK4, HM87/00, with 9MMW
(mixing adjacent and separated keys) still correct. Port score 292 -> 298.
libyaml makes every '?' in flow context a mapping-key indicator, so it
parses '{ ?foo: bar }' as a key 'foo' (dropping the '?') and '[?x]' as an
empty single-pair mapping. Per ns-plain-first a '?' followed by a
flow-safe character (anything but whitespace, end of input, or a flow
indicator) begins a plain scalar.
Gate the '?' key-indicator dispatch on isBlankZ or, in flow context, a
flow indicator, so '?foo' and '?x' stay plain-scalar content. Hoist the
flow-indicator predicate into CharacterClass and reuse it in the plain
scalar scanner. Surpasses libyaml on 652Z and HM87/01. Port score
290 -> 292.
YAML 1.2 separation lines may contain tabs, so a line of only spaces and
tabs is blank. libyaml leaves a leading tab and later rejects it; the
scanner now consumes the trailing blanks of a whitespace-only line so the
break is reached and the line is skipped.
Canonical event conformance: 258 -> 259 of 308 (DK95/03). 333 tests pass.
The scanner now errors with "found character that cannot start any token"
when a tab leads a token in block context, matching libyaml's final guard
in fetch_next_token (a plain scalar may not begin with a blank). Adds a
differential harness (env-gated) that dumps the ported pipeline's yeast
events for the whole suite.
Verified against the libyaml reference dumper across all 308 vendored
valid cases: the port is byte-for-byte identical to libyaml on 307, and
the single difference (L24T/01) is the port matching the canonical
test.event where libyaml drops a trailing line break. 333 tests.
Adds the %YAML and %TAG directive scanner, the final scanner piece.
Ports fetch_directive, scan_directive, scan_directive_name,
scan_version_directive_value/number, and scan_tag_directive_value,
reusing the already-ported tag handle/URI scanners. Adds
versionDirective and tagDirective token kinds.
The libyaml scanner is now fully ported: stream, document markers,
directives, block + flow structure, plain/quoted/block scalars,
anchors, aliases, and tags. 319 tests, no regression.
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_tag, scan_tag, scan_tag_handle, scan_tag_uri, and scan_uri_escapes: verbatim (!<uri>), shorthand (!handle!suffix), and local (!suffix) tags, producing the handle and suffix for the parser to resolve. URI %-escapes are decoded by collecting the UTF-8 octets and decoding them as a unit. Unit-tested for a secondary-handle shorthand tag, a local tag, and a verbatim tag. Only directives (%YAML/%TAG) remain unported in the scanner. 317 tests, no regression.
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_block_scalar, scan_block_scalar, and scan_block_scalar_breaks: literal (|) and folded (>) block scalars with the chomping (+/-) and indentation indicators in either order, indentation auto-detection from the first non-empty line (clamped to parser indent + 1), folded joining of equally-indented non-empty lines into a space, and clip/strip/keep chomping of the trailing breaks. Unit-tested for literal clip, folded join, and strip chomping. Only tags and directives remain unported in the scanner. 314 tests, no regression.
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_anchor and scan_anchor: an anchor (&name) or alias (*name) whose name is a run of IS_ALPHA characters, validated to be non-empty and followed by a blank, end of input, or a terminating indicator. The flow/quoted/anchor fetchers are consolidated into one indicator dispatch to stay within the complexity limit. Unit-tested for an anchored scalar value and a later alias. Remaining unported fetchers: tags, block scalars, directives. 311 tests, no regression.
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_flow_scalar and scan_flow_scalar: single- and double-quoted scalars with line-break folding (a single break folds to a space, blank lines to newlines), the single-quote '' escape, double-quoted backslash escapes (the full \0 \a \b \t \n \v \f \r \e space \" / \ \N \_ \L \P table plus \xXX/\uXXXX/\UXXXXXXXX hex), and the escaped end-of-line continuation that suppresses the break. Escapes are decoded to Unicode scalars directly rather than UTF-8 bytes. Unit-tested for the escaped single quote, the escape table including a hex unicode escape, and multi-line folding. The remaining unported fetchers are block scalars, anchors, tags, and directives. 310 tests, no regression.
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_flow_collection_start, fetch_flow_collection_end, and fetch_flow_entry (the [ ] { } , indicators), with the flow-level push/pop already ported. Combined with the plain-scalar scanner's flow-aware termination, the port now tokenizes flow sequences and mappings of plain scalars, and the simple-key splice promotes a flow mapping key. Unit-tested for [a, b] and {k: v}. Remaining unported fetchers: quoted/block scalars, anchors, tags, directives. 307 tests, no regression.
Stage 3 of the faithful libyaml scanner port: the token producer. Translates scan_to_next_token, fetch_more_tokens/fetch_next_token dispatch, and the fetchers for stream start/end, document indicators, block entry, key, value (with the simple-key splice and roll-indent), plus scan_plain_scalar with its line-break folding. Reuses the ported indent engine, simple-key mechanism, character classes, and reader. Constructs not yet ported (flow collections, quoted scalars, anchors, tags, directives, block scalars) throw an explicit 'not yet ported' error rather than mis-scan; the port is not yet wired to the public API.
Milestone: the ported scanner tokenizes '- a:' followed by a same-column 'b:' as siblings (one BLOCK-MAPPING-START), matching libyaml, fixing the original block-mapping bug by construction; a more-indented value still nests. Unit-tested for the sibling case, the nested case, a simple pair, and multi-line plain-scalar folding. 305 tests, no regression.
