libyaml refuses to skip a tab in the inter-token whitespace whenever a
simple key is allowed, so a tab used as valid separation (after a '-',
'?', or ':' indicator, or before a flow collection) is reported as a
character that cannot start a token. But a tab is legal in-line
separation; it is only illegal as indentation.
The scanner now skips tabs in scan_to_next_token and records in
tabbedSeparation whether one preceded the pending token. A block entry,
an explicit key, or an implicit mapping key promoted by ':' is rejected
when reached through a tab (there the tab would be indentation), while a
scalar value or a flow collection is accepted. Tab-indented keys and
'-\t-' nested entries still reject.
Surpasses libyaml on 6BCT, 6CA3, A2M4, DK95/00, Q5MG, Y79Y/010.
Port score 299 -> 305.
libyaml makes every ':' in a flow collection a value indicator, so it
mis-scans ':x', 'http://foo.com', '::vector', and 'omitted value:,', and
drops the value after an adjacent quoted or collection key. Per the spec a
':' only ends a node when followed by whitespace, end of input, or a flow
indicator, except an adjacent ':' after a JSON key (a quoted flow scalar
or a flow collection used as the key).
Gate the ':' value-indicator dispatch on that rule, tracking on the
pending simple key whether it is a JSON node (set for quoted flow scalars
and flow-collection starts). End the plain scalar at a ':' before a flow
indicator instead of rejecting it as unexpected.
Surpasses libyaml on 4ABK, 58MP, 5T43, DBG4, FRK4, HM87/00, with 9MMW
(mixing adjacent and separated keys) still correct. Port score 292 -> 298.
libyaml makes every '?' in flow context a mapping-key indicator, so it
parses '{ ?foo: bar }' as a key 'foo' (dropping the '?') and '[?x]' as an
empty single-pair mapping. Per ns-plain-first a '?' followed by a
flow-safe character (anything but whitespace, end of input, or a flow
indicator) begins a plain scalar.
Gate the '?' key-indicator dispatch on isBlankZ or, in flow context, a
flow indicator, so '?foo' and '?x' stay plain-scalar content. Hoist the
flow-indicator predicate into CharacterClass and reuse it in the plain
scalar scanner. Surpasses libyaml on 652Z and HM87/01. Port score
290 -> 292.
YAML 1.2 separation lines may contain tabs, so a line of only spaces and
tabs is blank. libyaml leaves a leading tab and later rejects it; the
scanner now consumes the trailing blanks of a whitespace-only line so the
break is reached and the line is skipped.
Canonical event conformance: 258 -> 259 of 308 (DK95/03). 333 tests pass.
The scanner now errors with "found character that cannot start any token"
when a tab leads a token in block context, matching libyaml's final guard
in fetch_next_token (a plain scalar may not begin with a blank). Adds a
differential harness (env-gated) that dumps the ported pipeline's yeast
events for the whole suite.
Verified against the libyaml reference dumper across all 308 vendored
valid cases: the port is byte-for-byte identical to libyaml on 307, and
the single difference (L24T/01) is the port matching the canonical
test.event where libyaml drops a trailing line break. 333 tests.
Adds the %YAML and %TAG directive scanner, the final scanner piece.
Ports fetch_directive, scan_directive, scan_directive_name,
scan_version_directive_value/number, and scan_tag_directive_value,
reusing the already-ported tag handle/URI scanners. Adds
versionDirective and tagDirective token kinds.
The libyaml scanner is now fully ported: stream, document markers,
directives, block + flow structure, plain/quoted/block scalars,
anchors, aliases, and tags. 319 tests, no regression.
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_tag, scan_tag, scan_tag_handle, scan_tag_uri, and scan_uri_escapes: verbatim (!<uri>), shorthand (!handle!suffix), and local (!suffix) tags, producing the handle and suffix for the parser to resolve. URI %-escapes are decoded by collecting the UTF-8 octets and decoding them as a unit. Unit-tested for a secondary-handle shorthand tag, a local tag, and a verbatim tag. Only directives (%YAML/%TAG) remain unported in the scanner. 317 tests, no regression.
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_block_scalar, scan_block_scalar, and scan_block_scalar_breaks: literal (|) and folded (>) block scalars with the chomping (+/-) and indentation indicators in either order, indentation auto-detection from the first non-empty line (clamped to parser indent + 1), folded joining of equally-indented non-empty lines into a space, and clip/strip/keep chomping of the trailing breaks. Unit-tested for literal clip, folded join, and strip chomping. Only tags and directives remain unported in the scanner. 314 tests, no regression.
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_anchor and scan_anchor: an anchor (&name) or alias (*name) whose name is a run of IS_ALPHA characters, validated to be non-empty and followed by a blank, end of input, or a terminating indicator. The flow/quoted/anchor fetchers are consolidated into one indicator dispatch to stay within the complexity limit. Unit-tested for an anchored scalar value and a later alias. Remaining unported fetchers: tags, block scalars, directives. 311 tests, no regression.
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_flow_scalar and scan_flow_scalar: single- and double-quoted scalars with line-break folding (a single break folds to a space, blank lines to newlines), the single-quote '' escape, double-quoted backslash escapes (the full \0 \a \b \t \n \v \f \r \e space \" / \ \N \_ \L \P table plus \xXX/\uXXXX/\UXXXXXXXX hex), and the escaped end-of-line continuation that suppresses the break. Escapes are decoded to Unicode scalars directly rather than UTF-8 bytes. Unit-tested for the escaped single quote, the escape table including a hex unicode escape, and multi-line folding. The remaining unported fetchers are block scalars, anchors, tags, and directives. 310 tests, no regression.
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_flow_collection_start, fetch_flow_collection_end, and fetch_flow_entry (the [ ] { } , indicators), with the flow-level push/pop already ported. Combined with the plain-scalar scanner's flow-aware termination, the port now tokenizes flow sequences and mappings of plain scalars, and the simple-key splice promotes a flow mapping key. Unit-tested for [a, b] and {k: v}. Remaining unported fetchers: quoted/block scalars, anchors, tags, directives. 307 tests, no regression.
Stage 3 of the faithful libyaml scanner port: the token producer. Translates scan_to_next_token, fetch_more_tokens/fetch_next_token dispatch, and the fetchers for stream start/end, document indicators, block entry, key, value (with the simple-key splice and roll-indent), plus scan_plain_scalar with its line-break folding. Reuses the ported indent engine, simple-key mechanism, character classes, and reader. Constructs not yet ported (flow collections, quoted scalars, anchors, tags, directives, block scalars) throw an explicit 'not yet ported' error rather than mis-scan; the port is not yet wired to the public API.
Milestone: the ported scanner tokenizes '- a:' followed by a same-column 'b:' as siblings (one BLOCK-MAPPING-START), matching libyaml, fixing the original block-mapping bug by construction; a more-indented value still nests. Unit-tested for the sibling case, the nested case, a simple pair, and multi-line plain-scalar folding. 305 tests, no regression.
