The double-quoted escape decoder recognized only a subset of escapes
(" \ \/ \b \f \n \r \t \u) and silently passed any other character
through. That accepted invalid escapes such as \. and \', and mangled
valid ones the subset omitted: \x and \U hex escapes, \v \e \0, the named
\N \_ \L \P, and a backslash before a space, were emitted as their literal
letters (\x0d became the text x0d).
The decoder now maps every escape in the YAML 1.2 table to its character,
reads \x (8-bit) and \U (32-bit) hex escapes alongside \u, treats a
backslash before a line break as an escaped line break that joins the
lines with no separator, and rejects an unknown or truncated escape with
a diagnostic. Single-character escapes move to a lookup table so the
decoder stays within the complexity limit.
Measured against the official YAML test-suite, parse-level conformance
rises from 247/333 (74.2%) to 249/333 (74.8%) as two invalid documents
are now rejected, and value-level from 166 to 170 of comparable cases
(90.4%) as the previously-mangled escapes decode correctly, with no
regression.
A '#' that begins a comment must be preceded by whitespace or open the
line; glued to earlier same-line content (e.g. "value"# c, [ a, b ]#c,
>#c, a flow entry then ,#c) it is neither a comment nor, in those
positions, plain-scalar content, so the document is malformed. Such input
was previously accepted as a comment.
A '#' that opens a line, follows a space, or sits inside a plain scalar
(a#b, where it stays literal content and never reaches this check) is
unaffected. The scanner tracks whether a real token has been scanned on
the current physical line, reset on each line break, rather than comparing
token line numbers: the streaming tokenizer drains its token buffer
between scans, and a block scalar's blank line emits an empty scalar
attributed to the next line, either of which would misread as content.
Measured against the official YAML test-suite, parse-level conformance
rises from 243/333 (73.0%) to 247/333 (74.2%): four wrongly-accepted
invalid documents are now rejected, with no valid document regressing.
The scalar-scanning routines move to Scanner.Scalars.swift to keep the
file within the length limit.
A block scalar terminated by a less-indented comment line failed with a
spurious 'unexpected indentation': once the smaller indentation ended the
scalar, the comment's own column was processed as a dedent, and because no
indentation level held that width the dedent raised an error. A comment's
indentation does not participate in block structure, so a comment that ends
a block scalar is now left for the main loop to scan, with the block
scalar's own first content line (which may itself begin with '#') still
treated as content.
Separately, a whitespace-only line indented past a block scalar's content
level lost its surplus spaces because the width was consumed before the
line was recognized as blank; the surplus is now preserved as the line's
content, so an over-indented blank contributes the correct value.
Measured against the official YAML test-suite, parse-level conformance
rises from 240/333 (72.1%) to 243/333 (73.0%) with matching value-level
gains, and no regression.
A quoted mapping key followed by a space and then ':' ('"key" : value')
failed to parse: the scanner synthesizes the implicit mapping-key token by
checking the character immediately after the key, but a quoted scalar
stops at its closing quote, leaving the separating space, whereas a plain
scalar consumes that space into itself and ends at the colon. The quoted
path now looks past spaces for the colon, which the spec allows between an
implicit key and its value indicator.
Quoted keys with no space, quoted values, and plain keys (which already
worked with a space) are unaffected. Measured against the official YAML
test-suite, parse-level conformance rises from 238/333 (71.5%) to
240/333 (72.1%), with no regression.
A block mapping pair after the first could not carry an anchor or tag on
its key (k: 1 then &b key2: 2): the first pair's key properties are
consumed when the mapping is entered, but subsequent pairs were not, so an
anchored key past the first failed with 'expected mapping key, found
anchor'. The pair parser now consumes leading key properties for every
pair, attaching them to the key node, and a block mapping continues across
an anchor or tag token that begins a key.
Normal mappings, anchored values, sequences, and explicit keys are
unaffected. Measured against the official YAML test-suite, parse-level
conformance rises from 235/333 (70.6%) to 238/333 (71.5%), with no
regression.
The block mapping parser required a ':' after every explicit key ('? key'),
so a key with no value, including several '? key' entries in a row (a set),
failed to parse with 'expected mapping value'. An explicit key with no
following ':' now takes an implicit null value, shared with the existing
'key:'-with-nothing-after path via one emitNullMappingValue helper.
Explicit keys with values and complex keys in general are unaffected. A
test that asserted '? [a, b]' followed by a block entry is a missing-value
error was corrected: the key is a valid null entry, and the trailing
sequence is now rejected as extra content (a second root node), which is
the more accurate diagnosis.
Measured against the official YAML test-suite, parse-level conformance
rises from 229/333 (68.8%) to 235/333 (70.6%), with several of the newly
parsed documents also value-correct and no regression.
A directive the processor does not recognize (e.g. %FOO) before a
document is now ignored, matching the YAML spec (emit a warning, otherwise
no effect), rather than failing the whole document with an
unsupportedDirective error. %YAML and %TAG handling is unchanged, and a
directive after content, where a directive cannot legally appear, is still
rejected via the existing guard.
Two tests and a fixture that pinned the old reject-everything behavior are
updated: the unknown-directive case moves from the unsupported-gaps list,
and the scanner-error test now exercises a directive-after-content case
(still an error) rather than an unknown directive before content.
Measured against the official YAML test-suite, parse-level conformance
rises from 227/333 (68.2%) to 229/333 (68.8%), with the two newly-parsed
documents also value-correct and no regression.
A plain scalar at the document root failed to parse when it spanned
multiple lines at column 1 (`a\nb\nc`): the continuation check required a
continuation to be indented more than the scalar, but a root scalar has
nothing enclosing it, so its enclosing-block indent is -1 and a
continuation may start at column 1. The base indentation now returns -1 at
the root, and the continuation stops at a document marker (--- / ...) so a
scalar never folds across a document boundary.
A stream test that asserted two bare plain lines (first / second) were
rejected as extra content was corrected: that input is a valid multi-line
scalar; genuine extra content (a flow node followed by a scalar) is still
rejected, which the test now uses.
Measured against the official YAML test-suite, parse-level conformance
rises from 221/333 (66.4%) to 227/333 (68.2%), with four more documents
also now value-correct and no regression across the existing suite.
Single-quoted scalars share the same flow line folding as double-quoted,
which the previous commit added: a single break folds to a space, a blank
line to one newline, whitespace adjacent to a break is stripped, and the
first-line leading and last-line trailing whitespace are kept. The
single-quoted decoder kept its raw breaks; it now reuses the shared
foldFlowScalarLineBreaks helper, while doubled quotes ('') still escape to
one quote.
Measured against the official YAML test-suite, value-level conformance
rises from 143/169 (84.6%) to 147/169 (87.0%) with no regression.
Parse-level is unchanged: these documents already parsed, just to the
wrong tree.
The double-quoted decoder appended real line breaks and continuation-line
indentation verbatim, so a multi-line "..." scalar kept its raw newlines
and leading whitespace instead of folding. It now applies YAML flow
folding at each unescaped break: a single break folds to a space, a blank
line to one newline, and whitespace adjacent to a break (the line's
trailing, the next line's leading) is stripped. First-line leading and
last-line trailing whitespace stay (they sit next to a quote, not a
break), and \n escapes are still decoded to literal newlines.
Measured against the official YAML test-suite, value-level conformance
rises from 135/169 (79.9%) to 143/169 (84.6%) with no regression.
Parse-level is unchanged: these documents already parsed, just to the
wrong tree, which only the value-level metric exposes.
A document-start marker scan skipped the rest of the '---' line
unconditionally, so a root node written on that same line was discarded:
a block-scalar header ('--- >' / '--- |') was dropped and its body
mis-read as a plain scalar (folded to spaces, no trailing newline), and a
same-line plain or flow node was lost entirely. The document start now
skips only trailing spaces and leaves any real same-line content for the
parser; an empty rest-of-line or a comment is still consumed, and a node
on the line below the marker is unaffected. Document end ('...') still
takes no same-line node.
Measured against the official YAML test-suite, value-level conformance
rises from 130/170 (76.5%) to 135/169 (79.9%) with no regression.
Parse-level is unchanged: these documents already parsed, just to the
wrong tree, which only the value-level metric exposes.
A multi-line plain scalar whose value sits one line below its mapping key
stopped after the first line: the continuation check compared the next
line's indent against the value's own indent, which had been pushed onto
the indentation stack, so a continuation at the same indent failed the
strict '>'. It now compares against the parent block's indent (the level
below the value's, when the scalar begins exactly at the indentation top),
matching the spec: a continuation must exceed the mapping's indentation,
not the value's own. Inline values are unaffected, since their top-of-
stack already is the parent block's indent.
Measured against the official YAML test-suite, parse-level conformance
rises from 211/333 (63.4%) to 221/333 (66.4%) with no regression across
the existing 306 tests. The +10 covers the multi-line plain scalar family
the harness flagged, not just the case investigated.
Blank lines between a block scalar header and its first content line come
before the indent token, so the earlier blank-line capture skipped them.
Buffer leading blanks while the block indentation is still pending and
flush them as empty content once the indent is established, in the shared
indentation path used by both scan loops. Also fix folded bodies to emit
a break for a leading blank rather than folding it into a leading space.
This closes the last block-scalar blank-line edge from #57.
emptyDocument, expectedMappingKey, and tabIndentation were never thrown
anywhere in the parser. Empty and comment-only documents parse to null,
the mapping-key path throws unexpectedToken with an "expected mapping
key" message, and tabs in indentation are reported by the validation
layer's PreflightScanner rather than the parser. Drop the dead cases and
their switch arms and description test assertions. No reachable behavior
changes; the PropertyMutation.tabIndentation test mutation is unrelated
and untouched.
A folded block scalar folded every non-blank line into the surrounding
text, including more-indented lines, which YAML keeps literal with their
line breaks. Detect a more-indented line by its retained leading
whitespace and keep the breaks adjacent to it instead of folding them.
Plain and blank-line folding are unchanged.
The scanner dropped every blank line within and after a block scalar, so
a literal block lost internal blank lines, a folded block turned a blank
line into a space rather than a newline, and the keep indicator could not
preserve trailing blank lines. Capture blank lines as empty content in
both the batch scan loop and the streaming TokenCursor loop, which had
drifted as duplicate copies. Rework blockScalarValue to apply literal
versus folded line semantics and clip/strip/keep chomping over the
captured lines; content without blank lines is unchanged. Re-pin the
multiline-scalar test whose expectation encoded the old dropped-blank
behavior, and add a chomping-and-folding test matrix.
The scanner consumed the `+` indicator of a `|+` / `>+` block scalar but
mapped it to nothing, so keep was silently treated as clip, and a scanner
test baked that in. Add a `keep` case to BlockScalarChomping, recognize
`+` in the header scanner, and correct the test. Full keep value semantics
(preserving trailing blank lines) depend on scanner blank-line capture and
are tracked in #57; this commit fixes the token classification only.
Split the six files that declared multiple sibling types under one
namespace extension so each file declares exactly one top-level type:
the Tagged node component types (Node, NodeKind, Scalar, Sequence,
Mapping, Pair), the validation report types (Report, Source,
SourceReport, BatchReport), DiagnosticKind, TokenKind, ScannerFlowContext
kind, and the event-stream composer helpers. Public type names are
unchanged; the shared report-rendering helpers move from file-private to
internal so the split report types still reach them.
Make check-namespacing.sh namespace-aware: the previous column-anchored
scan was blind to types declared inside indented extension blocks, so it
passed while the rule was violated. It now counts direct members of a
PureYAML namespace extension and ignores nested types and private helpers.
The buffered and streaming event composers each carried an identical
copy of the merge-key (<<) override logic and its invalid-value error.
Extract both into one internal Parsing.MergeResolution helper so the
precedence rules (local keys win, earlier sources win, inherited then
local order) have a single source of truth. No behavior change; all
tests pass.
Add structured validation diagnostics, JSON and Yams comparison gates, and lazy token/event parsing so production behavior is measured without retaining full intermediate parser arrays.
