enforceNestingLimit (Parser.Events.swift) bounds PHYSICAL collection
depth in the raw event stream, since it counts sequenceStart/
mappingStart events directly. It does nothing against an
anchor-expansion bomb: a handful of shallow, small anchors each
aliased many times, each alias nested inside the next, keeps raw
event depth at a constant 2 per line while the LOGICAL tree an alias
resolves to grows exponentially in the number of alias levels. A
25-level, fan-out-9 document (~1.5KB) expands past 9^25 logical
nodes; before this fix, parse accepted it in under 2ms, and any
full-tree walk downstream (JSON representability checking, JSON
writing, validation) never terminated.
Added a total expanded-node budget, charged incrementally as the
document streams through the composer rather than by walking
anything: every scalar charges 1, every completed container charges 1
for its own wrapper (children already charged their own cost as they
completed), and every alias resolution charges the target's CACHED
expanded size -- computed once when the anchor was defined, looked up
in O(1). This makes a nested bomb's true compounded size visible
arithmetically within a few alias levels, without ever materializing
or walking the exponential tree it would otherwise produce. A
document with genuinely wide but non-exponential alias reuse (the
same small anchor referenced thousands of times at one level, no
nesting) stays comfortably inside the budget and parses normally.
Verified via the full existing test suite (674 tests, all passing,
zero regressions -- including the YAML Test Suite conformance corpus
and the tagged round-trip corpus), two new tests proving the bomb is
rejected (in under 2ms) and that wide non-exponential fan-out still
parses, and swiftformat/swiftlint --strict clean on all five touched
files.
PureYAML.Parsing.EventComposer and Tagged.EventComposer (two other,
separate composer implementations in this package, used by the
diagnostic-validation and tagged-parsing entry points respectively)
share the same architectural gap and would benefit from the same
fix, but are not reachable from the confirmed exploit path this
change closes -- disclosed, not fixed here.
The ported engine wins every conflict: main's post-fork patches to the
hand-rolled Scanner and TokenEventParser land on deleted files and are
dropped; their behaviors are carried by the port and guarded by the
whole-suite conformance gate (402/402 passing).
Merge adaptations, all gates green (style, namespacing, forbidden
patterns, changelog, roadmap, swiftformat, swiftlint strict, build,
full test suite, whole-suite conformance sweep):
- PureYAMLJSON representability rules move onto the Check algebra,
replacing the retired legacy Rule type.
- The permissive tag-handle pre-pass is removed; the port rejects an
undefined handle at parse time (suite case QLJ7) and the validation
report carries it as a parse diagnostic.
- EventStreamComposer's nested types split one-per-file per the
namespacing gate.
- Tests pinned to old-engine behavior were re-verified against the
reference and updated: libyaml rejects an explicit-key continuation
ending at a value colon, and suite case QB6E requires rejecting a
quoted continuation at column 0.
- CHANGELOG unreleased entries from both lines are preserved in two
groups with a provenance note.
Closes#1, closes#3
Port the located value tree onto the hand-rolled engine, reusing the
engine-agnostic located layer (Model.LocatedValue, LocatedPair,
SourcePosition, SourceRange, and the located composer helper) authored
on the conformance branch. The streaming composer now builds a parallel
located tree behind an opt-in tracksLocations flag, exposed as
parseLocated / parseStreamLocated; the plain Value path stays
byte-identical, gated behind the flag.
The scalar event carries the scanner token's end mark, so a located
scalar spans the whole token (end exclusive, one past the last
character) rather than being start-anchored. The hand-rolled scanner
already tracked token end marks, so this surfaces information the engine
was already computing.
Also stop emitting a spurious null document for a comment-only or empty
stream: YAML 1.2 gives such a stream zero documents, while the
single-document value API still maps it to null, so parse("") is
unchanged.
A whole-suite test proves the located projection equals a plain parse
for every passing case; the value, event-dump, and conformance suites
stay green (507 tests).
An anchor on an empty mapping value (`a: &anchor`) failed to register, so a
later `*anchor` errored as an undefined alias. Two bugs compounded:
- The parser dispatched the anchor-only value position to a nested block
mapping, absorbing the sibling key as the anchored node's child instead of
emitting the empty anchored node. It now emits an empty anchored node when
only a sibling mapping key follows, since the indented and inline-nested
cases are already handled by earlier branches.
- The stream composer registered an anchor by its raw scalar text, so an
anchor on an empty scalar came back as "" and a typed scalar (`&a 42`) as
text. It now registers the composed value, fixing a latent typed-anchor bug.
Fixes yaml-test-suite 6KGN; raises parse-level conformance to 306/333 (91.9%)
and value-level to 199/199 (100.0%), with the should-error rate unchanged.
Split the six files that declared multiple sibling types under one
namespace extension so each file declares exactly one top-level type:
the Tagged node component types (Node, NodeKind, Scalar, Sequence,
Mapping, Pair), the validation report types (Report, Source,
SourceReport, BatchReport), DiagnosticKind, TokenKind, ScannerFlowContext
kind, and the event-stream composer helpers. Public type names are
unchanged; the shared report-rendering helpers move from file-private to
internal so the split report types still reach them.
Make check-namespacing.sh namespace-aware: the previous column-anchored
scan was blind to types declared inside indented extension blocks, so it
passed while the rule was violated. It now counts direct members of a
PureYAML namespace extension and ignores nested types and private helpers.
The buffered and streaming event composers each carried an identical
copy of the merge-key (<<) override logic and its invalid-value error.
Extract both into one internal Parsing.MergeResolution helper so the
precedence rules (local keys win, earlier sources win, inherited then
local order) have a single source of truth. No behavior change; all
tests pass.
The scalar event now carries the scanner token's end mark, so a located
scalar's range spans the whole token (end exclusive, one past the last
character) instead of being start-anchored. This gives consumers exact
highlight widths and is the prerequisite for surgical, position-based
text edits. Only the scalar event case gained a field; the value tree,
its equality, and the event dump (which compares value and style, not
marks) are unchanged, so the canonical-event and whole-suite gates stay
green (343 tests).
Add parseLocated / parseStreamLocated, returning a Model.LocatedValue
tree parallel in shape to Value, where every node carries its source
span (1-based line/column) and located children. Collections span from
their opening to closing event; scalars carry a start-anchored span;
aliases and merge-inherited pairs honestly point at their reference site.
The plain Value, its equality, and the conformance oracle are untouched:
locations are built only when the streaming composer is asked for them
(tracksLocations), behind new entry points, so the default value path is
byte-identical. A whole-suite test proves the located projection equals
a plain parse for every passing case (402/402), including through merge
keys and aliases, and the canonical-event and gap gates stay green.
A stream whose only content is comments or whitespace has zero documents in YAML 1.2 (confirmed by both libyaml and the suite's test.event), but PureYAML was emitting a spurious empty null document into the event stream. Stop emitting it; the single-document value API maps an empty document stream to null, so PureYAML.parse("") is unchanged.
Found two ways: the canonical event gate (8G76, 98YD, AVM7) and, independently, differential testing against libyaml built from the CYaml source over the real-world corpus, where a comment-only values.yaml diverged. Raises exact canonical-event conformance to 267 of 308; the real-world differential agreement rises to 113 of 114. Adds a gated PureYAML yeast-event dumper used to drive the libyaml differential.
The conformance number was measured against a lossy oracle: the JSON
projection (in.json) discards anchors, tags, and scalar styles, and exists
for only about half the cases, so "passing" often meant only "parse did not
throw". Measured against the official suite's own test.event event streams,
the true exact conformance is 254 of 308 valid cases (40 emit a wrong event
stream, 14 false-reject), well below the JSON-proxy figure.
Add a canonical-event gate (Tests/CanonicalEventConformanceTests.swift) that
serializes the parser's events into the yaml-test-suite yeast format and
compares them exactly to each case's test.event, checking anchors, tags,
scalar styles, flow vs block, and explicit document markers, for every valid
case. The gate records the gap set and fails on any regression or unrecorded
fix. All 402 test.event files are vendored alongside their inputs.
To make the comparison exact, the parser Event model now records whether a
document boundary was explicit (documentStart/documentEnd carry an `explicit`
flag), distinguishing `---`/`...` from an implicit boundary. This resolves 89
cases that were previously correct but unverifiable without normalization.
No parser behavior changed; this is measurement and model fidelity. The 40
wrong-event and 14 false-reject cases are the real remaining gaps, to be fixed
against this gate.
A mapping key is a full node: it may carry &anchor and !tag properties and may itself be an *alias. Fixes: (1) anchor/alias name scanning now stops at the first non-name character per yaml_parser_scan_anchor, so *a: reads as alias 'a' plus a mapping indicator instead of name 'a:'; (2) the block-mapping parser enters on a leading node property or alias, attaches same-line key properties to the key node, and emits alias keys directly; (3) an anchor or tag at value position with no same-line node composes an empty (null) value carrying the property, so 'a: &anchor' registers the anchor; (4) the composer registers an anchor's resolved model value (an empty anchored scalar registers null); (5) implicit keys with trailing spaces before the colon are recognized.
Flips 10 official YAML test suite cases to passing (280/402), including G9HC now correctly rejected. 2SXE no longer false-rejects but still value-mismatches on compact-mapping interpretation (a separate gap), so it is reclassified in the manifest rather than removed. Updates the manifest, fixtures, counts, and roadmap.
Add structured validation diagnostics, JSON and Yams comparison gates, and lazy token/event parsing so production behavior is measured without retaining full intermediate parser arrays.
