Compare commits

...

235 Commits
0.1.0 ... main

Author SHA1 Message Date
938ad85d9e ci: run the workflows on git.aleahim.com, and point the CI links to it
Some checks failed
rules-freshness / rules (push) Failing after 3s
2026-11-04 21:14:08 +01:00
a11858380b docs: point the links of this repository to git.aleahim.com
Some checks failed
Conformance / Conformance (push) Successful in 23s
Gates / Gates (push) Failing after 25s
Linux / Linux (push) Failing after 2s
macOS / macOS (push) Successful in 1m34s
No Cursor attribution / Scan commits (push) Successful in 3s
WASM / WASM (push) Failing after 1m2s
Windows / Windows (push) Has been cancelled
2026-11-04 20:50:07 +01:00
1572307de9 fix(build): read the symbol graph from SwiftPM and say why a build fails 2026-11-03 12:38:42 +01:00
7992c104a5 fix(hooks): accept the critic-fix commit type 2026-11-03 12:36:31 +01:00
28c89f3713 chore(rules): track rules-swift v2.14.0 (freshness marker) 2026-11-02 22:02:23 +01:00
d2421aa0ce fix(parsing): a zero followed by a sign is text, not an octal integer
Int(_, radix:) accepts a sign, so 0-3 (a CPU range, a port range) was read as the integer -3, and 0x-1, 0b-1 and 0o-1 as negative numbers. After a radix prefix only digits may follow. Octal, hex, binary and signed integers read as before.
2026-11-02 18:25:06 +01:00
ae85ec97c9 fix(emitting): quote every string the parser reads as another type
The emitter kept its own list of reserved words and number shapes,
apart from the parser's, and the two disagreed: the string yes was
written plain and read back as a boolean, as were 0x1F, 1_000 and
.inf. The parser's reading of a plain scalar is now one type that
both use, and the emitter quotes what does not read back as a string.

Also quoted: the merge key, the document end marker, and a string
with a CR+LF break, which was written raw.
2026-10-28 23:32:40 +01:00
93b90adfab fix(tests): unwrap url.path the same way as lastPathComponent
contentsOfFile: fixed the documented NSURL/URL bridging bug
(swiftlang/swift-corelibs-foundation#3707) in the previous commit,
confirmed by the Linux relay's error moving off String(contentsOf:)
entirely -- but url.path has the identical Darwin String vs Linux
String? split lastPathComponent already had. Same fix, same reason.

Verified via a full clean macOS build and the specific test passing,
plus swiftformat/swiftlint --strict clean.
2026-07-18 23:56:16 +02:00
7fcfdea68c fix(tests): read fixtures via contentsOfFile:, not String(contentsOf:)
String(contentsOf: url, encoding:) itself was the failure, independent
of how url was obtained -- a bare `for url in urls` with no sort at
all reproduces the same NSURL/URL error on this toolchain. This is a
documented swift-corelibs-foundation issue (SR-7620): String(contentsOf:)
has inconsistent NSURL/URL bridging on Linux.

contentsOfFile: takes a String path instead of a URL, so it has no
exposure to that bridging ambiguity at all.

Verified via a full clean macOS build and the specific test passing,
plus swiftformat/swiftlint --strict clean; the Linux relay confirms
or refutes the actual claim on the next push.
2026-07-18 23:54:27 +02:00
ecd1b34a18 fix(tests): sort plain Int indices, never a derived URL array
Precomputing (url, name) tuples and mapping back to [URL] via a
keypath still put a URL through a map/sorted chain, and the relay
showed the SAME NSURL/URL inference failure moved one line further
down (String(contentsOf:) on the loop variable this time).

Sorting names.indices (plain Int) and subscripting the ORIGINAL,
never-transformed urls array at the one point each URL is used
removes every map/sorted call from ever touching a URL-typed value
at all, keeping the one place a URL is read structurally identical
to a bare `for url in urls` (confirmed safe from the very first
relay run, before any of these fix attempts).

Verified via a full clean macOS build, the specific test passing,
and swiftformat/swiftlint --strict clean; the Linux relay is the
actual proof and needs the next push to confirm.
2026-07-18 23:51:57 +02:00
f1bf9e7a18 fix(tests): precompute the sort key outside sorted(by:) on Linux
The previous attempt still cast inside the sorted(by:) trailing
closure itself; the relay showed the SAME NSURL/URL inference failure
had moved to the next line (String(contentsOf: url, ...)), confirming
the cast inside that specific closure -- not the earlier named-helper
form -- was still what confused the resolver.

Precomputing (url, name) tuples via a separate map, then sorting on
the plain String name, keeps the sorted(by:) closure itself exactly
as simple as the original working comparison (no cast, no call
inside it), which is what compiled cleanly before any of this.

Verified via a full clean macOS build and the specific test passing,
plus swiftformat/swiftlint --strict clean; the actual claim (Linux
relay passes) needs the next push to confirm.
2026-07-18 23:49:16 +02:00
d29ac65d4d fix(tests): avoid a URL/NSURL bridging mismatch in the sort closure
The prior fix's separate `func name(_ url: URL) -> String` helper,
called as `name($0)`/`name($1)` inside the sort closure, made the
Linux relay fail differently: "'NSURL' is not implicitly convertible
to 'URL'" at both call sites, even though the surrounding loop's
`url` (same array, same element) type-checked fine everywhere else.
Declaring no explicit URL parameter type anywhere and inlining the
`as String? ?? ""` coercion directly at each of the three use sites
sidesteps whatever overload-resolution path the named helper's
concrete parameter type was forcing.

Verified via a full clean macOS build and test run (the specific test
passing) and swiftformat/swiftlint --strict clean; the actual claim
(Linux relay passes) still needs the next push to confirm, since this
Foundation/bridging behavior cannot be reproduced on macOS directly.
2026-07-18 23:46:48 +02:00
d465f500e3 fix(tests): normalize lastPathComponent for Linux Foundation
Darwin's Foundation returns String from URL.lastPathComponent;
swift-corelibs-foundation on this toolchain returns String?. The three
uses in TaggedNodeCorpusRoundTripTests (a sort comparator and two
Testing.Comment interpolations) compiled on Darwin but failed the
Linux build outright, an unrelated pre-existing gap surfaced while
verifying the anchor-expansion-bomb fix on the Linux CI relay.

An explicit `(url.lastPathComponent as String?) ?? ""` coercion makes
both platforms produce a uniform String? before the `??`, rather than
picking one platform's non-optional type and breaking the other.

Verified via a full clean macOS build and test run (674 tests, all
passing) and swiftformat/swiftlint --strict clean.
2026-07-18 23:43:59 +02:00
1bbf715fa5 docs(changelog): note the anchor-expansion bomb fix 2026-07-18 23:36:40 +02:00
20b96fbfd4 fix(parsing): reject anchor-expansion (billion-laughs) bombs
enforceNestingLimit (Parser.Events.swift) bounds PHYSICAL collection
depth in the raw event stream, since it counts sequenceStart/
mappingStart events directly. It does nothing against an
anchor-expansion bomb: a handful of shallow, small anchors each
aliased many times, each alias nested inside the next, keeps raw
event depth at a constant 2 per line while the LOGICAL tree an alias
resolves to grows exponentially in the number of alias levels. A
25-level, fan-out-9 document (~1.5KB) expands past 9^25 logical
nodes; before this fix, parse accepted it in under 2ms, and any
full-tree walk downstream (JSON representability checking, JSON
writing, validation) never terminated.

Added a total expanded-node budget, charged incrementally as the
document streams through the composer rather than by walking
anything: every scalar charges 1, every completed container charges 1
for its own wrapper (children already charged their own cost as they
completed), and every alias resolution charges the target's CACHED
expanded size -- computed once when the anchor was defined, looked up
in O(1). This makes a nested bomb's true compounded size visible
arithmetically within a few alias levels, without ever materializing
or walking the exponential tree it would otherwise produce. A
document with genuinely wide but non-exponential alias reuse (the
same small anchor referenced thousands of times at one level, no
nesting) stays comfortably inside the budget and parses normally.

Verified via the full existing test suite (674 tests, all passing,
zero regressions -- including the YAML Test Suite conformance corpus
and the tagged round-trip corpus), two new tests proving the bomb is
rejected (in under 2ms) and that wide non-exponential fan-out still
parses, and swiftformat/swiftlint --strict clean on all five touched
files.

PureYAML.Parsing.EventComposer and Tagged.EventComposer (two other,
separate composer implementations in this package, used by the
diagnostic-validation and tagged-parsing entry points respectively)
share the same architectural gap and would benefit from the same
fix, but are not reachable from the confirmed exploit path this
change closes -- disclosed, not fixed here.
2026-07-18 23:35:35 +02:00
2fcf54684e fix: correct emitter and tagged round-trip and validation bugs
Seven correctness fixes to the recently landed anchors/styles, canonical,
timestamp/binary, and full-node-key work, each reproduced and covered by a
regression test.

- Stream dump doubled the `---` marker under explicitDocumentStart or
  canonical: the single-document body now renders marker-free and the
  stream path owns marker emission, so a two-document stream reparses as
  two documents, not four.
- Tagged flow-mapping keys used block-context safety, so a key like `a,b`
  emitted bare and reparsed as several entries; flow keys now use the
  flow-aware plain-scalar check.
- Plain scalars ending in `:` reparse as an empty-valued mapping key, so
  both the model and tagged emit paths now fall back to quoting them.
- An alias mapping key pointing at a defined non-scalar anchor raised the
  misleading undefinedAlias; it now raises expectedScalarKey, and a truly
  undefined alias key still raises undefinedAlias.
- Tagged duplicate-key detection keyed on source spelling: it now keys on
  the resolved scalar value plus effective tag, so 0x1 and 1 collide while
  1 and "1" do not and custom-tagged keys stay distinct.
- Dumping a tree nested past the emitter limit overflowed the stack.

- Timestamp now accepts a leap-second value of 60, counted as 60 elapsed
  seconds in the epoch math.
- Tagged.Dumper honors sortKeys in its block and canonical paths for
  all-plain-string-key mappings, matching the model dumper.

Nesting-guard design: dump is public and non-throwing across 90+ call
sites, so making it throwing would be a breaking change. A tree nested
deeper than the parser's own maxNestingDepth cannot round-trip through
PureYAML (the parser rejects it), so no faithful emission exists. Each
dump entry point now measures depth iteratively (no recursion, so the
guard itself cannot overflow) and, past the limit, returns a documented,
reparseable marker document that resolves to null. Under the limit the
existing recursive renderers run byte-identically, so parsed trees and
every realistic hand-built tree are unaffected. This mirrors the parser's
enforceNestingLimit, which bounds depth before any recursive consumer.
2026-07-16 16:29:08 +02:00
e51df57c35 feat(tagged): carry mapping keys as full nodes
Closes #12 and closes #13. Tagged.Pair stores keyNode instead of a
bare string: alias mapping keys parse on the Tagged path (composer
resolves them through the anchors table, dumper re-emits the reference
with the mandatory space before the colon since a colon is a legal
anchor character), key anchors register for later aliases and re-emit,
and a key's presentation style survives dump and reparse. key and
keyTag remain as read-through computed accessors and the string-key
convenience init keeps most call sites source compatible. A style-less
hand-built key keeps the plain-when-safe emission convention. The
corpus round-trip law compares key nodes structurally; alias-key
documents move from rejected to round-tripping.
2026-07-16 15:42:56 +02:00
81e2794c9c docs(changelog): note the validation framework rename 2026-07-12 18:12:57 +02:00
8a3deaebb4 chore(validation): update coverage manifest for the renamed types
The pre-push validation-coverage gate caught the previous commit's renames
(Check/Issue/SubjectContext/BuiltinCheck/AnyCheck) as unclassified new
types and stale old entries. Reclassify them under their new names and add
the new Traversable protocol.
2026-07-12 18:11:40 +02:00
bf2dc4ece7 refactor(validation): rename framework to match the validation idiom
The validation module already implemented the composable-rule idiom under
its own names (Check/Issue/SubjectContext) and a value-type Validator with
duplicated traversal machinery per layer. Renamed the core types to the
canonical names (Validation/ValidationError/ValidationErrorCollection/
ValidationContext/BuiltinValidation/AnyValidation), converted Validator to
a reference-type class with the two-tier default-rule split, and genericized
it over both Subject and Document via a new Traversable protocol so the
model and tagged layers share one walk engine instead of two copies. Wired
the JSON-representability rule set into the same shape and closed the two
missing pieces (unwrap combinator, JSON coverage/config-pin tests).
2026-07-12 18:10:00 +02:00
e1c08260c4 feat(tagged): add typed timestamp and binary scalar accessors
Completes issue #11. Tagged.Scalar.timestampValue parses a !!timestamp
scalar under the YAML 1.1 grammar into the pure-Swift Tagged.Timestamp
(calendar components, timezone offset, secondsSinceEpoch via the
days-from-civil closed form; epoch test expectations measured against
Python's independent implementation). Tagged.Scalar.binaryValue base64
decodes a !!binary scalar into bytes, accepting unpadded and block
scalar wrapped content and rejecting corrupt input whole.

Yams hands back Foundation Date/Data; this package is mechanically
Foundation-free, so the bridges are caller-owned one-liners:
Date(timeIntervalSince1970:) and Data(_:). !!set and !!omap stay
skipped since Model.Mapping already preserves order and duplicates.

Closes #11
2026-07-12 14:20:23 +02:00
8df3f0b8d4 feat(emitting): add canonical mode, document markers, and lineWidth
Completes issue #10. Canonical mode follows libyaml's canonical layout
on both the Model and Tagged dump paths (tags explicit, scalars double
quoted, flow one entry per line, anchors and aliases preserved on the
Tagged path). explicitDocumentStart/End wrap the body in --- and ...
markers. lineWidth wraps overflowing flow collections one entry per
line and folds plain block scalars at single-space boundaries, vetoing
any fold that would manufacture syntax, so values always reparse
unchanged.

Closes #10
2026-07-12 14:13:19 +02:00
58872248d5 fix(scripts): let check-wasm skip when the SDK is absent
check-all.sh defines exit 77 as a non-fatal infrastructure skip and
check-linux.sh already uses it for an unreachable runner, but a missing
wasm SDK hard-failed with exit 2, blocking every push from a machine
without the toolchain. The hosted CI wasm job stays authoritative; a
machine with the SDK installed still fails hard on a real build break.
2026-07-12 13:32:17 +02:00
6fa49b7ffc style: bring four files back under the swiftformat gate
swiftformat --lint failed on these pre-existing files and the pre-push
hook checks the whole tree, blocking every push until they conform.
The wrapped if bodies pushed YAMLTestSuiteGapsTests over the 400-line
lint limit, so its private JSONStreamScanner helper moves to its own
support file.
2026-07-12 13:29:05 +02:00
20f080a9f7 fix(streaming): replace the token-kind trap with an exhaustive switch
The 22-case TokenType mapping was split across three initializers to
stay under the complexity lint, leaving a preconditionFailure default
the types could not prove unreachable, which also tripped the
forbidden-patterns gate on every push. One exhaustive switch makes a
new TokenType case a compile error instead of a runtime trap; the lint
now ignores case statements when counting complexity, since exhaustive
closed-enum switches are the point, not a smell.
2026-07-12 13:26:06 +02:00
b48a30f978 chore(validation): classify new public types in coverage ledger
Tagged.Dumper, Tagged.ScalarStyle, and Tagged.CollectionStyle from the
issue #9 work, plus Streaming.Token.Kind and Streaming.Token.ScalarStyle
which predate it but were never entered and block every push.
2026-07-12 13:23:24 +02:00
2a84d6616f feat(tagged): preserve anchors, aliases, and per-node styles end to end
Tagged.Scalar/Sequence/Mapping gain anchor and style fields
(Tagged.ScalarStyle, Tagged.CollectionStyle), Tagged.Node gains an
alias(anchor:node:) case carrying both the reference and its resolution,
and parseTagged populates all of it from the parse events instead of
discarding anchors and expanding aliases into copies.

PureYAML.dump(_ node: Tagged.Node, options:) re-emits anchors, alias
references, tags, and per-node styles over the document-wide policy.
Plain scalars re-emit their source spelling (NO stays NO, an empty null
stays empty); literal blocks cover keep chomping, single-line strip
chomping, leading blank lines, and whitespace-led first lines via an
explicit indentation indicator; folded scalars re-emit as > when the
folding rules invert exactly and degrade to a literal block otherwise.

The round-trip law parseTagged(dump(parseTagged(d))) == parseTagged(d)
is corpus-tested over every real-world fixture, with one documented
degradation: single-quoted content containing line breaks re-emits
double-quoted, value unchanged.

Closes #9
2026-07-12 13:20:12 +02:00
ec3642bacc feat(emitting): add indentWidth and sortKeys emitter options
Closes two of the four emitter gaps a feature-parity audit against Yams
found (PureYAMLYamsComparison's FEATURE-PARITY.md): no configurable
indentation and no sorted-key output.

indentWidth replaces every hardcoded indent + 2 in Dumper.swift/
Dumper.Mapping.swift, verified with round-trip tests since indentation is
purely a rendering choice and must never change what a document means.

sortKeys sorts a mapping's pairs by their string key in both block and flow
style, but only when every key in that mapping is a plain string; a mapping
with any complex (mapping/sequence) key stays in insertion order, since
there is no well-defined ordering across mixed key kinds. Off by default,
since insertion order is what makes a duplicate-key-preserving Mapping
round-trip meaningfully in the first place.

width-based line wrapping and canonical mode are each a correctness surface
of their own and are deliberately left for later, not folded into this.
2026-07-10 17:49:44 +02:00
f9fe2325f7 fix(parsing): reject raw control characters in parse(), matching libyaml
PureYAML.parse() silently accepted raw C0 control characters anywhere in
the input; Yams's underlying libyaml rejects them unconditionally at the
reader level. Found by comparing parse() directly against Yams on a
real-world OpenAPI corpus. The printable-character predicate this needed
already existed (CharacterClass.isPrintable, matching libyaml's
c-printable set) but was only ever wired into anchor-name validation, and
the separate PreflightScanner control-character check is opt-in via
validate(), not part of what parse() itself runs.

Reader now records the index of the first non-printable scalar as it
buffers input (covering both batch and streaming), and fetchNextToken()
throws once scanning has actually reached that position, not merely once
one exists, so an earlier real syntax error is not hidden behind a later
control character.

Verified with 5 new tests and a full re-run of the official yaml-test-suite
conformance (333/333 parse-level, 208/208 value-level, unchanged) and the
whole-suite streaming differential (still clean), confirming no regression.
2026-07-10 11:03:27 +02:00
41ca600d93 feat(parsing): persist mid-token suspension and expose it as a public streaming API
Plain scalars now suspend and resume mid-token instead of rolling back to a
checkpoint and rescanning from the token's start on every chunk boundary
inside them: scanPlainScalar's loop is an explicit PlainScalarPhase state
machine that throws PlainScalarSuspended the moment a lookahead runs dry, and
scanStep() resumes it directly from Scanner.suspendedPlainScalar. Getting
this right took four rounds of bugs, each caught by the existing whole-suite
streaming differential test: checking ranDry only once at the loop's end
instead of after every peek-dependent decision, resuming a dry-interrupted
content or break run through a pre-check meant only for a fresh entry point,
trusting dispatchToken's own dry-fooled document-marker check, and a nested
blankLineAhead() lookahead independently going dry before an outer check
could catch it.

A synthetic long-scalar test (needed since the real-world benchmark fixture's
scalars are too short to exercise this) then caught a fifth, genuine bug: the
first working resumePlainScalar took the whole suspended value by parameter,
which kept a reference to its accumulated string alive for the entire resume
call and defeated copy-on-write, copying the whole scalar on every resume.
That is O(N^2) in the scalar's length, and measured 29x slower than not
suspending at all at small chunk sizes. Passing state inout directly, with
the alias broken at the call site before the call, fixed it to 3-4x, verified
genuinely linear (not just a smaller quadratic constant) by a permanent test
scaling scalar length 16x at a fixed chunk size.

Also exposes this scanner as public API for the first time: scanStep() and
everything around it were internal-only, so no consumer outside this module,
including the separate PureYAMLStreaming package that actually wants a
resumable token scanner, could call it. PureYAML.Streaming.Scanner wraps
scanStep() directly with no independent logic, translating its tokens into
public Streaming.Token/Kind/ScalarStyle types that don't leak the libyaml
port's internal naming. Proven separately from the internal scanner with its
own differential test, since a wrong case mapping in the translation layer
wouldn't be caught by a proof that never looks at the public types.

Stops short of rewiring PureYAMLStreaming's own document-boundary scanner
onto this; that is separate, cross-repo work with its own build and CI cycle.
2026-07-10 09:10:56 +02:00
6d3f3009d2 perf(parsing): correct the scanStep() floor to a checkpoint cost, not rescanning
The previous commit concluded the ~1.75x floor needed persisted mid-token
suspension to close, since that removes the need for a checkpoint. One more
measurement shows that conclusion was wrong for the floor itself: feeding the
entire 9.5MB file as a single chunk, so isFinal stays false until the true
end and essentially no real rescanning can occur, still measures 1.75x,
identical to 1,000,000-scalar and 65536-scalar chunk sizes.

The overhead is actually two independent costs. A fixed floor (~1.75x),
present even with zero rescanning, from copying the whole Scanner on every
fetchNextToken() call whenever isFinal is false, whether or not that copy is
ever used to roll back; since isFinal only becomes true once, on the last
chunk, that copy is paid for virtually every token in a real streaming
session. And a rescan-driven tail that only appears below roughly
4096-scalar chunks (up to 11.54x at 16 scalars).

Mid-token suspension would eliminate the rescan tail, not the floor. A fix
for the floor is not yet designed: every field the checkpoint covers can be
mutated before a dry point is reached within a single fetch, so no smaller
subset is obviously safe to skip, and isFinal: true remains the only
provably safe condition for skipping it entirely.
2026-07-10 01:23:39 +02:00
c6740ca299 perf(parsing): correct the scanStep() finding to release-mode numbers
The prior commit's ~1.5x "unexplained floor" was measured in debug mode,
because release-mode test builds hit an unrelated PureYAMLJSON test-target
issue. sample(1) profiling of that debug build pointed at consumePlainContent,
a mutating throws method, which is a known trigger for defensive retain/copy
traffic under -Onone that the optimizer removes in release.

Worked around the release-test-build issue with -Xswiftc -enable-testing and
re-measured: the isFinal: true control now shows 1.04x, essentially batch
parity, so the checkpoint-skip fix was a full fix for that case, not the
small, unexplained one debug mode suggested.

What remains is now correctly explained rather than guessed at: a chunked
run's isFinal only becomes true once, on the last chunk, so for any input
that arrives incrementally, the checkpoint stays live for virtually the
entire scan regardless of chunk size (release-mode ratios: 11.70x at 16
scalars down to 1.75x at 1,000,000, never reaching the 1.04x floor). Closing
that needs the deferred persisted mid-token suspension, which removes the
need for a checkpoint rather than making it cheaper; isFinal: true is the
only condition under which skipping it is provably safe, since a single
scalar can span past any finite "probably buffered enough" margin.

Removed the temporary sample(1) profiling scratch test and documented how to
re-run the measurement in release mode, which the doc comment previously
did not call out as necessary.
2026-07-10 00:36:45 +02:00
7add6f0087 perf(parsing): measure scanStep() overhead, fix a real but partial cause
Add StreamingScannerPerformanceTests (issue #4), gated behind
PUREYAML_STREAMING_PERF_MEASURE=1, measuring scanStep() against scanAll() on
the largest real-world fixture (9.5MB) across a spread of chunk sizes.

Small chunk sizes (16-256 scalars) show a real, chunk-size-dependent rescan
cost, up to 6x at 16, consistent with the checkpoint-and-retry design
rescanning a fetch attempt once per chunk it spans. A control run with
isFinal: true from construction, where ranDry can never be set and no
rollback ever happens, still cost ~1.5x over scanAll(), proving a separate,
chunk-size-independent cost dominates at any realistic chunk size (4KB+): the
checkpoint was being copied on every fetch even when it could never be used.

Fixed by skipping the checkpoint entirely when reader.isFinal is true.
Re-measured rather than assumed fixed: the isolated control improved from
1.49x to 1.38x, a real but modest gain, while the large-chunk-size ratios
were unchanged within measurement noise. The remaining floor's cause is not
identified and is left open on issue #4 rather than patched further on
speculation; closing it needs a profiler, not another guess.

Whole-suite conformance (402/402) and the chunk-size differential proof both
still pass unchanged.
2026-07-10 00:14:19 +02:00
18fdf9b69a feat(parsing): add a resumable scanner increment and broaden the mutation suite
Add Scanner.scanStep(), the first increment of true chunked input streaming
(#4): Reader gains isFinal/append(_:)/markFinal(), and peek(_:) flags ranDry
instead of asserting end of input when the buffer runs out mid-fetch and more
chunks may still arrive. scanStep() snapshots the scanner before each
fetchNextToken() and restores it whenever a fetch runs dry, so a construct
spanning a chunk boundary is retried after more input lands rather than
misread as an early end of input. Every existing call site keeps the default
isFinal: true, so scanAll() and batch parsing are unchanged; ranDry can never
be set when isFinal is true, by construction. A token already queued is not
always safe to release, since libyaml's simple-key mechanism can still splice
a KEY token in front of it, so scanStep() ports libyaml's
yaml_parser_fetch_more_tokens release condition.

Proof: StreamingScannerDifferentialTests.swift drives scanStep() at several
chunk sizes over representative cases always, and over the entire
yaml-test-suite manifest (402 cases) behind PUREYAML_RUN_YAML_TEST_SUITE=1,
asserting the token sequence matches scanAll() exactly. That proof caught a
real bug: lastTokenForbidsInlineSequence() read indentEngine.tokens.last,
which goes empty once tokens are released incrementally, silently breaking
the key: - a rejection. Fixed by tracking IndentEngine.lastToken explicitly,
kept in sync with tokens.last so batch behavior (which never dequeues) is
unchanged.

Not yet wired into PureYAMLStreaming; scanStep() and its supporting types stay
internal until that integration is designed.

Also broaden the real-seed mutation property suite (#6) with the three
mutation classes it named but didn't yet cover: invalidControlCharacter,
mixedTabSpaceIndentation, and truncatedFlowCollection. The test now asserts
diagnostic quality, not just pass/fail: line must fall within the mutated
document, and column, when present, must be 1-based (some ParseError cases,
like tabIndentation, are line-only by design).
2026-07-09 23:38:39 +02:00
1dc986f146 docs(readme): retire the stale Mermaid roadmap and its dead CI gate
The roadmap diagrams referenced issue numbers (#1, #8-#13, #43-#55) that no
longer exist on Codeberg: the GitHub to Codeberg migration reset this repo's
issue tracker, and those numbers were never re-created. scripts/check-roadmap.sh
existed only to keep that roadmap in sync with GitHub epic-labeled issues via
gh, which can't resolve a Codeberg/Forgejo repo anyway, so the gate had been a
silent no-op since the move. Removed the roadmap section, the gate script, its
two call sites, and its now-pointless rule doc, and pointed the remaining
work at the real, live issues (#4, #5, #6, #7, tracked under epic #8). Also
fixed two leftover Mihaela/PureYAML references in docs/RELEASE.md.
2026-07-09 23:38:23 +02:00
8031568858 docs(codeberg): update links for NodetteHQ 2026-07-05 12:59:00 +02:00
630849e67a chore(release): stamp 0.2.0
Section the changelog, add the release notes with the Windows
verification disclosure, update the current-release stamps, and repoint
the release process from the retired GitHub commands to Codeberg.
2026-07-05 00:41:02 +02:00
26bf4a0e00 chore(rules): track shared-rules baseline v1.1.2
The v1.1.1 and v1.1.2 canon changes touch only the canon's leak-gate
script, which this repo does not vendor, so the stamp is the whole fold.
2026-07-05 00:27:44 +02:00
a8950e118c Merge conformance/yaml-test-suite-gaps: the libyaml port becomes the engine on main
The ported engine wins every conflict: main's post-fork patches to the
hand-rolled Scanner and TokenEventParser land on deleted files and are
dropped; their behaviors are carried by the port and guarded by the
whole-suite conformance gate (402/402 passing).

Merge adaptations, all gates green (style, namespacing, forbidden
patterns, changelog, roadmap, swiftformat, swiftlint strict, build,
full test suite, whole-suite conformance sweep):

- PureYAMLJSON representability rules move onto the Check algebra,
  replacing the retired legacy Rule type.
- The permissive tag-handle pre-pass is removed; the port rejects an
  undefined handle at parse time (suite case QLJ7) and the validation
  report carries it as a parse diagnostic.
- EventStreamComposer's nested types split one-per-file per the
  namespacing gate.
- Tests pinned to old-engine behavior were re-verified against the
  reference and updated: libyaml rejects an explicit-key continuation
  ending at a value colon, and suite case QB6E requires rejecting a
  quoted continuation at column 0.
- CHANGELOG unreleased entries from both lines are preserved in two
  groups with a provenance note.

Closes #1, closes #3
2026-07-05 00:23:48 +02:00
e76472da11 docs(conformance): align the suite narrative with the 402/402 manifest
The doc still narrated 329 passing with 73 gaps and listed those gaps as
open, contradicting the manifest. Update the headline to 402/0, name the
manifest as the source of truth over the prose, retire the remaining-gaps
narrative, mark bucket 6 done, and add a closing section recording the
libyaml-port cutover and the final fix series (the hand-rolled engine
files cited in buckets 1-8 are historical; the port is the sole engine).
2026-07-05 00:02:43 +02:00
9b60f415ac chore: repoint github.com/mihaelamj references to codeberg.org/Mihaela
Swap issue-template, usage, and release-note links to Codeberg and
replace the four shields.io GitHub badges with a single Forgejo
Actions badge for ci.yml.
2026-07-04 23:02:59 +02:00
Mihaela
f5fefc7d21 chore(rules): track shared-rules baseline v1.1.0 2026-07-02 21:40:13 +02:00
ebf2e2477c fix(parsing): keep tag-handle diagnostics across a scan-aborting error
The diagnostic accessor ran the scan through `try?`, which discarded the whole
scanner state, and the tag-handle issues collected on it, on any error: the
comment promised the issues found before a scan-aborting error were returned,
but the code returned none. Split the run into a state-capturing `collect` that
returns the partial state with the aborting error, so `scan` still throws while
`tagHandleIssues` reports the diagnostics gathered before the abort. A
regression test pins an undeclared handle followed by an unterminated quote.
2026-06-29 21:43:25 +02:00
e64bff7cdf Merge multi-line flow value fix onto main
A flow mapping value on the line after the colon is now parsed (flow is line
insensitive); fixes all 27 JSON Schema Test Suite files that failed to parse.
2026-06-29 21:41:15 +02:00
d1d5b1a37c fix(parsing): parse a flow mapping value on the next line
In a flow mapping the value is a flow node and may sit on any line; flow is line
insensitive. parseMappingPair gated value parsing on hasNodeOnSameLine, a
block-only rule, so a value after `:` on the next line was treated as null and
left unconsumed, failing the whole flow mapping. Thread an inFlow flag and, in
flow, parse the value as a flow node line-insensitively (only a flow terminator
leaves it null), bypassing the block-only same-line and indentation branches.

Pretty-printed JSON (a YAML 1.2 superset) relies on this. Against the JSON Schema
Test Suite corpus this fixes all 27 files that previously failed to parse
(432/432 now); the YAML test-suite (318/333, value-level 206/206) and the full
suite stay green.
2026-06-29 21:40:12 +02:00
a8c5de540a fix(tests): compile conformance on Linux without CoreFoundation
YAMLTestSuiteConformanceTests resolved a JSON bool vs number with CFGetTypeID and
CFBooleanGetTypeID, CoreFoundation functions absent on Linux and Windows, so the
test target failed to compile there and the Linux and Windows CI gates were red.
Make JSONNode Decodable and read the expected in.json with JSONDecoder, which
distinguishes a bool from a number natively and reads top-level fragments via a
single-value container. Value-level conformance is unchanged at 206/206 on
macOS, and the parse no longer touches CoreFoundation.
2026-06-29 13:31:54 +02:00
5e5e568e2c docs(changelog): record the converged located-parse and conformance work
The located-parse port, per-document %TAG scoping with tag-handle diagnostics,
and the parse-gap fixes landed on main without CHANGELOG entries, and the new
canonical event-stream conformance gate was undocumented. Record them under
Unreleased so the changelog gate passes and the release notes are complete.
2026-06-29 12:21:20 +02:00
b35e2a04f0 build: classify the located-parse public types for coverage
The located-parse port added four public Model types (LocatedValue, LocatedPair,
SourceRange, SourcePosition) that the validation-coverage gate flagged as
unclassified, blocking the push. They are positional projections and pure
position data with no validation invariant of their own: LocatedValue and
LocatedPair wrap an already-validated Model.Value, and SourceRange and
SourcePosition are data-transfer. Classify all four as excluded with reasons.
2026-06-29 12:19:17 +02:00
c1bb5aa8d6 ci: gate conformance on every push, fetching the test-suite
The parse, value, and canonical event-stream conformance suites were opt-in
(PUREYAML_RUN_YAML_TEST_SUITE=1) and so never ran in CI: an oracle that only
runs when asked is not a gate. Add a Conformance workflow that fetches the
canonical yaml-test-suite and runs all three suites, so the event gate's
recorded-gap set fails the build on any regression or silently-closed gap, and
add its README badge alongside the other quality gates.
2026-06-29 12:06:58 +02:00
93621b8c84 test(conformance): add the canonical event-stream conformance gate
Serialize parse events to the official yaml-test-suite yeast format and compare
exactly to each case's test.event: the strongest oracle for the parser, checking
anchors, tags, scalar styles, flow vs block, and aliases for every valid case,
including those without an in.json that the JSON-projection oracle cannot judge.

It runs opt-in (PUREYAML_RUN_YAML_TEST_SUITE=1) against the fetched suite and is
a gate, not a measurement: knownEventGaps records the 31 valid cases whose event
stream is not yet exact, so closing one must remove its id and a regression adds
one. No case is silently skipped. One documented normalization: the event API
does not carry the explicit document-marker flag, so +DOC/-DOC are compared
without their ---/... suffix.

The first run exposed real deviations the parse+value metric missed: 4ABK
truncates http://foo.com at the colon, FH7J and PW8X bind a property to the
collection instead of the empty scalar, 6BFJ binds the wrong anchor.
2026-06-29 10:13:45 +02:00
af4a00f229 Merge parse-gap fixes onto main
Fix three should-parse conformance gaps: a repeated document-end marker (M7A3),
a root block scalar ended by a document marker (W4TN), and an empty explicit key
in a flow mapping (DFF7). Threaded the located parse end mark through the new
explicit-key emit. Parse-level suite conformance reaches 318/333.
2026-06-29 09:15:50 +02:00
ba000d90fa Merge tag-handle diagnostics onto main
Per-document %TAG scoping and an undefinedTagHandle diagnostic via
validationReport, with Scanner.State and the tag helpers in their own files.
Resolved against main's Scanner.State by carrying its sawNonPropertyContentOnLine
field into the extracted State.
2026-06-29 09:14:21 +02:00
9620122759 Merge located parse onto main
Opt-in located parse (parseLocated / parseStreamLocated) on the hand-rolled
engine, with exact scalar end positions. Resolved against main's extracted
parseScalarNode by threading the token end mark through it.
2026-06-29 09:11:07 +02:00
990a2d2e42 fix(parsing): parse an empty explicit key in a flow mapping
An explicit key (`?`) may have no key node: `? : v`, a trailing `? }` / `? ]` /
`? ,`, or `?` at the end of a flow mapping all denote a null key. The parser now
emits an empty key in those positions instead of failing to find a node before
the value indicator or terminator.

Fixes suite case DFF7 (parse-level conformance 316 -> 317); adds tests.
2026-06-29 09:04:29 +02:00
8112954f4c fix(parsing): end a root block scalar at a document marker
A `---` or `...` at column 0 ends an active block scalar rather than being
consumed as its content. The scanner now closes the scalar's indentation at the
marker, dedenting back to the document level, so the token stream stays balanced:
the dedent precedes the documentEnd. Previously the scalar's indent lingered past
the marker and surfaced as a stray late dedent plus a phantom empty document, and
a following directive was misread as a scalar.

Fixes suite case W4TN (parse-level conformance 315 -> 316); adds tests.
2026-06-29 08:57:39 +02:00
ae29059530 fix(parsing): parse a property-only explicit mapping key as an empty key
An explicit key that is only a node property (`? &e` then `: &a`) parsed to an
`expectedNode` error. Unlike the implicit form (`&a : v`), the scanner emits no
synthesized empty scalar for an explicit key, so the block parser read the key
via parseNode, consumed the property, then hit the `:` value indicator with no
node and threw.

parseNode now treats a `:` in node position as an empty key carrying any
properties, reusing emitEmptyNodeOrThrow (the node it already produced for a
property at end of input) without consuming the `:`, so the mapping machinery
reads the `:` as the value indicator. To stay within the cyclomatic-complexity
limit, the scalar case is extracted to parseScalarNode.

yaml-test-suite PW8X now passes (parse-level 314/333 to 315/333). should-error
rejections and value-level matches are unchanged, and a value-level test pins
the produced tree for the anchored and tagged property-only explicit keys.
2026-06-29 08:56:58 +02:00
056095ca3d fix(parsing): allow a repeated document-end marker
A `...` after a document is already closed (only comments or blank lines since
the last `...`) ends nothing: a repeated document-end marker is valid YAML, so
consume it as a no-op rather than rejecting the stream as an unsupported
multi-document case. A second documentEnd is not emitted, so no phantom null
document is composed, and real content after the marker stays an error.

Fixes suite case M7A3 (parse-level conformance 313 -> 315); adds focused tests.
2026-06-29 08:32:26 +02:00
8d245ae335 fix(parsing): parse a property-only block mapping key as an empty key
A block key that is only a node property (`&a : v`, `!!str : v`) parsed to an
`expectedNode` error. The scanner synthesized an empty mapping key for a bare
`: v` line but not when the line's sole content before the `:` was an anchor or
tag, so no key marker was emitted and the parser rejected the `:`.

Track non-property content per line (a leading `&`/`!` does not count, and a
block entry opens a fresh node position) and synthesize the empty key whenever
only properties precede the `:`. The property tokens already bind to the
synthesized empty scalar through the existing mapping machinery, so no parser
change is needed. Relocate the synthesis helper to Scanner.Helpers to keep
Scanner under the file-length limit.

yaml-test-suite FH7J now passes (parse-level 313/333 to 314/333); PW8X still
needs the empty-anchored explicit-value form. should-error rejections and
value-level matches are unchanged, and a value-level regression test pins the
produced tree for each property-only-key shape.
2026-06-29 08:26:50 +02:00
df3d43804c feat(parsing): scope %TAG per document, diagnose undefined handles
A tag shorthand using a named handle (`!handle!suffix`) must be declared by a
`%TAG` directive in the document that uses it; the handle does not carry across
documents (YAML 1.2). The parser was permissive but silent here, so an
undeclared handle passed unnoticed.

- Scope tag handles per document: a `%TAG` handle now applies only to the
  document it precedes (merged onto the defaults at the document start, cleared
  at the boundary) instead of accumulating across the whole stream. A directive
  repeated for each document keeps working, as the suite requires.
- Detect the named-handle form before the primary-handle fallback (which
  previously masked `!prefix!x` as a local tag) and record an undeclared handle.
  The parse stays permissive: it still yields a value, and
  `PureYAML.validationReport` surfaces the issue as an `undefinedTagHandle`
  error. The check reads the scanner's tokens, so it is exact: a `!x!` inside a
  scalar or comment is never flagged.

Extracts `Scanner.State` and the tag helpers into their own files to keep each
source file within the 400-line limit. Adds focused tests; the full suite and
strict lint stay green.
2026-06-29 07:54:55 +02:00
941c3f948a test(parsing): assert an explicit-key compact sequence's value
The explicit-key compact block-sequence case (`? k\n: - a\n  - b\n`) was only
asserted to parse without error, with a comment deferring its value because the
two entries used to fold into a single `"a - b"` scalar. That defect is fixed:
it now parses to `{k: ["a", "b"]}`. Strengthen the test to a value-level
assertion so the correct tree is locked against regression, and drop the
obsolete carve-out comment.
2026-06-29 07:23:57 +02:00
d4ba594012 feat(parsing): opt-in located parse on the hand-rolled engine
Port the located value tree onto the hand-rolled engine, reusing the
engine-agnostic located layer (Model.LocatedValue, LocatedPair,
SourcePosition, SourceRange, and the located composer helper) authored
on the conformance branch. The streaming composer now builds a parallel
located tree behind an opt-in tracksLocations flag, exposed as
parseLocated / parseStreamLocated; the plain Value path stays
byte-identical, gated behind the flag.

The scalar event carries the scanner token's end mark, so a located
scalar spans the whole token (end exclusive, one past the last
character) rather than being start-anchored. The hand-rolled scanner
already tracked token end marks, so this surfaces information the engine
was already computing.

Also stop emitting a spurious null document for a comment-only or empty
stream: YAML 1.2 gives such a stream zero documents, while the
single-document value API still maps it to null, so parse("") is
unchanged.

A whole-suite test proves the located projection equals a plain parse
for every passing case; the value, event-dump, and conformance suites
stay green (507 tests).
2026-06-29 07:14:42 +02:00
5a3bf6518c build(verify): make the local Linux relay a skippable fallback gate
The pre-push gate ran the self-hosted claw relay (ssh + Lima Debian) as a
hard stage, so any time the host was unreachable the whole hook failed and
the only way past it was a blanket --no-verify, which also discarded the
local macOS Swift gate. The authoritative Linux gate is the GitHub-hosted
workflow (ubuntu-latest + swift:6.2); the relay is a convenience fallback.

check-linux.sh now separates connectivity and provisioning (Stage A, exit 77
to signal skip when the relay is unavailable) from the relayed build and test
(Stage B, exit 1 on a genuine Linux failure). check-all.sh treats 77 as a
non-fatal skip and still aborts on any other non-zero, so a runner outage no
longer blocks a green push and never masks a real failure.
2026-06-29 06:55:36 +02:00
942a7ff2bc fix(parsing): keep a leading percent in a block scalar literal
A zero-indent literal or folded block scalar sits at column 0, where the
directive scan runs, so a content line beginning with `%` (`%!PS-Adobe-2.0`)
was misread as a directive and rejected as unsupported.

The directive scan now runs only outside a block scalar, so a leading `%` in
block scalar content is literal text. A real `%YAML` or `%TAG` directive still
parses, and a stray `%` in other content is still rejected rather than silently
accepted, so the should-error behaviour is unchanged.

This is groundwork for yaml-test-suite W4TN, which additionally needs a block
scalar to be terminated by `...` and per-document directives across a multi-
document stream, so conformance counts are unchanged this step.
2026-06-29 05:04:32 +02:00
5ea6fa668b fix(parsing): end a tagged zero-indent sequence at a sibling key
A block sequence value carrying an inline property (`sequence: !!seq` then
`- entry` at the key's column) is a zero-indent sequence: its entries sit at the
mapping key's own indentation, so a following sibling key ends it.

The untagged form was already parsed as a zero-indent sequence that stops at a
sibling mapping key, but the property-bearing form fell through to the generic
node path, which parsed the sequence without that terminator and read the
sibling key as a mixed sequence-and-mapping entry. The zero-indent sequence is
now handled before the generic path, carrying any property to the sequence.

Fixes yaml-test-suite 57H4; raises parse-level conformance to 313/333 (94.0%)
and value-level to 206/206 (100.0%), with the should-error rate unchanged.
2026-06-29 04:48:47 +02:00
4dcedbd6aa fix(parsing): carry spanned properties to a deeper indented node
An inline anchor with a tag on the next indented line, above a mapping indented
deeper still (`key: &anchor` / ` !!map` / `  a: b`), left the node introduced by
a further indent token after the properties were consumed. The parser could not
read a node from that indent and rejected it as a missing node.

The indented mapping value now descends through the deeper indent, carrying the
accumulated anchor and tag to the node it introduces, so the properties attach
to the mapping rather than to an empty value.

Fixes yaml-test-suite BU8L; raises parse-level conformance to 312/333 (93.7%)
and value-level to 205/205 (100.0%), with the should-error rate unchanged.
2026-06-29 04:05:04 +02:00
5528ce9aa4 fix(parsing): bind an indented property to a zero-indent sequence
A node property written on its own indented line can sit above a block sequence
whose entries return to the mapping key's own indentation (`seq:` then
` &anchor` then `- a` at the key's column). The grammar allows a block-in-block
sequence at its parent's indent n while the property sits at n+1, so the
indent's matching dedent falls between the property and the entries.

The parser consumed the property as an empty anchored value and then rejected
the entries as an unexpected block entry. It now recognises the property-then-
dedent-then-entry shape, consumes that dedent, and binds the property to the
block sequence that follows.

Fixes yaml-test-suite SKE5; raises parse-level conformance to 311/333 (93.4%)
and value-level to 204/204 (100.0%), with the should-error rate unchanged.
2026-06-29 03:44:53 +02:00
b45b7a4b7d fix(parsing): accept an alias node as an implicit mapping key
An alias may stand where a mapping key is expected (`*a : value`). The scanner
emits the implicit-key indicator before a scalar key that a `:` follows, but
not before an alias, so the parser, which already accepts an alias node in key
position, never entered the pair and rejected the alias as an unexpected token.

The scanner now emits the same indicator before an alias that a `:` (after
optional spaces) follows, in block context. A `:` written directly against the
alias stays part of its name, since `:` is a valid anchor-name character, so
only a space-separated `:` marks the alias as a key.

Fixes yaml-test-suite E76Z and 26DV; raises parse-level conformance to 310/333
(93.1%) and value-level to 203/203 (100.0%), with the should-error rate
unchanged.
2026-06-29 03:35:56 +02:00
c858339307 fix(parsing): bind an inline first-key property to the key
A property written inline before a block mapping's first key (`&a key: v`) was
attached to the whole mapping, so an anchor registered only when the mapping
ended, too late for an alias inside one of the mapping's values to resolve.

The YAML grammar requires a block collection's own properties to be followed by
a line break (`s-l+block-collection`), so a property on the first key's line
belongs to the key, not the collection. The mapping now keeps only a property
written on its own line above the first key; an inline one is threaded back to
the first pair, where the key parses it and an anchor registers with the key.

Fixes yaml-test-suite 2SXE (`&a:` and `&a` are distinct anchors, since a name
includes a trailing `:`) and HMQ5 (a key carrying both a tag and an anchor).
Raises parse-level conformance to 308/333 (92.5%) and value-level to 201/201
(100.0%), with the should-error rate unchanged.
2026-06-29 03:24:30 +02:00
e3caa5be0d fix(parsing): resolve an alias to an anchor on a null value
An anchor on an empty mapping value (`a: &anchor`) failed to register, so a
later `*anchor` errored as an undefined alias. Two bugs compounded:

  - The parser dispatched the anchor-only value position to a nested block
    mapping, absorbing the sibling key as the anchored node's child instead of
    emitting the empty anchored node. It now emits an empty anchored node when
    only a sibling mapping key follows, since the indented and inline-nested
    cases are already handled by earlier branches.
  - The stream composer registered an anchor by its raw scalar text, so an
    anchor on an empty scalar came back as "" and a typed scalar (`&a 42`) as
    text. It now registers the composed value, fixing a latent typed-anchor bug.

Fixes yaml-test-suite 6KGN; raises parse-level conformance to 306/333 (91.9%)
and value-level to 199/199 (100.0%), with the should-error rate unchanged.
2026-06-29 02:56:17 +02:00
7879818ce5 fix(parsing): accept a higher-minor %YAML version
The %YAML version check accepted only 1.1 and 1.2, rejecting anything else,
so `%YAML 1.3` errored. The spec calls for processing a document whose minor
version is higher than the processor's, with a warning, rather than refusing
it: only a different major version is incompatible.

Accept any well-formed `1.minor` version and reject only a different major
(`2.0`) or a malformed version. Three tests that asserted `%YAML 1.3` was an
error are corrected to use `2.0`, the genuinely incompatible case, and a test
covers that 1.3 now parses.

Raises yaml-test-suite parse-level conformance to 305/333 (91.6%) with the
value-level (100%) and should-error rejection rates unchanged. 486 tests pass.
2026-06-29 02:34:55 +02:00
4b06f6e5fc fix(parsing): end an inline item value at a sibling block entry
A block entry with an inline node recorded only the node's indent (one past
the dash) as a valid implicit level, not the dash's own indent. So a plain
scalar that is an inline sequence item value could not tell a sibling entry
that returns to the sequence indent from its own content, and a compact nested
sequence as an explicit key's value folded into one scalar: `? a` / `: - b` /
`  - - c` / `   - d` became `{a: [b -  - c - d]}` instead of `{a: [b, [c, d]]}`.

Record the entry's own indent as a sibling level too, so the continuation
stops there. Skip it under a pending explicit key, whose own sequence key
drives the implicit indentation differently (an explicit key whose key is a
sequence, `? - x`, must keep parsing). A block entry more indented than its
container is still plain content, so a wrapped value beginning a line with
`- ` folds in. The block-entry recording moved to a helper to keep the file
within its length limit.

Brings yaml-test-suite value-level conformance to 197/197 (100.0%) with the
parse-level and should-error rejection rates unchanged. 485 tests pass.
2026-06-29 02:09:57 +02:00
3b504a9c3a fix(parsing): keep a tab on a leading blank of a block scalar
A blank line within a block scalar keeps any whitespace past the content
indentation as content, but the leading blank lines, those before the first
content line, were only counted and emitted as empty. So an otherwise-blank
`tab` line under a one-space content indent lost its tab, and a folded scalar
beginning with such a line dropped the leading tab.

Record each leading blank line's whitespace verbatim instead of a count, and
once the content indentation is established emit the surplus past it, the same
way an interior blank line does. A leading blank with no surplus still emits an
empty line, and an empty block scalar (no content indentation) still emits
empty lines for its blanks.

Raises yaml-test-suite value-level conformance to 196/197 (99.5%) with the
parse-level and should-error rejection rates unchanged. 482 tests pass.
2026-06-29 01:52:58 +02:00
31da488055 fix(parsing): allow a flow mapping entry with no value
A flow mapping entry may omit its `:` value indicator, in which case its key
maps to null: `{ a }` and `{ a, b }` are flow sets. parseFlowMappingPair
required a `:` after every key, so it rejected these entries, and rejected a
valueless key that spanned lines (`{ multi\n  line, a: b}`) with them.

Parse a value only when a `:` follows the key; otherwise emit a null value for
the key. A unit test that asserted `{ a, }` was an error is corrected to expect
the set `{a: null}`, the behaviour the yaml-test-suite cases 8KB6 and 9BXH
require.

Raises yaml-test-suite parse-level conformance to 304/333 (91.3%) and
value-level to 195/197, with the should-error rejection rate unchanged.
480 tests pass.
2026-06-29 01:42:02 +02:00
5c99086a38 fix(parsing): let a flow plain key span lines
A plain scalar continuation stops before a line ending at `:`, since a block
implicit key may not span lines and the next line is a new key. A flow key may
span lines, so the same rule wrongly split `{ multi\n  line: v }`: the scalar
returned just `multi` and the parser mis-read `line: v` as a separate entry.

Fold across the `:` in flow as well, leaving the `:` to be scanned as the
value indicator, so the key folds to `multi line`. A block implicit key keeps
the previous behaviour and still stops before the `:`.

Raises yaml-test-suite parse-level conformance to 301/333 (90.4%) and
value-level to 193/195, with the should-error rejection rate unchanged.
476 tests pass.
2026-06-29 01:38:33 +02:00
95c5ff3966 fix(parsing): parse an explicit key inside a flow collection
A flow `?` did not mark an explicit key, so the scalar after it synthesized
its own implicit key: `{? foo : bar}` parsed as `{{foo: bar}: null}` instead of
`{foo: bar}`, and `[? foo : bar]` the same way.

Track the explicit-key state per flow level on the flow context: set it at a
flow `?`, clear it at that level's `:` or a `,`, and discard it with the
context when the collection closes. The key scalar then is not turned into an
implicit key, the pair's `:` is its value indicator, and a nested collection's
`:` does not affect an outer key (`{? {a: b} : c}`). The continuation fold for
a multi-line explicit key now also consults the flow context, restoring
`[? foo\n bar : baz]` to the single key `foo bar`.

The block explicit-key flag is left untouched, since it is not cleared between
valueless `? a` / `? b` entries and folding it into this check regressed
block sets; only the flow path is changed here.

Raises yaml-test-suite value-level conformance to 192/194 (99.0%) with the
parse-level and should-error rejection rates unchanged. 472 tests pass.
2026-06-29 01:17:26 +02:00
143624be08 fix(parsing): fold an explicit key ending at its value colon
A plain scalar continuation that ends at a `:` is normally a new implicit
mapping key, so the scalar stops before it. Under an explicit key (`? foo`
then `bar :`) the `:` is the key's own value indicator instead, so the line is
still key content. The scalar dropped that line and the parser stalled, so
`? foo` then `bar : baz` failed rather than yielding the key `foo bar`.

When the explicit-key marker is pending, fold the continuation line in and
stop, leaving the `:` to be scanned as the value indicator. An implicit key
keeps the previous behaviour, so a continuation ending at `:` still begins a
new key.

The test-suite rates are unchanged, since no comparable case isolates the
block form and the flow form needs separate explicit-key work, but block
explicit keys spanning lines now fold correctly. 466 tests pass.
2026-06-29 00:59:18 +02:00
8bc855d7a0 fix(parsing): base an inline-mapping indent indicator on the mapping
An explicit block scalar indentation indicator is the content indentation
relative to the block scalar's parent node. For an inline mapping value
(`- aaa: |2`) the parent is the mapping, but a compact inline mapping never
pushes its indentation onto the main stack, so the indicator was added to the
stack level (0) rather than the mapping's (2). The content indentation then
came out too small and the next mapping key was pulled into the scalar.

Capture at the header whether the block scalar is an inline mapping value
(a mapping value was seen on its line). When it is, fold the deepest
applicable implicit indentation into the indicator's base; a direct sequence
item (`- |1`), which has no mapping value on its line, keeps the sequence's
own level, so the explicit indicators on sequence items stay correct.

Raises yaml-test-suite value-level conformance to 191/194 (98.5%) with the
parse-level and should-error rejection rates unchanged. 463 tests pass.
2026-06-28 23:24:21 +02:00
a33be7c37a fix(parsing): bound a block sequence to its own column for siblings
A compact nested block sequence (`- - x`) carries no indent or dedent tokens,
only block entries at their source columns. The sibling-entry loop in
parseBlockSequence collected every following block entry by kind alone, so the
inner sequence absorbed an entry that belonged to an enclosing one: `- - x`
then `- - y` parsed as a single outer item `[[x, [y]]]` instead of two items
`[[x], [y]]`.

Record the first entry's column as the sequence's column and continue the
sibling loop only while the next entry sits at it. An entry at a smaller
column ends this sequence and is handled by the enclosing one; entries at the
same column still group, and flat sequences are unchanged.

Raises yaml-test-suite value-level conformance to 190/194 (97.9%) with the
parse-level and should-error rejection rates unchanged. 460 tests pass.
2026-06-28 22:37:36 +02:00
2b3d95d214 fix(parsing): chomp the blank lines of an empty block scalar
A block scalar whose only lines are blank never establishes a content
indentation, so its blank lines stayed in the leading-blank buffer and were
discarded when the scalar ended. `keep: |+` over a blank line then resolved to
the empty string rather than the line break keep chomping should preserve.

At stream end, flush the buffered blank lines as content wrapped in an
indent/dedent pair, so the parser builds a block scalar of those lines and
chomping applies: keep preserves the breaks, clip and strip trim them. Only
the stream-end case is handled, where no following line constrains token
ordering; an empty keep scalar before a sibling key is left for later.

Raises yaml-test-suite value-level conformance to 189/194 (97.4%) with the
parse-level and should-error rejection rates unchanged. 456 tests pass.
2026-06-28 22:31:09 +02:00
64f3f71211 fix(parsing): keep a non-core-tagged scalar as a string
A node with an explicit tag is not subject to implicit plain-scalar
resolution, but composeScalarValue fell through to it whenever the tag was not
a recognized core type. So a local tag (`!foo`), the non-specific `!`, or an
unknown verbatim URI still let `! 12` resolve to the integer 12 and
`!Thing false` to a boolean.

When a tag is present but not a core type, fix the node as the string of its
content instead of resolving it. Recognized core tags and untagged scalars
are unchanged. Two compatibility tests asserted the old behavior (a custom-
tagged `true` as a boolean); they are corrected to expect the string, the
behavior the yaml-test-suite case S4JQ requires.

Raises yaml-test-suite value-level conformance to 188/194 (96.9%) with the
parse-level and should-error rejection rates unchanged. 452 tests pass.
2026-06-28 22:25:48 +02:00
6c322fc5ed fix(parsing): end a plain scalar at a tab before a comment
A comment opens after white space, a tab as well as a space, but the plain
scalar terminator checked only for a space before `#`. So `c: d\t#X` kept the
tab and hash as content, giving the value `d\t#X` rather than `d` with a
trailing comment.

Treat a tab before `#` as a space there, in both the lookahead from a white
space character and the check on the accumulated value's last character. A tab
inside a plain scalar with no `#` after it stays content, and a `#` glued to
content with no white space is still content.

Raises yaml-test-suite value-level conformance to 187/194 (96.4%) with the
parse-level and should-error rejection rates unchanged. 448 tests pass.
2026-06-28 21:24:59 +02:00
37ac4d862d fix(parsing): keep both breaks at a folded blank by a more-indented line
In a folded block scalar, a blank run of N lines between two non-more-indented
lines folds to N line breaks: the fold absorbs the first break. Next to a
more-indented line nothing folds, so that break survives and the run yields
N+1 breaks. The folder used N in every case, so paragraph spacing around an
indented block collapsed: a blank line before a more-indented block came out
as one break instead of two.

Track whether the line before the blank run, or the one after it, is
more-indented and add the extra break in that case. Folding between plain
lines and runs with no blank lines are unchanged.

Raises yaml-test-suite value-level conformance to 186/194 (95.9%) with the
parse-level and should-error rejection rates unchanged. 444 tests pass.
2026-06-28 21:14:14 +02:00
246e6b1891 fix(parsing): fold tabs in plain scalar continuation lines
Plain scalar line folding strips white space around continuation lines, but
the scanner stripped only spaces, so tabs leaked in. A tab is not indentation,
so the indentation scan left a continuation line's leading tab in place and it
joined the value (`\t3rd` folded as `\t3rd`, not `3rd`), and a line of only
white space holding a tab (`  \t`) was treated as a non-blank continuation,
folding to a space rather than the line break an empty line gives.

Skip leading white space, tabs included, before a continuation line's content,
and treat a whitespace-only line that holds a tab as blank when counting the
folded line breaks.

Raises yaml-test-suite value-level conformance to 183/194 (94.3%) with the
parse-level and should-error rejection rates unchanged. 441 tests pass.
2026-06-28 21:00:32 +02:00
4b0dc9375e fix(parsing): keep tabs in block scalar content verbatim
A block scalar is indented with spaces only, so a tab past the content
indentation is literal content. The indentation scan measured the leading
whitespace as a tab-expanded width and rebuilt the surplus beyond the
indentation as spaces, so a line like `␣\ttext` under a one-space indent came
out `   text` rather than `\ttext`: the tab became three spaces.

Record the literal leading-whitespace characters during the indentation scan
and reconstruct the surplus from them, walking columns the way the width scan
does and keeping the remainder verbatim. A tab stays a tab; surplus spaces
stay spaces; the blank-line surplus path uses the same reconstruction.

Raises yaml-test-suite value-level conformance to 180/194 (92.8%) with the
parse-level and should-error rejection rates unchanged. 438 tests pass.
2026-06-28 20:53:40 +02:00
67d72c8b15 fix(parsing): skip tabs as mid-line separation white space
skipSeparation consumed only spaces between tokens, so a tab after an
indicator leaked into the following node: an explicit key written `?\tkey`
became the key `\tkey` rather than `key`. A tab is separation white space in
YAML, forbidden only as indentation, which the line-start scan already handles
before this point.

Skip tabs there too. This closes the post-indicator tab leak left open when
the indicator boundary itself was changed, and handles a run of tabs between a
`:` and its value. The test-suite rates are unchanged, since no comparable
case isolates this, but the parser now matches the spec on tab separation.

435 tests pass.
2026-06-28 20:35:04 +02:00
738408d458 fix(parsing): accept a tab after the colon value indicator
A tab is separation white space in YAML, so it follows the `:` mapping-value
indicator exactly as a space does. isMappingValueBoundary (flow) and
isBlockMappingValueBoundary (block) accepted only a space, line break, comma,
or closing bracket, so `foo:\tbar` folded into the plain scalar `foo:\tbar`
instead of the mapping `{foo: bar}`.

Accept a tab in both. A `:` with no following white space (`foo:bar`) is still
plain scalar content, and the symmetric counterpart for a tab after a leading
indicator landed in the previous change.

Raises yaml-test-suite parse-level to 300/333 (90.1%) with the should-error
rejection rate unchanged. 433 tests pass.
2026-06-28 20:30:24 +02:00
37e84b5eb4 fix(parsing): accept a tab as a leading indicator boundary
A tab is separation white space in YAML, forbidden only for indentation, so it
ends a leading indicator exactly as a space does. isIndicatorBoundary treated
only a space, line break, or end of input as a boundary, so a document marker,
block entry, or explicit key followed by a tab was folded into a plain scalar:
`---\tscalar` parsed to the string `---\tscalar` rather than a document whose
root scalar is `scalar`.

Accept a tab there too. The document-marker and block-entry forms also strip
the tab as separation, so the content is clean; the explicit-key form is now
recognized as well, with post-indicator tab stripping left for a later change.

Raises yaml-test-suite value-level conformance to 175/192 (91.1%) with the
parse-level and should-error rejection rates unchanged. 429 tests pass.
2026-06-28 20:19:16 +02:00
c1610f1eec fix(parsing): treat colon-led flow sequence element as empty-key map
A flow sequence element that begins with `:` is a single-pair mapping with an
empty key, e.g. `[ : value ]` is `[{null: value}]`, but parseFlowSequence ran
the element through parseNode, which found the leading `:` and raised
"expected a YAML node".

Detect the leading mapping-value indicator and emit a single-pair flow mapping
with a null key and the following value (itself possibly empty), the sequence
counterpart of the scalar-key element `[ a: b ]` that already becomes a
mapping through its implicit-key marker. Plain elements and scalar-key mapping
elements are untouched, and the allowance is confined to flow sequence element
position.

Raises yaml-test-suite parse-level to 298/333 (89.5%) with the should-error
rejection rate unchanged. 426 tests pass.
2026-06-28 19:43:56 +02:00
956f42bbb7 fix(parsing): allow an empty key in a flow mapping
A flow mapping entry may begin with its `:` indicator for an empty key
(`{ : value }`) or carry only properties before the `:` (`{ !!str : value }`).
Both resolve to a null key, but parseFlowMappingPair required a node before
the `:` and raised "expected a YAML node".

Emit a null key in that position instead, mirroring the empty-value handling
already present after the `:`. The change consumes the key's properties first
so a property-only key keeps its tag or anchor, and is confined to flow
mapping key position, where a null key is unambiguously valid; an entry with
no `:` at all (`{ a, }`) still errors.

Raises yaml-test-suite parse-level to 297/333 (89.2%) and value-level to
174/192, with the should-error rejection rate unchanged (no invalid document
newly accepted). 422 tests pass.
2026-06-28 19:23:49 +02:00
5a0e6c732e fix(parsing): parse a flow collection before a colon as a mapping key
A flow collection followed by a same-line `: ` indicator is an implicit block
mapping key, e.g. `[a, b]: value` or `{a: 1}: value`, but the parser took the
flow node as the document's root value and then rejected the trailing `:` with
"expected document boundary".

The marker that tells the parser a node is a key is emitted before the node,
so for a flow key it must precede the flow tokens, which are already drained
by the time the `:` is seen (the streaming cursor drains after each token).
Resolve it in the scanner with a character lookahead from the flow opener to
its matching close, skipping quoted scalars so their brackets and colons do
not miscount the nesting; when a same-line `:` boundary follows, emit the same
implicit-key marker a scalar key gets, so the flow node parses as the key
through the ordinary mapping path with no parser change.

A block implicit key must lie on one line, so a flow collection spanning a
line break before its `:` (`[23 / ]: 42`) is rejected, which a first cut
wrongly accepted until the lookahead was made to stop at a line break. A flow
node that no `:` follows, or that is a mapping value, is unaffected.

Raises yaml-test-suite parse-level to 294/333 (88.3%) with the should-error
rejection rate unchanged. 416 tests pass.
2026-06-28 18:34:11 +02:00
a5b8986d5e fix(parsing): fold root block scalars with column-0 content
A block scalar that is the document's own root node (`>`, `|`, or after
`---` on the same line) may begin its content at column 0: the document node's
indentation is -1, so column-0 content is still more indented than its parent.
The scanner read the base level as 0 and required content strictly deeper than
it, so the lines fell through indentation handling and were mis-scanned as a
stray plain scalar after the document, raising "expected document boundary".

A root block scalar and an under-indented nested one (`- |` then unindented
content) present identical indentation state at the first content line, so the
two cannot be told apart there. Capture the distinction at header-scan time
instead: a root block scalar has the base indentation and no mapping value,
explicit key, block-entry implicit indent, or flow context introducing it on
its line. Only then is its parent the document; a sequence item or mapping
value keeps its parent's indent as the floor, so its under-indented content is
still rejected, unchanged.

Raises yaml-test-suite parse-level to 291/333 (87.4%) and value-level to
173/191 (90.6%). A regression suite covers the root cases (including a leading
'#' as literal content) and the nested under-indented cases that must error.
410 tests pass.
2026-06-28 17:42:38 +02:00
8f87a1ecd0 test(validation): extend the finding-coverage law to both layers
The finding-coverage meta-test governed only the model validator's default
checks. Bring the tagged validator's defaults under the same law, and make
the law precise as either layer grows past one check.

Each check now runs in isolation, in a validator holding only it, built from
the check's own AnyCheck. A fixture therefore cannot lean on a sibling check
to fail, which the previous model test (running the whole default validator)
would have allowed once a second model check shipped. Set-equality asserts
the fixtures cover exactly each layer's default checks.

The tagged set ships failing and near-miss fixtures for both built-in checks:
duplicate versus one-character-apart keys, and a built-in tag on the wrong
node kind versus a supported, kind-consistent one. A planted orphan fixture
was confirmed to fail both the set-equality and the isolation tests, then
removed. 405 tests pass.
2026-06-28 17:10:03 +02:00
47960a85c8 feat(validation): enforce tagged mapping-key uniqueness
The coverage manifest claimed Tagged.Mapping and Tagged.Node validate key
uniqueness, but the tagged validator's only default check was the built-in
tag check: the claim was false. Tagged.Mapping keeps every parsed pair, so a
duplicate key survives into the tree and is as invalid as in the plain model.

Add Tagged.BuiltinCheck.mappingKeysAreUnique, a Check<Mapping, Node> on the
one validation framework, to the tagged validator's defaults; the traversal
now offers each mapping to the checks. It reports each duplicate at its key
path, with failing, near-miss, and nested fixtures, and the two default
tagged checks are pinned by description.

Correct the Model.Value coverage reason: merges and aliases resolve at parse
time, so they are not a validation-phase invariant. Every validated manifest
entry now names an invariant a check actually enforces. 403 tests pass.
2026-06-28 17:03:20 +02:00
d195dd3fa1 refactor(validation): retire the legacy Rule for one check type
A Rule was exactly a Check<Model.Value, Model.Value>: it fired on every
node, switched on the value itself, and carried the root. It was therefore a
second copy of the one check machinery, the duplication the validation idiom
exists to prevent (principle 10), now also resolved for the tagged layer.

Remove Rule and its Context. The model validator holds only checks; its
default is the type-directed BuiltinCheck.mappingKeysAreUnique; validating
takes a Check. Custom whole-tree rules become Check<Model.Value, Model.Value>
with the former context.root now context.document.

Behaviour is byte-identical: the mapping-key builtin reports the same issues
at the same paths in the same DFS order, so the public validate/collect/blank
surface is unchanged. All 399 tests pass; the coverage gate reconciles to 60
public types.
2026-06-28 16:43:31 +02:00
fc1a6db1dc refactor(validation): fold the tagged validator onto one framework
The tagged-node validator carried its own copy of the validation machinery:
a Tagged.Rule cloning Validation's check type and a Tagged.Context cloning
its context, differing only in subject type. That is the per-layer
duplication the validation idiom exists to prevent (principle 10).

Generalise the type-directed framework over both the subject and the
document: SubjectContext, Check, and AnyCheck now take a Document type
parameter for the validated tree's root. The model layer validates with a
Model.Value document, the tagged layer with a Tagged.Node document, each
truthful rather than forcing one root type to stand in for the other.

The tagged layer keeps only what is genuinely its own: the Node document
type, its built-in check (now Tagged.BuiltinCheck, a Check<Node, Node>), and
its traversal. Tagged.Rule and the tagged Context are removed. The validator
surface is unchanged but validating now takes a Check<Node, Node>.

All 399 tests pass; the coverage gate reconciles to 62 public types.
2026-06-28 10:41:10 +02:00
f901a3f04c test(validation): make finding coverage a tested law
Add the meta-test the validation idiom mandates: every rule in the default
validator ships a failing fixture and a near-miss (just-valid) fixture, kept
in a registry keyed by rule description. One meta-test asserts the registry
covers exactly the default rule set, naming any default rule without
fixtures or any orphaned fixture; another exercises each rule in isolation,
failing its failing fixture and passing its near miss so nothing proves a
rule quiet on valid input by accident.

A planted orphan fixture was confirmed to fail the set-equality meta-test
before being removed. Adding a default rule without its fixtures now fails a
test before the change is deliberate.
2026-06-28 10:17:48 +02:00
1915a00fde feat(validation): add the combinator algebra for checks
Add the declarative combinators the validation idiom composes rules from,
as free functions returning closures: comparison operators (==, !=, <, <=,
>, >=) lifting a key path and a literal into a predicate; take to dig to a
value and run arbitrary logic; all to apply many checks to one context; lift
to run child-typed checks against a value reached by key path; and && / ||
to compose two checks (concatenating their errors, or short-circuiting so
errors surface only when both fail). Both the context-rooted and
subject-rooted key-path forms are provided so an author can pin the subject
type in either clause.

A key path is immutable and read-safe but not declared Sendable, so a
@Sendable predicate captures it through a small @unchecked Sendable box.
lookup is omitted: it resolves references against a component store the
generic model does not carry; a consumer with references supplies it.

Each combinator has a pure-function test against a hand-built context, per
the idiom's requirement to unit test the operators in isolation.
2026-06-28 09:27:17 +02:00
2e45ecaffe feat(validation): add type-directed Check dispatch
Introduce the core of the validation-rules idiom: a Check<Subject> with a
positive description, a when predicate, and a Bool or multi-error check
that fires only on values of its Subject type wherever they occur. AnyCheck
erases the subject and dispatches by runtime type, guarding an optional or
a wrong type from matching. The validator gains a checks list, a
validating(_:) overload, and a traversal that offers each node the values
it may specialise on. BuiltinCheck.mappingKeysAreUnique is the
type-directed counterpart of the legacy rule.

The machinery has its own negative tests (wrong type, optional, false
predicate yield nothing; same type twice yields two errors) and the builtin
ships a failing and a near-miss test. The coverage gate catches the five
new public types, which are classified as validation machinery. The default
validator is unchanged, so this is additive; retiring the legacy Rule,
moving the default onto the check, the combinator algebra, the error
rendering, and de-duplicating the Tagged validator are the remaining steps.
2026-06-28 07:23:23 +02:00
d3b4fab0d9 feat(validation): configure the validator by rule identity
Move the validator toward the validation-rules idiom's identity-based
configuration. withoutValidating(_:) removes a rule by its description, the
removal identity the idiom uses, so descriptions must be unique and stable.
validationDescriptions lists the active rules in application order.

A configuration-pin test asserts the exact descriptions of the blank and
default validators and that the default descriptions are unique, so adding,
removing, or rewording a rule is a deliberate change that a test catches.
Both additions are non-breaking; the type-directed Validation<Subject>
dispatch and the error-rendering change remain the larger migration steps.
2026-06-28 05:18:06 +02:00
85663682a2 build: enforce validation coverage over every public type
Adopt the validation-rules discipline that every public type is either
validated or excluded, enforced mechanically rather than by review. A new
gate enumerates the public surface from the compiler symbol graph and
fails the build, naming the type, if any public type is missing from the
coverage manifest or any manifest entry is no longer public.

scripts/validation-coverage.tsv classifies all 58 public types: the value
models with the mapping-key-uniqueness runtime invariant are validated,
and the namespaces, closed sum types, data-transfer values, error types,
and engine machinery are excluded with a one-line reason each. The gate
runs in check-all.sh and CI; a planted unclassified type was confirmed to
fail it.

This is the coverage-first phase of moving the validation layer to the
type-directed Validation idiom; the framework backing the validated
entries, the de-duplication of the Tagged validator machinery, and the
exhaustive tests follow.
2026-06-28 05:00:54 +02:00
2501cacd19 fix(parsing): parse a bare document after a document-end marker
Content after a '...' document-end marker begins a new bare document, but
the scanner rejected it as an unsupported multi-document stream and the
parser expected the document to end at the marker. A stream like
'--- scalar / ... / key: value' or 'a / ... / b' is two documents.

The scanner now reopens the stream when content follows '...', and
parseDocument returns at a '...' so the outer stream loop parses the next
document, bare or marked. A trailing '...' with nothing after it still
leaves a single document. The single-document composer still rejects a
multi-document stream, so 'parse' is unchanged. Two tests that asserted
content after '...' is an error are corrected, since the construct is valid
per the grammar and the official test-suite.

Measured against the official YAML test-suite, parse-level conformance
rises from 288/333 (86.5%) to 289/333 (86.8%) with no regression.
2026-06-28 04:23:06 +02:00
3291a3701f fix(parsing): treat a dangling anchor or tag as an empty node
A node property always attaches to a node, and an empty node is allowed, so
an anchor or tag with nothing after it (a bare '&anchor', a 'key: &a', or a
'!!str'/'!!null' with no value) denotes that property applied to a null
node. parseNode rejected it as a missing node.

parseNode now emits a null scalar carrying the properties when a node
position holds only an anchor or tag, via a small helper to stay within the
complexity limit; with no properties, a missing node is still an error. A
'!!str' empty node resolves to the empty string and '!!null' to null. The
error-path test that asserted a dangling property throws is corrected, as
the construct is valid per the grammar and the official test-suite.

Measured against the official YAML test-suite, parse-level conformance
rises from 287/333 (86.2%) to 288/333 (86.5%), with value-level up one and
no regression.
2026-06-28 03:21:04 +02:00
4941eac711 fix(parsing): parse implicit null mapping keys
A ':' that opens a line with no key before it is a mapping entry with a
null key (': value', a bare ':', or a null-keyed entry among others). The
scanner emitted the ':' with no preceding key, so the parser rejected it
as a missing node.

The scanner now synthesizes the empty key and its node for such a ':'. To
avoid mistaking an explicit key's value indicator for a null key, it tracks
a pending block-level '?' key: a ':' opening a line is that key's value
when one is pending, and only a synthesized null key otherwise. The flag
is set and cleared only at flow depth zero, so a ':' inside a flow complex
key ('? {a: b}' then ': c') does not disturb the pending block key.

Measured against the official YAML test-suite, parse-level conformance
rises from 283/333 (85.0%) to 287/333 (86.2%) with no regression.
2026-06-28 02:53:45 +02:00
7e89ba1877 fix(parsing): allow a directive after a document-end marker
A directive following a document-end marker ('...') begins the next
document's directive section, but the scanner rejected it: the
isDocumentClosed flag set by '...' tripped the directive guard, so a
multi-document stream where each document carries its own directives
('%TAG ...' / '--- doc' / '...' / '%TAG ...' / '--- doc') failed.

A directive now reopens the stream when it follows '...', clearing the
closed flag, while a directive after document content within the same
document stays rejected via the content guard. The directive still
requires its own '---', so a directive after '...' with no document start
is still an error. Multi-document streams already composed through the
stream mode; this removes the scanner-side block on per-document
directives.

Measured against the official YAML test-suite, parse-level conformance
rises from 278/333 (83.5%) to 283/333 (85.0%) with no regression.
2026-06-28 02:09:01 +02:00
25b1388dbb fix(parsing): reject a later under-indented property on a mapping value
The value-property indentation check used only the first property's mark,
so it caught a sole under-indented property ('seq:' then '&anchor') but not
a second one after an inline first ('key: &x' then a '!!tag' in column 1),
which still parsed.

The last consumed property is cursor.previous after consumeProperties, so
the check now uses it instead of the first mark, catching an under-indented
property whether it is the only one or a later one. Inline properties, a
property indented under the key, and a properly indented anchor-and-tag pair
are unaffected.

Measured against the official YAML test-suite, parse-level conformance
rises from 277/333 (83.2%) to 278/333 (83.5%) with no regression.
2026-06-28 01:44:48 +02:00
d79df8c4bc fix(parsing): reject a block entry after a flow collection on a line
A block sequence entry cannot follow a flow collection that closed earlier
on the same line ('- { y: z }- invalid', '- [a] - b'): the entry would
begin a sequence inline after a complete node. The scanner accepted it,
producing extra sequence items.

The scanner now records when a flow collection closes at the top level on
the current line and rejects a following block entry. The flag resets at
each line break, so a flow collection on its own line is unaffected, and a
nested block entry ('- -') has no preceding flow close, so it stays valid.

Measured against the official YAML test-suite, parse-level conformance
rises from 276/333 (82.9%) to 277/333 (83.2%) with no regression.
2026-06-28 01:34:55 +02:00
ecf634fdf7 fix(parsing): reject an under-indented property on a mapping value
A mapping value's anchor or tag on the line below the key must be indented
past the key. The parser consumed such a property regardless of its column,
so 'seq:' then '&anchor' in column 1, anchoring a sequence at the document
level, parsed instead of failing.

When the value's properties are consumed before the indent check, a
properly indented one would have been preceded by an indent token and
consumed inside the indented value; reaching that point with a property on
a later line therefore means it sits at the key's indentation, which is
rejected. The indented-value branch and the new guard move to their own
helpers so parseMappingPair stays within the complexity and length limits.
An inline property, an indented property, and a block sequence at the key's
indent are unaffected.

Measured against the official YAML test-suite, parse-level conformance
rises from 275/333 (82.6%) to 276/333 (82.9%) with no regression.
2026-06-28 01:28:00 +02:00
30dbd9316b fix(parsing): reject a lone dash before a flow indicator
In a flow collection a '-' that begins a plain scalar must be followed by
a non-indicator character. Immediately before a flow indicator ('[-]',
'[-, -]', '{a: -}') the dash is not a valid scalar, but the scanner
accepted it as the scalar '-'.

scanToken now rejects a '-' in flow context when the next character is a
flow indicator, before falling through to plain-scalar scanning. A dash
followed by ordinary content stays valid: negative numbers, a dash inside
a scalar, a dash starting a word, and a quoted dash are unaffected.

Measured against the official YAML test-suite, parse-level conformance
rises from 273/333 (82.0%) to 275/333 (82.6%) with no regression.
2026-06-28 01:00:57 +02:00
f1db1344da fix(parsing): reject an over-indented leading line in a block scalar
A block scalar's leading empty lines must not be more indented than its
first non-empty line, so a blank line wider than the content (a 5-space
blank above 2-space content, or 2,3-space blanks above 1-space content) is
malformed. The scanner buffered the leading blanks without comparing their
indentation to the content, folding such input into the value.

The scanner now records the widest leading blank and rejects the scalar
when it exceeds the content indentation, established for both auto-detected
and explicit indicators. Indentation is counted in leading spaces, not
tab-expanded width, so a tab on a blank line ('␣\t') is one space rather
than a jump to the next tab stop, keeping such valid block scalars intact.

Measured against the official YAML test-suite, parse-level conformance
rises from 270/333 (81.1%) to 273/333 (82.0%) with no regression.
2026-06-28 00:21:57 +02:00
c1dda8bbab fix(parsing): reject a document marker inside a multi-line scalar
A line that begins with a document marker ('---' or '...') followed by a
boundary is forbidden inside a multi-line scalar, since it is ambiguous
with a document boundary. A multi-line quoted scalar folded such a line
into its value instead of rejecting it.

scanQuotedScalar now checks, on crossing each line break, whether the next
line begins a document marker, and rejects the scalar if so. The check
requires the marker at the line start with a following boundary, so inline
dashes ('a --- b'), '---' with no boundary ('---b'), an indented '---',
and dots not at a line start stay ordinary content.

Measured against the official YAML test-suite, parse-level conformance
rises from 268/333 (80.5%) to 270/333 (81.1%) with no regression.
2026-06-27 23:20:52 +02:00
28f549311f fix(parsing): reject a block sequence inline after a node property
A block collection must begin on the line after its node properties, so a
block sequence whose first entry sits on the same line as its anchor or tag
('&a - x', '!!seq - x') is malformed. The parser accepted it, anchoring or
tagging the inline sequence.

parseBlockSequence now rejects a sequence whose first block entry shares a
line with the anchor or tag that introduced it, via a small helper to stay
within the complexity and brace-style limits. An anchor or tag on its own
line above the sequence, inside a sequence item ('- &a x'), before a flow
sequence ('&a [x]'), or on a scalar is unaffected; the mapping form is
left alone, since '&a key: value' is a valid anchored key.

Measured against the official YAML test-suite, parse-level conformance
rises from 267/333 (80.2%) to 268/333 (80.5%) with no regression.
2026-06-27 23:14:54 +02:00
779ebe596d fix(parsing): reject an anchor or tag on an alias node
An alias is a bare reference and cannot carry node properties, so '&b *a'
(an anchor on an alias) and a tag on an alias are malformed. parseNode
ignored any pending anchor or tag when it reached an alias token, dropping
the property and accepting the document.

The alias case now rejects a pending anchor or tag, and moves to its own
parseAlias helper so parseNode stays within the complexity limit. A bare
alias and an anchor or tag on a real node are unaffected.

Measured against the official YAML test-suite, parse-level conformance
rises from 266/333 (79.9%) to 267/333 (80.2%) with no regression.
2026-06-27 21:11:10 +02:00
8649f1722a fix(parsing): reject a comment interrupting a multi-line plain scalar
A comment terminates a plain scalar, so a multi-line plain scalar cannot
resume after one. The parser accepted 'key: word1' then a '#' line then
'  word2', and 'a' then 'b # ...' then 'c', merging the trailing content
back into the value ('word1 word2', 'a b c').

The scanner folds a genuine multi-line plain scalar into a single token
and splits it into a value-line scalar, an indent, and a further scalar
only when a comment broke its continuation folding. That token shape
therefore always denotes a comment-interrupted scalar, so the parser now
rejects it rather than re-merging. The re-merge path and its helper are
removed as they only ever served this malformed shape. A real multi-line
plain scalar and a trailing comment after a value stay valid.

Measured against the official YAML test-suite, parse-level conformance
rises from 264/333 (79.3%) to 266/333 (79.9%) with no regression.
2026-06-27 19:47:28 +02:00
b964fea138 fix(parsing): reject a multiline quoted scalar as a block implicit key
A block implicit key must lie on a single line, so a quoted scalar whose
opening and closing quotes sit on different lines cannot be one. The
scanner synthesized the implicit key regardless, so '"c\n d": 1' and the
single-quoted form parsed as a mapping with a folded key.

When synthesizing the implicit key for a quoted scalar followed by ':',
the scanner now rejects the construct if the scalar spanned a line break.
The restriction is block-only: a flow mapping key may span lines, so the
check is gated on block context, and a multiline quoted value, a single-
line quoted key, and an explicit key (which carries no ':' directly after
the scalar) are all unaffected.

Measured against the official YAML test-suite, parse-level conformance
rises from 261/333 (78.4%) to 264/333 (79.3%) with no regression.
2026-06-27 18:43:09 +02:00
b0dc31922b fix(parsing): reject a block sequence inline as a mapping value
A block sequence cannot be an implicit key's value on the same line as its
':' (key: - a); the value must begin on a following line, or use flow
style. The scanner accepted the inline block entry and parsed it as the
key's sequence value.

The block-entry case now rejects a '-' that starts a sequence when the
line already holds an implicit-key mapping value, reusing the same
same-line-pair flag as the inline nested mapping rule. As there, an
explicit- or null-key ':' opens its line and does not set the flag, so a
compact sequence value after it stays valid, as do a sequence on the next
line, a nested sequence, a flow sequence, and a negative number (key: -1).

Measured against the official YAML test-suite, parse-level conformance
rises from 260/333 (78.1%) to 261/333 (78.4%) with no regression.
2026-06-27 18:35:58 +02:00
735767ced9 fix(parsing): reject inline nested block mappings
Block style permits only one mapping pair per line, so a scalar followed
by ':' after a block-level value on the same line (a: b: c, a: 'b': c,
x: { y: z }in: valid, or trailing content after a quoted value) is not a
new key but a forbidden inline nested mapping. The scanner synthesized an
implicit key for any scalar followed by ':', regardless of position, so
these parsed as nested mappings instead of failing.

The scanner now records when a line already holds a block-level mapping
pair and suppresses a further implicit key on that line, in plain and
quoted scalars alike. The flag is set only for an implicit-key ':' (one
preceded by its key on the same line): a ':' that opens its line is an
explicit- or null-key value indicator, whose value may be a compact
mapping (? k then : a: 1), so that, flow mappings, next-line nesting, and
colons inside scalars stay valid.

Measured against the official YAML test-suite, parse-level conformance
rises from 256/333 (76.9%) to 260/333 (78.1%) with no regression.
2026-06-27 18:28:57 +02:00
fcba4e1f57 fix(parsing): reject content after a document end marker
A document-end marker takes no node on its own line: only a comment or
blank may follow. The scanner skipped the rest of the marker's line, so
content such as the 'invalid' in '... invalid' was silently dropped and
the document accepted.

scanDocumentMarker now distinguishes the two markers after skipping spaces:
a document start may carry the root node on its line and leaves it for the
main loop, while a document end rejects any real content (a comment or
blank still passes). A '...' not followed by an indicator boundary is not
a marker at all and remains an ordinary plain scalar, so '...junk' still
parses as content.

Measured against the official YAML test-suite, parse-level conformance
rises from 255/333 (76.6%) to 256/333 (76.9%) with no regression.
2026-06-27 17:56:37 +02:00
27c423dbb9 fix(parsing): require an explicit document after a directive
The YAML grammar requires every directive to head an explicit document,
so a directive must be followed by a '---' marker. The scanner accepted a
directive with no document at all: '%YAML 1.2' alone, or '%YAML 1.2' then
a '...' document-end marker, parsed without error.

The scanner now records the line of a directive awaiting its document,
clears it at a document start, and rejects a still-pending directive at a
document-end marker and at stream end. Both scan drivers finalize through
the same stream-end path, so the check lives in one helper called from
each. The rule applies to any directive, including an unrecognized one
(which is otherwise still ignored); the directive test that relied on an
unrecognized directive parsing without a '---' is corrected to include
one, since that leniency is not permitted by the grammar.

Measured against the official YAML test-suite, parse-level conformance
rises from 253/333 (76.0%) to 255/333 (76.6%) with no regression.
2026-06-27 17:18:47 +02:00
4c77b17f9f fix(parsing): validate %YAML directive parameters and reject duplicates
The %YAML directive takes exactly one parameter, the version, and at most
one may appear per document. The scanner checked only that a valid version
was present: it ignored any further tokens, so '%YAML 1.2 foo' was
accepted, and it did not track that a %YAML directive had already been
seen, so a repeated directive was accepted too. It now rejects an extra
parameter (allowing only a trailing comment) and a second %YAML directive,
resetting the per-document flag at each document boundary. A well-formed
directive, with or without a trailing comment, is unaffected.

Measured against the official YAML test-suite, parse-level conformance
rises from 251/333 (75.4%) to 253/333 (76.0%) with no regression.
2026-06-27 16:33:03 +02:00
97584dc57c fix(parsing): reject empty entries in flow collections
In a flow sequence or mapping a comma separates entries, so one must
precede it. The parser advanced past a comma without checking that an
entry came before, accepting a leading comma ([ , a ]) or a repeated comma
([ a, , b ]) as empty entries. Each flow loop now tracks whether an entry
precedes the comma and rejects it otherwise. A single trailing comma
([ a, b, ]) and empty collections ([], {}) remain valid, since the closing
bracket is handled before that check.

Measured against the official YAML test-suite, parse-level conformance
rises from 249/333 (74.8%) to 251/333 (75.4%) with no regression.
2026-06-27 16:26:46 +02:00
b7bcfe1a01 fix(parsing): enforce the full YAML 1.2 double-quoted escape table
The double-quoted escape decoder recognized only a subset of escapes
(" \ \/ \b \f \n \r \t \u) and silently passed any other character
through. That accepted invalid escapes such as \. and \', and mangled
valid ones the subset omitted: \x and \U hex escapes, \v \e \0, the named
\N \_ \L \P, and a backslash before a space, were emitted as their literal
letters (\x0d became the text x0d).

The decoder now maps every escape in the YAML 1.2 table to its character,
reads \x (8-bit) and \U (32-bit) hex escapes alongside \u, treats a
backslash before a line break as an escaped line break that joins the
lines with no separator, and rejects an unknown or truncated escape with
a diagnostic. Single-character escapes move to a lookup table so the
decoder stays within the complexity limit.

Measured against the official YAML test-suite, parse-level conformance
rises from 247/333 (74.2%) to 249/333 (74.8%) as two invalid documents
are now rejected, and value-level from 166 to 170 of comparable cases
(90.4%) as the previously-mangled escapes decode correctly, with no
regression.
2026-06-27 16:19:25 +02:00
1d07fce635 fix(parsing): reject a comment glued to earlier content on its line
A '#' that begins a comment must be preceded by whitespace or open the
line; glued to earlier same-line content (e.g. "value"# c, [ a, b ]#c,
>#c, a flow entry then ,#c) it is neither a comment nor, in those
positions, plain-scalar content, so the document is malformed. Such input
was previously accepted as a comment.

A '#' that opens a line, follows a space, or sits inside a plain scalar
(a#b, where it stays literal content and never reaches this check) is
unaffected. The scanner tracks whether a real token has been scanned on
the current physical line, reset on each line break, rather than comparing
token line numbers: the streaming tokenizer drains its token buffer
between scans, and a block scalar's blank line emits an empty scalar
attributed to the next line, either of which would misread as content.

Measured against the official YAML test-suite, parse-level conformance
rises from 243/333 (73.0%) to 247/333 (74.2%): four wrongly-accepted
invalid documents are now rejected, with no valid document regressing.
The scalar-scanning routines move to Scanner.Scalars.swift to keep the
file within the length limit.
2026-06-27 15:53:16 +02:00
b80e9c1c63 fix(parsing): comments and over-indented blanks ending block scalars
A block scalar terminated by a less-indented comment line failed with a
spurious 'unexpected indentation': once the smaller indentation ended the
scalar, the comment's own column was processed as a dedent, and because no
indentation level held that width the dedent raised an error. A comment's
indentation does not participate in block structure, so a comment that ends
a block scalar is now left for the main loop to scan, with the block
scalar's own first content line (which may itself begin with '#') still
treated as content.

Separately, a whitespace-only line indented past a block scalar's content
level lost its surplus spaces because the width was consumed before the
line was recognized as blank; the surplus is now preserved as the line's
content, so an over-indented blank contributes the correct value.

Measured against the official YAML test-suite, parse-level conformance
rises from 240/333 (72.1%) to 243/333 (73.0%) with matching value-level
gains, and no regression.
2026-06-27 15:34:13 +02:00
ceec6cca84 fix(parsing): allow a space before the colon of a quoted mapping key
A quoted mapping key followed by a space and then ':' ('"key" : value')
failed to parse: the scanner synthesizes the implicit mapping-key token by
checking the character immediately after the key, but a quoted scalar
stops at its closing quote, leaving the separating space, whereas a plain
scalar consumes that space into itself and ends at the colon. The quoted
path now looks past spaces for the colon, which the spec allows between an
implicit key and its value indicator.

Quoted keys with no space, quoted values, and plain keys (which already
worked with a space) are unaffected. Measured against the official YAML
test-suite, parse-level conformance rises from 238/333 (71.5%) to
240/333 (72.1%), with no regression.
2026-06-27 01:19:28 +02:00
0ae88855da fix(parsing): allow an anchor or tag on a non-first mapping key
A block mapping pair after the first could not carry an anchor or tag on
its key (k: 1 then &b key2: 2): the first pair's key properties are
consumed when the mapping is entered, but subsequent pairs were not, so an
anchored key past the first failed with 'expected mapping key, found
anchor'. The pair parser now consumes leading key properties for every
pair, attaching them to the key node, and a block mapping continues across
an anchor or tag token that begins a key.

Normal mappings, anchored values, sequences, and explicit keys are
unaffected. Measured against the official YAML test-suite, parse-level
conformance rises from 235/333 (70.6%) to 238/333 (71.5%), with no
regression.
2026-06-27 00:21:38 +02:00
58f30b4343 fix(parsing): an explicit key with no value maps to null
The block mapping parser required a ':' after every explicit key ('? key'),
so a key with no value, including several '? key' entries in a row (a set),
failed to parse with 'expected mapping value'. An explicit key with no
following ':' now takes an implicit null value, shared with the existing
'key:'-with-nothing-after path via one emitNullMappingValue helper.

Explicit keys with values and complex keys in general are unaffected. A
test that asserted '? [a, b]' followed by a block entry is a missing-value
error was corrected: the key is a valid null entry, and the trailing
sequence is now rejected as extra content (a second root node), which is
the more accurate diagnosis.

Measured against the official YAML test-suite, parse-level conformance
rises from 229/333 (68.8%) to 235/333 (70.6%), with several of the newly
parsed documents also value-correct and no regression.
2026-06-26 22:28:13 +02:00
82d5b57fd6 fix(parsing): ignore an unrecognized directive instead of rejecting it
A directive the processor does not recognize (e.g. %FOO) before a
document is now ignored, matching the YAML spec (emit a warning, otherwise
no effect), rather than failing the whole document with an
unsupportedDirective error. %YAML and %TAG handling is unchanged, and a
directive after content, where a directive cannot legally appear, is still
rejected via the existing guard.

Two tests and a fixture that pinned the old reject-everything behavior are
updated: the unknown-directive case moves from the unsupported-gaps list,
and the scanner-error test now exercises a directive-after-content case
(still an error) rather than an unknown directive before content.

Measured against the official YAML test-suite, parse-level conformance
rises from 227/333 (68.2%) to 229/333 (68.8%), with the two newly-parsed
documents also value-correct and no regression.
2026-06-26 22:20:18 +02:00
688f0d5c12 fix(parsing): fold a document-root plain scalar across column-1 lines
A plain scalar at the document root failed to parse when it spanned
multiple lines at column 1 (`a\nb\nc`): the continuation check required a
continuation to be indented more than the scalar, but a root scalar has
nothing enclosing it, so its enclosing-block indent is -1 and a
continuation may start at column 1. The base indentation now returns -1 at
the root, and the continuation stops at a document marker (--- / ...) so a
scalar never folds across a document boundary.

A stream test that asserted two bare plain lines (first / second) were
rejected as extra content was corrected: that input is a valid multi-line
scalar; genuine extra content (a flow node followed by a scalar) is still
rejected, which the test now uses.

Measured against the official YAML test-suite, parse-level conformance
rises from 221/333 (66.4%) to 227/333 (68.2%), with four more documents
also now value-correct and no regression across the existing suite.
2026-06-26 21:30:49 +02:00
27e8defac6 fix(parsing): fold line breaks in multi-line single-quoted scalars
Single-quoted scalars share the same flow line folding as double-quoted,
which the previous commit added: a single break folds to a space, a blank
line to one newline, whitespace adjacent to a break is stripped, and the
first-line leading and last-line trailing whitespace are kept. The
single-quoted decoder kept its raw breaks; it now reuses the shared
foldFlowScalarLineBreaks helper, while doubled quotes ('') still escape to
one quote.

Measured against the official YAML test-suite, value-level conformance
rises from 143/169 (84.6%) to 147/169 (87.0%) with no regression.
Parse-level is unchanged: these documents already parsed, just to the
wrong tree.
2026-06-26 21:18:55 +02:00
08e1979085 fix(parsing): fold line breaks in multi-line double-quoted scalars
The double-quoted decoder appended real line breaks and continuation-line
indentation verbatim, so a multi-line "..." scalar kept its raw newlines
and leading whitespace instead of folding. It now applies YAML flow
folding at each unescaped break: a single break folds to a space, a blank
line to one newline, and whitespace adjacent to a break (the line's
trailing, the next line's leading) is stripped. First-line leading and
last-line trailing whitespace stay (they sit next to a quote, not a
break), and \n escapes are still decoded to literal newlines.

Measured against the official YAML test-suite, value-level conformance
rises from 135/169 (79.9%) to 143/169 (84.6%) with no regression.
Parse-level is unchanged: these documents already parsed, just to the
wrong tree, which only the value-level metric exposes.
2026-06-26 21:13:13 +02:00
455cd07f4a fix(parsing): keep a root node that begins on the document-marker line
A document-start marker scan skipped the rest of the '---' line
unconditionally, so a root node written on that same line was discarded:
a block-scalar header ('--- >' / '--- |') was dropped and its body
mis-read as a plain scalar (folded to spaces, no trailing newline), and a
same-line plain or flow node was lost entirely. The document start now
skips only trailing spaces and leaves any real same-line content for the
parser; an empty rest-of-line or a comment is still consumed, and a node
on the line below the marker is unaffected. Document end ('...') still
takes no same-line node.

Measured against the official YAML test-suite, value-level conformance
rises from 130/170 (76.5%) to 135/169 (79.9%) with no regression.
Parse-level is unchanged: these documents already parsed, just to the
wrong tree, which only the value-level metric exposes.
2026-06-26 20:50:24 +02:00
c888a113f2 test: add value-level YAML conformance against in.json
Parse-level conformance only asks whether a document parses or is
rejected; a parser can accept a document and build the wrong tree and
still score. The harness now also compares the parsed tree to each case's
in.json, the canonical expected value, the honest metric.

Value-level baseline: 130/170 (76.5%) of comparable single-document
cases. It immediately exposes 40 documents that parse without error but to
the wrong tree, invisible to parse-level: double-quoted multi-line folding
(the newline+indent should fold to one space), scalar resolution (a value
read as null instead of a string), and block-literal indentation. 49 more
do not parse, and 12 multi-doc/non-string-key cases are not comparable
through a single JSON value and are counted as skipped, not failed.

Every future parsing fix is now judged against ground truth, not just
whether it threw.
2026-06-26 20:18:32 +02:00
d90a091ab0 fix(parsing): fold a plain scalar value that begins on its own line
A multi-line plain scalar whose value sits one line below its mapping key
stopped after the first line: the continuation check compared the next
line's indent against the value's own indent, which had been pushed onto
the indentation stack, so a continuation at the same indent failed the
strict '>'. It now compares against the parent block's indent (the level
below the value's, when the scalar begins exactly at the indentation top),
matching the spec: a continuation must exceed the mapping's indentation,
not the value's own. Inline values are unaffected, since their top-of-
stack already is the parent block's indent.

Measured against the official YAML test-suite, parse-level conformance
rises from 211/333 (63.4%) to 221/333 (66.4%) with no regression across
the existing 306 tests. The +10 covers the multi-line plain scalar family
the harness flagged, not just the case investigated.
2026-06-26 18:52:09 +02:00
eaf6a01a0e test: break down YAML test-suite parse failures by cause
The harness now histograms should-parse failures by ParseError
diagnosticCode, so the conformance gap is actionable rather than a single
number. The breakdown corrects an earlier heuristic guess: multi-document
streams are not the lever (1 of 78 failures). The real clusters are
unexpectedToken=49 (explicit/complex keys, multi-line plain scalars,
tagged keys, document-marker + node combinations), expectedNode=11, and
unsupportedDirective=10.
2026-06-26 18:26:21 +02:00
9298536e52 test: measure conformance against the official YAML test-suite
Adds an opt-in harness that runs PureYAML over the canonical YAML 1.2
corpus (github.com/yaml/yaml-test-suite) and reports a parse-level pass
rate: a should-parse case must parse, a should-error case must be
rejected. The data is fetched into a gitignored location by
scripts/fetch-yaml-test-suite.sh, never vendored, and the run is gated
behind PUREYAML_RUN_YAML_TEST_SUITE=1 so the default suite is unaffected.

This turns 'best YAML library' from an assertion into a tracked number.
Baseline: 211/333 (63.4%) parse-level: 177/255 valid documents parse,
34/78 invalid documents are rejected. So 78 valid docs are not yet parsed
and 44 invalid docs are wrongly accepted. The stricter value-level check
against each case's in.json is the next step.
2026-06-26 17:43:19 +02:00
985bd7baac test: round-trip the full scalar type matrix through Codable containers
Enumerate every scalar Codable type (Int8 through Int64, Int, UInt
through UInt64, Float, Double, Bool, String) through both a keyed
container, via an all-scalars struct, and an unkeyed container, via
per-type arrays. This drives the keyed encode and decode container
methods and the encoder's scalar ladder to near-full coverage and pins
exact round-trip equality for each type.
2026-06-26 09:04:54 +02:00
2f6de52654 test: cover the remaining low-level paths
Pin the last untested-but-correct paths surfaced by the coverage audit:
flow-style emission of double scalars, BatchReport.description across
multiple sources, the unkeyed superDecoder reading successive indexes,
the PreflightScanner scheme-colon exception that must not flag a missing
mapping space, and the tagged stream-validation throwing overload that
wraps failures in a document-indexed issue collection. Each assertion was
measured against actual behavior before being pinned.
2026-06-26 08:59:24 +02:00
ceb12295a4 fix(parsing): preserve leading blank lines in block scalars
Blank lines between a block scalar header and its first content line come
before the indent token, so the earlier blank-line capture skipped them.
Buffer leading blanks while the block indentation is still pending and
flush them as empty content once the indent is established, in the shared
indentation path used by both scan loops. Also fix folded bodies to emit
a break for a leading blank rather than folding it into a leading space.
This closes the last block-scalar blank-line edge from #57.
2026-06-26 08:42:16 +02:00
234e0382e6 refactor(parsing): remove three unreachable ParseError cases
emptyDocument, expectedMappingKey, and tabIndentation were never thrown
anywhere in the parser. Empty and comment-only documents parse to null,
the mapping-key path throws unexpectedToken with an "expected mapping
key" message, and tabs in indentation are reported by the validation
layer's PreflightScanner rather than the parser. Drop the dead cases and
their switch arms and description test assertions. No reachable behavior
changes; the PropertyMutation.tabIndentation test mutation is unrelated
and untouched.
2026-06-26 08:37:53 +02:00
4358cf02cc fix(parsing): keep more-indented folded lines literal
A folded block scalar folded every non-blank line into the surrounding
text, including more-indented lines, which YAML keeps literal with their
line breaks. Detect a more-indented line by its retained leading
whitespace and keep the breaks adjacent to it instead of folding them.
Plain and blank-line folding are unchanged.
2026-06-26 08:13:43 +02:00
d3ff3823a5 fix(parsing): preserve blank lines in block scalars
The scanner dropped every blank line within and after a block scalar, so
a literal block lost internal blank lines, a folded block turned a blank
line into a space rather than a newline, and the keep indicator could not
preserve trailing blank lines. Capture blank lines as empty content in
both the batch scan loop and the streaming TokenCursor loop, which had
drifted as duplicate copies. Rework blockScalarValue to apply literal
versus folded line semantics and clip/strip/keep chomping over the
captured lines; content without blank lines is unchanged. Re-pin the
multiline-scalar test whose expectation encoded the old dropped-blank
behavior, and add a chomping-and-folding test matrix.
2026-06-26 08:09:45 +02:00
934c9eafa3 test: cover numeric ladder, NaN hashing, unkeyed edges, error paths
Fill the highest-value coverage gaps found in the Codable bridge and
parser: the full integer-width decode ladder and per-width out-of-range
errors, Float round-trip, NaN equality and hash consistency (the bespoke
nan == nan invariant), unkeyed decodeNil false-branch and nested-index
mismatch, and the previously untested expectedNode, unterminated flow
mapping, explicit block-scalar under-indent, and buffered invalid-merge
error paths.
2026-06-26 07:50:08 +02:00
fadde56a05 fix(parsing): classify the keep block-scalar chomping indicator
The scanner consumed the `+` indicator of a `|+` / `>+` block scalar but
mapped it to nothing, so keep was silently treated as clip, and a scanner
test baked that in. Add a `keep` case to BlockScalarChomping, recognize
`+` in the header scanner, and correct the test. Full keep value semantics
(preserving trailing blank lines) depend on scanner blank-line capture and
are tracked in #57; this commit fixes the token classification only.
2026-06-26 07:50:07 +02:00
44abc12e80 docs: record rules conformance, import contract, and round-trip law
Add docs/rules-conformance.md documenting how the repo conforms to the
vendored canon, the documentation.md vs dependency-free conflict and its
scoped resolution, the domain-mismatched rules that do not apply, and the
conscious deviations. Add docs/package-import-contract-status.md. State
the round-trip law (Lambda1/Lambda2) in DESIGN.md.
2026-06-26 06:14:24 +02:00
7270359d14 ci(githooks): enforce format, lint, and conventional commits
Run SwiftFormat and SwiftLint on staged Swift files in the pre-commit
hook (formatting-and-linting.md), gracefully skipping when a tool is
absent since CI re-runs both. Enforce the Conventional Commits subject
format and the 72-character subject limit in the commit-msg hook
(commits.md). Correct the pre-push comment that named the removed ci.yml.
2026-06-26 06:14:24 +02:00
c4f7f9dccc refactor(code-style): one top-level type per source file
Split the six files that declared multiple sibling types under one
namespace extension so each file declares exactly one top-level type:
the Tagged node component types (Node, NodeKind, Scalar, Sequence,
Mapping, Pair), the validation report types (Report, Source,
SourceReport, BatchReport), DiagnosticKind, TokenKind, ScannerFlowContext
kind, and the event-stream composer helpers. Public type names are
unchanged; the shared report-rendering helpers move from file-private to
internal so the split report types still reach them.

Make check-namespacing.sh namespace-aware: the previous column-anchored
scan was blind to types declared inside indented extension blocks, so it
passed while the rule was violated. It now counts direct members of a
PureYAML namespace extension and ignores nested types and private helpers.
2026-06-26 06:14:21 +02:00
9ae6c3baff refactor(parsing): share merge-key resolution between composers
The buffered and streaming event composers each carried an identical
copy of the merge-key (<<) override logic and its invalid-value error.
Extract both into one internal Parsing.MergeResolution helper so the
precedence rules (local keys win, earlier sources win, inherited then
local order) have a single source of truth. No behavior change; all
tests pass.
2026-06-26 05:50:28 +02:00
87e3e5cbab docs(security): fix contact address and correct the scope section
The reporting address was missing its @ and was unusable. The scope
section described an unrelated static-site generator; replace it with
the YAML parsing, validation, and emitting surface that is actually in
scope (malformed-input DoS, misrepresented values, validation as a
trust boundary).
2026-06-26 05:50:08 +02:00
ea31bb5864 chore(ci): split CI into per-workflow files with truthful badges
Rule 8.1 forbids one multi-job workflow fronted by several cosmetically
labeled badges that all resolve to the same whole-workflow status. Split
ci.yml into one workflow per concern (gates, macOS, Linux, Windows,
WASM, corpus), each with a unique name, and point each README badge at
its own workflow through the native per-workflow badge. The WASM
workflow notes in a header that it is a build gate, not a test gate
(Rule 8.3). Add a static MIT license badge linking to LICENSE
(Rule 8.2). Step contents are unchanged from the previous jobs.
2026-06-26 05:48:11 +02:00
9acfd57acc chore(issues): add required complexity field and a form labeler
Add the Complexity dropdown that Rule 1.6 requires on both issue forms
(status date, priority, complexity), alongside the existing per-kind
fields. Add a labeler workflow that maps the Priority dropdown to the
priority: high label on issue open and edit, closing the Rule 1.6
gotcha where form dropdowns produce body text but never apply a label.
Complexity stays a body field only, never a label, per Rule 2.1.
2026-06-26 05:48:01 +02:00
5f3892f2c1 chore(rules): vendor the rules-swift canon into docs/rules-swift
Mirror the shared Swift rules canon (rules-swift, commit 4ebb892) into
docs/rules-swift/ as a read-only reference alongside the repo's tailored
docs/rules/. PROVENANCE.md records the source, commit, and what was
included or excluded; the upstream repo's own VCS, CI, and lint tooling
is omitted as machinery rather than rule content.

Exclude the vendored tree from the style gate (scripts/check-style.sh)
and the pre-commit hook: the canon's enforcement docs quote the
forbidden attribution phrases to define them, the same reason the
repo's own enforcement files are already excluded.
2026-06-26 05:36:12 +02:00
f7500400ee feat(parsing): exact scalar end positions in the located parse
The scalar event now carries the scanner token's end mark, so a located
scalar's range spans the whole token (end exclusive, one past the last
character) instead of being start-anchored. This gives consumers exact
highlight widths and is the prerequisite for surgical, position-based
text edits. Only the scalar event case gained a field; the value tree,
its equality, and the event dump (which compares value and style, not
marks) are unchanged, so the canonical-event and whole-suite gates stay
green (343 tests).
2026-06-26 02:25:01 +02:00
3f3ce310b1 feat(parsing): opt-in located parse with source spans on every node
Add parseLocated / parseStreamLocated, returning a Model.LocatedValue
tree parallel in shape to Value, where every node carries its source
span (1-based line/column) and located children. Collections span from
their opening to closing event; scalars carry a start-anchored span;
aliases and merge-inherited pairs honestly point at their reference site.

The plain Value, its equality, and the conformance oracle are untouched:
locations are built only when the streaming composer is asked for them
(tracksLocations), behind new entry points, so the default value path is
byte-identical. A whole-suite test proves the located projection equals
a plain parse for every passing case (402/402), including through merge
keys and aliases, and the canonical-event and gap gates stay green.
2026-06-26 00:57:55 +02:00
bf069fad39 feat(json): add PureYAMLJSON product, a standalone YAML <-> JSON converter
PureYAMLJSON is a separate library product layered on the core engine. JSON is
a subset of YAML 1.2, so the existing parser already reads both into one value
tree; this product adds the missing half and the value-level policy:

- JSON.Writer: a pure-Swift RFC-8259 serializer (no Foundation; WASI-clean),
  pretty or compact, rejecting non-string keys and non-finite numbers.
- JSON representability rules expressed in PureYAML's own Validation framework:
  mappingKeysAreStrings and numbersAreFinite join the core mappingKeysAreUnique.
- JSON.Validator: the representability rule set plus issues(for:)/isRepresentable.
- JSON.Convert: standalone convert(_:to:) with the round-trip law (parse ->
  validate representability -> emit -> re-parse must equal the source). To JSON
  rejects complex keys and non-finite numbers; to YAML rejects duplicate keys,
  which are legal in JSON but not in YAML.

19 tests cover the writer, the rules, and both conversion directions including
the dual-rejection edges. Carved out of the catch-all Sources/Tests paths via
exclude so the core PureYAML target is unchanged.
2026-06-25 00:33:40 +02:00
5c34ebd8b6 fix(parsing): reject a misindented multi-line quoted scalar continuation
A quoted scalar's continuation line must be indented past the parent
block, so '"a' / 'b"' (continuation in column 0) and '"bar' / '\tbaz'
(tab indentation) are malformed; libyaml accepts both.

Record the indentation error during the continuation scan but defer
raising it until the closing quote is found. An unterminated quote reaches
end of input first and still reports the missing quote (the root cause)
rather than the indentation symptom, preserving the multi-error diagnostic.

Surpasses libyaml on QB6E and DK95/01. Whole-suite conformance is now
402/402 with zero recorded gaps; canonical events stay 308/308.
2026-06-24 22:09:14 +02:00
98ca76e7af fix(parsing): reject a tab used to indent flow content
A tab is never indentation, so flow-collection content reached through a
tab below the continuation indent ('- [' / '\tfoo') is malformed even
though the tab inflates the column. A tab at or past the indent is
separation, and a tab-only blank line stays blank, so both remain valid.

Track a tab in a flow continuation's leading whitespace below the
parent + 1 indent and reject content (but not a closing bracket) reached
through it. Surpasses libyaml on Y79Y/003; whole-suite conformance
400/402, canonical events 308/308.
2026-06-24 22:04:30 +02:00
52b62ba390 fix(parsing): reject a block scalar with an over-indented empty line
A leading empty line in a block scalar may not be more indented than the
first content line. libyaml auto-detects the deeper indentation (from the
empty lines) and silently drops the content. The auto-detection now errors
when the content line is less indented than an earlier empty line on the
same scan.

Surpasses libyaml on S98Z; whole-suite conformance 399/402, canonical
events 308/308.
2026-06-24 21:58:15 +02:00
24bb2ea4dd fix(parsing): also reject a plain multi-line key in a flow sequence
A flow-sequence single-pair implicit key must be single-line, so its ':'
must sit on the key's line. The quoted form ('[ "key"\n :value ]') was
already rejected via a crossed line break, but a plain key folds its line
break, hiding it. Compare the simple key's mark line with the ':' line
instead, which catches both forms and still exempts flow mappings.

Surpasses libyaml on DK4H (and keeps ZXT5); whole-suite conformance
398/402, canonical events 308/308.
2026-06-24 21:53:48 +02:00
c236945323 fix(parsing): reject a multi-line implicit key in a flow sequence
A flow-sequence single-pair implicit key is single-line
(c-s-implicit-json-key uses s-separate-in-line), so its ':' may not cross
a line break: '[ "key"\n :value ]' is malformed. A flow MAPPING adjacent
value may still span lines ('{ "foo"\n :bar }' stays valid), which is why
the check is scoped to flow levels that are sequences.

Track per-flow-level whether it is a sequence ('[' vs '{') in a
flowIsSequence stack pushed/popped with the flow level, and in fetchValue
reject a pending simple key whose ':' crossed a line break when the
current flow level is a sequence.

Surpasses libyaml on ZXT5 (5MUD, the flow-mapping analogue, stays valid);
whole-suite conformance 397/402, canonical events 308/308.
2026-06-24 21:50:41 +02:00
57adb14f15 fix(parsing): reject a block sequence that opens mid-line
A block sequence must begin its own line. A '-' entry mid-line after an
inline mapping value ('key: - a') or a node property ('&anchor - x') is
invalid, but libyaml accepts both. A '-' after another '-' is still a
valid compact sequence ('- - a'), and an explicit key whose ':' is on a
later line may still open a sequence ('? k' / ': - a').

In fetchBlockEntry, when no line break was crossed, reject if the last
token is an anchor/tag, or a value that closed an inline implicit key
(told apart by a pending simple key: an explicit key's simple key has
gone stale across its line break). Extract the scanner guard helpers into
LibYAML.Scanner.Guards to stay within the file-length limit.

Surpasses libyaml on 5U3A, SY6V; whole-suite conformance 396/402,
canonical events 308/308.
2026-06-24 21:44:27 +02:00
eee7079efe fix(parsing): require flow continuation lines to be indented
YAML requires multi-line flow content to be indented past the enclosing
block context (s-flow-line-prefix). libyaml does not enforce this, so
'flow: [a,' / 'b]' with 'b' in column 0, and the equivalent multi-line
flow mapping, wrongly parsed.

Track whether reaching a token crossed a line break, and in flow context
reject a token at or below the parent block indent. A closing ']' / '}'
or a leading ',' may sit at any indentation, so those are exempt (a flow
mapping's closing brace in column 0 stays valid).

Surpasses libyaml on 9C9N, VJP3/00; whole-suite conformance 394/402,
canonical events 308/308.
2026-06-24 20:45:12 +02:00
4a6b168565 fix(parsing): reject content after a document-end marker
A '...' may be followed only by whitespace and a comment on its line.
Unlike '--- x', which carries inline document content, '... x' is
malformed, but the port (like libyaml) parsed the trailing text as a new
bare document.

After emitting the document-end token, require the rest of the line to be
blank or a comment. Surpasses libyaml on 3HFZ; whole-suite conformance
392/402, canonical events 308/308.
2026-06-24 20:13:51 +02:00
fca1441a27 fix(parsing): resolve a non-specific '!' tagged scalar as a string
The non-specific '!' tag resolves a node by kind alone, with no implicit
typing, so '! 12' is the string "12", not the integer 12. The event
stream already carried the tag correctly, but typed-value construction
fell through to plain-scalar resolution and produced an int.

Add the '!' case to tagged-scalar composition so it yields a string.
Surpasses libyaml on S4JQ (the last parse-ok gap); whole-suite
conformance 391/402, canonical events 308/308.
2026-06-24 20:10:02 +02:00
1b67030ed5 fix(parsing): reject a bare dash before a flow indicator
Per ns-plain-first a '-' starts a plain scalar in flow only when a
flow-safe character follows. A '-' before a flow indicator ('[-]',
'[-, -]') cannot start a node, but the port (like libyaml) read it as the
scalar '-'.

Reject that form in the indicator dispatch. Extract scan_to_next_token,
the comment validator, and blankLineAhead into LibYAML.Scanner.SkipToToken
to keep the fetch file within the length limit.

Surpasses libyaml on G5U8, YJV2; whole-suite conformance 390/402,
canonical events 308/308.
2026-06-24 20:06:30 +02:00
2e9975e977 fix(parsing): reject a comment with no preceding whitespace
A '#' starts a comment only after a space or tab, or at the start of a
line. A '#' adjacent to the previous token is not a comment: 'key: "v"#c',
'[ a ]#c', '[ a,#c', and a '>#c' block-scalar header are malformed, but
the port (like libyaml) accepted them.

Validate the character immediately before the '#' (a plain scalar already
consumes its own trailing blanks, so an adjacent '#' the next-token scan
sees genuinely had no separator). Applied in scan_to_next_token and the
block-scalar header tail.

Surpasses libyaml on 9JBA, CVW2, SU5Z, X4QW; whole-suite conformance
388/402. Canonical event stream stays 308/308.
2026-06-24 20:00:21 +02:00
b5129d71ee fix(parsing): reject misplaced directives, refresh suite manifest
Directives apply to the document that follows, so the previous document
must be closed with an explicit '...' end marker before them. After an
implicit end, a '%YAML'/'%TAG'/reserved directive is now rejected instead
of being silently accepted. Also reject a '%YAML' version trailed directly
by '#' with no separating space ('%YAML 1.1#...'), since a comment needs
leading separation; libyaml accepts both.

This restores reject-side conformance the suite manifest recorded as
passing but the libyaml port had dropped (9HCY, EB22, MUS6/00, MUS6/01,
RHX7), and updates the two PureYAML gap-error fixtures whose rejection now
fires earlier and more precisely at the misplaced directive.

Refresh yaml-test-suite-manifest.yaml to version 3: whole-suite
conformance is 384/402 (up from the recorded 331); 53 cases still marked
as gaps now pass, none regress.
2026-06-24 19:51:39 +02:00
7898dcd1a9 fix(parsing): keep the final break of a newline-less block scalar
A block scalar whose last line is whitespace only and ends at end of
input without a break still owns one line break, which keep (+) chomping
preserves. libyaml records no break because there is no break byte, so
'- |+' over a spaces-only unterminated line yields '' instead of '\n'.

In scan_block_scalar_breaks, when the break loop stops at end of input
after consuming whitespace on that line, append one implicit '\n' to the
trailing breaks (used only by keep chomping). Content lines are
unaffected because they do not skip whitespace in this loop.

With this the ported engine emits the exact canonical event stream for
every valid case in the vendored YAML test suite: 308/308, ahead of
libyaml (252) and the old hand-rolled parser (267). Port score 307 -> 308.
2026-06-24 19:28:58 +02:00
d222b8cc53 fix(parsing): end a root block scalar at a column-0 document marker
Now that a root-level block scalar can auto-detect indentation 0, its
content loop reaches lines in column 0, so a '---' or '...' document
marker there must end the scalar. Without the guard a '--- |' literal
folded the following '...', the next '%YAML' directive, and every later
document into its value.

Break out of the block-scalar content loop on a column-0 document marker.
The guard only fires at indent 0 (column == indent == 0), so nested block
scalars are unaffected. Surpasses libyaml on W4TN. Port score 306 -> 307.
2026-06-24 19:22:33 +02:00
825084dbff fix(parsing): fold over a tab-only blank line in a plain scalar
A plain scalar may span blank lines, and a blank line that contains a tab
is still blank, so the tab is not indentation. libyaml rejects any tab
below the scalar indentation with 'found a tab character that violates
indentation', which wrongly rejects 'foo: 1' / tab-only line / 'bar: 2'.

Guard the tab check with blankLineAhead so it fires only when the tab
indents real content. Surpasses libyaml on DK95/04. Port score 305 -> 306,
leaving zero false-rejects.
2026-06-24 19:18:05 +02:00
b2d2d2f595 fix(parsing): accept tabs as separation, reject them as indentation
libyaml refuses to skip a tab in the inter-token whitespace whenever a
simple key is allowed, so a tab used as valid separation (after a '-',
'?', or ':' indicator, or before a flow collection) is reported as a
character that cannot start a token. But a tab is legal in-line
separation; it is only illegal as indentation.

The scanner now skips tabs in scan_to_next_token and records in
tabbedSeparation whether one preceded the pending token. A block entry,
an explicit key, or an implicit mapping key promoted by ':' is rejected
when reached through a tab (there the tab would be indentation), while a
scalar value or a flow collection is accepted. Tab-indented keys and
'-\t-' nested entries still reject.

Surpasses libyaml on 6BCT, 6CA3, A2M4, DK95/00, Q5MG, Y79Y/010.
Port score 299 -> 305.
2026-06-24 19:14:12 +02:00
8092897612 fix(parsing): parse a leading ':' flow-sequence entry as an empty-key pair
A flow sequence entry that begins with ':' (e.g. '[ : v ]') is a
single-pair mapping with an empty key, which YAML 1.2 permits but libyaml
rejects by handing the leading VALUE token to its node parser.

Open the implicit single-pair mapping on a leading ':' in the
flow-sequence-entry parser, and leave the VALUE token for the mapping
value state to consume so the value is parsed rather than dropped.
Surpasses libyaml on CFD4. Port score 298 -> 299.
2026-06-24 19:07:04 +02:00
ff199516ac fix(parsing): apply the YAML 1.2 flow ':' rule with JSON-key adjacency
libyaml makes every ':' in a flow collection a value indicator, so it
mis-scans ':x', 'http://foo.com', '::vector', and 'omitted value:,', and
drops the value after an adjacent quoted or collection key. Per the spec a
':' only ends a node when followed by whitespace, end of input, or a flow
indicator, except an adjacent ':' after a JSON key (a quoted flow scalar
or a flow collection used as the key).

Gate the ':' value-indicator dispatch on that rule, tracking on the
pending simple key whether it is a JSON node (set for quoted flow scalars
and flow-collection starts). End the plain scalar at a ':' before a flow
indicator instead of rejecting it as unexpected.

Surpasses libyaml on 4ABK, 58MP, 5T43, DBG4, FRK4, HM87/00, with 9MMW
(mixing adjacent and separated keys) still correct. Port score 292 -> 298.
2026-06-24 19:02:43 +02:00
c3a1e6aaa4 fix(parsing): treat flow '?' before a non-indicator as scalar content
libyaml makes every '?' in flow context a mapping-key indicator, so it
parses '{ ?foo: bar }' as a key 'foo' (dropping the '?') and '[?x]' as an
empty single-pair mapping. Per ns-plain-first a '?' followed by a
flow-safe character (anything but whitespace, end of input, or a flow
indicator) begins a plain scalar.

Gate the '?' key-indicator dispatch on isBlankZ or, in flow context, a
flow indicator, so '?foo' and '?x' stay plain-scalar content. Hoist the
flow-indicator predicate into CharacterClass and reuse it in the plain
scalar scanner. Surpasses libyaml on 652Z and HM87/01. Port score
290 -> 292.
2026-06-24 18:50:46 +02:00
26e09103b4 fix(parsing): accept tabs as block-scalar content past the indentation
The block-scalar break scanner raised 'found a tab character where an
indentation space is expected' for every tab seen while auto-detecting
the content indentation. But a tab at or beyond the content column is
content, not indentation: only a tab below the content column is illegal.

Gate the tab check on a content-column floor (the resolved indent, or the
parent indent + 1 while auto-detecting) instead of firing on every tab in
auto-detect mode. A tab-only line shallower than the content is still
rejected, so tab-as-indentation errors (Y79Y/000 and friends) are
preserved. Surpasses libyaml on 96NN/00, 96NN/01, R4YG, Y79Y/001.
Port score 286 -> 290.
2026-06-24 18:44:25 +02:00
254905fe67 fix(parsing): auto-detect indentation 0 for a root block scalar
The block-scalar scanner used 0 as the 'indentation not yet detected'
sentinel and floored the auto-detected indent at 1, mirroring libyaml's
scan_block_scalar_breaks. A root-level block scalar (parent indent -1)
whose content starts in column 0 therefore detected no content, and the
first content line was reparsed as a fresh document, raising 'did not
find expected document start'.

Switch the sentinel to -1 so 0 is a representable detected indent, and
floor the auto-detected indent at parent indent + 1 (0 at the document
root) instead of 1. Nested block scalars keep a parent indent >= 0, so
their floor stays >= 1 and they are unaffected.

Surpasses libyaml on DK3J, FP8R, and M7A3. Port score 283 -> 286.
2026-06-24 18:36:17 +02:00
3fc3072544 feat(parsing): parse flow mappings with empty keys
A ':' value indicator with no preceding key is also valid inside a flow
mapping, denoting an entry whose key is empty (null). libyaml hands the
bare VALUE token to its node parser and fails with 'did not find expected
node content'. The flow-mapping-key parser now emits the empty scalar key
and lets the value state consume the indicator.

Fixes '{ : }' and '{ key: value, : empty key }', completing the
multi-document NKF9 case. Port score 282 -> 283.
2026-06-24 18:31:40 +02:00
70c3da85a3 feat(parsing): parse block mappings with empty keys
A ':' value indicator with no preceding key on its line denotes a
mapping entry whose key is empty (null), which YAML 1.2 permits but
libyaml rejects with 'did not find expected key'. The block-mapping-key
parser now emits the empty scalar key and leaves the VALUE token for the
value state to consume, rather than throwing.

Fixes the canonical event streams for plain empty-key mappings, explicit
empty keys with anchors and aliases, empty keys nested in sequences, and
mixed empty-key/value entries. Port score 275 -> 282; surpasses libyaml
on 2JQS, 6M2F, M2N8/00, NHX8, S3PD, SM9W/01, UKK6/00.
2026-06-24 18:27:45 +02:00
884ff7d9a1 docs: record the libyaml port as complete and the sole parsing engine
Marks every step of the port plan done and adds a status section: the port
is integrated as PureYAML's only parser, verified byte-for-byte against
libyaml on 307/308 cases, and scores 275/308 on the canonical event gate,
ahead of libyaml's own 252 and the prior hand-rolled parser's 267.
2026-06-24 18:17:12 +02:00
18101b8ddc feat(parsing): accept full YAML 1.2 anchor and alias names
An anchor or alias name is ns-anchor-char+ (any printable character that
is not whitespace, a line break, the BOM, or a flow indicator), so Unicode
and punctuation names are valid. libyaml restricts the name to
alphanumerics and rejects them.

Canonical event conformance: 271 -> 275 of 308 (2SXE, 8XYN, W5VH, Y2GN).
379 tests pass.
2026-06-24 18:11:58 +02:00
df3aa0b5d4 refactor(parsing): delete the hand-rolled parser, leaving the ported engine
The ported libyaml scanner and parser now serve every parsing path, so the
hand-rolled scanner, token cursor, and token event parser are dead. Removes
them and the old Token/NodeResult/NodeProperties types along with the unit
tests that exercised the old scanner directly. The remaining suite drives
the ported engine through the public API.

PureYAML now has a single parsing engine. 379 tests pass.
2026-06-24 18:07:20 +02:00
0588996506 feat(parsing): run the whole library on the ported engine
Routes the value and stream APIs through the ported scanner+parser (the
event API already was), so PureYAML now parses entirely on the libyaml
port. Adds the features needed to match and exceed the previous engine:
JSON UTF-16 surrogate-pair \u escapes, bare documents after an explicit
`...`, reserved-directive handling that still requires a following `---`,
and a 64-level nesting bound enforced on the event stream so deeply nested
input fails with nestingTooDeep instead of overflowing a recursive
consumer. Parser errors are translated to the diagnostic-aware ParseError,
recovering precise codes where identifiable and a readable malformed
fallback otherwise; error-reporting fixtures are rebased onto the port.

Canonical event conformance: 268 -> 271 of 308 (7Z25, HWV9, QT73).
393 tests pass.
2026-06-24 18:02:22 +02:00
d9bf2bda26 feat(parsing): support multi-line flow mapping keys
YAML 1.2 lets a flow-context key be separated from its ':' value indicator
by line breaks, so a flow-level simple key must survive a new line (it is
still bounded to 1024 characters). libyaml invalidates every simple key on
a new line, which wrongly rejects keys like `{"foo"\n: "bar"}` and
multi-line quoted keys.

Canonical event conformance: 259 -> 268 of 308 (4MUZ/00-02, 5MUD, 9SA2,
K3WX, NJ66, UT92, VJP3/01). This passes the hand-rolled parser's 267 and
exceeds libyaml's own 252. 333 tests pass.
2026-06-24 17:46:16 +02:00
c906993f00 feat(parsing): treat tab-containing blank lines as blank
YAML 1.2 separation lines may contain tabs, so a line of only spaces and
tabs is blank. libyaml leaves a leading tab and later rejects it; the
scanner now consumes the trailing blanks of a whitespace-only line so the
break is reached and the line is skipped.

Canonical event conformance: 258 -> 259 of 308 (DK95/03). 333 tests pass.
2026-06-24 17:41:21 +02:00
6ce1f906f9 feat(parsing): surpass libyaml on reserved directives and YAML versions
YAML 1.2 §6.8.1 requires a reserved directive (anything other than %YAML
or %TAG) to be ignored with a warning, and a document whose minor version
is newer than 1.2 to be parsed with a warning. libyaml rejects both. The
scanner now drops a reserved directive's whole line and emits no token,
and the parser accepts any major-version-1 document.

Canonical event conformance: 253 -> 258 of 308 (2LFX, 6LVF, MUS6/05,
MUS6/06, BEC7). 333 tests pass.
2026-06-24 17:37:32 +02:00
561921df42 feat(parsing): route the event API through the ported libyaml engine
parseEvents now runs on the ported scanner+parser instead of the
hand-rolled approximation. Adds 1-based mark conversion at the boundary
(the port stays faithfully 0-based internally), translates the port's
scanner and parser errors into the diagnostic-aware ParseError with
precise codes (unterminatedQuotedString, unterminatedTag, tabIndentation)
and a readable malformed fallback, and retargets the canonical event gate
to track the port's gap set.

The port matches the canonical test.event on 253 of 308 valid cases;
the remaining gaps match libyaml's own behaviour and are the targets for
surpassing libyaml. Golden event and diagnostic expectations are rebased
onto the port where it is faithful or more correct (resolved verbatim
tags, libyaml's stream-end marks, reaching the real unterminated quote).
333 tests pass.
2026-06-24 17:32:29 +02:00
6eddb64dcd fix(parsing): reject a tab where a token must start, add port differential
The scanner now errors with "found character that cannot start any token"
when a tab leads a token in block context, matching libyaml's final guard
in fetch_next_token (a plain scalar may not begin with a blank). Adds a
differential harness (env-gated) that dumps the ported pipeline's yeast
events for the whole suite.

Verified against the libyaml reference dumper across all 308 vendored
valid cases: the port is byte-for-byte identical to libyaml on 307, and
the single difference (L24T/01) is the port matching the canonical
test.event where libyaml drops a trailing line break. 333 tests.
2026-06-24 17:12:36 +02:00
521062d974 feat(parsing): port libyaml parser state machine to Swift
Ports parser.c: the one-event-at-a-time state machine that turns the
scanner's token stream into the YAML event stream. Covers stream and
document framing, directive processing with %TAG handle resolution and
default handles, the node property/tag/anchor production, block and
indentless sequences, block mappings, and flow sequences and mappings
including the single-pair `?` mapping forms.

Output verified against the libyaml reference event dumper: the empty
mapping value, %TAG resolution, and shorthand tags all match byte for
byte. 331 tests, no regression.
2026-06-24 17:05:59 +02:00
ab05580518 feat(parsing): port libyaml directive scanner to Swift
Adds the %YAML and %TAG directive scanner, the final scanner piece.
Ports fetch_directive, scan_directive, scan_directive_name,
scan_version_directive_value/number, and scan_tag_directive_value,
reusing the already-ported tag handle/URI scanners. Adds
versionDirective and tagDirective token kinds.

The libyaml scanner is now fully ported: stream, document markers,
directives, block + flow structure, plain/quoted/block scalars,
anchors, aliases, and tags. 319 tests, no regression.
2026-06-24 16:54:04 +02:00
ed1904534c feat(parsing): port libyaml tag scanner to Swift
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_tag, scan_tag, scan_tag_handle, scan_tag_uri, and scan_uri_escapes: verbatim (!<uri>), shorthand (!handle!suffix), and local (!suffix) tags, producing the handle and suffix for the parser to resolve. URI %-escapes are decoded by collecting the UTF-8 octets and decoding them as a unit. Unit-tested for a secondary-handle shorthand tag, a local tag, and a verbatim tag. Only directives (%YAML/%TAG) remain unported in the scanner. 317 tests, no regression.
2026-06-24 16:48:22 +02:00
7918eefe14 feat(parsing): port libyaml block-scalar scanner to Swift
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_block_scalar, scan_block_scalar, and scan_block_scalar_breaks: literal (|) and folded (>) block scalars with the chomping (+/-) and indentation indicators in either order, indentation auto-detection from the first non-empty line (clamped to parser indent + 1), folded joining of equally-indented non-empty lines into a space, and clip/strip/keep chomping of the trailing breaks. Unit-tested for literal clip, folded join, and strip chomping. Only tags and directives remain unported in the scanner. 314 tests, no regression.
2026-06-24 16:40:18 +02:00
526abe9998 feat(parsing): port libyaml anchor and alias scanner to Swift
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_anchor and scan_anchor: an anchor (&name) or alias (*name) whose name is a run of IS_ALPHA characters, validated to be non-empty and followed by a blank, end of input, or a terminating indicator. The flow/quoted/anchor fetchers are consolidated into one indicator dispatch to stay within the complexity limit. Unit-tested for an anchored scalar value and a later alias. Remaining unported fetchers: tags, block scalars, directives. 311 tests, no regression.
2026-06-24 15:26:49 +02:00
4e7ca8fcb2 feat(parsing): port libyaml quoted-scalar scanner to Swift
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_flow_scalar and scan_flow_scalar: single- and double-quoted scalars with line-break folding (a single break folds to a space, blank lines to newlines), the single-quote '' escape, double-quoted backslash escapes (the full \0 \a \b \t \n \v \f \r \e space \" / \ \N \_ \L \P table plus \xXX/\uXXXX/\UXXXXXXXX hex), and the escaped end-of-line continuation that suppresses the break. Escapes are decoded to Unicode scalars directly rather than UTF-8 bytes. Unit-tested for the escaped single quote, the escape table including a hex unicode escape, and multi-line folding. The remaining unported fetchers are block scalars, anchors, tags, and directives. 310 tests, no regression.
2026-06-24 15:22:41 +02:00
27514466d0 feat(parsing): port libyaml flow-collection fetchers to Swift
Stage 3 of the faithful libyaml scanner port, continued. Translates fetch_flow_collection_start, fetch_flow_collection_end, and fetch_flow_entry (the [ ] { } , indicators), with the flow-level push/pop already ported. Combined with the plain-scalar scanner's flow-aware termination, the port now tokenizes flow sequences and mappings of plain scalars, and the simple-key splice promotes a flow mapping key. Unit-tested for [a, b] and {k: v}. Remaining unported fetchers: quoted/block scalars, anchors, tags, directives. 307 tests, no regression.
2026-06-24 15:16:45 +02:00
43b20ab7bc feat(parsing): port libyaml token-fetch loop (block + plain core)
Stage 3 of the faithful libyaml scanner port: the token producer. Translates scan_to_next_token, fetch_more_tokens/fetch_next_token dispatch, and the fetchers for stream start/end, document indicators, block entry, key, value (with the simple-key splice and roll-indent), plus scan_plain_scalar with its line-break folding. Reuses the ported indent engine, simple-key mechanism, character classes, and reader. Constructs not yet ported (flow collections, quoted scalars, anchors, tags, directives, block scalars) throw an explicit 'not yet ported' error rather than mis-scan; the port is not yet wired to the public API.

Milestone: the ported scanner tokenizes '- a:' followed by a same-column 'b:' as siblings (one BLOCK-MAPPING-START), matching libyaml, fixing the original block-mapping bug by construction; a more-indented value still nests. Unit-tested for the sibling case, the nested case, a simple pair, and multi-line plain-scalar folding. 305 tests, no regression.
2026-06-24 10:37:35 +02:00
7f94e17e2f feat(parsing): port libyaml input reader to Swift
Stage 3 foundation of the faithful libyaml scanner port. Translates libyaml's input buffer and position tracking (the buffer/mark fields and the PEEK/SKIP/SKIP_LINE macros) into PureYAML.Parsing.LibYAML.Reader. It iterates Unicode scalars rather than grapheme clusters so a CR and an LF are distinct, matching libyaml's byte-oriented scanning (a CR+LF pair is still one line break), and tracks the mark in libyaml's 0-based line and column convention. Unit-tested for peek/skip column tracking and LF and CRLF line breaks. This is the input layer the token fetchers consume.
2026-06-24 03:04:08 +02:00
348af8b77b feat(parsing): port libyaml character-class predicates to Swift
Stage 3 foundation of the faithful libyaml scanner port. Translates the IS_* character-class macros from yaml_private.h (IS_Z, IS_SPACE, IS_TAB, IS_BLANK, IS_BREAK, IS_BREAKZ, IS_SPACEZ, IS_BLANKZ, IS_DIGIT, IS_ALPHA, IS_HEX, IS_ASCII, IS_PRINTABLE) into PureYAML.Parsing.LibYAML.CharacterClass. libyaml inspects UTF-8 bytes; the Swift port operates on Unicode.Scalar, where the multi-byte line breaks NEL (U+0085), LS (U+2028), and PS (U+2029) are single scalars, and end of input is nil, libyaml's \0 sentinel (so isZ(nil) is true and the others are false). These predicates are what the token fetchers dispatch on. Unit-tested for end-of-input, blanks, the full break set, the alpha/digit/hex sets, and the ascii/printable boundaries.
2026-06-24 03:00:28 +02:00
9ce57e86e5 feat(parsing): port libyaml simple-key mechanism to Swift
Stage 2 of the faithful libyaml scanner port. Translates yaml_parser_save_simple_key, yaml_parser_remove_simple_key, yaml_parser_stale_simple_keys, and the flow-level push/pop (yaml_parser_increase_flow_level / decrease) onto a new PureYAML.Parsing.LibYAML.Scanner aggregate that embeds the indent engine. The simple-key mechanism is what lets a quoted scalar, a nested collection, or a property-carrying node become a mapping key when a ':' follows, and is required exactly at the block indent. Modelled faithfully: a potential key goes stale once it spans a line break or exceeds 1024 characters, and a required key that is removed or goes stale is the 'could not find expected colon' error. Unit-tested for the required/not-required, removal-error, staleness, and flow-level cases. No public behavior changed yet.
2026-06-24 02:55:07 +02:00
c3033c356d feat(parsing): begin faithful libyaml scanner port (indent engine)
PureYAML's scanner is a hand-rolled approximation of libyaml's algorithm; its indentation and simple-key bugs are the gap between the two. Rather than keep patching the approximation, re-derive the scanner as a faithful, function-by-function Swift port of libyaml's scanner.c (parser.c to follow), so correctness is by construction. The no-C-runtime constraint is preserved: this translates the algorithm into pure Swift, it does not bind to libyaml.

First piece: the indentation engine under PureYAML.Parsing.LibYAML, a line-for-line port of yaml_parser_roll_indent and yaml_parser_unroll_indent. A block collection opens only when content is strictly more indented than the current level, which fixes the '- a:' / same-column 'b:' mis-nesting by construction. Unit-tested against that exact case (sibling vs nested vs flow-inert). Lands as a parallel implementation verified against libyaml built from the CYaml source; swapped in only once it matches the reference with no regression. docs/LIBYAML_PORT.md records the plan. No public behavior changed yet.
2026-06-24 02:46:29 +02:00
190b80316b fix(parsing): emit no document for comment-only or empty streams
A stream whose only content is comments or whitespace has zero documents in YAML 1.2 (confirmed by both libyaml and the suite's test.event), but PureYAML was emitting a spurious empty null document into the event stream. Stop emitting it; the single-document value API maps an empty document stream to null, so PureYAML.parse("") is unchanged.

Found two ways: the canonical event gate (8G76, 98YD, AVM7) and, independently, differential testing against libyaml built from the CYaml source over the real-world corpus, where a comment-only values.yaml diverged. Raises exact canonical-event conformance to 267 of 308; the real-world differential agreement rises to 113 of 114. Adds a gated PureYAML yeast-event dumper used to drive the libyaml differential.
2026-06-24 02:27:38 +02:00
71e53b7a4c fix(parsing): strip trailing fold from flow plain scalars
A plain scalar inside a flow collection whose next line begins with a flow indicator (',' ']' '}') or the ':' value indicator was retaining the folded line break as a trailing space, so [\nfoo: bar\n] produced the value 'bar ' instead of 'bar'. YAML 1.2 section 7.3.3 strips trailing whitespace from a plain scalar; end the scalar at such a line without folding the break.

Derived from the plain-scalar production, not pattern-matched per case, and verified against the canonical event gate (exact test.event comparison): fixes 4FJ6, 4MUZ/02, CT4Q, LP6E, NKF9, QF4Y, VJP3/01, Y79Y/002, ZF4X, and ZK9H with zero regressions. Exact canonical-event conformance rises from 254 to 264 of 308 valid cases. Adds a gated triage diagnostic to the event gate.
2026-06-24 02:01:08 +02:00
e8987e4cfd feat(parsing): add authoritative canonical-event conformance gate
The conformance number was measured against a lossy oracle: the JSON
projection (in.json) discards anchors, tags, and scalar styles, and exists
for only about half the cases, so "passing" often meant only "parse did not
throw". Measured against the official suite's own test.event event streams,
the true exact conformance is 254 of 308 valid cases (40 emit a wrong event
stream, 14 false-reject), well below the JSON-proxy figure.

Add a canonical-event gate (Tests/CanonicalEventConformanceTests.swift) that
serializes the parser's events into the yaml-test-suite yeast format and
compares them exactly to each case's test.event, checking anchors, tags,
scalar styles, flow vs block, and explicit document markers, for every valid
case. The gate records the gap set and fails on any regression or unrecorded
fix. All 402 test.event files are vendored alongside their inputs.

To make the comparison exact, the parser Event model now records whether a
document boundary was explicit (documentStart/documentEnd carry an `explicit`
flag), distinguishing `---`/`...` from an implicit boundary. This resolves 89
cases that were previously correct but unverifiable without normalization.

No parser behavior changed; this is measurement and model fidelity. The 40
wrong-event and 14 false-reject cases are the real remaining gaps, to be fixed
against this gate.
2026-06-24 01:24:24 +02:00
ce0585f77e test(conformance): track all 402 suite cases; fix 3 regressions
The gap-only manifest stopped guarding a case once it was fixed, so later
work silently regressed previously-fixed cases (caught only by an external
full-suite run, not the gated sweep or full corpus). Replace it with a full
manifest of all 402 official cases (Tests/Fixtures/yaml-test-suite-manifest.yaml),
each carrying its conformant expectation and a passing/gap status, with every
case's files vendored. The gated sweep now asserts every passing case still
passes (the regression guard) and every gap still reproduces, so any regression
fails the build.

The honest gate-enforced count is 329/402 passing (the earlier 348 came from a
more lenient one-time oracle).

Repairs three regressions the new gate revealed: the bucket-6 tab rule
over-rejected tabs in block-scalar content and as separation before a scalar
value (root cause: the batch scanner and the incremental TokenCursor had
divergent line-start loops; unified onto a single scanStep, fixing 96NN/00,
96NN/01, R4YG, Y79Y/001, DK95/00, DK95/03, DK95/07, 6CA3, Q5MG, J3BT); an anchor
on a prior line before an alias key wrongly rejected 26DV; and a tag-only flow
key mis-parsed WZ62. Genuine tab-indentation errors (4EJS, Y79Y/000) still reject.
2026-06-24 00:16:28 +02:00
16da7248c7 fix(parsing): complete flow-collection key and empty-node handling
Implement libyaml's simple-key mechanism for flow context: the scanner records a potential key position before a plain or quoted scalar, an anchor/tag, or a [/{ opener, and a later ':' splices a mappingKey token retroactively in front of that node. This lets a quoted scalar ("k":v), a nested collection ({k}:v), or a property-carrying node (&c c: d) become a flow mapping key. In flow context ':' is always a value indicator and '?' always an explicit-key indicator regardless of the next character (the block-only following-blank lookahead is disabled), so adjacent forms and trailing '?' parse, and multi-line plain keys fold before the ':' is read. The flow parser is rebuilt around the flow_sequence_entry / flow_mapping_key / flow_mapping_value productions, composing empty scalars for absent keys and values. The streaming token cursor now buffers an open flow collection's tokens until it closes so the retroactive key splice stays valid.

Flips 10 official YAML test suite cases (348/402): 4FJ6, 8KB6, 9BXH, 9MMW, CN3R, DFF7, FRK4, NJ66, UT92, CT4Q. Flow collections used as block mapping keys (LX3P, 6BFJ) need the splice in block context and stay tracked. Updates manifest, fixtures, counts, and roadmap.
2026-06-23 23:18:59 +02:00
1875d66d46 fix(parsing): parse multi-line and block scalars at the document root
Model libyaml's parser->indent (the enclosing block-collection column, -1 at the bare root) separately from per-line indentation, via a blockIndent stack that rolls on block-sequence entries and mapping keys, unrolls at each block-line start, and resets at document boundaries. Plain-scalar continuation folding now compares against blockIndent, so a root scalar folds its column-0 continuations (single break to space, blank line to newline, per-line whitespace stripped) while a scalar inside a collection still ends at a sibling line. The ':' value indicator deliberately does not roll the block indent.

A '---' document-start marker may now be followed by the document's root node on the same line ('--- foo', '--- [1, 2]', '--- |', '--- >'), and a block scalar whose body sits at the document root's column 0 is captured and ends at the next '---'/'...' marker.

Flips 33 official YAML test suite cases (338/402): 18 false-rejects and 12 value mismatches now match, and 3 invalid inputs (9KBC, ZVH3, 9MQT/01) are now correctly rejected. M7A3 and NB6Z now parse but with mismatched values (separate multi-doc and tab-folding gaps) and stay tracked. Adds root multi-line plain scalar regression tests; updates manifest, fixtures, counts, and roadmap.
2026-06-23 22:56:33 +02:00
318913baeb fix(emitting): escape CRLF per unicode scalar in double-quoted output
A CR+LF pair forms a single Swift extended grapheme cluster, so quote()'s Character-level switch matched neither '\\r' nor '\\n' and emitted the line break unescaped. The folding parser then read the dumped value back differently, breaking parse/dump/parse stability for any string containing CRLF (surfaced as a round-trip failure on the Zoom OpenAPI corpus seed under PUREYAML_RUN_FULL_CORPUS). Iterating over unicode scalars escapes CR and LF independently. Adds a dumper regression test.
2026-06-23 22:23:40 +02:00
1de1fcf2a8 fix(parsing): reject invalid input that was wrongly accepted
Tighten error detection per the YAML 1.2 spec across four rules: (1) a tab may never supply block-structure indentation (line-start indentation, block-scalar indentation, and separation before a nested block node are rejected; a tab separating an indicator from a scalar value stays valid); (2) a ---/... document marker inside an open quoted scalar or flow collection is rejected instead of silently ending the document; (3) a %YAML/%TAG line that opens a second explicit document with no intervening ... is rejected; (4) an anchor cannot be attached to an alias node.

This repairs 4 of the 5 error-detection regressions introduced by buckets 3-5 (MUS6/01, N782, SU74, Y79Y/009); 9MQT/01 is deferred (its invalidity is in same-line content after the --- marker, which needs document-content scanning that risks the baseline). Rejects 13 accepted-invalid suite cases and fixes 2 more by value (6BCT, Y79Y/010); 305/402 passing.

BEHAVIOR CHANGE: tab-indented block structure (previously parsed permissively) is now rejected per spec; no real-world corpus seed relied on this. Updates the manifest, fixtures, counts, two affected tests (with spec justification), and the roadmap (bucket 6 marked PARTIAL).
2026-06-23 22:13:35 +02:00
b426c04bfa test(conformance): track 5 error-detection regressions from buckets 3-5
A full-suite audit found 5 yaml-test-suite error cases that were correctly rejected at the original baseline but are now wrongly accepted, introduced by the empty-node (bucket 3), anchor/alias-key (bucket 4), and directive/document-boundary (bucket 5) work. They escaped both gates because they were never in the gap manifest. Vendoring them and recording current: accepted so the sweep guards them; they are fixed in the stricter-rejection pass.

Regressions: 9MQT/01 ('...' inside a double-quoted scalar split into documents), MUS6/01 (repeated %YAML across documents), N782 (---/... inside a flow sequence), SU74 (anchored alias as a mapping key), Y79Y/009 (tab after an empty explicit key).
2026-06-23 21:37:00 +02:00
cd05976fac feat(parsing): handle directives and document-end boundaries
Three root causes: (1) a '%' begins a directive only in directive position (before document content, outside flow); everywhere else it is plain content, fixing misfires inside plain scalars, flow mappings, and block scalars. (2) %YAML accepts any 1.x (a higher minor is treated as 1.2 without error), rejects major != 1 and duplicate %YAML, and requires a following '---'; reserved and %TAG directives are accepted and ignored. A directive not followed by a document start is rejected with the new directiveWithoutDocument error. (3) An explicit '...' end marker closes the current document so following content begins a fresh document in parseStream; single-document parse still rejects multi-document input at compose time.

Tests/UnsupportedYAMLGapsFixtures.swift, StreamParsingTests, ScannerTests, and ParsingErrorTests were updated in lockstep to assert the now-correct behavior (percent-after-content is plain text; '...' then content is two documents; %YAML 1.3 is valid while 2.0 and duplicate %YAML are rejected); no test was weakened and multi-document rejection for single-document parse is still covered.

Flips 15 official YAML test suite cases (295/402); SF5V, 9MMA, B63P, H7TQ now correctly rejected. 6LVF and W4TN are reclassified to json-mismatch (blocked by other scalar gaps). Updates the manifest, fixtures, counts, and roadmap.
2026-06-23 21:31:58 +02:00
dfb498a89c fix(parsing): accept anchors, tags, and aliases as mapping keys
A mapping key is a full node: it may carry &anchor and !tag properties and may itself be an *alias. Fixes: (1) anchor/alias name scanning now stops at the first non-name character per yaml_parser_scan_anchor, so *a: reads as alias 'a' plus a mapping indicator instead of name 'a:'; (2) the block-mapping parser enters on a leading node property or alias, attaches same-line key properties to the key node, and emits alias keys directly; (3) an anchor or tag at value position with no same-line node composes an empty (null) value carrying the property, so 'a: &anchor' registers the anchor; (4) the composer registers an anchor's resolved model value (an empty anchored scalar registers null); (5) implicit keys with trailing spaces before the colon are recognized.

Flips 10 official YAML test suite cases to passing (280/402), including G9HC now correctly rejected. 2SXE no longer false-rejects but still value-mismatches on compact-mapping interpretation (a separate gap), so it is reclassified in the manifest rather than removed. Updates the manifest, fixtures, counts, and roadmap.
2026-06-23 21:07:47 +02:00
cdf5f082f8 fix(parsing): support empty (null) nodes in every position
Compose an empty scalar wherever a node is grammatically required but the next token is a structural indicator or line/stream end, mirroring libyaml's process_empty_scalar: after a tag or anchor property with no node, as an absent mapping value (block and flow), as an absent mapping key (both the plain ': v' form and the explicit '?' form), in flow empty key/value forms ({ : }, { , : v }), and for an indented block node following properties. Also fixes a pre-existing block-sequence bug that nested a sibling '-' as a child when only properties remained on the entry line.

Empty scalars resolve through the existing resolver against the empty string, so !!str yields the empty string, bare empty or !!null yields null, and an empty key yields the empty string.

Flips 20 official YAML test suite cases to passing (270/402). CT4Q no longer false-rejects but still value-mismatches on flow plain-scalar folding (a separate gap), so it is reclassified in the manifest rather than removed. Updates the gap manifest, fixtures, counts, and roadmap.
2026-06-23 20:40:40 +02:00
88b67ff0b2 fix(parsing): apply flow folding to quoted scalars
Single (') and double (") quoted multi-line scalars now apply YAML 1.2 flow folding (sections 6.5, 7.3.1, 7.3.2): trailing whitespace before a break is stripped, leading whitespace on continuation lines is stripped, a single break between non-empty lines folds to one space, and N consecutive breaks yield N-1 literal line feeds (first and last lines keep their own leading/trailing spaces as content). A double-quoted backslash at end of line suppresses the break entirely.

Complete the double-quoted escape set: add \xXX and \UXXXXXXXX hex escapes plus the previously missing \0 \a \v \e, literal backslash-space, and named breaks \N \_ \L \P (existing \uXXXX surrogate-pair handling kept). The folding state machine mirrors scanner.c yaml_parser_scan_flow_scalar; implemented in Sources/Parsing/Scalar.swift.

Flips 30 official YAML test suite cases to passing (250/402). Updates the gap manifest, fixtures, count assertions, and roadmap.
2026-06-23 20:17:56 +02:00
3db3e87d39 fix(parsing): correct block scalar folding and chomping
Rewrite literal (|) and folded (>) block scalar content to YAML 1.2 section 8.1: preserve interior blank lines, apply 8.1.3 line folding (a single break between equally indented non-empty lines folds to a space, while blank lines and more-indented lines keep their breaks), preserve tabs in content (only block-indentation columns are stripped), and support the + keep chomping indicator alongside clip and strip.

The block scalar is now scanned as a single unit (header line plus all content lines) with indentation auto-detection in Scanner.BlockScalar.swift, mirroring the reference scanner, instead of being reassembled from a lossy per-line token array.

Flips 24 official YAML test suite cases to passing (220/402). Updates the gap manifest, vendored fixtures, count assertions, and the conformance roadmap.
2026-06-23 20:01:03 +02:00
13a3859e21 test(conformance): track YAML test suite gaps
Vendor the 206 official YAML test suite cases (yaml/yaml-test-suite, data branch, commit 6ad3d2c) that PureYAML does not yet handle correctly, under Tests/Fixtures/yaml-test-suite/, with a manifest recording each gap's category and current behavior.

Add an opt-in conformance sweep (scripts/check-yaml-test-suite.sh, gated behind PUREYAML_RUN_YAML_TEST_SUITE=1) that reproduces every recorded gap, so a future fix must update the manifest. Document the measured result (196/402 passing) and the ordered plan for closing the gaps in docs/YAML_TEST_SUITE_CONFORMANCE.md.

No parser behavior changed.
2026-06-23 19:34:18 +02:00
16a47df118 docs(parsing): document why the nesting bound is necessary and sufficient 2026-06-23 11:08:30 +02:00
64a171bff4 fix(parsing): bound nesting depth so pathological input errors instead of crashing 2026-06-23 09:29:45 +02:00
5d76d184cb fix(parsing): treat leading '#' in block scalars as literal content 2026-06-23 04:00:07 +02:00
c13b7c97fd test: demonstrate yaml/json parsing and generation round-trips
Add a suite covering the four supported conversions around a shared
Codable value: parsing YAML, parsing JSON (via the same API, since YAML
1.2 is a JSON superset), generating YAML, and generating strict JSON
through Foundation's JSONEncoder. Each path asserts exact output and
round-trips back to the original value.
2026-06-08 00:13:56 +02:00
dc8d977f78 fix(license): keep LICENSE to standard MIT text so it is detected as MIT
Move the attribution notice out of LICENSE into ATTRIBUTION.md. The appended
notice dropped the file below GitHub's license-match threshold, so it showed as
Other/NOASSERTION instead of MIT, which blocks dependency adoption.
2026-06-07 20:45:59 +02:00
28cfd3839d chore(hooks): block Cursor attribution locally and in CI
Reject Cursor co-author trailers, cursor.com agent emails, and machine
attribution tells in commit messages and staged content, plus a Cursor
author or committer identity. Add a workflow enforcing the same rule
server-side on push and pull request.
2026-06-07 09:43:29 +02:00
7006d1eb52 chore(style): skip vendored yaml fixture corpora
Keep repo prose checks focused on authored files while preserving upstream fixture contents verbatim.
2026-06-06 08:25:03 +02:00
ca96772207 test(corpus): add public real yaml seeds
Vendor the pinned public YAML seeds used by the real-corpus and Yams differential gates.
2026-06-06 08:24:12 +02:00
b9215d4015 chore(hooks): skip vendored yaml fixture corpora
Exclude checked-in fixture corpus data from prose-style checks so upstream YAML content can remain faithful.
2026-06-06 08:24:08 +02:00
aa74cf4332 test(corpus): add remaining acme catalog fixtures
Add the remaining sanitized ACME catalog modules for multi-file corpus coverage.
2026-06-06 08:16:37 +02:00
7c069dce66 test(corpus): add acme catalog service fixtures
Add sanitized ACME service modules that exercise cross-file OpenAPI-style YAML structures.
2026-06-06 08:03:57 +02:00
8184470cd6 test(corpus): add acme catalog core fixtures
Add the first sanitized ACME catalog modules used for multi-file real-world corpus coverage.
2026-06-06 08:03:27 +02:00
6f76f31c88 test(corpus): add real yaml corpus manifest
Record the checked corpus provenance and fixture metadata used by production hardening gates.
2026-06-06 08:03:21 +02:00
208b892019 feat(parser): harden diagnostics and streaming
Add structured validation diagnostics, JSON and Yams comparison gates, and lazy token/event parsing so production behavior is measured without retaining full intermediate parser arrays.
2026-06-06 07:57:48 +02:00
9f68671c16 docs(corpus): document real seed gates 2026-06-06 01:07:45 +02:00
5f420cdf02 docs(release): update 0.1.1 notes 2026-06-06 00:38:50 +02:00
1bcec9593b ci: use checkout action with node 24 2026-06-06 00:32:53 +02:00
0a649c4d55 docs(release): prepare 0.1.1 2026-06-06 00:28:38 +02:00
36364cfcac docs(readme): add production hardening epic 2026-06-06 00:02:29 +02:00
b47ca4fc31 ci: run optimized build gates 2026-06-05 22:36:28 +02:00
4bc772d807 docs: add platform CI badges 2026-06-05 22:30:41 +02:00
37239fe613 fix(parser): match block continuation parsing 2026-06-05 22:12:12 +02:00
2297 changed files with 270987 additions and 4334 deletions

1
.claude/commands/exp Symbolic link
View File

@ -0,0 +1 @@
../../../../private/mihaela-agents/skills/exp

1
.claude/commands/openapi Symbolic link
View File

@ -0,0 +1 @@
../../../../private/mihaela-agents/skills/openapi

1
.claude/commands/swift Symbolic link
View File

@ -0,0 +1 @@
../../../../private/mihaela-agents/skills/swift

1
.claude/commands/ui Symbolic link
View File

@ -0,0 +1 @@
../../../../private/mihaela-agents/skills/ui

View File

@ -0,0 +1,24 @@
# Managed by rules-swift (scripts/sync-rules.sh). Verifies only that this repo's
# freshness marker matches the current shared-rules version. It does not claim
# semantic rule conformance; each consumer owns and runs its project-specific gates.
name: rules-freshness
on:
push:
pull_request:
workflow_dispatch:
jobs:
rules:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Shared-rules freshness marker
run: |
BASE=https://codeberg.org/Mihaela/rules-swift/raw/branch/main
canon=$(curl -fsSL "$BASE/VERSION" | tr -d '[:space:]')
mine=$(tr -d '[:space:]' < .rules-version 2>/dev/null || echo MISSING)
if [ "$mine" != "$canon" ]; then
echo "STALE: shared-rules baseline is v$canon, this repo tracks v$mine."
echo "Review the baseline diff, fold it into this repo's rules, then set .rules-version to $canon."
exit 1
fi
echo "rules baseline FRESH at v$canon (repo-specific rules preserved, not checked)"

View File

@ -88,4 +88,61 @@ for p in "${PREAMBLE_PATTERNS[@]}"; do
fi
done
# Rule 5.1: Cursor attribution. Reject Cursor co-author trailers, the agent
# email, any @cursor.com address, and "generated with Cursor" style tells.
# Case-insensitive on purpose: cursoragent@cursor.com, Co-authored-by: Cursor.
CURSOR_LITERALS=(
'Co-Authored-By: Cursor'
'Co-authored-by: Cursor'
'Generated with Cursor'
'Generated by Cursor'
'with help from Cursor'
'Powered by Cursor'
)
for p in "${CURSOR_LITERALS[@]}"; do
if printf '%s' "$CONTENT" | grep -qiF -- "$p"; then
printf 'commit-msg: forbidden Cursor attribution: %s\n' "$p" >&2
printf ' Rule: github-discipline.md Rule 5.1\n' >&2
FAIL=1
fi
done
if printf '%s' "$CONTENT" | grep -qiE -- '@cursor\.com'; then
printf 'commit-msg: forbidden Cursor email (@cursor.com) in commit message.\n' >&2
printf ' Rule: github-discipline.md Rule 5.1\n' >&2
FAIL=1
fi
# Author / committer identity must not be Cursor. git var resolves the exact
# identity (including --author / GIT_*_IDENT overrides) this commit will record.
for ident in "$(git var GIT_AUTHOR_IDENT 2>/dev/null)" "$(git var GIT_COMMITTER_IDENT 2>/dev/null)"; do
[ -z "$ident" ] && continue
if printf '%s' "$ident" | grep -qiE -- '@cursor\.com|(^|[^[:alpha:]])cursor(agent)?([^[:alpha:]]|$)'; then
printf 'commit-msg: forbidden Cursor identity as author/committer: %s\n' "$ident" >&2
printf ' Fix: set git config user.name / user.email to your own identity.\n' >&2
FAIL=1
fi
done
# commits.md Rule 1/2: subject must follow Conventional Commits (lowercase type
# from the allowed set, optional scope, optional ! for breaking) and be at most
# 72 characters. Merge/revert/fixup/squash subjects are tooling-generated and
# exempt.
SUBJECT=$(printf '%s\n' "$CONTENT" | sed '/^[[:space:]]*$/d' | head -1)
case "$SUBJECT" in
"Merge "*|"Revert "*|"fixup! "*|"squash! "*|"") ;;
*)
if [ "${#SUBJECT}" -gt 72 ]; then
printf 'commit-msg: subject is %s chars (max 72).\n' "${#SUBJECT}" >&2
printf ' Rule: commits.md Rule 1\n' >&2
FAIL=1
fi
if ! printf '%s' "$SUBJECT" | grep -qE '^(feat|fix|critic-fix|docs|style|refactor|perf|test|build|ci|chore)(\([a-z0-9._-]+\))?!?: .+'; then
printf 'commit-msg: subject must be Conventional Commits: "type(scope): summary".\n' >&2
printf ' Allowed types: feat fix critic-fix docs style refactor perf test build ci chore\n' >&2
printf ' Rule: commits.md Rule 1, Rule 2\n' >&2
FAIL=1
fi
;;
esac
exit $FAIL

View File

@ -62,6 +62,12 @@ SKIP_PATTERNS=(
'^third_party/'
'^node_modules/'
'^_archive/'
'^Tests/Fixtures/acme/'
'^Tests/Fixtures/real-yaml/'
# Vendored upstream rules canon (mirror of rules-swift). Its enforcement docs
# legitimately quote the forbidden phrases. See docs/rules-swift/PROVENANCE.md.
'^docs/rules-swift/'
'^Tests/Fixtures/yaml-test-suite/'
# Rule files that DEFINE the prohibitions; they MUST be able to cite
# the forbidden strings to document them.
'/github-discipline\.md$'
@ -71,8 +77,12 @@ SKIP_PATTERNS=(
# .githooks/pre-commit) rather than the canonical *-style-check names.
'/commit-msg$'
'/pre-commit$'
'/pre-push$'
'/check-style\.sh$'
'/check-staged-style\.sh$'
# The CI workflow that enforces the same Cursor prohibition server-side; it
# must quote the forbidden tokens to scan for them.
'^\.github/workflows/no-cursor-attribution\.yml$'
)
# Filter STAGED_FILES through the skip list.
@ -145,6 +155,16 @@ LITERAL_PATTERNS=(
'This change was generated'
)
# Cursor-attribution literal tells (case-insensitive at the call site).
CURSOR_LITERALS=(
'Co-Authored-By: Cursor'
'Co-authored-by: Cursor'
'Generated with Cursor'
'Generated by Cursor'
'with help from Cursor'
'Powered by Cursor'
)
# AI-tell signature emojis (UTF-8 byte sequences kept in a variable so the
# script source stays clean of the glyphs themselves).
EMOJI_TELLS=$(printf '\xf0\x9f\xa4\x96\n\xe2\x9c\xa8\n\xf0\x9f\xaa\x84\n\xf0\x9f\xa7\xa0\n\xf0\x9f\xa6\xbe\n\xf0\x9f\xa4\x9d')
@ -178,6 +198,21 @@ while IFS= read -r file; do
fi
done
# Cursor-attribution check (mirrors commit-msg). Specific tells only, never
# the bare word "cursor" (a legitimate programming term: text/db cursor).
for p in "${CURSOR_LITERALS[@]}"; do
if printf '%s' "$ADDED" | grep -qiF -- "$p"; then
printf 'pre-commit: forbidden Cursor attribution in staged %s: %s\n' "$file" "$p" >&2
printf ' Rule: github-discipline.md Rule 5.1\n' >&2
FAIL=1
fi
done
if printf '%s' "$ADDED" | grep -qiE -- '@cursor\.com'; then
printf 'pre-commit: forbidden Cursor email (@cursor.com) in staged %s.\n' "$file" >&2
printf ' Rule: github-discipline.md Rule 5.1\n' >&2
FAIL=1
fi
# AI-signature emoji check.
while IFS= read -r emoji; do
[ -z "$emoji" ] && continue
@ -190,6 +225,32 @@ while IFS= read -r file; do
done <<<"$EMOJI_TELLS"
done <<<"$SCAN_FILES"
# SwiftFormat + SwiftLint on staged Swift files (formatting-and-linting.md
# acceptance item 2). Run only when the tool is installed; CI re-runs both as a
# required gate (gates.yml), so a machine lacking the tools cannot land
# unformatted or lint-dirty code regardless.
STAGED_SWIFT=$(printf '%s\n' "$STAGED_FILES" | grep -E '\.swift$' || true)
if [ -n "$STAGED_SWIFT" ]; then
if command -v swiftformat >/dev/null 2>&1; then
if ! printf '%s\n' "$STAGED_SWIFT" | xargs swiftformat --lint --config .swiftformat >/dev/null 2>&1; then
printf 'pre-commit: SwiftFormat found unformatted staged Swift.\n' >&2
printf ' Fix: swiftformat . --config .swiftformat\n' >&2
FAIL=1
fi
else
printf 'pre-commit: swiftformat not installed; format gate skipped (CI enforces).\n' >&2
fi
if command -v swiftlint >/dev/null 2>&1; then
if ! printf '%s\n' "$STAGED_SWIFT" | xargs swiftlint lint --strict --quiet --config .swiftlint.yml >/dev/null 2>&1; then
printf 'pre-commit: SwiftLint violations in staged Swift.\n' >&2
printf ' Fix: swiftlint --config .swiftlint.yml --strict\n' >&2
FAIL=1
fi
else
printf 'pre-commit: swiftlint not installed; lint gate skipped (CI enforces).\n' >&2
fi
fi
if [ "$FAIL" -ne 0 ]; then
printf '\n' >&2
printf 'Hook scope: only ADDED lines in the staged diff are gated.\n' >&2

View File

@ -1,6 +1,6 @@
#!/usr/bin/env bash
# pre-push hook: run the mechanical gates locally before a push reaches GitHub.
# These mirror the CI workflow (.github/workflows/ci.yml) so red is caught here
# These mirror the CI workflows (.github/workflows/) so red is caught here
# first. PureYAML is a root Swift package, so Swift gates run at repo root.
#
# Install (per repo): git config core.hooksPath .githooks

View File

@ -52,6 +52,17 @@ body:
- High
validations:
required: true
- type: dropdown
id: complexity
attributes:
label: Complexity
description: A rough effort estimate. Small is a focused fix; Large spans many files or needs design.
options:
- Small
- Medium
- Large
validations:
required: true
- type: input
id: version
attributes:

View File

@ -1,8 +1,8 @@
blank_issues_enabled: false
contact_links:
- name: Questions and ideas
url: https://github.com/mihaelamj/PureYAML/discussions
about: Ask questions, float ideas, or discuss design in Discussions.
url: https://git.aleahim.com/NodetteHQ/PureYAML/issues
about: Ask questions, float ideas, or discuss design in an issue.
- name: Security issue
url: https://github.com/mihaelamj/PureYAML/blob/main/SECURITY.md
url: https://git.aleahim.com/NodetteHQ/PureYAML/src/branch/main/SECURITY.md
about: Do not file security issues publicly. Follow the security policy instead.

View File

@ -41,3 +41,14 @@ body:
- High
validations:
required: true
- type: dropdown
id: complexity
attributes:
label: Complexity
description: A rough effort estimate. Small is a focused change; Large spans many files or needs design.
options:
- Small
- Medium
- Large
validations:
required: true

View File

@ -1,86 +0,0 @@
name: CI
# PureYAML is a root Swift package. CI must prove macOS, Linux, Windows, and WASM.
on:
push:
branches: [main]
pull_request:
permissions:
contents: read
issues: read
jobs:
macos:
name: macOS
runs-on: macos-latest
steps:
- uses: actions/checkout@v4
- name: Install format and lint tools
run: brew install swiftformat swiftlint
- name: Repository gates
env:
GH_TOKEN: ${{ github.token }}
run: |
bash scripts/check-style.sh
bash scripts/check-namespacing.sh
bash scripts/check-forbidden-patterns.sh
git fetch origin main --depth=1 || true
BASE_REF=origin/main bash scripts/check-changelog-touched.sh
- name: Format and lint
run: |
swiftformat . --config .swiftformat --lint
swiftlint --config .swiftlint.yml --strict
- name: Build and test
run: |
swift build
swift test
linux:
name: Linux
runs-on: ubuntu-latest
container: swift:6.2
steps:
- uses: actions/checkout@v4
- name: Build and test
run: bash scripts/check-linux.sh
windows:
name: Windows
runs-on: windows-2022
timeout-minutes: 45
steps:
- uses: actions/checkout@v4
- name: Install Swift
uses: compnerd/gha-setup-swift@main
with:
swift-version: swift-6.3.2-release
swift-build: 6.3.2-RELEASE
- name: Build and test
shell: pwsh
run: ./scripts/check-windows.ps1
wasm:
name: WASM
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Install Swift toolchain dependencies
run: |
sudo apt-get update
sudo apt-get install -y libcurl4-openssl-dev
- name: Install swiftly
run: |
curl -L https://download.swift.org/swiftly/linux/swiftly-$(uname -m).tar.gz > swiftly.tar.gz
tar zxf swiftly.tar.gz
./swiftly init --quiet-shell-followup --assume-yes --skip-install
- name: Install Swift 6.3.2 and Wasm SDK
run: |
. "${SWIFTLY_HOME_DIR:-$HOME/.local/share/swiftly}/env.sh"
swiftly install --use 6.3.2
swift sdk install https://download.swift.org/swift-6.3.2-release/wasm-sdk/swift-6.3.2-RELEASE/swift-6.3.2-RELEASE_wasm.artifactbundle.tar.gz --checksum a61f0584c93283589f8b2f42db05c1f9a182b506c2957271402992655591dd7c
- name: Build for WASM
run: |
. "${SWIFTLY_HOME_DIR:-$HOME/.local/share/swiftly}/env.sh"
bash scripts/check-wasm.sh

26
.github/workflows/conformance.yml vendored Normal file
View File

@ -0,0 +1,26 @@
name: Conformance
# Canonical YAML test-suite conformance: parse-level, value-level, and the canonical
# event-stream gate. The corpus is fetched, not vendored, so this job fetches it and then
# runs the opt-in conformance tests as a gate. The event gate's recorded-gap set fails the
# build on any regression or any silently-closed gap. One concern, one badge.
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
permissions:
contents: read
jobs:
conformance:
name: Conformance
runs-on: macos-latest
steps:
- uses: actions/checkout@v6
- name: Fetch the canonical yaml-test-suite
run: bash scripts/fetch-yaml-test-suite.sh
- name: Run conformance gates
run: PUREYAML_RUN_YAML_TEST_SUITE=1 swift test --filter Conformance

35
.github/workflows/corpus.yml vendored Normal file
View File

@ -0,0 +1,35 @@
name: Corpus
# Heavy corpus, differential, and throughput gates. Scheduled and manual only;
# not a platform claim, so it carries no README badge.
on:
workflow_dispatch:
schedule:
- cron: "0 6 * * 1"
permissions:
contents: read
jobs:
corpus:
name: Corpus
runs-on: macos-latest
steps:
- uses: actions/checkout@v6
- name: Run corpus gates
run: |
bash scripts/check-json-compatibility.sh
bash scripts/check-corpus.sh
bash scripts/check-yams-differential.sh
bash scripts/check-yams-diagnostics.sh
bash scripts/check-yams-validation.sh
bash scripts/check-throughput.sh
bash scripts/check-performance-phases.sh
- name: Upload corpus artifacts
if: always()
uses: actions/upload-artifact@v6
with:
name: pureyaml-corpus-${{ runner.os }}-${{ github.sha }}
path: .build/pureyaml-artifacts
retention-days: 30

35
.github/workflows/gates.yml vendored Normal file
View File

@ -0,0 +1,35 @@
name: Gates
# Repository style/structure gates plus format and lint. One concern, one badge.
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
permissions:
contents: read
issues: read
jobs:
gates:
name: Gates
runs-on: macos-latest
steps:
- uses: actions/checkout@v6
- name: Install format and lint tools
run: brew install swiftformat swiftlint
- name: Repository gates
env:
GH_TOKEN: ${{ github.token }}
run: |
bash scripts/check-style.sh
bash scripts/check-namespacing.sh
bash scripts/check-forbidden-patterns.sh
git fetch origin main --depth=1 || true
BASE_REF=origin/main bash scripts/check-changelog-touched.sh
- name: Format and lint
run: |
swiftformat . --config .swiftformat --lint
swiftlint --config .swiftlint.yml --strict

39
.github/workflows/labeler.yml vendored Normal file
View File

@ -0,0 +1,39 @@
name: Issue labeler
# Rule 1.6 backstop: issue-form dropdown selections produce body TEXT under
# `### <field>` headings, not labels. This translates the Priority dropdown into
# the `priority: high` label. Complexity is intentionally a body field only, not
# a label (Rule 2.1), so it is not mapped here.
on:
issues:
types: [opened, edited]
permissions:
issues: write
jobs:
label:
name: Apply form labels
runs-on: ubuntu-latest
steps:
- name: Apply priority label from the issue form
env:
GH_TOKEN: ${{ github.token }}
REPO: ${{ github.repository }}
NUMBER: ${{ github.event.issue.number }}
BODY: ${{ github.event.issue.body }}
run: |
# Print the first non-empty line after a "### <heading>" line.
value_under() {
printf '%s\n' "$BODY" | awk -v h="### $1" '
$0 == h { found = 1; next }
found && NF { print; exit }
'
}
priority=$(value_under "Priority")
if [ "$priority" = "High" ]; then
gh issue edit "$NUMBER" --repo "$REPO" --add-label "priority: high"
else
gh issue edit "$NUMBER" --repo "$REPO" --remove-label "priority: high" || true
fi

22
.github/workflows/linux.yml vendored Normal file
View File

@ -0,0 +1,22 @@
name: Linux
# Linux build and test in the official Swift container. One concern, one badge.
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
permissions:
contents: read
jobs:
linux:
name: Linux
runs-on: ubuntu-latest
container: swift:6.2
steps:
- uses: actions/checkout@v6
- name: Build and test
run: bash scripts/check-linux.sh

25
.github/workflows/macos.yml vendored Normal file
View File

@ -0,0 +1,25 @@
name: macOS
# macOS build and test (debug and release). One concern, one badge.
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
permissions:
contents: read
jobs:
macos:
name: macOS
runs-on: macos-latest
steps:
- uses: actions/checkout@v6
- name: Build and test
run: |
swift build
swift test
swift build -c release
swift test -c release

View File

@ -0,0 +1,60 @@
name: No Cursor attribution
# Server-side companion to the local .githooks/commit-msg and pre-commit guards.
# Fails the check if any commit pushed (or in a PR) carries Cursor attribution:
# a Cursor co-author trailer, an @cursor.com email, or a "generated with Cursor"
# tell, in the commit message or in the author/committer identity.
on:
push:
pull_request:
workflow_dispatch:
permissions:
contents: read
jobs:
scan:
name: Scan commits
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Reject Cursor attribution
env:
EVENT: ${{ github.event_name }}
PR_BASE: ${{ github.event.pull_request.base.sha }}
PR_HEAD: ${{ github.event.pull_request.head.sha }}
PUSH_BEFORE: ${{ github.event.before }}
SHA: ${{ github.sha }}
run: |
set -euo pipefail
# Resolve the set of commits to scan.
if [ "$EVENT" = "pull_request" ]; then
range="${PR_BASE}..${PR_HEAD}"
elif [ -n "${PUSH_BEFORE:-}" ] && [ "$PUSH_BEFORE" != "0000000000000000000000000000000000000000" ]; then
range="${PUSH_BEFORE}..${SHA}"
else
# New branch / tag push or manual run: scan just the tip commit.
range="${SHA}~1..${SHA}"
git rev-parse "${SHA}~1" >/dev/null 2>&1 || range="$SHA"
fi
echo "Scanning commit range: $range"
# Emit message + author + committer for each commit, then grep.
fmt='commit %H%nauthor: %an <%ae>%ncommitter: %cn <%ce>%n%B'
if printf '%s' "$range" | grep -q '\.\.'; then
log=$(git log --format="$fmt" "$range")
else
log=$(git log -1 --format="$fmt" "$range")
fi
pattern='cursoragent@cursor\.com|@cursor\.com|Co-authored-by:[[:space:]]*Cursor|Generated (with|by) Cursor|with help from Cursor|Powered by Cursor'
if printf '%s\n' "$log" | grep -niE -- "$pattern"; then
echo "::error::Cursor attribution found in the commits above. Strip it (rewrite the commits) before pushing."
exit 1
fi
echo "No Cursor attribution found."

38
.github/workflows/wasm.yml vendored Normal file
View File

@ -0,0 +1,38 @@
name: WASM
# WASM build gate (Rule 8.3): the WASI SDK compiles the package; it does not run
# the test suite. A green badge here means "compiles for WASM", not "tests pass".
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
permissions:
contents: read
jobs:
wasm:
name: WASM
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v6
- name: Install Swift toolchain dependencies
run: |
sudo apt-get update
sudo apt-get install -y libcurl4-openssl-dev
- name: Install swiftly
run: |
curl -L https://download.swift.org/swiftly/linux/swiftly-$(uname -m).tar.gz > swiftly.tar.gz
tar zxf swiftly.tar.gz
./swiftly init --quiet-shell-followup --assume-yes --skip-install
- name: Install Swift 6.3.2 and Wasm SDK
run: |
. "${SWIFTLY_HOME_DIR:-$HOME/.local/share/swiftly}/env.sh"
swiftly install --use 6.3.2
swift sdk install https://download.swift.org/swift-6.3.2-release/wasm-sdk/swift-6.3.2-RELEASE/swift-6.3.2-RELEASE_wasm.artifactbundle.tar.gz --checksum a61f0584c93283589f8b2f42db05c1f9a182b506c2957271402992655591dd7c
- name: Build for WASM
run: |
. "${SWIFTLY_HOME_DIR:-$HOME/.local/share/swiftly}/env.sh"
bash scripts/check-wasm.sh

28
.github/workflows/windows.yml vendored Normal file
View File

@ -0,0 +1,28 @@
name: Windows
# Windows build and test on the Swift 6.3.2 toolchain. One concern, one badge.
on:
push:
branches: [main]
pull_request:
workflow_dispatch:
permissions:
contents: read
jobs:
windows:
name: Windows
runs-on: windows-2022
timeout-minutes: 45
steps:
- uses: actions/checkout@v6
- name: Install Swift
uses: compnerd/gha-setup-swift@main
with:
swift-version: swift-6.3.2-release
swift-build: 6.3.2-RELEASE
- name: Build and test
shell: pwsh
run: ./scripts/check-windows.ps1

1
.rules-version Normal file
View File

@ -0,0 +1 @@
2.14.0

View File

@ -14,5 +14,10 @@ force_cast: error
line_length:
warning: 180
error: 180
# Exhaustive switches over closed enums are the intended way to make
# impossible states unrepresentable; counting their cases as complexity
# pushes code toward split mappings with trapping defaults instead.
cyclomatic_complexity:
ignores_case_statements: true
trailing_comma:
mandatory_comma: true

View File

@ -16,8 +16,20 @@ Yams and libyaml source remain available from their upstream projects:
- https://github.com/jpsim/Yams
- https://pyyaml.org/wiki/LibYAML
The license file includes a matching attribution notice.
This file is the canonical attribution notice for the project; the `LICENSE`
file is kept to the standard MIT text so it is correctly detected as MIT.
The private `PureYAMLResearch` repository contains a full Yams source snapshot
for study and compatibility research. That repository is not the implementation
source for PureYAML.
## Vendored test data
`Tests/Fixtures/yaml-test-suite/` contains input cases vendored from the official
[YAML test suite](https://github.com/yaml/yaml-test-suite) (`data` branch, commit
`6ad3d2c62885d82fc349026c136ef560838fdf3d`), which is distributed under the MIT
license. All 402 cases (numbered sub-cases counted) are vendored, each as its input
(`in.yaml`), expected JSON projection (`in.json`), and/or expected-error marker
(`error`). `Tests/Fixtures/yaml-test-suite-manifest.yaml` records the provenance and
each case's conformance status. These files are test data, not implementation
source.

File diff suppressed because it is too large Load Diff

16
LICENSE
View File

@ -19,19 +19,3 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Attribution Notice
------------------
PureYAML is an independent Swift implementation. It is informed by the behavior
and public API surface of Yams, and by the YAML parsing model embodied in Yams'
bundled CYaml / libyaml-derived C sources.
PureYAML does not copy Yams Swift source code, CYaml C source code, or libyaml C
source code into this public package.
Yams:
https://github.com/jpsim/Yams
libyaml:
https://pyyaml.org/wiki/LibYAML

View File

@ -9,20 +9,36 @@ let package = Package(
name: "PureYAML",
targets: ["PureYAML"],
),
.library(
name: "PureYAMLJSON",
targets: ["PureYAMLJSON"],
),
],
dependencies: [],
targets: [
.target(
name: "PureYAML",
path: "Sources",
exclude: ["PureYAMLJSON"],
),
.target(
name: "PureYAMLJSON",
dependencies: ["PureYAML"],
path: "Sources/PureYAMLJSON",
),
.testTarget(
name: "PureYAMLTests",
dependencies: ["PureYAML"],
path: "Tests",
exclude: ["PureYAMLJSONTests"],
resources: [
.copy("Fixtures"),
],
),
.testTarget(
name: "PureYAMLJSONTests",
dependencies: ["PureYAMLJSON"],
path: "Tests/PureYAMLJSONTests",
),
],
)

View File

@ -1,6 +1,12 @@
# PureYAML
[![CI](https://img.shields.io/github/actions/workflow/status/mihaelamj/PureYAML/ci.yml?branch=main&label=macOS%20Linux%20Windows%20WASM)](https://github.com/mihaelamj/PureYAML/actions/workflows/ci.yml)
[![Gates](https://git.aleahim.com/NodetteHQ/PureYAML/actions/workflows/gates.yml/badge.svg?branch=main)](https://git.aleahim.com/NodetteHQ/PureYAML/actions?workflow=gates.yml)
[![macOS](https://git.aleahim.com/NodetteHQ/PureYAML/actions/workflows/macos.yml/badge.svg?branch=main)](https://git.aleahim.com/NodetteHQ/PureYAML/actions?workflow=macos.yml)
[![Linux](https://git.aleahim.com/NodetteHQ/PureYAML/actions/workflows/linux.yml/badge.svg?branch=main)](https://git.aleahim.com/NodetteHQ/PureYAML/actions?workflow=linux.yml)
[![Windows](https://git.aleahim.com/NodetteHQ/PureYAML/actions/workflows/windows.yml/badge.svg?branch=main)](https://git.aleahim.com/NodetteHQ/PureYAML/actions?workflow=windows.yml)
[![WASM](https://git.aleahim.com/NodetteHQ/PureYAML/actions/workflows/wasm.yml/badge.svg?branch=main)](https://git.aleahim.com/NodetteHQ/PureYAML/actions?workflow=wasm.yml)
[![Conformance](https://git.aleahim.com/NodetteHQ/PureYAML/actions/workflows/conformance.yml/badge.svg?branch=main)](https://git.aleahim.com/NodetteHQ/PureYAML/actions?workflow=conformance.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
PureYAML is a dependency-free YAML package written entirely in Swift.
@ -14,34 +20,9 @@ C-backed parsers. The package is intentionally strict about portability:
- root Swift package layout
- macOS, Linux, Windows, and WASI build gates
## Roadmap
Mermaid status legend:
The roadmap uses the TileDown Mermaid palette: green for merged work, yellow for
review, purple for epic grouping, and gray for open work with no PR.
```mermaid
flowchart TB
classDef done fill:#ddf9e4,stroke:#34c759,color:#111827
classDef review fill:#fff7d6,stroke:#ffcc00,color:#111827
classDef epic fill:#f2e5ff,stroke:#af52de,color:#111827
classDef todo fill:#f2f4f7,stroke:#8e8e93,color:#111827
%% Completed parser replacement epics: #1 #8 #9 #10 #11 #12 #13
LDone["In main now"]:::done
LReview["PR in review"]:::review
LEpic["Epic grouping"]:::epic
LTodo["Open issue, no PR"]:::todo
LDone ~~~ LReview
LReview ~~~ LEpic
LEpic ~~~ LTodo
```
Parser Replacement Roadmap #8 is complete in main.
## Status
The current release is 0.1.0. It includes block mappings, block sequences,
The current release is 0.2.0. It includes block mappings, block sequences,
ordered mappings, common scalars, quoted strings, comments, flow collections,
literal and folded block scalars, anchors, aliases, YAML directives, document
markers, explicit built-in scalar tags, merge-key expansion, complex mapping keys,
@ -67,6 +48,18 @@ Unsupported gaps are pinned by executable tests so they produce exact errors,
exact validation issues, or exact fallback value trees instead of silent parser
drift.
PureYAML now has checked-corpus Yams parse-success parity, structured
diagnostics and validation, JSON compatibility proof gates, internal
token/event streaming, real-corpus property tests, and repeatable performance
comparison artifacts. Remaining work is tracked under the
[production hardening and validation readiness](https://git.aleahim.com/NodetteHQ/PureYAML/issues/8)
epic:
[true chunked input streaming](https://git.aleahim.com/NodetteHQ/PureYAML/issues/4),
[throughput toward libyaml parity](https://git.aleahim.com/NodetteHQ/PureYAML/issues/5),
[a broader bad-YAML mutation suite](https://git.aleahim.com/NodetteHQ/PureYAML/issues/6),
and an
[OpenAPIDoctor integration dry run](https://git.aleahim.com/NodetteHQ/PureYAML/issues/7).
## Attribution
PureYAML is informed by Yams and its bundled `CYaml` / libyaml-derived parser,
@ -77,6 +70,8 @@ but it does not copy their implementation into `Sources/`. See
- [Usage](docs/USAGE.md): public APIs, validation, emitter options, and
cross-platform gates.
- [Corpus gates](docs/CORPUS.md): real-world YAML seed sources, generated
mutations, artifact format, and hardening acceptance criteria.
- [Migration and support boundaries](docs/MIGRATION.md): what PureYAML can
replace today and which YAML features are deliberately unsupported.
- [Release process](docs/RELEASE.md): verification commands, tag commands, and
@ -262,7 +257,6 @@ bash scripts/check-style.sh
bash scripts/check-namespacing.sh
bash scripts/check-forbidden-patterns.sh
bash scripts/check-changelog-touched.sh
bash scripts/check-roadmap.sh
swiftformat . --config .swiftformat --lint
swiftlint --config .swiftlint.yml --strict
swift build

View File

@ -5,7 +5,7 @@
If you believe you have found a security issue in PureYAML, please report it
privately. Do not open a public issue for security problems.
Email **mihaelamj.com** with:
Email **mihaelamj@gmail.com** with:
- A description of the issue and its impact.
- Steps to reproduce, or a proof of concept.
@ -23,6 +23,13 @@ supported.
## Scope
PureYAML generates static sites. The generated output is plain HTML, CSS, and
client-side JavaScript for rich tiles. Reports about generated output (for
example, missing output escaping that enables script injection) are in scope.
PureYAML parses, validates, and emits YAML. Reports about how the library
handles untrusted input are in scope, for example:
- Malformed or adversarial input that causes a crash, an unbounded loop, stack
overflow, or excessive memory or CPU use.
- Parser or emitter behavior that misrepresents input in a way a consumer could
rely on for a security decision (for example, a value that round-trips to a
different meaning).
- Incorrect rejection or acceptance of input by the validation layer that a
consumer treats as a trust boundary.

View File

@ -9,6 +9,17 @@ public extension PureYAML {
try Parsing.Parser().parseStream(yaml)
}
/// Parses a YAML document into a located value tree, annotating every node with its source
/// span. The plain ``parse(_:)`` is unaffected; positions are strictly opt-in.
static func parseLocated(_ yaml: String) throws -> Model.LocatedValue {
try Parsing.Parser().parseLocated(yaml)
}
/// Parses a YAML stream into one located value tree per document, in document order.
static func parseStreamLocated(_ yaml: String) throws -> [Model.LocatedValue] {
try Parsing.Parser().parseStreamLocated(yaml)
}
/// Serializes a ``Model/Value`` tree into YAML with the selected options.
static func dump(
_ value: Model.Value,
@ -25,13 +36,23 @@ public extension PureYAML {
Emitting.Dumper(options: options).dump(documents)
}
/// Serializes a tag-preserving node tree into YAML, re-emitting anchors,
/// aliases (as `*name` references, not expanded copies), explicit tags,
/// and per-node presentation styles.
static func dump(
_ node: Tagged.Node,
options: Emitting.Options = .default,
) -> String {
Tagged.Dumper(options: options).dump(node)
}
/// Validates a parsed YAML value with the default validation rules.
@discardableResult
static func validate(
_ value: Model.Value,
using validator: Validation.Validator = .init(),
using validator: Validation.Validator<Model.Value, Model.Value> = .init(),
strict: Bool = true,
) throws -> [Validation.Issue] {
) throws -> [Validation.ValidationError] {
try validator.validate(value, strict: strict)
}
@ -39,7 +60,7 @@ public extension PureYAML {
@discardableResult
static func validate(
_ documents: [Stream.Document],
using validator: Validation.Validator = .init(),
using validator: Validation.Validator<Model.Value, Model.Value> = .init(),
strict: Bool = true,
) throws -> [Stream.Issue] {
try validator.validate(documents, strict: strict)

View File

@ -79,20 +79,20 @@ extension PureYAML.Decoding {
_ value: PureYAML.Model.Value,
codingPath: [any CodingKey],
) throws {
let result = PureYAML.Validation.Validator().collect(value)
guard !result.issues.isEmpty else {
let result = PureYAML.Validation.Validator<PureYAML.Model.Value, PureYAML.Model.Value>().collect(value)
guard !result.values.isEmpty else {
return
}
let pathPrefix = PureYAML.Validation.Path(codingPath: codingPath)
let issues = result.issues.map { issue in
PureYAML.Validation.Issue(
severity: issue.severity,
reason: issue.reason,
path: .init(pathPrefix.components + issue.path.components),
let errors = result.values.map { error in
PureYAML.Validation.ValidationError(
severity: error.severity,
reason: error.reason,
path: .init(pathPrefix.components + error.path.components),
)
}
throw PureYAML.Validation.Issue.Collection(issues)
throw PureYAML.Validation.ValidationErrorCollection(errors)
}
}

View File

@ -0,0 +1,93 @@
extension PureYAML.Emitting.Dumper {
/// The YAML-specification canonical form, matching libyaml's canonical
/// emitter layout: an explicit `---` document start, an explicit tag on
/// every node, double-quoted scalars, and flow collections with one
/// entry per line, mapping entries in the explicit `? key : value,`
/// form. Tags derive from each value's case, the same resolution the
/// parser applies in reverse.
func canonicalDocument(_ value: PureYAML.Model.Value) -> String {
"---\n" + canonicalBody(value)
}
/// The canonical body without the leading `---` marker, so the marker is
/// owned by the single-document and stream paths, not baked into the body.
func canonicalBody(_ value: PureYAML.Model.Value) -> String {
canonicalLines(value, indent: 0).joined(separator: "\n") + "\n"
}
func canonicalLines(
_ value: PureYAML.Model.Value,
indent: Int,
) -> [String] {
let prefix = String(repeating: " ", count: indent)
switch value {
case let .sequence(values):
return canonicalSequenceLines(values, indent: indent)
case let .mapping(mapping):
return canonicalMappingLines(mapping, indent: indent)
default:
return [prefix + canonicalScalar(value)]
}
}
func canonicalSequenceLines(
_ values: [PureYAML.Model.Value],
indent: Int,
) -> [String] {
let prefix = String(repeating: " ", count: indent)
guard !values.isEmpty else {
return [prefix + "!!seq []"]
}
var lines = [prefix + "!!seq ["]
for value in values {
var entry = canonicalLines(value, indent: indent + options.indentWidth)
entry[entry.count - 1] += ","
lines.append(contentsOf: entry)
}
lines.append(prefix + "]")
return lines
}
func canonicalMappingLines(
_ mapping: PureYAML.Model.Mapping,
indent: Int,
) -> [String] {
let prefix = String(repeating: " ", count: indent)
guard !mapping.pairs.isEmpty else {
return [prefix + "!!map {}"]
}
let inner = indent + options.indentWidth
let innerPrefix = String(repeating: " ", count: inner)
var lines = [prefix + "!!map {"]
for pair in orderedPairs(mapping) {
var keyLines = canonicalLines(pair.keyNode.value, indent: inner)
keyLines[0] = innerPrefix + "? " + String(keyLines[0].dropFirst(innerPrefix.count))
lines.append(contentsOf: keyLines)
var valueLines = canonicalLines(pair.value, indent: inner)
valueLines[0] = innerPrefix + ": " + String(valueLines[0].dropFirst(innerPrefix.count))
valueLines[valueLines.count - 1] += ","
lines.append(contentsOf: valueLines)
}
lines.append(prefix + "}")
return lines
}
/// A canonical scalar: the value's implicit tag made explicit, content
/// double-quoted so no resolution ambiguity survives.
func canonicalScalar(_ value: PureYAML.Model.Value) -> String {
switch value {
case .null:
"!!null \"null\""
case let .bool(flag):
"!!bool \"\(flag ? "true" : "false")\""
case let .int(number):
"!!int \"\(number)\""
case let .double(number):
"!!float \"\(number)\""
case let .string(string):
"!!str " + quote(string)
case .sequence, .mapping:
""
}
}
}

View File

@ -8,11 +8,30 @@ extension PureYAML.Emitting.Dumper {
return [prefix + "{}"]
}
return mapping.pairs.flatMap { pair in
return orderedPairs(mapping).flatMap { pair in
renderMappingPair(pair, indent: indent)
}
}
/// `mapping.pairs` in insertion order, or sorted by their rendered
/// string-keyed form when `options.sortKeys` is set. Only string keys
/// are sortable this way; a mapping with any non-string key is left in
/// insertion order regardless of the option, since there is no
/// well-defined lexicographic ordering across mixed key kinds
/// (`Model.Key.mapping`/`.sequence`) to sort by.
func orderedPairs(_ mapping: PureYAML.Model.Mapping) -> [PureYAML.Model.Pair] {
guard options.sortKeys, mapping.pairs.allSatisfy({ pair in
if case .string = pair.keyNode {
true
} else {
false
}
}) else {
return mapping.pairs
}
return mapping.pairs.sorted { $0.key < $1.key }
}
func renderMappingPair(
_ pair: PureYAML.Model.Pair,
indent: Int,
@ -36,7 +55,7 @@ extension PureYAML.Emitting.Dumper {
switch pair.value {
case .mapping, .sequence:
return [prefix + escapeKey(pair.key) + ":"] + render(pair.value, indent: indent + 2)
return [prefix + escapeKey(pair.key) + ":"] + render(pair.value, indent: indent + options.indentWidth)
default:
return renderMappingScalarPair(pair, indent: indent)
}
@ -54,11 +73,11 @@ extension PureYAML.Emitting.Dumper {
return [prefix + "? " + pair.keyNode.flowDescription]
+ [prefix + ":"]
+ render(pair.value, indent: indent + 2)
+ render(pair.value, indent: indent + options.indentWidth)
}
func renderFlowMapping(_ mapping: PureYAML.Model.Mapping) -> String {
let pairs = mapping.pairs
let pairs = orderedPairs(mapping)
.map { pair in
"\(renderFlowKey(pair.keyNode)): \(renderFlow(pair.value))"
}

View File

@ -0,0 +1,77 @@
extension PureYAML.Emitting {
/// The deepest collection nesting the recursive renderers walk before a
/// value is refused. It matches the parser's ``PureYAML/Parsing/Parser/maxNestingDepth``
/// exactly: a value nested deeper than the parser accepts can never round-trip
/// through PureYAML, so emitting it would produce YAML the paired parser
/// rejects. Parsed trees are always within this bound, so real round-trips
/// are unaffected.
static var maxNestingDepth: Int {
PureYAML.Parsing.Parser.maxNestingDepth
}
/// The reparseable marker returned for a value nested past ``maxNestingDepth``.
/// It is a valid YAML document that resolves to `null`, with a comment that
/// names the refused limit, so a caller never receives a crash or silent
/// truncation.
static var nestingLimitDocument: String {
"# value nesting exceeds the emitter limit of \(maxNestingDepth); not serialized\nnull\n"
}
/// Iteratively (no recursion, so this cannot itself overflow) reports whether
/// a model value nests collections deeper than ``maxNestingDepth``. Complex
/// keys count toward depth like values; scalars are leaves.
static func exceedsNestingLimit(_ value: PureYAML.Model.Value) -> Bool {
var stack: [(value: PureYAML.Model.Value, depth: Int)] = [(value, 1)]
while let (node, depth) = stack.popLast() {
switch node {
case let .sequence(values):
if depth > maxNestingDepth {
return true
}
for child in values {
stack.append((child, depth + 1))
}
case let .mapping(mapping):
if depth > maxNestingDepth {
return true
}
for pair in mapping.pairs {
stack.append((pair.value, depth + 1))
stack.append((pair.keyNode.value, depth + 1))
}
case .null, .bool, .int, .double, .string:
continue
}
}
return false
}
/// Iteratively reports whether a tagged node nests collections deeper than
/// ``maxNestingDepth``. An alias renders as a `*name` reference, not an
/// expanded subtree, so it is a leaf here and its aliased node is not walked.
static func exceedsNestingLimit(_ node: PureYAML.Tagged.Node) -> Bool {
var stack: [(node: PureYAML.Tagged.Node, depth: Int)] = [(node, 1)]
while let (current, depth) = stack.popLast() {
switch current {
case let .sequence(sequence):
if depth > maxNestingDepth {
return true
}
for child in sequence.values {
stack.append((child, depth + 1))
}
case let .mapping(mapping):
if depth > maxNestingDepth {
return true
}
for pair in mapping.pairs {
stack.append((pair.value, depth + 1))
stack.append((pair.keyNode, depth + 1))
}
case .scalar, .alias:
continue
}
}
return false
}
}

View File

@ -0,0 +1,188 @@
// MARK: - Scalar analysis
extension PureYAML.Emitting.Dumper {
func canRenderLiteralBlockString(_ value: String) -> Bool {
guard value.contains("\n"), !value.contains("\r"), !value.contains("\t") else {
return false
}
let contentSource = value.hasSuffix("\n") ? String(value.dropLast()) : value
guard !contentSource.isEmpty else {
return false
}
return contentSource
.split(separator: "\n", omittingEmptySubsequences: false)
.allSatisfy(canRenderLiteralBlockLine)
}
func canRenderLiteralBlockLine(_ line: Substring) -> Bool {
guard !line.isEmpty, line.first?.isWhitespace == false, line.last?.isWhitespace == false else {
return false
}
let value = String(line)
guard !containsCommentStart(value), !containsMappingSeparator(value) else {
return false
}
guard let first = line.first else {
return false
}
return canRenderLiteralBlockLineStarting(with: first, line: line)
}
func canRenderLiteralBlockLineStarting(
with first: Character,
line: Substring,
) -> Bool {
switch first {
case "#", ",", "[", "]", "{", "}", "&", "*", "!", "|", ">", "'", "\"", "`":
false
case "-":
!hasOnlyIndicator(first, line: line) && line.dropFirst().first != " "
case "?":
!hasOnlyIndicator(first, line: line) && line.dropFirst().first != " "
case ":":
!hasOnlyIndicator(first, line: line) && line.dropFirst().first != " "
default:
true
}
}
func hasOnlyIndicator(
_ indicator: Character,
line: Substring,
) -> Bool {
line.count == 1 && line.first == indicator
}
func containsCommentStart(_ value: String) -> Bool {
var previousWasWhitespace = true
for character in value {
if character == "#", previousWasWhitespace {
return true
}
previousWasWhitespace = character.isWhitespace
}
return false
}
func containsMappingSeparator(_ value: String) -> Bool {
var previousWasColon = false
for character in value {
if previousWasColon, isMappingSeparatorBoundary(character) {
return true
}
previousWasColon = character == ":"
}
return previousWasColon
}
func isMappingSeparatorBoundary(_ character: Character) -> Bool {
character.isWhitespace
|| character == ","
|| character == "]"
|| character == "}"
}
func canRenderPlainString(_ value: String) -> Bool {
guard !value.isEmpty else {
return false
}
guard value.first?.isWhitespace == false, value.last?.isWhitespace == false else {
return false
}
// By unicode scalar: CR+LF is one Character in Swift, so `contains("\n")` is false for it and a string with a CRLF break would be written raw.
guard !value.unicodeScalars.contains(where: { $0 == "\n" || $0 == "\r" || $0 == "\t" }) else {
return false
}
guard !isReservedPlainScalar(value), !isNumberLikePlainScalar(value) else {
return false
}
guard let first = value.first, !isPlainScalarIndicator(first) else {
return false
}
// A trailing `:` reparses as a mapping key with an empty value, so a
// scalar that ends in one cannot render plain even though no `:` is
// followed by a space inside it.
guard value.last != ":" else {
return false
}
// `<<` as a key is the merge key, and `...` at the start of a line ends the document.
guard value != "<<", value != "...", !value.hasPrefix("... ") else {
return false
}
return !containsColonSpace(value) && !value.contains("#")
}
func canRenderFlowPlainString(_ value: String) -> Bool {
canRenderPlainString(value) && !containsFlowDelimiter(value)
}
func containsFlowDelimiter(_ value: String) -> Bool {
value.contains(",")
|| value.contains("[")
|| value.contains("]")
|| value.contains("{")
|| value.contains("}")
}
func containsColonSpace(_ value: String) -> Bool {
var previousWasColon = false
for character in value {
if previousWasColon, character == " " {
return true
}
previousWasColon = character == ":"
}
return false
}
/// A word the parser reads as something other than a string (null, a boolean, a number): written plain, a string of it would come back changed.
func isReservedPlainScalar(_ value: String) -> Bool {
!PureYAML.Parsing.PlainScalar.readsAsString(value)
}
func isNumberLikePlainScalar(_ value: String) -> Bool {
if Int(value) != nil, !value.contains(".") {
return true
}
if value.contains(".") || value.contains("e") || value.contains("E") {
return Double(value) != nil
}
return false
}
func isPlainScalarIndicator(_ character: Character) -> Bool {
"-?:,[]{}#&*!|>'\"%@`".contains(character)
}
func escapeKey(_ key: String) -> String {
canRenderPlainString(key) ? key : quote(key)
}
func quote(_ value: String) -> String {
// Iterate over unicode scalars, not Characters: a CR+LF pair forms a
// single extended grapheme cluster in Swift, so a Character-level switch
// would match neither "\r" nor "\n" and emit the line break unescaped,
// which does not round-trip through the folding parser.
var output = "\""
for scalar in value.unicodeScalars {
switch scalar {
case "\"":
output += "\\\""
case "\\":
output += "\\\\"
case "\n":
output += "\\n"
case "\r":
output += "\\r"
case "\t":
output += "\\t"
default:
output.unicodeScalars.append(scalar)
}
}
output += "\""
return output
}
}

View File

@ -1,9 +1,18 @@
public extension PureYAML.Emitting.Dumper {
/// Serializes indexed YAML stream documents with explicit document starts.
extension PureYAML.Emitting.Dumper {
/// Serializes a stream of documents. The stream path owns the document
/// markers: it emits one `---` start per document and, when
/// `explicitDocumentEnd` is set, one `...` end per document. Each document
/// body is rendered marker-free so `explicitDocumentStart` and `canonical`
/// cannot stack a second `---` onto the separator, which previously turned
/// every document into two (an empty one plus the real one) on reparse.
func dump(_ documents: [PureYAML.Stream.Document]) -> String {
documents
.map { document in
"---\n" + dump(document.value)
let body = PureYAML.Emitting.exceedsNestingLimit(document.value)
? PureYAML.Emitting.nestingLimitDocument
: documentBody(document.value)
let end = options.explicitDocumentEnd ? "...\n" : ""
return "---\n" + body + end
}
.joined()
}

View File

@ -0,0 +1,147 @@
extension PureYAML.Emitting.Dumper {
/// Width-aware flow rendering: when a flow collection's single-line
/// form (with its indent) overflows `lineWidth`, it re-renders with one
/// entry per line, recursively, which is equally valid flow YAML. With
/// wrapping disabled or content that fits, the single-line form is
/// unchanged.
func renderFlowWrappedDocument(_ value: PureYAML.Model.Value) -> String {
renderFlowWrappedLines(value, indent: 0).joined(separator: "\n") + "\n"
}
func renderFlowWrappedLines(
_ value: PureYAML.Model.Value,
indent: Int,
) -> [String] {
let prefix = String(repeating: " ", count: indent)
let single = renderFlow(value)
if indent + single.count <= options.lineWidth {
return [prefix + single]
}
switch value {
case let .sequence(values):
guard !values.isEmpty else {
return [prefix + "[]"]
}
var lines = [prefix + "["]
for item in values {
var entry = renderFlowWrappedLines(item, indent: indent + options.indentWidth)
entry[entry.count - 1] += ","
lines.append(contentsOf: entry)
}
lines.append(prefix + "]")
return lines
case let .mapping(mapping):
guard !mapping.pairs.isEmpty else {
return [prefix + "{}"]
}
let inner = indent + options.indentWidth
let innerPrefix = String(repeating: " ", count: inner)
var lines = [prefix + "{"]
for pair in orderedPairs(mapping) {
let key = renderFlowKey(pair.keyNode)
var entry = renderFlowWrappedLines(pair.value, indent: inner)
entry[0] = innerPrefix + key + ": " + String(entry[0].dropFirst(innerPrefix.count))
entry[entry.count - 1] += ","
lines.append(contentsOf: entry)
}
lines.append(prefix + "}")
return lines
default:
// A scalar has no flow break points; an overlong one stays on
// its single line.
return [prefix + single]
}
}
/// Folds an overlong plain scalar line at single-space boundaries onto
/// more-indented continuation lines. Parsing folds each break back to
/// exactly one space, so the value is unchanged. Returns `nil` when
/// wrapping is off, the line fits, the scalar did not render plain, or
/// no safe break point exists.
func foldedPlainScalarLine(
head: String,
value: PureYAML.Model.Value,
rendered: String,
indent: Int,
) -> [String]? {
guard options.lineWidth > 0, head.count + rendered.count > options.lineWidth else {
return nil
}
guard case let .string(string) = value, rendered == string else {
return nil
}
let continuationIndent = indent + options.indentWidth
let continuationPrefix = String(repeating: " ", count: continuationIndent)
let segments = foldSegments(
string,
firstBudget: options.lineWidth - head.count,
continuationBudget: options.lineWidth - continuationIndent,
)
guard segments.count > 1, foldSegmentsAreSafe(segments) else {
return nil
}
var lines = [head + segments[0]]
for segment in segments.dropFirst() {
lines.append(continuationPrefix + segment)
}
return lines
}
/// A fold boundary must not manufacture YAML syntax: a segment ending
/// in `:` would put a colon before a line break (a value indicator), a
/// continuation line starting with `#` would open a comment (the fold
/// break is whitespace), and one starting with `-`, `?`, or `:` before
/// a space (or alone) would read as a block indicator. Any unsafe
/// boundary vetoes the whole fold; the line stays long.
func foldSegmentsAreSafe(_ segments: [String]) -> Bool {
for (index, segment) in segments.enumerated() {
if index < segments.count - 1, segment.hasSuffix(":") {
return false
}
guard index > 0 else {
continue
}
if segment.hasPrefix("#") {
return false
}
if let first = segment.first, "-?:".contains(first) {
let second = segment.dropFirst().first
if second == nil || second == " " {
return false
}
}
}
return true
}
/// Splits at spaces whose neighbors are non-space, so no segment gains
/// a leading or trailing space and every fold restores to exactly one
/// space on reparse. A word longer than the budget stays whole.
func foldSegments(
_ string: String,
firstBudget: Int,
continuationBudget: Int,
) -> [String] {
var segments: [String] = []
var current = ""
var budget = firstBudget
for word in string.split(separator: " ", omittingEmptySubsequences: false) {
if current.isEmpty {
current = String(word)
} else if current.count + 1 + word.count <= budget || word.isEmpty || current.hasSuffix(" ") {
// Empty words mark runs of consecutive spaces: those spaces
// must stay literal (a fold would swallow them), so they
// glue onto the current segment and forbid breaking here.
current += " " + word
} else {
segments.append(current)
current = String(word)
budget = continuationBudget
}
}
if !current.isEmpty {
segments.append(current)
}
return segments
}
}

View File

@ -7,10 +7,36 @@ public extension PureYAML.Emitting {
self.options = options
}
/// Serializes one document. A value nested deeper than the parser's own
/// ``PureYAML/Parsing/Parser/maxNestingDepth`` cannot round-trip through
/// PureYAML (the parser rejects it), and the recursive renderers would
/// overflow the stack on it, so such a value is refused with a
/// documented, reparseable marker document instead of crashing.
public func dump(_ value: PureYAML.Model.Value) -> String {
switch options.collectionStyle {
guard !PureYAML.Emitting.exceedsNestingLimit(value) else {
return PureYAML.Emitting.nestingLimitDocument
}
var body = documentBody(value)
if options.canonical || options.explicitDocumentStart {
body = "---\n" + body
}
if options.explicitDocumentEnd {
body += "...\n"
}
return body
}
/// The document body with no leading `---` or trailing `...` markers, so
/// the single-document and stream paths can each own marker emission.
func documentBody(_ value: PureYAML.Model.Value) -> String {
if options.canonical {
return canonicalBody(value)
}
return switch options.collectionStyle {
case .block:
render(value, indent: 0).joined(separator: "\n") + "\n"
case .flow where options.lineWidth > 0:
renderFlowWrappedDocument(value)
case .flow:
renderFlowDocument(value)
}
@ -49,7 +75,7 @@ extension PureYAML.Emitting.Dumper {
switch value {
case .mapping, .sequence:
return [prefix + "-"] + render(value, indent: indent + 2)
return [prefix + "-"] + render(value, indent: indent + options.indentWidth)
default:
return renderSequenceScalarItem(value, indent: indent)
}
@ -116,10 +142,14 @@ extension PureYAML.Emitting.Dumper {
indent: Int,
) -> [String] {
let prefix = String(repeating: " ", count: indent)
if let lines = renderLiteralBlockString(value, contentIndent: indent + 2) {
if let lines = renderLiteralBlockString(value, contentIndent: indent + options.indentWidth) {
return [prefix + lines.header] + lines.content
}
return [prefix + renderScalar(value)]
let rendered = renderScalar(value)
if let folded = foldedPlainScalarLine(head: prefix, value: value, rendered: rendered, indent: indent) {
return folded
}
return [prefix + rendered]
}
func renderMappingScalarPair(
@ -128,10 +158,15 @@ extension PureYAML.Emitting.Dumper {
) -> [String] {
let prefix = String(repeating: " ", count: indent)
let key = escapeKey(pair.key)
if let lines = renderLiteralBlockString(pair.value, contentIndent: indent + 2) {
if let lines = renderLiteralBlockString(pair.value, contentIndent: indent + options.indentWidth) {
return [prefix + key + ": " + lines.header] + lines.content
}
return [prefix + key + ": " + renderScalar(pair.value)]
let rendered = renderScalar(pair.value)
let head = prefix + key + ": "
if let folded = foldedPlainScalarLine(head: head, value: pair.value, rendered: rendered, indent: indent) {
return folded
}
return [head + rendered]
}
func renderSequenceScalarItem(
@ -139,10 +174,14 @@ extension PureYAML.Emitting.Dumper {
indent: Int,
) -> [String] {
let prefix = String(repeating: " ", count: indent)
if let lines = renderLiteralBlockString(value, contentIndent: indent + 2) {
if let lines = renderLiteralBlockString(value, contentIndent: indent + options.indentWidth) {
return [prefix + "- " + lines.header] + lines.content
}
return [prefix + "- " + renderScalar(value)]
let rendered = renderScalar(value)
if let folded = foldedPlainScalarLine(head: prefix + "- ", value: value, rendered: rendered, indent: indent) {
return folded
}
return [prefix + "- " + rendered]
}
func renderScalar(_ value: PureYAML.Model.Value) -> String {
@ -202,181 +241,4 @@ extension PureYAML.Emitting.Dumper {
.map { prefix + String($0) }
return (header, content)
}
func canRenderLiteralBlockString(_ value: String) -> Bool {
guard value.contains("\n"), !value.contains("\r"), !value.contains("\t") else {
return false
}
let contentSource = value.hasSuffix("\n") ? String(value.dropLast()) : value
guard !contentSource.isEmpty else {
return false
}
return contentSource
.split(separator: "\n", omittingEmptySubsequences: false)
.allSatisfy(canRenderLiteralBlockLine)
}
func canRenderLiteralBlockLine(_ line: Substring) -> Bool {
guard !line.isEmpty, line.first?.isWhitespace == false, line.last?.isWhitespace == false else {
return false
}
let value = String(line)
guard !containsCommentStart(value), !containsMappingSeparator(value) else {
return false
}
guard let first = line.first else {
return false
}
return canRenderLiteralBlockLineStarting(with: first, line: line)
}
func canRenderLiteralBlockLineStarting(
with first: Character,
line: Substring,
) -> Bool {
switch first {
case "#", ",", "[", "]", "{", "}", "&", "*", "!", "|", ">", "'", "\"", "`":
false
case "-":
!hasOnlyIndicator(first, line: line) && line.dropFirst().first != " "
case "?":
!hasOnlyIndicator(first, line: line) && line.dropFirst().first != " "
case ":":
!hasOnlyIndicator(first, line: line) && line.dropFirst().first != " "
default:
true
}
}
func hasOnlyIndicator(
_ indicator: Character,
line: Substring,
) -> Bool {
line.count == 1 && line.first == indicator
}
func containsCommentStart(_ value: String) -> Bool {
var previousWasWhitespace = true
for character in value {
if character == "#", previousWasWhitespace {
return true
}
previousWasWhitespace = character.isWhitespace
}
return false
}
func containsMappingSeparator(_ value: String) -> Bool {
var previousWasColon = false
for character in value {
if previousWasColon, isMappingSeparatorBoundary(character) {
return true
}
previousWasColon = character == ":"
}
return previousWasColon
}
func isMappingSeparatorBoundary(_ character: Character) -> Bool {
character.isWhitespace
|| character == ","
|| character == "]"
|| character == "}"
}
func canRenderPlainString(_ value: String) -> Bool {
guard !value.isEmpty else {
return false
}
guard value.first?.isWhitespace == false, value.last?.isWhitespace == false else {
return false
}
guard !value.contains("\n"), !value.contains("\r"), !value.contains("\t") else {
return false
}
guard !isReservedPlainScalar(value), !isNumberLikePlainScalar(value) else {
return false
}
guard let first = value.first, !isPlainScalarIndicator(first) else {
return false
}
return !containsColonSpace(value) && !value.contains("#")
}
func canRenderFlowPlainString(_ value: String) -> Bool {
canRenderPlainString(value) && !containsFlowDelimiter(value)
}
func containsFlowDelimiter(_ value: String) -> Bool {
value.contains(",")
|| value.contains("[")
|| value.contains("]")
|| value.contains("{")
|| value.contains("}")
}
func containsColonSpace(_ value: String) -> Bool {
var previousWasColon = false
for character in value {
if previousWasColon, character == " " {
return true
}
previousWasColon = character == ":"
}
return false
}
func isReservedPlainScalar(_ value: String) -> Bool {
switch value {
case "~", "null", "Null", "NULL", "true", "True", "TRUE", "false", "False", "FALSE":
true
default:
false
}
}
func isNumberLikePlainScalar(_ value: String) -> Bool {
if Int(value) != nil, !value.contains(".") {
return true
}
if value.contains(".") || value.contains("e") || value.contains("E") {
return Double(value) != nil
}
return false
}
func isPlainScalarIndicator(_ character: Character) -> Bool {
"-?:,[]{}#&*!|>'\"%@`".contains(character)
}
func escapeKey(_ key: String) -> String {
if key.isEmpty || key.contains(":") || key.contains("#") || key.contains("\n") {
return quote(key)
}
return key
}
func quote(_ value: String) -> String {
var output = "\""
for character in value {
switch character {
case "\"":
output += "\\\""
case "\\":
output += "\\\\"
case "\n":
output += "\\n"
case "\r":
output += "\\r"
case "\t":
output += "\\t"
default:
output.append(character)
}
}
output += "\""
return output
}
}

View File

@ -5,13 +5,62 @@ public extension PureYAML.Emitting {
public var scalarStyle: ScalarStyle
public var collectionStyle: CollectionStyle
/// Spaces per nesting level in block style. Has no effect on flow
/// style, which has no indentation to control. Must be at least 1:
/// block YAML distinguishes nesting only by indentation, so 0 would
/// make sibling and child content indistinguishable and produce
/// output that cannot round-trip back through the parser. Not
/// enforced at construction (a plain `Int`, matching this struct's
/// existing options), so passing 0 or negative is a caller error
/// that surfaces as unparseable output, not a thrown error.
public var indentWidth: Int
/// Sort mapping keys lexicographically by their rendered (already
/// quoted/escaped) form before emitting, instead of preserving
/// `Model.Mapping`'s insertion order. Off by default: insertion
/// order is what makes round-tripping a duplicate-key-preserving
/// `Mapping` back to itself meaningful, so sorting is opt-in, not
/// the default.
public var sortKeys: Bool
/// Emit the YAML-specification canonical form: an explicit `---`
/// document start, an explicit `!!tag` on every node, double-quoted
/// scalars, and flow collections with one `? key : value,` entry
/// per line. When set, `scalarStyle`, `collectionStyle`, and
/// `lineWidth` are ignored; `indentWidth` and `sortKeys` still
/// apply. Off by default.
public var canonical: Bool
/// Emit an explicit `---` document start marker. Implied by
/// `canonical`. Off by default.
public var explicitDocumentStart: Bool
/// Emit an explicit `...` document end marker. Off by default.
public var explicitDocumentEnd: Bool
/// The preferred maximum line width. `0` (the default) disables
/// wrapping. When positive, a flow collection whose single-line
/// form would overflow the width re-renders with one entry per
/// line, and a plain block scalar folds at spaces onto
/// more-indented continuation lines; folding restores each break
/// to the same single space on reparse, so the value is unchanged.
/// Content with no legal break point (a single overlong word, a
/// quoted or block scalar) stays on one line.
public var lineWidth: Int
public init(
scalarStyle: ScalarStyle = .quoted,
collectionStyle: CollectionStyle = .block,
indentWidth: Int = 2,
sortKeys: Bool = false,
canonical: Bool = false,
explicitDocumentStart: Bool = false,
explicitDocumentEnd: Bool = false,
lineWidth: Int = 0,
) {
self.scalarStyle = scalarStyle
self.collectionStyle = collectionStyle
self.indentWidth = indentWidth
self.sortKeys = sortKeys
self.canonical = canonical
self.explicitDocumentStart = explicitDocumentStart
self.explicitDocumentEnd = explicitDocumentEnd
self.lineWidth = lineWidth
}
}
}

View File

@ -0,0 +1,20 @@
public extension PureYAML.Model {
/// A key-value pair in a located mapping: the key (as a ``Key``), the source span of the
/// key node, and the located value. Parallel to ``Pair`` but carrying source coordinates.
struct LocatedPair: Equatable, Sendable {
public var keyNode: Key
public var keyRange: SourceRange
public var value: LocatedValue
public init(keyNode: Key, keyRange: SourceRange, value: LocatedValue) {
self.keyNode = keyNode
self.keyRange = keyRange
self.value = value
}
/// The position-free pair, projecting the located value back to a plain ``Value``.
public var pair: Pair {
Pair(keyNode: keyNode, value: value.value)
}
}
}

View File

@ -0,0 +1,58 @@
public extension PureYAML.Model {
/// A value tree annotated with source spans, parallel in shape to ``Value``: every node
/// carries its own ``SourceRange`` and its located children. It is produced by the opt-in
/// ``PureYAML/parseLocated(_:)`` and ``PureYAML/parseStreamLocated(_:)`` entry points; the
/// plain ``Value`` API is unchanged, so source positions never affect value equality, the
/// conformance oracle, or any existing consumer.
///
/// `value` projects the position-free ``Value`` (identical to what a plain parse returns),
/// and `range` is the node's own span, so a consumer can keep operating on ``Value`` and
/// consult the located tree only when it needs a location.
indirect enum LocatedValue: Equatable, Sendable {
/// A leaf scalar (`null`, `bool`, `int`, `double`, or `string`) with its start-anchored span.
case scalar(Value, SourceRange)
/// A sequence: its located items and the span from its opening to its closing.
case sequence([LocatedValue], SourceRange)
/// A mapping: its located pairs and the span from its opening to its closing.
case mapping([LocatedPair], SourceRange)
}
}
public extension PureYAML.Model.LocatedValue {
/// The span of this node.
var range: PureYAML.Model.SourceRange {
switch self {
case let .scalar(_, range), let .sequence(_, range), let .mapping(_, range):
range
}
}
/// The position-free value this located node projects to, identical to a plain parse.
var value: PureYAML.Model.Value {
switch self {
case let .scalar(value, _):
value
case let .sequence(items, _):
.sequence(items.map(\.value))
case let .mapping(pairs, _):
.mapping(PureYAML.Model.Mapping(pairs.map(\.pair)))
}
}
/// Wrap a plain ``Value`` so every node shares one span. Used where a real per-node span is
/// not available (an alias expansion or a merge-inherited pair, whose content is defined
/// elsewhere): the whole subtree honestly points at its single reference site.
static func synthesized(from value: PureYAML.Model.Value, at range: PureYAML.Model.SourceRange) -> Self {
switch value {
case .null, .bool, .int, .double, .string:
return .scalar(value, range)
case let .sequence(items):
return .sequence(items.map { synthesized(from: $0, at: range) }, range)
case let .mapping(mapping):
let pairs = mapping.pairs.map {
PureYAML.Model.LocatedPair(keyNode: $0.keyNode, keyRange: range, value: synthesized(from: $0.value, at: range))
}
return .mapping(pairs, range)
}
}
}

View File

@ -0,0 +1,19 @@
public extension PureYAML.Model {
/// A 1-based line/column position in the source text. Carried by ``LocatedValue`` so a
/// consumer (an editor, a linter, a CI report) can point at exactly where a value-tree
/// node was written. Positions live on the parallel located tree, never on ``Value``,
/// so the plain value tree and its equality stay free of source coordinates.
struct SourcePosition: Equatable, Hashable, Sendable, Comparable {
public var line: Int
public var column: Int
public init(line: Int, column: Int) {
self.line = line
self.column = column
}
public static func < (lhs: Self, rhs: Self) -> Bool {
(lhs.line, lhs.column) < (rhs.line, rhs.column)
}
}
}

View File

@ -0,0 +1,20 @@
public extension PureYAML.Model {
/// A source span for a value-tree node: a start and end ``SourcePosition``. A collection
/// spans from its opening to its closing event; a scalar carries a start-anchored span
/// (start equal to end), because the event stream records a scalar's start mark only.
struct SourceRange: Equatable, Hashable, Sendable {
public var start: SourcePosition
public var end: SourcePosition
public init(start: SourcePosition, end: SourcePosition) {
self.start = start
self.end = end
}
/// A zero-width span at a single position, used when only a point is known.
public init(at position: SourcePosition) {
start = position
end = position
}
}
}

View File

@ -0,0 +1,26 @@
extension PureYAML.Model.Value: PureYAML.Validation.Traversable {
/// For a mapping: the mapping itself (a `Validation<Mapping, _>` fires on this), and each pair's
/// key node at its own path (so a key-typed check can inspect it, e.g. "is this key a string").
/// Nothing extra for any other case.
public var extraOffers: [(component: PureYAML.Validation.Path.Component?, value: any PureYAML.Validation.Validatable)] {
guard case let .mapping(mapping) = self else {
return []
}
return [(nil, mapping)] + mapping.pairs.map { pair in
(component: pair.keyNode.pathComponent, value: pair.keyNode)
}
}
/// A mapping's children are its pairs' values, keyed by their key node's path component; a
/// sequence's children are its values, keyed by index. A scalar has none.
public var children: [(component: PureYAML.Validation.Path.Component, value: PureYAML.Model.Value)] {
switch self {
case let .mapping(mapping):
mapping.pairs.map { pair in (component: pair.keyNode.pathComponent, value: pair.value) }
case let .sequence(values):
values.indices.map { index in (component: .index(index), value: values[index]) }
case .null, .bool, .int, .double, .string:
[]
}
}
}

View File

@ -2,6 +2,7 @@ extension PureYAML.Parsing {
enum BlockScalarChomping: String, Equatable, CustomStringConvertible {
case clip
case strip
case keep
var description: String {
rawValue

View File

@ -2,13 +2,13 @@ extension PureYAML.Parsing {
enum Event: Equatable, CustomStringConvertible {
case streamStart(mark: Mark)
case streamEnd(mark: Mark)
case documentStart(mark: Mark)
case documentEnd(mark: Mark)
case documentStart(explicit: Bool, mark: Mark)
case documentEnd(explicit: Bool, mark: Mark)
case sequenceStart(anchor: String?, tag: String?, style: CollectionStyle, mark: Mark)
case sequenceEnd(mark: Mark)
case mappingStart(anchor: String?, tag: String?, style: CollectionStyle, mark: Mark)
case mappingEnd(mark: Mark)
case scalar(value: String, anchor: String?, tag: String?, style: ScalarStyle, mark: Mark)
case scalar(value: String, anchor: String?, tag: String?, style: ScalarStyle, mark: Mark, endMark: Mark)
case alias(anchor: String, mark: Mark)
var description: String {
@ -17,10 +17,10 @@ extension PureYAML.Parsing {
"streamStart @\(mark)"
case let .streamEnd(mark):
"streamEnd @\(mark)"
case let .documentStart(mark):
"documentStart @\(mark)"
case let .documentEnd(mark):
"documentEnd @\(mark)"
case let .documentStart(explicit, mark):
"documentStart explicit=\(explicit) @\(mark)"
case let .documentEnd(explicit, mark):
"documentEnd explicit=\(explicit) @\(mark)"
case let .sequenceStart(anchor, tag, style, mark):
"sequenceStart anchor=\(describe(anchor)) tag=\(describe(tag)) style=\(style) @\(mark)"
case let .sequenceEnd(mark):
@ -29,7 +29,7 @@ extension PureYAML.Parsing {
"mappingStart anchor=\(describe(anchor)) tag=\(describe(tag)) style=\(style) @\(mark)"
case let .mappingEnd(mark):
"mappingEnd @\(mark)"
case let .scalar(value, anchor, tag, style, mark):
case let .scalar(value, anchor, tag, style, mark, _):
"scalar value=\"\(escape(value))\" anchor=\(describe(anchor)) tag=\(describe(tag)) style=\(style) @\(mark)"
case let .alias(anchor, mark):
"alias anchor=\(anchor) @\(mark)"
@ -39,6 +39,36 @@ extension PureYAML.Parsing {
}
extension PureYAML.Parsing.Event {
/// Re-emit the event with libyaml's 0-based line/column mark converted to the
/// package's 1-based convention (the byte index is already 0-based in both).
func bumpingMark() -> Self {
let source = mark
let bumped = PureYAML.Parsing.Mark(line: source.line + 1, column: source.column + 1, index: source.index)
switch self {
case .streamStart:
return .streamStart(mark: bumped)
case .streamEnd:
return .streamEnd(mark: bumped)
case let .documentStart(explicit, _):
return .documentStart(explicit: explicit, mark: bumped)
case let .documentEnd(explicit, _):
return .documentEnd(explicit: explicit, mark: bumped)
case let .sequenceStart(anchor, tag, style, _):
return .sequenceStart(anchor: anchor, tag: tag, style: style, mark: bumped)
case .sequenceEnd:
return .sequenceEnd(mark: bumped)
case let .mappingStart(anchor, tag, style, _):
return .mappingStart(anchor: anchor, tag: tag, style: style, mark: bumped)
case .mappingEnd:
return .mappingEnd(mark: bumped)
case let .scalar(value, anchor, tag, style, _, endMark):
let bumpedEnd = PureYAML.Parsing.Mark(line: endMark.line + 1, column: endMark.column + 1, index: endMark.index)
return .scalar(value: value, anchor: anchor, tag: tag, style: style, mark: bumped, endMark: bumpedEnd)
case let .alias(anchor, _):
return .alias(anchor: anchor, mark: bumped)
}
}
var isMappingEnd: Bool {
if case .mappingEnd = self {
return true
@ -63,16 +93,17 @@ extension PureYAML.Parsing.Event {
var mark: PureYAML.Parsing.Mark {
switch self {
case let .alias(_, mark),
let .documentEnd(mark),
let .documentStart(mark),
let .mappingEnd(mark),
let .mappingStart(_, _, _, mark),
let .scalar(_, _, _, _, mark),
let .scalar(_, _, _, _, mark, _),
let .sequenceEnd(mark),
let .sequenceStart(_, _, _, mark),
let .streamEnd(mark),
let .streamStart(mark):
mark
case let .documentEnd(_, mark),
let .documentStart(_, mark):
mark
}
}

View File

@ -56,12 +56,12 @@ extension PureYAML.Parsing.EventComposer {
case let .sequence(values):
return try values.map { value in
guard case let .mapping(mapping) = value else {
throw invalidMergeValue(mark: mark)
throw PureYAML.Parsing.MergeResolution.invalidMergeValue(mark: mark)
}
return mapping.pairs
}
case .bool, .double, .int, .null, .string:
throw invalidMergeValue(mark: mark)
throw PureYAML.Parsing.MergeResolution.invalidMergeValue(mark: mark)
}
}
@ -70,42 +70,8 @@ extension PureYAML.Parsing.EventComposer {
mark: PureYAML.Parsing.Mark,
) throws -> [PureYAML.Model.Pair] {
guard case let .mapping(mapping) = value else {
throw invalidMergeValue(mark: mark)
throw PureYAML.Parsing.MergeResolution.invalidMergeValue(mark: mark)
}
return mapping.pairs
}
func mergedPairs(
from mergeSources: [[PureYAML.Model.Pair]],
localPairs: [PureYAML.Model.Pair],
) -> [PureYAML.Model.Pair] {
guard !mergeSources.isEmpty else {
return localPairs
}
let localKeys = Set(localPairs.map(\.keyNode))
var inheritedKeys = Set<PureYAML.Model.Key>()
var inheritedPairs: [PureYAML.Model.Pair] = []
for source in mergeSources {
var sourceKeys = Set<PureYAML.Model.Key>()
for pair in source {
guard !localKeys.contains(pair.keyNode),
!inheritedKeys.contains(pair.keyNode)
else {
continue
}
inheritedPairs.append(pair)
sourceKeys.insert(pair.keyNode)
}
inheritedKeys.formUnion(sourceKeys)
}
return inheritedPairs + localPairs
}
func invalidMergeValue(mark: PureYAML.Parsing.Mark) -> PureYAML.Parsing.ParseError {
PureYAML.Parsing.ParseError.invalidMergeValue(
line: mark.line,
column: mark.column,
)
}
}

View File

@ -22,6 +22,12 @@ extension PureYAML.Parsing {
}
return false
}
// An empty or comment-only stream has no document events; the
// single-document value API maps that to null.
if current?.isStreamEnd == true {
_ = try expectStreamEnd()
return .null
}
_ = try expect("document start") { event in
if case .documentStart = event {
return true
@ -35,7 +41,7 @@ extension PureYAML.Parsing {
}
return false
}
if case let .documentStart(mark)? = current {
if case let .documentStart(_, mark)? = current {
throw PureYAML.Parsing.ParseError.unsupportedMultiDocumentStream(line: mark.line)
}
_ = try expectStreamEnd()
@ -96,7 +102,7 @@ extension PureYAML.Parsing.EventComposer {
let value = try composeMapping()
store(value, anchor: anchor)
return value
case let .scalar(value, anchor, tag, style, mark):
case let .scalar(value, anchor, tag, style, mark, _):
index += 1
let model = try composeScalar(
value,
@ -149,7 +155,7 @@ extension PureYAML.Parsing.EventComposer {
}
}
_ = try expect("mapping end") { $0.isMappingEnd }
let pairs = mergedPairs(from: mergeSources, localPairs: localPairs)
let pairs = PureYAML.Parsing.MergeResolution.mergedPairs(from: mergeSources, localPairs: localPairs)
return .mapping(PureYAML.Model.Mapping(pairs))
}
@ -157,7 +163,7 @@ extension PureYAML.Parsing.EventComposer {
guard let event = current else {
throw unexpectedEvent(expected: "mapping key")
}
guard case let .scalar(value, anchor, tag, style, _) = event else {
guard case let .scalar(value, anchor, tag, style, _, _) = event else {
let value = try composeNode()
return ComposedMappingKey(
keyNode: PureYAML.Model.Key(value: value),
@ -167,7 +173,10 @@ extension PureYAML.Parsing.EventComposer {
)
}
index += 1
store(.string(value), anchor: anchor)
if let anchor {
let model = try composeScalar(value, tag: tag, style: style, mark: .start)
store(model, anchor: anchor)
}
return ComposedMappingKey(
keyNode: .string(value),
scalarValue: value,

View File

@ -0,0 +1,10 @@
extension PureYAML.Parsing {
struct EventStreamComposedNode {
var value: PureYAML.Model.Value
var mappingKey: PureYAML.Parsing.EventComposer.ComposedMappingKey
var mark: PureYAML.Parsing.Mark
var sequenceItems: [EventStreamComposedNode] = []
/// The located projection, populated only when the composer tracks locations.
var located: PureYAML.Model.LocatedValue?
}
}

View File

@ -0,0 +1,26 @@
extension PureYAML.Parsing.EventStreamComposer {
/// Charges `cost` new expanded nodes against the document's remaining
/// ``EventStreamComposer/expandedNodeBudget``, throwing `expansionTooLarge` the moment it
/// runs out.
///
/// `enforceNestingLimit` (`Parser.Events.swift`) bounds PHYSICAL collection depth in the raw
/// event stream; it does nothing against an anchor-expansion ("billion laughs") bomb, where a
/// handful of shallow, small anchors are each aliased many times, each alias nested inside the
/// next, so the LOGICAL tree an alias resolves to is exponential in the number of alias levels
/// while the physical document stays kilobytes -- a 25-level, fan-out-9 document already
/// expands past 9^25 logical nodes.
///
/// Called once per scalar, alias resolution, or completed container -- never re-derived from
/// a walk -- so a bomb is rejected within a few alias levels, long before its true size could
/// ever be materialized: each charge is the MARGINAL cost this one node adds, with any nested
/// alias expansion it contains already folded into a prior charge (cached in
/// ``EventStreamComposer/anchorSizes``) and looked up in O(1), never re-counted here.
/// ``EventStreamComposer/expandedNodeBudget``'s initial 5,000,000 is far past any real
/// document's node count.
mutating func chargeExpansion(_ cost: Int, mark: PureYAML.Parsing.Mark) throws {
expandedNodeBudget -= cost
guard expandedNodeBudget >= 0 else {
throw PureYAML.Parsing.ParseError.expansionTooLarge(line: mark.line, column: mark.column)
}
}
}

View File

@ -0,0 +1,63 @@
extension PureYAML.Parsing.EventStreamComposer {
func range(at mark: PureYAML.Parsing.Mark) -> PureYAML.Model.SourceRange {
.init(at: .init(line: mark.line, column: mark.column))
}
func range(start: PureYAML.Parsing.Mark, end: PureYAML.Parsing.Mark) -> PureYAML.Model.SourceRange {
.init(start: .init(line: start.line, column: start.column), end: .init(line: end.line, column: end.column))
}
/// The node's located projection, falling back to a synthesized one at the node's mark if
/// the located field was somehow not populated (never expected while tracking locations).
func locatedNode(_ node: PureYAML.Parsing.EventStreamComposedNode) -> PureYAML.Model.LocatedValue {
node.located ?? .synthesized(from: node.value, at: range(at: node.mark))
}
/// The located pairs a merge value (`<<`) contributes, mirroring ``mergeSources(from:)``.
/// The value-side merge validation has already run, so the located shape is a mapping or a
/// sequence of mappings; anything else contributes nothing.
func locatedMergeSources(from node: PureYAML.Parsing.EventStreamComposedNode) -> [[PureYAML.Model.LocatedPair]] {
switch locatedNode(node) {
case let .mapping(pairs, _):
[pairs]
case let .sequence(items, _):
items.compactMap { item in
guard case let .mapping(pairs, _) = item else {
return nil
}
return pairs
}
case .scalar:
[]
}
}
/// The final located pairs, mirroring ``mergedPairs(from:localPairs:)`` exactly (same dedup,
/// same order) so the located mapping has the identical shape to the value mapping.
func mergedLocatedPairs(
from mergeSources: [[PureYAML.Model.LocatedPair]],
localPairs: [PureYAML.Model.LocatedPair],
) -> [PureYAML.Model.LocatedPair] {
guard !mergeSources.isEmpty else {
return localPairs
}
let localKeys = Set(localPairs.map(\.keyNode))
var inheritedKeys = Set<PureYAML.Model.Key>()
var inheritedPairs: [PureYAML.Model.LocatedPair] = []
for source in mergeSources {
var sourceKeys = Set<PureYAML.Model.Key>()
for pair in source {
guard !localKeys.contains(pair.keyNode),
!inheritedKeys.contains(pair.keyNode)
else {
continue
}
inheritedPairs.append(pair)
sourceKeys.insert(pair.keyNode)
}
inheritedKeys.formUnion(sourceKeys)
}
return inheritedPairs + localPairs
}
}

View File

@ -0,0 +1,386 @@
extension PureYAML.Parsing {
struct EventStreamComposer {
let scalarParser: Parser
let mode: EventStreamComposerMode
// When true, the composer builds a parallel located tree alongside each value. The
// default value path never sets this, so it stays byte-identical to before.
var tracksLocations = false
var anchors: [String: PureYAML.Model.Value] = [:]
/// Each anchor's cached EXPANDED node count; see `chargeExpansion` for why.
var anchorSizes: [String: Int] = [:]
// The expanded-node budget remaining for this document; see `chargeExpansion`.
var expandedNodeBudget = 5_000_000
var containers: [EventStreamContainer] = []
var documentValues: [PureYAML.Model.Value] = []
var documentLocated: [PureYAML.Model.LocatedValue] = []
var currentDocumentValue: PureYAML.Model.Value?
var currentDocumentLocated: PureYAML.Model.LocatedValue?
var didSeeStreamStart = false
var didSeeStreamEnd = false
init(
scalarParser: Parser,
mode: EventStreamComposerMode,
tracksLocations: Bool = false,
) {
self.scalarParser = scalarParser
self.mode = mode
self.tracksLocations = tracksLocations
}
mutating func consume(_ event: PureYAML.Parsing.Event) throws {
switch event {
case .streamStart:
didSeeStreamStart = true
case .streamEnd:
didSeeStreamEnd = true
case let .documentStart(_, mark):
try startDocument(mark: mark)
case let .documentEnd(_, mark):
try endDocument(mark: mark)
case let .scalar(value, anchor, tag, style, mark, endMark):
try consumeScalar(value, anchor: anchor, tag: tag, style: style, marks: (mark, endMark))
case let .alias(anchor, mark):
try consumeAlias(anchor, mark: mark)
case let .sequenceStart(anchor, _, _, mark):
containers.append(.init(kind: .sequence, anchor: anchor, mark: mark))
case let .sequenceEnd(mark):
try completeSequence(mark: mark)
case let .mappingStart(anchor, _, _, mark):
containers.append(.init(kind: .mapping, anchor: anchor, mark: mark))
case let .mappingEnd(mark):
try completeMapping(mark: mark)
}
}
mutating func composedValue() throws -> PureYAML.Model.Value {
guard didSeeStreamStart, didSeeStreamEnd else {
throw unexpectedEvent(expected: "complete YAML stream", mark: .start)
}
// An empty or comment-only stream has zero documents and maps to null;
// a single document yields its value; more than one is not a single
// document and is rejected.
guard documentValues.count <= 1 else {
throw unexpectedEvent(expected: "single YAML document", mark: .start)
}
return documentValues.first ?? .null
}
mutating func composedStream() throws -> [PureYAML.Stream.Document] {
guard didSeeStreamStart, didSeeStreamEnd else {
throw unexpectedEvent(expected: "complete YAML stream", mark: .start)
}
return documentValues.enumerated().map { index, value in
PureYAML.Stream.Document(index: index, value: value)
}
}
/// The single document's located tree (mirrors ``composedValue()``). Only meaningful
/// when the composer was created with `tracksLocations: true`.
mutating func composedLocated() throws -> PureYAML.Model.LocatedValue {
guard didSeeStreamStart, didSeeStreamEnd else {
throw unexpectedEvent(expected: "complete YAML stream", mark: .start)
}
guard documentLocated.count <= 1 else {
throw unexpectedEvent(expected: "single YAML document", mark: .start)
}
return documentLocated.first ?? .scalar(.null, .init(at: .init(line: 1, column: 1)))
}
/// The stream's located trees, one per document (mirrors ``composedStream()``). Only
/// meaningful when the composer was created with `tracksLocations: true`.
mutating func composedLocatedStream() throws -> [PureYAML.Model.LocatedValue] {
guard didSeeStreamStart, didSeeStreamEnd else {
throw unexpectedEvent(expected: "complete YAML stream", mark: .start)
}
return documentLocated
}
}
}
private extension PureYAML.Parsing.EventStreamComposer {
mutating func startDocument(mark: PureYAML.Parsing.Mark) throws {
guard containers.isEmpty else {
throw unexpectedEvent(expected: "document content end", mark: mark)
}
if mode == .singleDocument, !documentValues.isEmpty {
throw PureYAML.Parsing.ParseError.unsupportedMultiDocumentStream(line: mark.line)
}
anchors = [:]
anchorSizes = [:]
expandedNodeBudget = 5_000_000
currentDocumentValue = nil
}
mutating func endDocument(mark: PureYAML.Parsing.Mark) throws {
guard containers.isEmpty else {
throw unexpectedEvent(expected: "document content end", mark: mark)
}
documentValues.append(currentDocumentValue ?? .null)
currentDocumentValue = nil
if tracksLocations {
documentLocated.append(currentDocumentLocated ?? .scalar(.null, range(at: mark)))
currentDocumentLocated = nil
}
}
mutating func consumeScalar(
_ value: String,
anchor: String?,
tag: String?,
style: PureYAML.Parsing.ScalarStyle,
marks: (start: PureYAML.Parsing.Mark, end: PureYAML.Parsing.Mark),
) throws {
let model = try scalarParser.composeScalarValue(value, tag: tag, style: style, mark: marks.start)
try chargeExpansion(1, mark: marks.start)
if let anchor {
anchors[anchor] = model
anchorSizes[anchor] = 1
}
try complete(.init(
value: model,
mappingKey: .init(
keyNode: .string(value),
scalarValue: value,
tag: scalarParser.normalizedTag(tag),
style: style,
),
mark: marks.start,
located: tracksLocations ? .scalar(model, range(start: marks.start, end: marks.end)) : nil,
), cost: 1)
}
mutating func consumeAlias(
_ anchor: String,
mark: PureYAML.Parsing.Mark,
) throws {
guard let value = anchors[anchor] else {
throw PureYAML.Parsing.ParseError.undefinedAlias(
anchor: anchor,
line: mark.line,
column: mark.column,
)
}
// The target's own cached expanded size (itself plus every descendant, its own nested
// aliases already resolved): an alias to a value built from other aliases charges the
// FULL compounded cost here, in O(1), which is exactly what makes a nested bomb's true
// size visible without ever materializing it.
let cost = anchorSizes[anchor] ?? 1
try chargeExpansion(cost, mark: mark)
try complete(.init(
value: value,
mappingKey: .init(
keyNode: PureYAML.Model.Key(value: value),
scalarValue: nil,
tag: nil,
style: nil,
),
mark: mark,
located: tracksLocations ? .synthesized(from: value, at: range(at: mark)) : nil,
), cost: cost)
}
mutating func completeSequence(mark: PureYAML.Parsing.Mark) throws {
guard let container = containers.popLast(), container.kind == .sequence else {
throw unexpectedEvent(expected: "sequence start", mark: mark)
}
// Only the container's own wrapper is a NEW charge here; every child already charged its
// own cost (including any alias expansion) as it completed, accumulated into
// `logicalSize`. The cache for a later alias to THIS container needs the full total.
try chargeExpansion(1, mark: container.mark)
let totalCost = container.logicalSize + 1
let value = PureYAML.Model.Value.sequence(container.values.map(\.value))
store(value, anchor: container.anchor, cost: totalCost)
try complete(.init(
value: value,
mappingKey: .init(
keyNode: PureYAML.Model.Key(value: value),
scalarValue: nil,
tag: nil,
style: nil,
),
mark: container.mark,
sequenceItems: container.values,
located: tracksLocations
? .sequence(container.locatedValues, range(start: container.mark, end: mark))
: nil,
), cost: totalCost)
}
mutating func completeMapping(mark: PureYAML.Parsing.Mark) throws {
guard let container = containers.popLast(), container.kind == .mapping else {
throw unexpectedEvent(expected: "mapping start", mark: mark)
}
guard container.pendingKey == nil else {
throw unexpectedEvent(expected: "mapping value", mark: mark)
}
try chargeExpansion(1, mark: container.mark)
let totalCost = container.logicalSize + 1
let pairs = mergedPairs(
from: container.mergeSources,
localPairs: container.localPairs,
)
let value = PureYAML.Model.Value.mapping(.init(pairs))
store(value, anchor: container.anchor, cost: totalCost)
let located: PureYAML.Model.LocatedValue? = tracksLocations
? .mapping(
mergedLocatedPairs(from: container.mergeLocatedSources, localPairs: container.localLocatedPairs),
range(start: container.mark, end: mark),
)
: nil
try complete(.init(
value: value,
mappingKey: .init(
keyNode: PureYAML.Model.Key(value: value),
scalarValue: nil,
tag: nil,
style: nil,
),
mark: container.mark,
located: located,
), cost: totalCost)
}
mutating func complete(_ node: PureYAML.Parsing.EventStreamComposedNode, cost: Int) throws {
guard !containers.isEmpty else {
currentDocumentValue = node.value
if tracksLocations {
currentDocumentLocated = node.located
}
return
}
switch containers[containers.count - 1].kind {
case .sequence:
containers[containers.count - 1].values.append(node)
containers[containers.count - 1].logicalSize += cost
if tracksLocations {
containers[containers.count - 1].locatedValues.append(locatedNode(node))
}
case .mapping:
try completeMappingNode(node)
containers[containers.count - 1].logicalSize += cost
}
}
mutating func completeMappingNode(_ node: PureYAML.Parsing.EventStreamComposedNode) throws {
let index = containers.count - 1
guard let key = containers[index].pendingKey else {
containers[index].pendingKey = node.mappingKey
if tracksLocations {
containers[index].pendingKeyNode = node
}
return
}
if key.isMergeKey {
try containers[index].mergeSources.append(contentsOf: mergeSources(
from: node,
))
if tracksLocations {
containers[index].mergeLocatedSources.append(contentsOf: locatedMergeSources(from: node))
}
} else {
containers[index].localPairs.append(.init(keyNode: key.keyNode, value: node.value))
if tracksLocations {
let keyNode = containers[index].pendingKeyNode
containers[index].localLocatedPairs.append(.init(
keyNode: key.keyNode,
keyRange: keyNode.flatMap(\.located)?.range ?? locatedNode(node).range,
value: locatedNode(node),
))
}
}
containers[index].pendingKey = nil
if tracksLocations {
containers[index].pendingKeyNode = nil
}
}
mutating func store(
_ value: PureYAML.Model.Value,
anchor: String?,
cost: Int,
) {
guard let anchor else {
return
}
anchors[anchor] = value
anchorSizes[anchor] = cost
}
func mergeSources(
from node: PureYAML.Parsing.EventStreamComposedNode,
) throws -> [[PureYAML.Model.Pair]] {
switch node.value {
case let .mapping(mapping):
return [mapping.pairs]
case let .sequence(values):
guard !node.sequenceItems.isEmpty else {
return try values.map { value in
guard case let .mapping(mapping) = value else {
throw invalidMergeValue(mark: node.mark)
}
return mapping.pairs
}
}
return try values.enumerated().map { index, value in
let mark = node.sequenceItems[index].mark
guard case let .mapping(mapping) = value else {
throw invalidMergeValue(mark: mark)
}
return mapping.pairs
}
case .bool, .double, .int, .null, .string:
throw invalidMergeValue(mark: node.mark)
}
}
func mergedPairs(
from mergeSources: [[PureYAML.Model.Pair]],
localPairs: [PureYAML.Model.Pair],
) -> [PureYAML.Model.Pair] {
guard !mergeSources.isEmpty else {
return localPairs
}
let localKeys = Set(localPairs.map(\.keyNode))
var inheritedKeys = Set<PureYAML.Model.Key>()
var inheritedPairs: [PureYAML.Model.Pair] = []
for source in mergeSources {
var sourceKeys = Set<PureYAML.Model.Key>()
for pair in source {
guard !localKeys.contains(pair.keyNode),
!inheritedKeys.contains(pair.keyNode)
else {
continue
}
inheritedPairs.append(pair)
sourceKeys.insert(pair.keyNode)
}
inheritedKeys.formUnion(sourceKeys)
}
return inheritedPairs + localPairs
}
func invalidMergeValue(
mark: PureYAML.Parsing.Mark,
) -> PureYAML.Parsing.ParseError {
PureYAML.Parsing.ParseError.invalidMergeValue(
line: mark.line,
column: mark.column,
)
}
func unexpectedEvent(
expected: String,
mark: PureYAML.Parsing.Mark,
) -> PureYAML.Parsing.ParseError {
PureYAML.Parsing.ParseError.unexpectedEvent(
expected: expected,
actual: "streaming event",
line: mark.line,
column: mark.column,
)
}
}

View File

@ -0,0 +1,6 @@
extension PureYAML.Parsing {
enum EventStreamComposerMode {
case singleDocument
case stream
}
}

View File

@ -0,0 +1,21 @@
extension PureYAML.Parsing {
struct EventStreamContainer {
var kind: EventStreamContainerKind
var anchor: String?
var values: [EventStreamComposedNode] = []
var mergeSources: [[PureYAML.Model.Pair]] = []
var localPairs: [PureYAML.Model.Pair] = []
var pendingKey: PureYAML.Parsing.EventComposer.ComposedMappingKey?
var mark: PureYAML.Parsing.Mark
/// The running sum of every child's own expanded-node cost (see
/// `EventStreamComposer.expandedNodeBudget`), so this container's total logical size --
/// itself plus every descendant, alias expansion already accounted for -- is available in
/// O(1) the moment it completes, without re-walking anything.
var logicalSize = 0
// Location tracking (parallel to the above, only when the composer tracks locations).
var locatedValues: [PureYAML.Model.LocatedValue] = []
var mergeLocatedSources: [[PureYAML.Model.LocatedPair]] = []
var localLocatedPairs: [PureYAML.Model.LocatedPair] = []
var pendingKeyNode: EventStreamComposedNode?
}
}

View File

@ -0,0 +1,6 @@
extension PureYAML.Parsing {
enum EventStreamContainerKind {
case sequence
case mapping
}
}

View File

@ -0,0 +1,103 @@
extension PureYAML.Parsing.LibYAML {
/// Port of libyaml's character-class macros (`IS_*` in `yaml_private.h`).
/// libyaml inspects UTF-8 bytes; this operates on `Unicode.Scalar`, where the
/// multi-byte line breaks NEL (`U+0085`), LS (`U+2028`), and PS (`U+2029`) are
/// single scalars. End of input is `nil`, which is libyaml's `\0` sentinel, so
/// `isZ(nil)` is true and every other predicate is false for `nil`.
enum CharacterClass {
/// `IS_Z`: the end of the input (libyaml's `\0`).
static func isZ(_ scalar: Unicode.Scalar?) -> Bool {
scalar == nil
}
/// `IS_SPACE`.
static func isSpace(_ scalar: Unicode.Scalar?) -> Bool {
scalar == " "
}
/// `IS_TAB`.
static func isTab(_ scalar: Unicode.Scalar?) -> Bool {
scalar == "\t"
}
/// `IS_BLANK`: space or tab.
static func isBlank(_ scalar: Unicode.Scalar?) -> Bool {
isSpace(scalar) || isTab(scalar)
}
/// `IS_BREAK`: CR, LF, NEL, LS, or PS.
static func isBreak(_ scalar: Unicode.Scalar?) -> Bool {
switch scalar {
case "\r", "\n", "\u{85}", "\u{2028}", "\u{2029}": true
default: false
}
}
/// `IS_BREAKZ`: a line break or the end of input.
static func isBreakZ(_ scalar: Unicode.Scalar?) -> Bool {
isBreak(scalar) || isZ(scalar)
}
/// `IS_SPACEZ`: a space, a line break, or the end of input.
static func isSpaceZ(_ scalar: Unicode.Scalar?) -> Bool {
isSpace(scalar) || isBreakZ(scalar)
}
/// `IS_BLANKZ`: blank (space or tab), a line break, or the end of input.
static func isBlankZ(_ scalar: Unicode.Scalar?) -> Bool {
isBlank(scalar) || isBreakZ(scalar)
}
/// A flow collection indicator (`,`, `[`, `]`, `{`, `}`); the YAML
/// `c-flow-indicator` set excluded from `ns-plain-safe` in flow context.
static func isFlowIndicator(_ scalar: Unicode.Scalar?) -> Bool {
switch scalar {
case ",", "[", "]", "{", "}": true
default: false
}
}
/// `IS_DIGIT`.
static func isDigit(_ scalar: Unicode.Scalar?) -> Bool {
guard let scalar else { return false }
return scalar >= "0" && scalar <= "9"
}
/// `IS_ALPHA`: a digit, ASCII letter, `_`, or `-` (the directive, anchor,
/// and tag-handle name characters).
static func isAlpha(_ scalar: Unicode.Scalar?) -> Bool {
guard let scalar else { return false }
return (scalar >= "0" && scalar <= "9")
|| (scalar >= "A" && scalar <= "Z")
|| (scalar >= "a" && scalar <= "z")
|| scalar == "_"
|| scalar == "-"
}
/// `IS_HEX`: a hexadecimal digit.
static func isHex(_ scalar: Unicode.Scalar?) -> Bool {
guard let scalar else { return false }
return (scalar >= "0" && scalar <= "9")
|| (scalar >= "A" && scalar <= "F")
|| (scalar >= "a" && scalar <= "f")
}
/// `IS_ASCII`.
static func isASCII(_ scalar: Unicode.Scalar?) -> Bool {
guard let scalar else { return false }
return scalar.value <= 0x7F
}
/// `IS_PRINTABLE`: the YAML `c-printable` set.
static func isPrintable(_ scalar: Unicode.Scalar?) -> Bool {
guard let scalar else { return false }
let value = scalar.value
return value == 0x09 || value == 0x0A || value == 0x0D
|| (value >= 0x20 && value <= 0x7E)
|| value == 0x85
|| (value >= 0xA0 && value <= 0xD7FF)
|| (value >= 0xE000 && value <= 0xFFFD)
|| (value >= 0x10000 && value <= 0x10FFFF)
}
}
}

View File

@ -0,0 +1,98 @@
extension PureYAML.Parsing.LibYAML {
/// Port of the indentation stack from `yaml_parser_t` plus
/// `yaml_parser_roll_indent` and `yaml_parser_unroll_indent`. libyaml tracks a
/// single `indent` column (the current block collection's indentation, -1 at
/// the document root) and a stack of the enclosing levels. A block collection
/// is opened only when content is strictly more indented than the current
/// level, which is exactly why `- a:` followed by a same-column `b:` keeps `b`
/// a sibling and leaves `a` an empty value.
struct IndentEngine {
/// `parser->indent`. -1 at the document root, matching libyaml.
private(set) var indent = -1
/// `parser->indents`, the stack of enclosing indentation levels.
private(set) var indents: [Int] = []
/// `parser->flow_level`; indentation is inert inside flow collections.
var flowLevel = 0
/// The token queue (`parser->tokens`) the engine appends to or, for
/// `roll_indent`, inserts a start token into at a recorded position.
private(set) var tokens: [Token] = []
/// `parser->tokens_parsed`, the count already dequeued, used to turn an
/// absolute token number into an index into the live queue.
var tokensParsed = 0
/// The most recently enqueued or spliced token, kept in sync with
/// `tokens.last` after every mutation below. Batch scanning never dequeues
/// (`tokensParsed` stays 0, `dequeue()` unused), so `tokens.last` alone
/// would suffice there; this field exists for a resumable scanner
/// (`Scanner.scanStep()`), which releases and discards tokens as soon as
/// they are safe to hand out, so `tokens` can be empty exactly when a check
/// like `lastTokenForbidsInlineSequence()` needs to know what the last
/// token was.
private(set) var lastToken: Token?
/// Port of `yaml_parser_roll_indent`. In flow context it does nothing.
/// When the column is greater than the current indent, it pushes the
/// current indent, adopts the new column, and emits the block start token.
/// `number == -1` appends; otherwise the token is inserted at the recorded
/// queue position so a `BLOCK-MAPPING-START` precedes a simple key that was
/// already queued.
mutating func rollIndent(
column: Int,
number: Int,
type: TokenType,
mark: PureYAML.Parsing.Mark,
) {
if flowLevel != 0 {
return
}
if indent < column {
indents.append(indent)
indent = column
let token = Token(type: type, startMark: mark, endMark: mark)
if number == -1 {
tokens.append(token)
} else {
tokens.insert(token, at: number - tokensParsed)
}
lastToken = tokens.last
}
}
/// Port of `yaml_parser_unroll_indent`. In flow context it does nothing.
/// While the current indent is deeper than the column, it emits a
/// `BLOCK-END` and pops the enclosing level, closing every block collection
/// that the dedent ends.
mutating func unrollIndent(column: Int, mark: PureYAML.Parsing.Mark) {
if flowLevel != 0 {
return
}
while indent > column {
tokens.append(Token(type: .blockEnd, startMark: mark, endMark: mark))
indent = indents.removeLast()
}
lastToken = tokens.last ?? lastToken
}
/// `ENQUEUE`: append a token to the queue.
mutating func enqueue(_ token: Token) {
tokens.append(token)
lastToken = token
}
/// `QUEUE_INSERT`: insert a token at an absolute token number, used to
/// splice a `KEY` token in front of a node already queued.
mutating func insert(_ token: Token, atTokenNumber number: Int) {
tokens.insert(token, at: number - tokensParsed)
lastToken = tokens.last
}
/// Dequeue the front token, advancing `tokensParsed` like libyaml's
/// `yaml_parser_scan` does when it hands a token to the parser.
mutating func dequeue() -> Token? {
guard !tokens.isEmpty else {
return nil
}
tokensParsed += 1
return tokens.removeFirst()
}
}
}

View File

@ -0,0 +1,12 @@
extension PureYAML.Parsing.LibYAML {
/// The anchor and tag scanned ahead of a node's content, with the marks needed
/// for events and tag-resolution errors. Mirrors the locals in
/// `yaml_parser_parse_node`.
struct NodeProperties {
var anchor: String?
var tagHandle: String?
var tagSuffix: String?
var startMark: PureYAML.Parsing.Mark
var tagMark: PureYAML.Parsing.Mark
}
}

View File

@ -0,0 +1,109 @@
extension PureYAML.Parsing.LibYAML.Parser {
/// Port of `yaml_parser_parse_block_sequence_entry`.
mutating func parseBlockSequenceEntry(first: Bool) throws -> Event {
if first {
marks.append(peekToken().startMark)
skipToken()
}
let token = peekToken()
if case .blockEntry = token.type {
let mark = token.endMark
skipToken()
let next = peekToken()
if !isBlockEntry(next), !isBlockEnd(next) {
states.append(.blockSequenceEntry)
return try parseNode(block: true, indentlessSequence: false)
}
state = .blockSequenceEntry
return processEmptyScalar(at: mark)
}
if case .blockEnd = token.type {
state = states.removeLast()
_ = marks.popLast()
skipToken()
return .sequenceEnd(mark: token.startMark)
}
throw ParserError(
context: "while parsing a block collection",
problem: "did not find expected '-' indicator",
mark: marks.popLast() ?? token.startMark,
)
}
/// Port of `yaml_parser_parse_indentless_sequence_entry`. The closing token
/// belongs to the enclosing collection, so the SEQUENCE-END is emitted without
/// consuming it.
mutating func parseIndentlessSequenceEntry() throws -> Event {
let token = peekToken()
guard case .blockEntry = token.type else {
state = states.removeLast()
return .sequenceEnd(mark: token.startMark)
}
let mark = token.endMark
skipToken()
let next = peekToken()
if !isBlockEntry(next), !isKey(next), !isValue(next), !isBlockEnd(next) {
states.append(.indentlessSequenceEntry)
return try parseNode(block: true, indentlessSequence: false)
}
state = .indentlessSequenceEntry
return processEmptyScalar(at: mark)
}
/// Port of `yaml_parser_parse_block_mapping_key`.
mutating func parseBlockMappingKey(first: Bool) throws -> Event {
if first {
marks.append(peekToken().startMark)
skipToken()
}
let token = peekToken()
if case .key = token.type {
let mark = token.endMark
skipToken()
let next = peekToken()
if !isKey(next), !isValue(next), !isBlockEnd(next) {
states.append(.blockMappingValue)
return try parseNode(block: true, indentlessSequence: true)
}
state = .blockMappingValue
return processEmptyScalar(at: mark)
}
if case .blockEnd = token.type {
state = states.removeLast()
_ = marks.popLast()
skipToken()
return .mappingEnd(mark: token.startMark)
}
// YAML 1.2 permits an empty (null) key: a VALUE indicator with no
// preceding key denotes a mapping entry whose key is empty. libyaml
// rejects this here; instead emit the empty scalar key and leave the
// VALUE token for `parseBlockMappingValue` to consume.
if case .value = token.type {
state = .blockMappingValue
return processEmptyScalar(at: token.startMark)
}
throw ParserError(
context: "while parsing a block mapping",
problem: "did not find expected key",
mark: marks.popLast() ?? token.startMark,
)
}
/// Port of `yaml_parser_parse_block_mapping_value`.
mutating func parseBlockMappingValue() throws -> Event {
let token = peekToken()
guard case .value = token.type else {
state = .blockMappingKey
return processEmptyScalar(at: token.startMark)
}
let mark = token.endMark
skipToken()
let next = peekToken()
if !isKey(next), !isValue(next), !isBlockEnd(next) {
states.append(.blockMappingKey)
return try parseNode(block: true, indentlessSequence: true)
}
state = .blockMappingKey
return processEmptyScalar(at: mark)
}
}

View File

@ -0,0 +1,70 @@
extension PureYAML.Parsing.LibYAML.Parser {
/// The implied tag handles, mirroring libyaml's `default_tag_directives`.
private var defaultTagDirectives: [(handle: String, prefix: String)] {
[("!", "!"), ("!!", "tag:yaml.org,2002:")]
}
/// Port of `yaml_parser_process_directives`: consume the `%YAML` and `%TAG`
/// directive tokens at the head of a document, validating the version and
/// recording the tag handles, then install the default tag handles. The
/// version is checked against a local so a duplicate `%YAML` is rejected.
mutating func processDirectives() throws {
var version: (major: Int, minor: Int)?
var token = peekToken()
while true {
if case let .versionDirective(major, minor) = token.type {
try recordVersion(major: major, minor: minor, into: &version, mark: token.startMark)
} else if case let .tagDirective(handle, prefix) = token.type {
try appendTagDirective(handle: handle, prefix: prefix, allowDuplicates: false, mark: token.startMark)
} else if case .reservedDirective = token.type {
// A reserved directive is ignored (YAML 1.2 §6.8.1) but still
// consumed here so the explicit-document path requires a `---`.
} else {
break
}
skipToken()
token = peekToken()
}
for directive in defaultTagDirectives {
try appendTagDirective(handle: directive.handle, prefix: directive.prefix, allowDuplicates: true, mark: token.startMark)
}
}
private mutating func recordVersion(major: Int, minor: Int, into version: inout (major: Int, minor: Int)?, mark: Mark) throws {
guard version == nil else {
throw ParserError(context: nil, problem: "found duplicate %YAML directive", mark: mark)
}
// YAML 1.2 §6.8.1: a document whose major version is 1 is parsed even when
// the minor version is newer than 1.2 (with a warning), rather than
// libyaml's hard rejection of any minor other than 1 or 2.
guard major == 1 else {
throw ParserError(context: nil, problem: "found incompatible YAML document", mark: mark)
}
version = (major, minor)
}
/// Port of `yaml_parser_append_tag_directive`: append a handle/prefix mapping,
/// rejecting a redefinition unless duplicates are allowed (the defaults are).
mutating func appendTagDirective(handle: String, prefix: String, allowDuplicates: Bool, mark: Mark) throws {
if tagDirectives.contains(where: { $0.handle == handle }) {
if allowDuplicates {
return
}
throw ParserError(context: nil, problem: "found duplicate %TAG directive", mark: mark)
}
tagDirectives.append((handle, prefix))
}
/// Port of the tag-resolution block in `yaml_parser_parse_node`: an empty
/// handle means the suffix is a verbatim tag; otherwise substitute the handle's
/// registered prefix, failing on an undefined handle.
func resolveTag(handle: String, suffix: String, startMark _: Mark, tagMark: Mark) throws -> String {
guard !handle.isEmpty else {
return suffix
}
guard let directive = tagDirectives.first(where: { $0.handle == handle }) else {
throw ParserError(context: "while parsing a node", problem: "found undefined tag handle", mark: tagMark)
}
return directive.prefix + suffix
}
}

View File

@ -0,0 +1,112 @@
extension PureYAML.Parsing.LibYAML.Parser {
/// Port of `yaml_parser_parse_stream_start`.
mutating func parseStreamStart() throws -> Event {
let token = peekToken()
guard case .streamStart = token.type else {
throw ParserError(context: nil, problem: "did not find expected <stream-start>", mark: token.startMark)
}
state = .implicitDocumentStart
skipToken()
return .streamStart(mark: token.startMark)
}
/// Port of `yaml_parser_parse_document_start`, extended for YAML 1.2: stray
/// `...` markers are skipped, then a bare document is opened when one is allowed
/// (the first document or after an explicit `...`), an explicit document is
/// opened after directives and a `---`, or STREAM-END is emitted. libyaml only
/// allows a bare document as the very first one.
mutating func parseDocumentStart() throws -> Event {
var token = peekToken()
while case .documentEnd = token.type {
skipToken()
token = peekToken()
}
// Directives apply to the document that follows, so the previous document
// must be closed with an explicit `...` end marker before them. After an
// implicit end (`allowBareDocument` is false) a directive is an error.
if isDirective(token), !allowBareDocument {
throw ParserError(
context: nil,
problem: "directives require a preceding document end marker",
mark: token.startMark,
)
}
if allowBareDocument, !isDirectiveOrBoundary(token) {
try processDirectives()
states.append(.documentEnd)
state = .blockNode
return .documentStart(explicit: false, mark: token.startMark)
}
if case .streamEnd = token.type {
state = .end
skipToken()
return .streamEnd(mark: token.startMark)
}
return try openExplicitDocument(startMark: token.startMark)
}
/// The explicit-document branch: directives then a required `---`.
private mutating func openExplicitDocument(startMark: Mark) throws -> Event {
try processDirectives()
let token = peekToken()
guard case .documentStart = token.type else {
throw ParserError(context: nil, problem: "did not find expected <document start>", mark: token.startMark)
}
states.append(.documentEnd)
state = .documentContent
skipToken()
return .documentStart(explicit: true, mark: startMark)
}
/// Port of `yaml_parser_parse_document_content`: an empty scalar stands in for
/// a document with no node, otherwise parse the root node.
mutating func parseDocumentContent() throws -> Event {
let token = peekToken()
if isDirectiveOrBoundary(token) || isDocumentEnd(token) {
state = states.removeLast()
return processEmptyScalar(at: token.startMark)
}
return try parseNode(block: true, indentlessSequence: false)
}
/// Port of `yaml_parser_parse_document_end`: reset the per-document tag
/// directives and emit DOCUMENT-END, explicit only when `...` was present.
mutating func parseDocumentEnd() throws -> Event {
let token = peekToken()
var explicit = false
if case .documentEnd = token.type {
skipToken()
explicit = true
}
tagDirectives.removeAll()
// A bare document may follow only an explicit `...` end marker.
allowBareDocument = explicit
state = .documentStart
return .documentEnd(explicit: explicit, mark: token.startMark)
}
private func isDirectiveOrBoundary(_ token: Token) -> Bool {
switch token.type {
case .versionDirective, .tagDirective, .reservedDirective, .documentStart, .streamEnd:
true
default:
false
}
}
private func isDirective(_ token: Token) -> Bool {
switch token.type {
case .versionDirective, .tagDirective, .reservedDirective:
true
default:
false
}
}
private func isDocumentEnd(_ token: Token) -> Bool {
if case .documentEnd = token.type {
return true
}
return false
}
}

View File

@ -0,0 +1,157 @@
extension PureYAML.Parsing.LibYAML.Parser {
/// Port of `yaml_parser_parse_flow_sequence_entry`. A `?` key inside a flow
/// sequence opens a single-pair flow mapping.
mutating func parseFlowSequenceEntry(first: Bool) throws -> Event {
if first {
marks.append(peekToken().startMark)
skipToken()
}
var token = peekToken()
if !isFlowSequenceEnd(token) {
if !first {
guard isFlowEntry(token) else {
throw ParserError(
context: "while parsing a flow sequence",
problem: "did not find expected ',' or ']'",
mark: marks.popLast() ?? token.startMark,
)
}
skipToken()
token = peekToken()
}
if isKey(token) {
state = .flowSequenceEntryMappingKey
skipToken()
return .mappingStart(anchor: nil, tag: nil, style: .flow, mark: token.startMark)
}
if isValue(token) {
// A leading `:` opens a single-pair mapping with an empty key
// (`[ : v ]`). Keep the VALUE token for the mapping-key state.
state = .flowSequenceEntryMappingKey
return .mappingStart(anchor: nil, tag: nil, style: .flow, mark: token.startMark)
}
if !isFlowSequenceEnd(token) {
states.append(.flowSequenceEntry)
return try parseNode(block: false, indentlessSequence: false)
}
}
state = states.removeLast()
_ = marks.popLast()
skipToken()
return .sequenceEnd(mark: token.startMark)
}
/// Port of `yaml_parser_parse_flow_sequence_entry_mapping_key`.
mutating func parseFlowSequenceEntryMappingKey() throws -> Event {
let token = peekToken()
if !isValue(token), !isFlowEntry(token), !isFlowSequenceEnd(token) {
states.append(.flowSequenceEntryMappingValue)
return try parseNode(block: false, indentlessSequence: false)
}
// An empty key: leave a VALUE token for the value state to consume (so the
// value is parsed); a closing `]` ends the pair with an empty value too.
if isValue(token) || isFlowSequenceEnd(token) {
state = .flowSequenceEntryMappingValue
return processEmptyScalar(at: token.startMark)
}
let mark = token.endMark
skipToken()
state = .flowSequenceEntryMappingValue
return processEmptyScalar(at: mark)
}
/// Port of `yaml_parser_parse_flow_sequence_entry_mapping_value`.
mutating func parseFlowSequenceEntryMappingValue() throws -> Event {
var token = peekToken()
if isValue(token) {
skipToken()
token = peekToken()
if !isFlowEntry(token), !isFlowSequenceEnd(token) {
states.append(.flowSequenceEntryMappingEnd)
return try parseNode(block: false, indentlessSequence: false)
}
}
state = .flowSequenceEntryMappingEnd
return processEmptyScalar(at: token.startMark)
}
/// Port of `yaml_parser_parse_flow_sequence_entry_mapping_end`.
mutating func parseFlowSequenceEntryMappingEnd() throws -> Event {
let token = peekToken()
state = .flowSequenceEntry
return .mappingEnd(mark: token.startMark)
}
/// Port of `yaml_parser_parse_flow_mapping_key`.
mutating func parseFlowMappingKey(first: Bool) throws -> Event {
if first {
marks.append(peekToken().startMark)
skipToken()
}
var token = peekToken()
if !isFlowMappingEnd(token) {
if !first {
guard isFlowEntry(token) else {
throw ParserError(
context: "while parsing a flow mapping",
problem: "did not find expected ',' or '}'",
mark: marks.popLast() ?? token.startMark,
)
}
skipToken()
token = peekToken()
}
if isKey(token) {
return try flowMappingKeyEvent()
}
// YAML 1.2 permits an empty (null) key in a flow mapping: a `:` with
// no preceding key denotes an entry whose key is empty. libyaml hands
// the VALUE token to `parse_node` and fails; instead emit the empty
// scalar key and let `parseFlowMappingValue` consume the indicator.
if isValue(token) {
state = .flowMappingValue
return processEmptyScalar(at: token.startMark)
}
if !isFlowMappingEnd(token) {
states.append(.flowMappingEmptyValue)
return try parseNode(block: false, indentlessSequence: false)
}
}
state = states.removeLast()
_ = marks.popLast()
skipToken()
return .mappingEnd(mark: token.startMark)
}
/// The KEY branch of `parse_flow_mapping_key`: parse the key node, or an empty
/// scalar when a value/entry/end follows the `?` immediately.
private mutating func flowMappingKeyEvent() throws -> Event {
skipToken()
let token = peekToken()
if !isValue(token), !isFlowEntry(token), !isFlowMappingEnd(token) {
states.append(.flowMappingValue)
return try parseNode(block: false, indentlessSequence: false)
}
state = .flowMappingValue
return processEmptyScalar(at: token.startMark)
}
/// Port of `yaml_parser_parse_flow_mapping_value`.
mutating func parseFlowMappingValue(empty: Bool) throws -> Event {
var token = peekToken()
if empty {
state = .flowMappingKey
return processEmptyScalar(at: token.startMark)
}
if isValue(token) {
skipToken()
token = peekToken()
if !isFlowEntry(token), !isFlowMappingEnd(token) {
states.append(.flowMappingKey)
return try parseNode(block: false, indentlessSequence: false)
}
}
state = .flowMappingKey
return processEmptyScalar(at: token.startMark)
}
}

View File

@ -0,0 +1,50 @@
extension PureYAML.Parsing.LibYAML.Parser {
func isBlockEntry(_ token: Token) -> Bool {
if case .blockEntry = token.type {
return true
}
return false
}
func isBlockEnd(_ token: Token) -> Bool {
if case .blockEnd = token.type {
return true
}
return false
}
func isKey(_ token: Token) -> Bool {
if case .key = token.type {
return true
}
return false
}
func isValue(_ token: Token) -> Bool {
if case .value = token.type {
return true
}
return false
}
func isFlowEntry(_ token: Token) -> Bool {
if case .flowEntry = token.type {
return true
}
return false
}
func isFlowSequenceEnd(_ token: Token) -> Bool {
if case .flowSequenceEnd = token.type {
return true
}
return false
}
func isFlowMappingEnd(_ token: Token) -> Bool {
if case .flowMappingEnd = token.type {
return true
}
return false
}
}

View File

@ -0,0 +1,99 @@
extension PureYAML.Parsing.LibYAML.Parser {
typealias NodeProperties = PureYAML.Parsing.LibYAML.NodeProperties
/// Port of `yaml_parser_parse_node`. An ALIAS is emitted directly; otherwise
/// the node's anchor/tag properties are consumed, the tag is resolved, and the
/// content token selects a scalar, a flow/block collection start, or an empty
/// scalar standing in for a property-only node.
mutating func parseNode(block: Bool, indentlessSequence: Bool) throws -> Event {
let token = peekToken()
if case let .alias(name) = token.type {
state = states.removeLast()
skipToken()
return .alias(anchor: name, mark: token.startMark)
}
let props = consumeNodeProperties()
let tag = try props.tagHandle.map {
try resolveTag(handle: $0, suffix: props.tagSuffix ?? "", startMark: props.startMark, tagMark: props.tagMark)
}
return try dispatchNodeContent(block: block, indentlessSequence: indentlessSequence, props: props, tag: tag)
}
/// Consume `properties ::= TAG ANCHOR? | ANCHOR TAG?` ahead of the content.
private mutating func consumeNodeProperties() -> NodeProperties {
var token = peekToken()
var props = NodeProperties(anchor: nil, tagHandle: nil, tagSuffix: nil, startMark: token.startMark, tagMark: token.startMark)
if case let .anchor(name) = token.type {
props.anchor = name
skipToken()
token = peekToken()
if case let .tag(handle, suffix) = token.type {
props.tagHandle = handle
props.tagSuffix = suffix
props.tagMark = token.startMark
skipToken()
}
} else if case let .tag(handle, suffix) = token.type {
props.tagHandle = handle
props.tagSuffix = suffix
props.tagMark = token.startMark
skipToken()
token = peekToken()
if case let .anchor(name) = token.type {
props.anchor = name
skipToken()
}
}
return props
}
/// Select the event for the content token following a node's properties.
private mutating func dispatchNodeContent(block: Bool, indentlessSequence: Bool, props: NodeProperties, tag: String?) throws -> Event {
let token = peekToken()
if indentlessSequence, case .blockEntry = token.type {
state = .indentlessSequenceEntry
return .sequenceStart(anchor: props.anchor, tag: tag, style: .block, mark: props.startMark)
}
if let event = contentEvent(token: token, block: block, props: props, tag: tag) {
return event
}
return try emptyNodeOrError(token: token, block: block, props: props, tag: tag)
}
/// A scalar or a flow/block collection start, or nil if the token is neither.
private mutating func contentEvent(token: Token, block: Bool, props: NodeProperties, tag: String?) -> Event? {
switch token.type {
case let .scalar(value, style):
state = states.removeLast()
skipToken()
return .scalar(value: value, anchor: props.anchor, tag: tag, style: style, mark: props.startMark, endMark: token.endMark)
case .flowSequenceStart:
state = .flowSequenceFirstEntry
return .sequenceStart(anchor: props.anchor, tag: tag, style: .flow, mark: props.startMark)
case .flowMappingStart:
state = .flowMappingFirstKey
return .mappingStart(anchor: props.anchor, tag: tag, style: .flow, mark: props.startMark)
case .blockSequenceStart where block:
state = .blockSequenceFirstEntry
return .sequenceStart(anchor: props.anchor, tag: tag, style: .block, mark: props.startMark)
case .blockMappingStart where block:
state = .blockMappingFirstKey
return .mappingStart(anchor: props.anchor, tag: tag, style: .block, mark: props.startMark)
default:
return nil
}
}
/// An empty scalar when the node carried only properties, else a parse error.
private mutating func emptyNodeOrError(token: Token, block: Bool, props: NodeProperties, tag: String?) throws -> Event {
if props.anchor != nil || tag != nil {
state = states.removeLast()
return .scalar(value: "", anchor: props.anchor, tag: tag, style: .plain, mark: props.startMark, endMark: props.startMark)
}
throw ParserError(
context: block ? "while parsing a block node" : "while parsing a flow node",
problem: "did not find expected node content",
mark: token.startMark,
)
}
}

View File

@ -0,0 +1,148 @@
extension PureYAML.Parsing.LibYAML {
/// A faithful port of libyaml's `parser.c`: a one-event-at-a-time state
/// machine that consumes the scanner's token stream and produces the YAML
/// event stream. The token queue is materialized up front by the scanner, so
/// `peekToken`/`skipToken` index into a fixed array rather than pulling lazily,
/// which is equivalent because simple-key resolution lives entirely in the
/// scanner.
struct Parser {
let tokens: [Token]
var cursor = 0
var state: ParserState = .streamStart
var states: [ParserState] = []
var marks: [Mark] = []
var tagDirectives: [(handle: String, prefix: String)] = []
/// Whether a bare (implicit) document may begin here: true for the first
/// document and after an explicit `...` end marker, false otherwise. YAML
/// 1.2 allows a bare document to follow `...`, which libyaml rejects.
var allowBareDocument = true
init(_ input: String = "") throws {
var scanner = Scanner(input)
tokens = try scanner.scanAll()
}
init(tokens: [Token]) {
self.tokens = tokens
}
/// Drive the state machine until STREAM-END, collecting every event.
mutating func parse() throws -> [Event] {
var events: [Event] = []
while state != .end {
try events.append(stateMachine())
}
return events
}
/// The head of the token queue. STREAM-END is always the final token and
/// the machine halts once it has been consumed, so the clamp is never the
/// value actually parsed; it only avoids an out-of-range index.
func peekToken() -> Token {
tokens[min(cursor, tokens.count - 1)]
}
mutating func skipToken() {
cursor += 1
}
/// Port of `yaml_parser_state_machine`: dispatch on the current state. The
/// flat 24-way switch is split into document/block/flow groups to keep each
/// dispatcher's branching within the complexity budget.
mutating func stateMachine() throws -> Event {
switch state {
case .streamStart, .implicitDocumentStart, .documentStart, .documentContent, .documentEnd:
try documentStateMachine()
case .blockNode, .blockNodeOrIndentlessSequence, .flowNode,
.blockSequenceFirstEntry, .blockSequenceEntry, .indentlessSequenceEntry,
.blockMappingFirstKey, .blockMappingKey, .blockMappingValue:
try nodeStateMachine()
case .flowSequenceFirstEntry, .flowSequenceEntry, .flowSequenceEntryMappingKey,
.flowSequenceEntryMappingValue, .flowSequenceEntryMappingEnd,
.flowMappingFirstKey, .flowMappingKey, .flowMappingValue, .flowMappingEmptyValue:
try flowStateMachine()
case .end:
throw endError()
}
}
private mutating func documentStateMachine() throws -> Event {
switch state {
case .streamStart:
try parseStreamStart()
case .implicitDocumentStart:
try parseDocumentStart()
case .documentStart:
try parseDocumentStart()
case .documentContent:
try parseDocumentContent()
default:
try parseDocumentEnd()
}
}
private mutating func nodeStateMachine() throws -> Event {
switch state {
case .blockNode:
try parseNode(block: true, indentlessSequence: false)
case .blockNodeOrIndentlessSequence:
try parseNode(block: true, indentlessSequence: true)
case .flowNode:
try parseNode(block: false, indentlessSequence: false)
case .blockSequenceFirstEntry:
try parseBlockSequenceEntry(first: true)
case .blockSequenceEntry:
try parseBlockSequenceEntry(first: false)
case .indentlessSequenceEntry:
try parseIndentlessSequenceEntry()
case .blockMappingFirstKey:
try parseBlockMappingKey(first: true)
case .blockMappingKey:
try parseBlockMappingKey(first: false)
default:
try parseBlockMappingValue()
}
}
private mutating func flowStateMachine() throws -> Event {
switch state {
case .flowSequenceFirstEntry:
try parseFlowSequenceEntry(first: true)
case .flowSequenceEntry:
try parseFlowSequenceEntry(first: false)
case .flowSequenceEntryMappingKey:
try parseFlowSequenceEntryMappingKey()
case .flowSequenceEntryMappingValue:
try parseFlowSequenceEntryMappingValue()
case .flowSequenceEntryMappingEnd:
try parseFlowSequenceEntryMappingEnd()
case .flowMappingFirstKey:
try parseFlowMappingKey(first: true)
case .flowMappingKey:
try parseFlowMappingKey(first: false)
case .flowMappingValue:
try parseFlowMappingValue(empty: false)
default:
try parseFlowMappingValue(empty: true)
}
}
private func endError() -> ParserError {
ParserError(context: nil, problem: "parsed past the end of the stream", mark: peekToken().startMark)
}
/// Port of `yaml_parser_process_empty_scalar`: an empty plain scalar event
/// stands in for an absent node.
func processEmptyScalar(at mark: Mark) -> Event {
.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark, endMark: mark)
}
}
}
extension PureYAML.Parsing.LibYAML.Parser {
typealias Token = PureYAML.Parsing.LibYAML.Token
typealias Mark = PureYAML.Parsing.Mark
typealias Event = PureYAML.Parsing.Event
typealias ParserError = PureYAML.Parsing.LibYAML.ParserError
typealias ParserState = PureYAML.Parsing.LibYAML.ParserState
}

View File

@ -0,0 +1,10 @@
extension PureYAML.Parsing.LibYAML {
/// A parser error, mirroring `yaml_parser_set_parser_error[_context]`: an
/// optional context describing the surrounding production, the immediate
/// problem, and the mark where it was detected.
struct ParserError: Error, Equatable {
var context: String?
var problem: String
var mark: PureYAML.Parsing.Mark
}
}

View File

@ -0,0 +1,30 @@
extension PureYAML.Parsing.LibYAML {
/// The parser state, mirroring `yaml_parser_state_e`. Each case names the
/// production the state machine is in the middle of recognizing.
enum ParserState: Equatable {
case streamStart
case implicitDocumentStart
case documentStart
case documentContent
case documentEnd
case blockNode
case blockNodeOrIndentlessSequence
case flowNode
case blockSequenceFirstEntry
case blockSequenceEntry
case indentlessSequenceEntry
case blockMappingFirstKey
case blockMappingKey
case blockMappingValue
case flowSequenceFirstEntry
case flowSequenceEntry
case flowSequenceEntryMappingKey
case flowSequenceEntryMappingValue
case flowSequenceEntryMappingEnd
case flowMappingFirstKey
case flowMappingKey
case flowMappingValue
case flowMappingEmptyValue
case end
}
}

View File

@ -0,0 +1,152 @@
extension PureYAML.Parsing.LibYAML {
/// Port of libyaml's input buffer and position tracking (the `buffer`/`mark`
/// fields with the `PEEK`, `SKIP`, and `SKIP_LINE` macros). It iterates
/// `Unicode.Scalar`s rather than grapheme clusters so a CR and an LF are
/// distinct, matching libyaml's byte-oriented scanning, and tracks the mark in
/// libyaml's 0-based line and column convention; conversion to the package's
/// 1-based `Mark` happens at token emission during integration.
struct Reader {
private var scalars: [Unicode.Scalar]
private(set) var index = 0
private(set) var line = 0
private(set) var column = 0
/// Whether the buffered `scalars` are the complete input. True for every
/// batch caller (the default), so `ranDry` can never be set and this type's
/// behavior is unchanged for `scanAll()`. A resumable caller passes `false`
/// and later calls `markFinal()` once the last chunk has been appended.
private(set) var isFinal: Bool
/// Set by `peek(_:)` when it was asked for a position past the buffered
/// scalars while `!isFinal`: the query is inconclusive, not a real end of
/// input, and the caller must suspend and retry after `append(_:)`. Batch
/// callers (`isFinal == true`) never set this, since the guard below is
/// unreachable when `isFinal` is true.
private(set) var ranDry = false
/// The index of the first non-`c-printable` scalar found in `scalars`
/// (libyaml's reader-level check, `yaml_parser_update_buffer`'s printable
/// validation), or nil if none has been found yet. Set once, by whichever
/// of `init`/`append(_:)` first buffers the offending scalar; never
/// cleared. `fetchNextToken()` checks this once per token and throws,
/// mirroring libyaml rejecting raw control characters (anything outside
/// `c-printable`, i.e. not a plain tab/CR/LF and not in the printable
/// ranges `CharacterClass.isPrintable` implements) wherever they occur in
/// the stream, not only inside specific constructs like anchor names.
private(set) var invalidControlCharacterIndex: Int?
init(_ string: String = "", isFinal: Bool = true) {
scalars = Array(string.unicodeScalars)
self.isFinal = isFinal
recordFirstInvalidControlCharacter(in: 0 ..< scalars.count)
}
/// `PEEK_TOKEN` / `CHECK_AT`: the scalar `offset` ahead, or nil past the
/// end (libyaml's `\0`). Past the buffered end while more chunks may still
/// arrive, this flags `ranDry` instead of asserting true end of input.
mutating func peek(_ offset: Int = 0) -> Unicode.Scalar? {
let position = index + offset
if position < scalars.count {
return scalars[position]
}
if !isFinal {
ranDry = true
}
return nil
}
/// Append a chunk of newly arrived input. Only meaningful before
/// `markFinal()`. Clears `ranDry`: new data may satisfy whatever `peek(_:)`
/// call set it, and nothing else ever clears it. The existing
/// checkpoint-restore path gets this for free (the restored `Reader` is a
/// copy from before the dry read), but a resumed scan that keeps its
/// reader position instead of restoring a snapshot (`PlainScalarSuspended`)
/// has no other point where a stale `true` would ever become false again.
mutating func append(_ chunk: String) {
let previousCount = scalars.count
scalars.append(contentsOf: chunk.unicodeScalars)
ranDry = false
recordFirstInvalidControlCharacter(in: previousCount ..< scalars.count)
}
/// Declare that no further chunks are coming: a `peek` past the buffered
/// end is now a real end of input, matching the batch reader's behavior.
mutating func markFinal() {
isFinal = true
}
var isAtEnd: Bool {
index >= scalars.count
}
/// The current position. Column is 0-based, as in libyaml.
var mark: PureYAML.Parsing.Mark {
PureYAML.Parsing.Mark(line: line, column: column, index: index)
}
/// `SKIP`: consume one non-break scalar, advancing the column.
mutating func skip() {
guard index < scalars.count else {
return
}
index += 1
column += 1
}
/// `SKIP_LINE`: consume a line break and start a new line. A CR+LF pair is
/// one break, matching libyaml.
mutating func skipLine() {
if peek() == "\r", peek(1) == "\n" {
index += 2
} else if CharacterClass.isBreak(peek()) {
index += 1
} else {
return
}
line += 1
column = 0
}
/// The 0-based line/column of `scalars[index]`, computed by counting line
/// breaks from the start of the buffer. Not the hot `mark` path (that
/// tracks `line`/`column` incrementally as scanning consumes characters);
/// used only once, lazily, when `fetchNextToken()` actually needs to report
/// `invalidControlCharacterIndex` as a `Mark`, since that index can be
/// anywhere in the buffer, not necessarily at the reader's current
/// scanning position.
func mark(atIndex target: Int) -> PureYAML.Parsing.Mark {
var line = 0
var column = 0
var cursor = 0
while cursor < target, cursor < scalars.count {
if scalars[cursor] == "\r", cursor + 1 < scalars.count, scalars[cursor + 1] == "\n" {
line += 1
column = 0
cursor += 2
} else if CharacterClass.isBreak(scalars[cursor]) {
line += 1
column = 0
cursor += 1
} else {
column += 1
cursor += 1
}
}
return PureYAML.Parsing.Mark(line: line, column: column, index: target)
}
/// Scans `range` (a slice of `scalars` just buffered by `init`/`append(_:)`)
/// for the first scalar outside `CharacterClass.isPrintable`, recording its
/// index if one is found and none was already recorded. Only the first
/// occurrence overall matters (matching libyaml, which reports the first
/// invalid byte and stops), so once `invalidControlCharacterIndex` is set
/// this is a no-op.
private mutating func recordFirstInvalidControlCharacter(in range: Range<Int>) {
guard invalidControlCharacterIndex == nil else {
return
}
for index in range where !CharacterClass.isPrintable(scalars[index]) {
invalidControlCharacterIndex = index
return
}
}
}
}

View File

@ -0,0 +1,45 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
/// Port of `yaml_parser_scan_anchor`, widened to YAML 1.2: an anchor (`&name`)
/// or alias (`*name`) name is `ns-anchor-char+`, i.e. any printable character
/// that is not whitespace, a line break, or a flow indicator (`,[]{}`). libyaml
/// restricts the name to alphanumerics, which rejects Unicode and punctuation
/// anchor names that YAML 1.2 allows.
mutating func scanAnchor(isAnchor: Bool) throws -> PureYAML.Parsing.LibYAML.Token {
let startMark = reader.mark
reader.skip()
var name = ""
while let character = reader.peek(), isAnchorCharacter(character) {
name.unicodeScalars.append(character)
reader.skip()
}
let endMark = reader.mark
guard !name.isEmpty else {
throw PureYAML.Parsing.LibYAML.ScannerError(
context: isAnchor ? "while scanning an anchor" : "while scanning an alias",
problem: "did not find expected alphabetic or numeric character",
mark: startMark,
)
}
return PureYAML.Parsing.LibYAML.Token(
type: isAnchor ? .anchor(name: name) : .alias(name: name),
startMark: startMark,
endMark: endMark,
)
}
/// `ns-anchor-char`: a printable, non-blank, non-break character that is not a
/// flow indicator or the byte-order mark.
private func isAnchorCharacter(_ scalar: Unicode.Scalar) -> Bool {
guard Classify.isPrintable(scalar), !Classify.isBlankZ(scalar), scalar != "\u{FEFF}" else {
return false
}
switch scalar {
case ",", "[", "]", "{", "}":
return false
default:
return true
}
}
}

View File

@ -0,0 +1,205 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
/// The accumulation state for `scan_block_scalar`.
private struct BlockState {
var string = ""
var leadingBreak = ""
var trailingBreaks = ""
var leadingBlank = false
}
/// Port of `yaml_parser_scan_block_scalar`. Scans a literal (`|`) or folded
/// (`>`) block scalar: parses the chomping (`+`/`-`) and indentation indicators
/// from the header, auto-detects the indentation if not given, then folds the
/// content (literal keeps breaks, folded joins equally-indented non-empty lines
/// with a space) and applies clip/strip/keep chomping to the trailing breaks.
mutating func scanBlockScalar(literal: Bool) throws -> PureYAML.Parsing.LibYAML.Token {
let startMark = reader.mark
reader.skip()
let header = try readBlockScalarHeader(startMark: startMark)
try consumeBlockHeaderTail(startMark: startMark)
var endMark = reader.mark
// -1 is the "indentation not yet auto-detected" sentinel. libyaml uses 0,
// but a root-level block scalar legitimately detects indentation 0, so 0
// must be a representable detected value rather than the sentinel.
var indent = -1
if header.increment != 0 {
indent = indentEngine.indent >= 0 ? indentEngine.indent + header.increment : header.increment
}
var state = BlockState()
try scanBlockScalarBreaks(indent: &indent, state: &state, startMark: startMark, endMark: &endMark)
while reader.column == indent, reader.peek() != nil {
// A root block scalar (indent 0) ends at a `---`/`...` document marker
// in column 0. libyaml never needs this because it floors the indent
// at 1, but indent 0 lets the loop reach a column-0 marker line.
if reader.column == 0, documentMarkerAhead() {
break
}
foldBlockLine(&state, literal: literal)
while let character = reader.peek(), !Classify.isBreakZ(character) {
state.string.unicodeScalars.append(character)
reader.skip()
}
state.leadingBreak += readBlockLine()
try scanBlockScalarBreaks(indent: &indent, state: &state, startMark: startMark, endMark: &endMark)
}
if header.chomping != -1 {
state.string += state.leadingBreak
}
if header.chomping == 1 {
state.string += state.trailingBreaks
}
return PureYAML.Parsing.LibYAML.Token(
type: .scalar(value: state.string, style: literal ? .literal : .folded),
startMark: startMark,
endMark: endMark,
)
}
/// Parse the chomping and indentation indicators in either order.
private mutating func readBlockScalarHeader(startMark: PureYAML.Parsing.Mark) throws -> (chomping: Int, increment: Int) {
var chomping = 0
var increment = 0
if reader.peek() == "+" || reader.peek() == "-" {
chomping = reader.peek() == "+" ? 1 : -1
reader.skip()
if Classify.isDigit(reader.peek()) {
increment = try blockIncrement(startMark: startMark)
}
} else if Classify.isDigit(reader.peek()) {
increment = try blockIncrement(startMark: startMark)
if reader.peek() == "+" || reader.peek() == "-" {
chomping = reader.peek() == "+" ? 1 : -1
reader.skip()
}
}
return (chomping, increment)
}
private mutating func blockIncrement(startMark: PureYAML.Parsing.Mark) throws -> Int {
if reader.peek() == "0" {
throw blockError("found an indentation indicator equal to 0", startMark)
}
let value = digitValue(reader.peek())
reader.skip()
return value
}
/// Eat trailing blanks and a comment to the end of the header line, then the
/// line break.
private mutating func consumeBlockHeaderTail(startMark: PureYAML.Parsing.Mark) throws {
while reader.peek() == " " {
reader.skip()
}
if reader.peek() == "#" {
// A comment needs a preceding space, so `>#` (a `#` adjacent to the
// block scalar indicator) is malformed, not a comment. libyaml accepts it.
if reader.column > 0, let previous = reader.peek(-1), !Classify.isBlank(previous) {
throw blockError("found a comment without a preceding space", startMark)
}
while !Classify.isBreakZ(reader.peek()) {
reader.skip()
}
}
guard Classify.isBreakZ(reader.peek()) else {
throw blockError("did not find expected comment or line break", startMark)
}
if Classify.isBreak(reader.peek()) {
reader.skipLine()
}
}
/// Port of `yaml_parser_scan_block_scalar_breaks`: eat indentation spaces and
/// blank lines, auto-detecting the indentation level when none was given.
private mutating func scanBlockScalarBreaks(indent: inout Int, state: inout BlockState, startMark: PureYAML.Parsing.Mark, endMark: inout PureYAML.Parsing.Mark) throws {
var maxIndent = 0
endMark = reader.mark
// A tab is content (or separation) once the column reaches the content
// indentation; it is illegal only while still inside the required
// indentation. During auto-detection the floor is the parent indent + 1.
let tabFloor = indent >= 0 ? indent : max(0, indentEngine.indent + 1)
while true {
let lineStart = reader.column
while indent < 0 || reader.column < indent, reader.peek() == " " {
reader.skip()
}
if reader.column > maxIndent {
maxIndent = reader.column
}
if reader.column < tabFloor, Classify.isTab(reader.peek()) {
throw blockError("found a tab character where an indentation space is expected", startMark)
}
if !Classify.isBreak(reader.peek()) {
// A trailing whitespace-only line that ends at end of input without
// a break still counts as one line break for keep chomping (as the
// test-suite reference does), even with no break byte present.
if reader.peek() == nil, reader.column > lineStart {
state.trailingBreaks += "\n"
}
break
}
state.trailingBreaks += readBlockLine()
endMark = reader.mark
}
// A leading empty line may not be more indented than the first content
// line; libyaml silently detects the deeper indent and drops the content.
if indent < 0, !Classify.isBreakZ(reader.peek()), reader.column < maxIndent {
throw blockError("found a leading empty line more indented than the content", startMark)
}
if indent < 0 {
// Auto-detected indentation. A root-level block scalar has a parent
// indent of -1, so the floor is 0 (allowing column-0 content), not
// libyaml's 1. maxIndent >= 0 keeps the result non-negative.
indent = max(maxIndent, indentEngine.indent + 1)
}
}
/// Fold the pending breaks at the start of a content line (folded style joins a
/// single break between non-blank lines into a space; otherwise breaks are
/// kept).
private mutating func foldBlockLine(_ state: inout BlockState, literal: Bool) {
let trailingBlank = Classify.isBlank(reader.peek())
if !literal, state.leadingBreak.first == "\n", !state.leadingBlank, !trailingBlank {
if state.trailingBreaks.isEmpty {
state.string += " "
}
state.leadingBreak = ""
} else {
state.string += state.leadingBreak
state.leadingBreak = ""
}
state.string += state.trailingBreaks
state.trailingBreaks = ""
state.leadingBlank = Classify.isBlank(reader.peek())
}
private mutating func readBlockLine() -> String {
reader.skipLine()
return "\n"
}
/// Whether a `---` or `...` document marker (three repeats then a blank, break,
/// or end of input) starts at the current position.
private mutating func documentMarkerAhead() -> Bool {
let first = reader.peek()
guard first == "-" || first == "." else {
return false
}
return reader.peek(1) == first && reader.peek(2) == first && Classify.isBlankZ(reader.peek(3))
}
private func digitValue(_ scalar: Unicode.Scalar?) -> Int {
guard let scalar, scalar >= "0", scalar <= "9" else {
return 0
}
return Int(scalar.value - Unicode.Scalar("0").value)
}
private func blockError(_ problem: String, _ mark: PureYAML.Parsing.Mark) -> PureYAML.Parsing.LibYAML.ScannerError {
PureYAML.Parsing.LibYAML.ScannerError(context: "while scanning a block scalar", problem: problem, mark: mark)
}
}

View File

@ -0,0 +1,153 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
private typealias Token = PureYAML.Parsing.LibYAML.Token
private typealias Mark = PureYAML.Parsing.Mark
/// The longest version number libyaml accepts (`MAX_NUMBER_LENGTH`).
private var maxVersionNumberLength: Int {
9
}
/// Port of `yaml_parser_fetch_directive`: reset the indentation and simple keys,
/// then scan a `%YAML` or `%TAG` directive token.
mutating func fetchDirective() throws {
indentEngine.unrollIndent(column: -1, mark: reader.mark)
try removeSimpleKey()
simpleKeyAllowed = false
let token = try scanDirective()
indentEngine.enqueue(token)
}
/// Port of `yaml_parser_scan_directive`, extended to YAML 1.2 §6.8.1: a `%YAML`
/// or `%TAG` directive produces its token; any other (reserved) directive is
/// ignored with a warning, producing a `reservedDirective` token whose content
/// the parser drops, rather than libyaml's hard rejection. The reserved token
/// still marks the document as a directives document, so it must be followed by
/// a `---`.
private mutating func scanDirective() throws -> Token {
let startMark = reader.mark
reader.skip()
let name = try scanDirectiveName(startMark: startMark)
if name == "YAML" {
let version = try scanVersionDirectiveValue(startMark: startMark)
let token = Token(type: .versionDirective(major: version.major, minor: version.minor), startMark: startMark, endMark: reader.mark)
try consumeDirectiveTail(startMark: startMark)
return token
}
if name == "TAG" {
let value = try scanTagDirectiveValue(startMark: startMark)
let token = Token(type: .tagDirective(handle: value.handle, prefix: value.prefix), startMark: startMark, endMark: reader.mark)
try consumeDirectiveTail(startMark: startMark)
return token
}
skipReservedDirectiveLine()
return Token(type: .reservedDirective, startMark: startMark, endMark: reader.mark)
}
/// Consume a reserved directive's parameters through the end of the line; the
/// whole line is ignored per YAML 1.2 §6.8.1.
private mutating func skipReservedDirectiveLine() {
while !Classify.isBreakZ(reader.peek()) {
reader.skip()
}
if Classify.isBreak(reader.peek()) {
reader.skipLine()
}
}
/// Eat trailing blanks and an optional comment, require a line break, eat it.
private mutating func consumeDirectiveTail(startMark: Mark) throws {
while Classify.isBlank(reader.peek()) {
reader.skip()
}
if reader.peek() == "#" {
while !Classify.isBreakZ(reader.peek()) {
reader.skip()
}
}
guard Classify.isBreakZ(reader.peek()) else {
throw directiveError("while scanning a directive", "did not find expected comment or line break", startMark)
}
if Classify.isBreak(reader.peek()) {
reader.skipLine()
}
}
/// Port of `yaml_parser_scan_directive_name`: the alphanumeric name after `%`,
/// which must be non-empty and followed by a blank or line break.
private mutating func scanDirectiveName(startMark: Mark) throws -> String {
var name = ""
while let character = reader.peek(), Classify.isAlpha(character) {
name.unicodeScalars.append(character)
reader.skip()
}
guard !name.isEmpty else {
throw directiveError("while scanning a directive", "could not find expected directive name", startMark)
}
guard Classify.isBlankZ(reader.peek()) else {
throw directiveError("while scanning a directive", "found unexpected non-alphabetical character", startMark)
}
return name
}
/// Port of `yaml_parser_scan_version_directive_value`: `major.minor`.
private mutating func scanVersionDirectiveValue(startMark: Mark) throws -> (major: Int, minor: Int) {
while Classify.isBlank(reader.peek()) {
reader.skip()
}
let major = try scanVersionDirectiveNumber(startMark: startMark)
guard reader.peek() == "." else {
throw directiveError("while scanning a %YAML directive", "did not find expected digit or '.' character", startMark)
}
reader.skip()
let minor = try scanVersionDirectiveNumber(startMark: startMark)
// The version must be followed by separation; a comment needs a space
// before its `#`, so `%YAML 1.1#...` (no space) is malformed. libyaml
// accepts it, treating the `#` as an adjacent comment.
guard Classify.isBlankZ(reader.peek()) else {
throw directiveError("while scanning a %YAML directive", "did not find expected digit or '.' character", startMark)
}
return (major, minor)
}
/// Port of `yaml_parser_scan_version_directive_number`: a bounded run of digits.
private mutating func scanVersionDirectiveNumber(startMark: Mark) throws -> Int {
var value = 0
var length = 0
while let character = reader.peek(), Classify.isDigit(character) {
length += 1
guard length <= maxVersionNumberLength else {
throw directiveError("while scanning a %YAML directive", "found extremely long version number", startMark)
}
value = value * 10 + Int(character.value - Unicode.Scalar("0").value)
reader.skip()
}
guard length > 0 else {
throw directiveError("while scanning a %YAML directive", "did not find expected version number", startMark)
}
return value
}
/// Port of `yaml_parser_scan_tag_directive_value`: `!handle! prefix`.
private mutating func scanTagDirectiveValue(startMark: Mark) throws -> (handle: String, prefix: String) {
while Classify.isBlank(reader.peek()) {
reader.skip()
}
let handle = try scanTagHandle(directive: true, startMark: startMark)
guard Classify.isBlank(reader.peek()) else {
throw directiveError("while scanning a %TAG directive", "did not find expected whitespace", startMark)
}
while Classify.isBlank(reader.peek()) {
reader.skip()
}
let prefix = try scanTagURI(uriChar: true, directive: true, head: "", startMark: startMark)
guard Classify.isBlankZ(reader.peek()) else {
throw directiveError("while scanning a %TAG directive", "did not find expected whitespace or line break", startMark)
}
return (handle, prefix)
}
private func directiveError(_ context: String, _ problem: String, _ mark: Mark) -> PureYAML.Parsing.LibYAML.ScannerError {
PureYAML.Parsing.LibYAML.ScannerError(context: context, problem: problem, mark: mark)
}
}

View File

@ -0,0 +1,380 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
private typealias Token = PureYAML.Parsing.LibYAML.Token
private typealias ScannerError = PureYAML.Parsing.LibYAML.ScannerError
/// Produce every token for the input, mirroring repeated calls to libyaml's
/// `yaml_parser_fetch_next_token` until STREAM-END. Tokens are never dequeued,
/// so `tokens_parsed` stays 0 and `QUEUE_INSERT` positions are absolute, which
/// matches the simple-key splice.
mutating func scanAll() throws -> [PureYAML.Parsing.LibYAML.Token] {
while !streamEndProduced {
try fetchNextToken()
}
return indentEngine.tokens
}
/// Port of `yaml_parser_fetch_next_token`: emit STREAM-START first, otherwise
/// skip to the next token, expire stale simple keys, unroll the indentation to
/// the current column, and dispatch on the leading character. Constructs not
/// yet ported throw an explicit error rather than mis-scan.
mutating func fetchNextToken() throws {
if !streamStartProduced {
fetchStreamStart()
return
}
// Gated on `reader.index >= badIndex`, not just "is one set", so a real
// syntax error earlier in the document (which normal scanning would reach
// and throw first) is not hidden behind a control character that only
// occurs later: this only fires once the reader has genuinely scanned up
// to the offending position, matching the order libyaml's own byte-by-byte
// reader would hit them in. `resumePlainScalar`'s own suspend/resume path
// (`scanStep()`, the streaming API) does not route through
// `fetchNextToken()` and so is not covered by this check; a token
// containing a control character can be returned to a streaming caller one
// token before this fires. `scanAll()`/`PureYAML.parse()`, the primary
// target, always calls `fetchNextToken()` again before returning, so this
// still surfaces before a batch parse completes.
if let badIndex = reader.invalidControlCharacterIndex, reader.index >= badIndex {
throw ScannerError(
context: "while scanning for the next token",
problem: "control characters are not allowed",
mark: reader.mark(atIndex: badIndex),
)
}
try scanToNextToken()
try checkFlowContinuationIndent()
mark = reader.mark
try staleSimpleKeys()
indentEngine.unrollIndent(column: reader.column, mark: reader.mark)
mark = reader.mark
try dispatchToken()
}
/// Dispatch on the leading character: stream end, directives and document
/// markers at column 0, then the indicators.
private mutating func dispatchToken() throws {
let character = reader.peek()
if Classify.isZ(character) {
fetchStreamEnd()
return
}
if reader.column == 0, character == "%" {
try fetchDirective()
return
}
if reader.column == 0, isDocumentMarker("-") {
try fetchDocumentIndicator(.documentStart)
return
}
if reader.column == 0, isDocumentMarker(".") {
try fetchDocumentIndicator(.documentEnd)
return
}
guard let character else {
return
}
try dispatchIndicator(character)
}
/// Dispatch the block-structure indicators and plain scalars; unported
/// constructs throw an explicit error.
private mutating func dispatchIndicator(_ character: Unicode.Scalar) throws {
if try fetchFlowToken(character) {
return
}
if try fetchNodeToken(character) {
return
}
switch character {
case "-" where Classify.isBlankZ(reader.peek(1)):
try fetchBlockEntry()
case "-" where flowLevel > 0 && Classify.isFlowIndicator(reader.peek(1)):
// Per `ns-plain-first` a `-` begins a plain scalar in flow only when a
// flow-safe character follows; before a flow indicator (`[-]`, `[-,`)
// it cannot start a node. libyaml reads it as the scalar `-`.
throw ScannerError(
context: "while scanning for the next token",
problem: "found unexpected '-' indicator",
mark: reader.mark,
)
case "?" where Classify.isBlankZ(reader.peek(1)) || (flowLevel > 0 && Classify.isFlowIndicator(reader.peek(1))):
// In flow context libyaml treats every `?` as a key indicator, but per
// `ns-plain-first` a `?` followed by a flow-safe character (anything but
// whitespace, end, or a flow indicator) begins a plain scalar (`?foo`).
try fetchKey()
case ":" where Classify.isBlankZ(reader.peek(1))
|| (flowLevel > 0 && (Classify.isFlowIndicator(reader.peek(1)) || flowKeyIsJSON)):
// In flow context libyaml makes every `:` a value indicator. Per the
// spec a `:` only ends a node when followed by whitespace, end of
// input, or a flow indicator, except an adjacent `:` after a JSON
// (quoted) key. Otherwise the `:` is plain-scalar content (`:x`).
try fetchValue()
default:
// A plain scalar may begin with any non-blank character; a blank (a
// tab here, since spaces and breaks were already consumed) cannot
// start a token, matching libyaml's final guard in fetch_next_token.
guard !Classify.isBlankZ(character) else {
throw ScannerError(
context: "while scanning for the next token",
problem: "found character that cannot start any token",
mark: reader.mark,
)
}
try fetchPlainScalar()
}
}
/// Fetch a flow-collection indicator (`[` `]` `{` `}` `,`); returns false
/// otherwise.
private mutating func fetchFlowToken(_ character: Unicode.Scalar) throws -> Bool {
switch character {
case "[": fetchFlowCollectionStart(.flowSequenceStart)
case "{": fetchFlowCollectionStart(.flowMappingStart)
case "]": try fetchFlowCollectionEnd(.flowSequenceEnd)
case "}": try fetchFlowCollectionEnd(.flowMappingEnd)
case ",": try fetchFlowEntry()
default: return false
}
return true
}
/// Fetch a quoted scalar, an anchor/alias, or a block scalar; returns false
/// otherwise.
private mutating func fetchNodeToken(_ character: Unicode.Scalar) throws -> Bool {
switch character {
case "'": try fetchFlowScalar(single: true)
case "\"": try fetchFlowScalar(single: false)
case "&": try fetchAnchor(isAnchor: true)
case "*": try fetchAnchor(isAnchor: false)
case "!": try fetchTag()
case "|" where flowLevel == 0: try fetchBlockScalar(literal: true)
case ">" where flowLevel == 0: try fetchBlockScalar(literal: false)
default: return false
}
return true
}
/// Port of `yaml_parser_fetch_tag`.
private mutating func fetchTag() throws {
mark = reader.mark
try saveSimpleKey()
simpleKeyAllowed = false
let token = try scanTag()
indentEngine.enqueue(token)
}
/// Port of `yaml_parser_fetch_block_scalar`.
private mutating func fetchBlockScalar(literal: Bool) throws {
try removeSimpleKey()
simpleKeyAllowed = true
let token = try scanBlockScalar(literal: literal)
indentEngine.enqueue(token)
}
/// Port of `yaml_parser_fetch_flow_scalar`.
private mutating func fetchFlowScalar(single: Bool) throws {
mark = reader.mark
let saved = simpleKeyAllowed
try saveSimpleKey()
// A quoted scalar that became this level's simple key is a JSON node, so a
// later adjacent `:` (no space) still opens its value.
if saved {
simpleKeys[simpleKeys.count - 1].isJSON = true
}
simpleKeyAllowed = false
let token = try scanFlowScalar(single: single)
indentEngine.enqueue(token)
}
/// Port of `yaml_parser_fetch_anchor`.
private mutating func fetchAnchor(isAnchor: Bool) throws {
mark = reader.mark
try saveSimpleKey()
simpleKeyAllowed = false
let token = try scanAnchor(isAnchor: isAnchor)
indentEngine.enqueue(token)
}
/// Port of `yaml_parser_fetch_flow_collection_start`.
private mutating func fetchFlowCollectionStart(_ type: PureYAML.Parsing.LibYAML.TokenType) {
// saveSimpleKey cannot throw in flow: a required key needs block context.
let saved = simpleKeyAllowed
try? saveSimpleKey()
// A flow collection is a JSON node, so when it is this level's simple key
// an adjacent `:` after its closing `]`/`}` still opens its value.
if saved {
simpleKeys[simpleKeys.count - 1].isJSON = true
}
increaseFlowLevel(isSequence: type == .flowSequenceStart)
simpleKeyAllowed = true
let startMark = reader.mark
reader.skip()
indentEngine.enqueue(Token(type: type, startMark: startMark, endMark: reader.mark))
}
/// Port of `yaml_parser_fetch_flow_collection_end`.
private mutating func fetchFlowCollectionEnd(_ type: PureYAML.Parsing.LibYAML.TokenType) throws {
try removeSimpleKey()
decreaseFlowLevel()
simpleKeyAllowed = false
let startMark = reader.mark
reader.skip()
indentEngine.enqueue(Token(type: type, startMark: startMark, endMark: reader.mark))
}
/// Port of `yaml_parser_fetch_flow_entry`.
private mutating func fetchFlowEntry() throws {
try removeSimpleKey()
simpleKeyAllowed = true
let startMark = reader.mark
reader.skip()
indentEngine.enqueue(Token(type: .flowEntry, startMark: startMark, endMark: reader.mark))
}
/// Port of `yaml_parser_fetch_stream_start`.
private mutating func fetchStreamStart() {
// indent is already -1 and simpleKeys already holds the block-context slot.
simpleKeyAllowed = true
streamStartProduced = true
indentEngine.enqueue(Token(type: .streamStart, startMark: reader.mark, endMark: reader.mark))
}
/// Port of `yaml_parser_fetch_stream_end`.
private mutating func fetchStreamEnd() {
if reader.column != 0 {
// Force a new line in the mark, as libyaml does.
mark = PureYAML.Parsing.Mark(line: reader.line + 1, column: 0, index: reader.index)
}
indentEngine.unrollIndent(column: -1, mark: mark)
// remove_simple_key cannot throw here: a required key would already have
// been resolved before stream end in the block + plain core.
simpleKeys[simpleKeys.count - 1].possible = false
simpleKeyAllowed = false
streamEndProduced = true
indentEngine.enqueue(Token(type: .streamEnd, startMark: mark, endMark: mark))
}
/// Port of `yaml_parser_fetch_document_indicator`.
private mutating func fetchDocumentIndicator(_ type: PureYAML.Parsing.LibYAML.TokenType) throws {
indentEngine.unrollIndent(column: -1, mark: reader.mark)
simpleKeys[simpleKeys.count - 1].possible = false
simpleKeyAllowed = false
let startMark = reader.mark
reader.skip()
reader.skip()
reader.skip()
indentEngine.enqueue(Token(type: type, startMark: startMark, endMark: reader.mark))
if case .documentEnd = type {
// Only whitespace and a comment may follow `...` on its line; a `--- x`
// document start can carry inline content, but `... x` cannot.
while reader.peek() == " " || Classify.isTab(reader.peek()) {
reader.skip()
}
if let next = reader.peek(), next != "#", !Classify.isBreak(next) {
throw ScannerError(
context: "while scanning a document end marker",
problem: "found content after the document end marker",
mark: reader.mark,
)
}
}
}
/// Port of `yaml_parser_fetch_block_entry`.
private mutating func fetchBlockEntry() throws {
if tabbedSeparation {
throw tabIndentationError()
}
// A block sequence must begin its own line. A `-` mid-line after a mapping
// value (`key: - a`) or a node property (`&a - x`) is invalid, but a `-`
// after another `-` is a valid compact sequence (`- - a`). libyaml accepts
// all three.
if flowLevel == 0, !crossedLineBreak, lastTokenForbidsInlineSequence() {
throw ScannerError(
context: "while scanning a node",
problem: "found a block sequence on the same line as an earlier node",
mark: reader.mark,
)
}
if flowLevel == 0 {
indentEngine.rollIndent(column: reader.column, number: -1, type: .blockSequenceStart, mark: reader.mark)
}
simpleKeys[simpleKeys.count - 1].possible = false
simpleKeyAllowed = true
let startMark = reader.mark
reader.skip()
indentEngine.enqueue(Token(type: .blockEntry, startMark: startMark, endMark: reader.mark))
}
/// Port of `yaml_parser_fetch_key`.
private mutating func fetchKey() throws {
if tabbedSeparation {
throw tabIndentationError()
}
if flowLevel == 0 {
indentEngine.rollIndent(column: reader.column, number: -1, type: .blockMappingStart, mark: reader.mark)
}
simpleKeys[simpleKeys.count - 1].possible = false
simpleKeyAllowed = flowLevel == 0
let startMark = reader.mark
reader.skip()
indentEngine.enqueue(Token(type: .key, startMark: startMark, endMark: reader.mark))
}
/// Port of `yaml_parser_fetch_value`, including the simple-key splice: a
/// possible simple key inserts its `KEY` token at the recorded position and
/// rolls the block mapping indent to the key's column.
private mutating func fetchValue() throws {
let top = simpleKeys.count - 1
// A pending simple key means this `:` closes an inline implicit key (key
// and `:` on one line). An explicit `?` key's `:` has no pending simple key
// (it went stale across the line break), so its value may open a sequence.
lastValueWasInline = simpleKeys[top].possible
// A flow-sequence single-pair implicit key must be single-line, so its `:`
// must sit on the key's line (`[ "k"\n :v ]` and `[ k\n : v ]` are both
// invalid); a flow-mapping adjacent value may span lines (`{ "k"\n :v }`).
if simpleKeys[top].possible, flowIsSequence.last == true, simpleKeys[top].mark.line != mark.line {
throw ScannerError(
context: "while scanning a simple key",
problem: "could not find expected ':'",
mark: simpleKeys[top].mark,
)
}
if simpleKeys[top].possible {
if simpleKeys[top].tabbed {
throw tabIndentationError()
}
let key = simpleKeys[top]
indentEngine.insert(Token(type: .key, startMark: key.mark, endMark: key.mark), atTokenNumber: key.tokenNumber)
indentEngine.rollIndent(column: key.mark.column, number: key.tokenNumber, type: .blockMappingStart, mark: key.mark)
simpleKeys[top].possible = false
simpleKeyAllowed = false
} else {
if flowLevel == 0 {
indentEngine.rollIndent(column: reader.column, number: -1, type: .blockMappingStart, mark: reader.mark)
}
simpleKeyAllowed = flowLevel == 0
}
let startMark = reader.mark
reader.skip()
indentEngine.enqueue(Token(type: .value, startMark: startMark, endMark: reader.mark))
}
/// Port of `yaml_parser_fetch_plain_scalar`.
private mutating func fetchPlainScalar() throws {
mark = reader.mark
try saveSimpleKey()
simpleKeyAllowed = false
let token = try scanPlainScalar()
indentEngine.enqueue(token)
}
private mutating func isDocumentMarker(_ character: Unicode.Scalar) -> Bool {
reader.peek(0) == character
&& reader.peek(1) == character
&& reader.peek(2) == character
&& Classify.isBlankZ(reader.peek(3))
}
}

View File

@ -0,0 +1,225 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
/// The folding state carried through `scan_flow_scalar`, the same shape as the
/// plain-scalar state.
private struct FlowState {
var string = ""
var leadingBreak = ""
var trailingBreaks = ""
var whitespaces = ""
var leadingBlanks = false
}
/// Decoded single-character escapes for double-quoted scalars (libyaml's escape
/// switch). libyaml emits UTF-8 bytes; here each maps to its Unicode scalar.
private static let doubleQuotedEscapes: [Unicode.Scalar: Unicode.Scalar] = [
"0": "\u{0}", "a": "\u{07}", "b": "\u{08}", "t": "\u{09}", "\t": "\u{09}",
"n": "\u{0A}", "v": "\u{0B}", "f": "\u{0C}", "r": "\u{0D}", "e": "\u{1B}",
" ": "\u{20}", "\"": "\"", "/": "/", "\\": "\\",
"N": "\u{85}", "_": "\u{A0}", "L": "\u{2028}", "P": "\u{2029}",
]
/// Port of `yaml_parser_scan_flow_scalar`. Scans a single- or double-quoted
/// scalar, folding line breaks like a plain scalar and, for double quotes,
/// decoding escape sequences.
mutating func scanFlowScalar(single: Bool) throws -> PureYAML.Parsing.LibYAML.Token {
let startMark = reader.mark
reader.skip()
var state = FlowState()
let quote: Unicode.Scalar = single ? "'" : "\""
let indent = indentEngine.indent + 1
// A continuation indented below `indent` is an error, but it is recorded
// and only raised once the quote is known to close; an unterminated quote
// reaches end of input first and reports that instead.
var indentError: PureYAML.Parsing.LibYAML.ScannerError?
while true {
if reader.column == 0, documentMarkerAhead() {
throw flowError("found unexpected document indicator", startMark)
}
if reader.peek() == nil {
throw flowError("found unexpected end of stream", startMark)
}
try consumeFlowContent(single: single, &state, startMark: startMark)
if reader.peek() == quote {
break
}
consumeFlowBreaks(&state, indent: indent, quote: quote, startMark: startMark, error: &indentError)
foldFlow(&state)
}
reader.skip()
if let indentError {
throw indentError
}
return PureYAML.Parsing.LibYAML.Token(
type: .scalar(value: state.string, style: single ? .singleQuoted : .doubleQuoted),
startMark: startMark,
endMark: reader.mark,
)
}
/// Consume the run of non-blank characters, decoding escapes and stopping at
/// the closing quote or an escaped line break.
private mutating func consumeFlowContent(single: Bool, _ state: inout FlowState, startMark: PureYAML.Parsing.Mark) throws {
state.leadingBlanks = false
let quote: Unicode.Scalar = single ? "'" : "\""
while let character = reader.peek(), !Classify.isBlankZ(character) {
if single, character == "'", reader.peek(1) == "'" {
state.string.unicodeScalars.append("'")
reader.skip()
reader.skip()
} else if character == quote {
return
} else if !single, character == "\\", Classify.isBreak(reader.peek(1)) {
reader.skip()
reader.skipLine()
state.leadingBlanks = true
return
} else if !single, character == "\\" {
try appendEscape(&state, startMark: startMark)
} else {
state.string.unicodeScalars.append(character)
reader.skip()
}
}
}
/// Decode a double-quoted escape sequence (`\` already at the reader).
private mutating func appendEscape(_ state: inout FlowState, startMark: PureYAML.Parsing.Mark) throws {
let escape = reader.peek(1)
if let escape, let decoded = Self.doubleQuotedEscapes[escape] {
state.string.unicodeScalars.append(decoded)
reader.skip()
reader.skip()
return
}
let codeLength = try hexEscapeLength(escape, startMark: startMark)
reader.skip()
reader.skip()
var value = try readHexEscape(count: codeLength, startMark: startMark)
if codeLength == 4 {
value = try combineSurrogateIfNeeded(high: value, startMark: startMark)
}
guard let scalar = Unicode.Scalar(value) else {
throw flowError("found invalid Unicode character escape code", startMark)
}
state.string.unicodeScalars.append(scalar)
}
/// The hex digit count for `\x` (2), `\u` (4), and `\U` (8) escapes.
private func hexEscapeLength(_ escape: Unicode.Scalar?, startMark: PureYAML.Parsing.Mark) throws -> Int {
switch escape {
case "x": 2
case "u": 4
case "U": 8
default: throw flowError("found unknown escape character", startMark)
}
}
/// Read and consume `count` hexadecimal digits, returning their value.
private mutating func readHexEscape(count: Int, startMark: PureYAML.Parsing.Mark) throws -> UInt32 {
var value: UInt32 = 0
for _ in 0 ..< count {
guard let digit = reader.peek(), Classify.isHex(digit) else {
throw flowError("did not find expected hexdecimal number", startMark)
}
value = (value << 4) + hexValue(digit)
reader.skip()
}
return value
}
/// Combine a UTF-16 surrogate pair (a high surrogate `\u` escape immediately
/// followed by a low surrogate `\u` escape) into a single astral code point,
/// for JSON compatibility. A lone high surrogate is left for the scalar check.
private mutating func combineSurrogateIfNeeded(high: UInt32, startMark: PureYAML.Parsing.Mark) throws -> UInt32 {
guard (0xD800 ... 0xDBFF).contains(high), reader.peek() == "\\", reader.peek(1) == "u" else {
return high
}
reader.skip()
reader.skip()
let low = try readHexEscape(count: 4, startMark: startMark)
guard (0xDC00 ... 0xDFFF).contains(low) else {
throw flowError("found invalid Unicode character escape code", startMark)
}
return 0x10000 + ((high - 0xD800) << 10) + (low - 0xDC00)
}
/// Consume blanks and line breaks between content, recording pending
/// whitespace and folded breaks.
private mutating func consumeFlowBreaks(
_ state: inout FlowState,
indent: Int,
quote: Unicode.Scalar,
startMark: PureYAML.Parsing.Mark,
error: inout PureYAML.Parsing.LibYAML.ScannerError?,
) {
while Classify.isBlank(reader.peek()) || Classify.isBreak(reader.peek()) {
if Classify.isBlank(reader.peek()) {
if error == nil, state.leadingBlanks, reader.column < indent, Classify.isTab(reader.peek()), !blankLineAhead() {
error = flowError("found a tab character that violates indentation", startMark)
}
if !state.leadingBlanks, let blank = reader.peek() {
state.whitespaces.unicodeScalars.append(blank)
}
reader.skip()
} else if !state.leadingBlanks {
state.whitespaces = ""
reader.skipLine()
state.leadingBreak = "\n"
state.leadingBlanks = true
} else {
reader.skipLine()
state.trailingBreaks += "\n"
}
}
// The continuation content must be indented at least `indent`; a less
// indented (non-closing, non-empty) line is malformed.
if error == nil, state.leadingBlanks, let next = reader.peek(), next != quote, !Classify.isBreakZ(next), reader.column < indent {
error = flowError("found wrong indentation in a multi-line quoted scalar", startMark)
}
}
/// Fold the pending breaks or whitespace into the value.
private func foldFlow(_ state: inout FlowState) {
guard state.leadingBlanks else {
state.string += state.whitespaces
state.whitespaces = ""
return
}
if state.leadingBreak.first == "\n" {
if state.trailingBreaks.isEmpty {
state.string += " "
} else {
state.string += state.trailingBreaks
state.trailingBreaks = ""
}
state.leadingBreak = ""
} else {
state.string += state.leadingBreak
state.string += state.trailingBreaks
state.leadingBreak = ""
state.trailingBreaks = ""
}
}
private func hexValue(_ scalar: Unicode.Scalar) -> UInt32 {
switch scalar {
case "0" ... "9": scalar.value - Unicode.Scalar("0").value
case "A" ... "F": scalar.value - Unicode.Scalar("A").value + 10
default: scalar.value - Unicode.Scalar("a").value + 10
}
}
private func flowError(_ problem: String, _ mark: PureYAML.Parsing.Mark) -> PureYAML.Parsing.LibYAML.ScannerError {
PureYAML.Parsing.LibYAML.ScannerError(context: "while scanning a quoted scalar", problem: problem, mark: mark)
}
private mutating func documentMarkerAhead() -> Bool {
let first = reader.peek()
guard first == "-" || first == "." else {
return false
}
return reader.peek(1) == first && reader.peek(2) == first && Classify.isBlankZ(reader.peek(3))
}
}

View File

@ -0,0 +1,59 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias ScannerError = PureYAML.Parsing.LibYAML.ScannerError
/// Reject a flow-collection continuation line that is not indented past the
/// enclosing block context. YAML requires `s-flow-line-prefix` to be more
/// indented than the parent block; libyaml does not enforce it, so multi-line
/// flow content in column 0 (`[a,\nb]` under a key) wrongly parses.
mutating func checkFlowContinuationIndent() throws {
guard flowLevel > 0, crossedLineBreak, let next = reader.peek() else {
return
}
// A closing `]`/`}` (or a `,` separator) may sit at any indentation; only
// flow CONTENT must be indented past the parent block.
if next == "]" || next == "}" || next == "," {
return
}
// A tab in the leading whitespace is never valid indentation, even where
// spaces of the same width would be (the tab inflates the column).
if reader.column <= indentEngine.indent || flowLeadingTab {
throw ScannerError(
context: "while parsing a flow node",
problem: "found insufficiently indented line in a flow collection",
mark: reader.mark,
)
}
}
/// Whether the last emitted token bars a block sequence from opening on the
/// same line: a mapping value indicator or a node property (anchor/tag). After
/// another block entry a compact sequence is allowed, so that is not barred.
func lastTokenForbidsInlineSequence() -> Bool {
switch indentEngine.lastToken?.type {
case .anchor, .tag:
true
case .value:
lastValueWasInline
default:
false
}
}
/// The error raised when a tab is used where indentation is required (before a
/// block indicator or an implicit mapping key). A tab is valid separation but
/// never indentation.
func tabIndentationError() -> PureYAML.Parsing.LibYAML.ScannerError {
ScannerError(
context: "while scanning for the next token",
problem: "found a tab character where an indentation space is expected",
mark: reader.mark,
)
}
/// Whether this flow level's pending simple key is a JSON (quoted) node, which
/// lets an adjacent `:` open its value without a separating space.
var flowKeyIsJSON: Bool {
let key = simpleKeys[simpleKeys.count - 1]
return key.possible && key.isJSON
}
}

View File

@ -0,0 +1,323 @@
extension PureYAML.Parsing.LibYAML.Scanner {
/// Which step of `runPlainScalarLoop`'s outer iteration a suspend happened
/// at. Resuming must land back on the step that was actually interrupted,
/// not on whatever step normally follows it, and in particular must never
/// redo a decision (document marker, comment, or "does a blank/break run
/// follow") that a *previous, already-passed* check already answered:
/// `.start`'s document-marker/`#` checks and `.afterContent`'s blank/break
/// guard are only meant to run once, at a genuine entry point, not at an
/// arbitrary position inside an already-in-progress content or break run.
/// A first version of this type resumed a dry `consumePlainContent` via
/// `.start` on the (wrong) assumption that redoing its checks was harmless;
/// it is not; if the resume position happens to land on a literal `#`
/// that is legitimate mid-scalar content (e.g. a value containing every
/// printable character), `.start`'s comment check wrongly ends the scalar
/// right there. `consumePlainContent` and `consumePlainBreaks` each get
/// their own dedicated "continue this same call, no pre-check" case
/// (`.consumingContent`, `.consumingBreaks`) for exactly this reason.
enum PlainScalarPhase: Equatable {
/// Check for a document marker or comment, then consume content.
case start
/// Inside consuming an already-started content run (the document
/// marker/comment checks that gated entry into it do not apply here).
case consumingContent
/// Content just finished; decide whether a blank/break run follows.
case afterContent
/// Inside consuming an already-confirmed blank/break run.
case consumingBreaks
/// A blank/break run just finished; check whether the scalar dedents.
case afterBreaks
}
/// Thrown by `runPlainScalarLoop` when accumulating a plain scalar's content
/// runs dry (`Reader.ranDry`) while `!Reader.isFinal`. Carries everything
/// `resumePlainScalar(_:)` needs to continue from exactly this point once more
/// input is appended, rather than rescanning the scalar's content from its
/// start. `scanStep()` catches this specifically and stores it on
/// `Scanner.suspendedPlainScalar`, distinct from a generic dry-run catch: no
/// checkpoint restore happens for this error, since the whole point is to keep
/// the reader position and accumulated content rather than discard them.
struct PlainScalarSuspended: Swift.Error {
var state: PlainState
var indent: Int
var startMark: PureYAML.Parsing.Mark
var phase: PlainScalarPhase
}
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
/// The folding state carried through `scan_plain_scalar`: the accumulated
/// value plus the pending leading break, trailing breaks, and whitespace that
/// are folded in before the next content character. Internal rather than
/// private: `PlainScalarSuspended` carries one across a `scanStep()` suspend
/// point, so `Scanner`'s own file needs to name the type for its stored
/// `suspendedPlainScalar` property.
struct PlainState {
var string = ""
var leadingBreak = ""
var trailingBreaks = ""
var whitespaces = ""
var leadingBlanks = false
var endMark: PureYAML.Parsing.Mark
}
/// Port of `yaml_parser_scan_plain_scalar`. Consumes a plain scalar, folding
/// line breaks: a single break between non-empty lines becomes a space, blank
/// lines become literal newlines, and leading/trailing whitespace on folded
/// lines is dropped. Ends at a `: ` value indicator, a comment, a less-indented
/// line (block context), a flow indicator (flow context), or a document marker.
mutating func scanPlainScalar() throws -> PureYAML.Parsing.LibYAML.Token {
let indent = indentEngine.indent + 1
let startMark = reader.mark
var state = PlainState(endMark: reader.mark)
try runPlainScalarLoop(&state, indent: indent, startMark: startMark)
return finishPlainScalar(state, startMark: startMark)
}
/// Resumes a plain scalar suspended by a previous `PlainScalarSuspended`,
/// continuing from its saved state, phase, and reader position rather than
/// rescanning the content already accumulated before the suspend.
///
/// Takes `state` `inout` rather than the whole `PlainScalarSuspended`, on
/// purpose: the caller (`scanStep()`) must disalias `state` from the
/// suspended value it came from *before* this call, since a `let
/// suspended = ...; resumePlainScalar(suspended)` call would keep the
/// caller's own `suspended.state` reference to the same string buffer
/// alive for this whole call's duration regardless of anything done
/// inside this function's own body (a value parameter is borrowed for
/// the call, not owned, so a local disalias inside the callee cannot
/// remove a reference the caller itself is still holding). With `state`
/// passed by direct reference and no wrapping struct crossing this call
/// boundary at all, there is exactly one owner of its string buffer
/// throughout, so appending inside the loop below never triggers a
/// spurious copy-on-write copy of the whole accumulated scalar. See
/// `scanStep()`'s resume branch for where the disalias happens.
mutating func resumePlainScalar(
_ state: inout PlainState,
indent: Int,
startMark: PureYAML.Parsing.Mark,
phase: PlainScalarPhase,
) throws -> PureYAML.Parsing.LibYAML.Token {
try runPlainScalarLoop(&state, indent: indent, startMark: startMark, from: phase)
return finishPlainScalar(state, startMark: startMark)
}
/// The scan loop shared by a fresh attempt and a resume, restructured as an
/// explicit `PlainScalarPhase` state machine (see its own doc comment for
/// why) with a `suspendIfDry` check after every step that peeked. Checking
/// only once, after the whole loop exits, is not enough: a dry read
/// (`reader.ranDry`, see `Reader.peek(_:)`) inside a lookahead
/// (`documentMarkerAhead()`, `consumePlainContent`'s `:`-followed-by-blank
/// check, `consumePlainBreaks`'s `blankLineAhead()` guard) reads as "blank or
/// end" (`CharacterClass.isBlankZ`/`isBreakZ` treat a missing scalar as
/// true), not as "not enough data yet", so the *next* step can reach a
/// wrong-but-plausible decision (e.g. `documentMarkerAhead` wrongly says
/// "not a marker", so `consumePlainContent` appends `---` as scalar content)
/// before a single end-of-loop check would ever see the taint that caused
/// it. Checking right after each peeking step, before its result is used for
/// anything further, catches the taint before it can influence another
/// decision. Batch parsing (`isFinal: true`) is unaffected: every check can
/// only throw when `!reader.isFinal`, which batch callers never set, so the
/// loop's own logic is otherwise identical to the original.
///
/// `state` and the reader position are always resumable at a suspend point:
/// nothing here calls `reader.skip()` on a dry `nil`, so accumulated content
/// and position both stop exactly at the last successfully read scalar.
private mutating func runPlainScalarLoop(
_ state: inout PlainState,
indent: Int,
startMark: PureYAML.Parsing.Mark,
from initialPhase: PlainScalarPhase = .start,
) throws {
var phase = initialPhase
while true {
switch phase {
case .start:
let atDocumentMarker = reader.column == 0 && documentMarkerAhead()
// A dry lookahead inside `documentMarkerAhead()` must resume by
// redoing this whole step, not by trusting `atDocumentMarker`
// (which `isBlankZ`'s nil-is-true bias could have gotten wrong)
// and moving on: hence `phase: .start` here, not `.consumingContent`.
try suspendIfDry(state, indent: indent, startMark: startMark, phase: .start)
if atDocumentMarker {
return
}
if reader.peek() == "#" {
return
}
try consumePlainContent(&state, startMark: startMark)
// consumePlainContent stopping is not yet known to be a real
// answer (terminator found) versus a dry cutoff: check before
// trusting it enough to move to `.afterContent`'s guard. On dry,
// resume via `.consumingContent`, not `.start`: the document-marker
// and `#` checks above only apply at a genuine entry point, and
// the reader position where a content run was interrupted can
// land on a literal `#` or `-` that is legitimate content, not a
// fresh comment or marker.
try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingContent)
phase = .afterContent
case .consumingContent:
try consumePlainContent(&state, startMark: startMark)
try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingContent)
phase = .afterContent
case .afterContent:
// Reached only once consumePlainContent is known to have stopped
// for a real reason, so this peek cannot itself be dry: it
// re-reads the exact position consumePlainContent's own last
// (already non-dry) peek confirmed, and nothing advanced between.
guard Classify.isBlank(reader.peek()) || Classify.isBreak(reader.peek()) else {
return
}
try consumePlainBreaks(&state, indent: indent, startMark: startMark)
// Same reasoning as consumePlainContent above, but resuming here
// must not redo `.afterContent`'s guard against new data: that
// guard gates *entry* into a blank/break run, and re-asking it
// after the run is already underway would wrongly treat "the run
// continues" as "the run never started" (or vice versa) instead
// of just continuing the same run. `.consumingBreaks` re-enters
// consumePlainBreaks directly.
try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingBreaks)
phase = .afterBreaks
case .consumingBreaks:
try consumePlainBreaks(&state, indent: indent, startMark: startMark)
try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingBreaks)
phase = .afterBreaks
case .afterBreaks:
// `reader.column` is a plain stored field, never read via
// `peek(_:)`, so this check cannot itself observe a dry read; by
// construction every path reaching this case already confirmed
// `!ranDry` before transitioning here.
if flowLevel == 0, reader.column < indent {
return
}
phase = .start
}
}
}
private func suspendIfDry(
_ state: PlainState,
indent: Int,
startMark: PureYAML.Parsing.Mark,
phase: PlainScalarPhase,
) throws {
if !reader.isFinal, reader.ranDry {
throw PlainScalarSuspended(state: state, indent: indent, startMark: startMark, phase: phase)
}
}
private mutating func finishPlainScalar(
_ state: PlainState,
startMark: PureYAML.Parsing.Mark,
) -> PureYAML.Parsing.LibYAML.Token {
if state.leadingBlanks {
simpleKeyAllowed = true
}
return PureYAML.Parsing.LibYAML.Token(
type: .scalar(value: state.string, style: .plain),
startMark: startMark,
endMark: state.endMark,
)
}
/// Consume the run of non-blank characters, folding any pending breaks or
/// whitespace ahead of each one.
private mutating func consumePlainContent(_ state: inout PlainState, startMark _: PureYAML.Parsing.Mark) throws {
while let character = reader.peek(), !Classify.isBlankZ(character) {
if character == ":", Classify.isBlankZ(reader.peek(1)) {
break
}
if flowLevel > 0, character == ":", Classify.isFlowIndicator(reader.peek(1)) {
// A `:` before a flow indicator ends the key; the value is empty.
// libyaml rejects this as an unexpected ':', but it is a valid
// value indicator in flow context.
break
}
if flowLevel > 0, Classify.isFlowIndicator(character) {
break
}
foldPending(&state)
state.string.unicodeScalars.append(character)
reader.skip()
state.endMark = reader.mark
}
}
/// Fold the pending leading break, trailing breaks, and whitespace into the
/// accumulated value (the join logic from libyaml's inner loop).
private func foldPending(_ state: inout PlainState) {
guard state.leadingBlanks || !state.whitespaces.isEmpty else {
return
}
guard state.leadingBlanks else {
state.string += state.whitespaces
state.whitespaces = ""
return
}
if state.leadingBreak.first == "\n" {
if state.trailingBreaks.isEmpty {
state.string += " "
} else {
state.string += state.trailingBreaks
state.trailingBreaks = ""
}
state.leadingBreak = ""
} else {
state.string += state.leadingBreak
state.string += state.trailingBreaks
state.leadingBreak = ""
state.trailingBreaks = ""
}
state.leadingBlanks = false
}
/// Consume the run of blanks and line breaks between content, recording them as
/// pending whitespace or folded breaks.
private mutating func consumePlainBreaks(_ state: inout PlainState, indent: Int, startMark: PureYAML.Parsing.Mark) throws {
while Classify.isBlank(reader.peek()) || Classify.isBreak(reader.peek()) {
if Classify.isBlank(reader.peek()) {
if state.leadingBlanks, reader.column < indent, Classify.isTab(reader.peek()) {
// `blankLineAhead()` is its own multi-character lookahead (not
// just the single already-confirmed-real `peek()` above), so it
// can independently run dry: check immediately, before the
// tab-violation decision below trusts its answer. Unchecked,
// `isBreakZ`'s nil-is-true bias makes an incomplete lookahead
// read as "blank line", suppressing a real violation until more
// data arrives proves otherwise.
let onBlankLine = blankLineAhead()
try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingBreaks)
if !onBlankLine {
// A tab below the indentation is illegal only when it indents
// real content; on an otherwise-blank line it is just a blank
// line the scalar folds over. libyaml rejects it either way.
throw scannerError("found a tab character that violates indentation", at: startMark)
}
}
if !state.leadingBlanks, let blank = reader.peek() {
state.whitespaces.unicodeScalars.append(blank)
}
reader.skip()
} else if !state.leadingBlanks {
state.whitespaces = ""
reader.skipLine()
state.leadingBreak = "\n"
state.leadingBlanks = true
} else {
reader.skipLine()
state.trailingBreaks += "\n"
}
}
}
private func scannerError(_ problem: String, at mark: PureYAML.Parsing.Mark) -> PureYAML.Parsing.LibYAML.ScannerError {
PureYAML.Parsing.LibYAML.ScannerError(context: "while scanning a plain scalar", problem: problem, mark: mark)
}
private mutating func documentMarkerAhead() -> Bool {
let first = reader.peek()
guard first == "-" || first == "." else {
return false
}
return reader.peek(1) == first && reader.peek(2) == first && Classify.isBlankZ(reader.peek(3))
}
}

View File

@ -0,0 +1,88 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
private typealias ScannerError = PureYAML.Parsing.LibYAML.ScannerError
/// Port of `yaml_parser_scan_to_next_token`: skip a leading BOM, spaces and
/// tabs, a comment, and a single line break, which re-allows a simple key in
/// the block context. A tab is valid in-line separation, so it is skipped
/// here; `tabbedSeparation` records when one preceded the token so a block
/// indicator or implicit key (where the tab would be indentation) can reject.
mutating func scanToNextToken() throws {
crossedLineBreak = false
flowLeadingTab = false
while true {
tabbedSeparation = false
if reader.column == 0, reader.peek() == "\u{FEFF}" {
reader.skip()
}
while reader.peek() == " " || Classify.isTab(reader.peek()) {
if Classify.isTab(reader.peek()) {
noteLeadingTab()
}
reader.skip()
}
// YAML 1.2: a line containing only whitespace is blank even when that
// whitespace includes tabs. libyaml leaves a leading tab in place and
// later rejects it; instead consume the trailing blanks so the line
// break below is reached and the blank line is skipped.
if blankLineAhead() {
while reader.peek() == " " || Classify.isTab(reader.peek()) {
reader.skip()
}
}
if reader.peek() == "#" {
try consumeComment()
}
if Classify.isBreak(reader.peek()) {
reader.skipLine()
crossedLineBreak = true
if flowLevel == 0 {
simpleKeyAllowed = true
}
} else {
break
}
}
}
/// Record what a tab in the leading whitespace implies. In block context with a
/// simple key allowed it is potential indentation before an indicator. In flow
/// context, below the continuation indent on a non-blank line it is indentation
/// abuse (a tab is never indentation); at or above the indent it is separation.
private mutating func noteLeadingTab() {
if flowLevel == 0, simpleKeyAllowed {
tabbedSeparation = true
}
if flowLevel > 0, crossedLineBreak, reader.column < indentEngine.indent + 1, !blankLineAhead() {
flowLeadingTab = true
}
}
/// Consume a comment to the end of the line. A comment must be preceded by
/// whitespace or start the line; a `#` adjacent to the previous token (e.g.
/// `"x"#`, `]#`, `,#`) is not a comment. The plain-scalar scanner consumes its
/// own trailing blanks, so the character just before the `#` is the reliable
/// signal. libyaml accepts the adjacent form.
private mutating func consumeComment() throws {
if reader.column > 0, let previous = reader.peek(-1), !Classify.isBlank(previous) {
throw ScannerError(
context: "while scanning for the next token",
problem: "found a comment without a preceding space",
mark: reader.mark,
)
}
while !Classify.isBreakZ(reader.peek()) {
reader.skip()
}
}
/// Whether the rest of the current line, from the current position, is only
/// spaces and tabs up to a line break or end of stream (a blank line).
mutating func blankLineAhead() -> Bool {
var offset = 0
while let character = reader.peek(offset), character == " " || Classify.isTab(character) {
offset += 1
}
return Classify.isBreakZ(reader.peek(offset))
}
}

View File

@ -0,0 +1,165 @@
extension PureYAML.Parsing.LibYAML.Scanner {
/// The result of one `scanStep()` call.
enum ScanOutcome: Equatable {
/// A token is safe to release to the caller now.
case token(PureYAML.Parsing.LibYAML.Token)
/// The buffered input was exhausted mid-fetch; append more input (via
/// `reader.append(_:)`) and call `scanStep()` again. The scanner's state is
/// unchanged from before this call, as if it had not been called.
case needMoreInput
/// STREAM-END has been released and no further tokens remain.
case done
}
/// Resumable, one-token-at-a-time counterpart to `scanAll()`. Where `scanAll()`
/// assumes the full input is already buffered, `scanStep()` may be driven across
/// a `Reader` still receiving chunks (`isFinal == false`): a fetch that runs off
/// the buffered end is rolled back in full (as if it had not been attempted)
/// rather than mistaken for true end of input.
///
/// A token already queued is not always safe to hand out immediately: libyaml's
/// simple-key mechanism can still splice a KEY (and a BLOCK-MAPPING-START) in
/// front of it once a later `:` is found (`saveSimpleKey`/`insert(_:atTokenNumber:)`
/// in `LibYAML.Scanner.swift`/`LibYAML.IndentEngine.swift`). Port of libyaml's
/// `yaml_parser_fetch_more_tokens` release condition: the token at the queue
/// front (position `tokensParsed`) is safe to release only when no currently
/// possible simple key (at any flow level) is still recorded at that exact
/// position; such a key could still resolve into a splice targeting that slot.
mutating func scanStep() throws -> ScanOutcome {
while true {
if isFrontSafeToRelease, let token = indentEngine.dequeue() {
return .token(token)
}
if streamEndProduced {
return .done
}
// A plain scalar in progress resumes from exactly where it stopped,
// with no checkpoint at all: `runPlainScalarLoop` never leaves `reader`
// or `state` in a state that needs rolling back (see
// `PlainScalarSuspended`), so there is nothing to restore if it runs
// dry again.
switch try resumeSuspendedPlainScalarIfNeeded() {
case .notSuspended:
break
case .tokenReady:
continue
case .stillDry:
return .needMoreInput
}
if let outcome = try fetchNextTokenCheckpointed() {
return outcome
}
}
}
/// A dry run can only ever be reported while `!reader.isFinal` (see
/// `Reader.peek(_:)`), so once the caller has appended every chunk and
/// called `markFinal()`, no fetch can ever need rolling back. Skipping
/// the snapshot in that case matters: it is the entire cost difference
/// between `scanStep()` and `scanAll()` once the whole input is already
/// buffered (measured in StreamingScannerPerformanceTests), since neither
/// reads nor mutations inside `fetchNextToken()` become any cheaper by
/// being wrapped in an unused checkpoint. Returns `nil` to mean "keep
/// looping in `scanStep()`", matching a fetch that produced no directly
/// returnable outcome (a non-plain-scalar token was merely enqueued, or a
/// document/directive token dispatched with no scalar involved).
private mutating func fetchNextTokenCheckpointed() throws -> ScanOutcome? {
guard !reader.isFinal else {
try fetchNextToken()
return nil
}
let checkpoint = self
do {
try fetchNextToken()
} catch let suspended as PlainScalarSuspended {
// `fetchPlainScalar()` calls `saveSimpleKey()` and sets
// `simpleKeyAllowed = false` before ever calling `scanPlainScalar()`,
// and a plain-scalar-only resume has no way to undo those: it is
// only safe to keep them applied and resume past this suspend if
// real progress (content) exists to preserve. `phase == .start`
// with an empty `state` means the very first check inside the loop
// (before `consumePlainContent` ever ran, e.g. `documentMarkerAhead()`'s
// own lookahead went dry) suspended: nothing would be saved by
// resuming instead of retrying from scratch, and retrying from
// scratch is what correctly re-derives whether this token should
// have been dispatched as a plain scalar at all. `dispatchToken()`'s
// own `isDocumentMarker` check, called before `fetchPlainScalar()`
// is ever reached, can suffer the exact same dry-lookahead problem
// `runPlainScalarLoop` guards against; only a full retry re-checks it
// with more data, since nothing here can undo a wrong dispatch.
if suspended.phase == .start, suspended.state.string.isEmpty {
self = checkpoint
return .needMoreInput
}
suspendedPlainScalar = suspended
return .needMoreInput
} catch {
if reader.ranDry {
self = checkpoint
return .needMoreInput
}
throw error
}
if reader.ranDry {
self = checkpoint
return .needMoreInput
}
return nil
}
private var isFrontSafeToRelease: Bool {
guard !indentEngine.tokens.isEmpty else {
return false
}
return !simpleKeys.contains { $0.possible && $0.tokenNumber == indentEngine.tokensParsed }
}
private enum ResumeOutcome {
/// No plain scalar was suspended; `scanStep()` should fall through
/// to its normal fetch path.
case notSuspended
/// The suspended scalar was resumed to completion and its token was
/// enqueued; `scanStep()` should loop back to the top and try to
/// release it, not fall through to a fresh fetch.
case tokenReady
/// The resume itself ran dry again; `scanStep()` should report
/// `.needMoreInput`.
case stillDry
}
/// Extracted out of `scanStep()` (which was tripping the cyclomatic
/// complexity limit) rather than inlined, this also fixes a real O(N^2)
/// bug the inline version had: extracting `state` and immediately
/// overwriting `suspended.state` with a fresh, unrelated value before
/// calling `resumePlainScalar` is required, not optional. A plain
/// `resumePlainScalar(suspended)` call, passing the whole suspended
/// value, would keep a reference to it alive (as an ordinary borrowed
/// value parameter) for this call's entire duration regardless of what
/// the callee does internally, leaving `state`'s string buffer with two
/// live owners throughout `runPlainScalarLoop` and forcing copy-on-write
/// to fully copy the accumulated scalar on every single append.
/// Overwriting `suspended.state` here, before the call, drops that
/// reference for good, so `state` is the sole owner by the time
/// `resumePlainScalar` starts mutating it (see
/// `StreamingScannerPerformanceTests.test_measureLongPlainScalar`, which
/// caught this at a measured 29x versus the ~3-4x this fix gives).
private mutating func resumeSuspendedPlainScalarIfNeeded() throws -> ResumeOutcome {
guard var suspended = suspendedPlainScalar else {
return .notSuspended
}
suspendedPlainScalar = nil
var state = suspended.state
suspended.state = PlainState(endMark: state.endMark)
let indent = suspended.indent
let startMark = suspended.startMark
let phase = suspended.phase
do {
let token = try resumePlainScalar(&state, indent: indent, startMark: startMark, phase: phase)
indentEngine.enqueue(token)
return .tokenReady
} catch let resuspended as PlainScalarSuspended {
suspendedPlainScalar = resuspended
return .stillDry
}
}
}

View File

@ -0,0 +1,163 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
/// Port of `yaml_parser_scan_tag`. Scans a tag in verbatim `!<uri>`, shorthand
/// `!handle!suffix`, or `!suffix` form, producing the handle and suffix; the
/// parser resolves them against the tag directives.
mutating func scanTag() throws -> PureYAML.Parsing.LibYAML.Token {
let startMark = reader.mark
var handle = ""
var suffix = ""
if reader.peek(1) == "<" {
reader.skip()
reader.skip()
suffix = try scanTagURI(uriChar: true, directive: false, head: "", startMark: startMark)
guard reader.peek() == ">" else {
throw tagError("did not find the expected '>'", startMark)
}
reader.skip()
} else {
handle = try scanTagHandle(directive: false, startMark: startMark)
if handle.first == "!", handle.count > 1, handle.last == "!" {
suffix = try scanTagURI(uriChar: false, directive: false, head: "", startMark: startMark)
} else {
suffix = try scanTagURI(uriChar: false, directive: false, head: handle, startMark: startMark)
handle = "!"
if suffix.isEmpty {
swap(&handle, &suffix)
}
}
}
guard Classify.isBlankZ(reader.peek()) || (flowLevel > 0 && reader.peek() == ",") else {
throw tagError("did not find expected whitespace or line break", startMark)
}
return PureYAML.Parsing.LibYAML.Token(type: .tag(handle: handle, suffix: suffix), startMark: startMark, endMark: reader.mark)
}
/// Port of `yaml_parser_scan_tag_handle`. A handle is `!`, optionally followed
/// by alphanumeric characters and a closing `!`.
mutating func scanTagHandle(directive: Bool, startMark: PureYAML.Parsing.Mark) throws -> String {
guard reader.peek() == "!" else {
throw tagError("did not find expected '!'", startMark)
}
var handle = "!"
reader.skip()
while let character = reader.peek(), Classify.isAlpha(character) {
handle.unicodeScalars.append(character)
reader.skip()
}
if reader.peek() == "!" {
handle.unicodeScalars.append("!")
reader.skip()
} else if directive, handle != "!" {
throw tagError("did not find expected '!'", startMark)
}
return handle
}
/// Port of `yaml_parser_scan_tag_uri`. Consumes the URI character set (plus the
/// flow indicators inside a verbatim tag), decoding `%`-escapes.
mutating func scanTagURI(uriChar: Bool, directive: Bool, head: String, startMark: PureYAML.Parsing.Mark) throws -> String {
var string = head.count > 1 ? String(head.dropFirst()) : ""
var length = head.isEmpty ? 0 : head.count
while let character = reader.peek(), isURICharacter(character, uriChar: uriChar) {
if character == "%" {
try scanURIEscapes(directive: directive, startMark: startMark, into: &string)
} else {
string.unicodeScalars.append(character)
reader.skip()
}
length += 1
}
guard length > 0 else {
throw tagError("did not find expected tag URI", startMark)
}
return string
}
/// Port of `yaml_parser_scan_uri_escapes`. Decodes one UTF-8 character from a
/// run of `%XX` octets.
private mutating func scanURIEscapes(directive _: Bool, startMark: PureYAML.Parsing.Mark, into string: inout String) throws {
var bytes: [UInt8] = []
var width = 0
repeat {
guard reader.peek() == "%", let high = reader.peek(1), Classify.isHex(high), let low = reader.peek(2), Classify.isHex(low) else {
throw tagError("did not find URI escaped octet", startMark)
}
let octet = UInt8((hexDigit(high) << 4) + hexDigit(low))
if width == 0 {
width = utf8Width(of: octet)
guard width > 0 else {
throw tagError("found an incorrect leading UTF-8 octet", startMark)
}
} else if octet & 0xC0 != 0x80 {
throw tagError("found an incorrect trailing UTF-8 octet", startMark)
}
bytes.append(octet)
reader.skip()
reader.skip()
reader.skip()
width -= 1
} while width > 0
guard let scalar = Unicode.Scalar(decodeUTF8(bytes)) else {
throw tagError("found invalid UTF-8 in a URI escape", startMark)
}
string.unicodeScalars.append(scalar)
}
/// Assemble a code point from the validated 1-to-4 UTF-8 octets of one escape.
private func decodeUTF8(_ bytes: [UInt8]) -> UInt32 {
switch bytes.count {
case 1:
UInt32(bytes[0])
case 2:
(UInt32(bytes[0] & 0x1F) << 6) | UInt32(bytes[1] & 0x3F)
case 3:
(UInt32(bytes[0] & 0x0F) << 12) | (UInt32(bytes[1] & 0x3F) << 6) | UInt32(bytes[2] & 0x3F)
default:
(UInt32(bytes[0] & 0x07) << 18) | (UInt32(bytes[1] & 0x3F) << 12) | (UInt32(bytes[2] & 0x3F) << 6) | UInt32(bytes[3] & 0x3F)
}
}
private func utf8Width(of octet: UInt8) -> Int {
if octet & 0x80 == 0x00 {
return 1
}
if octet & 0xE0 == 0xC0 {
return 2
}
if octet & 0xF0 == 0xE0 {
return 3
}
if octet & 0xF8 == 0xF0 {
return 4
}
return 0
}
private func isURICharacter(_ scalar: Unicode.Scalar, uriChar: Bool) -> Bool {
if Classify.isAlpha(scalar) {
return true
}
switch scalar {
case ";", "/", "?", ":", "@", "&", "=", "+", "$", ".", "%", "!", "~", "*", "'", "(", ")":
return true
case ",", "[", "]":
return uriChar
default:
return false
}
}
private func hexDigit(_ scalar: Unicode.Scalar) -> Int {
switch scalar {
case "0" ... "9": Int(scalar.value - Unicode.Scalar("0").value)
case "A" ... "F": Int(scalar.value - Unicode.Scalar("A").value) + 10
default: Int(scalar.value - Unicode.Scalar("a").value) + 10
}
}
private func tagError(_ problem: String, _ mark: PureYAML.Parsing.Mark) -> PureYAML.Parsing.LibYAML.ScannerError {
PureYAML.Parsing.LibYAML.ScannerError(context: "while scanning a tag", problem: problem, mark: mark)
}
}

View File

@ -0,0 +1,143 @@
extension PureYAML.Parsing.LibYAML {
/// Aggregate scanner state, mirroring the fields of `yaml_parser_t` that the
/// ported functions read and write. It is built up as the port proceeds;
/// currently it holds the indentation engine and the simple-key mechanism.
/// Column values follow libyaml's 0-based convention internally; conversion to
/// the package's 1-based `Mark` happens at token emission during integration.
struct Scanner {
/// The input buffer and position (`parser->buffer` / `parser->mark`).
var reader: Reader
/// The indentation stack and `roll_indent` / `unroll_indent` (also the
/// owner of `flow_level` and the token queue).
var indentEngine = IndentEngine()
/// `parser->simple_keys`. Always holds at least the block-context level.
var simpleKeys: [SimpleKey] = [.empty]
/// `parser->simple_key_allowed`. True at stream start.
var simpleKeyAllowed = true
/// Whether a tab appeared in the indentation before the pending token. A
/// tab is valid in-line separation but never indentation, so a block
/// indicator or implicit mapping key reached through one is an error.
var tabbedSeparation = false
/// Whether reaching the pending token crossed at least one line break. In
/// flow context a continuation line must be indented past the parent block,
/// so a token on a new line at or below that indent is an error.
var crossedLineBreak = false
/// Whether a tab appeared in the leading whitespace of a flow continuation
/// line. A tab is never indentation, so flow content reached through one is
/// an error even though the tab inflates the column.
var flowLeadingTab = false
/// Whether the most recent `:` value indicator sat on the same line as its
/// key (an inline implicit-key value). Such a value may not open a block
/// sequence on that line (`key: - a`), whereas an explicit key whose `:` is
/// on a later line may (`? k`\n`: - a`).
var lastValueWasInline = false
/// For each open flow level, whether it is a sequence (`[`) rather than a
/// mapping (`{`). A flow-sequence single-pair implicit key is single-line
/// (`c-s-implicit-json-key`), so its `:` may not cross a line break, unlike
/// a flow-mapping adjacent value (`{ "a"`\n`: b }` is allowed).
var flowIsSequence: [Bool] = []
/// `parser->mark`, kept in sync with the reader at each fetch step.
var mark: PureYAML.Parsing.Mark = .start
/// `parser->stream_start_produced`.
var streamStartProduced = false
/// `parser->stream_end_produced`.
var streamEndProduced = false
/// A plain scalar fetch that ran dry mid-content (`scanStep()` only, batch
/// scanning never sets this): resumed from here on the next `scanStep()`
/// call instead of restarting the scalar from its first scalar.
var suspendedPlainScalar: PlainScalarSuspended?
init(_ input: String = "", isFinal: Bool = true) {
reader = Reader(input, isFinal: isFinal)
}
/// `parser->flow_level`, kept on the indent engine.
var flowLevel: Int {
indentEngine.flowLevel
}
/// Port of `yaml_parser_stale_simple_keys`. A potential simple key is
/// invalid once it is on an earlier line or more than 1024 characters back
/// (a simple key is single-line and bounded); a required one going stale is
/// an error.
mutating func staleSimpleKeys() throws {
for index in simpleKeys.indices {
let key = simpleKeys[index]
guard key.possible else {
continue
}
// A block-context key (index 0) is single-line. A flow-context key
// (index >= 1) may be separated from its ':' by line breaks under
// YAML 1.2, so a new line alone does not invalidate it; both remain
// bounded to 1024 characters. libyaml invalidates every key on a new
// line, which wrongly rejects multi-line flow keys.
let tooFarBack = key.mark.index + 1024 < mark.index
let stale = index == 0 ? (key.mark.line < mark.line || tooFarBack) : tooFarBack
guard stale else {
continue
}
if key.required {
throw ScannerError(
context: "while scanning a simple key",
problem: "could not find expected ':'",
mark: key.mark,
)
}
simpleKeys[index].possible = false
}
}
/// Port of `yaml_parser_save_simple_key`. Record the current position as a
/// potential simple key for this flow level when one is allowed. A key is
/// required (a `:` must follow) when it sits exactly at the block indent.
mutating func saveSimpleKey() throws {
let required = flowLevel == 0 && indentEngine.indent == mark.column
guard simpleKeyAllowed else {
return
}
let key = SimpleKey(
possible: true,
required: required,
tokenNumber: indentEngine.tokensParsed + indentEngine.tokens.count,
mark: mark,
tabbed: tabbedSeparation,
)
try removeSimpleKey()
simpleKeys[simpleKeys.count - 1] = key
}
/// Port of `yaml_parser_remove_simple_key`. Drop the potential key at the
/// current flow level; removing a required key is an error.
mutating func removeSimpleKey() throws {
let top = simpleKeys.count - 1
if simpleKeys[top].possible, simpleKeys[top].required {
throw ScannerError(
context: "while scanning a simple key",
problem: "could not find expected ':'",
mark: simpleKeys[top].mark,
)
}
simpleKeys[top].possible = false
}
/// Port of `yaml_parser_increase_flow_level`. Push an empty simple key for
/// the new flow level, recording whether the level is a sequence.
mutating func increaseFlowLevel(isSequence: Bool) {
simpleKeys.append(.empty)
flowIsSequence.append(isSequence)
indentEngine.flowLevel += 1
}
/// Port of `yaml_parser_decrease_flow_level`. Pop the flow level's simple
/// key.
mutating func decreaseFlowLevel() {
if indentEngine.flowLevel > 0 {
indentEngine.flowLevel -= 1
simpleKeys.removeLast()
if !flowIsSequence.isEmpty {
flowIsSequence.removeLast()
}
}
}
}
}

View File

@ -0,0 +1,10 @@
extension PureYAML.Parsing.LibYAML {
/// A scanning error, mirroring libyaml's `yaml_parser_set_scanner_error`
/// (`context` plus `problem` at a mark). It is mapped to the package's public
/// `ParseError` when the port is integrated.
struct ScannerError: Swift.Error, Equatable {
var context: String
var problem: String
var mark: PureYAML.Parsing.Mark
}
}

View File

@ -0,0 +1,28 @@
extension PureYAML.Parsing.LibYAML {
/// Port of `yaml_simple_key_t`. A potential implicit ("simple") mapping key:
/// a node that may turn out to be a key once a `:` is seen on the same line.
/// `tokenNumber` records where its `KEY` token must be spliced into the queue.
struct SimpleKey: Equatable {
/// Whether a simple key is still possible at this flow level.
var possible: Bool
/// Whether a `:` is required (a key sitting exactly at the block indent).
var required: Bool
/// The absolute token number where the `KEY` token is inserted.
var tokenNumber: Int
/// The position the potential key started at.
var mark: PureYAML.Parsing.Mark
/// Whether this key is a JSON-style node (a quoted flow scalar). A `:`
/// adjacent to such a key is a value indicator even without a following
/// space, per the YAML flow "adjacent value" rule.
var isJSON = false
/// Whether a tab preceded this key in its indentation. A tab may be
/// separation but never indentation, so promoting such a key to a block
/// mapping key (a `:` follows) is an error.
var tabbed = false
/// The empty entry pushed for a new flow level (`{ 0, 0, 0, {0,0,0} }`).
static var empty: Self {
Self(possible: false, required: false, tokenNumber: 0, mark: .start)
}
}
}

View File

@ -0,0 +1,10 @@
extension PureYAML.Parsing.LibYAML {
/// A scanner token, mirroring `yaml_token_t`: a kind plus its start and end
/// marks. Token-specific payloads (scalar value, anchor name, tag) are added
/// as their fetchers are ported.
struct Token: Equatable {
var type: TokenType
var startMark: PureYAML.Parsing.Mark
var endMark: PureYAML.Parsing.Mark
}
}

View File

@ -0,0 +1,29 @@
extension PureYAML.Parsing.LibYAML {
/// A scanner token kind, mirroring `yaml_token_type_t`. The flow-collection,
/// anchor, tag, alias, and directive kinds are added as their fetchers are
/// ported; the block + plain-scalar core is present.
enum TokenType: Equatable {
case streamStart
case streamEnd
case documentStart
case documentEnd
case blockSequenceStart
case blockMappingStart
case blockEnd
case blockEntry
case flowSequenceStart
case flowMappingStart
case flowSequenceEnd
case flowMappingEnd
case flowEntry
case key
case value
case versionDirective(major: Int, minor: Int)
case tagDirective(handle: String, prefix: String)
case reservedDirective
case anchor(name: String)
case alias(name: String)
case tag(handle: String, suffix: String)
case scalar(value: String, style: PureYAML.Parsing.ScalarStyle)
}
}

View File

@ -0,0 +1,10 @@
extension PureYAML.Parsing {
/// Faithful Swift port of libyaml's scanner (`CYaml/src/scanner.c`). It is
/// built as a parallel implementation, verified token-for-token against
/// libyaml through the differential harness, to replace the hand-rolled
/// approximation whose indentation and simple-key handling the official test
/// suite surfaced as incorrect. The port proceeds function by function; each
/// piece mirrors its C counterpart so correctness is by construction rather
/// than re-derived. Members live under this namespace in their own files.
enum LibYAML {}
}

View File

@ -0,0 +1,46 @@
extension PureYAML.Parsing {
/// Shared merge-key (`<<`) resolution used by both the buffered and the
/// streaming event composers, so the override semantics live in one place.
enum MergeResolution {
/// Combine merge sources with local pairs.
///
/// Local keys win over inherited keys, and earlier merge sources win
/// over later ones. Insertion order is inherited pairs first, then the
/// document's own local pairs.
static func mergedPairs(
from mergeSources: [[PureYAML.Model.Pair]],
localPairs: [PureYAML.Model.Pair],
) -> [PureYAML.Model.Pair] {
guard !mergeSources.isEmpty else {
return localPairs
}
let localKeys = Set(localPairs.map(\.keyNode))
var inheritedKeys = Set<PureYAML.Model.Key>()
var inheritedPairs: [PureYAML.Model.Pair] = []
for source in mergeSources {
var sourceKeys = Set<PureYAML.Model.Key>()
for pair in source {
guard !localKeys.contains(pair.keyNode),
!inheritedKeys.contains(pair.keyNode)
else {
continue
}
inheritedPairs.append(pair)
sourceKeys.insert(pair.keyNode)
}
inheritedKeys.formUnion(sourceKeys)
}
return inheritedPairs + localPairs
}
/// The parse error raised when a merge value is neither a mapping nor a
/// sequence of mappings.
static func invalidMergeValue(mark: PureYAML.Parsing.Mark) -> PureYAML.Parsing.ParseError {
PureYAML.Parsing.ParseError.invalidMergeValue(
line: mark.line,
column: mark.column,
)
}
}
}

View File

@ -1,17 +0,0 @@
extension PureYAML.Parsing {
struct NodeProperties: Equatable {
var anchor: String?
var tag: String?
var mark: Mark?
static var none: Self {
Self(anchor: nil, tag: nil, mark: nil)
}
mutating func mergeUnset(from other: Self) {
anchor = anchor ?? other.anchor
tag = tag ?? other.tag
mark = mark ?? other.mark
}
}
}

View File

@ -1,6 +0,0 @@
extension PureYAML.Parsing {
struct NodeResult: Equatable {
var kind: NodeKind
var endMark: Mark
}
}

View File

@ -0,0 +1,41 @@
extension PureYAML.Parsing.ParseError {
/// Translate a ported-scanner error into the package's diagnostic-aware error,
/// converting libyaml's 0-based mark to the 1-based convention and mapping the
/// well-known scanner failures to precise diagnostic codes.
init(translating error: PureYAML.Parsing.LibYAML.ScannerError) {
let line = error.mark.line + 1
let column = error.mark.column + 1
let context = error.context ?? ""
if error.problem.hasPrefix("found a tab") {
self = .tabIndentation(line: line)
} else if error.problem == "control characters are not allowed" {
self = .invalidControlCharacter(line: line, column: column)
} else if context == "while scanning a quoted scalar", error.problem.hasSuffix("end of stream") {
self = .unterminatedQuotedString(line: line)
} else if context.hasPrefix("while scanning a tag") || context.hasSuffix("TAG directive") {
self = .unterminatedTag(line: line)
} else {
self = .malformed(reason: Self.reason(context: error.context, problem: error.problem), line: line, column: column)
}
}
/// Translate a ported-parser error into the package's diagnostic-aware error.
init(translating error: PureYAML.Parsing.LibYAML.ParserError) {
let line = error.mark.line + 1
let column = error.mark.column + 1
if error.problem == "found incompatible YAML document" {
self = .incompatibleYAMLDirective(line: line)
} else {
self = .malformed(reason: Self.reason(context: error.context, problem: error.problem), line: line, column: column)
}
}
/// Join a libyaml context and problem into one readable reason, e.g.
/// "while parsing a flow sequence, did not find expected ',' or ']'".
private static func reason(context: String?, problem: String) -> String {
guard let context else {
return problem
}
return "\(context), \(problem)"
}
}

View File

@ -3,6 +3,7 @@ public extension PureYAML.Parsing {
enum ParseError: Swift.Error, Equatable, Sendable, CustomStringConvertible {
case emptyDocument
case tabIndentation(line: Int)
case invalidControlCharacter(line: Int, column: Int)
case unexpectedIndentation(line: Int)
case mixedCollectionStyles(line: Int)
case expectedMappingKey(line: Int)
@ -11,6 +12,7 @@ public extension PureYAML.Parsing {
case expectedNode(line: Int, column: Int)
case expectedScalarKey(line: Int, column: Int)
case incompatibleYAMLDirective(line: Int)
case directiveWithoutDocument(line: Int)
case invalidTaggedScalar(tag: String, value: String, line: Int, column: Int)
case invalidMergeValue(line: Int, column: Int)
case unterminatedTag(line: Int)
@ -20,6 +22,15 @@ public extension PureYAML.Parsing {
case undefinedAlias(anchor: String, line: Int, column: Int)
case unexpectedEvent(expected: String, actual: String, line: Int, column: Int)
case unexpectedToken(expected: String, actual: String, line: Int, column: Int)
case nestingTooDeep(line: Int, column: Int)
/// The document's total EXPANDED node count (after resolving every `*alias` to its
/// target) exceeds the budget. `nestingTooDeep` bounds physical collection depth in the
/// event stream; it does nothing against an anchor-expansion ("billion laughs") bomb,
/// where a handful of shallow, small anchors are each aliased many times, each alias
/// nested inside the next, so the LOGICAL tree an alias resolves to is exponential in
/// the number of alias levels while the physical document stays tiny.
case expansionTooLarge(line: Int, column: Int)
case malformed(reason: String, line: Int, column: Int)
public var description: String {
switch self {
@ -27,6 +38,8 @@ public extension PureYAML.Parsing {
"document is empty"
case let .tabIndentation(line):
"tabs are not allowed for indentation at line \(line)"
case let .invalidControlCharacter(line, column):
"invalid control character at line \(line), column \(column)"
case let .unexpectedIndentation(line):
"unexpected indentation at line \(line)"
case let .mixedCollectionStyles(line):
@ -43,6 +56,8 @@ public extension PureYAML.Parsing {
"expected a scalar mapping key at line \(line), column \(column)"
case let .incompatibleYAMLDirective(line):
"incompatible YAML directive at line \(line)"
case let .directiveWithoutDocument(line):
"directive is not followed by a document start at line \(line)"
case let .invalidTaggedScalar(tag, value, line, column):
"invalid scalar '\(value)' for tag '\(tag)' at line \(line), column \(column)"
case let .invalidMergeValue(line, column):
@ -61,6 +76,128 @@ public extension PureYAML.Parsing {
"expected \(expected) at line \(line), column \(column), found \(actual)"
case let .unexpectedToken(expected, actual, line, column):
"expected \(expected) at line \(line), column \(column), found \(actual)"
case let .nestingTooDeep(line, column):
"nesting is too deep at line \(line), column \(column)"
case let .expansionTooLarge(line, column):
"the document's expanded size (after resolving aliases) is too large at line \(line), column \(column)"
case let .malformed(reason, line, column):
"\(reason) at line \(line), column \(column)"
}
}
public var diagnosticCode: String {
switch self {
case .emptyDocument:
"emptyDocument"
case .tabIndentation:
"tabIndentation"
case .invalidControlCharacter:
"invalidControlCharacter"
case .unexpectedIndentation:
"unexpectedIndentation"
case .mixedCollectionStyles:
"mixedCollectionStyles"
case .expectedMappingKey:
"expectedMappingKey"
case .expectedAnchorName:
"expectedAnchorName"
case .expectedAliasName:
"expectedAliasName"
case .expectedNode:
"expectedNode"
case .expectedScalarKey:
"expectedScalarKey"
case .incompatibleYAMLDirective:
"incompatibleYAMLDirective"
case .directiveWithoutDocument:
"directiveWithoutDocument"
case .invalidTaggedScalar:
"invalidTaggedScalar"
case .invalidMergeValue:
"invalidMergeValue"
case .unterminatedTag:
"unterminatedTag"
case .unterminatedQuotedString:
"unterminatedQuotedString"
case .unsupportedDirective:
"unsupportedDirective"
case .unsupportedMultiDocumentStream:
"unsupportedMultiDocumentStream"
case .undefinedAlias:
"undefinedAlias"
case .unexpectedEvent:
"unexpectedEvent"
case .unexpectedToken:
"unexpectedToken"
case .nestingTooDeep:
"nestingTooDeep"
case .expansionTooLarge:
"expansionTooLarge"
case .malformed:
"malformed"
}
}
public var sourceLine: Int? {
switch self {
case .emptyDocument:
nil
case let .tabIndentation(line),
let .unexpectedIndentation(line),
let .mixedCollectionStyles(line),
let .expectedMappingKey(line),
let .expectedAnchorName(line),
let .expectedAliasName(line),
let .incompatibleYAMLDirective(line),
let .directiveWithoutDocument(line),
let .unterminatedTag(line),
let .unterminatedQuotedString(line),
let .unsupportedDirective(_, line),
let .unsupportedMultiDocumentStream(line):
line
case let .expectedNode(line, _),
let .expectedScalarKey(line, _),
let .invalidTaggedScalar(_, _, line, _),
let .invalidMergeValue(line, _),
let .undefinedAlias(_, line, _),
let .unexpectedEvent(_, _, line, _),
let .unexpectedToken(_, _, line, _),
let .nestingTooDeep(line, _),
let .expansionTooLarge(line, _),
let .invalidControlCharacter(line, _),
let .malformed(_, line, _):
line
}
}
public var sourceColumn: Int? {
switch self {
case .emptyDocument,
.tabIndentation,
.unexpectedIndentation,
.mixedCollectionStyles,
.expectedMappingKey,
.expectedAnchorName,
.expectedAliasName,
.incompatibleYAMLDirective,
.directiveWithoutDocument,
.unterminatedTag,
.unterminatedQuotedString,
.unsupportedDirective,
.unsupportedMultiDocumentStream:
nil
case let .expectedNode(_, column),
let .expectedScalarKey(_, column),
let .invalidTaggedScalar(_, _, _, column),
let .invalidMergeValue(_, column),
let .undefinedAlias(_, _, column),
let .unexpectedEvent(_, _, _, column),
let .unexpectedToken(_, _, _, column),
let .nestingTooDeep(_, column),
let .expansionTooLarge(_, column),
let .invalidControlCharacter(_, column),
let .malformed(_, _, column):
column
}
}
}

View File

@ -1,7 +1,91 @@
extension PureYAML.Parsing.Parser {
func parseEvents(_ yaml: String) throws -> [PureYAML.Parsing.Event] {
let tokens = try PureYAML.Parsing.Scanner().scan(yaml)
var parser = PureYAML.Parsing.TokenEventParser(tokens: tokens, scalarParser: self)
return try parser.parse()
let events: [PureYAML.Parsing.Event]
do {
var parser = try PureYAML.Parsing.LibYAML.Parser(yaml)
events = try parser.parse().map { $0.bumpingMark() }
} catch let error as PureYAML.Parsing.LibYAML.ScannerError {
throw PureYAML.Parsing.ParseError(translating: error)
} catch let error as PureYAML.Parsing.LibYAML.ParserError {
throw PureYAML.Parsing.ParseError(translating: error)
}
try enforceNestingLimit(events)
return events
}
/// The deepest collection nesting the value/event composers will build. The
/// ported parser produces events iteratively without recursion, but the
/// composers, equality, hashing, and `Model/Value` teardown all recurse with
/// the nesting depth, so the event stream is bounded here before any recursive
/// consumer runs. 64 accepts every realistic document yet rejects pathological
/// input well before it can overflow the stack.
static var maxNestingDepth: Int {
64
}
/// Reject an event stream that nests collections past ``maxNestingDepth`` with
/// a ``ParseError/nestingTooDeep(line:column:)`` so deeply nested input fails
/// cleanly instead of overflowing a recursive consumer's stack.
private func enforceNestingLimit(_ events: [PureYAML.Parsing.Event]) throws {
var depth = 0
for event in events {
switch event {
case .sequenceStart, .mappingStart:
depth += 1
if depth > Self.maxNestingDepth {
throw PureYAML.Parsing.ParseError.nestingTooDeep(line: event.mark.line, column: event.mark.column)
}
case .sequenceEnd, .mappingEnd:
depth -= 1
default:
break
}
}
}
func parseStreaming(_ yaml: String) throws -> PureYAML.Model.Value {
var composer = PureYAML.Parsing.EventStreamComposer(
scalarParser: self,
mode: .singleDocument,
)
for event in try parseEvents(yaml) {
try composer.consume(event)
}
return try composer.composedValue()
}
func parseStreamStreaming(_ yaml: String) throws -> [PureYAML.Stream.Document] {
var composer = PureYAML.Parsing.EventStreamComposer(
scalarParser: self,
mode: .stream,
)
for event in try parseEvents(yaml) {
try composer.consume(event)
}
return try composer.composedStream()
}
func parseLocatedStreaming(_ yaml: String) throws -> PureYAML.Model.LocatedValue {
var composer = PureYAML.Parsing.EventStreamComposer(
scalarParser: self,
mode: .singleDocument,
tracksLocations: true,
)
for event in try parseEvents(yaml) {
try composer.consume(event)
}
return try composer.composedLocated()
}
func parseStreamLocatedStreaming(_ yaml: String) throws -> [PureYAML.Model.LocatedValue] {
var composer = PureYAML.Parsing.EventStreamComposer(
scalarParser: self,
mode: .stream,
tracksLocations: true,
)
for event in try parseEvents(yaml) {
try composer.consume(event)
}
return try composer.composedLocatedStream()
}
}

View File

@ -9,6 +9,13 @@ extension PureYAML.Parsing.Parser {
return tagged
}
// An explicit tag that is not a recognized core type, a local tag like `!foo` or the
// non-specific `!`, still fixes the node as a string: a tagged scalar is not subject to the
// implicit plain-scalar resolution that would otherwise read `! 12` as the integer 12.
if normalizedTag(tag) != nil {
return .string(value)
}
switch style {
case .folded, .literal:
return .string(value)
@ -28,6 +35,10 @@ extension PureYAML.Parsing.Parser {
return nil
}
switch tag {
case "!":
// The non-specific `!` tag resolves a scalar by kind alone: a scalar is
// a string, with no implicit typing, so `! 12` is "12" not the int 12.
return .string(value)
case "tag:yaml.org,2002:str":
return .string(value)
case "tag:yaml.org,2002:int":

View File

@ -4,15 +4,19 @@ public extension PureYAML.Parsing {
public init() {}
public func parse(_ yaml: String) throws -> PureYAML.Model.Value {
let events = try parseEvents(yaml)
var composer = PureYAML.Parsing.EventComposer(events: events, scalarParser: self)
return try composer.compose()
try parseStreaming(yaml)
}
public func parseStream(_ yaml: String) throws -> [PureYAML.Stream.Document] {
let events = try parseEvents(yaml)
var composer = PureYAML.Parsing.EventComposer(events: events, scalarParser: self)
return try composer.composeStream()
try parseStreamStreaming(yaml)
}
public func parseLocated(_ yaml: String) throws -> PureYAML.Model.LocatedValue {
try parseLocatedStreaming(yaml)
}
public func parseStreamLocated(_ yaml: String) throws -> [PureYAML.Model.LocatedValue] {
try parseStreamLocatedStreaming(yaml)
}
}
}

View File

@ -0,0 +1,99 @@
extension PureYAML.Parsing {
/// How a plain (unquoted) scalar reads: as null, a boolean, an integer, a floating-point number, or else a string.
///
/// This is the one definition of that reading. The parser takes the value of every plain scalar from it, and the emitter asks it whether a string can be
/// written without quotes, since a string that reads as anything else would come back changed. Two lists of the same words kept apart drift apart: the
/// emitter once wrote the string `yes` plain while the parser read `yes` as a boolean.
enum PlainScalar {
/// The value `text` has as a plain scalar. `text` has no surrounding whitespace.
static func resolve(_ text: String) -> PureYAML.Model.Value {
switch text {
case "", "~", "null", "Null", "NULL":
return .null
default:
break
}
if let bool = bool(text) {
return .bool(bool)
}
if let int = integer(text), !text.contains(".") {
return .int(int)
}
if text.contains(".") || text.contains("e") || text.contains("E"), let double = double(text) {
return .double(double)
}
return .string(text)
}
/// Whether `text` written plain reads back as the same string.
static func readsAsString(_ text: String) -> Bool {
if case .string = resolve(text) {
return true
}
return false
}
static func bool(_ text: String) -> Bool? {
switch text {
case "true", "True", "TRUE", "yes", "Yes", "YES", "on", "On", "ON":
true
case "false", "False", "FALSE", "no", "No", "NO", "off", "Off", "OFF":
false
default:
nil
}
}
static func integer(_ text: String) -> Int? {
let value = removingNumericSeparators(from: text)
let negative = value.hasPrefix("-")
let hasSign = negative || value.hasPrefix("+")
let unsigned = hasSign ? String(value.dropFirst()) : value
guard !unsigned.isEmpty else {
return nil
}
guard unsigned != "0" else {
return 0
}
let radixPrefixes: [(prefix: String, radix: Int)] = [
("0x", 16),
("0X", 16),
("0b", 2),
("0B", 2),
("0o", 8),
("0O", 8),
("0", 8),
]
for radixPrefix in radixPrefixes where unsigned.hasPrefix(radixPrefix.prefix) {
let digits = String(unsigned.dropFirst(radixPrefix.prefix.count))
// `Int(_, radix:)` accepts a sign, so `0-3` would read as minus three: after the prefix only digits may follow
guard let first = digits.first, first != "-", first != "+", let magnitude = Int(digits, radix: radixPrefix.radix) else {
return nil
}
return negative ? -magnitude : magnitude
}
return Int(value)
}
static func double(_ text: String) -> Double? {
switch text {
case ".inf", ".Inf", ".INF", "+.inf", "+.Inf", "+.INF":
.infinity
case "-.inf", "-.Inf", "-.INF":
-.infinity
case ".nan", ".NaN", ".NAN", "+.nan", "+.NaN", "+.NAN", "-.nan", "-.NaN", "-.NAN":
.nan
default:
Double(removingNumericSeparators(from: text))
}
}
private static func removingNumericSeparators(from text: String) -> String {
text.filter { $0 != "_" }
}
}
}

View File

@ -1,14 +1,18 @@
extension PureYAML.Parsing {
struct Reader {
private let source: String
private let utf8: String.UTF8View
private var position: String.Index
private var utf8Position: String.UTF8View.Index
private var line: Int
private var column: Int
private var utf8Index: Int
init(_ source: String) {
self.source = source
utf8 = source.utf8
position = source.startIndex
utf8Position = source.utf8.startIndex
line = 1
column = 1
utf8Index = 0
@ -42,22 +46,20 @@ extension PureYAML.Parsing {
return nil
}
let character = source[position]
position = source.index(after: position)
utf8Index += String(character).utf8.count
if character == "\r\n" {
switch utf8[utf8Position] {
case 13:
advanceCarriageReturn()
line += 1
column = 1
} else if character == "\r" {
if position < source.endIndex, source[position] == "\n" {
position = source.index(after: position)
utf8Index += 1
}
case 10:
advanceASCIICharacter()
line += 1
column = 1
} else if character == "\n" {
line += 1
column = 1
} else {
case 0 ..< 128:
advanceASCIICharacter()
column += 1
default:
advanceExtendedCharacter()
column += 1
}
return character
@ -71,5 +73,35 @@ extension PureYAML.Parsing {
}
return output
}
private mutating func advanceASCIICharacter() {
position = source.index(after: position)
utf8Position = utf8.index(after: utf8Position)
utf8Index += 1
}
private mutating func advanceCarriageReturn() {
position = source.index(after: position)
utf8Position = utf8.index(after: utf8Position)
utf8Index += 1
guard utf8Position < utf8.endIndex, utf8[utf8Position] == 10 else {
return
}
utf8Position = utf8.index(after: utf8Position)
utf8Index += 1
if position < source.endIndex, source[position] == "\n" {
position = source.index(after: position)
}
}
private mutating func advanceExtendedCharacter() {
position = source.index(after: position)
guard let nextUTF8Position = position.samePosition(in: utf8) else {
utf8Index += String(source[source.index(before: position)]).utf8.count
return
}
utf8Index += utf8.distance(from: utf8Position, to: nextUTF8Position)
utf8Position = nextUTF8Position
}
}
}

View File

@ -14,93 +14,25 @@ extension PureYAML.Parsing.Parser {
line: Int,
) throws -> PureYAML.Model.Value {
let value = trim(text)
switch value {
case "", "~", "null", "Null", "NULL":
return .null
default:
break
}
if let bool = parseBool(value) {
return .bool(bool)
}
if value.hasPrefix("\"") {
return try .string(parseDoubleQuoted(value, line: line))
}
if value.hasPrefix("'") {
return try .string(parseSingleQuoted(value, line: line))
}
if let int = parseInteger(value), !value.contains(".") {
return .int(int)
}
if value.contains(".") || value.contains("e") || value.contains("E") {
if let double = parseDouble(value) {
return .double(double)
}
}
return .string(value)
return PureYAML.Parsing.PlainScalar.resolve(value)
}
func parseBool(_ text: String) -> Bool? {
switch text {
case "true", "True", "TRUE", "yes", "Yes", "YES", "on", "On", "ON":
true
case "false", "False", "FALSE", "no", "No", "NO", "off", "Off", "OFF":
false
default:
nil
}
PureYAML.Parsing.PlainScalar.bool(text)
}
func parseInteger(_ text: String) -> Int? {
let value = removingNumericSeparators(from: text)
let negative = value.hasPrefix("-")
let hasSign = negative || value.hasPrefix("+")
let unsigned = hasSign ? String(value.dropFirst()) : value
guard !unsigned.isEmpty else {
return nil
}
guard unsigned != "0" else {
return 0
}
let radixPrefixes: [(prefix: String, radix: Int)] = [
("0x", 16),
("0X", 16),
("0b", 2),
("0B", 2),
("0o", 8),
("0O", 8),
("0", 8),
]
for radixPrefix in radixPrefixes where unsigned.hasPrefix(radixPrefix.prefix) {
let digits = String(unsigned.dropFirst(radixPrefix.prefix.count))
guard !digits.isEmpty, let magnitude = Int(digits, radix: radixPrefix.radix) else {
return nil
}
return negative ? -magnitude : magnitude
}
return Int(value)
PureYAML.Parsing.PlainScalar.integer(text)
}
func parseDouble(_ text: String) -> Double? {
switch text {
case ".inf", ".Inf", ".INF", "+.inf", "+.Inf", "+.INF":
.infinity
case "-.inf", "-.Inf", "-.INF":
-.infinity
case ".nan", ".NaN", ".NAN", "+.nan", "+.NaN", "+.NAN", "-.nan", "-.NaN", "-.NAN":
.nan
default:
Double(removingNumericSeparators(from: text))
}
}
func removingNumericSeparators(from text: String) -> String {
text.filter { $0 != "_" }
PureYAML.Parsing.PlainScalar.double(text)
}
func parseSingleQuoted(
@ -110,22 +42,8 @@ extension PureYAML.Parsing.Parser {
guard text.count >= 2, text.last == "'" else {
throw PureYAML.Parsing.ParseError.unterminatedQuotedString(line: line)
}
let inner = text.dropFirst().dropLast()
var output = ""
var iterator = inner.makeIterator()
while let character = iterator.next() {
if character == "'", let next = iterator.next() {
if next == "'" {
output.append("'")
} else {
output.append(character)
output.append(next)
}
} else {
output.append(character)
}
}
return output
let inner = Array(text.dropFirst().dropLast())
return foldFlowScalar(inner, single: true)
}
func parseDoubleQuoted(
@ -135,37 +53,271 @@ extension PureYAML.Parsing.Parser {
guard text.count >= 2, text.last == "\"" else {
throw PureYAML.Parsing.ParseError.unterminatedQuotedString(line: line)
}
let inner = text.dropFirst().dropLast()
var output = ""
var escaping = false
for character in inner {
if escaping {
appendDoubleQuotedEscape(character, to: &output)
escaping = false
} else if character == "\\" {
escaping = true
} else {
output.append(character)
}
}
if escaping {
output.append("\\")
}
return output
let inner = Array(text.dropFirst().dropLast())
return foldFlowScalar(inner, single: false)
}
func appendDoubleQuotedEscape(
_ character: Character,
to output: inout String,
/// Flow (quoted) scalar line folding, mirroring libyaml's
/// yaml_parser_scan_flow_scalar (single and double quoted). Trailing
/// whitespace before a break is dropped, leading whitespace on a
/// continuation line is dropped, a single break between non-empty lines
/// folds to one space, and N consecutive breaks yield N-1 line feeds.
/// For double quoted scalars a backslash at end of line suppresses the
/// break entirely.
func foldFlowScalar(
_ inner: [Character],
single: Bool,
) -> String {
var state = FlowFoldState()
let count = inner.count
while state.index < count {
state.leadingBlanks = false
consumeFlowContent(inner, single: single, state: &state)
consumeFlowBlanks(inner, state: &state)
joinFlowFold(state: &state)
}
return state.output
}
/// Mutable accumulators for flow scalar folding, matching the C reference's
/// string / leading_break / trailing_breaks / whitespaces buffers.
struct FlowFoldState {
var output = ""
var whitespaces = ""
var leadingBreak = ""
var trailingBreaks = ""
var leadingBlanks = false
var index = 0
}
/// Consume the run of non-blank characters at `state.index`, decoding
/// escapes. A double quoted backslash at end of line sets `leadingBlanks`
/// and stops, so the following break is suppressed during folding.
func consumeFlowContent(
_ inner: [Character],
single: Bool,
state: inout FlowFoldState,
) {
switch character {
case "\"": output.append("\"")
case "\\": output.append("\\")
case "/": output.append("/")
case "n": output.append("\n")
case "r": output.append("\r")
case "t": output.append("\t")
default: output.append(character)
let count = inner.count
while state.index < count, !isFlowBlank(inner[state.index]), !isFlowBreak(inner[state.index]) {
if single, inner[state.index] == "'" {
state.output.append("'")
let isEscapedQuote = state.index + 1 < count && inner[state.index + 1] == "'"
state.index += isEscapedQuote ? 2 : 1
continue
}
if !single, inner[state.index] == "\\" {
if state.index + 1 < count, isFlowBreak(inner[state.index + 1]) {
state.index += 2
state.leadingBlanks = true
break
}
state.index += 1
appendDoubleQuotedEscape(inner, index: &state.index, to: &state.output)
continue
}
state.output.append(inner[state.index])
state.index += 1
}
}
/// Consume the run of blanks and line breaks at `state.index`, sorting them
/// into the whitespace / leading-break / trailing-break buffers.
func consumeFlowBlanks(
_ inner: [Character],
state: inout FlowFoldState,
) {
let count = inner.count
while state.index < count, isFlowBlank(inner[state.index]) || isFlowBreak(inner[state.index]) {
if isFlowBlank(inner[state.index]) {
if !state.leadingBlanks {
state.whitespaces.append(inner[state.index])
}
} else if !state.leadingBlanks {
state.whitespaces = ""
state.leadingBreak = "\n"
state.leadingBlanks = true
} else {
state.trailingBreaks.append("\n")
}
state.index += 1
}
}
/// Join the captured whitespace or fold the captured line breaks into the
/// output, per YAML 1.2 flow folding.
func joinFlowFold(state: inout FlowFoldState) {
guard state.leadingBlanks else {
state.output.append(state.whitespaces)
state.whitespaces = ""
return
}
if state.leadingBreak.first == "\n" {
if state.trailingBreaks.isEmpty {
state.output.append(" ")
} else {
state.output.append(state.trailingBreaks)
state.trailingBreaks = ""
}
state.leadingBreak = ""
} else {
state.output.append(state.leadingBreak)
state.output.append(state.trailingBreaks)
state.leadingBreak = ""
state.trailingBreaks = ""
}
}
func isFlowBlank(_ character: Character) -> Bool {
character == " " || character == "\t"
}
func isFlowBreak(_ character: Character) -> Bool {
character == "\n" || character == "\r" || character == "\r\n"
}
/// Decode a double quoted escape. On entry index points at the escape
/// character (the position just after the backslash). On return index has
/// advanced past every character the escape consumed. The escape set
/// mirrors libyaml's yaml_parser_scan_flow_scalar.
func appendDoubleQuotedEscape(
_ text: [Character],
index: inout Int,
to output: inout String,
) {
guard index < text.count else {
output.append("\\")
return
}
let character = text[index]
if let simple = Self.simpleDoubleQuotedEscapes[character] {
output.append(simple)
index += 1
return
}
switch character {
case "x":
index += 1
appendHexEscape(text, index: &index, digits: 2, marker: "x", to: &output)
case "u":
index += 1
appendUnicodeEscape(text, index: &index, to: &output)
case "U":
index += 1
appendHexEscape(text, index: &index, digits: 8, marker: "U", to: &output)
default:
output.append(character)
index += 1
}
}
/// Single character double quoted escapes, matching libyaml's escape table.
static let simpleDoubleQuotedEscapes: [Character: Character] = [
"0": "\u{00}",
"a": "\u{07}",
"b": "\u{08}",
"t": "\t",
"\t": "\t",
"n": "\n",
"v": "\u{0B}",
"f": "\u{0C}",
"r": "\r",
"e": "\u{1B}",
" ": " ",
"\"": "\"",
"/": "/",
"\\": "\\",
"N": "\u{85}",
"_": "\u{A0}",
"L": "\u{2028}",
"P": "\u{2029}",
]
/// Decode \xXX or \UXXXXXXXX. On entry index points at the first hex digit.
func appendHexEscape(
_ text: [Character],
index: inout Int,
digits: Int,
marker: Character,
to output: inout String,
) {
guard let value = readHex(text, index: &index, digits: digits) else {
output.append(marker)
return
}
if let scalar = UnicodeScalar(value) {
output.unicodeScalars.append(scalar)
} else {
output.append(marker)
}
}
/// Decode \uXXXX, including surrogate pairs. On entry index points at the
/// first hex digit.
func appendUnicodeEscape(
_ text: [Character],
index: inout Int,
to output: inout String,
) {
let start = index
guard let codeUnit = readHex(text, index: &index, digits: 4) else {
output.append("u")
index = start
return
}
if isHighSurrogate(codeUnit), let scalar = readSurrogatePair(high: codeUnit, text: text, index: &index) {
output.unicodeScalars.append(scalar)
} else if let scalar = UnicodeScalar(codeUnit) {
output.unicodeScalars.append(scalar)
} else {
output.append("u")
}
}
func readSurrogatePair(
high: UInt32,
text: [Character],
index: inout Int,
) -> UnicodeScalar? {
var probe = index
guard probe + 1 < text.count, text[probe] == "\\", text[probe + 1] == "u" else {
return nil
}
probe += 2
guard let low = readHex(text, index: &probe, digits: 4), isLowSurrogate(low) else {
return nil
}
index = probe
let value = 0x10000 + ((high - 0xD800) << 10) + (low - 0xDC00)
return UnicodeScalar(value)
}
/// Read exactly `digits` hex characters starting at index. On success index
/// advances past them, otherwise index is left unchanged.
func readHex(
_ text: [Character],
index: inout Int,
digits: Int,
) -> UInt32? {
guard index + digits <= text.count else {
return nil
}
var value: UInt32 = 0
for offset in 0 ..< digits {
guard let digit = text[index + offset].hexDigitValue else {
return nil
}
value = (value << 4) + UInt32(digit)
}
index += digits
return value
}
func isHighSurrogate(_ value: UInt32) -> Bool {
(0xD800 ... 0xDBFF).contains(value)
}
func isLowSurrogate(_ value: UInt32) -> Bool {
(0xDC00 ... 0xDFFF).contains(value)
}
}

View File

@ -1,82 +0,0 @@
extension PureYAML.Parsing.Scanner {
func scanDirectiveOrDocumentMarker(_ state: inout State) throws -> Bool {
guard state.indentation.last == 0 else {
return false
}
if state.reader.peek() == "%" {
try scanDirective(&state)
return true
}
if state.reader.peek() == "#" {
return false
}
if startsWith("---", state.reader), isIndicatorBoundary(state.reader.peek(offset: 3)) {
scanDocumentMarker(.documentStart, state: &state)
resetForDocumentStart(&state)
return true
}
if startsWith("...", state.reader), isIndicatorBoundary(state.reader.peek(offset: 3)) {
guard !state.isDocumentClosed else {
throw PureYAML.Parsing.ParseError.unsupportedMultiDocumentStream(line: state.reader.mark.line)
}
scanDocumentMarker(.documentEnd, state: &state)
resetForDocumentEnd(&state)
return true
}
guard !state.isDocumentClosed else {
throw PureYAML.Parsing.ParseError.unsupportedMultiDocumentStream(line: state.reader.mark.line)
}
return false
}
func scanDocumentMarker(
_ kind: PureYAML.Parsing.TokenKind,
state: inout State,
) {
let start = state.reader.mark
state.reader.advance()
state.reader.advance()
state.reader.advance()
state.append(kind, mark: start, endMark: state.reader.mark)
skipLine(&state)
}
func resetForDocumentStart(_ state: inout State) {
state.hasDocumentContent = false
state.hasDocumentStartMarker = true
state.isDocumentClosed = false
}
func resetForDocumentEnd(_ state: inout State) {
state.hasDocumentContent = false
state.hasDocumentStartMarker = false
state.isDocumentClosed = true
}
func scanDirective(_ state: inout State) throws {
let start = state.reader.mark
let line = state.reader.consume { character in
!isLineBreak(character)
}
let parts = line.split(separator: " ").map(String.init)
guard let name = parts.first else {
return
}
guard !state.hasDocumentContent, !state.hasDocumentStartMarker, !state.isDocumentClosed else {
throw PureYAML.Parsing.ParseError.unsupportedDirective(name: name, line: start.line)
}
switch name {
case "%YAML":
guard parts.indices.contains(1), parts[1] == "1.1" || parts[1] == "1.2" else {
throw PureYAML.Parsing.ParseError.incompatibleYAMLDirective(line: start.line)
}
case "%TAG":
guard parts.count >= 3 else {
return
}
state.tagHandles[parts[1]] = parts[2]
default:
throw PureYAML.Parsing.ParseError.unsupportedDirective(name: name, line: start.line)
}
}
}

View File

@ -1,245 +0,0 @@
extension PureYAML.Parsing.Scanner {
func startsWith(
_ prefix: String,
_ reader: PureYAML.Parsing.Reader,
) -> Bool {
for (offset, character) in prefix.enumerated() {
guard reader.peek(offset: offset) == character else {
return false
}
}
return true
}
func isLineBreak(_ character: Character?) -> Bool {
character == "\n" || character == "\r" || character == "\r\n"
}
func isBlockEntry(_ reader: PureYAML.Parsing.Reader) -> Bool {
reader.peek() == "-" && isIndicatorBoundary(reader.peek(offset: 1))
}
func isIndicatorBoundary(_ character: Character?) -> Bool {
guard let character else {
return true
}
return character == " " || isLineBreak(character)
}
func isMappingValueBoundary(_ character: Character?) -> Bool {
guard let character else {
return true
}
return character == " "
|| isLineBreak(character)
|| character == ","
|| character == "]"
|| character == "}"
}
func isBlockMappingValueBoundary(_ character: Character?) -> Bool {
guard let character else {
return true
}
return character == " " || isLineBreak(character)
}
func isTokenTerminator(_ character: Character) -> Bool {
character == " "
|| isLineBreak(character)
|| character == ","
|| character == "["
|| character == "]"
|| character == "{"
|| character == "}"
|| character == "#"
}
func canStartAnchorName(_ character: Character?) -> Bool {
guard let character else {
return false
}
return !isTokenTerminator(character)
&& character != "*"
&& character != "&"
&& character != "!"
&& character != "\""
&& character != "'"
}
func hasNodeOnSameLine(afterCurrentPositionIn reader: PureYAML.Parsing.Reader) -> Bool {
var offset = 0
while reader.peek(offset: offset) == " " {
offset += 1
}
guard let character = reader.peek(offset: offset) else {
return false
}
return !isLineBreak(character) && character != "#"
}
func isFlowDelimiter(_ character: Character) -> Bool {
character == ","
|| character == "["
|| character == "]"
|| character == "{"
|| character == "}"
}
func isBlockScalarHeaderCharacter(_ character: Character) -> Bool {
character == "+"
|| character == "-"
|| character.isNumber
}
func trimTrailingSpaces(_ value: String) -> String {
var end = value.endIndex
while end > value.startIndex {
let previous = value.index(before: end)
guard value[previous] == " " else {
break
}
end = previous
}
return String(value[..<end])
}
func scanPlainScalarContinuations(
after firstLine: String,
state: inout State,
) -> String {
var value = firstLine
while true {
var probe = state.reader
guard isLineBreak(probe.peek()) else {
return value
}
probe.advance()
var blankLineCount = 0
while true {
var blankProbe = probe
_ = blankProbe.consume { character in
character == " "
}
guard isLineBreak(blankProbe.peek()) else {
break
}
blankProbe.advance()
probe = blankProbe
blankLineCount += 1
}
_ = probe.consume { character in
character == " "
}
let indentationWidth = probe.mark.column - 1
guard !probe.isAtEnd, !isLineBreak(probe.peek()), probe.peek() != "#" else {
return value
}
guard !isBlockEntry(probe) else {
return value
}
let baseIndentation = plainScalarContinuationBaseIndentation(state: state)
guard indentationWidth > baseIndentation else {
return value
}
var continuationState = state
continuationState.reader = probe
let continuation = scanPlainScalarContinuationLine(&continuationState)
guard !isAtMappingValueIndicator(state: continuationState) else {
return value
}
state = continuationState
if blankLineCount == 0 {
value += " "
} else {
value += String(repeating: "\n", count: blankLineCount)
}
value += continuation
}
}
func plainScalarContinuationBaseIndentation(state: State) -> Int {
state.indentation.last ?? 0
}
func scanPlainScalarContinuationLine(_ state: inout State) -> String {
var value = ""
while let character = state.reader.peek(), !isLineBreak(character) {
if character == " ", state.reader.peek(offset: 1) == "#" {
break
}
if character == "#", value.isEmpty || value.last == " " {
break
}
if character == ":", shouldEndPlainScalarAtMappingValue(state: state) {
break
}
if state.flowDepth > 0, isFlowDelimiter(character) {
break
}
value.append(character)
state.reader.advance()
}
return trimTrailingSpaces(value)
}
func skipLine(_ state: inout State) {
_ = state.reader.consume { character in
!isLineBreak(character)
}
}
func expandTag(
_ tag: String,
state: State,
) -> String {
let handles = state.tagHandles.keys.sorted { first, second in
first.count > second.count
}
for handle in handles where tag.hasPrefix(handle) {
let suffix = tag.dropFirst(handle.count)
return (state.tagHandles[handle] ?? handle) + suffix
}
return tag
}
func shouldEndPlainScalar(
_ character: Character,
state: State,
accumulated: String,
) -> Bool {
if isLineBreak(character) {
return true
}
if character == " ", state.reader.peek(offset: 1) == "#" {
return true
}
if character == "#", accumulated.isEmpty || accumulated.last == " " {
return true
}
if character == ":", shouldEndPlainScalarAtMappingValue(state: state) {
return true
}
if state.flowDepth > 0, isFlowDelimiter(character) {
return true
}
return false
}
func shouldEndPlainScalarAtMappingValue(state: State) -> Bool {
if state.flowDepth > 0 {
return isMappingValueBoundary(state.reader.peek(offset: 1))
}
return isBlockMappingValueBoundary(state.reader.peek(offset: 1))
}
func isAtMappingValueIndicator(state: State) -> Bool {
state.reader.peek() == ":" && shouldEndPlainScalarAtMappingValue(state: state)
}
}

View File

@ -1,158 +0,0 @@
extension PureYAML.Parsing.Scanner {
func scanIndentation(_ state: inout State) throws {
let start = state.reader.mark
let width = try scanIndentationWidth(&state)
guard shouldProcessIndentation(state) else {
return
}
let end = state.reader.mark
let current = state.indentation.last ?? 0
if consumeBlockScalarContentIndentation(width: width, state: &state) {
return
}
if let indicator = state.pendingBlockScalarIndentationIndicator {
try applyExplicitBlockScalarIndentation(
indicator: indicator,
width: width,
current: current,
marks: (start: start, end: end),
state: &state,
)
} else if width > current {
applyIncreasedIndentation(width: width, start: start, end: end, state: &state)
} else if width < current {
try applyDedentation(width: width, start: start, end: end, state: &state)
}
pruneImplicitIndentation(&state)
clearPendingBlockScalarIndentation(&state)
}
func scanIndentationWidth(_ state: inout State) throws -> Int {
var width = 0
while let character = state.reader.peek() {
if character == " " {
width += 1
state.reader.advance()
} else if character == "\t" {
width = nextTabStop(after: width)
state.reader.advance()
} else {
return width
}
}
return width
}
func nextTabStop(after width: Int) -> Int {
((width / 4) + 1) * 4
}
func shouldProcessIndentation(_ state: State) -> Bool {
guard !state.reader.isAtEnd, !isLineBreak(state.reader.peek()) else {
return false
}
guard state.reader.peek() != "#" else {
return false
}
return state.flowDepth == 0
}
func consumeBlockScalarContentIndentation(
width: Int,
state: inout State,
) -> Bool {
guard let blockScalarIndentation = state.blockScalarIndentation else {
return false
}
guard width < blockScalarIndentation else {
state.blockScalarContentPrefix = String(repeating: " ", count: width - blockScalarIndentation)
return true
}
state.blockScalarIndentation = nil
state.blockScalarContentPrefix = ""
return false
}
func applyExplicitBlockScalarIndentation(
indicator: Int,
width: Int,
current: Int,
marks: (start: PureYAML.Parsing.Mark, end: PureYAML.Parsing.Mark),
state: inout State,
) throws {
let blockScalarIndentation = current + indicator
guard width >= blockScalarIndentation else {
throw PureYAML.Parsing.ParseError.unexpectedIndentation(line: marks.end.line)
}
state.indentation.append(blockScalarIndentation)
state.append(.indent(width: blockScalarIndentation), mark: marks.start, endMark: marks.end)
state.blockScalarIndentation = blockScalarIndentation
state.blockScalarContentPrefix = String(repeating: " ", count: width - blockScalarIndentation)
clearPendingBlockScalarIndentation(&state)
}
func applyIncreasedIndentation(
width: Int,
start: PureYAML.Parsing.Mark,
end: PureYAML.Parsing.Mark,
state: inout State,
) {
state.indentation.append(width)
state.append(.indent(width: width), mark: start, endMark: end)
guard state.pendingBlockScalarIndentation else {
return
}
state.blockScalarIndentation = width
state.blockScalarContentPrefix = ""
clearPendingBlockScalarIndentation(&state)
}
func applyDedentation(
width: Int,
start: PureYAML.Parsing.Mark,
end: PureYAML.Parsing.Mark,
state: inout State,
) throws {
while state.indentation.count > 1, width < (state.indentation.last ?? 0) {
state.indentation.removeLast()
state.append(.dedent(width: state.indentation.last ?? 0), mark: end, endMark: end)
}
if canUseImplicitIndentation(width: width, state: state) {
state.indentation.append(width)
state.append(.indent(width: width), mark: start, endMark: end)
}
guard width == state.indentation.last else {
throw PureYAML.Parsing.ParseError.unexpectedIndentation(line: end.line)
}
}
func canUseImplicitIndentation(
width: Int,
state: State,
) -> Bool {
width != state.indentation.last
&& width > (state.indentation.last ?? 0)
&& state.validImplicitIndentation.contains(width)
}
func pruneImplicitIndentation(_ state: inout State) {
let current = state.indentation.last ?? 0
state.validImplicitIndentation = state.validImplicitIndentation.filter { $0 <= current }
}
func clearPendingBlockScalarIndentation(_ state: inout State) {
state.pendingBlockScalarIndentation = false
state.pendingBlockScalarIndentationIndicator = nil
}
func closeIndentation(_ state: inout State) {
let mark = state.reader.mark
while state.indentation.count > 1 {
state.indentation.removeLast()
state.append(.dedent(width: state.indentation.last ?? 0), mark: mark, endMark: mark)
}
}
}

View File

@ -1,386 +0,0 @@
extension PureYAML.Parsing {
struct Scanner {
init() {}
func scan(_ yaml: String) throws -> [Token] {
var state = State(reader: Reader(yaml))
state.append(.streamStart)
while !state.reader.isAtEnd {
if isLineBreak(state.reader.peek()) {
state.reader.advance()
state.isAtLineStart = true
continue
}
if state.isAtLineStart {
try scanIndentation(&state)
if state.reader.isAtEnd {
break
}
if isLineBreak(state.reader.peek()) {
continue
}
if try scanDirectiveOrDocumentMarker(&state) {
continue
}
state.isAtLineStart = false
}
try scanToken(&state)
}
closeIndentation(&state)
state.append(.streamEnd)
return state.tokens
}
}
}
extension PureYAML.Parsing.Scanner {
struct State {
var reader: PureYAML.Parsing.Reader
var tokens: [PureYAML.Parsing.Token] = []
var indentation: [Int] = [0]
var isAtLineStart = true
var hasDocumentContent = false
var hasDocumentStartMarker = false
var isDocumentClosed = false
var flowDepth = 0
var validImplicitIndentation: Set<Int> = []
var pendingBlockScalarIndentation = false
var pendingBlockScalarIndentationIndicator: Int?
var blockScalarIndentation: Int?
var blockScalarContentPrefix = ""
var tagHandles = [
"!": "!",
"!!": "tag:yaml.org,2002:",
]
mutating func append(_ kind: PureYAML.Parsing.TokenKind) {
let mark = reader.mark
tokens.append(PureYAML.Parsing.Token(kind: kind, mark: mark, endMark: mark))
markDocumentContentIfNeeded(kind)
}
mutating func append(
_ kind: PureYAML.Parsing.TokenKind,
mark: PureYAML.Parsing.Mark,
endMark: PureYAML.Parsing.Mark,
) {
tokens.append(PureYAML.Parsing.Token(kind: kind, mark: mark, endMark: endMark))
markDocumentContentIfNeeded(kind)
}
mutating func markDocumentContentIfNeeded(_ kind: PureYAML.Parsing.TokenKind) {
switch kind {
case .comment, .dedent, .documentEnd, .documentStart, .indent, .streamEnd, .streamStart:
return
default:
hasDocumentContent = true
}
}
}
}
extension PureYAML.Parsing.Scanner {
func scanToken(_ state: inout State) throws {
skipSeparation(&state)
guard let character = state.reader.peek(), !isLineBreak(character) else {
return
}
if state.blockScalarIndentation != nil {
scanBlockScalarContent(&state)
return
}
if try scanLineIndicatorToken(character, state: &state) {
return
}
if scanFlowToken(character, state: &state) {
return
}
if try scanQuotedOrBlockScalarToken(character, state: &state) {
return
}
if try scanNodePropertyToken(character, state: &state) {
return
}
scanPlainScalar(&state)
}
func scanLineIndicatorToken(
_ character: Character,
state: inout State,
) throws -> Bool {
switch character {
case "#":
scanComment(&state)
return true
case "-" where isBlockEntry(state.reader):
scanFixedToken(.blockEntry, state: &state)
return true
case "?" where isIndicatorBoundary(state.reader.peek(offset: 1)):
scanFixedToken(.mappingKey, state: &state)
return true
case ":" where isMappingValueBoundary(state.reader.peek(offset: 1)):
scanFixedToken(.mappingValue, state: &state)
return true
default:
return false
}
}
func scanFlowToken(
_ character: Character,
state: inout State,
) -> Bool {
switch character {
case "[":
state.flowDepth += 1
scanFixedToken(.flowSequenceStart, state: &state)
return true
case "]":
state.flowDepth = max(0, state.flowDepth - 1)
scanFixedToken(.flowSequenceEnd, state: &state)
return true
case "{":
state.flowDepth += 1
scanFixedToken(.flowMappingStart, state: &state)
return true
case "}":
state.flowDepth = max(0, state.flowDepth - 1)
scanFixedToken(.flowMappingEnd, state: &state)
return true
case ",":
scanFixedToken(.flowEntry, state: &state)
return true
default:
return false
}
}
func scanNodePropertyToken(
_ character: Character,
state: inout State,
) throws -> Bool {
switch character {
case "&":
try scanNamedToken(
kind: PureYAML.Parsing.TokenKind.anchor,
missingNameError: PureYAML.Parsing.ParseError.expectedAnchorName,
state: &state,
)
return true
case "*":
try scanNamedToken(
kind: PureYAML.Parsing.TokenKind.alias,
missingNameError: PureYAML.Parsing.ParseError.expectedAliasName,
state: &state,
)
return true
case "!":
try scanTag(&state)
return true
default:
return false
}
}
func scanQuotedOrBlockScalarToken(
_ character: Character,
state: inout State,
) throws -> Bool {
switch character {
case "\"":
try scanQuotedScalar(.doubleQuoted, state: &state)
return true
case "'":
try scanQuotedScalar(.singleQuoted, state: &state)
return true
case "|":
scanBlockScalarHeader(.literal, state: &state)
return true
case ">":
scanBlockScalarHeader(.folded, state: &state)
return true
default:
return false
}
}
func skipSeparation(_ state: inout State) {
while state.reader.peek() == " " {
state.reader.advance()
}
}
func scanFixedToken(
_ kind: PureYAML.Parsing.TokenKind,
state: inout State,
) {
let start = state.reader.mark
state.reader.advance()
state.append(kind, mark: start, endMark: state.reader.mark)
if case .blockEntry = kind, hasNodeOnSameLine(afterCurrentPositionIn: state.reader) {
state.validImplicitIndentation.insert(start.column + 1)
}
}
func scanComment(_ state: inout State) {
let start = state.reader.mark
state.reader.advance()
if state.reader.peek() == " " {
state.reader.advance()
}
let value = state.reader.consume { character in
!isLineBreak(character)
}
state.append(.comment(value), mark: start, endMark: state.reader.mark)
}
func scanQuotedScalar(
_ style: PureYAML.Parsing.ScalarStyle,
state: inout State,
) throws {
let start = state.reader.mark
let quote: Character = style == .doubleQuoted ? "\"" : "'"
state.reader.advance()
var value = ""
var previousWasEscape = false
while let character = state.reader.peek() {
if style == .singleQuoted, character == quote, state.reader.peek(offset: 1) == quote {
value.append(character)
state.reader.advance()
value.append(character)
state.reader.advance()
continue
}
if character == quote, !previousWasEscape {
break
}
value.append(character)
previousWasEscape = character == "\\" && style == .doubleQuoted && !previousWasEscape
state.reader.advance()
if character != "\\" {
previousWasEscape = false
}
}
guard state.reader.peek() == quote else {
throw PureYAML.Parsing.ParseError.unterminatedQuotedString(line: start.line)
}
state.reader.advance()
appendMappingKeyIfNeeded(start: start, state: &state)
state.append(.scalar(value: value, style: style), mark: start, endMark: state.reader.mark)
}
func scanBlockScalarHeader(
_ style: PureYAML.Parsing.BlockScalarStyle,
state: inout State,
) {
let start = state.reader.mark
var chomping = PureYAML.Parsing.BlockScalarChomping.clip
var indentationIndicator: Int?
state.reader.advance()
while let character = state.reader.peek(), isBlockScalarHeaderCharacter(character) {
if character == "-" {
chomping = .strip
} else if character.isNumber {
indentationIndicator = Int(String(character))
}
state.reader.advance()
}
state.append(
.blockScalarHeader(style: style, chomping: chomping),
mark: start,
endMark: state.reader.mark,
)
state.pendingBlockScalarIndentation = true
state.pendingBlockScalarIndentationIndicator = indentationIndicator
}
func scanBlockScalarContent(_ state: inout State) {
let start = state.reader.mark
let value = state.blockScalarContentPrefix + state.reader.consume { character in
!isLineBreak(character)
}
state.blockScalarContentPrefix = ""
state.append(.scalar(value: value, style: .plain), mark: start, endMark: state.reader.mark)
}
func scanNamedToken(
kind: (String) -> PureYAML.Parsing.TokenKind,
missingNameError: (Int) -> PureYAML.Parsing.ParseError,
state: inout State,
) throws {
let start = state.reader.mark
state.reader.advance()
let name = state.reader.consume { character in
!isTokenTerminator(character)
}
guard !name.isEmpty else {
throw missingNameError(start.line)
}
state.append(kind(name), mark: start, endMark: state.reader.mark)
}
func scanTag(_ state: inout State) throws {
let start = state.reader.mark
state.reader.advance()
var value = "!"
if state.reader.peek() == "<" {
value.append(state.reader.advance() ?? "<")
var didClose = false
while let character = state.reader.peek() {
value.append(character)
state.reader.advance()
if character == ">" {
didClose = true
break
}
}
guard didClose else {
throw PureYAML.Parsing.ParseError.unterminatedTag(line: start.line)
}
} else {
value += state.reader.consume { character in
!isTokenTerminator(character)
}
value = expandTag(value, state: state)
}
state.append(.tag(value), mark: start, endMark: state.reader.mark)
}
func scanPlainScalar(_ state: inout State) {
let start = state.reader.mark
var value = ""
while let character = state.reader.peek() {
if shouldEndPlainScalar(character, state: state, accumulated: value) {
break
}
value.append(character)
state.reader.advance()
}
var trimmed = trimTrailingSpaces(value)
let isMappingKey = appendMappingKeyIfNeeded(start: start, state: &state)
if !isMappingKey {
trimmed = scanPlainScalarContinuations(
after: trimmed,
state: &state,
)
}
state.append(.scalar(value: trimmed, style: .plain), mark: start, endMark: state.reader.mark)
}
@discardableResult
func appendMappingKeyIfNeeded(
start: PureYAML.Parsing.Mark,
state: inout State,
) -> Bool {
guard state.reader.peek() == ":", isMappingValueBoundary(state.reader.peek(offset: 1)) else {
return false
}
state.append(.mappingKey, mark: start, endMark: start)
return true
}
}

View File

@ -1,108 +0,0 @@
extension PureYAML.Parsing {
struct Token: Equatable, CustomStringConvertible {
var kind: TokenKind
var mark: Mark
var endMark: Mark
var description: String {
"\(kind) @\(mark)..\(endMark)"
}
}
}
extension PureYAML.Parsing {
enum TokenKind: Equatable {
case streamStart
case streamEnd
case documentStart
case documentEnd
case indent(width: Int)
case dedent(width: Int)
case blockEntry
case mappingKey
case mappingValue
case flowSequenceStart
case flowSequenceEnd
case flowMappingStart
case flowMappingEnd
case flowEntry
case scalar(value: String, style: ScalarStyle)
case blockScalarHeader(style: BlockScalarStyle, chomping: BlockScalarChomping)
case anchor(String)
case alias(String)
case tag(String)
case comment(String)
}
}
extension PureYAML.Parsing.TokenKind: CustomStringConvertible {
var description: String {
switch self {
case .streamStart:
"streamStart"
case .streamEnd:
"streamEnd"
case .documentStart:
"documentStart"
case .documentEnd:
"documentEnd"
case let .indent(width):
"indent width=\(width)"
case let .dedent(width):
"dedent width=\(width)"
case .blockEntry:
"blockEntry"
case .mappingKey:
"mappingKey"
case .mappingValue:
"mappingValue"
case .flowSequenceStart:
"flowSequenceStart"
case .flowSequenceEnd:
"flowSequenceEnd"
case .flowMappingStart:
"flowMappingStart"
case .flowMappingEnd:
"flowMappingEnd"
case .flowEntry:
"flowEntry"
case let .scalar(value, style):
"scalar value=\"\(escape(value))\" style=\(style)"
case let .blockScalarHeader(style, chomping):
if chomping == .clip {
"blockScalarHeader style=\(style)"
} else {
"blockScalarHeader style=\(style) chomping=\(chomping)"
}
case let .anchor(value):
"anchor name=\(value)"
case let .alias(value):
"alias name=\(value)"
case let .tag(value):
"tag value=\(value)"
case let .comment(value):
"comment value=\"\(escape(value))\""
}
}
private func escape(_ value: String) -> String {
var output = ""
for character in value {
switch character {
case "\\":
output += "\\\\"
case "\"":
output += "\\\""
case "\n":
output += "\\n"
case "\r":
output += "\\r"
case "\t":
output += "\\t"
default:
output.append(character)
}
}
return output
}
}

Some files were not shown because too many files have changed in this diff Show More