Merge conformance/yaml-test-suite-gaps: the libyaml port becomes the engine on main

The ported engine wins every conflict: main's post-fork patches to the
hand-rolled Scanner and TokenEventParser land on deleted files and are
dropped; their behaviors are carried by the port and guarded by the
whole-suite conformance gate (402/402 passing).

Merge adaptations, all gates green (style, namespacing, forbidden
patterns, changelog, roadmap, swiftformat, swiftlint strict, build,
full test suite, whole-suite conformance sweep):

- PureYAMLJSON representability rules move onto the Check algebra,
  replacing the retired legacy Rule type.
- The permissive tag-handle pre-pass is removed; the port rejects an
  undefined handle at parse time (suite case QLJ7) and the validation
  report carries it as a parse diagnostic.
- EventStreamComposer's nested types split one-per-file per the
  namespacing gate.
- Tests pinned to old-engine behavior were re-verified against the
  reference and updated: libyaml rejects an explicit-key continuation
  ending at a value colon, and suite case QB6E requires rejecting a
  quoted continuation at column 0.
- CHANGELOG unreleased entries from both lines are preserved in two
  groups with a provenance note.

Closes #1, closes #3
This commit is contained in:
Mihaela Mihaljevic 2026-07-05 00:23:48 +02:00
commit a8950e118c
1299 changed files with 16254 additions and 4586 deletions

1
.claude/commands/exp Symbolic link
View File

@ -0,0 +1 @@
../../../../private/mihaela-agents/skills/exp

1
.claude/commands/openapi Symbolic link
View File

@ -0,0 +1 @@
../../../../private/mihaela-agents/skills/openapi

1
.claude/commands/swift Symbolic link
View File

@ -0,0 +1 @@
../../../../private/mihaela-agents/skills/swift

1
.claude/commands/ui Symbolic link
View File

@ -0,0 +1 @@
../../../../private/mihaela-agents/skills/ui

View File

@ -67,6 +67,7 @@ SKIP_PATTERNS=(
# Vendored upstream rules canon (mirror of rules-swift). Its enforcement docs
# legitimately quote the forbidden phrases. See docs/rules-swift/PROVENANCE.md.
'^docs/rules-swift/'
'^Tests/Fixtures/yaml-test-suite/'
# Rule files that DEFINE the prohibitions; they MUST be able to cite
# the forbidden strings to document them.
'/github-discipline\.md$'

View File

@ -1,8 +1,8 @@
blank_issues_enabled: false
contact_links:
- name: Questions and ideas
url: https://github.com/mihaelamj/PureYAML/discussions
url: https://codeberg.org/Mihaela/PureYAML/discussions
about: Ask questions, float ideas, or discuss design in Discussions.
- name: Security issue
url: https://github.com/mihaelamj/PureYAML/blob/main/SECURITY.md
url: https://codeberg.org/Mihaela/PureYAML/blob/main/SECURITY.md
about: Do not file security issues publicly. Follow the security policy instead.

View File

@ -22,3 +22,14 @@ file is kept to the standard MIT text so it is correctly detected as MIT.
The private `PureYAMLResearch` repository contains a full Yams source snapshot
for study and compatibility research. That repository is not the implementation
source for PureYAML.
## Vendored test data
`Tests/Fixtures/yaml-test-suite/` contains input cases vendored from the official
[YAML test suite](https://github.com/yaml/yaml-test-suite) (`data` branch, commit
`6ad3d2c62885d82fc349026c136ef560838fdf3d`), which is distributed under the MIT
license. All 402 cases (numbered sub-cases counted) are vendored, each as its input
(`in.yaml`), expected JSON projection (`in.json`), and/or expected-error marker
(`error`). `Tests/Fixtures/yaml-test-suite-manifest.yaml` records the provenance and
each case's conformance status. These files are test data, not implementation
source.

View File

@ -7,6 +7,453 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Changed
- Merge the libyaml-port line into main. The ported engine is the sole parser;
the hand-rolled `Scanner`/`TokenEventParser` files and their post-fork
patches are removed. Adaptations made in the merge: the JSON-representability
rules move from the retired legacy `Rule` type onto the Check algebra
(`BuiltinCheck.mappingKeysAreStrings`, `BuiltinCheck.numbersAreFinite` in
`PureYAMLJSON`); the permissive tag-handle pre-pass
(`Scanner.tagHandleIssues`) is removed because the ported engine rejects an
undefined handle at parse time (official suite case QLJ7), so the validation
report carries it as a parse diagnostic; and tests pinned to old-engine
behavior or error text were updated to the reference-verified behavior
(libyaml rejects an explicit-key continuation ending at a value colon, and
suite case QB6E requires rejecting a quoted continuation at column 0).
The unreleased entries below come from two parallel lines of work, merged
in one step: the libyaml-port line (listed first), whose ported engine is
now the sole parsing engine, and the earlier hand-rolled-engine hardening
line (listed second). Engine-internal file references in the second group
are historical; the behaviors they describe are carried by the port and
guarded by the whole-suite conformance gate.
### Changed
- Refresh `docs/YAML_TEST_SUITE_CONFORMANCE.md` to match the manifest: the
headline is 402 passing, 0 gap; the "remaining gaps" narrative is retired; a
closing section records the libyaml-port cutover and the final fix series,
and the manifest is named as the source of truth over the prose.
### Added
- The located parse now reports an exact end position for scalars, not just a start: the
scalar event carries the scanner token's end mark, so a `LocatedValue.scalar`'s range covers
the whole token (end exclusive, one past the last character) instead of being start-anchored.
Collections already spanned open to close. The event dump compares value and style, not marks,
so conformance is unaffected; the whole-suite gates stay green.
- Add an opt-in located parse, `PureYAML.parseLocated(_:)` and
`PureYAML.parseStreamLocated(_:)`, returning a `Model.LocatedValue` tree parallel
in shape to `Value` where every node carries its source span (1-based line and
column) and located children. Collections span from their opening to closing
event, scalars carry a start-anchored span, and aliases and merge-inherited pairs
point at their reference site. The plain `Value`, its equality, and the
conformance oracle are untouched: locations are built only when the streaming
composer is asked for them, so the default value path is byte-identical. A
whole-suite test proves the located projection equals a plain parse for every
passing case, including through merge keys and aliases.
- Begin a faithful Swift port of libyaml's scanner under
`PureYAML.Parsing.LibYAML`, to replace the hand-rolled approximation whose
indentation and simple-key bugs the test suite surfaced. The first piece is the
indentation engine, a line-for-line translation of `yaml_parser_roll_indent` and
`yaml_parser_unroll_indent`: a block collection opens only when content is
strictly more indented than the current level, which fixes the `- a:` followed
by a same-column `b:` mis-nesting by construction. The port is verified against
libyaml (built from the `CYaml` source) and lands as a parallel implementation,
swapped in only when it matches the reference with no regression. See
`docs/LIBYAML_PORT.md`. No public behavior changed yet.
### Fixed
- Reject an under-indented or tab-indented continuation in a multi-line quoted
scalar. A quoted scalar's continuation must be indented past the parent block, so
`"a`\n`b"` (continuation in column 0) and `"bar`\n`\tbaz"` (tab indentation) are
malformed. The error is deferred until the closing quote is found, so an
unterminated quote still reports the missing quote (its root cause) rather than
the indentation symptom. Surpasses libyaml on QB6E and DK95/01, bringing
whole-suite conformance to 402/402.
- Reject a tab used to indent flow-collection content. A tab is never indentation,
so flow content reached through a tab below the continuation indent (`- [`\n
`\tfoo`) is malformed, even though the tab inflates the column. A tab at or past
the indent stays separation, and a tab-only blank line stays blank. Surpasses
libyaml on Y79Y/003.
- Reject a block scalar whose leading empty line is more indented than its first
content line. libyaml auto-detects the deeper indentation and silently drops the
content; the auto-detection now errors when the content line is less indented
than an earlier empty line. Surpasses libyaml on S98Z.
- Reject a multi-line implicit key in a flow sequence. A flow-sequence single-pair
implicit key is single-line (`c-s-implicit-json-key`), so its `:` must sit on the
key's line: both `[ "key"\n :value ]` and `[ key\n : value ]` are malformed. A
flow MAPPING adjacent value may still span lines (`{ "foo"\n :bar }` stays
valid), so the rejection is scoped to flow levels that are sequences, tracked
alongside the flow-level stack. The key/`:` line comparison catches the plain-key
form too, which folds its line break. Surpasses libyaml on ZXT5 and DK4H.
- Reject a block sequence that opens on the same line as an earlier node. A `-`
entry mid-line after an inline mapping value (`key: - a`) or a node property
(`&anchor - x`) is invalid, while a `-` after another `-` stays a valid compact
sequence (`- - a`) and an explicit key's value on a later line (`? k`\n`: - a`)
stays valid. The inline case is told apart by a pending simple key (an explicit
key's has gone stale across the line break). Surpasses libyaml on 5U3A and SY6V.
- Reject a flow-collection continuation line that is not indented past the parent
block. YAML requires multi-line flow content to be more indented than the
enclosing block context, so `flow: [a,` / `b]` (with `b` in column 0) and the
equivalent multi-line flow mapping are malformed; libyaml does not enforce it. A
closing `]`/`}` or a leading `,` may still sit at any indentation. Surpasses
libyaml on 9C9N and VJP3/00.
- Reject content after a `...` document-end marker on the same line. Only
whitespace and a comment may follow `...`; unlike `--- x`, a `... x` cannot
carry inline content. Surpasses libyaml on 3HFZ.
- Resolve a scalar with the non-specific `!` tag as a string. The `!` tag forces
resolution by kind alone, with no implicit typing, so `! 12` is the string "12"
rather than the integer 12. The typed value now honours it (the event stream
already did). Surpasses libyaml on S4JQ.
- Reject a bare `-` before a flow indicator inside a flow collection. Per
`ns-plain-first` a `-` begins a plain scalar in flow only when a flow-safe
character follows; `[-]` and `[-, -]` are malformed, not a sequence of `-`
scalars. libyaml reads the `-` as a scalar. Surpasses libyaml on G5U8 and YJV2.
- Reject a comment that is not preceded by whitespace. A `#` only begins a comment
after a space, a tab, or at the start of a line; a `#` adjacent to the previous
token (`key: "value"# c`, `[ a ]#c`, `[ a,#c`, a `>#c` block scalar header) is
malformed. The check uses the character immediately before the `#`, since a plain
scalar consumes its own trailing blanks. libyaml accepts the adjacent form.
Surpasses libyaml on 9JBA, CVW2, SU5Z, and X4QW. Whole-suite conformance is now
388/402.
- Reject a directive that follows a document without an explicit `...` end marker.
Directives apply to the document that follows, so the previous document must be
closed with `...` before them; after an implicit end a `%YAML`/`%TAG`/reserved
directive is now an error. Also reject a `%YAML` version followed immediately by
`#` with no separating space (`%YAML 1.1#...`), since a comment requires leading
separation. Restores reject-side conformance lost when the libyaml port became
the sole engine (9HCY, EB22, MUS6/00, MUS6/01, RHX7).
- Refresh `yaml-test-suite-manifest.yaml` (now version 3): the ported engine's
whole-suite conformance is 384/402, up from the recorded 331, after the
canonical-event and reject-side work. 53 cases the manifest still marked as gaps
now pass.
- Keep the final line break of a block scalar that ends at end of input without a
trailing newline. A whitespace-only last line still counts as one line break
under keep (`+`) chomping, so `- |+` followed by a spaces-only unterminated line
yields `"\n"` rather than `""`. With this the ported engine emits the exact
canonical event stream for every valid case in the vendored YAML test suite
(308/308), surpassing libyaml (252) and the previous hand-rolled parser (267).
Surpasses libyaml on JEF9/02.
- End a root-level block scalar at a `---` or `...` document marker in column 0.
Now that a root block scalar can take indentation 0, its content loop reaches
column-0 lines, so a marker line must end the scalar instead of being folded
into it. Without this a `--- |` literal swallowed the following `...`, the next
directives, and later documents. Surpasses libyaml on W4TN.
- Fold over a tab-only blank line inside a multi-line plain scalar. A plain
scalar may span blank lines, and a blank line that contains a tab is still
blank, so the tab is not indentation. libyaml rejects any tab below the scalar
indentation; the scanner now rejects it only when it indents real content, not
when the rest of the line is blank. Surpasses libyaml on DK95/04.
- Accept a tab as in-line separation while still rejecting it as indentation. A
tab after a `-`, `?`, or `:` indicator, or before a flow collection, is valid
separation, but libyaml refuses to skip it whenever a simple key is allowed and
reports "found character that cannot start any token". The scanner now skips
tabs in the inter-token whitespace and records whether one preceded the token;
a block indicator or an implicit mapping key reached through a tab is rejected
(the tab would be indentation), while a scalar value or flow collection is
accepted. So `- foo:\tbar`, `-\t-1`, `: -\tb`, `\t[ ]`, `\t{}`, and a
space-then-tab continuation parse, while tab-indented keys and `-\t-` style
nested entries are still rejected. Surpasses libyaml on 6BCT, 6CA3, A2M4,
DK95/00, Q5MG, and Y79Y/010.
- Parse a flow sequence entry that begins with `:` as a single-pair mapping with
an empty key (`[ : v ]`). libyaml hands the leading VALUE token to its node
parser and fails. The flow-sequence-entry parser now opens the implicit mapping
on a leading `:` and leaves the indicator for the value state, so the value is
parsed rather than dropped. Surpasses libyaml on CFD4.
- Apply the YAML 1.2 flow `:` rule instead of libyaml's "always a value
indicator in flow". In a flow collection a `:` only ends a node when it is
followed by whitespace, the end of input, or a flow indicator, or when it is
adjacent to a JSON key (a quoted scalar or a flow collection used as the key).
Otherwise the `:` is plain-scalar content. So `[:x]` and `{x: :x}` keep the
scalar `:x`, `http://foo.com` and `::vector` stay single scalars, `"key":value`
and `{JSON: like}:adjacent` bind the value across the adjacent `:`, and
`omitted value:,` is a key with an empty value rather than a scan error. The
pending simple key now records whether it is a JSON node, and the plain scalar
scanner ends (rather than rejecting) at a `:` before a flow indicator. Surpasses
libyaml on 4ABK, 58MP, 5T43, DBG4, FRK4, and HM87/00, with 9MMW (mixed
adjacent/separate keys) still correct.
- Treat `?` in a flow collection as plain-scalar content unless it is followed by
whitespace, the end of input, or a flow indicator. libyaml makes every `?` in
flow context a mapping-key indicator, dropping the `?` from `?foo`, but
`ns-plain-first` lets a `?` followed by a flow-safe character begin a plain
scalar. So `{ ?foo: bar }` keys on `?foo` and `[?x]` is the scalar `?x` rather
than an empty single-pair mapping. Surpasses libyaml on 652Z and HM87/01.
- Accept a tab as block-scalar content once the indentation is satisfied. The
block-scalar break scanner rejected any tab seen while auto-detecting the
content indentation, but a tab at or beyond the content column is content, not
indentation. The tab is now an error only below the content column (the parent
indent + 1 floor while auto-detecting), so `foo: |-` / `bar` lines whose
content begins with a tab parse, while a tab used as indentation (a tab-only
line shallower than the content) is still rejected. Surpasses libyaml on
96NN/00, 96NN/01, R4YG, and Y79Y/001.
- Auto-detect indentation 0 for a root-level block scalar. The block-scalar
scanner used 0 as the "indentation not yet detected" sentinel and floored the
detected indent at 1 (as libyaml does in `scan_block_scalar_breaks`), so a
`--- >` or `--- |` whose content begins in column 0 detected no content and the
following line was misread as a new document ("did not find expected document
start"). The sentinel is now -1, leaving 0 a representable detected indent, and
the floor is the parent indent + 1 (0 at the document root). Surpasses libyaml
on DK3J, FP8R, and M7A3.
- Parse block mappings with empty (null) keys. A `:` value indicator with no
preceding key on its line denotes a mapping entry whose key is empty, which
YAML 1.2 permits but libyaml rejects with "did not find expected key". The
block-mapping-key parser now emits the empty scalar key and lets the value
state consume the indicator, fixing the canonical event streams for `: a`
style mappings, explicit empty keys (`? &a a` / `: *a`), empty keys nested in
sequences (`- :`, `- ? : x`), and mixed empty-key/value entries. Surpasses
libyaml on these seven test-suite cases (2JQS, 6M2F, M2N8/00, NHX8, S3PD,
SM9W/01, UKK6/00).
- Parse flow mappings with empty (null) keys. The same `:`-without-a-key entry
is valid inside `{ ... }`, where libyaml hands the bare VALUE token to its node
parser and fails with "did not find expected node content". The flow-mapping-key
parser now emits the empty scalar key for a leading `:`, so `{ : }` and
`{ key: value, : empty key }` parse, completing the multi-document NKF9 case.
- Emit no document for a stream whose only content is comments or whitespace. A
comment-only or empty stream has zero documents in YAML 1.2 (matching libyaml
and the test suite), but the event stream was emitting a spurious empty null
document. Found by differential testing against libyaml over real-world YAML
(a comment-only `values.yaml`) in addition to the canonical event gate. The
single-document value API still maps an empty stream to `null`, so
`PureYAML.parse("")` is unchanged. Fixes 8G76, 98YD, and AVM7, raising exact
canonical-event conformance to 267 of 308.
- Strip the trailing folded line break from a flow-context plain scalar. A plain
scalar inside `[...]`/`{...}` whose next line begins with a flow indicator
(`,` `]` `}`) or the `:` value indicator was retaining the line break folded to
a trailing space (for example `[\nfoo: bar\n]` produced the value `"bar "`),
which YAML 1.2 section 7.3.3 forbids. Derived from the plain-scalar production
and verified against the canonical event gate: fixes 10 official YAML test suite
cases (4FJ6, 4MUZ/02, CT4Q, LP6E, NKF9, QF4Y, VJP3/01, Y79Y/002, ZF4X, ZK9H)
with no regression, raising exact canonical-event conformance to 264 of 308.
### Added
- Add an authoritative canonical-event conformance gate. The parser's event
stream is serialized into the official YAML test suite event (yeast) format and
compared exactly to each case's vendored `test.event`, which checks anchors,
tags, scalar styles, flow vs block, and explicit document markers for every
valid case (including the roughly half with no `in.json` projection). This
replaces the JSON-projection oracle, which was lossy: measured against ground
truth, 254 of 308 valid suite cases emit exactly correct events (40 emit a
wrong event stream, 14 false-reject), where the JSON proxy had reported a higher
figure. The gate records the gap set and fails on any regression or unrecorded
fix. All 402 cases' `test.event` files are vendored alongside their inputs.
### Changed
- Record whether a document boundary was explicit in the parser `Event` model
(`documentStart`/`documentEnd` now carry an `explicit` flag), so a `---`/`...`
marker is distinguished from an implicit boundary. This is required for exact
canonical-event comparison and resolves 89 cases that were previously correct
but unverifiable.
### Fixed
- Make the YAML test suite conformance gate comprehensive and repair the
regressions it revealed. The gap-only manifest (which deleted a case once fixed,
so nothing guarded it afterward and later work could silently regress it) is
replaced by a full manifest of all 402 official cases
(`Tests/Fixtures/yaml-test-suite-manifest.yaml`), each with its `id`, `name`,
`localPath`, conformant `expected` behavior, `oracle`, `byteCount`, `lineCount`,
and a `status` (`passing` | `gap`). Every case's files are vendored under
`Tests/Fixtures/yaml-test-suite/`. The gated sweep
(`PUREYAML_RUN_YAML_TEST_SUITE=1`) now asserts every `passing` case still passes
and every `gap` still reproduces, so a regression in any passing case fails the
sweep. Result at the pinned commit: **329 of 402 passing, 73 gap.**
- Stop over-rejecting tabs. A tab is invalid only as block-structure indentation
(`scanner.c` `scan_to_next_token` / `scan_block_scalar_breaks` /
`scan_plain_scalar`). Block indentation is now measured in spaces only; a tab is
left for the token scanner, which allows it inside block-scalar content once a
leading space fixes the content column (96NN/00, 96NN/01, R4YG, Y79Y/001), as
separation before a scalar value (DK95/00) or a flow node (6CA3, Q5MG, J3BT), and
on a whitespace-only blank line (DK95/03, DK95/07), while still rejecting genuine
tab indentation (4EJS, Y79Y/000, Y79Y/004-008, DK95/06). The batch scanner and the
incremental token cursor were unified onto a single `scanStep`, since their
divergence previously let the streaming parser miss a tab error the batch scanner
caught.
- Restore anchors/aliases as mapping keys for `26DV`: an anchor on a `key:` value
line followed by an indented `*alias :` key now anchors the nested mapping rather
than the alias key (which cannot carry properties); `&b *alias` on one line is
still rejected (SU74).
- Restore flow bare-tag/anchor keys for `WZ62` (`{ foo : !!str, !!str : bar, }` to
`{"foo":"", "":"bar"}`): a flow key introduced by a spliced `mappingKey` uses the
same key parser as the implicit form, so a bare tag before the `:` composes an
empty tagged scalar key instead of a mapping-as-key.
- Complete flow-collection (`[...]`, `{...}`) parsing for keys, empty nodes, and
multi-line scalars, fixing 10 official YAML test suite cases and removing them
from the conformance gap manifest. The scanner now implements libyaml's
simple-key mechanism inside flow context: a potential key position is recorded
before a plain or quoted scalar, an anchor or tag, or a nested `[`/`{` opener,
and a later `:` value indicator splices a `mappingKey` token in front of that
node so a collection or property-carrying node can be a flow-mapping key. In
flow context a `:` is now always a value indicator and a `?` always an explicit
key indicator regardless of the following character (matching libyaml's
`flow_level || IS_BLANKZ_AT` checks), so `"k":v`, `{k}:v`, and a trailing `?`
parse correctly; the implicit-key forward lookahead is restricted to block
context. A multi-line plain key folds in full before its `:` is read. The flow
parser was rebuilt around libyaml's flow productions (`flow_sequence_entry`,
`flow_mapping_key`/`value`), composing empty scalar keys and values where
absent. The streaming token cursor holds an open flow collection's tokens until
it closes so the retroactive key splice stays valid. Fixed false-rejects 4FJ6,
8KB6, 9BXH, 9MMW, CN3R, DFF7, FRK4, NJ66, UT92 and corrected values for CT4Q.
The flow-node-as-block-mapping-key cases LX3P and 6BFJ remain tracked (they need
the block-context simple-key splice, a separate change).
- Parse multi-line plain scalars and block scalars at the document root, fixing 35
official YAML test suite cases and removing them from the conformance gap
manifest. The scanner now tracks the enclosing block-collection indent
(libyaml's `parser->indent`) separately from per-line indentation, defaulting to
-1 at the bare document root, so a plain scalar's continuation lines fold using
the spec rule "more indented than the enclosing block": a root scalar folds its
column-0 continuations (single break to space, blank line to newline, per-line
whitespace stripped) while a scalar inside a collection still ends at a sibling
line. A `---` document-start marker may now be followed by the document's root
node on the same line (`--- foo`, `--- [1, 2]`, `--- |`, `--- >`), and a block
scalar whose body sits at the document root's column 0 captures that body and
ends at the next `---`/`...` marker. Fixed false-rejects 35KP, 4CQQ, 4Q9F, 6JQW,
82AN, 93WF, 96L6, 9MQT/00, 9YRD, B3HG, DK3J, EX5H, EXG3, FP8R, RZT7, T26H, UGM3,
XLQ9; corrected values for 27NA, 2G84/02, 2G84/03, 4WA9, 6FWR, 6LVF, A2M4, K54U,
KSS4, L383, T5N4, W4TN; and started correctly rejecting the invalid 9KBC, ZVH3,
9MQT/01. The still-imperfect M7A3 (multi-document `...` splitting) and NB6Z (tabs
on empty plain-scalar lines) now parse but with mismatched values and stay
tracked. Added root multi-line plain scalar regression tests.
- Escape carriage returns and line feeds in double-quoted output per unicode
scalar rather than per Character. A CR+LF pair forms a single Swift extended
grapheme cluster, so the previous Character-level escape switch matched neither
`\r` nor `\n` and emitted the pair unescaped; the folding parser then read it
back differently, breaking parse/dump/parse stability for values containing
CRLF (observed on the Zoom OpenAPI corpus seed). Added a dumper regression test.
- Tightened error detection so 15 more official YAML test suite cases are handled
correctly, repairing four error-detection regressions and removing them from the
conformance gap manifest. Tabs are now rejected when they would supply
block-structure indentation: a tab at the start of a block line, a tab in a
block scalar's indentation, and a tab in separation position that precedes a
nested block node (a `-` entry, a `?` key, or a compact mapping key) all fail,
while a tab used purely as in-line separation before a scalar value stays valid
(4EJS, Y79Y/000, Y79Y/004, Y79Y/005, Y79Y/006, Y79Y/008, Y79Y/009 now reject;
6BCT and Y79Y/010 now parse with matching values; DC7X and J3BT no longer
falsely reject). A `---`/`...` document marker at column 0 is now rejected when
it appears inside an open flow collection (N782) or inside an open quoted scalar
(RXY3, 5TRB) rather than silently ending the document. A second `%YAML`/`%TAG`
directive section that opens a new document without an intervening `...` suffix
is rejected (MUS6/01). An anchor or tag attached to an alias node, and an
anchor-plus-alias on one node, are rejected (SU74, SR86). Remaining
accepted-invalid cases that need `---`-line content scanning or full simple-key
tracking (for example 2G84/00, 2G84/01, 9MQT/01, ZCZ6) are left tracked in the
gap manifest.
- Added YAML directive handling and completed document-boundary support, fixing
three root causes. First, a `%` now begins a directive only in directive
position (before any document content and before the `---` that starts a
document, including immediately after a `...` end marker); everywhere else, and
in any flow context, `%` is ordinary plain text rather than a misfiring
directive. Second, `%YAML` is parsed leniently per the spec: a `1.x` version is
accepted (a higher minor is treated as 1.2 with a warning rather than an
error), while an unsupported major version, a duplicate `%YAML` in one
document, or extra non-comment fields are rejected; reserved and unknown
directives (and `%TAG` handle definitions) are accepted and ignored, and a
directive must be followed by a `---` document start. Third, a `...` end marker
now closes the current document so that following content begins a fresh
document in the multi-document stream parser (`parseStream`), while the
single-document `parse` still rejects multi-document input. This makes 11
official YAML test suite cases parse with matching values (2LFX, 5TYM, 6WLZ,
6ZKB, 7Z25, 9DXL, 9WXW, BEC7, MUS6/03, MUS6/05, MUS6/06), correctly rejects
four previously accepted-invalid cases (9MMA, B63P, H7TQ, SF5V), and unblocks
6LVF and W4TN from parse errors to value comparisons that now depend on
separate scalar gaps (`---`-line content and literal block chomping).
- Rewrote literal (`|`) and folded (`>`) block scalar content handling to follow
YAML 1.2 §8.1 semantics, fixing four root causes: interior blank lines are no
longer dropped from literal scalars; folded scalars now apply true §8.1.3 line
folding (a single break between equally indented non-empty lines folds to a
space, blank lines stay literal newlines, and more-indented lines keep both
their leading whitespace and their surrounding breaks unfolded); tabs inside
block scalar content are preserved instead of being rewritten as spaces (only
the block indentation columns are stripped); and the `+` keep chomping
indicator is now supported alongside clip and strip. The block scalar is now
scanned as a single unit (header line plus all content lines) with indentation
auto-detection, mirroring the reference scanner, rather than reassembled from a
lossy per-line token array. This makes 24 additional official YAML test suite
cases pass: 4QFQ, 5LLU, 6VJK, 7T8X, 96NN/00, 96NN/01, DWX9, F6MC, F8F9, H2RW,
JEF9/00, JEF9/01, K527, K858, L24T/00, L24T/01, M29M, M9B4, MJS9, P2AD, R4YG,
TS54, W9L4, and Y79Y/001.
- Implemented YAML 1.2 flow (quoted) scalar line folding for both single (`'`)
and double (`"`) quoted scalars, mirroring the reference scanner's
`yaml_parser_scan_flow_scalar`. Multi-line quoted scalars now fold correctly:
trailing whitespace before a break is stripped, leading whitespace on a
continuation line is stripped, a single break between non-empty lines folds to
one space, and N consecutive breaks yield N-1 literal line feeds (the first and
last lines keep their own leading and trailing spaces as content). A double
quoted backslash at end of line now suppresses the break entirely (no space, no
newline). Completed the double quoted escape set with `\xXX` (2 hex digits) and
`\UXXXXXXXX` (8 hex digits), and added the previously missing `\0 \a \v \e`,
literal `\<space>`, and the named breaks `\N \_ \L \P`, alongside the existing
`\uXXXX` (including surrogate pairs). This makes 30 additional official YAML
test suite cases pass: 3RLN/00, 3RLN/01, 3RLN/02, 3RLN/03, 3RLN/04, 3RLN/05,
4ZYM, 565N, 5GBF, 6WPF, 7A4E, 8UDB, 9SA2, 9TFX, DE56/00, DE56/01, DE56/02,
DE56/03, DE56/04, DE56/05, DK95/02, DK95/08, G4RS, NAT4, NP9H, PRH3, Q8AD,
T4YY, TL85, and XV9V.
- Composed an empty (null) plain scalar wherever a node is grammatically required
but absent, instead of raising "expected a YAML node". This single production
mirrors `parser.c`'s empty-scalar fallback and covers every position a node can
go missing: after a tag or anchor property with no following node, an absent
mapping value, an absent mapping key (both the plain `: value` form and the
explicit `?` form), and the empty key and value forms inside flow collections.
An empty node carrying an anchor or tag is explicit; a bare absent node with no
property in a grammatically-required position is still a parse error, matching
the reference parser. The empty scalar resolves through the normal scalar
resolver against an empty value, so `!!str` yields `""`, `!!null` and a bare
empty yield null, and an empty key resolves like any other empty scalar.
Implemented in `Sources/Parsing/TokenEventParser.EmptyNode.swift` with the
flow and block call sites in the sibling `TokenEventParser.*` files. This makes
20 additional official YAML test suite cases pass: 2JQS, 2XXW, 5WE3, 6M2F,
7W2P, BU8L, CFD4, JTV5, KK5P, LE5A, M2N8/00, NHX8, NKF9, PW8X, RZP5, S3PD,
SM9W/01, UKK6/02, WZ62, and XW4D. CT4Q now parses but still mismatches its
expected JSON (a separate flow multi-line plain-scalar folding gap) and is
re-recorded as `json-mismatch`.
- Accepted node properties (anchor, tag) and aliases in mapping-key position, so
a mapping key is parsed as the full node it is: it may carry `&anchor` and/or
`!tag` properties, and it may itself be an `*alias`. Fixed four root causes.
(1) The scanner now terminates an anchor or alias name at the first character
outside libyaml's `IS_ALPHA` set (ASCII letters, digits, `-`, `_`), mirroring
`yaml_parser_scan_anchor`, so `*alias:` reads the alias `alias` followed by a
`:` mapping indicator instead of swallowing the colon into the name. (2) The
block-mapping parser now begins a pair at leading properties or an alias, not
only at the key indicator the scanner emits after them, and properties on the
same line as the first key attach to that key node rather than to the mapping
(`&a a: b` anchors the key `a`, while `&a` alone on its own line anchors the
map). (3) An anchor or tag at value position with no node following on the same
line now composes an empty (null) value that still carries the property, so
`a: &anchor` registers `anchor` as null and a later `*anchor` resolves. (4) An
anchored scalar now registers its resolved value (so an empty anchored scalar
registers null, not `""`). The implicit-key scanner also now recognizes a key
with trailing spaces before its `:` (`"top1" :`). Implemented in
`Sources/Parsing/Scanner.swift`, `Scanner.Helpers.swift`,
`TokenEventParser.swift`, `TokenEventParser.EmptyNode.swift`,
`TokenEventParser.Helpers.swift`, `TokenEventParser.Block.swift`,
`TokenEventParser.Flow.swift`, `TokenCursor.swift`, and the anchor registration
in `EventStreamComposer.swift` and `EventComposer.swift`. This makes 10
additional official YAML test suite cases pass: 26DV, 6KGN, 74H7, 87E4, 9KAX,
E76Z, G9HC, HMQ5, LQZ7, and ZWK4 (G9HC is an invalid-anchor case that is now
correctly rejected). 2SXE now parses but still mismatches its expected JSON (a
separate compact block-mapping structural gap) and is re-recorded as
`json-mismatch`.
### Added
- Track conformance against the official YAML test suite
(github.com/yaml/yaml-test-suite, `data` branch). The 182 cases PureYAML does
not yet handle correctly are vendored under `Tests/Fixtures/yaml-test-suite/`
with a machine-readable manifest (`Tests/Fixtures/yaml-test-suite-gaps.yaml`)
recording each gap's category and current behavior. A new opt-in sweep
(`bash scripts/check-yaml-test-suite.sh`, gated behind
`PUREYAML_RUN_YAML_TEST_SUITE=1`) reproduces every recorded gap so future fixes
must update the manifest. `docs/YAML_TEST_SUITE_CONFORMANCE.md` documents the
measured result (220/402 passing) and the ordered plan for closing the gaps.
### Added
- Opt-in located parse: `parseStreamLocated` returns a source-span projection of every document,

View File

@ -9,20 +9,36 @@ let package = Package(
name: "PureYAML",
targets: ["PureYAML"],
),
.library(
name: "PureYAMLJSON",
targets: ["PureYAMLJSON"],
),
],
dependencies: [],
targets: [
.target(
name: "PureYAML",
path: "Sources",
exclude: ["PureYAMLJSON"],
),
.target(
name: "PureYAMLJSON",
dependencies: ["PureYAML"],
path: "Sources/PureYAMLJSON",
),
.testTarget(
name: "PureYAMLTests",
dependencies: ["PureYAML"],
path: "Tests",
exclude: ["PureYAMLJSONTests"],
resources: [
.copy("Fixtures"),
],
),
.testTarget(
name: "PureYAMLJSONTests",
dependencies: ["PureYAMLJSON"],
path: "Tests/PureYAMLJSONTests",
),
],
)

View File

@ -1,11 +1,11 @@
# PureYAML
[![Gates](https://github.com/mihaelamj/PureYAML/actions/workflows/gates.yml/badge.svg?branch=main)](https://github.com/mihaelamj/PureYAML/actions/workflows/gates.yml)
[![macOS](https://github.com/mihaelamj/PureYAML/actions/workflows/macos.yml/badge.svg?branch=main)](https://github.com/mihaelamj/PureYAML/actions/workflows/macos.yml)
[![Linux](https://github.com/mihaelamj/PureYAML/actions/workflows/linux.yml/badge.svg?branch=main)](https://github.com/mihaelamj/PureYAML/actions/workflows/linux.yml)
[![Windows](https://github.com/mihaelamj/PureYAML/actions/workflows/windows.yml/badge.svg?branch=main)](https://github.com/mihaelamj/PureYAML/actions/workflows/windows.yml)
[![WASM](https://github.com/mihaelamj/PureYAML/actions/workflows/wasm.yml/badge.svg?branch=main)](https://github.com/mihaelamj/PureYAML/actions/workflows/wasm.yml)
[![Conformance](https://github.com/mihaelamj/PureYAML/actions/workflows/conformance.yml/badge.svg?branch=main)](https://github.com/mihaelamj/PureYAML/actions/workflows/conformance.yml)
[![Gates](https://codeberg.org/Mihaela/PureYAML/actions/workflows/gates.yml/badge.svg?branch=main)](https://codeberg.org/Mihaela/PureYAML/actions?workflow=gates.yml)
[![macOS](https://codeberg.org/Mihaela/PureYAML/actions/workflows/macos.yml/badge.svg?branch=main)](https://codeberg.org/Mihaela/PureYAML/actions?workflow=macos.yml)
[![Linux](https://codeberg.org/Mihaela/PureYAML/actions/workflows/linux.yml/badge.svg?branch=main)](https://codeberg.org/Mihaela/PureYAML/actions?workflow=linux.yml)
[![Windows](https://codeberg.org/Mihaela/PureYAML/actions/workflows/windows.yml/badge.svg?branch=main)](https://codeberg.org/Mihaela/PureYAML/actions?workflow=windows.yml)
[![WASM](https://codeberg.org/Mihaela/PureYAML/actions/workflows/wasm.yml/badge.svg?branch=main)](https://codeberg.org/Mihaela/PureYAML/actions?workflow=wasm.yml)
[![Conformance](https://codeberg.org/Mihaela/PureYAML/actions/workflows/conformance.yml/badge.svg?branch=main)](https://codeberg.org/Mihaela/PureYAML/actions?workflow=conformance.yml)
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
PureYAML is a dependency-free YAML package written entirely in Swift.

View File

@ -356,9 +356,13 @@ extension PureYAML.Emitting.Dumper {
}
func quote(_ value: String) -> String {
// Iterate over unicode scalars, not Characters: a CR+LF pair forms a
// single extended grapheme cluster in Swift, so a Character-level switch
// would match neither "\r" nor "\n" and emit the line break unescaped,
// which does not round-trip through the folding parser.
var output = "\""
for character in value {
switch character {
for scalar in value.unicodeScalars {
switch scalar {
case "\"":
output += "\\\""
case "\\":
@ -370,7 +374,7 @@ extension PureYAML.Emitting.Dumper {
case "\t":
output += "\\t"
default:
output.append(character)
output.unicodeScalars.append(scalar)
}
}
output += "\""

View File

@ -2,8 +2,8 @@ extension PureYAML.Parsing {
enum Event: Equatable, CustomStringConvertible {
case streamStart(mark: Mark)
case streamEnd(mark: Mark)
case documentStart(mark: Mark)
case documentEnd(mark: Mark)
case documentStart(explicit: Bool, mark: Mark)
case documentEnd(explicit: Bool, mark: Mark)
case sequenceStart(anchor: String?, tag: String?, style: CollectionStyle, mark: Mark)
case sequenceEnd(mark: Mark)
case mappingStart(anchor: String?, tag: String?, style: CollectionStyle, mark: Mark)
@ -17,10 +17,10 @@ extension PureYAML.Parsing {
"streamStart @\(mark)"
case let .streamEnd(mark):
"streamEnd @\(mark)"
case let .documentStart(mark):
"documentStart @\(mark)"
case let .documentEnd(mark):
"documentEnd @\(mark)"
case let .documentStart(explicit, mark):
"documentStart explicit=\(explicit) @\(mark)"
case let .documentEnd(explicit, mark):
"documentEnd explicit=\(explicit) @\(mark)"
case let .sequenceStart(anchor, tag, style, mark):
"sequenceStart anchor=\(describe(anchor)) tag=\(describe(tag)) style=\(style) @\(mark)"
case let .sequenceEnd(mark):
@ -39,6 +39,36 @@ extension PureYAML.Parsing {
}
extension PureYAML.Parsing.Event {
/// Re-emit the event with libyaml's 0-based line/column mark converted to the
/// package's 1-based convention (the byte index is already 0-based in both).
func bumpingMark() -> Self {
let source = mark
let bumped = PureYAML.Parsing.Mark(line: source.line + 1, column: source.column + 1, index: source.index)
switch self {
case .streamStart:
return .streamStart(mark: bumped)
case .streamEnd:
return .streamEnd(mark: bumped)
case let .documentStart(explicit, _):
return .documentStart(explicit: explicit, mark: bumped)
case let .documentEnd(explicit, _):
return .documentEnd(explicit: explicit, mark: bumped)
case let .sequenceStart(anchor, tag, style, _):
return .sequenceStart(anchor: anchor, tag: tag, style: style, mark: bumped)
case .sequenceEnd:
return .sequenceEnd(mark: bumped)
case let .mappingStart(anchor, tag, style, _):
return .mappingStart(anchor: anchor, tag: tag, style: style, mark: bumped)
case .mappingEnd:
return .mappingEnd(mark: bumped)
case let .scalar(value, anchor, tag, style, _, endMark):
let bumpedEnd = PureYAML.Parsing.Mark(line: endMark.line + 1, column: endMark.column + 1, index: endMark.index)
return .scalar(value: value, anchor: anchor, tag: tag, style: style, mark: bumped, endMark: bumpedEnd)
case let .alias(anchor, _):
return .alias(anchor: anchor, mark: bumped)
}
}
var isMappingEnd: Bool {
if case .mappingEnd = self {
return true
@ -63,8 +93,6 @@ extension PureYAML.Parsing.Event {
var mark: PureYAML.Parsing.Mark {
switch self {
case let .alias(_, mark),
let .documentEnd(mark),
let .documentStart(mark),
let .mappingEnd(mark),
let .mappingStart(_, _, _, mark),
let .scalar(_, _, _, _, mark, _),
@ -73,6 +101,9 @@ extension PureYAML.Parsing.Event {
let .streamEnd(mark),
let .streamStart(mark):
mark
case let .documentEnd(_, mark),
let .documentStart(_, mark):
mark
}
}

View File

@ -41,7 +41,7 @@ extension PureYAML.Parsing {
}
return false
}
if case let .documentStart(mark)? = current {
if case let .documentStart(_, mark)? = current {
throw PureYAML.Parsing.ParseError.unsupportedMultiDocumentStream(line: mark.line)
}
_ = try expectStreamEnd()
@ -173,7 +173,10 @@ extension PureYAML.Parsing.EventComposer {
)
}
index += 1
store(.string(value), anchor: anchor)
if let anchor {
let model = try composeScalar(value, tag: tag, style: style, mark: .start)
store(model, anchor: anchor)
}
return ComposedMappingKey(
keyNode: .string(value),
scalarValue: value,

View File

@ -30,9 +30,9 @@ extension PureYAML.Parsing {
didSeeStreamStart = true
case .streamEnd:
didSeeStreamEnd = true
case let .documentStart(mark):
case let .documentStart(_, mark):
try startDocument(mark: mark)
case let .documentEnd(mark):
case let .documentEnd(_, mark):
try endDocument(mark: mark)
case let .scalar(value, anchor, tag, style, mark, endMark):
try consumeScalar(value, anchor: anchor, tag: tag, style: style, marks: (mark, endMark))
@ -104,7 +104,6 @@ private extension PureYAML.Parsing.EventStreamComposer {
}
anchors = [:]
currentDocumentValue = nil
currentDocumentLocated = nil
}
mutating func endDocument(mark: PureYAML.Parsing.Mark) throws {
@ -128,8 +127,6 @@ private extension PureYAML.Parsing.EventStreamComposer {
) throws {
let model = try scalarParser.composeScalarValue(value, tag: tag, style: style, mark: marks.start)
if let anchor {
// Register the resolved node, not the raw text: an alias resolves to the anchored node's
// value, so an empty scalar anchors null and a typed scalar (`&a 42`) anchors that value.
anchors[anchor] = model
}
try complete(.init(
@ -198,7 +195,7 @@ private extension PureYAML.Parsing.EventStreamComposer {
guard container.pendingKey == nil else {
throw unexpectedEvent(expected: "mapping value", mark: mark)
}
let pairs = PureYAML.Parsing.MergeResolution.mergedPairs(
let pairs = mergedPairs(
from: container.mergeSources,
localPairs: container.localPairs,
)
@ -297,7 +294,7 @@ private extension PureYAML.Parsing.EventStreamComposer {
guard !node.sequenceItems.isEmpty else {
return try values.map { value in
guard case let .mapping(mapping) = value else {
throw PureYAML.Parsing.MergeResolution.invalidMergeValue(mark: node.mark)
throw invalidMergeValue(mark: node.mark)
}
return mapping.pairs
}
@ -305,15 +302,51 @@ private extension PureYAML.Parsing.EventStreamComposer {
return try values.enumerated().map { index, value in
let mark = node.sequenceItems[index].mark
guard case let .mapping(mapping) = value else {
throw PureYAML.Parsing.MergeResolution.invalidMergeValue(mark: mark)
throw invalidMergeValue(mark: mark)
}
return mapping.pairs
}
case .bool, .double, .int, .null, .string:
throw PureYAML.Parsing.MergeResolution.invalidMergeValue(mark: node.mark)
throw invalidMergeValue(mark: node.mark)
}
}
func mergedPairs(
from mergeSources: [[PureYAML.Model.Pair]],
localPairs: [PureYAML.Model.Pair],
) -> [PureYAML.Model.Pair] {
guard !mergeSources.isEmpty else {
return localPairs
}
let localKeys = Set(localPairs.map(\.keyNode))
var inheritedKeys = Set<PureYAML.Model.Key>()
var inheritedPairs: [PureYAML.Model.Pair] = []
for source in mergeSources {
var sourceKeys = Set<PureYAML.Model.Key>()
for pair in source {
guard !localKeys.contains(pair.keyNode),
!inheritedKeys.contains(pair.keyNode)
else {
continue
}
inheritedPairs.append(pair)
sourceKeys.insert(pair.keyNode)
}
inheritedKeys.formUnion(sourceKeys)
}
return inheritedPairs + localPairs
}
func invalidMergeValue(
mark: PureYAML.Parsing.Mark,
) -> PureYAML.Parsing.ParseError {
PureYAML.Parsing.ParseError.invalidMergeValue(
line: mark.line,
column: mark.column,
)
}
func unexpectedEvent(
expected: String,
mark: PureYAML.Parsing.Mark,

View File

@ -0,0 +1,103 @@
extension PureYAML.Parsing.LibYAML {
/// Port of libyaml's character-class macros (`IS_*` in `yaml_private.h`).
/// libyaml inspects UTF-8 bytes; this operates on `Unicode.Scalar`, where the
/// multi-byte line breaks NEL (`U+0085`), LS (`U+2028`), and PS (`U+2029`) are
/// single scalars. End of input is `nil`, which is libyaml's `\0` sentinel, so
/// `isZ(nil)` is true and every other predicate is false for `nil`.
enum CharacterClass {
/// `IS_Z`: the end of the input (libyaml's `\0`).
static func isZ(_ scalar: Unicode.Scalar?) -> Bool {
scalar == nil
}
/// `IS_SPACE`.
static func isSpace(_ scalar: Unicode.Scalar?) -> Bool {
scalar == " "
}
/// `IS_TAB`.
static func isTab(_ scalar: Unicode.Scalar?) -> Bool {
scalar == "\t"
}
/// `IS_BLANK`: space or tab.
static func isBlank(_ scalar: Unicode.Scalar?) -> Bool {
isSpace(scalar) || isTab(scalar)
}
/// `IS_BREAK`: CR, LF, NEL, LS, or PS.
static func isBreak(_ scalar: Unicode.Scalar?) -> Bool {
switch scalar {
case "\r", "\n", "\u{85}", "\u{2028}", "\u{2029}": true
default: false
}
}
/// `IS_BREAKZ`: a line break or the end of input.
static func isBreakZ(_ scalar: Unicode.Scalar?) -> Bool {
isBreak(scalar) || isZ(scalar)
}
/// `IS_SPACEZ`: a space, a line break, or the end of input.
static func isSpaceZ(_ scalar: Unicode.Scalar?) -> Bool {
isSpace(scalar) || isBreakZ(scalar)
}
/// `IS_BLANKZ`: blank (space or tab), a line break, or the end of input.
static func isBlankZ(_ scalar: Unicode.Scalar?) -> Bool {
isBlank(scalar) || isBreakZ(scalar)
}
/// A flow collection indicator (`,`, `[`, `]`, `{`, `}`); the YAML
/// `c-flow-indicator` set excluded from `ns-plain-safe` in flow context.
static func isFlowIndicator(_ scalar: Unicode.Scalar?) -> Bool {
switch scalar {
case ",", "[", "]", "{", "}": true
default: false
}
}
/// `IS_DIGIT`.
static func isDigit(_ scalar: Unicode.Scalar?) -> Bool {
guard let scalar else { return false }
return scalar >= "0" && scalar <= "9"
}
/// `IS_ALPHA`: a digit, ASCII letter, `_`, or `-` (the directive, anchor,
/// and tag-handle name characters).
static func isAlpha(_ scalar: Unicode.Scalar?) -> Bool {
guard let scalar else { return false }
return (scalar >= "0" && scalar <= "9")
|| (scalar >= "A" && scalar <= "Z")
|| (scalar >= "a" && scalar <= "z")
|| scalar == "_"
|| scalar == "-"
}
/// `IS_HEX`: a hexadecimal digit.
static func isHex(_ scalar: Unicode.Scalar?) -> Bool {
guard let scalar else { return false }
return (scalar >= "0" && scalar <= "9")
|| (scalar >= "A" && scalar <= "F")
|| (scalar >= "a" && scalar <= "f")
}
/// `IS_ASCII`.
static func isASCII(_ scalar: Unicode.Scalar?) -> Bool {
guard let scalar else { return false }
return scalar.value <= 0x7F
}
/// `IS_PRINTABLE`: the YAML `c-printable` set.
static func isPrintable(_ scalar: Unicode.Scalar?) -> Bool {
guard let scalar else { return false }
let value = scalar.value
return value == 0x09 || value == 0x0A || value == 0x0D
|| (value >= 0x20 && value <= 0x7E)
|| value == 0x85
|| (value >= 0xA0 && value <= 0xD7FF)
|| (value >= 0xE000 && value <= 0xFFFD)
|| (value >= 0x10000 && value <= 0x10FFFF)
}
}
}

View File

@ -0,0 +1,85 @@
extension PureYAML.Parsing.LibYAML {
/// Port of the indentation stack from `yaml_parser_t` plus
/// `yaml_parser_roll_indent` and `yaml_parser_unroll_indent`. libyaml tracks a
/// single `indent` column (the current block collection's indentation, -1 at
/// the document root) and a stack of the enclosing levels. A block collection
/// is opened only when content is strictly more indented than the current
/// level, which is exactly why `- a:` followed by a same-column `b:` keeps `b`
/// a sibling and leaves `a` an empty value.
struct IndentEngine {
/// `parser->indent`. -1 at the document root, matching libyaml.
private(set) var indent = -1
/// `parser->indents`, the stack of enclosing indentation levels.
private(set) var indents: [Int] = []
/// `parser->flow_level`; indentation is inert inside flow collections.
var flowLevel = 0
/// The token queue (`parser->tokens`) the engine appends to or, for
/// `roll_indent`, inserts a start token into at a recorded position.
private(set) var tokens: [Token] = []
/// `parser->tokens_parsed`, the count already dequeued, used to turn an
/// absolute token number into an index into the live queue.
var tokensParsed = 0
/// Port of `yaml_parser_roll_indent`. In flow context it does nothing.
/// When the column is greater than the current indent, it pushes the
/// current indent, adopts the new column, and emits the block start token.
/// `number == -1` appends; otherwise the token is inserted at the recorded
/// queue position so a `BLOCK-MAPPING-START` precedes a simple key that was
/// already queued.
mutating func rollIndent(
column: Int,
number: Int,
type: TokenType,
mark: PureYAML.Parsing.Mark,
) {
if flowLevel != 0 {
return
}
if indent < column {
indents.append(indent)
indent = column
let token = Token(type: type, startMark: mark, endMark: mark)
if number == -1 {
tokens.append(token)
} else {
tokens.insert(token, at: number - tokensParsed)
}
}
}
/// Port of `yaml_parser_unroll_indent`. In flow context it does nothing.
/// While the current indent is deeper than the column, it emits a
/// `BLOCK-END` and pops the enclosing level, closing every block collection
/// that the dedent ends.
mutating func unrollIndent(column: Int, mark: PureYAML.Parsing.Mark) {
if flowLevel != 0 {
return
}
while indent > column {
tokens.append(Token(type: .blockEnd, startMark: mark, endMark: mark))
indent = indents.removeLast()
}
}
/// `ENQUEUE`: append a token to the queue.
mutating func enqueue(_ token: Token) {
tokens.append(token)
}
/// `QUEUE_INSERT`: insert a token at an absolute token number, used to
/// splice a `KEY` token in front of a node already queued.
mutating func insert(_ token: Token, atTokenNumber number: Int) {
tokens.insert(token, at: number - tokensParsed)
}
/// Dequeue the front token, advancing `tokensParsed` like libyaml's
/// `yaml_parser_scan` does when it hands a token to the parser.
mutating func dequeue() -> Token? {
guard !tokens.isEmpty else {
return nil
}
tokensParsed += 1
return tokens.removeFirst()
}
}
}

View File

@ -0,0 +1,12 @@
extension PureYAML.Parsing.LibYAML {
/// The anchor and tag scanned ahead of a node's content, with the marks needed
/// for events and tag-resolution errors. Mirrors the locals in
/// `yaml_parser_parse_node`.
struct NodeProperties {
var anchor: String?
var tagHandle: String?
var tagSuffix: String?
var startMark: PureYAML.Parsing.Mark
var tagMark: PureYAML.Parsing.Mark
}
}

View File

@ -0,0 +1,109 @@
extension PureYAML.Parsing.LibYAML.Parser {
/// Port of `yaml_parser_parse_block_sequence_entry`.
mutating func parseBlockSequenceEntry(first: Bool) throws -> Event {
if first {
marks.append(peekToken().startMark)
skipToken()
}
let token = peekToken()
if case .blockEntry = token.type {
let mark = token.endMark
skipToken()
let next = peekToken()
if !isBlockEntry(next), !isBlockEnd(next) {
states.append(.blockSequenceEntry)
return try parseNode(block: true, indentlessSequence: false)
}
state = .blockSequenceEntry
return processEmptyScalar(at: mark)
}
if case .blockEnd = token.type {
state = states.removeLast()
_ = marks.popLast()
skipToken()
return .sequenceEnd(mark: token.startMark)
}
throw ParserError(
context: "while parsing a block collection",
problem: "did not find expected '-' indicator",
mark: marks.popLast() ?? token.startMark,
)
}
/// Port of `yaml_parser_parse_indentless_sequence_entry`. The closing token
/// belongs to the enclosing collection, so the SEQUENCE-END is emitted without
/// consuming it.
mutating func parseIndentlessSequenceEntry() throws -> Event {
let token = peekToken()
guard case .blockEntry = token.type else {
state = states.removeLast()
return .sequenceEnd(mark: token.startMark)
}
let mark = token.endMark
skipToken()
let next = peekToken()
if !isBlockEntry(next), !isKey(next), !isValue(next), !isBlockEnd(next) {
states.append(.indentlessSequenceEntry)
return try parseNode(block: true, indentlessSequence: false)
}
state = .indentlessSequenceEntry
return processEmptyScalar(at: mark)
}
/// Port of `yaml_parser_parse_block_mapping_key`.
mutating func parseBlockMappingKey(first: Bool) throws -> Event {
if first {
marks.append(peekToken().startMark)
skipToken()
}
let token = peekToken()
if case .key = token.type {
let mark = token.endMark
skipToken()
let next = peekToken()
if !isKey(next), !isValue(next), !isBlockEnd(next) {
states.append(.blockMappingValue)
return try parseNode(block: true, indentlessSequence: true)
}
state = .blockMappingValue
return processEmptyScalar(at: mark)
}
if case .blockEnd = token.type {
state = states.removeLast()
_ = marks.popLast()
skipToken()
return .mappingEnd(mark: token.startMark)
}
// YAML 1.2 permits an empty (null) key: a VALUE indicator with no
// preceding key denotes a mapping entry whose key is empty. libyaml
// rejects this here; instead emit the empty scalar key and leave the
// VALUE token for `parseBlockMappingValue` to consume.
if case .value = token.type {
state = .blockMappingValue
return processEmptyScalar(at: token.startMark)
}
throw ParserError(
context: "while parsing a block mapping",
problem: "did not find expected key",
mark: marks.popLast() ?? token.startMark,
)
}
/// Port of `yaml_parser_parse_block_mapping_value`.
mutating func parseBlockMappingValue() throws -> Event {
let token = peekToken()
guard case .value = token.type else {
state = .blockMappingKey
return processEmptyScalar(at: token.startMark)
}
let mark = token.endMark
skipToken()
let next = peekToken()
if !isKey(next), !isValue(next), !isBlockEnd(next) {
states.append(.blockMappingKey)
return try parseNode(block: true, indentlessSequence: true)
}
state = .blockMappingKey
return processEmptyScalar(at: mark)
}
}

View File

@ -0,0 +1,70 @@
extension PureYAML.Parsing.LibYAML.Parser {
/// The implied tag handles, mirroring libyaml's `default_tag_directives`.
private var defaultTagDirectives: [(handle: String, prefix: String)] {
[("!", "!"), ("!!", "tag:yaml.org,2002:")]
}
/// Port of `yaml_parser_process_directives`: consume the `%YAML` and `%TAG`
/// directive tokens at the head of a document, validating the version and
/// recording the tag handles, then install the default tag handles. The
/// version is checked against a local so a duplicate `%YAML` is rejected.
mutating func processDirectives() throws {
var version: (major: Int, minor: Int)?
var token = peekToken()
while true {
if case let .versionDirective(major, minor) = token.type {
try recordVersion(major: major, minor: minor, into: &version, mark: token.startMark)
} else if case let .tagDirective(handle, prefix) = token.type {
try appendTagDirective(handle: handle, prefix: prefix, allowDuplicates: false, mark: token.startMark)
} else if case .reservedDirective = token.type {
// A reserved directive is ignored (YAML 1.2 §6.8.1) but still
// consumed here so the explicit-document path requires a `---`.
} else {
break
}
skipToken()
token = peekToken()
}
for directive in defaultTagDirectives {
try appendTagDirective(handle: directive.handle, prefix: directive.prefix, allowDuplicates: true, mark: token.startMark)
}
}
private mutating func recordVersion(major: Int, minor: Int, into version: inout (major: Int, minor: Int)?, mark: Mark) throws {
guard version == nil else {
throw ParserError(context: nil, problem: "found duplicate %YAML directive", mark: mark)
}
// YAML 1.2 §6.8.1: a document whose major version is 1 is parsed even when
// the minor version is newer than 1.2 (with a warning), rather than
// libyaml's hard rejection of any minor other than 1 or 2.
guard major == 1 else {
throw ParserError(context: nil, problem: "found incompatible YAML document", mark: mark)
}
version = (major, minor)
}
/// Port of `yaml_parser_append_tag_directive`: append a handle/prefix mapping,
/// rejecting a redefinition unless duplicates are allowed (the defaults are).
mutating func appendTagDirective(handle: String, prefix: String, allowDuplicates: Bool, mark: Mark) throws {
if tagDirectives.contains(where: { $0.handle == handle }) {
if allowDuplicates {
return
}
throw ParserError(context: nil, problem: "found duplicate %TAG directive", mark: mark)
}
tagDirectives.append((handle, prefix))
}
/// Port of the tag-resolution block in `yaml_parser_parse_node`: an empty
/// handle means the suffix is a verbatim tag; otherwise substitute the handle's
/// registered prefix, failing on an undefined handle.
func resolveTag(handle: String, suffix: String, startMark _: Mark, tagMark: Mark) throws -> String {
guard !handle.isEmpty else {
return suffix
}
guard let directive = tagDirectives.first(where: { $0.handle == handle }) else {
throw ParserError(context: "while parsing a node", problem: "found undefined tag handle", mark: tagMark)
}
return directive.prefix + suffix
}
}

View File

@ -0,0 +1,112 @@
extension PureYAML.Parsing.LibYAML.Parser {
/// Port of `yaml_parser_parse_stream_start`.
mutating func parseStreamStart() throws -> Event {
let token = peekToken()
guard case .streamStart = token.type else {
throw ParserError(context: nil, problem: "did not find expected <stream-start>", mark: token.startMark)
}
state = .implicitDocumentStart
skipToken()
return .streamStart(mark: token.startMark)
}
/// Port of `yaml_parser_parse_document_start`, extended for YAML 1.2: stray
/// `...` markers are skipped, then a bare document is opened when one is allowed
/// (the first document or after an explicit `...`), an explicit document is
/// opened after directives and a `---`, or STREAM-END is emitted. libyaml only
/// allows a bare document as the very first one.
mutating func parseDocumentStart() throws -> Event {
var token = peekToken()
while case .documentEnd = token.type {
skipToken()
token = peekToken()
}
// Directives apply to the document that follows, so the previous document
// must be closed with an explicit `...` end marker before them. After an
// implicit end (`allowBareDocument` is false) a directive is an error.
if isDirective(token), !allowBareDocument {
throw ParserError(
context: nil,
problem: "directives require a preceding document end marker",
mark: token.startMark,
)
}
if allowBareDocument, !isDirectiveOrBoundary(token) {
try processDirectives()
states.append(.documentEnd)
state = .blockNode
return .documentStart(explicit: false, mark: token.startMark)
}
if case .streamEnd = token.type {
state = .end
skipToken()
return .streamEnd(mark: token.startMark)
}
return try openExplicitDocument(startMark: token.startMark)
}
/// The explicit-document branch: directives then a required `---`.
private mutating func openExplicitDocument(startMark: Mark) throws -> Event {
try processDirectives()
let token = peekToken()
guard case .documentStart = token.type else {
throw ParserError(context: nil, problem: "did not find expected <document start>", mark: token.startMark)
}
states.append(.documentEnd)
state = .documentContent
skipToken()
return .documentStart(explicit: true, mark: startMark)
}
/// Port of `yaml_parser_parse_document_content`: an empty scalar stands in for
/// a document with no node, otherwise parse the root node.
mutating func parseDocumentContent() throws -> Event {
let token = peekToken()
if isDirectiveOrBoundary(token) || isDocumentEnd(token) {
state = states.removeLast()
return processEmptyScalar(at: token.startMark)
}
return try parseNode(block: true, indentlessSequence: false)
}
/// Port of `yaml_parser_parse_document_end`: reset the per-document tag
/// directives and emit DOCUMENT-END, explicit only when `...` was present.
mutating func parseDocumentEnd() throws -> Event {
let token = peekToken()
var explicit = false
if case .documentEnd = token.type {
skipToken()
explicit = true
}
tagDirectives.removeAll()
// A bare document may follow only an explicit `...` end marker.
allowBareDocument = explicit
state = .documentStart
return .documentEnd(explicit: explicit, mark: token.startMark)
}
private func isDirectiveOrBoundary(_ token: Token) -> Bool {
switch token.type {
case .versionDirective, .tagDirective, .reservedDirective, .documentStart, .streamEnd:
true
default:
false
}
}
private func isDirective(_ token: Token) -> Bool {
switch token.type {
case .versionDirective, .tagDirective, .reservedDirective:
true
default:
false
}
}
private func isDocumentEnd(_ token: Token) -> Bool {
if case .documentEnd = token.type {
return true
}
return false
}
}

View File

@ -0,0 +1,157 @@
extension PureYAML.Parsing.LibYAML.Parser {
/// Port of `yaml_parser_parse_flow_sequence_entry`. A `?` key inside a flow
/// sequence opens a single-pair flow mapping.
mutating func parseFlowSequenceEntry(first: Bool) throws -> Event {
if first {
marks.append(peekToken().startMark)
skipToken()
}
var token = peekToken()
if !isFlowSequenceEnd(token) {
if !first {
guard isFlowEntry(token) else {
throw ParserError(
context: "while parsing a flow sequence",
problem: "did not find expected ',' or ']'",
mark: marks.popLast() ?? token.startMark,
)
}
skipToken()
token = peekToken()
}
if isKey(token) {
state = .flowSequenceEntryMappingKey
skipToken()
return .mappingStart(anchor: nil, tag: nil, style: .flow, mark: token.startMark)
}
if isValue(token) {
// A leading `:` opens a single-pair mapping with an empty key
// (`[ : v ]`). Keep the VALUE token for the mapping-key state.
state = .flowSequenceEntryMappingKey
return .mappingStart(anchor: nil, tag: nil, style: .flow, mark: token.startMark)
}
if !isFlowSequenceEnd(token) {
states.append(.flowSequenceEntry)
return try parseNode(block: false, indentlessSequence: false)
}
}
state = states.removeLast()
_ = marks.popLast()
skipToken()
return .sequenceEnd(mark: token.startMark)
}
/// Port of `yaml_parser_parse_flow_sequence_entry_mapping_key`.
mutating func parseFlowSequenceEntryMappingKey() throws -> Event {
let token = peekToken()
if !isValue(token), !isFlowEntry(token), !isFlowSequenceEnd(token) {
states.append(.flowSequenceEntryMappingValue)
return try parseNode(block: false, indentlessSequence: false)
}
// An empty key: leave a VALUE token for the value state to consume (so the
// value is parsed); a closing `]` ends the pair with an empty value too.
if isValue(token) || isFlowSequenceEnd(token) {
state = .flowSequenceEntryMappingValue
return processEmptyScalar(at: token.startMark)
}
let mark = token.endMark
skipToken()
state = .flowSequenceEntryMappingValue
return processEmptyScalar(at: mark)
}
/// Port of `yaml_parser_parse_flow_sequence_entry_mapping_value`.
mutating func parseFlowSequenceEntryMappingValue() throws -> Event {
var token = peekToken()
if isValue(token) {
skipToken()
token = peekToken()
if !isFlowEntry(token), !isFlowSequenceEnd(token) {
states.append(.flowSequenceEntryMappingEnd)
return try parseNode(block: false, indentlessSequence: false)
}
}
state = .flowSequenceEntryMappingEnd
return processEmptyScalar(at: token.startMark)
}
/// Port of `yaml_parser_parse_flow_sequence_entry_mapping_end`.
mutating func parseFlowSequenceEntryMappingEnd() throws -> Event {
let token = peekToken()
state = .flowSequenceEntry
return .mappingEnd(mark: token.startMark)
}
/// Port of `yaml_parser_parse_flow_mapping_key`.
mutating func parseFlowMappingKey(first: Bool) throws -> Event {
if first {
marks.append(peekToken().startMark)
skipToken()
}
var token = peekToken()
if !isFlowMappingEnd(token) {
if !first {
guard isFlowEntry(token) else {
throw ParserError(
context: "while parsing a flow mapping",
problem: "did not find expected ',' or '}'",
mark: marks.popLast() ?? token.startMark,
)
}
skipToken()
token = peekToken()
}
if isKey(token) {
return try flowMappingKeyEvent()
}
// YAML 1.2 permits an empty (null) key in a flow mapping: a `:` with
// no preceding key denotes an entry whose key is empty. libyaml hands
// the VALUE token to `parse_node` and fails; instead emit the empty
// scalar key and let `parseFlowMappingValue` consume the indicator.
if isValue(token) {
state = .flowMappingValue
return processEmptyScalar(at: token.startMark)
}
if !isFlowMappingEnd(token) {
states.append(.flowMappingEmptyValue)
return try parseNode(block: false, indentlessSequence: false)
}
}
state = states.removeLast()
_ = marks.popLast()
skipToken()
return .mappingEnd(mark: token.startMark)
}
/// The KEY branch of `parse_flow_mapping_key`: parse the key node, or an empty
/// scalar when a value/entry/end follows the `?` immediately.
private mutating func flowMappingKeyEvent() throws -> Event {
skipToken()
let token = peekToken()
if !isValue(token), !isFlowEntry(token), !isFlowMappingEnd(token) {
states.append(.flowMappingValue)
return try parseNode(block: false, indentlessSequence: false)
}
state = .flowMappingValue
return processEmptyScalar(at: token.startMark)
}
/// Port of `yaml_parser_parse_flow_mapping_value`.
mutating func parseFlowMappingValue(empty: Bool) throws -> Event {
var token = peekToken()
if empty {
state = .flowMappingKey
return processEmptyScalar(at: token.startMark)
}
if isValue(token) {
skipToken()
token = peekToken()
if !isFlowEntry(token), !isFlowMappingEnd(token) {
states.append(.flowMappingKey)
return try parseNode(block: false, indentlessSequence: false)
}
}
state = .flowMappingKey
return processEmptyScalar(at: token.startMark)
}
}

View File

@ -0,0 +1,50 @@
extension PureYAML.Parsing.LibYAML.Parser {
func isBlockEntry(_ token: Token) -> Bool {
if case .blockEntry = token.type {
return true
}
return false
}
func isBlockEnd(_ token: Token) -> Bool {
if case .blockEnd = token.type {
return true
}
return false
}
func isKey(_ token: Token) -> Bool {
if case .key = token.type {
return true
}
return false
}
func isValue(_ token: Token) -> Bool {
if case .value = token.type {
return true
}
return false
}
func isFlowEntry(_ token: Token) -> Bool {
if case .flowEntry = token.type {
return true
}
return false
}
func isFlowSequenceEnd(_ token: Token) -> Bool {
if case .flowSequenceEnd = token.type {
return true
}
return false
}
func isFlowMappingEnd(_ token: Token) -> Bool {
if case .flowMappingEnd = token.type {
return true
}
return false
}
}

View File

@ -0,0 +1,99 @@
extension PureYAML.Parsing.LibYAML.Parser {
typealias NodeProperties = PureYAML.Parsing.LibYAML.NodeProperties
/// Port of `yaml_parser_parse_node`. An ALIAS is emitted directly; otherwise
/// the node's anchor/tag properties are consumed, the tag is resolved, and the
/// content token selects a scalar, a flow/block collection start, or an empty
/// scalar standing in for a property-only node.
mutating func parseNode(block: Bool, indentlessSequence: Bool) throws -> Event {
let token = peekToken()
if case let .alias(name) = token.type {
state = states.removeLast()
skipToken()
return .alias(anchor: name, mark: token.startMark)
}
let props = consumeNodeProperties()
let tag = try props.tagHandle.map {
try resolveTag(handle: $0, suffix: props.tagSuffix ?? "", startMark: props.startMark, tagMark: props.tagMark)
}
return try dispatchNodeContent(block: block, indentlessSequence: indentlessSequence, props: props, tag: tag)
}
/// Consume `properties ::= TAG ANCHOR? | ANCHOR TAG?` ahead of the content.
private mutating func consumeNodeProperties() -> NodeProperties {
var token = peekToken()
var props = NodeProperties(anchor: nil, tagHandle: nil, tagSuffix: nil, startMark: token.startMark, tagMark: token.startMark)
if case let .anchor(name) = token.type {
props.anchor = name
skipToken()
token = peekToken()
if case let .tag(handle, suffix) = token.type {
props.tagHandle = handle
props.tagSuffix = suffix
props.tagMark = token.startMark
skipToken()
}
} else if case let .tag(handle, suffix) = token.type {
props.tagHandle = handle
props.tagSuffix = suffix
props.tagMark = token.startMark
skipToken()
token = peekToken()
if case let .anchor(name) = token.type {
props.anchor = name
skipToken()
}
}
return props
}
/// Select the event for the content token following a node's properties.
private mutating func dispatchNodeContent(block: Bool, indentlessSequence: Bool, props: NodeProperties, tag: String?) throws -> Event {
let token = peekToken()
if indentlessSequence, case .blockEntry = token.type {
state = .indentlessSequenceEntry
return .sequenceStart(anchor: props.anchor, tag: tag, style: .block, mark: props.startMark)
}
if let event = contentEvent(token: token, block: block, props: props, tag: tag) {
return event
}
return try emptyNodeOrError(token: token, block: block, props: props, tag: tag)
}
/// A scalar or a flow/block collection start, or nil if the token is neither.
private mutating func contentEvent(token: Token, block: Bool, props: NodeProperties, tag: String?) -> Event? {
switch token.type {
case let .scalar(value, style):
state = states.removeLast()
skipToken()
return .scalar(value: value, anchor: props.anchor, tag: tag, style: style, mark: props.startMark, endMark: token.endMark)
case .flowSequenceStart:
state = .flowSequenceFirstEntry
return .sequenceStart(anchor: props.anchor, tag: tag, style: .flow, mark: props.startMark)
case .flowMappingStart:
state = .flowMappingFirstKey
return .mappingStart(anchor: props.anchor, tag: tag, style: .flow, mark: props.startMark)
case .blockSequenceStart where block:
state = .blockSequenceFirstEntry
return .sequenceStart(anchor: props.anchor, tag: tag, style: .block, mark: props.startMark)
case .blockMappingStart where block:
state = .blockMappingFirstKey
return .mappingStart(anchor: props.anchor, tag: tag, style: .block, mark: props.startMark)
default:
return nil
}
}
/// An empty scalar when the node carried only properties, else a parse error.
private mutating func emptyNodeOrError(token: Token, block: Bool, props: NodeProperties, tag: String?) throws -> Event {
if props.anchor != nil || tag != nil {
state = states.removeLast()
return .scalar(value: "", anchor: props.anchor, tag: tag, style: .plain, mark: props.startMark, endMark: props.startMark)
}
throw ParserError(
context: block ? "while parsing a block node" : "while parsing a flow node",
problem: "did not find expected node content",
mark: token.startMark,
)
}
}

View File

@ -0,0 +1,148 @@
extension PureYAML.Parsing.LibYAML {
/// A faithful port of libyaml's `parser.c`: a one-event-at-a-time state
/// machine that consumes the scanner's token stream and produces the YAML
/// event stream. The token queue is materialized up front by the scanner, so
/// `peekToken`/`skipToken` index into a fixed array rather than pulling lazily,
/// which is equivalent because simple-key resolution lives entirely in the
/// scanner.
struct Parser {
let tokens: [Token]
var cursor = 0
var state: ParserState = .streamStart
var states: [ParserState] = []
var marks: [Mark] = []
var tagDirectives: [(handle: String, prefix: String)] = []
/// Whether a bare (implicit) document may begin here: true for the first
/// document and after an explicit `...` end marker, false otherwise. YAML
/// 1.2 allows a bare document to follow `...`, which libyaml rejects.
var allowBareDocument = true
init(_ input: String = "") throws {
var scanner = Scanner(input)
tokens = try scanner.scanAll()
}
init(tokens: [Token]) {
self.tokens = tokens
}
/// Drive the state machine until STREAM-END, collecting every event.
mutating func parse() throws -> [Event] {
var events: [Event] = []
while state != .end {
try events.append(stateMachine())
}
return events
}
/// The head of the token queue. STREAM-END is always the final token and
/// the machine halts once it has been consumed, so the clamp is never the
/// value actually parsed; it only avoids an out-of-range index.
func peekToken() -> Token {
tokens[min(cursor, tokens.count - 1)]
}
mutating func skipToken() {
cursor += 1
}
/// Port of `yaml_parser_state_machine`: dispatch on the current state. The
/// flat 24-way switch is split into document/block/flow groups to keep each
/// dispatcher's branching within the complexity budget.
mutating func stateMachine() throws -> Event {
switch state {
case .streamStart, .implicitDocumentStart, .documentStart, .documentContent, .documentEnd:
try documentStateMachine()
case .blockNode, .blockNodeOrIndentlessSequence, .flowNode,
.blockSequenceFirstEntry, .blockSequenceEntry, .indentlessSequenceEntry,
.blockMappingFirstKey, .blockMappingKey, .blockMappingValue:
try nodeStateMachine()
case .flowSequenceFirstEntry, .flowSequenceEntry, .flowSequenceEntryMappingKey,
.flowSequenceEntryMappingValue, .flowSequenceEntryMappingEnd,
.flowMappingFirstKey, .flowMappingKey, .flowMappingValue, .flowMappingEmptyValue:
try flowStateMachine()
case .end:
throw endError()
}
}
private mutating func documentStateMachine() throws -> Event {
switch state {
case .streamStart:
try parseStreamStart()
case .implicitDocumentStart:
try parseDocumentStart()
case .documentStart:
try parseDocumentStart()
case .documentContent:
try parseDocumentContent()
default:
try parseDocumentEnd()
}
}
private mutating func nodeStateMachine() throws -> Event {
switch state {
case .blockNode:
try parseNode(block: true, indentlessSequence: false)
case .blockNodeOrIndentlessSequence:
try parseNode(block: true, indentlessSequence: true)
case .flowNode:
try parseNode(block: false, indentlessSequence: false)
case .blockSequenceFirstEntry:
try parseBlockSequenceEntry(first: true)
case .blockSequenceEntry:
try parseBlockSequenceEntry(first: false)
case .indentlessSequenceEntry:
try parseIndentlessSequenceEntry()
case .blockMappingFirstKey:
try parseBlockMappingKey(first: true)
case .blockMappingKey:
try parseBlockMappingKey(first: false)
default:
try parseBlockMappingValue()
}
}
private mutating func flowStateMachine() throws -> Event {
switch state {
case .flowSequenceFirstEntry:
try parseFlowSequenceEntry(first: true)
case .flowSequenceEntry:
try parseFlowSequenceEntry(first: false)
case .flowSequenceEntryMappingKey:
try parseFlowSequenceEntryMappingKey()
case .flowSequenceEntryMappingValue:
try parseFlowSequenceEntryMappingValue()
case .flowSequenceEntryMappingEnd:
try parseFlowSequenceEntryMappingEnd()
case .flowMappingFirstKey:
try parseFlowMappingKey(first: true)
case .flowMappingKey:
try parseFlowMappingKey(first: false)
case .flowMappingValue:
try parseFlowMappingValue(empty: false)
default:
try parseFlowMappingValue(empty: true)
}
}
private func endError() -> ParserError {
ParserError(context: nil, problem: "parsed past the end of the stream", mark: peekToken().startMark)
}
/// Port of `yaml_parser_process_empty_scalar`: an empty plain scalar event
/// stands in for an absent node.
func processEmptyScalar(at mark: Mark) -> Event {
.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark, endMark: mark)
}
}
}
extension PureYAML.Parsing.LibYAML.Parser {
typealias Token = PureYAML.Parsing.LibYAML.Token
typealias Mark = PureYAML.Parsing.Mark
typealias Event = PureYAML.Parsing.Event
typealias ParserError = PureYAML.Parsing.LibYAML.ParserError
typealias ParserState = PureYAML.Parsing.LibYAML.ParserState
}

View File

@ -0,0 +1,10 @@
extension PureYAML.Parsing.LibYAML {
/// A parser error, mirroring `yaml_parser_set_parser_error[_context]`: an
/// optional context describing the surrounding production, the immediate
/// problem, and the mark where it was detected.
struct ParserError: Error, Equatable {
var context: String?
var problem: String
var mark: PureYAML.Parsing.Mark
}
}

View File

@ -0,0 +1,30 @@
extension PureYAML.Parsing.LibYAML {
/// The parser state, mirroring `yaml_parser_state_e`. Each case names the
/// production the state machine is in the middle of recognizing.
enum ParserState: Equatable {
case streamStart
case implicitDocumentStart
case documentStart
case documentContent
case documentEnd
case blockNode
case blockNodeOrIndentlessSequence
case flowNode
case blockSequenceFirstEntry
case blockSequenceEntry
case indentlessSequenceEntry
case blockMappingFirstKey
case blockMappingKey
case blockMappingValue
case flowSequenceFirstEntry
case flowSequenceEntry
case flowSequenceEntryMappingKey
case flowSequenceEntryMappingValue
case flowSequenceEntryMappingEnd
case flowMappingFirstKey
case flowMappingKey
case flowMappingValue
case flowMappingEmptyValue
case end
}
}

View File

@ -0,0 +1,57 @@
extension PureYAML.Parsing.LibYAML {
/// Port of libyaml's input buffer and position tracking (the `buffer`/`mark`
/// fields with the `PEEK`, `SKIP`, and `SKIP_LINE` macros). It iterates
/// `Unicode.Scalar`s rather than grapheme clusters so a CR and an LF are
/// distinct, matching libyaml's byte-oriented scanning, and tracks the mark in
/// libyaml's 0-based line and column convention; conversion to the package's
/// 1-based `Mark` happens at token emission during integration.
struct Reader {
private let scalars: [Unicode.Scalar]
private(set) var index = 0
private(set) var line = 0
private(set) var column = 0
init(_ string: String) {
scalars = Array(string.unicodeScalars)
}
/// `PEEK_TOKEN` / `CHECK_AT`: the scalar `offset` ahead, or nil past the
/// end (libyaml's `\0`).
func peek(_ offset: Int = 0) -> Unicode.Scalar? {
let position = index + offset
return position < scalars.count ? scalars[position] : nil
}
var isAtEnd: Bool {
index >= scalars.count
}
/// The current position. Column is 0-based, as in libyaml.
var mark: PureYAML.Parsing.Mark {
PureYAML.Parsing.Mark(line: line, column: column, index: index)
}
/// `SKIP`: consume one non-break scalar, advancing the column.
mutating func skip() {
guard index < scalars.count else {
return
}
index += 1
column += 1
}
/// `SKIP_LINE`: consume a line break and start a new line. A CR+LF pair is
/// one break, matching libyaml.
mutating func skipLine() {
if peek() == "\r", peek(1) == "\n" {
index += 2
} else if CharacterClass.isBreak(peek()) {
index += 1
} else {
return
}
line += 1
column = 0
}
}
}

View File

@ -0,0 +1,45 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
/// Port of `yaml_parser_scan_anchor`, widened to YAML 1.2: an anchor (`&name`)
/// or alias (`*name`) name is `ns-anchor-char+`, i.e. any printable character
/// that is not whitespace, a line break, or a flow indicator (`,[]{}`). libyaml
/// restricts the name to alphanumerics, which rejects Unicode and punctuation
/// anchor names that YAML 1.2 allows.
mutating func scanAnchor(isAnchor: Bool) throws -> PureYAML.Parsing.LibYAML.Token {
let startMark = reader.mark
reader.skip()
var name = ""
while let character = reader.peek(), isAnchorCharacter(character) {
name.unicodeScalars.append(character)
reader.skip()
}
let endMark = reader.mark
guard !name.isEmpty else {
throw PureYAML.Parsing.LibYAML.ScannerError(
context: isAnchor ? "while scanning an anchor" : "while scanning an alias",
problem: "did not find expected alphabetic or numeric character",
mark: startMark,
)
}
return PureYAML.Parsing.LibYAML.Token(
type: isAnchor ? .anchor(name: name) : .alias(name: name),
startMark: startMark,
endMark: endMark,
)
}
/// `ns-anchor-char`: a printable, non-blank, non-break character that is not a
/// flow indicator or the byte-order mark.
private func isAnchorCharacter(_ scalar: Unicode.Scalar) -> Bool {
guard Classify.isPrintable(scalar), !Classify.isBlankZ(scalar), scalar != "\u{FEFF}" else {
return false
}
switch scalar {
case ",", "[", "]", "{", "}":
return false
default:
return true
}
}
}

View File

@ -0,0 +1,205 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
/// The accumulation state for `scan_block_scalar`.
private struct BlockState {
var string = ""
var leadingBreak = ""
var trailingBreaks = ""
var leadingBlank = false
}
/// Port of `yaml_parser_scan_block_scalar`. Scans a literal (`|`) or folded
/// (`>`) block scalar: parses the chomping (`+`/`-`) and indentation indicators
/// from the header, auto-detects the indentation if not given, then folds the
/// content (literal keeps breaks, folded joins equally-indented non-empty lines
/// with a space) and applies clip/strip/keep chomping to the trailing breaks.
mutating func scanBlockScalar(literal: Bool) throws -> PureYAML.Parsing.LibYAML.Token {
let startMark = reader.mark
reader.skip()
let header = try readBlockScalarHeader(startMark: startMark)
try consumeBlockHeaderTail(startMark: startMark)
var endMark = reader.mark
// -1 is the "indentation not yet auto-detected" sentinel. libyaml uses 0,
// but a root-level block scalar legitimately detects indentation 0, so 0
// must be a representable detected value rather than the sentinel.
var indent = -1
if header.increment != 0 {
indent = indentEngine.indent >= 0 ? indentEngine.indent + header.increment : header.increment
}
var state = BlockState()
try scanBlockScalarBreaks(indent: &indent, state: &state, startMark: startMark, endMark: &endMark)
while reader.column == indent, reader.peek() != nil {
// A root block scalar (indent 0) ends at a `---`/`...` document marker
// in column 0. libyaml never needs this because it floors the indent
// at 1, but indent 0 lets the loop reach a column-0 marker line.
if reader.column == 0, documentMarkerAhead() {
break
}
foldBlockLine(&state, literal: literal)
while let character = reader.peek(), !Classify.isBreakZ(character) {
state.string.unicodeScalars.append(character)
reader.skip()
}
state.leadingBreak += readBlockLine()
try scanBlockScalarBreaks(indent: &indent, state: &state, startMark: startMark, endMark: &endMark)
}
if header.chomping != -1 {
state.string += state.leadingBreak
}
if header.chomping == 1 {
state.string += state.trailingBreaks
}
return PureYAML.Parsing.LibYAML.Token(
type: .scalar(value: state.string, style: literal ? .literal : .folded),
startMark: startMark,
endMark: endMark,
)
}
/// Parse the chomping and indentation indicators in either order.
private mutating func readBlockScalarHeader(startMark: PureYAML.Parsing.Mark) throws -> (chomping: Int, increment: Int) {
var chomping = 0
var increment = 0
if reader.peek() == "+" || reader.peek() == "-" {
chomping = reader.peek() == "+" ? 1 : -1
reader.skip()
if Classify.isDigit(reader.peek()) {
increment = try blockIncrement(startMark: startMark)
}
} else if Classify.isDigit(reader.peek()) {
increment = try blockIncrement(startMark: startMark)
if reader.peek() == "+" || reader.peek() == "-" {
chomping = reader.peek() == "+" ? 1 : -1
reader.skip()
}
}
return (chomping, increment)
}
private mutating func blockIncrement(startMark: PureYAML.Parsing.Mark) throws -> Int {
if reader.peek() == "0" {
throw blockError("found an indentation indicator equal to 0", startMark)
}
let value = digitValue(reader.peek())
reader.skip()
return value
}
/// Eat trailing blanks and a comment to the end of the header line, then the
/// line break.
private mutating func consumeBlockHeaderTail(startMark: PureYAML.Parsing.Mark) throws {
while reader.peek() == " " {
reader.skip()
}
if reader.peek() == "#" {
// A comment needs a preceding space, so `>#` (a `#` adjacent to the
// block scalar indicator) is malformed, not a comment. libyaml accepts it.
if reader.column > 0, let previous = reader.peek(-1), !Classify.isBlank(previous) {
throw blockError("found a comment without a preceding space", startMark)
}
while !Classify.isBreakZ(reader.peek()) {
reader.skip()
}
}
guard Classify.isBreakZ(reader.peek()) else {
throw blockError("did not find expected comment or line break", startMark)
}
if Classify.isBreak(reader.peek()) {
reader.skipLine()
}
}
/// Port of `yaml_parser_scan_block_scalar_breaks`: eat indentation spaces and
/// blank lines, auto-detecting the indentation level when none was given.
private mutating func scanBlockScalarBreaks(indent: inout Int, state: inout BlockState, startMark: PureYAML.Parsing.Mark, endMark: inout PureYAML.Parsing.Mark) throws {
var maxIndent = 0
endMark = reader.mark
// A tab is content (or separation) once the column reaches the content
// indentation; it is illegal only while still inside the required
// indentation. During auto-detection the floor is the parent indent + 1.
let tabFloor = indent >= 0 ? indent : max(0, indentEngine.indent + 1)
while true {
let lineStart = reader.column
while indent < 0 || reader.column < indent, reader.peek() == " " {
reader.skip()
}
if reader.column > maxIndent {
maxIndent = reader.column
}
if reader.column < tabFloor, Classify.isTab(reader.peek()) {
throw blockError("found a tab character where an indentation space is expected", startMark)
}
if !Classify.isBreak(reader.peek()) {
// A trailing whitespace-only line that ends at end of input without
// a break still counts as one line break for keep chomping (as the
// test-suite reference does), even with no break byte present.
if reader.peek() == nil, reader.column > lineStart {
state.trailingBreaks += "\n"
}
break
}
state.trailingBreaks += readBlockLine()
endMark = reader.mark
}
// A leading empty line may not be more indented than the first content
// line; libyaml silently detects the deeper indent and drops the content.
if indent < 0, !Classify.isBreakZ(reader.peek()), reader.column < maxIndent {
throw blockError("found a leading empty line more indented than the content", startMark)
}
if indent < 0 {
// Auto-detected indentation. A root-level block scalar has a parent
// indent of -1, so the floor is 0 (allowing column-0 content), not
// libyaml's 1. maxIndent >= 0 keeps the result non-negative.
indent = max(maxIndent, indentEngine.indent + 1)
}
}
/// Fold the pending breaks at the start of a content line (folded style joins a
/// single break between non-blank lines into a space; otherwise breaks are
/// kept).
private func foldBlockLine(_ state: inout BlockState, literal: Bool) {
let trailingBlank = Classify.isBlank(reader.peek())
if !literal, state.leadingBreak.first == "\n", !state.leadingBlank, !trailingBlank {
if state.trailingBreaks.isEmpty {
state.string += " "
}
state.leadingBreak = ""
} else {
state.string += state.leadingBreak
state.leadingBreak = ""
}
state.string += state.trailingBreaks
state.trailingBreaks = ""
state.leadingBlank = Classify.isBlank(reader.peek())
}
private mutating func readBlockLine() -> String {
reader.skipLine()
return "\n"
}
/// Whether a `---` or `...` document marker (three repeats then a blank, break,
/// or end of input) starts at the current position.
private func documentMarkerAhead() -> Bool {
let first = reader.peek()
guard first == "-" || first == "." else {
return false
}
return reader.peek(1) == first && reader.peek(2) == first && Classify.isBlankZ(reader.peek(3))
}
private func digitValue(_ scalar: Unicode.Scalar?) -> Int {
guard let scalar, scalar >= "0", scalar <= "9" else {
return 0
}
return Int(scalar.value - Unicode.Scalar("0").value)
}
private func blockError(_ problem: String, _ mark: PureYAML.Parsing.Mark) -> PureYAML.Parsing.LibYAML.ScannerError {
PureYAML.Parsing.LibYAML.ScannerError(context: "while scanning a block scalar", problem: problem, mark: mark)
}
}

View File

@ -0,0 +1,153 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
private typealias Token = PureYAML.Parsing.LibYAML.Token
private typealias Mark = PureYAML.Parsing.Mark
/// The longest version number libyaml accepts (`MAX_NUMBER_LENGTH`).
private var maxVersionNumberLength: Int {
9
}
/// Port of `yaml_parser_fetch_directive`: reset the indentation and simple keys,
/// then scan a `%YAML` or `%TAG` directive token.
mutating func fetchDirective() throws {
indentEngine.unrollIndent(column: -1, mark: reader.mark)
try removeSimpleKey()
simpleKeyAllowed = false
let token = try scanDirective()
indentEngine.enqueue(token)
}
/// Port of `yaml_parser_scan_directive`, extended to YAML 1.2 §6.8.1: a `%YAML`
/// or `%TAG` directive produces its token; any other (reserved) directive is
/// ignored with a warning, producing a `reservedDirective` token whose content
/// the parser drops, rather than libyaml's hard rejection. The reserved token
/// still marks the document as a directives document, so it must be followed by
/// a `---`.
private mutating func scanDirective() throws -> Token {
let startMark = reader.mark
reader.skip()
let name = try scanDirectiveName(startMark: startMark)
if name == "YAML" {
let version = try scanVersionDirectiveValue(startMark: startMark)
let token = Token(type: .versionDirective(major: version.major, minor: version.minor), startMark: startMark, endMark: reader.mark)
try consumeDirectiveTail(startMark: startMark)
return token
}
if name == "TAG" {
let value = try scanTagDirectiveValue(startMark: startMark)
let token = Token(type: .tagDirective(handle: value.handle, prefix: value.prefix), startMark: startMark, endMark: reader.mark)
try consumeDirectiveTail(startMark: startMark)
return token
}
skipReservedDirectiveLine()
return Token(type: .reservedDirective, startMark: startMark, endMark: reader.mark)
}
/// Consume a reserved directive's parameters through the end of the line; the
/// whole line is ignored per YAML 1.2 §6.8.1.
private mutating func skipReservedDirectiveLine() {
while !Classify.isBreakZ(reader.peek()) {
reader.skip()
}
if Classify.isBreak(reader.peek()) {
reader.skipLine()
}
}
/// Eat trailing blanks and an optional comment, require a line break, eat it.
private mutating func consumeDirectiveTail(startMark: Mark) throws {
while Classify.isBlank(reader.peek()) {
reader.skip()
}
if reader.peek() == "#" {
while !Classify.isBreakZ(reader.peek()) {
reader.skip()
}
}
guard Classify.isBreakZ(reader.peek()) else {
throw directiveError("while scanning a directive", "did not find expected comment or line break", startMark)
}
if Classify.isBreak(reader.peek()) {
reader.skipLine()
}
}
/// Port of `yaml_parser_scan_directive_name`: the alphanumeric name after `%`,
/// which must be non-empty and followed by a blank or line break.
private mutating func scanDirectiveName(startMark: Mark) throws -> String {
var name = ""
while let character = reader.peek(), Classify.isAlpha(character) {
name.unicodeScalars.append(character)
reader.skip()
}
guard !name.isEmpty else {
throw directiveError("while scanning a directive", "could not find expected directive name", startMark)
}
guard Classify.isBlankZ(reader.peek()) else {
throw directiveError("while scanning a directive", "found unexpected non-alphabetical character", startMark)
}
return name
}
/// Port of `yaml_parser_scan_version_directive_value`: `major.minor`.
private mutating func scanVersionDirectiveValue(startMark: Mark) throws -> (major: Int, minor: Int) {
while Classify.isBlank(reader.peek()) {
reader.skip()
}
let major = try scanVersionDirectiveNumber(startMark: startMark)
guard reader.peek() == "." else {
throw directiveError("while scanning a %YAML directive", "did not find expected digit or '.' character", startMark)
}
reader.skip()
let minor = try scanVersionDirectiveNumber(startMark: startMark)
// The version must be followed by separation; a comment needs a space
// before its `#`, so `%YAML 1.1#...` (no space) is malformed. libyaml
// accepts it, treating the `#` as an adjacent comment.
guard Classify.isBlankZ(reader.peek()) else {
throw directiveError("while scanning a %YAML directive", "did not find expected digit or '.' character", startMark)
}
return (major, minor)
}
/// Port of `yaml_parser_scan_version_directive_number`: a bounded run of digits.
private mutating func scanVersionDirectiveNumber(startMark: Mark) throws -> Int {
var value = 0
var length = 0
while let character = reader.peek(), Classify.isDigit(character) {
length += 1
guard length <= maxVersionNumberLength else {
throw directiveError("while scanning a %YAML directive", "found extremely long version number", startMark)
}
value = value * 10 + Int(character.value - Unicode.Scalar("0").value)
reader.skip()
}
guard length > 0 else {
throw directiveError("while scanning a %YAML directive", "did not find expected version number", startMark)
}
return value
}
/// Port of `yaml_parser_scan_tag_directive_value`: `!handle! prefix`.
private mutating func scanTagDirectiveValue(startMark: Mark) throws -> (handle: String, prefix: String) {
while Classify.isBlank(reader.peek()) {
reader.skip()
}
let handle = try scanTagHandle(directive: true, startMark: startMark)
guard Classify.isBlank(reader.peek()) else {
throw directiveError("while scanning a %TAG directive", "did not find expected whitespace", startMark)
}
while Classify.isBlank(reader.peek()) {
reader.skip()
}
let prefix = try scanTagURI(uriChar: true, directive: true, head: "", startMark: startMark)
guard Classify.isBlankZ(reader.peek()) else {
throw directiveError("while scanning a %TAG directive", "did not find expected whitespace or line break", startMark)
}
return (handle, prefix)
}
private func directiveError(_ context: String, _ problem: String, _ mark: Mark) -> PureYAML.Parsing.LibYAML.ScannerError {
PureYAML.Parsing.LibYAML.ScannerError(context: context, problem: problem, mark: mark)
}
}

View File

@ -0,0 +1,361 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
private typealias Token = PureYAML.Parsing.LibYAML.Token
private typealias ScannerError = PureYAML.Parsing.LibYAML.ScannerError
/// Produce every token for the input, mirroring repeated calls to libyaml's
/// `yaml_parser_fetch_next_token` until STREAM-END. Tokens are never dequeued,
/// so `tokens_parsed` stays 0 and `QUEUE_INSERT` positions are absolute, which
/// matches the simple-key splice.
mutating func scanAll() throws -> [PureYAML.Parsing.LibYAML.Token] {
while !streamEndProduced {
try fetchNextToken()
}
return indentEngine.tokens
}
/// Port of `yaml_parser_fetch_next_token`: emit STREAM-START first, otherwise
/// skip to the next token, expire stale simple keys, unroll the indentation to
/// the current column, and dispatch on the leading character. Constructs not
/// yet ported throw an explicit error rather than mis-scan.
mutating func fetchNextToken() throws {
if !streamStartProduced {
fetchStreamStart()
return
}
try scanToNextToken()
try checkFlowContinuationIndent()
mark = reader.mark
try staleSimpleKeys()
indentEngine.unrollIndent(column: reader.column, mark: reader.mark)
mark = reader.mark
try dispatchToken()
}
/// Dispatch on the leading character: stream end, directives and document
/// markers at column 0, then the indicators.
private mutating func dispatchToken() throws {
let character = reader.peek()
if Classify.isZ(character) {
fetchStreamEnd()
return
}
if reader.column == 0, character == "%" {
try fetchDirective()
return
}
if reader.column == 0, isDocumentMarker("-") {
try fetchDocumentIndicator(.documentStart)
return
}
if reader.column == 0, isDocumentMarker(".") {
try fetchDocumentIndicator(.documentEnd)
return
}
guard let character else {
return
}
try dispatchIndicator(character)
}
/// Dispatch the block-structure indicators and plain scalars; unported
/// constructs throw an explicit error.
private mutating func dispatchIndicator(_ character: Unicode.Scalar) throws {
if try fetchFlowToken(character) {
return
}
if try fetchNodeToken(character) {
return
}
switch character {
case "-" where Classify.isBlankZ(reader.peek(1)):
try fetchBlockEntry()
case "-" where flowLevel > 0 && Classify.isFlowIndicator(reader.peek(1)):
// Per `ns-plain-first` a `-` begins a plain scalar in flow only when a
// flow-safe character follows; before a flow indicator (`[-]`, `[-,`)
// it cannot start a node. libyaml reads it as the scalar `-`.
throw ScannerError(
context: "while scanning for the next token",
problem: "found unexpected '-' indicator",
mark: reader.mark,
)
case "?" where Classify.isBlankZ(reader.peek(1)) || (flowLevel > 0 && Classify.isFlowIndicator(reader.peek(1))):
// In flow context libyaml treats every `?` as a key indicator, but per
// `ns-plain-first` a `?` followed by a flow-safe character (anything but
// whitespace, end, or a flow indicator) begins a plain scalar (`?foo`).
try fetchKey()
case ":" where Classify.isBlankZ(reader.peek(1))
|| (flowLevel > 0 && (Classify.isFlowIndicator(reader.peek(1)) || flowKeyIsJSON)):
// In flow context libyaml makes every `:` a value indicator. Per the
// spec a `:` only ends a node when followed by whitespace, end of
// input, or a flow indicator, except an adjacent `:` after a JSON
// (quoted) key. Otherwise the `:` is plain-scalar content (`:x`).
try fetchValue()
default:
// A plain scalar may begin with any non-blank character; a blank (a
// tab here, since spaces and breaks were already consumed) cannot
// start a token, matching libyaml's final guard in fetch_next_token.
guard !Classify.isBlankZ(character) else {
throw ScannerError(
context: "while scanning for the next token",
problem: "found character that cannot start any token",
mark: reader.mark,
)
}
try fetchPlainScalar()
}
}
/// Fetch a flow-collection indicator (`[` `]` `{` `}` `,`); returns false
/// otherwise.
private mutating func fetchFlowToken(_ character: Unicode.Scalar) throws -> Bool {
switch character {
case "[": fetchFlowCollectionStart(.flowSequenceStart)
case "{": fetchFlowCollectionStart(.flowMappingStart)
case "]": try fetchFlowCollectionEnd(.flowSequenceEnd)
case "}": try fetchFlowCollectionEnd(.flowMappingEnd)
case ",": try fetchFlowEntry()
default: return false
}
return true
}
/// Fetch a quoted scalar, an anchor/alias, or a block scalar; returns false
/// otherwise.
private mutating func fetchNodeToken(_ character: Unicode.Scalar) throws -> Bool {
switch character {
case "'": try fetchFlowScalar(single: true)
case "\"": try fetchFlowScalar(single: false)
case "&": try fetchAnchor(isAnchor: true)
case "*": try fetchAnchor(isAnchor: false)
case "!": try fetchTag()
case "|" where flowLevel == 0: try fetchBlockScalar(literal: true)
case ">" where flowLevel == 0: try fetchBlockScalar(literal: false)
default: return false
}
return true
}
/// Port of `yaml_parser_fetch_tag`.
private mutating func fetchTag() throws {
mark = reader.mark
try saveSimpleKey()
simpleKeyAllowed = false
let token = try scanTag()
indentEngine.enqueue(token)
}
/// Port of `yaml_parser_fetch_block_scalar`.
private mutating func fetchBlockScalar(literal: Bool) throws {
try removeSimpleKey()
simpleKeyAllowed = true
let token = try scanBlockScalar(literal: literal)
indentEngine.enqueue(token)
}
/// Port of `yaml_parser_fetch_flow_scalar`.
private mutating func fetchFlowScalar(single: Bool) throws {
mark = reader.mark
let saved = simpleKeyAllowed
try saveSimpleKey()
// A quoted scalar that became this level's simple key is a JSON node, so a
// later adjacent `:` (no space) still opens its value.
if saved {
simpleKeys[simpleKeys.count - 1].isJSON = true
}
simpleKeyAllowed = false
let token = try scanFlowScalar(single: single)
indentEngine.enqueue(token)
}
/// Port of `yaml_parser_fetch_anchor`.
private mutating func fetchAnchor(isAnchor: Bool) throws {
mark = reader.mark
try saveSimpleKey()
simpleKeyAllowed = false
let token = try scanAnchor(isAnchor: isAnchor)
indentEngine.enqueue(token)
}
/// Port of `yaml_parser_fetch_flow_collection_start`.
private mutating func fetchFlowCollectionStart(_ type: PureYAML.Parsing.LibYAML.TokenType) {
// saveSimpleKey cannot throw in flow: a required key needs block context.
let saved = simpleKeyAllowed
try? saveSimpleKey()
// A flow collection is a JSON node, so when it is this level's simple key
// an adjacent `:` after its closing `]`/`}` still opens its value.
if saved {
simpleKeys[simpleKeys.count - 1].isJSON = true
}
increaseFlowLevel(isSequence: type == .flowSequenceStart)
simpleKeyAllowed = true
let startMark = reader.mark
reader.skip()
indentEngine.enqueue(Token(type: type, startMark: startMark, endMark: reader.mark))
}
/// Port of `yaml_parser_fetch_flow_collection_end`.
private mutating func fetchFlowCollectionEnd(_ type: PureYAML.Parsing.LibYAML.TokenType) throws {
try removeSimpleKey()
decreaseFlowLevel()
simpleKeyAllowed = false
let startMark = reader.mark
reader.skip()
indentEngine.enqueue(Token(type: type, startMark: startMark, endMark: reader.mark))
}
/// Port of `yaml_parser_fetch_flow_entry`.
private mutating func fetchFlowEntry() throws {
try removeSimpleKey()
simpleKeyAllowed = true
let startMark = reader.mark
reader.skip()
indentEngine.enqueue(Token(type: .flowEntry, startMark: startMark, endMark: reader.mark))
}
/// Port of `yaml_parser_fetch_stream_start`.
private mutating func fetchStreamStart() {
// indent is already -1 and simpleKeys already holds the block-context slot.
simpleKeyAllowed = true
streamStartProduced = true
indentEngine.enqueue(Token(type: .streamStart, startMark: reader.mark, endMark: reader.mark))
}
/// Port of `yaml_parser_fetch_stream_end`.
private mutating func fetchStreamEnd() {
if reader.column != 0 {
// Force a new line in the mark, as libyaml does.
mark = PureYAML.Parsing.Mark(line: reader.line + 1, column: 0, index: reader.index)
}
indentEngine.unrollIndent(column: -1, mark: mark)
// remove_simple_key cannot throw here: a required key would already have
// been resolved before stream end in the block + plain core.
simpleKeys[simpleKeys.count - 1].possible = false
simpleKeyAllowed = false
streamEndProduced = true
indentEngine.enqueue(Token(type: .streamEnd, startMark: mark, endMark: mark))
}
/// Port of `yaml_parser_fetch_document_indicator`.
private mutating func fetchDocumentIndicator(_ type: PureYAML.Parsing.LibYAML.TokenType) throws {
indentEngine.unrollIndent(column: -1, mark: reader.mark)
simpleKeys[simpleKeys.count - 1].possible = false
simpleKeyAllowed = false
let startMark = reader.mark
reader.skip()
reader.skip()
reader.skip()
indentEngine.enqueue(Token(type: type, startMark: startMark, endMark: reader.mark))
if case .documentEnd = type {
// Only whitespace and a comment may follow `...` on its line; a `--- x`
// document start can carry inline content, but `... x` cannot.
while reader.peek() == " " || Classify.isTab(reader.peek()) {
reader.skip()
}
if let next = reader.peek(), next != "#", !Classify.isBreak(next) {
throw ScannerError(
context: "while scanning a document end marker",
problem: "found content after the document end marker",
mark: reader.mark,
)
}
}
}
/// Port of `yaml_parser_fetch_block_entry`.
private mutating func fetchBlockEntry() throws {
if tabbedSeparation {
throw tabIndentationError()
}
// A block sequence must begin its own line. A `-` mid-line after a mapping
// value (`key: - a`) or a node property (`&a - x`) is invalid, but a `-`
// after another `-` is a valid compact sequence (`- - a`). libyaml accepts
// all three.
if flowLevel == 0, !crossedLineBreak, lastTokenForbidsInlineSequence() {
throw ScannerError(
context: "while scanning a node",
problem: "found a block sequence on the same line as an earlier node",
mark: reader.mark,
)
}
if flowLevel == 0 {
indentEngine.rollIndent(column: reader.column, number: -1, type: .blockSequenceStart, mark: reader.mark)
}
simpleKeys[simpleKeys.count - 1].possible = false
simpleKeyAllowed = true
let startMark = reader.mark
reader.skip()
indentEngine.enqueue(Token(type: .blockEntry, startMark: startMark, endMark: reader.mark))
}
/// Port of `yaml_parser_fetch_key`.
private mutating func fetchKey() throws {
if tabbedSeparation {
throw tabIndentationError()
}
if flowLevel == 0 {
indentEngine.rollIndent(column: reader.column, number: -1, type: .blockMappingStart, mark: reader.mark)
}
simpleKeys[simpleKeys.count - 1].possible = false
simpleKeyAllowed = flowLevel == 0
let startMark = reader.mark
reader.skip()
indentEngine.enqueue(Token(type: .key, startMark: startMark, endMark: reader.mark))
}
/// Port of `yaml_parser_fetch_value`, including the simple-key splice: a
/// possible simple key inserts its `KEY` token at the recorded position and
/// rolls the block mapping indent to the key's column.
private mutating func fetchValue() throws {
let top = simpleKeys.count - 1
// A pending simple key means this `:` closes an inline implicit key (key
// and `:` on one line). An explicit `?` key's `:` has no pending simple key
// (it went stale across the line break), so its value may open a sequence.
lastValueWasInline = simpleKeys[top].possible
// A flow-sequence single-pair implicit key must be single-line, so its `:`
// must sit on the key's line (`[ "k"\n :v ]` and `[ k\n : v ]` are both
// invalid); a flow-mapping adjacent value may span lines (`{ "k"\n :v }`).
if simpleKeys[top].possible, flowIsSequence.last == true, simpleKeys[top].mark.line != mark.line {
throw ScannerError(
context: "while scanning a simple key",
problem: "could not find expected ':'",
mark: simpleKeys[top].mark,
)
}
if simpleKeys[top].possible {
if simpleKeys[top].tabbed {
throw tabIndentationError()
}
let key = simpleKeys[top]
indentEngine.insert(Token(type: .key, startMark: key.mark, endMark: key.mark), atTokenNumber: key.tokenNumber)
indentEngine.rollIndent(column: key.mark.column, number: key.tokenNumber, type: .blockMappingStart, mark: key.mark)
simpleKeys[top].possible = false
simpleKeyAllowed = false
} else {
if flowLevel == 0 {
indentEngine.rollIndent(column: reader.column, number: -1, type: .blockMappingStart, mark: reader.mark)
}
simpleKeyAllowed = flowLevel == 0
}
let startMark = reader.mark
reader.skip()
indentEngine.enqueue(Token(type: .value, startMark: startMark, endMark: reader.mark))
}
/// Port of `yaml_parser_fetch_plain_scalar`.
private mutating func fetchPlainScalar() throws {
mark = reader.mark
try saveSimpleKey()
simpleKeyAllowed = false
let token = try scanPlainScalar()
indentEngine.enqueue(token)
}
private func isDocumentMarker(_ character: Unicode.Scalar) -> Bool {
reader.peek(0) == character
&& reader.peek(1) == character
&& reader.peek(2) == character
&& Classify.isBlankZ(reader.peek(3))
}
}

View File

@ -0,0 +1,225 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
/// The folding state carried through `scan_flow_scalar`, the same shape as the
/// plain-scalar state.
private struct FlowState {
var string = ""
var leadingBreak = ""
var trailingBreaks = ""
var whitespaces = ""
var leadingBlanks = false
}
/// Decoded single-character escapes for double-quoted scalars (libyaml's escape
/// switch). libyaml emits UTF-8 bytes; here each maps to its Unicode scalar.
private static let doubleQuotedEscapes: [Unicode.Scalar: Unicode.Scalar] = [
"0": "\u{0}", "a": "\u{07}", "b": "\u{08}", "t": "\u{09}", "\t": "\u{09}",
"n": "\u{0A}", "v": "\u{0B}", "f": "\u{0C}", "r": "\u{0D}", "e": "\u{1B}",
" ": "\u{20}", "\"": "\"", "/": "/", "\\": "\\",
"N": "\u{85}", "_": "\u{A0}", "L": "\u{2028}", "P": "\u{2029}",
]
/// Port of `yaml_parser_scan_flow_scalar`. Scans a single- or double-quoted
/// scalar, folding line breaks like a plain scalar and, for double quotes,
/// decoding escape sequences.
mutating func scanFlowScalar(single: Bool) throws -> PureYAML.Parsing.LibYAML.Token {
let startMark = reader.mark
reader.skip()
var state = FlowState()
let quote: Unicode.Scalar = single ? "'" : "\""
let indent = indentEngine.indent + 1
// A continuation indented below `indent` is an error, but it is recorded
// and only raised once the quote is known to close; an unterminated quote
// reaches end of input first and reports that instead.
var indentError: PureYAML.Parsing.LibYAML.ScannerError?
while true {
if reader.column == 0, documentMarkerAhead() {
throw flowError("found unexpected document indicator", startMark)
}
if reader.peek() == nil {
throw flowError("found unexpected end of stream", startMark)
}
try consumeFlowContent(single: single, &state, startMark: startMark)
if reader.peek() == quote {
break
}
consumeFlowBreaks(&state, indent: indent, quote: quote, startMark: startMark, error: &indentError)
foldFlow(&state)
}
reader.skip()
if let indentError {
throw indentError
}
return PureYAML.Parsing.LibYAML.Token(
type: .scalar(value: state.string, style: single ? .singleQuoted : .doubleQuoted),
startMark: startMark,
endMark: reader.mark,
)
}
/// Consume the run of non-blank characters, decoding escapes and stopping at
/// the closing quote or an escaped line break.
private mutating func consumeFlowContent(single: Bool, _ state: inout FlowState, startMark: PureYAML.Parsing.Mark) throws {
state.leadingBlanks = false
let quote: Unicode.Scalar = single ? "'" : "\""
while let character = reader.peek(), !Classify.isBlankZ(character) {
if single, character == "'", reader.peek(1) == "'" {
state.string.unicodeScalars.append("'")
reader.skip()
reader.skip()
} else if character == quote {
return
} else if !single, character == "\\", Classify.isBreak(reader.peek(1)) {
reader.skip()
reader.skipLine()
state.leadingBlanks = true
return
} else if !single, character == "\\" {
try appendEscape(&state, startMark: startMark)
} else {
state.string.unicodeScalars.append(character)
reader.skip()
}
}
}
/// Decode a double-quoted escape sequence (`\` already at the reader).
private mutating func appendEscape(_ state: inout FlowState, startMark: PureYAML.Parsing.Mark) throws {
let escape = reader.peek(1)
if let escape, let decoded = Self.doubleQuotedEscapes[escape] {
state.string.unicodeScalars.append(decoded)
reader.skip()
reader.skip()
return
}
let codeLength = try hexEscapeLength(escape, startMark: startMark)
reader.skip()
reader.skip()
var value = try readHexEscape(count: codeLength, startMark: startMark)
if codeLength == 4 {
value = try combineSurrogateIfNeeded(high: value, startMark: startMark)
}
guard let scalar = Unicode.Scalar(value) else {
throw flowError("found invalid Unicode character escape code", startMark)
}
state.string.unicodeScalars.append(scalar)
}
/// The hex digit count for `\x` (2), `\u` (4), and `\U` (8) escapes.
private func hexEscapeLength(_ escape: Unicode.Scalar?, startMark: PureYAML.Parsing.Mark) throws -> Int {
switch escape {
case "x": 2
case "u": 4
case "U": 8
default: throw flowError("found unknown escape character", startMark)
}
}
/// Read and consume `count` hexadecimal digits, returning their value.
private mutating func readHexEscape(count: Int, startMark: PureYAML.Parsing.Mark) throws -> UInt32 {
var value: UInt32 = 0
for _ in 0 ..< count {
guard let digit = reader.peek(), Classify.isHex(digit) else {
throw flowError("did not find expected hexdecimal number", startMark)
}
value = (value << 4) + hexValue(digit)
reader.skip()
}
return value
}
/// Combine a UTF-16 surrogate pair (a high surrogate `\u` escape immediately
/// followed by a low surrogate `\u` escape) into a single astral code point,
/// for JSON compatibility. A lone high surrogate is left for the scalar check.
private mutating func combineSurrogateIfNeeded(high: UInt32, startMark: PureYAML.Parsing.Mark) throws -> UInt32 {
guard (0xD800 ... 0xDBFF).contains(high), reader.peek() == "\\", reader.peek(1) == "u" else {
return high
}
reader.skip()
reader.skip()
let low = try readHexEscape(count: 4, startMark: startMark)
guard (0xDC00 ... 0xDFFF).contains(low) else {
throw flowError("found invalid Unicode character escape code", startMark)
}
return 0x10000 + ((high - 0xD800) << 10) + (low - 0xDC00)
}
/// Consume blanks and line breaks between content, recording pending
/// whitespace and folded breaks.
private mutating func consumeFlowBreaks(
_ state: inout FlowState,
indent: Int,
quote: Unicode.Scalar,
startMark: PureYAML.Parsing.Mark,
error: inout PureYAML.Parsing.LibYAML.ScannerError?,
) {
while Classify.isBlank(reader.peek()) || Classify.isBreak(reader.peek()) {
if Classify.isBlank(reader.peek()) {
if error == nil, state.leadingBlanks, reader.column < indent, Classify.isTab(reader.peek()), !blankLineAhead() {
error = flowError("found a tab character that violates indentation", startMark)
}
if !state.leadingBlanks, let blank = reader.peek() {
state.whitespaces.unicodeScalars.append(blank)
}
reader.skip()
} else if !state.leadingBlanks {
state.whitespaces = ""
reader.skipLine()
state.leadingBreak = "\n"
state.leadingBlanks = true
} else {
reader.skipLine()
state.trailingBreaks += "\n"
}
}
// The continuation content must be indented at least `indent`; a less
// indented (non-closing, non-empty) line is malformed.
if error == nil, state.leadingBlanks, let next = reader.peek(), next != quote, !Classify.isBreakZ(next), reader.column < indent {
error = flowError("found wrong indentation in a multi-line quoted scalar", startMark)
}
}
/// Fold the pending breaks or whitespace into the value.
private func foldFlow(_ state: inout FlowState) {
guard state.leadingBlanks else {
state.string += state.whitespaces
state.whitespaces = ""
return
}
if state.leadingBreak.first == "\n" {
if state.trailingBreaks.isEmpty {
state.string += " "
} else {
state.string += state.trailingBreaks
state.trailingBreaks = ""
}
state.leadingBreak = ""
} else {
state.string += state.leadingBreak
state.string += state.trailingBreaks
state.leadingBreak = ""
state.trailingBreaks = ""
}
}
private func hexValue(_ scalar: Unicode.Scalar) -> UInt32 {
switch scalar {
case "0" ... "9": scalar.value - Unicode.Scalar("0").value
case "A" ... "F": scalar.value - Unicode.Scalar("A").value + 10
default: scalar.value - Unicode.Scalar("a").value + 10
}
}
private func flowError(_ problem: String, _ mark: PureYAML.Parsing.Mark) -> PureYAML.Parsing.LibYAML.ScannerError {
PureYAML.Parsing.LibYAML.ScannerError(context: "while scanning a quoted scalar", problem: problem, mark: mark)
}
private func documentMarkerAhead() -> Bool {
let first = reader.peek()
guard first == "-" || first == "." else {
return false
}
return reader.peek(1) == first && reader.peek(2) == first && Classify.isBlankZ(reader.peek(3))
}
}

View File

@ -0,0 +1,59 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias ScannerError = PureYAML.Parsing.LibYAML.ScannerError
/// Reject a flow-collection continuation line that is not indented past the
/// enclosing block context. YAML requires `s-flow-line-prefix` to be more
/// indented than the parent block; libyaml does not enforce it, so multi-line
/// flow content in column 0 (`[a,\nb]` under a key) wrongly parses.
mutating func checkFlowContinuationIndent() throws {
guard flowLevel > 0, crossedLineBreak, let next = reader.peek() else {
return
}
// A closing `]`/`}` (or a `,` separator) may sit at any indentation; only
// flow CONTENT must be indented past the parent block.
if next == "]" || next == "}" || next == "," {
return
}
// A tab in the leading whitespace is never valid indentation, even where
// spaces of the same width would be (the tab inflates the column).
if reader.column <= indentEngine.indent || flowLeadingTab {
throw ScannerError(
context: "while parsing a flow node",
problem: "found insufficiently indented line in a flow collection",
mark: reader.mark,
)
}
}
/// Whether the last emitted token bars a block sequence from opening on the
/// same line: a mapping value indicator or a node property (anchor/tag). After
/// another block entry a compact sequence is allowed, so that is not barred.
func lastTokenForbidsInlineSequence() -> Bool {
switch indentEngine.tokens.last?.type {
case .anchor, .tag:
true
case .value:
lastValueWasInline
default:
false
}
}
/// The error raised when a tab is used where indentation is required (before a
/// block indicator or an implicit mapping key). A tab is valid separation but
/// never indentation.
func tabIndentationError() -> PureYAML.Parsing.LibYAML.ScannerError {
ScannerError(
context: "while scanning for the next token",
problem: "found a tab character where an indentation space is expected",
mark: reader.mark,
)
}
/// Whether this flow level's pending simple key is a JSON (quoted) node, which
/// lets an adjacent `:` open its value without a separating space.
var flowKeyIsJSON: Bool {
let key = simpleKeys[simpleKeys.count - 1]
return key.possible && key.isJSON
}
}

View File

@ -0,0 +1,142 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
/// The folding state carried through `scan_plain_scalar`: the accumulated
/// value plus the pending leading break, trailing breaks, and whitespace that
/// are folded in before the next content character.
private struct PlainState {
var string = ""
var leadingBreak = ""
var trailingBreaks = ""
var whitespaces = ""
var leadingBlanks = false
var endMark: PureYAML.Parsing.Mark
}
/// Port of `yaml_parser_scan_plain_scalar`. Consumes a plain scalar, folding
/// line breaks: a single break between non-empty lines becomes a space, blank
/// lines become literal newlines, and leading/trailing whitespace on folded
/// lines is dropped. Ends at a `: ` value indicator, a comment, a less-indented
/// line (block context), a flow indicator (flow context), or a document marker.
mutating func scanPlainScalar() throws -> PureYAML.Parsing.LibYAML.Token {
let indent = indentEngine.indent + 1
let startMark = reader.mark
var state = PlainState(endMark: reader.mark)
while true {
if reader.column == 0, documentMarkerAhead() {
break
}
if reader.peek() == "#" {
break
}
try consumePlainContent(&state, startMark: startMark)
guard Classify.isBlank(reader.peek()) || Classify.isBreak(reader.peek()) else {
break
}
try consumePlainBreaks(&state, indent: indent, startMark: startMark)
if flowLevel == 0, reader.column < indent {
break
}
}
if state.leadingBlanks {
simpleKeyAllowed = true
}
return PureYAML.Parsing.LibYAML.Token(
type: .scalar(value: state.string, style: .plain),
startMark: startMark,
endMark: state.endMark,
)
}
/// Consume the run of non-blank characters, folding any pending breaks or
/// whitespace ahead of each one.
private mutating func consumePlainContent(_ state: inout PlainState, startMark _: PureYAML.Parsing.Mark) throws {
while let character = reader.peek(), !Classify.isBlankZ(character) {
if character == ":", Classify.isBlankZ(reader.peek(1)) {
break
}
if flowLevel > 0, character == ":", Classify.isFlowIndicator(reader.peek(1)) {
// A `:` before a flow indicator ends the key; the value is empty.
// libyaml rejects this as an unexpected ':', but it is a valid
// value indicator in flow context.
break
}
if flowLevel > 0, Classify.isFlowIndicator(character) {
break
}
foldPending(&state)
state.string.unicodeScalars.append(character)
reader.skip()
state.endMark = reader.mark
}
}
/// Fold the pending leading break, trailing breaks, and whitespace into the
/// accumulated value (the join logic from libyaml's inner loop).
private func foldPending(_ state: inout PlainState) {
guard state.leadingBlanks || !state.whitespaces.isEmpty else {
return
}
guard state.leadingBlanks else {
state.string += state.whitespaces
state.whitespaces = ""
return
}
if state.leadingBreak.first == "\n" {
if state.trailingBreaks.isEmpty {
state.string += " "
} else {
state.string += state.trailingBreaks
state.trailingBreaks = ""
}
state.leadingBreak = ""
} else {
state.string += state.leadingBreak
state.string += state.trailingBreaks
state.leadingBreak = ""
state.trailingBreaks = ""
}
state.leadingBlanks = false
}
/// Consume the run of blanks and line breaks between content, recording them as
/// pending whitespace or folded breaks.
private mutating func consumePlainBreaks(_ state: inout PlainState, indent: Int, startMark: PureYAML.Parsing.Mark) throws {
while Classify.isBlank(reader.peek()) || Classify.isBreak(reader.peek()) {
if Classify.isBlank(reader.peek()) {
if state.leadingBlanks, reader.column < indent, Classify.isTab(reader.peek()), !blankLineAhead() {
// A tab below the indentation is illegal only when it indents
// real content; on an otherwise-blank line it is just a blank
// line the scalar folds over. libyaml rejects it either way.
throw scannerError("found a tab character that violates indentation", at: startMark)
}
if !state.leadingBlanks, let blank = reader.peek() {
state.whitespaces.unicodeScalars.append(blank)
}
reader.skip()
} else if !state.leadingBlanks {
state.whitespaces = ""
reader.skipLine()
state.leadingBreak = "\n"
state.leadingBlanks = true
} else {
reader.skipLine()
state.trailingBreaks += "\n"
}
}
}
private func scannerError(_ problem: String, at mark: PureYAML.Parsing.Mark) -> PureYAML.Parsing.LibYAML.ScannerError {
PureYAML.Parsing.LibYAML.ScannerError(context: "while scanning a plain scalar", problem: problem, mark: mark)
}
private func documentMarkerAhead() -> Bool {
let first = reader.peek()
guard first == "-" || first == "." else {
return false
}
return reader.peek(1) == first && reader.peek(2) == first && Classify.isBlankZ(reader.peek(3))
}
}

View File

@ -0,0 +1,88 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
private typealias ScannerError = PureYAML.Parsing.LibYAML.ScannerError
/// Port of `yaml_parser_scan_to_next_token`: skip a leading BOM, spaces and
/// tabs, a comment, and a single line break, which re-allows a simple key in
/// the block context. A tab is valid in-line separation, so it is skipped
/// here; `tabbedSeparation` records when one preceded the token so a block
/// indicator or implicit key (where the tab would be indentation) can reject.
mutating func scanToNextToken() throws {
crossedLineBreak = false
flowLeadingTab = false
while true {
tabbedSeparation = false
if reader.column == 0, reader.peek() == "\u{FEFF}" {
reader.skip()
}
while reader.peek() == " " || Classify.isTab(reader.peek()) {
if Classify.isTab(reader.peek()) {
noteLeadingTab()
}
reader.skip()
}
// YAML 1.2: a line containing only whitespace is blank even when that
// whitespace includes tabs. libyaml leaves a leading tab in place and
// later rejects it; instead consume the trailing blanks so the line
// break below is reached and the blank line is skipped.
if blankLineAhead() {
while reader.peek() == " " || Classify.isTab(reader.peek()) {
reader.skip()
}
}
if reader.peek() == "#" {
try consumeComment()
}
if Classify.isBreak(reader.peek()) {
reader.skipLine()
crossedLineBreak = true
if flowLevel == 0 {
simpleKeyAllowed = true
}
} else {
break
}
}
}
/// Record what a tab in the leading whitespace implies. In block context with a
/// simple key allowed it is potential indentation before an indicator. In flow
/// context, below the continuation indent on a non-blank line it is indentation
/// abuse (a tab is never indentation); at or above the indent it is separation.
private mutating func noteLeadingTab() {
if flowLevel == 0, simpleKeyAllowed {
tabbedSeparation = true
}
if flowLevel > 0, crossedLineBreak, reader.column < indentEngine.indent + 1, !blankLineAhead() {
flowLeadingTab = true
}
}
/// Consume a comment to the end of the line. A comment must be preceded by
/// whitespace or start the line; a `#` adjacent to the previous token (e.g.
/// `"x"#`, `]#`, `,#`) is not a comment. The plain-scalar scanner consumes its
/// own trailing blanks, so the character just before the `#` is the reliable
/// signal. libyaml accepts the adjacent form.
private mutating func consumeComment() throws {
if reader.column > 0, let previous = reader.peek(-1), !Classify.isBlank(previous) {
throw ScannerError(
context: "while scanning for the next token",
problem: "found a comment without a preceding space",
mark: reader.mark,
)
}
while !Classify.isBreakZ(reader.peek()) {
reader.skip()
}
}
/// Whether the rest of the current line, from the current position, is only
/// spaces and tabs up to a line break or end of stream (a blank line).
func blankLineAhead() -> Bool {
var offset = 0
while let character = reader.peek(offset), character == " " || Classify.isTab(character) {
offset += 1
}
return Classify.isBreakZ(reader.peek(offset))
}
}

View File

@ -0,0 +1,155 @@
extension PureYAML.Parsing.LibYAML.Scanner {
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
/// Port of `yaml_parser_scan_tag`. Scans a tag in verbatim `!<uri>`, shorthand
/// `!handle!suffix`, or `!suffix` form, producing the handle and suffix; the
/// parser resolves them against the tag directives.
mutating func scanTag() throws -> PureYAML.Parsing.LibYAML.Token {
let startMark = reader.mark
var handle = ""
var suffix = ""
if reader.peek(1) == "<" {
reader.skip()
reader.skip()
suffix = try scanTagURI(uriChar: true, directive: false, head: "", startMark: startMark)
guard reader.peek() == ">" else {
throw tagError("did not find the expected '>'", startMark)
}
reader.skip()
} else {
handle = try scanTagHandle(directive: false, startMark: startMark)
if handle.first == "!", handle.count > 1, handle.last == "!" {
suffix = try scanTagURI(uriChar: false, directive: false, head: "", startMark: startMark)
} else {
suffix = try scanTagURI(uriChar: false, directive: false, head: handle, startMark: startMark)
handle = "!"
if suffix.isEmpty {
swap(&handle, &suffix)
}
}
}
guard Classify.isBlankZ(reader.peek()) || (flowLevel > 0 && reader.peek() == ",") else {
throw tagError("did not find expected whitespace or line break", startMark)
}
return PureYAML.Parsing.LibYAML.Token(type: .tag(handle: handle, suffix: suffix), startMark: startMark, endMark: reader.mark)
}
/// Port of `yaml_parser_scan_tag_handle`. A handle is `!`, optionally followed
/// by alphanumeric characters and a closing `!`.
mutating func scanTagHandle(directive: Bool, startMark: PureYAML.Parsing.Mark) throws -> String {
guard reader.peek() == "!" else {
throw tagError("did not find expected '!'", startMark)
}
var handle = "!"
reader.skip()
while let character = reader.peek(), Classify.isAlpha(character) {
handle.unicodeScalars.append(character)
reader.skip()
}
if reader.peek() == "!" {
handle.unicodeScalars.append("!")
reader.skip()
} else if directive, handle != "!" {
throw tagError("did not find expected '!'", startMark)
}
return handle
}
/// Port of `yaml_parser_scan_tag_uri`. Consumes the URI character set (plus the
/// flow indicators inside a verbatim tag), decoding `%`-escapes.
mutating func scanTagURI(uriChar: Bool, directive: Bool, head: String, startMark: PureYAML.Parsing.Mark) throws -> String {
var string = head.count > 1 ? String(head.dropFirst()) : ""
var length = head.isEmpty ? 0 : head.count
while let character = reader.peek(), isURICharacter(character, uriChar: uriChar) {
if character == "%" {
try scanURIEscapes(directive: directive, startMark: startMark, into: &string)
} else {
string.unicodeScalars.append(character)
reader.skip()
}
length += 1
}
guard length > 0 else {
throw tagError("did not find expected tag URI", startMark)
}
return string
}
/// Port of `yaml_parser_scan_uri_escapes`. Decodes one UTF-8 character from a
/// run of `%XX` octets.
private mutating func scanURIEscapes(directive _: Bool, startMark: PureYAML.Parsing.Mark, into string: inout String) throws {
var bytes: [UInt8] = []
var width = 0
repeat {
guard reader.peek() == "%", let high = reader.peek(1), Classify.isHex(high), let low = reader.peek(2), Classify.isHex(low) else {
throw tagError("did not find URI escaped octet", startMark)
}
let octet = UInt8((hexDigit(high) << 4) + hexDigit(low))
if width == 0 {
width = utf8Width(of: octet)
guard width > 0 else {
throw tagError("found an incorrect leading UTF-8 octet", startMark)
}
} else if octet & 0xC0 != 0x80 {
throw tagError("found an incorrect trailing UTF-8 octet", startMark)
}
bytes.append(octet)
reader.skip()
reader.skip()
reader.skip()
width -= 1
} while width > 0
guard let scalar = Unicode.Scalar(decodeUTF8(bytes)) else {
throw tagError("found invalid UTF-8 in a URI escape", startMark)
}
string.unicodeScalars.append(scalar)
}
/// Assemble a code point from the validated 1-to-4 UTF-8 octets of one escape.
private func decodeUTF8(_ bytes: [UInt8]) -> UInt32 {
switch bytes.count {
case 1:
UInt32(bytes[0])
case 2:
(UInt32(bytes[0] & 0x1F) << 6) | UInt32(bytes[1] & 0x3F)
case 3:
(UInt32(bytes[0] & 0x0F) << 12) | (UInt32(bytes[1] & 0x3F) << 6) | UInt32(bytes[2] & 0x3F)
default:
(UInt32(bytes[0] & 0x07) << 18) | (UInt32(bytes[1] & 0x3F) << 12) | (UInt32(bytes[2] & 0x3F) << 6) | UInt32(bytes[3] & 0x3F)
}
}
private func utf8Width(of octet: UInt8) -> Int {
if octet & 0x80 == 0x00 { return 1 }
if octet & 0xE0 == 0xC0 { return 2 }
if octet & 0xF0 == 0xE0 { return 3 }
if octet & 0xF8 == 0xF0 { return 4 }
return 0
}
private func isURICharacter(_ scalar: Unicode.Scalar, uriChar: Bool) -> Bool {
if Classify.isAlpha(scalar) {
return true
}
switch scalar {
case ";", "/", "?", ":", "@", "&", "=", "+", "$", ".", "%", "!", "~", "*", "'", "(", ")":
return true
case ",", "[", "]":
return uriChar
default:
return false
}
}
private func hexDigit(_ scalar: Unicode.Scalar) -> Int {
switch scalar {
case "0" ... "9": Int(scalar.value - Unicode.Scalar("0").value)
case "A" ... "F": Int(scalar.value - Unicode.Scalar("A").value) + 10
default: Int(scalar.value - Unicode.Scalar("a").value) + 10
}
}
private func tagError(_ problem: String, _ mark: PureYAML.Parsing.Mark) -> PureYAML.Parsing.LibYAML.ScannerError {
PureYAML.Parsing.LibYAML.ScannerError(context: "while scanning a tag", problem: problem, mark: mark)
}
}

View File

@ -0,0 +1,139 @@
extension PureYAML.Parsing.LibYAML {
/// Aggregate scanner state, mirroring the fields of `yaml_parser_t` that the
/// ported functions read and write. It is built up as the port proceeds;
/// currently it holds the indentation engine and the simple-key mechanism.
/// Column values follow libyaml's 0-based convention internally; conversion to
/// the package's 1-based `Mark` happens at token emission during integration.
struct Scanner {
/// The input buffer and position (`parser->buffer` / `parser->mark`).
var reader: Reader
/// The indentation stack and `roll_indent` / `unroll_indent` (also the
/// owner of `flow_level` and the token queue).
var indentEngine = IndentEngine()
/// `parser->simple_keys`. Always holds at least the block-context level.
var simpleKeys: [SimpleKey] = [.empty]
/// `parser->simple_key_allowed`. True at stream start.
var simpleKeyAllowed = true
/// Whether a tab appeared in the indentation before the pending token. A
/// tab is valid in-line separation but never indentation, so a block
/// indicator or implicit mapping key reached through one is an error.
var tabbedSeparation = false
/// Whether reaching the pending token crossed at least one line break. In
/// flow context a continuation line must be indented past the parent block,
/// so a token on a new line at or below that indent is an error.
var crossedLineBreak = false
/// Whether a tab appeared in the leading whitespace of a flow continuation
/// line. A tab is never indentation, so flow content reached through one is
/// an error even though the tab inflates the column.
var flowLeadingTab = false
/// Whether the most recent `:` value indicator sat on the same line as its
/// key (an inline implicit-key value). Such a value may not open a block
/// sequence on that line (`key: - a`), whereas an explicit key whose `:` is
/// on a later line may (`? k`\n`: - a`).
var lastValueWasInline = false
/// For each open flow level, whether it is a sequence (`[`) rather than a
/// mapping (`{`). A flow-sequence single-pair implicit key is single-line
/// (`c-s-implicit-json-key`), so its `:` may not cross a line break, unlike
/// a flow-mapping adjacent value (`{ "a"`\n`: b }` is allowed).
var flowIsSequence: [Bool] = []
/// `parser->mark`, kept in sync with the reader at each fetch step.
var mark: PureYAML.Parsing.Mark = .start
/// `parser->stream_start_produced`.
var streamStartProduced = false
/// `parser->stream_end_produced`.
var streamEndProduced = false
init(_ input: String = "") {
reader = Reader(input)
}
/// `parser->flow_level`, kept on the indent engine.
var flowLevel: Int {
indentEngine.flowLevel
}
/// Port of `yaml_parser_stale_simple_keys`. A potential simple key is
/// invalid once it is on an earlier line or more than 1024 characters back
/// (a simple key is single-line and bounded); a required one going stale is
/// an error.
mutating func staleSimpleKeys() throws {
for index in simpleKeys.indices {
let key = simpleKeys[index]
guard key.possible else {
continue
}
// A block-context key (index 0) is single-line. A flow-context key
// (index >= 1) may be separated from its ':' by line breaks under
// YAML 1.2, so a new line alone does not invalidate it; both remain
// bounded to 1024 characters. libyaml invalidates every key on a new
// line, which wrongly rejects multi-line flow keys.
let tooFarBack = key.mark.index + 1024 < mark.index
let stale = index == 0 ? (key.mark.line < mark.line || tooFarBack) : tooFarBack
guard stale else {
continue
}
if key.required {
throw ScannerError(
context: "while scanning a simple key",
problem: "could not find expected ':'",
mark: key.mark,
)
}
simpleKeys[index].possible = false
}
}
/// Port of `yaml_parser_save_simple_key`. Record the current position as a
/// potential simple key for this flow level when one is allowed. A key is
/// required (a `:` must follow) when it sits exactly at the block indent.
mutating func saveSimpleKey() throws {
let required = flowLevel == 0 && indentEngine.indent == mark.column
guard simpleKeyAllowed else {
return
}
let key = SimpleKey(
possible: true,
required: required,
tokenNumber: indentEngine.tokensParsed + indentEngine.tokens.count,
mark: mark,
tabbed: tabbedSeparation,
)
try removeSimpleKey()
simpleKeys[simpleKeys.count - 1] = key
}
/// Port of `yaml_parser_remove_simple_key`. Drop the potential key at the
/// current flow level; removing a required key is an error.
mutating func removeSimpleKey() throws {
let top = simpleKeys.count - 1
if simpleKeys[top].possible, simpleKeys[top].required {
throw ScannerError(
context: "while scanning a simple key",
problem: "could not find expected ':'",
mark: simpleKeys[top].mark,
)
}
simpleKeys[top].possible = false
}
/// Port of `yaml_parser_increase_flow_level`. Push an empty simple key for
/// the new flow level, recording whether the level is a sequence.
mutating func increaseFlowLevel(isSequence: Bool) {
simpleKeys.append(.empty)
flowIsSequence.append(isSequence)
indentEngine.flowLevel += 1
}
/// Port of `yaml_parser_decrease_flow_level`. Pop the flow level's simple
/// key.
mutating func decreaseFlowLevel() {
if indentEngine.flowLevel > 0 {
indentEngine.flowLevel -= 1
simpleKeys.removeLast()
if !flowIsSequence.isEmpty {
flowIsSequence.removeLast()
}
}
}
}
}

View File

@ -0,0 +1,10 @@
extension PureYAML.Parsing.LibYAML {
/// A scanning error, mirroring libyaml's `yaml_parser_set_scanner_error`
/// (`context` plus `problem` at a mark). It is mapped to the package's public
/// `ParseError` when the port is integrated.
struct ScannerError: Swift.Error, Equatable {
var context: String
var problem: String
var mark: PureYAML.Parsing.Mark
}
}

View File

@ -0,0 +1,28 @@
extension PureYAML.Parsing.LibYAML {
/// Port of `yaml_simple_key_t`. A potential implicit ("simple") mapping key:
/// a node that may turn out to be a key once a `:` is seen on the same line.
/// `tokenNumber` records where its `KEY` token must be spliced into the queue.
struct SimpleKey: Equatable {
/// Whether a simple key is still possible at this flow level.
var possible: Bool
/// Whether a `:` is required (a key sitting exactly at the block indent).
var required: Bool
/// The absolute token number where the `KEY` token is inserted.
var tokenNumber: Int
/// The position the potential key started at.
var mark: PureYAML.Parsing.Mark
/// Whether this key is a JSON-style node (a quoted flow scalar). A `:`
/// adjacent to such a key is a value indicator even without a following
/// space, per the YAML flow "adjacent value" rule.
var isJSON = false
/// Whether a tab preceded this key in its indentation. A tab may be
/// separation but never indentation, so promoting such a key to a block
/// mapping key (a `:` follows) is an error.
var tabbed = false
/// The empty entry pushed for a new flow level (`{ 0, 0, 0, {0,0,0} }`).
static var empty: Self {
Self(possible: false, required: false, tokenNumber: 0, mark: .start)
}
}
}

View File

@ -0,0 +1,10 @@
extension PureYAML.Parsing.LibYAML {
/// A scanner token, mirroring `yaml_token_t`: a kind plus its start and end
/// marks. Token-specific payloads (scalar value, anchor name, tag) are added
/// as their fetchers are ported.
struct Token: Equatable {
var type: TokenType
var startMark: PureYAML.Parsing.Mark
var endMark: PureYAML.Parsing.Mark
}
}

View File

@ -0,0 +1,29 @@
extension PureYAML.Parsing.LibYAML {
/// A scanner token kind, mirroring `yaml_token_type_t`. The flow-collection,
/// anchor, tag, alias, and directive kinds are added as their fetchers are
/// ported; the block + plain-scalar core is present.
enum TokenType: Equatable {
case streamStart
case streamEnd
case documentStart
case documentEnd
case blockSequenceStart
case blockMappingStart
case blockEnd
case blockEntry
case flowSequenceStart
case flowMappingStart
case flowSequenceEnd
case flowMappingEnd
case flowEntry
case key
case value
case versionDirective(major: Int, minor: Int)
case tagDirective(handle: String, prefix: String)
case reservedDirective
case anchor(name: String)
case alias(name: String)
case tag(handle: String, suffix: String)
case scalar(value: String, style: PureYAML.Parsing.ScalarStyle)
}
}

View File

@ -0,0 +1,10 @@
extension PureYAML.Parsing {
/// Faithful Swift port of libyaml's scanner (`CYaml/src/scanner.c`). It is
/// built as a parallel implementation, verified token-for-token against
/// libyaml through the differential harness, to replace the hand-rolled
/// approximation whose indentation and simple-key handling the official test
/// suite surfaced as incorrect. The port proceeds function by function; each
/// piece mirrors its C counterpart so correctness is by construction rather
/// than re-derived. Members live under this namespace in their own files.
enum LibYAML {}
}

View File

@ -1,17 +0,0 @@
extension PureYAML.Parsing {
struct NodeProperties: Equatable {
var anchor: String?
var tag: String?
var mark: Mark?
static var none: Self {
Self(anchor: nil, tag: nil, mark: nil)
}
mutating func mergeUnset(from other: Self) {
anchor = anchor ?? other.anchor
tag = tag ?? other.tag
mark = mark ?? other.mark
}
}
}

View File

@ -1,6 +0,0 @@
extension PureYAML.Parsing {
struct NodeResult: Equatable {
var kind: NodeKind
var endMark: Mark
}
}

View File

@ -0,0 +1,39 @@
extension PureYAML.Parsing.ParseError {
/// Translate a ported-scanner error into the package's diagnostic-aware error,
/// converting libyaml's 0-based mark to the 1-based convention and mapping the
/// well-known scanner failures to precise diagnostic codes.
init(translating error: PureYAML.Parsing.LibYAML.ScannerError) {
let line = error.mark.line + 1
let column = error.mark.column + 1
let context = error.context ?? ""
if error.problem.hasPrefix("found a tab") {
self = .tabIndentation(line: line)
} else if context == "while scanning a quoted scalar", error.problem.hasSuffix("end of stream") {
self = .unterminatedQuotedString(line: line)
} else if context.hasPrefix("while scanning a tag") || context.hasSuffix("TAG directive") {
self = .unterminatedTag(line: line)
} else {
self = .malformed(reason: Self.reason(context: error.context, problem: error.problem), line: line, column: column)
}
}
/// Translate a ported-parser error into the package's diagnostic-aware error.
init(translating error: PureYAML.Parsing.LibYAML.ParserError) {
let line = error.mark.line + 1
let column = error.mark.column + 1
if error.problem == "found incompatible YAML document" {
self = .incompatibleYAMLDirective(line: line)
} else {
self = .malformed(reason: Self.reason(context: error.context, problem: error.problem), line: line, column: column)
}
}
/// Join a libyaml context and problem into one readable reason, e.g.
/// "while parsing a flow sequence, did not find expected ',' or ']'".
private static func reason(context: String?, problem: String) -> String {
guard let context else {
return problem
}
return "\(context), \(problem)"
}
}

View File

@ -1,13 +1,17 @@
public extension PureYAML.Parsing {
/// Parser error with source line information when available.
enum ParseError: Swift.Error, Equatable, Sendable, CustomStringConvertible {
case emptyDocument
case tabIndentation(line: Int)
case unexpectedIndentation(line: Int)
case mixedCollectionStyles(line: Int)
case expectedMappingKey(line: Int)
case expectedAnchorName(line: Int)
case expectedAliasName(line: Int)
case expectedNode(line: Int, column: Int)
case expectedScalarKey(line: Int, column: Int)
case incompatibleYAMLDirective(line: Int)
case directiveWithoutDocument(line: Int)
case invalidTaggedScalar(tag: String, value: String, line: Int, column: Int)
case invalidMergeValue(line: Int, column: Int)
case unterminatedTag(line: Int)
@ -18,25 +22,20 @@ public extension PureYAML.Parsing {
case unexpectedEvent(expected: String, actual: String, line: Int, column: Int)
case unexpectedToken(expected: String, actual: String, line: Int, column: Int)
case nestingTooDeep(line: Int, column: Int)
case commentNotPrecededByWhitespace(line: Int, column: Int)
case invalidEscape(sequence: String, line: Int)
case malformedDirective(name: String, line: Int)
case duplicateDirective(name: String, line: Int)
case directiveWithoutDocument(line: Int)
case contentAfterDocumentEnd(line: Int)
case blockSequenceAsInlineMappingValue(line: Int)
case multilineImplicitKey(line: Int)
case commentInMultilineScalar(line: Int)
case aliasWithProperties(line: Int, column: Int)
case inlineBlockSequenceAfterProperty(line: Int, column: Int)
case documentMarkerInScalar(line: Int)
case malformed(reason: String, line: Int, column: Int)
public var description: String {
switch self {
case .emptyDocument:
"document is empty"
case let .tabIndentation(line):
"tabs are not allowed for indentation at line \(line)"
case let .unexpectedIndentation(line):
"unexpected indentation at line \(line)"
case let .mixedCollectionStyles(line):
"mapping and sequence entries are mixed at line \(line)"
case let .expectedMappingKey(line):
"expected a mapping key at line \(line)"
case let .expectedAnchorName(line):
"expected an anchor name at line \(line)"
case let .expectedAliasName(line):
@ -47,6 +46,8 @@ public extension PureYAML.Parsing {
"expected a scalar mapping key at line \(line), column \(column)"
case let .incompatibleYAMLDirective(line):
"incompatible YAML directive at line \(line)"
case let .directiveWithoutDocument(line):
"directive is not followed by a document start at line \(line)"
case let .invalidTaggedScalar(tag, value, line, column):
"invalid scalar '\(value)' for tag '\(tag)' at line \(line), column \(column)"
case let .invalidMergeValue(line, column):
@ -67,39 +68,23 @@ public extension PureYAML.Parsing {
"expected \(expected) at line \(line), column \(column), found \(actual)"
case let .nestingTooDeep(line, column):
"nesting is too deep at line \(line), column \(column)"
case let .commentNotPrecededByWhitespace(line, column):
"a comment must be preceded by whitespace at line \(line), column \(column)"
case let .invalidEscape(sequence, line):
"invalid escape sequence '\(sequence)' at line \(line)"
case let .malformedDirective(name, line):
"malformed directive '\(name)' at line \(line)"
case let .duplicateDirective(name, line):
"duplicate directive '\(name)' at line \(line)"
case let .directiveWithoutDocument(line):
"a directive must be followed by a document at line \(line)"
case let .contentAfterDocumentEnd(line):
"no content may follow a document end marker at line \(line)"
case let .blockSequenceAsInlineMappingValue(line):
"a block sequence cannot start on the same line as a mapping value at line \(line)"
case let .multilineImplicitKey(line):
"an implicit mapping key must not span multiple lines at line \(line)"
case let .commentInMultilineScalar(line):
"a comment cannot interrupt a multi-line plain scalar at line \(line)"
case let .aliasWithProperties(line, column):
"an alias node cannot have an anchor or tag at line \(line), column \(column)"
case let .inlineBlockSequenceAfterProperty(line, column):
"a block sequence cannot start on the same line as its anchor or tag at line \(line), column \(column)"
case let .documentMarkerInScalar(line):
"a document marker cannot appear inside a multi-line scalar at line \(line)"
case let .malformed(reason, line, column):
"\(reason) at line \(line), column \(column)"
}
}
public var diagnosticCode: String {
switch self {
case .emptyDocument:
"emptyDocument"
case .tabIndentation:
"tabIndentation"
case .unexpectedIndentation:
"unexpectedIndentation"
case .mixedCollectionStyles:
"mixedCollectionStyles"
case .expectedMappingKey:
"expectedMappingKey"
case .expectedAnchorName:
"expectedAnchorName"
case .expectedAliasName:
@ -110,6 +95,8 @@ public extension PureYAML.Parsing {
"expectedScalarKey"
case .incompatibleYAMLDirective:
"incompatibleYAMLDirective"
case .directiveWithoutDocument:
"directiveWithoutDocument"
case .invalidTaggedScalar:
"invalidTaggedScalar"
case .invalidMergeValue:
@ -130,53 +117,27 @@ public extension PureYAML.Parsing {
"unexpectedToken"
case .nestingTooDeep:
"nestingTooDeep"
case .commentNotPrecededByWhitespace:
"commentNotPrecededByWhitespace"
case .invalidEscape:
"invalidEscape"
case .malformedDirective:
"malformedDirective"
case .duplicateDirective:
"duplicateDirective"
case .directiveWithoutDocument:
"directiveWithoutDocument"
case .contentAfterDocumentEnd:
"contentAfterDocumentEnd"
case .blockSequenceAsInlineMappingValue:
"blockSequenceAsInlineMappingValue"
case .multilineImplicitKey:
"multilineImplicitKey"
case .commentInMultilineScalar:
"commentInMultilineScalar"
case .aliasWithProperties:
"aliasWithProperties"
case .inlineBlockSequenceAfterProperty:
"inlineBlockSequenceAfterProperty"
case .documentMarkerInScalar:
"documentMarkerInScalar"
case .malformed:
"malformed"
}
}
public var sourceLine: Int? {
switch self {
case let .unexpectedIndentation(line),
case .emptyDocument:
nil
case let .tabIndentation(line),
let .unexpectedIndentation(line),
let .mixedCollectionStyles(line),
let .expectedMappingKey(line),
let .expectedAnchorName(line),
let .expectedAliasName(line),
let .incompatibleYAMLDirective(line),
let .directiveWithoutDocument(line),
let .unterminatedTag(line),
let .unterminatedQuotedString(line),
let .unsupportedDirective(_, line),
let .unsupportedMultiDocumentStream(line),
let .documentMarkerInScalar(line),
let .invalidEscape(_, line),
let .malformedDirective(_, line),
let .duplicateDirective(_, line),
let .directiveWithoutDocument(line),
let .contentAfterDocumentEnd(line),
let .blockSequenceAsInlineMappingValue(line),
let .multilineImplicitKey(line),
let .commentInMultilineScalar(line):
let .unsupportedMultiDocumentStream(line):
line
case let .expectedNode(line, _),
let .expectedScalarKey(line, _),
@ -186,33 +147,26 @@ public extension PureYAML.Parsing {
let .unexpectedEvent(_, _, line, _),
let .unexpectedToken(_, _, line, _),
let .nestingTooDeep(line, _),
let .commentNotPrecededByWhitespace(line, _),
let .aliasWithProperties(line, _),
let .inlineBlockSequenceAfterProperty(line, _):
let .malformed(_, line, _):
line
}
}
public var sourceColumn: Int? {
switch self {
case .unexpectedIndentation,
case .emptyDocument,
.tabIndentation,
.unexpectedIndentation,
.mixedCollectionStyles,
.expectedMappingKey,
.expectedAnchorName,
.expectedAliasName,
.incompatibleYAMLDirective,
.directiveWithoutDocument,
.unterminatedTag,
.unterminatedQuotedString,
.unsupportedDirective,
.unsupportedMultiDocumentStream,
.documentMarkerInScalar,
.invalidEscape,
.malformedDirective,
.duplicateDirective,
.directiveWithoutDocument,
.contentAfterDocumentEnd,
.blockSequenceAsInlineMappingValue,
.multilineImplicitKey,
.commentInMultilineScalar:
.unsupportedMultiDocumentStream:
nil
case let .expectedNode(_, column),
let .expectedScalarKey(_, column),
@ -222,9 +176,7 @@ public extension PureYAML.Parsing {
let .unexpectedEvent(_, _, _, column),
let .unexpectedToken(_, _, _, column),
let .nestingTooDeep(_, column),
let .commentNotPrecededByWhitespace(_, column),
let .aliasWithProperties(_, column),
let .inlineBlockSequenceAfterProperty(_, column):
let .malformed(_, _, column):
column
}
}

View File

@ -1,8 +1,46 @@
extension PureYAML.Parsing.Parser {
func parseEvents(_ yaml: String) throws -> [PureYAML.Parsing.Event] {
let tokens = try PureYAML.Parsing.Scanner().scan(yaml)
var parser = PureYAML.Parsing.TokenEventParser(tokens: tokens, scalarParser: self)
return try parser.parse()
let events: [PureYAML.Parsing.Event]
do {
var parser = try PureYAML.Parsing.LibYAML.Parser(yaml)
events = try parser.parse().map { $0.bumpingMark() }
} catch let error as PureYAML.Parsing.LibYAML.ScannerError {
throw PureYAML.Parsing.ParseError(translating: error)
} catch let error as PureYAML.Parsing.LibYAML.ParserError {
throw PureYAML.Parsing.ParseError(translating: error)
}
try enforceNestingLimit(events)
return events
}
/// The deepest collection nesting the value/event composers will build. The
/// ported parser produces events iteratively without recursion, but the
/// composers, equality, hashing, and `Model/Value` teardown all recurse with
/// the nesting depth, so the event stream is bounded here before any recursive
/// consumer runs. 64 accepts every realistic document yet rejects pathological
/// input well before it can overflow the stack.
static var maxNestingDepth: Int {
64
}
/// Reject an event stream that nests collections past ``maxNestingDepth`` with
/// a ``ParseError/nestingTooDeep(line:column:)`` so deeply nested input fails
/// cleanly instead of overflowing a recursive consumer's stack.
private func enforceNestingLimit(_ events: [PureYAML.Parsing.Event]) throws {
var depth = 0
for event in events {
switch event {
case .sequenceStart, .mappingStart:
depth += 1
if depth > Self.maxNestingDepth {
throw PureYAML.Parsing.ParseError.nestingTooDeep(line: event.mark.line, column: event.mark.column)
}
case .sequenceEnd, .mappingEnd:
depth -= 1
default:
break
}
}
}
func parseStreaming(_ yaml: String) throws -> PureYAML.Model.Value {
@ -10,14 +48,9 @@ extension PureYAML.Parsing.Parser {
scalarParser: self,
mode: .singleDocument,
)
var parser = PureYAML.Parsing.TokenEventParser(
yaml: yaml,
scalarParser: self,
eventSink: { event in
try composer.consume(event)
},
)
try parser.parseStreaming()
for event in try parseEvents(yaml) {
try composer.consume(event)
}
return try composer.composedValue()
}
@ -26,14 +59,9 @@ extension PureYAML.Parsing.Parser {
scalarParser: self,
mode: .stream,
)
var parser = PureYAML.Parsing.TokenEventParser(
yaml: yaml,
scalarParser: self,
eventSink: { event in
try composer.consume(event)
},
)
try parser.parseStreaming()
for event in try parseEvents(yaml) {
try composer.consume(event)
}
return try composer.composedStream()
}
@ -43,14 +71,9 @@ extension PureYAML.Parsing.Parser {
mode: .singleDocument,
tracksLocations: true,
)
var parser = PureYAML.Parsing.TokenEventParser(
yaml: yaml,
scalarParser: self,
eventSink: { event in
try composer.consume(event)
},
)
try parser.parseStreaming()
for event in try parseEvents(yaml) {
try composer.consume(event)
}
return try composer.composedLocated()
}
@ -60,14 +83,9 @@ extension PureYAML.Parsing.Parser {
mode: .stream,
tracksLocations: true,
)
var parser = PureYAML.Parsing.TokenEventParser(
yaml: yaml,
scalarParser: self,
eventSink: { event in
try composer.consume(event)
},
)
try parser.parseStreaming()
for event in try parseEvents(yaml) {
try composer.consume(event)
}
return try composer.composedLocatedStream()
}
}

View File

@ -35,6 +35,10 @@ extension PureYAML.Parsing.Parser {
return nil
}
switch tag {
case "!":
// The non-specific `!` tag resolves a scalar by kind alone: a scalar is
// a string, with no implicit typing, so `! 12` is "12" not the int 12.
return .string(value)
case "tag:yaml.org,2002:str":
return .string(value)
case "tag:yaml.org,2002:int":

View File

@ -110,27 +110,8 @@ extension PureYAML.Parsing.Parser {
guard text.count >= 2, text.last == "'" else {
throw PureYAML.Parsing.ParseError.unterminatedQuotedString(line: line)
}
let inner = text.dropFirst().dropLast()
var output = ""
var index = inner.startIndex
while index < inner.endIndex {
let character = inner[index]
if character == "'" {
// A doubled quote escapes to a single quote; a lone quote (not normally present
// inside the stripped body) is kept as-is.
inner.formIndex(after: &index)
output.append("'")
if index < inner.endIndex, inner[index] == "'" {
inner.formIndex(after: &index)
}
} else if character == "\n" || character == "\r" {
foldFlowScalarLineBreaks(in: inner, index: &index, to: &output)
} else {
output.append(character)
inner.formIndex(after: &index)
}
}
return output
let inner = Array(text.dropFirst().dropLast())
return foldFlowScalar(inner, single: true)
}
func parseDoubleQuoted(
@ -140,184 +121,238 @@ extension PureYAML.Parsing.Parser {
guard text.count >= 2, text.last == "\"" else {
throw PureYAML.Parsing.ParseError.unterminatedQuotedString(line: line)
}
let inner = text.dropFirst().dropLast()
let inner = Array(text.dropFirst().dropLast())
return foldFlowScalar(inner, single: false)
}
/// Flow (quoted) scalar line folding, mirroring libyaml's
/// yaml_parser_scan_flow_scalar (single and double quoted). Trailing
/// whitespace before a break is dropped, leading whitespace on a
/// continuation line is dropped, a single break between non-empty lines
/// folds to one space, and N consecutive breaks yield N-1 line feeds.
/// For double quoted scalars a backslash at end of line suppresses the
/// break entirely.
func foldFlowScalar(
_ inner: [Character],
single: Bool,
) -> String {
var state = FlowFoldState()
let count = inner.count
while state.index < count {
state.leadingBlanks = false
consumeFlowContent(inner, single: single, state: &state)
consumeFlowBlanks(inner, state: &state)
joinFlowFold(state: &state)
}
return state.output
}
/// Mutable accumulators for flow scalar folding, matching the C reference's
/// string / leading_break / trailing_breaks / whitespaces buffers.
struct FlowFoldState {
var output = ""
var index = inner.startIndex
while index < inner.endIndex {
let character = inner[index]
if character == "\\" {
inner.formIndex(after: &index)
guard index < inner.endIndex else {
output.append("\\")
var whitespaces = ""
var leadingBreak = ""
var trailingBreaks = ""
var leadingBlanks = false
var index = 0
}
/// Consume the run of non-blank characters at `state.index`, decoding
/// escapes. A double quoted backslash at end of line sets `leadingBlanks`
/// and stops, so the following break is suppressed during folding.
func consumeFlowContent(
_ inner: [Character],
single: Bool,
state: inout FlowFoldState,
) {
let count = inner.count
while state.index < count, !isFlowBlank(inner[state.index]), !isFlowBreak(inner[state.index]) {
if single, inner[state.index] == "'" {
state.output.append("'")
let isEscapedQuote = state.index + 1 < count && inner[state.index + 1] == "'"
state.index += isEscapedQuote ? 2 : 1
continue
}
if !single, inner[state.index] == "\\" {
if state.index + 1 < count, isFlowBreak(inner[state.index + 1]) {
state.index += 2
state.leadingBlanks = true
break
}
try appendDoubleQuotedEscape(in: inner, index: &index, to: &output, line: line)
} else if character == "\n" || character == "\r" {
foldFlowScalarLineBreaks(in: inner, index: &index, to: &output)
} else {
output.append(character)
inner.formIndex(after: &index)
state.index += 1
appendDoubleQuotedEscape(inner, index: &state.index, to: &state.output)
continue
}
state.output.append(inner[state.index])
state.index += 1
}
return output
}
/// Flow line folding for a quoted (single- or double-) scalar at a line break. Whitespace
/// adjacent to the break is stripped (the current line's trailing, each following line's
/// leading), then a single break folds to one space and a run of breaks (blank lines) folds to
/// one fewer newline. The scalar's first-line leading whitespace and last-line trailing
/// whitespace are untouched, since they sit next to a quote, not a break.
func foldFlowScalarLineBreaks(
in text: Substring,
index: inout Substring.Index,
/// Consume the run of blanks and line breaks at `state.index`, sorting them
/// into the whitespace / leading-break / trailing-break buffers.
func consumeFlowBlanks(
_ inner: [Character],
state: inout FlowFoldState,
) {
let count = inner.count
while state.index < count, isFlowBlank(inner[state.index]) || isFlowBreak(inner[state.index]) {
if isFlowBlank(inner[state.index]) {
if !state.leadingBlanks {
state.whitespaces.append(inner[state.index])
}
} else if !state.leadingBlanks {
state.whitespaces = ""
state.leadingBreak = "\n"
state.leadingBlanks = true
} else {
state.trailingBreaks.append("\n")
}
state.index += 1
}
}
/// Join the captured whitespace or fold the captured line breaks into the
/// output, per YAML 1.2 flow folding.
func joinFlowFold(state: inout FlowFoldState) {
guard state.leadingBlanks else {
state.output.append(state.whitespaces)
state.whitespaces = ""
return
}
if state.leadingBreak.first == "\n" {
if state.trailingBreaks.isEmpty {
state.output.append(" ")
} else {
state.output.append(state.trailingBreaks)
state.trailingBreaks = ""
}
state.leadingBreak = ""
} else {
state.output.append(state.leadingBreak)
state.output.append(state.trailingBreaks)
state.leadingBreak = ""
state.trailingBreaks = ""
}
}
func isFlowBlank(_ character: Character) -> Bool {
character == " " || character == "\t"
}
func isFlowBreak(_ character: Character) -> Bool {
character == "\n" || character == "\r" || character == "\r\n"
}
/// Decode a double quoted escape. On entry index points at the escape
/// character (the position just after the backslash). On return index has
/// advanced past every character the escape consumed. The escape set
/// mirrors libyaml's yaml_parser_scan_flow_scalar.
func appendDoubleQuotedEscape(
_ text: [Character],
index: inout Int,
to output: inout String,
) {
while let last = output.last, last == " " || last == "\t" {
output.removeLast()
guard index < text.count else {
output.append("\\")
return
}
var breaks = 0
while index < text.endIndex {
let character = text[index]
if character == "\r" {
text.formIndex(after: &index)
if index < text.endIndex, text[index] == "\n" {
text.formIndex(after: &index)
}
breaks += 1
} else if character == "\n" {
text.formIndex(after: &index)
breaks += 1
} else if character == " " || character == "\t" {
text.formIndex(after: &index)
} else {
break
}
}
output.append(breaks == 1 ? " " : String(repeating: "\n", count: breaks - 1))
}
/// Decodes one double-quoted escape, the character after a `\`. The set is exactly YAML 1.2's
/// escape table; anything else is rejected rather than silently passed through. The index enters
/// at the escape character and is left on the last consumed character, so the caller's trailing
/// advance steps past it.
func appendDoubleQuotedEscape(
in text: Substring,
index: inout Substring.Index,
to output: inout String,
line: Int,
) throws {
let character = text[index]
if let replacement = Self.simpleDoubleQuotedEscapes[character] {
output.append(replacement)
} else {
switch character {
case "x": try appendHexEscape(in: text, index: &index, to: &output, digits: 2, line: line)
case "u": try appendUnicodeEscape(in: text, index: &index, to: &output, line: line)
case "U": try appendHexEscape(in: text, index: &index, to: &output, digits: 8, line: line)
case "\n", "\r": consumeEscapedLineBreak(in: text, index: &index)
default:
throw PureYAML.Parsing.ParseError.invalidEscape(sequence: "\\\(character)", line: line)
}
if let simple = Self.simpleDoubleQuotedEscapes[character] {
output.append(simple)
index += 1
return
}
switch character {
case "x":
index += 1
appendHexEscape(text, index: &index, digits: 2, marker: "x", to: &output)
case "u":
index += 1
appendUnicodeEscape(text, index: &index, to: &output)
case "U":
index += 1
appendHexEscape(text, index: &index, digits: 8, marker: "U", to: &output)
default:
output.append(character)
index += 1
}
text.formIndex(after: &index)
}
/// The YAML 1.2 escapes that map a single character after `\` to one character. Escapes that
/// consume following characters (`\x`, `\u`, `\U`, an escaped line break) are handled separately.
/// Single character double quoted escapes, matching libyaml's escape table.
static let simpleDoubleQuotedEscapes: [Character: Character] = [
"0": "\u{00}", "a": "\u{07}", "b": "\u{08}", "t": "\t", "\t": "\t",
"n": "\n", "v": "\u{0B}", "f": "\u{0C}", "r": "\r", "e": "\u{1B}",
" ": " ", "\"": "\"", "/": "/", "\\": "\\",
"N": "\u{85}", "_": "\u{A0}", "L": "\u{2028}", "P": "\u{2029}",
"0": "\u{00}",
"a": "\u{07}",
"b": "\u{08}",
"t": "\t",
"\t": "\t",
"n": "\n",
"v": "\u{0B}",
"f": "\u{0C}",
"r": "\r",
"e": "\u{1B}",
" ": " ",
"\"": "\"",
"/": "/",
"\\": "\\",
"N": "\u{85}",
"_": "\u{A0}",
"L": "\u{2028}",
"P": "\u{2029}",
]
/// Handles a `\` immediately before a line break: an escaped line break, where the two lines
/// join with no separator. Consumes the break (LF, CR, or CRLF) and the continuation line's
/// leading spaces and tabs, emitting nothing, and leaves the index on the last consumed
/// character so the caller's advance steps to the continuation's first content character.
func consumeEscapedLineBreak(
in text: Substring,
index: inout Substring.Index,
) {
if text[index] == "\r" {
var next = index
text.formIndex(after: &next)
if next < text.endIndex, text[next] == "\n" {
index = next
}
}
while true {
var next = index
text.formIndex(after: &next)
guard next < text.endIndex, text[next] == " " || text[next] == "\t" else {
break
}
index = next
}
}
/// Reads a fixed-width hex escape (`\xHH`, `\UHHHHHHHH`) and appends the scalar it denotes,
/// rejecting a truncated sequence or a value that is not a valid Unicode scalar.
/// Decode \xXX or \UXXXXXXXX. On entry index points at the first hex digit.
func appendHexEscape(
in text: Substring,
index: inout Substring.Index,
to output: inout String,
_ text: [Character],
index: inout Int,
digits: Int,
line: Int,
) throws {
let typeCharacter = text[index]
guard let value = readHexEscape(in: text, index: &index, digits: digits),
let scalar = UnicodeScalar(value)
else {
throw PureYAML.Parsing.ParseError.invalidEscape(sequence: "\\\(typeCharacter)", line: line)
marker: Character,
to output: inout String,
) {
guard let value = readHex(text, index: &index, digits: digits) else {
output.append(marker)
return
}
if let scalar = UnicodeScalar(value) {
output.unicodeScalars.append(scalar)
} else {
output.append(marker)
}
output.unicodeScalars.append(scalar)
}
/// Decode \uXXXX, including surrogate pairs. On entry index points at the
/// first hex digit.
func appendUnicodeEscape(
in text: Substring,
index: inout Substring.Index,
_ text: [Character],
index: inout Int,
to output: inout String,
line: Int,
) throws {
guard let codeUnit = readHexEscape(in: text, index: &index, digits: 4) else {
throw PureYAML.Parsing.ParseError.invalidEscape(sequence: "\\u", line: line)
) {
let start = index
guard let codeUnit = readHex(text, index: &index, digits: 4) else {
output.append("u")
index = start
return
}
if let scalar = readSurrogatePairIfNeeded(high: codeUnit, in: text, index: &index) {
if isHighSurrogate(codeUnit), let scalar = readSurrogatePair(high: codeUnit, text: text, index: &index) {
output.unicodeScalars.append(scalar)
} else if let scalar = UnicodeScalar(codeUnit) {
output.unicodeScalars.append(scalar)
} else {
throw PureYAML.Parsing.ParseError.invalidEscape(sequence: "\\u", line: line)
output.append("u")
}
}
func readSurrogatePairIfNeeded(
high: UInt32,
in text: Substring,
index: inout Substring.Index,
) -> UnicodeScalar? {
guard isHighSurrogate(high) else {
return nil
}
return readSurrogatePair(high: high, in: text, index: &index)
}
func readSurrogatePair(
high: UInt32,
in text: Substring,
index: inout Substring.Index,
text: [Character],
index: inout Int,
) -> UnicodeScalar? {
var probe = index
text.formIndex(after: &probe)
guard probe < text.endIndex, text[probe] == "\\" else {
guard probe + 1 < text.count, text[probe] == "\\", text[probe + 1] == "u" else {
return nil
}
text.formIndex(after: &probe)
guard probe < text.endIndex, text[probe] == "u",
let low = readHexEscape(in: text, index: &probe, digits: 4),
isLowSurrogate(low)
else {
probe += 2
guard let low = readHex(text, index: &probe, digits: 4), isLowSurrogate(low) else {
return nil
}
index = probe
@ -325,26 +360,25 @@ extension PureYAML.Parsing.Parser {
return UnicodeScalar(value)
}
/// Reads `digits` hex characters following the escape type character. The index enters at the
/// type character and is left on the final hex digit; returns nil if fewer than `digits` hex
/// characters are available.
func readHexEscape(
in text: Substring,
index: inout Substring.Index,
/// Read exactly `digits` hex characters starting at index. On success index
/// advances past them, otherwise index is left unchanged.
func readHex(
_ text: [Character],
index: inout Int,
digits: Int,
) -> UInt32? {
var probe = index
text.formIndex(after: &probe)
var hex = ""
for _ in 0 ..< digits {
guard probe < text.endIndex, text[probe].isHexDigit else {
guard index + digits <= text.count else {
return nil
}
var value: UInt32 = 0
for offset in 0 ..< digits {
guard let digit = text[index + offset].hexDigitValue else {
return nil
}
hex.append(text[probe])
text.formIndex(after: &probe)
value = (value << 4) + UInt32(digit)
}
index = text.index(before: probe)
return UInt32(hex, radix: 16)
index += digits
return value
}
func isHighSurrogate(_ value: UInt32) -> Bool {

View File

@ -1,167 +0,0 @@
extension PureYAML.Parsing.Scanner {
func scanDirectiveOrDocumentMarker(_ state: inout State) throws -> Bool {
guard state.indentation.last == 0 else {
return false
}
// A `%` opens a directive only outside a block scalar. Inside one (a zero-indent literal or
// folded scalar sits at column 0, where this directive check runs) a leading `%` is literal
// content, so let the main scanner read the line. Elsewhere the directive path still runs, so
// a stray `%` in other content is rejected as before, not silently accepted.
if state.reader.peek() == "%", state.blockScalarIndentation == nil {
try scanDirective(&state)
return true
}
if state.reader.peek() == "#" {
return false
}
if startsWith("---", state.reader), isIndicatorBoundary(state.reader.peek(offset: 3)) {
try scanDocumentMarker(.documentStart, state: &state)
resetForDocumentStart(&state)
return true
}
if startsWith("...", state.reader), isIndicatorBoundary(state.reader.peek(offset: 3)) {
// A `...` after a document is already closed (only comments or blank lines since the
// last `...`) ends nothing: it is a repeated document-end marker, which YAML allows. Skip
// it as a no-op rather than emitting a second documentEnd (which would compose a phantom
// null document) or rejecting a valid stream. Content after it is still an error.
if state.isDocumentClosed {
state.reader.advance()
state.reader.advance()
state.reader.advance()
while state.reader.peek() == " " {
state.reader.advance()
}
if let next = state.reader.peek(), !isLineBreak(next), next != "#" {
throw PureYAML.Parsing.ParseError.contentAfterDocumentEnd(line: state.reader.mark.line)
}
skipLine(&state)
return true
}
// A `...` ends a document, so a directive still awaiting its `---` never gets one.
try enforceDirectiveHasDocument(state)
try scanDocumentMarker(.documentEnd, state: &state)
resetForDocumentEnd(&state)
return true
}
// Content after a document-end marker (`...`) begins a new bare document, so reopen the
// stream rather than rejecting it.
if state.isDocumentClosed {
state.isDocumentClosed = false
}
return false
}
func scanDocumentMarker(
_ kind: PureYAML.Parsing.TokenKind,
state: inout State,
) throws {
let start = state.reader.mark
state.reader.advance()
state.reader.advance()
state.reader.advance()
state.append(kind, mark: start, endMark: state.reader.mark)
while state.reader.peek() == " " {
state.reader.advance()
}
if let next = state.reader.peek(), !isLineBreak(next), next != "#" {
// The marker's own line carries real content. A document start may carry the root node
// here (e.g. `--- >`, `--- foo`, `--- {a: 1}`), so leave it for the main loop to scan.
if case .documentStart = kind {
state.isAtLineStart = false
return
}
// A document end (`...`) takes no same-line node; only a comment or blank may follow it.
throw PureYAML.Parsing.ParseError.contentAfterDocumentEnd(line: state.reader.mark.line)
}
skipLine(&state)
}
func resetForDocumentStart(_ state: inout State) {
state.hasDocumentContent = false
state.hasDocumentStartMarker = true
state.isDocumentClosed = false
state.hasYAMLDirective = false
state.pendingDirectiveLine = nil
// The directives that preceded this `---` apply to this document only: merge them onto the
// defaults, then clear them so they do not leak into the next document.
state.tagHandles = State.defaultTagHandles.merging(state.pendingTagHandles) { _, new in new }
state.pendingTagHandles = [:]
}
/// A directive must be followed by an explicit `---` document start. Called at a document end
/// and at stream end, where a still-pending directive means none ever came.
func enforceDirectiveHasDocument(_ state: State) throws {
if let line = state.pendingDirectiveLine {
throw PureYAML.Parsing.ParseError.directiveWithoutDocument(line: line)
}
}
func resetForDocumentEnd(_ state: inout State) {
state.hasDocumentContent = false
state.hasDocumentStartMarker = false
state.isDocumentClosed = true
state.hasYAMLDirective = false
// A document boundary ends the scope of that document's tag handles.
state.tagHandles = State.defaultTagHandles
state.pendingTagHandles = [:]
}
func scanDirective(_ state: inout State) throws {
let start = state.reader.mark
let line = state.reader.consume { character in
!isLineBreak(character)
}
let parts = line.split(separator: " ").map(String.init)
guard let name = parts.first else {
return
}
// A directive after a document-end marker (`...`) begins the next document's directive
// section, so it reopens the stream for a new document rather than being rejected.
if state.isDocumentClosed {
state.isDocumentClosed = false
}
guard !state.hasDocumentContent, !state.hasDocumentStartMarker else {
throw PureYAML.Parsing.ParseError.unsupportedDirective(name: name, line: start.line)
}
// Every directive must be followed by an explicit `---`; record that one is now awaiting it.
state.pendingDirectiveLine = start.line
switch name {
case "%YAML":
// Any version with major 1 is processable: a higher minor than 1.2 (`%YAML 1.3`) is
// accepted, since the spec calls for processing it, with a warning, rather than rejecting
// it. Only a different major version, or a malformed one, is incompatible.
guard parts.indices.contains(1), isProcessableYAMLVersion(parts[1]) else {
throw PureYAML.Parsing.ParseError.incompatibleYAMLDirective(line: start.line)
}
// `%YAML` takes exactly one parameter, the version; only a comment may follow it.
guard parts.count == 2 || parts[2].hasPrefix("#") else {
throw PureYAML.Parsing.ParseError.malformedDirective(name: name, line: start.line)
}
// At most one `%YAML` directive is allowed per document.
guard !state.hasYAMLDirective else {
throw PureYAML.Parsing.ParseError.duplicateDirective(name: name, line: start.line)
}
state.hasYAMLDirective = true
case "%TAG":
guard parts.count >= 3 else {
return
}
state.pendingTagHandles[parts[1]] = parts[2]
default:
// A YAML processor ignores a directive it does not recognize (the spec calls for a
// warning and otherwise no effect), rather than rejecting the document. The line has
// already been consumed, so there is nothing more to do.
return
}
}
/// Whether a `%YAML` version is one this processor can attempt: any `major.minor` with major 1,
/// so 1.0 through 1.2 and a higher minor like 1.3 are all accepted; a different major (2.0) or a
/// non-numeric version is not.
func isProcessableYAMLVersion(_ version: String) -> Bool {
let components = version.split(separator: ".", omittingEmptySubsequences: false)
return components.count == 2
&& components[0] == "1"
&& components.allSatisfy { !$0.isEmpty && $0.allSatisfy(\.isNumber) }
}
}

View File

@ -1,11 +0,0 @@
extension PureYAML.Parsing {
struct ScannerFlowContext {
var kind: ScannerFlowContextKind
var isExpectingKey: Bool
/// Whether an explicit key indicator (`?`) is open at this flow level, so the following scalar
/// is the explicit key's content and must not synthesize its own implicit key. Cleared by the
/// pair's `:` value indicator or a `,`, and discarded with the context when the collection
/// closes, so it tracks per-level and a nested collection's `:` does not affect an outer key.
var hasExplicitKey = false
}
}

View File

@ -1,89 +0,0 @@
extension PureYAML.Parsing.Scanner {
/// Emits the implicit-key marker before a block-level flow collection that a `: ` follows, so the
/// flow node is parsed as a mapping key (e.g. `[a, b]: value`) through the same path a scalar key
/// takes. Skipped inside a flow context, after a mapping value already on the line (block style
/// allows one pair per line), and under a pending explicit `?` key, which marks its own key.
func appendFlowImplicitKeyMarkerIfNeeded(state: inout State) {
guard state.flowDepth == 0, !state.sawMappingValueOnLine, !state.pendingExplicitKey,
flowCollectionIsImplicitKey(state: state)
else {
return
}
let mark = state.reader.mark
state.append(.mappingKey, mark: mark, endMark: mark)
}
/// Looks ahead from a block-level flow collection opener (`[` or `{`) to decide whether the whole
/// collection is an implicit mapping key: a flow node immediately followed by a `: ` mapping-value
/// indicator, e.g. `[a, b]: value` or `{a: 1}: value`. Quoted scalars are skipped so brackets and
/// colons inside them do not affect the nesting count, and the `:` must sit on the same line as
/// the close (only spaces between), since a block implicit key carries its `:` on its own line.
/// Returns false if the brackets do not balance before end of input.
func flowCollectionIsImplicitKey(state: State) -> Bool {
var depth = 0
var offset = 0
while let character = state.reader.peek(offset: offset) {
// A block implicit key must lie on a single line, so a flow collection that spans a line
// break (outside any quoted scalar) before its `:` cannot be one: `[23\n]: 42` is an error.
if isLineBreak(character) {
return false
}
switch character {
case "'":
offset = skipSingleQuotedLookahead(after: offset, state: state)
case "\"":
offset = skipDoubleQuotedLookahead(after: offset, state: state)
case "[", "{":
depth += 1
offset += 1
case "]", "}":
depth -= 1
offset += 1
guard depth > 0 else {
while state.reader.peek(offset: offset) == " " {
offset += 1
}
return state.reader.peek(offset: offset) == ":"
&& isBlockMappingValueBoundary(state.reader.peek(offset: offset + 1))
}
default:
offset += 1
}
}
return false
}
/// Returns the offset just past the closing `'` of a single-quoted scalar that opened at `open`,
/// treating a doubled `''` as an escaped quote rather than a close.
private func skipSingleQuotedLookahead(after open: Int, state: State) -> Int {
var offset = open + 1
while let character = state.reader.peek(offset: offset) {
if character == "'" {
if state.reader.peek(offset: offset + 1) == "'" {
offset += 2
continue
}
return offset + 1
}
offset += 1
}
return offset
}
/// Returns the offset just past the closing `"` of a double-quoted scalar that opened at `open`,
/// skipping `\`-escaped characters so an escaped quote does not end the scan early.
private func skipDoubleQuotedLookahead(after open: Int, state: State) -> Int {
var offset = open + 1
while let character = state.reader.peek(offset: offset) {
if character == "\\" {
offset += 2
continue
}
if character == "\"" {
return offset + 1
}
offset += 1
}
return offset
}
}

View File

@ -1,379 +0,0 @@
extension PureYAML.Parsing.Scanner {
/// A blank line inside a block scalar is preserved as empty content so that
/// chomping and folding can apply to it. The guard keeps blank lines outside
/// block scalars unaffected; `isAtLineStart` is true here only when the
/// previous line already ended, so a content line's own terminating break
/// does not emit a blank. Blank lines between the header and the first
/// content line precede the indent token, so they are buffered and emitted
/// once the content indentation is established.
func captureBlockScalarBlankLine(_ state: inout State) {
guard state.isAtLineStart else {
return
}
if state.blockScalarIndentation != nil {
// The prefix holds any surplus indentation past the block's content level for an
// otherwise-empty line (see `preserveBlockScalarSurplusOnBlankLine`); empty otherwise.
state.append(.scalar(value: state.blockScalarContentPrefix, style: .plain))
state.blockScalarContentPrefix = ""
} else if state.pendingBlockScalarIndentation {
state.pendingBlockScalarLeadingBlankWhitespace.append(state.lastLeadingWhitespace)
}
}
func startsWith(
_ prefix: String,
_ reader: PureYAML.Parsing.Reader,
) -> Bool {
for (offset, character) in prefix.enumerated() {
guard reader.peek(offset: offset) == character else {
return false
}
}
return true
}
func isLineBreak(_ character: Character?) -> Bool {
character == "\n" || character == "\r" || character == "\r\n"
}
func isBlockEntry(_ reader: PureYAML.Parsing.Reader) -> Bool {
reader.peek() == "-" && isIndicatorBoundary(reader.peek(offset: 1))
}
func isIndicatorBoundary(_ character: Character?) -> Bool {
guard let character else {
return true
}
// A tab is separation white space in YAML (only indentation forbids it), so it ends an
// indicator just as a space does: `---\tscalar`, `-\titem`, `?\tkey`.
return character == " " || character == "\t" || isLineBreak(character)
}
func isMappingValueBoundary(_ character: Character?) -> Bool {
guard let character else {
return true
}
// A tab is separation white space, so it follows a `:` value indicator as a space does.
return character == " "
|| character == "\t"
|| isLineBreak(character)
|| character == ","
|| character == "]"
|| character == "}"
}
func isBlockMappingValueBoundary(_ character: Character?) -> Bool {
guard let character else {
return true
}
return character == " " || character == "\t" || isLineBreak(character)
}
func isTokenTerminator(_ character: Character) -> Bool {
character == " "
|| isLineBreak(character)
|| character == ","
|| character == "["
|| character == "]"
|| character == "{"
|| character == "}"
|| character == "#"
}
func canStartAnchorName(_ character: Character?) -> Bool {
guard let character else {
return false
}
return !isTokenTerminator(character)
&& character != "*"
&& character != "&"
&& character != "!"
&& character != "\""
&& character != "'"
}
func hasNodeOnSameLine(afterCurrentPositionIn reader: PureYAML.Parsing.Reader) -> Bool {
var offset = 0
while reader.peek(offset: offset) == " " {
offset += 1
}
guard let character = reader.peek(offset: offset) else {
return false
}
return !isLineBreak(character) && character != "#"
}
func isFlowDelimiter(_ character: Character) -> Bool {
character == ","
|| character == "["
|| character == "]"
|| character == "{"
|| character == "}"
}
func isBlockScalarHeaderCharacter(_ character: Character) -> Bool {
character == "+"
|| character == "-"
|| character.isNumber
}
func trimTrailingSpaces(_ value: String) -> String {
var end = value.endIndex
while end > value.startIndex {
let previous = value.index(before: end)
guard value[previous] == " " else {
break
}
end = previous
}
return String(value[..<end])
}
func scanPlainScalarContinuations(
after firstLine: String,
firstLineIndent: Int,
state: inout State,
) -> String {
var value = firstLine
while true {
var probe = state.reader
guard isLineBreak(probe.peek()) else {
return value
}
probe.advance()
let blankLineCount = consumeBlankLines(&probe)
_ = probe.consume { character in
character == " "
}
let indentationWidth = probe.mark.column - 1
guard !probe.isAtEnd, !isLineBreak(probe.peek()), probe.peek() != "#" else {
return value
}
// A plain scalar never spans a document marker (`---` / `...`); stop the continuation at
// one so it begins (or ends) the document rather than folding into the scalar's value.
if indentationWidth == 0, isAtDocumentMarker(probe) {
return value
}
let baseIndentation = plainScalarContinuationBaseIndentation(state: state, firstLineIndent: firstLineIndent)
guard indentationWidth > baseIndentation else {
return value
}
if isBlockEntry(probe), state.validImplicitIndentation.contains(indentationWidth) {
return value
}
var continuationState = state
continuationState.reader = probe
let continuation = scanPlainScalarContinuationLine(&continuationState)
if isAtMappingValueIndicator(state: continuationState) {
// A continuation line ending at `:` is normally a new implicit key, so the scalar
// stops before it. Two cases keep folding instead, with the `:` left to be scanned as
// the value indicator: a block explicit key (`? foo` then `bar :`), where the line is
// still the key's content, and any flow scalar, since a flow key may span lines
// (`{ multi`/`line: v }`) where a block implicit key may not.
let foldsAcrossMappingValue = state.flowDepth > 0 || state.pendingExplicitKey
guard foldsAcrossMappingValue else {
return value
}
state = continuationState
value += blankLineCount == 0 ? " " : String(repeating: "\n", count: blankLineCount)
value += continuation
return value
}
state = continuationState
if blankLineCount == 0 {
value += " "
} else {
value += String(repeating: "\n", count: blankLineCount)
}
value += continuation
}
}
/// Advance `probe` over a run of blank lines (lines that are empty or only spaces), returning
/// how many were consumed; the probe is left at the start of the next non-blank line.
func consumeBlankLines(_ probe: inout PureYAML.Parsing.Reader) -> Int {
var count = 0
while true {
var blankProbe = probe
// A line of only white space is empty for folding, including one that holds a tab: a tab
// is not indentation, but a whitespace-only line is still blank, so consume it here.
_ = blankProbe.consume { character in
character == " " || character == "\t"
}
guard isLineBreak(blankProbe.peek()) else {
break
}
blankProbe.advance()
probe = blankProbe
count += 1
}
return count
}
/// Whether the reader is positioned at a document marker (`---` or `...`) followed by an
/// indicator boundary, i.e. the marker that starts or ends a document.
func isAtDocumentMarker(_ reader: PureYAML.Parsing.Reader) -> Bool {
(startsWith("---", reader) || startsWith("...", reader)) && isIndicatorBoundary(reader.peek(offset: 3))
}
func plainScalarContinuationBaseIndentation(state: State, firstLineIndent: Int) -> Int {
// When the scalar begins exactly at the current indentation top, it is a block value on its
// own line, so its own indent was pushed onto the stack; its continuation lines sit at that
// same indent, and the level they must exceed is the enclosing block's: the indent below on
// the stack, or -1 at the document root, where nothing encloses the scalar and a
// continuation may sit at column 1. For an inline value the top of the stack already is the
// enclosing block's indent, so it is used directly.
if state.indentation.last == firstLineIndent {
return state.indentation.count >= 2 ? state.indentation[state.indentation.count - 2] : -1
}
return state.indentation.last ?? 0
}
func scanPlainScalarContinuationLine(_ state: inout State) -> String {
// Leading white space beyond the line's indentation folds away in a plain scalar. A tab is
// not indentation, so the indentation scan leaves it here; skip it (and any spaces after it)
// so a continuation line like `\t3rd` joins as `3rd`, not `\t3rd`.
while let character = state.reader.peek(), character == " " || character == "\t" {
state.reader.advance()
}
var value = ""
while let character = state.reader.peek(), !isLineBreak(character) {
if character == " ", state.reader.peek(offset: 1) == "#" {
break
}
if character == "#", value.isEmpty || value.last == " " {
break
}
if character == ":", shouldEndPlainScalarAtMappingValue(state: state) {
break
}
if state.flowDepth > 0, isFlowDelimiter(character) {
break
}
value.append(character)
state.reader.advance()
}
return trimTrailingSpaces(value)
}
func skipLine(_ state: inout State) {
_ = state.reader.consume { character in
!isLineBreak(character)
}
}
func shouldEndPlainScalar(
_ character: Character,
state: State,
accumulated: String,
) -> Bool {
if isLineBreak(character) {
return true
}
// A comment opens after white space, a tab as well as a space, so a `#` preceded by either
// ends the plain scalar: `d\t#X` is the value `d` with a trailing comment.
if character == " " || character == "\t", state.reader.peek(offset: 1) == "#" {
return true
}
if character == "#", accumulated.isEmpty || accumulated.last == " " || accumulated.last == "\t" {
return true
}
if character == ":", shouldEndPlainScalarAtMappingValue(state: state) {
return true
}
if state.flowDepth > 0, isFlowDelimiter(character) {
return true
}
return false
}
func shouldEndPlainScalarAtMappingValue(state: State) -> Bool {
if state.flowDepth > 0 {
return state.isExpectingFlowMappingKey
|| isMappingValueBoundary(state.reader.peek(offset: 1))
}
return isBlockMappingValueBoundary(state.reader.peek(offset: 1))
}
func isAtMappingValueIndicator(state: State) -> Bool {
state.reader.peek() == ":" && shouldEndPlainScalarAtMappingValue(state: state)
}
/// Whether a mapping-value indicator (`: ` or `:` at line end) follows, allowing spaces before
/// the colon. Used after a quoted scalar, where the reader stops at the closing quote and any
/// separating spaces remain, unlike a plain scalar that ends at the colon itself.
func isAtMappingValueIndicatorAfterSpaces(state: State) -> Bool {
var offset = 0
while state.reader.peek(offset: offset) == " " {
offset += 1
}
return state.reader.peek(offset: offset) == ":"
&& isBlockMappingValueBoundary(state.reader.peek(offset: offset + 1))
}
@discardableResult
func appendMappingKeyIfNeeded(
start: PureYAML.Parsing.Mark,
state: inout State,
) -> Bool {
guard isAtMappingValueIndicator(state: state), canSynthesizeImplicitBlockKey(state) else {
return false
}
state.append(.mappingKey, mark: start, endMark: start)
return true
}
/// Whether a scalar followed by `:` may start an implicit mapping key here. In block style a
/// line holds at most one mapping pair, so once a block-level `:` has appeared on the line a
/// further key would form a forbidden inline nested mapping; in flow style several pairs per
/// line are allowed, so that restriction does not apply. In flow a scalar that is the content of
/// an open explicit key (`?`) is the key itself, not a new implicit key.
/// Records the implicit indentation levels a block entry with an inline node introduces: the
/// node's own indent (one past the dash), and the dash's indent as a sibling level so an inline
/// item's plain value stops folding at a later entry that returns to it (`: -␣b` then ` - - c`).
/// The sibling level is skipped under a pending explicit key (`? - x`), whose sequence key drives
/// implicit indentation itself.
func recordBlockEntryImplicitIndentation(start: PureYAML.Parsing.Mark, state: inout State) {
guard hasNodeOnSameLine(afterCurrentPositionIn: state.reader) else {
return
}
state.validImplicitIndentation.insert(start.column + 1)
if !state.pendingExplicitKey {
state.validImplicitIndentation.insert(start.column - 1)
}
}
func canSynthesizeImplicitBlockKey(_ state: State) -> Bool {
if state.flowDepth > 0 {
return state.flowContexts.last?.hasExplicitKey != true
}
return !state.sawMappingValueOnLine
}
/// A `:` with no node before it on the line is an implicit empty mapping key, unless it is the
/// value indicator of a pending explicit `?` key. This covers both a bare `: value` and a line
/// whose only prior content is a node property (`&a : value`, `!!str : value`): the property
/// decorates the empty key, which the preceding property tokens bind to. Synthesizes the empty
/// key and its node so the block mapping machinery treats the pair uniformly.
func synthesizeNullMappingKeyIfNeeded(
_ character: Character,
hadNonPropertyContentOnLine: Bool,
state: inout State,
) {
guard character == ":", !hadNonPropertyContentOnLine, state.flowDepth == 0,
!state.pendingExplicitKey, isAtMappingValueIndicator(state: state)
else {
return
}
let mark = state.reader.mark
state.append(.mappingKey, mark: mark, endMark: mark)
state.append(.scalar(value: "", style: .plain), mark: mark, endMark: mark)
}
}

View File

@ -1,312 +0,0 @@
extension PureYAML.Parsing.Scanner {
func scanIndentation(_ state: inout State) throws {
let start = state.reader.mark
let width = try scanIndentationWidth(&state)
guard shouldProcessIndentation(state) else {
preserveBlockScalarSurplusOnBlankLine(width: width, state: &state)
recordPendingBlockScalarBlankIndent(width: width, state: &state)
return
}
let end = state.reader.mark
let current = state.indentation.last ?? 0
let wasInBlockScalar = state.blockScalarIndentation != nil
// A document marker (`---` / `...`) at column 0 ends an active block scalar rather than
// being its content. Close the scalar's indentation here, dedenting back to the document
// level, so the marker is then scanned balanced at the document level. Without this the
// scalar's indent lingers past the marker and surfaces later as a stray dedent plus a
// phantom empty node (the marker still ends the document, but the structure is unbalanced).
if state.blockScalarIndentation != nil, width == 0, isAtDocumentMarker(state.reader) {
state.blockScalarIndentation = nil
state.blockScalarContentPrefix = ""
while state.indentation.count > 1 {
state.indentation.removeLast()
state.append(.dedent(width: state.indentation.last ?? 0), mark: end, endMark: end)
}
return
}
if consumeBlockScalarContentIndentation(width: width, state: &state) {
return
}
// An active block scalar that just ended (its content was less indented this line) onto a
// line beginning with '#' is leaving an ordinary comment. A comment's own indentation does
// not participate in block structure, so leave it for the main loop to scan rather than
// forcing its column to match an indentation level, which would dedent onto a width no stack
// level holds and raise a spurious error. The guard on `wasInBlockScalar` keeps a block
// scalar's own first content line that begins with '#' (e.g. `|-` then `# text`) as content.
if wasInBlockScalar, state.reader.peek() == "#" {
return
}
if let indicator = state.pendingBlockScalarIndentationIndicator {
try applyExplicitBlockScalarIndentation(
indicator: indicator,
width: width,
current: current,
marks: (start: start, end: end),
state: &state,
)
} else if width > current {
try applyIncreasedIndentation(width: width, start: start, end: end, state: &state)
} else if state.pendingBlockScalarIndentation, state.pendingBlockScalarIsRootNode, width == current {
// A root block scalar (`>`/`|` as the whole document) has the document node as its
// parent, whose indentation is -1, so content at column 0 (width == current == 0) is
// still more indented than its parent and must establish the block scalar's content
// indentation here, where `width > current` (with the base level read as 0) cannot fire.
// A non-root block scalar (a sequence item or mapping value) keeps its parent's indent as
// the floor, so its under-indented content still falls through to an error, unchanged.
try applyIncreasedIndentation(width: width, start: start, end: end, state: &state)
} else if width < current {
try applyDedentation(width: width, start: start, end: end, state: &state)
}
pruneImplicitIndentation(&state)
clearPendingBlockScalarIndentation(&state)
}
/// Records a leading blank line's indentation while a block scalar's content indentation is
/// still pending (before its first content line), so an over-indented leading line can be
/// caught once the content indentation is established.
func recordPendingBlockScalarBlankIndent(width _: Int, state: inout State) {
guard state.pendingBlockScalarIndentation, state.blockScalarIndentation == nil else {
return
}
// Compare leading spaces, not the tab-expanded width: a tab on a blank line is not
// indentation, so `␣\t` counts as one space, not a jump to the next tab stop.
state.pendingBlockScalarMaxBlankIndent = max(state.pendingBlockScalarMaxBlankIndent, state.lastLeadingSpaceCount)
}
/// A block scalar's leading empty lines must not be more indented than its content, so reject
/// the scalar if the widest leading blank exceeds the content indentation.
func enforceLeadingBlankIndentation(contentIndent: Int, line: Int, state: State) throws {
guard state.pendingBlockScalarMaxBlankIndent > contentIndent else {
return
}
throw PureYAML.Parsing.ParseError.unexpectedIndentation(line: line)
}
/// A whitespace-only line inside an active block scalar that is indented past the block's
/// content indentation carries the surplus spaces as content (e.g. a 3-space line in a 2-space
/// block contributes `" "`). The width is consumed before the line is recognized as blank, so
/// stash the surplus in the content prefix for the blank-line emission to pick up; otherwise the
/// line collapses to empty and a space is dropped from the scalar's value.
func preserveBlockScalarSurplusOnBlankLine(width: Int, state: inout State) {
guard let blockScalarIndentation = state.blockScalarIndentation, width > blockScalarIndentation else {
return
}
state.blockScalarContentPrefix = blockScalarSurplusPrefix(beyond: blockScalarIndentation, whitespace: state.lastLeadingWhitespace)
}
func scanIndentationWidth(_ state: inout State) throws -> Int {
var width = 0
var leadingSpaces = 0
var sawTab = false
var whitespace = ""
while let character = state.reader.peek() {
if character == " " {
width += 1
if !sawTab {
leadingSpaces += 1
}
whitespace.append(character)
state.reader.advance()
} else if character == "\t" {
sawTab = true
width = nextTabStop(after: width)
whitespace.append(character)
state.reader.advance()
} else {
break
}
}
state.lastLeadingSpaceCount = leadingSpaces
state.lastLeadingWhitespace = whitespace
return width
}
func nextTabStop(after width: Int) -> Int {
((width / 4) + 1) * 4
}
/// The leading-whitespace characters that lie beyond a block scalar's space-only `indent`, kept
/// verbatim so a tab in the content (` \ttext` under a 2-space indent) stays a tab rather than
/// being re-expanded to spaces. Walks the recorded whitespace, counting columns the way the width
/// scan does, and returns the remainder once the indentation columns are covered.
func blockScalarSurplusPrefix(beyond indent: Int, whitespace: String) -> String {
var column = 0
var index = whitespace.startIndex
while index < whitespace.endIndex, column < indent {
let character = whitespace[index]
column = character == "\t" ? nextTabStop(after: column) : column + 1
index = whitespace.index(after: index)
}
return String(whitespace[index...])
}
func shouldProcessIndentation(_ state: State) -> Bool {
guard !state.reader.isAtEnd, !isLineBreak(state.reader.peek()) else {
return false
}
// A leading '#' normally starts a comment line, which must not drive
// indent/dedent processing. Inside a block scalar it is different: when
// the header is still pending or the content is active, a leading '#' is
// literal scalar content, so indentation must be processed so the block
// scalar's indentation resolves and its content lines are consumed.
// Bailing here would leave `blockScalarIndentation` unset and the main
// loop would mis-scan the line as a comment, dropping the content.
if state.reader.peek() == "#", !state.pendingBlockScalarIndentation, state.blockScalarIndentation == nil {
return false
}
return state.flowDepth == 0
}
func consumeBlockScalarContentIndentation(
width: Int,
state: inout State,
) -> Bool {
guard let blockScalarIndentation = state.blockScalarIndentation else {
return false
}
guard width < blockScalarIndentation else {
state.blockScalarContentPrefix = blockScalarSurplusPrefix(beyond: blockScalarIndentation, whitespace: state.lastLeadingWhitespace)
return true
}
state.blockScalarIndentation = nil
state.blockScalarContentPrefix = ""
return false
}
func applyExplicitBlockScalarIndentation(
indicator: Int,
width: Int,
current: Int,
marks: (start: PureYAML.Parsing.Mark, end: PureYAML.Parsing.Mark),
state: inout State,
) throws {
// The indicator is the content indentation relative to the block scalar's parent node. For an
// inline mapping value (`- aaa: |2`), the parent mapping sits at an implicit indentation that
// never reached the main stack, so take the deepest applicable implicit level as the base:
// `|2` under a mapping at column 2 makes the content indentation 4. A direct sequence item
// (`- |2`) has no such parent and keeps the stack's current level.
let parentIndentation = state.pendingBlockScalarInlineMappingValue
? max(current, state.validImplicitIndentation.filter { $0 <= width }.max() ?? current)
: current
let blockScalarIndentation = parentIndentation + indicator
guard width >= blockScalarIndentation else {
throw PureYAML.Parsing.ParseError.unexpectedIndentation(line: marks.end.line)
}
try enforceLeadingBlankIndentation(contentIndent: blockScalarIndentation, line: marks.end.line, state: state)
state.indentation.append(blockScalarIndentation)
state.append(.indent(width: blockScalarIndentation), mark: marks.start, endMark: marks.end)
state.blockScalarIndentation = blockScalarIndentation
state.blockScalarContentPrefix = String(repeating: " ", count: width - blockScalarIndentation)
emitBufferedBlockScalarLeadingBlanks(&state)
clearPendingBlockScalarIndentation(&state)
}
/// Leading blank lines (between the header and the first content line) are
/// buffered until the indent token is emitted, then flushed as empty content
/// so they land after the indent and before the first content line.
func emitBufferedBlockScalarLeadingBlanks(_ state: inout State) {
// Each leading blank line carries any whitespace past the (now known) content indentation as
// content: an otherwise-blank `␣\t` line under a one-space indent yields the tab. With no
// content indentation established (an empty block scalar), the line is simply empty.
let indent = state.blockScalarIndentation
for whitespace in state.pendingBlockScalarLeadingBlankWhitespace {
let surplus = indent.map { blockScalarSurplusPrefix(beyond: $0, whitespace: whitespace) } ?? ""
state.append(.scalar(value: surplus, style: .plain))
}
state.pendingBlockScalarLeadingBlankWhitespace = []
}
func applyIncreasedIndentation(
width: Int,
start: PureYAML.Parsing.Mark,
end: PureYAML.Parsing.Mark,
state: inout State,
) throws {
guard state.pendingBlockScalarIndentation else {
state.indentation.append(width)
state.append(.indent(width: width), mark: start, endMark: end)
return
}
try enforceLeadingBlankIndentation(contentIndent: width, line: end.line, state: state)
state.indentation.append(width)
state.append(.indent(width: width), mark: start, endMark: end)
state.blockScalarIndentation = width
state.blockScalarContentPrefix = ""
emitBufferedBlockScalarLeadingBlanks(&state)
clearPendingBlockScalarIndentation(&state)
}
func applyDedentation(
width: Int,
start: PureYAML.Parsing.Mark,
end: PureYAML.Parsing.Mark,
state: inout State,
) throws {
while state.indentation.count > 1, width < (state.indentation.last ?? 0) {
state.indentation.removeLast()
state.append(.dedent(width: state.indentation.last ?? 0), mark: end, endMark: end)
}
if canUseImplicitIndentation(width: width, state: state) {
state.indentation.append(width)
state.append(.indent(width: width), mark: start, endMark: end)
}
guard width == state.indentation.last else {
throw PureYAML.Parsing.ParseError.unexpectedIndentation(line: end.line)
}
}
func canUseImplicitIndentation(
width: Int,
state: State,
) -> Bool {
width != state.indentation.last
&& width > (state.indentation.last ?? 0)
&& state.validImplicitIndentation.contains(width)
}
func pruneImplicitIndentation(_ state: inout State) {
let current = state.indentation.last ?? 0
state.validImplicitIndentation = state.validImplicitIndentation.filter { $0 <= current }
}
func clearPendingBlockScalarIndentation(_ state: inout State) {
state.pendingBlockScalarIndentation = false
state.pendingBlockScalarIndentationIndicator = nil
state.pendingBlockScalarLeadingBlankWhitespace = []
state.pendingBlockScalarMaxBlankIndent = 0
state.pendingBlockScalarIsRootNode = false
state.pendingBlockScalarInlineMappingValue = false
}
func closeIndentation(_ state: inout State) {
flushEmptyBlockScalarBlanks(&state)
let mark = state.reader.mark
while state.indentation.count > 1 {
state.indentation.removeLast()
state.append(.dedent(width: state.indentation.last ?? 0), mark: mark, endMark: mark)
}
}
/// A block scalar whose only lines are blank never establishes a content indentation, so its
/// blank lines stay buffered and are lost. At stream end, flush them as content wrapped in an
/// indent/dedent pair, so the parser builds a block scalar of those blank lines and chomping
/// applies: `keep: |+` over one blank line keeps a trailing line break, while clip and strip trim
/// it away. (Only the stream-end case is handled here, where no later line constrains ordering.)
func flushEmptyBlockScalarBlanks(_ state: inout State) {
guard state.pendingBlockScalarIndentation, state.blockScalarIndentation == nil,
!state.pendingBlockScalarLeadingBlankWhitespace.isEmpty
else {
return
}
state.append(.indent(width: (state.indentation.last ?? 0) + 1))
emitBufferedBlockScalarLeadingBlanks(&state)
state.append(.dedent(width: state.indentation.last ?? 0))
clearPendingBlockScalarIndentation(&state)
}
}

View File

@ -1,191 +0,0 @@
extension PureYAML.Parsing.Scanner {
func scanComment(_ state: inout State) {
let start = state.reader.mark
state.reader.advance()
if state.reader.peek() == " " {
state.reader.advance()
}
let value = state.reader.consume { character in
!isLineBreak(character)
}
state.append(.comment(value), mark: start, endMark: state.reader.mark)
}
func scanQuotedScalar(
_ style: PureYAML.Parsing.ScalarStyle,
state: inout State,
) throws {
let start = state.reader.mark
let quote: Character = style == .doubleQuoted ? "\"" : "'"
state.reader.advance()
var value = ""
var previousWasEscape = false
while let character = state.reader.peek() {
if style == .singleQuoted, character == quote, state.reader.peek(offset: 1) == quote {
value.append(character)
state.reader.advance()
value.append(character)
state.reader.advance()
continue
}
if character == quote, !previousWasEscape {
break
}
value.append(character)
previousWasEscape = character == "\\" && style == .doubleQuoted && !previousWasEscape
state.reader.advance()
if character != "\\" {
previousWasEscape = false
}
// A line that begins with a document marker (`---` / `...`) is forbidden inside a
// multi-line scalar: the marker would be ambiguous with a document boundary.
if isLineBreak(character), isAtDocumentMarker(state.reader) {
throw PureYAML.Parsing.ParseError.documentMarkerInScalar(line: state.reader.mark.line)
}
}
guard state.reader.peek() == quote else {
throw PureYAML.Parsing.ParseError.unterminatedQuotedString(line: start.line)
}
state.reader.advance()
// Unlike a plain scalar (which consumes the space before a `:` into itself), a quoted scalar
// stops at its closing quote, so the reader sits at any spaces before a `:`. A space between
// an implicit key and its mapping-value indicator is allowed, so look past them.
if isAtMappingValueIndicatorAfterSpaces(state: state), canSynthesizeImplicitBlockKey(state) {
// A block implicit key must lie on a single line, so a quoted scalar that spanned a line
// break (its open and close quotes sit on different lines) cannot be one. The rule is
// block-only: a flow mapping key may span lines, so the check is gated on block context.
// An explicit key (`? ...`) carries no `:` directly after the scalar, so never reaches here.
guard start.line == state.reader.mark.line || state.flowDepth > 0 else {
throw PureYAML.Parsing.ParseError.multilineImplicitKey(line: start.line)
}
state.append(.mappingKey, mark: start, endMark: start)
}
state.append(.scalar(value: value, style: style), mark: start, endMark: state.reader.mark)
}
func scanBlockScalarHeader(
_ style: PureYAML.Parsing.BlockScalarStyle,
state: inout State,
) {
let start = state.reader.mark
var chomping = PureYAML.Parsing.BlockScalarChomping.clip
var indentationIndicator: Int?
state.reader.advance()
while let character = state.reader.peek(), isBlockScalarHeaderCharacter(character) {
if character == "-" {
chomping = .strip
} else if character == "+" {
chomping = .keep
} else if character.isNumber {
indentationIndicator = Int(String(character))
}
state.reader.advance()
}
state.append(
.blockScalarHeader(style: style, chomping: chomping),
mark: start,
endMark: state.reader.mark,
)
state.pendingBlockScalarIndentation = true
state.pendingBlockScalarIndentationIndicator = indentationIndicator
// A root block scalar (the document's own node) has no enclosing collection: it sits at the
// base indentation with no mapping value, explicit key, block-entry implicit indent, or flow
// context introducing it on this line. Only then is its parent the document (indentation -1),
// letting its content begin at column 0. A sequence item or mapping value keeps its parent's
// indent as the floor instead.
state.pendingBlockScalarIsRootNode = state.indentation == [0]
&& !state.sawMappingValueOnLine
&& !state.pendingExplicitKey
&& state.validImplicitIndentation.isEmpty
&& state.flowDepth == 0
// Whether this block scalar is an inline mapping value (`- key: |2`). Such a value's parent is
// the mapping, which sits at an implicit indentation off the main stack, so an explicit
// indicator is measured relative to it. A direct sequence item (`- |2`) has no mapping value
// on its line and keeps the sequence's own indentation as the base.
state.pendingBlockScalarInlineMappingValue = state.sawMappingValueOnLine && state.flowDepth == 0
}
func scanBlockScalarContent(_ state: inout State) {
let start = state.reader.mark
let value = state.blockScalarContentPrefix + state.reader.consume { character in
!isLineBreak(character)
}
state.blockScalarContentPrefix = ""
state.append(.scalar(value: value, style: .plain), mark: start, endMark: state.reader.mark)
}
func scanNamedToken(
kind: (String) -> PureYAML.Parsing.TokenKind,
missingNameError: (Int) -> PureYAML.Parsing.ParseError,
canBeImplicitKey: Bool = false,
state: inout State,
) throws {
let start = state.reader.mark
state.reader.advance()
let name = state.reader.consume { character in
!isTokenTerminator(character)
}
guard !name.isEmpty else {
throw missingNameError(start.line)
}
// An alias node may serve as an implicit mapping key (`*a : value`). As with a scalar key, a
// `:` after optional spaces marks it as the key, so emit the key indicator before the alias.
// An anchor is a node property, never a key, so it never asks for this. An alias name cannot
// span lines, so the single-line implicit-key rule needs no separate check here.
let isAliasImplicitKey = canBeImplicitKey && state.flowDepth == 0
&& isAtMappingValueIndicatorAfterSpaces(state: state) && canSynthesizeImplicitBlockKey(state)
if isAliasImplicitKey {
state.append(.mappingKey, mark: start, endMark: start)
}
state.append(kind(name), mark: start, endMark: state.reader.mark)
}
func scanTag(_ state: inout State) throws {
let start = state.reader.mark
state.reader.advance()
var value = "!"
if state.reader.peek() == "<" {
value.append(state.reader.advance() ?? "<")
var didClose = false
while let character = state.reader.peek() {
value.append(character)
state.reader.advance()
if character == ">" {
didClose = true
break
}
}
guard didClose else {
throw PureYAML.Parsing.ParseError.unterminatedTag(line: start.line)
}
} else {
value += state.reader.consume { character in
!isTokenTerminator(character)
}
value = expandTag(value, at: start, state: &state)
}
state.append(.tag(value), mark: start, endMark: state.reader.mark)
}
func scanPlainScalar(_ state: inout State) {
let start = state.reader.mark
var value = ""
while let character = state.reader.peek() {
if shouldEndPlainScalar(character, state: state, accumulated: value) {
break
}
value.append(character)
state.reader.advance()
}
var trimmed = trimTrailingSpaces(value)
let isMappingKey = appendMappingKeyIfNeeded(start: start, state: &state)
if !isMappingKey {
trimmed = scanPlainScalarContinuations(
after: trimmed,
firstLineIndent: start.column - 1,
state: &state,
)
}
state.append(.scalar(value: trimmed, style: .plain), mark: start, endMark: state.reader.mark)
}
}

View File

@ -1,112 +0,0 @@
extension PureYAML.Parsing.Scanner {
struct State {
var reader: PureYAML.Parsing.Reader
var tokens: [PureYAML.Parsing.Token] = []
var indentation: [Int] = [0]
var isAtLineStart = true
var hasDocumentContent = false
var hasDocumentStartMarker = false
var isDocumentClosed = false
var hasYAMLDirective = false
// The line of a directive awaiting its document. A directive must be followed by an explicit
// `---`; this is set when one is scanned, cleared at a document start, and checked at a
// document end and at stream end so a directive with no document is rejected.
var pendingDirectiveLine: Int?
var flowDepth = 0
var flowContexts: [PureYAML.Parsing.ScannerFlowContext] = []
var validImplicitIndentation: Set<Int> = []
var pendingBlockScalarIndentation = false
var pendingBlockScalarIndentationIndicator: Int?
/// Whether the pending block scalar is the document's own root node (no enclosing collection),
/// so its content may begin at column 0 (the document parent's indentation being -1).
var pendingBlockScalarIsRootNode = false
/// Whether the pending block scalar is an inline mapping value (`- key: |2`), whose parent
/// mapping sits at an implicit indentation off the main stack, so an explicit indentation
/// indicator is measured relative to that level rather than the stack's.
var pendingBlockScalarInlineMappingValue = false
/// The leading-whitespace characters of each blank line between a block scalar header and its
/// first content line. Stored verbatim (not just counted) so once the content indentation is
/// known, any surplus past it, notably a tab on an otherwise-blank line, survives as content.
var pendingBlockScalarLeadingBlankWhitespace: [String] = []
/// The widest indentation seen on a block scalar's leading blank lines (before its first
/// content line). The spec forbids a leading empty line from being more indented than the
/// content, so this is compared to the content indentation once that is known.
var pendingBlockScalarMaxBlankIndent = 0
/// The count of leading space characters (before any tab) from the most recent indentation
/// scan. Indentation is measured in spaces, so this, not the tab-expanded width, is what the
/// leading-blank comparison uses: a tab is not indentation.
var lastLeadingSpaceCount = 0
// The literal leading-whitespace characters from the most recent indentation scan. Block
// scalar content keeps tabs beyond its (space-only) indentation verbatim, so the surplus is
// reconstructed from these characters, not as spaces from the tab-expanded width.
var lastLeadingWhitespace = ""
var blockScalarIndentation: Int?
var blockScalarContentPrefix = ""
/// Whether a real token has been scanned on the current physical line. Reset when the scan
/// loop consumes a line break and set as each token is scanned, so the scanner can tell a
/// comment that opens a line (valid) from one glued to earlier content on it (invalid). A
/// simple flag, not a token-line comparison, because a blank line inside a block scalar
/// emits an empty scalar attributed to the next line, which would read as phantom content.
var sawContentOnLine = false
/// Like ``sawContentOnLine`` but not set by a node property (an anchor or tag). A `:` whose
/// only preceding same-line content is a property (`&a :`, `!!str :`) is an empty implicit
/// key carrying that property, so a null mapping key is synthesized for it exactly as for a
/// bare `: value`. Reset at each line break.
var sawNonPropertyContentOnLine = false
// Whether a block-level mapping-value indicator (`:`) has been emitted on the current line.
// Once one has, a later scalar followed by `:` is not a new implicit key: block style allows
// only one mapping pair per line, so the document would be a forbidden inline nested mapping
// (`a: b: c`). Reset at each line break. Flow `:` do not set it, since flow allows several
// pairs on a line (`{a: b, c: d}`).
var sawMappingValueOnLine = false
/// Whether a flow collection has closed on the current line at the top level. A block
/// sequence entry cannot follow a flow node on the same line (`{a: b}- x`), unlike a nested
/// block entry (`- -`), which is not preceded by a flow close. Reset at each line break.
var sawFlowCloseOnLine = false
// Whether an explicit `?` mapping key is awaiting its `:` value indicator. A `:` that opens
// a line is that key's value when one is pending, otherwise it is an implicit null key
// (`: value`). Only the `?` indicator sets this; a synthesized implicit key does not.
var pendingExplicitKey = false
/// The tag handles in scope for the current document. A `%TAG` directive's handle applies
/// only to the document it precedes (YAML 1.2), so handles are scoped per document, not
/// accumulated across the stream: `pendingTagHandles` collects a document's directives and is
/// merged onto the defaults at the document start, then both reset at the document boundary.
static let defaultTagHandles = [
"!": "!",
"!!": "tag:yaml.org,2002:",
]
var tagHandles = Self.defaultTagHandles
var pendingTagHandles: [String: String] = [:]
/// Tag shorthands using a handle never declared in their document; collected, not rejected.
var tagHandleIssues: [PureYAML.Parsing.TagHandleIssue] = []
mutating func append(_ kind: PureYAML.Parsing.TokenKind) {
let mark = reader.mark
tokens.append(PureYAML.Parsing.Token(kind: kind, mark: mark, endMark: mark))
markDocumentContentIfNeeded(kind)
}
mutating func append(
_ kind: PureYAML.Parsing.TokenKind,
mark: PureYAML.Parsing.Mark,
endMark: PureYAML.Parsing.Mark,
) {
tokens.append(PureYAML.Parsing.Token(kind: kind, mark: mark, endMark: endMark))
markDocumentContentIfNeeded(kind)
}
mutating func markDocumentContentIfNeeded(_ kind: PureYAML.Parsing.TokenKind) {
switch kind {
case .comment, .dedent, .documentEnd, .documentStart, .indent, .streamEnd, .streamStart:
return
default:
hasDocumentContent = true
}
}
var isExpectingFlowMappingKey: Bool {
flowContexts.last?.kind == .mapping
&& flowContexts.last?.isExpectingKey == true
}
}
}

View File

@ -1,45 +0,0 @@
extension PureYAML.Parsing.Scanner {
func expandTag(
_ tag: String,
at start: PureYAML.Parsing.Mark,
state: inout State,
) -> String {
// A named handle has the form `!name!suffix` with a non-empty name between two `!` (distinct
// from the secondary handle `!!suffix`, whose name is empty, and a local tag `!suffix`, which
// has no second `!`). Such a handle must be declared by a `%TAG` directive in this document;
// an undeclared one is recorded as a diagnostic and the tag is left verbatim, so the parse
// stays permissive while the issue can still be surfaced.
if let named = namedTagHandle(in: tag) {
if let prefix = state.tagHandles[named.handle] {
return prefix + named.suffix
}
state.tagHandleIssues.append(.init(line: start.line, column: start.column, handle: named.handle))
return tag
}
let handles = state.tagHandles.keys.sorted { first, second in
first.count > second.count
}
for handle in handles where tag.hasPrefix(handle) {
let suffix = tag.dropFirst(handle.count)
return (state.tagHandles[handle] ?? handle) + suffix
}
return tag
}
/// Splits a tag shorthand into its named handle and suffix when it has the `!name!suffix` form
/// (a non-empty name delimited by two `!`). Returns nil for a local tag (`!suffix`), the
/// secondary handle (`!!suffix`), or a verbatim tag.
func namedTagHandle(in tag: String) -> (handle: String, suffix: String)? {
guard tag.hasPrefix("!") else {
return nil
}
let afterFirst = tag.index(after: tag.startIndex)
guard let secondBang = tag[afterFirst...].firstIndex(of: "!"), secondBang > afterFirst else {
return nil
}
return (
handle: String(tag[...secondBang]),
suffix: String(tag[tag.index(after: secondBang)...]),
)
}
}

View File

@ -1,316 +0,0 @@
extension PureYAML.Parsing {
struct Scanner {
init() {}
func scan(_ yaml: String) throws -> [Token] {
let outcome = collect(yaml)
if let error = outcome.error {
throw error
}
return outcome.state.tokens
}
/// The tag-handle issues found while scanning, for the diagnostic layer. A scan-aborting error
/// still leaves the issues collected before it on the returned state, so they are reported; a
/// clean scan returns the complete set.
func tagHandleIssues(_ yaml: String) -> [PureYAML.Parsing.TagHandleIssue] {
collect(yaml).state.tagHandleIssues
}
/// Runs the scan and returns the resulting state with the error that aborted it, if any.
/// Returning the partial state keeps the diagnostics gathered before an abort, which a `try?`
/// over a throwing run would discard.
private func collect(_ yaml: String) -> (state: State, error: (any Error)?) {
var state = State(reader: Reader(yaml))
do {
try scanAll(&state)
return (state, nil)
} catch {
return (state, error)
}
}
private func scanAll(_ state: inout State) throws {
state.append(.streamStart)
while !state.reader.isAtEnd {
if isLineBreak(state.reader.peek()) {
state.reader.advance()
captureBlockScalarBlankLine(&state)
state.isAtLineStart = true
state.sawContentOnLine = false
state.sawNonPropertyContentOnLine = false
state.sawMappingValueOnLine = false
state.sawFlowCloseOnLine = false
continue
}
if state.isAtLineStart {
try scanIndentation(&state)
if state.reader.isAtEnd {
break
}
if isLineBreak(state.reader.peek()) {
continue
}
if try scanDirectiveOrDocumentMarker(&state) {
continue
}
state.isAtLineStart = false
}
try scanToken(&state)
}
closeIndentation(&state)
try enforceDirectiveHasDocument(state)
state.append(.streamEnd)
}
}
}
extension PureYAML.Parsing.Scanner {
func scanToken(_ state: inout State) throws {
let columnBeforeSeparation = state.reader.mark.column
skipSeparation(&state)
let consumedSeparation = state.reader.mark.column > columnBeforeSeparation
guard let character = state.reader.peek(), !isLineBreak(character) else {
return
}
if state.blockScalarIndentation != nil {
scanBlockScalarContent(&state)
return
}
// A `#` begins a comment only at the start of a line's content or after whitespace. Glued to
// earlier same-line content (e.g. `"x"#c`, `]#c`, `>#c`) it is not a plain-scalar character
// either, so the document is malformed; a plain scalar's own `#` is consumed before reaching
// here, so this rejects only the genuinely invalid case.
if character == "#", !consumedSeparation, state.sawContentOnLine {
throw PureYAML.Parsing.ParseError.commentNotPrecededByWhitespace(
line: state.reader.mark.line,
column: state.reader.mark.column,
)
}
let hadContentOnLine = state.sawContentOnLine
let hadNonPropertyContentOnLine = state.sawNonPropertyContentOnLine
state.sawContentOnLine = true
// A leading `&` or `!` begins a node property (anchor or tag), which is not itself a key or
// value; every other token character is real content. A property-only line followed by `:`
// is thus an empty implicit key (`&a :`), synthesized below like a bare `: value`.
if character != "&", character != "!" {
state.sawNonPropertyContentOnLine = true
}
// An implicit-key mapping value (`:` preceded by its key on the same line) means the line
// now holds a pair, so a later scalar followed by `:` cannot start another: that would be a
// forbidden inline nested block mapping (`a: b: c`). A `:` that opens the line is an explicit-
// or null-key value indicator, whose value may legitimately be a compact mapping, so it does
// not set the flag.
if character == ":", hadContentOnLine, state.flowDepth == 0, isAtMappingValueIndicator(state: state) {
state.sawMappingValueOnLine = true
}
synthesizeNullMappingKeyIfNeeded(character, hadNonPropertyContentOnLine: hadNonPropertyContentOnLine, state: &state)
if try scanLineIndicatorToken(character, state: &state) {
return
}
if scanFlowToken(character, state: &state) {
return
}
if try scanQuotedOrBlockScalarToken(character, state: &state) {
return
}
if try scanNodePropertyToken(character, state: &state) {
return
}
// In a flow collection, a `-` that begins a plain scalar must be followed by a non-indicator
// character. Immediately before a flow indicator (`[-]`, `[-, -]`) it is not a valid scalar.
if state.flowDepth > 0, character == "-", let next = state.reader.peek(offset: 1), isFlowDelimiter(next) {
throw PureYAML.Parsing.ParseError.unexpectedToken(
expected: "a flow value",
actual: "-",
line: state.reader.mark.line,
column: state.reader.mark.column,
)
}
scanPlainScalar(&state)
}
func scanLineIndicatorToken(
_ character: Character,
state: inout State,
) throws -> Bool {
switch character {
case "#":
scanComment(&state)
return true
case "-" where isBlockEntry(state.reader):
// A block sequence cannot be an implicit key's value on the same line as its `:`
// (`key: - a`); the value must begin on a following line. As with an inline nested
// mapping, an explicit- or null-key `:` opens its line and does not set the flag, so a
// compact sequence value after it stays valid.
if state.sawMappingValueOnLine, state.flowDepth == 0 {
throw PureYAML.Parsing.ParseError.blockSequenceAsInlineMappingValue(line: state.reader.mark.line)
}
// A block sequence entry cannot follow a flow collection that closed earlier on the
// line (`{a: b}- x`); the entry would begin a sequence inline after a complete node.
if state.sawFlowCloseOnLine {
throw PureYAML.Parsing.ParseError.unexpectedToken(
expected: "a line break or flow indicator",
actual: "-",
line: state.reader.mark.line,
column: state.reader.mark.column,
)
}
scanFixedToken(.blockEntry, state: &state)
return true
case "?" where isIndicatorBoundary(state.reader.peek(offset: 1)):
scanFixedToken(.mappingKey, state: &state)
return true
case ":" where isAtMappingValueIndicator(state: state):
scanFixedToken(.mappingValue, state: &state)
return true
default:
return false
}
}
func scanFlowToken(
_ character: Character,
state: inout State,
) -> Bool {
switch character {
case "[":
appendFlowImplicitKeyMarkerIfNeeded(state: &state)
state.flowDepth += 1
state.flowContexts.append(.init(kind: .sequence, isExpectingKey: false))
scanFixedToken(.flowSequenceStart, state: &state)
return true
case "]":
state.flowDepth = max(0, state.flowDepth - 1)
scanFixedToken(.flowSequenceEnd, state: &state)
_ = state.flowContexts.popLast()
if state.flowDepth == 0 {
state.sawFlowCloseOnLine = true
}
return true
case "{":
appendFlowImplicitKeyMarkerIfNeeded(state: &state)
state.flowDepth += 1
state.flowContexts.append(.init(kind: .mapping, isExpectingKey: true))
scanFixedToken(.flowMappingStart, state: &state)
return true
case "}":
state.flowDepth = max(0, state.flowDepth - 1)
scanFixedToken(.flowMappingEnd, state: &state)
_ = state.flowContexts.popLast()
if state.flowDepth == 0 {
state.sawFlowCloseOnLine = true
}
return true
case ",":
scanFixedToken(.flowEntry, state: &state)
if state.flowContexts.last?.kind == .mapping {
state.flowContexts[state.flowContexts.count - 1].isExpectingKey = true
}
if !state.flowContexts.isEmpty {
state.flowContexts[state.flowContexts.count - 1].hasExplicitKey = false
}
return true
default:
return false
}
}
func scanNodePropertyToken(
_ character: Character,
state: inout State,
) throws -> Bool {
switch character {
case "&":
try scanNamedToken(
kind: PureYAML.Parsing.TokenKind.anchor,
missingNameError: PureYAML.Parsing.ParseError.expectedAnchorName,
state: &state,
)
return true
case "*":
try scanNamedToken(
kind: PureYAML.Parsing.TokenKind.alias,
missingNameError: PureYAML.Parsing.ParseError.expectedAliasName,
canBeImplicitKey: true,
state: &state,
)
return true
case "!":
try scanTag(&state)
return true
default:
return false
}
}
func scanQuotedOrBlockScalarToken(
_ character: Character,
state: inout State,
) throws -> Bool {
switch character {
case "\"":
try scanQuotedScalar(.doubleQuoted, state: &state)
return true
case "'":
try scanQuotedScalar(.singleQuoted, state: &state)
return true
case "|":
scanBlockScalarHeader(.literal, state: &state)
return true
case ">":
scanBlockScalarHeader(.folded, state: &state)
return true
default:
return false
}
}
func skipSeparation(_ state: inout State) {
// Mid-line separation may be spaces or tabs: a tab is separation white space in YAML and is
// forbidden only as indentation, which the line-start indentation scan handles before this.
while state.reader.peek() == " " || state.reader.peek() == "\t" {
state.reader.advance()
}
}
func scanFixedToken(
_ kind: PureYAML.Parsing.TokenKind,
state: inout State,
) {
let start = state.reader.mark
state.reader.advance()
state.append(kind, mark: start, endMark: state.reader.mark)
if case .blockEntry = kind {
recordBlockEntryImplicitIndentation(start: start, state: &state)
// A block entry opens a fresh node position, so a property-only key after it on the same
// line (`- !!str : v`) is still an empty implicit key: the `-` is not content of the key.
state.sawNonPropertyContentOnLine = false
}
if case .mappingValue = kind, state.flowContexts.last?.kind == .mapping {
state.flowContexts[state.flowContexts.count - 1].isExpectingKey = false
}
// Only the explicit `?` indicator emits a mapping key through this path (an implicit key is
// synthesized directly), so track a block-level one as pending until its `:` value indicator
// is scanned. Flow `:`/`?` (inside a complex key, say) do not affect a pending block key.
if state.flowDepth == 0, case .mappingKey = kind {
state.pendingExplicitKey = true
}
if state.flowDepth == 0, case .mappingValue = kind {
state.pendingExplicitKey = false
}
// In flow the same role is tracked on the current flow context, so it is per-level and a
// nested collection's `:` does not affect an outer explicit key (`{? {a: b} : c}`).
if state.flowDepth > 0, !state.flowContexts.isEmpty, case .mappingKey = kind {
state.flowContexts[state.flowContexts.count - 1].hasExplicitKey = true
}
if state.flowDepth > 0, !state.flowContexts.isEmpty, case .mappingValue = kind {
state.flowContexts[state.flowContexts.count - 1].hasExplicitKey = false
}
}
}

View File

@ -1,6 +0,0 @@
extension PureYAML.Parsing {
enum ScannerFlowContextKind {
case sequence
case mapping
}
}

View File

@ -1,11 +0,0 @@
extension PureYAML.Parsing {
/// A tag shorthand (`!handle!suffix`) whose named handle was never declared by a `%TAG`
/// directive in the document where it is used. The scanner records these rather than rejecting,
/// so a permissive parse still produces a value while the diagnostic layer can surface the issue.
struct TagHandleIssue: Equatable {
var line: Int
var column: Int
/// The undeclared handle, including its surrounding `!`, e.g. `!e!`.
var handle: String
}
}

View File

@ -1,11 +0,0 @@
extension PureYAML.Parsing {
struct Token: Equatable, CustomStringConvertible {
var kind: TokenKind
var mark: Mark
var endMark: Mark
var description: String {
"\(kind) @\(mark)..\(endMark)"
}
}
}

View File

@ -1,208 +0,0 @@
extension PureYAML.Parsing {
private final class TokenScannerStateBox {
var state: Scanner.State
init(state: Scanner.State) {
self.state = state
}
}
struct TokenCursor {
private var tokens: [Token]
private(set) var index: Int
private var previousToken: Token?
private var scanner: Scanner?
private var scannerBox: TokenScannerStateBox?
init(tokens: [Token]) {
self.tokens = tokens.filter { !$0.kind.isComment }
index = 0
previousToken = nil
scanner = nil
scannerBox = nil
}
init(yaml: String, scanner: Scanner = .init()) {
var state = Scanner.State(reader: Reader(yaml))
state.append(.streamStart)
tokens = state.tokens.filter { !$0.kind.isComment }
state.tokens.removeAll(keepingCapacity: true)
index = 0
previousToken = nil
self.scanner = scanner
scannerBox = TokenScannerStateBox(state: state)
}
var current: Token? {
guard tokens.indices.contains(index) else {
return nil
}
return tokens[index]
}
var previous: Token? {
previousToken
}
mutating func peek(offset: Int = 1) throws -> Token? {
try ensureAvailable(offset: offset)
let targetIndex = index + offset
guard tokens.indices.contains(targetIndex) else {
return nil
}
return tokens[targetIndex]
}
mutating func advance() throws -> Token? {
try ensureAvailable(offset: 0)
guard tokens.indices.contains(index) else {
return nil
}
let token = tokens[index]
previousToken = token
index += 1
compactConsumedTokens()
try ensureAvailable(offset: 0)
return token
}
private mutating func ensureAvailable(offset: Int) throws {
while tokens.count <= index + offset, scannerBox != nil {
try scanMoreTokens()
}
}
private mutating func scanMoreTokens() throws {
guard let scanner, let scannerBox else {
return
}
while scannerBox.state.tokens.isEmpty {
guard !scannerBox.state.reader.isAtEnd else {
scanner.closeIndentation(&scannerBox.state)
try scanner.enforceDirectiveHasDocument(scannerBox.state)
scannerBox.state.append(.streamEnd)
drainScannedTokens(from: &scannerBox.state)
self.scannerBox = nil
return
}
if scanner.isLineBreak(scannerBox.state.reader.peek()) {
scannerBox.state.reader.advance()
scanner.captureBlockScalarBlankLine(&scannerBox.state)
scannerBox.state.isAtLineStart = true
scannerBox.state.sawContentOnLine = false
scannerBox.state.sawNonPropertyContentOnLine = false
scannerBox.state.sawMappingValueOnLine = false
scannerBox.state.sawFlowCloseOnLine = false
continue
}
if scannerBox.state.isAtLineStart {
try scanner.scanIndentation(&scannerBox.state)
if scannerBox.state.reader.isAtEnd {
continue
}
if scanner.isLineBreak(scannerBox.state.reader.peek()) {
continue
}
if try scanner.scanDirectiveOrDocumentMarker(&scannerBox.state) {
if !scannerBox.state.tokens.isEmpty {
break
}
continue
}
scannerBox.state.isAtLineStart = false
}
try scanner.scanToken(&scannerBox.state)
}
drainScannedTokens(from: &scannerBox.state)
}
private mutating func drainScannedTokens(from state: inout Scanner.State) {
tokens.append(contentsOf: state.tokens.filter { !$0.kind.isComment })
state.tokens.removeAll(keepingCapacity: true)
}
private mutating func compactConsumedTokens() {
guard index > 4096 else {
return
}
tokens.removeFirst(index)
index = 0
}
}
}
extension PureYAML.Parsing.TokenKind {
var isBlockEntry: Bool {
if case .blockEntry = self {
return true
}
return false
}
var isComment: Bool {
if case .comment = self {
return true
}
return false
}
var isDedent: Bool {
if case .dedent = self {
return true
}
return false
}
var isDocumentEnd: Bool {
if case .documentEnd = self {
return true
}
return false
}
var isDocumentStart: Bool {
if case .documentStart = self {
return true
}
return false
}
var isFlowTerminator: Bool {
switch self {
case .flowEntry, .flowMappingEnd, .flowSequenceEnd:
true
default:
false
}
}
var isIndent: Bool {
if case .indent = self {
return true
}
return false
}
var isMappingKey: Bool {
if case .mappingKey = self {
return true
}
return false
}
var isStreamEnd: Bool {
if case .streamEnd = self {
return true
}
return false
}
var isTerminator: Bool {
isDedent || isDocumentEnd || isDocumentStart || isFlowTerminator || isStreamEnd
}
}

View File

@ -1,367 +0,0 @@
extension PureYAML.Parsing.TokenEventParser {
mutating func parseBlockSequence(
properties: PureYAML.Parsing.NodeProperties,
allowMappingKeyTerminator: Bool = false,
) throws -> PureYAML.Parsing.NodeResult {
let start = try expect("block sequence entry") { $0.isBlockEntry }
try enforceBlockSequenceNotInlineAfterProperty(properties, firstEntry: start)
// A compact nested sequence (`- - x`) carries no indent/dedent tokens, only block entries at
// their columns, so the sibling-entry loop must stay at this sequence's own column: a later
// entry at a smaller column belongs to an enclosing sequence and ends this one.
let sequenceColumn = start.mark.column
let startMark = properties.mark ?? start.mark
var endMark = start.endMark
try emit(.sequenceStart(
anchor: properties.anchor,
tag: properties.tag,
style: .block,
mark: startMark,
))
try parseBlockSequenceItem(
after: start,
endMark: &endMark,
allowMappingKeyTerminator: allowMappingKeyTerminator,
)
try consumeSequenceItemBoundaryDedent(sequenceIndentWidth: start.mark.column - 1)
try consumeSequenceItemBoundaryIndent(sequenceIndentWidth: start.mark.column - 1)
while cursor.current?.kind.isBlockEntry == true, cursor.current?.mark.column == sequenceColumn {
let entry = try expect("block sequence entry") { $0.isBlockEntry }
try parseBlockSequenceItem(
after: entry,
endMark: &endMark,
allowMappingKeyTerminator: allowMappingKeyTerminator,
)
try consumeSequenceItemBoundaryDedent(sequenceIndentWidth: entry.mark.column - 1)
try consumeSequenceItemBoundaryIndent(sequenceIndentWidth: entry.mark.column - 1)
}
if let current = cursor.current, current.kind.isMappingKey, !allowMappingKeyTerminator {
throw PureYAML.Parsing.ParseError.mixedCollectionStyles(line: current.mark.line)
}
try emit(.sequenceEnd(mark: endMark))
return PureYAML.Parsing.NodeResult(kind: .sequence, endMark: endMark)
}
/// A block collection must begin on the line after its node properties, so an anchor or tag on
/// the same line as the sequence's first entry (`&a - x`) is malformed.
func enforceBlockSequenceNotInlineAfterProperty(
_ properties: PureYAML.Parsing.NodeProperties,
firstEntry: PureYAML.Parsing.Token,
) throws {
guard properties.anchor != nil || properties.tag != nil,
let propertyMark = properties.mark, propertyMark.line == firstEntry.mark.line
else {
return
}
throw PureYAML.Parsing.ParseError.inlineBlockSequenceAfterProperty(
line: propertyMark.line, column: propertyMark.column,
)
}
mutating func consumeSequenceItemBoundaryDedent(sequenceIndentWidth: Int) throws {
guard dedentWidth(cursor.current) == sequenceIndentWidth,
try cursor.peek()?.kind.isBlockEntry == true
else {
return
}
_ = try expect("dedent") { $0.isDedent }
}
mutating func consumeSequenceItemBoundaryIndent(sequenceIndentWidth: Int) throws {
guard indentWidth(cursor.current) == sequenceIndentWidth,
try cursor.peek()?.kind.isBlockEntry == true
else {
return
}
_ = try expect("indent") { $0.isIndent }
}
mutating func parseBlockSequenceItem(
after entry: PureYAML.Parsing.Token,
endMark: inout PureYAML.Parsing.Mark,
allowMappingKeyTerminator: Bool,
) throws {
if hasNodeOnSameLine(after: entry) {
let properties = try consumeProperties()
if cursor.current?.kind.isIndent == true {
_ = try expect("indent") { $0.isIndent }
endMark = try parseNode(properties: properties).endMark
_ = try expect("dedent") { $0.isDedent }
return
}
if cursor.current?.kind.isMappingKey == true {
endMark = try parseBlockMapping(
properties: properties,
includeIndentedContinuation: true,
minimumColumn: entry.mark.column + 1,
).endMark
} else if cursor.current?.kind.isBlockEntry == true {
endMark = try parseBlockSequence(
properties: properties,
allowMappingKeyTerminator: allowMappingKeyTerminator,
).endMark
} else {
endMark = try parseNode(properties: properties).endMark
}
} else if cursor.current?.kind.isIndent == true {
_ = try expect("indent") { $0.isIndent }
endMark = try parseNode().endMark
_ = try expect("dedent") { $0.isDedent }
} else {
let mark = entry.endMark
try emit(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark, endMark: mark))
endMark = mark
}
}
mutating func parseBlockMapping(
properties: PureYAML.Parsing.NodeProperties,
includeIndentedContinuation: Bool,
minimumColumn: Int? = nil,
) throws -> PureYAML.Parsing.NodeResult {
let startMark = properties.mark ?? cursor.current?.mark ?? .start
let tokenColumn = cursor.current?.mark.column ?? startMark.column
let contentColumn = min(startMark.column, tokenColumn)
let boundaryColumn = minimumColumn ?? contentColumn
var endMark = startMark
// A property written inline before the first key (`&a key: v`) binds to that key, not to the
// mapping: the spec requires a block collection's own properties to be followed by a line
// break (`s-l+block-collection`), so a property on the key's line is the key's. This matters
// for anchors, which must register with the key (early enough for an alias inside a value to
// resolve) rather than with the mapping, which registers only at its end. A property on its
// own line above the first key still binds to the mapping.
let firstKeyProperties: PureYAML.Parsing.NodeProperties
let mappingProperties: PureYAML.Parsing.NodeProperties
if properties != .none, properties.mark?.line == cursor.current?.mark.line {
firstKeyProperties = properties
mappingProperties = .none
} else {
firstKeyProperties = .none
mappingProperties = properties
}
try emit(.mappingStart(
anchor: mappingProperties.anchor,
tag: mappingProperties.tag,
style: .block,
mark: startMark,
))
try parseBlockMappingPairs(
boundaryColumn: boundaryColumn,
includeIndentedContinuation: includeIndentedContinuation,
firstKeyProperties: firstKeyProperties,
endMark: &endMark,
)
if let current = cursor.current, !canEndBlockMapping(at: current, minimumColumn: boundaryColumn) {
throw PureYAML.Parsing.ParseError.unexpectedToken(
expected: "mapping key",
actual: current.kind.description,
line: current.mark.line,
column: current.mark.column,
)
}
try emit(.mappingEnd(mark: endMark))
return PureYAML.Parsing.NodeResult(kind: .mapping, endMark: endMark)
}
mutating func parseBlockMappingPairs(
boundaryColumn: Int,
includeIndentedContinuation: Bool,
firstKeyProperties: PureYAML.Parsing.NodeProperties = .none,
endMark: inout PureYAML.Parsing.Mark,
) throws {
var pendingKeyProperties = firstKeyProperties
while canContinueBlockMapping(minimumColumn: boundaryColumn) {
endMark = try parseMappingPairAndBoundaries(
boundaryColumn: boundaryColumn,
includeIndentedContinuation: includeIndentedContinuation,
firstKeyProperties: pendingKeyProperties,
).endMark
pendingKeyProperties = .none
}
while includeIndentedContinuation, cursor.current?.kind.isIndent == true {
endMark = try parseIndentedBlockMappingPairs(boundaryColumn: boundaryColumn).endMark
}
}
mutating func parseIndentedBlockMappingPairs(
boundaryColumn: Int,
) throws -> PureYAML.Parsing.NodeResult {
_ = try expect("indent") { $0.isIndent }
guard cursor.current?.kind.isMappingKey == true else {
throw unexpectedToken(expected: "mapping key")
}
var endMark = cursor.current?.mark ?? .start
while canContinueBlockMapping(minimumColumn: boundaryColumn) {
endMark = try parseMappingPairAndBoundaries(
boundaryColumn: boundaryColumn,
includeIndentedContinuation: true,
).endMark
}
if !canEndBlockMapping(at: cursor.current, minimumColumn: boundaryColumn) {
_ = try expect("dedent") { $0.isDedent }
}
return PureYAML.Parsing.NodeResult(kind: .mapping, endMark: endMark)
}
mutating func parseMappingPairAndBoundaries(
boundaryColumn: Int,
includeIndentedContinuation: Bool,
firstKeyProperties: PureYAML.Parsing.NodeProperties = .none,
) throws -> PureYAML.Parsing.NodeResult {
let result = try parseMappingPair(firstKeyProperties: firstKeyProperties)
try consumeMappingSiblingBoundaryDedents(minimumColumn: boundaryColumn)
try consumeMappingSiblingBoundaryReindent(minimumColumn: boundaryColumn)
if includeIndentedContinuation {
try consumeDedents(atOrAbove: boundaryColumn)
try consumeMappingSiblingBoundaryDedents(minimumColumn: boundaryColumn)
try consumeMappingSiblingBoundaryReindent(minimumColumn: boundaryColumn)
}
return result
}
mutating func parseBlockScalar(
properties: PureYAML.Parsing.NodeProperties,
) throws -> PureYAML.Parsing.NodeResult {
let header = try expect("block scalar header") { kind in
if case .blockScalarHeader = kind {
return true
}
return false
}
var lines: [String] = []
var endMark = header.endMark
if cursor.current?.kind.isIndent == true {
_ = try expect("indent") { $0.isIndent }
while cursor.current != nil, cursor.current?.kind.isDedent != true {
guard case let .scalar(value, style) = cursor.current?.kind else {
throw unexpectedToken(expected: "block scalar content")
}
let token = try cursor.advance() ?? header
try lines.append(decodeScalar(value, style: style, line: token.mark.line))
endMark = token.endMark
}
_ = try expect("dedent") { $0.isDedent }
}
let value = blockScalarValue(lines: lines, header: header.kind)
let mark = properties.mark ?? header.mark
try emit(.scalar(
value: value,
anchor: properties.anchor,
tag: properties.tag,
style: scalarStyle(for: header.kind),
mark: mark,
endMark: endMark,
))
return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: endMark)
}
func scalarStyle(for kind: PureYAML.Parsing.TokenKind) -> PureYAML.Parsing.ScalarStyle {
switch kind {
case .blockScalarHeader(.folded, _):
.folded
case .blockScalarHeader(.literal, _):
.literal
default:
.plain
}
}
func blockScalarValue(
lines: [String],
header: PureYAML.Parsing.TokenKind,
) -> String {
let (style, chomping): (PureYAML.Parsing.BlockScalarStyle, PureYAML.Parsing.BlockScalarChomping)
switch header {
case let .blockScalarHeader(headerStyle, headerChomping):
style = headerStyle
chomping = headerChomping
default:
style = .literal
chomping = .clip
}
// Each content line carries its own line break. The folded style folds a
// single break between two non-empty lines into a space, while blank
// lines stay as line breaks. Chomping then trims the trailing breaks.
// For content with no blank lines (the only shape the scanner captures
// today) this is identical to the previous join-and-append behavior.
let body: String = switch style {
case .literal:
lines.map { $0 + "\n" }.joined()
case .folded:
foldedBlockBody(lines)
}
return chompedBlockScalar(body, chomping: chomping)
}
func foldedBlockBody(_ lines: [String]) -> String {
guard let firstContent = lines.firstIndex(where: { !$0.isEmpty }) else {
// The block is entirely blank lines: one break each, no folding.
return String(repeating: "\n", count: lines.count)
}
// Leading blank lines each contribute a line break before the content.
var result = String(repeating: "\n", count: firstContent) + lines[firstContent]
var index = firstContent + 1
while index < lines.count {
if lines[index].isEmpty {
// Between two non-more-indented lines a blank run of N folds to N line breaks: the
// first break is the absorbed fold, the rest are literal. Next to a more-indented line
// nothing folds, so the absorbed break survives too and the run yields N+1 breaks.
let beforeMoreIndented = isMoreIndentedBlockLine(lines[index - 1])
var blankCount = 0
while index < lines.count, lines[index].isEmpty {
blankCount += 1
index += 1
}
let afterMoreIndented = index < lines.count && isMoreIndentedBlockLine(lines[index])
let breaks = beforeMoreIndented || afterMoreIndented ? blankCount + 1 : blankCount
result += String(repeating: "\n", count: breaks)
if index < lines.count {
result += lines[index]
index += 1
}
} else if isMoreIndentedBlockLine(lines[index]) || isMoreIndentedBlockLine(lines[index - 1]) {
// A line break adjacent to a more-indented line is kept literal,
// so the more-indented line and its breaks survive folding.
result += "\n" + lines[index]
index += 1
} else {
result += " " + lines[index]
index += 1
}
}
return result + "\n"
}
func isMoreIndentedBlockLine(_ line: String) -> Bool {
// In a block scalar the scanner strips content lines to the block's
// indentation, so only a more-indented line retains leading whitespace.
line.first == " " || line.first == "\t"
}
func chompedBlockScalar(
_ body: String,
chomping: PureYAML.Parsing.BlockScalarChomping,
) -> String {
switch chomping {
case .keep:
return body
case .strip:
return blockScalarTrimmingTrailingNewlines(body)
case .clip:
let trimmed = blockScalarTrimmingTrailingNewlines(body)
return trimmed.isEmpty ? "" : trimmed + "\n"
}
}
func blockScalarTrimmingTrailingNewlines(_ value: String) -> String {
var result = value
while result.hasSuffix("\n") {
result.removeLast()
}
return result
}
}

View File

@ -1,26 +0,0 @@
extension PureYAML.Parsing.TokenEventParser {
/// The key node of an explicit (`?`) mapping entry. The key may be empty: `? : v`, a trailing
/// `? }` / `? ]` / `? ,`, or `?` at stream end all denote a null key. With no node and no
/// properties to anchor, emit a null scalar rather than failing to find a node.
mutating func parseExplicitMappingKey(
properties: PureYAML.Parsing.NodeProperties,
) throws -> PureYAML.Parsing.NodeResult {
guard properties == .none, isAtEmptyExplicitKey else {
return try parseNode(properties: properties)
}
let mark = cursor.current?.mark ?? cursor.previous?.endMark ?? .start
try emit(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark, endMark: mark))
return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: mark)
}
/// Whether an explicit key has no key node: the next token is the value indicator, a flow
/// terminator (`}`, `]`, `,`), or the stream has ended.
var isAtEmptyExplicitKey: Bool {
switch cursor.current?.kind {
case nil, .mappingValue, .flowMappingEnd, .flowSequenceEnd, .flowEntry:
true
default:
false
}
}
}

View File

@ -1,170 +0,0 @@
extension PureYAML.Parsing.TokenEventParser {
mutating func parseFlowSequence(
properties: PureYAML.Parsing.NodeProperties,
) throws -> PureYAML.Parsing.NodeResult {
let start = try expect("flow sequence start") { kind in
if case .flowSequenceStart = kind {
return true
}
return false
}
let startMark = properties.mark ?? start.mark
var endMark = start.endMark
try emit(.sequenceStart(
anchor: properties.anchor,
tag: properties.tag,
style: .flow,
mark: startMark,
))
var hasEntry = false
while cursor.current != nil {
if case .flowSequenceEnd = cursor.current?.kind {
let end = try expect("flow sequence end") { kind in
if case .flowSequenceEnd = kind {
return true
}
return false
}
endMark = end.endMark
try emit(.sequenceEnd(mark: endMark))
return PureYAML.Parsing.NodeResult(kind: .sequence, endMark: endMark)
}
if case .flowEntry = cursor.current?.kind {
// A comma separates entries, so one must precede it: a leading or repeated comma
// is an empty entry, which is not allowed. A single trailing comma before the
// closing bracket is fine, since the close is handled above regardless of `hasEntry`.
guard hasEntry else {
throw unexpectedToken(expected: "flow sequence entry or flow sequence end")
}
hasEntry = false
_ = try cursor.advance()
continue
}
// A flow sequence element that begins with `:` is a single-pair mapping with an empty
// key, e.g. `[ : value ]`; an element that begins with a node and is then followed by `:`
// already becomes a mapping through the scalar key's implicit-key marker (`[ a: b ]`).
if case .mappingValue = cursor.current?.kind {
endMark = try parseEmptyKeyFlowMappingElement().endMark
} else {
endMark = try parseNode().endMark
}
hasEntry = true
if cursor.current?.kind.isFlowTerminator == false {
throw unexpectedToken(expected: "flow entry or flow sequence end")
}
}
throw unexpectedToken(expected: "flow sequence end")
}
/// Emits a single-pair flow mapping with an empty (null) key for a flow sequence element that
/// begins with `:`. The value follows the `:` and may itself be empty before the next separator.
mutating func parseEmptyKeyFlowMappingElement() throws -> PureYAML.Parsing.NodeResult {
let mark = cursor.current?.mark ?? .start
try emit(.mappingStart(anchor: nil, tag: nil, style: .flow, mark: mark))
try emit(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark, endMark: mark))
let valueIndicator = try expect("mapping value") { kind in
if case .mappingValue = kind {
return true
}
return false
}
let properties = try consumeProperties()
let endMark: PureYAML.Parsing.Mark
if properties == .none, cursor.current?.kind.isFlowTerminator != false {
endMark = valueIndicator.endMark
try emit(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: endMark, endMark: endMark))
} else {
endMark = try parseNode(properties: properties).endMark
}
try emit(.mappingEnd(mark: endMark))
return PureYAML.Parsing.NodeResult(kind: .mapping, endMark: endMark)
}
mutating func parseFlowMapping(
properties: PureYAML.Parsing.NodeProperties,
) throws -> PureYAML.Parsing.NodeResult {
let start = try expect("flow mapping start") { kind in
if case .flowMappingStart = kind {
return true
}
return false
}
let startMark = properties.mark ?? start.mark
var endMark = start.endMark
try emit(.mappingStart(
anchor: properties.anchor,
tag: properties.tag,
style: .flow,
mark: startMark,
))
var hasEntry = false
while cursor.current != nil {
if case .flowMappingEnd = cursor.current?.kind {
let end = try expect("flow mapping end") { kind in
if case .flowMappingEnd = kind {
return true
}
return false
}
endMark = end.endMark
try emit(.mappingEnd(mark: endMark))
return PureYAML.Parsing.NodeResult(kind: .mapping, endMark: endMark)
}
if case .flowEntry = cursor.current?.kind {
// As in a flow sequence, a comma separates entries and so requires one before it;
// a leading or repeated comma is an empty entry. A trailing comma is fine.
guard hasEntry else {
throw unexpectedToken(expected: "flow mapping entry or flow mapping end")
}
hasEntry = false
_ = try cursor.advance()
continue
}
endMark = try parseFlowMappingPair().endMark
hasEntry = true
if cursor.current?.kind.isFlowTerminator == false {
throw unexpectedToken(expected: "flow entry or flow mapping end")
}
}
throw unexpectedToken(expected: "flow mapping end")
}
mutating func parseFlowMappingPair() throws -> PureYAML.Parsing.NodeResult {
if cursor.current?.kind.isMappingKey == true {
return try parseMappingPair(inFlow: true)
}
// A flow mapping key may be empty (`{ : v }`) or carry only properties (`{ !!str : v }`):
// both resolve to a null key, so emit one rather than requiring a node before the `:`. This is
// valid only in flow mapping key position, so the empty-node allowance stays local here.
let keyProperties = try consumeProperties()
let keyEndMark: PureYAML.Parsing.Mark
if case .mappingValue = cursor.current?.kind {
let mark = keyProperties.mark ?? cursor.current?.mark ?? .start
try emit(.scalar(value: "", anchor: keyProperties.anchor, tag: keyProperties.tag, style: .plain, mark: mark, endMark: mark))
keyEndMark = mark
} else {
keyEndMark = try parseNode(properties: keyProperties).endMark
}
// A flow mapping entry may omit its `:` value indicator: `{ a }` and `{ a, b }` are sets whose
// keys map to null. A value parses only when a `:` follows the key.
guard case .mappingValue = cursor.current?.kind else {
try emit(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: keyEndMark, endMark: keyEndMark))
return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: keyEndMark)
}
let valueIndicator = try expect("mapping value") { kind in
if case .mappingValue = kind {
return true
}
return false
}
let properties = try consumeProperties()
guard properties != .none || cursor.current?.kind.isFlowTerminator == false else {
let mark = valueIndicator.endMark
try emit(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark, endMark: mark))
return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: mark)
}
return try parseNode(properties: properties)
}
}

View File

@ -1,390 +0,0 @@
extension PureYAML.Parsing.TokenEventParser {
mutating func parseMappingPair(
firstKeyProperties: PureYAML.Parsing.NodeProperties = .none,
inFlow: Bool = false,
) throws -> PureYAML.Parsing.NodeResult {
// A subsequent pair's key may carry an anchor or tag (`&a key: …`). The first pair's inline
// properties are consumed when the mapping is entered and passed back in here; consuming
// again covers any on a later key, and is a no-op (`.none`) when there are none.
var keyProperties = firstKeyProperties
try keyProperties.mergeUnset(from: consumeProperties())
_ = try expect("mapping key") { $0.isMappingKey }
let key = try parseExplicitMappingKey(properties: keyProperties)
// An explicit key (`? key`) with no following `:` has an implicit null value, so several
// `? key` entries in a row (a set, for instance) each map to null rather than being an error.
guard case .mappingValue = cursor.current?.kind else {
return try emitNullMappingValue(at: key.endMark)
}
let valueIndicator = try expect("mapping value") { kind in
if case .mappingValue = kind {
return true
}
return false
}
let properties = try consumeProperties()
if inFlow {
// In a flow mapping the value is a flow node and may sit on any line: flow is line
// insensitive, so the block-only same-line and indentation rules below do not apply.
// Only a flow terminator (`,`, `}`, `]`) or the stream end leaves the value empty (null),
// mirroring the empty-value rule in `parseFlowMappingPair` for the non-`?` key path.
guard properties != .none || cursor.current?.kind.isFlowTerminator == false else {
return try emitNullMappingValue(at: valueIndicator.endMark)
}
return try parseNode(properties: properties)
}
try enforceValuePropertyIndented(properties, valueIndicator: valueIndicator)
if cursor.current?.kind.isIndent == true {
return try parseIndentedMappingValue(properties: properties)
}
if properties == .none, try isCommentInterruptedScalarValue(after: valueIndicator) {
return try rejectCommentInterruptedScalar()
}
if cursor.current?.kind.isMappingKey == true, hasNodeOnSameLine(after: valueIndicator) {
return try parseBlockMapping(
properties: properties,
includeIndentedContinuation: true,
minimumColumn: cursor.current?.mark.column,
)
}
// A value position holding only properties (`a: &anchor`) followed by a sibling mapping key
// has an empty (null) node carrying those properties, not a nested mapping: the indented
// nested case is consumed above by the indent branch, and an inline nested mapping is caught
// by the mapping-key-with-same-line-node branch, so a bare mapping key here is a sibling.
if properties != .none, cursor.current?.kind.isMappingKey == true {
return try emitEmptyNodeOrThrow(properties: properties)
}
if cursor.current?.kind.isBlockEntry == true {
// A block sequence value whose entries return to the mapping key's own indentation is a
// zero-indent sequence, which a sibling mapping key terminates. This holds whether or not
// the value carries properties (`seq: !!seq` then `- a`), so it is handled before the
// generic property-bearing node below, which would not stop at the sibling key and would
// misread it as a mixed collection.
return try parseBlockSequence(
properties: properties,
allowMappingKeyTerminator: true,
)
}
if properties != .none || hasNodeOnSameLine(after: valueIndicator) {
return try parseNode(properties: properties)
}
return try emitNullMappingValue(at: valueIndicator.endMark)
}
/// Parses a mapping value that begins on an indented line below the `:`. A leading block entry
/// is a block sequence; anything else is a node. The closing dedent is consumed unless the value
/// ended on a sibling mapping key, which the surrounding mapping continues from.
mutating func parseIndentedMappingValue(
properties: PureYAML.Parsing.NodeProperties,
) throws -> PureYAML.Parsing.NodeResult {
_ = try expect("indent") { $0.isIndent }
// A node property may sit alone on its own indented line above a block sequence whose entries
// return to the mapping key's indentation (`seq:` then ` &anchor` then `- a` back at the key's
// column). The grammar allows a block-in-block sequence at the parent indent n while the
// property is at n+1, so the indent's matching dedent falls between the property and the
// entries. Consume that dedent and bind the property to the sequence rather than emitting an
// empty anchored value and orphaning the entries.
if let kind = cursor.current?.kind, startsNodeProperties(kind) {
var carried = properties
try carried.mergeUnset(from: consumeProperties())
if cursor.current?.kind.isDedent == true, try cursor.peek(offset: 1)?.kind.isBlockEntry == true {
_ = try expect("dedent") { $0.isDedent }
return try parseBlockSequence(properties: carried, allowMappingKeyTerminator: true)
}
// The node carrying the properties may itself be indented further, when the properties
// span their own lines above it (`key: &anchor` / ` !!map` / ` a: b`): descend into that
// deeper indented value rather than expecting the node directly after the properties.
let node: PureYAML.Parsing.NodeResult = if cursor.current?.kind.isIndent == true {
try parseIndentedMappingValue(properties: carried)
} else {
try parseNode(properties: carried)
}
try consumeIndentedMappingValueClose()
return node
}
let result: PureYAML.Parsing.NodeResult = if cursor.current?.kind.isBlockEntry == true {
try parseBlockSequence(properties: properties, allowMappingKeyTerminator: true)
} else {
try parseNode(properties: properties)
}
try consumeIndentedMappingValueClose()
return result
}
/// Consume the dedent that closes an indented mapping value, unless the value ended on a sibling
/// mapping key, which the surrounding mapping continues from at the same indentation.
mutating func consumeIndentedMappingValueClose() throws {
if cursor.current?.kind.isDedent == true {
_ = try expect("dedent") { $0.isDedent }
} else if cursor.current?.kind.isMappingKey != true {
_ = try expect("dedent") { $0.isDedent }
}
}
/// A value's anchor or tag on a line below the `:` must be indented past the key. A properly
/// indented property is preceded by an indent token and consumed inside the indented value, so a
/// property consumed here on a line below the `:` sits at the key's level and is malformed. The
/// last consumed property is `cursor.previous`; checking it catches both a single under-indented
/// property (`seq:` then `&anchor`) and a second one after an inline first (`key: &x` then a
/// `!!tag` in column 1).
func enforceValuePropertyIndented(
_ properties: PureYAML.Parsing.NodeProperties,
valueIndicator: PureYAML.Parsing.Token,
) throws {
guard properties != .none, let lastProperty = cursor.previous,
lastProperty.mark.line != valueIndicator.mark.line
else {
return
}
throw PureYAML.Parsing.ParseError.unexpectedIndentation(line: lastProperty.mark.line)
}
/// Emit an empty (null) scalar as a mapping value and return its node result, used where a
/// mapping entry has no value: an explicit key with no `:`, or a `key:` with nothing after it.
mutating func emitNullMappingValue(at mark: PureYAML.Parsing.Mark) throws -> PureYAML.Parsing.NodeResult {
try emit(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark, endMark: mark))
return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: mark)
}
/// Rejects a plain scalar that the scanner split into a value-line scalar, an indent, and a
/// further scalar. The scanner folds a genuine multi-line plain scalar into a single token, so
/// this token shape arises only when a comment broke its continuation: a comment cannot
/// interrupt a multi-line plain scalar, so the construct is malformed.
mutating func rejectCommentInterruptedScalar() throws -> PureYAML.Parsing.NodeResult {
let first = try expect("plain scalar") { kind in
if case .scalar(_, .plain) = kind {
return true
}
return false
}
throw PureYAML.Parsing.ParseError.commentInMultilineScalar(line: first.mark.line)
}
mutating func consumeProperties() throws -> PureYAML.Parsing.NodeProperties {
var properties = PureYAML.Parsing.NodeProperties.none
while let token = cursor.current {
switch token.kind {
case let .anchor(anchor):
properties.anchor = anchor
properties.mark = properties.mark ?? token.mark
_ = try cursor.advance()
case let .tag(tag):
properties.tag = tag
properties.mark = properties.mark ?? token.mark
_ = try cursor.advance()
default:
return properties
}
}
return properties
}
func decodeScalar(
_ value: String,
style: PureYAML.Parsing.ScalarStyle,
line: Int,
) throws -> String {
switch style {
case .doubleQuoted:
try scalarParser.parseDoubleQuoted("\"\(value)\"", line: line)
case .folded, .literal:
value
case .plain:
value
case .singleQuoted:
try scalarParser.parseSingleQuoted("'\(value)'", line: line)
}
}
func decodedScalar(from token: PureYAML.Parsing.Token) throws -> String {
guard case let .scalar(value, style) = token.kind else {
throw unexpectedToken(expected: "scalar")
}
return try decodeScalar(value, style: style, line: token.mark.line)
}
func hasNodeOnSameLine(after token: PureYAML.Parsing.Token) -> Bool {
guard let current = cursor.current, !current.kind.isTerminator else {
return false
}
return current.mark.line == token.mark.line
}
mutating func isCommentInterruptedScalarValue(after token: PureYAML.Parsing.Token) throws -> Bool {
guard case .scalar(_, .plain) = cursor.current?.kind else {
return false
}
guard cursor.current?.mark.line == token.mark.line else {
return false
}
guard try cursor.peek()?.kind.isIndent == true else {
return false
}
guard try isScalar(cursor.peek(offset: 2)) else {
return false
}
return true
}
mutating func isCommentInterruptedScalar() throws -> Bool {
guard case .scalar(_, .plain) = cursor.current?.kind else {
return false
}
guard try cursor.peek()?.kind.isIndent == true else {
return false
}
guard try isScalar(cursor.peek(offset: 2)) else {
return false
}
return true
}
func isScalar(_ token: PureYAML.Parsing.Token?) -> Bool {
guard case .scalar = token?.kind else {
return false
}
return true
}
mutating func consumeDedents(atOrAbove minimumColumn: Int) throws {
while let width = dedentWidth(cursor.current), width >= minimumColumn {
_ = try expect("dedent") { $0.isDedent }
}
}
mutating func consumeMappingSiblingBoundaryDedents(minimumColumn: Int) throws {
while try canConsumeMappingSiblingBoundaryDedent(minimumColumn: minimumColumn) {
_ = try expect("dedent") { $0.isDedent }
}
}
mutating func canConsumeMappingSiblingBoundaryDedent(minimumColumn: Int) throws -> Bool {
guard let width = dedentWidth(cursor.current),
let next = try cursor.peek(),
next.kind.isMappingKey,
next.mark.column >= minimumColumn
else {
return false
}
return width == next.mark.column - 1
}
mutating func consumeMappingSiblingBoundaryReindent(minimumColumn: Int) throws {
guard let dedent = dedentWidth(cursor.current),
dedent < minimumColumn,
try indentWidth(cursor.peek()) == minimumColumn,
let mappingKey = try cursor.peek(offset: 2),
mappingKey.kind.isMappingKey,
mappingKey.mark.column == minimumColumn + 1
else {
return
}
_ = try expect("dedent") { $0.isDedent }
_ = try expect("indent") { $0.isIndent }
}
func dedentWidth(_ token: PureYAML.Parsing.Token?) -> Int? {
guard case let .dedent(width) = token?.kind else {
return nil
}
return width
}
func indentWidth(_ token: PureYAML.Parsing.Token?) -> Int? {
guard case let .indent(width) = token?.kind else {
return nil
}
return width
}
func canContinueBlockMapping(minimumColumn: Int?) -> Bool {
guard let current = cursor.current, current.kind.isMappingKey || startsNodeProperties(current.kind) else {
return false
}
guard let minimumColumn else {
return true
}
return current.mark.column >= minimumColumn
}
/// Whether a token kind begins a node's properties (an anchor or a tag), which a mapping key may
/// carry, so the next pair's key is `&a key: …` rather than a bare key.
func startsNodeProperties(_ kind: PureYAML.Parsing.TokenKind) -> Bool {
switch kind {
case .anchor, .tag:
true
default:
false
}
}
func canEndBlockMapping(
at token: PureYAML.Parsing.Token?,
minimumColumn: Int? = nil,
) -> Bool {
guard let token else {
return true
}
return token.kind.isBlockEntry
|| token.kind.isDedent
|| token.kind.isDocumentEnd
|| token.kind.isDocumentStart
|| token.kind.isFlowTerminator
|| isLowerColumnMappingKey(token, minimumColumn: minimumColumn)
|| token.kind.isStreamEnd
}
func isLowerColumnMappingKey(
_ token: PureYAML.Parsing.Token?,
minimumColumn: Int?,
) -> Bool {
guard let token, token.kind.isMappingKey, let minimumColumn else {
return false
}
return token.mark.column < minimumColumn
}
mutating func expect(
_ expected: String,
matching: (PureYAML.Parsing.TokenKind) -> Bool,
) throws -> PureYAML.Parsing.Token {
guard let token = cursor.current else {
let mark = cursor.previous?.endMark ?? .start
throw PureYAML.Parsing.ParseError.unexpectedToken(
expected: expected,
actual: "end of tokens",
line: mark.line,
column: mark.column,
)
}
guard matching(token.kind) else {
throw PureYAML.Parsing.ParseError.unexpectedToken(
expected: expected,
actual: token.kind.description,
line: token.mark.line,
column: token.mark.column,
)
}
_ = try cursor.advance()
return token
}
func unexpectedToken(expected: String) -> PureYAML.Parsing.ParseError {
guard let token = cursor.current else {
let mark = cursor.previous?.endMark ?? .start
return PureYAML.Parsing.ParseError.unexpectedToken(
expected: expected,
actual: "end of tokens",
line: mark.line,
column: mark.column,
)
}
return PureYAML.Parsing.ParseError.unexpectedToken(
expected: expected,
actual: token.kind.description,
line: token.mark.line,
column: token.mark.column,
)
}
}

View File

@ -1,305 +0,0 @@
extension PureYAML.Parsing {
struct TokenEventParser {
var cursor: TokenCursor
let scalarParser: Parser
var events: [Event]?
var eventSink: ((Event) throws -> Void)?
/// Current node recursion depth, tracked so deeply nested input fails
/// with a diagnostic instead of overflowing the stack.
var depth = 0
/// Maximum node nesting the parser will follow before failing with
/// ``ParseError/nestingTooDeep(line:column:)``.
///
/// This bound is the single safety invariant for deeply nested input,
/// and it is *necessary*, not merely convenient. A ``Model/Value`` is a
/// recursive tree, so composing it, emitting it, comparing it, hashing
/// it, and -- crucially -- deallocating it all recurse with the nesting
/// depth. Swift tears down a recursive value type recursively, and that
/// teardown cannot be made iterative cheaply, so the only way to keep
/// every operation within the call stack is to bound the depth of the
/// tree where it is built: here, while parsing.
///
/// Because every parsing entry point routes through ``parseNode``, this
/// one bound caps the depth of any parsed tree, which transitively keeps
/// the parser, the event composers, the emitter, equality, hashing, and
/// deallocation all well inside their limits. Measured on the main
/// thread's 8 MB stack: the parser overflows near depth 225 and the
/// tree operations near depth 50,000, while real-world specs nest below
/// ~40. So 64 accepts every realistic document yet rejects pathological
/// input -- for example a 100 KB `[[[...]]]` -- cleanly and early.
///
/// The bound cannot simply be raised: above ~225 the recursive parser
/// itself overflows before the guard fires (lifting it would require an
/// iterative parser), and a tree past ~5,000 deep crashes on `deinit`
/// regardless, so a higher limit would trade a parse-time failure for an
/// un-fixable teardown crash. 64 is the largest value that is both safe
/// here and sufficient for real input.
///
/// Caveat: a tree built *without* the parser -- for example by encoding
/// a pathologically deep `Codable` value -- bypasses this bound, and
/// operations on such a tree are O(depth) in stack like any recursive
/// value type. Keeping those shallow is the caller's responsibility, the
/// same way the standard library cannot prevent unbounded recursion in
/// user code.
static let maxNestingDepth = 64
init(
tokens: [Token],
scalarParser: Parser,
) {
cursor = TokenCursor(tokens: tokens)
self.scalarParser = scalarParser
events = []
eventSink = nil
}
mutating func parse() throws -> [Event] {
try parseEvents()
return events ?? []
}
init(
tokens: [Token],
scalarParser: Parser,
eventSink: @escaping (Event) throws -> Void,
) {
cursor = TokenCursor(tokens: tokens)
self.scalarParser = scalarParser
events = nil
self.eventSink = eventSink
}
init(
yaml: String,
scalarParser: Parser,
eventSink: @escaping (Event) throws -> Void,
) {
cursor = TokenCursor(yaml: yaml)
self.scalarParser = scalarParser
events = nil
self.eventSink = eventSink
}
mutating func parseStreaming() throws {
try parseEvents()
}
mutating func parseEvents() throws {
let streamStart = try expect("stream start") { kind in
if case .streamStart = kind {
return true
}
return false
}
try emit(.streamStart(mark: streamStart.mark))
// A stream whose only content is comments or whitespace has zero documents
// (YAML 1.2, matching libyaml and the test suite), so no document events are
// emitted. The value-level API maps an empty document stream to a single null
// value separately.
while cursor.current?.kind.isStreamEnd != true {
try parseDocument()
}
let streamEnd = try expect("stream end") { kind in
if case .streamEnd = kind {
return true
}
return false
}
try emit(.streamEnd(mark: streamEnd.mark))
}
mutating func emit(_ event: Event) throws {
if let eventSink {
try eventSink(event)
} else {
events?.append(event)
}
}
}
}
extension PureYAML.Parsing.TokenEventParser {
mutating func parseDocument() throws {
let startMark = try parseDocumentStartMark()
try emit(.documentStart(mark: startMark))
if cursor.current?.kind.isDocumentEnd == true {
let end = try expect("document end") { $0.isDocumentEnd }
try appendEmptyDocument(at: end.mark)
try emit(.documentEnd(mark: end.endMark))
return
}
if cursor.current?.kind.isDocumentStart == true || cursor.current?.kind.isStreamEnd == true {
try appendEmptyDocument(at: cursor.current?.mark ?? startMark)
try emit(.documentEnd(mark: cursor.current?.mark ?? startMark))
return
}
var endMark = try parseDocumentRootNode().endMark
while cursor.current?.kind.isDedent == true {
let dedent = try expect("dedent") { $0.isDedent }
endMark = dedent.endMark
}
if cursor.current?.kind.isDocumentEnd == true {
// A `...` ends this document. Content after it begins a new bare document, parsed by the
// outer stream loop, so emit the end and return rather than expecting a boundary here.
let end = try expect("document end") { $0.isDocumentEnd }
try emit(.documentEnd(mark: end.endMark))
return
}
if let current = cursor.current, !canEndDocument(at: current) {
throw PureYAML.Parsing.ParseError.unexpectedToken(
expected: "document boundary or stream end",
actual: current.kind.description,
line: current.mark.line,
column: current.mark.column,
)
}
try emit(.documentEnd(mark: endMark))
}
mutating func parseDocumentRootNode() throws -> PureYAML.Parsing.NodeResult {
guard cursor.current?.kind.isIndent == true else {
return try parseNode()
}
_ = try expect("indent") { $0.isIndent }
let result = try parseNode()
if cursor.current?.kind.isDedent == true {
_ = try expect("dedent") { $0.isDedent }
}
return result
}
mutating func parseDocumentStartMark() throws -> PureYAML.Parsing.Mark {
guard cursor.current?.kind.isDocumentStart == true else {
return cursor.current?.mark ?? cursor.previous?.endMark ?? .start
}
let start = try cursor.advance()
return start?.mark ?? .start
}
mutating func appendEmptyDocument(at mark: PureYAML.Parsing.Mark) throws {
try emit(.scalar(value: "", anchor: nil, tag: nil, style: .plain, mark: mark, endMark: mark))
}
func canEndDocument(at token: PureYAML.Parsing.Token) -> Bool {
token.kind.isDocumentStart || token.kind.isStreamEnd
}
mutating func parseNode() throws -> PureYAML.Parsing.NodeResult {
let properties = try consumeProperties()
return try parseNode(properties: properties)
}
/// Reject documents nested past ``maxNestingDepth`` with a diagnostic so a
/// pathologically nested input fails cleanly instead of overflowing the
/// recursive-descent stack.
func enforceNestingLimit(properties: PureYAML.Parsing.NodeProperties) throws {
guard depth > Self.maxNestingDepth else {
return
}
let mark = properties.mark ?? cursor.current?.mark ?? cursor.previous?.endMark ?? .start
throw PureYAML.Parsing.ParseError.nestingTooDeep(line: mark.line, column: mark.column)
}
/// Handles a node position with no node token. An anchor or tag with nothing after it
/// (`- !!str`, `&a` alone) applies to an empty node, so emit a null scalar carrying the
/// properties; with no properties, a missing node is an error.
mutating func emitEmptyNodeOrThrow(
properties: PureYAML.Parsing.NodeProperties,
) throws -> PureYAML.Parsing.NodeResult {
let mark = properties.mark ?? cursor.current?.mark ?? cursor.previous?.endMark ?? .start
guard properties.anchor != nil || properties.tag != nil else {
throw PureYAML.Parsing.ParseError.expectedNode(line: mark.line, column: mark.column)
}
try emit(.scalar(value: "", anchor: properties.anchor, tag: properties.tag, style: .plain, mark: mark, endMark: mark))
return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: mark)
}
/// An alias is a bare reference, so it cannot carry an anchor or tag of its own (`&b *a`).
mutating func parseAlias(
anchor: String,
token: PureYAML.Parsing.Token,
properties: PureYAML.Parsing.NodeProperties,
) throws -> PureYAML.Parsing.NodeResult {
guard properties.anchor == nil, properties.tag == nil else {
let mark = properties.mark ?? token.mark
throw PureYAML.Parsing.ParseError.aliasWithProperties(line: mark.line, column: mark.column)
}
_ = try cursor.advance()
try emit(.alias(anchor: anchor, mark: token.mark))
return PureYAML.Parsing.NodeResult(kind: .alias, endMark: token.endMark)
}
mutating func parseNode(
properties initialProperties: PureYAML.Parsing.NodeProperties,
) throws -> PureYAML.Parsing.NodeResult {
depth += 1
defer { depth -= 1 }
try enforceNestingLimit(properties: initialProperties)
var properties = initialProperties
try properties.mergeUnset(from: consumeProperties())
guard let token = cursor.current, !token.kind.isTerminator else {
return try emitEmptyNodeOrThrow(properties: properties)
}
switch token.kind {
case let .alias(anchor):
return try parseAlias(anchor: anchor, token: token, properties: properties)
case .blockEntry:
return try parseBlockSequence(properties: properties)
case .blockScalarHeader:
return try parseBlockScalar(properties: properties)
case .flowMappingStart:
return try parseFlowMapping(properties: properties)
case .flowSequenceStart:
return try parseFlowSequence(properties: properties)
case .mappingKey:
return try parseBlockMapping(
properties: properties,
includeIndentedContinuation: false,
)
case let .scalar(value, style):
return try parseScalarNode(value: value, style: style, token: token, properties: properties)
case .mappingValue:
// An explicit key that is only properties before its `:` (`? &e` then `: v`) is an empty
// key carrying those properties. Emit the empty scalar without consuming the `:`, so the
// mapping machinery reads it as the value indicator. With no properties this is a misplaced
// `:`, which `emitEmptyNodeOrThrow` keeps as an error. (The implicit form `&e : v` never
// reaches here: the scanner synthesizes its empty key as an explicit scalar token.)
return try emitEmptyNodeOrThrow(properties: properties)
default:
throw PureYAML.Parsing.ParseError.expectedNode(line: token.mark.line, column: token.mark.column)
}
}
/// Emits a scalar node from a scalar token, applying any node properties. A plain scalar that the
/// scanner split around a comment (the comment-interrupted shape) is rejected rather than emitted.
mutating func parseScalarNode(
value: String,
style: PureYAML.Parsing.ScalarStyle,
token: PureYAML.Parsing.Token,
properties: PureYAML.Parsing.NodeProperties,
) throws -> PureYAML.Parsing.NodeResult {
if try isCommentInterruptedScalar() {
return try rejectCommentInterruptedScalar()
}
_ = try cursor.advance()
let decoded = try decodeScalar(value, style: style, line: token.mark.line)
let mark = properties.mark ?? token.mark
try emit(.scalar(
value: decoded,
anchor: properties.anchor,
tag: properties.tag,
style: style,
mark: mark,
endMark: token.endMark,
))
return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: token.endMark)
}
}

View File

@ -1,96 +0,0 @@
extension PureYAML.Parsing {
enum TokenKind: Equatable {
case streamStart
case streamEnd
case documentStart
case documentEnd
case indent(width: Int)
case dedent(width: Int)
case blockEntry
case mappingKey
case mappingValue
case flowSequenceStart
case flowSequenceEnd
case flowMappingStart
case flowMappingEnd
case flowEntry
case scalar(value: String, style: ScalarStyle)
case blockScalarHeader(style: BlockScalarStyle, chomping: BlockScalarChomping)
case anchor(String)
case alias(String)
case tag(String)
case comment(String)
}
}
extension PureYAML.Parsing.TokenKind: CustomStringConvertible {
var description: String {
switch self {
case .streamStart:
"streamStart"
case .streamEnd:
"streamEnd"
case .documentStart:
"documentStart"
case .documentEnd:
"documentEnd"
case let .indent(width):
"indent width=\(width)"
case let .dedent(width):
"dedent width=\(width)"
case .blockEntry:
"blockEntry"
case .mappingKey:
"mappingKey"
case .mappingValue:
"mappingValue"
case .flowSequenceStart:
"flowSequenceStart"
case .flowSequenceEnd:
"flowSequenceEnd"
case .flowMappingStart:
"flowMappingStart"
case .flowMappingEnd:
"flowMappingEnd"
case .flowEntry:
"flowEntry"
case let .scalar(value, style):
"scalar value=\"\(escape(value))\" style=\(style)"
case let .blockScalarHeader(style, chomping):
if chomping == .clip {
"blockScalarHeader style=\(style)"
} else {
"blockScalarHeader style=\(style) chomping=\(chomping)"
}
case let .anchor(value):
"anchor name=\(value)"
case let .alias(value):
"alias name=\(value)"
case let .tag(value):
"tag value=\(value)"
case let .comment(value):
"comment value=\"\(escape(value))\""
}
}
private func escape(_ value: String) -> String {
var output = ""
for character in value {
switch character {
case "\\":
output += "\\\\"
case "\"":
output += "\\\""
case "\n":
output += "\\n"
case "\r":
output += "\\r"
case "\t":
output += "\\t"
default:
output.append(character)
}
}
return output
}
}

View File

@ -0,0 +1,44 @@
import PureYAML
public extension PureYAML.Validation.BuiltinCheck {
/// Reports any mapping key that is not a string. JSON object names must be
/// strings, so a YAML complex key (a sequence or mapping used as a key) has no
/// JSON form. PureYAML core already ships ``mappingKeysAreUnique``; this check and
/// ``numbersAreFinite`` complete the JSON-representability set.
static var mappingKeysAreStrings: PureYAML.Validation.Check<PureYAML.Model.Mapping, PureYAML.Model.Value> {
PureYAML.Validation.Check(description: "Mapping keys are strings") { context in
context.subject.pairs.compactMap { pair in
guard pair.keyNode.stringValue == nil else {
return nil
}
return PureYAML.Validation.Issue(
severity: .error,
reason: "Mapping key '\(pair.keyNode.description)' is not a string",
path: context.path.appending(.complexKey(pair.keyNode)),
)
}
}
}
/// Reports any non-finite number (NaN or infinity). JSON has no literal for
/// these, so a value tree carrying one is not representable as JSON.
static var numbersAreFinite: PureYAML.Validation.Check<PureYAML.Model.Value, PureYAML.Model.Value> {
PureYAML.Validation.Check(description: "Numbers are finite") { context in
guard case let .double(number) = context.subject, !number.isFinite else {
return []
}
return [
PureYAML.Validation.Issue(
severity: .error,
reason: "Number '\(number)' is not finite",
path: context.path,
),
]
} when: { context in
if case .double = context.subject {
return true
}
return false
}
}
}

View File

@ -0,0 +1,13 @@
import PureYAML
public extension PureYAML.JSON.Convert {
/// A reason a YAML <-> JSON conversion did not complete.
enum Failure: Error, Equatable, Sendable {
/// The source text did not parse.
case sourceInvalid(String)
/// The value cannot be represented in the target format.
case notRepresentable([PureYAML.Validation.Issue])
/// The emitted output did not parse back to the source value.
case roundTripMismatch(PureYAML.JSON.Format)
}
}

View File

@ -0,0 +1,64 @@
import PureYAML
public extension PureYAML.JSON {
/// Standalone YAML <-> JSON conversion. It depends only on the parsing/emitting
/// engine and the JSON writer; it knows nothing about any editor or UI, so it is
/// usable on its own (a CLI, a server, a build step, a test).
///
/// A conversion succeeds only when, in order:
/// 1. the source parses,
/// 2. the value is representable in the target format, and
/// 3. the emitted output parses back to a value identical to the source
/// (a lossless round-trip check that also guards against emitter defects).
///
/// JSON is a subset of YAML 1.2, so the same engine parses both the source and
/// the re-parsed output; only "to JSON" can be rejected for representability (a
/// value may use a non-string key or a non-finite number). "To YAML" still
/// rejects duplicate keys, which are legal in JSON but not in YAML.
enum Convert {
/// Convert source text to the target format, validated end to end.
public static func convert(_ source: String, to target: Format, pretty: Bool = true) throws -> String {
let value = try parse(source)
let output = try emit(value, as: target, pretty: pretty)
guard try parse(output) == value else {
throw Failure.roundTripMismatch(target)
}
return output
}
/// Emit a value tree in the target format. The value is validated for
/// representability first; the round-trip check is not run here (use
/// ``convert(_:to:pretty:)`` for the fully validated path).
public static func emit(_ value: PureYAML.Model.Value, as target: Format, pretty: Bool = true) throws -> String {
let issues = validator(for: target).collect(value).issues
guard issues.isEmpty else {
throw Failure.notRepresentable(issues)
}
switch target {
case .json:
return try PureYAML.JSON.Writer.string(from: value, options: .init(pretty: pretty, indent: 2))
case .yaml:
return PureYAML.dump(value, options: .init(scalarStyle: .plainWhenSafe, collectionStyle: .block))
}
}
/// The validator for a target format. JSON additionally requires string keys
/// and finite numbers; YAML allows complex keys but still forbids duplicates.
public static func validator(for target: Format) -> PureYAML.Validation.Validator {
switch target {
case .json:
PureYAML.JSON.Validator.representability
case .yaml:
PureYAML.Validation.Validator()
}
}
private static func parse(_ text: String) throws -> PureYAML.Model.Value {
do {
return try PureYAML.parse(text)
} catch {
throw Failure.sourceInvalid(String(describing: error))
}
}
}
}

View File

@ -0,0 +1,11 @@
import PureYAML
public extension PureYAML.JSON {
/// A serialization format the converter can emit. The source is always parsed by
/// PureYAML (JSON is a subset of YAML, so one parser reads both); only the target
/// needs to be named.
enum Format: String, CaseIterable, Sendable, Equatable {
case yaml
case json
}
}

View File

@ -0,0 +1,35 @@
import PureYAML
public extension PureYAML.JSON {
/// JSON-representability checks built on PureYAML's own validation framework.
/// The parser already guarantees the input is well-formed YAML (and so covers
/// JSON syntax); this layer reports the value-level reasons a tree cannot be
/// emitted as JSON: a non-string mapping key, a non-finite number, or (for a
/// deterministic single-meaning document) a duplicate key.
///
/// JSON the format tolerates duplicate object names, but a tree that carries two
/// pairs with the same key emits an ambiguous document, so the representability
/// validator rejects duplicates for determinism. Drop ``unique`` from the rule
/// set if you want the permissive reading.
enum Validator {
/// The full JSON-representability rule set: unique keys, string keys, finite
/// numbers. Suitable to pass straight to `PureYAML.JSON.Convert`.
public static var representability: PureYAML.Validation.Validator {
PureYAML.Validation.Validator(checks: [
PureYAML.Validation.AnyCheck(PureYAML.Validation.BuiltinCheck.mappingKeysAreUnique),
PureYAML.Validation.AnyCheck(PureYAML.Validation.BuiltinCheck.mappingKeysAreStrings),
PureYAML.Validation.AnyCheck(PureYAML.Validation.BuiltinCheck.numbersAreFinite),
])
}
/// Every reason the value cannot be emitted as strict JSON, in document order.
public static func issues(for value: PureYAML.Model.Value) -> [PureYAML.Validation.Issue] {
representability.collect(value).issues
}
/// Whether the value can be emitted as strict JSON.
public static func isRepresentable(_ value: PureYAML.Model.Value) -> Bool {
issues(for: value).isEmpty
}
}
}

View File

@ -0,0 +1,11 @@
import PureYAML
public extension PureYAML.JSON.Writer {
/// A reason a value tree cannot be written as strict JSON.
enum Failure: Error, Sendable, Equatable {
/// A YAML complex (sequence/mapping) key has no JSON object-name form.
case nonStringKey(String)
/// JSON cannot represent NaN or infinity.
case nonFiniteNumber(Double)
}
}

View File

@ -0,0 +1,19 @@
import PureYAML
public extension PureYAML.JSON.Writer {
/// Formatting options for the JSON writer.
struct Options: Sendable, Equatable {
/// Indent with newlines (`true`) or emit a single compact line (`false`).
public var pretty: Bool
/// Spaces per indentation level when `pretty`.
public var indent: Int
public init(pretty: Bool = true, indent: Int = 2) {
self.pretty = pretty
self.indent = indent
}
public static let pretty = Options(pretty: true, indent: 2)
public static let compact = Options(pretty: false, indent: 0)
}
}

View File

@ -0,0 +1,129 @@
import PureYAML
public extension PureYAML.JSON {
/// A pure-Swift JSON serializer for the YAML/JSON value tree. Converting a
/// parsed document to strict JSON is the core "emit as JSON" operation; since
/// JSON is a subset of YAML 1.2, any value PureYAML parses can be emitted as
/// JSON unless it uses a YAML-only feature with no JSON representation (a
/// non-string mapping key, or a non-finite number).
enum Writer {
/// Serialize a value tree to a JSON string.
public static func string(from value: PureYAML.Model.Value, options: Options = .pretty) throws -> String {
var out = ""
try write(value, into: &out, depth: 0, options: options)
return out
}
private static func write(
_ value: PureYAML.Model.Value,
into out: inout String,
depth: Int,
options: Options,
) throws {
switch value {
case .null:
out += "null"
case let .bool(flag):
out += flag ? "true" : "false"
case let .int(number):
out += String(number)
case let .double(number):
guard number.isFinite else { throw Failure.nonFiniteNumber(number) }
out += String(number)
case let .string(text):
out += escaped(text)
case let .sequence(items):
try writeArray(items, into: &out, depth: depth, options: options)
case let .mapping(mapping):
try writeObject(mapping, into: &out, depth: depth, options: options)
}
}
private static func writeArray(
_ items: [PureYAML.Model.Value],
into out: inout String,
depth: Int,
options: Options,
) throws {
guard !items.isEmpty else {
out += "[]"
return
}
out += "["
for (index, item) in items.enumerated() {
out += index == 0 ? "" : ","
newline(&out, depth: depth + 1, options: options)
try write(item, into: &out, depth: depth + 1, options: options)
}
newline(&out, depth: depth, options: options)
out += "]"
}
private static func writeObject(
_ mapping: PureYAML.Model.Mapping,
into out: inout String,
depth: Int,
options: Options,
) throws {
guard !mapping.pairs.isEmpty else {
out += "{}"
return
}
out += "{"
for (index, pair) in mapping.pairs.enumerated() {
guard let key = pair.keyNode.stringValue else {
throw Failure.nonStringKey(pair.keyNode.description)
}
out += index == 0 ? "" : ","
newline(&out, depth: depth + 1, options: options)
out += escaped(key)
out += options.pretty ? ": " : ":"
try write(pair.value, into: &out, depth: depth + 1, options: options)
}
newline(&out, depth: depth, options: options)
out += "}"
}
private static func newline(_ out: inout String, depth: Int, options: Options) {
guard options.pretty else { return }
out += "\n"
out += String(repeating: " ", count: depth * options.indent)
}
/// Quote and escape a string per RFC 8259: `"` and `\` are escaped, the
/// named short escapes are used where they apply, and any remaining control
/// character below U+0020 becomes a `\u00XX` sequence.
private static func escaped(_ text: String) -> String {
var out = "\""
for scalar in text.unicodeScalars {
switch scalar {
case "\"": out += "\\\""
case "\\": out += "\\\\"
case "\u{08}": out += "\\b"
case "\u{0C}": out += "\\f"
case "\n": out += "\\n"
case "\r": out += "\\r"
case "\t": out += "\\t"
default:
if scalar.value < 0x20 {
out += "\\u" + hex4(scalar.value)
} else {
out.unicodeScalars.append(scalar)
}
}
}
out += "\""
return out
}
private static func hex4(_ value: UInt32) -> String {
let digits = "0123456789abcdef"
var result = ""
for shift in stride(from: 12, through: 0, by: -4) {
let index = Int((value >> UInt32(shift)) & 0xF)
result.append(digits[digits.index(digits.startIndex, offsetBy: index)])
}
return result
}
}
}

View File

@ -0,0 +1,14 @@
import PureYAML
public extension PureYAML {
/// Strict-JSON support layered on the shared YAML value tree: a pure-Swift
/// RFC-8259 emitter, JSON-representability rules expressed in PureYAML's own
/// validation framework, and a standalone YAML <-> JSON converter.
///
/// JSON is a subset of YAML 1.2, so the parser PureYAML already ships reads both
/// JSON and YAML sources into the same ``Model/Value`` tree. This module adds the
/// missing half (emitting that tree as JSON) plus the value-level checks that
/// decide whether a tree is representable as JSON at all. Anchor file; concrete
/// types live in the sibling `JSON.*` files.
enum JSON {}
}

View File

@ -35,7 +35,7 @@ extension PureYAML.Tagged {
}
return false
}
if case let .documentStart(mark)? = current {
if case let .documentStart(_, mark)? = current {
throw PureYAML.Parsing.ParseError.unsupportedMultiDocumentStream(line: mark.line)
}
_ = try expect("stream end") { event in

View File

@ -213,10 +213,7 @@ public extension PureYAML {
reason: streamIssue.issue.reason,
)
}
// The parse is permissive: a tag whose handle was never declared by a `%TAG` directive
// in its document still yields a value, but it is a genuine error, so surface it here.
let tagHandleDiagnostics = tagHandleDiagnostics(in: yaml, file: file)
return Validation.Report(tagHandleDiagnostics + diagnostics)
return Validation.Report(diagnostics)
} catch let error as PureYAML.Parsing.ParseError {
return Validation.Report([
Validation.Diagnostic(
@ -241,26 +238,6 @@ public extension PureYAML {
}
}
/// Diagnostics for tag shorthands whose named handle was never declared by a `%TAG` directive
/// in the document that uses it. The scanner finds these with full lexical context (it knows a
/// tag token from a `!` inside a scalar or comment), so the check is exact, not a heuristic.
static func tagHandleDiagnostics(
in yaml: String,
file: String?,
) -> [Validation.Diagnostic] {
Parsing.Scanner().tagHandleIssues(yaml).map { issue in
Validation.Diagnostic(
kind: .parse,
code: "undefinedTagHandle",
severity: .error,
file: file,
line: issue.line,
column: issue.column,
reason: "tag handle '\(issue.handle)' is not declared by a %TAG directive in this document",
)
}
}
/// Validates many YAML inputs without throwing or stopping at the first failure.
static func validationReports(
_ sources: [Validation.Source],

View File

@ -2,105 +2,120 @@ import Foundation
@testable import PureYAML
import Testing
/// Canonical event-stream conformance gate: the strongest oracle for the parser. It serializes
/// PureYAML's parse events into the official YAML test-suite "event" (yeast) format and compares them
/// exactly to each case's `test.event`. Unlike the JSON-projection oracle, this checks anchors, tags,
/// scalar styles, flow vs block, and aliases, for every valid case including those without an `in.json`.
///
/// Opt-in like the other suite gates, since the corpus is fetched, not vendored:
/// bash scripts/fetch-yaml-test-suite.sh
/// PUREYAML_RUN_YAML_TEST_SUITE=1 swift test --filter CanonicalEventConformanceTests
///
/// `knownEventGaps` is the authoritative gap set: every valid case whose event stream PureYAML does not
/// yet emit exactly. It is a gate, not a measurement: a fix must remove an id (the test fails until it
/// is removed), and a regression adds one (the test fails). No case is ever silently skipped.
///
/// One documented normalization. `Parsing.Event` does not carry the explicit document-marker flag, so
/// `+DOC`/`-DOC` lines are compared without their ` ---`/` ...` suffix. The marker distinction is the
/// single property this gate does not yet check; enriching the event API to carry the flag removes the
/// normalization and this caveat with it.
/// Authoritative conformance gate: serialize PureYAML's parse events into the
/// official YAML test suite "event" (yeast) format and compare EXACTLY to each
/// case's vendored `test.event`. Unlike the JSON-projection oracle, this checks
/// anchors, tags, scalar styles, flow vs block, and explicit document markers,
/// for every valid case (including those with no `in.json`). No normalization.
@Suite("Canonical Event Conformance")
struct CanonicalEventConformanceTests {
private struct Case {
let id: String
let input: String
let expectedEvents: String
}
/// The authoritative event-level frontier: valid cases that parse+value-match but whose canonical
/// event stream PureYAML does not yet emit exactly. Each is a precise, recorded gap (first-line
/// event diff available via the triage below), not a silent skip. Closing one removes its id; a
/// regression adds one. Audited 2026-06-29: every entry is a real parser deviation (e.g. 4ABK
/// truncates `http://foo.com` at the `:`; FH7J/PW8X bind a property to the collection instead of
/// the empty scalar), not a serializer artifact.
static let knownEventGaps: Set<String> = [
"4ABK", "4FJ6", "6BFJ", "6CK3", "7BMT", "7FWL", "87E4", "8UDB", "9KAX", "9MMW",
"CN3R", "CT4Q", "DC7X", "FH7J", "HWV9", "L9U5", "LP6E", "LQZ7", "M5C3", "M5DY",
"PW8X", "QF4Y", "QT73", "RZP5", "U3XV", "UGM3", "UT92", "V9D5", "XW4D", "ZF4X", "ZK9H",
]
/// Valid suite cases whose event stream PureYAML does NOT yet emit exactly.
/// This is the authoritative gap set: a fix removes an id (and the test fails
/// until it is removed), a regression adds one (and the test fails).
///
/// The ported libyaml engine now emits the exact canonical event stream for
/// every valid case in the vendored YAML test suite, so this set is empty.
/// Each entry that used to live here was a libyaml deviation from YAML 1.2 that
/// the port has since surpassed; adding any id back means a regression.
static let knownEventGaps: Set<String> = []
@Test("Every valid case emits canonical events, except the recorded gaps")
func test_canonicalEventConformance() throws {
guard ProcessInfo.processInfo.environment["PUREYAML_RUN_YAML_TEST_SUITE"] == "1" else {
return
}
let cases = try #require(loadValidCases(), "fetch the suite first: bash scripts/fetch-yaml-test-suite.sh")
var observedGaps: Set<String> = []
var compared = 0
for testCase in cases {
for testCase in CanonicalEventCorpus.validCases() {
guard let yaml = testCase.input, let expected = testCase.expectedEvents else { continue }
let events: [PureYAML.Parsing.Event]
do {
events = try PureYAML.Parsing.Parser().parseEvents(testCase.input)
events = try PureYAML.Parsing.Parser().parseEvents(yaml)
} catch {
observedGaps.insert(testCase.id)
continue
}
compared += 1
let actual = Self.normalizeMarkers(events.map(Self.serialize).joined(separator: "\n"))
if actual != Self.normalizeMarkers(Self.trimmingTrailingNewlines(testCase.expectedEvents)) {
let actual = events.map(CanonicalEventCorpus.serialize).joined(separator: "\n")
if actual != expected.trimmingTrailingNewline {
observedGaps.insert(testCase.id)
}
}
let regressions = observedGaps.subtracting(Self.knownEventGaps).sorted()
let fixed = Self.knownEventGaps.subtracting(observedGaps).sorted()
report("""
Canonical event conformance: \(cases.count - observedGaps.count)/\(cases.count) valid cases emit exact canonical events
recorded gaps: \(observedGaps.count) (\(compared) parsed, \(cases.count - compared) threw)
""")
#expect(regressions.isEmpty, "Canonical-event REGRESSIONS (now wrong, were correct): \(regressions.joined(separator: " "))")
#expect(fixed.isEmpty, "Canonical-event cases now FIXED; remove from knownEventGaps: \(fixed.joined(separator: " "))")
#expect(compared > 100, "non-vacuity: only \(compared) cases compared; the corpus did not load")
}
private func loadValidCases() -> [Case]? {
let manager = FileManager.default
let root = ProcessInfo.processInfo.environment["PUREYAML_YAML_TEST_SUITE_DIR"]
?? manager.currentDirectoryPath + "/.build/yaml-test-suite"
guard let names = try? manager.contentsOfDirectory(atPath: root) else {
return nil
@Test("differential: dump PureYAML yeast events for files under a directory")
func test_dumpYeast() throws {
guard let dir = ProcessInfo.processInfo.environment["PUREYAML_YEAST_DIR"] else { return }
let fileManager = FileManager.default
guard let walker = fileManager.enumerator(atPath: dir) else { return }
for case let path as String in walker where path.hasSuffix(".yaml") || path.hasSuffix(".yml") {
let full = dir + "/" + path
guard let yaml = try? String(contentsOfFile: full, encoding: .utf8) else { continue }
print("@@@FILE@@@ \(path)")
do {
let events = try PureYAML.Parsing.Parser().parseEvents(yaml)
print(events.map(CanonicalEventCorpus.serialize).joined(separator: "\n"))
} catch {
print("@@@ERROR@@@ \(error)")
}
print("@@@END@@@")
}
var cases: [Case] = []
for name in names.sorted() {
let dir = root + "/" + name
guard !manager.fileExists(atPath: dir + "/error"),
let input = try? String(contentsOfFile: dir + "/in.yaml", encoding: .utf8),
let events = try? String(contentsOfFile: dir + "/test.event", encoding: .utf8)
else {
}
@Test("triage: dump first-line diff for each recorded gap")
func test_triage() throws {
guard ProcessInfo.processInfo.environment["PUREYAML_TRIAGE_EVENTS"] == "1" else { return }
for id in Self.knownEventGaps.sorted() {
let testCase = CanonicalEventCorpus.Case(id: id)
guard let yaml = testCase.input, let expectedRaw = testCase.expectedEvents else { continue }
let expected = expectedRaw.trimmingTrailingNewline.split(separator: "\n").map(String.init)
let events: [PureYAML.Parsing.Event]
do {
events = try PureYAML.Parsing.Parser().parseEvents(yaml)
} catch {
print("\(id) THROWS: \(error)")
continue
}
cases.append(Case(id: name, input: input, expectedEvents: events))
let actual = events.map(CanonicalEventCorpus.serialize)
var line = 0
while line < min(expected.count, actual.count), expected[line] == actual[line] {
line += 1
}
let exp = line < expected.count ? expected[line] : "(end)"
let act = line < actual.count ? actual[line] : "(end)"
print("\(id) @\(line): EXP[\(exp)] ACT[\(act)]")
}
return cases
}
}
enum CanonicalEventCorpus {
struct Case { let id: String }
static func validCases() -> [Case] {
// Every vendored case that is NOT an error case (no `error` marker).
guard let base = Bundle.module.url(forResource: "yaml-test-suite", withExtension: nil, subdirectory: "Fixtures") else { return [] }
let fileManager = FileManager.default
guard let walker = fileManager.enumerator(at: base, includingPropertiesForKeys: nil) else { return [] }
var out: [Case] = []
for case let url as URL in walker where url.lastPathComponent == "in.yaml" {
let dir = url.deletingLastPathComponent()
let id = dir.path.replacingOccurrences(of: base.path + "/", with: "")
if !fileManager.fileExists(atPath: dir.appendingPathComponent("error").path) {
out.append(Case(id: id))
}
}
return out.sorted { $0.id < $1.id }
}
// MARK: - Yeast serialization
static func url(_ id: String, _ name: String, _ ext: String?) -> URL? {
Bundle.module.url(forResource: name, withExtension: ext, subdirectory: "Fixtures/yaml-test-suite/" + id)
}
static func serialize(_ event: PureYAML.Parsing.Event) -> String {
switch event {
case .streamStart: "+STR"
case .streamEnd: "-STR"
case .documentStart: "+DOC"
case .documentEnd: "-DOC"
case let .documentStart(explicit, _): explicit ? "+DOC ---" : "+DOC"
case let .documentEnd(explicit, _): explicit ? "-DOC ..." : "-DOC"
case let .sequenceStart(anchor, tag, style, _): "+SEQ" + (style == .flow ? " []" : "") + props(anchor, tag)
case .sequenceEnd: "-SEQ"
case let .mappingStart(anchor, tag, style, _): "+MAP" + (style == .flow ? " {}" : "") + props(anchor, tag)
@ -142,29 +157,26 @@ struct CanonicalEventConformanceTests {
}
return out
}
}
/// Drops the explicit `---`/`...` suffix from document-boundary lines, since the event API does not
/// yet record whether a document marker was explicit. Applied to both sides for symmetry.
static func normalizeMarkers(_ events: String) -> String {
events
.split(separator: "\n", omittingEmptySubsequences: false)
.map { line in
if line == "+DOC ---" { return "+DOC" }
if line == "-DOC ..." { return "-DOC" }
return String(line)
}
.joined(separator: "\n")
extension CanonicalEventCorpus.Case {
var input: String? {
guard let url = CanonicalEventCorpus.url(id, "in", "yaml") else { return nil }
return try? String(contentsOf: url, encoding: .utf8)
}
static func trimmingTrailingNewlines(_ value: String) -> String {
var copy = value
var expectedEvents: String? {
guard let url = CanonicalEventCorpus.url(id, "test", "event") else { return nil }
return try? String(contentsOf: url, encoding: .utf8)
}
}
private extension String {
var trimmingTrailingNewline: String {
var copy = self
while copy.hasSuffix("\n") {
copy.removeLast()
}
return copy
}
private func report(_ text: String) {
FileHandle.standardError.write(Data("\n\(text)\n".utf8))
}
}

View File

@ -214,8 +214,12 @@ enum CollectionCompatibilityFixtures {
- one
key: value
""",
expected: .mixedCollectionStyles(line: 2),
expectedDescription: "mapping and sequence entries are mixed at line 2",
expected: .malformed(
reason: "while parsing a block collection, did not find expected '-' indicator",
line: 1,
column: 1,
),
expectedDescription: "while parsing a block collection, did not find expected '-' indicator at line 1, column 1",
),
]

View File

@ -136,17 +136,17 @@ struct ComplexMappingKeyParsingTests {
#expect(service?.rootMapping?["[\"a\", \"b\"]"] == nil)
}
@Test("A complex key with no value is null; a bare sequence after it is extra content")
func test_complexKeyWithoutValueIsNullThenRejectsExtraContent() {
// `? [a, b]` with no `:` is a valid null-valued entry, so the following block sequence is a
// second root node, rejected as extra content rather than as a missing mapping value.
@Test("Rejects a sequence dangling after a complex key mapping")
func test_rejectsSequenceDanglingAfterComplexKeyMapping() {
// `? [a, b]` is a valid explicit key with an implicit empty (null) value,
// so the mapping closes cleanly. The same-column `- invalid` then dangles
// outside any node and is rejected at the document boundary.
expectParseError("""
? [a, b]
- invalid
""", .unexpectedToken(
expected: "document boundary or stream end",
actual: "blockEntry",
line: 2,
""", .malformed(
reason: "while parsing a block mapping, did not find expected key",
line: 1,
column: 1,
))
}

View File

@ -217,6 +217,9 @@ struct DiagnosticValidationReportTests {
column: 12,
reason: "missing space after ':' in mapping entry",
))
// The ported engine treats the colon-less, space-broken lines as one long
// plain scalar and reaches the genuinely unterminated quote on line 11, so
// the trailing parse diagnostic is the unterminated-quoted-string failure.
#expect(report.diagnostics.last == .init(
kind: .parse,
code: "unterminatedQuotedString",

View File

@ -275,3 +275,21 @@ struct DumperTests {
#expect(dumper.dump(value) == PureYAML.dump(value))
}
}
@Suite("Dumper Line Break Escaping")
struct DumperLineBreakEscapingTests {
@Test("Escapes CR and LF in a CRLF pair so the value round-trips")
func test_escapesCarriageReturnLineFeedPair() throws {
// A CR+LF pair is a single Swift Character (extended grapheme cluster), so a
// Character-level escape switch would emit it unescaped, and the folding
// parser would not read it back. Each line break must escape independently.
let value = PureYAML.Model.Value.string("a\r\nb\r\n\r\nc")
let dumped = PureYAML.dump(value)
#expect(dumped == "\"a\\r\\nb\\r\\n\\r\\nc\"\n")
#expect(!dumped.unicodeScalars.contains("\r"))
let reparsed = try PureYAML.parse(dumped)
#expect(reparsed == value)
}
}

View File

@ -43,18 +43,6 @@ struct EventStreamingParserTests {
"error: Duplicate mapping key 'id' at $.flow.id",
])
}
@Test(
"Lazy token cursor matches full scanner tokens",
arguments: EventStreamingFixtures.singleDocuments + EventStreamingFixtures.streams,
)
func test_lazyTokenCursorMatchesFullScannerTokens(testCase: EventStreamingFixtures.DocumentCase) throws {
let full = try PureYAML.Parsing.Scanner().scan(testCase.yaml)
.filter { !$0.kind.isComment }
let lazy = try collectLazyTokens(testCase.yaml)
#expect(lazy == full)
}
}
enum EventStreamingFixtures {
@ -155,13 +143,3 @@ private func composeStreamWithFullEvents(_ yaml: String) throws -> [PureYAML.Str
var composer = PureYAML.Parsing.EventComposer(events: events, scalarParser: parser)
return try composer.composeStream()
}
private func collectLazyTokens(_ yaml: String) throws -> [PureYAML.Parsing.Token] {
var cursor = PureYAML.Parsing.TokenCursor(yaml: yaml)
var tokens: [PureYAML.Parsing.Token] = []
while let token = cursor.current {
tokens.append(token)
_ = try cursor.advance()
}
return tokens
}

View File

@ -6,11 +6,14 @@ import Testing
/// line ending at `:` is instead a new mapping key, so the scalar stops before it, unchanged.
@Suite("Explicit key multiline fold")
struct ExplicitKeyMultilineFoldTests {
@Test("An explicit key folds a continuation line that ends at the value colon")
func test_continuationEndingAtColon() throws {
#expect(try PureYAML.parse("? foo\n bar : baz\n") == .mapping(.init([
.init(key: "foo bar", value: .string("baz")),
])))
@Test("An explicit key continuation ending at a value colon is rejected")
func test_continuationEndingAtColon() {
// libyaml rejects a `: value` on a key continuation line ("mapping values are not
// allowed here"): the value indicator must sit at the key's own level, on a later
// line, as the next test proves. The ported engine agrees with the reference.
#expect(throws: PureYAML.Parsing.ParseError.self) {
try PureYAML.parse("? foo\n bar : baz\n")
}
}
@Test("An explicit key with the colon on its own line still folds")

File diff suppressed because it is too large Load Diff

View File

@ -0,0 +1,12 @@
[
{
"name": "Mark McGwire",
"hr": 65,
"avg": 0.278
},
{
"name": "Sammy Sosa",
"hr": 63,
"avg": 0.288
}
]

View File

@ -0,0 +1,8 @@
-
name: Mark McGwire
hr: 65
avg: 0.278
-
name: Sammy Sosa
hr: 63
avg: 0.288

View File

@ -0,0 +1,22 @@
+STR
+DOC
+SEQ
+MAP
=VAL :name
=VAL :Mark McGwire
=VAL :hr
=VAL :65
=VAL :avg
=VAL :0.278
-MAP
+MAP
=VAL :name
=VAL :Sammy Sosa
=VAL :hr
=VAL :63
=VAL :avg
=VAL :0.288
-MAP
-SEQ
-DOC
-STR

View File

@ -0,0 +1,3 @@
foo:
bar
invalid

View File

@ -0,0 +1,5 @@
+STR
+DOC
+MAP
=VAL :foo
=VAL :bar

View File

@ -0,0 +1,18 @@
{
"top1": {
"key1": "scalar1"
},
"top2": {
"key2": "scalar2"
},
"top3": {
"scalar1": "scalar3"
},
"top4": {
"scalar2": "scalar4"
},
"top5": "scalar5",
"top6": {
"key6": "scalar6"
}
}

View File

@ -0,0 +1,12 @@
"top1" :
"key1" : &alias1 scalar1
'top2' :
'key2' : &alias2 scalar2
top3: &node3
*alias1 : scalar3
top4:
*alias2 : scalar4
top5 :
scalar5
top6:
&anchor6 'key6' : scalar6

View File

@ -0,0 +1,33 @@
+STR
+DOC
+MAP
=VAL "top1
+MAP
=VAL "key1
=VAL &alias1 :scalar1
-MAP
=VAL 'top2
+MAP
=VAL 'key2
=VAL &alias2 :scalar2
-MAP
=VAL :top3
+MAP &node3
=ALI *alias1
=VAL :scalar3
-MAP
=VAL :top4
+MAP
=ALI *alias2
=VAL :scalar4
-MAP
=VAL :top5
=VAL :scalar5
=VAL :top6
+MAP
=VAL &anchor6 'key6
=VAL :scalar6
-MAP
-MAP
-DOC
-STR

View File

@ -0,0 +1 @@
"text"

View File

@ -0,0 +1,2 @@
%YAML 1.2
--- text

View File

@ -0,0 +1,5 @@
+STR
+DOC ---
=VAL :text
-DOC
-STR

View File

@ -0,0 +1,6 @@
[
"a",
"b",
42,
"d"
]

Some files were not shown because too many files have changed in this diff Show More