NodetteHQ/PureYAML
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat(parsing): persist mid-token suspension and expose it as a public streaming API

Browse Source 
Plain scalars now suspend and resume mid-token instead of rolling back to a
checkpoint and rescanning from the token's start on every chunk boundary
inside them: scanPlainScalar's loop is an explicit PlainScalarPhase state
machine that throws PlainScalarSuspended the moment a lookahead runs dry, and
scanStep() resumes it directly from Scanner.suspendedPlainScalar. Getting
this right took four rounds of bugs, each caught by the existing whole-suite
streaming differential test: checking ranDry only once at the loop's end
instead of after every peek-dependent decision, resuming a dry-interrupted
content or break run through a pre-check meant only for a fresh entry point,
trusting dispatchToken's own dry-fooled document-marker check, and a nested
blankLineAhead() lookahead independently going dry before an outer check
could catch it.

A synthetic long-scalar test (needed since the real-world benchmark fixture's
scalars are too short to exercise this) then caught a fifth, genuine bug: the
first working resumePlainScalar took the whole suspended value by parameter,
which kept a reference to its accumulated string alive for the entire resume
call and defeated copy-on-write, copying the whole scalar on every resume.
That is O(N^2) in the scalar's length, and measured 29x slower than not
suspending at all at small chunk sizes. Passing state inout directly, with
the alias broken at the call site before the call, fixed it to 3-4x, verified
genuinely linear (not just a smaller quadratic constant) by a permanent test
scaling scalar length 16x at a fixed chunk size.

Also exposes this scanner as public API for the first time: scanStep() and
everything around it were internal-only, so no consumer outside this module,
including the separate PureYAMLStreaming package that actually wants a
resumable token scanner, could call it. PureYAML.Streaming.Scanner wraps
scanStep() directly with no independent logic, translating its tokens into
public Streaming.Token/Kind/ScalarStyle types that don't leak the libyaml
port's internal naming. Proven separately from the internal scanner with its
own differential test, since a wrong case mapping in the translation layer
wouldn't be caught by a proof that never looks at the public types.

Stops short of rewiring PureYAMLStreaming's own document-boundary scanner
onto this; that is separate, cross-repo work with its own build and CI cycle.
This commit is contained in:
Mihaela Mihaljevic 2026-07-10 09:10:56 +02:00
parent 6d3f3009d2
commit 41ca600d93
14 changed files with 876 additions and 58 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

162
CHANGELOG.md
View File

@ -9,6 +9,130 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- Add a public `PureYAML.Streaming` namespace (issue #4), exposing the
internal `LibYAML.Scanner.scanStep()` proven below for the first time as a
callable public API rather than a module-internal implementation detail.
Needed because `scanStep()` and everything in
`LibYAML.Scanner.Streaming.swift` were internal-only: no consumer outside
the `PureYAML` module, including the separate `PureYAMLStreaming` incubator
package that issue #4 names as this feature's actual intended user, could
call it. `PureYAML.Streaming.Scanner` (`init()`, `feed(_:)`, `finish()`,
`next() throws -> Outcome`) is a thin wrapper with no logic of its own,
translating `scanStep()`'s `ScanOutcome`/internal `LibYAML.Token` into
public `Outcome`/`Streaming.Token` types under public naming that does not
leak the libyaml port's internal type names (`Streaming.Token.Kind` mirrors
`LibYAML.TokenType`'s 22 cases 1:1; `Streaming.Token.ScalarStyle` mirrors
the scanner's internal `Parsing.ScalarStyle`, a distinct 5-case type from
the existing public `Emitting.ScalarStyle`, which is an output rendering
policy, not a scanned style; `Streaming.Token.range` reuses the existing
public `Model.SourceRange`/`SourcePosition` rather than inventing a new
position type). `Token.Kind.init(_:)`'s mapping switch is split across
three chained initializers purely to stay under the project's cyclomatic
complexity lint limit (a mechanical 1:1 case mapping over 22 cases has no
way to be less than 22-branches complex in one function); the split carries
no semantic meaning.
Proven with a new differential test (`PublicStreamingScannerTests.swift`)
distinct from the existing `StreamingScannerDifferentialTests.swift`: that
existing proof covers whether `scanStep()` itself is correct, but does not
exercise the public wrapper or its translation layer at all, so a wrong
case mapping or a dropped field there would not have been caught by it.
The new proof drives the public `Streaming.Scanner` at several chunk sizes
and asserts it matches the batch scanner's own tokens, translated through
the identical `Streaming.Token.init(_:)` the public scanner itself uses,
clean across the whole yaml-test-suite corpus.
Deliberately scoped to stop here: this makes the public API exist and
proves it correct, but does not yet rewire the separate `PureYAMLStreaming`
package's own `ResumableDocumentScanner` (currently a naive line-based
`---` heuristic scanner with no relationship to real YAML grammar) onto it.
That rewiring is real, separate, cross-repo work with its own build/test/CI
cycle, deferred to a follow-up round rather than folded in here.
- Implement persisted mid-token suspension for plain scalars (issue #4), the
deferred piece the previous entry predicted would eliminate the checkpoint
floor entirely. `scanPlainScalar()`'s loop (`LibYAML.Scanner.PlainScalar.swift`)
is now an explicit `PlainScalarPhase` state machine (`.start`,
`.consumingContent`, `.afterContent`, `.consumingBreaks`, `.afterBreaks`)
that throws `PlainScalarSuspended` (carrying its accumulated `PlainState`,
indent, start mark, and phase) the moment a lookahead runs dry, and
`scanStep()` resumes it directly from `Scanner.suspendedPlainScalar` with no
checkpoint at all, rather than rolling back and rescanning the whole token
from its start on every chunk boundary inside it.
Getting this correct took four rounds of bugs, each found by the existing
whole-yaml-test-suite streaming differential test
(`PUREYAML_RUN_YAML_TEST_SUITE=1 swift test --filter
"StreamingScannerDifferentialTests|YAMLTestSuiteGapsTests" --no-parallel`),
none of them hypothetical: (1) checking `reader.ranDry` only once at the
loop's end is not enough, since a nested lookahead
(`documentMarkerAhead()`, the `:`-followed-by-blank check, `blankLineAhead()`)
can independently go dry and produce a wrong-but-plausible answer that gets
used by the next step before an end-of-loop check would ever see the taint;
fixed by checking immediately after every peek-dependent decision, before
its result is used for anything. (2) A resume must never redo a pre-check
that only applies at a fresh entry point (the document-marker/comment check
in `.start`, the blank-or-break guard in `.afterContent`): resuming a
dry-interrupted content or break run via `.start`/`.afterContent` again can
wrongly end a scalar right at a literal `#` or reinterpret an in-progress
break run, since those positions are not guaranteed to be fresh boundaries.
Fixed with dedicated `.consumingContent`/`.consumingBreaks` phases that
re-enter the same in-progress call with no pre-check. (3) `dispatchToken()`
itself has an `isDocumentMarker` pre-check, ahead of ever calling
`fetchPlainScalar()`, that can suffer the identical dry-lookahead problem;
since nothing can undo a wrong dispatch after the fact, `scanStep()` only
trusts a suspend enough to resume it when real content was already
accumulated (`phase == .start` with an empty `state.string` retries from
scratch instead). (4) `blankLineAhead()` inside `consumePlainBreaks` is
itself a multi-character lookahead distinct from the single
already-confirmed `peek()` before it, so it can independently go dry; fixed
by checking `ranDry` right after computing its result, before a
tab-violation decision trusts it.
A synthetic worst-case test was added for this specifically because the
9.5MB real-world fixture used above has no plain scalars long enough to
exercise it (`test_measureLongPlainScalar`, one 1MB single-line plain
scalar). It caught a fifth bug, a real one, not a suspicion: the first
working version of `resumePlainScalar` took the whole `PlainScalarSuspended`
as a by-value parameter and read `var state = suspended.state` inside it,
but `suspended.startMark` being read again at the very end kept the whole
parameter (including its `state.string` reference) alive as an ordinary
borrowed value for the call's entire duration, aliasing `state.string` to
it throughout `runPlainScalarLoop` and forcing copy-on-write to fully copy
the entire accumulated scalar on every single resume; for a scalar rebuilt
across many small chunks that is O(N^2) in the scalar's length, not the
O(N) this feature exists to guarantee, and measured **29.05x** at
chunk-size 16 on the synthetic case, worse than not suspending at all.
Passing `state` `inout` directly, with the alias broken at the call site
(`scanStep()`'s new `resumeSuspendedPlainScalarIfNeeded()`, overwriting
`suspended.state` with a fresh, unrelated value immediately after copying
it out and before ever calling into `resumePlainScalar`) fixed it to
**3.67x** at the same chunk size, with no correctness regression
(whole-suite differential clean in both debug and release, full default
suite of 597 tests passing).
A single before/after data point does not by itself distinguish "fixed" from
"smaller constant, still quadratic", so a second, permanent test
(`test_longPlainScalarStaysLinear`) checks the actual complexity claim: per-
byte cost at a fixed chunk size (16) across a 16x range of scalar lengths
(250KB to 4MB), asserting the largest measured per-byte cost stays under 3x
the smallest (generous headroom over the ~1.08x actually measured; a real
O(N^2) regression would blow well past 3x as length grows). Genuinely O(N)
work should hold per-byte cost flat regardless of total length, which this
confirms rather than assumes from the single 1MB case above.
Measured honestly rather than assumed to have delivered the predicted win:
on the 9.5MB real-world fixture, the checkpoint floor and rescan-driven
tail are statistically unchanged from before this feature existed
(floor 1.75x versus the prior entry's 1.75x-1.76x; 10.74x at chunk-size 16
versus the prior entry's 11.54x, both within measurement noise across
runs). The prior entry's prediction that mid-token suspension "would
eliminate the floor too" is corrected here: it eliminates rescanning
specifically for plain scalars that themselves span a chunk boundary, but
`github-rest-api.yaml`'s plain scalars are short enough that this was
never a meaningful contributor to its measured floor in the first place,
and the floor's actual cause (a `Scanner` struct copy taken on every
`fetchNextToken()` call whenever `!isFinal`, regardless of token type,
established in the entry below) is untouched by a plain-scalar-specific
fix. The synthetic long-scalar test above is the fixture that actually
demonstrates this feature's value (a 29x-worse-than-no-suspension bug,
fixed down to 3-4x); the real-world fixture simply does not have the
shape of input this feature targets.
- Add `Tests/StreamingScannerPerformanceTests.swift` (issue #4), measuring
`scanStep()`'s overhead over `scanAll()` on the largest real-world fixture
(github-rest-api.yaml, 9.5MB) across a spread of chunk sizes, gated behind
@ -39,15 +163,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
that copy is taken for virtually every token in the file. Only *shrinking*
chunk sizes below roughly 4096 scalars adds a second, genuinely
rescan-driven cost on top of that floor (11.54x at 16 scalars).
This corrects the previous entry's conclusion: the deferred persisted
mid-token suspension would eliminate the rescan-driven tail at small chunk
sizes, but would not touch the 1.75x floor, since that floor is not caused
by rescanning. Closing the floor needs a different fix, not yet designed:
something that avoids copying the whole scanner on every fetch while
`isFinal` is false, without weakening the correctness guarantee that only
`isFinal: true` currently provides (every other field the checkpoint covers
can be mutated before a dry point is reached within a single fetch, so no
smaller subset of fields is obviously safe to skip either).
Two follow-up checks, not assumed: ruled out buffer-population strategy as a
contributor (`Reader.init`'s `Array(string.unicodeScalars)` versus
`Reader.append(_:)`'s `append(contentsOf:)` measured 1.00x against each
other on the same 9.5MB string, no scanning involved), and re-profiled with
`sample`(1) on the release build this time (the first profile, in the entry
above, was debug-only and already shown to be misleading for magnitude; it
turns out to also be misleading for location). The release profile's
hottest symbols are String-building-adjacent
(`_StringGuts.append`, `UnicodeScalarView.distance(from:to:)`,
`consumePlainContent`) rather than a separately named "copy of Scanner"
symbol, most likely because the optimizer inlines the checkpoint's struct
copy directly into `scanStep()` (190 samples on its own, disproportionate to
its few lines of source) rather than outlining it as in debug, and the extra
live reference the checkpoint holds to `indentEngine.tokens` forces
conservative uniqueness handling that shows up as slower string mutation
nearby rather than as its own line item.
The structural cause, confirmed by an A/B the project already had (not a
new one): the checkpoint code path is still the only difference between the
1.04x `isFinal: true` control and the 1.75x-1.76x single-chunk/large-chunk
results, both driven through the identical per-token loop. This corrects the
previous entry's conclusion in an important way: the deferred persisted mid-
token suspension would not merely eliminate the rescan-driven tail while
leaving the floor untouched, it would eliminate the floor too, since it
removes the need for a checkpoint entirely rather than making one cheaper.
A cheaper checkpoint, absent mid-token suspension, is not yet designed:
every field the checkpoint covers can be mutated before a dry point is
reached within a single fetch, so no smaller subset of fields is obviously
safe to skip, and `isFinal: true` remains the only provably safe condition
for skipping it outright.
- Broaden the real-seed mutation property suite (issue #6) with three mutation
classes named in that issue but not yet covered: `invalidControlCharacter`

8
Sources/Parsing/LibYAML.Reader.swift
View File

@ -42,9 +42,15 @@ extension PureYAML.Parsing.LibYAML {
}
/// Append a chunk of newly arrived input. Only meaningful before
/// `markFinal()`.
/// `markFinal()`. Clears `ranDry`: new data may satisfy whatever `peek(_:)`
/// call set it, and nothing else ever clears it. The existing
/// checkpoint-restore path gets this for free (the restored `Reader` is a
/// copy from before the dry read), but a resumed scan that keeps its
/// reader position instead of restoring a snapshot (`PlainScalarSuspended`)
/// has no other point where a stale `true` would ever become false again.
mutating func append(_ chunk: String) {
scalars.append(contentsOf: chunk.unicodeScalars)
ranDry = false
}
/// Declare that no further chunks are coming: a `peek` past the buffered

221
Sources/Parsing/LibYAML.Scanner.PlainScalar.swift
View File

@ -1,10 +1,58 @@
extension PureYAML.Parsing.LibYAML.Scanner {
/// Which step of `runPlainScalarLoop`'s outer iteration a suspend happened
/// at. Resuming must land back on the step that was actually interrupted,
/// not on whatever step normally follows it, and in particular must never
/// redo a decision (document marker, comment, or "does a blank/break run
/// follow") that a *previous, already-passed* check already answered:
/// `.start`'s document-marker/`#` checks and `.afterContent`'s blank/break
/// guard are only meant to run once, at a genuine entry point, not at an
/// arbitrary position inside an already-in-progress content or break run.
/// A first version of this type resumed a dry `consumePlainContent` via
/// `.start` on the (wrong) assumption that redoing its checks was harmless;
/// it is not; if the resume position happens to land on a literal `#`
/// that is legitimate mid-scalar content (e.g. a value containing every
/// printable character), `.start`'s comment check wrongly ends the scalar
/// right there. `consumePlainContent` and `consumePlainBreaks` each get
/// their own dedicated "continue this same call, no pre-check" case
/// (`.consumingContent`, `.consumingBreaks`) for exactly this reason.
enum PlainScalarPhase: Equatable {
/// Check for a document marker or comment, then consume content.
case start
/// Inside consuming an already-started content run (the document
/// marker/comment checks that gated entry into it do not apply here).
case consumingContent
/// Content just finished; decide whether a blank/break run follows.
case afterContent
/// Inside consuming an already-confirmed blank/break run.
case consumingBreaks
/// A blank/break run just finished; check whether the scalar dedents.
case afterBreaks
}
/// Thrown by `runPlainScalarLoop` when accumulating a plain scalar's content
/// runs dry (`Reader.ranDry`) while `!Reader.isFinal`. Carries everything
/// `resumePlainScalar(_:)` needs to continue from exactly this point once more
/// input is appended, rather than rescanning the scalar's content from its
/// start. `scanStep()` catches this specifically and stores it on
/// `Scanner.suspendedPlainScalar`, distinct from a generic dry-run catch: no
/// checkpoint restore happens for this error, since the whole point is to keep
/// the reader position and accumulated content rather than discard them.
struct PlainScalarSuspended: Swift.Error {
var state: PlainState
var indent: Int
var startMark: PureYAML.Parsing.Mark
var phase: PlainScalarPhase
}
private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass
/// The folding state carried through `scan_plain_scalar`: the accumulated
/// value plus the pending leading break, trailing breaks, and whitespace that
/// are folded in before the next content character.
private struct PlainState {
/// are folded in before the next content character. Internal rather than
/// private: `PlainScalarSuspended` carries one across a `scanStep()` suspend
/// point, so `Scanner`'s own file needs to name the type for its stored
/// `suspendedPlainScalar` property.
struct PlainState {
var string = ""
var leadingBreak = ""
var trailingBreaks = ""
@ -22,24 +70,146 @@ extension PureYAML.Parsing.LibYAML.Scanner {
let indent = indentEngine.indent + 1
let startMark = reader.mark
var state = PlainState(endMark: reader.mark)
try runPlainScalarLoop(&state, indent: indent, startMark: startMark)
return finishPlainScalar(state, startMark: startMark)
}
/// Resumes a plain scalar suspended by a previous `PlainScalarSuspended`,
/// continuing from its saved state, phase, and reader position rather than
/// rescanning the content already accumulated before the suspend.
///
/// Takes `state` `inout` rather than the whole `PlainScalarSuspended`, on
/// purpose: the caller (`scanStep()`) must disalias `state` from the
/// suspended value it came from *before* this call, since a `let
/// suspended = ...; resumePlainScalar(suspended)` call would keep the
/// caller's own `suspended.state` reference to the same string buffer
/// alive for this whole call's duration regardless of anything done
/// inside this function's own body (a value parameter is borrowed for
/// the call, not owned, so a local disalias inside the callee cannot
/// remove a reference the caller itself is still holding). With `state`
/// passed by direct reference and no wrapping struct crossing this call
/// boundary at all, there is exactly one owner of its string buffer
/// throughout, so appending inside the loop below never triggers a
/// spurious copy-on-write copy of the whole accumulated scalar. See
/// `scanStep()`'s resume branch for where the disalias happens.
mutating func resumePlainScalar(
_ state: inout PlainState,
indent: Int,
startMark: PureYAML.Parsing.Mark,
phase: PlainScalarPhase,
) throws -> PureYAML.Parsing.LibYAML.Token {
try runPlainScalarLoop(&state, indent: indent, startMark: startMark, from: phase)
return finishPlainScalar(state, startMark: startMark)
}
/// The scan loop shared by a fresh attempt and a resume, restructured as an
/// explicit `PlainScalarPhase` state machine (see its own doc comment for
/// why) with a `suspendIfDry` check after every step that peeked. Checking
/// only once, after the whole loop exits, is not enough: a dry read
/// (`reader.ranDry`, see `Reader.peek(_:)`) inside a lookahead
/// (`documentMarkerAhead()`, `consumePlainContent`'s `:`-followed-by-blank
/// check, `consumePlainBreaks`'s `blankLineAhead()` guard) reads as "blank or
/// end" (`CharacterClass.isBlankZ`/`isBreakZ` treat a missing scalar as
/// true), not as "not enough data yet", so the *next* step can reach a
/// wrong-but-plausible decision (e.g. `documentMarkerAhead` wrongly says
/// "not a marker", so `consumePlainContent` appends `---` as scalar content)
/// before a single end-of-loop check would ever see the taint that caused
/// it. Checking right after each peeking step, before its result is used for
/// anything further, catches the taint before it can influence another
/// decision. Batch parsing (`isFinal: true`) is unaffected: every check can
/// only throw when `!reader.isFinal`, which batch callers never set, so the
/// loop's own logic is otherwise identical to the original.
///
/// `state` and the reader position are always resumable at a suspend point:
/// nothing here calls `reader.skip()` on a dry `nil`, so accumulated content
/// and position both stop exactly at the last successfully read scalar.
private mutating func runPlainScalarLoop(
_ state: inout PlainState,
indent: Int,
startMark: PureYAML.Parsing.Mark,
from initialPhase: PlainScalarPhase = .start,
) throws {
var phase = initialPhase
while true {
if reader.column == 0, documentMarkerAhead() {
break
}
if reader.peek() == "#" {
break
}
try consumePlainContent(&state, startMark: startMark)
guard Classify.isBlank(reader.peek()) || Classify.isBreak(reader.peek()) else {
break
}
try consumePlainBreaks(&state, indent: indent, startMark: startMark)
if flowLevel == 0, reader.column < indent {
break
switch phase {
case .start:
let atDocumentMarker = reader.column == 0 && documentMarkerAhead()
// A dry lookahead inside `documentMarkerAhead()` must resume by
// redoing this whole step, not by trusting `atDocumentMarker`
// (which `isBlankZ`'s nil-is-true bias could have gotten wrong)
// and moving on: hence `phase: .start` here, not `.consumingContent`.
try suspendIfDry(state, indent: indent, startMark: startMark, phase: .start)
if atDocumentMarker {
return
}
if reader.peek() == "#" {
return
}
try consumePlainContent(&state, startMark: startMark)
// consumePlainContent stopping is not yet known to be a real
// answer (terminator found) versus a dry cutoff: check before
// trusting it enough to move to `.afterContent`'s guard. On dry,
// resume via `.consumingContent`, not `.start`: the document-marker
// and `#` checks above only apply at a genuine entry point, and
// the reader position where a content run was interrupted can
// land on a literal `#` or `-` that is legitimate content, not a
// fresh comment or marker.
try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingContent)
phase = .afterContent
case .consumingContent:
try consumePlainContent(&state, startMark: startMark)
try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingContent)
phase = .afterContent
case .afterContent:
// Reached only once consumePlainContent is known to have stopped
// for a real reason, so this peek cannot itself be dry: it
// re-reads the exact position consumePlainContent's own last
// (already non-dry) peek confirmed, and nothing advanced between.
guard Classify.isBlank(reader.peek()) || Classify.isBreak(reader.peek()) else {
return
}
try consumePlainBreaks(&state, indent: indent, startMark: startMark)
// Same reasoning as consumePlainContent above, but resuming here
// must not redo `.afterContent`'s guard against new data: that
// guard gates *entry* into a blank/break run, and re-asking it
// after the run is already underway would wrongly treat "the run
// continues" as "the run never started" (or vice versa) instead
// of just continuing the same run. `.consumingBreaks` re-enters
// consumePlainBreaks directly.
try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingBreaks)
phase = .afterBreaks
case .consumingBreaks:
try consumePlainBreaks(&state, indent: indent, startMark: startMark)
try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingBreaks)
phase = .afterBreaks
case .afterBreaks:
// `reader.column` is a plain stored field, never read via
// `peek(_:)`, so this check cannot itself observe a dry read; by
// construction every path reaching this case already confirmed
// `!ranDry` before transitioning here.
if flowLevel == 0, reader.column < indent {
return
}
phase = .start
}
}
}
private func suspendIfDry(
_ state: PlainState,
indent: Int,
startMark: PureYAML.Parsing.Mark,
phase: PlainScalarPhase,
) throws {
if !reader.isFinal, reader.ranDry {
throw PlainScalarSuspended(state: state, indent: indent, startMark: startMark, phase: phase)
}
}
private mutating func finishPlainScalar(
_ state: PlainState,
startMark: PureYAML.Parsing.Mark,
) -> PureYAML.Parsing.LibYAML.Token {
if state.leadingBlanks {
simpleKeyAllowed = true
}
@ -106,11 +276,22 @@ extension PureYAML.Parsing.LibYAML.Scanner {
private mutating func consumePlainBreaks(_ state: inout PlainState, indent: Int, startMark: PureYAML.Parsing.Mark) throws {
while Classify.isBlank(reader.peek()) || Classify.isBreak(reader.peek()) {
if Classify.isBlank(reader.peek()) {
if state.leadingBlanks, reader.column < indent, Classify.isTab(reader.peek()), !blankLineAhead() {
// A tab below the indentation is illegal only when it indents
// real content; on an otherwise-blank line it is just a blank
// line the scalar folds over. libyaml rejects it either way.
throw scannerError("found a tab character that violates indentation", at: startMark)
if state.leadingBlanks, reader.column < indent, Classify.isTab(reader.peek()) {
// `blankLineAhead()` is its own multi-character lookahead (not
// just the single already-confirmed-real `peek()` above), so it
// can independently run dry: check immediately, before the
// tab-violation decision below trusts its answer. Unchecked,
// `isBreakZ`'s nil-is-true bias makes an incomplete lookahead
// read as "blank line", suppressing a real violation until more
// data arrives proves otherwise.
let onBlankLine = blankLineAhead()
try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingBreaks)
if !onBlankLine {
// A tab below the indentation is illegal only when it indents
// real content; on an otherwise-blank line it is just a blank
// line the scalar folds over. libyaml rejects it either way.
throw scannerError("found a tab character that violates indentation", at: startMark)
}
}
if !state.leadingBlanks, let blank = reader.peek() {
state.whitespaces.unicodeScalars.append(blank)

132
Sources/Parsing/LibYAML.Scanner.Streaming.swift
View File

@ -33,33 +33,78 @@ extension PureYAML.Parsing.LibYAML.Scanner {
if streamEndProduced {
return .done
}
// A dry run can only ever be reported while `!reader.isFinal` (see
// `Reader.peek(_:)`), so once the caller has appended every chunk and
// called `markFinal()`, no fetch can ever need rolling back. Skipping
// the snapshot in that case matters: it is the entire cost difference
// between `scanStep()` and `scanAll()` once the whole input is already
// buffered (measured in StreamingScannerPerformanceTests), since neither
// reads nor mutations inside `fetchNextToken()` become any cheaper by
// being wrapped in an unused checkpoint.
guard !reader.isFinal else {
try fetchNextToken()
// A plain scalar in progress resumes from exactly where it stopped,
// with no checkpoint at all: `runPlainScalarLoop` never leaves `reader`
// or `state` in a state that needs rolling back (see
// `PlainScalarSuspended`), so there is nothing to restore if it runs
// dry again.
switch try resumeSuspendedPlainScalarIfNeeded() {
case .notSuspended:
break
case .tokenReady:
continue
case .stillDry:
return .needMoreInput
}
let checkpoint = self
do {
try fetchNextToken()
} catch {
if reader.ranDry {
self = checkpoint
return .needMoreInput
}
throw error
if let outcome = try fetchNextTokenCheckpointed() {
return outcome
}
}
}
/// A dry run can only ever be reported while `!reader.isFinal` (see
/// `Reader.peek(_:)`), so once the caller has appended every chunk and
/// called `markFinal()`, no fetch can ever need rolling back. Skipping
/// the snapshot in that case matters: it is the entire cost difference
/// between `scanStep()` and `scanAll()` once the whole input is already
/// buffered (measured in StreamingScannerPerformanceTests), since neither
/// reads nor mutations inside `fetchNextToken()` become any cheaper by
/// being wrapped in an unused checkpoint. Returns `nil` to mean "keep
/// looping in `scanStep()`", matching a fetch that produced no directly
/// returnable outcome (a non-plain-scalar token was merely enqueued, or a
/// document/directive token dispatched with no scalar involved).
private mutating func fetchNextTokenCheckpointed() throws -> ScanOutcome? {
guard !reader.isFinal else {
try fetchNextToken()
return nil
}
let checkpoint = self
do {
try fetchNextToken()
} catch let suspended as PlainScalarSuspended {
// `fetchPlainScalar()` calls `saveSimpleKey()` and sets
// `simpleKeyAllowed = false` before ever calling `scanPlainScalar()`,
// and a plain-scalar-only resume has no way to undo those: it is
// only safe to keep them applied and resume past this suspend if
// real progress (content) exists to preserve. `phase == .start`
// with an empty `state` means the very first check inside the loop
// (before `consumePlainContent` ever ran, e.g. `documentMarkerAhead()`'s
// own lookahead went dry) suspended: nothing would be saved by
// resuming instead of retrying from scratch, and retrying from
// scratch is what correctly re-derives whether this token should
// have been dispatched as a plain scalar at all. `dispatchToken()`'s
// own `isDocumentMarker` check, called before `fetchPlainScalar()`
// is ever reached, can suffer the exact same dry-lookahead problem
// `runPlainScalarLoop` guards against; only a full retry re-checks it
// with more data, since nothing here can undo a wrong dispatch.
if suspended.phase == .start, suspended.state.string.isEmpty {
self = checkpoint
return .needMoreInput
}
suspendedPlainScalar = suspended
return .needMoreInput
} catch {
if reader.ranDry {
self = checkpoint
return .needMoreInput
}
throw error
}
if reader.ranDry {
self = checkpoint
return .needMoreInput
}
return nil
}
private var isFrontSafeToRelease: Bool {
@ -68,4 +113,53 @@ extension PureYAML.Parsing.LibYAML.Scanner {
}
return !simpleKeys.contains { $0.possible && $0.tokenNumber == indentEngine.tokensParsed }
}
private enum ResumeOutcome {
/// No plain scalar was suspended; `scanStep()` should fall through
/// to its normal fetch path.
case notSuspended
/// The suspended scalar was resumed to completion and its token was
/// enqueued; `scanStep()` should loop back to the top and try to
/// release it, not fall through to a fresh fetch.
case tokenReady
/// The resume itself ran dry again; `scanStep()` should report
/// `.needMoreInput`.
case stillDry
}
/// Extracted out of `scanStep()` (which was tripping the cyclomatic
/// complexity limit) rather than inlined, this also fixes a real O(N^2)
/// bug the inline version had: extracting `state` and immediately
/// overwriting `suspended.state` with a fresh, unrelated value before
/// calling `resumePlainScalar` is required, not optional. A plain
/// `resumePlainScalar(suspended)` call, passing the whole suspended
/// value, would keep a reference to it alive (as an ordinary borrowed
/// value parameter) for this call's entire duration regardless of what
/// the callee does internally, leaving `state`'s string buffer with two
/// live owners throughout `runPlainScalarLoop` and forcing copy-on-write
/// to fully copy the accumulated scalar on every single append.
/// Overwriting `suspended.state` here, before the call, drops that
/// reference for good, so `state` is the sole owner by the time
/// `resumePlainScalar` starts mutating it (see
/// `StreamingScannerPerformanceTests.test_measureLongPlainScalar`, which
/// caught this at a measured 29x versus the ~3-4x this fix gives).
private mutating func resumeSuspendedPlainScalarIfNeeded() throws -> ResumeOutcome {
guard var suspended = suspendedPlainScalar else {
return .notSuspended
}
suspendedPlainScalar = nil
var state = suspended.state
suspended.state = PlainState(endMark: state.endMark)
let indent = suspended.indent
let startMark = suspended.startMark
let phase = suspended.phase
do {
let token = try resumePlainScalar(&state, indent: indent, startMark: startMark, phase: phase)
indentEngine.enqueue(token)
return .tokenReady
} catch let resuspended as PlainScalarSuspended {
suspendedPlainScalar = resuspended
return .stillDry
}
}
}

4
Sources/Parsing/LibYAML.Scanner.swift
View File

@ -42,6 +42,10 @@ extension PureYAML.Parsing.LibYAML {
var streamStartProduced = false
/// `parser->stream_end_produced`.
var streamEndProduced = false
/// A plain scalar fetch that ran dry mid-content (`scanStep()` only, batch
/// scanning never sets this): resumed from here on the next `scanStep()`
/// call instead of restarting the scalar from its first scalar.
var suspendedPlainScalar: PlainScalarSuspended?
init(_ input: String = "", isFinal: Bool = true) {
reader = Reader(input, isFinal: isFinal)

3
Sources/PureYAML.swift
View File

@ -21,6 +21,9 @@ public enum PureYAML {
/// YAML stream types.
public enum Stream {}
/// Resumable, chunked YAML scanning types.
public enum Streaming {}
/// Tag-preserving YAML document model types.
public enum Tagged {}
}

13
Sources/Streaming/Streaming.Outcome.swift Normal file
View File

@ -0,0 +1,13 @@
public extension PureYAML.Streaming {
/// The result of one ``Scanner/next()`` call.
enum Outcome: Equatable, Sendable {
/// A token is ready.
case token(Token)
/// The buffered input was exhausted mid-scan; call ``Scanner/feed(_:)``
/// with more input (or ``Scanner/finish()`` if there is no more) and
/// call ``Scanner/next()`` again.
case needMoreInput
/// The stream has ended; no further tokens remain.
case done
}
}

117
Sources/Streaming/Streaming.Scanner.swift Normal file
View File

@ -0,0 +1,117 @@
public extension PureYAML.Streaming {
/// A resumable, chunked YAML token scanner: the public counterpart to the
/// internal libyaml-ported scanner's `scanStep()`, which this wraps
/// directly with no independent logic of its own. Proven to match batch
/// scanning byte-for-byte across the whole yaml-test-suite corpus and at
/// several chunk sizes (`StreamingScannerDifferentialTests` in the test
/// suite exercises the identical internal `scanStep()` this type calls).
struct Scanner: Sendable {
private var scanner: PureYAML.Parsing.LibYAML.Scanner
/// Creates a scanner with no input yet. Feed input with ``feed(_:)``
/// and mark the end of input with ``finish()``.
public init() {
scanner = PureYAML.Parsing.LibYAML.Scanner("", isFinal: false)
}
/// Appends a chunk of YAML source. Chunks are concatenated in the
/// order fed; a chunk need not align with any YAML grammar boundary.
public mutating func feed(_ chunk: String) {
scanner.reader.append(chunk)
}
/// Marks that no further input will be fed. After this, ``next()``
/// never returns ``Outcome/needMoreInput``: a fetch that would have
/// needed more input instead surfaces as a real scanning error, since
/// there is no more input coming that could resolve it.
public mutating func finish() {
scanner.reader.markFinal()
}
/// Advances the scan by one token. Call in a loop, feeding more input
/// on ``Outcome/needMoreInput`` and stopping on ``Outcome/done``.
public mutating func next() throws -> Outcome {
switch try scanner.scanStep() {
case let .token(token):
.token(Token(token))
case .needMoreInput:
.needMoreInput
case .done:
.done
}
}
}
}
extension PureYAML.Streaming.Token {
init(_ token: PureYAML.Parsing.LibYAML.Token) {
self.init(
kind: Kind(token.type),
range: PureYAML.Model.SourceRange(
start: PureYAML.Model.SourcePosition(line: token.startMark.line, column: token.startMark.column),
end: PureYAML.Model.SourcePosition(line: token.endMark.line, column: token.endMark.column),
),
)
}
}
extension PureYAML.Streaming.Token.Kind {
/// A mechanical 1:1 mapping over all 22 `TokenType` cases, split across
/// three initializers purely to keep each switch's cyclomatic complexity
/// under the project's lint limit; the split carries no meaning beyond
/// that; each `default:` either forwards to the next initializer (the
/// cases it doesn't itself handle) or, in the last one, is truly
/// unreachable because the first two together already handle every
/// non-payload-bearing case.
init(_ type: PureYAML.Parsing.LibYAML.TokenType) {
switch type {
case .streamStart: self = .streamStart
case .streamEnd: self = .streamEnd
case .documentStart: self = .documentStart
case .documentEnd: self = .documentEnd
case .blockSequenceStart: self = .blockSequenceStart
case .blockMappingStart: self = .blockMappingStart
case .blockEnd: self = .blockEnd
case .blockEntry: self = .blockEntry
default: self = Self(structural: type)
}
}
private init(structural type: PureYAML.Parsing.LibYAML.TokenType) {
switch type {
case .flowSequenceStart: self = .flowSequenceStart
case .flowMappingStart: self = .flowMappingStart
case .flowSequenceEnd: self = .flowSequenceEnd
case .flowMappingEnd: self = .flowMappingEnd
case .flowEntry: self = .flowEntry
case .key: self = .key
case .value: self = .value
case .reservedDirective: self = .reservedDirective
default: self = Self(payload: type)
}
}
private init(payload type: PureYAML.Parsing.LibYAML.TokenType) {
switch type {
case let .versionDirective(major, minor): self = .versionDirective(major: major, minor: minor)
case let .tagDirective(handle, prefix): self = .tagDirective(handle: handle, prefix: prefix)
case let .anchor(name): self = .anchor(name: name)
case let .alias(name): self = .alias(name: name)
case let .tag(handle, suffix): self = .tag(handle: handle, suffix: suffix)
case let .scalar(value, style): self = .scalar(value: value, style: PureYAML.Streaming.Token.ScalarStyle(style))
default: preconditionFailure("unreachable: init(_:) and init(structural:) handle every non-payload TokenType case")
}
}
}
extension PureYAML.Streaming.Token.ScalarStyle {
init(_ style: PureYAML.Parsing.ScalarStyle) {
switch style {
case .plain: self = .plain
case .singleQuoted: self = .singleQuoted
case .doubleQuoted: self = .doubleQuoted
case .literal: self = .literal
case .folded: self = .folded
}
}
}

29
Sources/Streaming/Streaming.Token.Kind.swift Normal file
View File

@ -0,0 +1,29 @@
public extension PureYAML.Streaming.Token {
/// A scanner token's kind, mirroring the internal libyaml-ported scanner's
/// own token kinds (`yaml_token_type_t`) one for one, under public naming
/// that does not leak the port's internal type names.
enum Kind: Equatable, Sendable {
case streamStart
case streamEnd
case documentStart
case documentEnd
case blockSequenceStart
case blockMappingStart
case blockEnd
case blockEntry
case flowSequenceStart
case flowMappingStart
case flowSequenceEnd
case flowMappingEnd
case flowEntry
case key
case value
case versionDirective(major: Int, minor: Int)
case tagDirective(handle: String, prefix: String)
case reservedDirective
case anchor(name: String)
case alias(name: String)
case tag(handle: String, suffix: String)
case scalar(value: String, style: PureYAML.Streaming.Token.ScalarStyle)
}
}

12
Sources/Streaming/Streaming.Token.ScalarStyle.swift Normal file
View File

@ -0,0 +1,12 @@
public extension PureYAML.Streaming.Token {
/// The quoting or block style a scalar token was scanned with. Distinct
/// from ``PureYAML/Emitting/ScalarStyle``, which is an output rendering
/// policy, not the style a scanned scalar was actually written in.
enum ScalarStyle: Equatable, Sendable {
case plain
case singleQuoted
case doubleQuoted
case literal
case folded
}
}

14
Sources/Streaming/Streaming.Token.swift Normal file
View File

@ -0,0 +1,14 @@
public extension PureYAML.Streaming {
/// A scanner token: a kind plus the source span it covers. Produced by
/// ``Scanner/next()`` one at a time, in the same order ``PureYAML/parse(_:)``
/// would see them internally.
struct Token: Equatable, Sendable {
public var kind: Kind
public var range: PureYAML.Model.SourceRange
public init(kind: Kind, range: PureYAML.Model.SourceRange) {
self.kind = kind
self.range = range
}
}
}

113
Tests/PublicStreamingScannerTests.swift Normal file
View File

@ -0,0 +1,113 @@
import Foundation
@testable import PureYAML
import Testing
/// Differential proof for the public `PureYAML.Streaming.Scanner`, the thin
/// translation wrapper around the internal `LibYAML.Scanner.scanStep()`
/// already proven by `StreamingScannerDifferentialTests`. That existing proof
/// covers `scanStep()` itself; this one covers the translation layer on top
/// of it (`Streaming.Token`/`Streaming.Token.Kind`/`Streaming.Token.ScalarStyle`
/// mapped from the internal `LibYAML.Token`/`TokenType`/`ScalarStyle`), since a
/// wrong case mapping or a dropped field would not be caught by a proof that
/// never looks at the public types at all.
@Suite("Public streaming scanner")
struct PublicStreamingScannerTests {
@Test("Public Scanner matches the internal scanAll() reference, translated, at several chunk sizes", arguments: [
"title: Example API",
"""
title: Example
servers:
- url: /
""",
"flow: {a: 1, b: [2, 3]}",
"""
base: &anchor
value: 1
derived: *anchor
""",
"block: |\n line one\n line two\n",
"&a a: b\nc: &d d\n",
])
func test_publicScannerMatchesTranslatedReference(yaml: String) throws {
try assertPublicScannerMatchesReference(yaml)
}
@Test("Public Scanner matches the internal scanAll() reference over the whole yaml-test-suite corpus")
func test_publicScannerMatchesReferenceOverWholeSuite() throws {
guard ProcessInfo.processInfo.environment["PUREYAML_RUN_YAML_TEST_SUITE"] == "1" else {
return
}
let manifest = try YAMLTestSuiteManifest.load()
var mismatches: [String] = []
for testCase in manifest.cases {
let yaml = try testCase.loadInput()
do {
try assertPublicScannerMatchesReference(yaml)
} catch {
mismatches.append("\(testCase.id): \(error)")
}
}
#expect(mismatches.isEmpty, "public/internal mismatches (\(mismatches.count)): \(mismatches.prefix(10).joined(separator: "; "))")
}
}
private func assertPublicScannerMatchesReference(_ yaml: String) throws {
let reference = translatedBatchReference(yaml)
for chunkSize in [1, 2, 3, 7, 64, yaml.unicodeScalars.count + 1] {
let observed = try publicScan(yaml, chunkSize: chunkSize)
#expect(observed == reference, "chunk size \(chunkSize) diverged from the translated reference for: \(yaml.debugDescription)")
}
}
private enum PublicScanResult: Equatable {
case tokens([PureYAML.Streaming.Token])
case threw(String)
}
/// The reference: the batch scanner's own tokens, run through the exact same
/// `Streaming.Token.init(_:)` translator the public `Scanner` uses, so this
/// proof isolates "does the public wrapper preserve what the internal scanner
/// already produces" from "is the internal scanner itself correct" (the
/// latter is `StreamingScannerDifferentialTests`'s job, not this one's).
private func translatedBatchReference(_ yaml: String) -> PublicScanResult {
var scanner = PureYAML.Parsing.LibYAML.Scanner(yaml)
do {
return try .tokens(scanner.scanAll().map(PureYAML.Streaming.Token.init))
} catch {
return .threw("\(error)")
}
}
private func publicScan(_ yaml: String, chunkSize: Int) throws -> PublicScanResult {
let scalars = Array(yaml.unicodeScalars)
var chunks: [String] = []
var index = 0
while index < scalars.count {
let end = min(index + chunkSize, scalars.count)
chunks.append(String(String.UnicodeScalarView(scalars[index ..< end])))
index = end
}
var scanner = PureYAML.Streaming.Scanner()
var nextChunk = 0
var tokens: [PureYAML.Streaming.Token] = []
do {
while true {
switch try scanner.next() {
case let .token(token):
tokens.append(token)
case .done:
return .tokens(tokens)
case .needMoreInput:
if nextChunk < chunks.count {
scanner.feed(chunks[nextChunk])
nextChunk += 1
} else {
scanner.finish()
}
}
}
} catch {
return .threw("\(error)")
}
}

1
Tests/StreamingScannerDifferentialTests.swift
View File

@ -33,6 +33,7 @@ struct StreamingScannerDifferentialTests {
""",
"block: |\n line one\n line two\n",
"title: \"unterminated",
"&a a: b\nc: &d d\n",
])
func test_representativeCasesMatchScanAllAtSeveralChunkSizes(yaml: String) throws {
try assertStreamingMatchesBatch(yaml)

105
Tests/StreamingScannerPerformanceTests.swift
View File

@ -4,15 +4,17 @@ import Testing
/// Measures the resumable scanner's checkpoint-and-retry cost (issue #4) against
/// the batch scanner, across a spread of chunk sizes, over the largest real-world
/// fixture, plus a `singleChunk` control that isolates the floor from real
/// rescanning (see CHANGELOG.md). The overhead splits into two independent
/// costs: a fixed floor (~1.75x release-mode, present even with a single chunk,
/// caused by copying the whole `Scanner` on every fetch whenever `isFinal` is
/// false) and a rescan-driven tail that only appears below roughly 4096-scalar
/// chunks. Persisted mid-token suspension, the deferred next increment, would
/// remove the rescan tail but not the floor; a fix for the floor is not yet
/// designed. Gated: release-mode timing on a 9.5MB fixture is not a default
/// `swift test` concern.
/// fixture, plus controls that isolate the floor from real rescanning and from
/// buffer-population strategy (see CHANGELOG.md). The overhead splits into two
/// independent costs: a fixed floor (~1.75x release-mode, present even with a
/// single chunk, structurally attributable to the checkpoint taken on every
/// fetch whenever `isFinal` is false, confirmed by the `isFinal: true` control
/// costing 1.04x with the identical per-token loop) and a rescan-driven tail
/// that only appears below roughly 4096-scalar chunks. Persisted mid-token
/// suspension, the deferred next increment, would remove both: it eliminates
/// the need for a checkpoint at all, not just the rescanning a checkpoint's
/// retry causes. Gated: release-mode timing on a 9.5MB fixture is not a
/// default `swift test` concern.
///
/// Run this in release mode; debug-mode numbers are not representative (Swift's
/// `-Onone` inserts defensive retain/copy traffic around `mutating throws`
@ -83,6 +85,91 @@ struct StreamingScannerPerformanceTests {
let singleChunkDuration = try streamingDuration(yaml: yaml, chunkSize: wholeFileAsOneChunk, clock: clock)
let singleChunkRatio = singleChunkDuration / batchDuration
print("STREAMING-PERF singleChunk(wholeFile) time=\(singleChunkDuration) ratio=\(String(format: "%.2f", singleChunkRatio))x")
// Isolates whether the floor is scanning-related at all, or an artifact of
// how the buffer is populated: `Reader.init` builds `scalars` in one shot
// via `Array(string.unicodeScalars)`; `Reader.append(_:)` (used by every
// streaming chunk feed, including the single-chunk case above) instead
// grows an existing array via `append(contentsOf:)`. No scanning happens
// here at all, just buffer construction.
let constructDuration = clock.measure {
_ = PureYAML.Parsing.LibYAML.Scanner(yaml)
}
var appendScanner = PureYAML.Parsing.LibYAML.Scanner("", isFinal: false)
let appendDuration = clock.measure {
appendScanner.reader.append(yaml)
}
print("STREAMING-PERF bufferConstruct time=\(constructDuration) bufferAppend time=\(appendDuration) ratio=\(String(format: "%.2f", appendDuration / constructDuration))x")
}
/// The `github-rest-api.yaml` fixture `test_measure` uses above has only
/// short plain scalars, so it cannot exercise mid-token suspension's
/// actual purpose: resuming a single plain scalar that itself spans many
/// chunks. This measures exactly that shape, on one synthetic 1MB plain
/// scalar. It is what caught a real O(N^2) bug in the first version of
/// `resumePlainScalar`/`scanStep()`'s resume branch: passing the
/// suspended `PlainState` across the `resumePlainScalar` call boundary
/// as a value parameter (even one immediately copied and overwritten
/// inside the callee) left the caller's own copy of `state.string`'s
/// buffer aliased to the callee's for the call's whole duration, so
/// copy-on-write copied the entire accumulated scalar on every resume;
/// at chunk size 16 that was a measured 29x, worse than not suspending
/// at all. Passing `state` directly `inout`, with the disalias done at
/// the call site before the call rather than inside the callee, fixed it
/// to 3-4x. See `resumePlainScalar` and `scanStep()`'s resume branch for
/// the fix itself.
@Test("measure scanStep() overhead on one plain scalar spanning many chunks")
func test_measureLongPlainScalar() throws {
guard ProcessInfo.processInfo.environment["PUREYAML_STREAMING_PERF_MEASURE"] == "1" else {
return
}
let longWord = String(repeating: "abcdefghij", count: 100_000)
let yaml = "key: \(longWord)\n"
let clock = ContinuousClock()
var batchScanner = PureYAML.Parsing.LibYAML.Scanner(yaml)
let batchDuration = try clock.measure {
_ = try batchScanner.scanAll()
}
print("LONGSCALAR-PERF batch bytes=\(yaml.utf8.count) time=\(batchDuration)")
for chunkSize in [16, 64, 256, 4096] {
let duration = try streamingDuration(yaml: yaml, chunkSize: chunkSize, clock: clock)
let ratio = duration / batchDuration
print("LONGSCALAR-PERF chunkSize=\(chunkSize) time=\(duration) ratio=\(String(format: "%.2f", ratio))x")
}
}
/// A regression guard for the O(N^2) aliasing bug `test_measureLongPlainScalar`'s
/// doc comment describes, not just an observational print like the
/// measurements above: that bug's signature is per-byte cost growing with
/// the scalar's total length, since each resume re-copied everything
/// accumulated so far, not just the new chunk. Genuinely O(N) work
/// (copying only each chunk's new characters, once) keeps per-byte cost
/// flat as length grows; measured flat within 8% across a 16x range
/// (250KB to 4MB) after the fix landed. `smallestPerByte` versus
/// `largestPerByte` is asserted at a generous 3x, not the ~1.08x actually
/// measured: this must catch a real quadratic regression, not fail on
/// ordinary run-to-run noise on a shared CI runner.
@Test("scanStep() cost on a long plain scalar stays linear as the scalar's length grows")
func test_longPlainScalarStaysLinear() throws {
guard ProcessInfo.processInfo.environment["PUREYAML_STREAMING_PERF_MEASURE"] == "1" else {
return
}
let clock = ContinuousClock()
var perByteCosts: [Double] = []
for wordCount in [25000, 400_000] {
let longWord = String(repeating: "abcdefghij", count: wordCount)
let yaml = "key: \(longWord)\n"
let duration = try streamingDuration(yaml: yaml, chunkSize: 16, clock: clock)
let seconds = Double(duration.components.seconds) + Double(duration.components.attoseconds) / 1e18
let perByte = seconds / Double(yaml.utf8.count)
perByteCosts.append(perByte)
print("LINEAR-CHECK bytes=\(yaml.utf8.count) time=\(duration) perByte=\(perByte)")
}
let smallestPerByte = perByteCosts[0]
let largestPerByte = perByteCosts[1]
#expect(largestPerByte < smallestPerByte * 3)
}
}