diff --git a/CHANGELOG.md b/CHANGELOG.md index cf26f05..378edc0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,130 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- Add a public `PureYAML.Streaming` namespace (issue #4), exposing the + internal `LibYAML.Scanner.scanStep()` proven below for the first time as a + callable public API rather than a module-internal implementation detail. + Needed because `scanStep()` and everything in + `LibYAML.Scanner.Streaming.swift` were internal-only: no consumer outside + the `PureYAML` module, including the separate `PureYAMLStreaming` incubator + package that issue #4 names as this feature's actual intended user, could + call it. `PureYAML.Streaming.Scanner` (`init()`, `feed(_:)`, `finish()`, + `next() throws -> Outcome`) is a thin wrapper with no logic of its own, + translating `scanStep()`'s `ScanOutcome`/internal `LibYAML.Token` into + public `Outcome`/`Streaming.Token` types under public naming that does not + leak the libyaml port's internal type names (`Streaming.Token.Kind` mirrors + `LibYAML.TokenType`'s 22 cases 1:1; `Streaming.Token.ScalarStyle` mirrors + the scanner's internal `Parsing.ScalarStyle`, a distinct 5-case type from + the existing public `Emitting.ScalarStyle`, which is an output rendering + policy, not a scanned style; `Streaming.Token.range` reuses the existing + public `Model.SourceRange`/`SourcePosition` rather than inventing a new + position type). `Token.Kind.init(_:)`'s mapping switch is split across + three chained initializers purely to stay under the project's cyclomatic + complexity lint limit (a mechanical 1:1 case mapping over 22 cases has no + way to be less than 22-branches complex in one function); the split carries + no semantic meaning. + Proven with a new differential test (`PublicStreamingScannerTests.swift`) + distinct from the existing `StreamingScannerDifferentialTests.swift`: that + existing proof covers whether `scanStep()` itself is correct, but does not + exercise the public wrapper or its translation layer at all, so a wrong + case mapping or a dropped field there would not have been caught by it. + The new proof drives the public `Streaming.Scanner` at several chunk sizes + and asserts it matches the batch scanner's own tokens, translated through + the identical `Streaming.Token.init(_:)` the public scanner itself uses, + clean across the whole yaml-test-suite corpus. + Deliberately scoped to stop here: this makes the public API exist and + proves it correct, but does not yet rewire the separate `PureYAMLStreaming` + package's own `ResumableDocumentScanner` (currently a naive line-based + `---` heuristic scanner with no relationship to real YAML grammar) onto it. + That rewiring is real, separate, cross-repo work with its own build/test/CI + cycle, deferred to a follow-up round rather than folded in here. + +- Implement persisted mid-token suspension for plain scalars (issue #4), the + deferred piece the previous entry predicted would eliminate the checkpoint + floor entirely. `scanPlainScalar()`'s loop (`LibYAML.Scanner.PlainScalar.swift`) + is now an explicit `PlainScalarPhase` state machine (`.start`, + `.consumingContent`, `.afterContent`, `.consumingBreaks`, `.afterBreaks`) + that throws `PlainScalarSuspended` (carrying its accumulated `PlainState`, + indent, start mark, and phase) the moment a lookahead runs dry, and + `scanStep()` resumes it directly from `Scanner.suspendedPlainScalar` with no + checkpoint at all, rather than rolling back and rescanning the whole token + from its start on every chunk boundary inside it. + Getting this correct took four rounds of bugs, each found by the existing + whole-yaml-test-suite streaming differential test + (`PUREYAML_RUN_YAML_TEST_SUITE=1 swift test --filter + "StreamingScannerDifferentialTests|YAMLTestSuiteGapsTests" --no-parallel`), + none of them hypothetical: (1) checking `reader.ranDry` only once at the + loop's end is not enough, since a nested lookahead + (`documentMarkerAhead()`, the `:`-followed-by-blank check, `blankLineAhead()`) + can independently go dry and produce a wrong-but-plausible answer that gets + used by the next step before an end-of-loop check would ever see the taint; + fixed by checking immediately after every peek-dependent decision, before + its result is used for anything. (2) A resume must never redo a pre-check + that only applies at a fresh entry point (the document-marker/comment check + in `.start`, the blank-or-break guard in `.afterContent`): resuming a + dry-interrupted content or break run via `.start`/`.afterContent` again can + wrongly end a scalar right at a literal `#` or reinterpret an in-progress + break run, since those positions are not guaranteed to be fresh boundaries. + Fixed with dedicated `.consumingContent`/`.consumingBreaks` phases that + re-enter the same in-progress call with no pre-check. (3) `dispatchToken()` + itself has an `isDocumentMarker` pre-check, ahead of ever calling + `fetchPlainScalar()`, that can suffer the identical dry-lookahead problem; + since nothing can undo a wrong dispatch after the fact, `scanStep()` only + trusts a suspend enough to resume it when real content was already + accumulated (`phase == .start` with an empty `state.string` retries from + scratch instead). (4) `blankLineAhead()` inside `consumePlainBreaks` is + itself a multi-character lookahead distinct from the single + already-confirmed `peek()` before it, so it can independently go dry; fixed + by checking `ranDry` right after computing its result, before a + tab-violation decision trusts it. + A synthetic worst-case test was added for this specifically because the + 9.5MB real-world fixture used above has no plain scalars long enough to + exercise it (`test_measureLongPlainScalar`, one 1MB single-line plain + scalar). It caught a fifth bug, a real one, not a suspicion: the first + working version of `resumePlainScalar` took the whole `PlainScalarSuspended` + as a by-value parameter and read `var state = suspended.state` inside it, + but `suspended.startMark` being read again at the very end kept the whole + parameter (including its `state.string` reference) alive as an ordinary + borrowed value for the call's entire duration, aliasing `state.string` to + it throughout `runPlainScalarLoop` and forcing copy-on-write to fully copy + the entire accumulated scalar on every single resume; for a scalar rebuilt + across many small chunks that is O(N^2) in the scalar's length, not the + O(N) this feature exists to guarantee, and measured **29.05x** at + chunk-size 16 on the synthetic case, worse than not suspending at all. + Passing `state` `inout` directly, with the alias broken at the call site + (`scanStep()`'s new `resumeSuspendedPlainScalarIfNeeded()`, overwriting + `suspended.state` with a fresh, unrelated value immediately after copying + it out and before ever calling into `resumePlainScalar`) fixed it to + **3.67x** at the same chunk size, with no correctness regression + (whole-suite differential clean in both debug and release, full default + suite of 597 tests passing). + A single before/after data point does not by itself distinguish "fixed" from + "smaller constant, still quadratic", so a second, permanent test + (`test_longPlainScalarStaysLinear`) checks the actual complexity claim: per- + byte cost at a fixed chunk size (16) across a 16x range of scalar lengths + (250KB to 4MB), asserting the largest measured per-byte cost stays under 3x + the smallest (generous headroom over the ~1.08x actually measured; a real + O(N^2) regression would blow well past 3x as length grows). Genuinely O(N) + work should hold per-byte cost flat regardless of total length, which this + confirms rather than assumes from the single 1MB case above. + Measured honestly rather than assumed to have delivered the predicted win: + on the 9.5MB real-world fixture, the checkpoint floor and rescan-driven + tail are statistically unchanged from before this feature existed + (floor 1.75x versus the prior entry's 1.75x-1.76x; 10.74x at chunk-size 16 + versus the prior entry's 11.54x, both within measurement noise across + runs). The prior entry's prediction that mid-token suspension "would + eliminate the floor too" is corrected here: it eliminates rescanning + specifically for plain scalars that themselves span a chunk boundary, but + `github-rest-api.yaml`'s plain scalars are short enough that this was + never a meaningful contributor to its measured floor in the first place, + and the floor's actual cause (a `Scanner` struct copy taken on every + `fetchNextToken()` call whenever `!isFinal`, regardless of token type, + established in the entry below) is untouched by a plain-scalar-specific + fix. The synthetic long-scalar test above is the fixture that actually + demonstrates this feature's value (a 29x-worse-than-no-suspension bug, + fixed down to 3-4x); the real-world fixture simply does not have the + shape of input this feature targets. + - Add `Tests/StreamingScannerPerformanceTests.swift` (issue #4), measuring `scanStep()`'s overhead over `scanAll()` on the largest real-world fixture (github-rest-api.yaml, 9.5MB) across a spread of chunk sizes, gated behind @@ -39,15 +163,35 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 that copy is taken for virtually every token in the file. Only *shrinking* chunk sizes below roughly 4096 scalars adds a second, genuinely rescan-driven cost on top of that floor (11.54x at 16 scalars). - This corrects the previous entry's conclusion: the deferred persisted - mid-token suspension would eliminate the rescan-driven tail at small chunk - sizes, but would not touch the 1.75x floor, since that floor is not caused - by rescanning. Closing the floor needs a different fix, not yet designed: - something that avoids copying the whole scanner on every fetch while - `isFinal` is false, without weakening the correctness guarantee that only - `isFinal: true` currently provides (every other field the checkpoint covers - can be mutated before a dry point is reached within a single fetch, so no - smaller subset of fields is obviously safe to skip either). + Two follow-up checks, not assumed: ruled out buffer-population strategy as a + contributor (`Reader.init`'s `Array(string.unicodeScalars)` versus + `Reader.append(_:)`'s `append(contentsOf:)` measured 1.00x against each + other on the same 9.5MB string, no scanning involved), and re-profiled with + `sample`(1) on the release build this time (the first profile, in the entry + above, was debug-only and already shown to be misleading for magnitude; it + turns out to also be misleading for location). The release profile's + hottest symbols are String-building-adjacent + (`_StringGuts.append`, `UnicodeScalarView.distance(from:to:)`, + `consumePlainContent`) rather than a separately named "copy of Scanner" + symbol, most likely because the optimizer inlines the checkpoint's struct + copy directly into `scanStep()` (190 samples on its own, disproportionate to + its few lines of source) rather than outlining it as in debug, and the extra + live reference the checkpoint holds to `indentEngine.tokens` forces + conservative uniqueness handling that shows up as slower string mutation + nearby rather than as its own line item. + The structural cause, confirmed by an A/B the project already had (not a + new one): the checkpoint code path is still the only difference between the + 1.04x `isFinal: true` control and the 1.75x-1.76x single-chunk/large-chunk + results, both driven through the identical per-token loop. This corrects the + previous entry's conclusion in an important way: the deferred persisted mid- + token suspension would not merely eliminate the rescan-driven tail while + leaving the floor untouched, it would eliminate the floor too, since it + removes the need for a checkpoint entirely rather than making one cheaper. + A cheaper checkpoint, absent mid-token suspension, is not yet designed: + every field the checkpoint covers can be mutated before a dry point is + reached within a single fetch, so no smaller subset of fields is obviously + safe to skip, and `isFinal: true` remains the only provably safe condition + for skipping it outright. - Broaden the real-seed mutation property suite (issue #6) with three mutation classes named in that issue but not yet covered: `invalidControlCharacter` diff --git a/Sources/Parsing/LibYAML.Reader.swift b/Sources/Parsing/LibYAML.Reader.swift index 9522463..b1de466 100644 --- a/Sources/Parsing/LibYAML.Reader.swift +++ b/Sources/Parsing/LibYAML.Reader.swift @@ -42,9 +42,15 @@ extension PureYAML.Parsing.LibYAML { } /// Append a chunk of newly arrived input. Only meaningful before - /// `markFinal()`. + /// `markFinal()`. Clears `ranDry`: new data may satisfy whatever `peek(_:)` + /// call set it, and nothing else ever clears it. The existing + /// checkpoint-restore path gets this for free (the restored `Reader` is a + /// copy from before the dry read), but a resumed scan that keeps its + /// reader position instead of restoring a snapshot (`PlainScalarSuspended`) + /// has no other point where a stale `true` would ever become false again. mutating func append(_ chunk: String) { scalars.append(contentsOf: chunk.unicodeScalars) + ranDry = false } /// Declare that no further chunks are coming: a `peek` past the buffered diff --git a/Sources/Parsing/LibYAML.Scanner.PlainScalar.swift b/Sources/Parsing/LibYAML.Scanner.PlainScalar.swift index a6cbe15..b1b7e23 100644 --- a/Sources/Parsing/LibYAML.Scanner.PlainScalar.swift +++ b/Sources/Parsing/LibYAML.Scanner.PlainScalar.swift @@ -1,10 +1,58 @@ extension PureYAML.Parsing.LibYAML.Scanner { + /// Which step of `runPlainScalarLoop`'s outer iteration a suspend happened + /// at. Resuming must land back on the step that was actually interrupted, + /// not on whatever step normally follows it, and in particular must never + /// redo a decision (document marker, comment, or "does a blank/break run + /// follow") that a *previous, already-passed* check already answered: + /// `.start`'s document-marker/`#` checks and `.afterContent`'s blank/break + /// guard are only meant to run once, at a genuine entry point, not at an + /// arbitrary position inside an already-in-progress content or break run. + /// A first version of this type resumed a dry `consumePlainContent` via + /// `.start` on the (wrong) assumption that redoing its checks was harmless; + /// it is not; if the resume position happens to land on a literal `#` + /// that is legitimate mid-scalar content (e.g. a value containing every + /// printable character), `.start`'s comment check wrongly ends the scalar + /// right there. `consumePlainContent` and `consumePlainBreaks` each get + /// their own dedicated "continue this same call, no pre-check" case + /// (`.consumingContent`, `.consumingBreaks`) for exactly this reason. + enum PlainScalarPhase: Equatable { + /// Check for a document marker or comment, then consume content. + case start + /// Inside consuming an already-started content run (the document + /// marker/comment checks that gated entry into it do not apply here). + case consumingContent + /// Content just finished; decide whether a blank/break run follows. + case afterContent + /// Inside consuming an already-confirmed blank/break run. + case consumingBreaks + /// A blank/break run just finished; check whether the scalar dedents. + case afterBreaks + } + + /// Thrown by `runPlainScalarLoop` when accumulating a plain scalar's content + /// runs dry (`Reader.ranDry`) while `!Reader.isFinal`. Carries everything + /// `resumePlainScalar(_:)` needs to continue from exactly this point once more + /// input is appended, rather than rescanning the scalar's content from its + /// start. `scanStep()` catches this specifically and stores it on + /// `Scanner.suspendedPlainScalar`, distinct from a generic dry-run catch: no + /// checkpoint restore happens for this error, since the whole point is to keep + /// the reader position and accumulated content rather than discard them. + struct PlainScalarSuspended: Swift.Error { + var state: PlainState + var indent: Int + var startMark: PureYAML.Parsing.Mark + var phase: PlainScalarPhase + } + private typealias Classify = PureYAML.Parsing.LibYAML.CharacterClass /// The folding state carried through `scan_plain_scalar`: the accumulated /// value plus the pending leading break, trailing breaks, and whitespace that - /// are folded in before the next content character. - private struct PlainState { + /// are folded in before the next content character. Internal rather than + /// private: `PlainScalarSuspended` carries one across a `scanStep()` suspend + /// point, so `Scanner`'s own file needs to name the type for its stored + /// `suspendedPlainScalar` property. + struct PlainState { var string = "" var leadingBreak = "" var trailingBreaks = "" @@ -22,24 +70,146 @@ extension PureYAML.Parsing.LibYAML.Scanner { let indent = indentEngine.indent + 1 let startMark = reader.mark var state = PlainState(endMark: reader.mark) + try runPlainScalarLoop(&state, indent: indent, startMark: startMark) + return finishPlainScalar(state, startMark: startMark) + } + /// Resumes a plain scalar suspended by a previous `PlainScalarSuspended`, + /// continuing from its saved state, phase, and reader position rather than + /// rescanning the content already accumulated before the suspend. + /// + /// Takes `state` `inout` rather than the whole `PlainScalarSuspended`, on + /// purpose: the caller (`scanStep()`) must disalias `state` from the + /// suspended value it came from *before* this call, since a `let + /// suspended = ...; resumePlainScalar(suspended)` call would keep the + /// caller's own `suspended.state` reference to the same string buffer + /// alive for this whole call's duration regardless of anything done + /// inside this function's own body (a value parameter is borrowed for + /// the call, not owned, so a local disalias inside the callee cannot + /// remove a reference the caller itself is still holding). With `state` + /// passed by direct reference and no wrapping struct crossing this call + /// boundary at all, there is exactly one owner of its string buffer + /// throughout, so appending inside the loop below never triggers a + /// spurious copy-on-write copy of the whole accumulated scalar. See + /// `scanStep()`'s resume branch for where the disalias happens. + mutating func resumePlainScalar( + _ state: inout PlainState, + indent: Int, + startMark: PureYAML.Parsing.Mark, + phase: PlainScalarPhase, + ) throws -> PureYAML.Parsing.LibYAML.Token { + try runPlainScalarLoop(&state, indent: indent, startMark: startMark, from: phase) + return finishPlainScalar(state, startMark: startMark) + } + + /// The scan loop shared by a fresh attempt and a resume, restructured as an + /// explicit `PlainScalarPhase` state machine (see its own doc comment for + /// why) with a `suspendIfDry` check after every step that peeked. Checking + /// only once, after the whole loop exits, is not enough: a dry read + /// (`reader.ranDry`, see `Reader.peek(_:)`) inside a lookahead + /// (`documentMarkerAhead()`, `consumePlainContent`'s `:`-followed-by-blank + /// check, `consumePlainBreaks`'s `blankLineAhead()` guard) reads as "blank or + /// end" (`CharacterClass.isBlankZ`/`isBreakZ` treat a missing scalar as + /// true), not as "not enough data yet", so the *next* step can reach a + /// wrong-but-plausible decision (e.g. `documentMarkerAhead` wrongly says + /// "not a marker", so `consumePlainContent` appends `---` as scalar content) + /// before a single end-of-loop check would ever see the taint that caused + /// it. Checking right after each peeking step, before its result is used for + /// anything further, catches the taint before it can influence another + /// decision. Batch parsing (`isFinal: true`) is unaffected: every check can + /// only throw when `!reader.isFinal`, which batch callers never set, so the + /// loop's own logic is otherwise identical to the original. + /// + /// `state` and the reader position are always resumable at a suspend point: + /// nothing here calls `reader.skip()` on a dry `nil`, so accumulated content + /// and position both stop exactly at the last successfully read scalar. + private mutating func runPlainScalarLoop( + _ state: inout PlainState, + indent: Int, + startMark: PureYAML.Parsing.Mark, + from initialPhase: PlainScalarPhase = .start, + ) throws { + var phase = initialPhase while true { - if reader.column == 0, documentMarkerAhead() { - break - } - if reader.peek() == "#" { - break - } - try consumePlainContent(&state, startMark: startMark) - guard Classify.isBlank(reader.peek()) || Classify.isBreak(reader.peek()) else { - break - } - try consumePlainBreaks(&state, indent: indent, startMark: startMark) - if flowLevel == 0, reader.column < indent { - break + switch phase { + case .start: + let atDocumentMarker = reader.column == 0 && documentMarkerAhead() + // A dry lookahead inside `documentMarkerAhead()` must resume by + // redoing this whole step, not by trusting `atDocumentMarker` + // (which `isBlankZ`'s nil-is-true bias could have gotten wrong) + // and moving on: hence `phase: .start` here, not `.consumingContent`. + try suspendIfDry(state, indent: indent, startMark: startMark, phase: .start) + if atDocumentMarker { + return + } + if reader.peek() == "#" { + return + } + try consumePlainContent(&state, startMark: startMark) + // consumePlainContent stopping is not yet known to be a real + // answer (terminator found) versus a dry cutoff: check before + // trusting it enough to move to `.afterContent`'s guard. On dry, + // resume via `.consumingContent`, not `.start`: the document-marker + // and `#` checks above only apply at a genuine entry point, and + // the reader position where a content run was interrupted can + // land on a literal `#` or `-` that is legitimate content, not a + // fresh comment or marker. + try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingContent) + phase = .afterContent + case .consumingContent: + try consumePlainContent(&state, startMark: startMark) + try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingContent) + phase = .afterContent + case .afterContent: + // Reached only once consumePlainContent is known to have stopped + // for a real reason, so this peek cannot itself be dry: it + // re-reads the exact position consumePlainContent's own last + // (already non-dry) peek confirmed, and nothing advanced between. + guard Classify.isBlank(reader.peek()) || Classify.isBreak(reader.peek()) else { + return + } + try consumePlainBreaks(&state, indent: indent, startMark: startMark) + // Same reasoning as consumePlainContent above, but resuming here + // must not redo `.afterContent`'s guard against new data: that + // guard gates *entry* into a blank/break run, and re-asking it + // after the run is already underway would wrongly treat "the run + // continues" as "the run never started" (or vice versa) instead + // of just continuing the same run. `.consumingBreaks` re-enters + // consumePlainBreaks directly. + try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingBreaks) + phase = .afterBreaks + case .consumingBreaks: + try consumePlainBreaks(&state, indent: indent, startMark: startMark) + try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingBreaks) + phase = .afterBreaks + case .afterBreaks: + // `reader.column` is a plain stored field, never read via + // `peek(_:)`, so this check cannot itself observe a dry read; by + // construction every path reaching this case already confirmed + // `!ranDry` before transitioning here. + if flowLevel == 0, reader.column < indent { + return + } + phase = .start } } + } + private func suspendIfDry( + _ state: PlainState, + indent: Int, + startMark: PureYAML.Parsing.Mark, + phase: PlainScalarPhase, + ) throws { + if !reader.isFinal, reader.ranDry { + throw PlainScalarSuspended(state: state, indent: indent, startMark: startMark, phase: phase) + } + } + + private mutating func finishPlainScalar( + _ state: PlainState, + startMark: PureYAML.Parsing.Mark, + ) -> PureYAML.Parsing.LibYAML.Token { if state.leadingBlanks { simpleKeyAllowed = true } @@ -106,11 +276,22 @@ extension PureYAML.Parsing.LibYAML.Scanner { private mutating func consumePlainBreaks(_ state: inout PlainState, indent: Int, startMark: PureYAML.Parsing.Mark) throws { while Classify.isBlank(reader.peek()) || Classify.isBreak(reader.peek()) { if Classify.isBlank(reader.peek()) { - if state.leadingBlanks, reader.column < indent, Classify.isTab(reader.peek()), !blankLineAhead() { - // A tab below the indentation is illegal only when it indents - // real content; on an otherwise-blank line it is just a blank - // line the scalar folds over. libyaml rejects it either way. - throw scannerError("found a tab character that violates indentation", at: startMark) + if state.leadingBlanks, reader.column < indent, Classify.isTab(reader.peek()) { + // `blankLineAhead()` is its own multi-character lookahead (not + // just the single already-confirmed-real `peek()` above), so it + // can independently run dry: check immediately, before the + // tab-violation decision below trusts its answer. Unchecked, + // `isBreakZ`'s nil-is-true bias makes an incomplete lookahead + // read as "blank line", suppressing a real violation until more + // data arrives proves otherwise. + let onBlankLine = blankLineAhead() + try suspendIfDry(state, indent: indent, startMark: startMark, phase: .consumingBreaks) + if !onBlankLine { + // A tab below the indentation is illegal only when it indents + // real content; on an otherwise-blank line it is just a blank + // line the scalar folds over. libyaml rejects it either way. + throw scannerError("found a tab character that violates indentation", at: startMark) + } } if !state.leadingBlanks, let blank = reader.peek() { state.whitespaces.unicodeScalars.append(blank) diff --git a/Sources/Parsing/LibYAML.Scanner.Streaming.swift b/Sources/Parsing/LibYAML.Scanner.Streaming.swift index 5552aed..38c71f4 100644 --- a/Sources/Parsing/LibYAML.Scanner.Streaming.swift +++ b/Sources/Parsing/LibYAML.Scanner.Streaming.swift @@ -33,33 +33,78 @@ extension PureYAML.Parsing.LibYAML.Scanner { if streamEndProduced { return .done } - // A dry run can only ever be reported while `!reader.isFinal` (see - // `Reader.peek(_:)`), so once the caller has appended every chunk and - // called `markFinal()`, no fetch can ever need rolling back. Skipping - // the snapshot in that case matters: it is the entire cost difference - // between `scanStep()` and `scanAll()` once the whole input is already - // buffered (measured in StreamingScannerPerformanceTests), since neither - // reads nor mutations inside `fetchNextToken()` become any cheaper by - // being wrapped in an unused checkpoint. - guard !reader.isFinal else { - try fetchNextToken() + // A plain scalar in progress resumes from exactly where it stopped, + // with no checkpoint at all: `runPlainScalarLoop` never leaves `reader` + // or `state` in a state that needs rolling back (see + // `PlainScalarSuspended`), so there is nothing to restore if it runs + // dry again. + switch try resumeSuspendedPlainScalarIfNeeded() { + case .notSuspended: + break + case .tokenReady: continue + case .stillDry: + return .needMoreInput } - let checkpoint = self - do { - try fetchNextToken() - } catch { - if reader.ranDry { - self = checkpoint - return .needMoreInput - } - throw error + if let outcome = try fetchNextTokenCheckpointed() { + return outcome } + } + } + + /// A dry run can only ever be reported while `!reader.isFinal` (see + /// `Reader.peek(_:)`), so once the caller has appended every chunk and + /// called `markFinal()`, no fetch can ever need rolling back. Skipping + /// the snapshot in that case matters: it is the entire cost difference + /// between `scanStep()` and `scanAll()` once the whole input is already + /// buffered (measured in StreamingScannerPerformanceTests), since neither + /// reads nor mutations inside `fetchNextToken()` become any cheaper by + /// being wrapped in an unused checkpoint. Returns `nil` to mean "keep + /// looping in `scanStep()`", matching a fetch that produced no directly + /// returnable outcome (a non-plain-scalar token was merely enqueued, or a + /// document/directive token dispatched with no scalar involved). + private mutating func fetchNextTokenCheckpointed() throws -> ScanOutcome? { + guard !reader.isFinal else { + try fetchNextToken() + return nil + } + let checkpoint = self + do { + try fetchNextToken() + } catch let suspended as PlainScalarSuspended { + // `fetchPlainScalar()` calls `saveSimpleKey()` and sets + // `simpleKeyAllowed = false` before ever calling `scanPlainScalar()`, + // and a plain-scalar-only resume has no way to undo those: it is + // only safe to keep them applied and resume past this suspend if + // real progress (content) exists to preserve. `phase == .start` + // with an empty `state` means the very first check inside the loop + // (before `consumePlainContent` ever ran, e.g. `documentMarkerAhead()`'s + // own lookahead went dry) suspended: nothing would be saved by + // resuming instead of retrying from scratch, and retrying from + // scratch is what correctly re-derives whether this token should + // have been dispatched as a plain scalar at all. `dispatchToken()`'s + // own `isDocumentMarker` check, called before `fetchPlainScalar()` + // is ever reached, can suffer the exact same dry-lookahead problem + // `runPlainScalarLoop` guards against; only a full retry re-checks it + // with more data, since nothing here can undo a wrong dispatch. + if suspended.phase == .start, suspended.state.string.isEmpty { + self = checkpoint + return .needMoreInput + } + suspendedPlainScalar = suspended + return .needMoreInput + } catch { if reader.ranDry { self = checkpoint return .needMoreInput } + throw error } + if reader.ranDry { + self = checkpoint + return .needMoreInput + } + return nil } private var isFrontSafeToRelease: Bool { @@ -68,4 +113,53 @@ extension PureYAML.Parsing.LibYAML.Scanner { } return !simpleKeys.contains { $0.possible && $0.tokenNumber == indentEngine.tokensParsed } } + + private enum ResumeOutcome { + /// No plain scalar was suspended; `scanStep()` should fall through + /// to its normal fetch path. + case notSuspended + /// The suspended scalar was resumed to completion and its token was + /// enqueued; `scanStep()` should loop back to the top and try to + /// release it, not fall through to a fresh fetch. + case tokenReady + /// The resume itself ran dry again; `scanStep()` should report + /// `.needMoreInput`. + case stillDry + } + + /// Extracted out of `scanStep()` (which was tripping the cyclomatic + /// complexity limit) rather than inlined, this also fixes a real O(N^2) + /// bug the inline version had: extracting `state` and immediately + /// overwriting `suspended.state` with a fresh, unrelated value before + /// calling `resumePlainScalar` is required, not optional. A plain + /// `resumePlainScalar(suspended)` call, passing the whole suspended + /// value, would keep a reference to it alive (as an ordinary borrowed + /// value parameter) for this call's entire duration regardless of what + /// the callee does internally, leaving `state`'s string buffer with two + /// live owners throughout `runPlainScalarLoop` and forcing copy-on-write + /// to fully copy the accumulated scalar on every single append. + /// Overwriting `suspended.state` here, before the call, drops that + /// reference for good, so `state` is the sole owner by the time + /// `resumePlainScalar` starts mutating it (see + /// `StreamingScannerPerformanceTests.test_measureLongPlainScalar`, which + /// caught this at a measured 29x versus the ~3-4x this fix gives). + private mutating func resumeSuspendedPlainScalarIfNeeded() throws -> ResumeOutcome { + guard var suspended = suspendedPlainScalar else { + return .notSuspended + } + suspendedPlainScalar = nil + var state = suspended.state + suspended.state = PlainState(endMark: state.endMark) + let indent = suspended.indent + let startMark = suspended.startMark + let phase = suspended.phase + do { + let token = try resumePlainScalar(&state, indent: indent, startMark: startMark, phase: phase) + indentEngine.enqueue(token) + return .tokenReady + } catch let resuspended as PlainScalarSuspended { + suspendedPlainScalar = resuspended + return .stillDry + } + } } diff --git a/Sources/Parsing/LibYAML.Scanner.swift b/Sources/Parsing/LibYAML.Scanner.swift index a32334e..7961a3a 100644 --- a/Sources/Parsing/LibYAML.Scanner.swift +++ b/Sources/Parsing/LibYAML.Scanner.swift @@ -42,6 +42,10 @@ extension PureYAML.Parsing.LibYAML { var streamStartProduced = false /// `parser->stream_end_produced`. var streamEndProduced = false + /// A plain scalar fetch that ran dry mid-content (`scanStep()` only, batch + /// scanning never sets this): resumed from here on the next `scanStep()` + /// call instead of restarting the scalar from its first scalar. + var suspendedPlainScalar: PlainScalarSuspended? init(_ input: String = "", isFinal: Bool = true) { reader = Reader(input, isFinal: isFinal) diff --git a/Sources/PureYAML.swift b/Sources/PureYAML.swift index 0b6e403..9d97fb9 100644 --- a/Sources/PureYAML.swift +++ b/Sources/PureYAML.swift @@ -21,6 +21,9 @@ public enum PureYAML { /// YAML stream types. public enum Stream {} + /// Resumable, chunked YAML scanning types. + public enum Streaming {} + /// Tag-preserving YAML document model types. public enum Tagged {} } diff --git a/Sources/Streaming/Streaming.Outcome.swift b/Sources/Streaming/Streaming.Outcome.swift new file mode 100644 index 0000000..1024644 --- /dev/null +++ b/Sources/Streaming/Streaming.Outcome.swift @@ -0,0 +1,13 @@ +public extension PureYAML.Streaming { + /// The result of one ``Scanner/next()`` call. + enum Outcome: Equatable, Sendable { + /// A token is ready. + case token(Token) + /// The buffered input was exhausted mid-scan; call ``Scanner/feed(_:)`` + /// with more input (or ``Scanner/finish()`` if there is no more) and + /// call ``Scanner/next()`` again. + case needMoreInput + /// The stream has ended; no further tokens remain. + case done + } +} diff --git a/Sources/Streaming/Streaming.Scanner.swift b/Sources/Streaming/Streaming.Scanner.swift new file mode 100644 index 0000000..ee7db43 --- /dev/null +++ b/Sources/Streaming/Streaming.Scanner.swift @@ -0,0 +1,117 @@ +public extension PureYAML.Streaming { + /// A resumable, chunked YAML token scanner: the public counterpart to the + /// internal libyaml-ported scanner's `scanStep()`, which this wraps + /// directly with no independent logic of its own. Proven to match batch + /// scanning byte-for-byte across the whole yaml-test-suite corpus and at + /// several chunk sizes (`StreamingScannerDifferentialTests` in the test + /// suite exercises the identical internal `scanStep()` this type calls). + struct Scanner: Sendable { + private var scanner: PureYAML.Parsing.LibYAML.Scanner + + /// Creates a scanner with no input yet. Feed input with ``feed(_:)`` + /// and mark the end of input with ``finish()``. + public init() { + scanner = PureYAML.Parsing.LibYAML.Scanner("", isFinal: false) + } + + /// Appends a chunk of YAML source. Chunks are concatenated in the + /// order fed; a chunk need not align with any YAML grammar boundary. + public mutating func feed(_ chunk: String) { + scanner.reader.append(chunk) + } + + /// Marks that no further input will be fed. After this, ``next()`` + /// never returns ``Outcome/needMoreInput``: a fetch that would have + /// needed more input instead surfaces as a real scanning error, since + /// there is no more input coming that could resolve it. + public mutating func finish() { + scanner.reader.markFinal() + } + + /// Advances the scan by one token. Call in a loop, feeding more input + /// on ``Outcome/needMoreInput`` and stopping on ``Outcome/done``. + public mutating func next() throws -> Outcome { + switch try scanner.scanStep() { + case let .token(token): + .token(Token(token)) + case .needMoreInput: + .needMoreInput + case .done: + .done + } + } + } +} + +extension PureYAML.Streaming.Token { + init(_ token: PureYAML.Parsing.LibYAML.Token) { + self.init( + kind: Kind(token.type), + range: PureYAML.Model.SourceRange( + start: PureYAML.Model.SourcePosition(line: token.startMark.line, column: token.startMark.column), + end: PureYAML.Model.SourcePosition(line: token.endMark.line, column: token.endMark.column), + ), + ) + } +} + +extension PureYAML.Streaming.Token.Kind { + /// A mechanical 1:1 mapping over all 22 `TokenType` cases, split across + /// three initializers purely to keep each switch's cyclomatic complexity + /// under the project's lint limit; the split carries no meaning beyond + /// that; each `default:` either forwards to the next initializer (the + /// cases it doesn't itself handle) or, in the last one, is truly + /// unreachable because the first two together already handle every + /// non-payload-bearing case. + init(_ type: PureYAML.Parsing.LibYAML.TokenType) { + switch type { + case .streamStart: self = .streamStart + case .streamEnd: self = .streamEnd + case .documentStart: self = .documentStart + case .documentEnd: self = .documentEnd + case .blockSequenceStart: self = .blockSequenceStart + case .blockMappingStart: self = .blockMappingStart + case .blockEnd: self = .blockEnd + case .blockEntry: self = .blockEntry + default: self = Self(structural: type) + } + } + + private init(structural type: PureYAML.Parsing.LibYAML.TokenType) { + switch type { + case .flowSequenceStart: self = .flowSequenceStart + case .flowMappingStart: self = .flowMappingStart + case .flowSequenceEnd: self = .flowSequenceEnd + case .flowMappingEnd: self = .flowMappingEnd + case .flowEntry: self = .flowEntry + case .key: self = .key + case .value: self = .value + case .reservedDirective: self = .reservedDirective + default: self = Self(payload: type) + } + } + + private init(payload type: PureYAML.Parsing.LibYAML.TokenType) { + switch type { + case let .versionDirective(major, minor): self = .versionDirective(major: major, minor: minor) + case let .tagDirective(handle, prefix): self = .tagDirective(handle: handle, prefix: prefix) + case let .anchor(name): self = .anchor(name: name) + case let .alias(name): self = .alias(name: name) + case let .tag(handle, suffix): self = .tag(handle: handle, suffix: suffix) + case let .scalar(value, style): self = .scalar(value: value, style: PureYAML.Streaming.Token.ScalarStyle(style)) + default: preconditionFailure("unreachable: init(_:) and init(structural:) handle every non-payload TokenType case") + } + } +} + +extension PureYAML.Streaming.Token.ScalarStyle { + init(_ style: PureYAML.Parsing.ScalarStyle) { + switch style { + case .plain: self = .plain + case .singleQuoted: self = .singleQuoted + case .doubleQuoted: self = .doubleQuoted + case .literal: self = .literal + case .folded: self = .folded + } + } +} diff --git a/Sources/Streaming/Streaming.Token.Kind.swift b/Sources/Streaming/Streaming.Token.Kind.swift new file mode 100644 index 0000000..99236c4 --- /dev/null +++ b/Sources/Streaming/Streaming.Token.Kind.swift @@ -0,0 +1,29 @@ +public extension PureYAML.Streaming.Token { + /// A scanner token's kind, mirroring the internal libyaml-ported scanner's + /// own token kinds (`yaml_token_type_t`) one for one, under public naming + /// that does not leak the port's internal type names. + enum Kind: Equatable, Sendable { + case streamStart + case streamEnd + case documentStart + case documentEnd + case blockSequenceStart + case blockMappingStart + case blockEnd + case blockEntry + case flowSequenceStart + case flowMappingStart + case flowSequenceEnd + case flowMappingEnd + case flowEntry + case key + case value + case versionDirective(major: Int, minor: Int) + case tagDirective(handle: String, prefix: String) + case reservedDirective + case anchor(name: String) + case alias(name: String) + case tag(handle: String, suffix: String) + case scalar(value: String, style: PureYAML.Streaming.Token.ScalarStyle) + } +} diff --git a/Sources/Streaming/Streaming.Token.ScalarStyle.swift b/Sources/Streaming/Streaming.Token.ScalarStyle.swift new file mode 100644 index 0000000..f9e4cba --- /dev/null +++ b/Sources/Streaming/Streaming.Token.ScalarStyle.swift @@ -0,0 +1,12 @@ +public extension PureYAML.Streaming.Token { + /// The quoting or block style a scalar token was scanned with. Distinct + /// from ``PureYAML/Emitting/ScalarStyle``, which is an output rendering + /// policy, not the style a scanned scalar was actually written in. + enum ScalarStyle: Equatable, Sendable { + case plain + case singleQuoted + case doubleQuoted + case literal + case folded + } +} diff --git a/Sources/Streaming/Streaming.Token.swift b/Sources/Streaming/Streaming.Token.swift new file mode 100644 index 0000000..b9cf38e --- /dev/null +++ b/Sources/Streaming/Streaming.Token.swift @@ -0,0 +1,14 @@ +public extension PureYAML.Streaming { + /// A scanner token: a kind plus the source span it covers. Produced by + /// ``Scanner/next()`` one at a time, in the same order ``PureYAML/parse(_:)`` + /// would see them internally. + struct Token: Equatable, Sendable { + public var kind: Kind + public var range: PureYAML.Model.SourceRange + + public init(kind: Kind, range: PureYAML.Model.SourceRange) { + self.kind = kind + self.range = range + } + } +} diff --git a/Tests/PublicStreamingScannerTests.swift b/Tests/PublicStreamingScannerTests.swift new file mode 100644 index 0000000..e73b0ad --- /dev/null +++ b/Tests/PublicStreamingScannerTests.swift @@ -0,0 +1,113 @@ +import Foundation +@testable import PureYAML +import Testing + +/// Differential proof for the public `PureYAML.Streaming.Scanner`, the thin +/// translation wrapper around the internal `LibYAML.Scanner.scanStep()` +/// already proven by `StreamingScannerDifferentialTests`. That existing proof +/// covers `scanStep()` itself; this one covers the translation layer on top +/// of it (`Streaming.Token`/`Streaming.Token.Kind`/`Streaming.Token.ScalarStyle` +/// mapped from the internal `LibYAML.Token`/`TokenType`/`ScalarStyle`), since a +/// wrong case mapping or a dropped field would not be caught by a proof that +/// never looks at the public types at all. +@Suite("Public streaming scanner") +struct PublicStreamingScannerTests { + @Test("Public Scanner matches the internal scanAll() reference, translated, at several chunk sizes", arguments: [ + "title: Example API", + """ + title: Example + servers: + - url: / + """, + "flow: {a: 1, b: [2, 3]}", + """ + base: &anchor + value: 1 + derived: *anchor + """, + "block: |\n line one\n line two\n", + "&a a: b\nc: &d d\n", + ]) + func test_publicScannerMatchesTranslatedReference(yaml: String) throws { + try assertPublicScannerMatchesReference(yaml) + } + + @Test("Public Scanner matches the internal scanAll() reference over the whole yaml-test-suite corpus") + func test_publicScannerMatchesReferenceOverWholeSuite() throws { + guard ProcessInfo.processInfo.environment["PUREYAML_RUN_YAML_TEST_SUITE"] == "1" else { + return + } + let manifest = try YAMLTestSuiteManifest.load() + var mismatches: [String] = [] + for testCase in manifest.cases { + let yaml = try testCase.loadInput() + do { + try assertPublicScannerMatchesReference(yaml) + } catch { + mismatches.append("\(testCase.id): \(error)") + } + } + #expect(mismatches.isEmpty, "public/internal mismatches (\(mismatches.count)): \(mismatches.prefix(10).joined(separator: "; "))") + } +} + +private func assertPublicScannerMatchesReference(_ yaml: String) throws { + let reference = translatedBatchReference(yaml) + for chunkSize in [1, 2, 3, 7, 64, yaml.unicodeScalars.count + 1] { + let observed = try publicScan(yaml, chunkSize: chunkSize) + #expect(observed == reference, "chunk size \(chunkSize) diverged from the translated reference for: \(yaml.debugDescription)") + } +} + +private enum PublicScanResult: Equatable { + case tokens([PureYAML.Streaming.Token]) + case threw(String) +} + +/// The reference: the batch scanner's own tokens, run through the exact same +/// `Streaming.Token.init(_:)` translator the public `Scanner` uses, so this +/// proof isolates "does the public wrapper preserve what the internal scanner +/// already produces" from "is the internal scanner itself correct" (the +/// latter is `StreamingScannerDifferentialTests`'s job, not this one's). +private func translatedBatchReference(_ yaml: String) -> PublicScanResult { + var scanner = PureYAML.Parsing.LibYAML.Scanner(yaml) + do { + return try .tokens(scanner.scanAll().map(PureYAML.Streaming.Token.init)) + } catch { + return .threw("\(error)") + } +} + +private func publicScan(_ yaml: String, chunkSize: Int) throws -> PublicScanResult { + let scalars = Array(yaml.unicodeScalars) + var chunks: [String] = [] + var index = 0 + while index < scalars.count { + let end = min(index + chunkSize, scalars.count) + chunks.append(String(String.UnicodeScalarView(scalars[index ..< end]))) + index = end + } + + var scanner = PureYAML.Streaming.Scanner() + var nextChunk = 0 + var tokens: [PureYAML.Streaming.Token] = [] + do { + while true { + switch try scanner.next() { + case let .token(token): + tokens.append(token) + case .done: + return .tokens(tokens) + case .needMoreInput: + if nextChunk < chunks.count { + scanner.feed(chunks[nextChunk]) + nextChunk += 1 + } else { + scanner.finish() + } + } + } + } catch { + return .threw("\(error)") + } +} diff --git a/Tests/StreamingScannerDifferentialTests.swift b/Tests/StreamingScannerDifferentialTests.swift index e4867aa..57f21e5 100644 --- a/Tests/StreamingScannerDifferentialTests.swift +++ b/Tests/StreamingScannerDifferentialTests.swift @@ -33,6 +33,7 @@ struct StreamingScannerDifferentialTests { """, "block: |\n line one\n line two\n", "title: \"unterminated", + "&a a: b\nc: &d d\n", ]) func test_representativeCasesMatchScanAllAtSeveralChunkSizes(yaml: String) throws { try assertStreamingMatchesBatch(yaml) diff --git a/Tests/StreamingScannerPerformanceTests.swift b/Tests/StreamingScannerPerformanceTests.swift index 99aa232..8687413 100644 --- a/Tests/StreamingScannerPerformanceTests.swift +++ b/Tests/StreamingScannerPerformanceTests.swift @@ -4,15 +4,17 @@ import Testing /// Measures the resumable scanner's checkpoint-and-retry cost (issue #4) against /// the batch scanner, across a spread of chunk sizes, over the largest real-world -/// fixture, plus a `singleChunk` control that isolates the floor from real -/// rescanning (see CHANGELOG.md). The overhead splits into two independent -/// costs: a fixed floor (~1.75x release-mode, present even with a single chunk, -/// caused by copying the whole `Scanner` on every fetch whenever `isFinal` is -/// false) and a rescan-driven tail that only appears below roughly 4096-scalar -/// chunks. Persisted mid-token suspension, the deferred next increment, would -/// remove the rescan tail but not the floor; a fix for the floor is not yet -/// designed. Gated: release-mode timing on a 9.5MB fixture is not a default -/// `swift test` concern. +/// fixture, plus controls that isolate the floor from real rescanning and from +/// buffer-population strategy (see CHANGELOG.md). The overhead splits into two +/// independent costs: a fixed floor (~1.75x release-mode, present even with a +/// single chunk, structurally attributable to the checkpoint taken on every +/// fetch whenever `isFinal` is false, confirmed by the `isFinal: true` control +/// costing 1.04x with the identical per-token loop) and a rescan-driven tail +/// that only appears below roughly 4096-scalar chunks. Persisted mid-token +/// suspension, the deferred next increment, would remove both: it eliminates +/// the need for a checkpoint at all, not just the rescanning a checkpoint's +/// retry causes. Gated: release-mode timing on a 9.5MB fixture is not a +/// default `swift test` concern. /// /// Run this in release mode; debug-mode numbers are not representative (Swift's /// `-Onone` inserts defensive retain/copy traffic around `mutating throws` @@ -83,6 +85,91 @@ struct StreamingScannerPerformanceTests { let singleChunkDuration = try streamingDuration(yaml: yaml, chunkSize: wholeFileAsOneChunk, clock: clock) let singleChunkRatio = singleChunkDuration / batchDuration print("STREAMING-PERF singleChunk(wholeFile) time=\(singleChunkDuration) ratio=\(String(format: "%.2f", singleChunkRatio))x") + + // Isolates whether the floor is scanning-related at all, or an artifact of + // how the buffer is populated: `Reader.init` builds `scalars` in one shot + // via `Array(string.unicodeScalars)`; `Reader.append(_:)` (used by every + // streaming chunk feed, including the single-chunk case above) instead + // grows an existing array via `append(contentsOf:)`. No scanning happens + // here at all, just buffer construction. + let constructDuration = clock.measure { + _ = PureYAML.Parsing.LibYAML.Scanner(yaml) + } + var appendScanner = PureYAML.Parsing.LibYAML.Scanner("", isFinal: false) + let appendDuration = clock.measure { + appendScanner.reader.append(yaml) + } + print("STREAMING-PERF bufferConstruct time=\(constructDuration) bufferAppend time=\(appendDuration) ratio=\(String(format: "%.2f", appendDuration / constructDuration))x") + } + + /// The `github-rest-api.yaml` fixture `test_measure` uses above has only + /// short plain scalars, so it cannot exercise mid-token suspension's + /// actual purpose: resuming a single plain scalar that itself spans many + /// chunks. This measures exactly that shape, on one synthetic 1MB plain + /// scalar. It is what caught a real O(N^2) bug in the first version of + /// `resumePlainScalar`/`scanStep()`'s resume branch: passing the + /// suspended `PlainState` across the `resumePlainScalar` call boundary + /// as a value parameter (even one immediately copied and overwritten + /// inside the callee) left the caller's own copy of `state.string`'s + /// buffer aliased to the callee's for the call's whole duration, so + /// copy-on-write copied the entire accumulated scalar on every resume; + /// at chunk size 16 that was a measured 29x, worse than not suspending + /// at all. Passing `state` directly `inout`, with the disalias done at + /// the call site before the call rather than inside the callee, fixed it + /// to 3-4x. See `resumePlainScalar` and `scanStep()`'s resume branch for + /// the fix itself. + @Test("measure scanStep() overhead on one plain scalar spanning many chunks") + func test_measureLongPlainScalar() throws { + guard ProcessInfo.processInfo.environment["PUREYAML_STREAMING_PERF_MEASURE"] == "1" else { + return + } + let longWord = String(repeating: "abcdefghij", count: 100_000) + let yaml = "key: \(longWord)\n" + let clock = ContinuousClock() + + var batchScanner = PureYAML.Parsing.LibYAML.Scanner(yaml) + let batchDuration = try clock.measure { + _ = try batchScanner.scanAll() + } + print("LONGSCALAR-PERF batch bytes=\(yaml.utf8.count) time=\(batchDuration)") + + for chunkSize in [16, 64, 256, 4096] { + let duration = try streamingDuration(yaml: yaml, chunkSize: chunkSize, clock: clock) + let ratio = duration / batchDuration + print("LONGSCALAR-PERF chunkSize=\(chunkSize) time=\(duration) ratio=\(String(format: "%.2f", ratio))x") + } + } + + /// A regression guard for the O(N^2) aliasing bug `test_measureLongPlainScalar`'s + /// doc comment describes, not just an observational print like the + /// measurements above: that bug's signature is per-byte cost growing with + /// the scalar's total length, since each resume re-copied everything + /// accumulated so far, not just the new chunk. Genuinely O(N) work + /// (copying only each chunk's new characters, once) keeps per-byte cost + /// flat as length grows; measured flat within 8% across a 16x range + /// (250KB to 4MB) after the fix landed. `smallestPerByte` versus + /// `largestPerByte` is asserted at a generous 3x, not the ~1.08x actually + /// measured: this must catch a real quadratic regression, not fail on + /// ordinary run-to-run noise on a shared CI runner. + @Test("scanStep() cost on a long plain scalar stays linear as the scalar's length grows") + func test_longPlainScalarStaysLinear() throws { + guard ProcessInfo.processInfo.environment["PUREYAML_STREAMING_PERF_MEASURE"] == "1" else { + return + } + let clock = ContinuousClock() + var perByteCosts: [Double] = [] + for wordCount in [25000, 400_000] { + let longWord = String(repeating: "abcdefghij", count: wordCount) + let yaml = "key: \(longWord)\n" + let duration = try streamingDuration(yaml: yaml, chunkSize: 16, clock: clock) + let seconds = Double(duration.components.seconds) + Double(duration.components.attoseconds) / 1e18 + let perByte = seconds / Double(yaml.utf8.count) + perByteCosts.append(perByte) + print("LINEAR-CHECK bytes=\(yaml.utf8.count) time=\(duration) perByte=\(perByte)") + } + let smallestPerByte = perByteCosts[0] + let largestPerByte = perByteCosts[1] + #expect(largestPerByte < smallestPerByte * 3) } }