perf(parsing): correct the scanStep() floor to a checkpoint cost, not rescanning

The previous commit concluded the ~1.75x floor needed persisted mid-token
suspension to close, since that removes the need for a checkpoint. One more
measurement shows that conclusion was wrong for the floor itself: feeding the
entire 9.5MB file as a single chunk, so isFinal stays false until the true
end and essentially no real rescanning can occur, still measures 1.75x,
identical to 1,000,000-scalar and 65536-scalar chunk sizes.

The overhead is actually two independent costs. A fixed floor (~1.75x),
present even with zero rescanning, from copying the whole Scanner on every
fetchNextToken() call whenever isFinal is false, whether or not that copy is
ever used to roll back; since isFinal only becomes true once, on the last
chunk, that copy is paid for virtually every token in a real streaming
session. And a rescan-driven tail that only appears below roughly
4096-scalar chunks (up to 11.54x at 16 scalars).

Mid-token suspension would eliminate the rescan tail, not the floor. A fix
for the floor is not yet designed: every field the checkpoint covers can be
mutated before a dry point is reached within a single fetch, so no smaller
subset is obviously safe to skip, and isFinal: true remains the only
provably safe condition for skipping it entirely.

CHANGELOG.md

@@ -26,18 +26,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   release, the `isFinal: true` control measured **1.04x**, essentially batch
   parity, meaning the checkpoint-skip fix was actually a full fix for that
   case, not the small, unexplained one it looked like in debug mode.
-  What remains, correctly explained rather than guessed at: a chunked run's
-  `isFinal` only becomes true once, on the last chunk, so for any input that
-  arrives incrementally, the checkpoint stays live for virtually the entire
-  scan no matter how large the chunks are. Release-mode chunked ratios: 11.70x
-  at 16-scalar chunks down to 1.75x at 1,000,000-scalar chunks (10 chunks
-  total for this file), never reaching the 1.04x no-chunking floor. Closing
-  that gap needs the deferred persisted mid-token suspension, which removes
-  the need for a checkpoint at all rather than making it cheaper; the
-  checkpoint's cost cannot be optimized away further without it, since
-  `isFinal: true` is the only condition under which skipping it is provably
-  safe (a single scalar can span past any finite "probably buffered enough"
-  margin).
+  What remains splits into two distinct costs, decomposed by one more
+  measurement rather than assumed: feeding the whole 9.5MB file as a single
+  chunk (`isFinal` stays false until the true end, so at most one token in the
+  entire file could ever actually be rescanned) still measured 1.75x, the same
+  as 1,000,000-scalar chunks (1.74x, ~10 chunks) and 65536-scalar chunks
+  (1.74x). Real rescanning is not the floor's cause, since there is
+  essentially no rescanning to have caused it here. The floor is the cost of
+  copying the whole `Scanner` on every `fetchNextToken()` call whenever
+  `isFinal` is false, whether or not that copy is ever used to roll back;
+  since a chunked run's `isFinal` only becomes true once, on the last chunk,
+  that copy is taken for virtually every token in the file. Only *shrinking*
+  chunk sizes below roughly 4096 scalars adds a second, genuinely
+  rescan-driven cost on top of that floor (11.54x at 16 scalars).
+  This corrects the previous entry's conclusion: the deferred persisted
+  mid-token suspension would eliminate the rescan-driven tail at small chunk
+  sizes, but would not touch the 1.75x floor, since that floor is not caused
+  by rescanning. Closing the floor needs a different fix, not yet designed:
+  something that avoids copying the whole scanner on every fetch while
+  `isFinal` is false, without weakening the correctness guarantee that only
+  `isFinal: true` currently provides (every other field the checkpoint covers
+  can be mutated before a dry point is reached within a single fetch, so no
+  smaller subset of fields is obviously safe to skip either).
 
 - Broaden the real-seed mutation property suite (issue #6) with three mutation
   classes named in that issue but not yet covered: `invalidControlCharacter`

Tests/StreamingScannerPerformanceTests.swift

@@ -4,14 +4,15 @@ import Testing
 
 /// Measures the resumable scanner's checkpoint-and-retry cost (issue #4) against
 /// the batch scanner, across a spread of chunk sizes, over the largest real-world
-/// fixture. `scanStep()`'s first increment restores the whole scanner and retries
-/// a fetch from its start whenever it runs dry, so a token that spans many small
-/// chunks is rescanned once per chunk it spans: correct (proven by
-/// `StreamingScannerDifferentialTests`) but worst-case quadratic in the number of
-/// chunks a single token spans. This measures whether that cost matters in
-/// practice before persisted mid-token suspension (a materially larger, riskier
-/// change to the scalar-scanning loops) is worth building. Gated: release-mode
-/// timing on a 9.5MB fixture is not a default `swift test` concern.
+/// fixture, plus a `singleChunk` control that isolates the floor from real
+/// rescanning (see CHANGELOG.md). The overhead splits into two independent
+/// costs: a fixed floor (~1.75x release-mode, present even with a single chunk,
+/// caused by copying the whole `Scanner` on every fetch whenever `isFinal` is
+/// false) and a rescan-driven tail that only appears below roughly 4096-scalar
+/// chunks. Persisted mid-token suspension, the deferred next increment, would
+/// remove the rescan tail but not the floor; a fix for the floor is not yet
+/// designed. Gated: release-mode timing on a 9.5MB fixture is not a default
+/// `swift test` concern.
 ///
 /// Run this in release mode; debug-mode numbers are not representative (Swift's
 /// `-Onone` inserts defensive retain/copy traffic around `mutating throws`
@@ -70,6 +71,18 @@ struct StreamingScannerPerformanceTests {
             let ratio = duration / batchDuration
             print("STREAMING-PERF chunkSize=\(chunkSize) time=\(duration) ratio=\(String(format: "%.2f", ratio))x")
         }
+
+        // Distinguishes two candidate explanations for the chunked-mode floor: (a)
+        // real rescanning at chunk boundaries, which should vanish as chunk count
+        // drops toward 1, or (b) a fixed per-fetchNextToken() cost paid on every
+        // token regardless of how many chunks exist, which would not. A single
+        // chunk covering the whole file means `isFinal` is false (so the checkpoint
+        // is taken every fetch) but `ranDry` can only ever fire once, at the true
+        // end: the only possible rescan is the very last token.
+        let wholeFileAsOneChunk = yaml.unicodeScalars.count + 1
+        let singleChunkDuration = try streamingDuration(yaml: yaml, chunkSize: wholeFileAsOneChunk, clock: clock)
+        let singleChunkRatio = singleChunkDuration / batchDuration
+        print("STREAMING-PERF singleChunk(wholeFile) time=\(singleChunkDuration) ratio=\(String(format: "%.2f", singleChunkRatio))x")
     }
 }