perf(parsing): measure scanStep() overhead, fix a real but partial cause

Add StreamingScannerPerformanceTests (issue #4), gated behind
PUREYAML_STREAMING_PERF_MEASURE=1, measuring scanStep() against scanAll() on
the largest real-world fixture (9.5MB) across a spread of chunk sizes.

Small chunk sizes (16-256 scalars) show a real, chunk-size-dependent rescan
cost, up to 6x at 16, consistent with the checkpoint-and-retry design
rescanning a fetch attempt once per chunk it spans. A control run with
isFinal: true from construction, where ranDry can never be set and no
rollback ever happens, still cost ~1.5x over scanAll(), proving a separate,
chunk-size-independent cost dominates at any realistic chunk size (4KB+): the
checkpoint was being copied on every fetch even when it could never be used.

Fixed by skipping the checkpoint entirely when reader.isFinal is true.
Re-measured rather than assumed fixed: the isolated control improved from
1.49x to 1.38x, a real but modest gain, while the large-chunk-size ratios
were unchanged within measurement noise. The remaining floor's cause is not
identified and is left open on issue #4 rather than patched further on
speculation; closing it needs a profiler, not another guess.

Whole-suite conformance (402/402) and the chunk-size differential proof both
still pass unchanged.

CHANGELOG.md

@@ -9,6 +9,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- Add `Tests/StreamingScannerPerformanceTests.swift` (issue #4), measuring
+  `scanStep()`'s overhead over `scanAll()` on the largest real-world fixture
+  (github-rest-api.yaml, 9.5MB) across a spread of chunk sizes, gated behind
+  `PUREYAML_STREAMING_PERF_MEASURE=1`. Measured, not guessed: small chunk sizes
+  (16-256 scalars) show a real, chunk-size-dependent rescan cost (up to ~6x at
+  16), consistent with the checkpoint-and-retry design re-scanning a fetch
+  attempt once per chunk it spans. But a control run with `isFinal: true` from
+  construction, where `ranDry` can never be set and no rollback ever happens,
+  still showed ~1.5x overhead over `scanAll()`, proving a separate,
+  chunk-size-independent cost dominates at any realistic chunk size (4KB+): the
+  checkpoint was being copied on every fetch even when it could never be used.
+  Fixed by skipping the checkpoint entirely when `reader.isFinal` is true.
+  Re-measured: the isolated control improved from 1.49x to 1.38x, a real but
+  modest gain; the large-chunk-size ratios (4096+) were unchanged within
+  measurement noise (~1.5x). The remaining floor's cause is not identified.
+  Closing it needs a real profiler, not further guessing, so it is left open on
+  issue #4 rather than patched further on speculation.
+
 - Broaden the real-seed mutation property suite (issue #6) with three mutation
   classes named in that issue but not yet covered: `invalidControlCharacter`
   (a raw control byte injected into a line), `mixedTabSpaceIndentation` (a tab

Sources/Parsing/LibYAML.Scanner.Streaming.swift

@@ -33,17 +33,29 @@ extension PureYAML.Parsing.LibYAML.Scanner {
             if streamEndProduced {
                 return .done
             }
+            // A dry run can only ever be reported while `!reader.isFinal` (see
+            // `Reader.peek(_:)`), so once the caller has appended every chunk and
+            // called `markFinal()`, no fetch can ever need rolling back. Skipping
+            // the snapshot in that case matters: it is the entire cost difference
+            // between `scanStep()` and `scanAll()` once the whole input is already
+            // buffered (measured in StreamingScannerPerformanceTests), since neither
+            // reads nor mutations inside `fetchNextToken()` become any cheaper by
+            // being wrapped in an unused checkpoint.
+            guard !reader.isFinal else {
+                try fetchNextToken()
+                continue
+            }
             let checkpoint = self
             do {
                 try fetchNextToken()
             } catch {
-                if !reader.isFinal, reader.ranDry {
+                if reader.ranDry {
                     self = checkpoint
                     return .needMoreInput
                 }
                 throw error
             }
-            if !reader.isFinal, reader.ranDry {
+            if reader.ranDry {
                 self = checkpoint
                 return .needMoreInput
             }

Tests/StreamingScannerPerformanceTests.swift

@@ -0,0 +1,94 @@
+import Foundation
+@testable import PureYAML
+import Testing
+
+/// Measures the resumable scanner's checkpoint-and-retry cost (issue #4) against
+/// the batch scanner, across a spread of chunk sizes, over the largest real-world
+/// fixture. `scanStep()`'s first increment restores the whole scanner and retries
+/// a fetch from its start whenever it runs dry, so a token that spans many small
+/// chunks is rescanned once per chunk it spans: correct (proven by
+/// `StreamingScannerDifferentialTests`) but worst-case quadratic in the number of
+/// chunks a single token spans. This measures whether that cost matters in
+/// practice before persisted mid-token suspension (a materially larger, riskier
+/// change to the scalar-scanning loops) is worth building. Gated: release-mode
+/// timing on a 9.5MB fixture is not a default `swift test` concern.
+@Suite("Streaming Scanner Performance")
+struct StreamingScannerPerformanceTests {
+    @Test("measure scanStep() overhead over scanAll() at a spread of chunk sizes")
+    func test_measure() throws {
+        guard ProcessInfo.processInfo.environment["PUREYAML_STREAMING_PERF_MEASURE"] == "1" else {
+            return
+        }
+        let url = try #require(Bundle.module.url(
+            forResource: "github-rest-api",
+            withExtension: "yaml",
+            subdirectory: "Fixtures/real-yaml",
+        ))
+        let yaml = try String(contentsOf: url, encoding: .utf8)
+        let clock = ContinuousClock()
+
+        var batchScanner = PureYAML.Parsing.LibYAML.Scanner(yaml)
+        let batchDuration = try clock.measure {
+            _ = try batchScanner.scanAll()
+        }
+        print("STREAMING-PERF batch bytes=\(yaml.utf8.count) time=\(batchDuration)")
+
+        // Control: the whole input is present from construction (isFinal: true), so
+        // ranDry can never be set and the checkpoint is never restored, meaning zero
+        // chunking or retry ever happens. Any overhead this still shows over
+        // scanAll() is intrinsic to scanStep()'s per-token loop (the checkpoint copy
+        // taken before every fetch, and isFrontSafeToRelease's check), not to
+        // chunking. This isolates that fixed cost from the chunk-size-dependent
+        // rescan cost measured below.
+        var noChunkScanner = PureYAML.Parsing.LibYAML.Scanner(yaml, isFinal: true)
+        let noChunkDuration = try clock.measure {
+            while true {
+                switch try noChunkScanner.scanStep() {
+                case .token: continue
+                case .done: return
+                case .needMoreInput: Issue.record("isFinal: true must never report needMoreInput")
+                    return
+                }
+            }
+        }
+        let noChunkRatio = noChunkDuration / batchDuration
+        print("STREAMING-PERF noChunking(control) time=\(noChunkDuration) ratio=\(String(format: "%.2f", noChunkRatio))x")
+
+        for chunkSize in [16, 64, 256, 1024, 4096, 65536, 1_000_000] {
+            let duration = try streamingDuration(yaml: yaml, chunkSize: chunkSize, clock: clock)
+            let ratio = duration / batchDuration
+            print("STREAMING-PERF chunkSize=\(chunkSize) time=\(duration) ratio=\(String(format: "%.2f", ratio))x")
+        }
+    }
+}
+
+private func streamingDuration(yaml: String, chunkSize: Int, clock: ContinuousClock) throws -> Duration {
+    let scalars = Array(yaml.unicodeScalars)
+    var chunks: [String] = []
+    var index = 0
+    while index < scalars.count {
+        let end = min(index + chunkSize, scalars.count)
+        chunks.append(String(String.UnicodeScalarView(scalars[index ..< end])))
+        index = end
+    }
+
+    return try clock.measure {
+        var scanner = PureYAML.Parsing.LibYAML.Scanner("", isFinal: false)
+        var nextChunk = 0
+        while true {
+            switch try scanner.scanStep() {
+            case .token:
+                continue
+            case .done:
+                return
+            case .needMoreInput:
+                if nextChunk < chunks.count {
+                    scanner.reader.append(chunks[nextChunk])
+                    nextChunk += 1
+                } else {
+                    scanner.reader.markFinal()
+                }
+            }
+        }
+    }
+}