Mihaela Mihaljevic
7add6f0087
Add StreamingScannerPerformanceTests (issue #4), gated behind PUREYAML_STREAMING_PERF_MEASURE=1, measuring scanStep() against scanAll() on the largest real-world fixture (9.5MB) across a spread of chunk sizes. Small chunk sizes (16-256 scalars) show a real, chunk-size-dependent rescan cost, up to 6x at 16, consistent with the checkpoint-and-retry design rescanning a fetch attempt once per chunk it spans. A control run with isFinal: true from construction, where ranDry can never be set and no rollback ever happens, still cost ~1.5x over scanAll(), proving a separate, chunk-size-independent cost dominates at any realistic chunk size (4KB+): the checkpoint was being copied on every fetch even when it could never be used. Fixed by skipping the checkpoint entirely when reader.isFinal is true. Re-measured rather than assumed fixed: the isolated control improved from 1.49x to 1.38x, a real but modest gain, while the large-chunk-size ratios were unchanged within measurement noise. The remaining floor's cause is not identified and is left open on issue #4 rather than patched further on speculation; closing it needs a profiler, not another guess. Whole-suite conformance (402/402) and the chunk-size differential proof both still pass unchanged.
72 lines
3.3 KiB
Swift
72 lines
3.3 KiB
Swift
extension PureYAML.Parsing.LibYAML.Scanner {
|
|
/// The result of one `scanStep()` call.
|
|
enum ScanOutcome: Equatable {
|
|
/// A token is safe to release to the caller now.
|
|
case token(PureYAML.Parsing.LibYAML.Token)
|
|
/// The buffered input was exhausted mid-fetch; append more input (via
|
|
/// `reader.append(_:)`) and call `scanStep()` again. The scanner's state is
|
|
/// unchanged from before this call, as if it had not been called.
|
|
case needMoreInput
|
|
/// STREAM-END has been released and no further tokens remain.
|
|
case done
|
|
}
|
|
|
|
/// Resumable, one-token-at-a-time counterpart to `scanAll()`. Where `scanAll()`
|
|
/// assumes the full input is already buffered, `scanStep()` may be driven across
|
|
/// a `Reader` still receiving chunks (`isFinal == false`): a fetch that runs off
|
|
/// the buffered end is rolled back in full (as if it had not been attempted)
|
|
/// rather than mistaken for true end of input.
|
|
///
|
|
/// A token already queued is not always safe to hand out immediately: libyaml's
|
|
/// simple-key mechanism can still splice a KEY (and a BLOCK-MAPPING-START) in
|
|
/// front of it once a later `:` is found (`saveSimpleKey`/`insert(_:atTokenNumber:)`
|
|
/// in `LibYAML.Scanner.swift`/`LibYAML.IndentEngine.swift`). Port of libyaml's
|
|
/// `yaml_parser_fetch_more_tokens` release condition: the token at the queue
|
|
/// front (position `tokensParsed`) is safe to release only when no currently
|
|
/// possible simple key (at any flow level) is still recorded at that exact
|
|
/// position; such a key could still resolve into a splice targeting that slot.
|
|
mutating func scanStep() throws -> ScanOutcome {
|
|
while true {
|
|
if isFrontSafeToRelease, let token = indentEngine.dequeue() {
|
|
return .token(token)
|
|
}
|
|
if streamEndProduced {
|
|
return .done
|
|
}
|
|
// A dry run can only ever be reported while `!reader.isFinal` (see
|
|
// `Reader.peek(_:)`), so once the caller has appended every chunk and
|
|
// called `markFinal()`, no fetch can ever need rolling back. Skipping
|
|
// the snapshot in that case matters: it is the entire cost difference
|
|
// between `scanStep()` and `scanAll()` once the whole input is already
|
|
// buffered (measured in StreamingScannerPerformanceTests), since neither
|
|
// reads nor mutations inside `fetchNextToken()` become any cheaper by
|
|
// being wrapped in an unused checkpoint.
|
|
guard !reader.isFinal else {
|
|
try fetchNextToken()
|
|
continue
|
|
}
|
|
let checkpoint = self
|
|
do {
|
|
try fetchNextToken()
|
|
} catch {
|
|
if reader.ranDry {
|
|
self = checkpoint
|
|
return .needMoreInput
|
|
}
|
|
throw error
|
|
}
|
|
if reader.ranDry {
|
|
self = checkpoint
|
|
return .needMoreInput
|
|
}
|
|
}
|
|
}
|
|
|
|
private var isFrontSafeToRelease: Bool {
|
|
guard !indentEngine.tokens.isEmpty else {
|
|
return false
|
|
}
|
|
return !simpleKeys.contains { $0.possible && $0.tokenNumber == indentEngine.tokensParsed }
|
|
}
|
|
}
|