Plain scalars now suspend and resume mid-token instead of rolling back to a checkpoint and rescanning from the token's start on every chunk boundary inside them: scanPlainScalar's loop is an explicit PlainScalarPhase state machine that throws PlainScalarSuspended the moment a lookahead runs dry, and scanStep() resumes it directly from Scanner.suspendedPlainScalar. Getting this right took four rounds of bugs, each caught by the existing whole-suite streaming differential test: checking ranDry only once at the loop's end instead of after every peek-dependent decision, resuming a dry-interrupted content or break run through a pre-check meant only for a fresh entry point, trusting dispatchToken's own dry-fooled document-marker check, and a nested blankLineAhead() lookahead independently going dry before an outer check could catch it. A synthetic long-scalar test (needed since the real-world benchmark fixture's scalars are too short to exercise this) then caught a fifth, genuine bug: the first working resumePlainScalar took the whole suspended value by parameter, which kept a reference to its accumulated string alive for the entire resume call and defeated copy-on-write, copying the whole scalar on every resume. That is O(N^2) in the scalar's length, and measured 29x slower than not suspending at all at small chunk sizes. Passing state inout directly, with the alias broken at the call site before the call, fixed it to 3-4x, verified genuinely linear (not just a smaller quadratic constant) by a permanent test scaling scalar length 16x at a fixed chunk size. Also exposes this scanner as public API for the first time: scanStep() and everything around it were internal-only, so no consumer outside this module, including the separate PureYAMLStreaming package that actually wants a resumable token scanner, could call it. PureYAML.Streaming.Scanner wraps scanStep() directly with no independent logic, translating its tokens into public Streaming.Token/Kind/ScalarStyle types that don't leak the libyaml port's internal naming. Proven separately from the internal scanner with its own differential test, since a wrong case mapping in the translation layer wouldn't be caught by a proof that never looks at the public types. Stops short of rewiring PureYAMLStreaming's own document-boundary scanner onto this; that is separate, cross-repo work with its own build and CI cycle.
95 lines
4.0 KiB
Swift
95 lines
4.0 KiB
Swift
extension PureYAML.Parsing.LibYAML {
|
|
/// Port of libyaml's input buffer and position tracking (the `buffer`/`mark`
|
|
/// fields with the `PEEK`, `SKIP`, and `SKIP_LINE` macros). It iterates
|
|
/// `Unicode.Scalar`s rather than grapheme clusters so a CR and an LF are
|
|
/// distinct, matching libyaml's byte-oriented scanning, and tracks the mark in
|
|
/// libyaml's 0-based line and column convention; conversion to the package's
|
|
/// 1-based `Mark` happens at token emission during integration.
|
|
struct Reader {
|
|
private var scalars: [Unicode.Scalar]
|
|
private(set) var index = 0
|
|
private(set) var line = 0
|
|
private(set) var column = 0
|
|
/// Whether the buffered `scalars` are the complete input. True for every
|
|
/// batch caller (the default), so `ranDry` can never be set and this type's
|
|
/// behavior is unchanged for `scanAll()`. A resumable caller passes `false`
|
|
/// and later calls `markFinal()` once the last chunk has been appended.
|
|
private(set) var isFinal: Bool
|
|
/// Set by `peek(_:)` when it was asked for a position past the buffered
|
|
/// scalars while `!isFinal`: the query is inconclusive, not a real end of
|
|
/// input, and the caller must suspend and retry after `append(_:)`. Batch
|
|
/// callers (`isFinal == true`) never set this, since the guard below is
|
|
/// unreachable when `isFinal` is true.
|
|
private(set) var ranDry = false
|
|
|
|
init(_ string: String = "", isFinal: Bool = true) {
|
|
scalars = Array(string.unicodeScalars)
|
|
self.isFinal = isFinal
|
|
}
|
|
|
|
/// `PEEK_TOKEN` / `CHECK_AT`: the scalar `offset` ahead, or nil past the
|
|
/// end (libyaml's `\0`). Past the buffered end while more chunks may still
|
|
/// arrive, this flags `ranDry` instead of asserting true end of input.
|
|
mutating func peek(_ offset: Int = 0) -> Unicode.Scalar? {
|
|
let position = index + offset
|
|
if position < scalars.count {
|
|
return scalars[position]
|
|
}
|
|
if !isFinal {
|
|
ranDry = true
|
|
}
|
|
return nil
|
|
}
|
|
|
|
/// Append a chunk of newly arrived input. Only meaningful before
|
|
/// `markFinal()`. Clears `ranDry`: new data may satisfy whatever `peek(_:)`
|
|
/// call set it, and nothing else ever clears it. The existing
|
|
/// checkpoint-restore path gets this for free (the restored `Reader` is a
|
|
/// copy from before the dry read), but a resumed scan that keeps its
|
|
/// reader position instead of restoring a snapshot (`PlainScalarSuspended`)
|
|
/// has no other point where a stale `true` would ever become false again.
|
|
mutating func append(_ chunk: String) {
|
|
scalars.append(contentsOf: chunk.unicodeScalars)
|
|
ranDry = false
|
|
}
|
|
|
|
/// Declare that no further chunks are coming: a `peek` past the buffered
|
|
/// end is now a real end of input, matching the batch reader's behavior.
|
|
mutating func markFinal() {
|
|
isFinal = true
|
|
}
|
|
|
|
var isAtEnd: Bool {
|
|
index >= scalars.count
|
|
}
|
|
|
|
/// The current position. Column is 0-based, as in libyaml.
|
|
var mark: PureYAML.Parsing.Mark {
|
|
PureYAML.Parsing.Mark(line: line, column: column, index: index)
|
|
}
|
|
|
|
/// `SKIP`: consume one non-break scalar, advancing the column.
|
|
mutating func skip() {
|
|
guard index < scalars.count else {
|
|
return
|
|
}
|
|
index += 1
|
|
column += 1
|
|
}
|
|
|
|
/// `SKIP_LINE`: consume a line break and start a new line. A CR+LF pair is
|
|
/// one break, matching libyaml.
|
|
mutating func skipLine() {
|
|
if peek() == "\r", peek(1) == "\n" {
|
|
index += 2
|
|
} else if CharacterClass.isBreak(peek()) {
|
|
index += 1
|
|
} else {
|
|
return
|
|
}
|
|
line += 1
|
|
column = 0
|
|
}
|
|
}
|
|
}
|