PureYAML/Sources/Parsing/LibYAML.Reader.swift
Mihaela Mihaljevic 18fdf9b69a feat(parsing): add a resumable scanner increment and broaden the mutation suite
Add Scanner.scanStep(), the first increment of true chunked input streaming
(#4): Reader gains isFinal/append(_:)/markFinal(), and peek(_:) flags ranDry
instead of asserting end of input when the buffer runs out mid-fetch and more
chunks may still arrive. scanStep() snapshots the scanner before each
fetchNextToken() and restores it whenever a fetch runs dry, so a construct
spanning a chunk boundary is retried after more input lands rather than
misread as an early end of input. Every existing call site keeps the default
isFinal: true, so scanAll() and batch parsing are unchanged; ranDry can never
be set when isFinal is true, by construction. A token already queued is not
always safe to release, since libyaml's simple-key mechanism can still splice
a KEY token in front of it, so scanStep() ports libyaml's
yaml_parser_fetch_more_tokens release condition.

Proof: StreamingScannerDifferentialTests.swift drives scanStep() at several
chunk sizes over representative cases always, and over the entire
yaml-test-suite manifest (402 cases) behind PUREYAML_RUN_YAML_TEST_SUITE=1,
asserting the token sequence matches scanAll() exactly. That proof caught a
real bug: lastTokenForbidsInlineSequence() read indentEngine.tokens.last,
which goes empty once tokens are released incrementally, silently breaking
the key: - a rejection. Fixed by tracking IndentEngine.lastToken explicitly,
kept in sync with tokens.last so batch behavior (which never dequeues) is
unchanged.

Not yet wired into PureYAMLStreaming; scanStep() and its supporting types stay
internal until that integration is designed.

Also broaden the real-seed mutation property suite (#6) with the three
mutation classes it named but didn't yet cover: invalidControlCharacter,
mixedTabSpaceIndentation, and truncatedFlowCollection. The test now asserts
diagnostic quality, not just pass/fail: line must fall within the mutated
document, and column, when present, must be 1-based (some ParseError cases,
like tabIndentation, are line-only by design).
2026-07-09 23:38:39 +02:00

89 lines
3.5 KiB
Swift

extension PureYAML.Parsing.LibYAML {
/// Port of libyaml's input buffer and position tracking (the `buffer`/`mark`
/// fields with the `PEEK`, `SKIP`, and `SKIP_LINE` macros). It iterates
/// `Unicode.Scalar`s rather than grapheme clusters so a CR and an LF are
/// distinct, matching libyaml's byte-oriented scanning, and tracks the mark in
/// libyaml's 0-based line and column convention; conversion to the package's
/// 1-based `Mark` happens at token emission during integration.
struct Reader {
private var scalars: [Unicode.Scalar]
private(set) var index = 0
private(set) var line = 0
private(set) var column = 0
/// Whether the buffered `scalars` are the complete input. True for every
/// batch caller (the default), so `ranDry` can never be set and this type's
/// behavior is unchanged for `scanAll()`. A resumable caller passes `false`
/// and later calls `markFinal()` once the last chunk has been appended.
private(set) var isFinal: Bool
/// Set by `peek(_:)` when it was asked for a position past the buffered
/// scalars while `!isFinal`: the query is inconclusive, not a real end of
/// input, and the caller must suspend and retry after `append(_:)`. Batch
/// callers (`isFinal == true`) never set this, since the guard below is
/// unreachable when `isFinal` is true.
private(set) var ranDry = false
init(_ string: String = "", isFinal: Bool = true) {
scalars = Array(string.unicodeScalars)
self.isFinal = isFinal
}
/// `PEEK_TOKEN` / `CHECK_AT`: the scalar `offset` ahead, or nil past the
/// end (libyaml's `\0`). Past the buffered end while more chunks may still
/// arrive, this flags `ranDry` instead of asserting true end of input.
mutating func peek(_ offset: Int = 0) -> Unicode.Scalar? {
let position = index + offset
if position < scalars.count {
return scalars[position]
}
if !isFinal {
ranDry = true
}
return nil
}
/// Append a chunk of newly arrived input. Only meaningful before
/// `markFinal()`.
mutating func append(_ chunk: String) {
scalars.append(contentsOf: chunk.unicodeScalars)
}
/// Declare that no further chunks are coming: a `peek` past the buffered
/// end is now a real end of input, matching the batch reader's behavior.
mutating func markFinal() {
isFinal = true
}
var isAtEnd: Bool {
index >= scalars.count
}
/// The current position. Column is 0-based, as in libyaml.
var mark: PureYAML.Parsing.Mark {
PureYAML.Parsing.Mark(line: line, column: column, index: index)
}
/// `SKIP`: consume one non-break scalar, advancing the column.
mutating func skip() {
guard index < scalars.count else {
return
}
index += 1
column += 1
}
/// `SKIP_LINE`: consume a line break and start a new line. A CR+LF pair is
/// one break, matching libyaml.
mutating func skipLine() {
if peek() == "\r", peek(1) == "\n" {
index += 2
} else if CharacterClass.isBreak(peek()) {
index += 1
} else {
return
}
line += 1
column = 0
}
}
}