Mihaela Mihaljevic
18fdf9b69a
Add Scanner.scanStep(), the first increment of true chunked input streaming (#4): Reader gains isFinal/append(_:)/markFinal(), and peek(_:) flags ranDry instead of asserting end of input when the buffer runs out mid-fetch and more chunks may still arrive. scanStep() snapshots the scanner before each fetchNextToken() and restores it whenever a fetch runs dry, so a construct spanning a chunk boundary is retried after more input lands rather than misread as an early end of input. Every existing call site keeps the default isFinal: true, so scanAll() and batch parsing are unchanged; ranDry can never be set when isFinal is true, by construction. A token already queued is not always safe to release, since libyaml's simple-key mechanism can still splice a KEY token in front of it, so scanStep() ports libyaml's yaml_parser_fetch_more_tokens release condition. Proof: StreamingScannerDifferentialTests.swift drives scanStep() at several chunk sizes over representative cases always, and over the entire yaml-test-suite manifest (402 cases) behind PUREYAML_RUN_YAML_TEST_SUITE=1, asserting the token sequence matches scanAll() exactly. That proof caught a real bug: lastTokenForbidsInlineSequence() read indentEngine.tokens.last, which goes empty once tokens are released incrementally, silently breaking the key: - a rejection. Fixed by tracking IndentEngine.lastToken explicitly, kept in sync with tokens.last so batch behavior (which never dequeues) is unchanged. Not yet wired into PureYAMLStreaming; scanStep() and its supporting types stay internal until that integration is designed. Also broaden the real-seed mutation property suite (#6) with the three mutation classes it named but didn't yet cover: invalidControlCharacter, mixedTabSpaceIndentation, and truncatedFlowCollection. The test now asserts diagnostic quality, not just pass/fail: line must fall within the mutated document, and column, when present, must be 1-based (some ParseError cases, like tabIndentation, are line-only by design).
60 lines
2.5 KiB
Swift
60 lines
2.5 KiB
Swift
extension PureYAML.Parsing.LibYAML.Scanner {
|
|
private typealias ScannerError = PureYAML.Parsing.LibYAML.ScannerError
|
|
|
|
/// Reject a flow-collection continuation line that is not indented past the
|
|
/// enclosing block context. YAML requires `s-flow-line-prefix` to be more
|
|
/// indented than the parent block; libyaml does not enforce it, so multi-line
|
|
/// flow content in column 0 (`[a,\nb]` under a key) wrongly parses.
|
|
mutating func checkFlowContinuationIndent() throws {
|
|
guard flowLevel > 0, crossedLineBreak, let next = reader.peek() else {
|
|
return
|
|
}
|
|
// A closing `]`/`}` (or a `,` separator) may sit at any indentation; only
|
|
// flow CONTENT must be indented past the parent block.
|
|
if next == "]" || next == "}" || next == "," {
|
|
return
|
|
}
|
|
// A tab in the leading whitespace is never valid indentation, even where
|
|
// spaces of the same width would be (the tab inflates the column).
|
|
if reader.column <= indentEngine.indent || flowLeadingTab {
|
|
throw ScannerError(
|
|
context: "while parsing a flow node",
|
|
problem: "found insufficiently indented line in a flow collection",
|
|
mark: reader.mark,
|
|
)
|
|
}
|
|
}
|
|
|
|
/// Whether the last emitted token bars a block sequence from opening on the
|
|
/// same line: a mapping value indicator or a node property (anchor/tag). After
|
|
/// another block entry a compact sequence is allowed, so that is not barred.
|
|
func lastTokenForbidsInlineSequence() -> Bool {
|
|
switch indentEngine.lastToken?.type {
|
|
case .anchor, .tag:
|
|
true
|
|
case .value:
|
|
lastValueWasInline
|
|
default:
|
|
false
|
|
}
|
|
}
|
|
|
|
/// The error raised when a tab is used where indentation is required (before a
|
|
/// block indicator or an implicit mapping key). A tab is valid separation but
|
|
/// never indentation.
|
|
func tabIndentationError() -> PureYAML.Parsing.LibYAML.ScannerError {
|
|
ScannerError(
|
|
context: "while scanning for the next token",
|
|
problem: "found a tab character where an indentation space is expected",
|
|
mark: reader.mark,
|
|
)
|
|
}
|
|
|
|
/// Whether this flow level's pending simple key is a JSON (quoted) node, which
|
|
/// lets an adjacent `:` open its value without a separating space.
|
|
var flowKeyIsJSON: Bool {
|
|
let key = simpleKeys[simpleKeys.count - 1]
|
|
return key.possible && key.isJSON
|
|
}
|
|
}
|