main
2 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
| 41ca600d93 |
feat(parsing): persist mid-token suspension and expose it as a public streaming API
Plain scalars now suspend and resume mid-token instead of rolling back to a checkpoint and rescanning from the token's start on every chunk boundary inside them: scanPlainScalar's loop is an explicit PlainScalarPhase state machine that throws PlainScalarSuspended the moment a lookahead runs dry, and scanStep() resumes it directly from Scanner.suspendedPlainScalar. Getting this right took four rounds of bugs, each caught by the existing whole-suite streaming differential test: checking ranDry only once at the loop's end instead of after every peek-dependent decision, resuming a dry-interrupted content or break run through a pre-check meant only for a fresh entry point, trusting dispatchToken's own dry-fooled document-marker check, and a nested blankLineAhead() lookahead independently going dry before an outer check could catch it. A synthetic long-scalar test (needed since the real-world benchmark fixture's scalars are too short to exercise this) then caught a fifth, genuine bug: the first working resumePlainScalar took the whole suspended value by parameter, which kept a reference to its accumulated string alive for the entire resume call and defeated copy-on-write, copying the whole scalar on every resume. That is O(N^2) in the scalar's length, and measured 29x slower than not suspending at all at small chunk sizes. Passing state inout directly, with the alias broken at the call site before the call, fixed it to 3-4x, verified genuinely linear (not just a smaller quadratic constant) by a permanent test scaling scalar length 16x at a fixed chunk size. Also exposes this scanner as public API for the first time: scanStep() and everything around it were internal-only, so no consumer outside this module, including the separate PureYAMLStreaming package that actually wants a resumable token scanner, could call it. PureYAML.Streaming.Scanner wraps scanStep() directly with no independent logic, translating its tokens into public Streaming.Token/Kind/ScalarStyle types that don't leak the libyaml port's internal naming. Proven separately from the internal scanner with its own differential test, since a wrong case mapping in the translation layer wouldn't be caught by a proof that never looks at the public types. Stops short of rewiring PureYAMLStreaming's own document-boundary scanner onto this; that is separate, cross-repo work with its own build and CI cycle. |
|||
| 18fdf9b69a |
feat(parsing): add a resumable scanner increment and broaden the mutation suite
Add Scanner.scanStep(), the first increment of true chunked input streaming (#4): Reader gains isFinal/append(_:)/markFinal(), and peek(_:) flags ranDry instead of asserting end of input when the buffer runs out mid-fetch and more chunks may still arrive. scanStep() snapshots the scanner before each fetchNextToken() and restores it whenever a fetch runs dry, so a construct spanning a chunk boundary is retried after more input lands rather than misread as an early end of input. Every existing call site keeps the default isFinal: true, so scanAll() and batch parsing are unchanged; ranDry can never be set when isFinal is true, by construction. A token already queued is not always safe to release, since libyaml's simple-key mechanism can still splice a KEY token in front of it, so scanStep() ports libyaml's yaml_parser_fetch_more_tokens release condition. Proof: StreamingScannerDifferentialTests.swift drives scanStep() at several chunk sizes over representative cases always, and over the entire yaml-test-suite manifest (402 cases) behind PUREYAML_RUN_YAML_TEST_SUITE=1, asserting the token sequence matches scanAll() exactly. That proof caught a real bug: lastTokenForbidsInlineSequence() read indentEngine.tokens.last, which goes empty once tokens are released incrementally, silently breaking the key: - a rejection. Fixed by tracking IndentEngine.lastToken explicitly, kept in sync with tokens.last so batch behavior (which never dequeues) is unchanged. Not yet wired into PureYAMLStreaming; scanStep() and its supporting types stay internal until that integration is designed. Also broaden the real-seed mutation property suite (#6) with the three mutation classes it named but didn't yet cover: invalidControlCharacter, mixedTabSpaceIndentation, and truncatedFlowCollection. The test now asserts diagnostic quality, not just pass/fail: line must fall within the mutated document, and column, when present, must be 1-based (some ParseError cases, like tabIndentation, are line-only by design). |
