fix(parsing): fold line breaks in multi-line double-quoted scalars

The double-quoted decoder appended real line breaks and continuation-line
indentation verbatim, so a multi-line "..." scalar kept its raw newlines
and leading whitespace instead of folding. It now applies YAML flow
folding at each unescaped break: a single break folds to a space, a blank
line to one newline, and whitespace adjacent to a break (the line's
trailing, the next line's leading) is stripped. First-line leading and
last-line trailing whitespace stay (they sit next to a quote, not a
break), and \n escapes are still decoded to literal newlines.

Measured against the official YAML test-suite, value-level conformance
rises from 135/169 (79.9%) to 143/169 (84.6%) with no regression.
Parse-level is unchanged: these documents already parsed, just to the
wrong tree, which only the value-level metric exposes.
This commit is contained in:
Mihaela M 2026-06-26 21:13:13 +02:00
parent 455cd07f4a
commit 08e1979085
3 changed files with 71 additions and 0 deletions

View File

@ -23,6 +23,14 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
- A multi-line double-quoted scalar now folds its line breaks instead of keeping them raw.
The decoder appended real line breaks and continuation-line indentation verbatim, so
`"a\n b"` came back as `"a\n b"` rather than `"a b"`. It now applies flow folding: a
single break folds to a space, a blank line to one newline, and whitespace adjacent to a
break (a line's trailing, the next line's leading) is stripped, while first-line leading
and last-line trailing whitespace and `\n` escapes are preserved. Lifts official test-suite
value-level conformance from 135/169 (79.9%) to 143/169 (84.6%), with no regression;
parse-level is unchanged because these documents already parsed, just to the wrong tree.
- A root node that begins on the document-marker line (`--- >`, `--- |`, `--- foo`,
`--- {a: 1}`) is no longer discarded. The marker scanner skipped the rest of the
`---` line unconditionally, so a block-scalar header was dropped and its body was

View File

@ -147,6 +147,8 @@ extension PureYAML.Parsing.Parser {
break
}
appendDoubleQuotedEscape(in: inner, index: &index, to: &output)
} else if character == "\n" || character == "\r" {
foldDoubleQuotedLineBreaks(in: inner, index: &index, to: &output)
} else {
output.append(character)
inner.formIndex(after: &index)
@ -155,6 +157,40 @@ extension PureYAML.Parsing.Parser {
return output
}
/// Flow line folding for a double-quoted scalar at an unescaped line break. Whitespace adjacent
/// to the break is stripped (the current line's trailing, each following line's leading), then a
/// single break folds to one space and a run of breaks (blank lines) folds to one fewer newline.
/// The scalar's first-line leading whitespace and last-line trailing whitespace are untouched,
/// since they sit next to a quote, not a break.
func foldDoubleQuotedLineBreaks(
in text: Substring,
index: inout Substring.Index,
to output: inout String,
) {
while let last = output.last, last == " " || last == "\t" {
output.removeLast()
}
var breaks = 0
while index < text.endIndex {
let character = text[index]
if character == "\r" {
text.formIndex(after: &index)
if index < text.endIndex, text[index] == "\n" {
text.formIndex(after: &index)
}
breaks += 1
} else if character == "\n" {
text.formIndex(after: &index)
breaks += 1
} else if character == " " || character == "\t" {
text.formIndex(after: &index)
} else {
break
}
}
output.append(breaks == 1 ? " " : String(repeating: "\n", count: breaks - 1))
}
func appendDoubleQuotedEscape(
in text: Substring,
index: inout Substring.Index,

View File

@ -0,0 +1,27 @@
@testable import PureYAML
import Testing
@Suite("Double-quoted scalar folding")
struct DoubleQuotedFoldingTests {
@Test("A multi-line double-quoted scalar folds breaks, blank lines, and adjacent whitespace")
func test_doubleQuotedLineFolding() throws {
// First-line leading and last-line trailing whitespace are kept (next to a quote, not a
// break); a single break folds to a space; a blank line folds to one newline; leading
// whitespace on continuation lines and trailing whitespace before a break are stripped.
let parsed = try PureYAML.parse("\" 1st non-empty\n\n 2nd non-empty \n\t3rd non-empty \"")
#expect(parsed == .string(" 1st non-empty\n2nd non-empty 3rd non-empty "))
}
@Test("An escaped newline stays literal while real breaks fold to a space")
func test_doubleQuotedEscapeVersusBreak() throws {
// A real line break between the two lines folds to a space; the trailing `\n` escape
// decodes to a literal newline.
let parsed = try PureYAML.parse("\"So does this\n quoted scalar.\\n\"")
#expect(parsed == .string("So does this quoted scalar.\n"))
}
@Test("A single-line double-quoted scalar is unchanged")
func test_doubleQuotedSingleLine() throws {
#expect(try PureYAML.parse("\"hello world\"") == .string("hello world"))
}
}