fix(parsing): complete flow-collection key and empty-node handling
Implement libyaml's simple-key mechanism for flow context: the scanner records a potential key position before a plain or quoted scalar, an anchor/tag, or a [/{ opener, and a later ':' splices a mappingKey token retroactively in front of that node. This lets a quoted scalar ("k":v), a nested collection ({k}:v), or a property-carrying node (&c c: d) become a flow mapping key. In flow context ':' is always a value indicator and '?' always an explicit-key indicator regardless of the next character (the block-only following-blank lookahead is disabled), so adjacent forms and trailing '?' parse, and multi-line plain keys fold before the ':' is read. The flow parser is rebuilt around the flow_sequence_entry / flow_mapping_key / flow_mapping_value productions, composing empty scalars for absent keys and values. The streaming token cursor now buffers an open flow collection's tokens until it closes so the retroactive key splice stays valid.
Flips 10 official YAML test suite cases (348/402): 4FJ6, 8KB6, 9BXH, 9MMW, CN3R, DFF7, FRK4, NJ66, UT92, CT4Q. Flow collections used as block mapping keys (LX3P, 6BFJ) need the splice in block context and stay tracked. Updates manifest, fixtures, counts, and roadmap.
This commit is contained in:
parent
1875d66d46
commit
16da7248c7
20
CHANGELOG.md
20
CHANGELOG.md
@ -9,6 +9,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
### Fixed
|
||||
|
||||
- Complete flow-collection (`[...]`, `{...}`) parsing for keys, empty nodes, and
|
||||
multi-line scalars, fixing 10 official YAML test suite cases and removing them
|
||||
from the conformance gap manifest. The scanner now implements libyaml's
|
||||
simple-key mechanism inside flow context: a potential key position is recorded
|
||||
before a plain or quoted scalar, an anchor or tag, or a nested `[`/`{` opener,
|
||||
and a later `:` value indicator splices a `mappingKey` token in front of that
|
||||
node so a collection or property-carrying node can be a flow-mapping key. In
|
||||
flow context a `:` is now always a value indicator and a `?` always an explicit
|
||||
key indicator regardless of the following character (matching libyaml's
|
||||
`flow_level || IS_BLANKZ_AT` checks), so `"k":v`, `{k}:v`, and a trailing `?`
|
||||
parse correctly; the implicit-key forward lookahead is restricted to block
|
||||
context. A multi-line plain key folds in full before its `:` is read. The flow
|
||||
parser was rebuilt around libyaml's flow productions (`flow_sequence_entry`,
|
||||
`flow_mapping_key`/`value`), composing empty scalar keys and values where
|
||||
absent. The streaming token cursor holds an open flow collection's tokens until
|
||||
it closes so the retroactive key splice stays valid. Fixed false-rejects 4FJ6,
|
||||
8KB6, 9BXH, 9MMW, CN3R, DFF7, FRK4, NJ66, UT92 and corrected values for CT4Q.
|
||||
The flow-node-as-block-mapping-key cases LX3P and 6BFJ remain tracked (they need
|
||||
the block-context simple-key splice, a separate change).
|
||||
|
||||
- Parse multi-line plain scalars and block scalars at the document root, fixing 35
|
||||
official YAML test suite cases and removing them from the conformance gap
|
||||
manifest. The scanner now tracks the enclosing block-collection indent
|
||||
|
||||
@ -8,4 +8,15 @@ extension PureYAML.Parsing {
|
||||
var kind: ScannerFlowContextKind
|
||||
var isExpectingKey: Bool
|
||||
}
|
||||
|
||||
/// A position that may be promoted to a flow-mapping key, mirroring libyaml's
|
||||
/// `yaml_simple_key_t`. When a `:` value indicator is later found, the scanner
|
||||
/// inserts a ``TokenKind/mappingKey`` token at ``tokenIndex`` so a node of any
|
||||
/// kind (plain or quoted scalar, or a nested `[...]`/`{...}` collection) that
|
||||
/// precedes the `:` becomes that pair's key. Only used inside flow collections;
|
||||
/// block context promotes keys with a forward lookahead instead.
|
||||
struct ScannerFlowSimpleKey {
|
||||
var tokenIndex: Int
|
||||
var mark: PureYAML.Parsing.Mark
|
||||
}
|
||||
}
|
||||
|
||||
105
Sources/Parsing/Scanner.FlowSimpleKey.swift
Normal file
105
Sources/Parsing/Scanner.FlowSimpleKey.swift
Normal file
@ -0,0 +1,105 @@
|
||||
extension PureYAML.Parsing.Scanner {
|
||||
func scanFlowToken(
|
||||
_ character: Character,
|
||||
state: inout State,
|
||||
) -> Bool {
|
||||
switch character {
|
||||
case "[":
|
||||
openFlowCollection(kind: .sequence, token: .flowSequenceStart, state: &state)
|
||||
return true
|
||||
case "]":
|
||||
closeFlowCollection(token: .flowSequenceEnd, state: &state)
|
||||
return true
|
||||
case "{":
|
||||
openFlowCollection(kind: .mapping, token: .flowMappingStart, state: &state)
|
||||
return true
|
||||
case "}":
|
||||
closeFlowCollection(token: .flowMappingEnd, state: &state)
|
||||
return true
|
||||
case ",":
|
||||
// A flow entry separator abandons any pending simple key and reopens the
|
||||
// door for a fresh one on the next entry (libyaml fetch_flow_entry).
|
||||
removeFlowSimpleKey(&state)
|
||||
scanFixedToken(.flowEntry, state: &state)
|
||||
state.simpleKeyAllowed = true
|
||||
if state.flowContexts.last?.kind == .mapping {
|
||||
state.flowContexts[state.flowContexts.count - 1].isExpectingKey = true
|
||||
}
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
/// Opens a flow collection (`[` or `{`). The opener itself may be a simple key
|
||||
/// at the enclosing level, so a key is saved before the flow level rises; a
|
||||
/// fresh simple-key slot is then pushed and a key is allowed inside.
|
||||
func openFlowCollection(
|
||||
kind: PureYAML.Parsing.ScannerFlowContextKind,
|
||||
token: PureYAML.Parsing.TokenKind,
|
||||
state: inout State,
|
||||
) {
|
||||
saveFlowSimpleKey(&state)
|
||||
state.flowDepth += 1
|
||||
state.flowContexts.append(.init(kind: kind, isExpectingKey: kind == .mapping))
|
||||
state.flowSimpleKeys.append(nil)
|
||||
scanFixedToken(token, state: &state)
|
||||
state.simpleKeyAllowed = true
|
||||
}
|
||||
|
||||
/// Closes a flow collection (`]` or `}`): the pending simple key on this level is
|
||||
/// dropped, the level is popped, and no simple key may immediately follow the
|
||||
/// closing indicator (libyaml fetch_flow_collection_end).
|
||||
func closeFlowCollection(
|
||||
token: PureYAML.Parsing.TokenKind,
|
||||
state: inout State,
|
||||
) {
|
||||
removeFlowSimpleKey(&state)
|
||||
state.flowDepth = max(0, state.flowDepth - 1)
|
||||
scanFixedToken(token, state: &state)
|
||||
_ = state.flowContexts.popLast()
|
||||
_ = state.flowSimpleKeys.popLast()
|
||||
state.simpleKeyAllowed = false
|
||||
}
|
||||
|
||||
/// Records a potential simple key at the current reader position for the
|
||||
/// innermost flow level, if a simple key is allowed there. Mirrors libyaml's
|
||||
/// `yaml_parser_save_simple_key`: the saved key remembers the index of the
|
||||
/// next token to be appended, so a later `:` can splice a ``mappingKey`` in
|
||||
/// front of the node that starts here. No-op outside flow context.
|
||||
func saveFlowSimpleKey(_ state: inout State) {
|
||||
guard state.flowDepth > 0, state.simpleKeyAllowed, !state.flowSimpleKeys.isEmpty else {
|
||||
return
|
||||
}
|
||||
state.flowSimpleKeys[state.flowSimpleKeys.count - 1] = PureYAML.Parsing.ScannerFlowSimpleKey(
|
||||
tokenIndex: state.tokens.count,
|
||||
mark: state.reader.mark,
|
||||
)
|
||||
}
|
||||
|
||||
/// Discards any potential simple key on the innermost flow level. Mirrors
|
||||
/// libyaml's `yaml_parser_remove_simple_key` for the flow context, where a
|
||||
/// simple key is never *required*, so abandoning it is not an error.
|
||||
func removeFlowSimpleKey(_ state: inout State) {
|
||||
guard !state.flowSimpleKeys.isEmpty else {
|
||||
return
|
||||
}
|
||||
state.flowSimpleKeys[state.flowSimpleKeys.count - 1] = nil
|
||||
}
|
||||
|
||||
/// Promotes a pending flow simple key to an actual mapping key by inserting a
|
||||
/// ``TokenKind/mappingKey`` token at the saved index, mirroring libyaml's
|
||||
/// `fetch_value` splicing a KEY token before a simple key. Returns true when a
|
||||
/// key was inserted so the caller can avoid emitting a redundant one.
|
||||
@discardableResult
|
||||
func promoteFlowSimpleKey(_ state: inout State) -> Bool {
|
||||
guard !state.flowSimpleKeys.isEmpty, let key = state.flowSimpleKeys[state.flowSimpleKeys.count - 1] else {
|
||||
return false
|
||||
}
|
||||
let token = PureYAML.Parsing.Token(kind: .mappingKey, mark: key.mark, endMark: key.mark)
|
||||
let index = min(key.tokenIndex, state.tokens.count)
|
||||
state.tokens.insert(token, at: index)
|
||||
state.flowSimpleKeys[state.flowSimpleKeys.count - 1] = nil
|
||||
return true
|
||||
}
|
||||
}
|
||||
@ -163,7 +163,11 @@ extension PureYAML.Parsing.Scanner {
|
||||
var continuationState = state
|
||||
continuationState.reader = probe
|
||||
let continuation = scanPlainScalarContinuationLine(&continuationState)
|
||||
guard !isAtMappingValueIndicator(state: continuationState) else {
|
||||
// In block context a continuation line that ends at a `:` is the next
|
||||
// mapping key, not part of this scalar, so stop folding. In flow context
|
||||
// a multi-line plain key folds in full and the trailing `:` then makes
|
||||
// the folded text the key (libyaml folds first, promotes the key after).
|
||||
guard state.flowDepth > 0 || !isAtMappingValueIndicator(state: continuationState) else {
|
||||
return value
|
||||
}
|
||||
|
||||
@ -315,6 +319,12 @@ extension PureYAML.Parsing.Scanner {
|
||||
start: PureYAML.Parsing.Mark,
|
||||
state: inout State,
|
||||
) -> Bool {
|
||||
// Flow-context keys are promoted by the simple-key mechanism (a `:` value
|
||||
// indicator splices the `mappingKey` in retroactively), so the block-only
|
||||
// forward lookahead must not also fire here.
|
||||
guard state.flowDepth == 0 else {
|
||||
return false
|
||||
}
|
||||
guard isAtMappingValueIndicator(state: state) || isAtSpacedMappingValueIndicator(state: state) else {
|
||||
return false
|
||||
}
|
||||
|
||||
@ -5,6 +5,11 @@ extension PureYAML.Parsing.Scanner {
|
||||
state: inout State,
|
||||
) throws {
|
||||
let start = state.reader.mark
|
||||
// An anchor or alias may itself begin a flow simple key, so record the
|
||||
// position before the token is appended; the following node then does not
|
||||
// start a new key, leaving `&anchor key :` anchored at the `&` (libyaml's
|
||||
// fetch_anchor saves the simple key before clearing simple_key_allowed).
|
||||
saveFlowSimpleKey(&state)
|
||||
state.reader.advance()
|
||||
// An anchor or alias name is a run of name characters that ends at the
|
||||
// first non-name character, mirroring `yaml_parser_scan_anchor`: the
|
||||
@ -22,6 +27,8 @@ extension PureYAML.Parsing.Scanner {
|
||||
|
||||
func scanTag(_ state: inout State) throws {
|
||||
let start = state.reader.mark
|
||||
// A tag may itself begin a flow simple key (see `scanNamedToken`).
|
||||
saveFlowSimpleKey(&state)
|
||||
state.reader.advance()
|
||||
var value = "!"
|
||||
if state.reader.peek() == "<" {
|
||||
|
||||
@ -81,6 +81,11 @@ extension PureYAML.Parsing.Scanner {
|
||||
/// positions provides block indentation and is rejected. See scanner.c:1948.
|
||||
var simpleKeyAllowed = true
|
||||
var flowContexts: [PureYAML.Parsing.ScannerFlowContext] = []
|
||||
/// A stack of potential simple keys, one entry per open flow collection,
|
||||
/// mirroring libyaml's `simple_keys`. The slot for the innermost flow level
|
||||
/// records the most recent position a `:` could promote to a mapping key.
|
||||
/// Empty outside flow context; block keys do not use this stack.
|
||||
var flowSimpleKeys: [PureYAML.Parsing.ScannerFlowSimpleKey?] = []
|
||||
var validImplicitIndentation: Set<Int> = []
|
||||
var tagHandles = [
|
||||
"!": "!",
|
||||
@ -162,52 +167,32 @@ extension PureYAML.Parsing.Scanner {
|
||||
state.simpleKeyAllowed = true
|
||||
}
|
||||
return true
|
||||
case "?" where isIndicatorBoundary(state.reader.peek(offset: 1)):
|
||||
case "?" where state.flowDepth > 0 || isIndicatorBoundary(state.reader.peek(offset: 1)):
|
||||
// In flow context a `?` is always an explicit-key indicator (libyaml's
|
||||
// fetch_more_tokens checks `flow_level || IS_BLANKZ_AT`); a pending
|
||||
// simple key is abandoned and no implicit key may follow on this entry.
|
||||
if state.flowDepth > 0 {
|
||||
removeFlowSimpleKey(&state)
|
||||
}
|
||||
scanFixedToken(.mappingKey, state: &state)
|
||||
if state.flowDepth == 0 {
|
||||
state.simpleKeyAllowed = true
|
||||
} else {
|
||||
state.simpleKeyAllowed = false
|
||||
}
|
||||
return true
|
||||
case ":" where isAtMappingValueIndicator(state: state):
|
||||
case ":" where state.flowDepth > 0 || isAtMappingValueIndicator(state: state):
|
||||
// In flow context a `:` is always a value indicator regardless of what
|
||||
// follows it (libyaml), so `"k":v` and `{k}:v` are key/value pairs. A
|
||||
// pending simple key is spliced into a `mappingKey` before its node.
|
||||
if state.flowDepth > 0 {
|
||||
promoteFlowSimpleKey(&state)
|
||||
}
|
||||
scanFixedToken(.mappingValue, state: &state)
|
||||
if state.flowDepth == 0 {
|
||||
state.simpleKeyAllowed = true
|
||||
}
|
||||
return true
|
||||
default:
|
||||
return false
|
||||
}
|
||||
}
|
||||
|
||||
func scanFlowToken(
|
||||
_ character: Character,
|
||||
state: inout State,
|
||||
) -> Bool {
|
||||
switch character {
|
||||
case "[":
|
||||
state.flowDepth += 1
|
||||
state.flowContexts.append(.init(kind: .sequence, isExpectingKey: false))
|
||||
scanFixedToken(.flowSequenceStart, state: &state)
|
||||
return true
|
||||
case "]":
|
||||
state.flowDepth = max(0, state.flowDepth - 1)
|
||||
scanFixedToken(.flowSequenceEnd, state: &state)
|
||||
_ = state.flowContexts.popLast()
|
||||
return true
|
||||
case "{":
|
||||
state.flowDepth += 1
|
||||
state.flowContexts.append(.init(kind: .mapping, isExpectingKey: true))
|
||||
scanFixedToken(.flowMappingStart, state: &state)
|
||||
return true
|
||||
case "}":
|
||||
state.flowDepth = max(0, state.flowDepth - 1)
|
||||
scanFixedToken(.flowMappingEnd, state: &state)
|
||||
_ = state.flowContexts.popLast()
|
||||
return true
|
||||
case ",":
|
||||
scanFixedToken(.flowEntry, state: &state)
|
||||
if state.flowContexts.last?.kind == .mapping {
|
||||
state.flowContexts[state.flowContexts.count - 1].isExpectingKey = true
|
||||
} else {
|
||||
state.simpleKeyAllowed = false
|
||||
}
|
||||
return true
|
||||
default:
|
||||
@ -314,6 +299,9 @@ extension PureYAML.Parsing.Scanner {
|
||||
state: inout State,
|
||||
) throws {
|
||||
let start = state.reader.mark
|
||||
// A quoted scalar may be a flow simple key: record its position before the
|
||||
// content is consumed so a following `:` can splice a key in front of it.
|
||||
saveFlowSimpleKey(&state)
|
||||
let quote: Character = style == .doubleQuoted ? "\"" : "'"
|
||||
state.reader.advance()
|
||||
var value = ""
|
||||
@ -352,6 +340,9 @@ extension PureYAML.Parsing.Scanner {
|
||||
|
||||
func scanPlainScalar(_ state: inout State) {
|
||||
let start = state.reader.mark
|
||||
// A plain scalar may be a flow simple key: record its position before the
|
||||
// content is consumed so a following `:` can splice a key in front of it.
|
||||
saveFlowSimpleKey(&state)
|
||||
var value = ""
|
||||
while let character = state.reader.peek() {
|
||||
if shouldEndPlainScalar(character, state: state, accumulated: value) {
|
||||
|
||||
@ -77,7 +77,12 @@ extension PureYAML.Parsing {
|
||||
return
|
||||
}
|
||||
|
||||
while scannerBox.state.tokens.isEmpty {
|
||||
// Scan at least one token, and keep scanning without draining while a
|
||||
// flow collection is open: a flow `:` may splice a `mappingKey` into an
|
||||
// already-scanned earlier token (the simple-key mechanism), so the whole
|
||||
// flow collection's tokens must stay in `state.tokens` until it closes
|
||||
// for the retroactive index to remain valid.
|
||||
while scannerBox.state.tokens.isEmpty || scannerBox.state.flowDepth > 0 {
|
||||
guard !scannerBox.state.reader.isAtEnd else {
|
||||
if scannerBox.state.hasPendingDirective {
|
||||
throw PureYAML.Parsing.ParseError.directiveWithoutDocument(
|
||||
|
||||
@ -1,4 +1,8 @@
|
||||
extension PureYAML.Parsing.TokenEventParser {
|
||||
/// Parses a flow sequence, mirroring libyaml's `flow_sequence_entry`
|
||||
/// productions. Each entry is either a flow node or, when introduced by a
|
||||
/// `mappingKey` (`?`) or recognized as a single-pair `key: value`, a one-entry
|
||||
/// flow mapping nested directly in the sequence.
|
||||
mutating func parseFlowSequence(
|
||||
properties: PureYAML.Parsing.NodeProperties,
|
||||
) throws -> PureYAML.Parsing.NodeResult {
|
||||
@ -32,7 +36,7 @@ extension PureYAML.Parsing.TokenEventParser {
|
||||
_ = try cursor.advance()
|
||||
continue
|
||||
}
|
||||
endMark = try parseNode().endMark
|
||||
endMark = try parseFlowSequenceEntry().endMark
|
||||
if cursor.current?.kind.isFlowTerminator == false {
|
||||
throw unexpectedToken(expected: "flow entry or flow sequence end")
|
||||
}
|
||||
@ -40,6 +44,28 @@ extension PureYAML.Parsing.TokenEventParser {
|
||||
throw unexpectedToken(expected: "flow sequence end")
|
||||
}
|
||||
|
||||
/// Parses a single flow-sequence entry. A `mappingKey` (`?`, or one spliced in
|
||||
/// by the simple-key mechanism before a `key: value` pair) produces a
|
||||
/// single-pair flow mapping wrapping the entry; otherwise the entry is a plain
|
||||
/// flow node.
|
||||
mutating func parseFlowSequenceEntry() throws -> PureYAML.Parsing.NodeResult {
|
||||
if cursor.current?.kind.isMappingKey == true {
|
||||
return try parseFlowSequenceSinglePair()
|
||||
}
|
||||
return try parseNode()
|
||||
}
|
||||
|
||||
/// Wraps an explicit-`?` single-pair entry of a flow sequence in a one-entry
|
||||
/// flow mapping, mirroring libyaml's `flow_sequence_entry_mapping_*` states.
|
||||
mutating func parseFlowSequenceSinglePair() throws -> PureYAML.Parsing.NodeResult {
|
||||
let keyIndicator = try expect("mapping key") { $0.isMappingKey }
|
||||
try emit(.mappingStart(anchor: nil, tag: nil, style: .flow, mark: keyIndicator.mark))
|
||||
try parseFlowSinglePairKeyValue(keyIndicator: keyIndicator)
|
||||
let mark = cursor.current?.mark ?? keyIndicator.endMark
|
||||
try emit(.mappingEnd(mark: mark))
|
||||
return PureYAML.Parsing.NodeResult(kind: .mapping, endMark: mark)
|
||||
}
|
||||
|
||||
mutating func parseFlowMapping(
|
||||
properties: PureYAML.Parsing.NodeProperties,
|
||||
) throws -> PureYAML.Parsing.NodeResult {
|
||||
@ -81,37 +107,38 @@ extension PureYAML.Parsing.TokenEventParser {
|
||||
throw unexpectedToken(expected: "flow mapping end")
|
||||
}
|
||||
|
||||
/// Parses one flow-mapping entry. With a `mappingKey` (`?`) indicator the key
|
||||
/// follows the explicit form; without one the entry is a `flow_node` that is
|
||||
/// itself the key (carrying an empty value unless a `:` provides one).
|
||||
mutating func parseFlowMappingPair() throws -> PureYAML.Parsing.NodeResult {
|
||||
if cursor.current?.kind.isMappingKey == true {
|
||||
return try parseMappingPair()
|
||||
let keyIndicator = try expect("mapping key") { $0.isMappingKey }
|
||||
return try parseFlowSinglePairKeyValue(keyIndicator: keyIndicator)
|
||||
}
|
||||
try parseFlowMappingImplicitKey()
|
||||
return try parseFlowMappingValue()
|
||||
}
|
||||
|
||||
try parseFlowMappingPairKey()
|
||||
|
||||
guard cursor.current?.kind.isMappingValue == true else {
|
||||
return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: cursor.previous?.endMark ?? .start)
|
||||
/// Parses the key and value of a flow pair introduced by an explicit `?`,
|
||||
/// where each side may be absent (composing an empty scalar). Shared by flow
|
||||
/// mappings and single-pair entries nested in a flow sequence.
|
||||
@discardableResult
|
||||
mutating func parseFlowSinglePairKeyValue(
|
||||
keyIndicator: PureYAML.Parsing.Token,
|
||||
) throws -> PureYAML.Parsing.NodeResult {
|
||||
if flowKeyIsEmpty() {
|
||||
try emitEmptyScalar(properties: .none, mark: keyIndicator.endMark)
|
||||
} else {
|
||||
_ = try parseNode()
|
||||
}
|
||||
let valueIndicator = try expect("mapping value") { kind in
|
||||
if case .mappingValue = kind {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
let properties = try consumeProperties()
|
||||
guard properties != .none || canBeginNode(cursor.current), cursor.current?.kind.isFlowTerminator == false else {
|
||||
let mark = valueIndicator.endMark
|
||||
try emitEmptyScalar(properties: properties, mark: mark)
|
||||
return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: mark)
|
||||
}
|
||||
return try parseNode(properties: properties)
|
||||
return try parseFlowMappingValue()
|
||||
}
|
||||
|
||||
/// Parses the key of an implicit flow-mapping entry (`key : value` with no
|
||||
/// explicit `?`). The key may be absent entirely (`{ : v }`, `{ , : v }`)
|
||||
/// or be only a tag/anchor with no content, both composing to an empty
|
||||
/// scalar key rather than routing into a block mapping.
|
||||
mutating func parseFlowMappingPairKey() throws {
|
||||
mutating func parseFlowMappingImplicitKey() throws {
|
||||
let properties = try consumeProperties()
|
||||
if properties == .none, cursor.current?.kind.isMappingValue == true {
|
||||
let mark = cursor.current?.mark ?? cursor.previous?.endMark ?? .start
|
||||
@ -132,4 +159,38 @@ extension PureYAML.Parsing.TokenEventParser {
|
||||
}
|
||||
_ = try parseNode(properties: properties)
|
||||
}
|
||||
|
||||
/// Parses the value half of a flow-mapping entry. A `:` consumes its node (or an
|
||||
/// empty scalar when the value is absent); without a `:` the entry has an
|
||||
/// implicit empty value (libyaml's `flow_mapping_value` with `empty == 1`).
|
||||
mutating func parseFlowMappingValue() throws -> PureYAML.Parsing.NodeResult {
|
||||
guard cursor.current?.kind.isMappingValue == true else {
|
||||
let mark = cursor.current?.mark ?? cursor.previous?.endMark ?? .start
|
||||
try emitEmptyScalar(properties: .none, mark: mark)
|
||||
return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: mark)
|
||||
}
|
||||
let valueIndicator = try expect("mapping value") { kind in
|
||||
if case .mappingValue = kind {
|
||||
return true
|
||||
}
|
||||
return false
|
||||
}
|
||||
let properties = try consumeProperties()
|
||||
guard properties != .none || canBeginNode(cursor.current), cursor.current?.kind.isFlowTerminator == false else {
|
||||
let mark = valueIndicator.endMark
|
||||
try emitEmptyScalar(properties: properties, mark: mark)
|
||||
return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: mark)
|
||||
}
|
||||
return try parseNode(properties: properties)
|
||||
}
|
||||
|
||||
/// True when the key position right after a `?` indicator is empty, i.e. the
|
||||
/// next token is a `:` value indicator, a flow terminator, or the end of input,
|
||||
/// so an empty scalar key must be composed (libyaml's `flow_mapping_key`).
|
||||
func flowKeyIsEmpty() -> Bool {
|
||||
guard let current = cursor.current else {
|
||||
return true
|
||||
}
|
||||
return current.kind.isMappingValue || current.kind.isFlowTerminator
|
||||
}
|
||||
}
|
||||
|
||||
@ -46,15 +46,6 @@ cases:
|
||||
current: accepted
|
||||
byteCount: 27
|
||||
lineCount: 3
|
||||
- id: "4FJ6"
|
||||
name: "Nested implicit complex keys"
|
||||
localPath: yaml-test-suite/4FJ6/in.yaml
|
||||
category: false-reject
|
||||
expected: parse-ok
|
||||
oracle: none
|
||||
current: parse-error
|
||||
byteCount: 37
|
||||
lineCount: 4
|
||||
- id: "4JVG"
|
||||
name: "Scalar value with two anchors"
|
||||
localPath: yaml-test-suite/4JVG/in.yaml
|
||||
@ -127,15 +118,6 @@ cases:
|
||||
current: value-mismatch
|
||||
byteCount: 58
|
||||
lineCount: 6
|
||||
- id: "8KB6"
|
||||
name: "Multiline plain flow mapping key without value"
|
||||
localPath: yaml-test-suite/8KB6/in.yaml
|
||||
category: false-reject
|
||||
expected: parse-ok
|
||||
oracle: json
|
||||
current: parse-error
|
||||
byteCount: 51
|
||||
lineCount: 4
|
||||
- id: "8XDJ"
|
||||
name: "Comment in plain multiline value"
|
||||
localPath: yaml-test-suite/8XDJ/in.yaml
|
||||
@ -145,15 +127,6 @@ cases:
|
||||
current: accepted
|
||||
byteCount: 26
|
||||
lineCount: 3
|
||||
- id: "9BXH"
|
||||
name: "Multiline doublequoted flow mapping key without value"
|
||||
localPath: yaml-test-suite/9BXH/in.yaml
|
||||
category: false-reject
|
||||
expected: parse-ok
|
||||
oracle: json
|
||||
current: parse-error
|
||||
byteCount: 55
|
||||
lineCount: 4
|
||||
- id: "9C9N"
|
||||
name: "Wrong indented flow sequence"
|
||||
localPath: yaml-test-suite/9C9N/in.yaml
|
||||
@ -181,15 +154,6 @@ cases:
|
||||
current: accepted
|
||||
byteCount: 18
|
||||
lineCount: 2
|
||||
- id: "9MMW"
|
||||
name: "Single Pair Implicit Entries"
|
||||
localPath: yaml-test-suite/9MMW/in.yaml
|
||||
category: false-reject
|
||||
expected: parse-ok
|
||||
oracle: none
|
||||
current: parse-error
|
||||
byteCount: 77
|
||||
lineCount: 3
|
||||
- id: "BF9H"
|
||||
name: "Trailing comment in multiline plain scalar"
|
||||
localPath: yaml-test-suite/BF9H/in.yaml
|
||||
@ -199,24 +163,6 @@ cases:
|
||||
current: accepted
|
||||
byteCount: 47
|
||||
lineCount: 4
|
||||
- id: "CN3R"
|
||||
name: "Various location of anchors in flow sequence"
|
||||
localPath: yaml-test-suite/CN3R/in.yaml
|
||||
category: false-reject
|
||||
expected: parse-ok
|
||||
oracle: json
|
||||
current: parse-error
|
||||
byteCount: 57
|
||||
lineCount: 6
|
||||
- id: "CT4Q"
|
||||
name: "Spec Example 7.20. Single Pair Explicit Entry"
|
||||
localPath: yaml-test-suite/CT4Q/in.yaml
|
||||
category: json-mismatch
|
||||
expected: parse-ok
|
||||
oracle: json
|
||||
current: value-mismatch
|
||||
byteCount: 21
|
||||
lineCount: 4
|
||||
- id: "CTN5"
|
||||
name: "Flow sequence with invalid extra comma"
|
||||
localPath: yaml-test-suite/CTN5/in.yaml
|
||||
@ -262,15 +208,6 @@ cases:
|
||||
current: value-mismatch
|
||||
byteCount: 26
|
||||
lineCount: 4
|
||||
- id: "DFF7"
|
||||
name: "Spec Example 7.16. Flow Mapping Entries"
|
||||
localPath: yaml-test-suite/DFF7/in.yaml
|
||||
category: false-reject
|
||||
expected: parse-ok
|
||||
oracle: none
|
||||
current: parse-error
|
||||
byteCount: 42
|
||||
lineCount: 5
|
||||
- id: "DK95/01"
|
||||
name: "Tabs that look like indentation"
|
||||
localPath: yaml-test-suite/DK95/01/in.yaml
|
||||
@ -289,15 +226,6 @@ cases:
|
||||
current: parse-error
|
||||
byteCount: 51
|
||||
lineCount: 5
|
||||
- id: "FRK4"
|
||||
name: "Spec Example 7.3. Completely Empty Flow Nodes"
|
||||
localPath: yaml-test-suite/FRK4/in.yaml
|
||||
category: false-reject
|
||||
expected: parse-ok
|
||||
oracle: none
|
||||
current: parse-error
|
||||
byteCount: 24
|
||||
lineCount: 4
|
||||
- id: "G5U8"
|
||||
name: "Plain dashes in flow sequence"
|
||||
localPath: yaml-test-suite/G5U8/in.yaml
|
||||
@ -406,15 +334,6 @@ cases:
|
||||
current: value-mismatch
|
||||
byteCount: 31
|
||||
lineCount: 5
|
||||
- id: "NJ66"
|
||||
name: "Multiline plain flow mapping key"
|
||||
localPath: yaml-test-suite/NJ66/in.yaml
|
||||
category: false-reject
|
||||
expected: parse-ok
|
||||
oracle: json
|
||||
current: parse-error
|
||||
byteCount: 53
|
||||
lineCount: 4
|
||||
- id: "P2EQ"
|
||||
name: "Invalid sequene item on same line as previous item"
|
||||
localPath: yaml-test-suite/P2EQ/in.yaml
|
||||
@ -505,15 +424,6 @@ cases:
|
||||
current: parse-error
|
||||
byteCount: 4
|
||||
lineCount: 1
|
||||
- id: "UT92"
|
||||
name: "Spec Example 9.4. Explicit Documents"
|
||||
localPath: yaml-test-suite/UT92/in.yaml
|
||||
category: false-reject
|
||||
expected: parse-ok
|
||||
oracle: json
|
||||
current: parse-error
|
||||
byteCount: 43
|
||||
lineCount: 7
|
||||
- id: "UV7Q"
|
||||
name: "Legal tab after indentation"
|
||||
localPath: yaml-test-suite/UV7Q/in.yaml
|
||||
|
||||
@ -1,4 +0,0 @@
|
||||
---
|
||||
[
|
||||
[ a, [ [[b,c]]: d, e]]: 23
|
||||
]
|
||||
@ -1,10 +0,0 @@
|
||||
[
|
||||
{
|
||||
"single line": null,
|
||||
"a": "b"
|
||||
},
|
||||
{
|
||||
"multi line": null,
|
||||
"a": "b"
|
||||
}
|
||||
]
|
||||
@ -1,4 +0,0 @@
|
||||
---
|
||||
- { single line, a: b}
|
||||
- { multi
|
||||
line, a: b}
|
||||
@ -1,10 +0,0 @@
|
||||
[
|
||||
{
|
||||
"single line": null,
|
||||
"a": "b"
|
||||
},
|
||||
{
|
||||
"multi line": null,
|
||||
"a": "b"
|
||||
}
|
||||
]
|
||||
@ -1,4 +0,0 @@
|
||||
---
|
||||
- { "single line", a: b}
|
||||
- { "multi
|
||||
line", a: b}
|
||||
@ -1,3 +0,0 @@
|
||||
- [ YAML : separate ]
|
||||
- [ "JSON like":adjacent ]
|
||||
- [ {JSON: like}:adjacent ]
|
||||
@ -1,14 +0,0 @@
|
||||
[
|
||||
{
|
||||
"a": "b"
|
||||
},
|
||||
{
|
||||
"c": "d"
|
||||
},
|
||||
{
|
||||
"e": "f"
|
||||
},
|
||||
{
|
||||
"g": "h"
|
||||
}
|
||||
]
|
||||
@ -1,6 +0,0 @@
|
||||
&flowseq [
|
||||
a: b,
|
||||
&c c: d,
|
||||
{ &e e: f },
|
||||
&g { g: h }
|
||||
]
|
||||
@ -1,5 +0,0 @@
|
||||
[
|
||||
{
|
||||
"foo bar": "baz"
|
||||
}
|
||||
]
|
||||
@ -1,4 +0,0 @@
|
||||
[
|
||||
? foo
|
||||
bar : baz
|
||||
]
|
||||
@ -1,5 +0,0 @@
|
||||
{
|
||||
? explicit: entry,
|
||||
implicit: entry,
|
||||
?
|
||||
}
|
||||
@ -1,4 +0,0 @@
|
||||
{
|
||||
? foo :,
|
||||
: bar,
|
||||
}
|
||||
@ -1,8 +0,0 @@
|
||||
[
|
||||
{
|
||||
"single line": "value"
|
||||
},
|
||||
{
|
||||
"multi line": "value"
|
||||
}
|
||||
]
|
||||
@ -1,4 +0,0 @@
|
||||
---
|
||||
- { single line: value}
|
||||
- { multi
|
||||
line: value}
|
||||
@ -1,4 +0,0 @@
|
||||
{
|
||||
"matches %": 20
|
||||
}
|
||||
null
|
||||
@ -1,7 +0,0 @@
|
||||
---
|
||||
{ matches
|
||||
% : 20 }
|
||||
...
|
||||
---
|
||||
# Empty
|
||||
...
|
||||
@ -18,9 +18,9 @@ struct YAMLTestSuiteGapsTests {
|
||||
let manifest = try YAMLTestSuiteManifest.load()
|
||||
|
||||
#expect(manifest.version == 1)
|
||||
#expect(manifest.cases.count == 64)
|
||||
#expect(manifest.cases.count(where: { $0.category == "false-reject" }) == 18)
|
||||
#expect(manifest.cases.count(where: { $0.category == "json-mismatch" }) == 10)
|
||||
#expect(manifest.cases.count == 54)
|
||||
#expect(manifest.cases.count(where: { $0.category == "false-reject" }) == 9)
|
||||
#expect(manifest.cases.count(where: { $0.category == "json-mismatch" }) == 9)
|
||||
#expect(manifest.cases.count(where: { $0.category == "accepted-invalid" }) == 36)
|
||||
#expect(Set(manifest.cases.map(\.id)).count == manifest.cases.count)
|
||||
|
||||
|
||||
@ -24,10 +24,11 @@ its own oracle:
|
||||
fail; after bucket 1 (block scalars, +24), bucket 2 (quoted-scalar folding, +30),
|
||||
bucket 3 (empty nodes and empty keys, +20), bucket 4 (anchors, tags, and
|
||||
aliases as mapping keys, +10), bucket 5 (directives and document boundaries,
|
||||
+15), bucket 6 (stricter rejection of invalid input, +15), and bucket 7
|
||||
(document-root multi-line plain and block scalars, +33) it is now
|
||||
**338 pass, 64 fail.**
|
||||
The 64 remaining failing cases are vendored under
|
||||
+15), bucket 6 (stricter rejection of invalid input, +15), bucket 7
|
||||
(document-root multi-line plain and block scalars, +33), and bucket 8
|
||||
(flow-collection completeness, +10) it is now
|
||||
**348 pass, 54 fail.**
|
||||
The 54 remaining failing cases are vendored under
|
||||
`Tests/Fixtures/yaml-test-suite/` and recorded in
|
||||
`Tests/Fixtures/yaml-test-suite-gaps.yaml`. Run the opt-in sweep with:
|
||||
|
||||
@ -45,8 +46,8 @@ updated, the deliberate forcing function for keeping this document honest.
|
||||
|
||||
| Category | Count | Meaning |
|
||||
|---|---|---|
|
||||
| `false-reject` | 18 | Well-formed YAML that PureYAML wrongly rejects at parse time. |
|
||||
| `json-mismatch` | 10 | Parses, but produces a value that disagrees with the expected JSON. |
|
||||
| `false-reject` | 9 | Well-formed YAML that PureYAML wrongly rejects at parse time. |
|
||||
| `json-mismatch` | 9 | Parses, but produces a value that disagrees with the expected JSON. |
|
||||
| `accepted-invalid` | 36 | Malformed YAML that PureYAML wrongly accepts. |
|
||||
|
||||
## Root-cause buckets and fix order
|
||||
@ -295,6 +296,41 @@ must fold to newlines rather than survive as literal tabs).
|
||||
roll/unroll logic (`yaml_parser_roll_indent`/`yaml_parser_unroll_indent`), plus
|
||||
block-scalar indent detection in `scanner.c`.
|
||||
|
||||
### 8. Flow-collection completeness (DONE)
|
||||
|
||||
Flow mappings and sequences now accept keys of any node kind, empty keys and
|
||||
values, and multi-line scalar keys, matching libyaml's flow productions. The
|
||||
scanner implements libyaml's simple-key mechanism for flow context: it records a
|
||||
potential key position before scanning a plain/quoted scalar, an anchor/tag, or a
|
||||
`[`/`{` opener (when a simple key is allowed there), and a later `:` value
|
||||
indicator splices a `mappingKey` token in front of that node. In flow context a
|
||||
`:` is always a value indicator and a `?` always an explicit key indicator
|
||||
regardless of the following character (libyaml's `flow_level || IS_BLANKZ_AT`),
|
||||
so adjacent forms (`"k":v`, `{k}:v`) and a trailing `?` parse; the block-only
|
||||
implicit-key forward lookahead no longer fires in flow. A multi-line plain key
|
||||
folds in full before its `:` is read. The flow parser was rebuilt around
|
||||
`flow_sequence_entry`/`flow_mapping_key`/`flow_mapping_value`, composing empty
|
||||
scalars where a key or value is absent. The streaming token cursor keeps an open
|
||||
flow collection's tokens buffered until it closes so the retroactive key splice
|
||||
stays valid.
|
||||
|
||||
Fixed false-rejects (now parse, values match where an oracle exists): `4FJ6`,
|
||||
`8KB6`, `9BXH`, `9MMW`, `CN3R`, `DFF7`, `FRK4`, `NJ66`, `UT92`. Corrected the
|
||||
`json-mismatch` `CT4Q` (spec example 7.20, single-pair explicit entry) to a
|
||||
matching value. Headline 338 -> 348 pass; categories drop to false-reject 9,
|
||||
json-mismatch 9, accepted-invalid 36.
|
||||
|
||||
Still tracked as `false-reject`: `LX3P` (`[flow]: block`) and `6BFJ`
|
||||
(`&key [ ... ]: value`) use a flow collection as a *block* mapping key, which
|
||||
needs the simple-key splice in block context, a separate change with its own
|
||||
regression surface. The flow-internal `7ZZ5` mismatch is a block-sequence
|
||||
nesting issue (`- - - []`), not flow, and remains `json-mismatch`.
|
||||
|
||||
- **Reference:** `yaml_parser_save_simple_key`, `yaml_parser_fetch_value`
|
||||
(KEY-token splice), `yaml_parser_fetch_key`, the flow-collection
|
||||
start/end and flow-entry fetches in `scanner.c`, and the
|
||||
`yaml_parser_parse_flow_*` productions in `parser.c`.
|
||||
|
||||
## Updating this document
|
||||
|
||||
When a bucket lands:
|
||||
|
||||
Loading…
Reference in New Issue
Block a user