diff --git a/CHANGELOG.md b/CHANGELOG.md index a6ba592..3d33963 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Fixed +- Complete flow-collection (`[...]`, `{...}`) parsing for keys, empty nodes, and + multi-line scalars, fixing 10 official YAML test suite cases and removing them + from the conformance gap manifest. The scanner now implements libyaml's + simple-key mechanism inside flow context: a potential key position is recorded + before a plain or quoted scalar, an anchor or tag, or a nested `[`/`{` opener, + and a later `:` value indicator splices a `mappingKey` token in front of that + node so a collection or property-carrying node can be a flow-mapping key. In + flow context a `:` is now always a value indicator and a `?` always an explicit + key indicator regardless of the following character (matching libyaml's + `flow_level || IS_BLANKZ_AT` checks), so `"k":v`, `{k}:v`, and a trailing `?` + parse correctly; the implicit-key forward lookahead is restricted to block + context. A multi-line plain key folds in full before its `:` is read. The flow + parser was rebuilt around libyaml's flow productions (`flow_sequence_entry`, + `flow_mapping_key`/`value`), composing empty scalar keys and values where + absent. The streaming token cursor holds an open flow collection's tokens until + it closes so the retroactive key splice stays valid. Fixed false-rejects 4FJ6, + 8KB6, 9BXH, 9MMW, CN3R, DFF7, FRK4, NJ66, UT92 and corrected values for CT4Q. + The flow-node-as-block-mapping-key cases LX3P and 6BFJ remain tracked (they need + the block-context simple-key splice, a separate change). + - Parse multi-line plain scalars and block scalars at the document root, fixing 35 official YAML test suite cases and removing them from the conformance gap manifest. The scanner now tracks the enclosing block-collection indent diff --git a/Sources/Parsing/Scanner.FlowContext.swift b/Sources/Parsing/Scanner.FlowContext.swift index 7f22627..d86bb80 100644 --- a/Sources/Parsing/Scanner.FlowContext.swift +++ b/Sources/Parsing/Scanner.FlowContext.swift @@ -8,4 +8,15 @@ extension PureYAML.Parsing { var kind: ScannerFlowContextKind var isExpectingKey: Bool } + + /// A position that may be promoted to a flow-mapping key, mirroring libyaml's + /// `yaml_simple_key_t`. When a `:` value indicator is later found, the scanner + /// inserts a ``TokenKind/mappingKey`` token at ``tokenIndex`` so a node of any + /// kind (plain or quoted scalar, or a nested `[...]`/`{...}` collection) that + /// precedes the `:` becomes that pair's key. Only used inside flow collections; + /// block context promotes keys with a forward lookahead instead. + struct ScannerFlowSimpleKey { + var tokenIndex: Int + var mark: PureYAML.Parsing.Mark + } } diff --git a/Sources/Parsing/Scanner.FlowSimpleKey.swift b/Sources/Parsing/Scanner.FlowSimpleKey.swift new file mode 100644 index 0000000..a15e99b --- /dev/null +++ b/Sources/Parsing/Scanner.FlowSimpleKey.swift @@ -0,0 +1,105 @@ +extension PureYAML.Parsing.Scanner { + func scanFlowToken( + _ character: Character, + state: inout State, + ) -> Bool { + switch character { + case "[": + openFlowCollection(kind: .sequence, token: .flowSequenceStart, state: &state) + return true + case "]": + closeFlowCollection(token: .flowSequenceEnd, state: &state) + return true + case "{": + openFlowCollection(kind: .mapping, token: .flowMappingStart, state: &state) + return true + case "}": + closeFlowCollection(token: .flowMappingEnd, state: &state) + return true + case ",": + // A flow entry separator abandons any pending simple key and reopens the + // door for a fresh one on the next entry (libyaml fetch_flow_entry). + removeFlowSimpleKey(&state) + scanFixedToken(.flowEntry, state: &state) + state.simpleKeyAllowed = true + if state.flowContexts.last?.kind == .mapping { + state.flowContexts[state.flowContexts.count - 1].isExpectingKey = true + } + return true + default: + return false + } + } + + /// Opens a flow collection (`[` or `{`). The opener itself may be a simple key + /// at the enclosing level, so a key is saved before the flow level rises; a + /// fresh simple-key slot is then pushed and a key is allowed inside. + func openFlowCollection( + kind: PureYAML.Parsing.ScannerFlowContextKind, + token: PureYAML.Parsing.TokenKind, + state: inout State, + ) { + saveFlowSimpleKey(&state) + state.flowDepth += 1 + state.flowContexts.append(.init(kind: kind, isExpectingKey: kind == .mapping)) + state.flowSimpleKeys.append(nil) + scanFixedToken(token, state: &state) + state.simpleKeyAllowed = true + } + + /// Closes a flow collection (`]` or `}`): the pending simple key on this level is + /// dropped, the level is popped, and no simple key may immediately follow the + /// closing indicator (libyaml fetch_flow_collection_end). + func closeFlowCollection( + token: PureYAML.Parsing.TokenKind, + state: inout State, + ) { + removeFlowSimpleKey(&state) + state.flowDepth = max(0, state.flowDepth - 1) + scanFixedToken(token, state: &state) + _ = state.flowContexts.popLast() + _ = state.flowSimpleKeys.popLast() + state.simpleKeyAllowed = false + } + + /// Records a potential simple key at the current reader position for the + /// innermost flow level, if a simple key is allowed there. Mirrors libyaml's + /// `yaml_parser_save_simple_key`: the saved key remembers the index of the + /// next token to be appended, so a later `:` can splice a ``mappingKey`` in + /// front of the node that starts here. No-op outside flow context. + func saveFlowSimpleKey(_ state: inout State) { + guard state.flowDepth > 0, state.simpleKeyAllowed, !state.flowSimpleKeys.isEmpty else { + return + } + state.flowSimpleKeys[state.flowSimpleKeys.count - 1] = PureYAML.Parsing.ScannerFlowSimpleKey( + tokenIndex: state.tokens.count, + mark: state.reader.mark, + ) + } + + /// Discards any potential simple key on the innermost flow level. Mirrors + /// libyaml's `yaml_parser_remove_simple_key` for the flow context, where a + /// simple key is never *required*, so abandoning it is not an error. + func removeFlowSimpleKey(_ state: inout State) { + guard !state.flowSimpleKeys.isEmpty else { + return + } + state.flowSimpleKeys[state.flowSimpleKeys.count - 1] = nil + } + + /// Promotes a pending flow simple key to an actual mapping key by inserting a + /// ``TokenKind/mappingKey`` token at the saved index, mirroring libyaml's + /// `fetch_value` splicing a KEY token before a simple key. Returns true when a + /// key was inserted so the caller can avoid emitting a redundant one. + @discardableResult + func promoteFlowSimpleKey(_ state: inout State) -> Bool { + guard !state.flowSimpleKeys.isEmpty, let key = state.flowSimpleKeys[state.flowSimpleKeys.count - 1] else { + return false + } + let token = PureYAML.Parsing.Token(kind: .mappingKey, mark: key.mark, endMark: key.mark) + let index = min(key.tokenIndex, state.tokens.count) + state.tokens.insert(token, at: index) + state.flowSimpleKeys[state.flowSimpleKeys.count - 1] = nil + return true + } +} diff --git a/Sources/Parsing/Scanner.Helpers.swift b/Sources/Parsing/Scanner.Helpers.swift index 52006c6..0aedd30 100644 --- a/Sources/Parsing/Scanner.Helpers.swift +++ b/Sources/Parsing/Scanner.Helpers.swift @@ -163,7 +163,11 @@ extension PureYAML.Parsing.Scanner { var continuationState = state continuationState.reader = probe let continuation = scanPlainScalarContinuationLine(&continuationState) - guard !isAtMappingValueIndicator(state: continuationState) else { + // In block context a continuation line that ends at a `:` is the next + // mapping key, not part of this scalar, so stop folding. In flow context + // a multi-line plain key folds in full and the trailing `:` then makes + // the folded text the key (libyaml folds first, promotes the key after). + guard state.flowDepth > 0 || !isAtMappingValueIndicator(state: continuationState) else { return value } @@ -315,6 +319,12 @@ extension PureYAML.Parsing.Scanner { start: PureYAML.Parsing.Mark, state: inout State, ) -> Bool { + // Flow-context keys are promoted by the simple-key mechanism (a `:` value + // indicator splices the `mappingKey` in retroactively), so the block-only + // forward lookahead must not also fire here. + guard state.flowDepth == 0 else { + return false + } guard isAtMappingValueIndicator(state: state) || isAtSpacedMappingValueIndicator(state: state) else { return false } diff --git a/Sources/Parsing/Scanner.PropertyTokens.swift b/Sources/Parsing/Scanner.PropertyTokens.swift index 7f635ab..e1b6ce7 100644 --- a/Sources/Parsing/Scanner.PropertyTokens.swift +++ b/Sources/Parsing/Scanner.PropertyTokens.swift @@ -5,6 +5,11 @@ extension PureYAML.Parsing.Scanner { state: inout State, ) throws { let start = state.reader.mark + // An anchor or alias may itself begin a flow simple key, so record the + // position before the token is appended; the following node then does not + // start a new key, leaving `&anchor key :` anchored at the `&` (libyaml's + // fetch_anchor saves the simple key before clearing simple_key_allowed). + saveFlowSimpleKey(&state) state.reader.advance() // An anchor or alias name is a run of name characters that ends at the // first non-name character, mirroring `yaml_parser_scan_anchor`: the @@ -22,6 +27,8 @@ extension PureYAML.Parsing.Scanner { func scanTag(_ state: inout State) throws { let start = state.reader.mark + // A tag may itself begin a flow simple key (see `scanNamedToken`). + saveFlowSimpleKey(&state) state.reader.advance() var value = "!" if state.reader.peek() == "<" { diff --git a/Sources/Parsing/Scanner.swift b/Sources/Parsing/Scanner.swift index 4bea3cd..adba375 100644 --- a/Sources/Parsing/Scanner.swift +++ b/Sources/Parsing/Scanner.swift @@ -81,6 +81,11 @@ extension PureYAML.Parsing.Scanner { /// positions provides block indentation and is rejected. See scanner.c:1948. var simpleKeyAllowed = true var flowContexts: [PureYAML.Parsing.ScannerFlowContext] = [] + /// A stack of potential simple keys, one entry per open flow collection, + /// mirroring libyaml's `simple_keys`. The slot for the innermost flow level + /// records the most recent position a `:` could promote to a mapping key. + /// Empty outside flow context; block keys do not use this stack. + var flowSimpleKeys: [PureYAML.Parsing.ScannerFlowSimpleKey?] = [] var validImplicitIndentation: Set = [] var tagHandles = [ "!": "!", @@ -162,52 +167,32 @@ extension PureYAML.Parsing.Scanner { state.simpleKeyAllowed = true } return true - case "?" where isIndicatorBoundary(state.reader.peek(offset: 1)): + case "?" where state.flowDepth > 0 || isIndicatorBoundary(state.reader.peek(offset: 1)): + // In flow context a `?` is always an explicit-key indicator (libyaml's + // fetch_more_tokens checks `flow_level || IS_BLANKZ_AT`); a pending + // simple key is abandoned and no implicit key may follow on this entry. + if state.flowDepth > 0 { + removeFlowSimpleKey(&state) + } scanFixedToken(.mappingKey, state: &state) if state.flowDepth == 0 { state.simpleKeyAllowed = true + } else { + state.simpleKeyAllowed = false } return true - case ":" where isAtMappingValueIndicator(state: state): + case ":" where state.flowDepth > 0 || isAtMappingValueIndicator(state: state): + // In flow context a `:` is always a value indicator regardless of what + // follows it (libyaml), so `"k":v` and `{k}:v` are key/value pairs. A + // pending simple key is spliced into a `mappingKey` before its node. + if state.flowDepth > 0 { + promoteFlowSimpleKey(&state) + } scanFixedToken(.mappingValue, state: &state) if state.flowDepth == 0 { state.simpleKeyAllowed = true - } - return true - default: - return false - } - } - - func scanFlowToken( - _ character: Character, - state: inout State, - ) -> Bool { - switch character { - case "[": - state.flowDepth += 1 - state.flowContexts.append(.init(kind: .sequence, isExpectingKey: false)) - scanFixedToken(.flowSequenceStart, state: &state) - return true - case "]": - state.flowDepth = max(0, state.flowDepth - 1) - scanFixedToken(.flowSequenceEnd, state: &state) - _ = state.flowContexts.popLast() - return true - case "{": - state.flowDepth += 1 - state.flowContexts.append(.init(kind: .mapping, isExpectingKey: true)) - scanFixedToken(.flowMappingStart, state: &state) - return true - case "}": - state.flowDepth = max(0, state.flowDepth - 1) - scanFixedToken(.flowMappingEnd, state: &state) - _ = state.flowContexts.popLast() - return true - case ",": - scanFixedToken(.flowEntry, state: &state) - if state.flowContexts.last?.kind == .mapping { - state.flowContexts[state.flowContexts.count - 1].isExpectingKey = true + } else { + state.simpleKeyAllowed = false } return true default: @@ -314,6 +299,9 @@ extension PureYAML.Parsing.Scanner { state: inout State, ) throws { let start = state.reader.mark + // A quoted scalar may be a flow simple key: record its position before the + // content is consumed so a following `:` can splice a key in front of it. + saveFlowSimpleKey(&state) let quote: Character = style == .doubleQuoted ? "\"" : "'" state.reader.advance() var value = "" @@ -352,6 +340,9 @@ extension PureYAML.Parsing.Scanner { func scanPlainScalar(_ state: inout State) { let start = state.reader.mark + // A plain scalar may be a flow simple key: record its position before the + // content is consumed so a following `:` can splice a key in front of it. + saveFlowSimpleKey(&state) var value = "" while let character = state.reader.peek() { if shouldEndPlainScalar(character, state: state, accumulated: value) { diff --git a/Sources/Parsing/TokenCursor.swift b/Sources/Parsing/TokenCursor.swift index c83ac7d..6e8cb5f 100644 --- a/Sources/Parsing/TokenCursor.swift +++ b/Sources/Parsing/TokenCursor.swift @@ -77,7 +77,12 @@ extension PureYAML.Parsing { return } - while scannerBox.state.tokens.isEmpty { + // Scan at least one token, and keep scanning without draining while a + // flow collection is open: a flow `:` may splice a `mappingKey` into an + // already-scanned earlier token (the simple-key mechanism), so the whole + // flow collection's tokens must stay in `state.tokens` until it closes + // for the retroactive index to remain valid. + while scannerBox.state.tokens.isEmpty || scannerBox.state.flowDepth > 0 { guard !scannerBox.state.reader.isAtEnd else { if scannerBox.state.hasPendingDirective { throw PureYAML.Parsing.ParseError.directiveWithoutDocument( diff --git a/Sources/Parsing/TokenEventParser.Flow.swift b/Sources/Parsing/TokenEventParser.Flow.swift index e888a1d..ad60110 100644 --- a/Sources/Parsing/TokenEventParser.Flow.swift +++ b/Sources/Parsing/TokenEventParser.Flow.swift @@ -1,4 +1,8 @@ extension PureYAML.Parsing.TokenEventParser { + /// Parses a flow sequence, mirroring libyaml's `flow_sequence_entry` + /// productions. Each entry is either a flow node or, when introduced by a + /// `mappingKey` (`?`) or recognized as a single-pair `key: value`, a one-entry + /// flow mapping nested directly in the sequence. mutating func parseFlowSequence( properties: PureYAML.Parsing.NodeProperties, ) throws -> PureYAML.Parsing.NodeResult { @@ -32,7 +36,7 @@ extension PureYAML.Parsing.TokenEventParser { _ = try cursor.advance() continue } - endMark = try parseNode().endMark + endMark = try parseFlowSequenceEntry().endMark if cursor.current?.kind.isFlowTerminator == false { throw unexpectedToken(expected: "flow entry or flow sequence end") } @@ -40,6 +44,28 @@ extension PureYAML.Parsing.TokenEventParser { throw unexpectedToken(expected: "flow sequence end") } + /// Parses a single flow-sequence entry. A `mappingKey` (`?`, or one spliced in + /// by the simple-key mechanism before a `key: value` pair) produces a + /// single-pair flow mapping wrapping the entry; otherwise the entry is a plain + /// flow node. + mutating func parseFlowSequenceEntry() throws -> PureYAML.Parsing.NodeResult { + if cursor.current?.kind.isMappingKey == true { + return try parseFlowSequenceSinglePair() + } + return try parseNode() + } + + /// Wraps an explicit-`?` single-pair entry of a flow sequence in a one-entry + /// flow mapping, mirroring libyaml's `flow_sequence_entry_mapping_*` states. + mutating func parseFlowSequenceSinglePair() throws -> PureYAML.Parsing.NodeResult { + let keyIndicator = try expect("mapping key") { $0.isMappingKey } + try emit(.mappingStart(anchor: nil, tag: nil, style: .flow, mark: keyIndicator.mark)) + try parseFlowSinglePairKeyValue(keyIndicator: keyIndicator) + let mark = cursor.current?.mark ?? keyIndicator.endMark + try emit(.mappingEnd(mark: mark)) + return PureYAML.Parsing.NodeResult(kind: .mapping, endMark: mark) + } + mutating func parseFlowMapping( properties: PureYAML.Parsing.NodeProperties, ) throws -> PureYAML.Parsing.NodeResult { @@ -81,37 +107,38 @@ extension PureYAML.Parsing.TokenEventParser { throw unexpectedToken(expected: "flow mapping end") } + /// Parses one flow-mapping entry. With a `mappingKey` (`?`) indicator the key + /// follows the explicit form; without one the entry is a `flow_node` that is + /// itself the key (carrying an empty value unless a `:` provides one). mutating func parseFlowMappingPair() throws -> PureYAML.Parsing.NodeResult { if cursor.current?.kind.isMappingKey == true { - return try parseMappingPair() + let keyIndicator = try expect("mapping key") { $0.isMappingKey } + return try parseFlowSinglePairKeyValue(keyIndicator: keyIndicator) } + try parseFlowMappingImplicitKey() + return try parseFlowMappingValue() + } - try parseFlowMappingPairKey() - - guard cursor.current?.kind.isMappingValue == true else { - return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: cursor.previous?.endMark ?? .start) + /// Parses the key and value of a flow pair introduced by an explicit `?`, + /// where each side may be absent (composing an empty scalar). Shared by flow + /// mappings and single-pair entries nested in a flow sequence. + @discardableResult + mutating func parseFlowSinglePairKeyValue( + keyIndicator: PureYAML.Parsing.Token, + ) throws -> PureYAML.Parsing.NodeResult { + if flowKeyIsEmpty() { + try emitEmptyScalar(properties: .none, mark: keyIndicator.endMark) + } else { + _ = try parseNode() } - let valueIndicator = try expect("mapping value") { kind in - if case .mappingValue = kind { - return true - } - return false - } - - let properties = try consumeProperties() - guard properties != .none || canBeginNode(cursor.current), cursor.current?.kind.isFlowTerminator == false else { - let mark = valueIndicator.endMark - try emitEmptyScalar(properties: properties, mark: mark) - return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: mark) - } - return try parseNode(properties: properties) + return try parseFlowMappingValue() } /// Parses the key of an implicit flow-mapping entry (`key : value` with no /// explicit `?`). The key may be absent entirely (`{ : v }`, `{ , : v }`) /// or be only a tag/anchor with no content, both composing to an empty /// scalar key rather than routing into a block mapping. - mutating func parseFlowMappingPairKey() throws { + mutating func parseFlowMappingImplicitKey() throws { let properties = try consumeProperties() if properties == .none, cursor.current?.kind.isMappingValue == true { let mark = cursor.current?.mark ?? cursor.previous?.endMark ?? .start @@ -132,4 +159,38 @@ extension PureYAML.Parsing.TokenEventParser { } _ = try parseNode(properties: properties) } + + /// Parses the value half of a flow-mapping entry. A `:` consumes its node (or an + /// empty scalar when the value is absent); without a `:` the entry has an + /// implicit empty value (libyaml's `flow_mapping_value` with `empty == 1`). + mutating func parseFlowMappingValue() throws -> PureYAML.Parsing.NodeResult { + guard cursor.current?.kind.isMappingValue == true else { + let mark = cursor.current?.mark ?? cursor.previous?.endMark ?? .start + try emitEmptyScalar(properties: .none, mark: mark) + return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: mark) + } + let valueIndicator = try expect("mapping value") { kind in + if case .mappingValue = kind { + return true + } + return false + } + let properties = try consumeProperties() + guard properties != .none || canBeginNode(cursor.current), cursor.current?.kind.isFlowTerminator == false else { + let mark = valueIndicator.endMark + try emitEmptyScalar(properties: properties, mark: mark) + return PureYAML.Parsing.NodeResult(kind: .scalar, endMark: mark) + } + return try parseNode(properties: properties) + } + + /// True when the key position right after a `?` indicator is empty, i.e. the + /// next token is a `:` value indicator, a flow terminator, or the end of input, + /// so an empty scalar key must be composed (libyaml's `flow_mapping_key`). + func flowKeyIsEmpty() -> Bool { + guard let current = cursor.current else { + return true + } + return current.kind.isMappingValue || current.kind.isFlowTerminator + } } diff --git a/Tests/Fixtures/yaml-test-suite-gaps.yaml b/Tests/Fixtures/yaml-test-suite-gaps.yaml index 3dca8de..492a6ad 100644 --- a/Tests/Fixtures/yaml-test-suite-gaps.yaml +++ b/Tests/Fixtures/yaml-test-suite-gaps.yaml @@ -46,15 +46,6 @@ cases: current: accepted byteCount: 27 lineCount: 3 - - id: "4FJ6" - name: "Nested implicit complex keys" - localPath: yaml-test-suite/4FJ6/in.yaml - category: false-reject - expected: parse-ok - oracle: none - current: parse-error - byteCount: 37 - lineCount: 4 - id: "4JVG" name: "Scalar value with two anchors" localPath: yaml-test-suite/4JVG/in.yaml @@ -127,15 +118,6 @@ cases: current: value-mismatch byteCount: 58 lineCount: 6 - - id: "8KB6" - name: "Multiline plain flow mapping key without value" - localPath: yaml-test-suite/8KB6/in.yaml - category: false-reject - expected: parse-ok - oracle: json - current: parse-error - byteCount: 51 - lineCount: 4 - id: "8XDJ" name: "Comment in plain multiline value" localPath: yaml-test-suite/8XDJ/in.yaml @@ -145,15 +127,6 @@ cases: current: accepted byteCount: 26 lineCount: 3 - - id: "9BXH" - name: "Multiline doublequoted flow mapping key without value" - localPath: yaml-test-suite/9BXH/in.yaml - category: false-reject - expected: parse-ok - oracle: json - current: parse-error - byteCount: 55 - lineCount: 4 - id: "9C9N" name: "Wrong indented flow sequence" localPath: yaml-test-suite/9C9N/in.yaml @@ -181,15 +154,6 @@ cases: current: accepted byteCount: 18 lineCount: 2 - - id: "9MMW" - name: "Single Pair Implicit Entries" - localPath: yaml-test-suite/9MMW/in.yaml - category: false-reject - expected: parse-ok - oracle: none - current: parse-error - byteCount: 77 - lineCount: 3 - id: "BF9H" name: "Trailing comment in multiline plain scalar" localPath: yaml-test-suite/BF9H/in.yaml @@ -199,24 +163,6 @@ cases: current: accepted byteCount: 47 lineCount: 4 - - id: "CN3R" - name: "Various location of anchors in flow sequence" - localPath: yaml-test-suite/CN3R/in.yaml - category: false-reject - expected: parse-ok - oracle: json - current: parse-error - byteCount: 57 - lineCount: 6 - - id: "CT4Q" - name: "Spec Example 7.20. Single Pair Explicit Entry" - localPath: yaml-test-suite/CT4Q/in.yaml - category: json-mismatch - expected: parse-ok - oracle: json - current: value-mismatch - byteCount: 21 - lineCount: 4 - id: "CTN5" name: "Flow sequence with invalid extra comma" localPath: yaml-test-suite/CTN5/in.yaml @@ -262,15 +208,6 @@ cases: current: value-mismatch byteCount: 26 lineCount: 4 - - id: "DFF7" - name: "Spec Example 7.16. Flow Mapping Entries" - localPath: yaml-test-suite/DFF7/in.yaml - category: false-reject - expected: parse-ok - oracle: none - current: parse-error - byteCount: 42 - lineCount: 5 - id: "DK95/01" name: "Tabs that look like indentation" localPath: yaml-test-suite/DK95/01/in.yaml @@ -289,15 +226,6 @@ cases: current: parse-error byteCount: 51 lineCount: 5 - - id: "FRK4" - name: "Spec Example 7.3. Completely Empty Flow Nodes" - localPath: yaml-test-suite/FRK4/in.yaml - category: false-reject - expected: parse-ok - oracle: none - current: parse-error - byteCount: 24 - lineCount: 4 - id: "G5U8" name: "Plain dashes in flow sequence" localPath: yaml-test-suite/G5U8/in.yaml @@ -406,15 +334,6 @@ cases: current: value-mismatch byteCount: 31 lineCount: 5 - - id: "NJ66" - name: "Multiline plain flow mapping key" - localPath: yaml-test-suite/NJ66/in.yaml - category: false-reject - expected: parse-ok - oracle: json - current: parse-error - byteCount: 53 - lineCount: 4 - id: "P2EQ" name: "Invalid sequene item on same line as previous item" localPath: yaml-test-suite/P2EQ/in.yaml @@ -505,15 +424,6 @@ cases: current: parse-error byteCount: 4 lineCount: 1 - - id: "UT92" - name: "Spec Example 9.4. Explicit Documents" - localPath: yaml-test-suite/UT92/in.yaml - category: false-reject - expected: parse-ok - oracle: json - current: parse-error - byteCount: 43 - lineCount: 7 - id: "UV7Q" name: "Legal tab after indentation" localPath: yaml-test-suite/UV7Q/in.yaml diff --git a/Tests/Fixtures/yaml-test-suite/4FJ6/in.yaml b/Tests/Fixtures/yaml-test-suite/4FJ6/in.yaml deleted file mode 100644 index 3f4aabe..0000000 --- a/Tests/Fixtures/yaml-test-suite/4FJ6/in.yaml +++ /dev/null @@ -1,4 +0,0 @@ ---- -[ - [ a, [ [[b,c]]: d, e]]: 23 -] diff --git a/Tests/Fixtures/yaml-test-suite/8KB6/in.json b/Tests/Fixtures/yaml-test-suite/8KB6/in.json deleted file mode 100644 index 2716276..0000000 --- a/Tests/Fixtures/yaml-test-suite/8KB6/in.json +++ /dev/null @@ -1,10 +0,0 @@ -[ - { - "single line": null, - "a": "b" - }, - { - "multi line": null, - "a": "b" - } -] diff --git a/Tests/Fixtures/yaml-test-suite/8KB6/in.yaml b/Tests/Fixtures/yaml-test-suite/8KB6/in.yaml deleted file mode 100644 index 9167fd6..0000000 --- a/Tests/Fixtures/yaml-test-suite/8KB6/in.yaml +++ /dev/null @@ -1,4 +0,0 @@ ---- -- { single line, a: b} -- { multi - line, a: b} diff --git a/Tests/Fixtures/yaml-test-suite/9BXH/in.json b/Tests/Fixtures/yaml-test-suite/9BXH/in.json deleted file mode 100644 index 2716276..0000000 --- a/Tests/Fixtures/yaml-test-suite/9BXH/in.json +++ /dev/null @@ -1,10 +0,0 @@ -[ - { - "single line": null, - "a": "b" - }, - { - "multi line": null, - "a": "b" - } -] diff --git a/Tests/Fixtures/yaml-test-suite/9BXH/in.yaml b/Tests/Fixtures/yaml-test-suite/9BXH/in.yaml deleted file mode 100644 index c007827..0000000 --- a/Tests/Fixtures/yaml-test-suite/9BXH/in.yaml +++ /dev/null @@ -1,4 +0,0 @@ ---- -- { "single line", a: b} -- { "multi - line", a: b} diff --git a/Tests/Fixtures/yaml-test-suite/9MMW/in.yaml b/Tests/Fixtures/yaml-test-suite/9MMW/in.yaml deleted file mode 100644 index b9fe5e1..0000000 --- a/Tests/Fixtures/yaml-test-suite/9MMW/in.yaml +++ /dev/null @@ -1,3 +0,0 @@ -- [ YAML : separate ] -- [ "JSON like":adjacent ] -- [ {JSON: like}:adjacent ] diff --git a/Tests/Fixtures/yaml-test-suite/CN3R/in.json b/Tests/Fixtures/yaml-test-suite/CN3R/in.json deleted file mode 100644 index 28eb29e..0000000 --- a/Tests/Fixtures/yaml-test-suite/CN3R/in.json +++ /dev/null @@ -1,14 +0,0 @@ -[ - { - "a": "b" - }, - { - "c": "d" - }, - { - "e": "f" - }, - { - "g": "h" - } -] diff --git a/Tests/Fixtures/yaml-test-suite/CN3R/in.yaml b/Tests/Fixtures/yaml-test-suite/CN3R/in.yaml deleted file mode 100644 index 8345dfc..0000000 --- a/Tests/Fixtures/yaml-test-suite/CN3R/in.yaml +++ /dev/null @@ -1,6 +0,0 @@ -&flowseq [ - a: b, - &c c: d, - { &e e: f }, - &g { g: h } -] diff --git a/Tests/Fixtures/yaml-test-suite/CT4Q/in.json b/Tests/Fixtures/yaml-test-suite/CT4Q/in.json deleted file mode 100644 index 6342cb2..0000000 --- a/Tests/Fixtures/yaml-test-suite/CT4Q/in.json +++ /dev/null @@ -1,5 +0,0 @@ -[ - { - "foo bar": "baz" - } -] diff --git a/Tests/Fixtures/yaml-test-suite/CT4Q/in.yaml b/Tests/Fixtures/yaml-test-suite/CT4Q/in.yaml deleted file mode 100644 index 19dc4f5..0000000 --- a/Tests/Fixtures/yaml-test-suite/CT4Q/in.yaml +++ /dev/null @@ -1,4 +0,0 @@ -[ -? foo - bar : baz -] diff --git a/Tests/Fixtures/yaml-test-suite/DFF7/in.yaml b/Tests/Fixtures/yaml-test-suite/DFF7/in.yaml deleted file mode 100644 index cb84a99..0000000 --- a/Tests/Fixtures/yaml-test-suite/DFF7/in.yaml +++ /dev/null @@ -1,5 +0,0 @@ -{ -? explicit: entry, -implicit: entry, -? -} diff --git a/Tests/Fixtures/yaml-test-suite/FRK4/in.yaml b/Tests/Fixtures/yaml-test-suite/FRK4/in.yaml deleted file mode 100644 index f46900d..0000000 --- a/Tests/Fixtures/yaml-test-suite/FRK4/in.yaml +++ /dev/null @@ -1,4 +0,0 @@ -{ - ? foo :, - : bar, -} diff --git a/Tests/Fixtures/yaml-test-suite/NJ66/in.json b/Tests/Fixtures/yaml-test-suite/NJ66/in.json deleted file mode 100644 index d214d88..0000000 --- a/Tests/Fixtures/yaml-test-suite/NJ66/in.json +++ /dev/null @@ -1,8 +0,0 @@ -[ - { - "single line": "value" - }, - { - "multi line": "value" - } -] diff --git a/Tests/Fixtures/yaml-test-suite/NJ66/in.yaml b/Tests/Fixtures/yaml-test-suite/NJ66/in.yaml deleted file mode 100644 index 6ded532..0000000 --- a/Tests/Fixtures/yaml-test-suite/NJ66/in.yaml +++ /dev/null @@ -1,4 +0,0 @@ ---- -- { single line: value} -- { multi - line: value} diff --git a/Tests/Fixtures/yaml-test-suite/UT92/in.json b/Tests/Fixtures/yaml-test-suite/UT92/in.json deleted file mode 100644 index 587bbf4..0000000 --- a/Tests/Fixtures/yaml-test-suite/UT92/in.json +++ /dev/null @@ -1,4 +0,0 @@ -{ - "matches %": 20 -} -null diff --git a/Tests/Fixtures/yaml-test-suite/UT92/in.yaml b/Tests/Fixtures/yaml-test-suite/UT92/in.yaml deleted file mode 100644 index bc363b1..0000000 --- a/Tests/Fixtures/yaml-test-suite/UT92/in.yaml +++ /dev/null @@ -1,7 +0,0 @@ ---- -{ matches -% : 20 } -... ---- -# Empty -... diff --git a/Tests/YAMLTestSuiteGapsTests.swift b/Tests/YAMLTestSuiteGapsTests.swift index bd0ed89..67b5606 100644 --- a/Tests/YAMLTestSuiteGapsTests.swift +++ b/Tests/YAMLTestSuiteGapsTests.swift @@ -18,9 +18,9 @@ struct YAMLTestSuiteGapsTests { let manifest = try YAMLTestSuiteManifest.load() #expect(manifest.version == 1) - #expect(manifest.cases.count == 64) - #expect(manifest.cases.count(where: { $0.category == "false-reject" }) == 18) - #expect(manifest.cases.count(where: { $0.category == "json-mismatch" }) == 10) + #expect(manifest.cases.count == 54) + #expect(manifest.cases.count(where: { $0.category == "false-reject" }) == 9) + #expect(manifest.cases.count(where: { $0.category == "json-mismatch" }) == 9) #expect(manifest.cases.count(where: { $0.category == "accepted-invalid" }) == 36) #expect(Set(manifest.cases.map(\.id)).count == manifest.cases.count) diff --git a/docs/YAML_TEST_SUITE_CONFORMANCE.md b/docs/YAML_TEST_SUITE_CONFORMANCE.md index 7ef599a..6a3a753 100644 --- a/docs/YAML_TEST_SUITE_CONFORMANCE.md +++ b/docs/YAML_TEST_SUITE_CONFORMANCE.md @@ -24,10 +24,11 @@ its own oracle: fail; after bucket 1 (block scalars, +24), bucket 2 (quoted-scalar folding, +30), bucket 3 (empty nodes and empty keys, +20), bucket 4 (anchors, tags, and aliases as mapping keys, +10), bucket 5 (directives and document boundaries, -+15), bucket 6 (stricter rejection of invalid input, +15), and bucket 7 -(document-root multi-line plain and block scalars, +33) it is now -**338 pass, 64 fail.** -The 64 remaining failing cases are vendored under ++15), bucket 6 (stricter rejection of invalid input, +15), bucket 7 +(document-root multi-line plain and block scalars, +33), and bucket 8 +(flow-collection completeness, +10) it is now +**348 pass, 54 fail.** +The 54 remaining failing cases are vendored under `Tests/Fixtures/yaml-test-suite/` and recorded in `Tests/Fixtures/yaml-test-suite-gaps.yaml`. Run the opt-in sweep with: @@ -45,8 +46,8 @@ updated, the deliberate forcing function for keeping this document honest. | Category | Count | Meaning | |---|---|---| -| `false-reject` | 18 | Well-formed YAML that PureYAML wrongly rejects at parse time. | -| `json-mismatch` | 10 | Parses, but produces a value that disagrees with the expected JSON. | +| `false-reject` | 9 | Well-formed YAML that PureYAML wrongly rejects at parse time. | +| `json-mismatch` | 9 | Parses, but produces a value that disagrees with the expected JSON. | | `accepted-invalid` | 36 | Malformed YAML that PureYAML wrongly accepts. | ## Root-cause buckets and fix order @@ -295,6 +296,41 @@ must fold to newlines rather than survive as literal tabs). roll/unroll logic (`yaml_parser_roll_indent`/`yaml_parser_unroll_indent`), plus block-scalar indent detection in `scanner.c`. +### 8. Flow-collection completeness (DONE) + +Flow mappings and sequences now accept keys of any node kind, empty keys and +values, and multi-line scalar keys, matching libyaml's flow productions. The +scanner implements libyaml's simple-key mechanism for flow context: it records a +potential key position before scanning a plain/quoted scalar, an anchor/tag, or a +`[`/`{` opener (when a simple key is allowed there), and a later `:` value +indicator splices a `mappingKey` token in front of that node. In flow context a +`:` is always a value indicator and a `?` always an explicit key indicator +regardless of the following character (libyaml's `flow_level || IS_BLANKZ_AT`), +so adjacent forms (`"k":v`, `{k}:v`) and a trailing `?` parse; the block-only +implicit-key forward lookahead no longer fires in flow. A multi-line plain key +folds in full before its `:` is read. The flow parser was rebuilt around +`flow_sequence_entry`/`flow_mapping_key`/`flow_mapping_value`, composing empty +scalars where a key or value is absent. The streaming token cursor keeps an open +flow collection's tokens buffered until it closes so the retroactive key splice +stays valid. + +Fixed false-rejects (now parse, values match where an oracle exists): `4FJ6`, +`8KB6`, `9BXH`, `9MMW`, `CN3R`, `DFF7`, `FRK4`, `NJ66`, `UT92`. Corrected the +`json-mismatch` `CT4Q` (spec example 7.20, single-pair explicit entry) to a +matching value. Headline 338 -> 348 pass; categories drop to false-reject 9, +json-mismatch 9, accepted-invalid 36. + +Still tracked as `false-reject`: `LX3P` (`[flow]: block`) and `6BFJ` +(`&key [ ... ]: value`) use a flow collection as a *block* mapping key, which +needs the simple-key splice in block context, a separate change with its own +regression surface. The flow-internal `7ZZ5` mismatch is a block-sequence +nesting issue (`- - - []`), not flow, and remains `json-mismatch`. + +- **Reference:** `yaml_parser_save_simple_key`, `yaml_parser_fetch_value` + (KEY-token splice), `yaml_parser_fetch_key`, the flow-collection + start/end and flow-entry fetches in `scanner.c`, and the + `yaml_parser_parse_flow_*` productions in `parser.c`. + ## Updating this document When a bucket lands: